All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-07  2:11 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

In version 3:

Changes suggested by Rob Herring <robh@kernel.org>:

  - prepended FIM node properties with vendor prefix "fsl,".

  - make mipi csi-2 receiver compatible string SoC specific:
    "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".

  - redundant "_clk" removed from mipi csi-2 receiver clock-names property.

  - removed board-specific info from the media driver binding doc. These
    were all related to sensor bindings, which already are (adv7180)
    or will be (ov564x) covered in separate binding docs. All reference
    board info not related to DT bindings has been moved to
    Documentation/media/v4l-drivers/imx.rst.

  - removed "_mipi" from the OV5640 compatible string.

Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:

  Mostly cosmetic/non-functional changes which I won't list here, except
  for the following:

  - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.

  - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.

  - check/handle return code from required reg property of CSI port nodes.

  - check/handle return code from clk_prepare_enable().

Changes suggested by Fabio Estevam <festevam@gmail.com>:

  - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.

  - finally got around to passing valid IOMUX pin config values to the
    pin groups.

Other changes:

  - removed the FIM properties that overrided the v4l2 FIM control defaults
    values. This was left-over from a requirement of a customer and is not
    necessary here.

  - The FIM must be explicitly enabled in the fim child node under the CSI
    port nodes, using the status property. If not enabled, FIM v4l2 controls
    will not appear in the video capture driver.

  - brought in additional media types patch from Philipp Zabel. Use new
    MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.

  - brought in latest platform generic video multiplexer subdevice driver
    from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).

  - removed imx-media-of.h, moved those prototypes into imx-media.h.


Philipp Zabel (3):
  ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
    connections
  add mux and video interface bridge entity functions
  platform: add video-multiplexer subdevice driver

Steve Longerbeam (21):
  [media] dt-bindings: Add bindings for i.MX media driver
  ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
  ARM: dts: imx6qdl: add media device
  ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
  ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
  ARM: dts: imx6-sabreauto: create i2cmux for i2c3
  ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
  ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
  ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
  UAPI: Add media UAPI Kbuild file
  media: Add userspace header file for i.MX
  media: Add i.MX media core driver
  media: imx: Add CSI subdev driver
  media: imx: Add SMFC subdev driver
  media: imx: Add IC subdev drivers
  media: imx: Add Camera Interface subdev driver
  media: imx: Add MIPI CSI-2 Receiver subdev driver
  media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  media: imx: Add Parallel OV5642 sensor subdev driver
  ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

 Documentation/devicetree/bindings/media/imx.txt    |   57 +
 .../bindings/media/video-multiplexer.txt           |   59 +
 Documentation/media/uapi/mediactl/media-types.rst  |   22 +
 Documentation/media/v4l-drivers/imx.rst            |  443 ++
 arch/arm/boot/dts/imx6dl-sabrelite.dts             |    5 +
 arch/arm/boot/dts/imx6dl-sabresd.dts               |    5 +
 arch/arm/boot/dts/imx6dl.dtsi                      |  187 +
 arch/arm/boot/dts/imx6q-sabrelite.dts              |    6 +
 arch/arm/boot/dts/imx6q-sabresd.dts                |    5 +
 arch/arm/boot/dts/imx6q.dtsi                       |  127 +
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi           |  147 +-
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi           |  122 +-
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi             |  114 +-
 arch/arm/boot/dts/imx6qdl.dtsi                     |   25 +-
 arch/arm/configs/imx_v6_v7_defconfig               |   12 +-
 drivers/media/platform/Kconfig                     |    8 +
 drivers/media/platform/Makefile                    |    2 +
 drivers/media/platform/video-multiplexer.c         |  472 +++
 drivers/staging/media/Kconfig                      |    2 +
 drivers/staging/media/Makefile                     |    1 +
 drivers/staging/media/imx/Kconfig                  |   36 +
 drivers/staging/media/imx/Makefile                 |   15 +
 drivers/staging/media/imx/TODO                     |   22 +
 drivers/staging/media/imx/imx-camif.c              | 1000 +++++
 drivers/staging/media/imx/imx-csi.c                |  644 +++
 drivers/staging/media/imx/imx-ic-common.c          |  109 +
 drivers/staging/media/imx/imx-ic-pp.c              |  636 +++
 drivers/staging/media/imx/imx-ic-prpenc.c          | 1033 +++++
 drivers/staging/media/imx/imx-ic-prpvf.c           | 1179 ++++++
 drivers/staging/media/imx/imx-ic.h                 |   38 +
 drivers/staging/media/imx/imx-media-common.c       |  981 +++++
 drivers/staging/media/imx/imx-media-dev.c          |  486 +++
 drivers/staging/media/imx/imx-media-fim.c          |  471 +++
 drivers/staging/media/imx/imx-media-internal-sd.c  |  457 ++
 drivers/staging/media/imx/imx-media-of.c           |  289 ++
 drivers/staging/media/imx/imx-media.h              |  310 ++
 drivers/staging/media/imx/imx-mipi-csi2.c          |  501 +++
 drivers/staging/media/imx/imx-smfc.c               |  737 ++++
 drivers/staging/media/imx/ov5640-mipi.c            | 2348 +++++++++++
 drivers/staging/media/imx/ov5642.c                 | 4363 ++++++++++++++++++++
 include/media/imx.h                                |   15 +
 include/uapi/Kbuild                                |    1 +
 include/uapi/linux/media.h                         |    6 +
 include/uapi/linux/v4l2-controls.h                 |    4 +
 include/uapi/media/Kbuild                          |    2 +
 include/uapi/media/imx.h                           |   30 +
 46 files changed, 17505 insertions(+), 29 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 Documentation/media/v4l-drivers/imx.rst
 create mode 100644 drivers/media/platform/video-multiplexer.c
 create mode 100644 drivers/staging/media/imx/Kconfig
 create mode 100644 drivers/staging/media/imx/Makefile
 create mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 drivers/staging/media/imx/imx-camif.c
 create mode 100644 drivers/staging/media/imx/imx-csi.c
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h
 create mode 100644 drivers/staging/media/imx/imx-media-common.c
 create mode 100644 drivers/staging/media/imx/imx-media-dev.c
 create mode 100644 drivers/staging/media/imx/imx-media-fim.c
 create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
 create mode 100644 drivers/staging/media/imx/imx-media-of.c
 create mode 100644 drivers/staging/media/imx/imx-media.h
 create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
 create mode 100644 drivers/staging/media/imx/imx-smfc.c
 create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
 create mode 100644 drivers/staging/media/imx/ov5642.c
 create mode 100644 include/media/imx.h
 create mode 100644 include/uapi/media/Kbuild
 create mode 100644 include/uapi/media/imx.h

-- 
2.7.4

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-07  2:11 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

In version 3:

Changes suggested by Rob Herring <robh@kernel.org>:

  - prepended FIM node properties with vendor prefix "fsl,".

  - make mipi csi-2 receiver compatible string SoC specific:
    "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".

  - redundant "_clk" removed from mipi csi-2 receiver clock-names property.

  - removed board-specific info from the media driver binding doc. These
    were all related to sensor bindings, which already are (adv7180)
    or will be (ov564x) covered in separate binding docs. All reference
    board info not related to DT bindings has been moved to
    Documentation/media/v4l-drivers/imx.rst.

  - removed "_mipi" from the OV5640 compatible string.

Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:

  Mostly cosmetic/non-functional changes which I won't list here, except
  for the following:

  - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.

  - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.

  - check/handle return code from required reg property of CSI port nodes.

  - check/handle return code from clk_prepare_enable().

Changes suggested by Fabio Estevam <festevam@gmail.com>:

  - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.

  - finally got around to passing valid IOMUX pin config values to the
    pin groups.

Other changes:

  - removed the FIM properties that overrided the v4l2 FIM control defaults
    values. This was left-over from a requirement of a customer and is not
    necessary here.

  - The FIM must be explicitly enabled in the fim child node under the CSI
    port nodes, using the status property. If not enabled, FIM v4l2 controls
    will not appear in the video capture driver.

  - brought in additional media types patch from Philipp Zabel. Use new
    MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.

  - brought in latest platform generic video multiplexer subdevice driver
    from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).

  - removed imx-media-of.h, moved those prototypes into imx-media.h.


Philipp Zabel (3):
  ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
    connections
  add mux and video interface bridge entity functions
  platform: add video-multiplexer subdevice driver

Steve Longerbeam (21):
  [media] dt-bindings: Add bindings for i.MX media driver
  ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
  ARM: dts: imx6qdl: add media device
  ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
  ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
  ARM: dts: imx6-sabreauto: create i2cmux for i2c3
  ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
  ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
  ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
  UAPI: Add media UAPI Kbuild file
  media: Add userspace header file for i.MX
  media: Add i.MX media core driver
  media: imx: Add CSI subdev driver
  media: imx: Add SMFC subdev driver
  media: imx: Add IC subdev drivers
  media: imx: Add Camera Interface subdev driver
  media: imx: Add MIPI CSI-2 Receiver subdev driver
  media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  media: imx: Add Parallel OV5642 sensor subdev driver
  ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

 Documentation/devicetree/bindings/media/imx.txt    |   57 +
 .../bindings/media/video-multiplexer.txt           |   59 +
 Documentation/media/uapi/mediactl/media-types.rst  |   22 +
 Documentation/media/v4l-drivers/imx.rst            |  443 ++
 arch/arm/boot/dts/imx6dl-sabrelite.dts             |    5 +
 arch/arm/boot/dts/imx6dl-sabresd.dts               |    5 +
 arch/arm/boot/dts/imx6dl.dtsi                      |  187 +
 arch/arm/boot/dts/imx6q-sabrelite.dts              |    6 +
 arch/arm/boot/dts/imx6q-sabresd.dts                |    5 +
 arch/arm/boot/dts/imx6q.dtsi                       |  127 +
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi           |  147 +-
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi           |  122 +-
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi             |  114 +-
 arch/arm/boot/dts/imx6qdl.dtsi                     |   25 +-
 arch/arm/configs/imx_v6_v7_defconfig               |   12 +-
 drivers/media/platform/Kconfig                     |    8 +
 drivers/media/platform/Makefile                    |    2 +
 drivers/media/platform/video-multiplexer.c         |  472 +++
 drivers/staging/media/Kconfig                      |    2 +
 drivers/staging/media/Makefile                     |    1 +
 drivers/staging/media/imx/Kconfig                  |   36 +
 drivers/staging/media/imx/Makefile                 |   15 +
 drivers/staging/media/imx/TODO                     |   22 +
 drivers/staging/media/imx/imx-camif.c              | 1000 +++++
 drivers/staging/media/imx/imx-csi.c                |  644 +++
 drivers/staging/media/imx/imx-ic-common.c          |  109 +
 drivers/staging/media/imx/imx-ic-pp.c              |  636 +++
 drivers/staging/media/imx/imx-ic-prpenc.c          | 1033 +++++
 drivers/staging/media/imx/imx-ic-prpvf.c           | 1179 ++++++
 drivers/staging/media/imx/imx-ic.h                 |   38 +
 drivers/staging/media/imx/imx-media-common.c       |  981 +++++
 drivers/staging/media/imx/imx-media-dev.c          |  486 +++
 drivers/staging/media/imx/imx-media-fim.c          |  471 +++
 drivers/staging/media/imx/imx-media-internal-sd.c  |  457 ++
 drivers/staging/media/imx/imx-media-of.c           |  289 ++
 drivers/staging/media/imx/imx-media.h              |  310 ++
 drivers/staging/media/imx/imx-mipi-csi2.c          |  501 +++
 drivers/staging/media/imx/imx-smfc.c               |  737 ++++
 drivers/staging/media/imx/ov5640-mipi.c            | 2348 +++++++++++
 drivers/staging/media/imx/ov5642.c                 | 4363 ++++++++++++++++++++
 include/media/imx.h                                |   15 +
 include/uapi/Kbuild                                |    1 +
 include/uapi/linux/media.h                         |    6 +
 include/uapi/linux/v4l2-controls.h                 |    4 +
 include/uapi/media/Kbuild                          |    2 +
 include/uapi/media/imx.h                           |   30 +
 46 files changed, 17505 insertions(+), 29 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 Documentation/media/v4l-drivers/imx.rst
 create mode 100644 drivers/media/platform/video-multiplexer.c
 create mode 100644 drivers/staging/media/imx/Kconfig
 create mode 100644 drivers/staging/media/imx/Makefile
 create mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 drivers/staging/media/imx/imx-camif.c
 create mode 100644 drivers/staging/media/imx/imx-csi.c
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h
 create mode 100644 drivers/staging/media/imx/imx-media-common.c
 create mode 100644 drivers/staging/media/imx/imx-media-dev.c
 create mode 100644 drivers/staging/media/imx/imx-media-fim.c
 create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
 create mode 100644 drivers/staging/media/imx/imx-media-of.c
 create mode 100644 drivers/staging/media/imx/imx-media.h
 create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
 create mode 100644 drivers/staging/media/imx/imx-smfc.c
 create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
 create mode 100644 drivers/staging/media/imx/ov5642.c
 create mode 100644 include/media/imx.h
 create mode 100644 include/uapi/media/Kbuild
 create mode 100644 include/uapi/media/imx.h

-- 
2.7.4

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-07  2:11 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

In version 3:

Changes suggested by Rob Herring <robh@kernel.org>:

  - prepended FIM node properties with vendor prefix "fsl,".

  - make mipi csi-2 receiver compatible string SoC specific:
    "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".

  - redundant "_clk" removed from mipi csi-2 receiver clock-names property.

  - removed board-specific info from the media driver binding doc. These
    were all related to sensor bindings, which already are (adv7180)
    or will be (ov564x) covered in separate binding docs. All reference
    board info not related to DT bindings has been moved to
    Documentation/media/v4l-drivers/imx.rst.

  - removed "_mipi" from the OV5640 compatible string.

Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:

  Mostly cosmetic/non-functional changes which I won't list here, except
  for the following:

  - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.

  - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.

  - check/handle return code from required reg property of CSI port nodes.

  - check/handle return code from clk_prepare_enable().

Changes suggested by Fabio Estevam <festevam@gmail.com>:

  - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.

  - finally got around to passing valid IOMUX pin config values to the
    pin groups.

Other changes:

  - removed the FIM properties that overrided the v4l2 FIM control defaults
    values. This was left-over from a requirement of a customer and is not
    necessary here.

  - The FIM must be explicitly enabled in the fim child node under the CSI
    port nodes, using the status property. If not enabled, FIM v4l2 controls
    will not appear in the video capture driver.

  - brought in additional media types patch from Philipp Zabel. Use new
    MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.

  - brought in latest platform generic video multiplexer subdevice driver
    from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).

  - removed imx-media-of.h, moved those prototypes into imx-media.h.


Philipp Zabel (3):
  ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
    connections
  add mux and video interface bridge entity functions
  platform: add video-multiplexer subdevice driver

Steve Longerbeam (21):
  [media] dt-bindings: Add bindings for i.MX media driver
  ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
  ARM: dts: imx6qdl: add media device
  ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
  ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
  ARM: dts: imx6-sabreauto: create i2cmux for i2c3
  ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
  ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
  ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
  UAPI: Add media UAPI Kbuild file
  media: Add userspace header file for i.MX
  media: Add i.MX media core driver
  media: imx: Add CSI subdev driver
  media: imx: Add SMFC subdev driver
  media: imx: Add IC subdev drivers
  media: imx: Add Camera Interface subdev driver
  media: imx: Add MIPI CSI-2 Receiver subdev driver
  media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  media: imx: Add Parallel OV5642 sensor subdev driver
  ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

 Documentation/devicetree/bindings/media/imx.txt    |   57 +
 .../bindings/media/video-multiplexer.txt           |   59 +
 Documentation/media/uapi/mediactl/media-types.rst  |   22 +
 Documentation/media/v4l-drivers/imx.rst            |  443 ++
 arch/arm/boot/dts/imx6dl-sabrelite.dts             |    5 +
 arch/arm/boot/dts/imx6dl-sabresd.dts               |    5 +
 arch/arm/boot/dts/imx6dl.dtsi                      |  187 +
 arch/arm/boot/dts/imx6q-sabrelite.dts              |    6 +
 arch/arm/boot/dts/imx6q-sabresd.dts                |    5 +
 arch/arm/boot/dts/imx6q.dtsi                       |  127 +
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi           |  147 +-
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi           |  122 +-
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi             |  114 +-
 arch/arm/boot/dts/imx6qdl.dtsi                     |   25 +-
 arch/arm/configs/imx_v6_v7_defconfig               |   12 +-
 drivers/media/platform/Kconfig                     |    8 +
 drivers/media/platform/Makefile                    |    2 +
 drivers/media/platform/video-multiplexer.c         |  472 +++
 drivers/staging/media/Kconfig                      |    2 +
 drivers/staging/media/Makefile                     |    1 +
 drivers/staging/media/imx/Kconfig                  |   36 +
 drivers/staging/media/imx/Makefile                 |   15 +
 drivers/staging/media/imx/TODO                     |   22 +
 drivers/staging/media/imx/imx-camif.c              | 1000 +++++
 drivers/staging/media/imx/imx-csi.c                |  644 +++
 drivers/staging/media/imx/imx-ic-common.c          |  109 +
 drivers/staging/media/imx/imx-ic-pp.c              |  636 +++
 drivers/staging/media/imx/imx-ic-prpenc.c          | 1033 +++++
 drivers/staging/media/imx/imx-ic-prpvf.c           | 1179 ++++++
 drivers/staging/media/imx/imx-ic.h                 |   38 +
 drivers/staging/media/imx/imx-media-common.c       |  981 +++++
 drivers/staging/media/imx/imx-media-dev.c          |  486 +++
 drivers/staging/media/imx/imx-media-fim.c          |  471 +++
 drivers/staging/media/imx/imx-media-internal-sd.c  |  457 ++
 drivers/staging/media/imx/imx-media-of.c           |  289 ++
 drivers/staging/media/imx/imx-media.h              |  310 ++
 drivers/staging/media/imx/imx-mipi-csi2.c          |  501 +++
 drivers/staging/media/imx/imx-smfc.c               |  737 ++++
 drivers/staging/media/imx/ov5640-mipi.c            | 2348 +++++++++++
 drivers/staging/media/imx/ov5642.c                 | 4363 ++++++++++++++++++++
 include/media/imx.h                                |   15 +
 include/uapi/Kbuild                                |    1 +
 include/uapi/linux/media.h                         |    6 +
 include/uapi/linux/v4l2-controls.h                 |    4 +
 include/uapi/media/Kbuild                          |    2 +
 include/uapi/media/imx.h                           |   30 +
 46 files changed, 17505 insertions(+), 29 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 Documentation/media/v4l-drivers/imx.rst
 create mode 100644 drivers/media/platform/video-multiplexer.c
 create mode 100644 drivers/staging/media/imx/Kconfig
 create mode 100644 drivers/staging/media/imx/Makefile
 create mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 drivers/staging/media/imx/imx-camif.c
 create mode 100644 drivers/staging/media/imx/imx-csi.c
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h
 create mode 100644 drivers/staging/media/imx/imx-media-common.c
 create mode 100644 drivers/staging/media/imx/imx-media-dev.c
 create mode 100644 drivers/staging/media/imx/imx-media-fim.c
 create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
 create mode 100644 drivers/staging/media/imx/imx-media-of.c
 create mode 100644 drivers/staging/media/imx/imx-media.h
 create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
 create mode 100644 drivers/staging/media/imx/imx-smfc.c
 create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
 create mode 100644 drivers/staging/media/imx/ov5642.c
 create mode 100644 include/media/imx.h
 create mode 100644 include/uapi/media/Kbuild
 create mode 100644 include/uapi/media/imx.h

-- 
2.7.4

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
  2017-01-07  2:11 ` Steve Longerbeam
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Add bindings documentation for the i.MX media driver.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt

diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
new file mode 100644
index 0000000..254b64a
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/imx.txt
@@ -0,0 +1,57 @@
+Freescale i.MX Media Video Devices
+
+Video Media Controller node
+---------------------------
+
+This is the parent media controller node for video capture support.
+
+Required properties:
+- compatible : "fsl,imx-media";
+- ports      : Should contain a list of phandles pointing to camera
+  	       sensor interface ports of IPU devices
+
+
+fim child node
+--------------
+
+This is an optional child node of the ipu_csi port nodes. If present and
+available, it enables the Frame Interval Monitor. Its properties can be
+used to modify the method in which the FIM measures frame intervals.
+Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
+Frame Interval Monitor.
+
+Optional properties:
+- fsl,input-capture-channel: an input capture channel and channel flags,
+			     specified as <chan flags>. The channel number
+			     must be 0 or 1. The flags can be
+			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
+			     IRQ_TYPE_EDGE_BOTH, and specify which input
+			     capture signal edge will trigger the input
+			     capture event. If an input capture channel is
+			     specified, the FIM will use this method to
+			     measure frame intervals instead of via the EOF
+			     interrupt. The input capture method is much
+			     preferred over EOF as it is not subject to
+			     interrupt latency errors. However it requires
+			     routing the VSYNC or FIELD output signals of
+			     the camera sensor to one of the i.MX input
+			     capture pads (SD1_DAT0, SD1_DAT1), which also
+			     gives up support for SD1.
+
+
+mipi_csi2 node
+--------------
+
+This is the device node for the MIPI CSI-2 Receiver, required for MIPI
+CSI-2 sensors.
+
+Required properties:
+- compatible	: "fsl,imx6-mipi-csi2";
+- reg           : physical base address and length of the register set;
+- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
+                  (the DPHY clock), video_27m, and eim_sel;
+- clock-names	: must contain "dphy", "cfg", "pix";
+
+Optional properties:
+- interrupts	: must contain two level-triggered interrupts,
+                  in order: 100 and 101;
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Add bindings documentation for the i.MX media driver.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt

diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
new file mode 100644
index 0000000..254b64a
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/imx.txt
@@ -0,0 +1,57 @@
+Freescale i.MX Media Video Devices
+
+Video Media Controller node
+---------------------------
+
+This is the parent media controller node for video capture support.
+
+Required properties:
+- compatible : "fsl,imx-media";
+- ports      : Should contain a list of phandles pointing to camera
+  	       sensor interface ports of IPU devices
+
+
+fim child node
+--------------
+
+This is an optional child node of the ipu_csi port nodes. If present and
+available, it enables the Frame Interval Monitor. Its properties can be
+used to modify the method in which the FIM measures frame intervals.
+Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
+Frame Interval Monitor.
+
+Optional properties:
+- fsl,input-capture-channel: an input capture channel and channel flags,
+			     specified as <chan flags>. The channel number
+			     must be 0 or 1. The flags can be
+			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
+			     IRQ_TYPE_EDGE_BOTH, and specify which input
+			     capture signal edge will trigger the input
+			     capture event. If an input capture channel is
+			     specified, the FIM will use this method to
+			     measure frame intervals instead of via the EOF
+			     interrupt. The input capture method is much
+			     preferred over EOF as it is not subject to
+			     interrupt latency errors. However it requires
+			     routing the VSYNC or FIELD output signals of
+			     the camera sensor to one of the i.MX input
+			     capture pads (SD1_DAT0, SD1_DAT1), which also
+			     gives up support for SD1.
+
+
+mipi_csi2 node
+--------------
+
+This is the device node for the MIPI CSI-2 Receiver, required for MIPI
+CSI-2 sensors.
+
+Required properties:
+- compatible	: "fsl,imx6-mipi-csi2";
+- reg           : physical base address and length of the register set;
+- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
+                  (the DPHY clock), video_27m, and eim_sel;
+- clock-names	: must contain "dphy", "cfg", "pix";
+
+Optional properties:
+- interrupts	: must contain two level-triggered interrupts,
+                  in order: 100 and 101;
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
clocks.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 53e6e63..42926e9 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1125,7 +1125,14 @@
 			};
 
 			mipi_csi: mipi@021dc000 {
+				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				interrupts = <0 100 0x04>, <0 101 0x04>;
+				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+					 <&clks IMX6QDL_CLK_VIDEO_27M>,
+					 <&clks IMX6QDL_CLK_EIM_SEL>;
+				clock-names = "dphy", "cfg", "pix";
+				status = "disabled";
 			};
 
 			mipi_dsi: mipi@021e0000 {
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
clocks.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 53e6e63..42926e9 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1125,7 +1125,14 @@
 			};
 
 			mipi_csi: mipi@021dc000 {
+				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				interrupts = <0 100 0x04>, <0 101 0x04>;
+				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+					 <&clks IMX6QDL_CLK_VIDEO_27M>,
+					 <&clks IMX6QDL_CLK_EIM_SEL>;
+				clock-names = "dphy", "cfg", "pix";
+				status = "disabled";
 			};
 
 			mipi_dsi: mipi@021e0000 {
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
clocks.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 53e6e63..42926e9 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1125,7 +1125,14 @@
 			};
 
 			mipi_csi: mipi at 021dc000 {
+				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				interrupts = <0 100 0x04>, <0 101 0x04>;
+				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+					 <&clks IMX6QDL_CLK_VIDEO_27M>,
+					 <&clks IMX6QDL_CLK_EIM_SEL>;
+				clock-names = "dphy", "cfg", "pix";
+				status = "disabled";
 			};
 
 			mipi_dsi: mipi at 021e0000 {
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 03/24] ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their connections
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

From: Philipp Zabel <p.zabel@pengutronix.de>

This patch adds the device tree graph connecting the input multiplexers
to the IPU CSIs and the MIPI-CSI2 gasket on i.MX6. The MIPI_IPU
multiplexers are added as children of the iomuxc-gpr syscon device node.
On i.MX6Q/D two two-input multiplexers in front of IPU1 CSI0 and IPU2
CSI1 allow to select between CSI0/1 parallel input pads and the MIPI
CSI-2 virtual channels 0/3.
On i.MX6DL/S two five-input multiplexers in front of IPU1 CSI0 and IPU1
CSI1 allow to select between CSI0/1 parallel input pads and any of the
four MIPI CSI-2 virtual channels.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>

--

- Removed some dangling/unused endpoints (ipu2_csi0_from_csi2ipu)
- Renamed the mipi virtual channel endpoint labels, from "mipi_csiX_..."
  to "mipi_vcX...".
- Added input endpoint anchors to the video muxes for the connections
  from parallel sensors.
- Added input endpoint anchors to the mipi_csi for the connections from
  mipi csi-2 sensors.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl.dtsi  | 187 +++++++++++++++++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6q.dtsi   | 123 +++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6qdl.dtsi |  10 ++-
 3 files changed, 319 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi
index 1ade195..0849e85 100644
--- a/arch/arm/boot/dts/imx6dl.dtsi
+++ b/arch/arm/boot/dts/imx6dl.dtsi
@@ -181,6 +181,193 @@
 		      "di0", "di1";
 };
 
+&gpr {
+	ipu1_csi0_mux: ipu1_csi0_mux@34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <0>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu1_csi0_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu1_csi0_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu1_csi0_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@3 {
+			reg = <3>;
+
+			ipu1_csi0_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@4 {
+			reg = <4>;
+
+			ipu1_csi0_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@5 {
+			reg = <5>;
+
+			ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+			};
+		};
+	};
+
+	ipu1_csi1_mux: ipu1_csi1_mux@34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu1_csi1_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu1_csi1_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu1_csi1_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@3 {
+			reg = <3>;
+
+			ipu1_csi1_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@4 {
+			reg = <4>;
+
+			ipu1_csi1_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@5 {
+			reg = <5>;
+
+			ipu1_csi1_mux_to_ipu1_csi1: endpoint {
+				remote-endpoint = <&ipu1_csi1_from_ipu1_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_ipu1_csi1_mux: endpoint {
+		remote-endpoint = <&ipu1_csi1_mux_to_ipu1_csi1>;
+	};
+};
+
+&mipi_csi {
+	port@0 {
+		reg = <0>;
+
+		mipi_csi_from_mipi_sensor: endpoint {
+		};
+	};
+
+	port@1 {
+		reg = <1>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc0_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+		};
+
+		mipi_vc0_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc0>;
+		};
+	};
+
+	port@2 {
+		reg = <2>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc1_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc1>;
+		};
+
+		mipi_vc1_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc1>;
+		};
+	};
+
+	port@3 {
+		reg = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc2_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc2>;
+		};
+
+		mipi_vc2_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc2>;
+		};
+	};
+
+	port@4 {
+		reg = <4>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc3_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc3>;
+		};
+
+		mipi_vc3_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc3>;
+		};
+	};
+};
+
 &vpu {
 	compatible = "fsl,imx6dl-vpu", "cnm,coda960";
 };
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index e9a5d0b..9b2ca32 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -143,10 +143,18 @@
 
 			ipu2_csi0: port@0 {
 				reg = <0>;
+
+				ipu2_csi0_from_mipi_vc2: endpoint {
+					remote-endpoint = <&mipi_vc2_to_ipu2_csi0>;
+				};
 			};
 
 			ipu2_csi1: port@1 {
 				reg = <1>;
+
+				ipu2_csi1_from_ipu2_csi1_mux: endpoint {
+					remote-endpoint = <&ipu2_csi1_mux_to_ipu2_csi1>;
+				};
 			};
 
 			ipu2_di0: port@2 {
@@ -266,6 +274,80 @@
 	};
 };
 
+&gpr {
+	ipu1_csi0_mux: ipu1_csi0_mux@4 {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu1_csi0_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu1_csi0_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+			};
+		};
+	};
+
+	ipu2_csi1_mux: ipu2_csi1_mux@4 {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <20>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu2_csi1_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu2_csi1_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu2_csi1_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu2_csi1_mux_to_ipu2_csi1: endpoint {
+				remote-endpoint = <&ipu2_csi1_from_ipu2_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_mipi_vc1: endpoint {
+		remote-endpoint = <&mipi_vc1_to_ipu1_csi1>;
+	};
+};
+
 &ldb {
 	clocks = <&clks IMX6QDL_CLK_LDB_DI0_SEL>, <&clks IMX6QDL_CLK_LDB_DI1_SEL>,
 		 <&clks IMX6QDL_CLK_IPU1_DI0_SEL>, <&clks IMX6QDL_CLK_IPU1_DI1_SEL>,
@@ -312,6 +394,47 @@
 	};
 };
 
+&mipi_csi {
+	port@0 {
+		reg = <0>;
+
+		mipi_csi_from_mipi_sensor: endpoint {
+		};
+	};
+
+	port@1 {
+		reg = <1>;
+
+		mipi_vc0_to_ipu1_csi0_mux: endpoint {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+		};
+	};
+
+	port@2 {
+		reg = <2>;
+
+		mipi_vc1_to_ipu1_csi1: endpoint {
+			remote-endpoint = <&ipu1_csi1_from_mipi_vc1>;
+		};
+	};
+
+	port@3 {
+		reg = <3>;
+
+		mipi_vc2_to_ipu2_csi0: endpoint {
+			remote-endpoint = <&ipu2_csi0_from_mipi_vc2>;
+		};
+	};
+
+	port@4 {
+		reg = <4>;
+
+		mipi_vc3_to_ipu2_csi1_mux: endpoint {
+			remote-endpoint = <&ipu2_csi1_mux_from_mipi_vc3>;
+		};
+	};
+};
+
 &mipi_dsi {
 	ports {
 		port@2 {
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 42926e9..010388c 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -799,8 +799,10 @@
 			};
 
 			gpr: iomuxc-gpr@020e0000 {
-				compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+				compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
 				reg = <0x020e0000 0x38>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 			};
 
 			iomuxc: iomuxc@020e0000 {
@@ -1127,6 +1129,8 @@
 			mipi_csi: mipi@021dc000 {
 				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 				interrupts = <0 100 0x04>, <0 101 0x04>;
 				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
 					 <&clks IMX6QDL_CLK_VIDEO_27M>,
@@ -1232,6 +1236,10 @@
 
 			ipu1_csi0: port@0 {
 				reg = <0>;
+
+				ipu1_csi0_from_ipu1_csi0_mux: endpoint {
+					remote-endpoint = <&ipu1_csi0_mux_to_ipu1_csi0>;
+				};
 			};
 
 			ipu1_csi1: port@1 {
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 03/24] ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their connections
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

From: Philipp Zabel <p.zabel@pengutronix.de>

This patch adds the device tree graph connecting the input multiplexers
to the IPU CSIs and the MIPI-CSI2 gasket on i.MX6. The MIPI_IPU
multiplexers are added as children of the iomuxc-gpr syscon device node.
On i.MX6Q/D two two-input multiplexers in front of IPU1 CSI0 and IPU2
CSI1 allow to select between CSI0/1 parallel input pads and the MIPI
CSI-2 virtual channels 0/3.
On i.MX6DL/S two five-input multiplexers in front of IPU1 CSI0 and IPU1
CSI1 allow to select between CSI0/1 parallel input pads and any of the
four MIPI CSI-2 virtual channels.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>

--

- Removed some dangling/unused endpoints (ipu2_csi0_from_csi2ipu)
- Renamed the mipi virtual channel endpoint labels, from "mipi_csiX_..."
  to "mipi_vcX...".
- Added input endpoint anchors to the video muxes for the connections
  from parallel sensors.
- Added input endpoint anchors to the mipi_csi for the connections from
  mipi csi-2 sensors.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl.dtsi  | 187 +++++++++++++++++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6q.dtsi   | 123 +++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6qdl.dtsi |  10 ++-
 3 files changed, 319 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi
index 1ade195..0849e85 100644
--- a/arch/arm/boot/dts/imx6dl.dtsi
+++ b/arch/arm/boot/dts/imx6dl.dtsi
@@ -181,6 +181,193 @@
 		      "di0", "di1";
 };
 
+&gpr {
+	ipu1_csi0_mux: ipu1_csi0_mux@34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <0>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu1_csi0_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu1_csi0_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu1_csi0_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@3 {
+			reg = <3>;
+
+			ipu1_csi0_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@4 {
+			reg = <4>;
+
+			ipu1_csi0_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@5 {
+			reg = <5>;
+
+			ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+			};
+		};
+	};
+
+	ipu1_csi1_mux: ipu1_csi1_mux@34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu1_csi1_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu1_csi1_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu1_csi1_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@3 {
+			reg = <3>;
+
+			ipu1_csi1_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port@4 {
+			reg = <4>;
+
+			ipu1_csi1_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@5 {
+			reg = <5>;
+
+			ipu1_csi1_mux_to_ipu1_csi1: endpoint {
+				remote-endpoint = <&ipu1_csi1_from_ipu1_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_ipu1_csi1_mux: endpoint {
+		remote-endpoint = <&ipu1_csi1_mux_to_ipu1_csi1>;
+	};
+};
+
+&mipi_csi {
+	port@0 {
+		reg = <0>;
+
+		mipi_csi_from_mipi_sensor: endpoint {
+		};
+	};
+
+	port@1 {
+		reg = <1>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc0_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+		};
+
+		mipi_vc0_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc0>;
+		};
+	};
+
+	port@2 {
+		reg = <2>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc1_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc1>;
+		};
+
+		mipi_vc1_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc1>;
+		};
+	};
+
+	port@3 {
+		reg = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc2_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc2>;
+		};
+
+		mipi_vc2_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc2>;
+		};
+	};
+
+	port@4 {
+		reg = <4>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc3_to_ipu1_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc3>;
+		};
+
+		mipi_vc3_to_ipu1_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc3>;
+		};
+	};
+};
+
 &vpu {
 	compatible = "fsl,imx6dl-vpu", "cnm,coda960";
 };
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index e9a5d0b..9b2ca32 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -143,10 +143,18 @@
 
 			ipu2_csi0: port@0 {
 				reg = <0>;
+
+				ipu2_csi0_from_mipi_vc2: endpoint {
+					remote-endpoint = <&mipi_vc2_to_ipu2_csi0>;
+				};
 			};
 
 			ipu2_csi1: port@1 {
 				reg = <1>;
+
+				ipu2_csi1_from_ipu2_csi1_mux: endpoint {
+					remote-endpoint = <&ipu2_csi1_mux_to_ipu2_csi1>;
+				};
 			};
 
 			ipu2_di0: port@2 {
@@ -266,6 +274,80 @@
 	};
 };
 
+&gpr {
+	ipu1_csi0_mux: ipu1_csi0_mux@4 {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu1_csi0_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu1_csi0_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+			};
+		};
+	};
+
+	ipu2_csi1_mux: ipu2_csi1_mux@4 {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <20>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port@0 {
+			reg = <0>;
+
+			ipu2_csi1_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu2_csi1_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu2_csi1_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu2_csi1_mux_to_ipu2_csi1: endpoint {
+				remote-endpoint = <&ipu2_csi1_from_ipu2_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_mipi_vc1: endpoint {
+		remote-endpoint = <&mipi_vc1_to_ipu1_csi1>;
+	};
+};
+
 &ldb {
 	clocks = <&clks IMX6QDL_CLK_LDB_DI0_SEL>, <&clks IMX6QDL_CLK_LDB_DI1_SEL>,
 		 <&clks IMX6QDL_CLK_IPU1_DI0_SEL>, <&clks IMX6QDL_CLK_IPU1_DI1_SEL>,
@@ -312,6 +394,47 @@
 	};
 };
 
+&mipi_csi {
+	port@0 {
+		reg = <0>;
+
+		mipi_csi_from_mipi_sensor: endpoint {
+		};
+	};
+
+	port@1 {
+		reg = <1>;
+
+		mipi_vc0_to_ipu1_csi0_mux: endpoint {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+		};
+	};
+
+	port@2 {
+		reg = <2>;
+
+		mipi_vc1_to_ipu1_csi1: endpoint {
+			remote-endpoint = <&ipu1_csi1_from_mipi_vc1>;
+		};
+	};
+
+	port@3 {
+		reg = <3>;
+
+		mipi_vc2_to_ipu2_csi0: endpoint {
+			remote-endpoint = <&ipu2_csi0_from_mipi_vc2>;
+		};
+	};
+
+	port@4 {
+		reg = <4>;
+
+		mipi_vc3_to_ipu2_csi1_mux: endpoint {
+			remote-endpoint = <&ipu2_csi1_mux_from_mipi_vc3>;
+		};
+	};
+};
+
 &mipi_dsi {
 	ports {
 		port@2 {
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 42926e9..010388c 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -799,8 +799,10 @@
 			};
 
 			gpr: iomuxc-gpr@020e0000 {
-				compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+				compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
 				reg = <0x020e0000 0x38>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 			};
 
 			iomuxc: iomuxc@020e0000 {
@@ -1127,6 +1129,8 @@
 			mipi_csi: mipi@021dc000 {
 				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 				interrupts = <0 100 0x04>, <0 101 0x04>;
 				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
 					 <&clks IMX6QDL_CLK_VIDEO_27M>,
@@ -1232,6 +1236,10 @@
 
 			ipu1_csi0: port@0 {
 				reg = <0>;
+
+				ipu1_csi0_from_ipu1_csi0_mux: endpoint {
+					remote-endpoint = <&ipu1_csi0_mux_to_ipu1_csi0>;
+				};
 			};
 
 			ipu1_csi1: port@1 {
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 03/24] ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their connections
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

From: Philipp Zabel <p.zabel@pengutronix.de>

This patch adds the device tree graph connecting the input multiplexers
to the IPU CSIs and the MIPI-CSI2 gasket on i.MX6. The MIPI_IPU
multiplexers are added as children of the iomuxc-gpr syscon device node.
On i.MX6Q/D two two-input multiplexers in front of IPU1 CSI0 and IPU2
CSI1 allow to select between CSI0/1 parallel input pads and the MIPI
CSI-2 virtual channels 0/3.
On i.MX6DL/S two five-input multiplexers in front of IPU1 CSI0 and IPU1
CSI1 allow to select between CSI0/1 parallel input pads and any of the
four MIPI CSI-2 virtual channels.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>

--

- Removed some dangling/unused endpoints (ipu2_csi0_from_csi2ipu)
- Renamed the mipi virtual channel endpoint labels, from "mipi_csiX_..."
  to "mipi_vcX...".
- Added input endpoint anchors to the video muxes for the connections
  from parallel sensors.
- Added input endpoint anchors to the mipi_csi for the connections from
  mipi csi-2 sensors.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl.dtsi  | 187 +++++++++++++++++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6q.dtsi   | 123 +++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6qdl.dtsi |  10 ++-
 3 files changed, 319 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi
index 1ade195..0849e85 100644
--- a/arch/arm/boot/dts/imx6dl.dtsi
+++ b/arch/arm/boot/dts/imx6dl.dtsi
@@ -181,6 +181,193 @@
 		      "di0", "di1";
 };
 
+&gpr {
+	ipu1_csi0_mux: ipu1_csi0_mux at 34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <0>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port at 0 {
+			reg = <0>;
+
+			ipu1_csi0_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			ipu1_csi0_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			ipu1_csi0_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 3 {
+			reg = <3>;
+
+			ipu1_csi0_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 4 {
+			reg = <4>;
+
+			ipu1_csi0_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port at 5 {
+			reg = <5>;
+
+			ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+			};
+		};
+	};
+
+	ipu1_csi1_mux: ipu1_csi1_mux at 34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port at 0 {
+			reg = <0>;
+
+			ipu1_csi1_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			ipu1_csi1_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			ipu1_csi1_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 3 {
+			reg = <3>;
+
+			ipu1_csi1_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 4 {
+			reg = <4>;
+
+			ipu1_csi1_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port at 5 {
+			reg = <5>;
+
+			ipu1_csi1_mux_to_ipu1_csi1: endpoint {
+				remote-endpoint = <&ipu1_csi1_from_ipu1_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_ipu1_csi1_mux: endpoint {
+		remote-endpoint = <&ipu1_csi1_mux_to_ipu1_csi1>;
+	};
+};
+
+&mipi_csi {
+	port at 0 {
+		reg = <0>;
+
+		mipi_csi_from_mipi_sensor: endpoint {
+		};
+	};
+
+	port at 1 {
+		reg = <1>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc0_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+		};
+
+		mipi_vc0_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc0>;
+		};
+	};
+
+	port at 2 {
+		reg = <2>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc1_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc1>;
+		};
+
+		mipi_vc1_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc1>;
+		};
+	};
+
+	port at 3 {
+		reg = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc2_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc2>;
+		};
+
+		mipi_vc2_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc2>;
+		};
+	};
+
+	port at 4 {
+		reg = <4>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc3_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc3>;
+		};
+
+		mipi_vc3_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc3>;
+		};
+	};
+};
+
 &vpu {
 	compatible = "fsl,imx6dl-vpu", "cnm,coda960";
 };
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index e9a5d0b..9b2ca32 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -143,10 +143,18 @@
 
 			ipu2_csi0: port at 0 {
 				reg = <0>;
+
+				ipu2_csi0_from_mipi_vc2: endpoint {
+					remote-endpoint = <&mipi_vc2_to_ipu2_csi0>;
+				};
 			};
 
 			ipu2_csi1: port at 1 {
 				reg = <1>;
+
+				ipu2_csi1_from_ipu2_csi1_mux: endpoint {
+					remote-endpoint = <&ipu2_csi1_mux_to_ipu2_csi1>;
+				};
 			};
 
 			ipu2_di0: port at 2 {
@@ -266,6 +274,80 @@
 	};
 };
 
+&gpr {
+	ipu1_csi0_mux: ipu1_csi0_mux at 4 {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port at 0 {
+			reg = <0>;
+
+			ipu1_csi0_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			ipu1_csi0_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+			};
+		};
+	};
+
+	ipu2_csi1_mux: ipu2_csi1_mux at 4 {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <20>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port at 0 {
+			reg = <0>;
+
+			ipu2_csi1_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu2_csi1_mux>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			ipu2_csi1_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			ipu2_csi1_mux_to_ipu2_csi1: endpoint {
+				remote-endpoint = <&ipu2_csi1_from_ipu2_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_mipi_vc1: endpoint {
+		remote-endpoint = <&mipi_vc1_to_ipu1_csi1>;
+	};
+};
+
 &ldb {
 	clocks = <&clks IMX6QDL_CLK_LDB_DI0_SEL>, <&clks IMX6QDL_CLK_LDB_DI1_SEL>,
 		 <&clks IMX6QDL_CLK_IPU1_DI0_SEL>, <&clks IMX6QDL_CLK_IPU1_DI1_SEL>,
@@ -312,6 +394,47 @@
 	};
 };
 
+&mipi_csi {
+	port at 0 {
+		reg = <0>;
+
+		mipi_csi_from_mipi_sensor: endpoint {
+		};
+	};
+
+	port at 1 {
+		reg = <1>;
+
+		mipi_vc0_to_ipu1_csi0_mux: endpoint {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+		};
+	};
+
+	port at 2 {
+		reg = <2>;
+
+		mipi_vc1_to_ipu1_csi1: endpoint {
+			remote-endpoint = <&ipu1_csi1_from_mipi_vc1>;
+		};
+	};
+
+	port at 3 {
+		reg = <3>;
+
+		mipi_vc2_to_ipu2_csi0: endpoint {
+			remote-endpoint = <&ipu2_csi0_from_mipi_vc2>;
+		};
+	};
+
+	port at 4 {
+		reg = <4>;
+
+		mipi_vc3_to_ipu2_csi1_mux: endpoint {
+			remote-endpoint = <&ipu2_csi1_mux_from_mipi_vc3>;
+		};
+	};
+};
+
 &mipi_dsi {
 	ports {
 		port at 2 {
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 42926e9..010388c 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -799,8 +799,10 @@
 			};
 
 			gpr: iomuxc-gpr at 020e0000 {
-				compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+				compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
 				reg = <0x020e0000 0x38>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 			};
 
 			iomuxc: iomuxc at 020e0000 {
@@ -1127,6 +1129,8 @@
 			mipi_csi: mipi at 021dc000 {
 				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 				interrupts = <0 100 0x04>, <0 101 0x04>;
 				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
 					 <&clks IMX6QDL_CLK_VIDEO_27M>,
@@ -1232,6 +1236,10 @@
 
 			ipu1_csi0: port at 0 {
 				reg = <0>;
+
+				ipu1_csi0_from_ipu1_csi0_mux: endpoint {
+					remote-endpoint = <&ipu1_csi0_mux_to_ipu1_csi0>;
+				};
 			};
 
 			ipu1_csi1: port at 1 {
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 04/24] ARM: dts: imx6qdl: add media device
  2017-01-07  2:11 ` Steve Longerbeam
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6q.dtsi   | 4 ++++
 arch/arm/boot/dts/imx6qdl.dtsi | 8 ++++++++
 2 files changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index 9b2ca32..8867e78 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -458,3 +458,7 @@
 &vpu {
 	compatible = "fsl,imx6q-vpu", "cnm,coda960";
 };
+
+&media0 {
+	ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 010388c..cbb42ec 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1296,5 +1296,13 @@
 				};
 			};
 		};
+
+		media0: media@0 {
+			compatible = "fsl,imx-media";
+			ports = <&ipu1_csi0>, <&ipu1_csi1>;
+			#address-cells = <1>;
+			#size-cells = <0>;
+			status = "okay";
+		};
 	};
 };
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 04/24] ARM: dts: imx6qdl: add media device
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6q.dtsi   | 4 ++++
 arch/arm/boot/dts/imx6qdl.dtsi | 8 ++++++++
 2 files changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index 9b2ca32..8867e78 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -458,3 +458,7 @@
 &vpu {
 	compatible = "fsl,imx6q-vpu", "cnm,coda960";
 };
+
+&media0 {
+	ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 010388c..cbb42ec 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1296,5 +1296,13 @@
 				};
 			};
 		};
+
+		media0: media at 0 {
+			compatible = "fsl,imx-media";
+			ports = <&ipu1_csi0>, <&ipu1_csi1>;
+			#address-cells = <1>;
+			#size-cells = <0>;
+			status = "okay";
+		};
 	};
 };
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 05/24] ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
  2017-01-07  2:11 ` Steve Longerbeam
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

There is a pin conflict with GPIO_6. This pin functions as a power
input pin to the OV5642 camera sensor, but ENET uses it as the h/w
workaround for erratum ERR006687, to wake-up the ARM cores on normal
RX and TX packet done events. So we need to remove the h/w workaround
to support the OV5642. The result is that the CPUidle driver will no
longer allow entering the deep idle states on the sabrelite.

This is a partial revert of

commit 6261c4c8f13e ("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC
			interrupt.")
commit a28eeb43ee57 ("ARM: dts: imx6: tag boards that have the HW workaround
			for ERR006687")

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
index 1f9076e..795b5a5 100644
--- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
@@ -271,9 +271,6 @@
 	txd1-skew-ps = <0>;
 	txd2-skew-ps = <0>;
 	txd3-skew-ps = <0>;
-	interrupts-extended = <&gpio1 6 IRQ_TYPE_LEVEL_HIGH>,
-			      <&intc 0 119 IRQ_TYPE_LEVEL_HIGH>;
-	fsl,err006687-workaround-present;
 	status = "okay";
 };
 
@@ -374,7 +371,6 @@
 				MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL	0x1b030
 				/* Phy reset */
 				MX6QDL_PAD_EIM_D23__GPIO3_IO23		0x000b0
-				MX6QDL_PAD_GPIO_6__ENET_IRQ		0x000b1
 			>;
 		};
 
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 05/24] ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

There is a pin conflict with GPIO_6. This pin functions as a power
input pin to the OV5642 camera sensor, but ENET uses it as the h/w
workaround for erratum ERR006687, to wake-up the ARM cores on normal
RX and TX packet done events. So we need to remove the h/w workaround
to support the OV5642. The result is that the CPUidle driver will no
longer allow entering the deep idle states on the sabrelite.

This is a partial revert of

commit 6261c4c8f13e ("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC
			interrupt.")
commit a28eeb43ee57 ("ARM: dts: imx6: tag boards that have the HW workaround
			for ERR006687")

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
index 1f9076e..795b5a5 100644
--- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
@@ -271,9 +271,6 @@
 	txd1-skew-ps = <0>;
 	txd2-skew-ps = <0>;
 	txd3-skew-ps = <0>;
-	interrupts-extended = <&gpio1 6 IRQ_TYPE_LEVEL_HIGH>,
-			      <&intc 0 119 IRQ_TYPE_LEVEL_HIGH>;
-	fsl,err006687-workaround-present;
 	status = "okay";
 };
 
@@ -374,7 +371,6 @@
 				MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL	0x1b030
 				/* Phy reset */
 				MX6QDL_PAD_EIM_D23__GPIO3_IO23		0x000b0
-				MX6QDL_PAD_GPIO_6__ENET_IRQ		0x000b1
 			>;
 		};
 
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
Both hang off the same i2c2 bus, so they require different (and non-
default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
 3 files changed, 129 insertions(+)

diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
index 0f06ca5..fec2524 100644
--- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
@@ -48,3 +48,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Lite Board";
 	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
index 66d10d8..9e2d26d 100644
--- a/arch/arm/boot/dts/imx6q-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
@@ -52,3 +52,9 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
index 795b5a5..bca9fed 100644
--- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
@@ -39,6 +39,8 @@
  *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
  *     OTHER DEALINGS IN THE SOFTWARE.
  */
+
+#include <dt-bindings/clock/imx6qdl-clock.h>
 #include <dt-bindings/gpio/gpio.h>
 #include <dt-bindings/input/input.h>
 
@@ -96,6 +98,15 @@
 		};
 	};
 
+	mipi_xclk: mipi_xclk {
+		compatible = "pwm-clock";
+		#clock-cells = <0>;
+		clock-frequency = <22000000>;
+		clock-output-names = "mipi_pwm3";
+		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
+		status = "okay";
+	};
+
 	gpio-keys {
 		compatible = "gpio-keys";
 		pinctrl-names = "default";
@@ -220,6 +231,22 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
 &audmux {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_audmux>;
@@ -299,6 +326,52 @@
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_i2c2>;
 	status = "okay";
+
+	ov5640: camera@40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;
+		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint@1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;
+			};
+		};
+	};
+
+	ov5642: camera@42 {
+		compatible = "ovti,ov5642";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5642>;
+		clocks = <&clks IMX6QDL_CLK_CKO2>;
+		clock-names = "xclk";
+		reg = <0x42>;
+		xclk = <24000000>;
+		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
+		pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
+		gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
+
+		port {
+			ov5642_to_ipu1_csi0_mux: endpoint {
+				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+				bus-width = <8>;
+				hsync-active = <1>;
+				vsync-active = <1>;
+			};
+		};
+	};
 };
 
 &i2c3 {
@@ -412,6 +485,23 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
+				MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
+			>;
+		};
+
 		pinctrl_j15: j15grp {
 			fsl,pins = <
 				MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
@@ -445,6 +535,22 @@
 			>;
 		};
 
+		pinctrl_ov5640: ov5640grp {
+			fsl,pins = <
+				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x000b0
+				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
+			>;
+		};
+
+		pinctrl_ov5642: ov5642grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+				MX6QDL_PAD_GPIO_6__GPIO1_IO06   0x1b0b0
+				MX6QDL_PAD_GPIO_8__GPIO1_IO08   0x130b0
+				MX6QDL_PAD_GPIO_3__CCM_CLKO2    0x000b0
+			>;
+		};
+
 		pinctrl_pwm1: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
@@ -601,3 +707,15 @@
 	vmmc-supply = <&reg_3p3v>;
 	status = "okay";
 };
+
+&mipi_csi {
+        status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+        remote-endpoint = <&ov5640_to_mipi_csi>;
+        data-lanes = <0 1>;
+        clock-lanes = <2>;
+};
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
Both hang off the same i2c2 bus, so they require different (and non-
default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
 3 files changed, 129 insertions(+)

diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
index 0f06ca5..fec2524 100644
--- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
@@ -48,3 +48,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Lite Board";
 	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
index 66d10d8..9e2d26d 100644
--- a/arch/arm/boot/dts/imx6q-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
@@ -52,3 +52,9 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
index 795b5a5..bca9fed 100644
--- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
@@ -39,6 +39,8 @@
  *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
  *     OTHER DEALINGS IN THE SOFTWARE.
  */
+
+#include <dt-bindings/clock/imx6qdl-clock.h>
 #include <dt-bindings/gpio/gpio.h>
 #include <dt-bindings/input/input.h>
 
@@ -96,6 +98,15 @@
 		};
 	};
 
+	mipi_xclk: mipi_xclk {
+		compatible = "pwm-clock";
+		#clock-cells = <0>;
+		clock-frequency = <22000000>;
+		clock-output-names = "mipi_pwm3";
+		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
+		status = "okay";
+	};
+
 	gpio-keys {
 		compatible = "gpio-keys";
 		pinctrl-names = "default";
@@ -220,6 +231,22 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
 &audmux {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_audmux>;
@@ -299,6 +326,52 @@
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_i2c2>;
 	status = "okay";
+
+	ov5640: camera@40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;
+		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint@1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;
+			};
+		};
+	};
+
+	ov5642: camera@42 {
+		compatible = "ovti,ov5642";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5642>;
+		clocks = <&clks IMX6QDL_CLK_CKO2>;
+		clock-names = "xclk";
+		reg = <0x42>;
+		xclk = <24000000>;
+		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
+		pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
+		gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
+
+		port {
+			ov5642_to_ipu1_csi0_mux: endpoint {
+				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+				bus-width = <8>;
+				hsync-active = <1>;
+				vsync-active = <1>;
+			};
+		};
+	};
 };
 
 &i2c3 {
@@ -412,6 +485,23 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
+				MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
+			>;
+		};
+
 		pinctrl_j15: j15grp {
 			fsl,pins = <
 				MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
@@ -445,6 +535,22 @@
 			>;
 		};
 
+		pinctrl_ov5640: ov5640grp {
+			fsl,pins = <
+				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x000b0
+				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
+			>;
+		};
+
+		pinctrl_ov5642: ov5642grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+				MX6QDL_PAD_GPIO_6__GPIO1_IO06   0x1b0b0
+				MX6QDL_PAD_GPIO_8__GPIO1_IO08   0x130b0
+				MX6QDL_PAD_GPIO_3__CCM_CLKO2    0x000b0
+			>;
+		};
+
 		pinctrl_pwm1: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
@@ -601,3 +707,15 @@
 	vmmc-supply = <&reg_3p3v>;
 	status = "okay";
 };
+
+&mipi_csi {
+        status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+        remote-endpoint = <&ov5640_to_mipi_csi>;
+        data-lanes = <0 1>;
+        clock-lanes = <2>;
+};
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
Both hang off the same i2c2 bus, so they require different (and non-
default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
 3 files changed, 129 insertions(+)

diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
index 0f06ca5..fec2524 100644
--- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
@@ -48,3 +48,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Lite Board";
 	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
index 66d10d8..9e2d26d 100644
--- a/arch/arm/boot/dts/imx6q-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
@@ -52,3 +52,9 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
index 795b5a5..bca9fed 100644
--- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
@@ -39,6 +39,8 @@
  *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
  *     OTHER DEALINGS IN THE SOFTWARE.
  */
+
+#include <dt-bindings/clock/imx6qdl-clock.h>
 #include <dt-bindings/gpio/gpio.h>
 #include <dt-bindings/input/input.h>
 
@@ -96,6 +98,15 @@
 		};
 	};
 
+	mipi_xclk: mipi_xclk {
+		compatible = "pwm-clock";
+		#clock-cells = <0>;
+		clock-frequency = <22000000>;
+		clock-output-names = "mipi_pwm3";
+		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
+		status = "okay";
+	};
+
 	gpio-keys {
 		compatible = "gpio-keys";
 		pinctrl-names = "default";
@@ -220,6 +231,22 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
 &audmux {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_audmux>;
@@ -299,6 +326,52 @@
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_i2c2>;
 	status = "okay";
+
+	ov5640: camera at 40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;
+		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint at 1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;
+			};
+		};
+	};
+
+	ov5642: camera at 42 {
+		compatible = "ovti,ov5642";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5642>;
+		clocks = <&clks IMX6QDL_CLK_CKO2>;
+		clock-names = "xclk";
+		reg = <0x42>;
+		xclk = <24000000>;
+		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
+		pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
+		gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
+
+		port {
+			ov5642_to_ipu1_csi0_mux: endpoint {
+				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+				bus-width = <8>;
+				hsync-active = <1>;
+				vsync-active = <1>;
+			};
+		};
+	};
 };
 
 &i2c3 {
@@ -412,6 +485,23 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
+				MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
+			>;
+		};
+
 		pinctrl_j15: j15grp {
 			fsl,pins = <
 				MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
@@ -445,6 +535,22 @@
 			>;
 		};
 
+		pinctrl_ov5640: ov5640grp {
+			fsl,pins = <
+				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x000b0
+				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
+			>;
+		};
+
+		pinctrl_ov5642: ov5642grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+				MX6QDL_PAD_GPIO_6__GPIO1_IO06   0x1b0b0
+				MX6QDL_PAD_GPIO_8__GPIO1_IO08   0x130b0
+				MX6QDL_PAD_GPIO_3__CCM_CLKO2    0x000b0
+			>;
+		};
+
 		pinctrl_pwm1: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
@@ -601,3 +707,15 @@
 	vmmc-supply = <&reg_3p3v>;
 	status = "okay";
 };
+
+&mipi_csi {
+        status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+        remote-endpoint = <&ov5640_to_mipi_csi>;
+        data-lanes = <0 1>;
+        clock-lanes = <2>;
+};
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 07/24] ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Until the OV5652 sensor module compatible with the SabreSD becomes
available for testing, the ov5642 node is currently disabled.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl-sabresd.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabresd.dts    |   5 ++
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 ++++++++++++++++++++++++++++++++-
 3 files changed, 123 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl-sabresd.dts b/arch/arm/boot/dts/imx6dl-sabresd.dts
index 1e45f2f..6cf7a50 100644
--- a/arch/arm/boot/dts/imx6dl-sabresd.dts
+++ b/arch/arm/boot/dts/imx6dl-sabresd.dts
@@ -15,3 +15,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Smart Device Board";
 	compatible = "fsl,imx6dl-sabresd", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6q-sabresd.dts b/arch/arm/boot/dts/imx6q-sabresd.dts
index 9cbdfe7..8c1d7ad 100644
--- a/arch/arm/boot/dts/imx6q-sabresd.dts
+++ b/arch/arm/boot/dts/imx6q-sabresd.dts
@@ -23,3 +23,8 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
index 55ef535..7c5dc99 100644
--- a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
@@ -10,6 +10,7 @@
  * http://www.gnu.org/copyleft/gpl.html
  */
 
+#include <dt-bindings/clock/imx6qdl-clock.h>
 #include <dt-bindings/gpio/gpio.h>
 #include <dt-bindings/input/input.h>
 
@@ -146,6 +147,33 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vsync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
+&mipi_csi {
+	status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+	remote-endpoint = <&ov5640_to_mipi_csi>;
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
 &audmux {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_audmux>;
@@ -214,7 +242,33 @@
 			0x8014 /* 4:FN_DMICCDAT */
 			0x0000 /* 5:Default */
 		>;
-       };
+	};
+
+	ov5642: camera@3c {
+		compatible = "ovti,ov5642";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5642>;
+		clocks = <&clks IMX6QDL_CLK_CKO>;
+		clock-names = "xclk";
+		reg = <0x3c>;
+		xclk = <24000000>;
+		DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+		AVDD-supply = <&vgen3_reg>;  /* 2.8v, rev C board is VGEN3
+						rev B board is VGEN5 */
+		DVDD-supply = <&vgen2_reg>;  /* 1.5v*/
+		pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
+		reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>;
+		status = "disabled";
+
+		port {
+			ov5642_to_ipu1_csi0_mux: endpoint {
+				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+				bus-width = <8>;
+				hsync-active = <1>;
+				vsync-active = <1>;
+			};
+		};
+	};
 };
 
 &i2c2 {
@@ -223,6 +277,34 @@
 	pinctrl-0 = <&pinctrl_i2c2>;
 	status = "okay";
 
+	ov5640: camera@3c {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		reg = <0x3c>;
+		clocks = <&clks IMX6QDL_CLK_CKO>;
+		clock-names = "xclk";
+		xclk = <24000000>;
+		DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+		AVDD-supply = <&vgen3_reg>;  /* 2.8v, rev C board is VGEN3
+						rev B board is VGEN5 */
+		DVDD-supply = <&vgen2_reg>;  /* 1.5v*/
+		pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>;
+		reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>;
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint@1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;
+			};
+		};
+	};
+
 	pmic: pfuze100@08 {
 		compatible = "fsl,pfuze100";
 		reg = <0x08>;
@@ -426,6 +508,36 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
+			>;
+		};
+
+		pinctrl_ov5640: ov5640grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x1b0b0
+				MX6QDL_PAD_SD1_CLK__GPIO1_IO20  0x1b0b0
+			>;
+		};
+
+		pinctrl_ov5642: ov5642grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+				MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x1b0b0
+			>;
+		};
+
 		pinctrl_pcie: pciegrp {
 			fsl,pins = <
 				MX6QDL_PAD_GPIO_17__GPIO7_IO12	0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 07/24] ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Until the OV5652 sensor module compatible with the SabreSD becomes
available for testing, the ov5642 node is currently disabled.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl-sabresd.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabresd.dts    |   5 ++
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 ++++++++++++++++++++++++++++++++-
 3 files changed, 123 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl-sabresd.dts b/arch/arm/boot/dts/imx6dl-sabresd.dts
index 1e45f2f..6cf7a50 100644
--- a/arch/arm/boot/dts/imx6dl-sabresd.dts
+++ b/arch/arm/boot/dts/imx6dl-sabresd.dts
@@ -15,3 +15,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Smart Device Board";
 	compatible = "fsl,imx6dl-sabresd", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6q-sabresd.dts b/arch/arm/boot/dts/imx6q-sabresd.dts
index 9cbdfe7..8c1d7ad 100644
--- a/arch/arm/boot/dts/imx6q-sabresd.dts
+++ b/arch/arm/boot/dts/imx6q-sabresd.dts
@@ -23,3 +23,8 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
index 55ef535..7c5dc99 100644
--- a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
@@ -10,6 +10,7 @@
  * http://www.gnu.org/copyleft/gpl.html
  */
 
+#include <dt-bindings/clock/imx6qdl-clock.h>
 #include <dt-bindings/gpio/gpio.h>
 #include <dt-bindings/input/input.h>
 
@@ -146,6 +147,33 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vsync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
+&mipi_csi {
+	status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+	remote-endpoint = <&ov5640_to_mipi_csi>;
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
 &audmux {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_audmux>;
@@ -214,7 +242,33 @@
 			0x8014 /* 4:FN_DMICCDAT */
 			0x0000 /* 5:Default */
 		>;
-       };
+	};
+
+	ov5642: camera@3c {
+		compatible = "ovti,ov5642";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5642>;
+		clocks = <&clks IMX6QDL_CLK_CKO>;
+		clock-names = "xclk";
+		reg = <0x3c>;
+		xclk = <24000000>;
+		DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+		AVDD-supply = <&vgen3_reg>;  /* 2.8v, rev C board is VGEN3
+						rev B board is VGEN5 */
+		DVDD-supply = <&vgen2_reg>;  /* 1.5v*/
+		pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
+		reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>;
+		status = "disabled";
+
+		port {
+			ov5642_to_ipu1_csi0_mux: endpoint {
+				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+				bus-width = <8>;
+				hsync-active = <1>;
+				vsync-active = <1>;
+			};
+		};
+	};
 };
 
 &i2c2 {
@@ -223,6 +277,34 @@
 	pinctrl-0 = <&pinctrl_i2c2>;
 	status = "okay";
 
+	ov5640: camera@3c {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		reg = <0x3c>;
+		clocks = <&clks IMX6QDL_CLK_CKO>;
+		clock-names = "xclk";
+		xclk = <24000000>;
+		DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+		AVDD-supply = <&vgen3_reg>;  /* 2.8v, rev C board is VGEN3
+						rev B board is VGEN5 */
+		DVDD-supply = <&vgen2_reg>;  /* 1.5v*/
+		pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>;
+		reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>;
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint@1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;
+			};
+		};
+	};
+
 	pmic: pfuze100@08 {
 		compatible = "fsl,pfuze100";
 		reg = <0x08>;
@@ -426,6 +508,36 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
+			>;
+		};
+
+		pinctrl_ov5640: ov5640grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x1b0b0
+				MX6QDL_PAD_SD1_CLK__GPIO1_IO20  0x1b0b0
+			>;
+		};
+
+		pinctrl_ov5642: ov5642grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+				MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x1b0b0
+			>;
+		};
+
 		pinctrl_pcie: pciegrp {
 			fsl,pins = <
 				MX6QDL_PAD_GPIO_17__GPIO7_IO12	0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 07/24] ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Until the OV5652 sensor module compatible with the SabreSD becomes
available for testing, the ov5642 node is currently disabled.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6dl-sabresd.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabresd.dts    |   5 ++
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 ++++++++++++++++++++++++++++++++-
 3 files changed, 123 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl-sabresd.dts b/arch/arm/boot/dts/imx6dl-sabresd.dts
index 1e45f2f..6cf7a50 100644
--- a/arch/arm/boot/dts/imx6dl-sabresd.dts
+++ b/arch/arm/boot/dts/imx6dl-sabresd.dts
@@ -15,3 +15,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Smart Device Board";
 	compatible = "fsl,imx6dl-sabresd", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6q-sabresd.dts b/arch/arm/boot/dts/imx6q-sabresd.dts
index 9cbdfe7..8c1d7ad 100644
--- a/arch/arm/boot/dts/imx6q-sabresd.dts
+++ b/arch/arm/boot/dts/imx6q-sabresd.dts
@@ -23,3 +23,8 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
index 55ef535..7c5dc99 100644
--- a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
@@ -10,6 +10,7 @@
  * http://www.gnu.org/copyleft/gpl.html
  */
 
+#include <dt-bindings/clock/imx6qdl-clock.h>
 #include <dt-bindings/gpio/gpio.h>
 #include <dt-bindings/input/input.h>
 
@@ -146,6 +147,33 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vsync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
+&mipi_csi {
+	status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+	remote-endpoint = <&ov5640_to_mipi_csi>;
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
 &audmux {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_audmux>;
@@ -214,7 +242,33 @@
 			0x8014 /* 4:FN_DMICCDAT */
 			0x0000 /* 5:Default */
 		>;
-       };
+	};
+
+	ov5642: camera at 3c {
+		compatible = "ovti,ov5642";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5642>;
+		clocks = <&clks IMX6QDL_CLK_CKO>;
+		clock-names = "xclk";
+		reg = <0x3c>;
+		xclk = <24000000>;
+		DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+		AVDD-supply = <&vgen3_reg>;  /* 2.8v, rev C board is VGEN3
+						rev B board is VGEN5 */
+		DVDD-supply = <&vgen2_reg>;  /* 1.5v*/
+		pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
+		reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>;
+		status = "disabled";
+
+		port {
+			ov5642_to_ipu1_csi0_mux: endpoint {
+				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+				bus-width = <8>;
+				hsync-active = <1>;
+				vsync-active = <1>;
+			};
+		};
+	};
 };
 
 &i2c2 {
@@ -223,6 +277,34 @@
 	pinctrl-0 = <&pinctrl_i2c2>;
 	status = "okay";
 
+	ov5640: camera at 3c {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		reg = <0x3c>;
+		clocks = <&clks IMX6QDL_CLK_CKO>;
+		clock-names = "xclk";
+		xclk = <24000000>;
+		DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+		AVDD-supply = <&vgen3_reg>;  /* 2.8v, rev C board is VGEN3
+						rev B board is VGEN5 */
+		DVDD-supply = <&vgen2_reg>;  /* 1.5v*/
+		pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>;
+		reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>;
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint at 1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;
+			};
+		};
+	};
+
 	pmic: pfuze100 at 08 {
 		compatible = "fsl,pfuze100";
 		reg = <0x08>;
@@ -426,6 +508,36 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
+			>;
+		};
+
+		pinctrl_ov5640: ov5640grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x1b0b0
+				MX6QDL_PAD_SD1_CLK__GPIO1_IO20  0x1b0b0
+			>;
+		};
+
+		pinctrl_ov5642: ov5642grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+				MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x1b0b0
+			>;
+		};
+
 		pinctrl_pcie: pciegrp {
 			fsl,pins = <
 				MX6QDL_PAD_GPIO_17__GPIO7_IO12	0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 08/24] ARM: dts: imx6-sabreauto: create i2cmux for i2c3
  2017-01-07  2:11 ` Steve Longerbeam
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

The sabreauto uses a steering pin to select between the SDA signal on
i2c3 bus, and a data-in pin for an SPI NOR chip. Use i2cmux to control
this steering pin. Idle state of the i2cmux selects SPI NOR. This is not
a classic way to use i2cmux, since one side of the mux selects something
other than an i2c bus, but it works and is probably the cleanest
solution. Note that if one thread is attempting to access SPI NOR while
another thread is accessing i2c3, the SPI NOR access will fail since the
i2cmux has selected the SDA pin rather than SPI NOR data-in. This couldn't
be avoided in any case, the board is not designed to allow concurrent
i2c3 and SPI NOR functions (and the default device-tree does not enable
SPI NOR anyway).

Devices hanging off i2c3 should now be defined under i2cmux, so
that the steering pin can be properly controlled to access those
devices. The port expanders (MAX7310) are thus moved into i2cmux.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 65 +++++++++++++++++++++-----------
 1 file changed, 44 insertions(+), 21 deletions(-)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 52390ba..cace88c 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -108,6 +108,44 @@
 		default-brightness-level = <7>;
 		status = "okay";
 	};
+
+	i2cmux {
+		compatible = "i2c-mux-gpio";
+		#address-cells = <1>;
+		#size-cells = <0>;
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_i2c3mux>;
+		mux-gpios = <&gpio5 4 0>;
+		i2c-parent = <&i2c3>;
+		idle-state = <0>;
+
+		i2c@1 {
+			#address-cells = <1>;
+			#size-cells = <0>;
+			reg = <1>;
+
+			max7310_a: gpio@30 {
+				compatible = "maxim,max7310";
+				reg = <0x30>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+
+			max7310_b: gpio@32 {
+				compatible = "maxim,max7310";
+				reg = <0x32>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+
+			max7310_c: gpio@34 {
+				compatible = "maxim,max7310";
+				reg = <0x34>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+		};
+	};
 };
 
 &clks {
@@ -291,27 +329,6 @@
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_i2c3>;
 	status = "okay";
-
-	max7310_a: gpio@30 {
-		compatible = "maxim,max7310";
-		reg = <0x30>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
-
-	max7310_b: gpio@32 {
-		compatible = "maxim,max7310";
-		reg = <0x32>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
-
-	max7310_c: gpio@34 {
-		compatible = "maxim,max7310";
-		reg = <0x34>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
 };
 
 &iomuxc {
@@ -419,6 +436,12 @@
 			>;
 		};
 
+		pinctrl_i2c3mux: i2c3muxgrp {
+			fsl,pins = <
+				MX6QDL_PAD_EIM_A24__GPIO5_IO04 0x0b0b1
+			>;
+		};
+
 		pinctrl_pwm3: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD4_DAT1__PWM3_OUT		0x1b0b1
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 08/24] ARM: dts: imx6-sabreauto: create i2cmux for i2c3
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

The sabreauto uses a steering pin to select between the SDA signal on
i2c3 bus, and a data-in pin for an SPI NOR chip. Use i2cmux to control
this steering pin. Idle state of the i2cmux selects SPI NOR. This is not
a classic way to use i2cmux, since one side of the mux selects something
other than an i2c bus, but it works and is probably the cleanest
solution. Note that if one thread is attempting to access SPI NOR while
another thread is accessing i2c3, the SPI NOR access will fail since the
i2cmux has selected the SDA pin rather than SPI NOR data-in. This couldn't
be avoided in any case, the board is not designed to allow concurrent
i2c3 and SPI NOR functions (and the default device-tree does not enable
SPI NOR anyway).

Devices hanging off i2c3 should now be defined under i2cmux, so
that the steering pin can be properly controlled to access those
devices. The port expanders (MAX7310) are thus moved into i2cmux.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 65 +++++++++++++++++++++-----------
 1 file changed, 44 insertions(+), 21 deletions(-)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 52390ba..cace88c 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -108,6 +108,44 @@
 		default-brightness-level = <7>;
 		status = "okay";
 	};
+
+	i2cmux {
+		compatible = "i2c-mux-gpio";
+		#address-cells = <1>;
+		#size-cells = <0>;
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_i2c3mux>;
+		mux-gpios = <&gpio5 4 0>;
+		i2c-parent = <&i2c3>;
+		idle-state = <0>;
+
+		i2c at 1 {
+			#address-cells = <1>;
+			#size-cells = <0>;
+			reg = <1>;
+
+			max7310_a: gpio at 30 {
+				compatible = "maxim,max7310";
+				reg = <0x30>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+
+			max7310_b: gpio at 32 {
+				compatible = "maxim,max7310";
+				reg = <0x32>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+
+			max7310_c: gpio at 34 {
+				compatible = "maxim,max7310";
+				reg = <0x34>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+		};
+	};
 };
 
 &clks {
@@ -291,27 +329,6 @@
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_i2c3>;
 	status = "okay";
-
-	max7310_a: gpio at 30 {
-		compatible = "maxim,max7310";
-		reg = <0x30>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
-
-	max7310_b: gpio at 32 {
-		compatible = "maxim,max7310";
-		reg = <0x32>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
-
-	max7310_c: gpio at 34 {
-		compatible = "maxim,max7310";
-		reg = <0x34>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
 };
 
 &iomuxc {
@@ -419,6 +436,12 @@
 			>;
 		};
 
+		pinctrl_i2c3mux: i2c3muxgrp {
+			fsl,pins = <
+				MX6QDL_PAD_EIM_A24__GPIO5_IO04 0x0b0b1
+			>;
+		};
+
 		pinctrl_pwm3: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD4_DAT1__PWM3_OUT		0x1b0b1
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 09/24] ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
  2017-01-07  2:11 ` Steve Longerbeam
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

The reset pin to the port expander chip (MAX7310) is controlled by a gpio,
so define a reset-gpios property to control it. There are three MAX7310's
on the SabreAuto CPU card (max7310_[abc]), but all use the same pin for
their reset. Since all can't acquire the same pin, assign it to max7310_b,
that chip is needed by more functions (usb and adv7180).

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index cace88c..967c3b8 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -136,6 +136,9 @@
 				reg = <0x32>;
 				gpio-controller;
 				#gpio-cells = <2>;
+				pinctrl-names = "default";
+				pinctrl-0 = <&pinctrl_max7310>;
+				reset-gpios = <&gpio1 15 GPIO_ACTIVE_LOW>;
 			};
 
 			max7310_c: gpio@34 {
@@ -442,6 +445,12 @@
 			>;
 		};
 
+		pinctrl_max7310: max7310grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
+			>;
+		};
+
 		pinctrl_pwm3: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD4_DAT1__PWM3_OUT		0x1b0b1
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 09/24] ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

The reset pin to the port expander chip (MAX7310) is controlled by a gpio,
so define a reset-gpios property to control it. There are three MAX7310's
on the SabreAuto CPU card (max7310_[abc]), but all use the same pin for
their reset. Since all can't acquire the same pin, assign it to max7310_b,
that chip is needed by more functions (usb and adv7180).

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index cace88c..967c3b8 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -136,6 +136,9 @@
 				reg = <0x32>;
 				gpio-controller;
 				#gpio-cells = <2>;
+				pinctrl-names = "default";
+				pinctrl-0 = <&pinctrl_max7310>;
+				reset-gpios = <&gpio1 15 GPIO_ACTIVE_LOW>;
 			};
 
 			max7310_c: gpio at 34 {
@@ -442,6 +445,12 @@
 			>;
 		};
 
+		pinctrl_max7310: max7310grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
+			>;
+		};
+
 		pinctrl_pwm3: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD4_DAT1__PWM3_OUT		0x1b0b1
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Add pinctrl groups for both GPT input capture channels.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 967c3b8..495709f 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -457,6 +457,18 @@
 			>;
 		};
 
+		pinctrl_gpt_input_capture0: gptinputcapture0grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1	0x1b0b0
+			>;
+		};
+
+		pinctrl_gpt_input_capture1: gptinputcapture1grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2	0x1b0b0
+			>;
+		};
+
 		pinctrl_spdif: spdifgrp {
 			fsl,pins = <
 				MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r
  Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

Add pinctrl groups for both GPT input capture channels.

Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 967c3b8..495709f 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -457,6 +457,18 @@
 			>;
 		};
 
+		pinctrl_gpt_input_capture0: gptinputcapture0grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1	0x1b0b0
+			>;
+		};
+
+		pinctrl_gpt_input_capture1: gptinputcapture1grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2	0x1b0b0
+			>;
+		};
+
 		pinctrl_spdif: spdifgrp {
 			fsl,pins = <
 				MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Add pinctrl groups for both GPT input capture channels.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 967c3b8..495709f 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -457,6 +457,18 @@
 			>;
 		};
 
+		pinctrl_gpt_input_capture0: gptinputcapture0grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1	0x1b0b0
+			>;
+		};
+
+		pinctrl_gpt_input_capture1: gptinputcapture1grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2	0x1b0b0
+			>;
+		};
+
 		pinctrl_spdif: spdifgrp {
 			fsl,pins = <
 				MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 11/24] ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Enables the ADV7180 decoder sensor. The ADV7180 connects to the
parallel-bus mux input on ipu1_csi0_mux.

On the sabreauto, two analog video inputs are routed to the ADV7180,
composite on Ain1, and composite on Ain3. Those inputs are defined
via inputs and input-names under the ADV7180 node. The ADV7180 power
pin is via max7310_b port expander.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 61 ++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 495709f..7999857 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -124,6 +124,24 @@
 			#size-cells = <0>;
 			reg = <1>;
 
+			adv7180: camera@21 {
+				compatible = "adi,adv7180";
+				reg = <0x21>;
+				powerdown-gpios = <&max7310_b 2 GPIO_ACTIVE_LOW>;
+				interrupt-parent = <&gpio1>;
+				interrupts = <27 0x8>;
+				inputs = <0x00 0x02>;
+				input-names = "ADV7180 Composite on Ain1",
+						"ADV7180 Composite on Ain3";
+
+				port {
+					adv7180_to_ipu1_csi0_mux: endpoint {
+						remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+						bus-width = <8>;
+					};
+				};
+			};
+
 			max7310_a: gpio@30 {
 				compatible = "maxim,max7310";
 				reg = <0x30>;
@@ -151,6 +169,25 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+	bus-width = <8>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+	/* enable frame interval monitor on this port */
+	fim {
+		status = "okay";
+	};
+};
+
 &clks {
 	assigned-clocks = <&clks IMX6QDL_PLL4_BYPASS_SRC>,
 			  <&clks IMX6QDL_PLL4_BYPASS>,
@@ -445,6 +482,30 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT4__IPU1_CSI0_DATA04   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT5__IPU1_CSI0_DATA05   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT6__IPU1_CSI0_DATA06   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT7__IPU1_CSI0_DATA07   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT8__IPU1_CSI0_DATA08   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT9__IPU1_CSI0_DATA09   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT10__IPU1_CSI0_DATA10  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT11__IPU1_CSI0_DATA11  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19  0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC    0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC   0x1b0b0
+			>;
+		};
+
 		pinctrl_max7310: max7310grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 11/24] ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

Enables the ADV7180 decoder sensor. The ADV7180 connects to the
parallel-bus mux input on ipu1_csi0_mux.

On the sabreauto, two analog video inputs are routed to the ADV7180,
composite on Ain1, and composite on Ain3. Those inputs are defined
via inputs and input-names under the ADV7180 node. The ADV7180 power
pin is via max7310_b port expander.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 61 ++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 495709f..7999857 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -124,6 +124,24 @@
 			#size-cells = <0>;
 			reg = <1>;
 
+			adv7180: camera@21 {
+				compatible = "adi,adv7180";
+				reg = <0x21>;
+				powerdown-gpios = <&max7310_b 2 GPIO_ACTIVE_LOW>;
+				interrupt-parent = <&gpio1>;
+				interrupts = <27 0x8>;
+				inputs = <0x00 0x02>;
+				input-names = "ADV7180 Composite on Ain1",
+						"ADV7180 Composite on Ain3";
+
+				port {
+					adv7180_to_ipu1_csi0_mux: endpoint {
+						remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+						bus-width = <8>;
+					};
+				};
+			};
+
 			max7310_a: gpio@30 {
 				compatible = "maxim,max7310";
 				reg = <0x30>;
@@ -151,6 +169,25 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+	bus-width = <8>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+	/* enable frame interval monitor on this port */
+	fim {
+		status = "okay";
+	};
+};
+
 &clks {
 	assigned-clocks = <&clks IMX6QDL_PLL4_BYPASS_SRC>,
 			  <&clks IMX6QDL_PLL4_BYPASS>,
@@ -445,6 +482,30 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT4__IPU1_CSI0_DATA04   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT5__IPU1_CSI0_DATA05   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT6__IPU1_CSI0_DATA06   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT7__IPU1_CSI0_DATA07   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT8__IPU1_CSI0_DATA08   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT9__IPU1_CSI0_DATA09   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT10__IPU1_CSI0_DATA10  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT11__IPU1_CSI0_DATA11  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19  0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC    0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC   0x1b0b0
+			>;
+		};
+
 		pinctrl_max7310: max7310grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 11/24] ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Enables the ADV7180 decoder sensor. The ADV7180 connects to the
parallel-bus mux input on ipu1_csi0_mux.

On the sabreauto, two analog video inputs are routed to the ADV7180,
composite on Ain1, and composite on Ain3. Those inputs are defined
via inputs and input-names under the ADV7180 node. The ADV7180 power
pin is via max7310_b port expander.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 61 ++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 495709f..7999857 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -124,6 +124,24 @@
 			#size-cells = <0>;
 			reg = <1>;
 
+			adv7180: camera at 21 {
+				compatible = "adi,adv7180";
+				reg = <0x21>;
+				powerdown-gpios = <&max7310_b 2 GPIO_ACTIVE_LOW>;
+				interrupt-parent = <&gpio1>;
+				interrupts = <27 0x8>;
+				inputs = <0x00 0x02>;
+				input-names = "ADV7180 Composite on Ain1",
+						"ADV7180 Composite on Ain3";
+
+				port {
+					adv7180_to_ipu1_csi0_mux: endpoint {
+						remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+						bus-width = <8>;
+					};
+				};
+			};
+
 			max7310_a: gpio at 30 {
 				compatible = "maxim,max7310";
 				reg = <0x30>;
@@ -151,6 +169,25 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+	bus-width = <8>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+	/* enable frame interval monitor on this port */
+	fim {
+		status = "okay";
+	};
+};
+
 &clks {
 	assigned-clocks = <&clks IMX6QDL_PLL4_BYPASS_SRC>,
 			  <&clks IMX6QDL_PLL4_BYPASS>,
@@ -445,6 +482,30 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT4__IPU1_CSI0_DATA04   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT5__IPU1_CSI0_DATA05   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT6__IPU1_CSI0_DATA06   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT7__IPU1_CSI0_DATA07   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT8__IPU1_CSI0_DATA08   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT9__IPU1_CSI0_DATA09   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT10__IPU1_CSI0_DATA10  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT11__IPU1_CSI0_DATA11  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19  0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC    0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC   0x1b0b0
+			>;
+		};
+
 		pinctrl_max7310: max7310grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 12/24] add mux and video interface bridge entity functions
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel

From: Philipp Zabel <p.zabel@pengutronix.de>

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)

diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
index 3e03dc2..023be29 100644
--- a/Documentation/media/uapi/mediactl/media-types.rst
+++ b/Documentation/media/uapi/mediactl/media-types.rst
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
 	  received on its sink pad and outputs the statistics data on
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad. Video
+         frame(s) from the inactive sink pads are discarded.
+
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have at
+         least one sink pad and one source pad. It receives video frame(s) on
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus format
+         (eDP, MIPI CSI-2, parallel, ...).
 
 ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
 
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 4890787..08a8bfa 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -105,6 +105,12 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
 
 /*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
+#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
+
+/*
  * Connectors
  */
 /* It is a responsibility of the entity drivers to add connectors and links */
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 12/24] add mux and video interface bridge entity functions
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, linux-kernel, linux-arm-kernel, linux-media

From: Philipp Zabel <p.zabel@pengutronix.de>

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)

diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
index 3e03dc2..023be29 100644
--- a/Documentation/media/uapi/mediactl/media-types.rst
+++ b/Documentation/media/uapi/mediactl/media-types.rst
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
 	  received on its sink pad and outputs the statistics data on
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad. Video
+         frame(s) from the inactive sink pads are discarded.
+
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have at
+         least one sink pad and one source pad. It receives video frame(s) on
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus format
+         (eDP, MIPI CSI-2, parallel, ...).
 
 ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
 
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 4890787..08a8bfa 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -105,6 +105,12 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
 
 /*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
+#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
+
+/*
  * Connectors
  */
 /* It is a responsibility of the entity drivers to add connectors and links */
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 12/24] add mux and video interface bridge entity functions
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

From: Philipp Zabel <p.zabel@pengutronix.de>

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)

diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
index 3e03dc2..023be29 100644
--- a/Documentation/media/uapi/mediactl/media-types.rst
+++ b/Documentation/media/uapi/mediactl/media-types.rst
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
 	  received on its sink pad and outputs the statistics data on
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad. Video
+         frame(s) from the inactive sink pads are discarded.
+
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have at
+         least one sink pad and one source pad. It receives video frame(s) on
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus format
+         (eDP, MIPI CSI-2, parallel, ...).
 
 ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
 
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 4890787..08a8bfa 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -105,6 +105,12 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
 
 /*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
+#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
+
+/*
  * Connectors
  */
 /* It is a responsibility of the entity drivers to add connectors and links */
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Sascha Hauer, Steve Longerbeam

From: Philipp Zabel <p.zabel@pengutronix.de>

This driver can handle SoC internal and external video bus multiplexers,
controlled either by register bit fields or by a GPIO. The subdevice
passes through frame interval and mbus configuration of the active input
to the output side.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
  should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev()
  to vidsw_remove().

- there was a line left over from a previous iteration that negated
  the new way of determining the pad count just before it which
  has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding
  to the subdev async notifier waiting list using a chaining method
  from the async registered callbacks (v4l2_of_subdev_registered()
  and the prep patches for that). For now, I've removed the use of
  v4l2_of_subdev_registered() for the vidmux driver's registered
  callback. This doesn't affect the functionality of this driver,
  but allows for it to be merged now, before adding the chaining
  support.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c

diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
new file mode 100644
index 0000000..9d133d9
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index d944421..65614b5 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
 	  To compile this driver as a module, choose M here: the
 	  module will be called arv.
 
+config VIDEO_MULTIPLEXER
+	tristate "Video Multiplexer"
+	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+	help
+	  This driver provides support for SoC internal N:1 video bus
+	  multiplexers controlled by register bitfields as well as external
+	  2:1 video multiplexers controlled by a single GPIO.
+
 config VIDEO_OMAP3
 	tristate "OMAP 3 Camera support"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index 5b3cb27..7cf0ee5 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
 
 obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
 
+obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
+
 obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
 obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
new file mode 100644
index 0000000..48980c4
--- /dev/null
+++ b/drivers/media/platform/video-multiplexer.c
@@ -0,0 +1,472 @@
+/*
+ * video stream multiplexer controlled via gpio or syscon
+ *
+ * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
+ * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+
+struct vidsw {
+	struct v4l2_subdev subdev;
+	unsigned int num_pads;
+	struct media_pad *pads;
+	struct v4l2_mbus_framefmt *format_mbus;
+	struct v4l2_fract timeperframe;
+	struct v4l2_of_endpoint *endpoint;
+	struct regmap_field *field;
+	struct gpio_desc *gpio;
+	int active;
+};
+
+static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct vidsw, subdev);
+}
+
+static void vidsw_set_active(struct vidsw *vidsw, int active)
+{
+	vidsw->active = active;
+	if (active < 0)
+		return;
+
+	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
+
+	if (vidsw->field)
+		regmap_field_write(vidsw->field, active);
+	else if (vidsw->gpio)
+		gpiod_set_value(vidsw->gpio, active);
+}
+
+static int vidsw_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	/* We have no limitations on enabling or disabling our output link */
+	if (local->index == vidsw->num_pads - 1)
+		return 0;
+
+	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+		if (local->index == vidsw->active) {
+			dev_dbg(sd->dev, "going inactive\n");
+			vidsw->active = -1;
+		}
+		return 0;
+	}
+
+	if (vidsw->active >= 0) {
+		struct media_pad *pad;
+
+		if (vidsw->active == local->index)
+			return 0;
+
+		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+		if (pad) {
+			struct media_link *link;
+			int ret;
+
+			link = media_entity_find_link(pad,
+						&vidsw->pads[vidsw->active]);
+			if (link) {
+				ret = __media_entity_setup_link(link, 0);
+				if (ret)
+					return ret;
+			}
+		}
+	}
+
+	vidsw_set_active(vidsw, local->index);
+
+	return 0;
+}
+
+static struct media_entity_operations vidsw_ops = {
+	.link_setup = vidsw_link_setup,
+};
+
+static bool vidsw_endpoint_disabled(struct device_node *ep)
+{
+	struct device_node *rpp;
+
+	if (!of_device_is_available(ep))
+		return true;
+
+	rpp = of_graph_get_remote_port_parent(ep);
+	if (!rpp)
+		return true;
+
+	return !of_device_is_available(rpp);
+}
+
+static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
+{
+	struct device_node *ep;
+	u32 portno;
+	int numports;
+	int ret;
+	int i;
+	bool active_link = false;
+
+	numports = vidsw->num_pads;
+
+	for (i = 0; i < numports - 1; i++)
+		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
+	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
+
+	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
+	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
+				     vidsw->pads);
+	if (ret < 0)
+		return ret;
+
+	vidsw->subdev.entity.ops = &vidsw_ops;
+
+	for_each_endpoint_of_node(node, ep) {
+		struct v4l2_of_endpoint endpoint;
+
+		v4l2_of_parse_endpoint(ep, &endpoint);
+
+		portno = endpoint.base.port;
+		if (portno >= numports - 1)
+			continue;
+
+		if (vidsw_endpoint_disabled(ep)) {
+			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
+			continue;
+		}
+
+		vidsw->endpoint[portno] = endpoint;
+
+		if (portno == vidsw->active)
+			active_link = true;
+	}
+
+	for (portno = 0; portno < numports - 1; portno++) {
+		if (!vidsw->endpoint[portno].base.local_node)
+			continue;
+
+		/* If the active input is not connected, use another */
+		if (!active_link) {
+			vidsw_set_active(vidsw, portno);
+			active_link = true;
+		}
+	}
+
+	return v4l2_async_register_subdev(&vidsw->subdev);
+}
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source configuration\n");
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
+	}
+
+	return 0;
+}
+
+static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_subdev *upstream_sd;
+	struct media_pad *pad;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "Can not start streaming on inactive mux\n");
+		return -EINVAL;
+	}
+
+	pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
+	if (!pad) {
+		dev_err(sd->dev, "Failed to find remote source pad\n");
+		return -ENOLINK;
+	}
+
+	if (!is_media_entity_v4l2_subdev(pad->entity)) {
+		dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
+		return -ENODEV;
+	}
+
+	upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
+
+	return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
+}
+
+static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	fi->interval = vidsw->timeperframe;
+
+	return 0;
+}
+
+static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	vidsw->timeperframe = fi->interval;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
+	.g_mbus_config = vidsw_g_mbus_config,
+	.s_stream = vidsw_s_stream,
+	.g_frame_interval = vidsw_g_frame_interval,
+	.s_frame_interval = vidsw_s_frame_interval,
+};
+
+static struct v4l2_mbus_framefmt *
+__vidsw_get_pad_format(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       unsigned int pad, u32 which)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	switch (which) {
+	case V4L2_SUBDEV_FORMAT_TRY:
+		return v4l2_subdev_get_try_format(sd, cfg, pad);
+	case V4L2_SUBDEV_FORMAT_ACTIVE:
+		return &vidsw->format_mbus[pad];
+	default:
+		return NULL;
+	}
+}
+
+static int vidsw_get_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
+						   sdformat->which);
+	return 0;
+}
+
+static int vidsw_set_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_mbus_framefmt *mbusformat;
+
+	if (sdformat->pad >= vidsw->num_pads)
+		return -EINVAL;
+
+	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
+					    sdformat->which);
+	if (!mbusformat)
+		return -EINVAL;
+
+	/* Output pad mirrors active input pad, no limitations on input pads */
+	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
+		sdformat->format = vidsw->format_mbus[vidsw->active];
+
+	*mbusformat = sdformat->format;
+
+	return 0;
+}
+
+static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
+	.get_fmt = vidsw_get_format,
+	.set_fmt = vidsw_set_format,
+};
+
+static struct v4l2_subdev_ops vidsw_subdev_ops = {
+	.pad = &vidsw_pad_ops,
+	.video = &vidsw_subdev_video_ops,
+};
+
+static int of_get_reg_field(struct device_node *node, struct reg_field *field)
+{
+	u32 bit_mask;
+	int ret;
+
+	ret = of_property_read_u32(node, "reg", &field->reg);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
+	if (ret < 0)
+		return ret;
+
+	field->msb = field->lsb + fls(bit_mask) - 1;
+
+	return 0;
+}
+
+static int vidsw_probe(struct platform_device *pdev)
+{
+	struct device_node *np = pdev->dev.of_node;
+	struct of_endpoint endpoint;
+	struct device_node *ep;
+	struct reg_field field;
+	struct vidsw *vidsw;
+	struct regmap *map;
+	unsigned int num_pads;
+	int ret;
+
+	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
+	if (!vidsw)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, vidsw);
+
+	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
+	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
+			np->name);
+	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	vidsw->subdev.dev = &pdev->dev;
+
+	/*
+	 * The largest numbered port is the output port. It determines
+	 * total number of pads
+	 */
+	num_pads = 0;
+	for_each_endpoint_of_node(np, ep) {
+		of_graph_parse_endpoint(ep, &endpoint);
+		num_pads = max(num_pads, endpoint.port + 1);
+	}
+
+	if (num_pads < 2) {
+		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
+		return -EINVAL;
+	}
+
+	ret = of_get_reg_field(np, &field);
+	if (ret == 0) {
+		map = syscon_node_to_regmap(np->parent);
+		if (!map) {
+			dev_err(&pdev->dev, "Failed to get syscon register map\n");
+			return PTR_ERR(map);
+		}
+
+		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
+		if (IS_ERR(vidsw->field)) {
+			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
+			return PTR_ERR(vidsw->field);
+		}
+
+		regmap_field_read(vidsw->field, &vidsw->active);
+	} else {
+		if (num_pads > 3) {
+			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
+			return -EINVAL;
+		}
+
+		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
+		if (IS_ERR(vidsw->gpio)) {
+			dev_warn(&pdev->dev,
+				 "could not request control gpio: %d\n", ret);
+			vidsw->gpio = NULL;
+		}
+
+		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
+	}
+
+	vidsw->num_pads = num_pads;
+	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
+			GFP_KERNEL);
+	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
+	vidsw->endpoint = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
+
+	ret = vidsw_async_init(vidsw, np);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int vidsw_remove(struct platform_device *pdev)
+{
+	struct vidsw *vidsw = platform_get_drvdata(pdev);
+	struct v4l2_subdev *sd = &vidsw->subdev;
+
+	v4l2_async_unregister_subdev(sd);
+	media_entity_cleanup(&sd->entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct of_device_id vidsw_dt_ids[] = {
+	{ .compatible = "video-multiplexer", },
+	{ /* sentinel */ }
+};
+
+static struct platform_driver vidsw_driver = {
+	.probe		= vidsw_probe,
+	.remove		= vidsw_remove,
+	.driver		= {
+		.of_match_table = vidsw_dt_ids,
+		.name = "video-multiplexer",
+	},
+};
+
+module_platform_driver(vidsw_driver);
+
+MODULE_DESCRIPTION("video stream multiplexer");
+MODULE_AUTHOR("Sascha Hauer, Pengutronix");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, Sascha Hauer, linux-kernel,
	linux-arm-kernel, linux-media

From: Philipp Zabel <p.zabel@pengutronix.de>

This driver can handle SoC internal and external video bus multiplexers,
controlled either by register bit fields or by a GPIO. The subdevice
passes through frame interval and mbus configuration of the active input
to the output side.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
  should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev()
  to vidsw_remove().

- there was a line left over from a previous iteration that negated
  the new way of determining the pad count just before it which
  has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding
  to the subdev async notifier waiting list using a chaining method
  from the async registered callbacks (v4l2_of_subdev_registered()
  and the prep patches for that). For now, I've removed the use of
  v4l2_of_subdev_registered() for the vidmux driver's registered
  callback. This doesn't affect the functionality of this driver,
  but allows for it to be merged now, before adding the chaining
  support.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c

diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
new file mode 100644
index 0000000..9d133d9
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index d944421..65614b5 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
 	  To compile this driver as a module, choose M here: the
 	  module will be called arv.
 
+config VIDEO_MULTIPLEXER
+	tristate "Video Multiplexer"
+	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+	help
+	  This driver provides support for SoC internal N:1 video bus
+	  multiplexers controlled by register bitfields as well as external
+	  2:1 video multiplexers controlled by a single GPIO.
+
 config VIDEO_OMAP3
 	tristate "OMAP 3 Camera support"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index 5b3cb27..7cf0ee5 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
 
 obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
 
+obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
+
 obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
 obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
new file mode 100644
index 0000000..48980c4
--- /dev/null
+++ b/drivers/media/platform/video-multiplexer.c
@@ -0,0 +1,472 @@
+/*
+ * video stream multiplexer controlled via gpio or syscon
+ *
+ * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
+ * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+
+struct vidsw {
+	struct v4l2_subdev subdev;
+	unsigned int num_pads;
+	struct media_pad *pads;
+	struct v4l2_mbus_framefmt *format_mbus;
+	struct v4l2_fract timeperframe;
+	struct v4l2_of_endpoint *endpoint;
+	struct regmap_field *field;
+	struct gpio_desc *gpio;
+	int active;
+};
+
+static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct vidsw, subdev);
+}
+
+static void vidsw_set_active(struct vidsw *vidsw, int active)
+{
+	vidsw->active = active;
+	if (active < 0)
+		return;
+
+	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
+
+	if (vidsw->field)
+		regmap_field_write(vidsw->field, active);
+	else if (vidsw->gpio)
+		gpiod_set_value(vidsw->gpio, active);
+}
+
+static int vidsw_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	/* We have no limitations on enabling or disabling our output link */
+	if (local->index == vidsw->num_pads - 1)
+		return 0;
+
+	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+		if (local->index == vidsw->active) {
+			dev_dbg(sd->dev, "going inactive\n");
+			vidsw->active = -1;
+		}
+		return 0;
+	}
+
+	if (vidsw->active >= 0) {
+		struct media_pad *pad;
+
+		if (vidsw->active == local->index)
+			return 0;
+
+		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+		if (pad) {
+			struct media_link *link;
+			int ret;
+
+			link = media_entity_find_link(pad,
+						&vidsw->pads[vidsw->active]);
+			if (link) {
+				ret = __media_entity_setup_link(link, 0);
+				if (ret)
+					return ret;
+			}
+		}
+	}
+
+	vidsw_set_active(vidsw, local->index);
+
+	return 0;
+}
+
+static struct media_entity_operations vidsw_ops = {
+	.link_setup = vidsw_link_setup,
+};
+
+static bool vidsw_endpoint_disabled(struct device_node *ep)
+{
+	struct device_node *rpp;
+
+	if (!of_device_is_available(ep))
+		return true;
+
+	rpp = of_graph_get_remote_port_parent(ep);
+	if (!rpp)
+		return true;
+
+	return !of_device_is_available(rpp);
+}
+
+static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
+{
+	struct device_node *ep;
+	u32 portno;
+	int numports;
+	int ret;
+	int i;
+	bool active_link = false;
+
+	numports = vidsw->num_pads;
+
+	for (i = 0; i < numports - 1; i++)
+		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
+	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
+
+	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
+	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
+				     vidsw->pads);
+	if (ret < 0)
+		return ret;
+
+	vidsw->subdev.entity.ops = &vidsw_ops;
+
+	for_each_endpoint_of_node(node, ep) {
+		struct v4l2_of_endpoint endpoint;
+
+		v4l2_of_parse_endpoint(ep, &endpoint);
+
+		portno = endpoint.base.port;
+		if (portno >= numports - 1)
+			continue;
+
+		if (vidsw_endpoint_disabled(ep)) {
+			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
+			continue;
+		}
+
+		vidsw->endpoint[portno] = endpoint;
+
+		if (portno == vidsw->active)
+			active_link = true;
+	}
+
+	for (portno = 0; portno < numports - 1; portno++) {
+		if (!vidsw->endpoint[portno].base.local_node)
+			continue;
+
+		/* If the active input is not connected, use another */
+		if (!active_link) {
+			vidsw_set_active(vidsw, portno);
+			active_link = true;
+		}
+	}
+
+	return v4l2_async_register_subdev(&vidsw->subdev);
+}
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source configuration\n");
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
+	}
+
+	return 0;
+}
+
+static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_subdev *upstream_sd;
+	struct media_pad *pad;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "Can not start streaming on inactive mux\n");
+		return -EINVAL;
+	}
+
+	pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
+	if (!pad) {
+		dev_err(sd->dev, "Failed to find remote source pad\n");
+		return -ENOLINK;
+	}
+
+	if (!is_media_entity_v4l2_subdev(pad->entity)) {
+		dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
+		return -ENODEV;
+	}
+
+	upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
+
+	return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
+}
+
+static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	fi->interval = vidsw->timeperframe;
+
+	return 0;
+}
+
+static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	vidsw->timeperframe = fi->interval;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
+	.g_mbus_config = vidsw_g_mbus_config,
+	.s_stream = vidsw_s_stream,
+	.g_frame_interval = vidsw_g_frame_interval,
+	.s_frame_interval = vidsw_s_frame_interval,
+};
+
+static struct v4l2_mbus_framefmt *
+__vidsw_get_pad_format(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       unsigned int pad, u32 which)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	switch (which) {
+	case V4L2_SUBDEV_FORMAT_TRY:
+		return v4l2_subdev_get_try_format(sd, cfg, pad);
+	case V4L2_SUBDEV_FORMAT_ACTIVE:
+		return &vidsw->format_mbus[pad];
+	default:
+		return NULL;
+	}
+}
+
+static int vidsw_get_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
+						   sdformat->which);
+	return 0;
+}
+
+static int vidsw_set_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_mbus_framefmt *mbusformat;
+
+	if (sdformat->pad >= vidsw->num_pads)
+		return -EINVAL;
+
+	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
+					    sdformat->which);
+	if (!mbusformat)
+		return -EINVAL;
+
+	/* Output pad mirrors active input pad, no limitations on input pads */
+	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
+		sdformat->format = vidsw->format_mbus[vidsw->active];
+
+	*mbusformat = sdformat->format;
+
+	return 0;
+}
+
+static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
+	.get_fmt = vidsw_get_format,
+	.set_fmt = vidsw_set_format,
+};
+
+static struct v4l2_subdev_ops vidsw_subdev_ops = {
+	.pad = &vidsw_pad_ops,
+	.video = &vidsw_subdev_video_ops,
+};
+
+static int of_get_reg_field(struct device_node *node, struct reg_field *field)
+{
+	u32 bit_mask;
+	int ret;
+
+	ret = of_property_read_u32(node, "reg", &field->reg);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
+	if (ret < 0)
+		return ret;
+
+	field->msb = field->lsb + fls(bit_mask) - 1;
+
+	return 0;
+}
+
+static int vidsw_probe(struct platform_device *pdev)
+{
+	struct device_node *np = pdev->dev.of_node;
+	struct of_endpoint endpoint;
+	struct device_node *ep;
+	struct reg_field field;
+	struct vidsw *vidsw;
+	struct regmap *map;
+	unsigned int num_pads;
+	int ret;
+
+	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
+	if (!vidsw)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, vidsw);
+
+	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
+	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
+			np->name);
+	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	vidsw->subdev.dev = &pdev->dev;
+
+	/*
+	 * The largest numbered port is the output port. It determines
+	 * total number of pads
+	 */
+	num_pads = 0;
+	for_each_endpoint_of_node(np, ep) {
+		of_graph_parse_endpoint(ep, &endpoint);
+		num_pads = max(num_pads, endpoint.port + 1);
+	}
+
+	if (num_pads < 2) {
+		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
+		return -EINVAL;
+	}
+
+	ret = of_get_reg_field(np, &field);
+	if (ret == 0) {
+		map = syscon_node_to_regmap(np->parent);
+		if (!map) {
+			dev_err(&pdev->dev, "Failed to get syscon register map\n");
+			return PTR_ERR(map);
+		}
+
+		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
+		if (IS_ERR(vidsw->field)) {
+			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
+			return PTR_ERR(vidsw->field);
+		}
+
+		regmap_field_read(vidsw->field, &vidsw->active);
+	} else {
+		if (num_pads > 3) {
+			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
+			return -EINVAL;
+		}
+
+		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
+		if (IS_ERR(vidsw->gpio)) {
+			dev_warn(&pdev->dev,
+				 "could not request control gpio: %d\n", ret);
+			vidsw->gpio = NULL;
+		}
+
+		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
+	}
+
+	vidsw->num_pads = num_pads;
+	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
+			GFP_KERNEL);
+	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
+	vidsw->endpoint = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
+
+	ret = vidsw_async_init(vidsw, np);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int vidsw_remove(struct platform_device *pdev)
+{
+	struct vidsw *vidsw = platform_get_drvdata(pdev);
+	struct v4l2_subdev *sd = &vidsw->subdev;
+
+	v4l2_async_unregister_subdev(sd);
+	media_entity_cleanup(&sd->entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct of_device_id vidsw_dt_ids[] = {
+	{ .compatible = "video-multiplexer", },
+	{ /* sentinel */ }
+};
+
+static struct platform_driver vidsw_driver = {
+	.probe		= vidsw_probe,
+	.remove		= vidsw_remove,
+	.driver		= {
+		.of_match_table = vidsw_dt_ids,
+		.name = "video-multiplexer",
+	},
+};
+
+module_platform_driver(vidsw_driver);
+
+MODULE_DESCRIPTION("video stream multiplexer");
+MODULE_AUTHOR("Sascha Hauer, Pengutronix");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

From: Philipp Zabel <p.zabel@pengutronix.de>

This driver can handle SoC internal and external video bus multiplexers,
controlled either by register bit fields or by a GPIO. The subdevice
passes through frame interval and mbus configuration of the active input
to the output side.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
  should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev()
  to vidsw_remove().

- there was a line left over from a previous iteration that negated
  the new way of determining the pad count just before it which
  has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding
  to the subdev async notifier waiting list using a chaining method
  from the async registered callbacks (v4l2_of_subdev_registered()
  and the prep patches for that). For now, I've removed the use of
  v4l2_of_subdev_registered() for the vidmux driver's registered
  callback. This doesn't affect the functionality of this driver,
  but allows for it to be merged now, before adding the chaining
  support.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c

diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
new file mode 100644
index 0000000..9d133d9
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port at 0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index d944421..65614b5 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
 	  To compile this driver as a module, choose M here: the
 	  module will be called arv.
 
+config VIDEO_MULTIPLEXER
+	tristate "Video Multiplexer"
+	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+	help
+	  This driver provides support for SoC internal N:1 video bus
+	  multiplexers controlled by register bitfields as well as external
+	  2:1 video multiplexers controlled by a single GPIO.
+
 config VIDEO_OMAP3
 	tristate "OMAP 3 Camera support"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index 5b3cb27..7cf0ee5 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
 
 obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
 
+obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
+
 obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
 obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
new file mode 100644
index 0000000..48980c4
--- /dev/null
+++ b/drivers/media/platform/video-multiplexer.c
@@ -0,0 +1,472 @@
+/*
+ * video stream multiplexer controlled via gpio or syscon
+ *
+ * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
+ * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+
+struct vidsw {
+	struct v4l2_subdev subdev;
+	unsigned int num_pads;
+	struct media_pad *pads;
+	struct v4l2_mbus_framefmt *format_mbus;
+	struct v4l2_fract timeperframe;
+	struct v4l2_of_endpoint *endpoint;
+	struct regmap_field *field;
+	struct gpio_desc *gpio;
+	int active;
+};
+
+static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct vidsw, subdev);
+}
+
+static void vidsw_set_active(struct vidsw *vidsw, int active)
+{
+	vidsw->active = active;
+	if (active < 0)
+		return;
+
+	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
+
+	if (vidsw->field)
+		regmap_field_write(vidsw->field, active);
+	else if (vidsw->gpio)
+		gpiod_set_value(vidsw->gpio, active);
+}
+
+static int vidsw_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	/* We have no limitations on enabling or disabling our output link */
+	if (local->index == vidsw->num_pads - 1)
+		return 0;
+
+	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+		if (local->index == vidsw->active) {
+			dev_dbg(sd->dev, "going inactive\n");
+			vidsw->active = -1;
+		}
+		return 0;
+	}
+
+	if (vidsw->active >= 0) {
+		struct media_pad *pad;
+
+		if (vidsw->active == local->index)
+			return 0;
+
+		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+		if (pad) {
+			struct media_link *link;
+			int ret;
+
+			link = media_entity_find_link(pad,
+						&vidsw->pads[vidsw->active]);
+			if (link) {
+				ret = __media_entity_setup_link(link, 0);
+				if (ret)
+					return ret;
+			}
+		}
+	}
+
+	vidsw_set_active(vidsw, local->index);
+
+	return 0;
+}
+
+static struct media_entity_operations vidsw_ops = {
+	.link_setup = vidsw_link_setup,
+};
+
+static bool vidsw_endpoint_disabled(struct device_node *ep)
+{
+	struct device_node *rpp;
+
+	if (!of_device_is_available(ep))
+		return true;
+
+	rpp = of_graph_get_remote_port_parent(ep);
+	if (!rpp)
+		return true;
+
+	return !of_device_is_available(rpp);
+}
+
+static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
+{
+	struct device_node *ep;
+	u32 portno;
+	int numports;
+	int ret;
+	int i;
+	bool active_link = false;
+
+	numports = vidsw->num_pads;
+
+	for (i = 0; i < numports - 1; i++)
+		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
+	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
+
+	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
+	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
+				     vidsw->pads);
+	if (ret < 0)
+		return ret;
+
+	vidsw->subdev.entity.ops = &vidsw_ops;
+
+	for_each_endpoint_of_node(node, ep) {
+		struct v4l2_of_endpoint endpoint;
+
+		v4l2_of_parse_endpoint(ep, &endpoint);
+
+		portno = endpoint.base.port;
+		if (portno >= numports - 1)
+			continue;
+
+		if (vidsw_endpoint_disabled(ep)) {
+			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
+			continue;
+		}
+
+		vidsw->endpoint[portno] = endpoint;
+
+		if (portno == vidsw->active)
+			active_link = true;
+	}
+
+	for (portno = 0; portno < numports - 1; portno++) {
+		if (!vidsw->endpoint[portno].base.local_node)
+			continue;
+
+		/* If the active input is not connected, use another */
+		if (!active_link) {
+			vidsw_set_active(vidsw, portno);
+			active_link = true;
+		}
+	}
+
+	return v4l2_async_register_subdev(&vidsw->subdev);
+}
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source configuration\n");
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
+	}
+
+	return 0;
+}
+
+static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_subdev *upstream_sd;
+	struct media_pad *pad;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "Can not start streaming on inactive mux\n");
+		return -EINVAL;
+	}
+
+	pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
+	if (!pad) {
+		dev_err(sd->dev, "Failed to find remote source pad\n");
+		return -ENOLINK;
+	}
+
+	if (!is_media_entity_v4l2_subdev(pad->entity)) {
+		dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
+		return -ENODEV;
+	}
+
+	upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
+
+	return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
+}
+
+static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	fi->interval = vidsw->timeperframe;
+
+	return 0;
+}
+
+static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	vidsw->timeperframe = fi->interval;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
+	.g_mbus_config = vidsw_g_mbus_config,
+	.s_stream = vidsw_s_stream,
+	.g_frame_interval = vidsw_g_frame_interval,
+	.s_frame_interval = vidsw_s_frame_interval,
+};
+
+static struct v4l2_mbus_framefmt *
+__vidsw_get_pad_format(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       unsigned int pad, u32 which)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	switch (which) {
+	case V4L2_SUBDEV_FORMAT_TRY:
+		return v4l2_subdev_get_try_format(sd, cfg, pad);
+	case V4L2_SUBDEV_FORMAT_ACTIVE:
+		return &vidsw->format_mbus[pad];
+	default:
+		return NULL;
+	}
+}
+
+static int vidsw_get_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
+						   sdformat->which);
+	return 0;
+}
+
+static int vidsw_set_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_mbus_framefmt *mbusformat;
+
+	if (sdformat->pad >= vidsw->num_pads)
+		return -EINVAL;
+
+	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
+					    sdformat->which);
+	if (!mbusformat)
+		return -EINVAL;
+
+	/* Output pad mirrors active input pad, no limitations on input pads */
+	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
+		sdformat->format = vidsw->format_mbus[vidsw->active];
+
+	*mbusformat = sdformat->format;
+
+	return 0;
+}
+
+static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
+	.get_fmt = vidsw_get_format,
+	.set_fmt = vidsw_set_format,
+};
+
+static struct v4l2_subdev_ops vidsw_subdev_ops = {
+	.pad = &vidsw_pad_ops,
+	.video = &vidsw_subdev_video_ops,
+};
+
+static int of_get_reg_field(struct device_node *node, struct reg_field *field)
+{
+	u32 bit_mask;
+	int ret;
+
+	ret = of_property_read_u32(node, "reg", &field->reg);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
+	if (ret < 0)
+		return ret;
+
+	field->msb = field->lsb + fls(bit_mask) - 1;
+
+	return 0;
+}
+
+static int vidsw_probe(struct platform_device *pdev)
+{
+	struct device_node *np = pdev->dev.of_node;
+	struct of_endpoint endpoint;
+	struct device_node *ep;
+	struct reg_field field;
+	struct vidsw *vidsw;
+	struct regmap *map;
+	unsigned int num_pads;
+	int ret;
+
+	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
+	if (!vidsw)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, vidsw);
+
+	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
+	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
+			np->name);
+	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	vidsw->subdev.dev = &pdev->dev;
+
+	/*
+	 * The largest numbered port is the output port. It determines
+	 * total number of pads
+	 */
+	num_pads = 0;
+	for_each_endpoint_of_node(np, ep) {
+		of_graph_parse_endpoint(ep, &endpoint);
+		num_pads = max(num_pads, endpoint.port + 1);
+	}
+
+	if (num_pads < 2) {
+		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
+		return -EINVAL;
+	}
+
+	ret = of_get_reg_field(np, &field);
+	if (ret == 0) {
+		map = syscon_node_to_regmap(np->parent);
+		if (!map) {
+			dev_err(&pdev->dev, "Failed to get syscon register map\n");
+			return PTR_ERR(map);
+		}
+
+		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
+		if (IS_ERR(vidsw->field)) {
+			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
+			return PTR_ERR(vidsw->field);
+		}
+
+		regmap_field_read(vidsw->field, &vidsw->active);
+	} else {
+		if (num_pads > 3) {
+			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
+			return -EINVAL;
+		}
+
+		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
+		if (IS_ERR(vidsw->gpio)) {
+			dev_warn(&pdev->dev,
+				 "could not request control gpio: %d\n", ret);
+			vidsw->gpio = NULL;
+		}
+
+		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
+	}
+
+	vidsw->num_pads = num_pads;
+	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
+			GFP_KERNEL);
+	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
+	vidsw->endpoint = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
+
+	ret = vidsw_async_init(vidsw, np);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int vidsw_remove(struct platform_device *pdev)
+{
+	struct vidsw *vidsw = platform_get_drvdata(pdev);
+	struct v4l2_subdev *sd = &vidsw->subdev;
+
+	v4l2_async_unregister_subdev(sd);
+	media_entity_cleanup(&sd->entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct of_device_id vidsw_dt_ids[] = {
+	{ .compatible = "video-multiplexer", },
+	{ /* sentinel */ }
+};
+
+static struct platform_driver vidsw_driver = {
+	.probe		= vidsw_probe,
+	.remove		= vidsw_remove,
+	.driver		= {
+		.of_match_table = vidsw_dt_ids,
+		.name = "video-multiplexer",
+	},
+};
+
+module_platform_driver(vidsw_driver);
+
+MODULE_DESCRIPTION("video stream multiplexer");
+MODULE_AUTHOR("Sascha Hauer, Pengutronix");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 14/24] UAPI: Add media UAPI Kbuild file
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Add an empty UAPI Kbuild file for media UAPI headers.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 include/uapi/Kbuild       | 1 +
 include/uapi/media/Kbuild | 1 +
 2 files changed, 2 insertions(+)
 create mode 100644 include/uapi/media/Kbuild

diff --git a/include/uapi/Kbuild b/include/uapi/Kbuild
index 245aa6e..9a51957 100644
--- a/include/uapi/Kbuild
+++ b/include/uapi/Kbuild
@@ -6,6 +6,7 @@
 header-y += asm-generic/
 header-y += linux/
 header-y += sound/
+header-y += media/
 header-y += mtd/
 header-y += rdma/
 header-y += video/
diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
new file mode 100644
index 0000000..aafaa5a
--- /dev/null
+++ b/include/uapi/media/Kbuild
@@ -0,0 +1 @@
+# UAPI Header export list
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 14/24] UAPI: Add media UAPI Kbuild file
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

Add an empty UAPI Kbuild file for media UAPI headers.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 include/uapi/Kbuild       | 1 +
 include/uapi/media/Kbuild | 1 +
 2 files changed, 2 insertions(+)
 create mode 100644 include/uapi/media/Kbuild

diff --git a/include/uapi/Kbuild b/include/uapi/Kbuild
index 245aa6e..9a51957 100644
--- a/include/uapi/Kbuild
+++ b/include/uapi/Kbuild
@@ -6,6 +6,7 @@
 header-y += asm-generic/
 header-y += linux/
 header-y += sound/
+header-y += media/
 header-y += mtd/
 header-y += rdma/
 header-y += video/
diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
new file mode 100644
index 0000000..aafaa5a
--- /dev/null
+++ b/include/uapi/media/Kbuild
@@ -0,0 +1 @@
+# UAPI Header export list
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 14/24] UAPI: Add media UAPI Kbuild file
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Add an empty UAPI Kbuild file for media UAPI headers.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 include/uapi/Kbuild       | 1 +
 include/uapi/media/Kbuild | 1 +
 2 files changed, 2 insertions(+)
 create mode 100644 include/uapi/media/Kbuild

diff --git a/include/uapi/Kbuild b/include/uapi/Kbuild
index 245aa6e..9a51957 100644
--- a/include/uapi/Kbuild
+++ b/include/uapi/Kbuild
@@ -6,6 +6,7 @@
 header-y += asm-generic/
 header-y += linux/
 header-y += sound/
+header-y += media/
 header-y += mtd/
 header-y += rdma/
 header-y += video/
diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
new file mode 100644
index 0000000..aafaa5a
--- /dev/null
+++ b/include/uapi/media/Kbuild
@@ -0,0 +1 @@
+# UAPI Header export list
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 15/24] media: Add userspace header file for i.MX
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

This adds a header file for use by userspace programs wanting to interact
with the i.MX media driver. It defines custom v4l2 controls and events
generated by the i.MX v4l2 subdevices.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 include/uapi/media/Kbuild |  1 +
 include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 include/uapi/media/imx.h

diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
index aafaa5a..fa78958 100644
--- a/include/uapi/media/Kbuild
+++ b/include/uapi/media/Kbuild
@@ -1 +1,2 @@
 # UAPI Header export list
+header-y += imx.h
diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
new file mode 100644
index 0000000..2421d9c
--- /dev/null
+++ b/include/uapi/media/imx.h
@@ -0,0 +1,30 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */
+
+#ifndef __UAPI_MEDIA_IMX_H__
+#define __UAPI_MEDIA_IMX_H__
+
+/*
+ * events from the subdevs
+ */
+#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
+#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
+#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
+#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
+
+enum imx_ctrl_id {
+	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
+	V4L2_CID_IMX_FIM_ENABLE,
+	V4L2_CID_IMX_FIM_NUM,
+	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
+	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
+	V4L2_CID_IMX_FIM_NUM_SKIP,
+};
+
+#endif
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

This adds a header file for use by userspace programs wanting to interact
with the i.MX media driver. It defines custom v4l2 controls and events
generated by the i.MX v4l2 subdevices.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 include/uapi/media/Kbuild |  1 +
 include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 include/uapi/media/imx.h

diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
index aafaa5a..fa78958 100644
--- a/include/uapi/media/Kbuild
+++ b/include/uapi/media/Kbuild
@@ -1 +1,2 @@
 # UAPI Header export list
+header-y += imx.h
diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
new file mode 100644
index 0000000..2421d9c
--- /dev/null
+++ b/include/uapi/media/imx.h
@@ -0,0 +1,30 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */
+
+#ifndef __UAPI_MEDIA_IMX_H__
+#define __UAPI_MEDIA_IMX_H__
+
+/*
+ * events from the subdevs
+ */
+#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
+#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
+#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
+#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
+
+enum imx_ctrl_id {
+	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
+	V4L2_CID_IMX_FIM_ENABLE,
+	V4L2_CID_IMX_FIM_NUM,
+	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
+	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
+	V4L2_CID_IMX_FIM_NUM_SKIP,
+};
+
+#endif
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

This adds a header file for use by userspace programs wanting to interact
with the i.MX media driver. It defines custom v4l2 controls and events
generated by the i.MX v4l2 subdevices.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 include/uapi/media/Kbuild |  1 +
 include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 include/uapi/media/imx.h

diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
index aafaa5a..fa78958 100644
--- a/include/uapi/media/Kbuild
+++ b/include/uapi/media/Kbuild
@@ -1 +1,2 @@
 # UAPI Header export list
+header-y += imx.h
diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
new file mode 100644
index 0000000..2421d9c
--- /dev/null
+++ b/include/uapi/media/imx.h
@@ -0,0 +1,30 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */
+
+#ifndef __UAPI_MEDIA_IMX_H__
+#define __UAPI_MEDIA_IMX_H__
+
+/*
+ * events from the subdevs
+ */
+#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
+#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
+#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
+#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
+
+enum imx_ctrl_id {
+	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
+	V4L2_CID_IMX_FIM_ENABLE,
+	V4L2_CID_IMX_FIM_NUM,
+	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
+	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
+	V4L2_CID_IMX_FIM_NUM_SKIP,
+};
+
+#endif
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-07  2:11 ` Steve Longerbeam
                   ` (16 preceding siblings ...)
  (?)
@ 2017-01-07  2:11 ` Steve Longerbeam
  2017-01-13 15:20     ` Philipp Zabel
                     ` (3 more replies)
  -1 siblings, 4 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Add the core media driver for i.MX SOC.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 Documentation/media/v4l-drivers/imx.rst           | 443 ++++++++++
 drivers/staging/media/Kconfig                     |   2 +
 drivers/staging/media/Makefile                    |   1 +
 drivers/staging/media/imx/Kconfig                 |   8 +
 drivers/staging/media/imx/Makefile                |   6 +
 drivers/staging/media/imx/TODO                    |  22 +
 drivers/staging/media/imx/imx-media-common.c      | 981 ++++++++++++++++++++++
 drivers/staging/media/imx/imx-media-dev.c         | 486 +++++++++++
 drivers/staging/media/imx/imx-media-fim.c         | 471 +++++++++++
 drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
 drivers/staging/media/imx/imx-media-of.c          | 289 +++++++
 drivers/staging/media/imx/imx-media.h             | 310 +++++++
 include/media/imx.h                               |  15 +
 include/uapi/linux/v4l2-controls.h                |   4 +
 14 files changed, 3495 insertions(+)
 create mode 100644 Documentation/media/v4l-drivers/imx.rst
 create mode 100644 drivers/staging/media/imx/Kconfig
 create mode 100644 drivers/staging/media/imx/Makefile
 create mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 drivers/staging/media/imx/imx-media-common.c
 create mode 100644 drivers/staging/media/imx/imx-media-dev.c
 create mode 100644 drivers/staging/media/imx/imx-media-fim.c
 create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
 create mode 100644 drivers/staging/media/imx/imx-media-of.c
 create mode 100644 drivers/staging/media/imx/imx-media.h
 create mode 100644 include/media/imx.h

diff --git a/Documentation/media/v4l-drivers/imx.rst b/Documentation/media/v4l-drivers/imx.rst
new file mode 100644
index 0000000..87b37b5
--- /dev/null
+++ b/Documentation/media/v4l-drivers/imx.rst
@@ -0,0 +1,443 @@
+i.MX Video Capture Driver
+=========================
+
+Introduction
+------------
+
+The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
+handles the flow of image frames to and from capture devices and
+display devices.
+
+For image capture, the IPU contains the following internal subunits:
+
+- Image DMA Controller (IDMAC)
+- Camera Serial Interface (CSI)
+- Image Converter (IC)
+- Sensor Multi-FIFO Controller (SMFC)
+- Image Rotator (IRT)
+- Video De-Interlace Controller (VDIC)
+
+The IDMAC is the DMA controller for transfer of image frames to and from
+memory. Various dedicated DMA channels exist for both video capture and
+display paths.
+
+The CSI is the frontend capture unit that interfaces directly with
+capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
+
+The IC handles color-space conversion, resizing, and rotation
+operations. There are three independent "tasks" within the IC that can
+carry out conversions concurrently: pre-processing encoding,
+pre-processing preview, and post-processing.
+
+The SMFC is composed of four independent channels that each can transfer
+captured frames from sensors directly to memory concurrently.
+
+The IRT carries out 90 and 270 degree image rotation operations.
+
+The VDIC handles the conversion of interlaced video to progressive, with
+support for different motion compensation modes (low, medium, and high
+motion). The deinterlaced output frames from the VDIC can be sent to the
+IC pre-process preview task for further conversions.
+
+In addition to the IPU internal subunits, there are also two units
+outside the IPU that are also involved in video capture on i.MX:
+
+- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
+  interface. This is a Synopsys DesignWare core.
+- A video multiplexer for selecting among multiple sensor inputs to
+  send to a CSI.
+
+For more info, refer to the latest versions of the i.MX5/6 reference
+manuals listed under References.
+
+
+Features
+--------
+
+Some of the features of this driver include:
+
+- Many different pipelines can be configured via media controller API,
+  that correspond to the hardware video capture pipelines supported in
+  the i.MX.
+
+- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
+
+- Up to four concurrent sensor acquisitions, by configuring each
+  sensor's pipeline using independent entities. This is currently
+  demonstrated with the SabreSD and SabreLite reference boards with
+  independent OV5642 and MIPI CSI-2 OV5640 sensor modules.
+
+- Scaling, color-space conversion, and image rotation via IC task
+  subdevs.
+
+- Many pixel formats supported (RGB, packed and planar YUV, partial
+  planar YUV).
+
+- The IC pre-process preview subdev supports motion compensated
+  de-interlacing using the VDIC, with three motion compensation modes:
+  low, medium, and high motion. The mode is specified with a custom
+  control. Pipelines are defined that allow sending frames to the
+  preview subdev directly from the CSI or from the SMFC.
+
+- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
+  problems with the ADV718x video decoders. See below for a description
+  of the FIM.
+
+
+Capture Pipelines
+-----------------
+
+The following describe the various use-cases supported by the pipelines.
+
+The links shown do not include the frontend sensor, video mux, or mipi
+csi-2 receiver links. This depends on the type of sensor interface
+(parallel or mipi csi-2). So in all cases, these pipelines begin with:
+
+sensor -> ipu_csi_mux -> ipu_csi -> ...
+
+for parallel sensors, or:
+
+sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
+
+for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
+to the video mux (ipu_csi_mux) before sending to the CSI, depending
+on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
+parenthesis.
+
+Unprocessed Video Capture:
+--------------------------
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif
+
+Note the ipu_smfc can do pixel reordering within the same colorspace.
+For example, its sink pad can take UYVY2X8, but its source pad can
+output YUYV2X8.
+
+IC Direct Conversions:
+----------------------
+
+This pipeline uses the preprocess encode entity to route frames directly
+from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
+1024x1024 resolution, CSC, and image rotation:
+
+-> ipu_ic_prpenc -> camif
+
+This can be a useful capture pipeline for heavily loaded memory bus
+traffic environments, since it has minimal IDMAC channel usage.
+
+Post-Processing Conversions:
+----------------------------
+
+This pipeline routes frames from the SMFC to the post-processing
+entity. In addition to CSC and rotation, this entity supports tiling
+which allows scaled output beyond the 1024x1024 limitation of the IC
+(up to 4096x4096 scaling output is supported):
+
+-> ipu_smfc -> ipu_ic_pp -> camif
+
+Motion Compensated De-interlace:
+--------------------------------
+
+This pipeline routes frames from the SMFC to the preprocess preview
+entity to support motion-compensated de-interlacing using the VDIC,
+scaling up to 1024x1024, and CSC:
+
+-> ipu_smfc -> ipu_ic_prpvf -> camif
+
+This pipeline also carries out the same conversions as above, but routes
+frames directly from the CSI to the IC preprocess preview entity for
+minimal memory bandwidth usage (note: this pipeline only works in
+"high motion" mode):
+
+-> ipu_ic_prpvf -> camif
+
+This pipeline takes the motion-compensated de-interlaced frames and
+sends them to the post-processor, to support motion-compensated
+de-interlacing, scaling up to 4096x4096, CSC, and rotation:
+
+-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif
+
+
+Usage Notes
+-----------
+
+Many of the subdevs require information from the active sensor in the
+current pipeline when configuring pad formats. Therefore the media links
+should be established before configuring the media pad formats.
+
+Similarly, the capture v4l2 interface subdev inherits controls from the
+active subdevs in the current pipeline at link-setup time. Therefore the
+capture links should be the last links established in order for capture
+to "see" and inherit all possible controls.
+
+The following are usage notes for Sabre- reference platforms:
+
+
+SabreLite with OV5642 and OV5640
+--------------------------------
+
+This platform requires the OmniVision OV5642 module with a parallel
+camera interface, and the OV5640 module with a MIPI CSI-2
+interface. Both modules are available from Boundary Devices:
+
+https://boundarydevices.com/products/nit6x_5mp
+https://boundarydevices.com/product/nit6x_5mp_mipi
+
+Note that if only one camera module is available, the other sensor
+node can be disabled in the device tree.
+
+The OV5642 module is connected to the parallel bus input on the i.MX
+internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
+
+The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
+receiver, and the four virtual channel outputs from the receiver are
+routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
+vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
+also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
+OV5640 must not share the same i2c slave address.
+
+The following basic example configures unprocessed video capture
+pipelines for both sensors. The OV5642 is routed to camif0
+(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
+virtual channel 1) is routed to camif1 (usually /dev/video1). Both
+sensors are configured to output 640x480, UYVY (not shown: all pad
+field types should be set to "NONE"):
+
+.. code-block:: none
+
+   # Setup links for OV5642
+   media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
+   media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Setup links for OV5640
+   media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
+   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
+   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
+   media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
+   media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
+   # Configure pads for OV5642 pipeline
+   media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
+   # Configure pads for OV5640 pipeline
+   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc1\":1 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
+
+Streaming can then begin independently on device nodes /dev/video0
+and /dev/video1.
+
+SabreAuto with ADV7180 decoder
+------------------------------
+
+On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
+parallel bus input on the internal video mux to IPU1 CSI0.
+
+The following example configures a pipeline to capture from the ADV7180
+video decoder, assuming NTSC 720x480 input signals, with Motion
+Compensated de-interlacing (not shown: all pad field types should be set
+as indicated). $outputfmt can be any format supported by the
+ipu1_ic_prpvf entity at its output pad:
+
+.. code-block:: none
+
+   # Setup links
+   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
+   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
+   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Configure pads
+   # pad field types for below pads must be an interlaced type
+   # such as "ALTERNATE"
+   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
+   # pad field types for below pads must be "NONE"
+   media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
+
+Streaming can then begin on /dev/video0.
+
+This platform accepts Composite Video analog inputs to the ADV7180 on
+Ain1 (connector J42) and Ain3 (connector J43).
+
+To switch to Ain1:
+
+.. code-block:: none
+
+   # v4l2-ctl -i0
+
+To switch to Ain3:
+
+.. code-block:: none
+
+   # v4l2-ctl -i1
+
+
+Frame Interval Monitor
+----------------------
+
+The adv718x decoders can occasionally send corrupt fields during
+NTSC/PAL signal re-sync (too little or too many video lines). When
+this happens, the IPU triggers a mechanism to re-establish vertical
+sync by adding 1 dummy line every frame, which causes a rolling effect
+from image to image, and can last a long time before a stable image is
+recovered. Or sometimes the mechanism doesn't work at all, causing a
+permanent split image (one frame contains lines from two consecutive
+captured images).
+
+From experiment it was found that during image rolling, the frame
+intervals (elapsed time between two EOF's) drop below the nominal
+value for the current standard, by about one frame time (60 usec),
+and remain at that value until rolling stops.
+
+While the reason for this observation isn't known (the IPU dummy
+line mechanism should show an increase in the intervals by 1 line
+time every frame, not a fixed value), we can use it to detect the
+corrupt fields using a frame interval monitor. If the FIM detects a
+bad frame interval, a subdev event is sent. In response, userland can
+issue a streaming restart to correct the rolling/split image.
+
+The FIM is implemented in the imx-csi entity, and the entities that have
+direct connections to the CSI call into the FIM to monitor the frame
+intervals: ipu_smfc, ipu_ic_prpenc, and ipu_prpvf (when configured with
+a direct link from ipu_csi). Userland can register with the FIM event
+notifications on the imx-csi subdev device node
+(V4L2_EVENT_IMX_FRAME_INTERVAL).
+
+The imx-csi entity includes custom controls to tweak some dials for FIM.
+If one of these controls is changed during streaming, the FIM will be
+reset and will continue at the new settings.
+
+- V4L2_CID_IMX_FIM_ENABLE
+
+Enable/disable the FIM.
+
+- V4L2_CID_IMX_FIM_NUM
+
+How many frame interval errors to average before comparing against the
+nominal frame interval reported by the sensor. This can reduce noise
+from interrupt latency.
+
+- V4L2_CID_IMX_FIM_TOLERANCE_MIN
+
+If the averaged intervals fall outside nominal by this amount, in
+microseconds, streaming will be restarted.
+
+- V4L2_CID_IMX_FIM_TOLERANCE_MAX
+
+If any interval errors are higher than this value, those error samples
+are discarded and do not enter into the average. This can be used to
+discard really high interval errors that might be due to very high
+system load, causing excessive interrupt latencies.
+
+- V4L2_CID_IMX_FIM_NUM_SKIP
+
+How many frames to skip after a FIM reset or stream restart before
+FIM begins to average intervals. It has been found that there can
+be a few bad frame intervals after stream restart which are not
+attributed to adv718x sending a corrupt field, so this is used to
+skip those frames to prevent unnecessary restarts.
+
+
+SabreSD with MIPI CSI-2 OV5640
+------------------------------
+
+Similarly to SabreLite, the SabreSD supports a parallel interface
+OV5642 module on IPU1 CSI0, and a MIPI CSI-2 OV5640 module. The OV5642
+connects to i2c bus 1 and the OV5640 to i2c bus 2.
+
+The device tree for SabreSD includes OF graphs for both the parallel
+OV5642 and the MIPI CSI-2 OV5640, but as of this writing only the MIPI
+CSI-2 OV5640 has been tested, so the OV5642 node is currently disabled.
+The OV5640 module connects to MIPI connector J5 (sorry I don't have the
+compatible module part number or URL).
+
+The following example configures a post-processing pipeline to capture
+from the OV5640 (not shown: all pad field types should be set to
+"NONE"). $sensorfmt can be any format supported by the
+OV5640. $outputfmt can be any format supported by the ipu1_ic_pp1
+entity at its output pad:
+
+
+.. code-block:: none
+
+   # Setup links
+   media-ctl -l '"ov5640_mipi 1-003c":0 -> "imx-mipi-csi2":0[1]'
+   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
+   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
+   media-ctl -l '"ipu1_smfc1":1 -> "ipu1_ic_pp1":0[1]'
+   media-ctl -l '"ipu1_ic_pp1":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Configure pads
+   media-ctl -V "\"ov5640_mipi 1-003c\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_smfc1\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_smfc1\":1 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_ic_pp1\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_ic_pp1\":1 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
+
+Streaming can then begin on /dev/video0.
+
+
+
+Known Issues
+------------
+
+1. When using 90 or 270 degree rotation control at capture resolutions
+   near the IC resizer limit of 1024x1024, and combined with planar
+   pixel formats (YUV420, YUV422p), frame capture will often fail with
+   no end-of-frame interrupts from the IDMAC channel. To work around
+   this, use lower resolution and/or packed formats (YUYV, RGB3, etc.)
+   when 90 or 270 rotations are needed.
+
+
+File list
+---------
+
+drivers/staging/media/imx/
+include/media/imx.h
+include/uapi/media/imx.h
+
+References
+----------
+
+[1] "i.MX 6Dual/6Quad Applications Processor Reference Manual"
+[2] "i.MX 6Solo/6DualLite Applications Processor Reference Manual"
+
+
+Author
+------
+Steve Longerbeam <steve_longerbeam@mentor.com>
+
+Copyright (C) 2012-2016 Mentor Graphics Inc.
diff --git a/drivers/staging/media/Kconfig b/drivers/staging/media/Kconfig
index ffb8fa7..05b55a8 100644
--- a/drivers/staging/media/Kconfig
+++ b/drivers/staging/media/Kconfig
@@ -25,6 +25,8 @@ source "drivers/staging/media/cxd2099/Kconfig"
 
 source "drivers/staging/media/davinci_vpfe/Kconfig"
 
+source "drivers/staging/media/imx/Kconfig"
+
 source "drivers/staging/media/omap4iss/Kconfig"
 
 source "drivers/staging/media/s5p-cec/Kconfig"
diff --git a/drivers/staging/media/Makefile b/drivers/staging/media/Makefile
index a28e82c..6f50ddd 100644
--- a/drivers/staging/media/Makefile
+++ b/drivers/staging/media/Makefile
@@ -1,6 +1,7 @@
 obj-$(CONFIG_I2C_BCM2048)	+= bcm2048/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_CEC) += s5p-cec/
 obj-$(CONFIG_DVB_CXD2099)	+= cxd2099/
+obj-$(CONFIG_VIDEO_IMX_MEDIA)	+= imx/
 obj-$(CONFIG_LIRC_STAGING)	+= lirc/
 obj-$(CONFIG_VIDEO_DM365_VPFE)	+= davinci_vpfe/
 obj-$(CONFIG_VIDEO_OMAP4)	+= omap4iss/
diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
new file mode 100644
index 0000000..bfde58d
--- /dev/null
+++ b/drivers/staging/media/imx/Kconfig
@@ -0,0 +1,8 @@
+config VIDEO_IMX_MEDIA
+	tristate "i.MX5/6 V4L2 media core driver"
+	depends on MEDIA_CONTROLLER && VIDEO_V4L2 && ARCH_MXC && IMX_IPUV3_CORE
+	default y
+	---help---
+	  Say yes here to enable support for video4linux media controller
+	  driver for the i.MX5/6 SOC.
+
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
new file mode 100644
index 0000000..ef9f11b
--- /dev/null
+++ b/drivers/staging/media/imx/Makefile
@@ -0,0 +1,6 @@
+imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
+	imx-media-of.o
+
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
+
diff --git a/drivers/staging/media/imx/TODO b/drivers/staging/media/imx/TODO
new file mode 100644
index 0000000..1f42381
--- /dev/null
+++ b/drivers/staging/media/imx/TODO
@@ -0,0 +1,22 @@
+
+- v4l2-compliance
+
+- imx-csi subdev is not being autoloaded as a kernel module, probably
+  because ipu_add_client_devices() does not register the IPU client
+  platform devices, but only allocates those devices.
+
+- Verify driver remove paths.
+
+- Currently registering with notifications from subdevs are only
+  available through the subdev device nodes and not through the main
+  capture device node. Need to come up with a way to find the camif in
+  the current pipeline that owns the subdev that sent the notify.
+
+- Need to decide whether a mem2mem device should be incorporated into
+  the media graph, or whether it should be a separate device that does
+  not link with any other entities.
+
+- Combine, clean up, and move the ov5640/ov5642 subdevs to
+  drivers/media/i2c. Once that is done the binding docs for ov564x
+  can be created under Documentation/devicetree/bindings/media/i2c.
+
diff --git a/drivers/staging/media/imx/imx-media-common.c b/drivers/staging/media/imx/imx-media-common.c
new file mode 100644
index 0000000..f19ffcf
--- /dev/null
+++ b/drivers/staging/media/imx/imx-media-common.c
@@ -0,0 +1,981 @@
+/*
+ * V4L2 Media Controller Driver for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include "imx-media.h"
+
+/*
+ * List of pixel formats for the subdevs. This must be a super-set of
+ * the formats supported by the ipu image converter.
+ */
+static const struct imx_media_pixfmt imx_media_formats[] = {
+	{
+		.fourcc	= V4L2_PIX_FMT_UYVY,
+		.codes  = {MEDIA_BUS_FMT_UYVY8_2X8, MEDIA_BUS_FMT_UYVY8_1X16},
+		.cs     = IPUV3_COLORSPACE_YUV,
+		.bpp    = 16,
+	}, {
+		.fourcc	= V4L2_PIX_FMT_YUYV,
+		.codes  = {MEDIA_BUS_FMT_YUYV8_2X8, MEDIA_BUS_FMT_YUYV8_1X16},
+		.cs     = IPUV3_COLORSPACE_YUV,
+		.bpp    = 16,
+	}, {
+		.fourcc	= V4L2_PIX_FMT_RGB565,
+		.codes  = {MEDIA_BUS_FMT_RGB565_2X8_LE},
+		.cs     = IPUV3_COLORSPACE_RGB,
+		.bpp    = 16,
+	}, {
+		.fourcc	= V4L2_PIX_FMT_RGB24,
+		.codes  = {MEDIA_BUS_FMT_RGB888_1X24,
+			   MEDIA_BUS_FMT_RGB888_2X12_LE},
+		.cs     = IPUV3_COLORSPACE_RGB,
+		.bpp    = 24,
+	}, {
+		.fourcc	= V4L2_PIX_FMT_BGR24,
+		.cs     = IPUV3_COLORSPACE_RGB,
+		.bpp    = 24,
+	}, {
+		.fourcc	= V4L2_PIX_FMT_RGB32,
+		.codes  = {MEDIA_BUS_FMT_ARGB8888_1X32},
+		.cs     = IPUV3_COLORSPACE_RGB,
+		.bpp    = 32,
+	}, {
+		.fourcc	= V4L2_PIX_FMT_BGR32,
+		.cs     = IPUV3_COLORSPACE_RGB,
+		.bpp    = 32,
+	}, {
+		.fourcc	= V4L2_PIX_FMT_YUV420,
+		.cs     = IPUV3_COLORSPACE_YUV,
+		.bpp    = 12,
+		.planar = true,
+	}, {
+		.fourcc = V4L2_PIX_FMT_YVU420,
+		.cs     = IPUV3_COLORSPACE_YUV,
+		.bpp    = 12,
+		.planar = true,
+	}, {
+		.fourcc = V4L2_PIX_FMT_YUV422P,
+		.cs     = IPUV3_COLORSPACE_YUV,
+		.bpp    = 16,
+		.planar = true,
+	}, {
+		.fourcc = V4L2_PIX_FMT_NV12,
+		.cs     = IPUV3_COLORSPACE_YUV,
+		.bpp    = 12,
+		.planar = true,
+	}, {
+		.fourcc = V4L2_PIX_FMT_NV16,
+		.cs     = IPUV3_COLORSPACE_YUV,
+		.bpp    = 16,
+		.planar = true,
+	},
+};
+
+const struct imx_media_pixfmt *imx_media_find_format(u32 fourcc, u32 code,
+						     bool allow_rgb,
+						     bool allow_planar)
+{
+	const struct imx_media_pixfmt *fmt, *ret = NULL;
+	int i, j;
+
+	for (i = 0; i < ARRAY_SIZE(imx_media_formats); i++) {
+		fmt = &imx_media_formats[i];
+
+		if (fourcc && fmt->fourcc == fourcc &&
+		    (fmt->cs != IPUV3_COLORSPACE_RGB || allow_rgb) &&
+		    (!fmt->planar || (allow_planar && fmt->codes[0]))) {
+			ret = fmt;
+			goto out;
+		}
+
+		for (j = 0; fmt->codes[j]; j++) {
+			if (fmt->codes[j] == code &&
+			    (fmt->cs != IPUV3_COLORSPACE_RGB || allow_rgb) &&
+			    (!fmt->planar || allow_planar)) {
+				ret = fmt;
+				goto out;
+			}
+		}
+	}
+out:
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_find_format);
+
+int imx_media_enum_format(u32 *code, u32 index, bool allow_rgb,
+			  bool allow_planar)
+{
+	const struct imx_media_pixfmt *fmt;
+
+	if (index >= ARRAY_SIZE(imx_media_formats))
+		return -EINVAL;
+
+	fmt = &imx_media_formats[index];
+	if ((fmt->cs == IPUV3_COLORSPACE_RGB && !allow_rgb) ||
+	    (fmt->planar && (!allow_planar || !fmt->codes[0])))
+		return -EINVAL;
+
+	*code = fmt->codes[0];
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_enum_format);
+
+int imx_media_init_mbus_fmt(struct v4l2_mbus_framefmt *mbus,
+			    u32 width, u32 height, u32 code, u32 field,
+			    const struct imx_media_pixfmt **cc)
+{
+	const struct imx_media_pixfmt *lcc;
+
+	mbus->width = width;
+	mbus->height = height;
+	mbus->field = field;
+	if (code == 0)
+		imx_media_enum_format(&code, 0, true, true);
+	lcc = imx_media_find_format(0, code, true, true);
+	if (!lcc)
+		return -EINVAL;
+	mbus->code = code;
+
+	if (cc)
+		*cc = lcc;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_init_mbus_fmt);
+
+int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix,
+				  struct v4l2_mbus_framefmt *mbus)
+{
+	const struct imx_media_pixfmt *fmt;
+	u32 stride;
+
+	fmt = imx_media_find_format(0, mbus->code, true, true);
+	if (!fmt)
+		return -EINVAL;
+
+	stride = fmt->planar ? mbus->width : (mbus->width * fmt->bpp) >> 3;
+
+	pix->width = mbus->width;
+	pix->height = mbus->height;
+	pix->pixelformat = fmt->fourcc;
+	pix->field = mbus->field;
+	pix->bytesperline = stride;
+	pix->sizeimage = (pix->width * pix->height * fmt->bpp) >> 3;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_mbus_fmt_to_pix_fmt);
+
+int imx_media_mbus_fmt_to_ipu_image(struct ipu_image *image,
+				    struct v4l2_mbus_framefmt *mbus)
+{
+	int ret;
+
+	memset(image, 0, sizeof(*image));
+
+	ret = imx_media_mbus_fmt_to_pix_fmt(&image->pix, mbus);
+	if (ret)
+		return ret;
+
+	image->rect.width = mbus->width;
+	image->rect.height = mbus->height;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_mbus_fmt_to_ipu_image);
+
+int imx_media_ipu_image_to_mbus_fmt(struct v4l2_mbus_framefmt *mbus,
+				    struct ipu_image *image)
+{
+	const struct imx_media_pixfmt *fmt;
+
+	fmt = imx_media_find_format(image->pix.pixelformat, 0, true, true);
+	if (!fmt)
+		return -EINVAL;
+
+	memset(mbus, 0, sizeof(*mbus));
+	mbus->width = image->pix.width;
+	mbus->height = image->pix.height;
+	mbus->code = fmt->codes[0];
+	mbus->field = image->pix.field;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_ipu_image_to_mbus_fmt);
+
+/*
+ * DMA buffer ring handling
+ */
+struct imx_media_dma_buf_ring {
+	struct imx_media_dev *imxmd;
+
+	/* the ring */
+	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
+	/* the scratch buffer for underruns */
+	struct imx_media_dma_buf scratch;
+
+	/* buffer generator */
+	struct media_entity *src;
+	/* buffer receiver */
+	struct media_entity *sink;
+
+	spinlock_t lock;
+
+	int num_bufs;
+	unsigned long last_seq;
+};
+
+void imx_media_free_dma_buf(struct imx_media_dev *imxmd,
+			    struct imx_media_dma_buf *buf)
+{
+	if (buf->virt && !buf->vb)
+		dma_free_coherent(imxmd->dev, buf->len, buf->virt, buf->phys);
+
+	buf->virt = NULL;
+	buf->phys = 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_free_dma_buf);
+
+int imx_media_alloc_dma_buf(struct imx_media_dev *imxmd,
+			    struct imx_media_dma_buf *buf,
+			    int size)
+{
+	imx_media_free_dma_buf(imxmd, buf);
+
+	buf->ring = NULL;
+	buf->vb = NULL;
+	buf->len = PAGE_ALIGN(size);
+	buf->virt = dma_alloc_coherent(imxmd->dev, buf->len, &buf->phys,
+				       GFP_DMA | GFP_KERNEL);
+	if (!buf->virt) {
+		dev_err(imxmd->dev, "failed to alloc dma buffer\n");
+		return -ENOMEM;
+	}
+
+	buf->state = IMX_MEDIA_BUF_STATUS_PREPARED;
+	buf->seq = 0;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_alloc_dma_buf);
+
+void imx_media_free_dma_buf_ring(struct imx_media_dma_buf_ring *ring)
+{
+	int i;
+
+	if (!ring)
+		return;
+
+	dev_dbg(ring->imxmd->dev, "freeing ring [%s -> %s]\n",
+		ring->src->name, ring->sink->name);
+
+	imx_media_free_dma_buf(ring->imxmd, &ring->scratch);
+
+	for (i = 0; i < ring->num_bufs; i++)
+		imx_media_free_dma_buf(ring->imxmd, &ring->buf[i]);
+	kfree(ring);
+}
+EXPORT_SYMBOL_GPL(imx_media_free_dma_buf_ring);
+
+struct imx_media_dma_buf_ring *
+imx_media_alloc_dma_buf_ring(struct imx_media_dev *imxmd,
+			     struct media_entity *src,
+			     struct media_entity *sink,
+			     int size, int num_bufs,
+			     bool alloc_bufs)
+{
+	struct imx_media_dma_buf_ring *ring;
+	int i, ret;
+
+	if (num_bufs < IMX_MEDIA_MIN_RING_BUFS ||
+	    num_bufs > IMX_MEDIA_MAX_RING_BUFS)
+		return ERR_PTR(-EINVAL);
+
+	ring = kzalloc(sizeof(*ring), GFP_KERNEL);
+	if (!ring)
+		return ERR_PTR(-ENOMEM);
+
+	spin_lock_init(&ring->lock);
+	ring->imxmd = imxmd;
+	ring->src = src;
+	ring->sink = sink;
+	ring->num_bufs = num_bufs;
+	ring->last_seq = 0;
+
+	for (i = 0; i < num_bufs; i++) {
+		if (alloc_bufs) {
+			ret = imx_media_alloc_dma_buf(imxmd, &ring->buf[i],
+						      size);
+			if (ret) {
+				ring->num_bufs = i;
+				goto free_ring;
+			}
+		}
+		ring->buf[i].ring = ring;
+		ring->buf[i].index = i;
+	}
+
+	/* now allocate the scratch buffer for underruns */
+	ret = imx_media_alloc_dma_buf(imxmd, &ring->scratch, size);
+	if (ret)
+		goto free_ring;
+	ring->scratch.ring = ring;
+	ring->scratch.index = 999;
+
+	dev_dbg(ring->imxmd->dev,
+		"created ring [%s -> %s], buf size %d, num bufs %d\n",
+		ring->src->name, ring->sink->name, size, num_bufs);
+
+	return ring;
+
+free_ring:
+	imx_media_free_dma_buf_ring(ring);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(imx_media_alloc_dma_buf_ring);
+
+static struct imx_media_dma_buf *
+__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
+{
+	struct imx_media_dma_buf *buf;
+
+	if (index >= ring->num_bufs)
+		return ERR_PTR(-EINVAL);
+
+	buf = &ring->buf[index];
+	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
+		return ERR_PTR(-EINVAL);
+
+	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
+	buf->seq = ring->last_seq++;
+
+	return buf;
+}
+
+int imx_media_dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
+{
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ring->lock, flags);
+	buf = __dma_buf_queue(ring, index);
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	if (IS_ERR(buf))
+		return PTR_ERR(buf);
+
+	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued\n",
+		index, ring->src->name, ring->sink->name);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue);
+
+int imx_media_dma_buf_queue_from_vb(struct imx_media_dma_buf_ring *ring,
+				    struct vb2_buffer *vb)
+{
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+	dma_addr_t phys;
+	void *virt;
+
+	if (vb->index >= ring->num_bufs)
+		return -EINVAL;
+
+	virt = vb2_plane_vaddr(vb, 0);
+	phys = vb2_dma_contig_plane_dma_addr(vb, 0);
+
+	spin_lock_irqsave(&ring->lock, flags);
+	buf = __dma_buf_queue(ring, vb->index);
+	if (IS_ERR(buf))
+		goto err_unlock;
+
+	buf->virt = virt;
+	buf->phys = phys;
+	buf->vb = vb;
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued from vb\n",
+		buf->index, ring->src->name, ring->sink->name);
+
+	return 0;
+err_unlock:
+	spin_unlock_irqrestore(&ring->lock, flags);
+	return PTR_ERR(buf);
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue_from_vb);
+
+void imx_media_dma_buf_done(struct imx_media_dma_buf *buf,
+			    enum imx_media_dma_buf_status status)
+{
+	struct imx_media_dma_buf_ring *ring = buf->ring;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ring->lock, flags);
+	WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_ACTIVE);
+	buf->state = buf->status = status;
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	if (buf == &ring->scratch)
+		dev_dbg(ring->imxmd->dev, "buf-scratch [%s -> %s] done\n",
+			ring->src->name, ring->sink->name);
+	else
+		dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] done\n",
+			buf->index, ring->src->name, ring->sink->name);
+
+	/* if the sink is a subdev, inform it that new buffers are available */
+	if (is_media_entity_v4l2_subdev(ring->sink)) {
+		struct v4l2_subdev *sd =
+			media_entity_to_v4l2_subdev(ring->sink);
+		v4l2_subdev_call(sd, core, ioctl, IMX_MEDIA_NEW_DMA_BUF, NULL);
+	}
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_done);
+
+/* find and return the oldest buffer in the done/error state */
+struct imx_media_dma_buf *
+imx_media_dma_buf_dequeue(struct imx_media_dma_buf_ring *ring)
+{
+	unsigned long flags, oldest_seq = (unsigned long)-1;
+	struct imx_media_dma_buf *buf = NULL, *scan;
+	int i;
+
+	spin_lock_irqsave(&ring->lock, flags);
+
+	for (i = 0; i < ring->num_bufs; i++) {
+		scan = &ring->buf[i];
+		if (scan->state != IMX_MEDIA_BUF_STATUS_DONE &&
+		    scan->state != IMX_MEDIA_BUF_STATUS_ERROR)
+			continue;
+		if (scan->seq < oldest_seq) {
+			buf = scan;
+			oldest_seq = scan->seq;
+		}
+	}
+
+	if (buf)
+		buf->state = IMX_MEDIA_BUF_STATUS_PREPARED;
+
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	if (buf)
+		dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] dequeued\n",
+			buf->index, ring->src->name, ring->sink->name);
+
+	return buf;
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_dequeue);
+
+/* find and return the active buffer, there can be only one! */
+struct imx_media_dma_buf *
+imx_media_dma_buf_get_active(struct imx_media_dma_buf_ring *ring)
+{
+	struct imx_media_dma_buf *buf = NULL;
+	unsigned long flags;
+	int i;
+
+	spin_lock_irqsave(&ring->lock, flags);
+
+	for (i = 0; i < ring->num_bufs; i++) {
+		if (ring->buf[i].state == IMX_MEDIA_BUF_STATUS_ACTIVE) {
+			buf = &ring->buf[i];
+			goto out;
+		}
+	}
+
+	if (ring->scratch.state == IMX_MEDIA_BUF_STATUS_ACTIVE)
+		buf = &ring->scratch;
+
+out:
+	spin_unlock_irqrestore(&ring->lock, flags);
+	return buf;
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_get_active);
+
+/* set this buffer as the active one */
+int imx_media_dma_buf_set_active(struct imx_media_dma_buf *buf)
+{
+	struct imx_media_dma_buf_ring *ring = buf->ring;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ring->lock, flags);
+	WARN_ON(buf != &ring->scratch &&
+		buf->state != IMX_MEDIA_BUF_STATUS_QUEUED);
+	buf->state = IMX_MEDIA_BUF_STATUS_ACTIVE;
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_set_active);
+
+/*
+ * find and return the oldest buffer in the queued state. If
+ * there are none, return the scratch buffer.
+ */
+struct imx_media_dma_buf *
+imx_media_dma_buf_get_next_queued(struct imx_media_dma_buf_ring *ring)
+{
+	unsigned long flags, oldest_seq = (unsigned long)-1;
+	struct imx_media_dma_buf *buf = NULL, *scan;
+	int i;
+
+	spin_lock_irqsave(&ring->lock, flags);
+
+	for (i = 0; i < ring->num_bufs; i++) {
+		scan = &ring->buf[i];
+		if (scan->state != IMX_MEDIA_BUF_STATUS_QUEUED)
+			continue;
+		if (scan->seq < oldest_seq) {
+			buf = scan;
+			oldest_seq = scan->seq;
+		}
+	}
+
+	if (!buf)
+		buf = &ring->scratch;
+
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	if (buf != &ring->scratch)
+		dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] next\n",
+			buf->index, ring->src->name, ring->sink->name);
+	else
+		dev_dbg(ring->imxmd->dev, "buf-scratch [%s -> %s] next\n",
+			ring->src->name, ring->sink->name);
+
+	return buf;
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_get_next_queued);
+
+struct imx_media_dma_buf *
+imx_media_dma_buf_get(struct imx_media_dma_buf_ring *ring, int index)
+{
+	if (index >= ring->num_bufs)
+		return ERR_PTR(-EINVAL);
+	return &ring->buf[index];
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_get);
+
+/* form a subdev name given a group id and ipu id */
+void imx_media_grp_id_to_sd_name(char *sd_name, int sz, u32 grp_id, int ipu_id)
+{
+	int id;
+
+	switch (grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0...IMX_MEDIA_GRP_ID_CSI1:
+		id = (grp_id >> IMX_MEDIA_GRP_ID_CSI_BIT) - 1;
+		snprintf(sd_name, sz, "ipu%d_csi%d", ipu_id + 1, id);
+		break;
+	case IMX_MEDIA_GRP_ID_SMFC0...IMX_MEDIA_GRP_ID_SMFC3:
+		id = (grp_id >> IMX_MEDIA_GRP_ID_SMFC_BIT) - 1;
+		snprintf(sd_name, sz, "ipu%d_smfc%d", ipu_id + 1, id);
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		snprintf(sd_name, sz, "ipu%d_ic_prpenc", ipu_id + 1);
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		snprintf(sd_name, sz, "ipu%d_ic_prpvf", ipu_id + 1);
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+		id = (grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
+		snprintf(sd_name, sz, "ipu%d_ic_pp%d", ipu_id + 1, id);
+		break;
+	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF3:
+		id = (grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
+		snprintf(sd_name, sz, "camif%d", id);
+		break;
+	default:
+		break;
+	}
+}
+EXPORT_SYMBOL_GPL(imx_media_grp_id_to_sd_name);
+
+struct imx_media_subdev *
+imx_media_find_subdev_by_sd(struct imx_media_dev *imxmd,
+			    struct v4l2_subdev *sd)
+{
+	struct imx_media_subdev *imxsd;
+	int i;
+
+	for (i = 0; i < imxmd->num_subdevs; i++) {
+		imxsd = &imxmd->subdev[i];
+		if (sd == imxsd->sd)
+			return imxsd;
+	}
+
+	return ERR_PTR(-ENODEV);
+}
+EXPORT_SYMBOL_GPL(imx_media_find_subdev_by_sd);
+
+struct imx_media_subdev *
+imx_media_find_subdev_by_id(struct imx_media_dev *imxmd, u32 grp_id)
+{
+	struct imx_media_subdev *imxsd;
+	int i;
+
+	for (i = 0; i < imxmd->num_subdevs; i++) {
+		imxsd = &imxmd->subdev[i];
+		if (imxsd->sd && imxsd->sd->grp_id == grp_id)
+			return imxsd;
+	}
+
+	return ERR_PTR(-ENODEV);
+}
+EXPORT_SYMBOL_GPL(imx_media_find_subdev_by_id);
+
+/*
+ * Search for an entity in the current pipeline with given grp_id.
+ * Called with mdev->graph_mutex held.
+ */
+static struct media_entity *
+find_pipeline_entity(struct imx_media_dev *imxmd,
+		     struct media_entity_graph *graph,
+		     struct media_entity *start_entity,
+		     u32 grp_id)
+{
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+
+	media_entity_graph_walk_start(graph, start_entity);
+
+	while ((entity = media_entity_graph_walk_next(graph))) {
+		if (is_media_entity_v4l2_video_device(entity))
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+		if (sd->grp_id & grp_id)
+			return entity;
+	}
+
+	return NULL;
+}
+
+/*
+ * Search for an entity in the current pipeline with given grp_id,
+ * then locate the remote enabled source pad from that entity.
+ * Called with mdev->graph_mutex held.
+ */
+static struct media_pad *
+find_pipeline_remote_source_pad(struct imx_media_dev *imxmd,
+				struct media_entity_graph *graph,
+				struct media_entity *start_entity,
+				u32 grp_id)
+{
+	struct media_pad *pad = NULL;
+	struct media_entity *me;
+	int i;
+
+	me = find_pipeline_entity(imxmd, graph, start_entity, grp_id);
+	if (!me)
+		return NULL;
+
+	/* Find remote source pad */
+	for (i = 0; i < me->num_pads; i++) {
+		struct media_pad *spad = &me->pads[i];
+
+		if (!(spad->flags & MEDIA_PAD_FL_SINK))
+			continue;
+		pad = media_entity_remote_pad(spad);
+		if (pad)
+			return pad;
+	}
+
+	return NULL;
+}
+
+/*
+ * Find the mipi-csi2 virtual channel reached from the given
+ * start entity in the current pipeline.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_find_mipi_csi2_channel(struct imx_media_dev *imxmd,
+				     struct media_entity *start_entity)
+{
+	struct media_entity_graph graph;
+	struct v4l2_subdev *sd;
+	struct media_pad *pad;
+	int ret;
+
+	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
+	if (ret)
+		return ret;
+
+	/* first try to locate the mipi-csi2 from the video mux */
+	pad = find_pipeline_remote_source_pad(imxmd, &graph, start_entity,
+					      IMX_MEDIA_GRP_ID_VIDMUX);
+	/* if couldn't reach it from there, try from a CSI */
+	if (!pad)
+		pad = find_pipeline_remote_source_pad(imxmd, &graph,
+						      start_entity,
+						      IMX_MEDIA_GRP_ID_CSI);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		if (sd->grp_id & IMX_MEDIA_GRP_ID_CSI2) {
+			ret = pad->index - 1; /* found it! */
+			dev_dbg(imxmd->dev, "found vc%d from %s\n",
+				ret, start_entity->name);
+			goto cleanup;
+		}
+	}
+
+	ret = -EPIPE;
+
+cleanup:
+	media_entity_graph_walk_cleanup(&graph);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_find_mipi_csi2_channel);
+
+/*
+ * Find a subdev reached from the given start entity in the
+ * current pipeline.
+ * Must be called with mdev->graph_mutex held.
+ */
+struct imx_media_subdev *
+imx_media_find_pipeline_subdev(struct imx_media_dev *imxmd,
+			       struct media_entity *start_entity,
+			       u32 grp_id)
+{
+	struct media_entity_graph graph;
+	struct imx_media_subdev *imxsd;
+	struct media_entity *me;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
+	if (ret)
+		return ERR_PTR(ret);
+
+	me = find_pipeline_entity(imxmd, &graph, start_entity, grp_id);
+	if (!me) {
+		imxsd = ERR_PTR(-ENODEV);
+		goto cleanup;
+	}
+
+	sd = media_entity_to_v4l2_subdev(me);
+	imxsd = imx_media_find_subdev_by_sd(imxmd, sd);
+cleanup:
+	media_entity_graph_walk_cleanup(&graph);
+	return imxsd;
+}
+EXPORT_SYMBOL_GPL(imx_media_find_pipeline_subdev);
+
+struct imx_media_subdev *
+__imx_media_find_sensor(struct imx_media_dev *imxmd,
+			struct media_entity *start_entity)
+{
+	return imx_media_find_pipeline_subdev(imxmd, start_entity,
+					      IMX_MEDIA_GRP_ID_SENSOR);
+}
+EXPORT_SYMBOL_GPL(__imx_media_find_sensor);
+
+struct imx_media_subdev *
+imx_media_find_sensor(struct imx_media_dev *imxmd,
+		      struct media_entity *start_entity)
+{
+	struct imx_media_subdev *sensor;
+
+	mutex_lock(&imxmd->md.graph_mutex);
+	sensor = __imx_media_find_sensor(imxmd, start_entity);
+	mutex_unlock(&imxmd->md.graph_mutex);
+
+	return sensor;
+}
+EXPORT_SYMBOL_GPL(imx_media_find_sensor);
+
+/*
+ * The subdevs have to be powered on/off, and streaming
+ * enabled/disabled, in a specific sequence.
+ */
+static const u32 stream_on_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+};
+
+static const u32 stream_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+};
+
+#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
+
+static const u32 power_on_seq[] = {
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PP,
+};
+
+static const u32 power_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+};
+
+#define NUM_POWER_ENTITIES ARRAY_SIZE(power_on_seq)
+
+static int imx_media_set_stream(struct imx_media_dev *imxmd,
+				struct media_entity *start_entity,
+				bool on)
+{
+	struct media_entity_graph graph;
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int i, ret;
+	u32 id;
+
+	mutex_lock(&imxmd->md.graph_mutex);
+
+	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
+	if (ret)
+		goto unlock;
+
+	for (i = 0; i < NUM_STREAM_ENTITIES; i++) {
+		id = on ? stream_on_seq[i] : stream_off_seq[i];
+		entity = find_pipeline_entity(imxmd, &graph,
+					      start_entity, id);
+		if (!entity)
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+		ret = v4l2_subdev_call(sd, video, s_stream, on);
+		if (ret && ret != -ENOIOCTLCMD)
+			break;
+	}
+
+	media_entity_graph_walk_cleanup(&graph);
+unlock:
+	mutex_unlock(&imxmd->md.graph_mutex);
+
+	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
+}
+
+/*
+ * Turn current pipeline streaming on/off starting from entity.
+ */
+int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
+				  struct media_entity *entity,
+				  struct media_pipeline *pipe,
+				  bool on)
+{
+	int ret = 0;
+
+	if (on) {
+		ret = media_entity_pipeline_start(entity, pipe);
+		if (ret)
+			return ret;
+		ret = imx_media_set_stream(imxmd, entity, true);
+		if (!ret)
+			return 0;
+		/* fall through */
+	}
+
+	imx_media_set_stream(imxmd, entity, false);
+	if (entity->pipe)
+		media_entity_pipeline_stop(entity);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_stream);
+
+/*
+ * Turn current pipeline power on/off starting from start_entity.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
+				 struct media_entity_graph *graph,
+				 struct media_entity *start_entity, bool on)
+{
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int i, ret = 0;
+	u32 id;
+
+	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
+		id = on ? power_on_seq[i] : power_off_seq[i];
+		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
+		if (!entity)
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		ret = v4l2_subdev_call(sd, core, s_power, on);
+		if (ret && ret != -ENOIOCTLCMD)
+			break;
+	}
+
+	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
+
+/*
+ * Inherit the v4l2 controls from all entities in a pipeline
+ * to the given video device.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_inherit_controls(struct imx_media_dev *imxmd,
+			       struct video_device *vfd,
+			       struct media_entity *start_entity)
+{
+	struct media_entity_graph graph;
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
+	if (ret)
+		return ret;
+
+	media_entity_graph_walk_start(&graph, start_entity);
+
+	while ((entity = media_entity_graph_walk_next(&graph))) {
+		if (is_media_entity_v4l2_video_device(entity))
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
+			__func__, sd->name);
+
+		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
+					    sd->ctrl_handler,
+					    NULL);
+		if (ret)
+			break;
+	}
+
+	media_entity_graph_walk_cleanup(&graph);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
+
+MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
new file mode 100644
index 0000000..357654d
--- /dev/null
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -0,0 +1,486 @@
+/*
+ * V4L2 Media Controller Driver for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-mc.h>
+#include <video/imx-ipu-v3.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+static inline struct imx_media_dev *notifier2dev(struct v4l2_async_notifier *n)
+{
+	return container_of(n, struct imx_media_dev, subdev_notifier);
+}
+
+/*
+ * Find a subdev by device node or device name. This is called during
+ * driver load to form the async subdev list and bind them.
+ */
+struct imx_media_subdev *
+imx_media_find_async_subdev(struct imx_media_dev *imxmd,
+			    struct device_node *np,
+			    const char *devname)
+{
+	struct imx_media_subdev *imxsd;
+	int i;
+
+	for (i = 0; i < imxmd->subdev_notifier.num_subdevs; i++) {
+		imxsd = &imxmd->subdev[i];
+		switch (imxsd->asd.match_type) {
+		case V4L2_ASYNC_MATCH_OF:
+			if (np && imxsd->asd.match.of.node == np)
+				return imxsd;
+			break;
+		case V4L2_ASYNC_MATCH_DEVNAME:
+			if (devname &&
+			    !strcmp(imxsd->asd.match.device_name.name, devname))
+				return imxsd;
+			break;
+		default:
+			break;
+		}
+	}
+
+	return NULL;
+}
+
+/*
+ * Adds a subdev to the async subdev list. If np is non-NULL, adds
+ * the async as a V4L2_ASYNC_MATCH_OF match type, otherwise as a
+ * V4L2_ASYNC_MATCH_DEVNAME match type using devname. This is called
+ * during driver load when forming the async subdev list.
+ */
+struct imx_media_subdev *
+imx_media_add_async_subdev(struct imx_media_dev *imxmd,
+			   struct device_node *np,
+			   const char *devname)
+{
+	struct imx_media_subdev *imxsd;
+	struct v4l2_async_subdev *asd;
+	int sd_idx;
+
+	/* return NULL if this subdev already added */
+	if (imx_media_find_async_subdev(imxmd, np, devname)) {
+		dev_dbg(imxmd->dev, "%s: already added %s\n",
+			__func__, np ? np->name : devname);
+		return NULL;
+	}
+
+	sd_idx = imxmd->subdev_notifier.num_subdevs;
+	if (sd_idx >= IMX_MEDIA_MAX_SUBDEVS) {
+		dev_err(imxmd->dev, "%s: too many subdevs! can't add %s\n",
+			__func__, np ? np->name : devname);
+		return ERR_PTR(-ENOSPC);
+	}
+
+	imxsd = &imxmd->subdev[sd_idx];
+
+	asd = &imxsd->asd;
+	if (np) {
+		asd->match_type = V4L2_ASYNC_MATCH_OF;
+		asd->match.of.node = np;
+	} else {
+		asd->match_type = V4L2_ASYNC_MATCH_DEVNAME;
+		strncpy(imxsd->devname, devname, sizeof(imxsd->devname));
+		asd->match.device_name.name = imxsd->devname;
+	}
+
+	imxmd->async_ptrs[sd_idx] = asd;
+	imxmd->subdev_notifier.num_subdevs++;
+
+	dev_dbg(imxmd->dev, "%s: added %s, match type %s\n",
+		__func__, np ? np->name : devname, np ? "OF" : "DEVNAME");
+
+	return imxsd;
+}
+
+/*
+ * Adds an imx-media link to a subdev pad's link list. This is called
+ * during driver load when forming the links between subdevs.
+ *
+ * @pad: the local pad
+ * @remote_node: the device node of the remote subdev
+ * @remote_devname: the device name of the remote subdev
+ * @local_pad: local pad index
+ * @remote_pad: remote pad index
+ */
+int imx_media_add_pad_link(struct imx_media_dev *imxmd,
+			   struct imx_media_pad *pad,
+			   struct device_node *remote_node,
+			   const char *remote_devname,
+			   int local_pad, int remote_pad)
+{
+	struct imx_media_link *link;
+	int link_idx;
+
+	link_idx = pad->num_links;
+	if (link_idx >= IMX_MEDIA_MAX_LINKS) {
+		dev_err(imxmd->dev, "%s: too many links!\n", __func__);
+		return -ENOSPC;
+	}
+
+	link = &pad->link[link_idx];
+
+	link->remote_sd_node = remote_node;
+	if (remote_devname)
+		strncpy(link->remote_devname, remote_devname,
+			sizeof(link->remote_devname));
+
+	link->local_pad = local_pad;
+	link->remote_pad = remote_pad;
+
+	pad->num_links++;
+
+	return 0;
+}
+
+/*
+ * get IPU from this CSI and add it to the list of IPUs
+ * the media driver will control.
+ */
+static int imx_media_get_ipu(struct imx_media_dev *imxmd,
+			     struct v4l2_subdev *csi_sd)
+{
+	struct ipu_soc *ipu;
+	int ipu_id;
+
+	ipu = dev_get_drvdata(csi_sd->dev->parent);
+	if (!ipu) {
+		v4l2_err(&imxmd->v4l2_dev,
+			 "CSI %s has no parent IPU!\n", csi_sd->name);
+		return -ENODEV;
+	}
+
+	ipu_id = ipu_get_num(ipu);
+	if (ipu_id > 1) {
+		v4l2_err(&imxmd->v4l2_dev, "invalid IPU id %d!\n", ipu_id);
+		return -ENODEV;
+	}
+
+	if (!imxmd->ipu[ipu_id])
+		imxmd->ipu[ipu_id] = ipu;
+
+	return 0;
+}
+
+/* async subdev bound notifier */
+static int imx_media_subdev_bound(struct v4l2_async_notifier *notifier,
+				  struct v4l2_subdev *sd,
+				  struct v4l2_async_subdev *asd)
+{
+	struct imx_media_dev *imxmd = notifier2dev(notifier);
+	struct imx_media_subdev *imxsd;
+	int i, ret = -EINVAL;
+
+	imxsd = imx_media_find_async_subdev(imxmd, sd->dev->of_node,
+					    dev_name(sd->dev));
+	if (!imxsd)
+		goto out;
+
+	imxsd->sd = sd;
+
+	if (sd->grp_id & IMX_MEDIA_GRP_ID_CSI) {
+		ret = imx_media_get_ipu(imxmd, sd);
+		if (ret)
+			return ret;
+	} else if (sd->entity.function == MEDIA_ENT_F_MUX) {
+		/* this is the video mux */
+		sd->grp_id = IMX_MEDIA_GRP_ID_VIDMUX;
+	} else if (imxsd->num_sink_pads == 0 &&
+		   ((sd->entity.flags & MEDIA_ENT_F_ATV_DECODER) ||
+		    sd->entity.function == MEDIA_ENT_F_CAM_SENSOR ||
+		    sd->entity.function == MEDIA_ENT_F_ATV_DECODER)) {
+		/* this is a sensor */
+		sd->grp_id = IMX_MEDIA_GRP_ID_SENSOR;
+
+		/* set sensor input names if needed */
+		for (i = 0; i < imxsd->input.num; i++) {
+			if (strlen(imxsd->input.name[i]))
+				continue;
+			snprintf(imxsd->input.name[i],
+				 sizeof(imxsd->input.name[i]),
+				 "%s-%d", sd->name, i);
+		}
+	}
+
+	ret = 0;
+out:
+	if (ret)
+		v4l2_warn(&imxmd->v4l2_dev,
+			  "Received unknown subdev %s\n", sd->name);
+	else
+		v4l2_info(&imxmd->v4l2_dev,
+			  "Registered subdev %s\n", sd->name);
+
+	return ret;
+}
+
+/*
+ * create a single media link given a local subdev, a single pad from that
+ * subdev, and a single link from that pad. Called after all subdevs have
+ * registered.
+ */
+static int imx_media_create_link(struct imx_media_dev *imxmd,
+				 struct imx_media_subdev *local_sd,
+				 struct imx_media_pad *pad,
+				 struct imx_media_link *link)
+{
+	struct imx_media_subdev *remote_sd;
+	struct v4l2_subdev *source, *sink;
+	u16 source_pad, sink_pad;
+	int ret;
+
+	/* only create the source->sink links */
+	if (pad->pad.flags & MEDIA_PAD_FL_SINK)
+		return 0;
+
+	remote_sd = imx_media_find_async_subdev(imxmd, link->remote_sd_node,
+						link->remote_devname);
+	if (!remote_sd) {
+		v4l2_warn(&imxmd->v4l2_dev, "%s: no remote for %s:%d\n",
+			  __func__, local_sd->sd->name, link->local_pad);
+		return 0;
+	}
+
+	source = local_sd->sd;
+	sink = remote_sd->sd;
+	source_pad = link->local_pad;
+	sink_pad = link->remote_pad;
+
+	v4l2_info(&imxmd->v4l2_dev, "%s: %s:%d -> %s:%d\n", __func__,
+		  source->name, source_pad, sink->name, sink_pad);
+
+	ret = media_create_pad_link(&source->entity, source_pad,
+				    &sink->entity, sink_pad, 0);
+	if (ret)
+		v4l2_err(&imxmd->v4l2_dev,
+			 "create_pad_link failed: %d\n", ret);
+
+	return ret;
+}
+
+/*
+ * create the media links from all imx-media pads and their links.
+ * Called after all subdevs have registered.
+ */
+static int imx_media_create_links(struct imx_media_dev *imxmd)
+{
+	struct imx_media_subdev *local_sd;
+	struct imx_media_link *link;
+	struct imx_media_pad *pad;
+	int num_pads, i, j, k;
+	int ret = 0;
+
+	for (i = 0; i < imxmd->num_subdevs; i++) {
+		local_sd = &imxmd->subdev[i];
+		num_pads = local_sd->num_sink_pads + local_sd->num_src_pads;
+
+		for (j = 0; j < num_pads; j++) {
+			pad = &local_sd->pad[j];
+
+			for (k = 0; k < pad->num_links; k++) {
+				link = &pad->link[k];
+
+				ret = imx_media_create_link(imxmd, local_sd,
+							    pad, link);
+				if (ret)
+					goto out;
+			}
+		}
+	}
+
+out:
+	return ret;
+}
+
+/* async subdev complete notifier */
+static int imx_media_probe_complete(struct v4l2_async_notifier *notifier)
+{
+	struct imx_media_dev *imxmd = notifier2dev(notifier);
+	int ret;
+
+	mutex_lock(&imxmd->md.graph_mutex);
+
+	ret = imx_media_create_links(imxmd);
+	if (ret)
+		goto unlock;
+
+	ret = v4l2_device_register_subdev_nodes(&imxmd->v4l2_dev);
+unlock:
+	mutex_unlock(&imxmd->md.graph_mutex);
+	if (ret)
+		return ret;
+
+	return media_device_register(&imxmd->md);
+}
+
+static int imx_media_link_notify(struct media_link *link, unsigned int flags,
+				 unsigned int notification)
+{
+	struct media_entity *sink = link->sink->entity;
+	struct media_entity_graph *graph;
+	struct v4l2_subdev *sink_sd;
+	struct imx_media_dev *imxmd;
+	int ret = 0;
+
+	if (is_media_entity_v4l2_video_device(sink))
+		return 0;
+	sink_sd = media_entity_to_v4l2_subdev(sink);
+	imxmd = dev_get_drvdata(sink_sd->v4l2_dev->dev);
+	graph = &imxmd->link_notify_graph;
+
+	if (notification == MEDIA_DEV_NOTIFY_PRE_LINK_CH) {
+		ret = media_entity_graph_walk_init(graph, &imxmd->md);
+		if (ret)
+			return ret;
+
+		if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+			/* Before link disconnection */
+			ret = imx_media_pipeline_set_power(imxmd, graph,
+							   sink, false);
+		}
+	} else if (notification == MEDIA_DEV_NOTIFY_POST_LINK_CH) {
+		if (link->flags & MEDIA_LNK_FL_ENABLED) {
+			/* After link activation */
+			ret = imx_media_pipeline_set_power(imxmd, graph,
+							   sink, true);
+		}
+
+		media_entity_graph_walk_cleanup(graph);
+	}
+
+	return ret ? -EPIPE : 0;
+}
+
+static const struct media_device_ops imx_media_md_ops = {
+	.link_notify = imx_media_link_notify,
+};
+
+static int imx_media_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *node = dev->of_node;
+	struct imx_media_subdev *csi[4];
+	struct imx_media_dev *imxmd;
+	int ret;
+
+	imxmd = devm_kzalloc(dev, sizeof(*imxmd), GFP_KERNEL);
+	if (!imxmd)
+		return -ENOMEM;
+
+	imxmd->dev = dev;
+	dev_set_drvdata(dev, imxmd);
+
+	strlcpy(imxmd->md.model, "imx-media", sizeof(imxmd->md.model));
+	imxmd->md.ops = &imx_media_md_ops;
+	imxmd->md.dev = dev;
+
+	imxmd->v4l2_dev.mdev = &imxmd->md;
+	strlcpy(imxmd->v4l2_dev.name, "imx-media",
+		sizeof(imxmd->v4l2_dev.name));
+
+	media_device_init(&imxmd->md);
+
+	ret = v4l2_device_register(dev, &imxmd->v4l2_dev);
+	if (ret < 0) {
+		v4l2_err(&imxmd->v4l2_dev,
+			 "Failed to register v4l2_device: %d\n", ret);
+		return ret;
+	}
+
+	dev_set_drvdata(imxmd->v4l2_dev.dev, imxmd);
+
+	ret = imx_media_of_parse(imxmd, &csi, node);
+	if (ret) {
+		v4l2_err(&imxmd->v4l2_dev,
+			 "imx_media_of_parse failed with %d\n", ret);
+		goto unreg_dev;
+	}
+
+	ret = imx_media_add_internal_subdevs(imxmd, csi);
+	if (ret) {
+		v4l2_err(&imxmd->v4l2_dev,
+			 "add_internal_subdevs failed with %d\n", ret);
+		goto unreg_dev;
+	}
+
+	/* no subdevs? just bail for this media device */
+	imxmd->num_subdevs = imxmd->subdev_notifier.num_subdevs;
+	if (imxmd->num_subdevs == 0) {
+		ret = -ENODEV;
+		goto unreg_dev;
+	}
+
+	/* prepare the async subdev notifier and register it */
+	imxmd->subdev_notifier.subdevs = imxmd->async_ptrs;
+	imxmd->subdev_notifier.bound = imx_media_subdev_bound;
+	imxmd->subdev_notifier.complete = imx_media_probe_complete;
+	ret = v4l2_async_notifier_register(&imxmd->v4l2_dev,
+					   &imxmd->subdev_notifier);
+	if (ret) {
+		v4l2_err(&imxmd->v4l2_dev,
+			 "v4l2_async_notifier_register failed with %d\n", ret);
+		goto unreg_dev;
+	}
+
+	return 0;
+
+unreg_dev:
+	v4l2_device_unregister(&imxmd->v4l2_dev);
+	return ret;
+}
+
+static int imx_media_remove(struct platform_device *pdev)
+{
+	struct imx_media_dev *imxmd =
+		(struct imx_media_dev *)platform_get_drvdata(pdev);
+
+	v4l2_info(&imxmd->v4l2_dev, "Removing imx-media\n");
+
+	v4l2_async_notifier_unregister(&imxmd->subdev_notifier);
+	v4l2_device_unregister(&imxmd->v4l2_dev);
+	media_device_unregister(&imxmd->md);
+	media_device_cleanup(&imxmd->md);
+
+	return 0;
+}
+
+static const struct of_device_id imx_media_dt_ids[] = {
+	{ .compatible = "fsl,imx-media" },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, imx_media_dt_ids);
+
+static struct platform_driver imx_media_pdrv = {
+	.probe		= imx_media_probe,
+	.remove		= imx_media_remove,
+	.driver		= {
+		.name	= "imx-media",
+		.of_match_table	= imx_media_dt_ids,
+	},
+};
+
+module_platform_driver(imx_media_pdrv);
+
+MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/media/imx/imx-media-fim.c b/drivers/staging/media/imx/imx-media-fim.c
new file mode 100644
index 0000000..acc7e39
--- /dev/null
+++ b/drivers/staging/media/imx/imx-media-fim.c
@@ -0,0 +1,471 @@
+/*
+ * Frame Interval Monitor.
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+enum {
+	FIM_CL_ENABLE = 0,
+	FIM_CL_NUM,
+	FIM_CL_TOLERANCE_MIN,
+	FIM_CL_TOLERANCE_MAX,
+	FIM_CL_NUM_SKIP,
+	FIM_NUM_CONTROLS,
+};
+
+#define FIM_CL_ENABLE_DEF          1 /* FIM enabled by default */
+#define FIM_CL_NUM_DEF             8 /* average 8 frames */
+#define FIM_CL_NUM_SKIP_DEF        2 /* skip 2 frames after restart */
+#define FIM_CL_TOLERANCE_MIN_DEF  50 /* usec */
+#define FIM_CL_TOLERANCE_MAX_DEF   0 /* no max tolerance (unbounded) */
+
+struct imx_media_fim {
+	struct imx_media_dev *md;
+
+	/* the owning subdev of this fim instance */
+	struct v4l2_subdev *sd;
+
+	/* FIM's control handler */
+	struct v4l2_ctrl_handler ctrl_handler;
+
+	/* control cluster */
+	struct v4l2_ctrl  *ctrl[FIM_NUM_CONTROLS];
+
+	/* current control values */
+	bool              enabled;
+	int               num_avg;
+	int               num_skip;
+	unsigned long     tolerance_min; /* usec */
+	unsigned long     tolerance_max; /* usec */
+
+	int               counter;
+	struct timespec   last_ts;
+	unsigned long     sum;       /* usec */
+	unsigned long     nominal;   /* usec */
+
+	/*
+	 * input capture method of measuring FI (channel and flags
+	 * from device tree)
+	 */
+	int               icap_channel;
+	int               icap_flags;
+	struct completion icap_first_event;
+};
+
+static void update_fim_nominal(struct imx_media_fim *fim,
+			       struct imx_media_subdev *sensor)
+{
+	struct v4l2_streamparm parm;
+	struct v4l2_fract tpf;
+	int ret;
+
+	parm.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	ret = v4l2_subdev_call(sensor->sd, video, g_parm, &parm);
+	tpf = parm.parm.capture.timeperframe;
+
+	if (ret || tpf.denominator == 0) {
+		dev_dbg(fim->sd->dev, "no tpf from sensor, FIM disabled\n");
+		fim->enabled = false;
+		return;
+	}
+
+	fim->nominal = DIV_ROUND_CLOSEST(1000 * 1000 * tpf.numerator,
+					 tpf.denominator);
+
+	dev_dbg(fim->sd->dev, "sensor FI=%lu usec\n", fim->nominal);
+}
+
+static void reset_fim(struct imx_media_fim *fim, bool curval)
+{
+	struct v4l2_ctrl *en = fim->ctrl[FIM_CL_ENABLE];
+	struct v4l2_ctrl *num = fim->ctrl[FIM_CL_NUM];
+	struct v4l2_ctrl *skip = fim->ctrl[FIM_CL_NUM_SKIP];
+	struct v4l2_ctrl *tol_min = fim->ctrl[FIM_CL_TOLERANCE_MIN];
+	struct v4l2_ctrl *tol_max = fim->ctrl[FIM_CL_TOLERANCE_MAX];
+
+	if (curval) {
+		fim->enabled = en->cur.val;
+		fim->num_avg = num->cur.val;
+		fim->num_skip = skip->cur.val;
+		fim->tolerance_min = tol_min->cur.val;
+		fim->tolerance_max = tol_max->cur.val;
+	} else {
+		fim->enabled = en->val;
+		fim->num_avg = num->val;
+		fim->num_skip = skip->val;
+		fim->tolerance_min = tol_min->val;
+		fim->tolerance_max = tol_max->val;
+	}
+
+	/* disable tolerance range if max <= min */
+	if (fim->tolerance_max <= fim->tolerance_min)
+		fim->tolerance_max = 0;
+
+	fim->counter = -fim->num_skip;
+	fim->sum = 0;
+}
+
+static void send_fim_event(struct imx_media_fim *fim, unsigned long error)
+{
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_FRAME_INTERVAL,
+	};
+
+	v4l2_subdev_notify_event(fim->sd, &ev);
+}
+
+/*
+ * Monitor an averaged frame interval. If the average deviates too much
+ * from the sensor's nominal frame rate, send the frame interval error
+ * event. The frame intervals are averaged in order to quiet noise from
+ * (presumably random) interrupt latency.
+ */
+static void frame_interval_monitor(struct imx_media_fim *fim,
+				   struct timespec *ts)
+{
+	unsigned long interval, error, error_avg;
+	struct timespec diff;
+	bool send_event = false;
+
+	if (!fim->enabled || ++fim->counter <= 0)
+		goto out_update_ts;
+
+	diff = timespec_sub(*ts, fim->last_ts);
+	interval = diff.tv_sec * 1000 * 1000 + diff.tv_nsec / 1000;
+	error = abs(interval - fim->nominal);
+
+	if (fim->tolerance_max && error >= fim->tolerance_max) {
+		dev_dbg(fim->sd->dev,
+			"FIM: %lu ignored, out of tolerance bounds\n",
+			error);
+		fim->counter--;
+		goto out_update_ts;
+	}
+
+	fim->sum += error;
+
+	if (fim->counter == fim->num_avg) {
+		error_avg = DIV_ROUND_CLOSEST(fim->sum, fim->num_avg);
+
+		if (error_avg > fim->tolerance_min)
+			send_event = true;
+
+		dev_dbg(fim->sd->dev, "FIM: error: %lu usec%s\n",
+			error_avg, send_event ? " (!!!)" : "");
+
+		fim->counter = 0;
+		fim->sum = 0;
+	}
+
+out_update_ts:
+	fim->last_ts = *ts;
+	if (send_event)
+		send_fim_event(fim, error_avg);
+}
+
+#ifdef CONFIG_IMX_GPT_ICAP
+/*
+ * Input Capture method of measuring frame intervals. Not subject
+ * to interrupt latency.
+ */
+static void fim_input_capture_handler(int channel, void *dev_id,
+				      struct timespec *ts)
+{
+	struct imx_media_fim *fim = dev_id;
+
+	frame_interval_monitor(fim, ts);
+
+	if (!completion_done(&fim->icap_first_event))
+		complete(&fim->icap_first_event);
+}
+
+static int fim_request_input_capture(struct imx_media_fim *fim)
+{
+	init_completion(&fim->icap_first_event);
+
+	return mxc_request_input_capture(fim->icap_channel,
+					 fim_input_capture_handler,
+					 fim->icap_flags, fim);
+}
+
+static void fim_free_input_capture(struct imx_media_fim *fim)
+{
+	mxc_free_input_capture(fim->icap_channel, fim);
+}
+
+#else /* CONFIG_IMX_GPT_ICAP */
+
+static int fim_request_input_capture(struct imx_media_fim *fim)
+{
+	return 0;
+}
+
+static void fim_free_input_capture(struct imx_media_fim *fim)
+{
+}
+
+#endif /* CONFIG_IMX_GPT_ICAP */
+
+/*
+ * In case we are monitoring the first frame interval after streamon
+ * (when fim->num_skip = 0), we need a valid fim->last_ts before we
+ * can begin. This only applies to the input capture method. It is not
+ * possible to accurately measure the first FI after streamon using the
+ * EOF method, so fim->num_skip minimum is set to 1 in that case, so this
+ * function is a noop when the EOF method is used.
+ */
+static void fim_acquire_first_ts(struct imx_media_fim *fim)
+{
+	unsigned long ret;
+
+	if (!fim->enabled || fim->num_skip > 0)
+		return;
+
+	ret = wait_for_completion_timeout(
+		&fim->icap_first_event,
+		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(fim->sd, "wait first icap event timeout\n");
+}
+
+/* FIM Controls */
+static int fim_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct imx_media_fim *fim = container_of(ctrl->handler,
+						 struct imx_media_fim,
+						 ctrl_handler);
+
+	switch (ctrl->id) {
+	case V4L2_CID_IMX_FIM_ENABLE:
+		reset_fim(fim, false);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops fim_ctrl_ops = {
+	.s_ctrl = fim_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config imx_media_fim_ctrl[] = {
+	[FIM_CL_ENABLE] = {
+		.ops = &fim_ctrl_ops,
+		.id = V4L2_CID_IMX_FIM_ENABLE,
+		.name = "FIM Enable",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def = FIM_CL_ENABLE_DEF,
+		.min = 0,
+		.max = 1,
+		.step = 1,
+	},
+	[FIM_CL_NUM] = {
+		.ops = &fim_ctrl_ops,
+		.id = V4L2_CID_IMX_FIM_NUM,
+		.name = "FIM Num Average",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_NUM_DEF,
+		.min =  1, /* no averaging */
+		.max = 64, /* average 64 frames */
+		.step = 1,
+	},
+	[FIM_CL_TOLERANCE_MIN] = {
+		.ops = &fim_ctrl_ops,
+		.id = V4L2_CID_IMX_FIM_TOLERANCE_MIN,
+		.name = "FIM Tolerance Min",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_TOLERANCE_MIN_DEF,
+		.min =    2,
+		.max =  200,
+		.step =   1,
+	},
+	[FIM_CL_TOLERANCE_MAX] = {
+		.ops = &fim_ctrl_ops,
+		.id = V4L2_CID_IMX_FIM_TOLERANCE_MAX,
+		.name = "FIM Tolerance Max",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_TOLERANCE_MAX_DEF,
+		.min =    0,
+		.max =  500,
+		.step =   1,
+	},
+	[FIM_CL_NUM_SKIP] = {
+		.ops = &fim_ctrl_ops,
+		.id = V4L2_CID_IMX_FIM_NUM_SKIP,
+		.name = "FIM Num Skip",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = FIM_CL_NUM_SKIP_DEF,
+		.min =   0, /* skip no frames */
+		.max = 256, /* skip 256 frames */
+		.step =  1,
+	},
+};
+
+static int init_fim_controls(struct imx_media_fim *fim)
+{
+	struct v4l2_ctrl_handler *hdlr = &fim->ctrl_handler;
+	struct v4l2_ctrl_config fim_c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, FIM_NUM_CONTROLS);
+
+	for (i = 0; i < FIM_NUM_CONTROLS; i++) {
+		fim_c = imx_media_fim_ctrl[i];
+
+		/*
+		 * it's not possible to accurately measure the first
+		 * FI after streamon using the EOF method, so force
+		 * num_skip minimum to 1 in that case.
+		 */
+		if (i == FIM_CL_NUM_SKIP && fim->icap_channel < 0)
+			fim_c.min = 1;
+
+		fim->ctrl[i] = v4l2_ctrl_new_custom(hdlr, &fim_c, NULL);
+	}
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		goto err_free;
+	}
+
+	v4l2_ctrl_cluster(FIM_NUM_CONTROLS, fim->ctrl);
+
+	/* add the FIM controls to the calling subdev ctrl handler */
+	ret = v4l2_ctrl_add_handler(fim->sd->ctrl_handler,
+				    &fim->ctrl_handler, NULL);
+	if (ret)
+		goto err_free;
+
+	return 0;
+err_free:
+	v4l2_ctrl_handler_free(hdlr);
+	return ret;
+}
+
+static int of_parse_fim(struct imx_media_fim *fim, struct device_node *np)
+{
+	struct device_node *fim_np;
+	u32 icap[2];
+	int ret;
+
+	/* by default EOF method is used */
+	fim->icap_channel = -1;
+
+	fim_np = of_get_child_by_name(np, "fim");
+	if (!fim_np || !of_device_is_available(fim_np)) {
+		of_node_put(fim_np);
+		return -ENODEV;
+	}
+
+	if (IS_ENABLED(CONFIG_IMX_GPT_ICAP)) {
+		ret = of_property_read_u32_array(fim_np,
+						 "fsl,input-capture-channel",
+						 icap, 2);
+		if (!ret) {
+			fim->icap_channel = icap[0];
+			fim->icap_flags = icap[1];
+		}
+	}
+
+	of_node_put(fim_np);
+	return 0;
+}
+
+/*
+ * Monitor frame intervals via EOF interrupt. This method is
+ * subject to uncertainty errors introduced by interrupt latency.
+ *
+ * This is a noop if the Input Capture method is being used, since
+ * the frame_interval_monitor() is called by the input capture event
+ * callback handler in that case.
+ */
+void imx_media_fim_eof_monitor(struct imx_media_fim *fim, struct timespec *ts)
+{
+	if (fim->icap_channel >= 0)
+		return;
+
+	frame_interval_monitor(fim, ts);
+}
+EXPORT_SYMBOL_GPL(imx_media_fim_eof_monitor);
+
+/* Called by the subdev in its s_power callback */
+int imx_media_fim_set_power(struct imx_media_fim *fim, bool on)
+{
+	int ret = 0;
+
+	if (fim->icap_channel >= 0) {
+		if (on)
+			ret = fim_request_input_capture(fim);
+		else
+			fim_free_input_capture(fim);
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_fim_set_power);
+
+/* Called by the subdev in its s_stream callback */
+int imx_media_fim_set_stream(struct imx_media_fim *fim,
+			     struct imx_media_subdev *sensor,
+			     bool on)
+{
+	if (on) {
+		reset_fim(fim, true);
+		update_fim_nominal(fim, sensor);
+
+		if (fim->icap_channel >= 0)
+			fim_acquire_first_ts(fim);
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_fim_set_stream);
+
+/* Called by the subdev in its subdev registered callback */
+struct imx_media_fim *imx_media_fim_init(struct v4l2_subdev *sd)
+{
+	struct device_node *node = sd->of_node;
+	struct imx_media_fim *fim;
+	int ret;
+
+	fim = devm_kzalloc(sd->dev, sizeof(*fim), GFP_KERNEL);
+	if (!fim)
+		return ERR_PTR(-ENOMEM);
+
+	/* get media device */
+	fim->md = dev_get_drvdata(sd->v4l2_dev->dev);
+	fim->sd = sd;
+
+	ret = of_parse_fim(fim, node);
+	if (ret)
+		return (ret == -ENODEV) ? NULL : ERR_PTR(ret);
+
+	ret = init_fim_controls(fim);
+	if (ret)
+		return ERR_PTR(ret);
+
+	return fim;
+}
+EXPORT_SYMBOL_GPL(imx_media_fim_init);
+
+void imx_media_fim_free(struct imx_media_fim *fim)
+{
+	v4l2_ctrl_handler_free(&fim->ctrl_handler);
+}
+EXPORT_SYMBOL_GPL(imx_media_fim_free);
diff --git a/drivers/staging/media/imx/imx-media-internal-sd.c b/drivers/staging/media/imx/imx-media-internal-sd.c
new file mode 100644
index 0000000..fd3e020
--- /dev/null
+++ b/drivers/staging/media/imx/imx-media-internal-sd.c
@@ -0,0 +1,457 @@
+/*
+ * Media driver for Freescale i.MX5/6 SOC
+ *
+ * Adds the internal subdevices and the media links between them.
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/platform_device.h>
+#include "imx-media.h"
+
+enum isd_enum {
+	isd_csi0 = 0,
+	isd_csi1,
+	isd_smfc0,
+	isd_smfc1,
+	isd_ic_prpenc,
+	isd_ic_prpvf,
+	isd_ic_pp0,
+	isd_ic_pp1,
+	isd_camif0,
+	isd_camif1,
+	num_isd,
+};
+
+static const struct internal_subdev_id {
+	enum isd_enum index;
+	const char *name;
+	u32 grp_id;
+} isd_id[num_isd] = {
+	[isd_csi0] = {
+		.index = isd_csi0,
+		.grp_id = IMX_MEDIA_GRP_ID_CSI0,
+		.name = "imx-ipuv3-csi",
+	},
+	[isd_csi1] = {
+		.index = isd_csi1,
+		.grp_id = IMX_MEDIA_GRP_ID_CSI1,
+		.name = "imx-ipuv3-csi",
+	},
+	[isd_smfc0] = {
+		.index = isd_smfc0,
+		.grp_id = IMX_MEDIA_GRP_ID_SMFC0,
+		.name = "imx-ipuv3-smfc",
+	},
+	[isd_smfc1] = {
+		.index = isd_smfc1,
+		.grp_id = IMX_MEDIA_GRP_ID_SMFC1,
+		.name = "imx-ipuv3-smfc",
+	},
+	[isd_ic_prpenc] = {
+		.index = isd_ic_prpenc,
+		.grp_id = IMX_MEDIA_GRP_ID_IC_PRPENC,
+		.name = "imx-ipuv3-ic",
+	},
+	[isd_ic_prpvf] = {
+		.index = isd_ic_prpvf,
+		.grp_id = IMX_MEDIA_GRP_ID_IC_PRPVF,
+		.name = "imx-ipuv3-ic",
+	},
+	[isd_ic_pp0] = {
+		.index = isd_ic_pp0,
+		.grp_id = IMX_MEDIA_GRP_ID_IC_PP0,
+		.name = "imx-ipuv3-ic",
+	},
+	[isd_ic_pp1] = {
+		.index = isd_ic_pp1,
+		.grp_id = IMX_MEDIA_GRP_ID_IC_PP1,
+		.name = "imx-ipuv3-ic",
+	},
+	[isd_camif0] = {
+		.index = isd_camif0,
+		.grp_id = IMX_MEDIA_GRP_ID_CAMIF0,
+		.name = "imx-media-camif",
+	},
+	[isd_camif1] = {
+		.index = isd_camif1,
+		.grp_id = IMX_MEDIA_GRP_ID_CAMIF1,
+		.name = "imx-media-camif",
+	},
+};
+
+struct internal_link {
+	const struct internal_subdev_id *remote_id;
+	int remote_pad;
+};
+
+struct internal_pad {
+	int num_links;
+	bool devnode; /* does this pad link to a device node */
+	struct internal_link link[IMX_MEDIA_MAX_LINKS];
+};
+
+static const struct internal_subdev {
+	const struct internal_subdev_id *id;
+	struct internal_pad pad[IMX_MEDIA_MAX_PADS];
+	int num_sink_pads;
+	int num_src_pads;
+} internal_subdev[num_isd] = {
+	[isd_csi0] = {
+		.id = &isd_id[isd_csi0],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 3,
+			.link[0] = {
+				.remote_id = &isd_id[isd_ic_prpenc],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id =  &isd_id[isd_ic_prpvf],
+				.remote_pad = 0,
+			},
+			.link[2] = {
+				.remote_id =  &isd_id[isd_smfc0],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_csi1] = {
+		.id = &isd_id[isd_csi1],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 3,
+			.link[0] = {
+				.remote_id = &isd_id[isd_ic_prpenc],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id =  &isd_id[isd_ic_prpvf],
+				.remote_pad = 0,
+			},
+			.link[2] = {
+				.remote_id =  &isd_id[isd_smfc1],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_smfc0] = {
+		.id = &isd_id[isd_smfc0],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 4,
+			.link[0] = {
+				.remote_id =  &isd_id[isd_ic_prpvf],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id =  &isd_id[isd_ic_pp0],
+				.remote_pad = 0,
+			},
+			.link[2] = {
+				.remote_id = &isd_id[isd_camif0],
+				.remote_pad = 0,
+			},
+			.link[3] = {
+				.remote_id = &isd_id[isd_camif1],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_smfc1] = {
+		.id = &isd_id[isd_smfc1],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 4,
+			.link[0] = {
+				.remote_id =  &isd_id[isd_ic_prpvf],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id =  &isd_id[isd_ic_pp1],
+				.remote_pad = 0,
+			},
+			.link[2] = {
+				.remote_id = &isd_id[isd_camif0],
+				.remote_pad = 0,
+			},
+			.link[3] = {
+				.remote_id = &isd_id[isd_camif1],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_ic_prpenc] = {
+		.id = &isd_id[isd_ic_prpenc],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 2,
+			.link[0] = {
+				.remote_id = &isd_id[isd_camif0],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id = &isd_id[isd_camif1],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_ic_prpvf] = {
+		.id = &isd_id[isd_ic_prpvf],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 4,
+			.link[0] = {
+				.remote_id = &isd_id[isd_camif0],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id = &isd_id[isd_camif1],
+				.remote_pad = 0,
+			},
+			.link[2] = {
+				.remote_id =  &isd_id[isd_ic_pp0],
+				.remote_pad = 0,
+			},
+			.link[3] = {
+				.remote_id =  &isd_id[isd_ic_pp1],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_ic_pp0] = {
+		.id = &isd_id[isd_ic_pp0],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 2,
+			.link[0] = {
+				.remote_id = &isd_id[isd_camif0],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id = &isd_id[isd_camif1],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_ic_pp1] = {
+		.id = &isd_id[isd_ic_pp1],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.num_links = 2,
+			.link[0] = {
+				.remote_id = &isd_id[isd_camif0],
+				.remote_pad = 0,
+			},
+			.link[1] = {
+				.remote_id = &isd_id[isd_camif1],
+				.remote_pad = 0,
+			},
+		},
+	},
+
+	[isd_camif0] = {
+		.id = &isd_id[isd_camif0],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.devnode = true,
+		},
+	},
+
+	[isd_camif1] = {
+		.id = &isd_id[isd_camif1],
+		.num_sink_pads = 1,
+		.num_src_pads = 1,
+		.pad[1] = {
+			.devnode = true,
+		},
+	},
+};
+
+/* form a device name given a group id and ipu id */
+static inline void isd_id_to_devname(char *devname, int sz,
+				     const struct internal_subdev_id *id,
+				     int ipu_id)
+{
+	int pdev_id = ipu_id * num_isd + id->index;
+
+	snprintf(devname, sz, "%s.%d", id->name, pdev_id);
+}
+
+/* adds the links from given internal subdev */
+static int add_internal_links(struct imx_media_dev *imxmd,
+			      const struct internal_subdev *isd,
+			      struct imx_media_subdev *imxsd,
+			      int ipu_id)
+{
+	int i, num_pads, ret;
+
+	num_pads = isd->num_sink_pads + isd->num_src_pads;
+
+	for (i = 0; i < num_pads; i++) {
+		const struct internal_pad *intpad = &isd->pad[i];
+		struct imx_media_pad *pad = &imxsd->pad[i];
+		int j;
+
+		/* init the pad flags for this internal subdev */
+		pad->pad.flags = (i < isd->num_sink_pads) ?
+			MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
+		/* export devnode pad flag to the subdevs */
+		pad->devnode = intpad->devnode;
+
+		for (j = 0; j < intpad->num_links; j++) {
+			const struct internal_link *link;
+			char remote_devname[32];
+
+			link = &intpad->link[j];
+
+			if (link->remote_id->grp_id == 0)
+				continue;
+
+			isd_id_to_devname(remote_devname,
+					  sizeof(remote_devname),
+					  link->remote_id, ipu_id);
+
+			ret = imx_media_add_pad_link(imxmd, pad,
+						     NULL, remote_devname,
+						     i, link->remote_pad);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
+/* register an internal subdev as a platform device */
+static struct imx_media_subdev *
+add_internal_subdev(struct imx_media_dev *imxmd,
+		    const struct internal_subdev *isd,
+		    int ipu_id)
+{
+	struct imx_media_internal_sd_platformdata pdata;
+	struct platform_device_info pdevinfo = {0};
+	struct imx_media_subdev *imxsd;
+	struct platform_device *pdev;
+
+	switch (isd->id->grp_id) {
+	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
+		pdata.grp_id = isd->id->grp_id +
+			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
+		break;
+	default:
+		pdata.grp_id = isd->id->grp_id;
+		break;
+	}
+
+	/* the id of IPU this subdev will control */
+	pdata.ipu_id = ipu_id;
+
+	/* create subdev name */
+	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
+				    pdata.grp_id, ipu_id);
+
+	pdevinfo.name = isd->id->name;
+	pdevinfo.id = ipu_id * num_isd + isd->id->index;
+	pdevinfo.parent = imxmd->dev;
+	pdevinfo.data = &pdata;
+	pdevinfo.size_data = sizeof(pdata);
+	pdevinfo.dma_mask = DMA_BIT_MASK(32);
+
+	pdev = platform_device_register_full(&pdevinfo);
+	if (IS_ERR(pdev))
+		return ERR_CAST(pdev);
+
+	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
+	if (IS_ERR(imxsd))
+		return imxsd;
+
+	imxsd->num_sink_pads = isd->num_sink_pads;
+	imxsd->num_src_pads = isd->num_src_pads;
+
+	return imxsd;
+}
+
+/* adds the internal subdevs in one ipu */
+static int add_ipu_internal_subdevs(struct imx_media_dev *imxmd,
+				    struct imx_media_subdev *csi0,
+				    struct imx_media_subdev *csi1,
+				    int ipu_id)
+{
+	enum isd_enum i;
+	int ret;
+
+	for (i = 0; i < num_isd; i++) {
+		const struct internal_subdev *isd = &internal_subdev[i];
+		struct imx_media_subdev *imxsd;
+
+		/*
+		 * the CSIs are represented in the device-tree, so those
+		 * devices are added already, and are added to the async
+		 * subdev list by of_parse_subdev(), so we are given those
+		 * subdevs as csi0 and csi1.
+		 */
+		switch (isd->id->grp_id) {
+		case IMX_MEDIA_GRP_ID_CSI0:
+			imxsd = csi0;
+			break;
+		case IMX_MEDIA_GRP_ID_CSI1:
+			imxsd = csi1;
+			break;
+		default:
+			imxsd = add_internal_subdev(imxmd, isd, ipu_id);
+			break;
+		}
+
+		if (IS_ERR(imxsd))
+			return PTR_ERR(imxsd);
+
+		/* add the links from this subdev */
+		if (imxsd) {
+			ret = add_internal_links(imxmd, isd, imxsd, ipu_id);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
+int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *csi[4])
+{
+	int ret;
+
+	/* there must be at least one CSI in first IPU */
+	if (!(csi[0] || csi[1]))
+		return -EINVAL;
+
+	ret = add_ipu_internal_subdevs(imxmd, csi[0], csi[1], 0);
+	if (ret)
+		return ret;
+
+	if (csi[2] || csi[3])
+		ret = add_ipu_internal_subdevs(imxmd, csi[2], csi[3], 1);
+
+	return ret;
+}
diff --git a/drivers/staging/media/imx/imx-media-of.c b/drivers/staging/media/imx/imx-media-of.c
new file mode 100644
index 0000000..a939c34
--- /dev/null
+++ b/drivers/staging/media/imx/imx-media-of.c
@@ -0,0 +1,289 @@
+/*
+ * Media driver for Freescale i.MX5/6 SOC
+ *
+ * Open Firmware parsing.
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/of_platform.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+static int of_add_pad_link(struct imx_media_dev *imxmd,
+			   struct imx_media_pad *pad,
+			   struct device_node *local_sd_node,
+			   struct device_node *remote_sd_node,
+			   int local_pad, int remote_pad)
+{
+	dev_dbg(imxmd->dev, "%s: adding %s:%d -> %s:%d\n", __func__,
+		local_sd_node->name, local_pad,
+		remote_sd_node->name, remote_pad);
+
+	return imx_media_add_pad_link(imxmd, pad, remote_sd_node, NULL,
+				      local_pad, remote_pad);
+}
+
+/* parse inputs property from a sensor node */
+static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *sensor,
+				   struct device_node *sensor_np)
+{
+	struct imx_media_sensor_input *sinput = &sensor->input;
+	int ret, i;
+
+	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
+		const char *input_name;
+		u32 val;
+
+		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
+		if (ret)
+			break;
+
+		sinput->value[i] = val;
+
+		ret = of_property_read_string_index(sensor_np, "input-names",
+						    i, &input_name);
+		/*
+		 * if input-names not provided, they will be set using
+		 * the subdev name once the sensor is known during
+		 * async bind
+		 */
+		if (!ret)
+			strncpy(sinput->name[i], input_name,
+				sizeof(sinput->name[i]));
+	}
+
+	sinput->num = i;
+
+	/* if no inputs provided just assume a single input */
+	if (sinput->num == 0)
+		sinput->num = 1;
+}
+
+static void of_parse_sensor(struct imx_media_dev *imxmd,
+			    struct imx_media_subdev *sensor,
+			    struct device_node *sensor_np)
+{
+	struct device_node *endpoint;
+
+	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
+
+	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
+	if (endpoint) {
+		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
+		of_node_put(endpoint);
+	}
+}
+
+static int of_get_port_count(const struct device_node *np)
+{
+	struct device_node *child;
+	int num = 0;
+
+	/* if this node is itself a port, return 1 */
+	if (of_node_cmp(np->name, "port") == 0)
+		return 1;
+
+	for_each_child_of_node(np, child)
+		if (of_node_cmp(child->name, "port") == 0)
+			num++;
+
+	return num;
+}
+
+/*
+ * find the remote device node and remote port id (remote pad #)
+ * given local endpoint node
+ */
+static void of_get_remote_pad(struct device_node *epnode,
+			      struct device_node **remote_node,
+			      int *remote_pad)
+{
+	struct device_node *rp, *rpp;
+	struct device_node *remote;
+
+	rp = of_graph_get_remote_port(epnode);
+	rpp = of_graph_get_remote_port_parent(epnode);
+
+	if (of_device_is_compatible(rpp, "fsl,imx6q-ipu")) {
+		/* the remote is one of the CSI ports */
+		remote = rp;
+		*remote_pad = 0;
+		of_node_put(rpp);
+	} else {
+		remote = rpp;
+		of_property_read_u32(rp, "reg", remote_pad);
+		of_node_put(rp);
+	}
+
+	if (!remote || !of_device_is_available(remote)) {
+		of_node_put(remote);
+		*remote_node = NULL;
+	} else {
+		*remote_node = remote;
+	}
+}
+
+static struct imx_media_subdev *
+of_parse_subdev(struct imx_media_dev *imxmd, struct device_node *sd_np,
+		bool is_csi_port)
+{
+	struct imx_media_subdev *imxsd;
+	int i, num_pads, ret;
+
+	if (!of_device_is_available(sd_np)) {
+		dev_dbg(imxmd->dev, "%s: %s not enabled\n", __func__,
+			sd_np->name);
+		return NULL;
+	}
+
+	/* register this subdev with async notifier */
+	imxsd = imx_media_add_async_subdev(imxmd, sd_np, NULL);
+	if (IS_ERR_OR_NULL(imxsd))
+		return imxsd;
+
+	if (is_csi_port) {
+		/*
+		 * the ipu-csi has one sink port and one source port.
+		 * The source port is not represented in the device tree,
+		 * but is described by the internal pads and links later.
+		 */
+		num_pads = 2;
+		imxsd->num_sink_pads = 1;
+	} else if (of_device_is_compatible(sd_np, "fsl,imx6-mipi-csi2")) {
+		num_pads = of_get_port_count(sd_np);
+		/* the mipi csi2 receiver has only one sink port */
+		imxsd->num_sink_pads = 1;
+	} else if (of_device_is_compatible(sd_np, "video-multiplexer")) {
+		num_pads = of_get_port_count(sd_np);
+		/* for the video mux, all but the last port are sinks */
+		imxsd->num_sink_pads = num_pads - 1;
+	} else {
+		/* must be a sensor */
+		num_pads = 1;
+		imxsd->num_sink_pads = 0;
+	}
+
+	if (imxsd->num_sink_pads >= num_pads)
+		return ERR_PTR(-EINVAL);
+
+	imxsd->num_src_pads = num_pads - imxsd->num_sink_pads;
+
+	dev_dbg(imxmd->dev, "%s: %s has %d pads (%d sink, %d src)\n",
+		__func__, sd_np->name, num_pads,
+		imxsd->num_sink_pads, imxsd->num_src_pads);
+
+	if (imxsd->num_sink_pads == 0)
+		of_parse_sensor(imxmd, imxsd, sd_np);
+
+	for (i = 0; i < num_pads; i++) {
+		struct device_node *epnode = NULL, *port, *remote_np;
+		struct imx_media_subdev *remote_imxsd;
+		struct imx_media_pad *pad;
+		int remote_pad;
+
+		/* init this pad */
+		pad = &imxsd->pad[i];
+		pad->pad.flags = (i < imxsd->num_sink_pads) ?
+			MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
+
+		if (is_csi_port)
+			port = (i < imxsd->num_sink_pads) ? sd_np : NULL;
+		else
+			port = of_graph_get_port_by_id(sd_np, i);
+		if (!port)
+			continue;
+
+		for_each_child_of_node(port, epnode) {
+			of_get_remote_pad(epnode, &remote_np, &remote_pad);
+			if (!remote_np)
+				continue;
+
+			ret = of_add_pad_link(imxmd, pad, sd_np, remote_np,
+					      i, remote_pad);
+			if (ret) {
+				imxsd = ERR_PTR(ret);
+				break;
+			}
+
+			if (i < imxsd->num_sink_pads) {
+				/* follow sink endpoints upstream */
+				remote_imxsd = of_parse_subdev(imxmd,
+							       remote_np,
+							       false);
+				if (IS_ERR(remote_imxsd)) {
+					imxsd = remote_imxsd;
+					break;
+				}
+			}
+
+			of_node_put(remote_np);
+		}
+
+		if (port != sd_np)
+			of_node_put(port);
+		if (IS_ERR(imxsd)) {
+			of_node_put(remote_np);
+			of_node_put(epnode);
+			break;
+		}
+	}
+
+	return imxsd;
+}
+
+int imx_media_of_parse(struct imx_media_dev *imxmd,
+		       struct imx_media_subdev *(*csi)[4],
+		       struct device_node *np)
+{
+	struct imx_media_subdev *lcsi;
+	struct device_node *csi_np;
+	u32 ipu_id, csi_id;
+	int i, ret;
+
+	for (i = 0; ; i++) {
+		csi_np = of_parse_phandle(np, "ports", i);
+		if (!csi_np)
+			break;
+
+		lcsi = of_parse_subdev(imxmd, csi_np, true);
+		if (IS_ERR(lcsi)) {
+			ret = PTR_ERR(lcsi);
+			goto err_put;
+		}
+
+		ret = of_property_read_u32(csi_np, "reg", &csi_id);
+		if (ret) {
+			dev_err(imxmd->dev,
+				"%s: csi port missing reg property!\n",
+				__func__);
+			goto err_put;
+		}
+
+		ipu_id = of_alias_get_id(csi_np->parent, "ipu");
+		of_node_put(csi_np);
+
+		if (ipu_id > 1 || csi_id > 1) {
+			dev_err(imxmd->dev, "%s: invalid ipu/csi id (%u/%u)\n",
+				__func__, ipu_id, csi_id);
+			return -EINVAL;
+		}
+
+		(*csi)[ipu_id * 2 + csi_id] = lcsi;
+	}
+
+	return 0;
+err_put:
+	of_node_put(csi_np);
+	return ret;
+}
diff --git a/drivers/staging/media/imx/imx-media.h b/drivers/staging/media/imx/imx-media.h
new file mode 100644
index 0000000..dbb2d8a
--- /dev/null
+++ b/drivers/staging/media/imx/imx-media.h
@@ -0,0 +1,310 @@
+/*
+ * V4L2 Media Controller Driver for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#ifndef _IMX_MEDIA_H
+#define _IMX_MEDIA_H
+
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+
+/*
+ * This is somewhat arbitrary, but we need at least:
+ * - 2 camera interface subdevs
+ * - 3 IC subdevs
+ * - 2 CSI subdevs
+ * - 1 mipi-csi2 receiver subdev
+ * - 2 video-mux subdevs
+ * - 3 camera sensor subdevs (2 parallel, 1 mipi-csi2)
+ *
+ * And double the above numbers for quad i.mx!
+ */
+#define IMX_MEDIA_MAX_SUBDEVS       48
+/* max pads per subdev */
+#define IMX_MEDIA_MAX_PADS          16
+/* max links per pad */
+#define IMX_MEDIA_MAX_LINKS          8
+
+/* How long to wait for EOF interrupts in the buffer-capture subdevs */
+#define IMX_MEDIA_EOF_TIMEOUT       1000
+
+/* A sensor's inputs parsed from a sensor node */
+#define IMX_MEDIA_MAX_SENSOR_INPUTS 16
+struct imx_media_sensor_input {
+	/* number of inputs */
+	int num;
+	/* input values passed to s_routing */
+	u32 value[IMX_MEDIA_MAX_SENSOR_INPUTS];
+	/* input names */
+	char name[IMX_MEDIA_MAX_SENSOR_INPUTS][32];
+};
+
+struct imx_media_pixfmt {
+	u32     fourcc;
+	u32     codes[4];
+	int     bpp;     /* total bpp */
+	enum ipu_color_space cs;
+	bool    planar;  /* is a planar format */
+};
+
+struct imx_media_buffer {
+	struct vb2_v4l2_buffer vbuf; /* v4l buffer must be first */
+	struct list_head  list;
+};
+
+static inline struct imx_media_buffer *to_imx_media_vb(struct vb2_buffer *vb)
+{
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+
+	return container_of(vbuf, struct imx_media_buffer, vbuf);
+}
+
+struct imx_media_link {
+	struct device_node *remote_sd_node;
+	char               remote_devname[32];
+	int                local_pad;
+	int                remote_pad;
+};
+
+struct imx_media_pad {
+	struct media_pad  pad;
+	struct imx_media_link link[IMX_MEDIA_MAX_LINKS];
+	bool devnode; /* does this pad link to a device node */
+	int num_links;
+};
+
+struct imx_media_internal_sd_platformdata {
+	char sd_name[V4L2_SUBDEV_NAME_SIZE];
+	u32 grp_id;
+	int ipu_id;
+};
+
+struct imx_media_subdev {
+	struct v4l2_async_subdev asd;
+	struct v4l2_subdev       *sd; /* set when bound */
+
+	struct imx_media_pad     pad[IMX_MEDIA_MAX_PADS];
+	int num_sink_pads;
+	int num_src_pads;
+
+	/* the devname is needed for async devname match */
+	char devname[32];
+
+	/* if this is a sensor */
+	struct imx_media_sensor_input input;
+	struct v4l2_of_endpoint sensor_ep;
+};
+
+struct imx_media_dev {
+	struct media_device md;
+	struct v4l2_device  v4l2_dev;
+	struct device *dev;
+
+	/* master subdev list */
+	struct imx_media_subdev subdev[IMX_MEDIA_MAX_SUBDEVS];
+	int num_subdevs;
+
+	/* IPUs this media driver control, valid after subdevs bound */
+	struct ipu_soc *ipu[2];
+
+	/* used during link_notify */
+	struct media_entity_graph link_notify_graph;
+
+	/* for async subdev registration */
+	struct v4l2_async_subdev *async_ptrs[IMX_MEDIA_MAX_SUBDEVS];
+	struct v4l2_async_notifier subdev_notifier;
+};
+
+const struct imx_media_pixfmt *imx_media_find_format(u32 fourcc, u32 code,
+						     bool allow_rgb,
+						     bool allow_planar);
+int imx_media_enum_format(u32 *code, u32 index,
+			  bool allow_rgb, bool allow_planar);
+
+int imx_media_init_mbus_fmt(struct v4l2_mbus_framefmt *mbus,
+			    u32 width, u32 height, u32 code, u32 field,
+			    const struct imx_media_pixfmt **cc);
+
+int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *fmt,
+				  struct v4l2_mbus_framefmt *mbus);
+int imx_media_mbus_fmt_to_ipu_image(struct ipu_image *image,
+				    struct v4l2_mbus_framefmt *mbus);
+int imx_media_ipu_image_to_mbus_fmt(struct v4l2_mbus_framefmt *mbus,
+				    struct ipu_image *image);
+
+struct imx_media_subdev *
+imx_media_find_async_subdev(struct imx_media_dev *imxmd,
+			    struct device_node *np,
+			    const char *devname);
+struct imx_media_subdev *
+imx_media_add_async_subdev(struct imx_media_dev *imxmd,
+			   struct device_node *np,
+			   const char *devname);
+int imx_media_add_pad_link(struct imx_media_dev *imxmd,
+			   struct imx_media_pad *pad,
+			   struct device_node *remote_node,
+			   const char *remote_devname,
+			   int local_pad, int remote_pad);
+
+void imx_media_grp_id_to_sd_name(char *sd_name, int sz,
+				 u32 grp_id, int ipu_id);
+
+int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *csi[4]);
+
+struct imx_media_subdev *
+imx_media_find_subdev_by_sd(struct imx_media_dev *imxmd,
+			    struct v4l2_subdev *sd);
+struct imx_media_subdev *
+imx_media_find_subdev_by_id(struct imx_media_dev *imxmd,
+			    u32 grp_id);
+int imx_media_find_mipi_csi2_channel(struct imx_media_dev *imxmd,
+				     struct media_entity *start_entity);
+struct imx_media_subdev *
+imx_media_find_pipeline_subdev(struct imx_media_dev *imxmd,
+			       struct media_entity *start_entity,
+			       u32 grp_id);
+struct imx_media_subdev *
+__imx_media_find_sensor(struct imx_media_dev *imxmd,
+			struct media_entity *start_entity);
+struct imx_media_subdev *
+imx_media_find_sensor(struct imx_media_dev *imxmd,
+		      struct media_entity *start_entity);
+
+enum imx_media_dma_buf_status {
+	IMX_MEDIA_BUF_STATUS_PREPARED = 0,
+	IMX_MEDIA_BUF_STATUS_QUEUED,
+	IMX_MEDIA_BUF_STATUS_ACTIVE,
+	IMX_MEDIA_BUF_STATUS_DONE,
+	IMX_MEDIA_BUF_STATUS_ERROR,
+};
+
+struct imx_media_dma_buf_ring;
+
+struct imx_media_dma_buf {
+	/* owning ring if any */
+	struct imx_media_dma_buf_ring *ring;
+	/* if !NULL this is a vb2_buffer */
+	struct vb2_buffer *vb;
+	void          *virt;
+	dma_addr_t     phys;
+	unsigned long  len;
+	int            index;
+	unsigned long  seq;
+	/* buffer state */
+	enum imx_media_dma_buf_status state;
+	/* completion status */
+	enum imx_media_dma_buf_status status;
+};
+
+enum imx_media_priv_ioctl {
+	IMX_MEDIA_REQ_DMA_BUF_SINK_RING = 1, /* src requests ring from sink */
+	IMX_MEDIA_REQ_DMA_BUF_SRC_RING,  /* sink requests ring from src */
+	IMX_MEDIA_NEW_DMA_BUF,           /* src hands new buffer to sink */
+	IMX_MEDIA_REL_DMA_BUF_SINK_RING, /* src informs sink that its ring
+					    can be released */
+	IMX_MEDIA_REL_DMA_BUF_SRC_RING,  /* sink informs src that its ring
+					    can be released */
+};
+
+#define IMX_MEDIA_MIN_RING_BUFS 2
+/* prpvf needs at least 3 buffers */
+#define IMX_MEDIA_MIN_RING_BUFS_PRPVF 3
+#define IMX_MEDIA_MAX_RING_BUFS 8
+void imx_media_free_dma_buf(struct imx_media_dev *imxmd,
+			    struct imx_media_dma_buf *buf);
+int imx_media_alloc_dma_buf(struct imx_media_dev *imxmd,
+			    struct imx_media_dma_buf *buf,
+			    int size);
+void imx_media_free_dma_buf_ring(struct imx_media_dma_buf_ring *ring);
+struct imx_media_dma_buf_ring *
+imx_media_alloc_dma_buf_ring(struct imx_media_dev *imxmd,
+			     struct media_entity *src,
+			     struct media_entity *sink,
+			     int size, int num_bufs,
+			     bool alloc_bufs);
+int imx_media_dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index);
+int imx_media_dma_buf_queue_from_vb(struct imx_media_dma_buf_ring *ring,
+				    struct vb2_buffer *vb);
+void imx_media_dma_buf_done(struct imx_media_dma_buf *buf,
+			    enum imx_media_dma_buf_status status);
+struct imx_media_dma_buf *
+imx_media_dma_buf_dequeue(struct imx_media_dma_buf_ring *ring);
+struct imx_media_dma_buf *
+imx_media_dma_buf_get_active(struct imx_media_dma_buf_ring *ring);
+int imx_media_dma_buf_set_active(struct imx_media_dma_buf *buf);
+struct imx_media_dma_buf *
+imx_media_dma_buf_get_next_queued(struct imx_media_dma_buf_ring *ring);
+struct imx_media_dma_buf *
+imx_media_dma_buf_get(struct imx_media_dma_buf_ring *ring, int index);
+
+int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
+				 struct media_entity_graph *graph,
+				 struct media_entity *entity, bool on);
+int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
+				  struct media_entity *entity,
+				  struct media_pipeline *pipe,
+				  bool on);
+int imx_media_inherit_controls(struct imx_media_dev *imxmd,
+			       struct video_device *vfd,
+			       struct media_entity *start_entity);
+
+/* imx-media-fim.c */
+struct imx_media_fim;
+void imx_media_fim_eof_monitor(struct imx_media_fim *fim, struct timespec *ts);
+int imx_media_fim_set_power(struct imx_media_fim *fim, bool on);
+int imx_media_fim_set_stream(struct imx_media_fim *fim,
+			     struct imx_media_subdev *sensor,
+			     bool on);
+struct imx_media_fim *imx_media_fim_init(struct v4l2_subdev *sd);
+void imx_media_fim_free(struct imx_media_fim *fim);
+
+/* imx-media-of.c */
+struct imx_media_subdev *
+imx_media_of_find_subdev(struct imx_media_dev *imxmd,
+			 struct device_node *np,
+			 const char *name);
+int imx_media_of_parse(struct imx_media_dev *dev,
+		       struct imx_media_subdev *(*csi)[4],
+		       struct device_node *np);
+
+/* subdev group ids */
+#define IMX_MEDIA_GRP_ID_SENSOR    (1 << 8)
+#define IMX_MEDIA_GRP_ID_VIDMUX    (1 << 9)
+#define IMX_MEDIA_GRP_ID_CSI2      (1 << 10)
+#define IMX_MEDIA_GRP_ID_CSI_BIT   11
+#define IMX_MEDIA_GRP_ID_CSI       (0x3 << 11)
+#define IMX_MEDIA_GRP_ID_CSI0      (1 << 11)
+#define IMX_MEDIA_GRP_ID_CSI1      (2 << 11)
+#define IMX_MEDIA_GRP_ID_SMFC_BIT  13
+#define IMX_MEDIA_GRP_ID_SMFC      (0x7 << 13)
+#define IMX_MEDIA_GRP_ID_SMFC0     (1 << 13)
+#define IMX_MEDIA_GRP_ID_SMFC1     (2 << 13)
+#define IMX_MEDIA_GRP_ID_SMFC2     (3 << 13)
+#define IMX_MEDIA_GRP_ID_SMFC3     (4 << 13)
+#define IMX_MEDIA_GRP_ID_IC_PRPENC (1 << 16)
+#define IMX_MEDIA_GRP_ID_IC_PRPVF  (1 << 17)
+#define IMX_MEDIA_GRP_ID_IC_PP_BIT 18
+#define IMX_MEDIA_GRP_ID_IC_PP     (0x7 << 18)
+#define IMX_MEDIA_GRP_ID_IC_PP0    (1 << 18)
+#define IMX_MEDIA_GRP_ID_IC_PP1    (2 << 18)
+#define IMX_MEDIA_GRP_ID_IC_PP2    (3 << 18)
+#define IMX_MEDIA_GRP_ID_IC_PP3    (4 << 18)
+#define IMX_MEDIA_GRP_ID_CAMIF_BIT 21
+#define IMX_MEDIA_GRP_ID_CAMIF     (0x7 << 21)
+#define IMX_MEDIA_GRP_ID_CAMIF0    (1 << 21)
+#define IMX_MEDIA_GRP_ID_CAMIF1    (2 << 21)
+#define IMX_MEDIA_GRP_ID_CAMIF2    (3 << 21)
+#define IMX_MEDIA_GRP_ID_CAMIF3    (4 << 21)
+
+#endif
diff --git a/include/media/imx.h b/include/media/imx.h
new file mode 100644
index 0000000..5025a72
--- /dev/null
+++ b/include/media/imx.h
@@ -0,0 +1,15 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */
+
+#ifndef __MEDIA_IMX_H__
+#define __MEDIA_IMX_H__
+
+#include <uapi/media/imx.h>
+
+#endif
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index 0d2e1e0..6c29f42 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -180,6 +180,10 @@ enum v4l2_colorfx {
  * We reserve 16 controls for this driver. */
 #define V4L2_CID_USER_TC358743_BASE		(V4L2_CID_USER_BASE + 0x1080)
 
+/* The base for the imx driver controls.
+ * We reserve 16 controls for this driver. */
+#define V4L2_CID_USER_IMX_BASE			(V4L2_CID_USER_BASE + 0x1090)
+
 /* MPEG-class control IDs */
 /* The MPEG controls are applicable to all codec controls
  * and the 'MPEG' part of the define is historical */
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

This is a media entity subdevice for the i.MX Camera
Serial Interface module.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Kconfig   |  13 +
 drivers/staging/media/imx/Makefile  |   2 +
 drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
 3 files changed, 659 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-csi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index bfde58d..ce2d2c8 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
 	  Say yes here to enable support for video4linux media controller
 	  driver for the i.MX5/6 SOC.
 
+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA
+	tristate "i.MX5/6 Camera driver"
+	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
+	select VIDEOBUF2_DMA_CONTIG
+	default y
+	---help---
+	  A video4linux camera capture driver for i.MX5/6.
+
+endmenu
+endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index ef9f11b..133672a 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
new file mode 100644
index 0000000..64ef862
--- /dev/null
+++ b/drivers/staging/media/imx/imx-csi.c
@@ -0,0 +1,644 @@
+/*
+ * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+#define CSI_NUM_PADS 2
+
+struct csi_priv {
+	struct device *dev;
+	struct ipu_soc *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev sd;
+	struct media_pad pad[CSI_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct v4l2_rect crop;
+	struct ipu_csi *csi;
+	int csi_id;
+	int input_pad;
+	int output_pad;
+	bool power_on;  /* power is on */
+	bool stream_on; /* streaming is on */
+
+	/* the sink for the captured frames */
+	struct v4l2_subdev *sink_sd;
+	enum ipu_csi_dest dest;
+	struct v4l2_subdev *src_sd;
+
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	struct imx_media_fim *fim;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+};
+
+static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct csi_priv, sd);
+}
+
+/* Update the CSI whole sensor and active windows */
+static int csi_setup(struct csi_priv *priv)
+{
+	struct v4l2_mbus_framefmt infmt;
+
+	ipu_csi_set_window(priv->csi, &priv->crop);
+
+	/*
+	 * the ipu-csi doesn't understand ALTERNATE, but it only
+	 * needs to know whether the stream is interlaced, so set
+	 * to INTERLACED if infmt field is ALTERNATE.
+	 */
+	infmt = priv->format_mbus[priv->input_pad];
+	if (infmt.field == V4L2_FIELD_ALTERNATE)
+		infmt.field = V4L2_FIELD_INTERLACED;
+
+	ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
+
+	ipu_csi_set_dest(priv->csi, priv->dest);
+
+	ipu_csi_dump(priv->csi);
+
+	return 0;
+}
+
+static int csi_start(struct csi_priv *priv)
+{
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = csi_setup(priv);
+	if (ret)
+		return ret;
+
+	/* start the frame interval monitor */
+	if (priv->fim) {
+		ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);
+		if (ret)
+			return ret;
+	}
+
+	ret = ipu_csi_enable(priv->csi);
+	if (ret) {
+		v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
+		goto fim_off;
+	}
+
+	return 0;
+
+fim_off:
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+	return ret;
+}
+
+static void csi_stop(struct csi_priv *priv)
+{
+	/* stop the frame interval monitor */
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+
+	ipu_csi_disable(priv->csi);
+}
+
+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = csi_start(priv);
+	else if (!enable && priv->stream_on)
+		csi_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (priv->fim && on != priv->power_on)
+		ret = imx_media_fim_set_power(priv->fim, on);
+
+	if (!ret)
+		priv->power_on = on;
+	return ret;
+}
+
+static int csi_link_setup(struct media_entity *entity,
+			  const struct media_pad *local,
+			  const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SINK) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->sink_sd)
+			return -EBUSY;
+		priv->sink_sd = remote_sd;
+	} else {
+		priv->sink_sd = NULL;
+		return 0;
+	}
+
+	/* set CSI destination */
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_SMFC0:
+	case IMX_MEDIA_GRP_ID_SMFC1:
+	case IMX_MEDIA_GRP_ID_SMFC2:
+	case IMX_MEDIA_GRP_ID_SMFC3:
+		priv->dest = IPU_CSI_DEST_IDMAC;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		priv->dest = IPU_CSI_DEST_VDIC;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		priv->dest = IPU_CSI_DEST_IC;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int csi_link_validate(struct v4l2_subdev *sd,
+			     struct media_link *link,
+			     struct v4l2_subdev_format *source_fmt,
+			     struct v4l2_subdev_format *sink_fmt)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	bool is_csi2;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+	if (is_csi2) {
+		int vc_num = 0;
+		/*
+		 * NOTE! It seems the virtual channels from the mipi csi-2
+		 * receiver are used only for routing by the video mux's,
+		 * or for hard-wired routing to the CSI's. Once the stream
+		 * enters the CSI's however, they are treated internally
+		 * in the IPU as virtual channel 0.
+		 */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+							  &priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
+					  &priv->format_mbus[priv->input_pad]);
+	}
+
+	/* select either parallel or MIPI-CSI2 as input to CSI */
+	ipu_set_csi_src_mux(priv->ipu, priv->csi_id, is_csi2);
+
+	return 0;
+}
+
+static int csi_eof_isr(struct v4l2_subdev *sd, u32 status, bool *handled)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (priv->fim) {
+		struct timespec cur_ts;
+
+		ktime_get_ts(&cur_ts);
+		/* call frame interval monitor */
+		imx_media_fim_eof_monitor(priv->fim, &cur_ts);
+	}
+
+	return 0;
+}
+
+static int csi_try_crop(struct csi_priv *priv, struct v4l2_rect *crop)
+{
+	struct v4l2_mbus_framefmt *infmt;
+	struct imx_media_subdev *sensor;
+	v4l2_std_id std;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	ret = v4l2_subdev_call(sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+
+	crop->width = min_t(__u32, infmt->width, crop->width);
+	if (crop->left + crop->width > infmt->width)
+		crop->left = infmt->width - crop->width;
+	/* adjust crop left/width to h/w alignment restrictions */
+	crop->left &= ~0x3;
+	crop->width &= ~0x7;
+
+	/*
+	 * FIXME: not sure why yet, but on interlaced bt.656,
+	 * changing the vertical cropping causes loss of vertical
+	 * sync, so fix it to NTSC/PAL active lines. NTSC contains
+	 * 2 extra lines of active video that need to be cropped.
+	 */
+	if (priv->sensor_mbus_cfg.type == V4L2_MBUS_BT656) {
+		ret = v4l2_subdev_call(sensor->sd, video, g_std, &std);
+		if (ret)
+			return ret;
+		if (std & V4L2_STD_525_60) {
+			crop->top = 2;
+			crop->height = 480;
+		} else {
+			crop->top = 0;
+			crop->height = 576;
+		}
+	} else {
+		crop->height = min_t(__u32, infmt->height, crop->height);
+		if (crop->top + crop->height > infmt->height)
+			crop->top = infmt->height - crop->height;
+	}
+
+	return 0;
+}
+
+static int csi_get_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);
+		if (ret)
+			return ret;
+		sdformat->format.width = crop.width;
+		sdformat->format.height = crop.height;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		/* Update the crop window if this is output pad  */
+		if (sdformat->pad == priv->output_pad)
+			priv->crop = crop;
+	}
+
+	return 0;
+}
+
+static int csi_get_selection(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_selection *sel)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt;
+
+	if (sel->pad != priv->output_pad)
+		return -EINVAL;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+
+	switch (sel->target) {
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		sel->r.left = 0;
+		sel->r.top = 0;
+		sel->r.width = infmt->width;
+		sel->r.height = infmt->height;
+		break;
+	case V4L2_SEL_TGT_CROP:
+		sel->r = priv->crop;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int csi_set_selection(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_selection *sel)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *outfmt;
+	int ret;
+
+	if (sel->pad != priv->output_pad ||
+	    sel->target != V4L2_SEL_TGT_CROP)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	/*
+	 * Modifying the crop rectangle always changes the format on the source
+	 * pad. If the KEEP_CONFIG flag is set, just return the current crop
+	 * rectangle.
+	 */
+	if (sel->flags & V4L2_SEL_FLAG_KEEP_CONFIG) {
+		sel->r = priv->crop;
+		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+			cfg->try_crop = sel->r;
+		return 0;
+	}
+
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	ret = csi_try_crop(priv, &sel->r);
+	if (ret)
+		return ret;
+
+	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_crop = sel->r;
+	} else {
+		priv->crop = sel->r;
+		/* Update the source format */
+		outfmt->width = sel->r.width;
+		outfmt->height = sel->r.height;
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int csi_registered(struct v4l2_subdev *sd)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	/* get handle to IPU CSI */
+	priv->csi = ipu_csi_get(priv->ipu, priv->csi_id);
+	if (IS_ERR(priv->csi)) {
+		v4l2_err(&priv->sd, "failed to get CSI %d\n", priv->csi_id);
+		return PTR_ERR(priv->csi);
+	}
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd)) {
+		ret = PTR_ERR(imxsd);
+		goto put_csi;
+	}
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+		ret = -EINVAL;
+		goto put_csi;
+	}
+
+	for (i = 0; i < CSI_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      NULL);
+		if (ret)
+			goto put_csi;
+	}
+
+	priv->fim = imx_media_fim_init(&priv->sd);
+	if (IS_ERR(priv->fim)) {
+		ret = PTR_ERR(priv->fim);
+		goto put_csi;
+	}
+
+	ret = media_entity_pads_init(&sd->entity, CSI_NUM_PADS, priv->pad);
+	if (ret)
+		goto free_fim;
+
+	return 0;
+free_fim:
+	if (priv->fim)
+		imx_media_fim_free(priv->fim);
+put_csi:
+	ipu_csi_put(priv->csi);
+	return ret;
+}
+
+static struct media_entity_operations csi_entity_ops = {
+	.link_setup = csi_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops csi_core_ops = {
+	.s_power = csi_s_power,
+	.interrupt_service_routine = csi_eof_isr,
+};
+
+static struct v4l2_subdev_video_ops csi_video_ops = {
+	.s_stream = csi_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops csi_pad_ops = {
+	.get_fmt = csi_get_fmt,
+	.set_fmt = csi_set_fmt,
+	.get_selection = csi_get_selection,
+	.set_selection = csi_set_selection,
+	.link_validate = csi_link_validate,
+};
+
+static struct v4l2_subdev_ops csi_subdev_ops = {
+	.core = &csi_core_ops,
+	.video = &csi_video_ops,
+	.pad = &csi_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops csi_internal_ops = {
+	.registered = csi_registered,
+};
+
+static int imx_csi_probe(struct platform_device *pdev)
+{
+	struct ipu_client_platformdata *pdata;
+	struct csi_priv *priv;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	/* get parent IPU */
+	priv->ipu = dev_get_drvdata(priv->dev->parent);
+
+	/* get our CSI id */
+	pdata = priv->dev->platform_data;
+	priv->csi_id = pdata->csi;
+
+	v4l2_subdev_init(&priv->sd, &csi_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &csi_internal_ops;
+	priv->sd.entity.ops = &csi_entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.grp_id = priv->csi_id ?
+		IMX_MEDIA_GRP_ID_CSI1 : IMX_MEDIA_GRP_ID_CSI0;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	imx_media_grp_id_to_sd_name(priv->sd.name, sizeof(priv->sd.name),
+				    priv->sd.grp_id, ipu_get_num(priv->ipu));
+
+	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+	priv->sd.ctrl_handler = &priv->ctrl_hdlr;
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static int imx_csi_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct csi_priv *priv = sd_to_dev(sd);
+
+	if (priv->fim)
+		imx_media_fim_free(priv->fim);
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	if (!IS_ERR_OR_NULL(priv->csi))
+		ipu_csi_put(priv->csi);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_csi_ids[] = {
+	{ .name = "imx-ipuv3-csi" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_csi_ids);
+
+static struct platform_driver imx_csi_driver = {
+	.probe = imx_csi_probe,
+	.remove = imx_csi_remove,
+	.id_table = imx_csi_ids,
+	.driver = {
+		.name = "imx-ipuv3-csi",
+	},
+};
+module_platform_driver(imx_csi_driver);
+
+MODULE_DESCRIPTION("i.MX CSI subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-csi");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r
  Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

This is a media entity subdevice for the i.MX Camera
Serial Interface module.

Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
---
 drivers/staging/media/imx/Kconfig   |  13 +
 drivers/staging/media/imx/Makefile  |   2 +
 drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
 3 files changed, 659 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-csi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index bfde58d..ce2d2c8 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
 	  Say yes here to enable support for video4linux media controller
 	  driver for the i.MX5/6 SOC.
 
+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA
+	tristate "i.MX5/6 Camera driver"
+	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
+	select VIDEOBUF2_DMA_CONTIG
+	default y
+	---help---
+	  A video4linux camera capture driver for i.MX5/6.
+
+endmenu
+endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index ef9f11b..133672a 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
new file mode 100644
index 0000000..64ef862
--- /dev/null
+++ b/drivers/staging/media/imx/imx-csi.c
@@ -0,0 +1,644 @@
+/*
+ * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+#define CSI_NUM_PADS 2
+
+struct csi_priv {
+	struct device *dev;
+	struct ipu_soc *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev sd;
+	struct media_pad pad[CSI_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct v4l2_rect crop;
+	struct ipu_csi *csi;
+	int csi_id;
+	int input_pad;
+	int output_pad;
+	bool power_on;  /* power is on */
+	bool stream_on; /* streaming is on */
+
+	/* the sink for the captured frames */
+	struct v4l2_subdev *sink_sd;
+	enum ipu_csi_dest dest;
+	struct v4l2_subdev *src_sd;
+
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	struct imx_media_fim *fim;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+};
+
+static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct csi_priv, sd);
+}
+
+/* Update the CSI whole sensor and active windows */
+static int csi_setup(struct csi_priv *priv)
+{
+	struct v4l2_mbus_framefmt infmt;
+
+	ipu_csi_set_window(priv->csi, &priv->crop);
+
+	/*
+	 * the ipu-csi doesn't understand ALTERNATE, but it only
+	 * needs to know whether the stream is interlaced, so set
+	 * to INTERLACED if infmt field is ALTERNATE.
+	 */
+	infmt = priv->format_mbus[priv->input_pad];
+	if (infmt.field == V4L2_FIELD_ALTERNATE)
+		infmt.field = V4L2_FIELD_INTERLACED;
+
+	ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
+
+	ipu_csi_set_dest(priv->csi, priv->dest);
+
+	ipu_csi_dump(priv->csi);
+
+	return 0;
+}
+
+static int csi_start(struct csi_priv *priv)
+{
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = csi_setup(priv);
+	if (ret)
+		return ret;
+
+	/* start the frame interval monitor */
+	if (priv->fim) {
+		ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);
+		if (ret)
+			return ret;
+	}
+
+	ret = ipu_csi_enable(priv->csi);
+	if (ret) {
+		v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
+		goto fim_off;
+	}
+
+	return 0;
+
+fim_off:
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+	return ret;
+}
+
+static void csi_stop(struct csi_priv *priv)
+{
+	/* stop the frame interval monitor */
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+
+	ipu_csi_disable(priv->csi);
+}
+
+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = csi_start(priv);
+	else if (!enable && priv->stream_on)
+		csi_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (priv->fim && on != priv->power_on)
+		ret = imx_media_fim_set_power(priv->fim, on);
+
+	if (!ret)
+		priv->power_on = on;
+	return ret;
+}
+
+static int csi_link_setup(struct media_entity *entity,
+			  const struct media_pad *local,
+			  const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SINK) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->sink_sd)
+			return -EBUSY;
+		priv->sink_sd = remote_sd;
+	} else {
+		priv->sink_sd = NULL;
+		return 0;
+	}
+
+	/* set CSI destination */
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_SMFC0:
+	case IMX_MEDIA_GRP_ID_SMFC1:
+	case IMX_MEDIA_GRP_ID_SMFC2:
+	case IMX_MEDIA_GRP_ID_SMFC3:
+		priv->dest = IPU_CSI_DEST_IDMAC;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		priv->dest = IPU_CSI_DEST_VDIC;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		priv->dest = IPU_CSI_DEST_IC;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int csi_link_validate(struct v4l2_subdev *sd,
+			     struct media_link *link,
+			     struct v4l2_subdev_format *source_fmt,
+			     struct v4l2_subdev_format *sink_fmt)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	bool is_csi2;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+	if (is_csi2) {
+		int vc_num = 0;
+		/*
+		 * NOTE! It seems the virtual channels from the mipi csi-2
+		 * receiver are used only for routing by the video mux's,
+		 * or for hard-wired routing to the CSI's. Once the stream
+		 * enters the CSI's however, they are treated internally
+		 * in the IPU as virtual channel 0.
+		 */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+							  &priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
+					  &priv->format_mbus[priv->input_pad]);
+	}
+
+	/* select either parallel or MIPI-CSI2 as input to CSI */
+	ipu_set_csi_src_mux(priv->ipu, priv->csi_id, is_csi2);
+
+	return 0;
+}
+
+static int csi_eof_isr(struct v4l2_subdev *sd, u32 status, bool *handled)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (priv->fim) {
+		struct timespec cur_ts;
+
+		ktime_get_ts(&cur_ts);
+		/* call frame interval monitor */
+		imx_media_fim_eof_monitor(priv->fim, &cur_ts);
+	}
+
+	return 0;
+}
+
+static int csi_try_crop(struct csi_priv *priv, struct v4l2_rect *crop)
+{
+	struct v4l2_mbus_framefmt *infmt;
+	struct imx_media_subdev *sensor;
+	v4l2_std_id std;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	ret = v4l2_subdev_call(sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+
+	crop->width = min_t(__u32, infmt->width, crop->width);
+	if (crop->left + crop->width > infmt->width)
+		crop->left = infmt->width - crop->width;
+	/* adjust crop left/width to h/w alignment restrictions */
+	crop->left &= ~0x3;
+	crop->width &= ~0x7;
+
+	/*
+	 * FIXME: not sure why yet, but on interlaced bt.656,
+	 * changing the vertical cropping causes loss of vertical
+	 * sync, so fix it to NTSC/PAL active lines. NTSC contains
+	 * 2 extra lines of active video that need to be cropped.
+	 */
+	if (priv->sensor_mbus_cfg.type == V4L2_MBUS_BT656) {
+		ret = v4l2_subdev_call(sensor->sd, video, g_std, &std);
+		if (ret)
+			return ret;
+		if (std & V4L2_STD_525_60) {
+			crop->top = 2;
+			crop->height = 480;
+		} else {
+			crop->top = 0;
+			crop->height = 576;
+		}
+	} else {
+		crop->height = min_t(__u32, infmt->height, crop->height);
+		if (crop->top + crop->height > infmt->height)
+			crop->top = infmt->height - crop->height;
+	}
+
+	return 0;
+}
+
+static int csi_get_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);
+		if (ret)
+			return ret;
+		sdformat->format.width = crop.width;
+		sdformat->format.height = crop.height;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		/* Update the crop window if this is output pad  */
+		if (sdformat->pad == priv->output_pad)
+			priv->crop = crop;
+	}
+
+	return 0;
+}
+
+static int csi_get_selection(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_selection *sel)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt;
+
+	if (sel->pad != priv->output_pad)
+		return -EINVAL;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+
+	switch (sel->target) {
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		sel->r.left = 0;
+		sel->r.top = 0;
+		sel->r.width = infmt->width;
+		sel->r.height = infmt->height;
+		break;
+	case V4L2_SEL_TGT_CROP:
+		sel->r = priv->crop;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int csi_set_selection(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_selection *sel)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *outfmt;
+	int ret;
+
+	if (sel->pad != priv->output_pad ||
+	    sel->target != V4L2_SEL_TGT_CROP)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	/*
+	 * Modifying the crop rectangle always changes the format on the source
+	 * pad. If the KEEP_CONFIG flag is set, just return the current crop
+	 * rectangle.
+	 */
+	if (sel->flags & V4L2_SEL_FLAG_KEEP_CONFIG) {
+		sel->r = priv->crop;
+		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+			cfg->try_crop = sel->r;
+		return 0;
+	}
+
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	ret = csi_try_crop(priv, &sel->r);
+	if (ret)
+		return ret;
+
+	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_crop = sel->r;
+	} else {
+		priv->crop = sel->r;
+		/* Update the source format */
+		outfmt->width = sel->r.width;
+		outfmt->height = sel->r.height;
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int csi_registered(struct v4l2_subdev *sd)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	/* get handle to IPU CSI */
+	priv->csi = ipu_csi_get(priv->ipu, priv->csi_id);
+	if (IS_ERR(priv->csi)) {
+		v4l2_err(&priv->sd, "failed to get CSI %d\n", priv->csi_id);
+		return PTR_ERR(priv->csi);
+	}
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd)) {
+		ret = PTR_ERR(imxsd);
+		goto put_csi;
+	}
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+		ret = -EINVAL;
+		goto put_csi;
+	}
+
+	for (i = 0; i < CSI_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      NULL);
+		if (ret)
+			goto put_csi;
+	}
+
+	priv->fim = imx_media_fim_init(&priv->sd);
+	if (IS_ERR(priv->fim)) {
+		ret = PTR_ERR(priv->fim);
+		goto put_csi;
+	}
+
+	ret = media_entity_pads_init(&sd->entity, CSI_NUM_PADS, priv->pad);
+	if (ret)
+		goto free_fim;
+
+	return 0;
+free_fim:
+	if (priv->fim)
+		imx_media_fim_free(priv->fim);
+put_csi:
+	ipu_csi_put(priv->csi);
+	return ret;
+}
+
+static struct media_entity_operations csi_entity_ops = {
+	.link_setup = csi_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops csi_core_ops = {
+	.s_power = csi_s_power,
+	.interrupt_service_routine = csi_eof_isr,
+};
+
+static struct v4l2_subdev_video_ops csi_video_ops = {
+	.s_stream = csi_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops csi_pad_ops = {
+	.get_fmt = csi_get_fmt,
+	.set_fmt = csi_set_fmt,
+	.get_selection = csi_get_selection,
+	.set_selection = csi_set_selection,
+	.link_validate = csi_link_validate,
+};
+
+static struct v4l2_subdev_ops csi_subdev_ops = {
+	.core = &csi_core_ops,
+	.video = &csi_video_ops,
+	.pad = &csi_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops csi_internal_ops = {
+	.registered = csi_registered,
+};
+
+static int imx_csi_probe(struct platform_device *pdev)
+{
+	struct ipu_client_platformdata *pdata;
+	struct csi_priv *priv;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	/* get parent IPU */
+	priv->ipu = dev_get_drvdata(priv->dev->parent);
+
+	/* get our CSI id */
+	pdata = priv->dev->platform_data;
+	priv->csi_id = pdata->csi;
+
+	v4l2_subdev_init(&priv->sd, &csi_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &csi_internal_ops;
+	priv->sd.entity.ops = &csi_entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.grp_id = priv->csi_id ?
+		IMX_MEDIA_GRP_ID_CSI1 : IMX_MEDIA_GRP_ID_CSI0;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	imx_media_grp_id_to_sd_name(priv->sd.name, sizeof(priv->sd.name),
+				    priv->sd.grp_id, ipu_get_num(priv->ipu));
+
+	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+	priv->sd.ctrl_handler = &priv->ctrl_hdlr;
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static int imx_csi_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct csi_priv *priv = sd_to_dev(sd);
+
+	if (priv->fim)
+		imx_media_fim_free(priv->fim);
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	if (!IS_ERR_OR_NULL(priv->csi))
+		ipu_csi_put(priv->csi);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_csi_ids[] = {
+	{ .name = "imx-ipuv3-csi" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_csi_ids);
+
+static struct platform_driver imx_csi_driver = {
+	.probe = imx_csi_probe,
+	.remove = imx_csi_remove,
+	.id_table = imx_csi_ids,
+	.driver = {
+		.name = "imx-ipuv3-csi",
+	},
+};
+module_platform_driver(imx_csi_driver);
+
+MODULE_DESCRIPTION("i.MX CSI subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-csi");
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

This is a media entity subdevice for the i.MX Camera
Serial Interface module.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Kconfig   |  13 +
 drivers/staging/media/imx/Makefile  |   2 +
 drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
 3 files changed, 659 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-csi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index bfde58d..ce2d2c8 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
 	  Say yes here to enable support for video4linux media controller
 	  driver for the i.MX5/6 SOC.
 
+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA
+	tristate "i.MX5/6 Camera driver"
+	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
+	select VIDEOBUF2_DMA_CONTIG
+	default y
+	---help---
+	  A video4linux camera capture driver for i.MX5/6.
+
+endmenu
+endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index ef9f11b..133672a 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
new file mode 100644
index 0000000..64ef862
--- /dev/null
+++ b/drivers/staging/media/imx/imx-csi.c
@@ -0,0 +1,644 @@
+/*
+ * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+#define CSI_NUM_PADS 2
+
+struct csi_priv {
+	struct device *dev;
+	struct ipu_soc *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev sd;
+	struct media_pad pad[CSI_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct v4l2_rect crop;
+	struct ipu_csi *csi;
+	int csi_id;
+	int input_pad;
+	int output_pad;
+	bool power_on;  /* power is on */
+	bool stream_on; /* streaming is on */
+
+	/* the sink for the captured frames */
+	struct v4l2_subdev *sink_sd;
+	enum ipu_csi_dest dest;
+	struct v4l2_subdev *src_sd;
+
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	struct imx_media_fim *fim;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+};
+
+static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct csi_priv, sd);
+}
+
+/* Update the CSI whole sensor and active windows */
+static int csi_setup(struct csi_priv *priv)
+{
+	struct v4l2_mbus_framefmt infmt;
+
+	ipu_csi_set_window(priv->csi, &priv->crop);
+
+	/*
+	 * the ipu-csi doesn't understand ALTERNATE, but it only
+	 * needs to know whether the stream is interlaced, so set
+	 * to INTERLACED if infmt field is ALTERNATE.
+	 */
+	infmt = priv->format_mbus[priv->input_pad];
+	if (infmt.field == V4L2_FIELD_ALTERNATE)
+		infmt.field = V4L2_FIELD_INTERLACED;
+
+	ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
+
+	ipu_csi_set_dest(priv->csi, priv->dest);
+
+	ipu_csi_dump(priv->csi);
+
+	return 0;
+}
+
+static int csi_start(struct csi_priv *priv)
+{
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = csi_setup(priv);
+	if (ret)
+		return ret;
+
+	/* start the frame interval monitor */
+	if (priv->fim) {
+		ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);
+		if (ret)
+			return ret;
+	}
+
+	ret = ipu_csi_enable(priv->csi);
+	if (ret) {
+		v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
+		goto fim_off;
+	}
+
+	return 0;
+
+fim_off:
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+	return ret;
+}
+
+static void csi_stop(struct csi_priv *priv)
+{
+	/* stop the frame interval monitor */
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+
+	ipu_csi_disable(priv->csi);
+}
+
+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = csi_start(priv);
+	else if (!enable && priv->stream_on)
+		csi_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (priv->fim && on != priv->power_on)
+		ret = imx_media_fim_set_power(priv->fim, on);
+
+	if (!ret)
+		priv->power_on = on;
+	return ret;
+}
+
+static int csi_link_setup(struct media_entity *entity,
+			  const struct media_pad *local,
+			  const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SINK) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->sink_sd)
+			return -EBUSY;
+		priv->sink_sd = remote_sd;
+	} else {
+		priv->sink_sd = NULL;
+		return 0;
+	}
+
+	/* set CSI destination */
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_SMFC0:
+	case IMX_MEDIA_GRP_ID_SMFC1:
+	case IMX_MEDIA_GRP_ID_SMFC2:
+	case IMX_MEDIA_GRP_ID_SMFC3:
+		priv->dest = IPU_CSI_DEST_IDMAC;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		priv->dest = IPU_CSI_DEST_VDIC;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		priv->dest = IPU_CSI_DEST_IC;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int csi_link_validate(struct v4l2_subdev *sd,
+			     struct media_link *link,
+			     struct v4l2_subdev_format *source_fmt,
+			     struct v4l2_subdev_format *sink_fmt)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	bool is_csi2;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+	if (is_csi2) {
+		int vc_num = 0;
+		/*
+		 * NOTE! It seems the virtual channels from the mipi csi-2
+		 * receiver are used only for routing by the video mux's,
+		 * or for hard-wired routing to the CSI's. Once the stream
+		 * enters the CSI's however, they are treated internally
+		 * in the IPU as virtual channel 0.
+		 */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+							  &priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
+					  &priv->format_mbus[priv->input_pad]);
+	}
+
+	/* select either parallel or MIPI-CSI2 as input to CSI */
+	ipu_set_csi_src_mux(priv->ipu, priv->csi_id, is_csi2);
+
+	return 0;
+}
+
+static int csi_eof_isr(struct v4l2_subdev *sd, u32 status, bool *handled)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (priv->fim) {
+		struct timespec cur_ts;
+
+		ktime_get_ts(&cur_ts);
+		/* call frame interval monitor */
+		imx_media_fim_eof_monitor(priv->fim, &cur_ts);
+	}
+
+	return 0;
+}
+
+static int csi_try_crop(struct csi_priv *priv, struct v4l2_rect *crop)
+{
+	struct v4l2_mbus_framefmt *infmt;
+	struct imx_media_subdev *sensor;
+	v4l2_std_id std;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	ret = v4l2_subdev_call(sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+
+	crop->width = min_t(__u32, infmt->width, crop->width);
+	if (crop->left + crop->width > infmt->width)
+		crop->left = infmt->width - crop->width;
+	/* adjust crop left/width to h/w alignment restrictions */
+	crop->left &= ~0x3;
+	crop->width &= ~0x7;
+
+	/*
+	 * FIXME: not sure why yet, but on interlaced bt.656,
+	 * changing the vertical cropping causes loss of vertical
+	 * sync, so fix it to NTSC/PAL active lines. NTSC contains
+	 * 2 extra lines of active video that need to be cropped.
+	 */
+	if (priv->sensor_mbus_cfg.type == V4L2_MBUS_BT656) {
+		ret = v4l2_subdev_call(sensor->sd, video, g_std, &std);
+		if (ret)
+			return ret;
+		if (std & V4L2_STD_525_60) {
+			crop->top = 2;
+			crop->height = 480;
+		} else {
+			crop->top = 0;
+			crop->height = 576;
+		}
+	} else {
+		crop->height = min_t(__u32, infmt->height, crop->height);
+		if (crop->top + crop->height > infmt->height)
+			crop->top = infmt->height - crop->height;
+	}
+
+	return 0;
+}
+
+static int csi_get_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);
+		if (ret)
+			return ret;
+		sdformat->format.width = crop.width;
+		sdformat->format.height = crop.height;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		/* Update the crop window if this is output pad  */
+		if (sdformat->pad == priv->output_pad)
+			priv->crop = crop;
+	}
+
+	return 0;
+}
+
+static int csi_get_selection(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_selection *sel)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt;
+
+	if (sel->pad != priv->output_pad)
+		return -EINVAL;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+
+	switch (sel->target) {
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		sel->r.left = 0;
+		sel->r.top = 0;
+		sel->r.width = infmt->width;
+		sel->r.height = infmt->height;
+		break;
+	case V4L2_SEL_TGT_CROP:
+		sel->r = priv->crop;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int csi_set_selection(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_selection *sel)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *outfmt;
+	int ret;
+
+	if (sel->pad != priv->output_pad ||
+	    sel->target != V4L2_SEL_TGT_CROP)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	/*
+	 * Modifying the crop rectangle always changes the format on the source
+	 * pad. If the KEEP_CONFIG flag is set, just return the current crop
+	 * rectangle.
+	 */
+	if (sel->flags & V4L2_SEL_FLAG_KEEP_CONFIG) {
+		sel->r = priv->crop;
+		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+			cfg->try_crop = sel->r;
+		return 0;
+	}
+
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	ret = csi_try_crop(priv, &sel->r);
+	if (ret)
+		return ret;
+
+	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_crop = sel->r;
+	} else {
+		priv->crop = sel->r;
+		/* Update the source format */
+		outfmt->width = sel->r.width;
+		outfmt->height = sel->r.height;
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int csi_registered(struct v4l2_subdev *sd)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	/* get handle to IPU CSI */
+	priv->csi = ipu_csi_get(priv->ipu, priv->csi_id);
+	if (IS_ERR(priv->csi)) {
+		v4l2_err(&priv->sd, "failed to get CSI %d\n", priv->csi_id);
+		return PTR_ERR(priv->csi);
+	}
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd)) {
+		ret = PTR_ERR(imxsd);
+		goto put_csi;
+	}
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+		ret = -EINVAL;
+		goto put_csi;
+	}
+
+	for (i = 0; i < CSI_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      NULL);
+		if (ret)
+			goto put_csi;
+	}
+
+	priv->fim = imx_media_fim_init(&priv->sd);
+	if (IS_ERR(priv->fim)) {
+		ret = PTR_ERR(priv->fim);
+		goto put_csi;
+	}
+
+	ret = media_entity_pads_init(&sd->entity, CSI_NUM_PADS, priv->pad);
+	if (ret)
+		goto free_fim;
+
+	return 0;
+free_fim:
+	if (priv->fim)
+		imx_media_fim_free(priv->fim);
+put_csi:
+	ipu_csi_put(priv->csi);
+	return ret;
+}
+
+static struct media_entity_operations csi_entity_ops = {
+	.link_setup = csi_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops csi_core_ops = {
+	.s_power = csi_s_power,
+	.interrupt_service_routine = csi_eof_isr,
+};
+
+static struct v4l2_subdev_video_ops csi_video_ops = {
+	.s_stream = csi_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops csi_pad_ops = {
+	.get_fmt = csi_get_fmt,
+	.set_fmt = csi_set_fmt,
+	.get_selection = csi_get_selection,
+	.set_selection = csi_set_selection,
+	.link_validate = csi_link_validate,
+};
+
+static struct v4l2_subdev_ops csi_subdev_ops = {
+	.core = &csi_core_ops,
+	.video = &csi_video_ops,
+	.pad = &csi_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops csi_internal_ops = {
+	.registered = csi_registered,
+};
+
+static int imx_csi_probe(struct platform_device *pdev)
+{
+	struct ipu_client_platformdata *pdata;
+	struct csi_priv *priv;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	/* get parent IPU */
+	priv->ipu = dev_get_drvdata(priv->dev->parent);
+
+	/* get our CSI id */
+	pdata = priv->dev->platform_data;
+	priv->csi_id = pdata->csi;
+
+	v4l2_subdev_init(&priv->sd, &csi_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &csi_internal_ops;
+	priv->sd.entity.ops = &csi_entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.grp_id = priv->csi_id ?
+		IMX_MEDIA_GRP_ID_CSI1 : IMX_MEDIA_GRP_ID_CSI0;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	imx_media_grp_id_to_sd_name(priv->sd.name, sizeof(priv->sd.name),
+				    priv->sd.grp_id, ipu_get_num(priv->ipu));
+
+	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+	priv->sd.ctrl_handler = &priv->ctrl_hdlr;
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static int imx_csi_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct csi_priv *priv = sd_to_dev(sd);
+
+	if (priv->fim)
+		imx_media_fim_free(priv->fim);
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	if (!IS_ERR_OR_NULL(priv->csi))
+		ipu_csi_put(priv->csi);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_csi_ids[] = {
+	{ .name = "imx-ipuv3-csi" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_csi_ids);
+
+static struct platform_driver imx_csi_driver = {
+	.probe = imx_csi_probe,
+	.remove = imx_csi_remove,
+	.id_table = imx_csi_ids,
+	.driver = {
+		.name = "imx-ipuv3-csi",
+	},
+};
+module_platform_driver(imx_csi_driver);
+
+MODULE_DESCRIPTION("i.MX CSI subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-csi");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 18/24] media: imx: Add SMFC subdev driver
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

This is a media entity subdevice driver for the i.MX Sensor Multi-FIFO
Controller module. Video frames are received from the CSI and can
be routed to various sinks including the i.MX Image Converter for
scaling, color-space conversion, motion compensated deinterlacing,
and image rotation.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile   |   1 +
 drivers/staging/media/imx/imx-smfc.c | 737 +++++++++++++++++++++++++++++++++++
 2 files changed, 738 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-smfc.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 133672a..3559d7b 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 
diff --git a/drivers/staging/media/imx/imx-smfc.c b/drivers/staging/media/imx/imx-smfc.c
new file mode 100644
index 0000000..614a4381
--- /dev/null
+++ b/drivers/staging/media/imx/imx-smfc.c
@@ -0,0 +1,737 @@
+/*
+ * V4L2 Capture SMFC Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of raw/unconverted video frames
+ * from the CSI, directly to memory via the Sensor Multi-FIFO Controller.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+/*
+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W       176
+#define MIN_H       144
+#define MAX_W      8192
+#define MAX_H      4096
+#define W_ALIGN    4 /* multiple of 16 pixels */
+#define H_ALIGN    1 /* multiple of 2 lines */
+#define S_ALIGN    1 /* multiple of 2 */
+
+#define SMFC_NUM_PADS 2
+
+struct imx_smfc_priv {
+	struct device        *dev;
+	struct ipu_soc       *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev   sd;
+	struct media_pad pad[SMFC_NUM_PADS];
+	int ipu_id;
+	int smfc_id;
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *smfc_ch;
+	struct ipu_smfc *smfc;
+
+	struct v4l2_mbus_framefmt format_mbus[SMFC_NUM_PADS];
+	const struct imx_media_pixfmt *cc[SMFC_NUM_PADS];
+
+	struct v4l2_mbus_config sensor_mbus_cfg;
+
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct imx_media_dma_buf *next;
+
+	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
+
+	/* the sink that will receive the dma buffers */
+	struct v4l2_subdev *sink_sd;
+	struct v4l2_subdev *src_sd;
+
+	/*
+	 * the CSI id and mipi virtual channel number at
+	 * link validate
+	 */
+	int csi_id;
+	int vc_num;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	spinlock_t irqlock;
+	struct timer_list eof_timeout_timer;
+	int eof_irq;
+	int nfb4eof_irq;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static void imx_smfc_put_ipu_resources(struct imx_smfc_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->smfc_ch))
+		ipu_idmac_put(priv->smfc_ch);
+	priv->smfc_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->smfc))
+		ipu_smfc_put(priv->smfc);
+	priv->smfc = NULL;
+}
+
+static int imx_smfc_get_ipu_resources(struct imx_smfc_priv *priv)
+{
+	int ch_num, ret;
+
+	priv->ipu = priv->md->ipu[priv->ipu_id];
+
+	ch_num = IPUV3_CHANNEL_CSI0 + priv->smfc_id;
+
+	priv->smfc = ipu_smfc_get(priv->ipu, ch_num);
+	if (IS_ERR(priv->smfc)) {
+		v4l2_err(&priv->sd, "failed to get SMFC\n");
+		ret = PTR_ERR(priv->smfc);
+		goto out;
+	}
+
+	priv->smfc_ch = ipu_idmac_get(priv->ipu, ch_num);
+	if (IS_ERR(priv->smfc_ch)) {
+		v4l2_err(&priv->sd, "could not get IDMAC channel %u\n", ch_num);
+		ret = PTR_ERR(priv->smfc_ch);
+		goto out;
+	}
+
+	return 0;
+out:
+	imx_smfc_put_ipu_resources(priv);
+	return ret;
+}
+
+static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
+{
+	struct imx_smfc_priv *priv = dev_id;
+	struct imx_media_dma_buf *done, *next;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	/* inform CSI of this EOF so it can monitor frame intervals */
+	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+			 0, NULL);
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* priv->next buffer is now the active one */
+	imx_media_dma_buf_set_active(priv->next);
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (ipu_idmac_buffer_is_ready(priv->smfc_ch, priv->ipu_buf_num))
+		ipu_idmac_clear_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+	/* get next queued buffer */
+	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+	ipu_cpmem_set_buffer(priv->smfc_ch, priv->ipu_buf_num, next->phys);
+	ipu_idmac_select_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+	/* toggle IPU double-buffer index */
+	priv->ipu_buf_num ^= 1;
+	priv->next = next;
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t imx_smfc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct imx_smfc_priv *priv = dev_id;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void imx_smfc_eof_timeout(unsigned long data)
+{
+	struct imx_smfc_priv *priv = (struct imx_smfc_priv *)data;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&priv->sd, &ev);
+}
+
+/* init the SMFC IDMAC channel */
+static void imx_smfc_setup_channel(struct imx_smfc_priv *priv)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct imx_media_dma_buf *buf0, *buf1;
+	unsigned int burst_size;
+	struct ipu_image image;
+	bool passthrough;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	ipu_cpmem_zero(priv->smfc_ch);
+
+	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	image.phys0 = buf0->phys;
+	image.phys1 = buf1->phys;
+	ipu_cpmem_set_image(priv->smfc_ch, &image);
+
+	burst_size = (outfmt->width & 0xf) ? 8 : 16;
+
+	ipu_cpmem_set_burstsize(priv->smfc_ch, burst_size);
+
+	/*
+	 * If the sensor uses 16-bit parallel CSI bus, we must handle
+	 * the data internally in the IPU as 16-bit generic, aka
+	 * passthrough mode.
+	 */
+	passthrough = (priv->sensor_mbus_cfg.type != V4L2_MBUS_CSI2 &&
+		       priv->sensor->sensor_ep.bus.parallel.bus_width >= 16);
+
+	if (passthrough)
+		ipu_cpmem_set_format_passthrough(priv->smfc_ch, 16);
+
+	/*
+	 * Set the channel for the direct CSI-->memory via SMFC
+	 * use-case to very high priority, by enabling the watermark
+	 * signal in the SMFC, enabling WM in the channel, and setting
+	 * the channel priority to high.
+	 *
+	 * Refer to the i.mx6 rev. D TRM Table 36-8: Calculated priority
+	 * value.
+	 *
+	 * The WM's are set very low by intention here to ensure that
+	 * the SMFC FIFOs do not overflow.
+	 */
+	ipu_smfc_set_watermark(priv->smfc, 0x02, 0x01);
+	ipu_cpmem_set_high_priority(priv->smfc_ch);
+	ipu_idmac_enable_watermark(priv->smfc_ch, true);
+	ipu_cpmem_set_axi_id(priv->smfc_ch, 0);
+	ipu_idmac_lock_enable(priv->smfc_ch, 8);
+
+	burst_size = ipu_cpmem_get_burstsize(priv->smfc_ch);
+	burst_size = passthrough ?
+		(burst_size >> 3) - 1 : (burst_size >> 2) - 1;
+
+	ipu_smfc_set_burstsize(priv->smfc, burst_size);
+
+	if (outfmt->field == V4L2_FIELD_NONE &&
+	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+	     infmt->field == V4L2_FIELD_ALTERNATE))
+		ipu_cpmem_interlaced_scan(priv->smfc_ch,
+					  image.pix.bytesperline);
+
+	ipu_idmac_set_double_buffer(priv->smfc_ch, true);
+}
+
+static void imx_smfc_unsetup(struct imx_smfc_priv *priv)
+{
+	ipu_idmac_disable_channel(priv->smfc_ch);
+	ipu_smfc_disable(priv->smfc);
+}
+
+static void imx_smfc_setup(struct imx_smfc_priv *priv)
+{
+	imx_smfc_setup_channel(priv);
+
+	ipu_cpmem_dump(priv->smfc_ch);
+	ipu_dump(priv->ipu);
+
+	ipu_smfc_enable(priv->smfc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->smfc_ch, 0);
+	ipu_idmac_select_buffer(priv->smfc_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->smfc_ch);
+}
+
+static int imx_smfc_start(struct imx_smfc_priv *priv)
+{
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = imx_smfc_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	ipu_smfc_map_channel(priv->smfc, priv->csi_id, priv->vc_num);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->ipu_buf_num = 0;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	imx_smfc_setup(priv);
+
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						 priv->smfc_ch,
+						 IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(priv->dev, priv->nfb4eof_irq,
+			       imx_smfc_nfb4eof_interrupt, 0,
+			       "imx-smfc-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_unsetup;
+	}
+
+	priv->eof_irq = ipu_idmac_channel_irq(priv->ipu, priv->smfc_ch,
+					      IPU_IRQ_EOF);
+
+	ret = devm_request_irq(priv->dev, priv->eof_irq,
+			       imx_smfc_eof_interrupt, 0,
+			       "imx-smfc-eof", priv);
+	if (ret) {
+		v4l2_err(&priv->sd,
+			 "Error registering eof irq: %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+	imx_smfc_unsetup(priv);
+out_put_ipu:
+	imx_smfc_put_ipu_resources(priv);
+	return ret;
+}
+
+static void imx_smfc_stop(struct imx_smfc_priv *priv)
+{
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&priv->sd, "wait last EOF timeout\n");
+
+	devm_free_irq(priv->dev, priv->eof_irq, priv);
+	devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+
+	imx_smfc_unsetup(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+
+	imx_smfc_put_ipu_resources(priv);
+}
+
+static int imx_smfc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = imx_smfc_start(priv);
+	else if (!enable && priv->stream_on)
+		imx_smfc_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int imx_smfc_enum_mbus_code(struct v4l2_subdev *sd,
+				   struct v4l2_subdev_pad_config *cfg,
+				   struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (code->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	return imx_media_enum_format(&code->code, code->index,
+				     true, code->pad == priv->output_pad);
+}
+
+static int imx_smfc_get_fmt(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int imx_smfc_set_fmt(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc, *incc;
+	bool allow_planar;
+	u32 code;
+
+	if (sdformat->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   true, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, false);
+		cc = imx_media_find_format(0, code, true, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	v4l_bound_align_image(&sdformat->format.width, MIN_W, MAX_W,
+			      W_ALIGN, &sdformat->format.height,
+			      MIN_H, MAX_H, H_ALIGN, S_ALIGN);
+
+	if (sdformat->pad == priv->output_pad) {
+		incc = priv->cc[priv->input_pad];
+		sdformat->format.width = infmt->width;
+		sdformat->format.height = infmt->height;
+		if (sdformat->format.field != V4L2_FIELD_NONE)
+			sdformat->format.field = infmt->field;
+		if (cc->cs != incc->cs) {
+			sdformat->format.code = infmt->code;
+			cc = imx_media_find_format(0, sdformat->format.code,
+						   true, false);
+		}
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int imx_smfc_link_setup(struct media_entity *entity,
+			       const struct media_pad *local,
+			       const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+	} else {
+		priv->src_sd = NULL;
+		return 0;
+	}
+
+	/* must attach to CSI source */
+	if (!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_CSI))
+		return -EINVAL;
+
+	return 0;
+}
+
+static int imx_smfc_link_validate(struct v4l2_subdev *sd,
+				  struct media_link *link,
+				  struct v4l2_subdev_format *source_fmt,
+				  struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	switch (priv->src_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0:
+		priv->csi_id = 0;
+		break;
+	case IMX_MEDIA_GRP_ID_CSI1:
+		priv->csi_id = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	priv->vc_num = 0;
+	if (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		/* see NOTE in imx-csi.c */
+#if 0
+		priv->vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &priv->sd.entity);
+		if (priv->vc_num < 0)
+			return vc_num;
+#endif
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int imx_smfc_registered(struct v4l2_subdev *sd)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < SMFC_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					640, 480, 0, V4L2_FIELD_NONE,
+					&priv->cc[i]);
+	}
+
+	return media_entity_pads_init(&sd->entity, SMFC_NUM_PADS, priv->pad);
+}
+
+static struct media_entity_operations imx_smfc_entity_ops = {
+	.link_setup = imx_smfc_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_video_ops imx_smfc_video_ops = {
+	.s_stream = imx_smfc_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imx_smfc_pad_ops = {
+	.enum_mbus_code = imx_smfc_enum_mbus_code,
+	.get_fmt = imx_smfc_get_fmt,
+	.set_fmt = imx_smfc_set_fmt,
+	.link_validate = imx_smfc_link_validate,
+};
+
+static struct v4l2_subdev_ops imx_smfc_subdev_ops = {
+	.video = &imx_smfc_video_ops,
+	.pad = &imx_smfc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imx_smfc_internal_ops = {
+	.registered = imx_smfc_registered,
+};
+
+static int imx_smfc_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct imx_smfc_priv *priv;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	pdata = priv->dev->platform_data;
+	priv->ipu_id = pdata->ipu_id;
+
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = imx_smfc_eof_timeout;
+	spin_lock_init(&priv->irqlock);
+
+	v4l2_subdev_init(&priv->sd, &imx_smfc_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &imx_smfc_internal_ops;
+	priv->sd.entity.ops = &imx_smfc_entity_ops;
+	/* FIXME: this the right function? */
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	/* get our group id and SMFC id */
+	priv->sd.grp_id = pdata->grp_id;
+	priv->smfc_id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_SMFC_BIT) - 1;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+	return v4l2_async_register_subdev(&priv->sd);
+}
+
+static int imx_smfc_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imx_smfc_priv *priv = container_of(sd, struct imx_smfc_priv, sd);
+
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_smfc_ids[] = {
+	{ .name = "imx-ipuv3-smfc" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
+
+static struct platform_driver imx_smfc_driver = {
+	.probe = imx_smfc_probe,
+	.remove = imx_smfc_remove,
+	.id_table = imx_smfc_ids,
+	.driver = {
+		.name = "imx-ipuv3-smfc",
+	},
+};
+module_platform_driver(imx_smfc_driver);
+
+MODULE_DESCRIPTION("i.MX SMFC subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-smfc");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 18/24] media: imx: Add SMFC subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

This is a media entity subdevice driver for the i.MX Sensor Multi-FIFO
Controller module. Video frames are received from the CSI and can
be routed to various sinks including the i.MX Image Converter for
scaling, color-space conversion, motion compensated deinterlacing,
and image rotation.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile   |   1 +
 drivers/staging/media/imx/imx-smfc.c | 737 +++++++++++++++++++++++++++++++++++
 2 files changed, 738 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-smfc.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 133672a..3559d7b 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 
diff --git a/drivers/staging/media/imx/imx-smfc.c b/drivers/staging/media/imx/imx-smfc.c
new file mode 100644
index 0000000..614a4381
--- /dev/null
+++ b/drivers/staging/media/imx/imx-smfc.c
@@ -0,0 +1,737 @@
+/*
+ * V4L2 Capture SMFC Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of raw/unconverted video frames
+ * from the CSI, directly to memory via the Sensor Multi-FIFO Controller.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+/*
+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W       176
+#define MIN_H       144
+#define MAX_W      8192
+#define MAX_H      4096
+#define W_ALIGN    4 /* multiple of 16 pixels */
+#define H_ALIGN    1 /* multiple of 2 lines */
+#define S_ALIGN    1 /* multiple of 2 */
+
+#define SMFC_NUM_PADS 2
+
+struct imx_smfc_priv {
+	struct device        *dev;
+	struct ipu_soc       *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev   sd;
+	struct media_pad pad[SMFC_NUM_PADS];
+	int ipu_id;
+	int smfc_id;
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *smfc_ch;
+	struct ipu_smfc *smfc;
+
+	struct v4l2_mbus_framefmt format_mbus[SMFC_NUM_PADS];
+	const struct imx_media_pixfmt *cc[SMFC_NUM_PADS];
+
+	struct v4l2_mbus_config sensor_mbus_cfg;
+
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct imx_media_dma_buf *next;
+
+	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
+
+	/* the sink that will receive the dma buffers */
+	struct v4l2_subdev *sink_sd;
+	struct v4l2_subdev *src_sd;
+
+	/*
+	 * the CSI id and mipi virtual channel number at
+	 * link validate
+	 */
+	int csi_id;
+	int vc_num;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	spinlock_t irqlock;
+	struct timer_list eof_timeout_timer;
+	int eof_irq;
+	int nfb4eof_irq;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static void imx_smfc_put_ipu_resources(struct imx_smfc_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->smfc_ch))
+		ipu_idmac_put(priv->smfc_ch);
+	priv->smfc_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->smfc))
+		ipu_smfc_put(priv->smfc);
+	priv->smfc = NULL;
+}
+
+static int imx_smfc_get_ipu_resources(struct imx_smfc_priv *priv)
+{
+	int ch_num, ret;
+
+	priv->ipu = priv->md->ipu[priv->ipu_id];
+
+	ch_num = IPUV3_CHANNEL_CSI0 + priv->smfc_id;
+
+	priv->smfc = ipu_smfc_get(priv->ipu, ch_num);
+	if (IS_ERR(priv->smfc)) {
+		v4l2_err(&priv->sd, "failed to get SMFC\n");
+		ret = PTR_ERR(priv->smfc);
+		goto out;
+	}
+
+	priv->smfc_ch = ipu_idmac_get(priv->ipu, ch_num);
+	if (IS_ERR(priv->smfc_ch)) {
+		v4l2_err(&priv->sd, "could not get IDMAC channel %u\n", ch_num);
+		ret = PTR_ERR(priv->smfc_ch);
+		goto out;
+	}
+
+	return 0;
+out:
+	imx_smfc_put_ipu_resources(priv);
+	return ret;
+}
+
+static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
+{
+	struct imx_smfc_priv *priv = dev_id;
+	struct imx_media_dma_buf *done, *next;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	/* inform CSI of this EOF so it can monitor frame intervals */
+	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+			 0, NULL);
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* priv->next buffer is now the active one */
+	imx_media_dma_buf_set_active(priv->next);
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (ipu_idmac_buffer_is_ready(priv->smfc_ch, priv->ipu_buf_num))
+		ipu_idmac_clear_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+	/* get next queued buffer */
+	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+	ipu_cpmem_set_buffer(priv->smfc_ch, priv->ipu_buf_num, next->phys);
+	ipu_idmac_select_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+	/* toggle IPU double-buffer index */
+	priv->ipu_buf_num ^= 1;
+	priv->next = next;
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t imx_smfc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct imx_smfc_priv *priv = dev_id;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void imx_smfc_eof_timeout(unsigned long data)
+{
+	struct imx_smfc_priv *priv = (struct imx_smfc_priv *)data;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&priv->sd, &ev);
+}
+
+/* init the SMFC IDMAC channel */
+static void imx_smfc_setup_channel(struct imx_smfc_priv *priv)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct imx_media_dma_buf *buf0, *buf1;
+	unsigned int burst_size;
+	struct ipu_image image;
+	bool passthrough;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	ipu_cpmem_zero(priv->smfc_ch);
+
+	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	image.phys0 = buf0->phys;
+	image.phys1 = buf1->phys;
+	ipu_cpmem_set_image(priv->smfc_ch, &image);
+
+	burst_size = (outfmt->width & 0xf) ? 8 : 16;
+
+	ipu_cpmem_set_burstsize(priv->smfc_ch, burst_size);
+
+	/*
+	 * If the sensor uses 16-bit parallel CSI bus, we must handle
+	 * the data internally in the IPU as 16-bit generic, aka
+	 * passthrough mode.
+	 */
+	passthrough = (priv->sensor_mbus_cfg.type != V4L2_MBUS_CSI2 &&
+		       priv->sensor->sensor_ep.bus.parallel.bus_width >= 16);
+
+	if (passthrough)
+		ipu_cpmem_set_format_passthrough(priv->smfc_ch, 16);
+
+	/*
+	 * Set the channel for the direct CSI-->memory via SMFC
+	 * use-case to very high priority, by enabling the watermark
+	 * signal in the SMFC, enabling WM in the channel, and setting
+	 * the channel priority to high.
+	 *
+	 * Refer to the i.mx6 rev. D TRM Table 36-8: Calculated priority
+	 * value.
+	 *
+	 * The WM's are set very low by intention here to ensure that
+	 * the SMFC FIFOs do not overflow.
+	 */
+	ipu_smfc_set_watermark(priv->smfc, 0x02, 0x01);
+	ipu_cpmem_set_high_priority(priv->smfc_ch);
+	ipu_idmac_enable_watermark(priv->smfc_ch, true);
+	ipu_cpmem_set_axi_id(priv->smfc_ch, 0);
+	ipu_idmac_lock_enable(priv->smfc_ch, 8);
+
+	burst_size = ipu_cpmem_get_burstsize(priv->smfc_ch);
+	burst_size = passthrough ?
+		(burst_size >> 3) - 1 : (burst_size >> 2) - 1;
+
+	ipu_smfc_set_burstsize(priv->smfc, burst_size);
+
+	if (outfmt->field == V4L2_FIELD_NONE &&
+	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+	     infmt->field == V4L2_FIELD_ALTERNATE))
+		ipu_cpmem_interlaced_scan(priv->smfc_ch,
+					  image.pix.bytesperline);
+
+	ipu_idmac_set_double_buffer(priv->smfc_ch, true);
+}
+
+static void imx_smfc_unsetup(struct imx_smfc_priv *priv)
+{
+	ipu_idmac_disable_channel(priv->smfc_ch);
+	ipu_smfc_disable(priv->smfc);
+}
+
+static void imx_smfc_setup(struct imx_smfc_priv *priv)
+{
+	imx_smfc_setup_channel(priv);
+
+	ipu_cpmem_dump(priv->smfc_ch);
+	ipu_dump(priv->ipu);
+
+	ipu_smfc_enable(priv->smfc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->smfc_ch, 0);
+	ipu_idmac_select_buffer(priv->smfc_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->smfc_ch);
+}
+
+static int imx_smfc_start(struct imx_smfc_priv *priv)
+{
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = imx_smfc_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	ipu_smfc_map_channel(priv->smfc, priv->csi_id, priv->vc_num);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->ipu_buf_num = 0;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	imx_smfc_setup(priv);
+
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						 priv->smfc_ch,
+						 IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(priv->dev, priv->nfb4eof_irq,
+			       imx_smfc_nfb4eof_interrupt, 0,
+			       "imx-smfc-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_unsetup;
+	}
+
+	priv->eof_irq = ipu_idmac_channel_irq(priv->ipu, priv->smfc_ch,
+					      IPU_IRQ_EOF);
+
+	ret = devm_request_irq(priv->dev, priv->eof_irq,
+			       imx_smfc_eof_interrupt, 0,
+			       "imx-smfc-eof", priv);
+	if (ret) {
+		v4l2_err(&priv->sd,
+			 "Error registering eof irq: %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+	imx_smfc_unsetup(priv);
+out_put_ipu:
+	imx_smfc_put_ipu_resources(priv);
+	return ret;
+}
+
+static void imx_smfc_stop(struct imx_smfc_priv *priv)
+{
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&priv->sd, "wait last EOF timeout\n");
+
+	devm_free_irq(priv->dev, priv->eof_irq, priv);
+	devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+
+	imx_smfc_unsetup(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+
+	imx_smfc_put_ipu_resources(priv);
+}
+
+static int imx_smfc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = imx_smfc_start(priv);
+	else if (!enable && priv->stream_on)
+		imx_smfc_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int imx_smfc_enum_mbus_code(struct v4l2_subdev *sd,
+				   struct v4l2_subdev_pad_config *cfg,
+				   struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (code->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	return imx_media_enum_format(&code->code, code->index,
+				     true, code->pad == priv->output_pad);
+}
+
+static int imx_smfc_get_fmt(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int imx_smfc_set_fmt(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc, *incc;
+	bool allow_planar;
+	u32 code;
+
+	if (sdformat->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   true, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, false);
+		cc = imx_media_find_format(0, code, true, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	v4l_bound_align_image(&sdformat->format.width, MIN_W, MAX_W,
+			      W_ALIGN, &sdformat->format.height,
+			      MIN_H, MAX_H, H_ALIGN, S_ALIGN);
+
+	if (sdformat->pad == priv->output_pad) {
+		incc = priv->cc[priv->input_pad];
+		sdformat->format.width = infmt->width;
+		sdformat->format.height = infmt->height;
+		if (sdformat->format.field != V4L2_FIELD_NONE)
+			sdformat->format.field = infmt->field;
+		if (cc->cs != incc->cs) {
+			sdformat->format.code = infmt->code;
+			cc = imx_media_find_format(0, sdformat->format.code,
+						   true, false);
+		}
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int imx_smfc_link_setup(struct media_entity *entity,
+			       const struct media_pad *local,
+			       const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+	} else {
+		priv->src_sd = NULL;
+		return 0;
+	}
+
+	/* must attach to CSI source */
+	if (!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_CSI))
+		return -EINVAL;
+
+	return 0;
+}
+
+static int imx_smfc_link_validate(struct v4l2_subdev *sd,
+				  struct media_link *link,
+				  struct v4l2_subdev_format *source_fmt,
+				  struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	switch (priv->src_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0:
+		priv->csi_id = 0;
+		break;
+	case IMX_MEDIA_GRP_ID_CSI1:
+		priv->csi_id = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	priv->vc_num = 0;
+	if (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		/* see NOTE in imx-csi.c */
+#if 0
+		priv->vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &priv->sd.entity);
+		if (priv->vc_num < 0)
+			return vc_num;
+#endif
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int imx_smfc_registered(struct v4l2_subdev *sd)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < SMFC_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					640, 480, 0, V4L2_FIELD_NONE,
+					&priv->cc[i]);
+	}
+
+	return media_entity_pads_init(&sd->entity, SMFC_NUM_PADS, priv->pad);
+}
+
+static struct media_entity_operations imx_smfc_entity_ops = {
+	.link_setup = imx_smfc_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_video_ops imx_smfc_video_ops = {
+	.s_stream = imx_smfc_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imx_smfc_pad_ops = {
+	.enum_mbus_code = imx_smfc_enum_mbus_code,
+	.get_fmt = imx_smfc_get_fmt,
+	.set_fmt = imx_smfc_set_fmt,
+	.link_validate = imx_smfc_link_validate,
+};
+
+static struct v4l2_subdev_ops imx_smfc_subdev_ops = {
+	.video = &imx_smfc_video_ops,
+	.pad = &imx_smfc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imx_smfc_internal_ops = {
+	.registered = imx_smfc_registered,
+};
+
+static int imx_smfc_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct imx_smfc_priv *priv;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	pdata = priv->dev->platform_data;
+	priv->ipu_id = pdata->ipu_id;
+
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = imx_smfc_eof_timeout;
+	spin_lock_init(&priv->irqlock);
+
+	v4l2_subdev_init(&priv->sd, &imx_smfc_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &imx_smfc_internal_ops;
+	priv->sd.entity.ops = &imx_smfc_entity_ops;
+	/* FIXME: this the right function? */
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	/* get our group id and SMFC id */
+	priv->sd.grp_id = pdata->grp_id;
+	priv->smfc_id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_SMFC_BIT) - 1;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+	return v4l2_async_register_subdev(&priv->sd);
+}
+
+static int imx_smfc_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imx_smfc_priv *priv = container_of(sd, struct imx_smfc_priv, sd);
+
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_smfc_ids[] = {
+	{ .name = "imx-ipuv3-smfc" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
+
+static struct platform_driver imx_smfc_driver = {
+	.probe = imx_smfc_probe,
+	.remove = imx_smfc_remove,
+	.id_table = imx_smfc_ids,
+	.driver = {
+		.name = "imx-ipuv3-smfc",
+	},
+};
+module_platform_driver(imx_smfc_driver);
+
+MODULE_DESCRIPTION("i.MX SMFC subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-smfc");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 18/24] media: imx: Add SMFC subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

This is a media entity subdevice driver for the i.MX Sensor Multi-FIFO
Controller module. Video frames are received from the CSI and can
be routed to various sinks including the i.MX Image Converter for
scaling, color-space conversion, motion compensated deinterlacing,
and image rotation.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile   |   1 +
 drivers/staging/media/imx/imx-smfc.c | 737 +++++++++++++++++++++++++++++++++++
 2 files changed, 738 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-smfc.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 133672a..3559d7b 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 
diff --git a/drivers/staging/media/imx/imx-smfc.c b/drivers/staging/media/imx/imx-smfc.c
new file mode 100644
index 0000000..614a4381
--- /dev/null
+++ b/drivers/staging/media/imx/imx-smfc.c
@@ -0,0 +1,737 @@
+/*
+ * V4L2 Capture SMFC Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of raw/unconverted video frames
+ * from the CSI, directly to memory via the Sensor Multi-FIFO Controller.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+/*
+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W       176
+#define MIN_H       144
+#define MAX_W      8192
+#define MAX_H      4096
+#define W_ALIGN    4 /* multiple of 16 pixels */
+#define H_ALIGN    1 /* multiple of 2 lines */
+#define S_ALIGN    1 /* multiple of 2 */
+
+#define SMFC_NUM_PADS 2
+
+struct imx_smfc_priv {
+	struct device        *dev;
+	struct ipu_soc       *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev   sd;
+	struct media_pad pad[SMFC_NUM_PADS];
+	int ipu_id;
+	int smfc_id;
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *smfc_ch;
+	struct ipu_smfc *smfc;
+
+	struct v4l2_mbus_framefmt format_mbus[SMFC_NUM_PADS];
+	const struct imx_media_pixfmt *cc[SMFC_NUM_PADS];
+
+	struct v4l2_mbus_config sensor_mbus_cfg;
+
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct imx_media_dma_buf *next;
+
+	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
+
+	/* the sink that will receive the dma buffers */
+	struct v4l2_subdev *sink_sd;
+	struct v4l2_subdev *src_sd;
+
+	/*
+	 * the CSI id and mipi virtual channel number at
+	 * link validate
+	 */
+	int csi_id;
+	int vc_num;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	spinlock_t irqlock;
+	struct timer_list eof_timeout_timer;
+	int eof_irq;
+	int nfb4eof_irq;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static void imx_smfc_put_ipu_resources(struct imx_smfc_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->smfc_ch))
+		ipu_idmac_put(priv->smfc_ch);
+	priv->smfc_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->smfc))
+		ipu_smfc_put(priv->smfc);
+	priv->smfc = NULL;
+}
+
+static int imx_smfc_get_ipu_resources(struct imx_smfc_priv *priv)
+{
+	int ch_num, ret;
+
+	priv->ipu = priv->md->ipu[priv->ipu_id];
+
+	ch_num = IPUV3_CHANNEL_CSI0 + priv->smfc_id;
+
+	priv->smfc = ipu_smfc_get(priv->ipu, ch_num);
+	if (IS_ERR(priv->smfc)) {
+		v4l2_err(&priv->sd, "failed to get SMFC\n");
+		ret = PTR_ERR(priv->smfc);
+		goto out;
+	}
+
+	priv->smfc_ch = ipu_idmac_get(priv->ipu, ch_num);
+	if (IS_ERR(priv->smfc_ch)) {
+		v4l2_err(&priv->sd, "could not get IDMAC channel %u\n", ch_num);
+		ret = PTR_ERR(priv->smfc_ch);
+		goto out;
+	}
+
+	return 0;
+out:
+	imx_smfc_put_ipu_resources(priv);
+	return ret;
+}
+
+static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
+{
+	struct imx_smfc_priv *priv = dev_id;
+	struct imx_media_dma_buf *done, *next;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	/* inform CSI of this EOF so it can monitor frame intervals */
+	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+			 0, NULL);
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* priv->next buffer is now the active one */
+	imx_media_dma_buf_set_active(priv->next);
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (ipu_idmac_buffer_is_ready(priv->smfc_ch, priv->ipu_buf_num))
+		ipu_idmac_clear_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+	/* get next queued buffer */
+	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+	ipu_cpmem_set_buffer(priv->smfc_ch, priv->ipu_buf_num, next->phys);
+	ipu_idmac_select_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+	/* toggle IPU double-buffer index */
+	priv->ipu_buf_num ^= 1;
+	priv->next = next;
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t imx_smfc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct imx_smfc_priv *priv = dev_id;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void imx_smfc_eof_timeout(unsigned long data)
+{
+	struct imx_smfc_priv *priv = (struct imx_smfc_priv *)data;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&priv->sd, &ev);
+}
+
+/* init the SMFC IDMAC channel */
+static void imx_smfc_setup_channel(struct imx_smfc_priv *priv)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct imx_media_dma_buf *buf0, *buf1;
+	unsigned int burst_size;
+	struct ipu_image image;
+	bool passthrough;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	ipu_cpmem_zero(priv->smfc_ch);
+
+	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	image.phys0 = buf0->phys;
+	image.phys1 = buf1->phys;
+	ipu_cpmem_set_image(priv->smfc_ch, &image);
+
+	burst_size = (outfmt->width & 0xf) ? 8 : 16;
+
+	ipu_cpmem_set_burstsize(priv->smfc_ch, burst_size);
+
+	/*
+	 * If the sensor uses 16-bit parallel CSI bus, we must handle
+	 * the data internally in the IPU as 16-bit generic, aka
+	 * passthrough mode.
+	 */
+	passthrough = (priv->sensor_mbus_cfg.type != V4L2_MBUS_CSI2 &&
+		       priv->sensor->sensor_ep.bus.parallel.bus_width >= 16);
+
+	if (passthrough)
+		ipu_cpmem_set_format_passthrough(priv->smfc_ch, 16);
+
+	/*
+	 * Set the channel for the direct CSI-->memory via SMFC
+	 * use-case to very high priority, by enabling the watermark
+	 * signal in the SMFC, enabling WM in the channel, and setting
+	 * the channel priority to high.
+	 *
+	 * Refer to the i.mx6 rev. D TRM Table 36-8: Calculated priority
+	 * value.
+	 *
+	 * The WM's are set very low by intention here to ensure that
+	 * the SMFC FIFOs do not overflow.
+	 */
+	ipu_smfc_set_watermark(priv->smfc, 0x02, 0x01);
+	ipu_cpmem_set_high_priority(priv->smfc_ch);
+	ipu_idmac_enable_watermark(priv->smfc_ch, true);
+	ipu_cpmem_set_axi_id(priv->smfc_ch, 0);
+	ipu_idmac_lock_enable(priv->smfc_ch, 8);
+
+	burst_size = ipu_cpmem_get_burstsize(priv->smfc_ch);
+	burst_size = passthrough ?
+		(burst_size >> 3) - 1 : (burst_size >> 2) - 1;
+
+	ipu_smfc_set_burstsize(priv->smfc, burst_size);
+
+	if (outfmt->field == V4L2_FIELD_NONE &&
+	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+	     infmt->field == V4L2_FIELD_ALTERNATE))
+		ipu_cpmem_interlaced_scan(priv->smfc_ch,
+					  image.pix.bytesperline);
+
+	ipu_idmac_set_double_buffer(priv->smfc_ch, true);
+}
+
+static void imx_smfc_unsetup(struct imx_smfc_priv *priv)
+{
+	ipu_idmac_disable_channel(priv->smfc_ch);
+	ipu_smfc_disable(priv->smfc);
+}
+
+static void imx_smfc_setup(struct imx_smfc_priv *priv)
+{
+	imx_smfc_setup_channel(priv);
+
+	ipu_cpmem_dump(priv->smfc_ch);
+	ipu_dump(priv->ipu);
+
+	ipu_smfc_enable(priv->smfc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->smfc_ch, 0);
+	ipu_idmac_select_buffer(priv->smfc_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->smfc_ch);
+}
+
+static int imx_smfc_start(struct imx_smfc_priv *priv)
+{
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = imx_smfc_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	ipu_smfc_map_channel(priv->smfc, priv->csi_id, priv->vc_num);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->ipu_buf_num = 0;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	imx_smfc_setup(priv);
+
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						 priv->smfc_ch,
+						 IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(priv->dev, priv->nfb4eof_irq,
+			       imx_smfc_nfb4eof_interrupt, 0,
+			       "imx-smfc-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_unsetup;
+	}
+
+	priv->eof_irq = ipu_idmac_channel_irq(priv->ipu, priv->smfc_ch,
+					      IPU_IRQ_EOF);
+
+	ret = devm_request_irq(priv->dev, priv->eof_irq,
+			       imx_smfc_eof_interrupt, 0,
+			       "imx-smfc-eof", priv);
+	if (ret) {
+		v4l2_err(&priv->sd,
+			 "Error registering eof irq: %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+	imx_smfc_unsetup(priv);
+out_put_ipu:
+	imx_smfc_put_ipu_resources(priv);
+	return ret;
+}
+
+static void imx_smfc_stop(struct imx_smfc_priv *priv)
+{
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&priv->sd, "wait last EOF timeout\n");
+
+	devm_free_irq(priv->dev, priv->eof_irq, priv);
+	devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+
+	imx_smfc_unsetup(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+
+	imx_smfc_put_ipu_resources(priv);
+}
+
+static int imx_smfc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = imx_smfc_start(priv);
+	else if (!enable && priv->stream_on)
+		imx_smfc_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int imx_smfc_enum_mbus_code(struct v4l2_subdev *sd,
+				   struct v4l2_subdev_pad_config *cfg,
+				   struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (code->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	return imx_media_enum_format(&code->code, code->index,
+				     true, code->pad == priv->output_pad);
+}
+
+static int imx_smfc_get_fmt(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int imx_smfc_set_fmt(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc, *incc;
+	bool allow_planar;
+	u32 code;
+
+	if (sdformat->pad >= SMFC_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   true, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, false);
+		cc = imx_media_find_format(0, code, true, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	v4l_bound_align_image(&sdformat->format.width, MIN_W, MAX_W,
+			      W_ALIGN, &sdformat->format.height,
+			      MIN_H, MAX_H, H_ALIGN, S_ALIGN);
+
+	if (sdformat->pad == priv->output_pad) {
+		incc = priv->cc[priv->input_pad];
+		sdformat->format.width = infmt->width;
+		sdformat->format.height = infmt->height;
+		if (sdformat->format.field != V4L2_FIELD_NONE)
+			sdformat->format.field = infmt->field;
+		if (cc->cs != incc->cs) {
+			sdformat->format.code = infmt->code;
+			cc = imx_media_find_format(0, sdformat->format.code,
+						   true, false);
+		}
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int imx_smfc_link_setup(struct media_entity *entity,
+			       const struct media_pad *local,
+			       const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+	} else {
+		priv->src_sd = NULL;
+		return 0;
+	}
+
+	/* must attach to CSI source */
+	if (!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_CSI))
+		return -EINVAL;
+
+	return 0;
+}
+
+static int imx_smfc_link_validate(struct v4l2_subdev *sd,
+				  struct media_link *link,
+				  struct v4l2_subdev_format *source_fmt,
+				  struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	switch (priv->src_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0:
+		priv->csi_id = 0;
+		break;
+	case IMX_MEDIA_GRP_ID_CSI1:
+		priv->csi_id = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	priv->vc_num = 0;
+	if (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		/* see NOTE in imx-csi.c */
+#if 0
+		priv->vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &priv->sd.entity);
+		if (priv->vc_num < 0)
+			return vc_num;
+#endif
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int imx_smfc_registered(struct v4l2_subdev *sd)
+{
+	struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < SMFC_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					640, 480, 0, V4L2_FIELD_NONE,
+					&priv->cc[i]);
+	}
+
+	return media_entity_pads_init(&sd->entity, SMFC_NUM_PADS, priv->pad);
+}
+
+static struct media_entity_operations imx_smfc_entity_ops = {
+	.link_setup = imx_smfc_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_video_ops imx_smfc_video_ops = {
+	.s_stream = imx_smfc_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imx_smfc_pad_ops = {
+	.enum_mbus_code = imx_smfc_enum_mbus_code,
+	.get_fmt = imx_smfc_get_fmt,
+	.set_fmt = imx_smfc_set_fmt,
+	.link_validate = imx_smfc_link_validate,
+};
+
+static struct v4l2_subdev_ops imx_smfc_subdev_ops = {
+	.video = &imx_smfc_video_ops,
+	.pad = &imx_smfc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imx_smfc_internal_ops = {
+	.registered = imx_smfc_registered,
+};
+
+static int imx_smfc_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct imx_smfc_priv *priv;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	pdata = priv->dev->platform_data;
+	priv->ipu_id = pdata->ipu_id;
+
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = imx_smfc_eof_timeout;
+	spin_lock_init(&priv->irqlock);
+
+	v4l2_subdev_init(&priv->sd, &imx_smfc_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &imx_smfc_internal_ops;
+	priv->sd.entity.ops = &imx_smfc_entity_ops;
+	/* FIXME: this the right function? */
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	/* get our group id and SMFC id */
+	priv->sd.grp_id = pdata->grp_id;
+	priv->smfc_id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_SMFC_BIT) - 1;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+	return v4l2_async_register_subdev(&priv->sd);
+}
+
+static int imx_smfc_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imx_smfc_priv *priv = container_of(sd, struct imx_smfc_priv, sd);
+
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_smfc_ids[] = {
+	{ .name = "imx-ipuv3-smfc" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
+
+static struct platform_driver imx_smfc_driver = {
+	.probe = imx_smfc_probe,
+	.remove = imx_smfc_remove,
+	.id_table = imx_smfc_ids,
+	.driver = {
+		.name = "imx-ipuv3-smfc",
+	},
+};
+module_platform_driver(imx_smfc_driver);
+
+MODULE_DESCRIPTION("i.MX SMFC subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-smfc");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 19/24] media: imx: Add IC subdev drivers
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

This is a set of three media entity subdevice drivers for the i.MX
Image Converter. The i.MX IC module contains three independent
"tasks":

- Pre-processing Encode task: video frames are routed directly from
  the CSI and can be scaled, color-space converted, and rotated.
  Scaled output is limited to 1024x1024 resolution. Output frames
  are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same
  conversions as the pre-process encode task, but in addition can
  be used for hardware motion compensated deinterlacing. Frames can
  come either directly from the CSI or from the SMFC entities (memory
  buffers via IDMAC channels). Scaled output is limited to 1024x1024
  resolution. Output frames can be routed to various sinks including
  the post-processing task entities.

- Post-processing task: same conversions as pre-process encode. However
  this entity sends frames to the i.MX IPU image converter which supports
  image tiling, which allows scaled output up to 4096x4096 resolution.
  Output frames can be routed to the camera interfaces.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile        |    2 +
 drivers/staging/media/imx/imx-ic-common.c |  109 +++
 drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic.h        |   38 +
 6 files changed, 2997 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 3559d7b..d2a962c 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -1,8 +1,10 @@
 imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 	imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
 
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
new file mode 100644
index 0000000..45706ca
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-common.c
@@ -0,0 +1,109 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
+	[IC_TASK_ENCODER]        = &imx_ic_prpenc_ops,
+	[IC_TASK_VIEWFINDER]     = &imx_ic_prpvf_ops,
+	[IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
+};
+
+static int imx_ic_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct imx_ic_priv *priv;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	/* get our ipu_id, grp_id and IC task id */
+	pdata = priv->dev->platform_data;
+	priv->ipu_id = pdata->ipu_id;
+	switch (pdata->grp_id) {
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		priv->task_id = IC_TASK_ENCODER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		priv->task_id = IC_TASK_VIEWFINDER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+		priv->task_id = IC_TASK_POST_PROCESSOR;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
+	priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	priv->sd.grp_id = pdata->grp_id;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+	ret = ic_ops[priv->task_id]->init(priv);
+	if (ret)
+		return ret;
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		ic_ops[priv->task_id]->remove(priv);
+
+	return ret;
+}
+
+static int imx_ic_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
+
+	ic_ops[priv->task_id]->remove(priv);
+
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_ic_ids[] = {
+	{ .name = "imx-ipuv3-ic" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_ic_ids);
+
+static struct platform_driver imx_ic_driver = {
+	.probe = imx_ic_probe,
+	.remove = imx_ic_remove,
+	.id_table = imx_ic_ids,
+	.driver = {
+		.name = "imx-ipuv3-ic",
+	},
+};
+module_platform_driver(imx_ic_driver);
+
+MODULE_DESCRIPTION("i.MX IC subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-ic");
diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
new file mode 100644
index 0000000..1f75616
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-pp.c
@@ -0,0 +1,636 @@
+/*
+ * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-image-convert.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PP_NUM_PADS 2
+
+struct pp_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+	int pp_id;
+
+	struct ipu_soc *ipu;
+	struct ipu_image_convert_ctx *ic_ctx;
+
+	struct media_pad pad[PP_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	/* our dma buffer sink ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	/* the dma buffer ring we send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct ipu_image_convert_run *out_run;
+
+	struct imx_media_dma_buf *inbuf; /* last input buffer */
+
+	bool stream_on;    /* streaming is on */
+	bool stop;         /* streaming is stopping */
+	spinlock_t irqlock;
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PP_NUM_PADS];
+
+	/* motion select control */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+};
+
+static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void pp_convert_complete(struct ipu_image_convert_run *run,
+				void *data)
+{
+	struct pp_priv *priv = data;
+	struct imx_media_dma_buf *done;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, run->status ?
+				       IMX_MEDIA_BUF_STATUS_ERROR :
+				       IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* we're done with the inbuf, queue it back */
+	imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
+
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+}
+
+static void pp_queue_conversion(struct pp_priv *priv,
+				struct imx_media_dma_buf *inbuf)
+{
+	struct ipu_image_convert_run *run;
+	struct imx_media_dma_buf *outbuf;
+
+	/* get next queued buffer and make it active */
+	outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(outbuf);
+	priv->inbuf = inbuf;
+
+	run = &priv->out_run[outbuf->index];
+	run->ctx = priv->ic_ctx;
+	run->in_phys = inbuf->phys;
+	run->out_phys = outbuf->phys;
+	ipu_image_convert_queue(run);
+}
+
+static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_dma_buf_ring **ring;
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		/* src asks for a buffer ring */
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		/* src hands us a new buffer */
+		spin_lock_irqsave(&priv->irqlock, flags);
+		if (!priv->stop &&
+		    !imx_media_dma_buf_get_active(priv->out_ring)) {
+			buf = imx_media_dma_buf_dequeue(priv->in_ring);
+			if (buf)
+				pp_queue_conversion(priv, buf);
+		}
+		spin_unlock_irqrestore(&priv->irqlock, flags);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int pp_start(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct ipu_image image_in, image_out;
+	const struct imx_media_pixfmt *incc;
+	struct v4l2_mbus_framefmt *infmt;
+	int i, in_size, ret;
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		return ret;
+
+	imx_media_mbus_fmt_to_ipu_image(&image_in,
+					&priv->format_mbus[priv->input_pad]);
+	imx_media_mbus_fmt_to_ipu_image(&image_out,
+					&priv->format_mbus[priv->output_pad]);
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+	priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
+						 IC_TASK_POST_PROCESSOR,
+						 &image_in, &image_out,
+						 priv->rot_mode,
+						 pp_convert_complete, priv);
+	if (IS_ERR(priv->ic_ctx))
+		return PTR_ERR(priv->ic_ctx);
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	incc = priv->cc[priv->input_pad];
+	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+	if (priv->in_ring) {
+		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+	}
+
+	priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
+						     &priv->src_sd->entity,
+						     &ic_priv->sd.entity,
+						     in_size,
+						     IMX_MEDIA_MIN_RING_BUFS,
+						     true);
+	if (IS_ERR(priv->in_ring)) {
+		v4l2_err(&ic_priv->sd,
+			 "failed to alloc dma-buf ring\n");
+		ret = PTR_ERR(priv->in_ring);
+		priv->in_ring = NULL;
+		goto out_unprep;
+	}
+
+	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
+		imx_media_dma_buf_queue(priv->in_ring, i);
+
+	priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
+				sizeof(*priv->out_run), GFP_KERNEL);
+	if (!priv->out_run) {
+		v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
+		ret = -ENOMEM;
+		goto out_free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+out_free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+out_unprep:
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	return ret;
+}
+
+static void pp_stop(struct pp_priv *priv)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	kfree(priv->out_run);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that its sink buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int pp_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = pp_start(priv);
+	else if (!enable && priv->stream_on)
+		pp_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int pp_enum_mbus_code(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_mbus_code_enum *code)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 fourcc;
+	int ret;
+
+	if (code->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	ret = ipu_image_convert_enum_format(code->index, &fourcc);
+	if (ret)
+		return ret;
+
+	/* convert returned fourcc to mbus code */
+	cc = imx_media_find_format(fourcc, 0, true, true);
+	if (WARN_ON(!cc))
+		return -EINVAL;
+
+	code->code = cc->codes[0];
+	return 0;
+}
+
+static int pp_get_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int pp_set_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	struct ipu_image test_in, test_out;
+	u32 code;
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
+	} else {
+		imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		if (sdformat->pad == priv->output_pad) {
+			*outfmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
+		} else {
+			*infmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
+		}
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int pp_link_setup(struct media_entity *entity,
+			 const struct media_pad *local,
+			 const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct pp_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+	} else {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+	}
+
+	return 0;
+}
+
+static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct pp_priv *priv = container_of(ctrl->handler,
+					       struct pp_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		struct v4l2_mbus_framefmt *infmt, *outfmt;
+		struct ipu_image test_in, test_out;
+
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		/*
+		 * make sure this rotation will work with current input/output
+		 * formats before setting
+		 */
+		infmt = &priv->format_mbus[priv->input_pad];
+		outfmt = &priv->format_mbus[priv->output_pad];
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+
+		ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);
+		if (ret)
+			return ret;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops pp_ctrl_ops = {
+	.s_ctrl = pp_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config pp_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};
+
+#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
+
+static int pp_init_controls(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
+
+	for (i = 0; i < PP_NUM_CONTROLS; i++) {
+		c = &pp_std_ctrl[i];
+		v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
+				  c->id, c->min, c->max, c->step, c->def);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		v4l2_ctrl_handler_free(hdlr);
+		return ret;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int pp_registered(struct v4l2_subdev *sd)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PP_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	ret = pp_init_controls(priv);
+	if (ret)
+		return ret;
+
+	ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+free_ctrls:
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	return ret;
+}
+
+static struct v4l2_subdev_pad_ops pp_pad_ops = {
+	.enum_mbus_code = pp_enum_mbus_code,
+	.get_fmt = pp_get_fmt,
+	.set_fmt = pp_set_fmt,
+};
+
+static struct v4l2_subdev_video_ops pp_video_ops = {
+	.s_stream = pp_s_stream,
+};
+
+static struct v4l2_subdev_core_ops pp_core_ops = {
+	.ioctl = pp_ioctl,
+};
+
+static struct media_entity_operations pp_entity_ops = {
+	.link_setup = pp_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops pp_subdev_ops = {
+	.video = &pp_video_ops,
+	.pad = &pp_pad_ops,
+	.core = &pp_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops pp_internal_ops = {
+	.registered = pp_registered,
+};
+
+static int pp_init(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+	spin_lock_init(&priv->irqlock);
+
+	/* get our PP id */
+	priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
+
+	return 0;
+}
+
+static void pp_remove(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_pp_ops = {
+	.subdev_ops = &pp_subdev_ops,
+	.internal_ops = &pp_internal_ops,
+	.entity_ops = &pp_entity_ops,
+	.init = pp_init,
+	.remove = pp_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
new file mode 100644
index 0000000..3d85a82
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpenc.c
@@ -0,0 +1,1033 @@
+/*
+ * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of video frames from the CSI, which
+ * are routed directly to the Image Converter preprocess encode task,
+ * for resizing, colorspace conversion, and rotation.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PRPENC_NUM_PADS 2
+
+#define MAX_W_IC   1024
+#define MAX_H_IC   1024
+#define MAX_W_SINK 4096
+#define MAX_H_SINK 4096
+
+struct prpenc_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+
+	/* IPU units we require */
+	struct ipu_soc *ipu;
+	struct ipu_ic *ic_enc;
+
+	struct media_pad pad[PRPENC_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *enc_ch;
+	struct ipuv3_channel *enc_rot_in_ch;
+	struct ipuv3_channel *enc_rot_out_ch;
+
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct imx_media_dma_buf *next;
+
+	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	/* the CSI id at link validate */
+	int csi_id;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
+
+	struct imx_media_dma_buf rot_buf[2];
+
+	/* controls */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+
+	spinlock_t irqlock;
+
+	struct timer_list eof_timeout_timer;
+	int eof_irq;
+	int nfb4eof_irq;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->ic_enc))
+		ipu_ic_put(priv->ic_enc);
+	priv->ic_enc = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_ch))
+		ipu_idmac_put(priv->enc_ch);
+	priv->enc_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
+		ipu_idmac_put(priv->enc_rot_in_ch);
+	priv->enc_rot_in_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
+		ipu_idmac_put(priv->enc_rot_out_ch);
+	priv->enc_rot_out_ch = NULL;
+}
+
+static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+	priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
+	if (IS_ERR(priv->ic_enc)) {
+		v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
+		ret = PTR_ERR(priv->ic_enc);
+		goto out;
+	}
+
+	priv->enc_ch = ipu_idmac_get(priv->ipu,
+				     IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+	if (IS_ERR(priv->enc_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+		ret = PTR_ERR(priv->enc_ch);
+		goto out;
+	}
+
+	priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
+					    IPUV3_CHANNEL_MEM_ROT_ENC);
+	if (IS_ERR(priv->enc_rot_in_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_MEM_ROT_ENC);
+		ret = PTR_ERR(priv->enc_rot_in_ch);
+		goto out;
+	}
+
+	priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
+					     IPUV3_CHANNEL_ROT_ENC_MEM);
+	if (IS_ERR(priv->enc_rot_out_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_ROT_ENC_MEM);
+		ret = PTR_ERR(priv->enc_rot_out_ch);
+		goto out;
+	}
+
+	return 0;
+out:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_media_dma_buf *done, *next;
+	struct ipuv3_channel *channel;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	/* inform CSI of this EOF so it can monitor frame intervals */
+	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+			 0, NULL);
+
+	channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
+		priv->enc_rot_out_ch : priv->enc_ch;
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* priv->next buffer is now the active one */
+	imx_media_dma_buf_set_active(priv->next);
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
+		ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);
+
+	/* get next queued buffer */
+	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+	ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
+	ipu_idmac_select_buffer(channel, priv->ipu_buf_num);
+
+	/* toggle IPU double-buffer index */
+	priv->ipu_buf_num ^= 1;
+	priv->next = next;
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void prpenc_eof_timeout(unsigned long data)
+{
+	struct prpenc_priv *priv = (struct prpenc_priv *)data;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void prpenc_setup_channel(struct prpenc_priv *priv,
+				 struct ipuv3_channel *channel,
+				 enum ipu_rotate_mode rot_mode,
+				 dma_addr_t addr0, dma_addr_t addr1,
+				 bool rot_swap_width_height)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	unsigned int burst_size;
+	struct ipu_image image;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+
+	ipu_cpmem_zero(channel);
+
+	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+	image.phys0 = addr0;
+	image.phys1 = addr1;
+	ipu_cpmem_set_image(channel, &image);
+
+	if (channel == priv->enc_rot_in_ch ||
+	    channel == priv->enc_rot_out_ch) {
+		burst_size = 8;
+		ipu_cpmem_set_block_mode(channel);
+	} else {
+		burst_size = (outfmt->width & 0xf) ? 8 : 16;
+	}
+
+	ipu_cpmem_set_burstsize(channel, burst_size);
+
+	if (rot_mode)
+		ipu_cpmem_set_rotation(channel, rot_mode);
+
+	if (outfmt->field == V4L2_FIELD_NONE &&
+	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+	     infmt->field == V4L2_FIELD_ALTERNATE) &&
+	    channel == priv->enc_ch)
+		ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
+
+	ipu_ic_task_idma_init(priv->ic_enc, channel,
+			      outfmt->width, outfmt->height,
+			      burst_size, rot_mode);
+	ipu_cpmem_set_axi_id(channel, 1);
+
+	ipu_idmac_set_double_buffer(channel, true);
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+}
+
+static int prpenc_setup_rotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int out_size, ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
+
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
+		return ret;
+	}
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
+		goto free_rot0;
+	}
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->height, outfmt->width,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		goto free_rot1;
+	}
+
+	/* init the IC ENC-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch,
+			     IPU_ROTATE_NONE,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	/* init the MEM-->IC ENC ROT IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_in_ch,
+			     priv->rot_mode,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the destination IC ENC ROT-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_out_ch,
+			     IPU_ROTATE_NONE,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	/* now link IC ENC-->MEM to MEM-->IC ENC ROT */
+	ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
+
+	/* enable the IC */
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_out_ch);
+
+	/* and finally enable the IC PRPENC task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+
+free_rot1:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+free_rot0:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	return ret;
+}
+
+static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_out_ch);
+
+	ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
+
+	ipu_ic_disable(priv->ic_enc);
+
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+}
+
+static int prpenc_setup_norotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->width, outfmt->height,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		return ret;
+	}
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the IC PRP-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	ipu_cpmem_dump(priv->enc_ch);
+	ipu_ic_dump(priv->ic_enc);
+	ipu_dump(priv->ipu);
+
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+
+	/* enable the IC ENCODE task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+}
+
+static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_ic_disable(priv->ic_enc);
+}
+
+static int prpenc_start(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = prpenc_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	/* set IC to receive from CSI */
+	ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->ipu_buf_num = 0;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		ret = prpenc_setup_rotation(priv);
+	else
+		ret = prpenc_setup_norotation(priv);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->enc_ch,
+						  IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+			       prpenc_nfb4eof_interrupt, 0,
+			       "imx-ic-prpenc-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_unsetup;
+	}
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
+	else
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
+
+	ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
+			       prpenc_eof_interrupt, 0,
+			       "imx-ic-prpenc-eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering eof irq: %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+out_put_ipu:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prpenc_stop(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp,
+		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+	devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+
+	prpenc_put_ipu_resources(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_pad_config *cfg,
+				 struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	bool allow_planar;
+
+	if (code->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	allow_planar = (code->pad == priv->output_pad);
+
+	return imx_media_enum_format(&code->code, code->index,
+				     true, allow_planar);
+}
+
+static int prpenc_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int prpenc_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	bool allow_planar;
+	u32 code;
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   true, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, false);
+		cc = imx_media_find_format(0, code, true, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_IC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_IC);
+
+		if (sdformat->format.field != V4L2_FIELD_NONE)
+			sdformat->format.field = infmt->field;
+
+		/* IC resizer cannot downsize more than 4:1 */
+		if (ipu_rot_mode_is_irt(priv->rot_mode)) {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->height / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->width / 4);
+		} else {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->width / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->height / 4);
+		}
+	} else {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_SINK);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_SINK);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_setup(struct media_entity *entity,
+			     const struct media_pad *local,
+			     const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	/* this is sink pad */
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+	} else {
+		priv->src_sd = NULL;
+		return 0;
+	}
+
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0:
+		priv->csi_id = 0;
+		break;
+	case IMX_MEDIA_GRP_ID_CSI1:
+		priv->csi_id = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_validate(struct v4l2_subdev *sd,
+				struct media_link *link,
+				struct v4l2_subdev_format *source_fmt,
+				struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link,
+						source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		int vc_num = 0;
+		/* see NOTE in imx-csi.c */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &ic_priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		/* only virtual channel 0 can be sent to IC */
+		if (vc_num != 0)
+			return -EINVAL;
+	} else {
+		/*
+		 * only 8-bit pixels can be sent to IC for parallel
+		 * busses
+		 */
+		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct prpenc_priv *priv = container_of(ctrl->handler,
+					       struct prpenc_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
+	.s_ctrl = prpenc_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};
+
+#define PRPENC_NUM_CONTROLS ARRAY_SIZE(prpenc_std_ctrl)
+
+static int prpenc_init_controls(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, PRPENC_NUM_CONTROLS);
+
+	for (i = 0; i < PRPENC_NUM_CONTROLS; i++) {
+		c = &prpenc_std_ctrl[i];
+		v4l2_ctrl_new_std(hdlr, &prpenc_ctrl_ops,
+				  c->id, c->min, c->max, c->step, c->def);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		goto out_free;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+	return 0;
+
+out_free:
+	v4l2_ctrl_handler_free(hdlr);
+	return ret;
+}
+
+static int prpenc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = prpenc_start(priv);
+	else if (!enable && priv->stream_on)
+		prpenc_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int prpenc_registered(struct v4l2_subdev *sd)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PRPENC_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	ret = prpenc_init_controls(priv);
+	if (ret)
+		return ret;
+
+	ret = media_entity_pads_init(&sd->entity, PRPENC_NUM_PADS, priv->pad);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static struct v4l2_subdev_pad_ops prpenc_pad_ops = {
+	.enum_mbus_code = prpenc_enum_mbus_code,
+	.get_fmt = prpenc_get_fmt,
+	.set_fmt = prpenc_set_fmt,
+	.link_validate = prpenc_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpenc_video_ops = {
+	.s_stream = prpenc_s_stream,
+};
+
+static struct media_entity_operations prpenc_entity_ops = {
+	.link_setup = prpenc_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops prpenc_subdev_ops = {
+	.video = &prpenc_video_ops,
+	.pad = &prpenc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpenc_internal_ops = {
+	.registered = prpenc_registered,
+};
+
+static int prpenc_init(struct imx_ic_priv *ic_priv)
+{
+	struct prpenc_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+
+	spin_lock_init(&priv->irqlock);
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = prpenc_eof_timeout;
+
+	return 0;
+}
+
+static void prpenc_remove(struct imx_ic_priv *ic_priv)
+{
+	struct prpenc_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_prpenc_ops = {
+	.subdev_ops = &prpenc_subdev_ops,
+	.internal_ops = &prpenc_internal_ops,
+	.entity_ops = &prpenc_entity_ops,
+	.init = prpenc_init,
+	.remove = prpenc_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpvf.c b/drivers/staging/media/imx/imx-ic-prpvf.c
new file mode 100644
index 0000000..dc2d402
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpvf.c
@@ -0,0 +1,1179 @@
+/*
+ * V4L2 IC Deinterlacer Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+/*
+ * This subdev implements two different video pipelines:
+ *
+ * CSI -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends a single interlaced field F(n-1)
+ * directly to the VDIC (and optionally the following field F(n)
+ * can be sent to memory via IDMAC channel 13). So only two fields
+ * can be processed by the VDIC. This pipeline only works in VDIC's
+ * high motion mode, which only requires a single field for processing.
+ * The other motion modes (low and medium) require three fields, so this
+ * pipeline does not work in those modes. Also, it is not clear how this
+ * pipeline can deal with the various field orders (sequential BT/TB,
+ * interlaced BT/TB).
+ *
+ * CSI -> CH[0-3] -> MEM -> CH8,9,10 -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends raw and full frames to memory buffers
+ * via the SMFC channels 0-3. Fields from these frames are then
+ * transferred to the VDIC via IDMAC channels 8,9,10. The VDIC requires
+ * three fields: previous field F(n-1), current field F(n), and next
+ * field F(n+1), so we need three raw frames in memory: two completed frames
+ * to send F(n-1), F(n), F(n+1) to the VDIC, and a third frame for active
+ * CSI capture while the completed fields are sent through the VDIC->IC for
+ * processing.
+ *
+ * The "direct" CSI->VDIC pipeline requires less memory bus bandwidth
+ * (just 1 channel vs. 5 channels for indirect pipeline), but it can
+ * only be used in high motion mode, and it only processes a single
+ * field (so half the original image resolution is lost).
+ */
+
+struct prpvf_priv;
+
+struct prpvf_pipeline_ops {
+	int (*setup)(struct prpvf_priv *priv);
+	void (*start)(struct prpvf_priv *priv);
+	void (*stop)(struct prpvf_priv *priv);
+	void (*disable)(struct prpvf_priv *priv);
+};
+
+#define PRPVF_NUM_PADS 2
+
+#define MAX_W_IC   1024
+#define MAX_H_IC   1024
+#define MAX_W_VDIC  968
+#define MAX_H_VDIC 2048
+
+struct prpvf_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+
+	/* IPU units we require */
+	struct ipu_soc *ipu;
+	struct ipu_ic *ic_vf;
+	struct ipu_vdi *vdi;
+
+	struct media_pad pad[PRPVF_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *vdi_in_ch_p; /* F(n-1) transfer channel */
+	struct ipuv3_channel *vdi_in_ch;   /* F(n) transfer channel */
+	struct ipuv3_channel *vdi_in_ch_n; /* F(n+1) transfer channel */
+	struct ipuv3_channel *prpvf_out_ch;/* final progressive frame channel */
+
+	/* pipeline operations */
+	struct prpvf_pipeline_ops *ops;
+
+	/* our dma buffer sink ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+
+	/* ipu buf num for double-buffering (csi-direct path only) */
+	int ipu_buf_num;
+	struct imx_media_dma_buf *next_out_buf;
+
+	/* current and last input buffers indirect path */
+	struct imx_media_dma_buf *curr_in_buf;
+	struct imx_media_dma_buf *last_in_buf;
+
+	/*
+	 * translated field type, input line stride, and field size
+	 * for indirect path
+	 */
+	u32 fieldtype;
+	u32 in_stride;
+	u32 field_size;
+
+	struct v4l2_subdev *src_sd;
+	/* the sink that will receive the progressive out buffers */
+	struct v4l2_subdev *sink_sd;
+
+	/* the attached CSI at stream on */
+	struct v4l2_subdev *csi_sd;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	/* the video standard from sensor at time of streamon */
+	v4l2_std_id std;
+
+	struct v4l2_mbus_framefmt format_mbus[PRPVF_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PRPVF_NUM_PADS];
+
+	bool csi_direct;  /* using direct CSI->VDIC->IC pipeline */
+
+	/* motion select control */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	enum ipu_motion_sel motion;
+
+	struct timer_list eof_timeout_timer;
+
+	int nfb4eof_irq; /* CSI or PRPVF channel NFB4EOF IRQ */
+	int out_eof_irq; /* PRPVF channel EOF IRQ */
+	spinlock_t irqlock;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static inline struct prpvf_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void prpvf_put_ipu_resources(struct prpvf_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->ic_vf))
+		ipu_ic_put(priv->ic_vf);
+	priv->ic_vf = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch_p))
+		ipu_idmac_put(priv->vdi_in_ch_p);
+	priv->vdi_in_ch_p = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch))
+		ipu_idmac_put(priv->vdi_in_ch);
+	priv->vdi_in_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch_n))
+		ipu_idmac_put(priv->vdi_in_ch_n);
+	priv->vdi_in_ch_n = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->prpvf_out_ch))
+		ipu_idmac_put(priv->prpvf_out_ch);
+	priv->prpvf_out_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi))
+		ipu_vdi_put(priv->vdi);
+	priv->vdi = NULL;
+}
+
+static int prpvf_get_ipu_resources(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret, err_chan;
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+	priv->ic_vf = ipu_ic_get(priv->ipu, IC_TASK_VIEWFINDER);
+	if (IS_ERR(priv->ic_vf)) {
+		v4l2_err(&ic_priv->sd, "failed to get IC VF\n");
+		ret = PTR_ERR(priv->ic_vf);
+		goto out;
+	}
+
+	priv->vdi = ipu_vdi_get(priv->ipu);
+	if (IS_ERR(priv->vdi)) {
+		v4l2_err(&ic_priv->sd, "failed to get VDIC\n");
+		ret = PTR_ERR(priv->vdi);
+		goto out;
+	}
+
+	priv->prpvf_out_ch = ipu_idmac_get(priv->ipu,
+					   IPUV3_CHANNEL_IC_PRP_VF_MEM);
+	if (IS_ERR(priv->prpvf_out_ch)) {
+		err_chan = IPUV3_CHANNEL_IC_PRP_VF_MEM;
+		ret = PTR_ERR(priv->prpvf_out_ch);
+		goto out_err_chan;
+	}
+
+	if (!priv->csi_direct) {
+		priv->vdi_in_ch_p = ipu_idmac_get(priv->ipu,
+						  IPUV3_CHANNEL_MEM_VDI_PREV);
+		if (IS_ERR(priv->vdi_in_ch_p)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_PREV;
+			ret = PTR_ERR(priv->vdi_in_ch_p);
+			goto out_err_chan;
+		}
+
+		priv->vdi_in_ch = ipu_idmac_get(priv->ipu,
+						IPUV3_CHANNEL_MEM_VDI_CUR);
+		if (IS_ERR(priv->vdi_in_ch)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_CUR;
+			ret = PTR_ERR(priv->vdi_in_ch);
+			goto out_err_chan;
+		}
+
+		priv->vdi_in_ch_n = ipu_idmac_get(priv->ipu,
+						  IPUV3_CHANNEL_MEM_VDI_NEXT);
+		if (IS_ERR(priv->vdi_in_ch_n)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_NEXT;
+			ret = PTR_ERR(priv->vdi_in_ch_n);
+			goto out_err_chan;
+		}
+	}
+
+	return 0;
+
+out_err_chan:
+	v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n", err_chan);
+out:
+	prpvf_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prepare_vdi_in_buffers(struct prpvf_priv *priv,
+				   struct imx_media_dma_buf *curr)
+{
+	dma_addr_t prev_phys, curr_phys, next_phys;
+	struct imx_media_dma_buf *last;
+
+	last = priv->last_in_buf ? priv->last_in_buf : curr;
+	priv->curr_in_buf = curr;
+
+	switch (priv->fieldtype) {
+	case V4L2_FIELD_SEQ_TB:
+		prev_phys = last->phys;
+		curr_phys = curr->phys + priv->field_size;
+		next_phys = curr->phys;
+		break;
+	case V4L2_FIELD_SEQ_BT:
+		prev_phys = last->phys + priv->field_size;
+		curr_phys = curr->phys;
+		next_phys = curr->phys + priv->field_size;
+		break;
+	case V4L2_FIELD_INTERLACED_BT:
+		prev_phys = last->phys + priv->in_stride;
+		curr_phys = curr->phys;
+		next_phys = curr->phys + priv->in_stride;
+		break;
+	default:
+		/* assume V4L2_FIELD_INTERLACED_TB */
+		prev_phys = last->phys;
+		curr_phys = curr->phys + priv->in_stride;
+		next_phys = curr->phys;
+		break;
+	}
+
+	ipu_cpmem_set_buffer(priv->vdi_in_ch_p, 0, prev_phys);
+	ipu_cpmem_set_buffer(priv->vdi_in_ch,   0, curr_phys);
+	ipu_cpmem_set_buffer(priv->vdi_in_ch_n, 0, next_phys);
+
+	ipu_idmac_select_buffer(priv->vdi_in_ch_p, 0);
+	ipu_idmac_select_buffer(priv->vdi_in_ch, 0);
+	ipu_idmac_select_buffer(priv->vdi_in_ch_n, 0);
+}
+
+static void prepare_prpvf_out_buffer(struct prpvf_priv *priv)
+{
+	struct imx_media_dma_buf *buf;
+
+	/* get next buffer to prepare */
+	buf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	if (!priv->csi_direct) {
+		/*
+		 * indirect does not use double-buffering, so this
+		 * buffer is now the active one
+		 */
+		imx_media_dma_buf_set_active(buf);
+	} else {
+		priv->next_out_buf = buf;
+	}
+
+	ipu_cpmem_set_buffer(priv->prpvf_out_ch, priv->ipu_buf_num, buf->phys);
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, priv->ipu_buf_num);
+}
+
+/* prpvf_out_ch EOF interrupt (progressive frame ready) */
+static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
+{
+	struct prpvf_priv *priv = dev_id;
+	struct imx_media_dma_buf *done;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	if (priv->csi_direct) {
+		/* inform CSI of this EOF so it can monitor frame intervals */
+		/* FIXME: frames are coming in twice as fast in direct path! */
+		v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+				 0, NULL);
+	}
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	if (!priv->csi_direct) {
+		/* we're done with the input buffer, queue it back */
+		imx_media_dma_buf_queue(priv->in_ring,
+					priv->curr_in_buf->index);
+
+		/* current input buffer is now last */
+		priv->last_in_buf = priv->curr_in_buf;
+	} else {
+		/*
+		 * priv->next buffer is now the active one due
+		 * to IPU double-buffering
+		 */
+		imx_media_dma_buf_set_active(priv->next_out_buf);
+	}
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (priv->csi_direct) {
+		prepare_prpvf_out_buffer(priv);
+		/* toggle IPU double-buffer index */
+		priv->ipu_buf_num ^= 1;
+	}
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static long prpvf_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct imx_media_dma_buf_ring **ring;
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		spin_lock_irqsave(&priv->irqlock, flags);
+		if (!imx_media_dma_buf_get_active(priv->out_ring)) {
+			buf = imx_media_dma_buf_dequeue(priv->in_ring);
+			if (buf) {
+				prepare_vdi_in_buffers(priv, buf);
+				prepare_prpvf_out_buffer(priv);
+			}
+		}
+		spin_unlock_irqrestore(&priv->irqlock, flags);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static irqreturn_t nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct prpvf_priv *priv = dev_id;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void prpvf_eof_timeout(unsigned long data)
+{
+	struct prpvf_priv *priv = (struct prpvf_priv *)data;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void setup_vdi_channel(struct prpvf_priv *priv,
+			      struct ipuv3_channel *channel,
+			      dma_addr_t phys0, dma_addr_t phys1,
+			      bool out_chan)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	unsigned int burst_size;
+	struct ipu_image image;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (out_chan) {
+		imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+	} else {
+		/* one field to VDIC channels */
+		infmt->height /= 2;
+		imx_media_mbus_fmt_to_ipu_image(&image, infmt);
+		infmt->height *= 2;
+	}
+	image.phys0 = phys0;
+	image.phys1 = phys1;
+
+	ipu_cpmem_zero(channel);
+	ipu_cpmem_set_image(channel, &image);
+
+	if (out_chan) {
+		burst_size = (outfmt->width & 0xf) ? 8 : 16;
+		ipu_cpmem_set_burstsize(channel, burst_size);
+		ipu_ic_task_idma_init(priv->ic_vf, channel,
+				      outfmt->width, outfmt->height,
+				      burst_size, IPU_ROTATE_NONE);
+	} else {
+		burst_size = (infmt->width & 0xf) ? 8 : 16;
+		ipu_cpmem_set_burstsize(channel, burst_size);
+	}
+
+	ipu_cpmem_set_axi_id(channel, 1);
+
+	ipu_idmac_set_double_buffer(channel, priv->csi_direct && out_chan);
+}
+
+static int prpvf_setup_direct(struct prpvf_priv *priv)
+{
+	struct imx_media_dma_buf *buf0, *buf1;
+
+	/* set VDIC to receive from CSI for direct path */
+	ipu_fsu_link(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+		     IPUV3_CHANNEL_CSI_VDI_PREV);
+
+	priv->ipu_buf_num = 0;
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next_out_buf = buf1;
+
+	/* init the prpvf out channel */
+	setup_vdi_channel(priv, priv->prpvf_out_ch,
+			  buf0->phys, buf1->phys, true);
+
+	return 0;
+}
+
+static void prpvf_start_direct(struct prpvf_priv *priv)
+{
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, 0);
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_stop_direct(struct prpvf_priv *priv)
+{
+	ipu_idmac_disable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_disable_direct(struct prpvf_priv *priv)
+{
+	ipu_fsu_unlink(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+		       IPUV3_CHANNEL_CSI_VDI_PREV);
+}
+
+static int prpvf_setup_indirect(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt;
+	const struct imx_media_pixfmt *incc;
+	int in_size, i, ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	incc = priv->cc[priv->input_pad];
+
+	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+	/* 1/2 full image size */
+	priv->field_size = in_size / 2;
+	priv->in_stride = incc->planar ?
+		infmt->width : (infmt->width * incc->bpp) >> 3;
+
+	priv->ipu_buf_num = 0;
+
+	if (priv->in_ring) {
+		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+	}
+
+	priv->in_ring = imx_media_alloc_dma_buf_ring(
+		priv->md, &priv->src_sd->entity,
+		&ic_priv->sd.entity,
+		in_size, IMX_MEDIA_MIN_RING_BUFS_PRPVF, true);
+	if (IS_ERR(priv->in_ring)) {
+		v4l2_err(&ic_priv->sd, "failed to alloc dma-buf ring\n");
+		ret = PTR_ERR(priv->in_ring);
+		priv->in_ring = NULL;
+		return ret;
+	}
+
+	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS_PRPVF; i++)
+		imx_media_dma_buf_queue(priv->in_ring, i);
+
+	priv->last_in_buf = NULL;
+	priv->curr_in_buf = NULL;
+
+	/* translate V4L2_FIELD_ALTERNATE to SEQ_TB or SEQ_BT */
+	priv->fieldtype = infmt->field;
+	if (infmt->field == V4L2_FIELD_ALTERNATE)
+		priv->fieldtype = (priv->std & V4L2_STD_525_60) ?
+			V4L2_FIELD_SEQ_TB : V4L2_FIELD_SEQ_BT;
+
+	/* init the vdi-in channels */
+	setup_vdi_channel(priv, priv->vdi_in_ch_p, 0, 0, false);
+	setup_vdi_channel(priv, priv->vdi_in_ch, 0, 0, false);
+	setup_vdi_channel(priv, priv->vdi_in_ch_n, 0, 0, false);
+
+	/* init the prpvf out channel */
+	setup_vdi_channel(priv, priv->prpvf_out_ch, 0, 0, true);
+
+	return 0;
+}
+
+static void prpvf_start_indirect(struct prpvf_priv *priv)
+{
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->prpvf_out_ch);
+	ipu_idmac_enable_channel(priv->vdi_in_ch_p);
+	ipu_idmac_enable_channel(priv->vdi_in_ch);
+	ipu_idmac_enable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_stop_indirect(struct prpvf_priv *priv)
+{
+	/* disable channels */
+	ipu_idmac_disable_channel(priv->prpvf_out_ch);
+	ipu_idmac_disable_channel(priv->vdi_in_ch_p);
+	ipu_idmac_disable_channel(priv->vdi_in_ch);
+	ipu_idmac_disable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_disable_indirect(struct prpvf_priv *priv)
+{
+}
+
+static struct prpvf_pipeline_ops direct_ops = {
+	.setup = prpvf_setup_direct,
+	.start = prpvf_start_direct,
+	.stop = prpvf_stop_direct,
+	.disable = prpvf_disable_direct,
+};
+
+static struct prpvf_pipeline_ops indirect_ops = {
+	.setup = prpvf_setup_indirect,
+	.start = prpvf_start_indirect,
+	.stop = prpvf_stop_indirect,
+	.disable = prpvf_disable_indirect,
+};
+
+static int prpvf_start(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	priv->ops = priv->csi_direct ? &direct_ops : &indirect_ops;
+
+	ret = prpvf_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	/* set IC to receive from VDIC */
+	ipu_set_ic_src_mux(priv->ipu, 0, true);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	/* request EOF irq for prpvf out channel */
+	priv->out_eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->prpvf_out_ch,
+						  IPU_IRQ_EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->out_eof_irq,
+			       prpvf_out_eof_interrupt, 0,
+			       "imx-ic-prpvf-out-eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering out eof irq: %d\n", ret);
+		goto out_put_ipu;
+	}
+
+	/* request NFB4EOF irq */
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->prpvf_out_ch,
+						  IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+			       nfb4eof_interrupt, 0,
+			       "imx-ic-prpvf-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_free_eof_irq;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_std, &priv->std);
+	if (ret)
+		goto out_free_nfb4eof_irq;
+
+	/* init the VDIC */
+	ipu_vdi_setup(priv->vdi, infmt->code,
+		      infmt->width, infmt->height);
+	ipu_vdi_set_field_order(priv->vdi, priv->std, infmt->field);
+	ipu_vdi_set_motion(priv->vdi, priv->motion);
+
+	ret = ipu_ic_task_init(priv->ic_vf,
+			       infmt->width, infmt->height,
+			       outfmt->width, outfmt->height,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	ret = priv->ops->setup(priv);
+	if (ret)
+		goto out_free_nfb4eof_irq;
+
+	ipu_vdi_enable(priv->vdi);
+	ipu_ic_enable(priv->ic_vf);
+
+	priv->ops->start(priv);
+
+	/* enable the IC VF task */
+	ipu_ic_task_enable(priv->ic_vf);
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_free_eof_irq:
+	devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+out_put_ipu:
+	prpvf_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prpvf_stop(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+	ipu_ic_task_disable(priv->ic_vf);
+	priv->ops->stop(priv);
+	ipu_ic_disable(priv->ic_vf);
+	ipu_vdi_disable(priv->vdi);
+	priv->ops->disable(priv);
+
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+	devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+	prpvf_put_ipu_resources(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int prpvf_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct prpvf_priv *priv = container_of(ctrl->handler,
+					       struct prpvf_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_motion_sel motion;
+
+	switch (ctrl->id) {
+	case V4L2_CID_IMX_MOTION:
+		motion = ctrl->val;
+		if (motion != priv->motion) {
+			/* can't change motion control mid-streaming */
+			if (priv->stream_on)
+				return -EBUSY;
+			priv->motion = motion;
+		}
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops prpvf_ctrl_ops = {
+	.s_ctrl = prpvf_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpvf_custom_ctrl[] = {
+	{
+		.ops = &prpvf_ctrl_ops,
+		.id = V4L2_CID_IMX_MOTION,
+		.name = "Motion Compensation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = MOTION_NONE,
+		.min = MOTION_NONE,
+		.max = HIGH_MOTION,
+		.step = 1,
+	},
+};
+
+#define PRPVF_NUM_CONTROLS ARRAY_SIZE(prpvf_custom_ctrl)
+
+static int prpvf_init_controls(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_free(hdlr);
+	v4l2_ctrl_handler_init(hdlr, PRPVF_NUM_CONTROLS);
+
+	for (i = 0; i < PRPVF_NUM_CONTROLS; i++) {
+		c = &prpvf_custom_ctrl[i];
+		v4l2_ctrl_new_custom(hdlr, c, NULL);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		goto out_free;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+	return 0;
+
+out_free:
+	v4l2_ctrl_handler_free(hdlr);
+	return ret;
+}
+
+static int prpvf_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = prpvf_start(priv);
+	else if (!enable && priv->stream_on)
+		prpvf_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int prpvf_enum_mbus_code(struct v4l2_subdev *sd,
+				struct v4l2_subdev_pad_config *cfg,
+				struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	bool allow_planar, allow_rgb;
+
+	if (code->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	allow_planar = (code->pad == priv->output_pad);
+	allow_rgb = allow_planar;
+
+	return imx_media_enum_format(&code->code, code->index,
+				     allow_rgb, allow_planar);
+}
+
+static int prpvf_get_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int prpvf_set_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	bool allow_planar, allow_rgb;
+	u32 code;
+
+	if (sdformat->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+	allow_rgb = allow_planar;
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   allow_rgb, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, false, false);
+		cc = imx_media_find_format(0, code, false, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_IC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_IC);
+		/* IC resizer cannot downsize more than 4:1 */
+		sdformat->format.width = max_t(__u32, sdformat->format.width,
+					       infmt->width / 4);
+		sdformat->format.height = max_t(__u32, sdformat->format.height,
+						infmt->height / 4);
+
+		/* output is always progressive! */
+		sdformat->format.field = V4L2_FIELD_NONE;
+	} else {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_VDIC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_VDIC);
+
+		/* input must be interlaced! Choose alternate if not */
+		if (!V4L2_FIELD_HAS_BOTH(sdformat->format.field))
+			sdformat->format.field = V4L2_FIELD_ALTERNATE;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int prpvf_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpvf_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+	int ret;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	/* this is sink pad */
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+
+		priv->csi_direct = ((priv->src_sd->grp_id &
+				     IMX_MEDIA_GRP_ID_CSI) != 0);
+
+		ret = prpvf_init_controls(priv);
+		if (ret)
+			return ret;
+	} else {
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+		priv->src_sd = NULL;
+	}
+
+	return 0;
+}
+
+static int prpvf_link_validate(struct v4l2_subdev *sd,
+			       struct media_link *link,
+			       struct v4l2_subdev_format *source_fmt,
+			       struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpvf_priv *priv = ic_priv->task_priv;
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct imx_media_subdev *csi;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link,
+						source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	if (!priv->csi_direct) {
+		csi = imx_media_find_pipeline_subdev(
+			priv->md, &ic_priv->sd.entity, IMX_MEDIA_GRP_ID_CSI);
+		if (IS_ERR(csi)) {
+			v4l2_err(&ic_priv->sd, "no CSI attached\n");
+			ret = PTR_ERR(csi);
+			return ret;
+		}
+
+		priv->csi_sd = csi->sd;
+		return 0;
+	}
+
+	priv->csi_sd = priv->src_sd;
+
+	if (priv->motion != HIGH_MOTION) {
+		v4l2_err(&ic_priv->sd,
+			 "direct CSI pipeline requires HIGH_MOTION\n");
+		return -EINVAL;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		int vc_num = 0;
+		/* see NOTE in imx-csi.c */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &ic_priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		/* only virtual channel 0 can be sent to IC */
+		if (vc_num != 0)
+			return -EINVAL;
+	} else {
+		/* only 8-bit pixels can be sent to IC for parallel busses */
+		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int prpvf_registered(struct v4l2_subdev *sd)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PRPVF_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	return media_entity_pads_init(&sd->entity, PRPVF_NUM_PADS, priv->pad);
+}
+
+static struct v4l2_subdev_pad_ops prpvf_pad_ops = {
+	.enum_mbus_code = prpvf_enum_mbus_code,
+	.get_fmt = prpvf_get_fmt,
+	.set_fmt = prpvf_set_fmt,
+	.link_validate = prpvf_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpvf_video_ops = {
+	.s_stream = prpvf_s_stream,
+};
+
+static struct v4l2_subdev_core_ops prpvf_core_ops = {
+	.ioctl = prpvf_ioctl,
+};
+
+static struct media_entity_operations prpvf_entity_ops = {
+	.link_setup = prpvf_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops prpvf_subdev_ops = {
+	.video = &prpvf_video_ops,
+	.pad = &prpvf_pad_ops,
+	.core = &prpvf_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpvf_internal_ops = {
+	.registered = prpvf_registered,
+};
+
+static int prpvf_init(struct imx_ic_priv *ic_priv)
+{
+	struct prpvf_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+
+	spin_lock_init(&priv->irqlock);
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = prpvf_eof_timeout;
+
+	return 0;
+}
+
+static void prpvf_remove(struct imx_ic_priv *ic_priv)
+{
+	struct prpvf_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_prpvf_ops = {
+	.subdev_ops = &prpvf_subdev_ops,
+	.internal_ops = &prpvf_internal_ops,
+	.entity_ops = &prpvf_entity_ops,
+	.init = prpvf_init,
+	.remove = prpvf_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic.h b/drivers/staging/media/imx/imx-ic.h
new file mode 100644
index 0000000..1ab222b
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic.h
@@ -0,0 +1,38 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#ifndef _IMX_IC_H
+#define _IMX_IC_H
+
+#include <media/v4l2-subdev.h>
+
+struct imx_ic_priv {
+	struct device *dev;
+	struct v4l2_subdev sd;
+	int    ipu_id;
+	int    task_id;
+	void   *task_priv;
+};
+
+struct imx_ic_ops {
+	struct v4l2_subdev_ops *subdev_ops;
+	struct v4l2_subdev_internal_ops *internal_ops;
+	struct media_entity_operations *entity_ops;
+
+	int (*init)(struct imx_ic_priv *ic_priv);
+	void (*remove)(struct imx_ic_priv *ic_priv);
+};
+
+extern struct imx_ic_ops imx_ic_prpenc_ops;
+extern struct imx_ic_ops imx_ic_prpvf_ops;
+extern struct imx_ic_ops imx_ic_pp_ops;
+
+#endif
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

This is a set of three media entity subdevice drivers for the i.MX
Image Converter. The i.MX IC module contains three independent
"tasks":

- Pre-processing Encode task: video frames are routed directly from
  the CSI and can be scaled, color-space converted, and rotated.
  Scaled output is limited to 1024x1024 resolution. Output frames
  are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same
  conversions as the pre-process encode task, but in addition can
  be used for hardware motion compensated deinterlacing. Frames can
  come either directly from the CSI or from the SMFC entities (memory
  buffers via IDMAC channels). Scaled output is limited to 1024x1024
  resolution. Output frames can be routed to various sinks including
  the post-processing task entities.

- Post-processing task: same conversions as pre-process encode. However
  this entity sends frames to the i.MX IPU image converter which supports
  image tiling, which allows scaled output up to 4096x4096 resolution.
  Output frames can be routed to the camera interfaces.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile        |    2 +
 drivers/staging/media/imx/imx-ic-common.c |  109 +++
 drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic.h        |   38 +
 6 files changed, 2997 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 3559d7b..d2a962c 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -1,8 +1,10 @@
 imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 	imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
 
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
new file mode 100644
index 0000000..45706ca
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-common.c
@@ -0,0 +1,109 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
+	[IC_TASK_ENCODER]        = &imx_ic_prpenc_ops,
+	[IC_TASK_VIEWFINDER]     = &imx_ic_prpvf_ops,
+	[IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
+};
+
+static int imx_ic_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct imx_ic_priv *priv;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	/* get our ipu_id, grp_id and IC task id */
+	pdata = priv->dev->platform_data;
+	priv->ipu_id = pdata->ipu_id;
+	switch (pdata->grp_id) {
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		priv->task_id = IC_TASK_ENCODER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		priv->task_id = IC_TASK_VIEWFINDER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+		priv->task_id = IC_TASK_POST_PROCESSOR;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
+	priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	priv->sd.grp_id = pdata->grp_id;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+	ret = ic_ops[priv->task_id]->init(priv);
+	if (ret)
+		return ret;
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		ic_ops[priv->task_id]->remove(priv);
+
+	return ret;
+}
+
+static int imx_ic_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
+
+	ic_ops[priv->task_id]->remove(priv);
+
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_ic_ids[] = {
+	{ .name = "imx-ipuv3-ic" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_ic_ids);
+
+static struct platform_driver imx_ic_driver = {
+	.probe = imx_ic_probe,
+	.remove = imx_ic_remove,
+	.id_table = imx_ic_ids,
+	.driver = {
+		.name = "imx-ipuv3-ic",
+	},
+};
+module_platform_driver(imx_ic_driver);
+
+MODULE_DESCRIPTION("i.MX IC subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-ic");
diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
new file mode 100644
index 0000000..1f75616
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-pp.c
@@ -0,0 +1,636 @@
+/*
+ * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-image-convert.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PP_NUM_PADS 2
+
+struct pp_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+	int pp_id;
+
+	struct ipu_soc *ipu;
+	struct ipu_image_convert_ctx *ic_ctx;
+
+	struct media_pad pad[PP_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	/* our dma buffer sink ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	/* the dma buffer ring we send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct ipu_image_convert_run *out_run;
+
+	struct imx_media_dma_buf *inbuf; /* last input buffer */
+
+	bool stream_on;    /* streaming is on */
+	bool stop;         /* streaming is stopping */
+	spinlock_t irqlock;
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PP_NUM_PADS];
+
+	/* motion select control */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+};
+
+static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void pp_convert_complete(struct ipu_image_convert_run *run,
+				void *data)
+{
+	struct pp_priv *priv = data;
+	struct imx_media_dma_buf *done;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, run->status ?
+				       IMX_MEDIA_BUF_STATUS_ERROR :
+				       IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* we're done with the inbuf, queue it back */
+	imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
+
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+}
+
+static void pp_queue_conversion(struct pp_priv *priv,
+				struct imx_media_dma_buf *inbuf)
+{
+	struct ipu_image_convert_run *run;
+	struct imx_media_dma_buf *outbuf;
+
+	/* get next queued buffer and make it active */
+	outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(outbuf);
+	priv->inbuf = inbuf;
+
+	run = &priv->out_run[outbuf->index];
+	run->ctx = priv->ic_ctx;
+	run->in_phys = inbuf->phys;
+	run->out_phys = outbuf->phys;
+	ipu_image_convert_queue(run);
+}
+
+static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_dma_buf_ring **ring;
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		/* src asks for a buffer ring */
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		/* src hands us a new buffer */
+		spin_lock_irqsave(&priv->irqlock, flags);
+		if (!priv->stop &&
+		    !imx_media_dma_buf_get_active(priv->out_ring)) {
+			buf = imx_media_dma_buf_dequeue(priv->in_ring);
+			if (buf)
+				pp_queue_conversion(priv, buf);
+		}
+		spin_unlock_irqrestore(&priv->irqlock, flags);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int pp_start(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct ipu_image image_in, image_out;
+	const struct imx_media_pixfmt *incc;
+	struct v4l2_mbus_framefmt *infmt;
+	int i, in_size, ret;
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		return ret;
+
+	imx_media_mbus_fmt_to_ipu_image(&image_in,
+					&priv->format_mbus[priv->input_pad]);
+	imx_media_mbus_fmt_to_ipu_image(&image_out,
+					&priv->format_mbus[priv->output_pad]);
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+	priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
+						 IC_TASK_POST_PROCESSOR,
+						 &image_in, &image_out,
+						 priv->rot_mode,
+						 pp_convert_complete, priv);
+	if (IS_ERR(priv->ic_ctx))
+		return PTR_ERR(priv->ic_ctx);
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	incc = priv->cc[priv->input_pad];
+	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+	if (priv->in_ring) {
+		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+	}
+
+	priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
+						     &priv->src_sd->entity,
+						     &ic_priv->sd.entity,
+						     in_size,
+						     IMX_MEDIA_MIN_RING_BUFS,
+						     true);
+	if (IS_ERR(priv->in_ring)) {
+		v4l2_err(&ic_priv->sd,
+			 "failed to alloc dma-buf ring\n");
+		ret = PTR_ERR(priv->in_ring);
+		priv->in_ring = NULL;
+		goto out_unprep;
+	}
+
+	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
+		imx_media_dma_buf_queue(priv->in_ring, i);
+
+	priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
+				sizeof(*priv->out_run), GFP_KERNEL);
+	if (!priv->out_run) {
+		v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
+		ret = -ENOMEM;
+		goto out_free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+out_free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+out_unprep:
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	return ret;
+}
+
+static void pp_stop(struct pp_priv *priv)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	kfree(priv->out_run);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that its sink buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int pp_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = pp_start(priv);
+	else if (!enable && priv->stream_on)
+		pp_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int pp_enum_mbus_code(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_mbus_code_enum *code)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 fourcc;
+	int ret;
+
+	if (code->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	ret = ipu_image_convert_enum_format(code->index, &fourcc);
+	if (ret)
+		return ret;
+
+	/* convert returned fourcc to mbus code */
+	cc = imx_media_find_format(fourcc, 0, true, true);
+	if (WARN_ON(!cc))
+		return -EINVAL;
+
+	code->code = cc->codes[0];
+	return 0;
+}
+
+static int pp_get_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int pp_set_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	struct ipu_image test_in, test_out;
+	u32 code;
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
+	} else {
+		imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		if (sdformat->pad == priv->output_pad) {
+			*outfmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
+		} else {
+			*infmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
+		}
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int pp_link_setup(struct media_entity *entity,
+			 const struct media_pad *local,
+			 const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct pp_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+	} else {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+	}
+
+	return 0;
+}
+
+static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct pp_priv *priv = container_of(ctrl->handler,
+					       struct pp_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		struct v4l2_mbus_framefmt *infmt, *outfmt;
+		struct ipu_image test_in, test_out;
+
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		/*
+		 * make sure this rotation will work with current input/output
+		 * formats before setting
+		 */
+		infmt = &priv->format_mbus[priv->input_pad];
+		outfmt = &priv->format_mbus[priv->output_pad];
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+
+		ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);
+		if (ret)
+			return ret;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops pp_ctrl_ops = {
+	.s_ctrl = pp_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config pp_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};
+
+#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
+
+static int pp_init_controls(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
+
+	for (i = 0; i < PP_NUM_CONTROLS; i++) {
+		c = &pp_std_ctrl[i];
+		v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
+				  c->id, c->min, c->max, c->step, c->def);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		v4l2_ctrl_handler_free(hdlr);
+		return ret;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int pp_registered(struct v4l2_subdev *sd)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PP_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	ret = pp_init_controls(priv);
+	if (ret)
+		return ret;
+
+	ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+free_ctrls:
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	return ret;
+}
+
+static struct v4l2_subdev_pad_ops pp_pad_ops = {
+	.enum_mbus_code = pp_enum_mbus_code,
+	.get_fmt = pp_get_fmt,
+	.set_fmt = pp_set_fmt,
+};
+
+static struct v4l2_subdev_video_ops pp_video_ops = {
+	.s_stream = pp_s_stream,
+};
+
+static struct v4l2_subdev_core_ops pp_core_ops = {
+	.ioctl = pp_ioctl,
+};
+
+static struct media_entity_operations pp_entity_ops = {
+	.link_setup = pp_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops pp_subdev_ops = {
+	.video = &pp_video_ops,
+	.pad = &pp_pad_ops,
+	.core = &pp_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops pp_internal_ops = {
+	.registered = pp_registered,
+};
+
+static int pp_init(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+	spin_lock_init(&priv->irqlock);
+
+	/* get our PP id */
+	priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
+
+	return 0;
+}
+
+static void pp_remove(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_pp_ops = {
+	.subdev_ops = &pp_subdev_ops,
+	.internal_ops = &pp_internal_ops,
+	.entity_ops = &pp_entity_ops,
+	.init = pp_init,
+	.remove = pp_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
new file mode 100644
index 0000000..3d85a82
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpenc.c
@@ -0,0 +1,1033 @@
+/*
+ * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of video frames from the CSI, which
+ * are routed directly to the Image Converter preprocess encode task,
+ * for resizing, colorspace conversion, and rotation.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PRPENC_NUM_PADS 2
+
+#define MAX_W_IC   1024
+#define MAX_H_IC   1024
+#define MAX_W_SINK 4096
+#define MAX_H_SINK 4096
+
+struct prpenc_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+
+	/* IPU units we require */
+	struct ipu_soc *ipu;
+	struct ipu_ic *ic_enc;
+
+	struct media_pad pad[PRPENC_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *enc_ch;
+	struct ipuv3_channel *enc_rot_in_ch;
+	struct ipuv3_channel *enc_rot_out_ch;
+
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct imx_media_dma_buf *next;
+
+	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	/* the CSI id at link validate */
+	int csi_id;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
+
+	struct imx_media_dma_buf rot_buf[2];
+
+	/* controls */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+
+	spinlock_t irqlock;
+
+	struct timer_list eof_timeout_timer;
+	int eof_irq;
+	int nfb4eof_irq;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->ic_enc))
+		ipu_ic_put(priv->ic_enc);
+	priv->ic_enc = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_ch))
+		ipu_idmac_put(priv->enc_ch);
+	priv->enc_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
+		ipu_idmac_put(priv->enc_rot_in_ch);
+	priv->enc_rot_in_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
+		ipu_idmac_put(priv->enc_rot_out_ch);
+	priv->enc_rot_out_ch = NULL;
+}
+
+static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+	priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
+	if (IS_ERR(priv->ic_enc)) {
+		v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
+		ret = PTR_ERR(priv->ic_enc);
+		goto out;
+	}
+
+	priv->enc_ch = ipu_idmac_get(priv->ipu,
+				     IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+	if (IS_ERR(priv->enc_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+		ret = PTR_ERR(priv->enc_ch);
+		goto out;
+	}
+
+	priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
+					    IPUV3_CHANNEL_MEM_ROT_ENC);
+	if (IS_ERR(priv->enc_rot_in_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_MEM_ROT_ENC);
+		ret = PTR_ERR(priv->enc_rot_in_ch);
+		goto out;
+	}
+
+	priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
+					     IPUV3_CHANNEL_ROT_ENC_MEM);
+	if (IS_ERR(priv->enc_rot_out_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_ROT_ENC_MEM);
+		ret = PTR_ERR(priv->enc_rot_out_ch);
+		goto out;
+	}
+
+	return 0;
+out:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_media_dma_buf *done, *next;
+	struct ipuv3_channel *channel;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	/* inform CSI of this EOF so it can monitor frame intervals */
+	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+			 0, NULL);
+
+	channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
+		priv->enc_rot_out_ch : priv->enc_ch;
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* priv->next buffer is now the active one */
+	imx_media_dma_buf_set_active(priv->next);
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
+		ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);
+
+	/* get next queued buffer */
+	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+	ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
+	ipu_idmac_select_buffer(channel, priv->ipu_buf_num);
+
+	/* toggle IPU double-buffer index */
+	priv->ipu_buf_num ^= 1;
+	priv->next = next;
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void prpenc_eof_timeout(unsigned long data)
+{
+	struct prpenc_priv *priv = (struct prpenc_priv *)data;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void prpenc_setup_channel(struct prpenc_priv *priv,
+				 struct ipuv3_channel *channel,
+				 enum ipu_rotate_mode rot_mode,
+				 dma_addr_t addr0, dma_addr_t addr1,
+				 bool rot_swap_width_height)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	unsigned int burst_size;
+	struct ipu_image image;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+
+	ipu_cpmem_zero(channel);
+
+	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+	image.phys0 = addr0;
+	image.phys1 = addr1;
+	ipu_cpmem_set_image(channel, &image);
+
+	if (channel == priv->enc_rot_in_ch ||
+	    channel == priv->enc_rot_out_ch) {
+		burst_size = 8;
+		ipu_cpmem_set_block_mode(channel);
+	} else {
+		burst_size = (outfmt->width & 0xf) ? 8 : 16;
+	}
+
+	ipu_cpmem_set_burstsize(channel, burst_size);
+
+	if (rot_mode)
+		ipu_cpmem_set_rotation(channel, rot_mode);
+
+	if (outfmt->field == V4L2_FIELD_NONE &&
+	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+	     infmt->field == V4L2_FIELD_ALTERNATE) &&
+	    channel == priv->enc_ch)
+		ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
+
+	ipu_ic_task_idma_init(priv->ic_enc, channel,
+			      outfmt->width, outfmt->height,
+			      burst_size, rot_mode);
+	ipu_cpmem_set_axi_id(channel, 1);
+
+	ipu_idmac_set_double_buffer(channel, true);
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+}
+
+static int prpenc_setup_rotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int out_size, ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
+
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
+		return ret;
+	}
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
+		goto free_rot0;
+	}
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->height, outfmt->width,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		goto free_rot1;
+	}
+
+	/* init the IC ENC-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch,
+			     IPU_ROTATE_NONE,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	/* init the MEM-->IC ENC ROT IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_in_ch,
+			     priv->rot_mode,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the destination IC ENC ROT-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_out_ch,
+			     IPU_ROTATE_NONE,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	/* now link IC ENC-->MEM to MEM-->IC ENC ROT */
+	ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
+
+	/* enable the IC */
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_out_ch);
+
+	/* and finally enable the IC PRPENC task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+
+free_rot1:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+free_rot0:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	return ret;
+}
+
+static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_out_ch);
+
+	ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
+
+	ipu_ic_disable(priv->ic_enc);
+
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+}
+
+static int prpenc_setup_norotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->width, outfmt->height,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		return ret;
+	}
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the IC PRP-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	ipu_cpmem_dump(priv->enc_ch);
+	ipu_ic_dump(priv->ic_enc);
+	ipu_dump(priv->ipu);
+
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+
+	/* enable the IC ENCODE task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+}
+
+static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_ic_disable(priv->ic_enc);
+}
+
+static int prpenc_start(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = prpenc_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	/* set IC to receive from CSI */
+	ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->ipu_buf_num = 0;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		ret = prpenc_setup_rotation(priv);
+	else
+		ret = prpenc_setup_norotation(priv);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->enc_ch,
+						  IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+			       prpenc_nfb4eof_interrupt, 0,
+			       "imx-ic-prpenc-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_unsetup;
+	}
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
+	else
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
+
+	ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
+			       prpenc_eof_interrupt, 0,
+			       "imx-ic-prpenc-eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering eof irq: %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+out_put_ipu:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prpenc_stop(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp,
+		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+	devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+
+	prpenc_put_ipu_resources(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_pad_config *cfg,
+				 struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	bool allow_planar;
+
+	if (code->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	allow_planar = (code->pad == priv->output_pad);
+
+	return imx_media_enum_format(&code->code, code->index,
+				     true, allow_planar);
+}
+
+static int prpenc_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int prpenc_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	bool allow_planar;
+	u32 code;
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   true, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, false);
+		cc = imx_media_find_format(0, code, true, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_IC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_IC);
+
+		if (sdformat->format.field != V4L2_FIELD_NONE)
+			sdformat->format.field = infmt->field;
+
+		/* IC resizer cannot downsize more than 4:1 */
+		if (ipu_rot_mode_is_irt(priv->rot_mode)) {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->height / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->width / 4);
+		} else {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->width / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->height / 4);
+		}
+	} else {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_SINK);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_SINK);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_setup(struct media_entity *entity,
+			     const struct media_pad *local,
+			     const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	/* this is sink pad */
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+	} else {
+		priv->src_sd = NULL;
+		return 0;
+	}
+
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0:
+		priv->csi_id = 0;
+		break;
+	case IMX_MEDIA_GRP_ID_CSI1:
+		priv->csi_id = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_validate(struct v4l2_subdev *sd,
+				struct media_link *link,
+				struct v4l2_subdev_format *source_fmt,
+				struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link,
+						source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		int vc_num = 0;
+		/* see NOTE in imx-csi.c */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &ic_priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		/* only virtual channel 0 can be sent to IC */
+		if (vc_num != 0)
+			return -EINVAL;
+	} else {
+		/*
+		 * only 8-bit pixels can be sent to IC for parallel
+		 * busses
+		 */
+		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct prpenc_priv *priv = container_of(ctrl->handler,
+					       struct prpenc_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
+	.s_ctrl = prpenc_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};
+
+#define PRPENC_NUM_CONTROLS ARRAY_SIZE(prpenc_std_ctrl)
+
+static int prpenc_init_controls(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, PRPENC_NUM_CONTROLS);
+
+	for (i = 0; i < PRPENC_NUM_CONTROLS; i++) {
+		c = &prpenc_std_ctrl[i];
+		v4l2_ctrl_new_std(hdlr, &prpenc_ctrl_ops,
+				  c->id, c->min, c->max, c->step, c->def);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		goto out_free;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+	return 0;
+
+out_free:
+	v4l2_ctrl_handler_free(hdlr);
+	return ret;
+}
+
+static int prpenc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = prpenc_start(priv);
+	else if (!enable && priv->stream_on)
+		prpenc_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int prpenc_registered(struct v4l2_subdev *sd)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PRPENC_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	ret = prpenc_init_controls(priv);
+	if (ret)
+		return ret;
+
+	ret = media_entity_pads_init(&sd->entity, PRPENC_NUM_PADS, priv->pad);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static struct v4l2_subdev_pad_ops prpenc_pad_ops = {
+	.enum_mbus_code = prpenc_enum_mbus_code,
+	.get_fmt = prpenc_get_fmt,
+	.set_fmt = prpenc_set_fmt,
+	.link_validate = prpenc_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpenc_video_ops = {
+	.s_stream = prpenc_s_stream,
+};
+
+static struct media_entity_operations prpenc_entity_ops = {
+	.link_setup = prpenc_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops prpenc_subdev_ops = {
+	.video = &prpenc_video_ops,
+	.pad = &prpenc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpenc_internal_ops = {
+	.registered = prpenc_registered,
+};
+
+static int prpenc_init(struct imx_ic_priv *ic_priv)
+{
+	struct prpenc_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+
+	spin_lock_init(&priv->irqlock);
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = prpenc_eof_timeout;
+
+	return 0;
+}
+
+static void prpenc_remove(struct imx_ic_priv *ic_priv)
+{
+	struct prpenc_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_prpenc_ops = {
+	.subdev_ops = &prpenc_subdev_ops,
+	.internal_ops = &prpenc_internal_ops,
+	.entity_ops = &prpenc_entity_ops,
+	.init = prpenc_init,
+	.remove = prpenc_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpvf.c b/drivers/staging/media/imx/imx-ic-prpvf.c
new file mode 100644
index 0000000..dc2d402
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpvf.c
@@ -0,0 +1,1179 @@
+/*
+ * V4L2 IC Deinterlacer Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+/*
+ * This subdev implements two different video pipelines:
+ *
+ * CSI -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends a single interlaced field F(n-1)
+ * directly to the VDIC (and optionally the following field F(n)
+ * can be sent to memory via IDMAC channel 13). So only two fields
+ * can be processed by the VDIC. This pipeline only works in VDIC's
+ * high motion mode, which only requires a single field for processing.
+ * The other motion modes (low and medium) require three fields, so this
+ * pipeline does not work in those modes. Also, it is not clear how this
+ * pipeline can deal with the various field orders (sequential BT/TB,
+ * interlaced BT/TB).
+ *
+ * CSI -> CH[0-3] -> MEM -> CH8,9,10 -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends raw and full frames to memory buffers
+ * via the SMFC channels 0-3. Fields from these frames are then
+ * transferred to the VDIC via IDMAC channels 8,9,10. The VDIC requires
+ * three fields: previous field F(n-1), current field F(n), and next
+ * field F(n+1), so we need three raw frames in memory: two completed frames
+ * to send F(n-1), F(n), F(n+1) to the VDIC, and a third frame for active
+ * CSI capture while the completed fields are sent through the VDIC->IC for
+ * processing.
+ *
+ * The "direct" CSI->VDIC pipeline requires less memory bus bandwidth
+ * (just 1 channel vs. 5 channels for indirect pipeline), but it can
+ * only be used in high motion mode, and it only processes a single
+ * field (so half the original image resolution is lost).
+ */
+
+struct prpvf_priv;
+
+struct prpvf_pipeline_ops {
+	int (*setup)(struct prpvf_priv *priv);
+	void (*start)(struct prpvf_priv *priv);
+	void (*stop)(struct prpvf_priv *priv);
+	void (*disable)(struct prpvf_priv *priv);
+};
+
+#define PRPVF_NUM_PADS 2
+
+#define MAX_W_IC   1024
+#define MAX_H_IC   1024
+#define MAX_W_VDIC  968
+#define MAX_H_VDIC 2048
+
+struct prpvf_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+
+	/* IPU units we require */
+	struct ipu_soc *ipu;
+	struct ipu_ic *ic_vf;
+	struct ipu_vdi *vdi;
+
+	struct media_pad pad[PRPVF_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *vdi_in_ch_p; /* F(n-1) transfer channel */
+	struct ipuv3_channel *vdi_in_ch;   /* F(n) transfer channel */
+	struct ipuv3_channel *vdi_in_ch_n; /* F(n+1) transfer channel */
+	struct ipuv3_channel *prpvf_out_ch;/* final progressive frame channel */
+
+	/* pipeline operations */
+	struct prpvf_pipeline_ops *ops;
+
+	/* our dma buffer sink ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+
+	/* ipu buf num for double-buffering (csi-direct path only) */
+	int ipu_buf_num;
+	struct imx_media_dma_buf *next_out_buf;
+
+	/* current and last input buffers indirect path */
+	struct imx_media_dma_buf *curr_in_buf;
+	struct imx_media_dma_buf *last_in_buf;
+
+	/*
+	 * translated field type, input line stride, and field size
+	 * for indirect path
+	 */
+	u32 fieldtype;
+	u32 in_stride;
+	u32 field_size;
+
+	struct v4l2_subdev *src_sd;
+	/* the sink that will receive the progressive out buffers */
+	struct v4l2_subdev *sink_sd;
+
+	/* the attached CSI at stream on */
+	struct v4l2_subdev *csi_sd;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	/* the video standard from sensor at time of streamon */
+	v4l2_std_id std;
+
+	struct v4l2_mbus_framefmt format_mbus[PRPVF_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PRPVF_NUM_PADS];
+
+	bool csi_direct;  /* using direct CSI->VDIC->IC pipeline */
+
+	/* motion select control */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	enum ipu_motion_sel motion;
+
+	struct timer_list eof_timeout_timer;
+
+	int nfb4eof_irq; /* CSI or PRPVF channel NFB4EOF IRQ */
+	int out_eof_irq; /* PRPVF channel EOF IRQ */
+	spinlock_t irqlock;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static inline struct prpvf_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void prpvf_put_ipu_resources(struct prpvf_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->ic_vf))
+		ipu_ic_put(priv->ic_vf);
+	priv->ic_vf = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch_p))
+		ipu_idmac_put(priv->vdi_in_ch_p);
+	priv->vdi_in_ch_p = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch))
+		ipu_idmac_put(priv->vdi_in_ch);
+	priv->vdi_in_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch_n))
+		ipu_idmac_put(priv->vdi_in_ch_n);
+	priv->vdi_in_ch_n = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->prpvf_out_ch))
+		ipu_idmac_put(priv->prpvf_out_ch);
+	priv->prpvf_out_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi))
+		ipu_vdi_put(priv->vdi);
+	priv->vdi = NULL;
+}
+
+static int prpvf_get_ipu_resources(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret, err_chan;
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+	priv->ic_vf = ipu_ic_get(priv->ipu, IC_TASK_VIEWFINDER);
+	if (IS_ERR(priv->ic_vf)) {
+		v4l2_err(&ic_priv->sd, "failed to get IC VF\n");
+		ret = PTR_ERR(priv->ic_vf);
+		goto out;
+	}
+
+	priv->vdi = ipu_vdi_get(priv->ipu);
+	if (IS_ERR(priv->vdi)) {
+		v4l2_err(&ic_priv->sd, "failed to get VDIC\n");
+		ret = PTR_ERR(priv->vdi);
+		goto out;
+	}
+
+	priv->prpvf_out_ch = ipu_idmac_get(priv->ipu,
+					   IPUV3_CHANNEL_IC_PRP_VF_MEM);
+	if (IS_ERR(priv->prpvf_out_ch)) {
+		err_chan = IPUV3_CHANNEL_IC_PRP_VF_MEM;
+		ret = PTR_ERR(priv->prpvf_out_ch);
+		goto out_err_chan;
+	}
+
+	if (!priv->csi_direct) {
+		priv->vdi_in_ch_p = ipu_idmac_get(priv->ipu,
+						  IPUV3_CHANNEL_MEM_VDI_PREV);
+		if (IS_ERR(priv->vdi_in_ch_p)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_PREV;
+			ret = PTR_ERR(priv->vdi_in_ch_p);
+			goto out_err_chan;
+		}
+
+		priv->vdi_in_ch = ipu_idmac_get(priv->ipu,
+						IPUV3_CHANNEL_MEM_VDI_CUR);
+		if (IS_ERR(priv->vdi_in_ch)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_CUR;
+			ret = PTR_ERR(priv->vdi_in_ch);
+			goto out_err_chan;
+		}
+
+		priv->vdi_in_ch_n = ipu_idmac_get(priv->ipu,
+						  IPUV3_CHANNEL_MEM_VDI_NEXT);
+		if (IS_ERR(priv->vdi_in_ch_n)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_NEXT;
+			ret = PTR_ERR(priv->vdi_in_ch_n);
+			goto out_err_chan;
+		}
+	}
+
+	return 0;
+
+out_err_chan:
+	v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n", err_chan);
+out:
+	prpvf_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prepare_vdi_in_buffers(struct prpvf_priv *priv,
+				   struct imx_media_dma_buf *curr)
+{
+	dma_addr_t prev_phys, curr_phys, next_phys;
+	struct imx_media_dma_buf *last;
+
+	last = priv->last_in_buf ? priv->last_in_buf : curr;
+	priv->curr_in_buf = curr;
+
+	switch (priv->fieldtype) {
+	case V4L2_FIELD_SEQ_TB:
+		prev_phys = last->phys;
+		curr_phys = curr->phys + priv->field_size;
+		next_phys = curr->phys;
+		break;
+	case V4L2_FIELD_SEQ_BT:
+		prev_phys = last->phys + priv->field_size;
+		curr_phys = curr->phys;
+		next_phys = curr->phys + priv->field_size;
+		break;
+	case V4L2_FIELD_INTERLACED_BT:
+		prev_phys = last->phys + priv->in_stride;
+		curr_phys = curr->phys;
+		next_phys = curr->phys + priv->in_stride;
+		break;
+	default:
+		/* assume V4L2_FIELD_INTERLACED_TB */
+		prev_phys = last->phys;
+		curr_phys = curr->phys + priv->in_stride;
+		next_phys = curr->phys;
+		break;
+	}
+
+	ipu_cpmem_set_buffer(priv->vdi_in_ch_p, 0, prev_phys);
+	ipu_cpmem_set_buffer(priv->vdi_in_ch,   0, curr_phys);
+	ipu_cpmem_set_buffer(priv->vdi_in_ch_n, 0, next_phys);
+
+	ipu_idmac_select_buffer(priv->vdi_in_ch_p, 0);
+	ipu_idmac_select_buffer(priv->vdi_in_ch, 0);
+	ipu_idmac_select_buffer(priv->vdi_in_ch_n, 0);
+}
+
+static void prepare_prpvf_out_buffer(struct prpvf_priv *priv)
+{
+	struct imx_media_dma_buf *buf;
+
+	/* get next buffer to prepare */
+	buf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	if (!priv->csi_direct) {
+		/*
+		 * indirect does not use double-buffering, so this
+		 * buffer is now the active one
+		 */
+		imx_media_dma_buf_set_active(buf);
+	} else {
+		priv->next_out_buf = buf;
+	}
+
+	ipu_cpmem_set_buffer(priv->prpvf_out_ch, priv->ipu_buf_num, buf->phys);
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, priv->ipu_buf_num);
+}
+
+/* prpvf_out_ch EOF interrupt (progressive frame ready) */
+static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
+{
+	struct prpvf_priv *priv = dev_id;
+	struct imx_media_dma_buf *done;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	if (priv->csi_direct) {
+		/* inform CSI of this EOF so it can monitor frame intervals */
+		/* FIXME: frames are coming in twice as fast in direct path! */
+		v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+				 0, NULL);
+	}
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	if (!priv->csi_direct) {
+		/* we're done with the input buffer, queue it back */
+		imx_media_dma_buf_queue(priv->in_ring,
+					priv->curr_in_buf->index);
+
+		/* current input buffer is now last */
+		priv->last_in_buf = priv->curr_in_buf;
+	} else {
+		/*
+		 * priv->next buffer is now the active one due
+		 * to IPU double-buffering
+		 */
+		imx_media_dma_buf_set_active(priv->next_out_buf);
+	}
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (priv->csi_direct) {
+		prepare_prpvf_out_buffer(priv);
+		/* toggle IPU double-buffer index */
+		priv->ipu_buf_num ^= 1;
+	}
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static long prpvf_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct imx_media_dma_buf_ring **ring;
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		spin_lock_irqsave(&priv->irqlock, flags);
+		if (!imx_media_dma_buf_get_active(priv->out_ring)) {
+			buf = imx_media_dma_buf_dequeue(priv->in_ring);
+			if (buf) {
+				prepare_vdi_in_buffers(priv, buf);
+				prepare_prpvf_out_buffer(priv);
+			}
+		}
+		spin_unlock_irqrestore(&priv->irqlock, flags);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static irqreturn_t nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct prpvf_priv *priv = dev_id;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void prpvf_eof_timeout(unsigned long data)
+{
+	struct prpvf_priv *priv = (struct prpvf_priv *)data;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void setup_vdi_channel(struct prpvf_priv *priv,
+			      struct ipuv3_channel *channel,
+			      dma_addr_t phys0, dma_addr_t phys1,
+			      bool out_chan)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	unsigned int burst_size;
+	struct ipu_image image;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (out_chan) {
+		imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+	} else {
+		/* one field to VDIC channels */
+		infmt->height /= 2;
+		imx_media_mbus_fmt_to_ipu_image(&image, infmt);
+		infmt->height *= 2;
+	}
+	image.phys0 = phys0;
+	image.phys1 = phys1;
+
+	ipu_cpmem_zero(channel);
+	ipu_cpmem_set_image(channel, &image);
+
+	if (out_chan) {
+		burst_size = (outfmt->width & 0xf) ? 8 : 16;
+		ipu_cpmem_set_burstsize(channel, burst_size);
+		ipu_ic_task_idma_init(priv->ic_vf, channel,
+				      outfmt->width, outfmt->height,
+				      burst_size, IPU_ROTATE_NONE);
+	} else {
+		burst_size = (infmt->width & 0xf) ? 8 : 16;
+		ipu_cpmem_set_burstsize(channel, burst_size);
+	}
+
+	ipu_cpmem_set_axi_id(channel, 1);
+
+	ipu_idmac_set_double_buffer(channel, priv->csi_direct && out_chan);
+}
+
+static int prpvf_setup_direct(struct prpvf_priv *priv)
+{
+	struct imx_media_dma_buf *buf0, *buf1;
+
+	/* set VDIC to receive from CSI for direct path */
+	ipu_fsu_link(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+		     IPUV3_CHANNEL_CSI_VDI_PREV);
+
+	priv->ipu_buf_num = 0;
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next_out_buf = buf1;
+
+	/* init the prpvf out channel */
+	setup_vdi_channel(priv, priv->prpvf_out_ch,
+			  buf0->phys, buf1->phys, true);
+
+	return 0;
+}
+
+static void prpvf_start_direct(struct prpvf_priv *priv)
+{
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, 0);
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_stop_direct(struct prpvf_priv *priv)
+{
+	ipu_idmac_disable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_disable_direct(struct prpvf_priv *priv)
+{
+	ipu_fsu_unlink(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+		       IPUV3_CHANNEL_CSI_VDI_PREV);
+}
+
+static int prpvf_setup_indirect(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt;
+	const struct imx_media_pixfmt *incc;
+	int in_size, i, ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	incc = priv->cc[priv->input_pad];
+
+	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+	/* 1/2 full image size */
+	priv->field_size = in_size / 2;
+	priv->in_stride = incc->planar ?
+		infmt->width : (infmt->width * incc->bpp) >> 3;
+
+	priv->ipu_buf_num = 0;
+
+	if (priv->in_ring) {
+		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+	}
+
+	priv->in_ring = imx_media_alloc_dma_buf_ring(
+		priv->md, &priv->src_sd->entity,
+		&ic_priv->sd.entity,
+		in_size, IMX_MEDIA_MIN_RING_BUFS_PRPVF, true);
+	if (IS_ERR(priv->in_ring)) {
+		v4l2_err(&ic_priv->sd, "failed to alloc dma-buf ring\n");
+		ret = PTR_ERR(priv->in_ring);
+		priv->in_ring = NULL;
+		return ret;
+	}
+
+	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS_PRPVF; i++)
+		imx_media_dma_buf_queue(priv->in_ring, i);
+
+	priv->last_in_buf = NULL;
+	priv->curr_in_buf = NULL;
+
+	/* translate V4L2_FIELD_ALTERNATE to SEQ_TB or SEQ_BT */
+	priv->fieldtype = infmt->field;
+	if (infmt->field == V4L2_FIELD_ALTERNATE)
+		priv->fieldtype = (priv->std & V4L2_STD_525_60) ?
+			V4L2_FIELD_SEQ_TB : V4L2_FIELD_SEQ_BT;
+
+	/* init the vdi-in channels */
+	setup_vdi_channel(priv, priv->vdi_in_ch_p, 0, 0, false);
+	setup_vdi_channel(priv, priv->vdi_in_ch, 0, 0, false);
+	setup_vdi_channel(priv, priv->vdi_in_ch_n, 0, 0, false);
+
+	/* init the prpvf out channel */
+	setup_vdi_channel(priv, priv->prpvf_out_ch, 0, 0, true);
+
+	return 0;
+}
+
+static void prpvf_start_indirect(struct prpvf_priv *priv)
+{
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->prpvf_out_ch);
+	ipu_idmac_enable_channel(priv->vdi_in_ch_p);
+	ipu_idmac_enable_channel(priv->vdi_in_ch);
+	ipu_idmac_enable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_stop_indirect(struct prpvf_priv *priv)
+{
+	/* disable channels */
+	ipu_idmac_disable_channel(priv->prpvf_out_ch);
+	ipu_idmac_disable_channel(priv->vdi_in_ch_p);
+	ipu_idmac_disable_channel(priv->vdi_in_ch);
+	ipu_idmac_disable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_disable_indirect(struct prpvf_priv *priv)
+{
+}
+
+static struct prpvf_pipeline_ops direct_ops = {
+	.setup = prpvf_setup_direct,
+	.start = prpvf_start_direct,
+	.stop = prpvf_stop_direct,
+	.disable = prpvf_disable_direct,
+};
+
+static struct prpvf_pipeline_ops indirect_ops = {
+	.setup = prpvf_setup_indirect,
+	.start = prpvf_start_indirect,
+	.stop = prpvf_stop_indirect,
+	.disable = prpvf_disable_indirect,
+};
+
+static int prpvf_start(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	priv->ops = priv->csi_direct ? &direct_ops : &indirect_ops;
+
+	ret = prpvf_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	/* set IC to receive from VDIC */
+	ipu_set_ic_src_mux(priv->ipu, 0, true);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	/* request EOF irq for prpvf out channel */
+	priv->out_eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->prpvf_out_ch,
+						  IPU_IRQ_EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->out_eof_irq,
+			       prpvf_out_eof_interrupt, 0,
+			       "imx-ic-prpvf-out-eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering out eof irq: %d\n", ret);
+		goto out_put_ipu;
+	}
+
+	/* request NFB4EOF irq */
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->prpvf_out_ch,
+						  IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+			       nfb4eof_interrupt, 0,
+			       "imx-ic-prpvf-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_free_eof_irq;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_std, &priv->std);
+	if (ret)
+		goto out_free_nfb4eof_irq;
+
+	/* init the VDIC */
+	ipu_vdi_setup(priv->vdi, infmt->code,
+		      infmt->width, infmt->height);
+	ipu_vdi_set_field_order(priv->vdi, priv->std, infmt->field);
+	ipu_vdi_set_motion(priv->vdi, priv->motion);
+
+	ret = ipu_ic_task_init(priv->ic_vf,
+			       infmt->width, infmt->height,
+			       outfmt->width, outfmt->height,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	ret = priv->ops->setup(priv);
+	if (ret)
+		goto out_free_nfb4eof_irq;
+
+	ipu_vdi_enable(priv->vdi);
+	ipu_ic_enable(priv->ic_vf);
+
+	priv->ops->start(priv);
+
+	/* enable the IC VF task */
+	ipu_ic_task_enable(priv->ic_vf);
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_free_eof_irq:
+	devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+out_put_ipu:
+	prpvf_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prpvf_stop(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+	ipu_ic_task_disable(priv->ic_vf);
+	priv->ops->stop(priv);
+	ipu_ic_disable(priv->ic_vf);
+	ipu_vdi_disable(priv->vdi);
+	priv->ops->disable(priv);
+
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+	devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+	prpvf_put_ipu_resources(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int prpvf_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct prpvf_priv *priv = container_of(ctrl->handler,
+					       struct prpvf_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_motion_sel motion;
+
+	switch (ctrl->id) {
+	case V4L2_CID_IMX_MOTION:
+		motion = ctrl->val;
+		if (motion != priv->motion) {
+			/* can't change motion control mid-streaming */
+			if (priv->stream_on)
+				return -EBUSY;
+			priv->motion = motion;
+		}
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops prpvf_ctrl_ops = {
+	.s_ctrl = prpvf_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpvf_custom_ctrl[] = {
+	{
+		.ops = &prpvf_ctrl_ops,
+		.id = V4L2_CID_IMX_MOTION,
+		.name = "Motion Compensation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = MOTION_NONE,
+		.min = MOTION_NONE,
+		.max = HIGH_MOTION,
+		.step = 1,
+	},
+};
+
+#define PRPVF_NUM_CONTROLS ARRAY_SIZE(prpvf_custom_ctrl)
+
+static int prpvf_init_controls(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_free(hdlr);
+	v4l2_ctrl_handler_init(hdlr, PRPVF_NUM_CONTROLS);
+
+	for (i = 0; i < PRPVF_NUM_CONTROLS; i++) {
+		c = &prpvf_custom_ctrl[i];
+		v4l2_ctrl_new_custom(hdlr, c, NULL);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		goto out_free;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+	return 0;
+
+out_free:
+	v4l2_ctrl_handler_free(hdlr);
+	return ret;
+}
+
+static int prpvf_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = prpvf_start(priv);
+	else if (!enable && priv->stream_on)
+		prpvf_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int prpvf_enum_mbus_code(struct v4l2_subdev *sd,
+				struct v4l2_subdev_pad_config *cfg,
+				struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	bool allow_planar, allow_rgb;
+
+	if (code->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	allow_planar = (code->pad == priv->output_pad);
+	allow_rgb = allow_planar;
+
+	return imx_media_enum_format(&code->code, code->index,
+				     allow_rgb, allow_planar);
+}
+
+static int prpvf_get_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int prpvf_set_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	bool allow_planar, allow_rgb;
+	u32 code;
+
+	if (sdformat->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+	allow_rgb = allow_planar;
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   allow_rgb, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, false, false);
+		cc = imx_media_find_format(0, code, false, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_IC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_IC);
+		/* IC resizer cannot downsize more than 4:1 */
+		sdformat->format.width = max_t(__u32, sdformat->format.width,
+					       infmt->width / 4);
+		sdformat->format.height = max_t(__u32, sdformat->format.height,
+						infmt->height / 4);
+
+		/* output is always progressive! */
+		sdformat->format.field = V4L2_FIELD_NONE;
+	} else {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_VDIC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_VDIC);
+
+		/* input must be interlaced! Choose alternate if not */
+		if (!V4L2_FIELD_HAS_BOTH(sdformat->format.field))
+			sdformat->format.field = V4L2_FIELD_ALTERNATE;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int prpvf_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpvf_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+	int ret;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	/* this is sink pad */
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+
+		priv->csi_direct = ((priv->src_sd->grp_id &
+				     IMX_MEDIA_GRP_ID_CSI) != 0);
+
+		ret = prpvf_init_controls(priv);
+		if (ret)
+			return ret;
+	} else {
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+		priv->src_sd = NULL;
+	}
+
+	return 0;
+}
+
+static int prpvf_link_validate(struct v4l2_subdev *sd,
+			       struct media_link *link,
+			       struct v4l2_subdev_format *source_fmt,
+			       struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpvf_priv *priv = ic_priv->task_priv;
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct imx_media_subdev *csi;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link,
+						source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	if (!priv->csi_direct) {
+		csi = imx_media_find_pipeline_subdev(
+			priv->md, &ic_priv->sd.entity, IMX_MEDIA_GRP_ID_CSI);
+		if (IS_ERR(csi)) {
+			v4l2_err(&ic_priv->sd, "no CSI attached\n");
+			ret = PTR_ERR(csi);
+			return ret;
+		}
+
+		priv->csi_sd = csi->sd;
+		return 0;
+	}
+
+	priv->csi_sd = priv->src_sd;
+
+	if (priv->motion != HIGH_MOTION) {
+		v4l2_err(&ic_priv->sd,
+			 "direct CSI pipeline requires HIGH_MOTION\n");
+		return -EINVAL;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		int vc_num = 0;
+		/* see NOTE in imx-csi.c */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &ic_priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		/* only virtual channel 0 can be sent to IC */
+		if (vc_num != 0)
+			return -EINVAL;
+	} else {
+		/* only 8-bit pixels can be sent to IC for parallel busses */
+		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int prpvf_registered(struct v4l2_subdev *sd)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PRPVF_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	return media_entity_pads_init(&sd->entity, PRPVF_NUM_PADS, priv->pad);
+}
+
+static struct v4l2_subdev_pad_ops prpvf_pad_ops = {
+	.enum_mbus_code = prpvf_enum_mbus_code,
+	.get_fmt = prpvf_get_fmt,
+	.set_fmt = prpvf_set_fmt,
+	.link_validate = prpvf_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpvf_video_ops = {
+	.s_stream = prpvf_s_stream,
+};
+
+static struct v4l2_subdev_core_ops prpvf_core_ops = {
+	.ioctl = prpvf_ioctl,
+};
+
+static struct media_entity_operations prpvf_entity_ops = {
+	.link_setup = prpvf_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops prpvf_subdev_ops = {
+	.video = &prpvf_video_ops,
+	.pad = &prpvf_pad_ops,
+	.core = &prpvf_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpvf_internal_ops = {
+	.registered = prpvf_registered,
+};
+
+static int prpvf_init(struct imx_ic_priv *ic_priv)
+{
+	struct prpvf_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+
+	spin_lock_init(&priv->irqlock);
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = prpvf_eof_timeout;
+
+	return 0;
+}
+
+static void prpvf_remove(struct imx_ic_priv *ic_priv)
+{
+	struct prpvf_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_prpvf_ops = {
+	.subdev_ops = &prpvf_subdev_ops,
+	.internal_ops = &prpvf_internal_ops,
+	.entity_ops = &prpvf_entity_ops,
+	.init = prpvf_init,
+	.remove = prpvf_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic.h b/drivers/staging/media/imx/imx-ic.h
new file mode 100644
index 0000000..1ab222b
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic.h
@@ -0,0 +1,38 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#ifndef _IMX_IC_H
+#define _IMX_IC_H
+
+#include <media/v4l2-subdev.h>
+
+struct imx_ic_priv {
+	struct device *dev;
+	struct v4l2_subdev sd;
+	int    ipu_id;
+	int    task_id;
+	void   *task_priv;
+};
+
+struct imx_ic_ops {
+	struct v4l2_subdev_ops *subdev_ops;
+	struct v4l2_subdev_internal_ops *internal_ops;
+	struct media_entity_operations *entity_ops;
+
+	int (*init)(struct imx_ic_priv *ic_priv);
+	void (*remove)(struct imx_ic_priv *ic_priv);
+};
+
+extern struct imx_ic_ops imx_ic_prpenc_ops;
+extern struct imx_ic_ops imx_ic_prpvf_ops;
+extern struct imx_ic_ops imx_ic_pp_ops;
+
+#endif
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

This is a set of three media entity subdevice drivers for the i.MX
Image Converter. The i.MX IC module contains three independent
"tasks":

- Pre-processing Encode task: video frames are routed directly from
  the CSI and can be scaled, color-space converted, and rotated.
  Scaled output is limited to 1024x1024 resolution. Output frames
  are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same
  conversions as the pre-process encode task, but in addition can
  be used for hardware motion compensated deinterlacing. Frames can
  come either directly from the CSI or from the SMFC entities (memory
  buffers via IDMAC channels). Scaled output is limited to 1024x1024
  resolution. Output frames can be routed to various sinks including
  the post-processing task entities.

- Post-processing task: same conversions as pre-process encode. However
  this entity sends frames to the i.MX IPU image converter which supports
  image tiling, which allows scaled output up to 4096x4096 resolution.
  Output frames can be routed to the camera interfaces.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile        |    2 +
 drivers/staging/media/imx/imx-ic-common.c |  109 +++
 drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic.h        |   38 +
 6 files changed, 2997 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 3559d7b..d2a962c 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -1,8 +1,10 @@
 imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 	imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
 
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
new file mode 100644
index 0000000..45706ca
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-common.c
@@ -0,0 +1,109 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
+	[IC_TASK_ENCODER]        = &imx_ic_prpenc_ops,
+	[IC_TASK_VIEWFINDER]     = &imx_ic_prpvf_ops,
+	[IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
+};
+
+static int imx_ic_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct imx_ic_priv *priv;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	/* get our ipu_id, grp_id and IC task id */
+	pdata = priv->dev->platform_data;
+	priv->ipu_id = pdata->ipu_id;
+	switch (pdata->grp_id) {
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		priv->task_id = IC_TASK_ENCODER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		priv->task_id = IC_TASK_VIEWFINDER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+		priv->task_id = IC_TASK_POST_PROCESSOR;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
+	priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	priv->sd.grp_id = pdata->grp_id;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+	ret = ic_ops[priv->task_id]->init(priv);
+	if (ret)
+		return ret;
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		ic_ops[priv->task_id]->remove(priv);
+
+	return ret;
+}
+
+static int imx_ic_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
+
+	ic_ops[priv->task_id]->remove(priv);
+
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_ic_ids[] = {
+	{ .name = "imx-ipuv3-ic" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_ic_ids);
+
+static struct platform_driver imx_ic_driver = {
+	.probe = imx_ic_probe,
+	.remove = imx_ic_remove,
+	.id_table = imx_ic_ids,
+	.driver = {
+		.name = "imx-ipuv3-ic",
+	},
+};
+module_platform_driver(imx_ic_driver);
+
+MODULE_DESCRIPTION("i.MX IC subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-ic");
diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
new file mode 100644
index 0000000..1f75616
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-pp.c
@@ -0,0 +1,636 @@
+/*
+ * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-image-convert.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PP_NUM_PADS 2
+
+struct pp_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+	int pp_id;
+
+	struct ipu_soc *ipu;
+	struct ipu_image_convert_ctx *ic_ctx;
+
+	struct media_pad pad[PP_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	/* our dma buffer sink ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	/* the dma buffer ring we send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct ipu_image_convert_run *out_run;
+
+	struct imx_media_dma_buf *inbuf; /* last input buffer */
+
+	bool stream_on;    /* streaming is on */
+	bool stop;         /* streaming is stopping */
+	spinlock_t irqlock;
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PP_NUM_PADS];
+
+	/* motion select control */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+};
+
+static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void pp_convert_complete(struct ipu_image_convert_run *run,
+				void *data)
+{
+	struct pp_priv *priv = data;
+	struct imx_media_dma_buf *done;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, run->status ?
+				       IMX_MEDIA_BUF_STATUS_ERROR :
+				       IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* we're done with the inbuf, queue it back */
+	imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
+
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+}
+
+static void pp_queue_conversion(struct pp_priv *priv,
+				struct imx_media_dma_buf *inbuf)
+{
+	struct ipu_image_convert_run *run;
+	struct imx_media_dma_buf *outbuf;
+
+	/* get next queued buffer and make it active */
+	outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(outbuf);
+	priv->inbuf = inbuf;
+
+	run = &priv->out_run[outbuf->index];
+	run->ctx = priv->ic_ctx;
+	run->in_phys = inbuf->phys;
+	run->out_phys = outbuf->phys;
+	ipu_image_convert_queue(run);
+}
+
+static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_dma_buf_ring **ring;
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		/* src asks for a buffer ring */
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		/* src hands us a new buffer */
+		spin_lock_irqsave(&priv->irqlock, flags);
+		if (!priv->stop &&
+		    !imx_media_dma_buf_get_active(priv->out_ring)) {
+			buf = imx_media_dma_buf_dequeue(priv->in_ring);
+			if (buf)
+				pp_queue_conversion(priv, buf);
+		}
+		spin_unlock_irqrestore(&priv->irqlock, flags);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int pp_start(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct ipu_image image_in, image_out;
+	const struct imx_media_pixfmt *incc;
+	struct v4l2_mbus_framefmt *infmt;
+	int i, in_size, ret;
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		return ret;
+
+	imx_media_mbus_fmt_to_ipu_image(&image_in,
+					&priv->format_mbus[priv->input_pad]);
+	imx_media_mbus_fmt_to_ipu_image(&image_out,
+					&priv->format_mbus[priv->output_pad]);
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+	priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
+						 IC_TASK_POST_PROCESSOR,
+						 &image_in, &image_out,
+						 priv->rot_mode,
+						 pp_convert_complete, priv);
+	if (IS_ERR(priv->ic_ctx))
+		return PTR_ERR(priv->ic_ctx);
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	incc = priv->cc[priv->input_pad];
+	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+	if (priv->in_ring) {
+		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+	}
+
+	priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
+						     &priv->src_sd->entity,
+						     &ic_priv->sd.entity,
+						     in_size,
+						     IMX_MEDIA_MIN_RING_BUFS,
+						     true);
+	if (IS_ERR(priv->in_ring)) {
+		v4l2_err(&ic_priv->sd,
+			 "failed to alloc dma-buf ring\n");
+		ret = PTR_ERR(priv->in_ring);
+		priv->in_ring = NULL;
+		goto out_unprep;
+	}
+
+	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
+		imx_media_dma_buf_queue(priv->in_ring, i);
+
+	priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
+				sizeof(*priv->out_run), GFP_KERNEL);
+	if (!priv->out_run) {
+		v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
+		ret = -ENOMEM;
+		goto out_free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+out_free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+out_unprep:
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	return ret;
+}
+
+static void pp_stop(struct pp_priv *priv)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	kfree(priv->out_run);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that its sink buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int pp_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = pp_start(priv);
+	else if (!enable && priv->stream_on)
+		pp_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int pp_enum_mbus_code(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_mbus_code_enum *code)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 fourcc;
+	int ret;
+
+	if (code->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	ret = ipu_image_convert_enum_format(code->index, &fourcc);
+	if (ret)
+		return ret;
+
+	/* convert returned fourcc to mbus code */
+	cc = imx_media_find_format(fourcc, 0, true, true);
+	if (WARN_ON(!cc))
+		return -EINVAL;
+
+	code->code = cc->codes[0];
+	return 0;
+}
+
+static int pp_get_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int pp_set_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	struct ipu_image test_in, test_out;
+	u32 code;
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
+	} else {
+		imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		if (sdformat->pad == priv->output_pad) {
+			*outfmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
+		} else {
+			*infmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
+		}
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int pp_link_setup(struct media_entity *entity,
+			 const struct media_pad *local,
+			 const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct pp_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+	} else {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+	}
+
+	return 0;
+}
+
+static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct pp_priv *priv = container_of(ctrl->handler,
+					       struct pp_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		struct v4l2_mbus_framefmt *infmt, *outfmt;
+		struct ipu_image test_in, test_out;
+
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		/*
+		 * make sure this rotation will work with current input/output
+		 * formats before setting
+		 */
+		infmt = &priv->format_mbus[priv->input_pad];
+		outfmt = &priv->format_mbus[priv->output_pad];
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+
+		ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);
+		if (ret)
+			return ret;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops pp_ctrl_ops = {
+	.s_ctrl = pp_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config pp_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};
+
+#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
+
+static int pp_init_controls(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
+
+	for (i = 0; i < PP_NUM_CONTROLS; i++) {
+		c = &pp_std_ctrl[i];
+		v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
+				  c->id, c->min, c->max, c->step, c->def);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		v4l2_ctrl_handler_free(hdlr);
+		return ret;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int pp_registered(struct v4l2_subdev *sd)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PP_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	ret = pp_init_controls(priv);
+	if (ret)
+		return ret;
+
+	ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+free_ctrls:
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	return ret;
+}
+
+static struct v4l2_subdev_pad_ops pp_pad_ops = {
+	.enum_mbus_code = pp_enum_mbus_code,
+	.get_fmt = pp_get_fmt,
+	.set_fmt = pp_set_fmt,
+};
+
+static struct v4l2_subdev_video_ops pp_video_ops = {
+	.s_stream = pp_s_stream,
+};
+
+static struct v4l2_subdev_core_ops pp_core_ops = {
+	.ioctl = pp_ioctl,
+};
+
+static struct media_entity_operations pp_entity_ops = {
+	.link_setup = pp_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops pp_subdev_ops = {
+	.video = &pp_video_ops,
+	.pad = &pp_pad_ops,
+	.core = &pp_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops pp_internal_ops = {
+	.registered = pp_registered,
+};
+
+static int pp_init(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+	spin_lock_init(&priv->irqlock);
+
+	/* get our PP id */
+	priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
+
+	return 0;
+}
+
+static void pp_remove(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_pp_ops = {
+	.subdev_ops = &pp_subdev_ops,
+	.internal_ops = &pp_internal_ops,
+	.entity_ops = &pp_entity_ops,
+	.init = pp_init,
+	.remove = pp_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
new file mode 100644
index 0000000..3d85a82
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpenc.c
@@ -0,0 +1,1033 @@
+/*
+ * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of video frames from the CSI, which
+ * are routed directly to the Image Converter preprocess encode task,
+ * for resizing, colorspace conversion, and rotation.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PRPENC_NUM_PADS 2
+
+#define MAX_W_IC   1024
+#define MAX_H_IC   1024
+#define MAX_W_SINK 4096
+#define MAX_H_SINK 4096
+
+struct prpenc_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+
+	/* IPU units we require */
+	struct ipu_soc *ipu;
+	struct ipu_ic *ic_enc;
+
+	struct media_pad pad[PRPENC_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *enc_ch;
+	struct ipuv3_channel *enc_rot_in_ch;
+	struct ipuv3_channel *enc_rot_out_ch;
+
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct imx_media_dma_buf *next;
+
+	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	/* the CSI id at link validate */
+	int csi_id;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
+
+	struct imx_media_dma_buf rot_buf[2];
+
+	/* controls */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+
+	spinlock_t irqlock;
+
+	struct timer_list eof_timeout_timer;
+	int eof_irq;
+	int nfb4eof_irq;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->ic_enc))
+		ipu_ic_put(priv->ic_enc);
+	priv->ic_enc = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_ch))
+		ipu_idmac_put(priv->enc_ch);
+	priv->enc_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
+		ipu_idmac_put(priv->enc_rot_in_ch);
+	priv->enc_rot_in_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
+		ipu_idmac_put(priv->enc_rot_out_ch);
+	priv->enc_rot_out_ch = NULL;
+}
+
+static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+	priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
+	if (IS_ERR(priv->ic_enc)) {
+		v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
+		ret = PTR_ERR(priv->ic_enc);
+		goto out;
+	}
+
+	priv->enc_ch = ipu_idmac_get(priv->ipu,
+				     IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+	if (IS_ERR(priv->enc_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+		ret = PTR_ERR(priv->enc_ch);
+		goto out;
+	}
+
+	priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
+					    IPUV3_CHANNEL_MEM_ROT_ENC);
+	if (IS_ERR(priv->enc_rot_in_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_MEM_ROT_ENC);
+		ret = PTR_ERR(priv->enc_rot_in_ch);
+		goto out;
+	}
+
+	priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
+					     IPUV3_CHANNEL_ROT_ENC_MEM);
+	if (IS_ERR(priv->enc_rot_out_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_ROT_ENC_MEM);
+		ret = PTR_ERR(priv->enc_rot_out_ch);
+		goto out;
+	}
+
+	return 0;
+out:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_media_dma_buf *done, *next;
+	struct ipuv3_channel *channel;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	/* inform CSI of this EOF so it can monitor frame intervals */
+	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+			 0, NULL);
+
+	channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
+		priv->enc_rot_out_ch : priv->enc_ch;
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* priv->next buffer is now the active one */
+	imx_media_dma_buf_set_active(priv->next);
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
+		ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);
+
+	/* get next queued buffer */
+	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+	ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
+	ipu_idmac_select_buffer(channel, priv->ipu_buf_num);
+
+	/* toggle IPU double-buffer index */
+	priv->ipu_buf_num ^= 1;
+	priv->next = next;
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void prpenc_eof_timeout(unsigned long data)
+{
+	struct prpenc_priv *priv = (struct prpenc_priv *)data;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void prpenc_setup_channel(struct prpenc_priv *priv,
+				 struct ipuv3_channel *channel,
+				 enum ipu_rotate_mode rot_mode,
+				 dma_addr_t addr0, dma_addr_t addr1,
+				 bool rot_swap_width_height)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	unsigned int burst_size;
+	struct ipu_image image;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+
+	ipu_cpmem_zero(channel);
+
+	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+	image.phys0 = addr0;
+	image.phys1 = addr1;
+	ipu_cpmem_set_image(channel, &image);
+
+	if (channel == priv->enc_rot_in_ch ||
+	    channel == priv->enc_rot_out_ch) {
+		burst_size = 8;
+		ipu_cpmem_set_block_mode(channel);
+	} else {
+		burst_size = (outfmt->width & 0xf) ? 8 : 16;
+	}
+
+	ipu_cpmem_set_burstsize(channel, burst_size);
+
+	if (rot_mode)
+		ipu_cpmem_set_rotation(channel, rot_mode);
+
+	if (outfmt->field == V4L2_FIELD_NONE &&
+	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+	     infmt->field == V4L2_FIELD_ALTERNATE) &&
+	    channel == priv->enc_ch)
+		ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
+
+	ipu_ic_task_idma_init(priv->ic_enc, channel,
+			      outfmt->width, outfmt->height,
+			      burst_size, rot_mode);
+	ipu_cpmem_set_axi_id(channel, 1);
+
+	ipu_idmac_set_double_buffer(channel, true);
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+}
+
+static int prpenc_setup_rotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int out_size, ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
+
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
+		return ret;
+	}
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
+		goto free_rot0;
+	}
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->height, outfmt->width,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		goto free_rot1;
+	}
+
+	/* init the IC ENC-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch,
+			     IPU_ROTATE_NONE,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	/* init the MEM-->IC ENC ROT IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_in_ch,
+			     priv->rot_mode,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the destination IC ENC ROT-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_out_ch,
+			     IPU_ROTATE_NONE,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	/* now link IC ENC-->MEM to MEM-->IC ENC ROT */
+	ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
+
+	/* enable the IC */
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_out_ch);
+
+	/* and finally enable the IC PRPENC task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+
+free_rot1:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+free_rot0:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	return ret;
+}
+
+static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_out_ch);
+
+	ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
+
+	ipu_ic_disable(priv->ic_enc);
+
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+}
+
+static int prpenc_setup_norotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->width, outfmt->height,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		return ret;
+	}
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the IC PRP-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	ipu_cpmem_dump(priv->enc_ch);
+	ipu_ic_dump(priv->ic_enc);
+	ipu_dump(priv->ipu);
+
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+
+	/* enable the IC ENCODE task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+}
+
+static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_ic_disable(priv->ic_enc);
+}
+
+static int prpenc_start(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = prpenc_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	/* set IC to receive from CSI */
+	ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->ipu_buf_num = 0;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		ret = prpenc_setup_rotation(priv);
+	else
+		ret = prpenc_setup_norotation(priv);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->enc_ch,
+						  IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+			       prpenc_nfb4eof_interrupt, 0,
+			       "imx-ic-prpenc-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_unsetup;
+	}
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
+	else
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
+
+	ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
+			       prpenc_eof_interrupt, 0,
+			       "imx-ic-prpenc-eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering eof irq: %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+out_put_ipu:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prpenc_stop(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp,
+		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+	devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+
+	prpenc_put_ipu_resources(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_pad_config *cfg,
+				 struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	bool allow_planar;
+
+	if (code->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	allow_planar = (code->pad == priv->output_pad);
+
+	return imx_media_enum_format(&code->code, code->index,
+				     true, allow_planar);
+}
+
+static int prpenc_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int prpenc_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	bool allow_planar;
+	u32 code;
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   true, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, false);
+		cc = imx_media_find_format(0, code, true, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_IC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_IC);
+
+		if (sdformat->format.field != V4L2_FIELD_NONE)
+			sdformat->format.field = infmt->field;
+
+		/* IC resizer cannot downsize more than 4:1 */
+		if (ipu_rot_mode_is_irt(priv->rot_mode)) {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->height / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->width / 4);
+		} else {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->width / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->height / 4);
+		}
+	} else {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_SINK);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_SINK);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_setup(struct media_entity *entity,
+			     const struct media_pad *local,
+			     const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	/* this is sink pad */
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+	} else {
+		priv->src_sd = NULL;
+		return 0;
+	}
+
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0:
+		priv->csi_id = 0;
+		break;
+	case IMX_MEDIA_GRP_ID_CSI1:
+		priv->csi_id = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_validate(struct v4l2_subdev *sd,
+				struct media_link *link,
+				struct v4l2_subdev_format *source_fmt,
+				struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link,
+						source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		int vc_num = 0;
+		/* see NOTE in imx-csi.c */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &ic_priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		/* only virtual channel 0 can be sent to IC */
+		if (vc_num != 0)
+			return -EINVAL;
+	} else {
+		/*
+		 * only 8-bit pixels can be sent to IC for parallel
+		 * busses
+		 */
+		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct prpenc_priv *priv = container_of(ctrl->handler,
+					       struct prpenc_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
+	.s_ctrl = prpenc_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};
+
+#define PRPENC_NUM_CONTROLS ARRAY_SIZE(prpenc_std_ctrl)
+
+static int prpenc_init_controls(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, PRPENC_NUM_CONTROLS);
+
+	for (i = 0; i < PRPENC_NUM_CONTROLS; i++) {
+		c = &prpenc_std_ctrl[i];
+		v4l2_ctrl_new_std(hdlr, &prpenc_ctrl_ops,
+				  c->id, c->min, c->max, c->step, c->def);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		goto out_free;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+	return 0;
+
+out_free:
+	v4l2_ctrl_handler_free(hdlr);
+	return ret;
+}
+
+static int prpenc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = prpenc_start(priv);
+	else if (!enable && priv->stream_on)
+		prpenc_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int prpenc_registered(struct v4l2_subdev *sd)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PRPENC_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	ret = prpenc_init_controls(priv);
+	if (ret)
+		return ret;
+
+	ret = media_entity_pads_init(&sd->entity, PRPENC_NUM_PADS, priv->pad);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static struct v4l2_subdev_pad_ops prpenc_pad_ops = {
+	.enum_mbus_code = prpenc_enum_mbus_code,
+	.get_fmt = prpenc_get_fmt,
+	.set_fmt = prpenc_set_fmt,
+	.link_validate = prpenc_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpenc_video_ops = {
+	.s_stream = prpenc_s_stream,
+};
+
+static struct media_entity_operations prpenc_entity_ops = {
+	.link_setup = prpenc_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops prpenc_subdev_ops = {
+	.video = &prpenc_video_ops,
+	.pad = &prpenc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpenc_internal_ops = {
+	.registered = prpenc_registered,
+};
+
+static int prpenc_init(struct imx_ic_priv *ic_priv)
+{
+	struct prpenc_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+
+	spin_lock_init(&priv->irqlock);
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = prpenc_eof_timeout;
+
+	return 0;
+}
+
+static void prpenc_remove(struct imx_ic_priv *ic_priv)
+{
+	struct prpenc_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_prpenc_ops = {
+	.subdev_ops = &prpenc_subdev_ops,
+	.internal_ops = &prpenc_internal_ops,
+	.entity_ops = &prpenc_entity_ops,
+	.init = prpenc_init,
+	.remove = prpenc_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpvf.c b/drivers/staging/media/imx/imx-ic-prpvf.c
new file mode 100644
index 0000000..dc2d402
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpvf.c
@@ -0,0 +1,1179 @@
+/*
+ * V4L2 IC Deinterlacer Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+/*
+ * This subdev implements two different video pipelines:
+ *
+ * CSI -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends a single interlaced field F(n-1)
+ * directly to the VDIC (and optionally the following field F(n)
+ * can be sent to memory via IDMAC channel 13). So only two fields
+ * can be processed by the VDIC. This pipeline only works in VDIC's
+ * high motion mode, which only requires a single field for processing.
+ * The other motion modes (low and medium) require three fields, so this
+ * pipeline does not work in those modes. Also, it is not clear how this
+ * pipeline can deal with the various field orders (sequential BT/TB,
+ * interlaced BT/TB).
+ *
+ * CSI -> CH[0-3] -> MEM -> CH8,9,10 -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends raw and full frames to memory buffers
+ * via the SMFC channels 0-3. Fields from these frames are then
+ * transferred to the VDIC via IDMAC channels 8,9,10. The VDIC requires
+ * three fields: previous field F(n-1), current field F(n), and next
+ * field F(n+1), so we need three raw frames in memory: two completed frames
+ * to send F(n-1), F(n), F(n+1) to the VDIC, and a third frame for active
+ * CSI capture while the completed fields are sent through the VDIC->IC for
+ * processing.
+ *
+ * The "direct" CSI->VDIC pipeline requires less memory bus bandwidth
+ * (just 1 channel vs. 5 channels for indirect pipeline), but it can
+ * only be used in high motion mode, and it only processes a single
+ * field (so half the original image resolution is lost).
+ */
+
+struct prpvf_priv;
+
+struct prpvf_pipeline_ops {
+	int (*setup)(struct prpvf_priv *priv);
+	void (*start)(struct prpvf_priv *priv);
+	void (*stop)(struct prpvf_priv *priv);
+	void (*disable)(struct prpvf_priv *priv);
+};
+
+#define PRPVF_NUM_PADS 2
+
+#define MAX_W_IC   1024
+#define MAX_H_IC   1024
+#define MAX_W_VDIC  968
+#define MAX_H_VDIC 2048
+
+struct prpvf_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+
+	/* IPU units we require */
+	struct ipu_soc *ipu;
+	struct ipu_ic *ic_vf;
+	struct ipu_vdi *vdi;
+
+	struct media_pad pad[PRPVF_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *vdi_in_ch_p; /* F(n-1) transfer channel */
+	struct ipuv3_channel *vdi_in_ch;   /* F(n) transfer channel */
+	struct ipuv3_channel *vdi_in_ch_n; /* F(n+1) transfer channel */
+	struct ipuv3_channel *prpvf_out_ch;/* final progressive frame channel */
+
+	/* pipeline operations */
+	struct prpvf_pipeline_ops *ops;
+
+	/* our dma buffer sink ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+
+	/* ipu buf num for double-buffering (csi-direct path only) */
+	int ipu_buf_num;
+	struct imx_media_dma_buf *next_out_buf;
+
+	/* current and last input buffers indirect path */
+	struct imx_media_dma_buf *curr_in_buf;
+	struct imx_media_dma_buf *last_in_buf;
+
+	/*
+	 * translated field type, input line stride, and field size
+	 * for indirect path
+	 */
+	u32 fieldtype;
+	u32 in_stride;
+	u32 field_size;
+
+	struct v4l2_subdev *src_sd;
+	/* the sink that will receive the progressive out buffers */
+	struct v4l2_subdev *sink_sd;
+
+	/* the attached CSI at stream on */
+	struct v4l2_subdev *csi_sd;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	/* the video standard from sensor at time of streamon */
+	v4l2_std_id std;
+
+	struct v4l2_mbus_framefmt format_mbus[PRPVF_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PRPVF_NUM_PADS];
+
+	bool csi_direct;  /* using direct CSI->VDIC->IC pipeline */
+
+	/* motion select control */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	enum ipu_motion_sel motion;
+
+	struct timer_list eof_timeout_timer;
+
+	int nfb4eof_irq; /* CSI or PRPVF channel NFB4EOF IRQ */
+	int out_eof_irq; /* PRPVF channel EOF IRQ */
+	spinlock_t irqlock;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static inline struct prpvf_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void prpvf_put_ipu_resources(struct prpvf_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->ic_vf))
+		ipu_ic_put(priv->ic_vf);
+	priv->ic_vf = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch_p))
+		ipu_idmac_put(priv->vdi_in_ch_p);
+	priv->vdi_in_ch_p = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch))
+		ipu_idmac_put(priv->vdi_in_ch);
+	priv->vdi_in_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi_in_ch_n))
+		ipu_idmac_put(priv->vdi_in_ch_n);
+	priv->vdi_in_ch_n = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->prpvf_out_ch))
+		ipu_idmac_put(priv->prpvf_out_ch);
+	priv->prpvf_out_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->vdi))
+		ipu_vdi_put(priv->vdi);
+	priv->vdi = NULL;
+}
+
+static int prpvf_get_ipu_resources(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret, err_chan;
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+	priv->ic_vf = ipu_ic_get(priv->ipu, IC_TASK_VIEWFINDER);
+	if (IS_ERR(priv->ic_vf)) {
+		v4l2_err(&ic_priv->sd, "failed to get IC VF\n");
+		ret = PTR_ERR(priv->ic_vf);
+		goto out;
+	}
+
+	priv->vdi = ipu_vdi_get(priv->ipu);
+	if (IS_ERR(priv->vdi)) {
+		v4l2_err(&ic_priv->sd, "failed to get VDIC\n");
+		ret = PTR_ERR(priv->vdi);
+		goto out;
+	}
+
+	priv->prpvf_out_ch = ipu_idmac_get(priv->ipu,
+					   IPUV3_CHANNEL_IC_PRP_VF_MEM);
+	if (IS_ERR(priv->prpvf_out_ch)) {
+		err_chan = IPUV3_CHANNEL_IC_PRP_VF_MEM;
+		ret = PTR_ERR(priv->prpvf_out_ch);
+		goto out_err_chan;
+	}
+
+	if (!priv->csi_direct) {
+		priv->vdi_in_ch_p = ipu_idmac_get(priv->ipu,
+						  IPUV3_CHANNEL_MEM_VDI_PREV);
+		if (IS_ERR(priv->vdi_in_ch_p)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_PREV;
+			ret = PTR_ERR(priv->vdi_in_ch_p);
+			goto out_err_chan;
+		}
+
+		priv->vdi_in_ch = ipu_idmac_get(priv->ipu,
+						IPUV3_CHANNEL_MEM_VDI_CUR);
+		if (IS_ERR(priv->vdi_in_ch)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_CUR;
+			ret = PTR_ERR(priv->vdi_in_ch);
+			goto out_err_chan;
+		}
+
+		priv->vdi_in_ch_n = ipu_idmac_get(priv->ipu,
+						  IPUV3_CHANNEL_MEM_VDI_NEXT);
+		if (IS_ERR(priv->vdi_in_ch_n)) {
+			err_chan = IPUV3_CHANNEL_MEM_VDI_NEXT;
+			ret = PTR_ERR(priv->vdi_in_ch_n);
+			goto out_err_chan;
+		}
+	}
+
+	return 0;
+
+out_err_chan:
+	v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n", err_chan);
+out:
+	prpvf_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prepare_vdi_in_buffers(struct prpvf_priv *priv,
+				   struct imx_media_dma_buf *curr)
+{
+	dma_addr_t prev_phys, curr_phys, next_phys;
+	struct imx_media_dma_buf *last;
+
+	last = priv->last_in_buf ? priv->last_in_buf : curr;
+	priv->curr_in_buf = curr;
+
+	switch (priv->fieldtype) {
+	case V4L2_FIELD_SEQ_TB:
+		prev_phys = last->phys;
+		curr_phys = curr->phys + priv->field_size;
+		next_phys = curr->phys;
+		break;
+	case V4L2_FIELD_SEQ_BT:
+		prev_phys = last->phys + priv->field_size;
+		curr_phys = curr->phys;
+		next_phys = curr->phys + priv->field_size;
+		break;
+	case V4L2_FIELD_INTERLACED_BT:
+		prev_phys = last->phys + priv->in_stride;
+		curr_phys = curr->phys;
+		next_phys = curr->phys + priv->in_stride;
+		break;
+	default:
+		/* assume V4L2_FIELD_INTERLACED_TB */
+		prev_phys = last->phys;
+		curr_phys = curr->phys + priv->in_stride;
+		next_phys = curr->phys;
+		break;
+	}
+
+	ipu_cpmem_set_buffer(priv->vdi_in_ch_p, 0, prev_phys);
+	ipu_cpmem_set_buffer(priv->vdi_in_ch,   0, curr_phys);
+	ipu_cpmem_set_buffer(priv->vdi_in_ch_n, 0, next_phys);
+
+	ipu_idmac_select_buffer(priv->vdi_in_ch_p, 0);
+	ipu_idmac_select_buffer(priv->vdi_in_ch, 0);
+	ipu_idmac_select_buffer(priv->vdi_in_ch_n, 0);
+}
+
+static void prepare_prpvf_out_buffer(struct prpvf_priv *priv)
+{
+	struct imx_media_dma_buf *buf;
+
+	/* get next buffer to prepare */
+	buf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	if (!priv->csi_direct) {
+		/*
+		 * indirect does not use double-buffering, so this
+		 * buffer is now the active one
+		 */
+		imx_media_dma_buf_set_active(buf);
+	} else {
+		priv->next_out_buf = buf;
+	}
+
+	ipu_cpmem_set_buffer(priv->prpvf_out_ch, priv->ipu_buf_num, buf->phys);
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, priv->ipu_buf_num);
+}
+
+/* prpvf_out_ch EOF interrupt (progressive frame ready) */
+static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
+{
+	struct prpvf_priv *priv = dev_id;
+	struct imx_media_dma_buf *done;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	if (priv->csi_direct) {
+		/* inform CSI of this EOF so it can monitor frame intervals */
+		/* FIXME: frames are coming in twice as fast in direct path! */
+		v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+				 0, NULL);
+	}
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	if (!priv->csi_direct) {
+		/* we're done with the input buffer, queue it back */
+		imx_media_dma_buf_queue(priv->in_ring,
+					priv->curr_in_buf->index);
+
+		/* current input buffer is now last */
+		priv->last_in_buf = priv->curr_in_buf;
+	} else {
+		/*
+		 * priv->next buffer is now the active one due
+		 * to IPU double-buffering
+		 */
+		imx_media_dma_buf_set_active(priv->next_out_buf);
+	}
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (priv->csi_direct) {
+		prepare_prpvf_out_buffer(priv);
+		/* toggle IPU double-buffer index */
+		priv->ipu_buf_num ^= 1;
+	}
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static long prpvf_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct imx_media_dma_buf_ring **ring;
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		spin_lock_irqsave(&priv->irqlock, flags);
+		if (!imx_media_dma_buf_get_active(priv->out_ring)) {
+			buf = imx_media_dma_buf_dequeue(priv->in_ring);
+			if (buf) {
+				prepare_vdi_in_buffers(priv, buf);
+				prepare_prpvf_out_buffer(priv);
+			}
+		}
+		spin_unlock_irqrestore(&priv->irqlock, flags);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static irqreturn_t nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct prpvf_priv *priv = dev_id;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void prpvf_eof_timeout(unsigned long data)
+{
+	struct prpvf_priv *priv = (struct prpvf_priv *)data;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void setup_vdi_channel(struct prpvf_priv *priv,
+			      struct ipuv3_channel *channel,
+			      dma_addr_t phys0, dma_addr_t phys1,
+			      bool out_chan)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	unsigned int burst_size;
+	struct ipu_image image;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (out_chan) {
+		imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+	} else {
+		/* one field to VDIC channels */
+		infmt->height /= 2;
+		imx_media_mbus_fmt_to_ipu_image(&image, infmt);
+		infmt->height *= 2;
+	}
+	image.phys0 = phys0;
+	image.phys1 = phys1;
+
+	ipu_cpmem_zero(channel);
+	ipu_cpmem_set_image(channel, &image);
+
+	if (out_chan) {
+		burst_size = (outfmt->width & 0xf) ? 8 : 16;
+		ipu_cpmem_set_burstsize(channel, burst_size);
+		ipu_ic_task_idma_init(priv->ic_vf, channel,
+				      outfmt->width, outfmt->height,
+				      burst_size, IPU_ROTATE_NONE);
+	} else {
+		burst_size = (infmt->width & 0xf) ? 8 : 16;
+		ipu_cpmem_set_burstsize(channel, burst_size);
+	}
+
+	ipu_cpmem_set_axi_id(channel, 1);
+
+	ipu_idmac_set_double_buffer(channel, priv->csi_direct && out_chan);
+}
+
+static int prpvf_setup_direct(struct prpvf_priv *priv)
+{
+	struct imx_media_dma_buf *buf0, *buf1;
+
+	/* set VDIC to receive from CSI for direct path */
+	ipu_fsu_link(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+		     IPUV3_CHANNEL_CSI_VDI_PREV);
+
+	priv->ipu_buf_num = 0;
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next_out_buf = buf1;
+
+	/* init the prpvf out channel */
+	setup_vdi_channel(priv, priv->prpvf_out_ch,
+			  buf0->phys, buf1->phys, true);
+
+	return 0;
+}
+
+static void prpvf_start_direct(struct prpvf_priv *priv)
+{
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, 0);
+	ipu_idmac_select_buffer(priv->prpvf_out_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_stop_direct(struct prpvf_priv *priv)
+{
+	ipu_idmac_disable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_disable_direct(struct prpvf_priv *priv)
+{
+	ipu_fsu_unlink(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+		       IPUV3_CHANNEL_CSI_VDI_PREV);
+}
+
+static int prpvf_setup_indirect(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt;
+	const struct imx_media_pixfmt *incc;
+	int in_size, i, ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	incc = priv->cc[priv->input_pad];
+
+	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+	/* 1/2 full image size */
+	priv->field_size = in_size / 2;
+	priv->in_stride = incc->planar ?
+		infmt->width : (infmt->width * incc->bpp) >> 3;
+
+	priv->ipu_buf_num = 0;
+
+	if (priv->in_ring) {
+		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+	}
+
+	priv->in_ring = imx_media_alloc_dma_buf_ring(
+		priv->md, &priv->src_sd->entity,
+		&ic_priv->sd.entity,
+		in_size, IMX_MEDIA_MIN_RING_BUFS_PRPVF, true);
+	if (IS_ERR(priv->in_ring)) {
+		v4l2_err(&ic_priv->sd, "failed to alloc dma-buf ring\n");
+		ret = PTR_ERR(priv->in_ring);
+		priv->in_ring = NULL;
+		return ret;
+	}
+
+	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS_PRPVF; i++)
+		imx_media_dma_buf_queue(priv->in_ring, i);
+
+	priv->last_in_buf = NULL;
+	priv->curr_in_buf = NULL;
+
+	/* translate V4L2_FIELD_ALTERNATE to SEQ_TB or SEQ_BT */
+	priv->fieldtype = infmt->field;
+	if (infmt->field == V4L2_FIELD_ALTERNATE)
+		priv->fieldtype = (priv->std & V4L2_STD_525_60) ?
+			V4L2_FIELD_SEQ_TB : V4L2_FIELD_SEQ_BT;
+
+	/* init the vdi-in channels */
+	setup_vdi_channel(priv, priv->vdi_in_ch_p, 0, 0, false);
+	setup_vdi_channel(priv, priv->vdi_in_ch, 0, 0, false);
+	setup_vdi_channel(priv, priv->vdi_in_ch_n, 0, 0, false);
+
+	/* init the prpvf out channel */
+	setup_vdi_channel(priv, priv->prpvf_out_ch, 0, 0, true);
+
+	return 0;
+}
+
+static void prpvf_start_indirect(struct prpvf_priv *priv)
+{
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->prpvf_out_ch);
+	ipu_idmac_enable_channel(priv->vdi_in_ch_p);
+	ipu_idmac_enable_channel(priv->vdi_in_ch);
+	ipu_idmac_enable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_stop_indirect(struct prpvf_priv *priv)
+{
+	/* disable channels */
+	ipu_idmac_disable_channel(priv->prpvf_out_ch);
+	ipu_idmac_disable_channel(priv->vdi_in_ch_p);
+	ipu_idmac_disable_channel(priv->vdi_in_ch);
+	ipu_idmac_disable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_disable_indirect(struct prpvf_priv *priv)
+{
+}
+
+static struct prpvf_pipeline_ops direct_ops = {
+	.setup = prpvf_setup_direct,
+	.start = prpvf_start_direct,
+	.stop = prpvf_stop_direct,
+	.disable = prpvf_disable_direct,
+};
+
+static struct prpvf_pipeline_ops indirect_ops = {
+	.setup = prpvf_setup_indirect,
+	.start = prpvf_start_indirect,
+	.stop = prpvf_stop_indirect,
+	.disable = prpvf_disable_indirect,
+};
+
+static int prpvf_start(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	priv->ops = priv->csi_direct ? &direct_ops : &indirect_ops;
+
+	ret = prpvf_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	/* set IC to receive from VDIC */
+	ipu_set_ic_src_mux(priv->ipu, 0, true);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	/* request EOF irq for prpvf out channel */
+	priv->out_eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->prpvf_out_ch,
+						  IPU_IRQ_EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->out_eof_irq,
+			       prpvf_out_eof_interrupt, 0,
+			       "imx-ic-prpvf-out-eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering out eof irq: %d\n", ret);
+		goto out_put_ipu;
+	}
+
+	/* request NFB4EOF irq */
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->prpvf_out_ch,
+						  IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+			       nfb4eof_interrupt, 0,
+			       "imx-ic-prpvf-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_free_eof_irq;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_std, &priv->std);
+	if (ret)
+		goto out_free_nfb4eof_irq;
+
+	/* init the VDIC */
+	ipu_vdi_setup(priv->vdi, infmt->code,
+		      infmt->width, infmt->height);
+	ipu_vdi_set_field_order(priv->vdi, priv->std, infmt->field);
+	ipu_vdi_set_motion(priv->vdi, priv->motion);
+
+	ret = ipu_ic_task_init(priv->ic_vf,
+			       infmt->width, infmt->height,
+			       outfmt->width, outfmt->height,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	ret = priv->ops->setup(priv);
+	if (ret)
+		goto out_free_nfb4eof_irq;
+
+	ipu_vdi_enable(priv->vdi);
+	ipu_ic_enable(priv->ic_vf);
+
+	priv->ops->start(priv);
+
+	/* enable the IC VF task */
+	ipu_ic_task_enable(priv->ic_vf);
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_free_eof_irq:
+	devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+out_put_ipu:
+	prpvf_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prpvf_stop(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+	ipu_ic_task_disable(priv->ic_vf);
+	priv->ops->stop(priv);
+	ipu_ic_disable(priv->ic_vf);
+	ipu_vdi_disable(priv->vdi);
+	priv->ops->disable(priv);
+
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+	devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+	prpvf_put_ipu_resources(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int prpvf_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct prpvf_priv *priv = container_of(ctrl->handler,
+					       struct prpvf_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_motion_sel motion;
+
+	switch (ctrl->id) {
+	case V4L2_CID_IMX_MOTION:
+		motion = ctrl->val;
+		if (motion != priv->motion) {
+			/* can't change motion control mid-streaming */
+			if (priv->stream_on)
+				return -EBUSY;
+			priv->motion = motion;
+		}
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops prpvf_ctrl_ops = {
+	.s_ctrl = prpvf_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpvf_custom_ctrl[] = {
+	{
+		.ops = &prpvf_ctrl_ops,
+		.id = V4L2_CID_IMX_MOTION,
+		.name = "Motion Compensation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def = MOTION_NONE,
+		.min = MOTION_NONE,
+		.max = HIGH_MOTION,
+		.step = 1,
+	},
+};
+
+#define PRPVF_NUM_CONTROLS ARRAY_SIZE(prpvf_custom_ctrl)
+
+static int prpvf_init_controls(struct prpvf_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_free(hdlr);
+	v4l2_ctrl_handler_init(hdlr, PRPVF_NUM_CONTROLS);
+
+	for (i = 0; i < PRPVF_NUM_CONTROLS; i++) {
+		c = &prpvf_custom_ctrl[i];
+		v4l2_ctrl_new_custom(hdlr, c, NULL);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		goto out_free;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+	return 0;
+
+out_free:
+	v4l2_ctrl_handler_free(hdlr);
+	return ret;
+}
+
+static int prpvf_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = prpvf_start(priv);
+	else if (!enable && priv->stream_on)
+		prpvf_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int prpvf_enum_mbus_code(struct v4l2_subdev *sd,
+				struct v4l2_subdev_pad_config *cfg,
+				struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	bool allow_planar, allow_rgb;
+
+	if (code->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	allow_planar = (code->pad == priv->output_pad);
+	allow_rgb = allow_planar;
+
+	return imx_media_enum_format(&code->code, code->index,
+				     allow_rgb, allow_planar);
+}
+
+static int prpvf_get_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int prpvf_set_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	bool allow_planar, allow_rgb;
+	u32 code;
+
+	if (sdformat->pad >= PRPVF_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+	allow_rgb = allow_planar;
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   allow_rgb, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, false, false);
+		cc = imx_media_find_format(0, code, false, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_IC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_IC);
+		/* IC resizer cannot downsize more than 4:1 */
+		sdformat->format.width = max_t(__u32, sdformat->format.width,
+					       infmt->width / 4);
+		sdformat->format.height = max_t(__u32, sdformat->format.height,
+						infmt->height / 4);
+
+		/* output is always progressive! */
+		sdformat->format.field = V4L2_FIELD_NONE;
+	} else {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_VDIC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_VDIC);
+
+		/* input must be interlaced! Choose alternate if not */
+		if (!V4L2_FIELD_HAS_BOTH(sdformat->format.field))
+			sdformat->format.field = V4L2_FIELD_ALTERNATE;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int prpvf_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpvf_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+	int ret;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	/* this is sink pad */
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+
+		priv->csi_direct = ((priv->src_sd->grp_id &
+				     IMX_MEDIA_GRP_ID_CSI) != 0);
+
+		ret = prpvf_init_controls(priv);
+		if (ret)
+			return ret;
+	} else {
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+		priv->src_sd = NULL;
+	}
+
+	return 0;
+}
+
+static int prpvf_link_validate(struct v4l2_subdev *sd,
+			       struct media_link *link,
+			       struct v4l2_subdev_format *source_fmt,
+			       struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpvf_priv *priv = ic_priv->task_priv;
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct imx_media_subdev *csi;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link,
+						source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	if (!priv->csi_direct) {
+		csi = imx_media_find_pipeline_subdev(
+			priv->md, &ic_priv->sd.entity, IMX_MEDIA_GRP_ID_CSI);
+		if (IS_ERR(csi)) {
+			v4l2_err(&ic_priv->sd, "no CSI attached\n");
+			ret = PTR_ERR(csi);
+			return ret;
+		}
+
+		priv->csi_sd = csi->sd;
+		return 0;
+	}
+
+	priv->csi_sd = priv->src_sd;
+
+	if (priv->motion != HIGH_MOTION) {
+		v4l2_err(&ic_priv->sd,
+			 "direct CSI pipeline requires HIGH_MOTION\n");
+		return -EINVAL;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		int vc_num = 0;
+		/* see NOTE in imx-csi.c */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &ic_priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		/* only virtual channel 0 can be sent to IC */
+		if (vc_num != 0)
+			return -EINVAL;
+	} else {
+		/* only 8-bit pixels can be sent to IC for parallel busses */
+		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int prpvf_registered(struct v4l2_subdev *sd)
+{
+	struct prpvf_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PRPVF_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	return media_entity_pads_init(&sd->entity, PRPVF_NUM_PADS, priv->pad);
+}
+
+static struct v4l2_subdev_pad_ops prpvf_pad_ops = {
+	.enum_mbus_code = prpvf_enum_mbus_code,
+	.get_fmt = prpvf_get_fmt,
+	.set_fmt = prpvf_set_fmt,
+	.link_validate = prpvf_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpvf_video_ops = {
+	.s_stream = prpvf_s_stream,
+};
+
+static struct v4l2_subdev_core_ops prpvf_core_ops = {
+	.ioctl = prpvf_ioctl,
+};
+
+static struct media_entity_operations prpvf_entity_ops = {
+	.link_setup = prpvf_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops prpvf_subdev_ops = {
+	.video = &prpvf_video_ops,
+	.pad = &prpvf_pad_ops,
+	.core = &prpvf_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpvf_internal_ops = {
+	.registered = prpvf_registered,
+};
+
+static int prpvf_init(struct imx_ic_priv *ic_priv)
+{
+	struct prpvf_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+
+	spin_lock_init(&priv->irqlock);
+	init_timer(&priv->eof_timeout_timer);
+	priv->eof_timeout_timer.data = (unsigned long)priv;
+	priv->eof_timeout_timer.function = prpvf_eof_timeout;
+
+	return 0;
+}
+
+static void prpvf_remove(struct imx_ic_priv *ic_priv)
+{
+	struct prpvf_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_prpvf_ops = {
+	.subdev_ops = &prpvf_subdev_ops,
+	.internal_ops = &prpvf_internal_ops,
+	.entity_ops = &prpvf_entity_ops,
+	.init = prpvf_init,
+	.remove = prpvf_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic.h b/drivers/staging/media/imx/imx-ic.h
new file mode 100644
index 0000000..1ab222b
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic.h
@@ -0,0 +1,38 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#ifndef _IMX_IC_H
+#define _IMX_IC_H
+
+#include <media/v4l2-subdev.h>
+
+struct imx_ic_priv {
+	struct device *dev;
+	struct v4l2_subdev sd;
+	int    ipu_id;
+	int    task_id;
+	void   *task_priv;
+};
+
+struct imx_ic_ops {
+	struct v4l2_subdev_ops *subdev_ops;
+	struct v4l2_subdev_internal_ops *internal_ops;
+	struct media_entity_operations *entity_ops;
+
+	int (*init)(struct imx_ic_priv *ic_priv);
+	void (*remove)(struct imx_ic_priv *ic_priv);
+};
+
+extern struct imx_ic_ops imx_ic_prpenc_ops;
+extern struct imx_ic_ops imx_ic_prpvf_ops;
+extern struct imx_ic_ops imx_ic_pp_ops;
+
+#endif
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

This is the camera interface driver that provides the v4l2
user interface. Frames can be received from various sources:

- directly from SMFC for capturing unconverted images directly from
  camera sensors.

- from the IC pre-process encode task.

- from the IC pre-process viewfinder task.

- from the IC post-process task.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile    |    2 +-
 drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
 2 files changed, 1001 insertions(+), 1 deletion(-)
 create mode 100644 drivers/staging/media/imx/imx-camif.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index d2a962c..fe9e992 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
new file mode 100644
index 0000000..404f724
--- /dev/null
+++ b/drivers/staging/media/imx/imx-camif.c
@@ -0,0 +1,1000 @@
+/*
+ * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+#define CAMIF_NUM_PADS 2
+
+struct camif_priv {
+	struct device         *dev;
+	struct video_device    vfd;
+	struct media_pipeline  mp;
+	struct imx_media_dev  *md;
+	struct v4l2_subdev     sd;
+	struct media_pad       pad[CAMIF_NUM_PADS];
+	struct media_pad       vd_pad;
+	int id;
+	int input_pad;
+	int output_pad;
+
+	struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
+	const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
+
+	/* dma buffer ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	struct v4l2_subdev     *src_sd;
+
+	struct mutex           mutex;       /* capture device mutex */
+	spinlock_t             q_lock;      /* protect ready_q */
+
+	/* buffer queue used in videobuf2 */
+	struct vb2_queue       buffer_queue;
+
+	/* streaming buffer queue */
+	struct list_head       ready_q;
+
+	/* controls inherited from subdevs */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+
+	/* misc status */
+	int                    current_input; /* the current input */
+	v4l2_std_id            current_std;   /* current standard */
+	bool                   stop;          /* streaming is stopping */
+};
+
+/* In bytes, per queue */
+#define VID_MEM_LIMIT	SZ_64M
+
+static struct vb2_ops camif_qops;
+
+/*
+ * Video ioctls follow
+ */
+
+static int vidioc_querycap(struct file *file, void *fh,
+			   struct v4l2_capability *cap)
+{
+	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
+	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
+	cap->bus_info[0] = 0;
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
+
+	return 0;
+}
+
+static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
+				  struct v4l2_fmtdesc *f)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+	int ret;
+
+	ret = imx_media_enum_format(&code, f->index, true, true);
+	if (ret)
+		return ret;
+	cc = imx_media_find_format(0, code, true, true);
+	if (!cc)
+		return -EINVAL;
+
+	f->pixelformat = cc->fourcc;
+
+	return 0;
+}
+
+static int camif_g_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct v4l2_mbus_framefmt *outfmt;
+
+	/* user format is the same as the format from output pad */
+	outfmt = &priv->format_mbus[priv->output_pad];
+	return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
+}
+
+static int camif_try_fmt_vid_cap(struct file *file, void *fh,
+				 struct v4l2_format *f)
+{
+	return camif_g_fmt_vid_cap(file, fh, f);
+}
+
+static int camif_s_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	return camif_try_fmt_vid_cap(file, priv, f);
+}
+
+static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	return v4l2_subdev_call(sensor->sd, video, querystd, std);
+}
+
+static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*std = priv->current_std;
+	return 0;
+}
+
+static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (vb2_is_busy(&priv->buffer_queue))
+		return -EBUSY;
+
+	ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
+	if (ret < 0)
+		return ret;
+
+	priv->current_std = std;
+	return 0;
+}
+
+static int camif_enum_input(struct file *file, void *fh,
+			    struct v4l2_input *input)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int index = input->index;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	input->type = V4L2_INPUT_TYPE_CAMERA;
+	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+	if (index == priv->current_input) {
+		v4l2_subdev_call(sensor->sd, video, g_input_status,
+				 &input->status);
+		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
+	} else {
+		input->status = V4L2_IN_ST_NO_SIGNAL;
+		input->std = V4L2_STD_UNKNOWN;
+	}
+
+	return 0;
+}
+
+static int camif_g_input(struct file *file, void *fh, unsigned int *index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*index = priv->current_input;
+	return 0;
+}
+
+static int camif_s_input(struct file *file, void *fh, unsigned int index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	if (index == priv->current_input)
+		return 0;
+
+	/* select the sensor's input */
+	ret = v4l2_subdev_call(sensor->sd, video, s_routing,
+			       sensor->input.value[index], 0, 0);
+	if (!ret)
+		priv->current_input = index;
+
+	return ret;
+}
+
+static int camif_g_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, g_parm, a);
+}
+
+static int camif_s_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, s_parm, a);
+}
+
+static const struct v4l2_ioctl_ops camif_ioctl_ops = {
+	.vidioc_querycap	= vidioc_querycap,
+
+	.vidioc_enum_fmt_vid_cap        = camif_enum_fmt_vid_cap,
+	.vidioc_g_fmt_vid_cap           = camif_g_fmt_vid_cap,
+	.vidioc_try_fmt_vid_cap         = camif_try_fmt_vid_cap,
+	.vidioc_s_fmt_vid_cap           = camif_s_fmt_vid_cap,
+
+	.vidioc_querystd        = camif_querystd,
+	.vidioc_g_std           = camif_g_std,
+	.vidioc_s_std           = camif_s_std,
+
+	.vidioc_enum_input      = camif_enum_input,
+	.vidioc_g_input         = camif_g_input,
+	.vidioc_s_input         = camif_s_input,
+
+	.vidioc_g_parm          = camif_g_parm,
+	.vidioc_s_parm          = camif_s_parm,
+
+	.vidioc_reqbufs		= vb2_ioctl_reqbufs,
+	.vidioc_create_bufs     = vb2_ioctl_create_bufs,
+	.vidioc_prepare_buf     = vb2_ioctl_prepare_buf,
+	.vidioc_querybuf	= vb2_ioctl_querybuf,
+	.vidioc_qbuf		= vb2_ioctl_qbuf,
+	.vidioc_dqbuf		= vb2_ioctl_dqbuf,
+	.vidioc_expbuf		= vb2_ioctl_expbuf,
+	.vidioc_streamon	= vb2_ioctl_streamon,
+	.vidioc_streamoff	= vb2_ioctl_streamoff,
+};
+
+/*
+ * Queue operations
+ */
+
+static u32 camif_get_sizeimage(struct camif_priv *priv)
+{
+	struct v4l2_mbus_framefmt *outfmt;
+	const struct imx_media_pixfmt *outcc;
+
+	outfmt = &priv->format_mbus[priv->output_pad];
+	outcc = priv->cc[priv->output_pad];
+	return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
+}
+
+static int camif_queue_setup(struct vb2_queue *vq,
+			     unsigned int *nbuffers, unsigned int *nplanes,
+			     unsigned int sizes[], struct device *alloc_devs[])
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	unsigned int count = *nbuffers;
+	u32 sizeimage;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	sizeimage = camif_get_sizeimage(priv);
+
+	count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
+	count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
+
+	while (sizeimage * count > VID_MEM_LIMIT)
+		count--;
+
+	if (count < IMX_MEDIA_MIN_RING_BUFS)
+		return -EINVAL;
+
+	*nplanes = 1;
+	*nbuffers = count;
+	sizes[0] = sizeimage;
+
+	return 0;
+}
+
+static int camif_buf_init(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	INIT_LIST_HEAD(&buf->list);
+
+	return 0;
+}
+
+static int camif_buf_prepare(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	int ret;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vb2_plane_size(vb, 0) < sizeimage) {
+		v4l2_err(&priv->sd,
+			 "data will not fit into plane (%lu < %lu)\n",
+			 vb2_plane_size(vb, 0), (long)sizeimage);
+		return -EINVAL;
+	}
+
+	vb2_set_plane_payload(vb, 0, sizeimage);
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			return ret;
+		}
+	}
+
+	ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
+	if (ret)
+		goto free_ring;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+	return ret;
+}
+
+static void camif_buf_queue(struct vb2_buffer *vb)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	list_add_tail(&buf->list, &priv->ready_q);
+
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	struct imx_media_buffer *buf, *tmp;
+	unsigned long flags;
+	int ret;
+
+	if (vb2_is_streaming(vq))
+		return 0;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			goto return_bufs;
+		}
+	}
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, true);
+	if (ret) {
+		v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
+			 ret);
+		goto free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+return_bufs:
+	spin_lock_irqsave(&priv->q_lock, flags);
+	list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
+		list_del(&buf->list);
+		vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+	return ret;
+}
+
+static void camif_stop_streaming(struct vb2_queue *vq)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *frame;
+	unsigned long flags;
+	int ret;
+
+	if (!vb2_is_streaming(vq))
+		return;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, false);
+	if (ret)
+		v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
+			  ret);
+
+	if (priv->in_ring) {
+		v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+	}
+
+	/* release all active buffers */
+	spin_lock_irqsave(&priv->q_lock, flags);
+	while (!list_empty(&priv->ready_q)) {
+		frame = list_entry(priv->ready_q.next,
+				   struct imx_media_buffer, list);
+		list_del(&frame->list);
+		vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static struct vb2_ops camif_qops = {
+	.queue_setup	 = camif_queue_setup,
+	.buf_init        = camif_buf_init,
+	.buf_prepare	 = camif_buf_prepare,
+	.buf_queue	 = camif_buf_queue,
+	.wait_prepare	 = vb2_ops_wait_prepare,
+	.wait_finish	 = vb2_ops_wait_finish,
+	.start_streaming = camif_start_streaming,
+	.stop_streaming  = camif_stop_streaming,
+};
+
+/*
+ * File operations
+ */
+static int camif_open(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	int ret;
+
+	if (mutex_lock_interruptible(&priv->mutex))
+		return -ERESTARTSYS;
+
+	ret = v4l2_fh_open(file);
+	if (ret)
+		v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
+
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static int camif_release(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	int ret = 0;
+
+	mutex_lock(&priv->mutex);
+
+	if (file->private_data == vq->owner) {
+		vb2_queue_release(vq);
+		vq->owner = NULL;
+	}
+
+	v4l2_fh_release(file);
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static const struct v4l2_file_operations camif_fops = {
+	.owner		= THIS_MODULE,
+	.open		= camif_open,
+	.release	= camif_release,
+	.poll		= vb2_fop_poll,
+	.unlocked_ioctl	= video_ioctl2,
+	.mmap		= vb2_fop_mmap,
+};
+
+static struct video_device camif_videodev = {
+	.fops		= &camif_fops,
+	.ioctl_ops	= &camif_ioctl_ops,
+	.minor		= -1,
+	.release	= video_device_release,
+	.vfl_dir	= VFL_DIR_RX,
+	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
+};
+
+/*
+ * Subdev and media entity operations
+ */
+
+static void camif_new_dma_buf(struct camif_priv *priv)
+{
+	struct imx_media_dma_buf *dmabuf;
+	struct imx_media_buffer *buf;
+	enum vb2_buffer_state state;
+	struct vb2_buffer *vb;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	if (priv->stop || list_empty(&priv->ready_q))
+		goto unlock;
+
+	dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
+	if (!dmabuf)
+		goto unlock;
+
+	vb = dmabuf->vb;
+	buf = to_imx_media_vb(vb);
+	if (list_empty(&buf->list)) {
+		dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
+			vb->index);
+		goto unlock;
+	}
+
+	dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
+	vb->timestamp = ktime_get_ns();
+	list_del_init(&buf->list);
+	state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
+		VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
+	vb2_buffer_done(vb, state);
+unlock:
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_dma_buf_ring **ring;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		camif_new_dma_buf(priv);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int camif_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_subdev *remote_sd;
+	int ret = 0;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (is_media_entity_v4l2_video_device(remote->entity))
+		return 0;
+
+	WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	/* reset controls to refresh with inherited from subdevs */
+	v4l2_ctrl_handler_free(vfd->ctrl_handler);
+	v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+		ret = imx_media_inherit_controls(priv->md, vfd,
+						 &priv->src_sd->entity);
+	} else {
+		priv->src_sd = NULL;
+	}
+
+	return ret;
+}
+
+static int camif_enum_mbus_code(struct v4l2_subdev *sd,
+				struct v4l2_subdev_pad_config *cfg,
+				struct v4l2_subdev_mbus_code_enum *code)
+{
+	if (code->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	return imx_media_enum_format(&code->code, code->index, true, true);
+}
+
+static int camif_get_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int camif_set_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	/* Output pad mirrors input pad, no limitations on input pads */
+	if (sdformat->pad == priv->output_pad)
+		sdformat->format = priv->format_mbus[priv->input_pad];
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int camif_init_pads(struct camif_priv *priv)
+{
+	struct video_device *vfd = &priv->vfd;
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+		v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
+			 imxsd->num_sink_pads, imxsd->num_src_pads);
+		return -EINVAL;
+	}
+
+	priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
+	ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
+	if (ret) {
+		v4l2_err(&priv->sd, "failed to init device node pad\n");
+		return ret;
+	}
+
+	for (i = 0; i < CAMIF_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
+				      priv->pad);
+}
+
+static int camif_registered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	int ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	vfd->v4l2_dev = sd->v4l2_dev;
+
+	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+	if (ret) {
+		v4l2_err(sd, "Failed to register video device\n");
+		return ret;
+	}
+
+	vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	vq->io_modes = VB2_MMAP | VB2_DMABUF;
+	vq->drv_priv = priv;
+	vq->buf_struct_size = sizeof(struct imx_media_buffer);
+	vq->ops = &camif_qops;
+	vq->mem_ops = &vb2_dma_contig_memops;
+	vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	vq->lock = &priv->mutex;
+	vq->min_buffers_needed = 2;
+	vq->dev = priv->dev;
+
+	ret = vb2_queue_init(vq);
+	if (ret) {
+		v4l2_err(sd, "vb2_queue_init failed\n");
+		goto unreg;
+	}
+
+	INIT_LIST_HEAD(&priv->ready_q);
+
+	ret = camif_init_pads(priv);
+	if (ret) {
+		v4l2_err(sd, "camif_init_pads failed\n");
+		goto unreg;
+	}
+
+	/* create the link to our device node */
+	ret = media_create_pad_link(&sd->entity, priv->output_pad,
+				    &vfd->entity, 0,
+				    MEDIA_LNK_FL_IMMUTABLE |
+				    MEDIA_LNK_FL_ENABLED);
+	if (ret) {
+		v4l2_err(sd, "failed to create link to device node\n");
+		goto unreg;
+	}
+
+	/* setup default pad formats */
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
+				      &priv->cc[priv->output_pad]);
+	if (ret)
+		goto unreg;
+
+	*infmt = *outfmt;
+	priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
+
+	priv->current_std = V4L2_STD_UNKNOWN;
+
+	v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
+		  video_device_node_name(vfd));
+
+	vfd->ctrl_handler = &priv->ctrl_hdlr;
+
+	return 0;
+unreg:
+	video_unregister_device(vfd);
+	return ret;
+}
+
+static void camif_unregistered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+
+	mutex_lock(&priv->mutex);
+
+	if (video_is_registered(vfd)) {
+		video_unregister_device(vfd);
+		media_entity_cleanup(&vfd->entity);
+	}
+
+	mutex_unlock(&priv->mutex);
+}
+
+static const struct v4l2_subdev_internal_ops camif_internal_ops = {
+	.registered = camif_registered,
+	.unregistered = camif_unregistered,
+};
+
+static struct v4l2_subdev_core_ops camif_core_ops = {
+	.ioctl = camif_ioctl,
+};
+
+static struct media_entity_operations camif_entity_ops = {
+	.link_setup = camif_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_pad_ops camif_pad_ops = {
+	.enum_mbus_code = camif_enum_mbus_code,
+	.get_fmt = camif_get_fmt,
+	.set_fmt = camif_set_fmt,
+};
+
+static struct v4l2_subdev_ops camif_subdev_ops = {
+	.pad = &camif_pad_ops,
+	.core = &camif_core_ops,
+};
+
+static int camif_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct camif_priv *priv;
+	struct video_device *vfd;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, priv);
+	priv->dev = &pdev->dev;
+
+	pdata = priv->dev->platform_data;
+
+	mutex_init(&priv->mutex);
+	spin_lock_init(&priv->q_lock);
+
+	v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &camif_internal_ops;
+	priv->sd.entity.ops = &camif_entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	/* get our group id and camif id */
+	priv->sd.grp_id = pdata->grp_id;
+	priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+	snprintf(camif_videodev.name, sizeof(camif_videodev.name),
+		 "%s devnode", pdata->sd_name);
+
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+
+	vfd = &priv->vfd;
+	*vfd = camif_videodev;
+	vfd->lock = &priv->mutex;
+	vfd->queue = &priv->buffer_queue;
+
+	video_set_drvdata(vfd, priv);
+
+	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static int camif_remove(struct platform_device *pdev)
+{
+	struct camif_priv *priv =
+		(struct camif_priv *)platform_get_drvdata(pdev);
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(&priv->sd);
+
+	return 0;
+}
+
+static const struct platform_device_id camif_ids[] = {
+	{ .name = "imx-media-camif" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, camif_ids);
+
+static struct platform_driver imx_camif_driver = {
+	.probe		= camif_probe,
+	.remove		= camif_remove,
+	.driver		= {
+		.name	= "imx-media-camif",
+	},
+};
+
+module_platform_driver(imx_camif_driver);
+
+MODULE_DESCRIPTION("i.MX camera interface subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

This is the camera interface driver that provides the v4l2
user interface. Frames can be received from various sources:

- directly from SMFC for capturing unconverted images directly from
  camera sensors.

- from the IC pre-process encode task.

- from the IC pre-process viewfinder task.

- from the IC post-process task.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile    |    2 +-
 drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
 2 files changed, 1001 insertions(+), 1 deletion(-)
 create mode 100644 drivers/staging/media/imx/imx-camif.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index d2a962c..fe9e992 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
new file mode 100644
index 0000000..404f724
--- /dev/null
+++ b/drivers/staging/media/imx/imx-camif.c
@@ -0,0 +1,1000 @@
+/*
+ * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+#define CAMIF_NUM_PADS 2
+
+struct camif_priv {
+	struct device         *dev;
+	struct video_device    vfd;
+	struct media_pipeline  mp;
+	struct imx_media_dev  *md;
+	struct v4l2_subdev     sd;
+	struct media_pad       pad[CAMIF_NUM_PADS];
+	struct media_pad       vd_pad;
+	int id;
+	int input_pad;
+	int output_pad;
+
+	struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
+	const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
+
+	/* dma buffer ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	struct v4l2_subdev     *src_sd;
+
+	struct mutex           mutex;       /* capture device mutex */
+	spinlock_t             q_lock;      /* protect ready_q */
+
+	/* buffer queue used in videobuf2 */
+	struct vb2_queue       buffer_queue;
+
+	/* streaming buffer queue */
+	struct list_head       ready_q;
+
+	/* controls inherited from subdevs */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+
+	/* misc status */
+	int                    current_input; /* the current input */
+	v4l2_std_id            current_std;   /* current standard */
+	bool                   stop;          /* streaming is stopping */
+};
+
+/* In bytes, per queue */
+#define VID_MEM_LIMIT	SZ_64M
+
+static struct vb2_ops camif_qops;
+
+/*
+ * Video ioctls follow
+ */
+
+static int vidioc_querycap(struct file *file, void *fh,
+			   struct v4l2_capability *cap)
+{
+	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
+	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
+	cap->bus_info[0] = 0;
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
+
+	return 0;
+}
+
+static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
+				  struct v4l2_fmtdesc *f)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+	int ret;
+
+	ret = imx_media_enum_format(&code, f->index, true, true);
+	if (ret)
+		return ret;
+	cc = imx_media_find_format(0, code, true, true);
+	if (!cc)
+		return -EINVAL;
+
+	f->pixelformat = cc->fourcc;
+
+	return 0;
+}
+
+static int camif_g_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct v4l2_mbus_framefmt *outfmt;
+
+	/* user format is the same as the format from output pad */
+	outfmt = &priv->format_mbus[priv->output_pad];
+	return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
+}
+
+static int camif_try_fmt_vid_cap(struct file *file, void *fh,
+				 struct v4l2_format *f)
+{
+	return camif_g_fmt_vid_cap(file, fh, f);
+}
+
+static int camif_s_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	return camif_try_fmt_vid_cap(file, priv, f);
+}
+
+static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	return v4l2_subdev_call(sensor->sd, video, querystd, std);
+}
+
+static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*std = priv->current_std;
+	return 0;
+}
+
+static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (vb2_is_busy(&priv->buffer_queue))
+		return -EBUSY;
+
+	ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
+	if (ret < 0)
+		return ret;
+
+	priv->current_std = std;
+	return 0;
+}
+
+static int camif_enum_input(struct file *file, void *fh,
+			    struct v4l2_input *input)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int index = input->index;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	input->type = V4L2_INPUT_TYPE_CAMERA;
+	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+	if (index == priv->current_input) {
+		v4l2_subdev_call(sensor->sd, video, g_input_status,
+				 &input->status);
+		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
+	} else {
+		input->status = V4L2_IN_ST_NO_SIGNAL;
+		input->std = V4L2_STD_UNKNOWN;
+	}
+
+	return 0;
+}
+
+static int camif_g_input(struct file *file, void *fh, unsigned int *index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*index = priv->current_input;
+	return 0;
+}
+
+static int camif_s_input(struct file *file, void *fh, unsigned int index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	if (index == priv->current_input)
+		return 0;
+
+	/* select the sensor's input */
+	ret = v4l2_subdev_call(sensor->sd, video, s_routing,
+			       sensor->input.value[index], 0, 0);
+	if (!ret)
+		priv->current_input = index;
+
+	return ret;
+}
+
+static int camif_g_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, g_parm, a);
+}
+
+static int camif_s_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, s_parm, a);
+}
+
+static const struct v4l2_ioctl_ops camif_ioctl_ops = {
+	.vidioc_querycap	= vidioc_querycap,
+
+	.vidioc_enum_fmt_vid_cap        = camif_enum_fmt_vid_cap,
+	.vidioc_g_fmt_vid_cap           = camif_g_fmt_vid_cap,
+	.vidioc_try_fmt_vid_cap         = camif_try_fmt_vid_cap,
+	.vidioc_s_fmt_vid_cap           = camif_s_fmt_vid_cap,
+
+	.vidioc_querystd        = camif_querystd,
+	.vidioc_g_std           = camif_g_std,
+	.vidioc_s_std           = camif_s_std,
+
+	.vidioc_enum_input      = camif_enum_input,
+	.vidioc_g_input         = camif_g_input,
+	.vidioc_s_input         = camif_s_input,
+
+	.vidioc_g_parm          = camif_g_parm,
+	.vidioc_s_parm          = camif_s_parm,
+
+	.vidioc_reqbufs		= vb2_ioctl_reqbufs,
+	.vidioc_create_bufs     = vb2_ioctl_create_bufs,
+	.vidioc_prepare_buf     = vb2_ioctl_prepare_buf,
+	.vidioc_querybuf	= vb2_ioctl_querybuf,
+	.vidioc_qbuf		= vb2_ioctl_qbuf,
+	.vidioc_dqbuf		= vb2_ioctl_dqbuf,
+	.vidioc_expbuf		= vb2_ioctl_expbuf,
+	.vidioc_streamon	= vb2_ioctl_streamon,
+	.vidioc_streamoff	= vb2_ioctl_streamoff,
+};
+
+/*
+ * Queue operations
+ */
+
+static u32 camif_get_sizeimage(struct camif_priv *priv)
+{
+	struct v4l2_mbus_framefmt *outfmt;
+	const struct imx_media_pixfmt *outcc;
+
+	outfmt = &priv->format_mbus[priv->output_pad];
+	outcc = priv->cc[priv->output_pad];
+	return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
+}
+
+static int camif_queue_setup(struct vb2_queue *vq,
+			     unsigned int *nbuffers, unsigned int *nplanes,
+			     unsigned int sizes[], struct device *alloc_devs[])
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	unsigned int count = *nbuffers;
+	u32 sizeimage;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	sizeimage = camif_get_sizeimage(priv);
+
+	count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
+	count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
+
+	while (sizeimage * count > VID_MEM_LIMIT)
+		count--;
+
+	if (count < IMX_MEDIA_MIN_RING_BUFS)
+		return -EINVAL;
+
+	*nplanes = 1;
+	*nbuffers = count;
+	sizes[0] = sizeimage;
+
+	return 0;
+}
+
+static int camif_buf_init(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	INIT_LIST_HEAD(&buf->list);
+
+	return 0;
+}
+
+static int camif_buf_prepare(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	int ret;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vb2_plane_size(vb, 0) < sizeimage) {
+		v4l2_err(&priv->sd,
+			 "data will not fit into plane (%lu < %lu)\n",
+			 vb2_plane_size(vb, 0), (long)sizeimage);
+		return -EINVAL;
+	}
+
+	vb2_set_plane_payload(vb, 0, sizeimage);
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			return ret;
+		}
+	}
+
+	ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
+	if (ret)
+		goto free_ring;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+	return ret;
+}
+
+static void camif_buf_queue(struct vb2_buffer *vb)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	list_add_tail(&buf->list, &priv->ready_q);
+
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	struct imx_media_buffer *buf, *tmp;
+	unsigned long flags;
+	int ret;
+
+	if (vb2_is_streaming(vq))
+		return 0;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			goto return_bufs;
+		}
+	}
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, true);
+	if (ret) {
+		v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
+			 ret);
+		goto free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+return_bufs:
+	spin_lock_irqsave(&priv->q_lock, flags);
+	list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
+		list_del(&buf->list);
+		vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+	return ret;
+}
+
+static void camif_stop_streaming(struct vb2_queue *vq)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *frame;
+	unsigned long flags;
+	int ret;
+
+	if (!vb2_is_streaming(vq))
+		return;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, false);
+	if (ret)
+		v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
+			  ret);
+
+	if (priv->in_ring) {
+		v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+	}
+
+	/* release all active buffers */
+	spin_lock_irqsave(&priv->q_lock, flags);
+	while (!list_empty(&priv->ready_q)) {
+		frame = list_entry(priv->ready_q.next,
+				   struct imx_media_buffer, list);
+		list_del(&frame->list);
+		vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static struct vb2_ops camif_qops = {
+	.queue_setup	 = camif_queue_setup,
+	.buf_init        = camif_buf_init,
+	.buf_prepare	 = camif_buf_prepare,
+	.buf_queue	 = camif_buf_queue,
+	.wait_prepare	 = vb2_ops_wait_prepare,
+	.wait_finish	 = vb2_ops_wait_finish,
+	.start_streaming = camif_start_streaming,
+	.stop_streaming  = camif_stop_streaming,
+};
+
+/*
+ * File operations
+ */
+static int camif_open(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	int ret;
+
+	if (mutex_lock_interruptible(&priv->mutex))
+		return -ERESTARTSYS;
+
+	ret = v4l2_fh_open(file);
+	if (ret)
+		v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
+
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static int camif_release(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	int ret = 0;
+
+	mutex_lock(&priv->mutex);
+
+	if (file->private_data == vq->owner) {
+		vb2_queue_release(vq);
+		vq->owner = NULL;
+	}
+
+	v4l2_fh_release(file);
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static const struct v4l2_file_operations camif_fops = {
+	.owner		= THIS_MODULE,
+	.open		= camif_open,
+	.release	= camif_release,
+	.poll		= vb2_fop_poll,
+	.unlocked_ioctl	= video_ioctl2,
+	.mmap		= vb2_fop_mmap,
+};
+
+static struct video_device camif_videodev = {
+	.fops		= &camif_fops,
+	.ioctl_ops	= &camif_ioctl_ops,
+	.minor		= -1,
+	.release	= video_device_release,
+	.vfl_dir	= VFL_DIR_RX,
+	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
+};
+
+/*
+ * Subdev and media entity operations
+ */
+
+static void camif_new_dma_buf(struct camif_priv *priv)
+{
+	struct imx_media_dma_buf *dmabuf;
+	struct imx_media_buffer *buf;
+	enum vb2_buffer_state state;
+	struct vb2_buffer *vb;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	if (priv->stop || list_empty(&priv->ready_q))
+		goto unlock;
+
+	dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
+	if (!dmabuf)
+		goto unlock;
+
+	vb = dmabuf->vb;
+	buf = to_imx_media_vb(vb);
+	if (list_empty(&buf->list)) {
+		dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
+			vb->index);
+		goto unlock;
+	}
+
+	dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
+	vb->timestamp = ktime_get_ns();
+	list_del_init(&buf->list);
+	state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
+		VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
+	vb2_buffer_done(vb, state);
+unlock:
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_dma_buf_ring **ring;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		camif_new_dma_buf(priv);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int camif_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_subdev *remote_sd;
+	int ret = 0;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (is_media_entity_v4l2_video_device(remote->entity))
+		return 0;
+
+	WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	/* reset controls to refresh with inherited from subdevs */
+	v4l2_ctrl_handler_free(vfd->ctrl_handler);
+	v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+		ret = imx_media_inherit_controls(priv->md, vfd,
+						 &priv->src_sd->entity);
+	} else {
+		priv->src_sd = NULL;
+	}
+
+	return ret;
+}
+
+static int camif_enum_mbus_code(struct v4l2_subdev *sd,
+				struct v4l2_subdev_pad_config *cfg,
+				struct v4l2_subdev_mbus_code_enum *code)
+{
+	if (code->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	return imx_media_enum_format(&code->code, code->index, true, true);
+}
+
+static int camif_get_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int camif_set_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	/* Output pad mirrors input pad, no limitations on input pads */
+	if (sdformat->pad == priv->output_pad)
+		sdformat->format = priv->format_mbus[priv->input_pad];
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int camif_init_pads(struct camif_priv *priv)
+{
+	struct video_device *vfd = &priv->vfd;
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+		v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
+			 imxsd->num_sink_pads, imxsd->num_src_pads);
+		return -EINVAL;
+	}
+
+	priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
+	ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
+	if (ret) {
+		v4l2_err(&priv->sd, "failed to init device node pad\n");
+		return ret;
+	}
+
+	for (i = 0; i < CAMIF_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
+				      priv->pad);
+}
+
+static int camif_registered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	int ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	vfd->v4l2_dev = sd->v4l2_dev;
+
+	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+	if (ret) {
+		v4l2_err(sd, "Failed to register video device\n");
+		return ret;
+	}
+
+	vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	vq->io_modes = VB2_MMAP | VB2_DMABUF;
+	vq->drv_priv = priv;
+	vq->buf_struct_size = sizeof(struct imx_media_buffer);
+	vq->ops = &camif_qops;
+	vq->mem_ops = &vb2_dma_contig_memops;
+	vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	vq->lock = &priv->mutex;
+	vq->min_buffers_needed = 2;
+	vq->dev = priv->dev;
+
+	ret = vb2_queue_init(vq);
+	if (ret) {
+		v4l2_err(sd, "vb2_queue_init failed\n");
+		goto unreg;
+	}
+
+	INIT_LIST_HEAD(&priv->ready_q);
+
+	ret = camif_init_pads(priv);
+	if (ret) {
+		v4l2_err(sd, "camif_init_pads failed\n");
+		goto unreg;
+	}
+
+	/* create the link to our device node */
+	ret = media_create_pad_link(&sd->entity, priv->output_pad,
+				    &vfd->entity, 0,
+				    MEDIA_LNK_FL_IMMUTABLE |
+				    MEDIA_LNK_FL_ENABLED);
+	if (ret) {
+		v4l2_err(sd, "failed to create link to device node\n");
+		goto unreg;
+	}
+
+	/* setup default pad formats */
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
+				      &priv->cc[priv->output_pad]);
+	if (ret)
+		goto unreg;
+
+	*infmt = *outfmt;
+	priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
+
+	priv->current_std = V4L2_STD_UNKNOWN;
+
+	v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
+		  video_device_node_name(vfd));
+
+	vfd->ctrl_handler = &priv->ctrl_hdlr;
+
+	return 0;
+unreg:
+	video_unregister_device(vfd);
+	return ret;
+}
+
+static void camif_unregistered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+
+	mutex_lock(&priv->mutex);
+
+	if (video_is_registered(vfd)) {
+		video_unregister_device(vfd);
+		media_entity_cleanup(&vfd->entity);
+	}
+
+	mutex_unlock(&priv->mutex);
+}
+
+static const struct v4l2_subdev_internal_ops camif_internal_ops = {
+	.registered = camif_registered,
+	.unregistered = camif_unregistered,
+};
+
+static struct v4l2_subdev_core_ops camif_core_ops = {
+	.ioctl = camif_ioctl,
+};
+
+static struct media_entity_operations camif_entity_ops = {
+	.link_setup = camif_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_pad_ops camif_pad_ops = {
+	.enum_mbus_code = camif_enum_mbus_code,
+	.get_fmt = camif_get_fmt,
+	.set_fmt = camif_set_fmt,
+};
+
+static struct v4l2_subdev_ops camif_subdev_ops = {
+	.pad = &camif_pad_ops,
+	.core = &camif_core_ops,
+};
+
+static int camif_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct camif_priv *priv;
+	struct video_device *vfd;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, priv);
+	priv->dev = &pdev->dev;
+
+	pdata = priv->dev->platform_data;
+
+	mutex_init(&priv->mutex);
+	spin_lock_init(&priv->q_lock);
+
+	v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &camif_internal_ops;
+	priv->sd.entity.ops = &camif_entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	/* get our group id and camif id */
+	priv->sd.grp_id = pdata->grp_id;
+	priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+	snprintf(camif_videodev.name, sizeof(camif_videodev.name),
+		 "%s devnode", pdata->sd_name);
+
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+
+	vfd = &priv->vfd;
+	*vfd = camif_videodev;
+	vfd->lock = &priv->mutex;
+	vfd->queue = &priv->buffer_queue;
+
+	video_set_drvdata(vfd, priv);
+
+	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static int camif_remove(struct platform_device *pdev)
+{
+	struct camif_priv *priv =
+		(struct camif_priv *)platform_get_drvdata(pdev);
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(&priv->sd);
+
+	return 0;
+}
+
+static const struct platform_device_id camif_ids[] = {
+	{ .name = "imx-media-camif" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, camif_ids);
+
+static struct platform_driver imx_camif_driver = {
+	.probe		= camif_probe,
+	.remove		= camif_remove,
+	.driver		= {
+		.name	= "imx-media-camif",
+	},
+};
+
+module_platform_driver(imx_camif_driver);
+
+MODULE_DESCRIPTION("i.MX camera interface subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

This is the camera interface driver that provides the v4l2
user interface. Frames can be received from various sources:

- directly from SMFC for capturing unconverted images directly from
  camera sensors.

- from the IC pre-process encode task.

- from the IC pre-process viewfinder task.

- from the IC post-process task.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile    |    2 +-
 drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
 2 files changed, 1001 insertions(+), 1 deletion(-)
 create mode 100644 drivers/staging/media/imx/imx-camif.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index d2a962c..fe9e992 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
new file mode 100644
index 0000000..404f724
--- /dev/null
+++ b/drivers/staging/media/imx/imx-camif.c
@@ -0,0 +1,1000 @@
+/*
+ * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+#define CAMIF_NUM_PADS 2
+
+struct camif_priv {
+	struct device         *dev;
+	struct video_device    vfd;
+	struct media_pipeline  mp;
+	struct imx_media_dev  *md;
+	struct v4l2_subdev     sd;
+	struct media_pad       pad[CAMIF_NUM_PADS];
+	struct media_pad       vd_pad;
+	int id;
+	int input_pad;
+	int output_pad;
+
+	struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
+	const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
+
+	/* dma buffer ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	struct v4l2_subdev     *src_sd;
+
+	struct mutex           mutex;       /* capture device mutex */
+	spinlock_t             q_lock;      /* protect ready_q */
+
+	/* buffer queue used in videobuf2 */
+	struct vb2_queue       buffer_queue;
+
+	/* streaming buffer queue */
+	struct list_head       ready_q;
+
+	/* controls inherited from subdevs */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+
+	/* misc status */
+	int                    current_input; /* the current input */
+	v4l2_std_id            current_std;   /* current standard */
+	bool                   stop;          /* streaming is stopping */
+};
+
+/* In bytes, per queue */
+#define VID_MEM_LIMIT	SZ_64M
+
+static struct vb2_ops camif_qops;
+
+/*
+ * Video ioctls follow
+ */
+
+static int vidioc_querycap(struct file *file, void *fh,
+			   struct v4l2_capability *cap)
+{
+	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
+	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
+	cap->bus_info[0] = 0;
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
+
+	return 0;
+}
+
+static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
+				  struct v4l2_fmtdesc *f)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+	int ret;
+
+	ret = imx_media_enum_format(&code, f->index, true, true);
+	if (ret)
+		return ret;
+	cc = imx_media_find_format(0, code, true, true);
+	if (!cc)
+		return -EINVAL;
+
+	f->pixelformat = cc->fourcc;
+
+	return 0;
+}
+
+static int camif_g_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct v4l2_mbus_framefmt *outfmt;
+
+	/* user format is the same as the format from output pad */
+	outfmt = &priv->format_mbus[priv->output_pad];
+	return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
+}
+
+static int camif_try_fmt_vid_cap(struct file *file, void *fh,
+				 struct v4l2_format *f)
+{
+	return camif_g_fmt_vid_cap(file, fh, f);
+}
+
+static int camif_s_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	return camif_try_fmt_vid_cap(file, priv, f);
+}
+
+static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	return v4l2_subdev_call(sensor->sd, video, querystd, std);
+}
+
+static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*std = priv->current_std;
+	return 0;
+}
+
+static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (vb2_is_busy(&priv->buffer_queue))
+		return -EBUSY;
+
+	ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
+	if (ret < 0)
+		return ret;
+
+	priv->current_std = std;
+	return 0;
+}
+
+static int camif_enum_input(struct file *file, void *fh,
+			    struct v4l2_input *input)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int index = input->index;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	input->type = V4L2_INPUT_TYPE_CAMERA;
+	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+	if (index == priv->current_input) {
+		v4l2_subdev_call(sensor->sd, video, g_input_status,
+				 &input->status);
+		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
+	} else {
+		input->status = V4L2_IN_ST_NO_SIGNAL;
+		input->std = V4L2_STD_UNKNOWN;
+	}
+
+	return 0;
+}
+
+static int camif_g_input(struct file *file, void *fh, unsigned int *index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*index = priv->current_input;
+	return 0;
+}
+
+static int camif_s_input(struct file *file, void *fh, unsigned int index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	if (index == priv->current_input)
+		return 0;
+
+	/* select the sensor's input */
+	ret = v4l2_subdev_call(sensor->sd, video, s_routing,
+			       sensor->input.value[index], 0, 0);
+	if (!ret)
+		priv->current_input = index;
+
+	return ret;
+}
+
+static int camif_g_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, g_parm, a);
+}
+
+static int camif_s_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, s_parm, a);
+}
+
+static const struct v4l2_ioctl_ops camif_ioctl_ops = {
+	.vidioc_querycap	= vidioc_querycap,
+
+	.vidioc_enum_fmt_vid_cap        = camif_enum_fmt_vid_cap,
+	.vidioc_g_fmt_vid_cap           = camif_g_fmt_vid_cap,
+	.vidioc_try_fmt_vid_cap         = camif_try_fmt_vid_cap,
+	.vidioc_s_fmt_vid_cap           = camif_s_fmt_vid_cap,
+
+	.vidioc_querystd        = camif_querystd,
+	.vidioc_g_std           = camif_g_std,
+	.vidioc_s_std           = camif_s_std,
+
+	.vidioc_enum_input      = camif_enum_input,
+	.vidioc_g_input         = camif_g_input,
+	.vidioc_s_input         = camif_s_input,
+
+	.vidioc_g_parm          = camif_g_parm,
+	.vidioc_s_parm          = camif_s_parm,
+
+	.vidioc_reqbufs		= vb2_ioctl_reqbufs,
+	.vidioc_create_bufs     = vb2_ioctl_create_bufs,
+	.vidioc_prepare_buf     = vb2_ioctl_prepare_buf,
+	.vidioc_querybuf	= vb2_ioctl_querybuf,
+	.vidioc_qbuf		= vb2_ioctl_qbuf,
+	.vidioc_dqbuf		= vb2_ioctl_dqbuf,
+	.vidioc_expbuf		= vb2_ioctl_expbuf,
+	.vidioc_streamon	= vb2_ioctl_streamon,
+	.vidioc_streamoff	= vb2_ioctl_streamoff,
+};
+
+/*
+ * Queue operations
+ */
+
+static u32 camif_get_sizeimage(struct camif_priv *priv)
+{
+	struct v4l2_mbus_framefmt *outfmt;
+	const struct imx_media_pixfmt *outcc;
+
+	outfmt = &priv->format_mbus[priv->output_pad];
+	outcc = priv->cc[priv->output_pad];
+	return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
+}
+
+static int camif_queue_setup(struct vb2_queue *vq,
+			     unsigned int *nbuffers, unsigned int *nplanes,
+			     unsigned int sizes[], struct device *alloc_devs[])
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	unsigned int count = *nbuffers;
+	u32 sizeimage;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	sizeimage = camif_get_sizeimage(priv);
+
+	count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
+	count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
+
+	while (sizeimage * count > VID_MEM_LIMIT)
+		count--;
+
+	if (count < IMX_MEDIA_MIN_RING_BUFS)
+		return -EINVAL;
+
+	*nplanes = 1;
+	*nbuffers = count;
+	sizes[0] = sizeimage;
+
+	return 0;
+}
+
+static int camif_buf_init(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	INIT_LIST_HEAD(&buf->list);
+
+	return 0;
+}
+
+static int camif_buf_prepare(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	int ret;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vb2_plane_size(vb, 0) < sizeimage) {
+		v4l2_err(&priv->sd,
+			 "data will not fit into plane (%lu < %lu)\n",
+			 vb2_plane_size(vb, 0), (long)sizeimage);
+		return -EINVAL;
+	}
+
+	vb2_set_plane_payload(vb, 0, sizeimage);
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			return ret;
+		}
+	}
+
+	ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
+	if (ret)
+		goto free_ring;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+	return ret;
+}
+
+static void camif_buf_queue(struct vb2_buffer *vb)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	list_add_tail(&buf->list, &priv->ready_q);
+
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	struct imx_media_buffer *buf, *tmp;
+	unsigned long flags;
+	int ret;
+
+	if (vb2_is_streaming(vq))
+		return 0;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			goto return_bufs;
+		}
+	}
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, true);
+	if (ret) {
+		v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
+			 ret);
+		goto free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+return_bufs:
+	spin_lock_irqsave(&priv->q_lock, flags);
+	list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
+		list_del(&buf->list);
+		vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+	return ret;
+}
+
+static void camif_stop_streaming(struct vb2_queue *vq)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *frame;
+	unsigned long flags;
+	int ret;
+
+	if (!vb2_is_streaming(vq))
+		return;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, false);
+	if (ret)
+		v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
+			  ret);
+
+	if (priv->in_ring) {
+		v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+	}
+
+	/* release all active buffers */
+	spin_lock_irqsave(&priv->q_lock, flags);
+	while (!list_empty(&priv->ready_q)) {
+		frame = list_entry(priv->ready_q.next,
+				   struct imx_media_buffer, list);
+		list_del(&frame->list);
+		vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static struct vb2_ops camif_qops = {
+	.queue_setup	 = camif_queue_setup,
+	.buf_init        = camif_buf_init,
+	.buf_prepare	 = camif_buf_prepare,
+	.buf_queue	 = camif_buf_queue,
+	.wait_prepare	 = vb2_ops_wait_prepare,
+	.wait_finish	 = vb2_ops_wait_finish,
+	.start_streaming = camif_start_streaming,
+	.stop_streaming  = camif_stop_streaming,
+};
+
+/*
+ * File operations
+ */
+static int camif_open(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	int ret;
+
+	if (mutex_lock_interruptible(&priv->mutex))
+		return -ERESTARTSYS;
+
+	ret = v4l2_fh_open(file);
+	if (ret)
+		v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
+
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static int camif_release(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	int ret = 0;
+
+	mutex_lock(&priv->mutex);
+
+	if (file->private_data == vq->owner) {
+		vb2_queue_release(vq);
+		vq->owner = NULL;
+	}
+
+	v4l2_fh_release(file);
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static const struct v4l2_file_operations camif_fops = {
+	.owner		= THIS_MODULE,
+	.open		= camif_open,
+	.release	= camif_release,
+	.poll		= vb2_fop_poll,
+	.unlocked_ioctl	= video_ioctl2,
+	.mmap		= vb2_fop_mmap,
+};
+
+static struct video_device camif_videodev = {
+	.fops		= &camif_fops,
+	.ioctl_ops	= &camif_ioctl_ops,
+	.minor		= -1,
+	.release	= video_device_release,
+	.vfl_dir	= VFL_DIR_RX,
+	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
+};
+
+/*
+ * Subdev and media entity operations
+ */
+
+static void camif_new_dma_buf(struct camif_priv *priv)
+{
+	struct imx_media_dma_buf *dmabuf;
+	struct imx_media_buffer *buf;
+	enum vb2_buffer_state state;
+	struct vb2_buffer *vb;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	if (priv->stop || list_empty(&priv->ready_q))
+		goto unlock;
+
+	dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
+	if (!dmabuf)
+		goto unlock;
+
+	vb = dmabuf->vb;
+	buf = to_imx_media_vb(vb);
+	if (list_empty(&buf->list)) {
+		dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
+			vb->index);
+		goto unlock;
+	}
+
+	dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
+	vb->timestamp = ktime_get_ns();
+	list_del_init(&buf->list);
+	state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
+		VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
+	vb2_buffer_done(vb, state);
+unlock:
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_dma_buf_ring **ring;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		camif_new_dma_buf(priv);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int camif_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_subdev *remote_sd;
+	int ret = 0;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (is_media_entity_v4l2_video_device(remote->entity))
+		return 0;
+
+	WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	/* reset controls to refresh with inherited from subdevs */
+	v4l2_ctrl_handler_free(vfd->ctrl_handler);
+	v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+		ret = imx_media_inherit_controls(priv->md, vfd,
+						 &priv->src_sd->entity);
+	} else {
+		priv->src_sd = NULL;
+	}
+
+	return ret;
+}
+
+static int camif_enum_mbus_code(struct v4l2_subdev *sd,
+				struct v4l2_subdev_pad_config *cfg,
+				struct v4l2_subdev_mbus_code_enum *code)
+{
+	if (code->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	return imx_media_enum_format(&code->code, code->index, true, true);
+}
+
+static int camif_get_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int camif_set_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	/* Output pad mirrors input pad, no limitations on input pads */
+	if (sdformat->pad == priv->output_pad)
+		sdformat->format = priv->format_mbus[priv->input_pad];
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int camif_init_pads(struct camif_priv *priv)
+{
+	struct video_device *vfd = &priv->vfd;
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+		v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
+			 imxsd->num_sink_pads, imxsd->num_src_pads);
+		return -EINVAL;
+	}
+
+	priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
+	ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
+	if (ret) {
+		v4l2_err(&priv->sd, "failed to init device node pad\n");
+		return ret;
+	}
+
+	for (i = 0; i < CAMIF_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
+				      priv->pad);
+}
+
+static int camif_registered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	int ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	vfd->v4l2_dev = sd->v4l2_dev;
+
+	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+	if (ret) {
+		v4l2_err(sd, "Failed to register video device\n");
+		return ret;
+	}
+
+	vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	vq->io_modes = VB2_MMAP | VB2_DMABUF;
+	vq->drv_priv = priv;
+	vq->buf_struct_size = sizeof(struct imx_media_buffer);
+	vq->ops = &camif_qops;
+	vq->mem_ops = &vb2_dma_contig_memops;
+	vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	vq->lock = &priv->mutex;
+	vq->min_buffers_needed = 2;
+	vq->dev = priv->dev;
+
+	ret = vb2_queue_init(vq);
+	if (ret) {
+		v4l2_err(sd, "vb2_queue_init failed\n");
+		goto unreg;
+	}
+
+	INIT_LIST_HEAD(&priv->ready_q);
+
+	ret = camif_init_pads(priv);
+	if (ret) {
+		v4l2_err(sd, "camif_init_pads failed\n");
+		goto unreg;
+	}
+
+	/* create the link to our device node */
+	ret = media_create_pad_link(&sd->entity, priv->output_pad,
+				    &vfd->entity, 0,
+				    MEDIA_LNK_FL_IMMUTABLE |
+				    MEDIA_LNK_FL_ENABLED);
+	if (ret) {
+		v4l2_err(sd, "failed to create link to device node\n");
+		goto unreg;
+	}
+
+	/* setup default pad formats */
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
+				      &priv->cc[priv->output_pad]);
+	if (ret)
+		goto unreg;
+
+	*infmt = *outfmt;
+	priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
+
+	priv->current_std = V4L2_STD_UNKNOWN;
+
+	v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
+		  video_device_node_name(vfd));
+
+	vfd->ctrl_handler = &priv->ctrl_hdlr;
+
+	return 0;
+unreg:
+	video_unregister_device(vfd);
+	return ret;
+}
+
+static void camif_unregistered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+
+	mutex_lock(&priv->mutex);
+
+	if (video_is_registered(vfd)) {
+		video_unregister_device(vfd);
+		media_entity_cleanup(&vfd->entity);
+	}
+
+	mutex_unlock(&priv->mutex);
+}
+
+static const struct v4l2_subdev_internal_ops camif_internal_ops = {
+	.registered = camif_registered,
+	.unregistered = camif_unregistered,
+};
+
+static struct v4l2_subdev_core_ops camif_core_ops = {
+	.ioctl = camif_ioctl,
+};
+
+static struct media_entity_operations camif_entity_ops = {
+	.link_setup = camif_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_pad_ops camif_pad_ops = {
+	.enum_mbus_code = camif_enum_mbus_code,
+	.get_fmt = camif_get_fmt,
+	.set_fmt = camif_set_fmt,
+};
+
+static struct v4l2_subdev_ops camif_subdev_ops = {
+	.pad = &camif_pad_ops,
+	.core = &camif_core_ops,
+};
+
+static int camif_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct camif_priv *priv;
+	struct video_device *vfd;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, priv);
+	priv->dev = &pdev->dev;
+
+	pdata = priv->dev->platform_data;
+
+	mutex_init(&priv->mutex);
+	spin_lock_init(&priv->q_lock);
+
+	v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &camif_internal_ops;
+	priv->sd.entity.ops = &camif_entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	/* get our group id and camif id */
+	priv->sd.grp_id = pdata->grp_id;
+	priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+	snprintf(camif_videodev.name, sizeof(camif_videodev.name),
+		 "%s devnode", pdata->sd_name);
+
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+
+	vfd = &priv->vfd;
+	*vfd = camif_videodev;
+	vfd->lock = &priv->mutex;
+	vfd->queue = &priv->buffer_queue;
+
+	video_set_drvdata(vfd, priv);
+
+	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static int camif_remove(struct platform_device *pdev)
+{
+	struct camif_priv *priv =
+		(struct camif_priv *)platform_get_drvdata(pdev);
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(&priv->sd);
+
+	return 0;
+}
+
+static const struct platform_device_id camif_ids[] = {
+	{ .name = "imx-media-camif" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, camif_ids);
+
+static struct platform_driver imx_camif_driver = {
+	.probe		= camif_probe,
+	.remove		= camif_remove,
+	.driver		= {
+		.name	= "imx-media-camif",
+	},
+};
+
+module_platform_driver(imx_camif_driver);
+
+MODULE_DESCRIPTION("i.MX camera interface subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-07  2:11   ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
for sensors with a MIPI CSI2 interface.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile        |   1 +
 drivers/staging/media/imx/imx-mipi-csi2.c | 501 ++++++++++++++++++++++++++++++
 2 files changed, 502 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index fe9e992..0decef7 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
new file mode 100644
index 0000000..daa6e1d
--- /dev/null
+++ b/drivers/staging/media/imx/imx-mipi-csi2.c
@@ -0,0 +1,501 @@
+/*
+ * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
+ *
+ * Copyright (c) 2012-2014 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+/*
+ * there must be 5 pads: 1 input pad from sensor, and
+ * the 4 virtual channel output pads
+ */
+#define CSI2_NUM_SINK_PADS  1
+#define CSI2_NUM_SRC_PADS   4
+#define CSI2_NUM_PADS       5
+
+struct imxcsi2_dev {
+	struct device          *dev;
+	struct imx_media_dev   *md;
+	struct v4l2_subdev      sd;
+	struct media_pad       pad[CSI2_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus;
+	struct v4l2_subdev     *src_sd;
+	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];
+	int                    input_pad;
+	struct clk             *dphy_clk;
+	struct clk             *cfg_clk;
+	struct clk             *pix_clk; /* what is this? */
+	void __iomem           *base;
+	int                     intr1;
+	int                     intr2;
+	struct v4l2_of_bus_mipi_csi2 bus;
+	bool                    on;
+	bool                    stream_on;
+};
+
+#define DEVICE_NAME "imx6-mipi-csi2"
+
+/* Register offsets */
+#define CSI2_VERSION            0x000
+#define CSI2_N_LANES            0x004
+#define CSI2_PHY_SHUTDOWNZ      0x008
+#define CSI2_DPHY_RSTZ          0x00c
+#define CSI2_RESETN             0x010
+#define CSI2_PHY_STATE          0x014
+#define CSI2_DATA_IDS_1         0x018
+#define CSI2_DATA_IDS_2         0x01c
+#define CSI2_ERR1               0x020
+#define CSI2_ERR2               0x024
+#define CSI2_MSK1               0x028
+#define CSI2_MSK2               0x02c
+#define CSI2_PHY_TST_CTRL0      0x030
+#define CSI2_PHY_TST_CTRL1      0x034
+#define CSI2_SFT_RESET          0xf00
+
+static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct imxcsi2_dev, sd);
+}
+
+static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
+{
+	return readl(csi2->base + regoff);
+}
+
+static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
+				 unsigned int regoff)
+{
+	writel(val, csi2->base + regoff);
+}
+
+static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
+{
+	int lanes = csi2->bus.num_data_lanes;
+
+	imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
+}
+
+static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
+{
+	if (enable) {
+		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
+		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
+		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
+	} else {
+		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
+		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
+		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
+	}
+}
+
+static void imxcsi2_reset(struct imxcsi2_dev *csi2)
+{
+	imxcsi2_enable(csi2, false);
+
+	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+
+	imxcsi2_enable(csi2, true);
+}
+
+static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
+{
+	u32 reg;
+	int i;
+
+	/* wait for mipi sensor ready */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+		if (reg != 0x200)
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for clock lane timeout, phy_state = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	/* wait for mipi stable */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_ERR1);
+		if (reg == 0x0)
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for controller timeout, err1 = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	/* finally let's wait for active clock on the clock lane */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+		if (reg & (1 << 8))
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for active clock timeout, phy_state = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
+		  imxcsi2_read(csi2, CSI2_VERSION));
+
+	return 0;
+}
+
+/*
+ * V4L2 subdev operations
+ */
+
+static int imxcsi2_link_setup(struct media_entity *entity,
+			      const struct media_pad *local,
+			      const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (csi2->sink_sd[local->index])
+				return -EBUSY;
+			csi2->sink_sd[local->index] = remote_sd;
+		} else {
+			csi2->sink_sd[local->index] = NULL;
+		}
+	} else {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (csi2->src_sd)
+				return -EBUSY;
+			csi2->src_sd = remote_sd;
+		} else {
+			csi2->src_sd = NULL;
+		}
+	}
+
+	return 0;
+}
+
+static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	if (on && !csi2->on) {
+		v4l2_info(&csi2->sd, "power ON\n");
+		clk_prepare_enable(csi2->cfg_clk);
+		clk_prepare_enable(csi2->dphy_clk);
+		imxcsi2_set_lanes(csi2);
+		imxcsi2_reset(csi2);
+	} else if (!on && csi2->on) {
+		v4l2_info(&csi2->sd, "power OFF\n");
+		imxcsi2_enable(csi2, false);
+		clk_disable_unprepare(csi2->dphy_clk);
+		clk_disable_unprepare(csi2->cfg_clk);
+	}
+
+	csi2->on = on;
+	return 0;
+}
+
+static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	int i, ret = 0;
+
+	if (!csi2->src_sd)
+		return -EPIPE;
+	for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
+		if (csi2->sink_sd[i])
+			break;
+	}
+	if (i >= CSI2_NUM_SRC_PADS)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !csi2->stream_on) {
+		ret = clk_prepare_enable(csi2->pix_clk);
+		if (ret)
+			return ret;
+
+		ret = imxcsi2_dphy_wait(csi2);
+		if (ret) {
+			clk_disable_unprepare(csi2->pix_clk);
+			return ret;
+		}
+	} else if (!enable && csi2->stream_on) {
+		clk_disable_unprepare(csi2->pix_clk);
+	}
+
+	csi2->stream_on = enable;
+	return 0;
+}
+
+static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_pad_config *cfg,
+			   struct v4l2_subdev_format *sdformat)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	sdformat->format = csi2->format_mbus;
+
+	return 0;
+}
+
+static int imxcsi2_set_fmt(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_pad_config *cfg,
+			   struct v4l2_subdev_format *sdformat)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	if (sdformat->pad >= CSI2_NUM_PADS)
+		return -EINVAL;
+
+	if (csi2->stream_on)
+		return -EBUSY;
+
+	/* Output pads mirror active input pad, no limits on input pads */
+	if (sdformat->pad != csi2->input_pad)
+		sdformat->format = csi2->format_mbus;
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY)
+		cfg->try_fmt = sdformat->format;
+	else
+		csi2->format_mbus = sdformat->format;
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int imxcsi2_registered(struct v4l2_subdev *sd)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	csi2->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(csi2->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 4)
+		return -EINVAL;
+
+	for (i = 0; i < CSI2_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		csi2->pad[i] = pad->pad;
+		if (csi2->pad[i].flags & MEDIA_PAD_FL_SINK)
+			csi2->input_pad = i;
+	}
+
+	/* set a default mbus format  */
+	ret = imx_media_init_mbus_fmt(&csi2->format_mbus,
+				      640, 480, 0, V4L2_FIELD_NONE, NULL);
+	if (ret)
+		return ret;
+
+	return media_entity_pads_init(&sd->entity, CSI2_NUM_PADS, csi2->pad);
+}
+
+static struct media_entity_operations imxcsi2_entity_ops = {
+	.link_setup = imxcsi2_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops imxcsi2_core_ops = {
+	.s_power = imxcsi2_s_power,
+};
+
+static struct v4l2_subdev_video_ops imxcsi2_video_ops = {
+	.s_stream = imxcsi2_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imxcsi2_pad_ops = {
+	.get_fmt = imxcsi2_get_fmt,
+	.set_fmt = imxcsi2_set_fmt,
+};
+
+static struct v4l2_subdev_ops imxcsi2_subdev_ops = {
+	.core = &imxcsi2_core_ops,
+	.video = &imxcsi2_video_ops,
+	.pad = &imxcsi2_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imxcsi2_internal_ops = {
+	.registered = imxcsi2_registered,
+};
+
+static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
+{
+	struct device_node *node = csi2->dev->of_node;
+	struct device_node *epnode;
+	struct v4l2_of_endpoint ep;
+
+	epnode = of_graph_get_next_endpoint(node, NULL);
+	if (!epnode) {
+		v4l2_err(&csi2->sd, "failed to get endpoint node\n");
+		return -EINVAL;
+	}
+
+	v4l2_of_parse_endpoint(epnode, &ep);
+	of_node_put(epnode);
+
+	if (ep.bus_type != V4L2_MBUS_CSI2) {
+		v4l2_err(&csi2->sd, "invalid bus type, must be MIPI CSI2\n");
+		return -EINVAL;
+	}
+
+	csi2->bus = ep.bus.mipi_csi2;
+
+	v4l2_info(&csi2->sd, "data lanes: %d\n", csi2->bus.num_data_lanes);
+	v4l2_info(&csi2->sd, "flags: 0x%08x\n", csi2->bus.flags);
+	return 0;
+}
+
+static int imxcsi2_probe(struct platform_device *pdev)
+{
+	struct imxcsi2_dev *csi2;
+	struct resource *res;
+	int ret;
+
+	csi2 = devm_kzalloc(&pdev->dev, sizeof(*csi2), GFP_KERNEL);
+	if (!csi2)
+		return -ENOMEM;
+
+	csi2->dev = &pdev->dev;
+
+	v4l2_subdev_init(&csi2->sd, &imxcsi2_subdev_ops);
+	v4l2_set_subdevdata(&csi2->sd, &pdev->dev);
+	csi2->sd.internal_ops = &imxcsi2_internal_ops;
+	csi2->sd.entity.ops = &imxcsi2_entity_ops;
+	csi2->sd.dev = &pdev->dev;
+	csi2->sd.owner = THIS_MODULE;
+	csi2->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	strcpy(csi2->sd.name, DEVICE_NAME);
+	csi2->sd.entity.function = MEDIA_ENT_F_VID_IF_BRIDGE;
+	csi2->sd.grp_id = IMX_MEDIA_GRP_ID_CSI2;
+
+	ret = imxcsi2_parse_endpoints(csi2);
+	if (ret)
+		return ret;
+
+	csi2->cfg_clk = devm_clk_get(&pdev->dev, "cfg");
+	if (IS_ERR(csi2->cfg_clk)) {
+		v4l2_err(&csi2->sd, "failed to get cfg clock\n");
+		ret = PTR_ERR(csi2->cfg_clk);
+		return ret;
+	}
+
+	csi2->dphy_clk = devm_clk_get(&pdev->dev, "dphy");
+	if (IS_ERR(csi2->dphy_clk)) {
+		v4l2_err(&csi2->sd, "failed to get dphy clock\n");
+		ret = PTR_ERR(csi2->dphy_clk);
+		return ret;
+	}
+
+	csi2->pix_clk = devm_clk_get(&pdev->dev, "pix");
+	if (IS_ERR(csi2->pix_clk)) {
+		v4l2_err(&csi2->sd, "failed to get pixel clock\n");
+		ret = PTR_ERR(csi2->pix_clk);
+		return ret;
+	}
+
+	csi2->intr1 = platform_get_irq(pdev, 0);
+	csi2->intr2 = platform_get_irq(pdev, 1);
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+
+	if (!res || csi2->intr1 < 0 || csi2->intr2 < 0) {
+		v4l2_err(&csi2->sd, "failed to get platform resources\n");
+		return -ENODEV;
+	}
+
+	csi2->base = devm_ioremap(&pdev->dev, res->start, PAGE_SIZE);
+	if (!csi2->base) {
+		v4l2_err(&csi2->sd, "failed to map CSI-2 registers\n");
+		return -ENOMEM;
+	}
+
+	platform_set_drvdata(pdev, &csi2->sd);
+
+	return v4l2_async_register_subdev(&csi2->sd);
+}
+
+static int imxcsi2_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	imxcsi2_s_power(sd, 0);
+
+	v4l2_async_unregister_subdev(&csi2->sd);
+	media_entity_cleanup(&csi2->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct of_device_id imxcsi2_dt_ids[] = {
+	{ .compatible = "fsl,imx6-mipi-csi2", },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
+
+static struct platform_driver imxcsi2_driver = {
+	.driver = {
+		.name = DEVICE_NAME,
+		.of_match_table = imxcsi2_dt_ids,
+	},
+	.probe = imxcsi2_probe,
+	.remove = imxcsi2_remove,
+};
+
+module_platform_driver(imxcsi2_driver);
+
+MODULE_DESCRIPTION("i.MX5/6 MIPI CSI-2 Receiver driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
for sensors with a MIPI CSI2 interface.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile        |   1 +
 drivers/staging/media/imx/imx-mipi-csi2.c | 501 ++++++++++++++++++++++++++++++
 2 files changed, 502 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index fe9e992..0decef7 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
new file mode 100644
index 0000000..daa6e1d
--- /dev/null
+++ b/drivers/staging/media/imx/imx-mipi-csi2.c
@@ -0,0 +1,501 @@
+/*
+ * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
+ *
+ * Copyright (c) 2012-2014 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+/*
+ * there must be 5 pads: 1 input pad from sensor, and
+ * the 4 virtual channel output pads
+ */
+#define CSI2_NUM_SINK_PADS  1
+#define CSI2_NUM_SRC_PADS   4
+#define CSI2_NUM_PADS       5
+
+struct imxcsi2_dev {
+	struct device          *dev;
+	struct imx_media_dev   *md;
+	struct v4l2_subdev      sd;
+	struct media_pad       pad[CSI2_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus;
+	struct v4l2_subdev     *src_sd;
+	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];
+	int                    input_pad;
+	struct clk             *dphy_clk;
+	struct clk             *cfg_clk;
+	struct clk             *pix_clk; /* what is this? */
+	void __iomem           *base;
+	int                     intr1;
+	int                     intr2;
+	struct v4l2_of_bus_mipi_csi2 bus;
+	bool                    on;
+	bool                    stream_on;
+};
+
+#define DEVICE_NAME "imx6-mipi-csi2"
+
+/* Register offsets */
+#define CSI2_VERSION            0x000
+#define CSI2_N_LANES            0x004
+#define CSI2_PHY_SHUTDOWNZ      0x008
+#define CSI2_DPHY_RSTZ          0x00c
+#define CSI2_RESETN             0x010
+#define CSI2_PHY_STATE          0x014
+#define CSI2_DATA_IDS_1         0x018
+#define CSI2_DATA_IDS_2         0x01c
+#define CSI2_ERR1               0x020
+#define CSI2_ERR2               0x024
+#define CSI2_MSK1               0x028
+#define CSI2_MSK2               0x02c
+#define CSI2_PHY_TST_CTRL0      0x030
+#define CSI2_PHY_TST_CTRL1      0x034
+#define CSI2_SFT_RESET          0xf00
+
+static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct imxcsi2_dev, sd);
+}
+
+static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
+{
+	return readl(csi2->base + regoff);
+}
+
+static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
+				 unsigned int regoff)
+{
+	writel(val, csi2->base + regoff);
+}
+
+static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
+{
+	int lanes = csi2->bus.num_data_lanes;
+
+	imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
+}
+
+static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
+{
+	if (enable) {
+		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
+		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
+		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
+	} else {
+		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
+		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
+		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
+	}
+}
+
+static void imxcsi2_reset(struct imxcsi2_dev *csi2)
+{
+	imxcsi2_enable(csi2, false);
+
+	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+
+	imxcsi2_enable(csi2, true);
+}
+
+static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
+{
+	u32 reg;
+	int i;
+
+	/* wait for mipi sensor ready */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+		if (reg != 0x200)
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for clock lane timeout, phy_state = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	/* wait for mipi stable */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_ERR1);
+		if (reg == 0x0)
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for controller timeout, err1 = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	/* finally let's wait for active clock on the clock lane */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+		if (reg & (1 << 8))
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for active clock timeout, phy_state = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
+		  imxcsi2_read(csi2, CSI2_VERSION));
+
+	return 0;
+}
+
+/*
+ * V4L2 subdev operations
+ */
+
+static int imxcsi2_link_setup(struct media_entity *entity,
+			      const struct media_pad *local,
+			      const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (csi2->sink_sd[local->index])
+				return -EBUSY;
+			csi2->sink_sd[local->index] = remote_sd;
+		} else {
+			csi2->sink_sd[local->index] = NULL;
+		}
+	} else {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (csi2->src_sd)
+				return -EBUSY;
+			csi2->src_sd = remote_sd;
+		} else {
+			csi2->src_sd = NULL;
+		}
+	}
+
+	return 0;
+}
+
+static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	if (on && !csi2->on) {
+		v4l2_info(&csi2->sd, "power ON\n");
+		clk_prepare_enable(csi2->cfg_clk);
+		clk_prepare_enable(csi2->dphy_clk);
+		imxcsi2_set_lanes(csi2);
+		imxcsi2_reset(csi2);
+	} else if (!on && csi2->on) {
+		v4l2_info(&csi2->sd, "power OFF\n");
+		imxcsi2_enable(csi2, false);
+		clk_disable_unprepare(csi2->dphy_clk);
+		clk_disable_unprepare(csi2->cfg_clk);
+	}
+
+	csi2->on = on;
+	return 0;
+}
+
+static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	int i, ret = 0;
+
+	if (!csi2->src_sd)
+		return -EPIPE;
+	for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
+		if (csi2->sink_sd[i])
+			break;
+	}
+	if (i >= CSI2_NUM_SRC_PADS)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !csi2->stream_on) {
+		ret = clk_prepare_enable(csi2->pix_clk);
+		if (ret)
+			return ret;
+
+		ret = imxcsi2_dphy_wait(csi2);
+		if (ret) {
+			clk_disable_unprepare(csi2->pix_clk);
+			return ret;
+		}
+	} else if (!enable && csi2->stream_on) {
+		clk_disable_unprepare(csi2->pix_clk);
+	}
+
+	csi2->stream_on = enable;
+	return 0;
+}
+
+static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_pad_config *cfg,
+			   struct v4l2_subdev_format *sdformat)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	sdformat->format = csi2->format_mbus;
+
+	return 0;
+}
+
+static int imxcsi2_set_fmt(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_pad_config *cfg,
+			   struct v4l2_subdev_format *sdformat)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	if (sdformat->pad >= CSI2_NUM_PADS)
+		return -EINVAL;
+
+	if (csi2->stream_on)
+		return -EBUSY;
+
+	/* Output pads mirror active input pad, no limits on input pads */
+	if (sdformat->pad != csi2->input_pad)
+		sdformat->format = csi2->format_mbus;
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY)
+		cfg->try_fmt = sdformat->format;
+	else
+		csi2->format_mbus = sdformat->format;
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int imxcsi2_registered(struct v4l2_subdev *sd)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	csi2->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(csi2->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 4)
+		return -EINVAL;
+
+	for (i = 0; i < CSI2_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		csi2->pad[i] = pad->pad;
+		if (csi2->pad[i].flags & MEDIA_PAD_FL_SINK)
+			csi2->input_pad = i;
+	}
+
+	/* set a default mbus format  */
+	ret = imx_media_init_mbus_fmt(&csi2->format_mbus,
+				      640, 480, 0, V4L2_FIELD_NONE, NULL);
+	if (ret)
+		return ret;
+
+	return media_entity_pads_init(&sd->entity, CSI2_NUM_PADS, csi2->pad);
+}
+
+static struct media_entity_operations imxcsi2_entity_ops = {
+	.link_setup = imxcsi2_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops imxcsi2_core_ops = {
+	.s_power = imxcsi2_s_power,
+};
+
+static struct v4l2_subdev_video_ops imxcsi2_video_ops = {
+	.s_stream = imxcsi2_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imxcsi2_pad_ops = {
+	.get_fmt = imxcsi2_get_fmt,
+	.set_fmt = imxcsi2_set_fmt,
+};
+
+static struct v4l2_subdev_ops imxcsi2_subdev_ops = {
+	.core = &imxcsi2_core_ops,
+	.video = &imxcsi2_video_ops,
+	.pad = &imxcsi2_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imxcsi2_internal_ops = {
+	.registered = imxcsi2_registered,
+};
+
+static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
+{
+	struct device_node *node = csi2->dev->of_node;
+	struct device_node *epnode;
+	struct v4l2_of_endpoint ep;
+
+	epnode = of_graph_get_next_endpoint(node, NULL);
+	if (!epnode) {
+		v4l2_err(&csi2->sd, "failed to get endpoint node\n");
+		return -EINVAL;
+	}
+
+	v4l2_of_parse_endpoint(epnode, &ep);
+	of_node_put(epnode);
+
+	if (ep.bus_type != V4L2_MBUS_CSI2) {
+		v4l2_err(&csi2->sd, "invalid bus type, must be MIPI CSI2\n");
+		return -EINVAL;
+	}
+
+	csi2->bus = ep.bus.mipi_csi2;
+
+	v4l2_info(&csi2->sd, "data lanes: %d\n", csi2->bus.num_data_lanes);
+	v4l2_info(&csi2->sd, "flags: 0x%08x\n", csi2->bus.flags);
+	return 0;
+}
+
+static int imxcsi2_probe(struct platform_device *pdev)
+{
+	struct imxcsi2_dev *csi2;
+	struct resource *res;
+	int ret;
+
+	csi2 = devm_kzalloc(&pdev->dev, sizeof(*csi2), GFP_KERNEL);
+	if (!csi2)
+		return -ENOMEM;
+
+	csi2->dev = &pdev->dev;
+
+	v4l2_subdev_init(&csi2->sd, &imxcsi2_subdev_ops);
+	v4l2_set_subdevdata(&csi2->sd, &pdev->dev);
+	csi2->sd.internal_ops = &imxcsi2_internal_ops;
+	csi2->sd.entity.ops = &imxcsi2_entity_ops;
+	csi2->sd.dev = &pdev->dev;
+	csi2->sd.owner = THIS_MODULE;
+	csi2->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	strcpy(csi2->sd.name, DEVICE_NAME);
+	csi2->sd.entity.function = MEDIA_ENT_F_VID_IF_BRIDGE;
+	csi2->sd.grp_id = IMX_MEDIA_GRP_ID_CSI2;
+
+	ret = imxcsi2_parse_endpoints(csi2);
+	if (ret)
+		return ret;
+
+	csi2->cfg_clk = devm_clk_get(&pdev->dev, "cfg");
+	if (IS_ERR(csi2->cfg_clk)) {
+		v4l2_err(&csi2->sd, "failed to get cfg clock\n");
+		ret = PTR_ERR(csi2->cfg_clk);
+		return ret;
+	}
+
+	csi2->dphy_clk = devm_clk_get(&pdev->dev, "dphy");
+	if (IS_ERR(csi2->dphy_clk)) {
+		v4l2_err(&csi2->sd, "failed to get dphy clock\n");
+		ret = PTR_ERR(csi2->dphy_clk);
+		return ret;
+	}
+
+	csi2->pix_clk = devm_clk_get(&pdev->dev, "pix");
+	if (IS_ERR(csi2->pix_clk)) {
+		v4l2_err(&csi2->sd, "failed to get pixel clock\n");
+		ret = PTR_ERR(csi2->pix_clk);
+		return ret;
+	}
+
+	csi2->intr1 = platform_get_irq(pdev, 0);
+	csi2->intr2 = platform_get_irq(pdev, 1);
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+
+	if (!res || csi2->intr1 < 0 || csi2->intr2 < 0) {
+		v4l2_err(&csi2->sd, "failed to get platform resources\n");
+		return -ENODEV;
+	}
+
+	csi2->base = devm_ioremap(&pdev->dev, res->start, PAGE_SIZE);
+	if (!csi2->base) {
+		v4l2_err(&csi2->sd, "failed to map CSI-2 registers\n");
+		return -ENOMEM;
+	}
+
+	platform_set_drvdata(pdev, &csi2->sd);
+
+	return v4l2_async_register_subdev(&csi2->sd);
+}
+
+static int imxcsi2_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	imxcsi2_s_power(sd, 0);
+
+	v4l2_async_unregister_subdev(&csi2->sd);
+	media_entity_cleanup(&csi2->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct of_device_id imxcsi2_dt_ids[] = {
+	{ .compatible = "fsl,imx6-mipi-csi2", },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
+
+static struct platform_driver imxcsi2_driver = {
+	.driver = {
+		.name = DEVICE_NAME,
+		.of_match_table = imxcsi2_dt_ids,
+	},
+	.probe = imxcsi2_probe,
+	.remove = imxcsi2_remove,
+};
+
+module_platform_driver(imxcsi2_driver);
+
+MODULE_DESCRIPTION("i.MX5/6 MIPI CSI-2 Receiver driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
for sensors with a MIPI CSI2 interface.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Makefile        |   1 +
 drivers/staging/media/imx/imx-mipi-csi2.c | 501 ++++++++++++++++++++++++++++++
 2 files changed, 502 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index fe9e992..0decef7 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
new file mode 100644
index 0000000..daa6e1d
--- /dev/null
+++ b/drivers/staging/media/imx/imx-mipi-csi2.c
@@ -0,0 +1,501 @@
+/*
+ * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
+ *
+ * Copyright (c) 2012-2014 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+/*
+ * there must be 5 pads: 1 input pad from sensor, and
+ * the 4 virtual channel output pads
+ */
+#define CSI2_NUM_SINK_PADS  1
+#define CSI2_NUM_SRC_PADS   4
+#define CSI2_NUM_PADS       5
+
+struct imxcsi2_dev {
+	struct device          *dev;
+	struct imx_media_dev   *md;
+	struct v4l2_subdev      sd;
+	struct media_pad       pad[CSI2_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus;
+	struct v4l2_subdev     *src_sd;
+	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];
+	int                    input_pad;
+	struct clk             *dphy_clk;
+	struct clk             *cfg_clk;
+	struct clk             *pix_clk; /* what is this? */
+	void __iomem           *base;
+	int                     intr1;
+	int                     intr2;
+	struct v4l2_of_bus_mipi_csi2 bus;
+	bool                    on;
+	bool                    stream_on;
+};
+
+#define DEVICE_NAME "imx6-mipi-csi2"
+
+/* Register offsets */
+#define CSI2_VERSION            0x000
+#define CSI2_N_LANES            0x004
+#define CSI2_PHY_SHUTDOWNZ      0x008
+#define CSI2_DPHY_RSTZ          0x00c
+#define CSI2_RESETN             0x010
+#define CSI2_PHY_STATE          0x014
+#define CSI2_DATA_IDS_1         0x018
+#define CSI2_DATA_IDS_2         0x01c
+#define CSI2_ERR1               0x020
+#define CSI2_ERR2               0x024
+#define CSI2_MSK1               0x028
+#define CSI2_MSK2               0x02c
+#define CSI2_PHY_TST_CTRL0      0x030
+#define CSI2_PHY_TST_CTRL1      0x034
+#define CSI2_SFT_RESET          0xf00
+
+static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct imxcsi2_dev, sd);
+}
+
+static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
+{
+	return readl(csi2->base + regoff);
+}
+
+static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
+				 unsigned int regoff)
+{
+	writel(val, csi2->base + regoff);
+}
+
+static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
+{
+	int lanes = csi2->bus.num_data_lanes;
+
+	imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
+}
+
+static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
+{
+	if (enable) {
+		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
+		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
+		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
+	} else {
+		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
+		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
+		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
+	}
+}
+
+static void imxcsi2_reset(struct imxcsi2_dev *csi2)
+{
+	imxcsi2_enable(csi2, false);
+
+	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
+	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+
+	imxcsi2_enable(csi2, true);
+}
+
+static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
+{
+	u32 reg;
+	int i;
+
+	/* wait for mipi sensor ready */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+		if (reg != 0x200)
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for clock lane timeout, phy_state = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	/* wait for mipi stable */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_ERR1);
+		if (reg == 0x0)
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for controller timeout, err1 = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	/* finally let's wait for active clock on the clock lane */
+	for (i = 0; i < 50; i++) {
+		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+		if (reg & (1 << 8))
+			break;
+		usleep_range(10000, 20000);
+	}
+
+	if (i >= 50) {
+		v4l2_err(&csi2->sd,
+			 "wait for active clock timeout, phy_state = 0x%08x\n",
+			 reg);
+		return -ETIME;
+	}
+
+	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
+		  imxcsi2_read(csi2, CSI2_VERSION));
+
+	return 0;
+}
+
+/*
+ * V4L2 subdev operations
+ */
+
+static int imxcsi2_link_setup(struct media_entity *entity,
+			      const struct media_pad *local,
+			      const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (csi2->sink_sd[local->index])
+				return -EBUSY;
+			csi2->sink_sd[local->index] = remote_sd;
+		} else {
+			csi2->sink_sd[local->index] = NULL;
+		}
+	} else {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (csi2->src_sd)
+				return -EBUSY;
+			csi2->src_sd = remote_sd;
+		} else {
+			csi2->src_sd = NULL;
+		}
+	}
+
+	return 0;
+}
+
+static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	if (on && !csi2->on) {
+		v4l2_info(&csi2->sd, "power ON\n");
+		clk_prepare_enable(csi2->cfg_clk);
+		clk_prepare_enable(csi2->dphy_clk);
+		imxcsi2_set_lanes(csi2);
+		imxcsi2_reset(csi2);
+	} else if (!on && csi2->on) {
+		v4l2_info(&csi2->sd, "power OFF\n");
+		imxcsi2_enable(csi2, false);
+		clk_disable_unprepare(csi2->dphy_clk);
+		clk_disable_unprepare(csi2->cfg_clk);
+	}
+
+	csi2->on = on;
+	return 0;
+}
+
+static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	int i, ret = 0;
+
+	if (!csi2->src_sd)
+		return -EPIPE;
+	for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
+		if (csi2->sink_sd[i])
+			break;
+	}
+	if (i >= CSI2_NUM_SRC_PADS)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !csi2->stream_on) {
+		ret = clk_prepare_enable(csi2->pix_clk);
+		if (ret)
+			return ret;
+
+		ret = imxcsi2_dphy_wait(csi2);
+		if (ret) {
+			clk_disable_unprepare(csi2->pix_clk);
+			return ret;
+		}
+	} else if (!enable && csi2->stream_on) {
+		clk_disable_unprepare(csi2->pix_clk);
+	}
+
+	csi2->stream_on = enable;
+	return 0;
+}
+
+static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_pad_config *cfg,
+			   struct v4l2_subdev_format *sdformat)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	sdformat->format = csi2->format_mbus;
+
+	return 0;
+}
+
+static int imxcsi2_set_fmt(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_pad_config *cfg,
+			   struct v4l2_subdev_format *sdformat)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	if (sdformat->pad >= CSI2_NUM_PADS)
+		return -EINVAL;
+
+	if (csi2->stream_on)
+		return -EBUSY;
+
+	/* Output pads mirror active input pad, no limits on input pads */
+	if (sdformat->pad != csi2->input_pad)
+		sdformat->format = csi2->format_mbus;
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY)
+		cfg->try_fmt = sdformat->format;
+	else
+		csi2->format_mbus = sdformat->format;
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int imxcsi2_registered(struct v4l2_subdev *sd)
+{
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	csi2->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(csi2->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 4)
+		return -EINVAL;
+
+	for (i = 0; i < CSI2_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		csi2->pad[i] = pad->pad;
+		if (csi2->pad[i].flags & MEDIA_PAD_FL_SINK)
+			csi2->input_pad = i;
+	}
+
+	/* set a default mbus format  */
+	ret = imx_media_init_mbus_fmt(&csi2->format_mbus,
+				      640, 480, 0, V4L2_FIELD_NONE, NULL);
+	if (ret)
+		return ret;
+
+	return media_entity_pads_init(&sd->entity, CSI2_NUM_PADS, csi2->pad);
+}
+
+static struct media_entity_operations imxcsi2_entity_ops = {
+	.link_setup = imxcsi2_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops imxcsi2_core_ops = {
+	.s_power = imxcsi2_s_power,
+};
+
+static struct v4l2_subdev_video_ops imxcsi2_video_ops = {
+	.s_stream = imxcsi2_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imxcsi2_pad_ops = {
+	.get_fmt = imxcsi2_get_fmt,
+	.set_fmt = imxcsi2_set_fmt,
+};
+
+static struct v4l2_subdev_ops imxcsi2_subdev_ops = {
+	.core = &imxcsi2_core_ops,
+	.video = &imxcsi2_video_ops,
+	.pad = &imxcsi2_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imxcsi2_internal_ops = {
+	.registered = imxcsi2_registered,
+};
+
+static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
+{
+	struct device_node *node = csi2->dev->of_node;
+	struct device_node *epnode;
+	struct v4l2_of_endpoint ep;
+
+	epnode = of_graph_get_next_endpoint(node, NULL);
+	if (!epnode) {
+		v4l2_err(&csi2->sd, "failed to get endpoint node\n");
+		return -EINVAL;
+	}
+
+	v4l2_of_parse_endpoint(epnode, &ep);
+	of_node_put(epnode);
+
+	if (ep.bus_type != V4L2_MBUS_CSI2) {
+		v4l2_err(&csi2->sd, "invalid bus type, must be MIPI CSI2\n");
+		return -EINVAL;
+	}
+
+	csi2->bus = ep.bus.mipi_csi2;
+
+	v4l2_info(&csi2->sd, "data lanes: %d\n", csi2->bus.num_data_lanes);
+	v4l2_info(&csi2->sd, "flags: 0x%08x\n", csi2->bus.flags);
+	return 0;
+}
+
+static int imxcsi2_probe(struct platform_device *pdev)
+{
+	struct imxcsi2_dev *csi2;
+	struct resource *res;
+	int ret;
+
+	csi2 = devm_kzalloc(&pdev->dev, sizeof(*csi2), GFP_KERNEL);
+	if (!csi2)
+		return -ENOMEM;
+
+	csi2->dev = &pdev->dev;
+
+	v4l2_subdev_init(&csi2->sd, &imxcsi2_subdev_ops);
+	v4l2_set_subdevdata(&csi2->sd, &pdev->dev);
+	csi2->sd.internal_ops = &imxcsi2_internal_ops;
+	csi2->sd.entity.ops = &imxcsi2_entity_ops;
+	csi2->sd.dev = &pdev->dev;
+	csi2->sd.owner = THIS_MODULE;
+	csi2->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	strcpy(csi2->sd.name, DEVICE_NAME);
+	csi2->sd.entity.function = MEDIA_ENT_F_VID_IF_BRIDGE;
+	csi2->sd.grp_id = IMX_MEDIA_GRP_ID_CSI2;
+
+	ret = imxcsi2_parse_endpoints(csi2);
+	if (ret)
+		return ret;
+
+	csi2->cfg_clk = devm_clk_get(&pdev->dev, "cfg");
+	if (IS_ERR(csi2->cfg_clk)) {
+		v4l2_err(&csi2->sd, "failed to get cfg clock\n");
+		ret = PTR_ERR(csi2->cfg_clk);
+		return ret;
+	}
+
+	csi2->dphy_clk = devm_clk_get(&pdev->dev, "dphy");
+	if (IS_ERR(csi2->dphy_clk)) {
+		v4l2_err(&csi2->sd, "failed to get dphy clock\n");
+		ret = PTR_ERR(csi2->dphy_clk);
+		return ret;
+	}
+
+	csi2->pix_clk = devm_clk_get(&pdev->dev, "pix");
+	if (IS_ERR(csi2->pix_clk)) {
+		v4l2_err(&csi2->sd, "failed to get pixel clock\n");
+		ret = PTR_ERR(csi2->pix_clk);
+		return ret;
+	}
+
+	csi2->intr1 = platform_get_irq(pdev, 0);
+	csi2->intr2 = platform_get_irq(pdev, 1);
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+
+	if (!res || csi2->intr1 < 0 || csi2->intr2 < 0) {
+		v4l2_err(&csi2->sd, "failed to get platform resources\n");
+		return -ENODEV;
+	}
+
+	csi2->base = devm_ioremap(&pdev->dev, res->start, PAGE_SIZE);
+	if (!csi2->base) {
+		v4l2_err(&csi2->sd, "failed to map CSI-2 registers\n");
+		return -ENOMEM;
+	}
+
+	platform_set_drvdata(pdev, &csi2->sd);
+
+	return v4l2_async_register_subdev(&csi2->sd);
+}
+
+static int imxcsi2_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+	imxcsi2_s_power(sd, 0);
+
+	v4l2_async_unregister_subdev(&csi2->sd);
+	media_entity_cleanup(&csi2->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct of_device_id imxcsi2_dt_ids[] = {
+	{ .compatible = "fsl,imx6-mipi-csi2", },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
+
+static struct platform_driver imxcsi2_driver = {
+	.driver = {
+		.name = DEVICE_NAME,
+		.of_match_table = imxcsi2_dt_ids,
+	},
+	.probe = imxcsi2_probe,
+	.remove = imxcsi2_remove,
+};
+
+module_platform_driver(imxcsi2_driver);
+
+MODULE_DESCRIPTION("i.MX5/6 MIPI CSI-2 Receiver driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_LICENSE("GPL");
+
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
branch, modified heavily to bring forward to latest interfaces and code
cleanup.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Kconfig       |    8 +
 drivers/staging/media/imx/Makefile      |    2 +
 drivers/staging/media/imx/ov5640-mipi.c | 2348 +++++++++++++++++++++++++++++++
 3 files changed, 2358 insertions(+)
 create mode 100644 drivers/staging/media/imx/ov5640-mipi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index ce2d2c8..09f373d 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
 	---help---
 	  A video4linux camera capture driver for i.MX5/6.
 
+config IMX_OV5640_MIPI
+       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
+       depends on GPIOLIB && VIDEO_IMX_CAMERA
+       select IMX_MIPI_CSI2
+       default y
+       ---help---
+         MIPI CSI-2 OV5640 Camera support.
+
 endmenu
 endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 0decef7..aa954c1 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -10,3 +10,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
+
+obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
new file mode 100644
index 0000000..54647a7
--- /dev/null
+++ b/drivers/staging/media/imx/ov5640-mipi.c
@@ -0,0 +1,2348 @@
+/*
+ * Copyright (c) 2014 Mentor Graphics Inc.
+ * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/ctype.h>
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/of_device.h>
+#include <linux/gpio/consumer.h>
+#include <linux/regulator/consumer.h>
+#include <linux/clk.h>
+#include <linux/clk-provider.h>
+#include <linux/clkdev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+
+#define OV5640_VOLTAGE_ANALOG               2800000
+#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
+#define OV5640_VOLTAGE_DIGITAL_IO           1800000
+
+#define MIN_FPS 15
+#define MAX_FPS 30
+#define DEFAULT_FPS 30
+
+/* min/typical/max system clock (xclk) frequencies */
+#define OV5640_XCLK_MIN  6000000
+#define OV5640_XCLK_TYP 24000000
+#define OV5640_XCLK_MAX 54000000
+
+/* min/typical/max pixel clock (mclk) frequencies */
+#define OV5640_MCLK_MIN 48000000
+#define OV5640_MCLK_TYP 48000000
+#define OV5640_MCLK_MAX 96000000
+
+#define OV5640_CHIP_ID  0x300A
+#define OV5640_SLAVE_ID 0x3100
+#define OV5640_DEFAULT_SLAVE_ID 0x3c
+
+#define OV5640_MAX_CONTROLS 64
+
+enum ov5640_mode {
+	ov5640_mode_MIN = 0,
+	ov5640_mode_QCIF_176_144 = 0,
+	ov5640_mode_QVGA_320_240,
+	ov5640_mode_VGA_640_480,
+	ov5640_mode_NTSC_720_480,
+	ov5640_mode_PAL_720_576,
+	ov5640_mode_XGA_1024_768,
+	ov5640_mode_720P_1280_720,
+	ov5640_mode_1080P_1920_1080,
+	ov5640_mode_QSXGA_2592_1944,
+	ov5640_num_modes,
+	ov5640_mode_INIT = 0xff, /*only for sensor init*/
+};
+
+enum ov5640_frame_rate {
+	ov5640_15_fps,
+	ov5640_30_fps
+};
+
+static int ov5640_framerates[] = {
+	[ov5640_15_fps] = 15,
+	[ov5640_30_fps] = 30,
+};
+#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
+
+/* image size under 1280 * 960 are SUBSAMPLING
+ * image size upper 1280 * 960 are SCALING
+ */
+enum ov5640_downsize_mode {
+	SUBSAMPLING,
+	SCALING,
+};
+
+struct reg_value {
+	u16 reg_addr;
+	u8 val;
+	u8 mask;
+	u32 delay_ms;
+};
+
+struct ov5640_mode_info {
+	enum ov5640_mode mode;
+	enum ov5640_downsize_mode dn_mode;
+	u32 width;
+	u32 height;
+	struct reg_value *init_data_ptr;
+	u32 init_data_size;
+};
+
+struct ov5640_dev {
+	struct i2c_client *i2c_client;
+	struct device *dev;
+	struct v4l2_subdev sd;
+	struct media_pad pad;
+	struct v4l2_ctrl_handler ctrl_hdl;
+	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_captureparm streamcap;
+	struct clk *xclk; /* system clock to OV5640 */
+	int xclk_freq;    /* requested xclk freq from devicetree */
+
+	enum ov5640_mode current_mode;
+	enum ov5640_frame_rate current_fr;
+
+	bool on;
+	bool awb_on;
+	bool agc_on;
+
+	/* cached control settings */
+	int ctrl_cache[OV5640_MAX_CONTROLS];
+
+	struct gpio_desc *reset_gpio;
+	struct gpio_desc *pwdn_gpio;
+	struct gpio_desc *gp_gpio;
+
+	int prev_sysclk, prev_hts;
+	int ae_low, ae_high, ae_target;
+
+	struct regulator *io_regulator;
+	struct regulator *core_regulator;
+	struct regulator *analog_regulator;
+	struct regulator *gpo_regulator;
+};
+
+static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct ov5640_dev, sd);
+}
+
+static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
+{
+	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
+}
+
+struct ov5640_control {
+	struct v4l2_queryctrl ctrl;
+	int (*set)(struct ov5640_dev *sensor, int value);
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable);
+static void ov5640_reset(struct ov5640_dev *sensor);
+static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
+static int ov5640_get_exposure(struct ov5640_dev *sensor);
+static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
+static int ov5640_get_gain(struct ov5640_dev *sensor);
+
+static struct reg_value ov5640_init_setting_30fps_VGA[] = {
+
+	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
+	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
+	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
+	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
+	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
+	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
+	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
+	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
+	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
+	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
+	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
+	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
+	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
+	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
+	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
+	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
+	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
+	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
+	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
+	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
+	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
+	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
+	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
+	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
+	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
+	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
+	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
+	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
+	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
+	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
+	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
+	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
+	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
+	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
+	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
+	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
+	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
+	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
+	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
+	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
+	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
+	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
+	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
+	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
+	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
+	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
+	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
+	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
+	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
+	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
+	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
+	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
+	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
+	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
+	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
+	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
+	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
+	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
+	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
+	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
+	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
+	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
+	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
+	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
+	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
+};
+
+static struct reg_value ov5640_setting_30fps_VGA_640_480[] = {
+
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_VGA_640_480[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_XGA_1024_768[] = {
+
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
+	{0x380b, 0x00, 0, 0}, {0x3035, 0x12, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_XGA_1024_768[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3808, 0x04, 0, 0},
+	{0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0}, {0x380b, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QVGA_320_240[] = {
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QVGA_320_240[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QCIF_176_144[] = {
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+static struct reg_value ov5640_setting_15fps_QCIF_176_144[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_NTSC_720_480[] = {
+	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_NTSC_720_480[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_PAL_720_576[] = {
+	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_PAL_720_576[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_720P_1280_720[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0}, {0x4005, 0x1a, 0, 0},
+	{0x3008, 0x02, 0, 0}, {0x3503, 0,    0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_720P_1280_720[] = {
+	{0x3035, 0x41, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_1080P_1920_1080[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x11, 0, 0},
+	{0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0},
+	{0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_1080P_1920_1080[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x21, 0, 0},
+	{0x3036, 0x54, 0, 1}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
+	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
+	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
+};
+
+static struct ov5640_mode_info
+ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
+	{
+		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+		 ov5640_setting_15fps_QCIF_176_144,
+		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
+		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
+		 ov5640_setting_15fps_QVGA_320_240,
+		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
+		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
+		 ov5640_setting_15fps_VGA_640_480,
+		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
+		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+		 ov5640_setting_15fps_NTSC_720_480,
+		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
+		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+		 ov5640_setting_15fps_PAL_720_576,
+		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
+		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+		 ov5640_setting_15fps_XGA_1024_768,
+		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
+		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+		 ov5640_setting_15fps_720P_1280_720,
+		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
+		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+		 ov5640_setting_15fps_1080P_1920_1080,
+		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
+		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
+		 ov5640_setting_15fps_QSXGA_2592_1944,
+		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
+	}, {
+		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+		 ov5640_setting_30fps_QCIF_176_144,
+		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
+		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
+		 ov5640_setting_30fps_QVGA_320_240,
+		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
+		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
+		 ov5640_setting_30fps_VGA_640_480,
+		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
+		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+		 ov5640_setting_30fps_NTSC_720_480,
+		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
+		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+		 ov5640_setting_30fps_PAL_720_576,
+		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
+		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+		 ov5640_setting_30fps_XGA_1024_768,
+		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
+		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+		 ov5640_setting_30fps_720P_1280_720,
+		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
+		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+		 ov5640_setting_30fps_1080P_1920_1080,
+		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
+		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
+	},
+};
+
+static int ov5640_probe(struct i2c_client *adapter,
+			const struct i2c_device_id *device_id);
+static int ov5640_remove(struct i2c_client *client);
+
+static int ov5640_init_slave_id(struct ov5640_dev *sensor)
+{
+	struct i2c_msg msg;
+	u8 buf[4];
+	int ret;
+
+	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
+		return 0;
+
+	buf[0] = OV5640_SLAVE_ID >> 8;
+	buf[1] = OV5640_SLAVE_ID & 0xff;
+	buf[2] = sensor->i2c_client->addr << 1;
+	msg.addr = OV5640_DEFAULT_SLAVE_ID;
+	msg.flags = 0;
+	msg.len = 3;
+	msg.buf = buf;
+
+	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
+	if (ret < 0) {
+		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
+{
+	u8 buf[3] = {0};
+	int ret;
+
+	buf[0] = reg >> 8;
+	buf[1] = reg & 0xff;
+	buf[2] = val;
+
+	ret = i2c_master_send(sensor->i2c_client, buf, 3);
+	if (ret < 0) {
+		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
+			__func__, reg, val);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
+{
+	u8 reg_buf[2] = {0};
+	u8 read_val = 0;
+
+	reg_buf[0] = reg >> 8;
+	reg_buf[1] = reg & 0xff;
+
+	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
+		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
+			__func__, reg);
+		return -EIO;
+	}
+
+	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
+		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
+			__func__, reg, read_val);
+		return -EIO;
+	}
+
+	*val = read_val;
+	return 0;
+}
+
+#define OV5640_READ_REG(s, r, v) {				\
+		ret = ov5640_read_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+#define OV5640_WRITE_REG(s, r, v) {				\
+		ret = ov5640_write_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
+{
+	u8 hi, lo;
+	int ret;
+
+	OV5640_READ_REG(sensor, reg, &hi);
+	OV5640_READ_REG(sensor, reg+1, &lo);
+
+	*val = ((u16)hi << 8) | (u16)lo;
+	return 0;
+}
+#define OV5640_READ_REG16(s, r, v) {				\
+		ret = ov5640_read_reg16((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
+{
+	int ret;
+
+	OV5640_WRITE_REG(sensor, reg, val >> 8);
+	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
+	return 0;
+}
+#define OV5640_WRITE_REG16(s, r, v) {				\
+		ret = ov5640_write_reg16((s), (r), (v));	\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
+			  u8 mask, u8 val)
+{
+	u8 readval;
+	int ret;
+
+	OV5640_READ_REG(sensor, reg, &readval);
+
+	readval &= ~mask;
+	val &= mask;
+	val |= readval;
+
+	OV5640_WRITE_REG(sensor, reg, val);
+	return 0;
+}
+#define OV5640_MOD_REG(s, r, m, v) {				\
+		ret = ov5640_mod_reg((s), (r), (m), (v));	\
+		if (ret)					\
+			return ret;				\
+	}
+
+/* download ov5640 settings to sensor through i2c */
+static int ov5640_load_regs(struct ov5640_dev *sensor,
+			    struct reg_value *regs,
+			    int size)
+{
+	register u32 delay_ms = 0;
+	register u16 reg_addr = 0;
+	register u8 mask = 0;
+	register u8 val = 0;
+	int i, ret;
+
+	for (i = 0; i < size; ++i, ++regs) {
+		delay_ms = regs->delay_ms;
+		reg_addr = regs->reg_addr;
+		val = regs->val;
+		mask = regs->mask;
+
+		if (mask) {
+			OV5640_MOD_REG(sensor, reg_addr, mask, val);
+		} else {
+			OV5640_WRITE_REG(sensor, reg_addr, val);
+		}
+		if (delay_ms)
+			usleep_range(1000*delay_ms, 1000*delay_ms+100);
+	}
+
+	return 0;
+}
+
+static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
+{
+	int ret;
+
+	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
+	return 0;
+}
+
+static int ov5640_get_sysclk(struct ov5640_dev *sensor)
+{
+	 /* calculate sysclk */
+	int xvclk = sensor->xclk_freq / 10000;
+	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
+	int sclk_rdiv_map[] = {1, 2, 4, 8};
+	int bit_div2x = 1, sclk_rdiv, sysclk;
+	u8 temp1, temp2;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3034, &temp1);
+	temp2 = temp1 & 0x0f;
+	if (temp2 == 8 || temp2 == 10)
+		bit_div2x = temp2 / 2;
+
+	OV5640_READ_REG(sensor, 0x3035, &temp1);
+	sysdiv = temp1>>4;
+	if (sysdiv == 0)
+		sysdiv = 16;
+
+	OV5640_READ_REG(sensor, 0x3036, &temp1);
+	multiplier = temp1;
+
+	OV5640_READ_REG(sensor, 0x3037, &temp1);
+	prediv = temp1 & 0x0f;
+	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
+
+	OV5640_READ_REG(sensor, 0x3108, &temp1);
+	temp2 = temp1 & 0x03;
+	sclk_rdiv = sclk_rdiv_map[temp2];
+
+	VCO = xvclk * multiplier / prediv;
+
+	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
+
+	return sysclk;
+}
+
+static int ov5640_set_night_mode(struct ov5640_dev *sensor)
+{
+	 /* read HTS from register settings */
+	u8 mode;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3a00, &mode);
+	mode &= 0xfb;
+	OV5640_WRITE_REG(sensor, 0x3a00, mode);
+	return 0;
+}
+
+static int ov5640_get_HTS(struct ov5640_dev *sensor)
+{
+	 /* read HTS from register settings */
+	u16 HTS;
+	int ret;
+
+	OV5640_READ_REG16(sensor, 0x380c, &HTS);
+	return HTS;
+}
+
+static int ov5640_get_VTS(struct ov5640_dev *sensor)
+{
+	u16 VTS;
+	int ret;
+
+	OV5640_READ_REG16(sensor, 0x380e, &VTS);
+	return VTS;
+}
+
+static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
+{
+	int ret;
+
+	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
+	return 0;
+}
+
+static int ov5640_get_light_freq(struct ov5640_dev *sensor)
+{
+	/* get banding filter value */
+	u8 temp, temp1;
+	int light_freq = 0;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3c01, &temp);
+
+	if (temp & 0x80) {
+		/* manual */
+		OV5640_READ_REG(sensor, 0x3c00, &temp1);
+		if (temp1 & 0x04) {
+			/* 50Hz */
+			light_freq = 50;
+		} else {
+			/* 60Hz */
+			light_freq = 60;
+		}
+	} else {
+		/* auto */
+		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
+		if (temp1 & 0x01) {
+			/* 50Hz */
+			light_freq = 50;
+		} else {
+			/* 60Hz */
+		}
+	}
+
+	return light_freq;
+}
+
+static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
+{
+	int prev_vts;
+	int band_step60, max_band60, band_step50, max_band50;
+	int ret;
+
+	/* read preview PCLK */
+	ret = ov5640_get_sysclk(sensor);
+	if (ret < 0)
+		return ret;
+	sensor->prev_sysclk = ret;
+	/* read preview HTS */
+	ret = ov5640_get_HTS(sensor);
+	if (ret < 0)
+		return ret;
+	sensor->prev_hts = ret;
+
+	/* read preview VTS */
+	ret = ov5640_get_VTS(sensor);
+	if (ret < 0)
+		return ret;
+	prev_vts = ret;
+
+	/* calculate banding filter */
+	/* 60Hz */
+	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
+	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
+
+	max_band60 = (int)((prev_vts-4)/band_step60);
+	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
+
+	/* 50Hz */
+	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
+	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
+
+	max_band50 = (int)((prev_vts-4)/band_step50);
+	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
+
+	return 0;
+}
+
+static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
+{
+	/* stable in high */
+	int fast_high, fast_low;
+	int ret;
+
+	sensor->ae_low = target * 23 / 25;	/* 0.92 */
+	sensor->ae_high = target * 27 / 25;	/* 1.08 */
+
+	fast_high = sensor->ae_high<<1;
+	if (fast_high > 255)
+		fast_high = 255;
+
+	fast_low = sensor->ae_low >> 1;
+
+	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
+	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
+	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
+	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
+	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
+	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
+
+	return 0;
+}
+
+static int ov5640_binning_on(struct ov5640_dev *sensor)
+{
+	u8 temp;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3821, &temp);
+	temp &= 0xfe;
+
+	return temp ? 1 : 0;
+}
+
+static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
+{
+	u8 temp, channel = sensor->ep.base.id;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x4814, &temp);
+	temp &= ~(3 << 6);
+	temp |= (channel << 6);
+	OV5640_WRITE_REG(sensor, 0x4814, temp);
+
+	return 0;
+}
+
+static enum ov5640_mode
+ov5640_find_nearest_mode(struct ov5640_dev *sensor,
+			 int width, int height)
+{
+	int i;
+
+	for (i = ov5640_num_modes - 1; i >= 0; i--) {
+		if (ov5640_mode_info_data[0][i].width <= width &&
+		    ov5640_mode_info_data[0][i].height <= height)
+			break;
+	}
+
+	if (i < 0)
+		i = 0;
+
+	return (enum ov5640_mode)i;
+}
+
+/*
+ * sensor changes between scaling and subsampling, go through
+ * exposure calculation
+ */
+static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
+					    enum ov5640_frame_rate frame_rate,
+					    enum ov5640_mode mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	u8 average;
+	int prev_shutter, prev_gain16;
+	int cap_shutter, cap_gain16;
+	int cap_sysclk, cap_hts, cap_vts;
+	int light_freq, cap_bandfilt, cap_maxband;
+	long cap_gain16_shutter;
+	int ret = 0;
+
+	/* check if the input mode and frame rate is valid */
+	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
+	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
+
+	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
+	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
+
+	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+	    mode_data == NULL || mode_size == 0)
+		return -EINVAL;
+
+	/* auto focus */
+	/* ov5640_auto_focus();//if no af function, just skip it */
+
+	/* turn off AE/AG */
+	ret = ov5640_set_agc(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* read preview shutter */
+	ret = ov5640_get_exposure(sensor);
+	if (ret < 0)
+		return ret;
+	prev_shutter = ret;
+	ret = ov5640_binning_on(sensor);
+	if (ret < 0)
+		return ret;
+	if (ret && mode != ov5640_mode_720P_1280_720 &&
+	    mode != ov5640_mode_1080P_1920_1080)
+		prev_shutter *= 2;
+
+	/* read preview gain */
+	ret = ov5640_get_gain(sensor);
+	if (ret < 0)
+		return ret;
+	prev_gain16 = ret;
+
+	/* get average */
+	OV5640_READ_REG(sensor, 0x56a1, &average);
+
+	/* turn off night mode for capture */
+	ret = ov5640_set_night_mode(sensor);
+	if (ret < 0)
+		return ret;
+
+	/* turn off overlay */
+	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
+	   just skip it */
+
+	ret = ov5640_set_stream(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* Write capture setting */
+	ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	if (ret < 0)
+		return ret;
+
+	/* read capture VTS */
+	ret = ov5640_get_VTS(sensor);
+	if (ret < 0)
+		return ret;
+	cap_vts = ret;
+	ret = ov5640_get_HTS(sensor);
+	if (ret < 0)
+		return ret;
+	cap_hts = ret;
+	ret = ov5640_get_sysclk(sensor);
+	if (ret < 0)
+		return ret;
+	cap_sysclk = ret;
+
+	/* calculate capture banding filter */
+	ret = ov5640_get_light_freq(sensor);
+	if (ret < 0)
+		return ret;
+	light_freq = ret;
+
+	if (light_freq == 60) {
+		/* 60Hz */
+		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
+	} else {
+		/* 50Hz */
+		cap_bandfilt = cap_sysclk * 100 / cap_hts;
+	}
+	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
+
+	/* calculate capture shutter/gain16 */
+	if (average > sensor->ae_low && average < sensor->ae_high) {
+		/* in stable range */
+		cap_gain16_shutter =
+			prev_gain16 * prev_shutter *
+			cap_sysclk / sensor->prev_sysclk *
+			sensor->prev_hts / cap_hts *
+			sensor->ae_target / average;
+	} else {
+		cap_gain16_shutter =
+			prev_gain16 * prev_shutter *
+			cap_sysclk / sensor->prev_sysclk *
+			sensor->prev_hts / cap_hts;
+	}
+
+	/* gain to shutter */
+	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
+		/* shutter < 1/100 */
+		cap_shutter = cap_gain16_shutter / 16;
+		if (cap_shutter < 1)
+			cap_shutter = 1;
+
+		cap_gain16 = cap_gain16_shutter / cap_shutter;
+		if (cap_gain16 < 16)
+			cap_gain16 = 16;
+	} else {
+		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
+			/* exposure reach max */
+			cap_shutter = cap_bandfilt * cap_maxband;
+			cap_gain16 = cap_gain16_shutter / cap_shutter;
+		} else {
+			/* 1/100 < (cap_shutter = n/100) =< max */
+			cap_shutter =
+				((int)(cap_gain16_shutter / 16 / cap_bandfilt))
+				* cap_bandfilt;
+			cap_gain16 = cap_gain16_shutter / cap_shutter;
+		}
+	}
+
+	/* write capture gain */
+	ret = ov5640_set_gain(sensor, cap_gain16);
+	if (ret < 0)
+		return ret;
+
+	/* write capture shutter */
+	if (cap_shutter > (cap_vts - 4)) {
+		cap_vts = cap_shutter + 4;
+		ret = ov5640_set_VTS(sensor, cap_vts);
+		if (ret < 0)
+			return ret;
+	}
+
+	ret = ov5640_set_exposure(sensor, cap_shutter);
+	if (ret < 0)
+		return ret;
+
+	return ov5640_set_stream(sensor, true);
+}
+
+/*
+ * if sensor changes inside scaling or subsampling
+ * change mode directly
+ */
+static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
+				     enum ov5640_frame_rate frame_rate,
+				     enum ov5640_mode mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	int ret = 0;
+
+	/* check if the input mode and frame rate is valid */
+	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
+	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
+
+	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
+	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
+
+	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+	    mode_data == NULL || mode_size == 0)
+		return -EINVAL;
+
+	/* turn off AE/AG */
+	ret = ov5640_set_agc(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_stream(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* Write capture setting */
+	ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_stream(sensor, true);
+	if (ret < 0)
+		return ret;
+
+	return ov5640_set_agc(sensor, true);
+}
+
+static int ov5640_change_mode(struct ov5640_dev *sensor,
+			      enum ov5640_frame_rate frame_rate,
+			      enum ov5640_mode mode,
+			      enum ov5640_mode orig_mode)
+{
+	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	int ret = 0;
+
+	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
+	    mode != ov5640_mode_INIT) {
+		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
+		return -EINVAL;
+	}
+
+	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
+	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
+	if (mode == ov5640_mode_INIT) {
+		mode_data = ov5640_init_setting_30fps_VGA;
+		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
+
+		sensor->fmt.width = 640;
+		sensor->fmt.height = 480;
+		ret = ov5640_load_regs(sensor, mode_data, mode_size);
+		if (ret < 0)
+			return ret;
+
+		mode_data = ov5640_setting_30fps_VGA_640_480;
+		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
+		ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
+			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
+		/* change between subsampling and scaling
+		 * go through exposure calucation */
+		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
+							  mode);
+	} else {
+		/* change inside subsampling or scaling
+		 * download firmware directly */
+		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
+	}
+
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_get_light_freq(sensor);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_set_bandingfilter(sensor);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_set_virtual_channel(sensor);
+	if (ret < 0)
+		return ret;
+
+	/* restore controls */
+	ov5640_restore_ctrls(sensor);
+
+	if (ret >= 0 && mode != ov5640_mode_INIT) {
+		sensor->current_mode = mode;
+		sensor->current_fr = frame_rate;
+	}
+
+	return 0;
+}
+
+/* restore the last set video mode after chip power-on */
+static int ov5640_restore_mode(struct ov5640_dev *sensor)
+{
+	int ret = 0;
+
+	/* first we need to set some initial register values */
+	ret = ov5640_change_mode(sensor, sensor->current_fr,
+				    ov5640_mode_INIT, ov5640_mode_INIT);
+	if (ret < 0)
+		return ret;
+
+	/* now restore the last capture mode */
+	return ov5640_change_mode(sensor,
+				  sensor->current_fr,
+				  sensor->current_mode,
+				  ov5640_mode_VGA_640_480);
+}
+
+static int ov5640_regulators_on(struct ov5640_dev *sensor)
+{
+	int ret;
+
+	if (sensor->io_regulator) {
+		ret = regulator_enable(sensor->io_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "io reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->core_regulator) {
+		ret = regulator_enable(sensor->core_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "core reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->gpo_regulator) {
+		ret = regulator_enable(sensor->gpo_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->analog_regulator) {
+		ret = regulator_enable(sensor->analog_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "analog reg enable failed\n");
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static void ov5640_regulators_off(struct ov5640_dev *sensor)
+{
+	if (sensor->analog_regulator)
+		regulator_disable(sensor->analog_regulator);
+	if (sensor->core_regulator)
+		regulator_disable(sensor->core_regulator);
+	if (sensor->io_regulator)
+		regulator_disable(sensor->io_regulator);
+	if (sensor->gpo_regulator)
+		regulator_disable(sensor->gpo_regulator);
+}
+
+/* --------------- Subdev Operations --------------- */
+
+static int ov5640_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	int ret;
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (on && !sensor->on) {
+		if (sensor->xclk)
+			clk_prepare_enable(sensor->xclk);
+
+		ret = ov5640_regulators_on(sensor);
+		if (ret)
+			return ret;
+
+		ov5640_reset(sensor);
+		ov5640_power(sensor, true);
+
+		ret = ov5640_init_slave_id(sensor);
+		if (ret)
+			return ret;
+
+		ret = ov5640_restore_mode(sensor);
+		if (ret)
+			return ret;
+
+		/*
+		 * NOTE: Freescale adds a long delay (600 msec) after
+		 * powering up and programming a mode on the ov5640-mipi
+		 * camera (search for "msec_wait4stable" in FSL's
+		 * ov5640_mipi.c), which equivalently would need to go
+		 * right here. If we run into MIPI CSI-2 receiver dphy
+		 * ready timeouts, it might be a clue to add that delay
+		 * here.
+		 */
+	} else if (!on && sensor->on) {
+		ov5640_power(sensor, false);
+
+		ov5640_regulators_off(sensor);
+
+		if (sensor->xclk)
+			clk_disable_unprepare(sensor->xclk);
+	}
+
+	sensor->on = on;
+
+	return 0;
+}
+
+static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	struct v4l2_captureparm *cparm = &a->parm.capture;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	/* This is the only case currently handled. */
+	memset(a, 0, sizeof(*a));
+	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	cparm->capability = sensor->streamcap.capability;
+	cparm->timeperframe = sensor->streamcap.timeperframe;
+	cparm->capturemode = sensor->streamcap.capturemode;
+
+	return 0;
+}
+
+static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
+	enum ov5640_frame_rate frame_rate;
+	u32 tgt_fps;	/* target frames per secound */
+	int ret = 0;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	/* Check that the new frame rate is allowed. */
+	if ((timeperframe->numerator == 0) ||
+	    (timeperframe->denominator == 0)) {
+		timeperframe->denominator = DEFAULT_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps > MAX_FPS) {
+		timeperframe->denominator = MAX_FPS;
+		timeperframe->numerator = 1;
+	} else if (tgt_fps < MIN_FPS) {
+		timeperframe->denominator = MIN_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	/* Actual frame rate we use */
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps == 15)
+		frame_rate = ov5640_15_fps;
+	else if (tgt_fps == 30)
+		frame_rate = ov5640_30_fps;
+	else {
+		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
+			 tgt_fps);
+		return -EINVAL;
+	}
+
+	ret = ov5640_change_mode(sensor, frame_rate,
+				 sensor->current_mode,
+				 sensor->current_mode);
+	if (ret < 0)
+		return ret;
+
+	sensor->streamcap.timeperframe = *timeperframe;
+
+	return 0;
+}
+
+static int ov5640_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	format->format = sensor->fmt;
+
+	return 0;
+}
+
+static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
+				   struct v4l2_mbus_framefmt *fmt,
+				   enum ov5640_mode *new_mode)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode mode;
+
+	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
+
+	fmt->width = ov5640_mode_info_data[0][mode].width;
+	fmt->height = ov5640_mode_info_data[0][mode].height;
+	fmt->code = sensor->fmt.code;
+
+	if (new_mode)
+		*new_mode = mode;
+	return 0;
+}
+
+static int ov5640_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode new_mode;
+	int ret;
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
+		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
+		if (ret)
+			return ret;
+		cfg->try_fmt = format->format;
+		return 0;
+	}
+
+	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
+	if (ret)
+		return ret;
+
+	ret = ov5640_change_mode(sensor, sensor->current_fr,
+				 new_mode, sensor->current_mode);
+	if (ret >= 0)
+		sensor->fmt = format->format;
+
+	return ret;
+}
+
+
+/*
+ * Sensor Controls.
+ */
+
+static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
+		OV5640_WRITE_REG16(sensor, 0x5581, value);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
+
+	return 0;
+}
+
+static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
+		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);
+
+	return 0;
+}
+
+static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
+		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
+		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);
+
+	return 0;
+}
+
+static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	sensor->awb_on = value ? true : false;
+	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
+	return 0;
+}
+
+static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
+	return 0;
+}
+
+#if 0
+static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
+	return 0;
+}
+#endif
+
+static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
+	return 0;
+}
+
+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
+{
+	u16 max_exp = 0;
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
+	if (value < max_exp) {
+		u32 exp = value << 4;
+
+		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
+		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
+		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
+	}
+
+	return 0;
+}
+
+/* read exposure, in number of line periods */
+static int ov5640_get_exposure(struct ov5640_dev *sensor)
+{
+	u8 temp;
+	int exp, ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG(sensor, 0x3500, &temp);
+	exp = ((int)temp & 0x0f) << 16;
+	OV5640_READ_REG(sensor, 0x3501, &temp);
+	exp |= ((int)temp << 8);
+	OV5640_READ_REG(sensor, 0x3502, &temp);
+	exp |= (int)temp;
+
+	return exp >> 4;
+}
+
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	/* this enables/disables both AEC and AGC */
+	sensor->agc_on = value ? true : false;
+	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
+
+	return 0;
+}
+
+static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
+	return 0;
+}
+
+static int ov5640_get_gain(struct ov5640_dev *sensor)
+{
+	u16 gain;
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG16(sensor, 0x350a, &gain);
+
+	return gain & 0x3ff;
+}
+
+#if 0
+static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
+	return 0;
+}
+#endif
+
+static struct ov5640_control ov5640_ctrls[] = {
+	{
+		.set = ov5640_set_agc,
+		.ctrl = {
+			.id = V4L2_CID_AUTOGAIN,
+			.name = "Auto Gain/Exposure Control",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5640_set_exposure,
+		.ctrl = {
+			.id = V4L2_CID_EXPOSURE,
+			.name = "Exposure",
+			.minimum = 0,
+			.maximum = 65535,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_gain,
+		.ctrl = {
+			.id = V4L2_CID_GAIN,
+			.name = "Gain",
+			.minimum = 0,
+			.maximum = 1023,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_hue,
+		.ctrl = {
+			.id = V4L2_CID_HUE,
+			.name = "Hue",
+			.minimum = 0,
+			.maximum = 359,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_contrast,
+		.ctrl = {
+			.id = V4L2_CID_CONTRAST,
+			.name = "Contrast",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_saturation,
+		.ctrl = {
+			.id = V4L2_CID_SATURATION,
+			.name = "Saturation",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 64,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_awb,
+		.ctrl = {
+			.id = V4L2_CID_AUTO_WHITE_BALANCE,
+			.name = "Auto White Balance",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5640_set_red_balance,
+		.ctrl = {
+			.id = V4L2_CID_RED_BALANCE,
+			.name = "Red Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_blue_balance,
+		.ctrl = {
+			.id = V4L2_CID_BLUE_BALANCE,
+			.name = "Blue Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	},
+};
+#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
+
+static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
+{
+	struct ov5640_control *ret = NULL;
+	int i;
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		if (id == ov5640_ctrls[i].ctrl.id) {
+			ret = &ov5640_ctrls[i];
+			break;
+		}
+	}
+
+	if (ret && index)
+		*index = i;
+	return ret;
+}
+
+static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
+{
+	struct ov5640_control *c;
+	int i;
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		c = &ov5640_ctrls[i];
+		c->set(sensor, sensor->ctrl_cache[i]);
+	}
+
+	return 0;
+}
+
+static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
+	struct ov5640_control *c;
+	int ret = 0;
+	int i;
+
+	c = ov5640_get_ctrl(ctrl->id, &i);
+	if (!c)
+		return -EINVAL;
+
+	ret = c->set(sensor, ctrl->val);
+	/* update cached value if no error */
+	if (!ret)
+		sensor->ctrl_cache[i] = ctrl->val;
+
+	return ret;
+}
+
+static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
+	.s_ctrl = ov5640_s_ctrl,
+};
+
+static int ov5640_init_controls(struct ov5640_dev *sensor)
+{
+	struct ov5640_control *c;
+	int i;
+
+	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		c = &ov5640_ctrls[i];
+
+		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
+				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
+				  c->ctrl.step, c->ctrl.default_value);
+	}
+
+	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
+	if (sensor->ctrl_hdl.error) {
+		int err = sensor->ctrl_hdl.error;
+
+		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
+		return err;
+	}
+	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_frame_size_enum *fse)
+{
+	if (fse->pad != 0)
+		return -EINVAL;
+	if (fse->index >= ov5640_num_modes)
+		return -EINVAL;
+
+	fse->min_width = fse->max_width =
+		ov5640_mode_info_data[0][fse->index].width;
+	fse->min_height = fse->max_height =
+		ov5640_mode_info_data[0][fse->index].height;
+
+	return 0;
+}
+
+static int ov5640_enum_frame_interval(
+	struct v4l2_subdev *sd,
+	struct v4l2_subdev_pad_config *cfg,
+	struct v4l2_subdev_frame_interval_enum *fie)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode mode;
+
+	if (fie->pad != 0)
+		return -EINVAL;
+	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
+		return -EINVAL;
+
+	if (fie->width == 0 || fie->height == 0)
+		return -EINVAL;
+
+	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
+
+	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
+		return -EINVAL;
+
+	fie->interval.numerator = 1;
+	fie->interval.denominator = ov5640_framerates[fie->index];
+
+	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
+		fie->width, fie->height, fie->index, fie->interval.denominator);
+	return 0;
+}
+
+static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
+
+	return 0;
+}
+
+static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
+			    u32 output, u32 config)
+{
+	return (input != 0) ? -EINVAL : 0;
+}
+
+static int ov5640_enum_mbus_code(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	if (code->pad != 0)
+		return -EINVAL;
+	if (code->index != 0)
+		return -EINVAL;
+
+	code->code = sensor->fmt.code;
+
+	return 0;
+}
+
+static int ov5640_g_mbus_config(struct v4l2_subdev *sd,
+				struct v4l2_mbus_config *cfg)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	cfg->type = V4L2_MBUS_CSI2;
+	cfg->flags = sensor->ep.bus.mipi_csi2.flags;
+	cfg->flags |= (1 << (sensor->ep.bus.mipi_csi2.num_data_lanes - 1));
+	cfg->flags |= V4L2_MBUS_CSI2_CHANNEL_0;
+
+	return 0;
+}
+
+static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	return ov5640_set_stream(sensor, enable);
+}
+
+static struct v4l2_subdev_core_ops ov5640_core_ops = {
+	.s_power = ov5640_s_power,
+};
+
+static struct v4l2_subdev_video_ops ov5640_video_ops = {
+	.s_parm = ov5640_s_parm,
+	.g_parm = ov5640_g_parm,
+	.g_input_status = ov5640_g_input_status,
+	.s_routing = ov5640_s_routing,
+	.g_mbus_config  = ov5640_g_mbus_config,
+	.s_stream = ov5640_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
+	.enum_mbus_code = ov5640_enum_mbus_code,
+	.get_fmt = ov5640_get_fmt,
+	.set_fmt = ov5640_set_fmt,
+	.enum_frame_size = ov5640_enum_frame_size,
+	.enum_frame_interval = ov5640_enum_frame_interval,
+};
+
+static struct v4l2_subdev_ops ov5640_subdev_ops = {
+	.core = &ov5640_core_ops,
+	.video = &ov5640_video_ops,
+	.pad = &ov5640_pad_ops,
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable)
+{
+	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
+}
+
+static void ov5640_reset(struct ov5640_dev *sensor)
+{
+	gpiod_set_value(sensor->reset_gpio, 0);
+
+	/* camera power cycle */
+	ov5640_power(sensor, false);
+	usleep_range(5000, 10000);
+	ov5640_power(sensor, true);
+	usleep_range(5000, 10000);
+
+	gpiod_set_value(sensor->reset_gpio, 1);
+	usleep_range(1000, 2000);
+
+	gpiod_set_value(sensor->reset_gpio, 0);
+	usleep_range(5000, 10000);
+}
+
+static void ov5640_get_regulators(struct ov5640_dev *sensor)
+{
+	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
+	if (!IS_ERR(sensor->io_regulator)) {
+		regulator_set_voltage(sensor->io_regulator,
+				      OV5640_VOLTAGE_DIGITAL_IO,
+				      OV5640_VOLTAGE_DIGITAL_IO);
+	} else {
+		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
+			__func__);
+		sensor->io_regulator = NULL;
+	}
+
+	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
+	if (!IS_ERR(sensor->core_regulator)) {
+		regulator_set_voltage(sensor->core_regulator,
+				      OV5640_VOLTAGE_DIGITAL_CORE,
+				      OV5640_VOLTAGE_DIGITAL_CORE);
+	} else {
+		sensor->core_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
+			__func__);
+	}
+
+	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
+	if (!IS_ERR(sensor->analog_regulator)) {
+		regulator_set_voltage(sensor->analog_regulator,
+				      OV5640_VOLTAGE_ANALOG,
+				      OV5640_VOLTAGE_ANALOG);
+	} else {
+		sensor->analog_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
+			__func__);
+	}
+}
+
+static int ov5640_probe(struct i2c_client *client,
+			const struct i2c_device_id *id)
+{
+	struct device *dev = &client->dev;
+	struct device_node *endpoint;
+	struct ov5640_dev *sensor;
+	int i, xclk, ret;
+
+	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
+	if (!sensor)
+		return -ENOMEM;
+
+	sensor->i2c_client = client;
+	sensor->dev = dev;
+	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
+	sensor->fmt.width = 640;
+	sensor->fmt.height = 480;
+	sensor->fmt.field = V4L2_FIELD_NONE;
+	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
+					   V4L2_CAP_TIMEPERFRAME;
+	sensor->streamcap.capturemode = 0;
+	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
+	sensor->streamcap.timeperframe.numerator = 1;
+
+	sensor->current_mode = ov5640_mode_VGA_640_480;
+	sensor->current_fr = ov5640_30_fps;
+
+	sensor->ae_target = 52;
+
+	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
+	if (!endpoint) {
+		dev_err(dev, "endpoint node not found\n");
+		return -EINVAL;
+	}
+
+	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
+	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
+		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
+		return -EINVAL;
+	}
+	of_node_put(endpoint);
+
+	/* get system clock (xclk) frequency */
+	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
+	if (!ret) {
+		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
+			dev_err(dev, "invalid xclk frequency\n");
+			return -EINVAL;
+		}
+		sensor->xclk_freq = xclk;
+	}
+
+	/* get system clock (xclk) */
+	sensor->xclk = devm_clk_get(dev, "xclk");
+	if (!IS_ERR(sensor->xclk)) {
+		if (!sensor->xclk_freq) {
+			dev_err(dev, "xclk requires xclk frequency!\n");
+			return -EINVAL;
+		}
+		clk_set_rate(sensor->xclk, sensor->xclk_freq);
+	} else {
+		/* assume system clock enabled by default */
+		sensor->xclk = NULL;
+	}
+
+	/* request power down pin */
+	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
+	if (IS_ERR(sensor->pwdn_gpio)) {
+		dev_err(dev, "request for power down gpio failed\n");
+		return PTR_ERR(sensor->pwdn_gpio);
+	}
+
+	/* request reset pin */
+	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
+	if (IS_ERR(sensor->reset_gpio)) {
+		dev_err(dev, "request for reset gpio failed\n");
+		return PTR_ERR(sensor->reset_gpio);
+	}
+
+	/* initialize the cached controls to their defaults */
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		struct ov5640_control *c = &ov5640_ctrls[i];
+
+		sensor->ctrl_cache[i] = c->ctrl.default_value;
+	}
+	sensor->awb_on = sensor->agc_on = true;
+
+	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
+
+	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
+	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
+	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
+	if (ret)
+		return ret;
+
+	ov5640_get_regulators(sensor);
+
+	ret = ov5640_s_power(&sensor->sd, 1);
+	if (ret)
+		goto entity_cleanup;
+	ret = ov5640_init_controls(sensor);
+	if (ret)
+		goto power_off;
+
+	ret = ov5640_s_power(&sensor->sd, 0);
+	if (ret)
+		goto free_ctrls;
+
+	ret = v4l2_async_register_subdev(&sensor->sd);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+
+free_ctrls:
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+power_off:
+	ov5640_s_power(&sensor->sd, 0);
+entity_cleanup:
+	media_entity_cleanup(&sensor->sd.entity);
+	ov5640_regulators_off(sensor);
+	return ret;
+}
+
+/*!
+ * ov5640 I2C detach function
+ *
+ * @param client            struct i2c_client *
+ * @return  Error code indicating success or failure
+ */
+static int ov5640_remove(struct i2c_client *client)
+{
+	struct v4l2_subdev *sd = i2c_get_clientdata(client);
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	ov5640_regulators_off(sensor);
+
+	v4l2_async_unregister_subdev(&sensor->sd);
+	media_entity_cleanup(&sensor->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static const struct i2c_device_id ov5640_id[] = {
+	{"ov5640", 0},
+	{},
+};
+MODULE_DEVICE_TABLE(i2c, ov5640_id);
+
+static const struct of_device_id ov5640_dt_ids[] = {
+	{ .compatible = "ovti,ov5640" },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, ov5640_dt_ids);
+
+static struct i2c_driver ov5640_i2c_driver = {
+	.driver = {
+		.name  = "ov5640",
+		.of_match_table	= ov5640_dt_ids,
+	},
+	.id_table = ov5640_id,
+	.probe    = ov5640_probe,
+	.remove   = ov5640_remove,
+};
+
+module_i2c_driver(ov5640_i2c_driver);
+
+MODULE_AUTHOR("Freescale Semiconductor, Inc.");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r
  Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
branch, modified heavily to bring forward to latest interfaces and code
cleanup.

Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
---
 drivers/staging/media/imx/Kconfig       |    8 +
 drivers/staging/media/imx/Makefile      |    2 +
 drivers/staging/media/imx/ov5640-mipi.c | 2348 +++++++++++++++++++++++++++++++
 3 files changed, 2358 insertions(+)
 create mode 100644 drivers/staging/media/imx/ov5640-mipi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index ce2d2c8..09f373d 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
 	---help---
 	  A video4linux camera capture driver for i.MX5/6.
 
+config IMX_OV5640_MIPI
+       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
+       depends on GPIOLIB && VIDEO_IMX_CAMERA
+       select IMX_MIPI_CSI2
+       default y
+       ---help---
+         MIPI CSI-2 OV5640 Camera support.
+
 endmenu
 endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 0decef7..aa954c1 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -10,3 +10,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
+
+obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
new file mode 100644
index 0000000..54647a7
--- /dev/null
+++ b/drivers/staging/media/imx/ov5640-mipi.c
@@ -0,0 +1,2348 @@
+/*
+ * Copyright (c) 2014 Mentor Graphics Inc.
+ * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/ctype.h>
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/of_device.h>
+#include <linux/gpio/consumer.h>
+#include <linux/regulator/consumer.h>
+#include <linux/clk.h>
+#include <linux/clk-provider.h>
+#include <linux/clkdev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+
+#define OV5640_VOLTAGE_ANALOG               2800000
+#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
+#define OV5640_VOLTAGE_DIGITAL_IO           1800000
+
+#define MIN_FPS 15
+#define MAX_FPS 30
+#define DEFAULT_FPS 30
+
+/* min/typical/max system clock (xclk) frequencies */
+#define OV5640_XCLK_MIN  6000000
+#define OV5640_XCLK_TYP 24000000
+#define OV5640_XCLK_MAX 54000000
+
+/* min/typical/max pixel clock (mclk) frequencies */
+#define OV5640_MCLK_MIN 48000000
+#define OV5640_MCLK_TYP 48000000
+#define OV5640_MCLK_MAX 96000000
+
+#define OV5640_CHIP_ID  0x300A
+#define OV5640_SLAVE_ID 0x3100
+#define OV5640_DEFAULT_SLAVE_ID 0x3c
+
+#define OV5640_MAX_CONTROLS 64
+
+enum ov5640_mode {
+	ov5640_mode_MIN = 0,
+	ov5640_mode_QCIF_176_144 = 0,
+	ov5640_mode_QVGA_320_240,
+	ov5640_mode_VGA_640_480,
+	ov5640_mode_NTSC_720_480,
+	ov5640_mode_PAL_720_576,
+	ov5640_mode_XGA_1024_768,
+	ov5640_mode_720P_1280_720,
+	ov5640_mode_1080P_1920_1080,
+	ov5640_mode_QSXGA_2592_1944,
+	ov5640_num_modes,
+	ov5640_mode_INIT = 0xff, /*only for sensor init*/
+};
+
+enum ov5640_frame_rate {
+	ov5640_15_fps,
+	ov5640_30_fps
+};
+
+static int ov5640_framerates[] = {
+	[ov5640_15_fps] = 15,
+	[ov5640_30_fps] = 30,
+};
+#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
+
+/* image size under 1280 * 960 are SUBSAMPLING
+ * image size upper 1280 * 960 are SCALING
+ */
+enum ov5640_downsize_mode {
+	SUBSAMPLING,
+	SCALING,
+};
+
+struct reg_value {
+	u16 reg_addr;
+	u8 val;
+	u8 mask;
+	u32 delay_ms;
+};
+
+struct ov5640_mode_info {
+	enum ov5640_mode mode;
+	enum ov5640_downsize_mode dn_mode;
+	u32 width;
+	u32 height;
+	struct reg_value *init_data_ptr;
+	u32 init_data_size;
+};
+
+struct ov5640_dev {
+	struct i2c_client *i2c_client;
+	struct device *dev;
+	struct v4l2_subdev sd;
+	struct media_pad pad;
+	struct v4l2_ctrl_handler ctrl_hdl;
+	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_captureparm streamcap;
+	struct clk *xclk; /* system clock to OV5640 */
+	int xclk_freq;    /* requested xclk freq from devicetree */
+
+	enum ov5640_mode current_mode;
+	enum ov5640_frame_rate current_fr;
+
+	bool on;
+	bool awb_on;
+	bool agc_on;
+
+	/* cached control settings */
+	int ctrl_cache[OV5640_MAX_CONTROLS];
+
+	struct gpio_desc *reset_gpio;
+	struct gpio_desc *pwdn_gpio;
+	struct gpio_desc *gp_gpio;
+
+	int prev_sysclk, prev_hts;
+	int ae_low, ae_high, ae_target;
+
+	struct regulator *io_regulator;
+	struct regulator *core_regulator;
+	struct regulator *analog_regulator;
+	struct regulator *gpo_regulator;
+};
+
+static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct ov5640_dev, sd);
+}
+
+static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
+{
+	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
+}
+
+struct ov5640_control {
+	struct v4l2_queryctrl ctrl;
+	int (*set)(struct ov5640_dev *sensor, int value);
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable);
+static void ov5640_reset(struct ov5640_dev *sensor);
+static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
+static int ov5640_get_exposure(struct ov5640_dev *sensor);
+static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
+static int ov5640_get_gain(struct ov5640_dev *sensor);
+
+static struct reg_value ov5640_init_setting_30fps_VGA[] = {
+
+	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
+	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
+	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
+	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
+	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
+	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
+	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
+	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
+	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
+	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
+	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
+	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
+	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
+	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
+	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
+	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
+	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
+	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
+	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
+	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
+	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
+	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
+	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
+	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
+	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
+	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
+	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
+	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
+	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
+	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
+	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
+	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
+	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
+	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
+	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
+	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
+	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
+	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
+	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
+	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
+	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
+	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
+	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
+	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
+	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
+	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
+	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
+	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
+	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
+	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
+	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
+	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
+	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
+	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
+	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
+	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
+	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
+	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
+	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
+	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
+	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
+	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
+	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
+	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
+	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
+};
+
+static struct reg_value ov5640_setting_30fps_VGA_640_480[] = {
+
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_VGA_640_480[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_XGA_1024_768[] = {
+
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
+	{0x380b, 0x00, 0, 0}, {0x3035, 0x12, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_XGA_1024_768[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3808, 0x04, 0, 0},
+	{0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0}, {0x380b, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QVGA_320_240[] = {
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QVGA_320_240[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QCIF_176_144[] = {
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+static struct reg_value ov5640_setting_15fps_QCIF_176_144[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_NTSC_720_480[] = {
+	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_NTSC_720_480[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_PAL_720_576[] = {
+	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_PAL_720_576[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_720P_1280_720[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0}, {0x4005, 0x1a, 0, 0},
+	{0x3008, 0x02, 0, 0}, {0x3503, 0,    0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_720P_1280_720[] = {
+	{0x3035, 0x41, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_1080P_1920_1080[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x11, 0, 0},
+	{0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0},
+	{0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_1080P_1920_1080[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x21, 0, 0},
+	{0x3036, 0x54, 0, 1}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
+	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
+	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
+};
+
+static struct ov5640_mode_info
+ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
+	{
+		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+		 ov5640_setting_15fps_QCIF_176_144,
+		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
+		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
+		 ov5640_setting_15fps_QVGA_320_240,
+		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
+		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
+		 ov5640_setting_15fps_VGA_640_480,
+		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
+		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+		 ov5640_setting_15fps_NTSC_720_480,
+		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
+		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+		 ov5640_setting_15fps_PAL_720_576,
+		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
+		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+		 ov5640_setting_15fps_XGA_1024_768,
+		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
+		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+		 ov5640_setting_15fps_720P_1280_720,
+		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
+		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+		 ov5640_setting_15fps_1080P_1920_1080,
+		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
+		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
+		 ov5640_setting_15fps_QSXGA_2592_1944,
+		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
+	}, {
+		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+		 ov5640_setting_30fps_QCIF_176_144,
+		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
+		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
+		 ov5640_setting_30fps_QVGA_320_240,
+		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
+		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
+		 ov5640_setting_30fps_VGA_640_480,
+		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
+		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+		 ov5640_setting_30fps_NTSC_720_480,
+		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
+		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+		 ov5640_setting_30fps_PAL_720_576,
+		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
+		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+		 ov5640_setting_30fps_XGA_1024_768,
+		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
+		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+		 ov5640_setting_30fps_720P_1280_720,
+		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
+		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+		 ov5640_setting_30fps_1080P_1920_1080,
+		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
+		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
+	},
+};
+
+static int ov5640_probe(struct i2c_client *adapter,
+			const struct i2c_device_id *device_id);
+static int ov5640_remove(struct i2c_client *client);
+
+static int ov5640_init_slave_id(struct ov5640_dev *sensor)
+{
+	struct i2c_msg msg;
+	u8 buf[4];
+	int ret;
+
+	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
+		return 0;
+
+	buf[0] = OV5640_SLAVE_ID >> 8;
+	buf[1] = OV5640_SLAVE_ID & 0xff;
+	buf[2] = sensor->i2c_client->addr << 1;
+	msg.addr = OV5640_DEFAULT_SLAVE_ID;
+	msg.flags = 0;
+	msg.len = 3;
+	msg.buf = buf;
+
+	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
+	if (ret < 0) {
+		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
+{
+	u8 buf[3] = {0};
+	int ret;
+
+	buf[0] = reg >> 8;
+	buf[1] = reg & 0xff;
+	buf[2] = val;
+
+	ret = i2c_master_send(sensor->i2c_client, buf, 3);
+	if (ret < 0) {
+		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
+			__func__, reg, val);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
+{
+	u8 reg_buf[2] = {0};
+	u8 read_val = 0;
+
+	reg_buf[0] = reg >> 8;
+	reg_buf[1] = reg & 0xff;
+
+	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
+		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
+			__func__, reg);
+		return -EIO;
+	}
+
+	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
+		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
+			__func__, reg, read_val);
+		return -EIO;
+	}
+
+	*val = read_val;
+	return 0;
+}
+
+#define OV5640_READ_REG(s, r, v) {				\
+		ret = ov5640_read_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+#define OV5640_WRITE_REG(s, r, v) {				\
+		ret = ov5640_write_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
+{
+	u8 hi, lo;
+	int ret;
+
+	OV5640_READ_REG(sensor, reg, &hi);
+	OV5640_READ_REG(sensor, reg+1, &lo);
+
+	*val = ((u16)hi << 8) | (u16)lo;
+	return 0;
+}
+#define OV5640_READ_REG16(s, r, v) {				\
+		ret = ov5640_read_reg16((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
+{
+	int ret;
+
+	OV5640_WRITE_REG(sensor, reg, val >> 8);
+	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
+	return 0;
+}
+#define OV5640_WRITE_REG16(s, r, v) {				\
+		ret = ov5640_write_reg16((s), (r), (v));	\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
+			  u8 mask, u8 val)
+{
+	u8 readval;
+	int ret;
+
+	OV5640_READ_REG(sensor, reg, &readval);
+
+	readval &= ~mask;
+	val &= mask;
+	val |= readval;
+
+	OV5640_WRITE_REG(sensor, reg, val);
+	return 0;
+}
+#define OV5640_MOD_REG(s, r, m, v) {				\
+		ret = ov5640_mod_reg((s), (r), (m), (v));	\
+		if (ret)					\
+			return ret;				\
+	}
+
+/* download ov5640 settings to sensor through i2c */
+static int ov5640_load_regs(struct ov5640_dev *sensor,
+			    struct reg_value *regs,
+			    int size)
+{
+	register u32 delay_ms = 0;
+	register u16 reg_addr = 0;
+	register u8 mask = 0;
+	register u8 val = 0;
+	int i, ret;
+
+	for (i = 0; i < size; ++i, ++regs) {
+		delay_ms = regs->delay_ms;
+		reg_addr = regs->reg_addr;
+		val = regs->val;
+		mask = regs->mask;
+
+		if (mask) {
+			OV5640_MOD_REG(sensor, reg_addr, mask, val);
+		} else {
+			OV5640_WRITE_REG(sensor, reg_addr, val);
+		}
+		if (delay_ms)
+			usleep_range(1000*delay_ms, 1000*delay_ms+100);
+	}
+
+	return 0;
+}
+
+static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
+{
+	int ret;
+
+	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
+	return 0;
+}
+
+static int ov5640_get_sysclk(struct ov5640_dev *sensor)
+{
+	 /* calculate sysclk */
+	int xvclk = sensor->xclk_freq / 10000;
+	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
+	int sclk_rdiv_map[] = {1, 2, 4, 8};
+	int bit_div2x = 1, sclk_rdiv, sysclk;
+	u8 temp1, temp2;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3034, &temp1);
+	temp2 = temp1 & 0x0f;
+	if (temp2 == 8 || temp2 == 10)
+		bit_div2x = temp2 / 2;
+
+	OV5640_READ_REG(sensor, 0x3035, &temp1);
+	sysdiv = temp1>>4;
+	if (sysdiv == 0)
+		sysdiv = 16;
+
+	OV5640_READ_REG(sensor, 0x3036, &temp1);
+	multiplier = temp1;
+
+	OV5640_READ_REG(sensor, 0x3037, &temp1);
+	prediv = temp1 & 0x0f;
+	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
+
+	OV5640_READ_REG(sensor, 0x3108, &temp1);
+	temp2 = temp1 & 0x03;
+	sclk_rdiv = sclk_rdiv_map[temp2];
+
+	VCO = xvclk * multiplier / prediv;
+
+	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
+
+	return sysclk;
+}
+
+static int ov5640_set_night_mode(struct ov5640_dev *sensor)
+{
+	 /* read HTS from register settings */
+	u8 mode;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3a00, &mode);
+	mode &= 0xfb;
+	OV5640_WRITE_REG(sensor, 0x3a00, mode);
+	return 0;
+}
+
+static int ov5640_get_HTS(struct ov5640_dev *sensor)
+{
+	 /* read HTS from register settings */
+	u16 HTS;
+	int ret;
+
+	OV5640_READ_REG16(sensor, 0x380c, &HTS);
+	return HTS;
+}
+
+static int ov5640_get_VTS(struct ov5640_dev *sensor)
+{
+	u16 VTS;
+	int ret;
+
+	OV5640_READ_REG16(sensor, 0x380e, &VTS);
+	return VTS;
+}
+
+static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
+{
+	int ret;
+
+	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
+	return 0;
+}
+
+static int ov5640_get_light_freq(struct ov5640_dev *sensor)
+{
+	/* get banding filter value */
+	u8 temp, temp1;
+	int light_freq = 0;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3c01, &temp);
+
+	if (temp & 0x80) {
+		/* manual */
+		OV5640_READ_REG(sensor, 0x3c00, &temp1);
+		if (temp1 & 0x04) {
+			/* 50Hz */
+			light_freq = 50;
+		} else {
+			/* 60Hz */
+			light_freq = 60;
+		}
+	} else {
+		/* auto */
+		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
+		if (temp1 & 0x01) {
+			/* 50Hz */
+			light_freq = 50;
+		} else {
+			/* 60Hz */
+		}
+	}
+
+	return light_freq;
+}
+
+static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
+{
+	int prev_vts;
+	int band_step60, max_band60, band_step50, max_band50;
+	int ret;
+
+	/* read preview PCLK */
+	ret = ov5640_get_sysclk(sensor);
+	if (ret < 0)
+		return ret;
+	sensor->prev_sysclk = ret;
+	/* read preview HTS */
+	ret = ov5640_get_HTS(sensor);
+	if (ret < 0)
+		return ret;
+	sensor->prev_hts = ret;
+
+	/* read preview VTS */
+	ret = ov5640_get_VTS(sensor);
+	if (ret < 0)
+		return ret;
+	prev_vts = ret;
+
+	/* calculate banding filter */
+	/* 60Hz */
+	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
+	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
+
+	max_band60 = (int)((prev_vts-4)/band_step60);
+	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
+
+	/* 50Hz */
+	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
+	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
+
+	max_band50 = (int)((prev_vts-4)/band_step50);
+	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
+
+	return 0;
+}
+
+static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
+{
+	/* stable in high */
+	int fast_high, fast_low;
+	int ret;
+
+	sensor->ae_low = target * 23 / 25;	/* 0.92 */
+	sensor->ae_high = target * 27 / 25;	/* 1.08 */
+
+	fast_high = sensor->ae_high<<1;
+	if (fast_high > 255)
+		fast_high = 255;
+
+	fast_low = sensor->ae_low >> 1;
+
+	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
+	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
+	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
+	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
+	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
+	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
+
+	return 0;
+}
+
+static int ov5640_binning_on(struct ov5640_dev *sensor)
+{
+	u8 temp;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3821, &temp);
+	temp &= 0xfe;
+
+	return temp ? 1 : 0;
+}
+
+static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
+{
+	u8 temp, channel = sensor->ep.base.id;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x4814, &temp);
+	temp &= ~(3 << 6);
+	temp |= (channel << 6);
+	OV5640_WRITE_REG(sensor, 0x4814, temp);
+
+	return 0;
+}
+
+static enum ov5640_mode
+ov5640_find_nearest_mode(struct ov5640_dev *sensor,
+			 int width, int height)
+{
+	int i;
+
+	for (i = ov5640_num_modes - 1; i >= 0; i--) {
+		if (ov5640_mode_info_data[0][i].width <= width &&
+		    ov5640_mode_info_data[0][i].height <= height)
+			break;
+	}
+
+	if (i < 0)
+		i = 0;
+
+	return (enum ov5640_mode)i;
+}
+
+/*
+ * sensor changes between scaling and subsampling, go through
+ * exposure calculation
+ */
+static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
+					    enum ov5640_frame_rate frame_rate,
+					    enum ov5640_mode mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	u8 average;
+	int prev_shutter, prev_gain16;
+	int cap_shutter, cap_gain16;
+	int cap_sysclk, cap_hts, cap_vts;
+	int light_freq, cap_bandfilt, cap_maxband;
+	long cap_gain16_shutter;
+	int ret = 0;
+
+	/* check if the input mode and frame rate is valid */
+	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
+	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
+
+	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
+	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
+
+	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+	    mode_data == NULL || mode_size == 0)
+		return -EINVAL;
+
+	/* auto focus */
+	/* ov5640_auto_focus();//if no af function, just skip it */
+
+	/* turn off AE/AG */
+	ret = ov5640_set_agc(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* read preview shutter */
+	ret = ov5640_get_exposure(sensor);
+	if (ret < 0)
+		return ret;
+	prev_shutter = ret;
+	ret = ov5640_binning_on(sensor);
+	if (ret < 0)
+		return ret;
+	if (ret && mode != ov5640_mode_720P_1280_720 &&
+	    mode != ov5640_mode_1080P_1920_1080)
+		prev_shutter *= 2;
+
+	/* read preview gain */
+	ret = ov5640_get_gain(sensor);
+	if (ret < 0)
+		return ret;
+	prev_gain16 = ret;
+
+	/* get average */
+	OV5640_READ_REG(sensor, 0x56a1, &average);
+
+	/* turn off night mode for capture */
+	ret = ov5640_set_night_mode(sensor);
+	if (ret < 0)
+		return ret;
+
+	/* turn off overlay */
+	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
+	   just skip it */
+
+	ret = ov5640_set_stream(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* Write capture setting */
+	ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	if (ret < 0)
+		return ret;
+
+	/* read capture VTS */
+	ret = ov5640_get_VTS(sensor);
+	if (ret < 0)
+		return ret;
+	cap_vts = ret;
+	ret = ov5640_get_HTS(sensor);
+	if (ret < 0)
+		return ret;
+	cap_hts = ret;
+	ret = ov5640_get_sysclk(sensor);
+	if (ret < 0)
+		return ret;
+	cap_sysclk = ret;
+
+	/* calculate capture banding filter */
+	ret = ov5640_get_light_freq(sensor);
+	if (ret < 0)
+		return ret;
+	light_freq = ret;
+
+	if (light_freq == 60) {
+		/* 60Hz */
+		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
+	} else {
+		/* 50Hz */
+		cap_bandfilt = cap_sysclk * 100 / cap_hts;
+	}
+	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
+
+	/* calculate capture shutter/gain16 */
+	if (average > sensor->ae_low && average < sensor->ae_high) {
+		/* in stable range */
+		cap_gain16_shutter =
+			prev_gain16 * prev_shutter *
+			cap_sysclk / sensor->prev_sysclk *
+			sensor->prev_hts / cap_hts *
+			sensor->ae_target / average;
+	} else {
+		cap_gain16_shutter =
+			prev_gain16 * prev_shutter *
+			cap_sysclk / sensor->prev_sysclk *
+			sensor->prev_hts / cap_hts;
+	}
+
+	/* gain to shutter */
+	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
+		/* shutter < 1/100 */
+		cap_shutter = cap_gain16_shutter / 16;
+		if (cap_shutter < 1)
+			cap_shutter = 1;
+
+		cap_gain16 = cap_gain16_shutter / cap_shutter;
+		if (cap_gain16 < 16)
+			cap_gain16 = 16;
+	} else {
+		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
+			/* exposure reach max */
+			cap_shutter = cap_bandfilt * cap_maxband;
+			cap_gain16 = cap_gain16_shutter / cap_shutter;
+		} else {
+			/* 1/100 < (cap_shutter = n/100) =< max */
+			cap_shutter =
+				((int)(cap_gain16_shutter / 16 / cap_bandfilt))
+				* cap_bandfilt;
+			cap_gain16 = cap_gain16_shutter / cap_shutter;
+		}
+	}
+
+	/* write capture gain */
+	ret = ov5640_set_gain(sensor, cap_gain16);
+	if (ret < 0)
+		return ret;
+
+	/* write capture shutter */
+	if (cap_shutter > (cap_vts - 4)) {
+		cap_vts = cap_shutter + 4;
+		ret = ov5640_set_VTS(sensor, cap_vts);
+		if (ret < 0)
+			return ret;
+	}
+
+	ret = ov5640_set_exposure(sensor, cap_shutter);
+	if (ret < 0)
+		return ret;
+
+	return ov5640_set_stream(sensor, true);
+}
+
+/*
+ * if sensor changes inside scaling or subsampling
+ * change mode directly
+ */
+static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
+				     enum ov5640_frame_rate frame_rate,
+				     enum ov5640_mode mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	int ret = 0;
+
+	/* check if the input mode and frame rate is valid */
+	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
+	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
+
+	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
+	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
+
+	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+	    mode_data == NULL || mode_size == 0)
+		return -EINVAL;
+
+	/* turn off AE/AG */
+	ret = ov5640_set_agc(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_stream(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* Write capture setting */
+	ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_stream(sensor, true);
+	if (ret < 0)
+		return ret;
+
+	return ov5640_set_agc(sensor, true);
+}
+
+static int ov5640_change_mode(struct ov5640_dev *sensor,
+			      enum ov5640_frame_rate frame_rate,
+			      enum ov5640_mode mode,
+			      enum ov5640_mode orig_mode)
+{
+	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	int ret = 0;
+
+	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
+	    mode != ov5640_mode_INIT) {
+		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
+		return -EINVAL;
+	}
+
+	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
+	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
+	if (mode == ov5640_mode_INIT) {
+		mode_data = ov5640_init_setting_30fps_VGA;
+		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
+
+		sensor->fmt.width = 640;
+		sensor->fmt.height = 480;
+		ret = ov5640_load_regs(sensor, mode_data, mode_size);
+		if (ret < 0)
+			return ret;
+
+		mode_data = ov5640_setting_30fps_VGA_640_480;
+		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
+		ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
+			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
+		/* change between subsampling and scaling
+		 * go through exposure calucation */
+		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
+							  mode);
+	} else {
+		/* change inside subsampling or scaling
+		 * download firmware directly */
+		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
+	}
+
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_get_light_freq(sensor);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_set_bandingfilter(sensor);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_set_virtual_channel(sensor);
+	if (ret < 0)
+		return ret;
+
+	/* restore controls */
+	ov5640_restore_ctrls(sensor);
+
+	if (ret >= 0 && mode != ov5640_mode_INIT) {
+		sensor->current_mode = mode;
+		sensor->current_fr = frame_rate;
+	}
+
+	return 0;
+}
+
+/* restore the last set video mode after chip power-on */
+static int ov5640_restore_mode(struct ov5640_dev *sensor)
+{
+	int ret = 0;
+
+	/* first we need to set some initial register values */
+	ret = ov5640_change_mode(sensor, sensor->current_fr,
+				    ov5640_mode_INIT, ov5640_mode_INIT);
+	if (ret < 0)
+		return ret;
+
+	/* now restore the last capture mode */
+	return ov5640_change_mode(sensor,
+				  sensor->current_fr,
+				  sensor->current_mode,
+				  ov5640_mode_VGA_640_480);
+}
+
+static int ov5640_regulators_on(struct ov5640_dev *sensor)
+{
+	int ret;
+
+	if (sensor->io_regulator) {
+		ret = regulator_enable(sensor->io_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "io reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->core_regulator) {
+		ret = regulator_enable(sensor->core_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "core reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->gpo_regulator) {
+		ret = regulator_enable(sensor->gpo_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->analog_regulator) {
+		ret = regulator_enable(sensor->analog_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "analog reg enable failed\n");
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static void ov5640_regulators_off(struct ov5640_dev *sensor)
+{
+	if (sensor->analog_regulator)
+		regulator_disable(sensor->analog_regulator);
+	if (sensor->core_regulator)
+		regulator_disable(sensor->core_regulator);
+	if (sensor->io_regulator)
+		regulator_disable(sensor->io_regulator);
+	if (sensor->gpo_regulator)
+		regulator_disable(sensor->gpo_regulator);
+}
+
+/* --------------- Subdev Operations --------------- */
+
+static int ov5640_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	int ret;
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (on && !sensor->on) {
+		if (sensor->xclk)
+			clk_prepare_enable(sensor->xclk);
+
+		ret = ov5640_regulators_on(sensor);
+		if (ret)
+			return ret;
+
+		ov5640_reset(sensor);
+		ov5640_power(sensor, true);
+
+		ret = ov5640_init_slave_id(sensor);
+		if (ret)
+			return ret;
+
+		ret = ov5640_restore_mode(sensor);
+		if (ret)
+			return ret;
+
+		/*
+		 * NOTE: Freescale adds a long delay (600 msec) after
+		 * powering up and programming a mode on the ov5640-mipi
+		 * camera (search for "msec_wait4stable" in FSL's
+		 * ov5640_mipi.c), which equivalently would need to go
+		 * right here. If we run into MIPI CSI-2 receiver dphy
+		 * ready timeouts, it might be a clue to add that delay
+		 * here.
+		 */
+	} else if (!on && sensor->on) {
+		ov5640_power(sensor, false);
+
+		ov5640_regulators_off(sensor);
+
+		if (sensor->xclk)
+			clk_disable_unprepare(sensor->xclk);
+	}
+
+	sensor->on = on;
+
+	return 0;
+}
+
+static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	struct v4l2_captureparm *cparm = &a->parm.capture;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	/* This is the only case currently handled. */
+	memset(a, 0, sizeof(*a));
+	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	cparm->capability = sensor->streamcap.capability;
+	cparm->timeperframe = sensor->streamcap.timeperframe;
+	cparm->capturemode = sensor->streamcap.capturemode;
+
+	return 0;
+}
+
+static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
+	enum ov5640_frame_rate frame_rate;
+	u32 tgt_fps;	/* target frames per secound */
+	int ret = 0;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	/* Check that the new frame rate is allowed. */
+	if ((timeperframe->numerator == 0) ||
+	    (timeperframe->denominator == 0)) {
+		timeperframe->denominator = DEFAULT_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps > MAX_FPS) {
+		timeperframe->denominator = MAX_FPS;
+		timeperframe->numerator = 1;
+	} else if (tgt_fps < MIN_FPS) {
+		timeperframe->denominator = MIN_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	/* Actual frame rate we use */
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps == 15)
+		frame_rate = ov5640_15_fps;
+	else if (tgt_fps == 30)
+		frame_rate = ov5640_30_fps;
+	else {
+		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
+			 tgt_fps);
+		return -EINVAL;
+	}
+
+	ret = ov5640_change_mode(sensor, frame_rate,
+				 sensor->current_mode,
+				 sensor->current_mode);
+	if (ret < 0)
+		return ret;
+
+	sensor->streamcap.timeperframe = *timeperframe;
+
+	return 0;
+}
+
+static int ov5640_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	format->format = sensor->fmt;
+
+	return 0;
+}
+
+static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
+				   struct v4l2_mbus_framefmt *fmt,
+				   enum ov5640_mode *new_mode)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode mode;
+
+	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
+
+	fmt->width = ov5640_mode_info_data[0][mode].width;
+	fmt->height = ov5640_mode_info_data[0][mode].height;
+	fmt->code = sensor->fmt.code;
+
+	if (new_mode)
+		*new_mode = mode;
+	return 0;
+}
+
+static int ov5640_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode new_mode;
+	int ret;
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
+		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
+		if (ret)
+			return ret;
+		cfg->try_fmt = format->format;
+		return 0;
+	}
+
+	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
+	if (ret)
+		return ret;
+
+	ret = ov5640_change_mode(sensor, sensor->current_fr,
+				 new_mode, sensor->current_mode);
+	if (ret >= 0)
+		sensor->fmt = format->format;
+
+	return ret;
+}
+
+
+/*
+ * Sensor Controls.
+ */
+
+static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
+		OV5640_WRITE_REG16(sensor, 0x5581, value);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
+
+	return 0;
+}
+
+static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
+		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);
+
+	return 0;
+}
+
+static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
+		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
+		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);
+
+	return 0;
+}
+
+static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	sensor->awb_on = value ? true : false;
+	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
+	return 0;
+}
+
+static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
+	return 0;
+}
+
+#if 0
+static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
+	return 0;
+}
+#endif
+
+static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
+	return 0;
+}
+
+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
+{
+	u16 max_exp = 0;
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
+	if (value < max_exp) {
+		u32 exp = value << 4;
+
+		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
+		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
+		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
+	}
+
+	return 0;
+}
+
+/* read exposure, in number of line periods */
+static int ov5640_get_exposure(struct ov5640_dev *sensor)
+{
+	u8 temp;
+	int exp, ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG(sensor, 0x3500, &temp);
+	exp = ((int)temp & 0x0f) << 16;
+	OV5640_READ_REG(sensor, 0x3501, &temp);
+	exp |= ((int)temp << 8);
+	OV5640_READ_REG(sensor, 0x3502, &temp);
+	exp |= (int)temp;
+
+	return exp >> 4;
+}
+
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	/* this enables/disables both AEC and AGC */
+	sensor->agc_on = value ? true : false;
+	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
+
+	return 0;
+}
+
+static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
+	return 0;
+}
+
+static int ov5640_get_gain(struct ov5640_dev *sensor)
+{
+	u16 gain;
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG16(sensor, 0x350a, &gain);
+
+	return gain & 0x3ff;
+}
+
+#if 0
+static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
+	return 0;
+}
+#endif
+
+static struct ov5640_control ov5640_ctrls[] = {
+	{
+		.set = ov5640_set_agc,
+		.ctrl = {
+			.id = V4L2_CID_AUTOGAIN,
+			.name = "Auto Gain/Exposure Control",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5640_set_exposure,
+		.ctrl = {
+			.id = V4L2_CID_EXPOSURE,
+			.name = "Exposure",
+			.minimum = 0,
+			.maximum = 65535,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_gain,
+		.ctrl = {
+			.id = V4L2_CID_GAIN,
+			.name = "Gain",
+			.minimum = 0,
+			.maximum = 1023,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_hue,
+		.ctrl = {
+			.id = V4L2_CID_HUE,
+			.name = "Hue",
+			.minimum = 0,
+			.maximum = 359,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_contrast,
+		.ctrl = {
+			.id = V4L2_CID_CONTRAST,
+			.name = "Contrast",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_saturation,
+		.ctrl = {
+			.id = V4L2_CID_SATURATION,
+			.name = "Saturation",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 64,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_awb,
+		.ctrl = {
+			.id = V4L2_CID_AUTO_WHITE_BALANCE,
+			.name = "Auto White Balance",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5640_set_red_balance,
+		.ctrl = {
+			.id = V4L2_CID_RED_BALANCE,
+			.name = "Red Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_blue_balance,
+		.ctrl = {
+			.id = V4L2_CID_BLUE_BALANCE,
+			.name = "Blue Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	},
+};
+#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
+
+static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
+{
+	struct ov5640_control *ret = NULL;
+	int i;
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		if (id == ov5640_ctrls[i].ctrl.id) {
+			ret = &ov5640_ctrls[i];
+			break;
+		}
+	}
+
+	if (ret && index)
+		*index = i;
+	return ret;
+}
+
+static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
+{
+	struct ov5640_control *c;
+	int i;
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		c = &ov5640_ctrls[i];
+		c->set(sensor, sensor->ctrl_cache[i]);
+	}
+
+	return 0;
+}
+
+static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
+	struct ov5640_control *c;
+	int ret = 0;
+	int i;
+
+	c = ov5640_get_ctrl(ctrl->id, &i);
+	if (!c)
+		return -EINVAL;
+
+	ret = c->set(sensor, ctrl->val);
+	/* update cached value if no error */
+	if (!ret)
+		sensor->ctrl_cache[i] = ctrl->val;
+
+	return ret;
+}
+
+static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
+	.s_ctrl = ov5640_s_ctrl,
+};
+
+static int ov5640_init_controls(struct ov5640_dev *sensor)
+{
+	struct ov5640_control *c;
+	int i;
+
+	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		c = &ov5640_ctrls[i];
+
+		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
+				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
+				  c->ctrl.step, c->ctrl.default_value);
+	}
+
+	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
+	if (sensor->ctrl_hdl.error) {
+		int err = sensor->ctrl_hdl.error;
+
+		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
+		return err;
+	}
+	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_frame_size_enum *fse)
+{
+	if (fse->pad != 0)
+		return -EINVAL;
+	if (fse->index >= ov5640_num_modes)
+		return -EINVAL;
+
+	fse->min_width = fse->max_width =
+		ov5640_mode_info_data[0][fse->index].width;
+	fse->min_height = fse->max_height =
+		ov5640_mode_info_data[0][fse->index].height;
+
+	return 0;
+}
+
+static int ov5640_enum_frame_interval(
+	struct v4l2_subdev *sd,
+	struct v4l2_subdev_pad_config *cfg,
+	struct v4l2_subdev_frame_interval_enum *fie)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode mode;
+
+	if (fie->pad != 0)
+		return -EINVAL;
+	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
+		return -EINVAL;
+
+	if (fie->width == 0 || fie->height == 0)
+		return -EINVAL;
+
+	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
+
+	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
+		return -EINVAL;
+
+	fie->interval.numerator = 1;
+	fie->interval.denominator = ov5640_framerates[fie->index];
+
+	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
+		fie->width, fie->height, fie->index, fie->interval.denominator);
+	return 0;
+}
+
+static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
+
+	return 0;
+}
+
+static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
+			    u32 output, u32 config)
+{
+	return (input != 0) ? -EINVAL : 0;
+}
+
+static int ov5640_enum_mbus_code(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	if (code->pad != 0)
+		return -EINVAL;
+	if (code->index != 0)
+		return -EINVAL;
+
+	code->code = sensor->fmt.code;
+
+	return 0;
+}
+
+static int ov5640_g_mbus_config(struct v4l2_subdev *sd,
+				struct v4l2_mbus_config *cfg)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	cfg->type = V4L2_MBUS_CSI2;
+	cfg->flags = sensor->ep.bus.mipi_csi2.flags;
+	cfg->flags |= (1 << (sensor->ep.bus.mipi_csi2.num_data_lanes - 1));
+	cfg->flags |= V4L2_MBUS_CSI2_CHANNEL_0;
+
+	return 0;
+}
+
+static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	return ov5640_set_stream(sensor, enable);
+}
+
+static struct v4l2_subdev_core_ops ov5640_core_ops = {
+	.s_power = ov5640_s_power,
+};
+
+static struct v4l2_subdev_video_ops ov5640_video_ops = {
+	.s_parm = ov5640_s_parm,
+	.g_parm = ov5640_g_parm,
+	.g_input_status = ov5640_g_input_status,
+	.s_routing = ov5640_s_routing,
+	.g_mbus_config  = ov5640_g_mbus_config,
+	.s_stream = ov5640_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
+	.enum_mbus_code = ov5640_enum_mbus_code,
+	.get_fmt = ov5640_get_fmt,
+	.set_fmt = ov5640_set_fmt,
+	.enum_frame_size = ov5640_enum_frame_size,
+	.enum_frame_interval = ov5640_enum_frame_interval,
+};
+
+static struct v4l2_subdev_ops ov5640_subdev_ops = {
+	.core = &ov5640_core_ops,
+	.video = &ov5640_video_ops,
+	.pad = &ov5640_pad_ops,
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable)
+{
+	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
+}
+
+static void ov5640_reset(struct ov5640_dev *sensor)
+{
+	gpiod_set_value(sensor->reset_gpio, 0);
+
+	/* camera power cycle */
+	ov5640_power(sensor, false);
+	usleep_range(5000, 10000);
+	ov5640_power(sensor, true);
+	usleep_range(5000, 10000);
+
+	gpiod_set_value(sensor->reset_gpio, 1);
+	usleep_range(1000, 2000);
+
+	gpiod_set_value(sensor->reset_gpio, 0);
+	usleep_range(5000, 10000);
+}
+
+static void ov5640_get_regulators(struct ov5640_dev *sensor)
+{
+	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
+	if (!IS_ERR(sensor->io_regulator)) {
+		regulator_set_voltage(sensor->io_regulator,
+				      OV5640_VOLTAGE_DIGITAL_IO,
+				      OV5640_VOLTAGE_DIGITAL_IO);
+	} else {
+		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
+			__func__);
+		sensor->io_regulator = NULL;
+	}
+
+	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
+	if (!IS_ERR(sensor->core_regulator)) {
+		regulator_set_voltage(sensor->core_regulator,
+				      OV5640_VOLTAGE_DIGITAL_CORE,
+				      OV5640_VOLTAGE_DIGITAL_CORE);
+	} else {
+		sensor->core_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
+			__func__);
+	}
+
+	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
+	if (!IS_ERR(sensor->analog_regulator)) {
+		regulator_set_voltage(sensor->analog_regulator,
+				      OV5640_VOLTAGE_ANALOG,
+				      OV5640_VOLTAGE_ANALOG);
+	} else {
+		sensor->analog_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
+			__func__);
+	}
+}
+
+static int ov5640_probe(struct i2c_client *client,
+			const struct i2c_device_id *id)
+{
+	struct device *dev = &client->dev;
+	struct device_node *endpoint;
+	struct ov5640_dev *sensor;
+	int i, xclk, ret;
+
+	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
+	if (!sensor)
+		return -ENOMEM;
+
+	sensor->i2c_client = client;
+	sensor->dev = dev;
+	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
+	sensor->fmt.width = 640;
+	sensor->fmt.height = 480;
+	sensor->fmt.field = V4L2_FIELD_NONE;
+	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
+					   V4L2_CAP_TIMEPERFRAME;
+	sensor->streamcap.capturemode = 0;
+	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
+	sensor->streamcap.timeperframe.numerator = 1;
+
+	sensor->current_mode = ov5640_mode_VGA_640_480;
+	sensor->current_fr = ov5640_30_fps;
+
+	sensor->ae_target = 52;
+
+	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
+	if (!endpoint) {
+		dev_err(dev, "endpoint node not found\n");
+		return -EINVAL;
+	}
+
+	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
+	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
+		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
+		return -EINVAL;
+	}
+	of_node_put(endpoint);
+
+	/* get system clock (xclk) frequency */
+	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
+	if (!ret) {
+		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
+			dev_err(dev, "invalid xclk frequency\n");
+			return -EINVAL;
+		}
+		sensor->xclk_freq = xclk;
+	}
+
+	/* get system clock (xclk) */
+	sensor->xclk = devm_clk_get(dev, "xclk");
+	if (!IS_ERR(sensor->xclk)) {
+		if (!sensor->xclk_freq) {
+			dev_err(dev, "xclk requires xclk frequency!\n");
+			return -EINVAL;
+		}
+		clk_set_rate(sensor->xclk, sensor->xclk_freq);
+	} else {
+		/* assume system clock enabled by default */
+		sensor->xclk = NULL;
+	}
+
+	/* request power down pin */
+	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
+	if (IS_ERR(sensor->pwdn_gpio)) {
+		dev_err(dev, "request for power down gpio failed\n");
+		return PTR_ERR(sensor->pwdn_gpio);
+	}
+
+	/* request reset pin */
+	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
+	if (IS_ERR(sensor->reset_gpio)) {
+		dev_err(dev, "request for reset gpio failed\n");
+		return PTR_ERR(sensor->reset_gpio);
+	}
+
+	/* initialize the cached controls to their defaults */
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		struct ov5640_control *c = &ov5640_ctrls[i];
+
+		sensor->ctrl_cache[i] = c->ctrl.default_value;
+	}
+	sensor->awb_on = sensor->agc_on = true;
+
+	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
+
+	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
+	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
+	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
+	if (ret)
+		return ret;
+
+	ov5640_get_regulators(sensor);
+
+	ret = ov5640_s_power(&sensor->sd, 1);
+	if (ret)
+		goto entity_cleanup;
+	ret = ov5640_init_controls(sensor);
+	if (ret)
+		goto power_off;
+
+	ret = ov5640_s_power(&sensor->sd, 0);
+	if (ret)
+		goto free_ctrls;
+
+	ret = v4l2_async_register_subdev(&sensor->sd);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+
+free_ctrls:
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+power_off:
+	ov5640_s_power(&sensor->sd, 0);
+entity_cleanup:
+	media_entity_cleanup(&sensor->sd.entity);
+	ov5640_regulators_off(sensor);
+	return ret;
+}
+
+/*!
+ * ov5640 I2C detach function
+ *
+ * @param client            struct i2c_client *
+ * @return  Error code indicating success or failure
+ */
+static int ov5640_remove(struct i2c_client *client)
+{
+	struct v4l2_subdev *sd = i2c_get_clientdata(client);
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	ov5640_regulators_off(sensor);
+
+	v4l2_async_unregister_subdev(&sensor->sd);
+	media_entity_cleanup(&sensor->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static const struct i2c_device_id ov5640_id[] = {
+	{"ov5640", 0},
+	{},
+};
+MODULE_DEVICE_TABLE(i2c, ov5640_id);
+
+static const struct of_device_id ov5640_dt_ids[] = {
+	{ .compatible = "ovti,ov5640" },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, ov5640_dt_ids);
+
+static struct i2c_driver ov5640_i2c_driver = {
+	.driver = {
+		.name  = "ov5640",
+		.of_match_table	= ov5640_dt_ids,
+	},
+	.id_table = ov5640_id,
+	.probe    = ov5640_probe,
+	.remove   = ov5640_remove,
+};
+
+module_i2c_driver(ov5640_i2c_driver);
+
+MODULE_AUTHOR("Freescale Semiconductor, Inc.");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>");
+MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
branch, modified heavily to bring forward to latest interfaces and code
cleanup.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Kconfig       |    8 +
 drivers/staging/media/imx/Makefile      |    2 +
 drivers/staging/media/imx/ov5640-mipi.c | 2348 +++++++++++++++++++++++++++++++
 3 files changed, 2358 insertions(+)
 create mode 100644 drivers/staging/media/imx/ov5640-mipi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index ce2d2c8..09f373d 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
 	---help---
 	  A video4linux camera capture driver for i.MX5/6.
 
+config IMX_OV5640_MIPI
+       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
+       depends on GPIOLIB && VIDEO_IMX_CAMERA
+       select IMX_MIPI_CSI2
+       default y
+       ---help---
+         MIPI CSI-2 OV5640 Camera support.
+
 endmenu
 endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 0decef7..aa954c1 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -10,3 +10,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
+
+obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
new file mode 100644
index 0000000..54647a7
--- /dev/null
+++ b/drivers/staging/media/imx/ov5640-mipi.c
@@ -0,0 +1,2348 @@
+/*
+ * Copyright (c) 2014 Mentor Graphics Inc.
+ * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/ctype.h>
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/of_device.h>
+#include <linux/gpio/consumer.h>
+#include <linux/regulator/consumer.h>
+#include <linux/clk.h>
+#include <linux/clk-provider.h>
+#include <linux/clkdev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+
+#define OV5640_VOLTAGE_ANALOG               2800000
+#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
+#define OV5640_VOLTAGE_DIGITAL_IO           1800000
+
+#define MIN_FPS 15
+#define MAX_FPS 30
+#define DEFAULT_FPS 30
+
+/* min/typical/max system clock (xclk) frequencies */
+#define OV5640_XCLK_MIN  6000000
+#define OV5640_XCLK_TYP 24000000
+#define OV5640_XCLK_MAX 54000000
+
+/* min/typical/max pixel clock (mclk) frequencies */
+#define OV5640_MCLK_MIN 48000000
+#define OV5640_MCLK_TYP 48000000
+#define OV5640_MCLK_MAX 96000000
+
+#define OV5640_CHIP_ID  0x300A
+#define OV5640_SLAVE_ID 0x3100
+#define OV5640_DEFAULT_SLAVE_ID 0x3c
+
+#define OV5640_MAX_CONTROLS 64
+
+enum ov5640_mode {
+	ov5640_mode_MIN = 0,
+	ov5640_mode_QCIF_176_144 = 0,
+	ov5640_mode_QVGA_320_240,
+	ov5640_mode_VGA_640_480,
+	ov5640_mode_NTSC_720_480,
+	ov5640_mode_PAL_720_576,
+	ov5640_mode_XGA_1024_768,
+	ov5640_mode_720P_1280_720,
+	ov5640_mode_1080P_1920_1080,
+	ov5640_mode_QSXGA_2592_1944,
+	ov5640_num_modes,
+	ov5640_mode_INIT = 0xff, /*only for sensor init*/
+};
+
+enum ov5640_frame_rate {
+	ov5640_15_fps,
+	ov5640_30_fps
+};
+
+static int ov5640_framerates[] = {
+	[ov5640_15_fps] = 15,
+	[ov5640_30_fps] = 30,
+};
+#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
+
+/* image size under 1280 * 960 are SUBSAMPLING
+ * image size upper 1280 * 960 are SCALING
+ */
+enum ov5640_downsize_mode {
+	SUBSAMPLING,
+	SCALING,
+};
+
+struct reg_value {
+	u16 reg_addr;
+	u8 val;
+	u8 mask;
+	u32 delay_ms;
+};
+
+struct ov5640_mode_info {
+	enum ov5640_mode mode;
+	enum ov5640_downsize_mode dn_mode;
+	u32 width;
+	u32 height;
+	struct reg_value *init_data_ptr;
+	u32 init_data_size;
+};
+
+struct ov5640_dev {
+	struct i2c_client *i2c_client;
+	struct device *dev;
+	struct v4l2_subdev sd;
+	struct media_pad pad;
+	struct v4l2_ctrl_handler ctrl_hdl;
+	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_captureparm streamcap;
+	struct clk *xclk; /* system clock to OV5640 */
+	int xclk_freq;    /* requested xclk freq from devicetree */
+
+	enum ov5640_mode current_mode;
+	enum ov5640_frame_rate current_fr;
+
+	bool on;
+	bool awb_on;
+	bool agc_on;
+
+	/* cached control settings */
+	int ctrl_cache[OV5640_MAX_CONTROLS];
+
+	struct gpio_desc *reset_gpio;
+	struct gpio_desc *pwdn_gpio;
+	struct gpio_desc *gp_gpio;
+
+	int prev_sysclk, prev_hts;
+	int ae_low, ae_high, ae_target;
+
+	struct regulator *io_regulator;
+	struct regulator *core_regulator;
+	struct regulator *analog_regulator;
+	struct regulator *gpo_regulator;
+};
+
+static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct ov5640_dev, sd);
+}
+
+static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
+{
+	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
+}
+
+struct ov5640_control {
+	struct v4l2_queryctrl ctrl;
+	int (*set)(struct ov5640_dev *sensor, int value);
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable);
+static void ov5640_reset(struct ov5640_dev *sensor);
+static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
+static int ov5640_get_exposure(struct ov5640_dev *sensor);
+static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
+static int ov5640_get_gain(struct ov5640_dev *sensor);
+
+static struct reg_value ov5640_init_setting_30fps_VGA[] = {
+
+	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
+	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
+	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
+	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
+	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
+	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
+	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
+	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
+	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
+	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
+	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
+	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
+	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
+	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
+	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
+	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
+	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
+	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
+	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
+	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
+	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
+	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
+	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
+	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
+	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
+	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
+	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
+	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
+	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
+	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
+	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
+	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
+	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
+	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
+	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
+	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
+	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
+	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
+	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
+	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
+	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
+	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
+	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
+	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
+	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
+	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
+	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
+	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
+	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
+	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
+	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
+	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
+	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
+	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
+	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
+	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
+	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
+	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
+	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
+	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
+	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
+	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
+	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
+	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
+	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
+};
+
+static struct reg_value ov5640_setting_30fps_VGA_640_480[] = {
+
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_VGA_640_480[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_XGA_1024_768[] = {
+
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
+	{0x380b, 0x00, 0, 0}, {0x3035, 0x12, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_XGA_1024_768[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3808, 0x04, 0, 0},
+	{0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0}, {0x380b, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QVGA_320_240[] = {
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QVGA_320_240[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QCIF_176_144[] = {
+	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+static struct reg_value ov5640_setting_15fps_QCIF_176_144[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_NTSC_720_480[] = {
+	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_NTSC_720_480[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_PAL_720_576[] = {
+	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_PAL_720_576[] = {
+	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_720P_1280_720[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0}, {0x4005, 0x1a, 0, 0},
+	{0x3008, 0x02, 0, 0}, {0x3503, 0,    0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_720P_1280_720[] = {
+	{0x3035, 0x41, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_1080P_1920_1080[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x11, 0, 0},
+	{0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0},
+	{0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_1080P_1920_1080[] = {
+	{0x3008, 0x42, 0, 0},
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x21, 0, 0},
+	{0x3036, 0x54, 0, 1}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
+	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
+	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
+	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
+};
+
+static struct ov5640_mode_info
+ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
+	{
+		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+		 ov5640_setting_15fps_QCIF_176_144,
+		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
+		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
+		 ov5640_setting_15fps_QVGA_320_240,
+		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
+		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
+		 ov5640_setting_15fps_VGA_640_480,
+		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
+		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+		 ov5640_setting_15fps_NTSC_720_480,
+		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
+		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+		 ov5640_setting_15fps_PAL_720_576,
+		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
+		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+		 ov5640_setting_15fps_XGA_1024_768,
+		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
+		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+		 ov5640_setting_15fps_720P_1280_720,
+		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
+		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+		 ov5640_setting_15fps_1080P_1920_1080,
+		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
+		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
+		 ov5640_setting_15fps_QSXGA_2592_1944,
+		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
+	}, {
+		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+		 ov5640_setting_30fps_QCIF_176_144,
+		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
+		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
+		 ov5640_setting_30fps_QVGA_320_240,
+		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
+		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
+		 ov5640_setting_30fps_VGA_640_480,
+		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
+		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+		 ov5640_setting_30fps_NTSC_720_480,
+		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
+		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+		 ov5640_setting_30fps_PAL_720_576,
+		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
+		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+		 ov5640_setting_30fps_XGA_1024_768,
+		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
+		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+		 ov5640_setting_30fps_720P_1280_720,
+		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
+		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+		 ov5640_setting_30fps_1080P_1920_1080,
+		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
+		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
+	},
+};
+
+static int ov5640_probe(struct i2c_client *adapter,
+			const struct i2c_device_id *device_id);
+static int ov5640_remove(struct i2c_client *client);
+
+static int ov5640_init_slave_id(struct ov5640_dev *sensor)
+{
+	struct i2c_msg msg;
+	u8 buf[4];
+	int ret;
+
+	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
+		return 0;
+
+	buf[0] = OV5640_SLAVE_ID >> 8;
+	buf[1] = OV5640_SLAVE_ID & 0xff;
+	buf[2] = sensor->i2c_client->addr << 1;
+	msg.addr = OV5640_DEFAULT_SLAVE_ID;
+	msg.flags = 0;
+	msg.len = 3;
+	msg.buf = buf;
+
+	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
+	if (ret < 0) {
+		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
+{
+	u8 buf[3] = {0};
+	int ret;
+
+	buf[0] = reg >> 8;
+	buf[1] = reg & 0xff;
+	buf[2] = val;
+
+	ret = i2c_master_send(sensor->i2c_client, buf, 3);
+	if (ret < 0) {
+		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
+			__func__, reg, val);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
+{
+	u8 reg_buf[2] = {0};
+	u8 read_val = 0;
+
+	reg_buf[0] = reg >> 8;
+	reg_buf[1] = reg & 0xff;
+
+	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
+		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
+			__func__, reg);
+		return -EIO;
+	}
+
+	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
+		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
+			__func__, reg, read_val);
+		return -EIO;
+	}
+
+	*val = read_val;
+	return 0;
+}
+
+#define OV5640_READ_REG(s, r, v) {				\
+		ret = ov5640_read_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+#define OV5640_WRITE_REG(s, r, v) {				\
+		ret = ov5640_write_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
+{
+	u8 hi, lo;
+	int ret;
+
+	OV5640_READ_REG(sensor, reg, &hi);
+	OV5640_READ_REG(sensor, reg+1, &lo);
+
+	*val = ((u16)hi << 8) | (u16)lo;
+	return 0;
+}
+#define OV5640_READ_REG16(s, r, v) {				\
+		ret = ov5640_read_reg16((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
+{
+	int ret;
+
+	OV5640_WRITE_REG(sensor, reg, val >> 8);
+	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
+	return 0;
+}
+#define OV5640_WRITE_REG16(s, r, v) {				\
+		ret = ov5640_write_reg16((s), (r), (v));	\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
+			  u8 mask, u8 val)
+{
+	u8 readval;
+	int ret;
+
+	OV5640_READ_REG(sensor, reg, &readval);
+
+	readval &= ~mask;
+	val &= mask;
+	val |= readval;
+
+	OV5640_WRITE_REG(sensor, reg, val);
+	return 0;
+}
+#define OV5640_MOD_REG(s, r, m, v) {				\
+		ret = ov5640_mod_reg((s), (r), (m), (v));	\
+		if (ret)					\
+			return ret;				\
+	}
+
+/* download ov5640 settings to sensor through i2c */
+static int ov5640_load_regs(struct ov5640_dev *sensor,
+			    struct reg_value *regs,
+			    int size)
+{
+	register u32 delay_ms = 0;
+	register u16 reg_addr = 0;
+	register u8 mask = 0;
+	register u8 val = 0;
+	int i, ret;
+
+	for (i = 0; i < size; ++i, ++regs) {
+		delay_ms = regs->delay_ms;
+		reg_addr = regs->reg_addr;
+		val = regs->val;
+		mask = regs->mask;
+
+		if (mask) {
+			OV5640_MOD_REG(sensor, reg_addr, mask, val);
+		} else {
+			OV5640_WRITE_REG(sensor, reg_addr, val);
+		}
+		if (delay_ms)
+			usleep_range(1000*delay_ms, 1000*delay_ms+100);
+	}
+
+	return 0;
+}
+
+static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
+{
+	int ret;
+
+	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
+	return 0;
+}
+
+static int ov5640_get_sysclk(struct ov5640_dev *sensor)
+{
+	 /* calculate sysclk */
+	int xvclk = sensor->xclk_freq / 10000;
+	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
+	int sclk_rdiv_map[] = {1, 2, 4, 8};
+	int bit_div2x = 1, sclk_rdiv, sysclk;
+	u8 temp1, temp2;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3034, &temp1);
+	temp2 = temp1 & 0x0f;
+	if (temp2 == 8 || temp2 == 10)
+		bit_div2x = temp2 / 2;
+
+	OV5640_READ_REG(sensor, 0x3035, &temp1);
+	sysdiv = temp1>>4;
+	if (sysdiv == 0)
+		sysdiv = 16;
+
+	OV5640_READ_REG(sensor, 0x3036, &temp1);
+	multiplier = temp1;
+
+	OV5640_READ_REG(sensor, 0x3037, &temp1);
+	prediv = temp1 & 0x0f;
+	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
+
+	OV5640_READ_REG(sensor, 0x3108, &temp1);
+	temp2 = temp1 & 0x03;
+	sclk_rdiv = sclk_rdiv_map[temp2];
+
+	VCO = xvclk * multiplier / prediv;
+
+	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
+
+	return sysclk;
+}
+
+static int ov5640_set_night_mode(struct ov5640_dev *sensor)
+{
+	 /* read HTS from register settings */
+	u8 mode;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3a00, &mode);
+	mode &= 0xfb;
+	OV5640_WRITE_REG(sensor, 0x3a00, mode);
+	return 0;
+}
+
+static int ov5640_get_HTS(struct ov5640_dev *sensor)
+{
+	 /* read HTS from register settings */
+	u16 HTS;
+	int ret;
+
+	OV5640_READ_REG16(sensor, 0x380c, &HTS);
+	return HTS;
+}
+
+static int ov5640_get_VTS(struct ov5640_dev *sensor)
+{
+	u16 VTS;
+	int ret;
+
+	OV5640_READ_REG16(sensor, 0x380e, &VTS);
+	return VTS;
+}
+
+static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
+{
+	int ret;
+
+	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
+	return 0;
+}
+
+static int ov5640_get_light_freq(struct ov5640_dev *sensor)
+{
+	/* get banding filter value */
+	u8 temp, temp1;
+	int light_freq = 0;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3c01, &temp);
+
+	if (temp & 0x80) {
+		/* manual */
+		OV5640_READ_REG(sensor, 0x3c00, &temp1);
+		if (temp1 & 0x04) {
+			/* 50Hz */
+			light_freq = 50;
+		} else {
+			/* 60Hz */
+			light_freq = 60;
+		}
+	} else {
+		/* auto */
+		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
+		if (temp1 & 0x01) {
+			/* 50Hz */
+			light_freq = 50;
+		} else {
+			/* 60Hz */
+		}
+	}
+
+	return light_freq;
+}
+
+static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
+{
+	int prev_vts;
+	int band_step60, max_band60, band_step50, max_band50;
+	int ret;
+
+	/* read preview PCLK */
+	ret = ov5640_get_sysclk(sensor);
+	if (ret < 0)
+		return ret;
+	sensor->prev_sysclk = ret;
+	/* read preview HTS */
+	ret = ov5640_get_HTS(sensor);
+	if (ret < 0)
+		return ret;
+	sensor->prev_hts = ret;
+
+	/* read preview VTS */
+	ret = ov5640_get_VTS(sensor);
+	if (ret < 0)
+		return ret;
+	prev_vts = ret;
+
+	/* calculate banding filter */
+	/* 60Hz */
+	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
+	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
+
+	max_band60 = (int)((prev_vts-4)/band_step60);
+	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
+
+	/* 50Hz */
+	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
+	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
+
+	max_band50 = (int)((prev_vts-4)/band_step50);
+	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
+
+	return 0;
+}
+
+static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
+{
+	/* stable in high */
+	int fast_high, fast_low;
+	int ret;
+
+	sensor->ae_low = target * 23 / 25;	/* 0.92 */
+	sensor->ae_high = target * 27 / 25;	/* 1.08 */
+
+	fast_high = sensor->ae_high<<1;
+	if (fast_high > 255)
+		fast_high = 255;
+
+	fast_low = sensor->ae_low >> 1;
+
+	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
+	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
+	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
+	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
+	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
+	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
+
+	return 0;
+}
+
+static int ov5640_binning_on(struct ov5640_dev *sensor)
+{
+	u8 temp;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x3821, &temp);
+	temp &= 0xfe;
+
+	return temp ? 1 : 0;
+}
+
+static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
+{
+	u8 temp, channel = sensor->ep.base.id;
+	int ret;
+
+	OV5640_READ_REG(sensor, 0x4814, &temp);
+	temp &= ~(3 << 6);
+	temp |= (channel << 6);
+	OV5640_WRITE_REG(sensor, 0x4814, temp);
+
+	return 0;
+}
+
+static enum ov5640_mode
+ov5640_find_nearest_mode(struct ov5640_dev *sensor,
+			 int width, int height)
+{
+	int i;
+
+	for (i = ov5640_num_modes - 1; i >= 0; i--) {
+		if (ov5640_mode_info_data[0][i].width <= width &&
+		    ov5640_mode_info_data[0][i].height <= height)
+			break;
+	}
+
+	if (i < 0)
+		i = 0;
+
+	return (enum ov5640_mode)i;
+}
+
+/*
+ * sensor changes between scaling and subsampling, go through
+ * exposure calculation
+ */
+static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
+					    enum ov5640_frame_rate frame_rate,
+					    enum ov5640_mode mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	u8 average;
+	int prev_shutter, prev_gain16;
+	int cap_shutter, cap_gain16;
+	int cap_sysclk, cap_hts, cap_vts;
+	int light_freq, cap_bandfilt, cap_maxband;
+	long cap_gain16_shutter;
+	int ret = 0;
+
+	/* check if the input mode and frame rate is valid */
+	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
+	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
+
+	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
+	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
+
+	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+	    mode_data == NULL || mode_size == 0)
+		return -EINVAL;
+
+	/* auto focus */
+	/* ov5640_auto_focus();//if no af function, just skip it */
+
+	/* turn off AE/AG */
+	ret = ov5640_set_agc(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* read preview shutter */
+	ret = ov5640_get_exposure(sensor);
+	if (ret < 0)
+		return ret;
+	prev_shutter = ret;
+	ret = ov5640_binning_on(sensor);
+	if (ret < 0)
+		return ret;
+	if (ret && mode != ov5640_mode_720P_1280_720 &&
+	    mode != ov5640_mode_1080P_1920_1080)
+		prev_shutter *= 2;
+
+	/* read preview gain */
+	ret = ov5640_get_gain(sensor);
+	if (ret < 0)
+		return ret;
+	prev_gain16 = ret;
+
+	/* get average */
+	OV5640_READ_REG(sensor, 0x56a1, &average);
+
+	/* turn off night mode for capture */
+	ret = ov5640_set_night_mode(sensor);
+	if (ret < 0)
+		return ret;
+
+	/* turn off overlay */
+	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
+	   just skip it */
+
+	ret = ov5640_set_stream(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* Write capture setting */
+	ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	if (ret < 0)
+		return ret;
+
+	/* read capture VTS */
+	ret = ov5640_get_VTS(sensor);
+	if (ret < 0)
+		return ret;
+	cap_vts = ret;
+	ret = ov5640_get_HTS(sensor);
+	if (ret < 0)
+		return ret;
+	cap_hts = ret;
+	ret = ov5640_get_sysclk(sensor);
+	if (ret < 0)
+		return ret;
+	cap_sysclk = ret;
+
+	/* calculate capture banding filter */
+	ret = ov5640_get_light_freq(sensor);
+	if (ret < 0)
+		return ret;
+	light_freq = ret;
+
+	if (light_freq == 60) {
+		/* 60Hz */
+		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
+	} else {
+		/* 50Hz */
+		cap_bandfilt = cap_sysclk * 100 / cap_hts;
+	}
+	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
+
+	/* calculate capture shutter/gain16 */
+	if (average > sensor->ae_low && average < sensor->ae_high) {
+		/* in stable range */
+		cap_gain16_shutter =
+			prev_gain16 * prev_shutter *
+			cap_sysclk / sensor->prev_sysclk *
+			sensor->prev_hts / cap_hts *
+			sensor->ae_target / average;
+	} else {
+		cap_gain16_shutter =
+			prev_gain16 * prev_shutter *
+			cap_sysclk / sensor->prev_sysclk *
+			sensor->prev_hts / cap_hts;
+	}
+
+	/* gain to shutter */
+	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
+		/* shutter < 1/100 */
+		cap_shutter = cap_gain16_shutter / 16;
+		if (cap_shutter < 1)
+			cap_shutter = 1;
+
+		cap_gain16 = cap_gain16_shutter / cap_shutter;
+		if (cap_gain16 < 16)
+			cap_gain16 = 16;
+	} else {
+		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
+			/* exposure reach max */
+			cap_shutter = cap_bandfilt * cap_maxband;
+			cap_gain16 = cap_gain16_shutter / cap_shutter;
+		} else {
+			/* 1/100 < (cap_shutter = n/100) =< max */
+			cap_shutter =
+				((int)(cap_gain16_shutter / 16 / cap_bandfilt))
+				* cap_bandfilt;
+			cap_gain16 = cap_gain16_shutter / cap_shutter;
+		}
+	}
+
+	/* write capture gain */
+	ret = ov5640_set_gain(sensor, cap_gain16);
+	if (ret < 0)
+		return ret;
+
+	/* write capture shutter */
+	if (cap_shutter > (cap_vts - 4)) {
+		cap_vts = cap_shutter + 4;
+		ret = ov5640_set_VTS(sensor, cap_vts);
+		if (ret < 0)
+			return ret;
+	}
+
+	ret = ov5640_set_exposure(sensor, cap_shutter);
+	if (ret < 0)
+		return ret;
+
+	return ov5640_set_stream(sensor, true);
+}
+
+/*
+ * if sensor changes inside scaling or subsampling
+ * change mode directly
+ */
+static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
+				     enum ov5640_frame_rate frame_rate,
+				     enum ov5640_mode mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	int ret = 0;
+
+	/* check if the input mode and frame rate is valid */
+	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
+	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
+
+	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
+	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
+
+	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+	    mode_data == NULL || mode_size == 0)
+		return -EINVAL;
+
+	/* turn off AE/AG */
+	ret = ov5640_set_agc(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_stream(sensor, false);
+	if (ret < 0)
+		return ret;
+
+	/* Write capture setting */
+	ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_stream(sensor, true);
+	if (ret < 0)
+		return ret;
+
+	return ov5640_set_agc(sensor, true);
+}
+
+static int ov5640_change_mode(struct ov5640_dev *sensor,
+			      enum ov5640_frame_rate frame_rate,
+			      enum ov5640_mode mode,
+			      enum ov5640_mode orig_mode)
+{
+	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	int ret = 0;
+
+	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
+	    mode != ov5640_mode_INIT) {
+		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
+		return -EINVAL;
+	}
+
+	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
+	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
+	if (mode == ov5640_mode_INIT) {
+		mode_data = ov5640_init_setting_30fps_VGA;
+		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
+
+		sensor->fmt.width = 640;
+		sensor->fmt.height = 480;
+		ret = ov5640_load_regs(sensor, mode_data, mode_size);
+		if (ret < 0)
+			return ret;
+
+		mode_data = ov5640_setting_30fps_VGA_640_480;
+		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
+		ret = ov5640_load_regs(sensor, mode_data, mode_size);
+	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
+			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
+		/* change between subsampling and scaling
+		 * go through exposure calucation */
+		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
+							  mode);
+	} else {
+		/* change inside subsampling or scaling
+		 * download firmware directly */
+		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
+	}
+
+	if (ret < 0)
+		return ret;
+
+	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_get_light_freq(sensor);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_set_bandingfilter(sensor);
+	if (ret < 0)
+		return ret;
+	ret = ov5640_set_virtual_channel(sensor);
+	if (ret < 0)
+		return ret;
+
+	/* restore controls */
+	ov5640_restore_ctrls(sensor);
+
+	if (ret >= 0 && mode != ov5640_mode_INIT) {
+		sensor->current_mode = mode;
+		sensor->current_fr = frame_rate;
+	}
+
+	return 0;
+}
+
+/* restore the last set video mode after chip power-on */
+static int ov5640_restore_mode(struct ov5640_dev *sensor)
+{
+	int ret = 0;
+
+	/* first we need to set some initial register values */
+	ret = ov5640_change_mode(sensor, sensor->current_fr,
+				    ov5640_mode_INIT, ov5640_mode_INIT);
+	if (ret < 0)
+		return ret;
+
+	/* now restore the last capture mode */
+	return ov5640_change_mode(sensor,
+				  sensor->current_fr,
+				  sensor->current_mode,
+				  ov5640_mode_VGA_640_480);
+}
+
+static int ov5640_regulators_on(struct ov5640_dev *sensor)
+{
+	int ret;
+
+	if (sensor->io_regulator) {
+		ret = regulator_enable(sensor->io_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "io reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->core_regulator) {
+		ret = regulator_enable(sensor->core_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "core reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->gpo_regulator) {
+		ret = regulator_enable(sensor->gpo_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->analog_regulator) {
+		ret = regulator_enable(sensor->analog_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "analog reg enable failed\n");
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static void ov5640_regulators_off(struct ov5640_dev *sensor)
+{
+	if (sensor->analog_regulator)
+		regulator_disable(sensor->analog_regulator);
+	if (sensor->core_regulator)
+		regulator_disable(sensor->core_regulator);
+	if (sensor->io_regulator)
+		regulator_disable(sensor->io_regulator);
+	if (sensor->gpo_regulator)
+		regulator_disable(sensor->gpo_regulator);
+}
+
+/* --------------- Subdev Operations --------------- */
+
+static int ov5640_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	int ret;
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (on && !sensor->on) {
+		if (sensor->xclk)
+			clk_prepare_enable(sensor->xclk);
+
+		ret = ov5640_regulators_on(sensor);
+		if (ret)
+			return ret;
+
+		ov5640_reset(sensor);
+		ov5640_power(sensor, true);
+
+		ret = ov5640_init_slave_id(sensor);
+		if (ret)
+			return ret;
+
+		ret = ov5640_restore_mode(sensor);
+		if (ret)
+			return ret;
+
+		/*
+		 * NOTE: Freescale adds a long delay (600 msec) after
+		 * powering up and programming a mode on the ov5640-mipi
+		 * camera (search for "msec_wait4stable" in FSL's
+		 * ov5640_mipi.c), which equivalently would need to go
+		 * right here. If we run into MIPI CSI-2 receiver dphy
+		 * ready timeouts, it might be a clue to add that delay
+		 * here.
+		 */
+	} else if (!on && sensor->on) {
+		ov5640_power(sensor, false);
+
+		ov5640_regulators_off(sensor);
+
+		if (sensor->xclk)
+			clk_disable_unprepare(sensor->xclk);
+	}
+
+	sensor->on = on;
+
+	return 0;
+}
+
+static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	struct v4l2_captureparm *cparm = &a->parm.capture;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	/* This is the only case currently handled. */
+	memset(a, 0, sizeof(*a));
+	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	cparm->capability = sensor->streamcap.capability;
+	cparm->timeperframe = sensor->streamcap.timeperframe;
+	cparm->capturemode = sensor->streamcap.capturemode;
+
+	return 0;
+}
+
+static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
+	enum ov5640_frame_rate frame_rate;
+	u32 tgt_fps;	/* target frames per secound */
+	int ret = 0;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	/* Check that the new frame rate is allowed. */
+	if ((timeperframe->numerator == 0) ||
+	    (timeperframe->denominator == 0)) {
+		timeperframe->denominator = DEFAULT_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps > MAX_FPS) {
+		timeperframe->denominator = MAX_FPS;
+		timeperframe->numerator = 1;
+	} else if (tgt_fps < MIN_FPS) {
+		timeperframe->denominator = MIN_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	/* Actual frame rate we use */
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps == 15)
+		frame_rate = ov5640_15_fps;
+	else if (tgt_fps == 30)
+		frame_rate = ov5640_30_fps;
+	else {
+		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
+			 tgt_fps);
+		return -EINVAL;
+	}
+
+	ret = ov5640_change_mode(sensor, frame_rate,
+				 sensor->current_mode,
+				 sensor->current_mode);
+	if (ret < 0)
+		return ret;
+
+	sensor->streamcap.timeperframe = *timeperframe;
+
+	return 0;
+}
+
+static int ov5640_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	format->format = sensor->fmt;
+
+	return 0;
+}
+
+static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
+				   struct v4l2_mbus_framefmt *fmt,
+				   enum ov5640_mode *new_mode)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode mode;
+
+	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
+
+	fmt->width = ov5640_mode_info_data[0][mode].width;
+	fmt->height = ov5640_mode_info_data[0][mode].height;
+	fmt->code = sensor->fmt.code;
+
+	if (new_mode)
+		*new_mode = mode;
+	return 0;
+}
+
+static int ov5640_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode new_mode;
+	int ret;
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
+		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
+		if (ret)
+			return ret;
+		cfg->try_fmt = format->format;
+		return 0;
+	}
+
+	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
+	if (ret)
+		return ret;
+
+	ret = ov5640_change_mode(sensor, sensor->current_fr,
+				 new_mode, sensor->current_mode);
+	if (ret >= 0)
+		sensor->fmt = format->format;
+
+	return ret;
+}
+
+
+/*
+ * Sensor Controls.
+ */
+
+static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
+		OV5640_WRITE_REG16(sensor, 0x5581, value);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
+
+	return 0;
+}
+
+static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
+		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);
+
+	return 0;
+}
+
+static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
+		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
+		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
+	} else
+		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);
+
+	return 0;
+}
+
+static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	sensor->awb_on = value ? true : false;
+	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
+	return 0;
+}
+
+static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
+	return 0;
+}
+
+#if 0
+static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
+	return 0;
+}
+#endif
+
+static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
+	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
+	return 0;
+}
+
+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
+{
+	u16 max_exp = 0;
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
+	if (value < max_exp) {
+		u32 exp = value << 4;
+
+		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
+		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
+		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
+	}
+
+	return 0;
+}
+
+/* read exposure, in number of line periods */
+static int ov5640_get_exposure(struct ov5640_dev *sensor)
+{
+	u8 temp;
+	int exp, ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG(sensor, 0x3500, &temp);
+	exp = ((int)temp & 0x0f) << 16;
+	OV5640_READ_REG(sensor, 0x3501, &temp);
+	exp |= ((int)temp << 8);
+	OV5640_READ_REG(sensor, 0x3502, &temp);
+	exp |= (int)temp;
+
+	return exp >> 4;
+}
+
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	/* this enables/disables both AEC and AGC */
+	sensor->agc_on = value ? true : false;
+	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
+
+	return 0;
+}
+
+static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
+	return 0;
+}
+
+static int ov5640_get_gain(struct ov5640_dev *sensor)
+{
+	u16 gain;
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5640_READ_REG16(sensor, 0x350a, &gain);
+
+	return gain & 0x3ff;
+}
+
+#if 0
+static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
+{
+	int ret;
+
+	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
+	return 0;
+}
+#endif
+
+static struct ov5640_control ov5640_ctrls[] = {
+	{
+		.set = ov5640_set_agc,
+		.ctrl = {
+			.id = V4L2_CID_AUTOGAIN,
+			.name = "Auto Gain/Exposure Control",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5640_set_exposure,
+		.ctrl = {
+			.id = V4L2_CID_EXPOSURE,
+			.name = "Exposure",
+			.minimum = 0,
+			.maximum = 65535,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_gain,
+		.ctrl = {
+			.id = V4L2_CID_GAIN,
+			.name = "Gain",
+			.minimum = 0,
+			.maximum = 1023,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_hue,
+		.ctrl = {
+			.id = V4L2_CID_HUE,
+			.name = "Hue",
+			.minimum = 0,
+			.maximum = 359,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_contrast,
+		.ctrl = {
+			.id = V4L2_CID_CONTRAST,
+			.name = "Contrast",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_saturation,
+		.ctrl = {
+			.id = V4L2_CID_SATURATION,
+			.name = "Saturation",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 64,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_awb,
+		.ctrl = {
+			.id = V4L2_CID_AUTO_WHITE_BALANCE,
+			.name = "Auto White Balance",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5640_set_red_balance,
+		.ctrl = {
+			.id = V4L2_CID_RED_BALANCE,
+			.name = "Red Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5640_set_blue_balance,
+		.ctrl = {
+			.id = V4L2_CID_BLUE_BALANCE,
+			.name = "Blue Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	},
+};
+#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
+
+static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
+{
+	struct ov5640_control *ret = NULL;
+	int i;
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		if (id == ov5640_ctrls[i].ctrl.id) {
+			ret = &ov5640_ctrls[i];
+			break;
+		}
+	}
+
+	if (ret && index)
+		*index = i;
+	return ret;
+}
+
+static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
+{
+	struct ov5640_control *c;
+	int i;
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		c = &ov5640_ctrls[i];
+		c->set(sensor, sensor->ctrl_cache[i]);
+	}
+
+	return 0;
+}
+
+static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
+	struct ov5640_control *c;
+	int ret = 0;
+	int i;
+
+	c = ov5640_get_ctrl(ctrl->id, &i);
+	if (!c)
+		return -EINVAL;
+
+	ret = c->set(sensor, ctrl->val);
+	/* update cached value if no error */
+	if (!ret)
+		sensor->ctrl_cache[i] = ctrl->val;
+
+	return ret;
+}
+
+static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
+	.s_ctrl = ov5640_s_ctrl,
+};
+
+static int ov5640_init_controls(struct ov5640_dev *sensor)
+{
+	struct ov5640_control *c;
+	int i;
+
+	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
+
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		c = &ov5640_ctrls[i];
+
+		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
+				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
+				  c->ctrl.step, c->ctrl.default_value);
+	}
+
+	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
+	if (sensor->ctrl_hdl.error) {
+		int err = sensor->ctrl_hdl.error;
+
+		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
+		return err;
+	}
+	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_frame_size_enum *fse)
+{
+	if (fse->pad != 0)
+		return -EINVAL;
+	if (fse->index >= ov5640_num_modes)
+		return -EINVAL;
+
+	fse->min_width = fse->max_width =
+		ov5640_mode_info_data[0][fse->index].width;
+	fse->min_height = fse->max_height =
+		ov5640_mode_info_data[0][fse->index].height;
+
+	return 0;
+}
+
+static int ov5640_enum_frame_interval(
+	struct v4l2_subdev *sd,
+	struct v4l2_subdev_pad_config *cfg,
+	struct v4l2_subdev_frame_interval_enum *fie)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+	enum ov5640_mode mode;
+
+	if (fie->pad != 0)
+		return -EINVAL;
+	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
+		return -EINVAL;
+
+	if (fie->width == 0 || fie->height == 0)
+		return -EINVAL;
+
+	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
+
+	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
+		return -EINVAL;
+
+	fie->interval.numerator = 1;
+	fie->interval.denominator = ov5640_framerates[fie->index];
+
+	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
+		fie->width, fie->height, fie->index, fie->interval.denominator);
+	return 0;
+}
+
+static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
+
+	return 0;
+}
+
+static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
+			    u32 output, u32 config)
+{
+	return (input != 0) ? -EINVAL : 0;
+}
+
+static int ov5640_enum_mbus_code(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	if (code->pad != 0)
+		return -EINVAL;
+	if (code->index != 0)
+		return -EINVAL;
+
+	code->code = sensor->fmt.code;
+
+	return 0;
+}
+
+static int ov5640_g_mbus_config(struct v4l2_subdev *sd,
+				struct v4l2_mbus_config *cfg)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	cfg->type = V4L2_MBUS_CSI2;
+	cfg->flags = sensor->ep.bus.mipi_csi2.flags;
+	cfg->flags |= (1 << (sensor->ep.bus.mipi_csi2.num_data_lanes - 1));
+	cfg->flags |= V4L2_MBUS_CSI2_CHANNEL_0;
+
+	return 0;
+}
+
+static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	return ov5640_set_stream(sensor, enable);
+}
+
+static struct v4l2_subdev_core_ops ov5640_core_ops = {
+	.s_power = ov5640_s_power,
+};
+
+static struct v4l2_subdev_video_ops ov5640_video_ops = {
+	.s_parm = ov5640_s_parm,
+	.g_parm = ov5640_g_parm,
+	.g_input_status = ov5640_g_input_status,
+	.s_routing = ov5640_s_routing,
+	.g_mbus_config  = ov5640_g_mbus_config,
+	.s_stream = ov5640_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
+	.enum_mbus_code = ov5640_enum_mbus_code,
+	.get_fmt = ov5640_get_fmt,
+	.set_fmt = ov5640_set_fmt,
+	.enum_frame_size = ov5640_enum_frame_size,
+	.enum_frame_interval = ov5640_enum_frame_interval,
+};
+
+static struct v4l2_subdev_ops ov5640_subdev_ops = {
+	.core = &ov5640_core_ops,
+	.video = &ov5640_video_ops,
+	.pad = &ov5640_pad_ops,
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable)
+{
+	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
+}
+
+static void ov5640_reset(struct ov5640_dev *sensor)
+{
+	gpiod_set_value(sensor->reset_gpio, 0);
+
+	/* camera power cycle */
+	ov5640_power(sensor, false);
+	usleep_range(5000, 10000);
+	ov5640_power(sensor, true);
+	usleep_range(5000, 10000);
+
+	gpiod_set_value(sensor->reset_gpio, 1);
+	usleep_range(1000, 2000);
+
+	gpiod_set_value(sensor->reset_gpio, 0);
+	usleep_range(5000, 10000);
+}
+
+static void ov5640_get_regulators(struct ov5640_dev *sensor)
+{
+	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
+	if (!IS_ERR(sensor->io_regulator)) {
+		regulator_set_voltage(sensor->io_regulator,
+				      OV5640_VOLTAGE_DIGITAL_IO,
+				      OV5640_VOLTAGE_DIGITAL_IO);
+	} else {
+		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
+			__func__);
+		sensor->io_regulator = NULL;
+	}
+
+	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
+	if (!IS_ERR(sensor->core_regulator)) {
+		regulator_set_voltage(sensor->core_regulator,
+				      OV5640_VOLTAGE_DIGITAL_CORE,
+				      OV5640_VOLTAGE_DIGITAL_CORE);
+	} else {
+		sensor->core_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
+			__func__);
+	}
+
+	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
+	if (!IS_ERR(sensor->analog_regulator)) {
+		regulator_set_voltage(sensor->analog_regulator,
+				      OV5640_VOLTAGE_ANALOG,
+				      OV5640_VOLTAGE_ANALOG);
+	} else {
+		sensor->analog_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
+			__func__);
+	}
+}
+
+static int ov5640_probe(struct i2c_client *client,
+			const struct i2c_device_id *id)
+{
+	struct device *dev = &client->dev;
+	struct device_node *endpoint;
+	struct ov5640_dev *sensor;
+	int i, xclk, ret;
+
+	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
+	if (!sensor)
+		return -ENOMEM;
+
+	sensor->i2c_client = client;
+	sensor->dev = dev;
+	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
+	sensor->fmt.width = 640;
+	sensor->fmt.height = 480;
+	sensor->fmt.field = V4L2_FIELD_NONE;
+	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
+					   V4L2_CAP_TIMEPERFRAME;
+	sensor->streamcap.capturemode = 0;
+	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
+	sensor->streamcap.timeperframe.numerator = 1;
+
+	sensor->current_mode = ov5640_mode_VGA_640_480;
+	sensor->current_fr = ov5640_30_fps;
+
+	sensor->ae_target = 52;
+
+	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
+	if (!endpoint) {
+		dev_err(dev, "endpoint node not found\n");
+		return -EINVAL;
+	}
+
+	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
+	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
+		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
+		return -EINVAL;
+	}
+	of_node_put(endpoint);
+
+	/* get system clock (xclk) frequency */
+	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
+	if (!ret) {
+		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
+			dev_err(dev, "invalid xclk frequency\n");
+			return -EINVAL;
+		}
+		sensor->xclk_freq = xclk;
+	}
+
+	/* get system clock (xclk) */
+	sensor->xclk = devm_clk_get(dev, "xclk");
+	if (!IS_ERR(sensor->xclk)) {
+		if (!sensor->xclk_freq) {
+			dev_err(dev, "xclk requires xclk frequency!\n");
+			return -EINVAL;
+		}
+		clk_set_rate(sensor->xclk, sensor->xclk_freq);
+	} else {
+		/* assume system clock enabled by default */
+		sensor->xclk = NULL;
+	}
+
+	/* request power down pin */
+	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
+	if (IS_ERR(sensor->pwdn_gpio)) {
+		dev_err(dev, "request for power down gpio failed\n");
+		return PTR_ERR(sensor->pwdn_gpio);
+	}
+
+	/* request reset pin */
+	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
+	if (IS_ERR(sensor->reset_gpio)) {
+		dev_err(dev, "request for reset gpio failed\n");
+		return PTR_ERR(sensor->reset_gpio);
+	}
+
+	/* initialize the cached controls to their defaults */
+	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+		struct ov5640_control *c = &ov5640_ctrls[i];
+
+		sensor->ctrl_cache[i] = c->ctrl.default_value;
+	}
+	sensor->awb_on = sensor->agc_on = true;
+
+	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
+
+	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
+	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
+	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
+	if (ret)
+		return ret;
+
+	ov5640_get_regulators(sensor);
+
+	ret = ov5640_s_power(&sensor->sd, 1);
+	if (ret)
+		goto entity_cleanup;
+	ret = ov5640_init_controls(sensor);
+	if (ret)
+		goto power_off;
+
+	ret = ov5640_s_power(&sensor->sd, 0);
+	if (ret)
+		goto free_ctrls;
+
+	ret = v4l2_async_register_subdev(&sensor->sd);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+
+free_ctrls:
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+power_off:
+	ov5640_s_power(&sensor->sd, 0);
+entity_cleanup:
+	media_entity_cleanup(&sensor->sd.entity);
+	ov5640_regulators_off(sensor);
+	return ret;
+}
+
+/*!
+ * ov5640 I2C detach function
+ *
+ * @param client            struct i2c_client *
+ * @return  Error code indicating success or failure
+ */
+static int ov5640_remove(struct i2c_client *client)
+{
+	struct v4l2_subdev *sd = i2c_get_clientdata(client);
+	struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+	ov5640_regulators_off(sensor);
+
+	v4l2_async_unregister_subdev(&sensor->sd);
+	media_entity_cleanup(&sensor->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static const struct i2c_device_id ov5640_id[] = {
+	{"ov5640", 0},
+	{},
+};
+MODULE_DEVICE_TABLE(i2c, ov5640_id);
+
+static const struct of_device_id ov5640_dt_ids[] = {
+	{ .compatible = "ovti,ov5640" },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, ov5640_dt_ids);
+
+static struct i2c_driver ov5640_i2c_driver = {
+	.driver = {
+		.name  = "ov5640",
+		.of_match_table	= ov5640_dt_ids,
+	},
+	.id_table = ov5640_id,
+	.probe    = ov5640_probe,
+	.remove   = ov5640_remove,
+};
+
+module_i2c_driver(ov5640_i2c_driver);
+
+MODULE_AUTHOR("Freescale Semiconductor, Inc.");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 23/24] media: imx: Add Parallel OV5642 sensor subdev driver
  2017-01-07  2:11 ` Steve Longerbeam
                   ` (23 preceding siblings ...)
  (?)
@ 2017-01-07  2:11 ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

This driver is based on ov5642.c from Freescale imx_3.10.17_1.0.0_beta
branch, modified heavily to bring forward to latest interfaces and code
cleanup.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 drivers/staging/media/imx/Kconfig  |    7 +
 drivers/staging/media/imx/Makefile |    1 +
 drivers/staging/media/imx/ov5642.c | 4363 ++++++++++++++++++++++++++++++++++++
 3 files changed, 4371 insertions(+)
 create mode 100644 drivers/staging/media/imx/ov5642.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index 09f373d..607e0bf 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -25,5 +25,12 @@ config IMX_OV5640_MIPI
        ---help---
          MIPI CSI-2 OV5640 Camera support.
 
+config IMX_OV5642
+       tristate "OmniVision OV5642 Parallel camera support"
+       depends on GPIOLIB && VIDEO_IMX_CAMERA
+       default y
+       ---help---
+         Parallel interface OV5642 camera support.
+
 endmenu
 endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index aa954c1..499924f 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -12,3 +12,4 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
 
 obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
+obj-$(CONFIG_IMX_OV5642) += ov5642.o
diff --git a/drivers/staging/media/imx/ov5642.c b/drivers/staging/media/imx/ov5642.c
new file mode 100644
index 0000000..a36ef41
--- /dev/null
+++ b/drivers/staging/media/imx/ov5642.c
@@ -0,0 +1,4363 @@
+/*
+ * Copyright (c) 2012-2014 Mentor Graphics Inc.
+ * Copyright (C) 2012 Freescale Semiconductor, Inc. All Rights Reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/ctype.h>
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/of_device.h>
+#include <linux/gpio/consumer.h>
+#include <linux/regulator/consumer.h>
+#include <linux/clk.h>
+#include <linux/clk-provider.h>
+#include <linux/clkdev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+
+#define OV5642_VOLTAGE_ANALOG               2800000
+#define OV5642_VOLTAGE_DIGITAL_CORE         1500000
+#define OV5642_VOLTAGE_DIGITAL_IO           1800000
+
+#define MIN_FPS 15
+#define MAX_FPS 30
+#define DEFAULT_FPS 30
+
+/* min/typical/max system clock (xclk) frequencies */
+#define OV5642_XCLK_MIN  6000000
+#define OV5642_XCLK_TYP 24000000
+#define OV5642_XCLK_MAX 54000000
+
+/* min/typical/max pixel clock (mclk) frequencies */
+#define OV5642_MCLK_MIN 48000000
+#define OV5642_MCLK_TYP 48000000
+#define OV5642_MCLK_MAX 96000000
+
+#define OV5642_CHIP_ID  0x300A
+#define OV5642_SLAVE_ID 0x3100
+#define OV5642_DEFAULT_SLAVE_ID 0x3c
+
+#define OV5642_MAX_CONTROLS 64
+
+enum ov5642_mode {
+	ov5642_mode_MIN = 0,
+	ov5642_mode_QCIF_176_144 = 0,
+	ov5642_mode_QVGA_320_240,
+	ov5642_mode_VGA_640_480,
+	ov5642_mode_NTSC_720_480,
+	ov5642_mode_PAL_720_576,
+	ov5642_mode_XGA_1024_768,
+	ov5642_mode_720P_1280_720,
+	ov5642_mode_1080P_1920_1080,
+	ov5642_mode_QSXGA_2592_1944,
+	ov5642_num_modes,
+};
+
+enum ov5642_frame_rate {
+	ov5642_15_fps,
+	ov5642_30_fps
+};
+
+static int ov5642_framerates[] = {
+	[ov5642_15_fps] = 15,
+	[ov5642_30_fps] = 30,
+};
+#define ov5642_num_framerates ARRAY_SIZE(ov5642_framerates)
+
+struct reg_value {
+	u16 reg_addr;
+	u8 val;
+	u8 mask;
+	u32 delay_ms;
+};
+
+struct ov5642_mode_info {
+	enum ov5642_mode mode;
+	u32 width;
+	u32 height;
+	struct reg_value *init_data_ptr;
+	u32 init_data_size;
+};
+
+struct ov5642_dev {
+	struct i2c_client *i2c_client;
+	struct device *dev;
+	struct v4l2_subdev sd;
+	struct media_pad pad;
+	struct v4l2_ctrl_handler ctrl_hdl;
+	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_captureparm streamcap;
+	struct clk *xclk; /* system clock to OV5642 */
+	int xclk_freq;    /* requested xclk freq from devicetree */
+
+	enum ov5642_mode current_mode;
+	enum ov5642_frame_rate current_fr;
+
+	bool on;
+	bool awb_on;
+	bool agc_on;
+
+	/* cached control settings */
+	int ctrl_cache[OV5642_MAX_CONTROLS];
+
+	struct gpio_desc *reset_gpio;
+	struct gpio_desc *pwdn_gpio;
+	struct gpio_desc *gp_gpio;
+
+	struct regulator *io_regulator;
+	struct regulator *core_regulator;
+	struct regulator *analog_regulator;
+	struct regulator *gpo_regulator;
+};
+
+static inline struct ov5642_dev *to_ov5642_dev(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct ov5642_dev, sd);
+}
+
+static inline struct ov5642_dev *ctrl_to_ov5642_dev(struct v4l2_ctrl *ctrl)
+{
+	return container_of(ctrl->handler, struct ov5642_dev, ctrl_hdl);
+}
+
+struct ov5642_control {
+	struct v4l2_queryctrl ctrl;
+	int (*set)(struct ov5642_dev *sensor, int value);
+};
+
+static void ov5642_power(struct ov5642_dev *sensor, bool enable);
+static void ov5642_reset(struct ov5642_dev *sensor);
+static int ov5642_restore_ctrls(struct ov5642_dev *sensor);
+static int ov5642_set_agc(struct ov5642_dev *sensor, int value);
+static int ov5642_set_exposure(struct ov5642_dev *sensor, int value);
+static int ov5642_set_gain(struct ov5642_dev *sensor, int value);
+
+#if 0
+/* not used, but keep around */
+static struct reg_value ov5642_rotate_none_VGA[] = {
+	{0x3818, 0xc1, 0x00, 0x00}, {0x3621, 0x87, 0x00, 0x00},
+};
+static struct reg_value ov5642_rotate_vert_flip_VGA[] = {
+	{0x3818, 0x20, 0xbf, 0x00}, {0x3621, 0x20, 0xff, 0x00},
+};
+static struct reg_value ov5642_rotate_horiz_flip_VGA[] = {
+	{0x3818, 0x81, 0x00, 0x01}, {0x3621, 0xa7, 0x00, 0x00},
+};
+static struct reg_value ov5642_rotate_180_VGA[] = {
+	{0x3818, 0x60, 0xff, 0x00}, {0x3621, 0x00, 0xdf, 0x00},
+};
+static struct reg_value ov5642_rotate_none_FULL[] = {
+	{0x3818, 0xc0, 0x00, 0x00}, {0x3621, 0x09, 0x00, 0x00},
+};
+static struct reg_value ov5642_rotate_vert_flip_FULL[] = {
+	{0x3818, 0x20, 0xbf, 0x01}, {0x3621, 0x20, 0xff, 0x00},
+};
+static struct reg_value ov5642_rotate_horiz_flip_FULL[] = {
+	{0x3818, 0x80, 0x00, 0x01}, {0x3621, 0x29, 0x00, 0x00},
+};
+static struct reg_value ov5642_rotate_180_FULL[] = {
+	{0x3818, 0x60, 0xff, 0x00}, {0x3621, 0x00, 0xdf, 0x00},
+};
+#endif
+
+static struct reg_value ov5642_initial_setting[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0},
+	{0x3006, 0x43, 0, 0}, {0x3007, 0x37, 0, 0}, {0x3011, 0x08, 0, 0},
+	{0x3010, 0x00, 0, 0}, {0x460c, 0x22, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c00, 0x04, 0, 0}, {0x3c01, 0x80, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0},
+	{0x5182, 0x00, 0, 0}, {0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0},
+	{0x5001, 0xff, 0, 0}, {0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0},
+	{0x5505, 0x7f, 0, 0}, {0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0},
+	{0x4610, 0x00, 0, 0}, {0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0},
+	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+	{0x380b, 0xe0, 0, 0}, {0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0},
+	{0x501f, 0x00, 0, 0}, {0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0},
+	{0x350b, 0x00, 0, 0}, {0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x0c, 0, 0}, {0x380d, 0x80, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x08, 0, 0}, {0x3810, 0x40, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x0b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 200},
+};
+
+static struct reg_value ov5642_setting_15fps_QCIF_176_144[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0},
+	{0x3006, 0x43, 0, 0}, {0x3007, 0x37, 0, 0}, {0x3011, 0x08, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x460c, 0x22, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3501, 0x1e, 0, 0},
+	{0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0}, {0x380c, 0x0c, 0, 0},
+	{0x380d, 0x80, 0, 0}, {0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0}, {0x3818, 0xc1, 0, 0},
+	{0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0}, {0x3801, 0x80, 0, 0},
+	{0x3621, 0x87, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3803, 0x08, 0, 0},
+	{0x3827, 0x08, 0, 0}, {0x3810, 0x40, 0, 0}, {0x3804, 0x05, 0, 0},
+	{0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0},
+	{0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0}, {0x5686, 0x03, 0, 0},
+	{0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a1a, 0x05, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0},
+	{0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0}, {0x3502, 0x00, 0, 0},
+	{0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0}, {0x528f, 0x10, 0, 0},
+	{0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x02, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x02, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x02, 0, 0},
+	{0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3c, 0, 0},
+	{0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0}, {0x3a03, 0x7d, 0, 0},
+	{0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0}, {0x3a15, 0x7d, 0, 0},
+	{0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a08, 0x09, 0, 0},
+	{0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0}, {0x3a0b, 0xd0, 0, 0},
+	{0x3a0d, 0x08, 0, 0}, {0x3a0e, 0x06, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0}, {0x401e, 0x20, 0, 0},
+	{0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0}, {0x528a, 0x01, 0, 0},
+	{0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0}, {0x528d, 0x10, 0, 0},
+	{0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0}, {0x5290, 0x30, 0, 0},
+	{0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0}, {0x5294, 0x00, 0, 0},
+	{0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0}, {0x5297, 0x08, 0, 0},
+	{0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0}, {0x529a, 0x00, 0, 0},
+	{0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0}, {0x529d, 0x28, 0, 0},
+	{0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0}, {0x5282, 0x00, 0, 0},
+	{0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0}, {0x5302, 0x00, 0, 0},
+	{0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0}, {0x530d, 0x0c, 0, 0},
+	{0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0}, {0x5310, 0x20, 0, 0},
+	{0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0}, {0x5309, 0x40, 0, 0},
+	{0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x00, 0, 0},
+	{0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0}, {0x5315, 0x20, 0, 0},
+	{0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0}, {0x5317, 0x00, 0, 0},
+	{0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0}, {0x5381, 0x00, 0, 0},
+	{0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0}, {0x5384, 0x00, 0, 0},
+	{0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0}, {0x5387, 0x00, 0, 0},
+	{0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0}, {0x538a, 0x00, 0, 0},
+	{0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0}, {0x538d, 0x00, 0, 0},
+	{0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0}, {0x5390, 0x00, 0, 0},
+	{0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0}, {0x5393, 0xa2, 0, 0},
+	{0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0}, {0x5481, 0x21, 0, 0},
+	{0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0}, {0x5484, 0x65, 0, 0},
+	{0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0}, {0x5487, 0x87, 0, 0},
+	{0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0}, {0x548a, 0xaa, 0, 0},
+	{0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0}, {0x548d, 0xdd, 0, 0},
+	{0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0}, {0x5490, 0x05, 0, 0},
+	{0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0}, {0x5493, 0x20, 0, 0},
+	{0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0}, {0x5496, 0x02, 0, 0},
+	{0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0}, {0x5499, 0x86, 0, 0},
+	{0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0}, {0x549c, 0x02, 0, 0},
+	{0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0}, {0x549f, 0x1c, 0, 0},
+	{0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0}, {0x54a2, 0x01, 0, 0},
+	{0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0}, {0x54a5, 0xc5, 0, 0},
+	{0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0}, {0x54a8, 0x01, 0, 0},
+	{0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0}, {0x54ab, 0x41, 0, 0},
+	{0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0}, {0x54ae, 0x00, 0, 0},
+	{0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0}, {0x54b1, 0x20, 0, 0},
+	{0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0}, {0x54b4, 0x00, 0, 0},
+	{0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0}, {0x54b7, 0xdf, 0, 0},
+	{0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0}, {0x3406, 0x00, 0, 0},
+	{0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0}, {0x5182, 0x11, 0, 0},
+	{0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0},
+	{0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0}, {0x5188, 0x08, 0, 0},
+	{0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0}, {0x518b, 0xb2, 0, 0},
+	{0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0}, {0x518e, 0x3d, 0, 0},
+	{0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0},
+	{0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0},
+	{0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0},
+	{0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0}, {0x519a, 0x04, 0, 0},
+	{0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0}, {0x519d, 0x82, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0}, {0x3a0f, 0x38, 0, 0},
+	{0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0}, {0x3a1e, 0x2e, 0, 0},
+	{0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0}, {0x5688, 0xa6, 0, 0},
+	{0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0}, {0x568b, 0xae, 0, 0},
+	{0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0}, {0x568e, 0x62, 0, 0},
+	{0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0}, {0x5584, 0x40, 0, 0},
+	{0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0}, {0x5800, 0x27, 0, 0},
+	{0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0}, {0x5803, 0x0f, 0, 0},
+	{0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0}, {0x5806, 0x1e, 0, 0},
+	{0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0}, {0x5809, 0x0d, 0, 0},
+	{0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0}, {0x580c, 0x0a, 0, 0},
+	{0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0}, {0x580f, 0x19, 0, 0},
+	{0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0}, {0x5812, 0x04, 0, 0},
+	{0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0}, {0x5815, 0x06, 0, 0},
+	{0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0}, {0x5818, 0x0a, 0, 0},
+	{0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0}, {0x581b, 0x00, 0, 0},
+	{0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0}, {0x581e, 0x08, 0, 0},
+	{0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0}, {0x5821, 0x05, 0, 0},
+	{0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0}, {0x5824, 0x00, 0, 0},
+	{0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0}, {0x5827, 0x0c, 0, 0},
+	{0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0}, {0x582a, 0x06, 0, 0},
+	{0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0}, {0x582d, 0x07, 0, 0},
+	{0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0}, {0x5830, 0x18, 0, 0},
+	{0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0}, {0x5833, 0x0a, 0, 0},
+	{0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0}, {0x5836, 0x15, 0, 0},
+	{0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0}, {0x5839, 0x1f, 0, 0},
+	{0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0}, {0x583c, 0x17, 0, 0},
+	{0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0}, {0x583f, 0x53, 0, 0},
+	{0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0}, {0x5842, 0x0d, 0, 0},
+	{0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0}, {0x5845, 0x09, 0, 0},
+	{0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0}, {0x5848, 0x10, 0, 0},
+	{0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0}, {0x584b, 0x0e, 0, 0},
+	{0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0}, {0x584e, 0x11, 0, 0},
+	{0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0}, {0x5851, 0x0c, 0, 0},
+	{0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0}, {0x5854, 0x10, 0, 0},
+	{0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0}, {0x5857, 0x0b, 0, 0},
+	{0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0}, {0x585a, 0x0d, 0, 0},
+	{0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0}, {0x585d, 0x0c, 0, 0},
+	{0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0}, {0x5860, 0x0c, 0, 0},
+	{0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0}, {0x5863, 0x08, 0, 0},
+	{0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0}, {0x5866, 0x18, 0, 0},
+	{0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0}, {0x5869, 0x19, 0, 0},
+	{0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0}, {0x586c, 0x13, 0, 0},
+	{0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0}, {0x586f, 0x16, 0, 0},
+	{0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0}, {0x5872, 0x10, 0, 0},
+	{0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0}, {0x5875, 0x16, 0, 0},
+	{0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0}, {0x5878, 0x10, 0, 0},
+	{0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0}, {0x587b, 0x14, 0, 0},
+	{0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0}, {0x587e, 0x11, 0, 0},
+	{0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0}, {0x5881, 0x15, 0, 0},
+	{0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0}, {0x5884, 0x15, 0, 0},
+	{0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0}, {0x5887, 0x17, 0, 0},
+	{0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0}, {0x3702, 0x10, 0, 0},
+	{0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0}, {0x370b, 0x40, 0, 0},
+	{0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0}, {0x3632, 0x52, 0, 0},
+	{0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0}, {0x5785, 0x07, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0}, {0x3604, 0x48, 0, 0},
+	{0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0}, {0x370f, 0xc0, 0, 0},
+	{0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0}, {0x5007, 0x00, 0, 0},
+	{0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0}, {0x5013, 0x00, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0}, {0x5087, 0x00, 0, 0},
+	{0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0}, {0x302b, 0x00, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x3a00, 0x78, 0, 0},
+};
+
+static struct reg_value ov5642_setting_30fps_QCIF_176_144[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0},
+	{0x3006, 0x43, 0, 0}, {0x3007, 0x37, 0, 0}, {0x3011, 0x10, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x460c, 0x22, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3501, 0x1e, 0, 0},
+	{0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0}, {0x380c, 0x0c, 0, 0},
+	{0x380d, 0x80, 0, 0}, {0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0}, {0x3818, 0xc1, 0, 0},
+	{0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0}, {0x3801, 0x80, 0, 0},
+	{0x3621, 0x87, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3803, 0x08, 0, 0},
+	{0x3827, 0x08, 0, 0}, {0x3810, 0x40, 0, 0}, {0x3804, 0x05, 0, 0},
+	{0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0},
+	{0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0}, {0x5686, 0x03, 0, 0},
+	{0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a1a, 0x05, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0},
+	{0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0}, {0x3502, 0x00, 0, 0},
+	{0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0}, {0x528f, 0x10, 0, 0},
+	{0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x02, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x02, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x02, 0, 0},
+	{0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3c, 0, 0},
+	{0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0}, {0x3a03, 0x7d, 0, 0},
+	{0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0}, {0x3a15, 0x7d, 0, 0},
+	{0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a08, 0x09, 0, 0},
+	{0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0}, {0x3a0b, 0xd0, 0, 0},
+	{0x3a0d, 0x08, 0, 0}, {0x3a0e, 0x06, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0}, {0x401e, 0x20, 0, 0},
+	{0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0}, {0x528a, 0x01, 0, 0},
+	{0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0}, {0x528d, 0x10, 0, 0},
+	{0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0}, {0x5290, 0x30, 0, 0},
+	{0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0}, {0x5294, 0x00, 0, 0},
+	{0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0}, {0x5297, 0x08, 0, 0},
+	{0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0}, {0x529a, 0x00, 0, 0},
+	{0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0}, {0x529d, 0x28, 0, 0},
+	{0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0}, {0x5282, 0x00, 0, 0},
+	{0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0}, {0x5302, 0x00, 0, 0},
+	{0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0}, {0x530d, 0x0c, 0, 0},
+	{0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0}, {0x5310, 0x20, 0, 0},
+	{0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0}, {0x5309, 0x40, 0, 0},
+	{0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x00, 0, 0},
+	{0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0}, {0x5315, 0x20, 0, 0},
+	{0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0}, {0x5317, 0x00, 0, 0},
+	{0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0}, {0x5381, 0x00, 0, 0},
+	{0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0}, {0x5384, 0x00, 0, 0},
+	{0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0}, {0x5387, 0x00, 0, 0},
+	{0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0}, {0x538a, 0x00, 0, 0},
+	{0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0}, {0x538d, 0x00, 0, 0},
+	{0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0}, {0x5390, 0x00, 0, 0},
+	{0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0}, {0x5393, 0xa2, 0, 0},
+	{0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0}, {0x5481, 0x21, 0, 0},
+	{0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0}, {0x5484, 0x65, 0, 0},
+	{0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0}, {0x5487, 0x87, 0, 0},
+	{0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0}, {0x548a, 0xaa, 0, 0},
+	{0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0}, {0x548d, 0xdd, 0, 0},
+	{0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0}, {0x5490, 0x05, 0, 0},
+	{0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0}, {0x5493, 0x20, 0, 0},
+	{0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0}, {0x5496, 0x02, 0, 0},
+	{0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0}, {0x5499, 0x86, 0, 0},
+	{0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0}, {0x549c, 0x02, 0, 0},
+	{0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0}, {0x549f, 0x1c, 0, 0},
+	{0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0}, {0x54a2, 0x01, 0, 0},
+	{0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0}, {0x54a5, 0xc5, 0, 0},
+	{0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0}, {0x54a8, 0x01, 0, 0},
+	{0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0}, {0x54ab, 0x41, 0, 0},
+	{0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0}, {0x54ae, 0x00, 0, 0},
+	{0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0}, {0x54b1, 0x20, 0, 0},
+	{0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0}, {0x54b4, 0x00, 0, 0},
+	{0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0}, {0x54b7, 0xdf, 0, 0},
+	{0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0}, {0x3406, 0x00, 0, 0},
+	{0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0}, {0x5182, 0x11, 0, 0},
+	{0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0},
+	{0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0}, {0x5188, 0x08, 0, 0},
+	{0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0}, {0x518b, 0xb2, 0, 0},
+	{0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0}, {0x518e, 0x3d, 0, 0},
+	{0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0},
+	{0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0},
+	{0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0},
+	{0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0}, {0x519a, 0x04, 0, 0},
+	{0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0}, {0x519d, 0x82, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0}, {0x3a0f, 0x38, 0, 0},
+	{0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0}, {0x3a1e, 0x2e, 0, 0},
+	{0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0}, {0x5688, 0xa6, 0, 0},
+	{0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0}, {0x568b, 0xae, 0, 0},
+	{0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0}, {0x568e, 0x62, 0, 0},
+	{0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0}, {0x5584, 0x40, 0, 0},
+	{0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0}, {0x5800, 0x27, 0, 0},
+	{0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0}, {0x5803, 0x0f, 0, 0},
+	{0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0}, {0x5806, 0x1e, 0, 0},
+	{0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0}, {0x5809, 0x0d, 0, 0},
+	{0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0}, {0x580c, 0x0a, 0, 0},
+	{0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0}, {0x580f, 0x19, 0, 0},
+	{0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0}, {0x5812, 0x04, 0, 0},
+	{0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0}, {0x5815, 0x06, 0, 0},
+	{0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0}, {0x5818, 0x0a, 0, 0},
+	{0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0}, {0x581b, 0x00, 0, 0},
+	{0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0}, {0x581e, 0x08, 0, 0},
+	{0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0}, {0x5821, 0x05, 0, 0},
+	{0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0}, {0x5824, 0x00, 0, 0},
+	{0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0}, {0x5827, 0x0c, 0, 0},
+	{0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0}, {0x582a, 0x06, 0, 0},
+	{0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0}, {0x582d, 0x07, 0, 0},
+	{0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0}, {0x5830, 0x18, 0, 0},
+	{0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0}, {0x5833, 0x0a, 0, 0},
+	{0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0}, {0x5836, 0x15, 0, 0},
+	{0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0}, {0x5839, 0x1f, 0, 0},
+	{0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0}, {0x583c, 0x17, 0, 0},
+	{0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0}, {0x583f, 0x53, 0, 0},
+	{0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0}, {0x5842, 0x0d, 0, 0},
+	{0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0}, {0x5845, 0x09, 0, 0},
+	{0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0}, {0x5848, 0x10, 0, 0},
+	{0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0}, {0x584b, 0x0e, 0, 0},
+	{0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0}, {0x584e, 0x11, 0, 0},
+	{0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0}, {0x5851, 0x0c, 0, 0},
+	{0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0}, {0x5854, 0x10, 0, 0},
+	{0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0}, {0x5857, 0x0b, 0, 0},
+	{0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0}, {0x585a, 0x0d, 0, 0},
+	{0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0}, {0x585d, 0x0c, 0, 0},
+	{0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0}, {0x5860, 0x0c, 0, 0},
+	{0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0}, {0x5863, 0x08, 0, 0},
+	{0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0}, {0x5866, 0x18, 0, 0},
+	{0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0}, {0x5869, 0x19, 0, 0},
+	{0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0}, {0x586c, 0x13, 0, 0},
+	{0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0}, {0x586f, 0x16, 0, 0},
+	{0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0}, {0x5872, 0x10, 0, 0},
+	{0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0}, {0x5875, 0x16, 0, 0},
+	{0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0}, {0x5878, 0x10, 0, 0},
+	{0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0}, {0x587b, 0x14, 0, 0},
+	{0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0}, {0x587e, 0x11, 0, 0},
+	{0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0}, {0x5881, 0x15, 0, 0},
+	{0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0}, {0x5884, 0x15, 0, 0},
+	{0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0}, {0x5887, 0x17, 0, 0},
+	{0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0}, {0x3702, 0x10, 0, 0},
+	{0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0}, {0x370b, 0x40, 0, 0},
+	{0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0}, {0x3632, 0x52, 0, 0},
+	{0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0}, {0x5785, 0x07, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0}, {0x3604, 0x48, 0, 0},
+	{0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0}, {0x370f, 0xc0, 0, 0},
+	{0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0}, {0x5007, 0x00, 0, 0},
+	{0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0}, {0x5013, 0x00, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0}, {0x5087, 0x00, 0, 0},
+	{0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0}, {0x302b, 0x00, 0, 0},
+	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0x90, 0, 0}, {0x3a00, 0x78, 0, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_QSXGA_2592_1944[] = {
+	{0x3503, 0x07, 0, 0}, {0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0},
+	{0x3002, 0x00, 0, 0}, {0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0},
+	{0x3005, 0xff, 0, 0}, {0x3006, 0xff, 0, 0}, {0x3007, 0x3f, 0, 0},
+	{0x3011, 0x08, 0, 0}, {0x3010, 0x10, 0, 0}, {0x3818, 0xc0, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0},
+	{0x3602, 0xe4, 0, 0}, {0x3612, 0xac, 0, 0}, {0x3613, 0x44, 0, 0},
+	{0x3622, 0x60, 0, 0}, {0x3623, 0x22, 0, 0}, {0x3604, 0x48, 0, 0},
+	{0x3705, 0xda, 0, 0}, {0x370a, 0x80, 0, 0}, {0x3801, 0x95, 0, 0},
+	{0x3803, 0x0e, 0, 0}, {0x3804, 0x0a, 0, 0}, {0x3805, 0x20, 0, 0},
+	{0x3806, 0x07, 0, 0}, {0x3807, 0x98, 0, 0}, {0x3808, 0x0a, 0, 0},
+	{0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0}, {0x380b, 0x98, 0, 0},
+	{0x380c, 0x0c, 0, 0}, {0x380d, 0x80, 0, 0}, {0x380e, 0x07, 0, 0},
+	{0x380f, 0xd0, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3815, 0x44, 0, 0},
+	{0x3824, 0x11, 0, 0}, {0x3825, 0xac, 0, 0}, {0x3827, 0x0c, 0, 0},
+	{0x3a00, 0x78, 0, 0}, {0x3a0d, 0x10, 0, 0}, {0x3a0e, 0x0d, 0, 0},
+	{0x5682, 0x0a, 0, 0}, {0x5683, 0x20, 0, 0}, {0x5686, 0x07, 0, 0},
+	{0x5687, 0x98, 0, 0}, {0x5001, 0xff, 0, 0}, {0x589b, 0x00, 0, 0},
+	{0x589a, 0xc0, 0, 0}, {0x4407, 0x04, 0, 0}, {0x3008, 0x02, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x460c, 0x22, 0, 0}, {0x471d, 0x05, 0, 0},
+	{0x4713, 0x03, 0, 0}, {0x471c, 0xd0, 0, 0}, {0x3815, 0x01, 0, 0},
+	{0x501f, 0x00, 0, 0}, {0x3002, 0x1c, 0, 0}, {0x3819, 0x80, 0, 0},
+	{0x5002, 0xe0, 0, 0}, {0x530a, 0x01, 0, 0}, {0x530d, 0x10, 0, 0},
+	{0x530c, 0x04, 0, 0}, {0x5312, 0x20, 0, 0}, {0x5282, 0x01, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x3012, 0x00, 0, 0},
+};
+
+
+static struct reg_value ov5642_setting_VGA_2_QVGA[] = {
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x3815, 0x04, 0, 0},
+};
+
+static struct reg_value ov5642_setting_QSXGA_2_VGA[] = {
+	{0x3503, 0x00, 0, 0}, {0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0},
+	{0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0},
+	{0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0}, {0x3007, 0x37, 0, 0},
+	{0x3010, 0x00, 0, 0}, {0x3818, 0xc1, 0, 0}, {0x3621, 0x87, 0, 0},
+	{0x350c, 0x03, 0, 0}, {0x350d, 0xe8, 0, 0}, {0x3602, 0xfc, 0, 0},
+	{0x3612, 0xff, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3622, 0x60, 0, 0},
+	{0x3623, 0x01, 0, 0}, {0x3604, 0x48, 0, 0}, {0x3705, 0xdb, 0, 0},
+	{0x370a, 0x81, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3803, 0x08, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x3806, 0x03, 0, 0},
+	{0x3807, 0xc0, 0, 0}, {0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0},
+	{0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0}, {0x380c, 0x0c, 0, 0},
+	{0x380d, 0x80, 0, 0}, {0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0},
+	{0x3810, 0x40, 0, 0}, {0x3815, 0x04, 0, 0}, {0x3824, 0x11, 0, 0},
+	{0x3825, 0xb4, 0, 0}, {0x3827, 0x08, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x5686, 0x03, 0, 0}, {0x5687, 0xbc, 0, 0},
+	{0x5001, 0xff, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x4407, 0x0c, 0, 0}, {0x3008, 0x02, 0, 0}, {0x460b, 0x37, 0, 0},
+	{0x460c, 0x22, 0, 0}, {0x471d, 0x05, 0, 0}, {0x4713, 0x02, 0, 0},
+	{0x471c, 0xd0, 0, 0}, {0x3815, 0x04, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x3002, 0x5c, 0, 0}, {0x3819, 0x80, 0, 0}, {0x5002, 0xe0, 0, 0},
+	{0x530a, 0x01, 0, 0}, {0x530d, 0x0c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x5312, 0x40, 0, 0}, {0x5282, 0x00, 0, 0},
+	{0x3012, 0x02, 0, 0}, {0x3010, 0x00, 0, 0},
+};
+
+static struct reg_value ov5642_setting_30fps_VGA_640_480[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3615, 0xf0, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0},
+	{0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0},
+	{0x3007, 0x37, 0, 0}, {0x3011, 0x09, 0, 0}, {0x3012, 0x02, 0, 0},
+	{0x3010, 0x00, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3825, 0xb0, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x07, 0, 0}, {0x380d, 0x2a, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0xc7, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x08, 0, 0}, {0x3810, 0x80, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3a00, 0x78, 0, 0},
+};
+
+static struct reg_value ov5642_setting_30fps_QVGA_320_240[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3615, 0xf0, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0},
+	{0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0},
+	{0x3007, 0x37, 0, 0}, {0x3011, 0x09, 0, 0}, {0x3012, 0x02, 0, 0},
+	{0x3010, 0x00, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3825, 0xb0, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x07, 0, 0}, {0x380d, 0x2a, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0xc7, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x08, 0, 0}, {0x3810, 0x80, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3808, 0x01, 0, 0},
+	{0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0}, {0x380b, 0xf0, 0, 0},
+};
+
+static struct reg_value ov5642_setting_30fps_NTSC_720_480[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3615, 0xf0, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0},
+	{0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0},
+	{0x3007, 0x37, 0, 0}, {0x3011, 0x09, 0, 0}, {0x3012, 0x02, 0, 0},
+	{0x3010, 0x00, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3825, 0xb0, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x07, 0, 0}, {0x380d, 0x2a, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0xc7, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x3c, 0, 0}, {0x3810, 0x80, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0x58, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0x58, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x302c, 0x60, 0x60, 0},
+};
+
+static struct reg_value ov5642_setting_30fps_PAL_720_576[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3615, 0xf0, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0},
+	{0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0},
+	{0x3007, 0x37, 0, 0}, {0x3011, 0x09, 0, 0}, {0x3012, 0x02, 0, 0},
+	{0x3010, 0x00, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0}, {0x380b, 0x40, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3825, 0xd8, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x07, 0, 0}, {0x380d, 0x2a, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0xc7, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x3c, 0, 0}, {0x3810, 0x80, 0, 0},
+	{0x3804, 0x04, 0, 0}, {0x3805, 0xb0, 0, 0}, {0x5682, 0x04, 0, 0},
+	{0x5683, 0xb0, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0x58, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0x58, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x302c, 0x60, 0x60, 0},
+};
+
+static struct reg_value ov5642_setting_30fps_720P_1280_720[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x00, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3030, 0x2b, 0, 0},
+	{0x3011, 0x08, 0, 0}, {0x3010, 0x10, 0, 0}, {0x3604, 0x60, 0, 0},
+	{0x3622, 0x60, 0, 0}, {0x3621, 0x09, 0, 0}, {0x3709, 0x00, 0, 0},
+	{0x4000, 0x21, 0, 0}, {0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0},
+	{0x3605, 0x04, 0, 0}, {0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0},
+	{0x300d, 0x22, 0, 0}, {0x3623, 0x22, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5500, 0x0a, 0, 0},
+	{0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0}, {0x5080, 0x08, 0, 0},
+	{0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0}, {0x471d, 0x05, 0, 0},
+	{0x4708, 0x06, 0, 0}, {0x370c, 0xa0, 0, 0}, {0x3808, 0x0a, 0, 0},
+	{0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0}, {0x380b, 0x98, 0, 0},
+	{0x380c, 0x0c, 0, 0}, {0x380d, 0x80, 0, 0}, {0x380e, 0x07, 0, 0},
+	{0x380f, 0xd0, 0, 0}, {0x5687, 0x94, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x5001, 0xcf, 0, 0}, {0x4300, 0x30, 0, 0},
+	{0x4300, 0x30, 0, 0}, {0x460b, 0x35, 0, 0}, {0x471d, 0x00, 0, 0},
+	{0x3002, 0x0c, 0, 0}, {0x3002, 0x00, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x471c, 0x50, 0, 0}, {0x4721, 0x02, 0, 0}, {0x4402, 0x90, 0, 0},
+	{0x460c, 0x22, 0, 0}, {0x3815, 0x44, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3818, 0xc8, 0, 0}, {0x3801, 0x88, 0, 0}, {0x3824, 0x11, 0, 0},
+	{0x3a00, 0x78, 0, 0}, {0x3a1a, 0x04, 0, 0}, {0x3a13, 0x30, 0, 0},
+	{0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0},
+	{0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0},
+	{0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0}, {0x3a0d, 0x08, 0, 0},
+	{0x3a0e, 0x06, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x32, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x32, 0, 0}, {0x3a11, 0x80, 0, 0},
+	{0x3a1f, 0x20, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x09, 0, 0}, {0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0},
+	{0x3a0b, 0xd0, 0, 0}, {0x3a0d, 0x10, 0, 0}, {0x3a0e, 0x0d, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x589b, 0x00, 0, 0},
+	{0x589a, 0xc0, 0, 0}, {0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0},
+	{0x401c, 0x06, 0, 0}, {0x3825, 0xac, 0, 0}, {0x3827, 0x0c, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3503, 0x07, 0, 0}, {0x3011, 0x08, 0, 0},
+	{0x350c, 0x02, 0, 0}, {0x350d, 0xe4, 0, 0}, {0x3621, 0xc9, 0, 0},
+	{0x370a, 0x81, 0, 0}, {0x3803, 0x08, 0, 0}, {0x3804, 0x05, 0, 0},
+	{0x3805, 0x00, 0, 0}, {0x3806, 0x02, 0, 0}, {0x3807, 0xd0, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x08, 0, 0}, {0x380d, 0x72, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0xc0, 0, 0},
+	{0x3818, 0xc9, 0, 0}, {0x381c, 0x10, 0, 0}, {0x381d, 0xa0, 0, 0},
+	{0x381e, 0x05, 0, 0}, {0x381f, 0xb0, 0, 0}, {0x3820, 0x00, 0, 0},
+	{0x3821, 0x00, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3a08, 0x1b, 0, 0},
+	{0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x17, 0, 0}, {0x3a0b, 0x20, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a0e, 0x01, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0}, {0x5686, 0x02, 0, 0},
+	{0x5687, 0xcc, 0, 0}, {0x5001, 0x7f, 0, 0}, {0x589b, 0x06, 0, 0},
+	{0x589a, 0xc5, 0, 0}, {0x3503, 0x00, 0, 0}, {0x3010, 0x10, 0, 0},
+	{0x460c, 0x20, 0, 0}, {0x460b, 0x37, 0, 0}, {0x471c, 0xd0, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x3815, 0x01, 0, 0}, {0x3818, 0x00, 0x08, 0},
+	{0x501f, 0x00, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3002, 0x1c, 0, 0},
+	{0x3819, 0x80, 0, 0}, {0x5002, 0xe0, 0, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_1080P_1920_1080[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x00, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3030, 0x2b, 0, 0},
+	{0x3011, 0x08, 0, 0}, {0x3010, 0x10, 0, 0}, {0x3604, 0x60, 0, 0},
+	{0x3622, 0x60, 0, 0}, {0x3621, 0x09, 0, 0}, {0x3709, 0x00, 0, 0},
+	{0x4000, 0x21, 0, 0}, {0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0},
+	{0x3605, 0x04, 0, 0}, {0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0},
+	{0x300d, 0x22, 0, 0}, {0x3623, 0x22, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5500, 0x0a, 0, 0},
+	{0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0}, {0x5080, 0x08, 0, 0},
+	{0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0}, {0x471d, 0x05, 0, 0},
+	{0x4708, 0x06, 0, 0}, {0x370c, 0xa0, 0, 0}, {0x3808, 0x0a, 0, 0},
+	{0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0}, {0x380b, 0x98, 0, 0},
+	{0x380c, 0x0c, 0, 0}, {0x380d, 0x80, 0, 0}, {0x380e, 0x07, 0, 0},
+	{0x380f, 0xd0, 0, 0}, {0x5687, 0x94, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x5001, 0xcf, 0, 0}, {0x4300, 0x30, 0, 0},
+	{0x4300, 0x30, 0, 0}, {0x460b, 0x35, 0, 0}, {0x471d, 0x00, 0, 0},
+	{0x3002, 0x0c, 0, 0}, {0x3002, 0x00, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x471c, 0x50, 0, 0}, {0x4721, 0x02, 0, 0}, {0x4402, 0x90, 0, 0},
+	{0x460c, 0x22, 0, 0}, {0x3815, 0x44, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3818, 0xc8, 0, 0}, {0x3801, 0x88, 0, 0}, {0x3824, 0x11, 0, 0},
+	{0x3a00, 0x78, 0, 0}, {0x3a1a, 0x04, 0, 0}, {0x3a13, 0x30, 0, 0},
+	{0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0},
+	{0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0},
+	{0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0}, {0x3a0d, 0x08, 0, 0},
+	{0x3a0e, 0x06, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x32, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x32, 0, 0}, {0x3a11, 0x80, 0, 0},
+	{0x3a1f, 0x20, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x09, 0, 0}, {0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0},
+	{0x3a0b, 0xd0, 0, 0}, {0x3a0d, 0x10, 0, 0}, {0x3a0e, 0x0d, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x589b, 0x00, 0, 0},
+	{0x589a, 0xc0, 0, 0}, {0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0},
+	{0x401c, 0x06, 0, 0}, {0x3825, 0xac, 0, 0}, {0x3827, 0x0c, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3503, 0x07, 0, 0}, {0x3011, 0x07, 0, 0},
+	{0x350c, 0x04, 0, 0}, {0x350d, 0x58, 0, 0}, {0x3801, 0x8a, 0, 0},
+	{0x3803, 0x0a, 0, 0}, {0x3804, 0x07, 0, 0}, {0x3805, 0x80, 0, 0},
+	{0x3806, 0x04, 0, 0}, {0x3807, 0x39, 0, 0}, {0x3808, 0x07, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+	{0x380c, 0x09, 0, 0}, {0x380d, 0xd6, 0, 0}, {0x380e, 0x04, 0, 0},
+	{0x380f, 0x58, 0, 0}, {0x381c, 0x11, 0, 0}, {0x381d, 0xba, 0, 0},
+	{0x381e, 0x04, 0, 0}, {0x381f, 0x48, 0, 0}, {0x3820, 0x04, 0, 0},
+	{0x3821, 0x18, 0, 0}, {0x3a08, 0x14, 0, 0}, {0x3a09, 0xe0, 0, 0},
+	{0x3a0a, 0x11, 0, 0}, {0x3a0b, 0x60, 0, 0}, {0x3a0d, 0x04, 0, 0},
+	{0x3a0e, 0x03, 0, 0}, {0x5682, 0x07, 0, 0}, {0x5683, 0x60, 0, 0},
+	{0x5686, 0x04, 0, 0}, {0x5687, 0x1c, 0, 0}, {0x5001, 0x7f, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x3010, 0x10, 0, 0}, {0x460c, 0x20, 0, 0},
+	{0x460b, 0x37, 0, 0}, {0x471c, 0xd0, 0, 0}, {0x471d, 0x05, 0, 0},
+	{0x3815, 0x01, 0, 0}, {0x3818, 0x00, 0x08, 0}, {0x501f, 0x00, 0, 0},
+	{0x4300, 0x30, 0, 0}, {0x3002, 0x1c, 0, 0}, {0x3819, 0x80, 0, 0},
+	{0x5002, 0xe0, 0, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_VGA_640_480[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3615, 0xf0, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0},
+	{0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0},
+	{0x3007, 0x37, 0, 0}, {0x3011, 0x09, 0, 0}, {0x3012, 0x02, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3825, 0xb0, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x07, 0, 0}, {0x380d, 0x2a, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0xc7, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x08, 0, 0}, {0x3810, 0x80, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3a00, 0x78, 0, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_QVGA_320_240[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0},
+	{0x3006, 0x43, 0, 0}, {0x3007, 0x37, 0, 0}, {0x3011, 0x08, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x460c, 0x22, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3501, 0x1e, 0, 0},
+	{0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0}, {0x380c, 0x0c, 0, 0},
+	{0x380d, 0x80, 0, 0}, {0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0}, {0x3818, 0xc1, 0, 0},
+	{0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0}, {0x3801, 0x80, 0, 0},
+	{0x3621, 0x87, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3803, 0x08, 0, 0},
+	{0x3827, 0x08, 0, 0}, {0x3810, 0x40, 0, 0}, {0x3804, 0x05, 0, 0},
+	{0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0},
+	{0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0}, {0x5686, 0x03, 0, 0},
+	{0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a1a, 0x05, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0},
+	{0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0}, {0x3502, 0x00, 0, 0},
+	{0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0}, {0x528f, 0x10, 0, 0},
+	{0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x02, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x02, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x02, 0, 0},
+	{0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3c, 0, 0},
+	{0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0}, {0x3a03, 0x7d, 0, 0},
+	{0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0}, {0x3a15, 0x7d, 0, 0},
+	{0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a08, 0x09, 0, 0},
+	{0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0}, {0x3a0b, 0xd0, 0, 0},
+	{0x3a0d, 0x08, 0, 0}, {0x3a0e, 0x06, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0}, {0x401e, 0x20, 0, 0},
+	{0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0}, {0x528a, 0x01, 0, 0},
+	{0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0}, {0x528d, 0x10, 0, 0},
+	{0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0}, {0x5290, 0x30, 0, 0},
+	{0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0}, {0x5294, 0x00, 0, 0},
+	{0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0}, {0x5297, 0x08, 0, 0},
+	{0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0}, {0x529a, 0x00, 0, 0},
+	{0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0}, {0x529d, 0x28, 0, 0},
+	{0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0}, {0x5282, 0x00, 0, 0},
+	{0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0}, {0x5302, 0x00, 0, 0},
+	{0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0}, {0x530d, 0x0c, 0, 0},
+	{0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0}, {0x5310, 0x20, 0, 0},
+	{0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0}, {0x5309, 0x40, 0, 0},
+	{0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x00, 0, 0},
+	{0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0}, {0x5315, 0x20, 0, 0},
+	{0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0}, {0x5317, 0x00, 0, 0},
+	{0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0}, {0x5381, 0x00, 0, 0},
+	{0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0}, {0x5384, 0x00, 0, 0},
+	{0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0}, {0x5387, 0x00, 0, 0},
+	{0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0}, {0x538a, 0x00, 0, 0},
+	{0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0}, {0x538d, 0x00, 0, 0},
+	{0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0}, {0x5390, 0x00, 0, 0},
+	{0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0}, {0x5393, 0xa2, 0, 0},
+	{0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0}, {0x5481, 0x21, 0, 0},
+	{0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0}, {0x5484, 0x65, 0, 0},
+	{0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0}, {0x5487, 0x87, 0, 0},
+	{0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0}, {0x548a, 0xaa, 0, 0},
+	{0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0}, {0x548d, 0xdd, 0, 0},
+	{0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0}, {0x5490, 0x05, 0, 0},
+	{0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0}, {0x5493, 0x20, 0, 0},
+	{0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0}, {0x5496, 0x02, 0, 0},
+	{0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0}, {0x5499, 0x86, 0, 0},
+	{0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0}, {0x549c, 0x02, 0, 0},
+	{0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0}, {0x549f, 0x1c, 0, 0},
+	{0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0}, {0x54a2, 0x01, 0, 0},
+	{0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0}, {0x54a5, 0xc5, 0, 0},
+	{0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0}, {0x54a8, 0x01, 0, 0},
+	{0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0}, {0x54ab, 0x41, 0, 0},
+	{0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0}, {0x54ae, 0x00, 0, 0},
+	{0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0}, {0x54b1, 0x20, 0, 0},
+	{0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0}, {0x54b4, 0x00, 0, 0},
+	{0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0}, {0x54b7, 0xdf, 0, 0},
+	{0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0}, {0x3406, 0x00, 0, 0},
+	{0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0}, {0x5182, 0x11, 0, 0},
+	{0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0},
+	{0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0}, {0x5188, 0x08, 0, 0},
+	{0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0}, {0x518b, 0xb2, 0, 0},
+	{0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0}, {0x518e, 0x3d, 0, 0},
+	{0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0},
+	{0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0},
+	{0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0},
+	{0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0}, {0x519a, 0x04, 0, 0},
+	{0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0}, {0x519d, 0x82, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0}, {0x3a0f, 0x38, 0, 0},
+	{0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0}, {0x3a1e, 0x2e, 0, 0},
+	{0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0}, {0x5688, 0xa6, 0, 0},
+	{0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0}, {0x568b, 0xae, 0, 0},
+	{0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0}, {0x568e, 0x62, 0, 0},
+	{0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0}, {0x5584, 0x40, 0, 0},
+	{0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0}, {0x5800, 0x27, 0, 0},
+	{0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0}, {0x5803, 0x0f, 0, 0},
+	{0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0}, {0x5806, 0x1e, 0, 0},
+	{0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0}, {0x5809, 0x0d, 0, 0},
+	{0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0}, {0x580c, 0x0a, 0, 0},
+	{0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0}, {0x580f, 0x19, 0, 0},
+	{0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0}, {0x5812, 0x04, 0, 0},
+	{0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0}, {0x5815, 0x06, 0, 0},
+	{0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0}, {0x5818, 0x0a, 0, 0},
+	{0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0}, {0x581b, 0x00, 0, 0},
+	{0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0}, {0x581e, 0x08, 0, 0},
+	{0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0}, {0x5821, 0x05, 0, 0},
+	{0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0}, {0x5824, 0x00, 0, 0},
+	{0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0}, {0x5827, 0x0c, 0, 0},
+	{0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0}, {0x582a, 0x06, 0, 0},
+	{0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0}, {0x582d, 0x07, 0, 0},
+	{0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0}, {0x5830, 0x18, 0, 0},
+	{0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0}, {0x5833, 0x0a, 0, 0},
+	{0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0}, {0x5836, 0x15, 0, 0},
+	{0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0}, {0x5839, 0x1f, 0, 0},
+	{0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0}, {0x583c, 0x17, 0, 0},
+	{0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0}, {0x583f, 0x53, 0, 0},
+	{0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0}, {0x5842, 0x0d, 0, 0},
+	{0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0}, {0x5845, 0x09, 0, 0},
+	{0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0}, {0x5848, 0x10, 0, 0},
+	{0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0}, {0x584b, 0x0e, 0, 0},
+	{0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0}, {0x584e, 0x11, 0, 0},
+	{0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0}, {0x5851, 0x0c, 0, 0},
+	{0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0}, {0x5854, 0x10, 0, 0},
+	{0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0}, {0x5857, 0x0b, 0, 0},
+	{0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0}, {0x585a, 0x0d, 0, 0},
+	{0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0}, {0x585d, 0x0c, 0, 0},
+	{0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0}, {0x5860, 0x0c, 0, 0},
+	{0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0}, {0x5863, 0x08, 0, 0},
+	{0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0}, {0x5866, 0x18, 0, 0},
+	{0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0}, {0x5869, 0x19, 0, 0},
+	{0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0}, {0x586c, 0x13, 0, 0},
+	{0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0}, {0x586f, 0x16, 0, 0},
+	{0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0}, {0x5872, 0x10, 0, 0},
+	{0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0}, {0x5875, 0x16, 0, 0},
+	{0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0}, {0x5878, 0x10, 0, 0},
+	{0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0}, {0x587b, 0x14, 0, 0},
+	{0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0}, {0x587e, 0x11, 0, 0},
+	{0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0}, {0x5881, 0x15, 0, 0},
+	{0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0}, {0x5884, 0x15, 0, 0},
+	{0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0}, {0x5887, 0x17, 0, 0},
+	{0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0}, {0x3702, 0x10, 0, 0},
+	{0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0}, {0x370b, 0x40, 0, 0},
+	{0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0}, {0x3632, 0x52, 0, 0},
+	{0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0}, {0x5785, 0x07, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0}, {0x3604, 0x48, 0, 0},
+	{0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0}, {0x370f, 0xc0, 0, 0},
+	{0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0}, {0x5007, 0x00, 0, 0},
+	{0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0}, {0x5013, 0x00, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0}, {0x5087, 0x00, 0, 0},
+	{0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0}, {0x302b, 0x00, 0, 0},
+	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+	{0x380b, 0xf0, 0, 0}, {0x3a00, 0x78, 0, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_NTSC_720_480[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0},
+	{0x3006, 0x43, 0, 0}, {0x3007, 0x37, 0, 0}, {0x3011, 0x08, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x460c, 0x22, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3501, 0x1e, 0, 0},
+	{0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0}, {0x380c, 0x0c, 0, 0},
+	{0x380d, 0x80, 0, 0}, {0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0}, {0x3818, 0xc1, 0, 0},
+	{0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0}, {0x3801, 0x80, 0, 0},
+	{0x3621, 0x87, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3803, 0x08, 0, 0},
+	{0x3827, 0x08, 0, 0}, {0x3810, 0x40, 0, 0}, {0x3804, 0x05, 0, 0},
+	{0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0},
+	{0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0}, {0x5686, 0x03, 0, 0},
+	{0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a1a, 0x05, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0},
+	{0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0}, {0x3502, 0x00, 0, 0},
+	{0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0}, {0x528f, 0x10, 0, 0},
+	{0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x02, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x02, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x02, 0, 0},
+	{0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3c, 0, 0},
+	{0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0}, {0x3a03, 0x7d, 0, 0},
+	{0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0}, {0x3a15, 0x7d, 0, 0},
+	{0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a08, 0x09, 0, 0},
+	{0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0}, {0x3a0b, 0xd0, 0, 0},
+	{0x3a0d, 0x08, 0, 0}, {0x3a0e, 0x06, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0}, {0x401e, 0x20, 0, 0},
+	{0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0}, {0x528a, 0x01, 0, 0},
+	{0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0}, {0x528d, 0x10, 0, 0},
+	{0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0}, {0x5290, 0x30, 0, 0},
+	{0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0}, {0x5294, 0x00, 0, 0},
+	{0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0}, {0x5297, 0x08, 0, 0},
+	{0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0}, {0x529a, 0x00, 0, 0},
+	{0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0}, {0x529d, 0x28, 0, 0},
+	{0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0}, {0x5282, 0x00, 0, 0},
+	{0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0}, {0x5302, 0x00, 0, 0},
+	{0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0}, {0x530d, 0x0c, 0, 0},
+	{0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0}, {0x5310, 0x20, 0, 0},
+	{0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0}, {0x5309, 0x40, 0, 0},
+	{0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x00, 0, 0},
+	{0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0}, {0x5315, 0x20, 0, 0},
+	{0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0}, {0x5317, 0x00, 0, 0},
+	{0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0}, {0x5381, 0x00, 0, 0},
+	{0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0}, {0x5384, 0x00, 0, 0},
+	{0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0}, {0x5387, 0x00, 0, 0},
+	{0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0}, {0x538a, 0x00, 0, 0},
+	{0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0}, {0x538d, 0x00, 0, 0},
+	{0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0}, {0x5390, 0x00, 0, 0},
+	{0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0}, {0x5393, 0xa2, 0, 0},
+	{0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0}, {0x5481, 0x21, 0, 0},
+	{0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0}, {0x5484, 0x65, 0, 0},
+	{0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0}, {0x5487, 0x87, 0, 0},
+	{0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0}, {0x548a, 0xaa, 0, 0},
+	{0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0}, {0x548d, 0xdd, 0, 0},
+	{0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0}, {0x5490, 0x05, 0, 0},
+	{0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0}, {0x5493, 0x20, 0, 0},
+	{0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0}, {0x5496, 0x02, 0, 0},
+	{0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0}, {0x5499, 0x86, 0, 0},
+	{0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0}, {0x549c, 0x02, 0, 0},
+	{0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0}, {0x549f, 0x1c, 0, 0},
+	{0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0}, {0x54a2, 0x01, 0, 0},
+	{0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0}, {0x54a5, 0xc5, 0, 0},
+	{0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0}, {0x54a8, 0x01, 0, 0},
+	{0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0}, {0x54ab, 0x41, 0, 0},
+	{0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0}, {0x54ae, 0x00, 0, 0},
+	{0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0}, {0x54b1, 0x20, 0, 0},
+	{0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0}, {0x54b4, 0x00, 0, 0},
+	{0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0}, {0x54b7, 0xdf, 0, 0},
+	{0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0}, {0x3406, 0x00, 0, 0},
+	{0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0}, {0x5182, 0x11, 0, 0},
+	{0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0},
+	{0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0}, {0x5188, 0x08, 0, 0},
+	{0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0}, {0x518b, 0xb2, 0, 0},
+	{0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0}, {0x518e, 0x3d, 0, 0},
+	{0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0},
+	{0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0},
+	{0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0},
+	{0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0}, {0x519a, 0x04, 0, 0},
+	{0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0}, {0x519d, 0x82, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0}, {0x3a0f, 0x38, 0, 0},
+	{0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0}, {0x3a1e, 0x2e, 0, 0},
+	{0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0}, {0x5688, 0xa6, 0, 0},
+	{0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0}, {0x568b, 0xae, 0, 0},
+	{0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0}, {0x568e, 0x62, 0, 0},
+	{0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0}, {0x5584, 0x40, 0, 0},
+	{0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0}, {0x5800, 0x27, 0, 0},
+	{0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0}, {0x5803, 0x0f, 0, 0},
+	{0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0}, {0x5806, 0x1e, 0, 0},
+	{0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0}, {0x5809, 0x0d, 0, 0},
+	{0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0}, {0x580c, 0x0a, 0, 0},
+	{0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0}, {0x580f, 0x19, 0, 0},
+	{0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0}, {0x5812, 0x04, 0, 0},
+	{0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0}, {0x5815, 0x06, 0, 0},
+	{0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0}, {0x5818, 0x0a, 0, 0},
+	{0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0}, {0x581b, 0x00, 0, 0},
+	{0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0}, {0x581e, 0x08, 0, 0},
+	{0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0}, {0x5821, 0x05, 0, 0},
+	{0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0}, {0x5824, 0x00, 0, 0},
+	{0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0}, {0x5827, 0x0c, 0, 0},
+	{0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0}, {0x582a, 0x06, 0, 0},
+	{0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0}, {0x582d, 0x07, 0, 0},
+	{0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0}, {0x5830, 0x18, 0, 0},
+	{0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0}, {0x5833, 0x0a, 0, 0},
+	{0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0}, {0x5836, 0x15, 0, 0},
+	{0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0}, {0x5839, 0x1f, 0, 0},
+	{0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0}, {0x583c, 0x17, 0, 0},
+	{0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0}, {0x583f, 0x53, 0, 0},
+	{0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0}, {0x5842, 0x0d, 0, 0},
+	{0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0}, {0x5845, 0x09, 0, 0},
+	{0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0}, {0x5848, 0x10, 0, 0},
+	{0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0}, {0x584b, 0x0e, 0, 0},
+	{0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0}, {0x584e, 0x11, 0, 0},
+	{0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0}, {0x5851, 0x0c, 0, 0},
+	{0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0}, {0x5854, 0x10, 0, 0},
+	{0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0}, {0x5857, 0x0b, 0, 0},
+	{0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0}, {0x585a, 0x0d, 0, 0},
+	{0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0}, {0x585d, 0x0c, 0, 0},
+	{0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0}, {0x5860, 0x0c, 0, 0},
+	{0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0}, {0x5863, 0x08, 0, 0},
+	{0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0}, {0x5866, 0x18, 0, 0},
+	{0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0}, {0x5869, 0x19, 0, 0},
+	{0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0}, {0x586c, 0x13, 0, 0},
+	{0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0}, {0x586f, 0x16, 0, 0},
+	{0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0}, {0x5872, 0x10, 0, 0},
+	{0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0}, {0x5875, 0x16, 0, 0},
+	{0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0}, {0x5878, 0x10, 0, 0},
+	{0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0}, {0x587b, 0x14, 0, 0},
+	{0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0}, {0x587e, 0x11, 0, 0},
+	{0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0}, {0x5881, 0x15, 0, 0},
+	{0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0}, {0x5884, 0x15, 0, 0},
+	{0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0}, {0x5887, 0x17, 0, 0},
+	{0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0}, {0x3702, 0x10, 0, 0},
+	{0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0}, {0x370b, 0x40, 0, 0},
+	{0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0}, {0x3632, 0x52, 0, 0},
+	{0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0}, {0x5785, 0x07, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0}, {0x3604, 0x48, 0, 0},
+	{0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0}, {0x370f, 0xc0, 0, 0},
+	{0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0}, {0x5007, 0x00, 0, 0},
+	{0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0}, {0x5013, 0x00, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0}, {0x5087, 0x00, 0, 0},
+	{0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0}, {0x302b, 0x00, 0, 0},
+	{0x3824, 0x11, 0, 0}, {0x3825, 0xb4, 0, 0}, {0x3826, 0x00, 0, 0},
+	{0x3827, 0x3d, 0, 0}, {0x380c, 0x0c, 0, 0}, {0x380d, 0x80, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0xd0, 0, 0}, {0x380A, 0x01, 0, 0}, {0x380B, 0xe0, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x3806, 0x03, 0, 0},
+	{0x3807, 0x55, 0, 0}, {0x5686, 0x03, 0, 0}, {0x5687, 0x55, 0, 0},
+	{0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_PAL_720_576[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0},
+	{0x3006, 0x43, 0, 0}, {0x3007, 0x37, 0, 0}, {0x3011, 0x08, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x460c, 0x22, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3501, 0x1e, 0, 0},
+	{0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0}, {0x380c, 0x0c, 0, 0},
+	{0x380d, 0x80, 0, 0}, {0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0},
+	{0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0}, {0x3818, 0xc1, 0, 0},
+	{0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0}, {0x3801, 0x80, 0, 0},
+	{0x3621, 0x87, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3803, 0x08, 0, 0},
+	{0x3827, 0x08, 0, 0}, {0x3810, 0x40, 0, 0}, {0x3804, 0x05, 0, 0},
+	{0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0},
+	{0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0}, {0x5686, 0x03, 0, 0},
+	{0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a1a, 0x05, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0},
+	{0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0}, {0x3502, 0x00, 0, 0},
+	{0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0}, {0x3503, 0x00, 0, 0},
+	{0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0}, {0x528f, 0x10, 0, 0},
+	{0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x02, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x02, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x02, 0, 0},
+	{0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3c, 0, 0},
+	{0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0}, {0x3a03, 0x7d, 0, 0},
+	{0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0}, {0x3a15, 0x7d, 0, 0},
+	{0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0}, {0x3a08, 0x09, 0, 0},
+	{0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0}, {0x3a0b, 0xd0, 0, 0},
+	{0x3a0d, 0x08, 0, 0}, {0x3a0e, 0x06, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0}, {0x401e, 0x20, 0, 0},
+	{0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0}, {0x528a, 0x01, 0, 0},
+	{0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0}, {0x528d, 0x10, 0, 0},
+	{0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0}, {0x5290, 0x30, 0, 0},
+	{0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0}, {0x5294, 0x00, 0, 0},
+	{0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0}, {0x5297, 0x08, 0, 0},
+	{0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0}, {0x529a, 0x00, 0, 0},
+	{0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0}, {0x529d, 0x28, 0, 0},
+	{0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0}, {0x5282, 0x00, 0, 0},
+	{0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0}, {0x5302, 0x00, 0, 0},
+	{0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0}, {0x530d, 0x0c, 0, 0},
+	{0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0}, {0x5310, 0x20, 0, 0},
+	{0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0}, {0x5309, 0x40, 0, 0},
+	{0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x00, 0, 0},
+	{0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0}, {0x5315, 0x20, 0, 0},
+	{0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0}, {0x5317, 0x00, 0, 0},
+	{0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0}, {0x5381, 0x00, 0, 0},
+	{0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0}, {0x5384, 0x00, 0, 0},
+	{0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0}, {0x5387, 0x00, 0, 0},
+	{0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0}, {0x538a, 0x00, 0, 0},
+	{0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0}, {0x538d, 0x00, 0, 0},
+	{0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0}, {0x5390, 0x00, 0, 0},
+	{0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0}, {0x5393, 0xa2, 0, 0},
+	{0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0}, {0x5481, 0x21, 0, 0},
+	{0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0}, {0x5484, 0x65, 0, 0},
+	{0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0}, {0x5487, 0x87, 0, 0},
+	{0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0}, {0x548a, 0xaa, 0, 0},
+	{0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0}, {0x548d, 0xdd, 0, 0},
+	{0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0}, {0x5490, 0x05, 0, 0},
+	{0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0}, {0x5493, 0x20, 0, 0},
+	{0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0}, {0x5496, 0x02, 0, 0},
+	{0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0}, {0x5499, 0x86, 0, 0},
+	{0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0}, {0x549c, 0x02, 0, 0},
+	{0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0}, {0x549f, 0x1c, 0, 0},
+	{0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0}, {0x54a2, 0x01, 0, 0},
+	{0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0}, {0x54a5, 0xc5, 0, 0},
+	{0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0}, {0x54a8, 0x01, 0, 0},
+	{0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0}, {0x54ab, 0x41, 0, 0},
+	{0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0}, {0x54ae, 0x00, 0, 0},
+	{0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0}, {0x54b1, 0x20, 0, 0},
+	{0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0}, {0x54b4, 0x00, 0, 0},
+	{0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0}, {0x54b7, 0xdf, 0, 0},
+	{0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0}, {0x3406, 0x00, 0, 0},
+	{0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0}, {0x5182, 0x11, 0, 0},
+	{0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0},
+	{0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0}, {0x5188, 0x08, 0, 0},
+	{0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0}, {0x518b, 0xb2, 0, 0},
+	{0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0}, {0x518e, 0x3d, 0, 0},
+	{0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0},
+	{0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0},
+	{0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0},
+	{0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0}, {0x519a, 0x04, 0, 0},
+	{0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0}, {0x519d, 0x82, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0}, {0x3a0f, 0x38, 0, 0},
+	{0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0}, {0x3a1e, 0x2e, 0, 0},
+	{0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0}, {0x5688, 0xa6, 0, 0},
+	{0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0}, {0x568b, 0xae, 0, 0},
+	{0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0}, {0x568e, 0x62, 0, 0},
+	{0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0}, {0x5584, 0x40, 0, 0},
+	{0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0}, {0x5800, 0x27, 0, 0},
+	{0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0}, {0x5803, 0x0f, 0, 0},
+	{0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0}, {0x5806, 0x1e, 0, 0},
+	{0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0}, {0x5809, 0x0d, 0, 0},
+	{0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0}, {0x580c, 0x0a, 0, 0},
+	{0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0}, {0x580f, 0x19, 0, 0},
+	{0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0}, {0x5812, 0x04, 0, 0},
+	{0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0}, {0x5815, 0x06, 0, 0},
+	{0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0}, {0x5818, 0x0a, 0, 0},
+	{0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0}, {0x581b, 0x00, 0, 0},
+	{0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0}, {0x581e, 0x08, 0, 0},
+	{0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0}, {0x5821, 0x05, 0, 0},
+	{0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0}, {0x5824, 0x00, 0, 0},
+	{0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0}, {0x5827, 0x0c, 0, 0},
+	{0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0}, {0x582a, 0x06, 0, 0},
+	{0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0}, {0x582d, 0x07, 0, 0},
+	{0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0}, {0x5830, 0x18, 0, 0},
+	{0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0}, {0x5833, 0x0a, 0, 0},
+	{0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0}, {0x5836, 0x15, 0, 0},
+	{0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0}, {0x5839, 0x1f, 0, 0},
+	{0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0}, {0x583c, 0x17, 0, 0},
+	{0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0}, {0x583f, 0x53, 0, 0},
+	{0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0}, {0x5842, 0x0d, 0, 0},
+	{0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0}, {0x5845, 0x09, 0, 0},
+	{0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0}, {0x5848, 0x10, 0, 0},
+	{0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0}, {0x584b, 0x0e, 0, 0},
+	{0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0}, {0x584e, 0x11, 0, 0},
+	{0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0}, {0x5851, 0x0c, 0, 0},
+	{0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0}, {0x5854, 0x10, 0, 0},
+	{0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0}, {0x5857, 0x0b, 0, 0},
+	{0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0}, {0x585a, 0x0d, 0, 0},
+	{0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0}, {0x585d, 0x0c, 0, 0},
+	{0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0}, {0x5860, 0x0c, 0, 0},
+	{0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0}, {0x5863, 0x08, 0, 0},
+	{0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0}, {0x5866, 0x18, 0, 0},
+	{0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0}, {0x5869, 0x19, 0, 0},
+	{0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0}, {0x586c, 0x13, 0, 0},
+	{0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0}, {0x586f, 0x16, 0, 0},
+	{0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0}, {0x5872, 0x10, 0, 0},
+	{0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0}, {0x5875, 0x16, 0, 0},
+	{0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0}, {0x5878, 0x10, 0, 0},
+	{0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0}, {0x587b, 0x14, 0, 0},
+	{0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0}, {0x587e, 0x11, 0, 0},
+	{0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0}, {0x5881, 0x15, 0, 0},
+	{0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0}, {0x5884, 0x15, 0, 0},
+	{0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0}, {0x5887, 0x17, 0, 0},
+	{0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0}, {0x3702, 0x10, 0, 0},
+	{0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0}, {0x370b, 0x40, 0, 0},
+	{0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0}, {0x3632, 0x52, 0, 0},
+	{0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0}, {0x5785, 0x07, 0, 0},
+	{0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0}, {0x3604, 0x48, 0, 0},
+	{0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0}, {0x370f, 0xc0, 0, 0},
+	{0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0}, {0x5007, 0x00, 0, 0},
+	{0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0}, {0x5013, 0x00, 0, 0},
+	{0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0}, {0x5087, 0x00, 0, 0},
+	{0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0}, {0x302b, 0x00, 0, 0},
+	{0x3824, 0x11, 0, 0}, {0x3825, 0xdc, 0, 0}, {0x3826, 0x00, 0, 0},
+	{0x3827, 0x08, 0, 0}, {0x380c, 0x0c, 0, 0}, {0x380d, 0x80, 0, 0},
+	{0x380e, 0x03, 0, 0}, {0x380f, 0xe8, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0xd0, 0, 0}, {0x380A, 0x02, 0, 0}, {0x380B, 0x40, 0, 0},
+	{0x3804, 0x04, 0, 0}, {0x3805, 0xb0, 0, 0}, {0x3806, 0x03, 0, 0},
+	{0x3807, 0xc0, 0, 0}, {0x5686, 0x03, 0, 0}, {0x5687, 0xc0, 0, 0},
+	{0x5682, 0x04, 0, 0}, {0x5683, 0xb0, 0, 0},
+};
+
+static struct reg_value ov5642_setting_30fps_XGA_1024_768[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3615, 0xf0, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0},
+	{0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0},
+	{0x3007, 0x37, 0, 0}, {0x3011, 0x09, 0, 0}, {0x3012, 0x02, 0, 0},
+	{0x3010, 0x00, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3825, 0xb0, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x07, 0, 0}, {0x380d, 0x2a, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0xc7, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x08, 0, 0}, {0x3810, 0x80, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
+	{0x380b, 0x00, 0, 0}, {0x3815, 0x02, 0, 0}, {0x302c, 0x60, 0x60, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_XGA_1024_768[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3615, 0xf0, 0, 0}, {0x3000, 0x00, 0, 0},
+	{0x3001, 0x00, 0, 0}, {0x3002, 0x5c, 0, 0}, {0x3003, 0x00, 0, 0},
+	{0x3004, 0xff, 0, 0}, {0x3005, 0xff, 0, 0}, {0x3006, 0x43, 0, 0},
+	{0x3007, 0x37, 0, 0}, {0x3011, 0x09, 0, 0}, {0x3012, 0x02, 0, 0},
+	{0x3010, 0x10, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3815, 0x04, 0, 0},
+	{0x370c, 0xa0, 0, 0}, {0x3602, 0xfc, 0, 0}, {0x3612, 0xff, 0, 0},
+	{0x3634, 0xc0, 0, 0}, {0x3613, 0x00, 0, 0}, {0x3605, 0x7c, 0, 0},
+	{0x3621, 0x09, 0, 0}, {0x3622, 0x60, 0, 0}, {0x3604, 0x40, 0, 0},
+	{0x3603, 0xa7, 0, 0}, {0x3603, 0x27, 0, 0}, {0x4000, 0x21, 0, 0},
+	{0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0}, {0x3605, 0x04, 0, 0},
+	{0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5001, 0xff, 0, 0},
+	{0x5500, 0x0a, 0, 0}, {0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0},
+	{0x5080, 0x08, 0, 0}, {0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x4708, 0x06, 0, 0}, {0x3808, 0x02, 0, 0},
+	{0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0}, {0x380b, 0xe0, 0, 0},
+	{0x380e, 0x07, 0, 0}, {0x380f, 0xd0, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x07, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3825, 0xb0, 0, 0},
+	{0x3501, 0x1e, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x7f, 0, 0},
+	{0x380c, 0x07, 0, 0}, {0x380d, 0x2a, 0, 0}, {0x380e, 0x03, 0, 0},
+	{0x380f, 0xe8, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x3818, 0xc1, 0, 0}, {0x3705, 0xdb, 0, 0}, {0x370a, 0x81, 0, 0},
+	{0x3801, 0x80, 0, 0}, {0x3621, 0xc7, 0, 0}, {0x3801, 0x50, 0, 0},
+	{0x3803, 0x08, 0, 0}, {0x3827, 0x08, 0, 0}, {0x3810, 0x80, 0, 0},
+	{0x3804, 0x05, 0, 0}, {0x3805, 0x00, 0, 0}, {0x5682, 0x05, 0, 0},
+	{0x5683, 0x00, 0, 0}, {0x3806, 0x03, 0, 0}, {0x3807, 0xc0, 0, 0},
+	{0x5686, 0x03, 0, 0}, {0x5687, 0xbc, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a1a, 0x05, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3a18, 0x00, 0, 0},
+	{0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0},
+	{0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0}, {0x350c, 0x07, 0, 0},
+	{0x350d, 0xd0, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x528a, 0x02, 0, 0}, {0x528b, 0x04, 0, 0},
+	{0x528c, 0x08, 0, 0}, {0x528d, 0x08, 0, 0}, {0x528e, 0x08, 0, 0},
+	{0x528f, 0x10, 0, 0}, {0x5290, 0x10, 0, 0}, {0x5292, 0x00, 0, 0},
+	{0x5293, 0x02, 0, 0}, {0x5294, 0x00, 0, 0}, {0x5295, 0x02, 0, 0},
+	{0x5296, 0x00, 0, 0}, {0x5297, 0x02, 0, 0}, {0x5298, 0x00, 0, 0},
+	{0x5299, 0x02, 0, 0}, {0x529a, 0x00, 0, 0}, {0x529b, 0x02, 0, 0},
+	{0x529c, 0x00, 0, 0}, {0x529d, 0x02, 0, 0}, {0x529e, 0x00, 0, 0},
+	{0x529f, 0x02, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x30, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x30, 0, 0}, {0x3a11, 0x70, 0, 0},
+	{0x3a1f, 0x10, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x12, 0, 0}, {0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0},
+	{0x3a0b, 0xa0, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a0e, 0x03, 0, 0},
+	{0x5193, 0x70, 0, 0}, {0x589b, 0x04, 0, 0}, {0x589a, 0xc5, 0, 0},
+	{0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3621, 0x87, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
+	{0x380b, 0x00, 0, 0}, {0x3815, 0x02, 0, 0},
+};
+
+static struct reg_value ov5642_setting_15fps_720P_1280_720[] = {
+	{0x3103, 0x93, 0, 0}, {0x3008, 0x82, 0, 0}, {0x3017, 0x7f, 0, 0},
+	{0x3018, 0xfc, 0, 0}, {0x3810, 0xc2, 0, 0}, {0x3615, 0xf0, 0, 0},
+	{0x3000, 0x00, 0, 0}, {0x3001, 0x00, 0, 0}, {0x3002, 0x00, 0, 0},
+	{0x3003, 0x00, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3030, 0x2b, 0, 0},
+	{0x3011, 0x08, 0, 0}, {0x3010, 0x10, 0, 0}, {0x3604, 0x60, 0, 0},
+	{0x3622, 0x60, 0, 0}, {0x3621, 0x09, 0, 0}, {0x3709, 0x00, 0, 0},
+	{0x4000, 0x21, 0, 0}, {0x401d, 0x22, 0, 0}, {0x3600, 0x54, 0, 0},
+	{0x3605, 0x04, 0, 0}, {0x3606, 0x3f, 0, 0}, {0x3c01, 0x80, 0, 0},
+	{0x300d, 0x22, 0, 0}, {0x3623, 0x22, 0, 0}, {0x5000, 0x4f, 0, 0},
+	{0x5020, 0x04, 0, 0}, {0x5181, 0x79, 0, 0}, {0x5182, 0x00, 0, 0},
+	{0x5185, 0x22, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5500, 0x0a, 0, 0},
+	{0x5504, 0x00, 0, 0}, {0x5505, 0x7f, 0, 0}, {0x5080, 0x08, 0, 0},
+	{0x300e, 0x18, 0, 0}, {0x4610, 0x00, 0, 0}, {0x471d, 0x05, 0, 0},
+	{0x4708, 0x06, 0, 0}, {0x370c, 0xa0, 0, 0}, {0x3808, 0x0a, 0, 0},
+	{0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0}, {0x380b, 0x98, 0, 0},
+	{0x380c, 0x0c, 0, 0}, {0x380d, 0x80, 0, 0}, {0x380e, 0x07, 0, 0},
+	{0x380f, 0xd0, 0, 0}, {0x5687, 0x94, 0, 0}, {0x501f, 0x00, 0, 0},
+	{0x5000, 0x4f, 0, 0}, {0x5001, 0xcf, 0, 0}, {0x4300, 0x30, 0, 0},
+	{0x4300, 0x30, 0, 0}, {0x460b, 0x35, 0, 0}, {0x471d, 0x00, 0, 0},
+	{0x3002, 0x0c, 0, 0}, {0x3002, 0x00, 0, 0}, {0x4713, 0x03, 0, 0},
+	{0x471c, 0x50, 0, 0}, {0x4721, 0x02, 0, 0}, {0x4402, 0x90, 0, 0},
+	{0x460c, 0x22, 0, 0}, {0x3815, 0x44, 0, 0}, {0x3503, 0x07, 0, 0},
+	{0x3501, 0x73, 0, 0}, {0x3502, 0x80, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3818, 0xc8, 0, 0}, {0x3801, 0x88, 0, 0}, {0x3824, 0x11, 0, 0},
+	{0x3a00, 0x78, 0, 0}, {0x3a1a, 0x04, 0, 0}, {0x3a13, 0x30, 0, 0},
+	{0x3a18, 0x00, 0, 0}, {0x3a19, 0x7c, 0, 0}, {0x3a08, 0x12, 0, 0},
+	{0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x0f, 0, 0}, {0x3a0b, 0xa0, 0, 0},
+	{0x350c, 0x07, 0, 0}, {0x350d, 0xd0, 0, 0}, {0x3a0d, 0x08, 0, 0},
+	{0x3a0e, 0x06, 0, 0}, {0x3500, 0x00, 0, 0}, {0x3501, 0x00, 0, 0},
+	{0x3502, 0x00, 0, 0}, {0x350a, 0x00, 0, 0}, {0x350b, 0x00, 0, 0},
+	{0x3503, 0x00, 0, 0}, {0x3a0f, 0x3c, 0, 0}, {0x3a10, 0x32, 0, 0},
+	{0x3a1b, 0x3c, 0, 0}, {0x3a1e, 0x32, 0, 0}, {0x3a11, 0x80, 0, 0},
+	{0x3a1f, 0x20, 0, 0}, {0x3030, 0x2b, 0, 0}, {0x3a02, 0x00, 0, 0},
+	{0x3a03, 0x7d, 0, 0}, {0x3a04, 0x00, 0, 0}, {0x3a14, 0x00, 0, 0},
+	{0x3a15, 0x7d, 0, 0}, {0x3a16, 0x00, 0, 0}, {0x3a00, 0x78, 0, 0},
+	{0x3a08, 0x09, 0, 0}, {0x3a09, 0x60, 0, 0}, {0x3a0a, 0x07, 0, 0},
+	{0x3a0b, 0xd0, 0, 0}, {0x3a0d, 0x10, 0, 0}, {0x3a0e, 0x0d, 0, 0},
+	{0x4407, 0x04, 0, 0}, {0x5193, 0x70, 0, 0}, {0x589b, 0x00, 0, 0},
+	{0x589a, 0xc0, 0, 0}, {0x401e, 0x20, 0, 0}, {0x4001, 0x42, 0, 0},
+	{0x401c, 0x06, 0, 0}, {0x3825, 0xac, 0, 0}, {0x3827, 0x0c, 0, 0},
+	{0x528a, 0x01, 0, 0}, {0x528b, 0x04, 0, 0}, {0x528c, 0x08, 0, 0},
+	{0x528d, 0x10, 0, 0}, {0x528e, 0x20, 0, 0}, {0x528f, 0x28, 0, 0},
+	{0x5290, 0x30, 0, 0}, {0x5292, 0x00, 0, 0}, {0x5293, 0x01, 0, 0},
+	{0x5294, 0x00, 0, 0}, {0x5295, 0x04, 0, 0}, {0x5296, 0x00, 0, 0},
+	{0x5297, 0x08, 0, 0}, {0x5298, 0x00, 0, 0}, {0x5299, 0x10, 0, 0},
+	{0x529a, 0x00, 0, 0}, {0x529b, 0x20, 0, 0}, {0x529c, 0x00, 0, 0},
+	{0x529d, 0x28, 0, 0}, {0x529e, 0x00, 0, 0}, {0x529f, 0x30, 0, 0},
+	{0x5282, 0x00, 0, 0}, {0x5300, 0x00, 0, 0}, {0x5301, 0x20, 0, 0},
+	{0x5302, 0x00, 0, 0}, {0x5303, 0x7c, 0, 0}, {0x530c, 0x00, 0, 0},
+	{0x530d, 0x0c, 0, 0}, {0x530e, 0x20, 0, 0}, {0x530f, 0x80, 0, 0},
+	{0x5310, 0x20, 0, 0}, {0x5311, 0x80, 0, 0}, {0x5308, 0x20, 0, 0},
+	{0x5309, 0x40, 0, 0}, {0x5304, 0x00, 0, 0}, {0x5305, 0x30, 0, 0},
+	{0x5306, 0x00, 0, 0}, {0x5307, 0x80, 0, 0}, {0x5314, 0x08, 0, 0},
+	{0x5315, 0x20, 0, 0}, {0x5319, 0x30, 0, 0}, {0x5316, 0x10, 0, 0},
+	{0x5317, 0x00, 0, 0}, {0x5318, 0x02, 0, 0}, {0x5380, 0x01, 0, 0},
+	{0x5381, 0x00, 0, 0}, {0x5382, 0x00, 0, 0}, {0x5383, 0x4e, 0, 0},
+	{0x5384, 0x00, 0, 0}, {0x5385, 0x0f, 0, 0}, {0x5386, 0x00, 0, 0},
+	{0x5387, 0x00, 0, 0}, {0x5388, 0x01, 0, 0}, {0x5389, 0x15, 0, 0},
+	{0x538a, 0x00, 0, 0}, {0x538b, 0x31, 0, 0}, {0x538c, 0x00, 0, 0},
+	{0x538d, 0x00, 0, 0}, {0x538e, 0x00, 0, 0}, {0x538f, 0x0f, 0, 0},
+	{0x5390, 0x00, 0, 0}, {0x5391, 0xab, 0, 0}, {0x5392, 0x00, 0, 0},
+	{0x5393, 0xa2, 0, 0}, {0x5394, 0x08, 0, 0}, {0x5480, 0x14, 0, 0},
+	{0x5481, 0x21, 0, 0}, {0x5482, 0x36, 0, 0}, {0x5483, 0x57, 0, 0},
+	{0x5484, 0x65, 0, 0}, {0x5485, 0x71, 0, 0}, {0x5486, 0x7d, 0, 0},
+	{0x5487, 0x87, 0, 0}, {0x5488, 0x91, 0, 0}, {0x5489, 0x9a, 0, 0},
+	{0x548a, 0xaa, 0, 0}, {0x548b, 0xb8, 0, 0}, {0x548c, 0xcd, 0, 0},
+	{0x548d, 0xdd, 0, 0}, {0x548e, 0xea, 0, 0}, {0x548f, 0x1d, 0, 0},
+	{0x5490, 0x05, 0, 0}, {0x5491, 0x00, 0, 0}, {0x5492, 0x04, 0, 0},
+	{0x5493, 0x20, 0, 0}, {0x5494, 0x03, 0, 0}, {0x5495, 0x60, 0, 0},
+	{0x5496, 0x02, 0, 0}, {0x5497, 0xb8, 0, 0}, {0x5498, 0x02, 0, 0},
+	{0x5499, 0x86, 0, 0}, {0x549a, 0x02, 0, 0}, {0x549b, 0x5b, 0, 0},
+	{0x549c, 0x02, 0, 0}, {0x549d, 0x3b, 0, 0}, {0x549e, 0x02, 0, 0},
+	{0x549f, 0x1c, 0, 0}, {0x54a0, 0x02, 0, 0}, {0x54a1, 0x04, 0, 0},
+	{0x54a2, 0x01, 0, 0}, {0x54a3, 0xed, 0, 0}, {0x54a4, 0x01, 0, 0},
+	{0x54a5, 0xc5, 0, 0}, {0x54a6, 0x01, 0, 0}, {0x54a7, 0xa5, 0, 0},
+	{0x54a8, 0x01, 0, 0}, {0x54a9, 0x6c, 0, 0}, {0x54aa, 0x01, 0, 0},
+	{0x54ab, 0x41, 0, 0}, {0x54ac, 0x01, 0, 0}, {0x54ad, 0x20, 0, 0},
+	{0x54ae, 0x00, 0, 0}, {0x54af, 0x16, 0, 0}, {0x54b0, 0x01, 0, 0},
+	{0x54b1, 0x20, 0, 0}, {0x54b2, 0x00, 0, 0}, {0x54b3, 0x10, 0, 0},
+	{0x54b4, 0x00, 0, 0}, {0x54b5, 0xf0, 0, 0}, {0x54b6, 0x00, 0, 0},
+	{0x54b7, 0xdf, 0, 0}, {0x5402, 0x3f, 0, 0}, {0x5403, 0x00, 0, 0},
+	{0x3406, 0x00, 0, 0}, {0x5180, 0xff, 0, 0}, {0x5181, 0x52, 0, 0},
+	{0x5182, 0x11, 0, 0}, {0x5183, 0x14, 0, 0}, {0x5184, 0x25, 0, 0},
+	{0x5185, 0x24, 0, 0}, {0x5186, 0x06, 0, 0}, {0x5187, 0x08, 0, 0},
+	{0x5188, 0x08, 0, 0}, {0x5189, 0x7c, 0, 0}, {0x518a, 0x60, 0, 0},
+	{0x518b, 0xb2, 0, 0}, {0x518c, 0xb2, 0, 0}, {0x518d, 0x44, 0, 0},
+	{0x518e, 0x3d, 0, 0}, {0x518f, 0x58, 0, 0}, {0x5190, 0x46, 0, 0},
+	{0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0}, {0x5193, 0x70, 0, 0},
+	{0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0}, {0x5196, 0x03, 0, 0},
+	{0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0}, {0x5199, 0x12, 0, 0},
+	{0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0}, {0x519c, 0x06, 0, 0},
+	{0x519d, 0x82, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5025, 0x80, 0, 0},
+	{0x3a0f, 0x38, 0, 0}, {0x3a10, 0x30, 0, 0}, {0x3a1b, 0x3a, 0, 0},
+	{0x3a1e, 0x2e, 0, 0}, {0x3a11, 0x60, 0, 0}, {0x3a1f, 0x10, 0, 0},
+	{0x5688, 0xa6, 0, 0}, {0x5689, 0x6a, 0, 0}, {0x568a, 0xea, 0, 0},
+	{0x568b, 0xae, 0, 0}, {0x568c, 0xa6, 0, 0}, {0x568d, 0x6a, 0, 0},
+	{0x568e, 0x62, 0, 0}, {0x568f, 0x26, 0, 0}, {0x5583, 0x40, 0, 0},
+	{0x5584, 0x40, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5000, 0xcf, 0, 0},
+	{0x5800, 0x27, 0, 0}, {0x5801, 0x19, 0, 0}, {0x5802, 0x12, 0, 0},
+	{0x5803, 0x0f, 0, 0}, {0x5804, 0x10, 0, 0}, {0x5805, 0x15, 0, 0},
+	{0x5806, 0x1e, 0, 0}, {0x5807, 0x2f, 0, 0}, {0x5808, 0x15, 0, 0},
+	{0x5809, 0x0d, 0, 0}, {0x580a, 0x0a, 0, 0}, {0x580b, 0x09, 0, 0},
+	{0x580c, 0x0a, 0, 0}, {0x580d, 0x0c, 0, 0}, {0x580e, 0x12, 0, 0},
+	{0x580f, 0x19, 0, 0}, {0x5810, 0x0b, 0, 0}, {0x5811, 0x07, 0, 0},
+	{0x5812, 0x04, 0, 0}, {0x5813, 0x03, 0, 0}, {0x5814, 0x03, 0, 0},
+	{0x5815, 0x06, 0, 0}, {0x5816, 0x0a, 0, 0}, {0x5817, 0x0f, 0, 0},
+	{0x5818, 0x0a, 0, 0}, {0x5819, 0x05, 0, 0}, {0x581a, 0x01, 0, 0},
+	{0x581b, 0x00, 0, 0}, {0x581c, 0x00, 0, 0}, {0x581d, 0x03, 0, 0},
+	{0x581e, 0x08, 0, 0}, {0x581f, 0x0c, 0, 0}, {0x5820, 0x0a, 0, 0},
+	{0x5821, 0x05, 0, 0}, {0x5822, 0x01, 0, 0}, {0x5823, 0x00, 0, 0},
+	{0x5824, 0x00, 0, 0}, {0x5825, 0x03, 0, 0}, {0x5826, 0x08, 0, 0},
+	{0x5827, 0x0c, 0, 0}, {0x5828, 0x0e, 0, 0}, {0x5829, 0x08, 0, 0},
+	{0x582a, 0x06, 0, 0}, {0x582b, 0x04, 0, 0}, {0x582c, 0x05, 0, 0},
+	{0x582d, 0x07, 0, 0}, {0x582e, 0x0b, 0, 0}, {0x582f, 0x12, 0, 0},
+	{0x5830, 0x18, 0, 0}, {0x5831, 0x10, 0, 0}, {0x5832, 0x0c, 0, 0},
+	{0x5833, 0x0a, 0, 0}, {0x5834, 0x0b, 0, 0}, {0x5835, 0x0e, 0, 0},
+	{0x5836, 0x15, 0, 0}, {0x5837, 0x19, 0, 0}, {0x5838, 0x32, 0, 0},
+	{0x5839, 0x1f, 0, 0}, {0x583a, 0x18, 0, 0}, {0x583b, 0x16, 0, 0},
+	{0x583c, 0x17, 0, 0}, {0x583d, 0x1e, 0, 0}, {0x583e, 0x26, 0, 0},
+	{0x583f, 0x53, 0, 0}, {0x5840, 0x10, 0, 0}, {0x5841, 0x0f, 0, 0},
+	{0x5842, 0x0d, 0, 0}, {0x5843, 0x0c, 0, 0}, {0x5844, 0x0e, 0, 0},
+	{0x5845, 0x09, 0, 0}, {0x5846, 0x11, 0, 0}, {0x5847, 0x10, 0, 0},
+	{0x5848, 0x10, 0, 0}, {0x5849, 0x10, 0, 0}, {0x584a, 0x10, 0, 0},
+	{0x584b, 0x0e, 0, 0}, {0x584c, 0x10, 0, 0}, {0x584d, 0x10, 0, 0},
+	{0x584e, 0x11, 0, 0}, {0x584f, 0x10, 0, 0}, {0x5850, 0x0f, 0, 0},
+	{0x5851, 0x0c, 0, 0}, {0x5852, 0x0f, 0, 0}, {0x5853, 0x10, 0, 0},
+	{0x5854, 0x10, 0, 0}, {0x5855, 0x0f, 0, 0}, {0x5856, 0x0e, 0, 0},
+	{0x5857, 0x0b, 0, 0}, {0x5858, 0x10, 0, 0}, {0x5859, 0x0d, 0, 0},
+	{0x585a, 0x0d, 0, 0}, {0x585b, 0x0c, 0, 0}, {0x585c, 0x0c, 0, 0},
+	{0x585d, 0x0c, 0, 0}, {0x585e, 0x0b, 0, 0}, {0x585f, 0x0c, 0, 0},
+	{0x5860, 0x0c, 0, 0}, {0x5861, 0x0c, 0, 0}, {0x5862, 0x0d, 0, 0},
+	{0x5863, 0x08, 0, 0}, {0x5864, 0x11, 0, 0}, {0x5865, 0x18, 0, 0},
+	{0x5866, 0x18, 0, 0}, {0x5867, 0x19, 0, 0}, {0x5868, 0x17, 0, 0},
+	{0x5869, 0x19, 0, 0}, {0x586a, 0x16, 0, 0}, {0x586b, 0x13, 0, 0},
+	{0x586c, 0x13, 0, 0}, {0x586d, 0x12, 0, 0}, {0x586e, 0x13, 0, 0},
+	{0x586f, 0x16, 0, 0}, {0x5870, 0x14, 0, 0}, {0x5871, 0x12, 0, 0},
+	{0x5872, 0x10, 0, 0}, {0x5873, 0x11, 0, 0}, {0x5874, 0x11, 0, 0},
+	{0x5875, 0x16, 0, 0}, {0x5876, 0x14, 0, 0}, {0x5877, 0x11, 0, 0},
+	{0x5878, 0x10, 0, 0}, {0x5879, 0x0f, 0, 0}, {0x587a, 0x10, 0, 0},
+	{0x587b, 0x14, 0, 0}, {0x587c, 0x13, 0, 0}, {0x587d, 0x12, 0, 0},
+	{0x587e, 0x11, 0, 0}, {0x587f, 0x11, 0, 0}, {0x5880, 0x12, 0, 0},
+	{0x5881, 0x15, 0, 0}, {0x5882, 0x14, 0, 0}, {0x5883, 0x15, 0, 0},
+	{0x5884, 0x15, 0, 0}, {0x5885, 0x15, 0, 0}, {0x5886, 0x13, 0, 0},
+	{0x5887, 0x17, 0, 0}, {0x3710, 0x10, 0, 0}, {0x3632, 0x51, 0, 0},
+	{0x3702, 0x10, 0, 0}, {0x3703, 0xb2, 0, 0}, {0x3704, 0x18, 0, 0},
+	{0x370b, 0x40, 0, 0}, {0x370d, 0x03, 0, 0}, {0x3631, 0x01, 0, 0},
+	{0x3632, 0x52, 0, 0}, {0x3606, 0x24, 0, 0}, {0x3620, 0x96, 0, 0},
+	{0x5785, 0x07, 0, 0}, {0x3a13, 0x30, 0, 0}, {0x3600, 0x52, 0, 0},
+	{0x3604, 0x48, 0, 0}, {0x3606, 0x1b, 0, 0}, {0x370d, 0x0b, 0, 0},
+	{0x370f, 0xc0, 0, 0}, {0x3709, 0x01, 0, 0}, {0x3823, 0x00, 0, 0},
+	{0x5007, 0x00, 0, 0}, {0x5009, 0x00, 0, 0}, {0x5011, 0x00, 0, 0},
+	{0x5013, 0x00, 0, 0}, {0x519e, 0x00, 0, 0}, {0x5086, 0x00, 0, 0},
+	{0x5087, 0x00, 0, 0}, {0x5088, 0x00, 0, 0}, {0x5089, 0x00, 0, 0},
+	{0x302b, 0x00, 0, 0}, {0x3503, 0x07, 0, 0}, {0x3011, 0x08, 0, 0},
+	{0x350c, 0x02, 0, 0}, {0x350d, 0xe4, 0, 0}, {0x3621, 0xc9, 0, 0},
+	{0x370a, 0x81, 0, 0}, {0x3803, 0x08, 0, 0}, {0x3804, 0x05, 0, 0},
+	{0x3805, 0x00, 0, 0}, {0x3806, 0x02, 0, 0}, {0x3807, 0xd0, 0, 0},
+	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+	{0x380b, 0xd0, 0, 0}, {0x380c, 0x08, 0, 0}, {0x380d, 0x72, 0, 0},
+	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0xc0, 0, 0},
+	{0x3818, 0xc9, 0, 0}, {0x381c, 0x10, 0, 0}, {0x381d, 0xa0, 0, 0},
+	{0x381e, 0x05, 0, 0}, {0x381f, 0xb0, 0, 0}, {0x3820, 0x00, 0, 0},
+	{0x3821, 0x00, 0, 0}, {0x3824, 0x11, 0, 0}, {0x3a08, 0x1b, 0, 0},
+	{0x3a09, 0xc0, 0, 0}, {0x3a0a, 0x17, 0, 0}, {0x3a0b, 0x20, 0, 0},
+	{0x3a0d, 0x02, 0, 0}, {0x3a0e, 0x01, 0, 0}, {0x401c, 0x04, 0, 0},
+	{0x5682, 0x05, 0, 0}, {0x5683, 0x00, 0, 0}, {0x5686, 0x02, 0, 0},
+	{0x5687, 0xcc, 0, 0}, {0x5001, 0x7f, 0, 0}, {0x589b, 0x06, 0, 0},
+	{0x589a, 0xc5, 0, 0}, {0x3503, 0x00, 0, 0}, {0x3010, 0x10, 0, 0},
+	{0x460c, 0x20, 0, 0}, {0x460b, 0x37, 0, 0}, {0x471c, 0xd0, 0, 0},
+	{0x471d, 0x05, 0, 0}, {0x3815, 0x01, 0, 0}, {0x3818, 0x00, 0x08, 0},
+	{0x501f, 0x00, 0, 0}, {0x4300, 0x30, 0, 0}, {0x3002, 0x1c, 0, 0},
+	{0x3819, 0x80, 0, 0}, {0x5002, 0xe0, 0, 0}, {0x3010, 0x30, 0, 0},
+	{0x3a08, 0x06, 0, 0}, {0x3a09, 0x60, 0, 0}, {0x3a0a, 0x05, 0, 0},
+	{0x3a0b, 0x50, 0, 0}, {0x3a0d, 0x08, 0, 0}, {0x3a0e, 0x07, 0, 0},
+};
+
+static struct ov5642_mode_info
+ov5642_mode_info_data[ov5642_num_framerates][ov5642_num_modes] = {
+	{
+		{ov5642_mode_QCIF_176_144, 176, 144,
+		ov5642_setting_15fps_QCIF_176_144,
+		ARRAY_SIZE(ov5642_setting_15fps_QCIF_176_144)},
+		{ov5642_mode_QVGA_320_240,   320,  240,
+		ov5642_setting_15fps_QVGA_320_240,
+		ARRAY_SIZE(ov5642_setting_15fps_QVGA_320_240)},
+		{ov5642_mode_VGA_640_480,    640,  480,
+		ov5642_setting_15fps_VGA_640_480,
+		ARRAY_SIZE(ov5642_setting_15fps_VGA_640_480)},
+		{ov5642_mode_NTSC_720_480,   720,  480,
+		ov5642_setting_15fps_NTSC_720_480,
+		ARRAY_SIZE(ov5642_setting_15fps_NTSC_720_480)},
+		{ov5642_mode_PAL_720_576,   720,  576,
+		ov5642_setting_15fps_PAL_720_576,
+		ARRAY_SIZE(ov5642_setting_15fps_PAL_720_576)},
+		{ov5642_mode_XGA_1024_768, 1024, 768,
+		ov5642_setting_15fps_XGA_1024_768,
+		ARRAY_SIZE(ov5642_setting_15fps_XGA_1024_768)},
+		{ov5642_mode_720P_1280_720,  1280, 720,
+		ov5642_setting_15fps_720P_1280_720,
+		ARRAY_SIZE(ov5642_setting_15fps_720P_1280_720)},
+		{ov5642_mode_1080P_1920_1080, 1920, 1080,
+		ov5642_setting_15fps_1080P_1920_1080,
+		ARRAY_SIZE(ov5642_setting_15fps_1080P_1920_1080)},
+		{ov5642_mode_QSXGA_2592_1944, 2592, 1944,
+		ov5642_setting_15fps_QSXGA_2592_1944,
+		ARRAY_SIZE(ov5642_setting_15fps_QSXGA_2592_1944)},
+	},
+	{
+		{ov5642_mode_QCIF_176_144, 176, 144,
+		ov5642_setting_30fps_QCIF_176_144,
+		ARRAY_SIZE(ov5642_setting_30fps_QCIF_176_144)},
+		{ov5642_mode_QVGA_320_240,   320,  240,
+		ov5642_setting_30fps_QVGA_320_240,
+		ARRAY_SIZE(ov5642_setting_30fps_QVGA_320_240)},
+		{ov5642_mode_VGA_640_480,    640,  480,
+		ov5642_setting_30fps_VGA_640_480,
+		ARRAY_SIZE(ov5642_setting_30fps_VGA_640_480)},
+		{ov5642_mode_NTSC_720_480,   720, 480,
+		ov5642_setting_30fps_NTSC_720_480,
+		ARRAY_SIZE(ov5642_setting_30fps_NTSC_720_480)},
+		{ov5642_mode_PAL_720_576,    720, 576,
+		ov5642_setting_30fps_PAL_720_576,
+		ARRAY_SIZE(ov5642_setting_30fps_PAL_720_576)},
+		{ov5642_mode_XGA_1024_768, 1024, 768,
+		ov5642_setting_30fps_XGA_1024_768,
+		ARRAY_SIZE(ov5642_setting_30fps_XGA_1024_768)},
+		{ov5642_mode_720P_1280_720,  1280, 720,
+		ov5642_setting_30fps_720P_1280_720,
+		ARRAY_SIZE(ov5642_setting_30fps_720P_1280_720)},
+		{ov5642_mode_1080P_1920_1080, 0, 0, NULL, 0},
+		{ov5642_mode_QSXGA_2592_1944, 0, 0, NULL, 0},
+	},
+};
+
+static int ov5642_init_slave_id(struct ov5642_dev *sensor)
+{
+	struct i2c_msg msg;
+	u8 buf[4];
+	int ret;
+
+	if (sensor->i2c_client->addr == OV5642_DEFAULT_SLAVE_ID)
+		return 0;
+
+	buf[0] = OV5642_SLAVE_ID >> 8;
+	buf[1] = OV5642_SLAVE_ID & 0xff;
+	buf[2] = sensor->i2c_client->addr << 1;
+	msg.addr = OV5642_DEFAULT_SLAVE_ID;
+	msg.flags = 0;
+	msg.len = 3;
+	msg.buf = buf;
+
+	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
+	if (ret < 0) {
+		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int ov5642_write_reg(struct ov5642_dev *sensor, u16 reg, u8 val)
+{
+	u8 au8Buf[3] = {0};
+	int ret;
+
+	au8Buf[0] = reg >> 8;
+	au8Buf[1] = reg & 0xff;
+	au8Buf[2] = val;
+
+	ret = i2c_master_send(sensor->i2c_client, au8Buf, 3);
+	if (ret < 0) {
+		dev_err(sensor->dev, "%s:write reg error:reg=%x,val=%x\n",
+			__func__, reg, val);
+		return ret;
+	}
+
+	/* if this is a s/w reset we need to reprogram the i2c slave address */
+	if (reg == 0x3008 && (val & 0x80))
+		return ov5642_init_slave_id(sensor);
+
+	return 0;
+}
+
+static int ov5642_read_reg(struct ov5642_dev *sensor, u16 reg, u8 *val)
+{
+	u8 au8RegBuf[2] = {0};
+	u8 u8RdVal = 0;
+
+	au8RegBuf[0] = reg >> 8;
+	au8RegBuf[1] = reg & 0xff;
+
+	if (2 != i2c_master_send(sensor->i2c_client, au8RegBuf, 2)) {
+		dev_err(sensor->dev, "%s:write reg error:reg=%x\n",
+			__func__, reg);
+		return -EIO;
+	}
+
+	if (1 != i2c_master_recv(sensor->i2c_client, &u8RdVal, 1)) {
+		dev_err(sensor->dev, "%s:read reg error:reg=%x,val=%x\n",
+			__func__, reg, u8RdVal);
+		return -EIO;
+	}
+
+	*val = u8RdVal;
+	return 0;
+}
+
+#define OV5642_READ_REG(s, r, v) {				\
+		ret = ov5642_read_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+#define OV5642_WRITE_REG(s, r, v) {				\
+		ret = ov5642_write_reg((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5642_read_reg16(struct ov5642_dev *sensor, u16 reg, u16 *val)
+{
+	u8 hi, lo;
+	int ret;
+
+	OV5642_READ_REG(sensor, reg, &hi);
+	OV5642_READ_REG(sensor, reg+1, &lo);
+
+	*val = ((u16)hi << 8) | (u16)lo;
+	return 0;
+}
+
+#define OV5642_READ_REG16(s, r, v) {				\
+		ret = ov5642_read_reg16((s), (r), (v));		\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5642_mod_reg(struct ov5642_dev *sensor, u16 reg,
+			  u8 mask, u8 val)
+{
+	u8 readval;
+	int ret;
+
+	OV5642_READ_REG(sensor, reg, &readval);
+
+	readval &= ~mask;
+	val &= mask;
+	val |= readval;
+
+	OV5642_WRITE_REG(sensor, reg, val);
+	return 0;
+}
+
+#define OV5642_MOD_REG(s, r, m, v) {				\
+		ret = ov5642_mod_reg((s), (r), (m), (v));	\
+		if (ret)					\
+			return ret;				\
+	}
+
+static int ov5642_load_regs(struct ov5642_dev *sensor,
+			    struct reg_value *regs,
+			    int size)
+{
+	register u32 delay_ms = 0;
+	register u16 reg_addr = 0;
+	register u8 mask = 0;
+	register u8 val = 0;
+	int i, ret = 0;
+
+	for (i = 0; i < size; ++i, ++regs) {
+		delay_ms = regs->delay_ms;
+		reg_addr = regs->reg_addr;
+		val = regs->val;
+		mask = regs->mask;
+
+		if (mask) {
+			OV5642_MOD_REG(sensor, reg_addr, mask, val);
+		} else {
+			OV5642_WRITE_REG(sensor, reg_addr, val);
+		}
+		if (delay_ms)
+			usleep_range(1000*delay_ms, 1000*delay_ms+100);
+	}
+
+	return 0;
+}
+
+static int ov5642_dump_format(struct ov5642_dev *sensor)
+{
+	u8 scaling, l_pix, r_pix, pix_fmt;
+	u16 hs, vs, hw, vh, dvp_w, dvp_h;
+	int ret;
+
+	OV5642_READ_REG16(sensor, 0x3800, &hs);
+	OV5642_READ_REG16(sensor, 0x3802, &vs);
+	OV5642_READ_REG16(sensor, 0x3804, &hw);
+	OV5642_READ_REG16(sensor, 0x3806, &vh);
+
+	OV5642_READ_REG(sensor, 0x5001, &scaling);
+	OV5642_READ_REG16(sensor, 0x3808, &dvp_w);
+	OV5642_READ_REG16(sensor, 0x380a, &dvp_h);
+
+	OV5642_READ_REG(sensor, 0x4711, &l_pix);
+	OV5642_READ_REG(sensor, 0x4712, &r_pix);
+	OV5642_READ_REG(sensor, 0x4300, &pix_fmt);
+
+	dev_dbg(sensor->dev, "Image Window:\n");
+	dev_dbg(sensor->dev, "\thoriz: %u@%u\n", hw, hs);
+	dev_dbg(sensor->dev, "\tvert : %u@%u\n", vh, vs);
+	dev_dbg(sensor->dev, "DVP Scaling:\n");
+	dev_dbg(sensor->dev, "\thoriz %s, vert %s, %ux%u\n",
+		(scaling & (1 << 4)) ? "enabled" : "disabled",
+		(scaling & (1 << 5)) ? "enabled" : "disabled",
+		dvp_w, dvp_h);
+	dev_dbg(sensor->dev, "DVP Padding:\n");
+	dev_dbg(sensor->dev, "\tleft %u pixels, right %u pixels\n",
+		l_pix, r_pix);
+	dev_dbg(sensor->dev, "Pixel Format: %02x\n", pix_fmt);
+
+	return 0;
+}
+
+static int ov5642_init_mode(struct ov5642_dev *sensor,
+			    enum ov5642_frame_rate frame_rate,
+			    enum ov5642_mode mode);
+static int ov5642_write_snapshot_para(struct ov5642_dev *sensor,
+				      enum ov5642_frame_rate frame_rate,
+				      enum ov5642_mode mode);
+
+static enum ov5642_mode
+ov5642_find_nearest_mode(struct ov5642_dev *sensor,
+			 int width, int height)
+{
+	int i;
+
+	for (i = ov5642_num_modes - 1; i >= 0; i--) {
+		if (ov5642_mode_info_data[0][i].width <= width &&
+		    ov5642_mode_info_data[0][i].height <= height)
+			break;
+	}
+
+	if (i < 0)
+		i = 0;
+
+	return (enum ov5642_mode)i;
+}
+
+static int ov5642_change_mode(struct ov5642_dev *sensor,
+			      enum ov5642_frame_rate new_frame_rate,
+			      enum ov5642_frame_rate old_frame_rate,
+			      enum ov5642_mode new_mode,
+			      enum ov5642_mode orig_mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+	int ret = 0;
+
+	if (new_mode >= ov5642_num_modes || new_mode < ov5642_mode_MIN) {
+		dev_err(sensor->dev, "Wrong ov5642 mode detected!\n");
+		return -EINVAL;
+	}
+
+	if (new_frame_rate == old_frame_rate) {
+		if (new_mode == orig_mode)
+			return 0;
+		else if (new_mode == ov5642_mode_VGA_640_480 &&
+			 orig_mode == ov5642_mode_QSXGA_2592_1944) {
+			mode_data = ov5642_setting_QSXGA_2_VGA;
+			mode_size = ARRAY_SIZE(ov5642_setting_QSXGA_2_VGA);
+			sensor->fmt.width = 640;
+			sensor->fmt.height = 480;
+		} else if (new_mode == ov5642_mode_QVGA_320_240 &&
+			   orig_mode == ov5642_mode_VGA_640_480) {
+			mode_data = ov5642_setting_VGA_2_QVGA;
+			mode_size = ARRAY_SIZE(ov5642_setting_VGA_2_QVGA);
+			sensor->fmt.width = 320;
+			sensor->fmt.height = 240;
+		} else
+			goto load_full;
+
+		ret = ov5642_load_regs(sensor, mode_data, mode_size);
+	} else {
+load_full:
+		ret = ov5642_write_snapshot_para(sensor, new_frame_rate,
+						    new_mode);
+	}
+
+	if (ret)
+		return ret;
+
+	/* restore controls */
+	ov5642_restore_ctrls(sensor);
+
+	if (ret >= 0) {
+		sensor->current_mode = new_mode;
+		sensor->current_fr = new_frame_rate;
+
+		ov5642_dump_format(sensor);
+	}
+
+	return ret;
+}
+
+static int ov5642_restore_mode(struct ov5642_dev *sensor)
+{
+	int ret = 0;
+
+	/* first we need to set some initial register values */
+	ret = ov5642_load_regs(sensor, ov5642_initial_setting,
+				  ARRAY_SIZE(ov5642_initial_setting));
+	if (ret < 0)
+		return ret;
+
+	/* now restore the last capture mode */
+	return ov5642_change_mode(sensor,
+				  sensor->current_fr,
+				  sensor->current_fr,
+				  sensor->current_mode,
+				  ov5642_mode_VGA_640_480);
+}
+
+static int ov5642_init_mode(struct ov5642_dev *sensor,
+			    enum ov5642_frame_rate frame_rate,
+			    enum ov5642_mode mode)
+{
+	struct reg_value *mode_data = NULL;
+	int mode_size = 0;
+
+	if (mode >= ov5642_num_modes || mode < ov5642_mode_MIN) {
+		dev_err(sensor->dev, "Wrong ov5642 mode detected!\n");
+		return -EINVAL;
+	}
+
+	mode_data = ov5642_mode_info_data[frame_rate][mode].init_data_ptr;
+	mode_size = ov5642_mode_info_data[frame_rate][mode].init_data_size;
+
+	sensor->fmt.width = ov5642_mode_info_data[frame_rate][mode].width;
+	sensor->fmt.height = ov5642_mode_info_data[frame_rate][mode].height;
+
+	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+	    mode_data == NULL || mode_size == 0)
+		return -EINVAL;
+
+	return ov5642_load_regs(sensor, mode_data, mode_size);
+}
+
+static int ov5642_write_snapshot_para(struct ov5642_dev *sensor,
+				      enum ov5642_frame_rate frame_rate,
+				      enum ov5642_mode mode)
+{
+	bool m_60Hz = false;
+	u16 capture_frame_rate = 50;
+	u16 g_preview_frame_rate = 225;
+	u8 ret_l, ret_m, ret_h, gain, lines_10ms;
+	u16 ulcapture_exposure, capture_maxlines;
+	u16 icapture_gain, preview_maxlines;
+	u32 ulcapture_exposure_gain, g_preview_exposure;
+	int ret;
+
+	ov5642_set_agc(sensor, 0);
+
+	ret_h = ret_m = ret_l = 0;
+	g_preview_exposure = 0;
+	OV5642_READ_REG(sensor, 0x3500, &ret_h);
+	OV5642_READ_REG(sensor, 0x3501, &ret_m);
+	OV5642_READ_REG(sensor, 0x3502, &ret_l);
+	g_preview_exposure = (ret_h << 12) + (ret_m << 4) + (ret_l >> 4);
+
+	OV5642_READ_REG16(sensor, 0x380e, &preview_maxlines);
+	/*Read back AGC Gain for preview*/
+	gain = 0;
+	OV5642_READ_REG(sensor, 0x350b, &gain);
+
+	ret = ov5642_init_mode(sensor, frame_rate, mode);
+	if (ret < 0)
+		return ret;
+
+	OV5642_READ_REG16(sensor, 0x380e, &capture_maxlines);
+	if (m_60Hz)
+		lines_10ms = capture_frame_rate * (u32)capture_maxlines/12000;
+	else
+		lines_10ms = capture_frame_rate * (u32)capture_maxlines/10000;
+
+	if (preview_maxlines == 0)
+		preview_maxlines = 1;
+
+	ulcapture_exposure = (g_preview_exposure * capture_frame_rate *
+			      (u32)capture_maxlines) /
+		(preview_maxlines * g_preview_frame_rate);
+	icapture_gain = (gain & 0x0f) + 16;
+	if (gain & 0x10)
+		icapture_gain = icapture_gain << 1;
+
+	if (gain & 0x20)
+		icapture_gain = icapture_gain << 1;
+
+	if (gain & 0x40)
+		icapture_gain = icapture_gain << 1;
+
+	if (gain & 0x80)
+		icapture_gain = icapture_gain << 1;
+
+	ulcapture_exposure_gain = 2 * ulcapture_exposure * icapture_gain;
+
+	if (ulcapture_exposure_gain < (u32)capture_maxlines*16) {
+		ulcapture_exposure = ulcapture_exposure_gain/16;
+		if (ulcapture_exposure > lines_10ms) {
+			ulcapture_exposure /= lines_10ms;
+			ulcapture_exposure *= lines_10ms;
+		}
+	} else
+		ulcapture_exposure = (u32)capture_maxlines;
+
+	if (ulcapture_exposure == 0)
+		ulcapture_exposure = 1;
+
+	icapture_gain = (ulcapture_exposure_gain*2/ulcapture_exposure + 1)/2;
+	gain = 0;
+	if (icapture_gain > 31) {
+		gain |= 0x10;
+		icapture_gain = icapture_gain >> 1;
+	}
+	if (icapture_gain > 31) {
+		gain |= 0x20;
+		icapture_gain = icapture_gain >> 1;
+	}
+	if (icapture_gain > 31) {
+		gain |= 0x40;
+		icapture_gain = icapture_gain >> 1;
+	}
+	if (icapture_gain > 31) {
+		gain |= 0x80;
+		icapture_gain = icapture_gain >> 1;
+	}
+	if (icapture_gain > 16)
+		gain |= ((icapture_gain - 16) & 0x0f);
+
+	if (gain == 0x10)
+		gain = 0x11;
+
+	ov5642_set_gain(sensor, gain);
+	ov5642_set_exposure(sensor, ulcapture_exposure);
+
+	return 0;
+}
+
+static int ov5642_regulators_on(struct ov5642_dev *sensor)
+{
+	int ret;
+
+	if (sensor->io_regulator) {
+		ret = regulator_enable(sensor->io_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "io reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->core_regulator) {
+		ret = regulator_enable(sensor->core_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "core reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->gpo_regulator) {
+		ret = regulator_enable(sensor->gpo_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
+			return ret;
+		}
+	}
+	if (sensor->analog_regulator) {
+		ret = regulator_enable(sensor->analog_regulator);
+		if (ret) {
+			v4l2_err(&sensor->sd, "analog reg enable failed\n");
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static void ov5642_regulators_off(struct ov5642_dev *sensor)
+{
+	if (sensor->analog_regulator)
+		regulator_disable(sensor->analog_regulator);
+	if (sensor->core_regulator)
+		regulator_disable(sensor->core_regulator);
+	if (sensor->io_regulator)
+		regulator_disable(sensor->io_regulator);
+	if (sensor->gpo_regulator)
+		regulator_disable(sensor->gpo_regulator);
+}
+
+/* --------------- Subdev Operations --------------- */
+
+static int ov5642_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+	int ret;
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (on && !sensor->on) {
+		if (sensor->xclk)
+			clk_prepare_enable(sensor->xclk);
+
+		ret = ov5642_regulators_on(sensor);
+		if (ret)
+			return ret;
+
+		ov5642_reset(sensor);
+		ov5642_power(sensor, true);
+
+		ret = ov5642_init_slave_id(sensor);
+		if (ret)
+			return ret;
+
+		ret = ov5642_restore_mode(sensor);
+		if (ret)
+			return ret;
+	} else if (!on && sensor->on) {
+		ov5642_power(sensor, false);
+
+		ov5642_regulators_off(sensor);
+
+		if (sensor->xclk)
+			clk_disable_unprepare(sensor->xclk);
+	}
+
+	sensor->on = on;
+
+	return 0;
+}
+
+static int ov5642_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+	struct v4l2_captureparm *cparm = &a->parm.capture;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	memset(a, 0, sizeof(*a));
+	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	cparm->capability = sensor->streamcap.capability;
+	cparm->timeperframe = sensor->streamcap.timeperframe;
+	cparm->capturemode = sensor->streamcap.capturemode;
+
+	return 0;
+}
+
+static int ov5642_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
+	enum ov5642_frame_rate new_frame_rate;
+	u32 tgt_fps;
+	int ret;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	/* Check that the new frame rate is allowed. */
+	if ((timeperframe->numerator == 0) ||
+	    (timeperframe->denominator == 0)) {
+		timeperframe->denominator = DEFAULT_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps > MAX_FPS) {
+		timeperframe->denominator = MAX_FPS;
+		timeperframe->numerator = 1;
+	} else if (tgt_fps < MIN_FPS) {
+		timeperframe->denominator = MIN_FPS;
+		timeperframe->numerator = 1;
+	}
+
+	/* Actual frame rate we use */
+	tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+	if (tgt_fps == 15)
+		new_frame_rate = ov5642_15_fps;
+	else if (tgt_fps == 30)
+		new_frame_rate = ov5642_30_fps;
+	else {
+		dev_err(sensor->dev, "frame rate is not supported!\n");
+		return -EINVAL;
+	}
+
+	ret = ov5642_change_mode(
+		sensor, new_frame_rate, sensor->current_fr,
+		sensor->current_mode, sensor->current_mode);
+	if (ret < 0)
+		return ret;
+
+	sensor->streamcap.timeperframe = *timeperframe;
+
+	return 0;
+}
+
+static int ov5642_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	format->format = sensor->fmt;
+
+	return 0;
+}
+
+static int ov5642_try_fmt_internal(struct v4l2_subdev *sd,
+				   struct v4l2_mbus_framefmt *fmt,
+				   enum ov5642_mode *new_mode)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+	enum ov5642_mode mode;
+
+	mode = ov5642_find_nearest_mode(sensor, fmt->width, fmt->height);
+
+	fmt->width = ov5642_mode_info_data[0][mode].width;
+	fmt->height = ov5642_mode_info_data[0][mode].height;
+	fmt->code = sensor->fmt.code;
+
+	if (new_mode)
+		*new_mode = mode;
+	return 0;
+}
+
+static int ov5642_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *format)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+	enum ov5642_mode new_mode;
+	int ret;
+
+	if (format->pad != 0)
+		return -EINVAL;
+
+	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
+		ret = ov5642_try_fmt_internal(sd, &format->format, NULL);
+		if (ret)
+			return ret;
+		cfg->try_fmt = format->format;
+		return 0;
+	}
+
+	ret = ov5642_try_fmt_internal(sd, &format->format, &new_mode);
+	if (ret)
+		return ret;
+
+	ret = ov5642_change_mode(sensor,
+				 sensor->current_fr,
+				 sensor->current_fr,
+				 new_mode, sensor->current_mode);
+	if (ret >= 0)
+		sensor->fmt = format->format;
+
+	return ret;
+}
+
+
+/*
+ * Sensor Controls.
+ */
+
+static int ov5642_set_hue(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5642_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
+		OV5642_MOD_REG(sensor, 0x558a, 1 << 6, 1 << 6);
+		OV5642_WRITE_REG(sensor, 0x5581, value & 0xff);
+		OV5642_WRITE_REG(sensor, 0x5582, (value & 0x100) >> 8);
+	} else
+		OV5642_MOD_REG(sensor, 0x5580, 1 << 0, 0);
+	return 0;
+}
+
+static int ov5642_set_contrast(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5642_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
+		OV5642_WRITE_REG(sensor, 0x5589, value & 0xff);
+	} else
+		OV5642_MOD_REG(sensor, 0x5580, 1 << 2, 0);
+	return 0;
+}
+
+static int ov5642_set_saturation(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	if (value) {
+		OV5642_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
+		OV5642_WRITE_REG(sensor, 0x5583, value & 0xff);
+		OV5642_WRITE_REG(sensor, 0x5584, value & 0xff);
+	} else
+		OV5642_MOD_REG(sensor, 0x5580, 1 << 1, 0);
+	return 0;
+}
+
+static int ov5642_set_awb(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	sensor->awb_on = value ? true : false;
+	OV5642_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
+	return 0;
+}
+
+static int ov5642_set_red_balance(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5642_WRITE_REG(sensor, 0x3401, value & 0xff);
+	OV5642_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
+	return 0;
+}
+
+#if 0
+static int ov5642_set_green_balance(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5642_WRITE_REG(sensor, 0x3403, value & 0xff);
+	OV5642_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
+	return 0;
+}
+#endif
+
+static int ov5642_set_blue_balance(struct ov5642_dev *sensor, int value)
+{
+	int ret = 0;
+
+	if (sensor->awb_on)
+		return -EINVAL;
+
+	OV5642_WRITE_REG(sensor, 0x3405, value & 0xff);
+	OV5642_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
+	return 0;
+}
+
+static int ov5642_set_exposure(struct ov5642_dev *sensor, int value)
+{
+	u16 max_exp = 0;
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5642_READ_REG16(sensor, 0x350c, &max_exp);
+	if (value < max_exp) {
+		u32 exp = value << 4;
+
+		OV5642_WRITE_REG(sensor, 0x3502, exp & 0xff);
+		OV5642_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
+		OV5642_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
+	}
+
+	return 0;
+}
+
+static int ov5642_set_agc(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	/* this enables/disables both AEC and AGC */
+	sensor->agc_on = value ? true : false;
+	OV5642_MOD_REG(sensor, 0x3503, 0x7, sensor->agc_on ? 0 : 0x7);
+	return 0;
+}
+
+static int ov5642_set_gain(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	if (sensor->agc_on)
+		return -EINVAL;
+
+	OV5642_WRITE_REG(sensor, 0x350b, value & 0xff);
+	OV5642_WRITE_REG(sensor, 0x350a, (value & 0x100) >> 8);
+	return 0;
+}
+
+#if 0
+static int ov5642_set_test_pattern(struct ov5642_dev *sensor, int value)
+{
+	int ret;
+
+	OV5642_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
+	return 0;
+}
+#endif
+
+static struct ov5642_control ov5642_ctrls[] = {
+	{
+		.set = ov5642_set_agc,
+		.ctrl = {
+			.id = V4L2_CID_AUTOGAIN,
+			.name = "Auto Gain/Exposure Control",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5642_set_exposure,
+		.ctrl = {
+			.id = V4L2_CID_EXPOSURE,
+			.name = "Exposure",
+			.minimum = 0,
+			.maximum = 65535,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5642_set_gain,
+		.ctrl = {
+			.id = V4L2_CID_GAIN,
+			.name = "Gain",
+			.minimum = 0,
+			.maximum = 511,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5642_set_hue,
+		.ctrl = {
+			.id = V4L2_CID_HUE,
+			.name = "Hue",
+			.minimum = 0,
+			.maximum = 359,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5642_set_contrast,
+		.ctrl = {
+			.id = V4L2_CID_CONTRAST,
+			.name = "Contrast",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 0,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5642_set_saturation,
+		.ctrl = {
+			.id = V4L2_CID_SATURATION,
+			.name = "Saturation",
+			.minimum = 0,
+			.maximum = 255,
+			.step = 1,
+			.default_value = 64,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5642_set_awb,
+		.ctrl = {
+			.id = V4L2_CID_AUTO_WHITE_BALANCE,
+			.name = "Auto White Balance",
+			.minimum = 0,
+			.maximum = 1,
+			.step = 1,
+			.default_value = 1,
+			.type = V4L2_CTRL_TYPE_BOOLEAN,
+		},
+	}, {
+		.set = ov5642_set_red_balance,
+		.ctrl = {
+			.id = V4L2_CID_RED_BALANCE,
+			.name = "Red Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 1024,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	}, {
+		.set = ov5642_set_blue_balance,
+		.ctrl = {
+			.id = V4L2_CID_BLUE_BALANCE,
+			.name = "Blue Balance",
+			.minimum = 0,
+			.maximum = 4095,
+			.step = 1,
+			.default_value = 1024,
+			.type = V4L2_CTRL_TYPE_INTEGER,
+		},
+	},
+};
+#define OV5642_NUM_CONTROLS ARRAY_SIZE(ov5642_ctrls)
+
+static struct ov5642_control *ov5642_get_ctrl(int id, int *index)
+{
+	struct ov5642_control *ret = NULL;
+	int i;
+
+	for (i = 0; i < OV5642_NUM_CONTROLS; i++) {
+		if (id == ov5642_ctrls[i].ctrl.id) {
+			ret = &ov5642_ctrls[i];
+			break;
+		}
+	}
+
+	if (ret && index)
+		*index = i;
+	return ret;
+}
+
+static int ov5642_restore_ctrls(struct ov5642_dev *sensor)
+{
+	struct ov5642_control *c;
+	int i;
+
+	for (i = 0; i < OV5642_NUM_CONTROLS; i++) {
+		c = &ov5642_ctrls[i];
+		c->set(sensor, sensor->ctrl_cache[i]);
+	}
+
+	return 0;
+}
+
+static int ov5642_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct ov5642_dev *sensor = ctrl_to_ov5642_dev(ctrl);
+	struct ov5642_control *c;
+	int ret = 0;
+	int i;
+
+	c = ov5642_get_ctrl(ctrl->id, &i);
+	if (!c)
+		return -EINVAL;
+
+	ret = c->set(sensor, ctrl->val);
+	/* update cached value if no error */
+	if (!ret)
+		sensor->ctrl_cache[i] = ctrl->val;
+
+	return ret;
+}
+
+static const struct v4l2_ctrl_ops ov5642_ctrl_ops = {
+	.s_ctrl = ov5642_s_ctrl,
+};
+
+static int ov5642_init_controls(struct ov5642_dev *sensor)
+{
+	struct ov5642_control *c;
+	int i;
+
+	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5642_NUM_CONTROLS);
+
+	for (i = 0; i < OV5642_NUM_CONTROLS; i++) {
+		c = &ov5642_ctrls[i];
+
+		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5642_ctrl_ops,
+				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
+				  c->ctrl.step, c->ctrl.default_value);
+	}
+
+	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
+	if (sensor->ctrl_hdl.error) {
+		int err = sensor->ctrl_hdl.error;
+
+		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
+		return err;
+	}
+	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static int ov5642_enum_frame_size(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_frame_size_enum *fse)
+{
+	if (fse->pad != 0)
+		return -EINVAL;
+	if (fse->index >= ov5642_num_modes)
+		return -EINVAL;
+
+	fse->min_width = fse->max_width =
+		ov5642_mode_info_data[0][fse->index].width;
+	fse->min_height = fse->max_height =
+		ov5642_mode_info_data[0][fse->index].height;
+
+	return 0;
+}
+
+static int ov5642_enum_frame_interval(
+	struct v4l2_subdev *sd,
+	struct v4l2_subdev_pad_config *cfg,
+	struct v4l2_subdev_frame_interval_enum *fie)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+	enum ov5642_mode mode;
+
+	if (fie->pad != 0)
+		return -EINVAL;
+	if (fie->index < 0 || fie->index >= ov5642_num_framerates)
+		return -EINVAL;
+
+	if (fie->width == 0 || fie->height == 0)
+		return -EINVAL;
+
+	mode = ov5642_find_nearest_mode(sensor, fie->width, fie->height);
+
+	if (ov5642_mode_info_data[fie->index][mode].init_data_ptr == NULL)
+		return -EINVAL;
+
+	fie->interval.numerator = 1;
+	fie->interval.denominator = ov5642_framerates[fie->index];
+
+	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
+		fie->width, fie->height, fie->index, fie->interval.denominator);
+	return 0;
+}
+
+static int ov5642_g_input_status(struct v4l2_subdev *sd, u32 *status)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+
+	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
+
+	return 0;
+}
+
+static int ov5642_s_routing(struct v4l2_subdev *sd, u32 input,
+			    u32 output, u32 config)
+{
+	return (input != 0) ? -EINVAL : 0;
+}
+
+static int ov5642_enum_mbus_code(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+
+	if (code->pad != 0)
+		return -EINVAL;
+	if (code->index != 0)
+		return -EINVAL;
+
+	code->code = sensor->fmt.code;
+
+	return 0;
+}
+
+static int ov5642_g_mbus_config(struct v4l2_subdev *sd,
+				struct v4l2_mbus_config *cfg)
+{
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+
+	cfg->type = V4L2_MBUS_PARALLEL;
+	cfg->flags = sensor->ep.bus.parallel.flags;
+
+	return 0;
+}
+
+static int ov5642_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+	return 0;
+}
+
+static struct v4l2_subdev_core_ops ov5642_core_ops = {
+	.s_power = ov5642_s_power,
+};
+
+static struct v4l2_subdev_video_ops ov5642_video_ops = {
+	.s_parm = ov5642_s_parm,
+	.g_parm = ov5642_g_parm,
+	.g_input_status = ov5642_g_input_status,
+	.s_routing = ov5642_s_routing,
+	.g_mbus_config  = ov5642_g_mbus_config,
+	.s_stream = ov5642_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops ov5642_pad_ops = {
+	.enum_mbus_code = ov5642_enum_mbus_code,
+	.get_fmt = ov5642_get_fmt,
+	.set_fmt = ov5642_set_fmt,
+	.enum_frame_size = ov5642_enum_frame_size,
+	.enum_frame_interval = ov5642_enum_frame_interval,
+};
+
+static struct v4l2_subdev_ops ov5642_subdev_ops = {
+	.core = &ov5642_core_ops,
+	.video = &ov5642_video_ops,
+	.pad = &ov5642_pad_ops,
+};
+
+static void ov5642_power(struct ov5642_dev *sensor, bool enable)
+{
+	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
+}
+
+static void ov5642_reset(struct ov5642_dev *sensor)
+{
+	gpiod_set_value(sensor->reset_gpio, 0);
+
+	/* camera power cycle */
+	ov5642_power(sensor, false);
+	usleep_range(5000, 10000);
+	ov5642_power(sensor, true);
+	usleep_range(5000, 10000);
+
+	gpiod_set_value(sensor->reset_gpio, 1);
+	usleep_range(1000, 2000);
+
+	gpiod_set_value(sensor->reset_gpio, 0);
+	usleep_range(5000, 10000);
+}
+
+static void ov5642_get_regulators(struct ov5642_dev *sensor)
+{
+	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
+	if (!IS_ERR(sensor->io_regulator)) {
+		regulator_set_voltage(sensor->io_regulator,
+				      OV5642_VOLTAGE_DIGITAL_IO,
+				      OV5642_VOLTAGE_DIGITAL_IO);
+	} else {
+		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
+			__func__);
+		sensor->io_regulator = NULL;
+	}
+
+	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
+	if (!IS_ERR(sensor->core_regulator)) {
+		regulator_set_voltage(sensor->core_regulator,
+				      OV5642_VOLTAGE_DIGITAL_CORE,
+				      OV5642_VOLTAGE_DIGITAL_CORE);
+	} else {
+		sensor->core_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
+			__func__);
+	}
+
+	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
+	if (!IS_ERR(sensor->analog_regulator)) {
+		regulator_set_voltage(sensor->analog_regulator,
+				      OV5642_VOLTAGE_ANALOG,
+				      OV5642_VOLTAGE_ANALOG);
+	} else {
+		sensor->analog_regulator = NULL;
+		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
+			__func__);
+	}
+}
+
+/*!
+ * ov5642 I2C probe function
+ *
+ * @param adapter            struct i2c_adapter *
+ * @return  Error code indicating success or failure
+ */
+static int ov5642_probe(struct i2c_client *client,
+			const struct i2c_device_id *id)
+{
+	struct device *dev = &client->dev;
+	struct device_node *endpoint;
+	struct ov5642_dev *sensor;
+	int i, xclk, ret;
+
+	sensor = devm_kzalloc(dev, sizeof(struct ov5642_dev),
+			      GFP_KERNEL);
+	if (!sensor)
+		return -ENOMEM;
+
+	sensor->i2c_client = client;
+	sensor->dev = dev;
+	sensor->fmt.code = MEDIA_BUS_FMT_YUYV8_2X8;
+	sensor->fmt.width = 640;
+	sensor->fmt.height = 480;
+	sensor->fmt.field = V4L2_FIELD_NONE;
+	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
+					   V4L2_CAP_TIMEPERFRAME;
+	sensor->streamcap.capturemode = 0;
+	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
+	sensor->streamcap.timeperframe.numerator = 1;
+
+	sensor->current_mode = ov5642_mode_VGA_640_480;
+	sensor->current_fr = ov5642_30_fps;
+
+	endpoint = of_graph_get_next_endpoint(dev->of_node, NULL);
+	if (!endpoint) {
+		dev_err(dev, "endpoint node not found\n");
+		return -EINVAL;
+	}
+
+	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
+	if (sensor->ep.bus_type != V4L2_MBUS_PARALLEL) {
+		dev_err(dev, "invalid bus type, must be parallel\n");
+		return -EINVAL;
+	}
+	of_node_put(endpoint);
+
+	/* get system clock (xclk) frequency */
+	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
+	if (!ret) {
+		if (xclk < OV5642_XCLK_MIN || xclk > OV5642_XCLK_MAX) {
+			dev_err(dev, "invalid xclk frequency\n");
+			return -EINVAL;
+		}
+		sensor->xclk_freq = xclk;
+	}
+
+	/* get system clock (xclk) */
+	sensor->xclk = devm_clk_get(dev, "xclk");
+	if (!IS_ERR(sensor->xclk)) {
+		if (!sensor->xclk_freq) {
+			dev_err(dev,
+				"xclk requires xclk frequency!\n");
+			return -EINVAL;
+		}
+		clk_set_rate(sensor->xclk, sensor->xclk_freq);
+	} else
+		sensor->xclk = NULL;
+
+	/* request power down pin */
+	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
+	if (IS_ERR(sensor->pwdn_gpio)) {
+		dev_err(dev, "request for power down gpio failed\n");
+		return PTR_ERR(sensor->pwdn_gpio);
+	}
+
+	/* request reset pin */
+	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_LOW);
+	if (IS_ERR(sensor->reset_gpio)) {
+		dev_err(dev, "request for reset gpio failed\n");
+		return PTR_ERR(sensor->reset_gpio);
+	}
+
+	/*
+	 * No idea what this "gp" gpio to the sensor is used for,
+	 * but optionally acquire it and set it high.
+	 */
+	sensor->gp_gpio = devm_gpiod_get_optional(dev, "gp", GPIOD_OUT_HIGH);
+	if (!IS_ERR_OR_NULL(sensor->gp_gpio))
+		gpiod_set_value(sensor->gp_gpio, 1);
+
+	/* initialize the cached controls to their defaults */
+	for (i = 0; i < OV5642_NUM_CONTROLS; i++) {
+		struct ov5642_control *c = &ov5642_ctrls[i];
+
+		sensor->ctrl_cache[i] = c->ctrl.default_value;
+	}
+	sensor->awb_on = sensor->agc_on = true;
+
+	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5642_subdev_ops);
+
+	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
+	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
+	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
+	if (ret)
+		return ret;
+
+	ov5642_get_regulators(sensor);
+
+	ret = ov5642_s_power(&sensor->sd, 1);
+	if (ret)
+		goto entity_cleanup;
+	ret = ov5642_init_controls(sensor);
+	if (ret)
+		goto power_off;
+	ret = ov5642_s_power(&sensor->sd, 0);
+	if (ret)
+		goto free_ctrls;
+
+	ret = v4l2_async_register_subdev(&sensor->sd);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+
+free_ctrls:
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+power_off:
+	ov5642_s_power(&sensor->sd, 0);
+entity_cleanup:
+	media_entity_cleanup(&sensor->sd.entity);
+	ov5642_regulators_off(sensor);
+	return ret;
+}
+
+/*!
+ * ov5642 I2C detach function
+ *
+ * @param client            struct i2c_client *
+ * @return  Error code indicating success or failure
+ */
+static int ov5642_remove(struct i2c_client *client)
+{
+	struct v4l2_subdev *sd = i2c_get_clientdata(client);
+	struct ov5642_dev *sensor = to_ov5642_dev(sd);
+
+	ov5642_regulators_off(sensor);
+
+	v4l2_async_unregister_subdev(&sensor->sd);
+	media_entity_cleanup(&sensor->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+	return 0;
+}
+
+static const struct i2c_device_id ov5642_id[] = {
+	{ "ov5642", 0 },
+	{}
+};
+MODULE_DEVICE_TABLE(i2c, ov5642_id);
+
+static const struct of_device_id ov5642_dt_ids[] = {
+	{ .compatible = "ovti,ov5642" },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, ov5642_dt_ids);
+
+static struct i2c_driver ov5642_driver = {
+	.driver = {
+		.name	= "ov5642",
+		.of_match_table	= ov5642_dt_ids,
+	},
+	.id_table	= ov5642_id,
+	.probe		= ov5642_probe,
+	.remove		= ov5642_remove,
+};
+
+module_i2c_driver(ov5642_driver);
+
+MODULE_AUTHOR("Freescale Semiconductor, Inc.");
+MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
+MODULE_DESCRIPTION("OV5642 Camera Subdev Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 24/24] ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Enable imx v4l2 staging drivers. For video capture on i.MX, the
video multiplexer subdev is required. On the SabreAuto, the ADV7180
video decoder is required along with i2c-mux-gpio. The Sabrelite
requires PWM clocks for the OV5640.

Increase max zoneorder to allow larger video buffer allocations.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/configs/imx_v6_v7_defconfig | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig
index cbe7faf..b67b99f 100644
--- a/arch/arm/configs/imx_v6_v7_defconfig
+++ b/arch/arm/configs/imx_v6_v7_defconfig
@@ -51,6 +51,7 @@ CONFIG_PREEMPT_VOLUNTARY=y
 CONFIG_AEABI=y
 CONFIG_HIGHMEM=y
 CONFIG_CMA=y
+CONFIG_FORCE_MAX_ZONEORDER=14
 CONFIG_CMDLINE="noinitrd console=ttymxc0,115200"
 CONFIG_KEXEC=y
 CONFIG_CPU_FREQ=y
@@ -174,13 +175,13 @@ CONFIG_INPUT_MISC=y
 CONFIG_INPUT_MMA8450=y
 CONFIG_SERIO_SERPORT=m
 # CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
 CONFIG_SERIAL_IMX=y
 CONFIG_SERIAL_IMX_CONSOLE=y
 CONFIG_SERIAL_FSL_LPUART=y
 CONFIG_SERIAL_FSL_LPUART_CONSOLE=y
 # CONFIG_I2C_COMPAT is not set
 CONFIG_I2C_CHARDEV=y
+CONFIG_I2C_MUX=y
 CONFIG_I2C_MUX_GPIO=y
 # CONFIG_I2C_HELPER_AUTO is not set
 CONFIG_I2C_ALGOPCF=m
@@ -194,11 +195,11 @@ CONFIG_GPIO_SYSFS=y
 CONFIG_GPIO_MC9S08DZ60=y
 CONFIG_GPIO_PCA953X=y
 CONFIG_GPIO_STMPE=y
-CONFIG_POWER_SUPPLY=y
 CONFIG_POWER_RESET=y
 CONFIG_POWER_RESET_IMX=y
 CONFIG_POWER_RESET_SYSCON=y
 CONFIG_POWER_RESET_SYSCON_POWEROFF=y
+CONFIG_POWER_SUPPLY=y
 CONFIG_SENSORS_GPIO_FAN=y
 CONFIG_SENSORS_IIO_HWMON=y
 CONFIG_THERMAL=y
@@ -221,14 +222,19 @@ CONFIG_REGULATOR_PFUZE100=y
 CONFIG_MEDIA_SUPPORT=y
 CONFIG_MEDIA_CAMERA_SUPPORT=y
 CONFIG_MEDIA_RC_SUPPORT=y
+CONFIG_MEDIA_CONTROLLER=y
+CONFIG_VIDEO_V4L2_SUBDEV_API=y
 CONFIG_RC_DEVICES=y
 CONFIG_IR_GPIO_CIR=y
 CONFIG_MEDIA_USB_SUPPORT=y
 CONFIG_USB_VIDEO_CLASS=m
 CONFIG_V4L_PLATFORM_DRIVERS=y
+CONFIG_VIDEO_MULTIPLEXER=y
 CONFIG_SOC_CAMERA=y
 CONFIG_V4L_MEM2MEM_DRIVERS=y
 CONFIG_VIDEO_CODA=y
+# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
+CONFIG_VIDEO_ADV7180=m
 CONFIG_SOC_CAMERA_OV2640=y
 CONFIG_IMX_IPUV3_CORE=y
 CONFIG_DRM=y
@@ -338,6 +344,8 @@ CONFIG_FSL_EDMA=y
 CONFIG_IMX_SDMA=y
 CONFIG_MXS_DMA=y
 CONFIG_STAGING=y
+CONFIG_STAGING_MEDIA=y
+CONFIG_COMMON_CLK_PWM=y
 CONFIG_IIO=y
 CONFIG_VF610_ADC=y
 CONFIG_MPL3115=y
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 24/24] ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r
  Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

Enable imx v4l2 staging drivers. For video capture on i.MX, the
video multiplexer subdev is required. On the SabreAuto, the ADV7180
video decoder is required along with i2c-mux-gpio. The Sabrelite
requires PWM clocks for the OV5640.

Increase max zoneorder to allow larger video buffer allocations.

Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
---
 arch/arm/configs/imx_v6_v7_defconfig | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig
index cbe7faf..b67b99f 100644
--- a/arch/arm/configs/imx_v6_v7_defconfig
+++ b/arch/arm/configs/imx_v6_v7_defconfig
@@ -51,6 +51,7 @@ CONFIG_PREEMPT_VOLUNTARY=y
 CONFIG_AEABI=y
 CONFIG_HIGHMEM=y
 CONFIG_CMA=y
+CONFIG_FORCE_MAX_ZONEORDER=14
 CONFIG_CMDLINE="noinitrd console=ttymxc0,115200"
 CONFIG_KEXEC=y
 CONFIG_CPU_FREQ=y
@@ -174,13 +175,13 @@ CONFIG_INPUT_MISC=y
 CONFIG_INPUT_MMA8450=y
 CONFIG_SERIO_SERPORT=m
 # CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
 CONFIG_SERIAL_IMX=y
 CONFIG_SERIAL_IMX_CONSOLE=y
 CONFIG_SERIAL_FSL_LPUART=y
 CONFIG_SERIAL_FSL_LPUART_CONSOLE=y
 # CONFIG_I2C_COMPAT is not set
 CONFIG_I2C_CHARDEV=y
+CONFIG_I2C_MUX=y
 CONFIG_I2C_MUX_GPIO=y
 # CONFIG_I2C_HELPER_AUTO is not set
 CONFIG_I2C_ALGOPCF=m
@@ -194,11 +195,11 @@ CONFIG_GPIO_SYSFS=y
 CONFIG_GPIO_MC9S08DZ60=y
 CONFIG_GPIO_PCA953X=y
 CONFIG_GPIO_STMPE=y
-CONFIG_POWER_SUPPLY=y
 CONFIG_POWER_RESET=y
 CONFIG_POWER_RESET_IMX=y
 CONFIG_POWER_RESET_SYSCON=y
 CONFIG_POWER_RESET_SYSCON_POWEROFF=y
+CONFIG_POWER_SUPPLY=y
 CONFIG_SENSORS_GPIO_FAN=y
 CONFIG_SENSORS_IIO_HWMON=y
 CONFIG_THERMAL=y
@@ -221,14 +222,19 @@ CONFIG_REGULATOR_PFUZE100=y
 CONFIG_MEDIA_SUPPORT=y
 CONFIG_MEDIA_CAMERA_SUPPORT=y
 CONFIG_MEDIA_RC_SUPPORT=y
+CONFIG_MEDIA_CONTROLLER=y
+CONFIG_VIDEO_V4L2_SUBDEV_API=y
 CONFIG_RC_DEVICES=y
 CONFIG_IR_GPIO_CIR=y
 CONFIG_MEDIA_USB_SUPPORT=y
 CONFIG_USB_VIDEO_CLASS=m
 CONFIG_V4L_PLATFORM_DRIVERS=y
+CONFIG_VIDEO_MULTIPLEXER=y
 CONFIG_SOC_CAMERA=y
 CONFIG_V4L_MEM2MEM_DRIVERS=y
 CONFIG_VIDEO_CODA=y
+# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
+CONFIG_VIDEO_ADV7180=m
 CONFIG_SOC_CAMERA_OV2640=y
 CONFIG_IMX_IPUV3_CORE=y
 CONFIG_DRM=y
@@ -338,6 +344,8 @@ CONFIG_FSL_EDMA=y
 CONFIG_IMX_SDMA=y
 CONFIG_MXS_DMA=y
 CONFIG_STAGING=y
+CONFIG_STAGING_MEDIA=y
+CONFIG_COMMON_CLK_PWM=y
 CONFIG_IIO=y
 CONFIG_VF610_ADC=y
 CONFIG_MPL3115=y
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 24/24] ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers
@ 2017-01-07  2:11   ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-07  2:11 UTC (permalink / raw)
  To: linux-arm-kernel

Enable imx v4l2 staging drivers. For video capture on i.MX, the
video multiplexer subdev is required. On the SabreAuto, the ADV7180
video decoder is required along with i2c-mux-gpio. The Sabrelite
requires PWM clocks for the OV5640.

Increase max zoneorder to allow larger video buffer allocations.

Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
---
 arch/arm/configs/imx_v6_v7_defconfig | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig
index cbe7faf..b67b99f 100644
--- a/arch/arm/configs/imx_v6_v7_defconfig
+++ b/arch/arm/configs/imx_v6_v7_defconfig
@@ -51,6 +51,7 @@ CONFIG_PREEMPT_VOLUNTARY=y
 CONFIG_AEABI=y
 CONFIG_HIGHMEM=y
 CONFIG_CMA=y
+CONFIG_FORCE_MAX_ZONEORDER=14
 CONFIG_CMDLINE="noinitrd console=ttymxc0,115200"
 CONFIG_KEXEC=y
 CONFIG_CPU_FREQ=y
@@ -174,13 +175,13 @@ CONFIG_INPUT_MISC=y
 CONFIG_INPUT_MMA8450=y
 CONFIG_SERIO_SERPORT=m
 # CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
 CONFIG_SERIAL_IMX=y
 CONFIG_SERIAL_IMX_CONSOLE=y
 CONFIG_SERIAL_FSL_LPUART=y
 CONFIG_SERIAL_FSL_LPUART_CONSOLE=y
 # CONFIG_I2C_COMPAT is not set
 CONFIG_I2C_CHARDEV=y
+CONFIG_I2C_MUX=y
 CONFIG_I2C_MUX_GPIO=y
 # CONFIG_I2C_HELPER_AUTO is not set
 CONFIG_I2C_ALGOPCF=m
@@ -194,11 +195,11 @@ CONFIG_GPIO_SYSFS=y
 CONFIG_GPIO_MC9S08DZ60=y
 CONFIG_GPIO_PCA953X=y
 CONFIG_GPIO_STMPE=y
-CONFIG_POWER_SUPPLY=y
 CONFIG_POWER_RESET=y
 CONFIG_POWER_RESET_IMX=y
 CONFIG_POWER_RESET_SYSCON=y
 CONFIG_POWER_RESET_SYSCON_POWEROFF=y
+CONFIG_POWER_SUPPLY=y
 CONFIG_SENSORS_GPIO_FAN=y
 CONFIG_SENSORS_IIO_HWMON=y
 CONFIG_THERMAL=y
@@ -221,14 +222,19 @@ CONFIG_REGULATOR_PFUZE100=y
 CONFIG_MEDIA_SUPPORT=y
 CONFIG_MEDIA_CAMERA_SUPPORT=y
 CONFIG_MEDIA_RC_SUPPORT=y
+CONFIG_MEDIA_CONTROLLER=y
+CONFIG_VIDEO_V4L2_SUBDEV_API=y
 CONFIG_RC_DEVICES=y
 CONFIG_IR_GPIO_CIR=y
 CONFIG_MEDIA_USB_SUPPORT=y
 CONFIG_USB_VIDEO_CLASS=m
 CONFIG_V4L_PLATFORM_DRIVERS=y
+CONFIG_VIDEO_MULTIPLEXER=y
 CONFIG_SOC_CAMERA=y
 CONFIG_V4L_MEM2MEM_DRIVERS=y
 CONFIG_VIDEO_CODA=y
+# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
+CONFIG_VIDEO_ADV7180=m
 CONFIG_SOC_CAMERA_OV2640=y
 CONFIG_IMX_IPUV3_CORE=y
 CONFIG_DRM=y
@@ -338,6 +344,8 @@ CONFIG_FSL_EDMA=y
 CONFIG_IMX_SDMA=y
 CONFIG_MXS_DMA=y
 CONFIG_STAGING=y
+CONFIG_STAGING_MEDIA=y
+CONFIG_COMMON_CLK_PWM=y
 CONFIG_IIO=y
 CONFIG_VF610_ADC=y
 CONFIG_MPL3115=y
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-10  5:35     ` Rob Herring
  -1 siblings, 0 replies; 549+ messages in thread
From: Rob Herring @ 2017-01-10  5:35 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, shawnguo, kernel, fabio.estevam, linux, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"

This should have an SoC/chip specific compatible string additionally. 
Just need a note to that effect here, but i.MX will need a compat 
string.

> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-10  5:35     ` Rob Herring
  0 siblings, 0 replies; 549+ messages in thread
From: Rob Herring @ 2017-01-10  5:35 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, Sascha Hauer,
	linux-media, devicetree, kernel, arnd, mchehab, bparrot,
	Steve Longerbeam, horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"

This should have an SoC/chip specific compatible string additionally. 
Just need a note to that effect here, but i.MX will need a compat 
string.

> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-10  5:35     ` Rob Herring
  0 siblings, 0 replies; 549+ messages in thread
From: Rob Herring @ 2017-01-10  5:35 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"

This should have an SoC/chip specific compatible string additionally. 
Just need a note to that effect here, but i.MX will need a compat 
string.

> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-11 23:14   ` Tim Harvey
  -1 siblings, 0 replies; 549+ messages in thread
From: Tim Harvey @ 2017-01-11 23:14 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King - ARM Linux, mchehab, Hans Verkuil,
	nick, markus.heiser, Philipp Zabel, laurent.pinchart+renesas,
	bparrot, geert, Arnd Bergmann, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, Simon Horman,
	Niklas Söderlund, robert.jarzmik, songjun.wu,
	andrew-ct.chen, Greg Kroah-Hartman, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> In version 3:
>
> Changes suggested by Rob Herring <robh@kernel.org>:
>
>   - prepended FIM node properties with vendor prefix "fsl,".
>
>   - make mipi csi-2 receiver compatible string SoC specific:
>     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>
>   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>
>   - removed board-specific info from the media driver binding doc. These
>     were all related to sensor bindings, which already are (adv7180)
>     or will be (ov564x) covered in separate binding docs. All reference
>     board info not related to DT bindings has been moved to
>     Documentation/media/v4l-drivers/imx.rst.
>
>   - removed "_mipi" from the OV5640 compatible string.
>
> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>
>   Mostly cosmetic/non-functional changes which I won't list here, except
>   for the following:
>
>   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>
>   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>
>   - check/handle return code from required reg property of CSI port nodes.
>
>   - check/handle return code from clk_prepare_enable().
>
> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>
>   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>
>   - finally got around to passing valid IOMUX pin config values to the
>     pin groups.
>
> Other changes:
>
>   - removed the FIM properties that overrided the v4l2 FIM control defaults
>     values. This was left-over from a requirement of a customer and is not
>     necessary here.
>
>   - The FIM must be explicitly enabled in the fim child node under the CSI
>     port nodes, using the status property. If not enabled, FIM v4l2 controls
>     will not appear in the video capture driver.
>
>   - brought in additional media types patch from Philipp Zabel. Use new
>     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>
>   - brought in latest platform generic video multiplexer subdevice driver
>     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>
>   - removed imx-media-of.h, moved those prototypes into imx-media.h.
>
>
<snip>

Hi Steve,

I took a stab at testing this today on a gw51xx which has an adv7180
hooked up as follows:
- i2c3@0x20
- 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
(CSI0_IPU1)
- PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
- IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
- all three analog inputs available to off-board connector

My patch to the imx6qdl-gw51xx dtsi is:
diff --git a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
index afec2c7..2583d72 100644
--- a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
@@ -165,6 +174,52 @@
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_i2c3>;
        status = "okay";
+
+       camera: adv7180@20 {
+               compatible = "adi,adv7180";
+               pinctrl-names = "default";
+               pinctrl-0 = <&pinctrl_adv7180>;
+               reg = <0x20>;
+               powerdown-gpio = <&gpio5 20 GPIO_ACTIVE_LOW>;
+               interrupt-parent = <&gpio5>;
+               interrupts = <23 GPIO_ACTIVE_LOW>;
+               inputs = <0x00 0x01 0x02>;
+               input-names = "ADV7180 Composite on Ain1",
+                             "ADV7180 Composite on Ain2",
+                             "ADV7180 Composite on Ain3";
+
+               port {
+                       adv7180_to_ipu1_csi0_mux: endpoint {
+                               remote-endpoint =
<&ipu1_csi0_mux_from_parallel_sensor>;
+                               bus-width = <8>;
+                       };
+               };
+       };
+};
+
+&ipu1_csi0_from_ipu1_csi0_mux {
+       bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+       remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+       bus-width = <8>;
+};
+
+&ipu1_csi0 {
+       pinctrl-names = "default";
+       pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+       /* enable frame interval monitor on this port */
+       fim {
+               status = "okay";
+       };
 };

 &pcie {
@@ -236,6 +291,13 @@

 &iomuxc {
        imx6qdl-gw51xx {
+               pinctrl_adv7180: adv7180grp {
+                       fsl,pins = <
+                               MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
 0x0001b0b0 /* VIDDEC_IRQ# */
+                               MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
 0x4001b0b0 /* VIDDEC_EN */
+                       >;
+               };
+
                pinctrl_enet: enetgrp {
                        fsl,pins = <
                                MX6QDL_PAD_RGMII_RXC__RGMII_RXC         0x1b030
@@ -306,6 +368,22 @@
                        >;
                };

+               pinctrl_ipu1_csi0: ipu1csi0grp { /* IPU1_CSI0: 8-bit input */
+                       fsl,pins = <
+
MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+                               MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC
    0x1b0b0
+                               MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC
    0x1b0b0
+
MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+                       >;
+               };
+
                pinctrl_pcie: pciegrp {
                        fsl,pins = <
                                MX6QDL_PAD_GPIO_0__GPIO1_IO00           0x1b0b0

On an IMX6Q I'm getting the following when the adv7180 module loads:
[   12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
[   12.907767] imx-media: Registered subdev adv7180 2-0020
[   12.907793] imx-media soc:media@0: Entity type for entity adv7180
2-0020 was not initialized!
[   12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
ipu1_csi0_mux:1

Is the warning that adv7180 was not initialized expected and or an issue?

Now that your driver is hooking into the current media framework, I'm
not at all clear on how to link and configure the media entities.
Using 'media-ctl -p' to list the entities I see:
Media controller API version 0.1.0

Media device information
------------------------
driver          imx-media
model           imx-media
serial
bus info
hw revision     0x0
driver version  0.0.0

Device topology
- entity 1: ipu2_csi1_mux (3 pads, 1 link)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:unknown/0x0]
        pad1: Sink
                [fmt:unknown/0x0]
        pad2: Source
                [fmt:unknown/0x0]
                -> "ipu2_csi1":0 []

- entity 5: ipu1_csi0_mux (3 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:unknown/0x0]
        pad1: Sink
                [fmt:unknown/0x0]
                <- "adv7180 2-0020":0 []
        pad2: Source
                [fmt:unknown/0x0]
                -> "ipu1_csi0":0 []

- entity 9: ipu1_ic_prpenc (2 pads, 4 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev2
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
                <- "ipu1_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 12: ipu1_ic_prpvf (2 pads, 8 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev3
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
                <- "ipu1_csi1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []
                -> "ipu1_ic_pp0":0 []
                -> "ipu1_ic_pp1":0 []

- entity 15: ipu1_ic_pp0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev4
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_smfc0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 18: ipu1_ic_pp1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev5
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 21: ipu2_ic_prpenc (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev6
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
                <- "ipu2_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 24: ipu2_ic_prpvf (2 pads, 8 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev7
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
                <- "ipu2_csi1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []
                -> "ipu2_ic_pp0":0 []
                -> "ipu2_ic_pp1":0 []

- entity 27: ipu2_ic_pp0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev8
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_smfc0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 30: ipu2_ic_pp1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev9
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 33: ipu1_csi0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev10
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0_mux":2 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu1_ic_prpenc":0 []
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_smfc0":0 []

- entity 36: ipu1_csi1 (2 pads, 3 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev11
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu1_ic_prpenc":0 []
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_smfc1":0 []

- entity 39: ipu2_csi0 (2 pads, 3 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev12
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu2_ic_prpenc":0 []
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_smfc0":0 []

- entity 42: ipu2_csi1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev13
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi1_mux":2 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu2_ic_prpenc":0 []
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_smfc1":0 []

- entity 45: ipu1_smfc0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev14
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_ic_pp0":0 []
                -> "camif0":0 []
                -> "camif1":0 []

- entity 48: ipu1_smfc1 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev15
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_ic_pp1":0 []
                -> "camif0":0 []
                -> "camif1":0 []

- entity 51: ipu2_smfc0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev16
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_ic_pp0":0 []
                -> "camif2":0 []
                -> "camif3":0 []

- entity 54: ipu2_smfc1 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev17
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_ic_pp1":0 []
                -> "camif2":0 []
                -> "camif3":0 []

- entity 57: camif0 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev18
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpenc":1 []
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_ic_pp0":1 []
                <- "ipu1_ic_pp1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0 devnode":0 [ENABLED,IMMUTABLE]

- entity 58: camif0 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
        pad0: Sink
                <- "camif0":1 [ENABLED,IMMUTABLE]

- entity 66: camif1 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev19
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpenc":1 []
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_ic_pp0":1 []
                <- "ipu1_ic_pp1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif1 devnode":0 [ENABLED,IMMUTABLE]

- entity 67: camif1 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "camif1":1 [ENABLED,IMMUTABLE]

- entity 75: camif2 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev20
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpenc":1 []
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_ic_pp0":1 []
                <- "ipu2_ic_pp1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2 devnode":0 [ENABLED,IMMUTABLE]

- entity 76: camif2 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "camif2":1 [ENABLED,IMMUTABLE]

- entity 84: camif3 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev21
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpenc":1 []
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_ic_pp0":1 []
                <- "ipu2_ic_pp1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif3 devnode":0 [ENABLED,IMMUTABLE]

- entity 85: camif3 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Sink
                <- "camif3":1 [ENABLED,IMMUTABLE]

- entity 93: adv7180 2-0020 (1 pad, 1 link)
             type V4L2 subdev subtype Unknown flags 20004
             device node name /dev/v4l-subdev22
        pad0: Source
                [fmt:UYVY2X8/720x480 field:interlaced]
                -> "ipu1_csi0_mux":1 []

How do I link the entities here to be able to capture frames with
v4l2-ctl or gstreamer?

Additionally I've found that on an IMX6S/IMX6DL we crash while
registering the media-ic subdev's:
[    3.975473] imx-media: Registered subdev ipu1_csi1_mux
[    3.980921] imx-media: Registered subdev ipu1_csi0_mux
[    4.003205] imx-media: Registered subdev ipu1_ic_prpenc
[    4.025373] imx-media: Registered subdev ipu1_ic_prpvf
[    4.037944] ------------[ cut here ]------------
[    4.042571] Kernel BUG at c06717dc [verbose debug info unavailable]
[    4.048845] Internal error: Oops - BUG: 0 [#1] SMP ARM
[    4.053990] Modules linked in:
[    4.057076] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
4.9.0-rc6-00524-g84dad6e-dirty #446
[    4.065260] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
...
[    4.296250] [<c0671780>] (v4l2_subdev_init) from [<c06fb02c>]
(imx_ic_probe+0x94/0x1ac)
[    4.304271] [<c06faf98>] (imx_ic_probe) from [<c05173d8>]
(platform_drv_probe+0x54/0xb8)
[    4.312373]  r9:c0d5e858 r8:00000000 r7:fffffdfb r6:c0e5dbf8
r5:da603810 r4:c16738d8
[    4.320129] [<c0517384>] (platform_drv_probe) from [<c0515978>]
(driver_probe_device+0x20c/0x2c0)
[    4.329010]  r7:c0e5dbf8 r6:00000000 r5:da603810 r4:c16738d8
[    4.334681] [<c051576c>] (driver_probe_device) from [<c0515af4>]
(__driver_attach+0xc8/0xcc)
[    4.343129]  r9:c0d5e858 r8:00000000 r7:00000000 r6:da603844
r5:c0e5dbf8 r4:da603810
[    4.350889] [<c0515a2c>] (__driver_attach) from [<c0513adc>]
(bus_for_each_dev+0x74/0xa8)
[    4.359078]  r7:00000000 r6:c0515a2c r5:c0e5dbf8 r4:00000000
[    4.364753] [<c0513a68>] (bus_for_each_dev) from [<c05151d4>]
(driver_attach+0x20/0x28)

I assume there is an iteration that needs a test on a missing pointer
only available on chips with both IPU's or PRP

Regards,

Tim

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-11 23:14   ` Tim Harvey
  0 siblings, 0 replies; 549+ messages in thread
From: Tim Harvey @ 2017-01-11 23:14 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King - ARM Linux, mchehab, Hans Verkuil,
	nick, markus.heiser, Philipp Zabel, laurent.pinchart+renesas,
	bparrot, geert, Arnd Bergmann, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, Simon Horman,
	Niklas Söderlund, robert.jarzmik, songjun.wu

On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> In version 3:
>
> Changes suggested by Rob Herring <robh@kernel.org>:
>
>   - prepended FIM node properties with vendor prefix "fsl,".
>
>   - make mipi csi-2 receiver compatible string SoC specific:
>     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>
>   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>
>   - removed board-specific info from the media driver binding doc. These
>     were all related to sensor bindings, which already are (adv7180)
>     or will be (ov564x) covered in separate binding docs. All reference
>     board info not related to DT bindings has been moved to
>     Documentation/media/v4l-drivers/imx.rst.
>
>   - removed "_mipi" from the OV5640 compatible string.
>
> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>
>   Mostly cosmetic/non-functional changes which I won't list here, except
>   for the following:
>
>   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>
>   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>
>   - check/handle return code from required reg property of CSI port nodes.
>
>   - check/handle return code from clk_prepare_enable().
>
> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>
>   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>
>   - finally got around to passing valid IOMUX pin config values to the
>     pin groups.
>
> Other changes:
>
>   - removed the FIM properties that overrided the v4l2 FIM control defaults
>     values. This was left-over from a requirement of a customer and is not
>     necessary here.
>
>   - The FIM must be explicitly enabled in the fim child node under the CSI
>     port nodes, using the status property. If not enabled, FIM v4l2 controls
>     will not appear in the video capture driver.
>
>   - brought in additional media types patch from Philipp Zabel. Use new
>     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>
>   - brought in latest platform generic video multiplexer subdevice driver
>     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>
>   - removed imx-media-of.h, moved those prototypes into imx-media.h.
>
>
<snip>

Hi Steve,

I took a stab at testing this today on a gw51xx which has an adv7180
hooked up as follows:
- i2c3@0x20
- 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
(CSI0_IPU1)
- PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
- IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
- all three analog inputs available to off-board connector

My patch to the imx6qdl-gw51xx dtsi is:
diff --git a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
index afec2c7..2583d72 100644
--- a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
@@ -165,6 +174,52 @@
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_i2c3>;
        status = "okay";
+
+       camera: adv7180@20 {
+               compatible = "adi,adv7180";
+               pinctrl-names = "default";
+               pinctrl-0 = <&pinctrl_adv7180>;
+               reg = <0x20>;
+               powerdown-gpio = <&gpio5 20 GPIO_ACTIVE_LOW>;
+               interrupt-parent = <&gpio5>;
+               interrupts = <23 GPIO_ACTIVE_LOW>;
+               inputs = <0x00 0x01 0x02>;
+               input-names = "ADV7180 Composite on Ain1",
+                             "ADV7180 Composite on Ain2",
+                             "ADV7180 Composite on Ain3";
+
+               port {
+                       adv7180_to_ipu1_csi0_mux: endpoint {
+                               remote-endpoint =
<&ipu1_csi0_mux_from_parallel_sensor>;
+                               bus-width = <8>;
+                       };
+               };
+       };
+};
+
+&ipu1_csi0_from_ipu1_csi0_mux {
+       bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+       remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+       bus-width = <8>;
+};
+
+&ipu1_csi0 {
+       pinctrl-names = "default";
+       pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+       /* enable frame interval monitor on this port */
+       fim {
+               status = "okay";
+       };
 };

 &pcie {
@@ -236,6 +291,13 @@

 &iomuxc {
        imx6qdl-gw51xx {
+               pinctrl_adv7180: adv7180grp {
+                       fsl,pins = <
+                               MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
 0x0001b0b0 /* VIDDEC_IRQ# */
+                               MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
 0x4001b0b0 /* VIDDEC_EN */
+                       >;
+               };
+
                pinctrl_enet: enetgrp {
                        fsl,pins = <
                                MX6QDL_PAD_RGMII_RXC__RGMII_RXC         0x1b030
@@ -306,6 +368,22 @@
                        >;
                };

+               pinctrl_ipu1_csi0: ipu1csi0grp { /* IPU1_CSI0: 8-bit input */
+                       fsl,pins = <
+
MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+                               MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC
    0x1b0b0
+                               MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC
    0x1b0b0
+
MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+                       >;
+               };
+
                pinctrl_pcie: pciegrp {
                        fsl,pins = <
                                MX6QDL_PAD_GPIO_0__GPIO1_IO00           0x1b0b0

On an IMX6Q I'm getting the following when the adv7180 module loads:
[   12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
[   12.907767] imx-media: Registered subdev adv7180 2-0020
[   12.907793] imx-media soc:media@0: Entity type for entity adv7180
2-0020 was not initialized!
[   12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
ipu1_csi0_mux:1

Is the warning that adv7180 was not initialized expected and or an issue?

Now that your driver is hooking into the current media framework, I'm
not at all clear on how to link and configure the media entities.
Using 'media-ctl -p' to list the entities I see:
Media controller API version 0.1.0

Media device information
------------------------
driver          imx-media
model           imx-media
serial
bus info
hw revision     0x0
driver version  0.0.0

Device topology
- entity 1: ipu2_csi1_mux (3 pads, 1 link)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:unknown/0x0]
        pad1: Sink
                [fmt:unknown/0x0]
        pad2: Source
                [fmt:unknown/0x0]
                -> "ipu2_csi1":0 []

- entity 5: ipu1_csi0_mux (3 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:unknown/0x0]
        pad1: Sink
                [fmt:unknown/0x0]
                <- "adv7180 2-0020":0 []
        pad2: Source
                [fmt:unknown/0x0]
                -> "ipu1_csi0":0 []

- entity 9: ipu1_ic_prpenc (2 pads, 4 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev2
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
                <- "ipu1_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 12: ipu1_ic_prpvf (2 pads, 8 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev3
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
                <- "ipu1_csi1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []
                -> "ipu1_ic_pp0":0 []
                -> "ipu1_ic_pp1":0 []

- entity 15: ipu1_ic_pp0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev4
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_smfc0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 18: ipu1_ic_pp1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev5
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 21: ipu2_ic_prpenc (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev6
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
                <- "ipu2_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 24: ipu2_ic_prpvf (2 pads, 8 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev7
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
                <- "ipu2_csi1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []
                -> "ipu2_ic_pp0":0 []
                -> "ipu2_ic_pp1":0 []

- entity 27: ipu2_ic_pp0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev8
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_smfc0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 30: ipu2_ic_pp1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev9
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 33: ipu1_csi0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev10
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0_mux":2 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu1_ic_prpenc":0 []
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_smfc0":0 []

- entity 36: ipu1_csi1 (2 pads, 3 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev11
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu1_ic_prpenc":0 []
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_smfc1":0 []

- entity 39: ipu2_csi0 (2 pads, 3 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev12
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu2_ic_prpenc":0 []
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_smfc0":0 []

- entity 42: ipu2_csi1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev13
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi1_mux":2 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu2_ic_prpenc":0 []
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_smfc1":0 []

- entity 45: ipu1_smfc0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev14
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_ic_pp0":0 []
                -> "camif0":0 []
                -> "camif1":0 []

- entity 48: ipu1_smfc1 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev15
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_ic_pp1":0 []
                -> "camif0":0 []
                -> "camif1":0 []

- entity 51: ipu2_smfc0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev16
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_ic_pp0":0 []
                -> "camif2":0 []
                -> "camif3":0 []

- entity 54: ipu2_smfc1 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev17
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_ic_pp1":0 []
                -> "camif2":0 []
                -> "camif3":0 []

- entity 57: camif0 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev18
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpenc":1 []
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_ic_pp0":1 []
                <- "ipu1_ic_pp1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0 devnode":0 [ENABLED,IMMUTABLE]

- entity 58: camif0 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
        pad0: Sink
                <- "camif0":1 [ENABLED,IMMUTABLE]

- entity 66: camif1 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev19
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpenc":1 []
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_ic_pp0":1 []
                <- "ipu1_ic_pp1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif1 devnode":0 [ENABLED,IMMUTABLE]

- entity 67: camif1 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "camif1":1 [ENABLED,IMMUTABLE]

- entity 75: camif2 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev20
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpenc":1 []
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_ic_pp0":1 []
                <- "ipu2_ic_pp1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2 devnode":0 [ENABLED,IMMUTABLE]

- entity 76: camif2 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "camif2":1 [ENABLED,IMMUTABLE]

- entity 84: camif3 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev21
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpenc":1 []
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_ic_pp0":1 []
                <- "ipu2_ic_pp1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif3 devnode":0 [ENABLED,IMMUTABLE]

- entity 85: camif3 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Sink
                <- "camif3":1 [ENABLED,IMMUTABLE]

- entity 93: adv7180 2-0020 (1 pad, 1 link)
             type V4L2 subdev subtype Unknown flags 20004
             device node name /dev/v4l-subdev22
        pad0: Source
                [fmt:UYVY2X8/720x480 field:interlaced]
                -> "ipu1_csi0_mux":1 []

How do I link the entities here to be able to capture frames with
v4l2-ctl or gstreamer?

Additionally I've found that on an IMX6S/IMX6DL we crash while
registering the media-ic subdev's:
[    3.975473] imx-media: Registered subdev ipu1_csi1_mux
[    3.980921] imx-media: Registered subdev ipu1_csi0_mux
[    4.003205] imx-media: Registered subdev ipu1_ic_prpenc
[    4.025373] imx-media: Registered subdev ipu1_ic_prpvf
[    4.037944] ------------[ cut here ]------------
[    4.042571] Kernel BUG at c06717dc [verbose debug info unavailable]
[    4.048845] Internal error: Oops - BUG: 0 [#1] SMP ARM
[    4.053990] Modules linked in:
[    4.057076] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
4.9.0-rc6-00524-g84dad6e-dirty #446
[    4.065260] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
...
[    4.296250] [<c0671780>] (v4l2_subdev_init) from [<c06fb02c>]
(imx_ic_probe+0x94/0x1ac)
[    4.304271] [<c06faf98>] (imx_ic_probe) from [<c05173d8>]
(platform_drv_probe+0x54/0xb8)
[    4.312373]  r9:c0d5e858 r8:00000000 r7:fffffdfb r6:c0e5dbf8
r5:da603810 r4:c16738d8
[    4.320129] [<c0517384>] (platform_drv_probe) from [<c0515978>]
(driver_probe_device+0x20c/0x2c0)
[    4.329010]  r7:c0e5dbf8 r6:00000000 r5:da603810 r4:c16738d8
[    4.334681] [<c051576c>] (driver_probe_device) from [<c0515af4>]
(__driver_attach+0xc8/0xcc)
[    4.343129]  r9:c0d5e858 r8:00000000 r7:00000000 r6:da603844
r5:c0e5dbf8 r4:da603810
[    4.350889] [<c0515a2c>] (__driver_attach) from [<c0513adc>]
(bus_for_each_dev+0x74/0xa8)
[    4.359078]  r7:00000000 r6:c0515a2c r5:c0e5dbf8 r4:00000000
[    4.364753] [<c0513a68>] (bus_for_each_dev) from [<c05151d4>]
(driver_attach+0x20/0x28)

I assume there is an iteration that needs a test on a missing pointer
only available on chips with both IPU's or PRP

Regards,

Tim

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-11 23:14   ` Tim Harvey
  0 siblings, 0 replies; 549+ messages in thread
From: Tim Harvey @ 2017-01-11 23:14 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> In version 3:
>
> Changes suggested by Rob Herring <robh@kernel.org>:
>
>   - prepended FIM node properties with vendor prefix "fsl,".
>
>   - make mipi csi-2 receiver compatible string SoC specific:
>     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>
>   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>
>   - removed board-specific info from the media driver binding doc. These
>     were all related to sensor bindings, which already are (adv7180)
>     or will be (ov564x) covered in separate binding docs. All reference
>     board info not related to DT bindings has been moved to
>     Documentation/media/v4l-drivers/imx.rst.
>
>   - removed "_mipi" from the OV5640 compatible string.
>
> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>
>   Mostly cosmetic/non-functional changes which I won't list here, except
>   for the following:
>
>   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>
>   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>
>   - check/handle return code from required reg property of CSI port nodes.
>
>   - check/handle return code from clk_prepare_enable().
>
> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>
>   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>
>   - finally got around to passing valid IOMUX pin config values to the
>     pin groups.
>
> Other changes:
>
>   - removed the FIM properties that overrided the v4l2 FIM control defaults
>     values. This was left-over from a requirement of a customer and is not
>     necessary here.
>
>   - The FIM must be explicitly enabled in the fim child node under the CSI
>     port nodes, using the status property. If not enabled, FIM v4l2 controls
>     will not appear in the video capture driver.
>
>   - brought in additional media types patch from Philipp Zabel. Use new
>     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>
>   - brought in latest platform generic video multiplexer subdevice driver
>     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>
>   - removed imx-media-of.h, moved those prototypes into imx-media.h.
>
>
<snip>

Hi Steve,

I took a stab at testing this today on a gw51xx which has an adv7180
hooked up as follows:
- i2c3 at 0x20
- 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
(CSI0_IPU1)
- PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
- IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
- all three analog inputs available to off-board connector

My patch to the imx6qdl-gw51xx dtsi is:
diff --git a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
index afec2c7..2583d72 100644
--- a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
@@ -165,6 +174,52 @@
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_i2c3>;
        status = "okay";
+
+       camera: adv7180 at 20 {
+               compatible = "adi,adv7180";
+               pinctrl-names = "default";
+               pinctrl-0 = <&pinctrl_adv7180>;
+               reg = <0x20>;
+               powerdown-gpio = <&gpio5 20 GPIO_ACTIVE_LOW>;
+               interrupt-parent = <&gpio5>;
+               interrupts = <23 GPIO_ACTIVE_LOW>;
+               inputs = <0x00 0x01 0x02>;
+               input-names = "ADV7180 Composite on Ain1",
+                             "ADV7180 Composite on Ain2",
+                             "ADV7180 Composite on Ain3";
+
+               port {
+                       adv7180_to_ipu1_csi0_mux: endpoint {
+                               remote-endpoint =
<&ipu1_csi0_mux_from_parallel_sensor>;
+                               bus-width = <8>;
+                       };
+               };
+       };
+};
+
+&ipu1_csi0_from_ipu1_csi0_mux {
+       bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+       remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+       bus-width = <8>;
+};
+
+&ipu1_csi0 {
+       pinctrl-names = "default";
+       pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+       /* enable frame interval monitor on this port */
+       fim {
+               status = "okay";
+       };
 };

 &pcie {
@@ -236,6 +291,13 @@

 &iomuxc {
        imx6qdl-gw51xx {
+               pinctrl_adv7180: adv7180grp {
+                       fsl,pins = <
+                               MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
 0x0001b0b0 /* VIDDEC_IRQ# */
+                               MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
 0x4001b0b0 /* VIDDEC_EN */
+                       >;
+               };
+
                pinctrl_enet: enetgrp {
                        fsl,pins = <
                                MX6QDL_PAD_RGMII_RXC__RGMII_RXC         0x1b030
@@ -306,6 +368,22 @@
                        >;
                };

+               pinctrl_ipu1_csi0: ipu1csi0grp { /* IPU1_CSI0: 8-bit input */
+                       fsl,pins = <
+
MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+
MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+                               MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC
    0x1b0b0
+                               MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC
    0x1b0b0
+
MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+                       >;
+               };
+
                pinctrl_pcie: pciegrp {
                        fsl,pins = <
                                MX6QDL_PAD_GPIO_0__GPIO1_IO00           0x1b0b0

On an IMX6Q I'm getting the following when the adv7180 module loads:
[   12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
[   12.907767] imx-media: Registered subdev adv7180 2-0020
[   12.907793] imx-media soc:media@0: Entity type for entity adv7180
2-0020 was not initialized!
[   12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
ipu1_csi0_mux:1

Is the warning that adv7180 was not initialized expected and or an issue?

Now that your driver is hooking into the current media framework, I'm
not at all clear on how to link and configure the media entities.
Using 'media-ctl -p' to list the entities I see:
Media controller API version 0.1.0

Media device information
------------------------
driver          imx-media
model           imx-media
serial
bus info
hw revision     0x0
driver version  0.0.0

Device topology
- entity 1: ipu2_csi1_mux (3 pads, 1 link)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:unknown/0x0]
        pad1: Sink
                [fmt:unknown/0x0]
        pad2: Source
                [fmt:unknown/0x0]
                -> "ipu2_csi1":0 []

- entity 5: ipu1_csi0_mux (3 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:unknown/0x0]
        pad1: Sink
                [fmt:unknown/0x0]
                <- "adv7180 2-0020":0 []
        pad2: Source
                [fmt:unknown/0x0]
                -> "ipu1_csi0":0 []

- entity 9: ipu1_ic_prpenc (2 pads, 4 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev2
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
                <- "ipu1_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 12: ipu1_ic_prpvf (2 pads, 8 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev3
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
                <- "ipu1_csi1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []
                -> "ipu1_ic_pp0":0 []
                -> "ipu1_ic_pp1":0 []

- entity 15: ipu1_ic_pp0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev4
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_smfc0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 18: ipu1_ic_pp1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev5
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0":0 []
                -> "camif1":0 []

- entity 21: ipu2_ic_prpenc (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev6
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
                <- "ipu2_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 24: ipu2_ic_prpvf (2 pads, 8 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev7
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
                <- "ipu2_csi1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []
                -> "ipu2_ic_pp0":0 []
                -> "ipu2_ic_pp1":0 []

- entity 27: ipu2_ic_pp0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev8
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_smfc0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 30: ipu2_ic_pp1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev9
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2":0 []
                -> "camif3":0 []

- entity 33: ipu1_csi0 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev10
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0_mux":2 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu1_ic_prpenc":0 []
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_smfc0":0 []

- entity 36: ipu1_csi1 (2 pads, 3 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev11
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu1_ic_prpenc":0 []
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_smfc1":0 []

- entity 39: ipu2_csi0 (2 pads, 3 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev12
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu2_ic_prpenc":0 []
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_smfc0":0 []

- entity 42: ipu2_csi1 (2 pads, 4 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev13
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi1_mux":2 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/0x0]
                -> "ipu2_ic_prpenc":0 []
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_smfc1":0 []

- entity 45: ipu1_smfc0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev14
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_ic_pp0":0 []
                -> "camif0":0 []
                -> "camif1":0 []

- entity 48: ipu1_smfc1 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev15
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu1_ic_prpvf":0 []
                -> "ipu1_ic_pp1":0 []
                -> "camif0":0 []
                -> "camif1":0 []

- entity 51: ipu2_smfc0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev16
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi0":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_ic_pp0":0 []
                -> "camif2":0 []
                -> "camif3":0 []

- entity 54: ipu2_smfc1 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev17
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_csi1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "ipu2_ic_prpvf":0 []
                -> "ipu2_ic_pp1":0 []
                -> "camif2":0 []
                -> "camif3":0 []

- entity 57: camif0 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev18
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpenc":1 []
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_ic_pp0":1 []
                <- "ipu1_ic_pp1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif0 devnode":0 [ENABLED,IMMUTABLE]

- entity 58: camif0 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
        pad0: Sink
                <- "camif0":1 [ENABLED,IMMUTABLE]

- entity 66: camif1 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev19
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu1_ic_prpenc":1 []
                <- "ipu1_ic_prpvf":1 []
                <- "ipu1_ic_pp0":1 []
                <- "ipu1_ic_pp1":1 []
                <- "ipu1_smfc0":1 []
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif1 devnode":0 [ENABLED,IMMUTABLE]

- entity 67: camif1 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "camif1":1 [ENABLED,IMMUTABLE]

- entity 75: camif2 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev20
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpenc":1 []
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_ic_pp0":1 []
                <- "ipu2_ic_pp1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif2 devnode":0 [ENABLED,IMMUTABLE]

- entity 76: camif2 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "camif2":1 [ENABLED,IMMUTABLE]

- entity 84: camif3 (2 pads, 7 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev21
        pad0: Sink
                [fmt:UYVY2X8/640x480 field:none]
                <- "ipu2_ic_prpenc":1 []
                <- "ipu2_ic_prpvf":1 []
                <- "ipu2_ic_pp0":1 []
                <- "ipu2_ic_pp1":1 []
                <- "ipu2_smfc0":1 []
                <- "ipu2_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/640x480 field:none]
                -> "camif3 devnode":0 [ENABLED,IMMUTABLE]

- entity 85: camif3 devnode (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Sink
                <- "camif3":1 [ENABLED,IMMUTABLE]

- entity 93: adv7180 2-0020 (1 pad, 1 link)
             type V4L2 subdev subtype Unknown flags 20004
             device node name /dev/v4l-subdev22
        pad0: Source
                [fmt:UYVY2X8/720x480 field:interlaced]
                -> "ipu1_csi0_mux":1 []

How do I link the entities here to be able to capture frames with
v4l2-ctl or gstreamer?

Additionally I've found that on an IMX6S/IMX6DL we crash while
registering the media-ic subdev's:
[    3.975473] imx-media: Registered subdev ipu1_csi1_mux
[    3.980921] imx-media: Registered subdev ipu1_csi0_mux
[    4.003205] imx-media: Registered subdev ipu1_ic_prpenc
[    4.025373] imx-media: Registered subdev ipu1_ic_prpvf
[    4.037944] ------------[ cut here ]------------
[    4.042571] Kernel BUG at c06717dc [verbose debug info unavailable]
[    4.048845] Internal error: Oops - BUG: 0 [#1] SMP ARM
[    4.053990] Modules linked in:
[    4.057076] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
4.9.0-rc6-00524-g84dad6e-dirty #446
[    4.065260] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
...
[    4.296250] [<c0671780>] (v4l2_subdev_init) from [<c06fb02c>]
(imx_ic_probe+0x94/0x1ac)
[    4.304271] [<c06faf98>] (imx_ic_probe) from [<c05173d8>]
(platform_drv_probe+0x54/0xb8)
[    4.312373]  r9:c0d5e858 r8:00000000 r7:fffffdfb r6:c0e5dbf8
r5:da603810 r4:c16738d8
[    4.320129] [<c0517384>] (platform_drv_probe) from [<c0515978>]
(driver_probe_device+0x20c/0x2c0)
[    4.329010]  r7:c0e5dbf8 r6:00000000 r5:da603810 r4:c16738d8
[    4.334681] [<c051576c>] (driver_probe_device) from [<c0515af4>]
(__driver_attach+0xc8/0xcc)
[    4.343129]  r9:c0d5e858 r8:00000000 r7:00000000 r6:da603844
r5:c0e5dbf8 r4:da603810
[    4.350889] [<c0515a2c>] (__driver_attach) from [<c0513adc>]
(bus_for_each_dev+0x74/0xa8)
[    4.359078]  r7:00000000 r6:c0515a2c r5:c0e5dbf8 r4:00000000
[    4.364753] [<c0513a68>] (bus_for_each_dev) from [<c05151d4>]
(driver_attach+0x20/0x28)

I assume there is an iteration that needs a test on a missing pointer
only available on chips with both IPU's or PRP

Regards,

Tim

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-11 23:14   ` Tim Harvey
  (?)
@ 2017-01-12  3:22     ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-12  3:22 UTC (permalink / raw)
  To: Tim Harvey, Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King - ARM Linux, mchehab, Hans Verkuil,
	nick, markus.heiser, Philipp Zabel, laurent.pinchart+renesas,
	bparrot, geert, Arnd Bergmann, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, Simon Horman,
	Niklas Söderlund, robert.jarzmik, songjun.wu,
	andrew-ct.chen, Greg Kroah-Hartman, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

Hi Tim,


On 01/11/2017 03:14 PM, Tim Harvey wrote:
>
> <snip>
>
> Hi Steve,
>
> I took a stab at testing this today on a gw51xx which has an adv7180
> hooked up as follows:
> - i2c3@0x20
> - 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
> (CSI0_IPU1)
> - PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
> - IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
> - all three analog inputs available to off-board connector
>
> My patch to the imx6qdl-gw51xx dtsi is:

As long as you used the patch to imx6qdl-sabreauto.dtsti that adds
the adv7180 support as a guide, you should be ok here.

> <snip>
>
>
>
> On an IMX6Q I'm getting the following when the adv7180 module loads:
> [   12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
> [   12.907767] imx-media: Registered subdev adv7180 2-0020
> [   12.907793] imx-media soc:media@0: Entity type for entity adv7180
> 2-0020 was not initialized!
> [   12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
> ipu1_csi0_mux:1
>
> Is the warning that adv7180 was not initialized expected and or an issue?

Yeah it's still a bug in the adv7180 driver, needs fixing.

>
> Now that your driver is hooking into the current media framework, I'm
> not at all clear on how to link and configure the media entities.

It's all documented at Documentation/media/v4l-drivers/imx.rst.
Follow the SabreAuto pipeline setup example.



> <snip>
>
>
> Additionally I've found that on an IMX6S/IMX6DL we crash while
> registering the media-ic subdev's:
> [    3.975473] imx-media: Registered subdev ipu1_csi1_mux
> [    3.980921] imx-media: Registered subdev ipu1_csi0_mux
> [    4.003205] imx-media: Registered subdev ipu1_ic_prpenc
> [    4.025373] imx-media: Registered subdev ipu1_ic_prpvf
> [    4.037944] ------------[ cut here ]------------
> [    4.042571] Kernel BUG at c06717dc [verbose debug info unavailable]
> [    4.048845] Internal error: Oops - BUG: 0 [#1] SMP ARM
> [    4.053990] Modules linked in:
> [    4.057076] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
> 4.9.0-rc6-00524-g84dad6e-dirty #446
> [    4.065260] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> ...
> [    4.296250] [<c0671780>] (v4l2_subdev_init) from [<c06fb02c>]
> (imx_ic_probe+0x94/0x1ac)
> [    4.304271] [<c06faf98>] (imx_ic_probe) from [<c05173d8>]
> (platform_drv_probe+0x54/0xb8)
> [    4.312373]  r9:c0d5e858 r8:00000000 r7:fffffdfb r6:c0e5dbf8
> r5:da603810 r4:c16738d8
> [    4.320129] [<c0517384>] (platform_drv_probe) from [<c0515978>]
> (driver_probe_device+0x20c/0x2c0)
> [    4.329010]  r7:c0e5dbf8 r6:00000000 r5:da603810 r4:c16738d8
> [    4.334681] [<c051576c>] (driver_probe_device) from [<c0515af4>]
> (__driver_attach+0xc8/0xcc)
> [    4.343129]  r9:c0d5e858 r8:00000000 r7:00000000 r6:da603844
> r5:c0e5dbf8 r4:da603810
> [    4.350889] [<c0515a2c>] (__driver_attach) from [<c0513adc>]
> (bus_for_each_dev+0x74/0xa8)
> [    4.359078]  r7:00000000 r6:c0515a2c r5:c0e5dbf8 r4:00000000
> [    4.364753] [<c0513a68>] (bus_for_each_dev) from [<c05151d4>]
> (driver_attach+0x20/0x28)
>
> I assume there is an iteration that needs a test on a missing pointer
> only available on chips with both IPU's or PRP

Yep, I only have quad boards here so I haven't gotten around to
testing on S/DL.

But it looks like I forgot to clear out the csi subdev pointer array before
passing it to imx_media_of_parse(). I think that might explain the OOPS
above. Try this patch:

diff --git a/drivers/staging/media/imx/imx-media-dev.c 
b/drivers/staging/media/imx/imx-media-dev.c
index 357654d..0cf2d61 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev)
  {
         struct device *dev = &pdev->dev;
         struct device_node *node = dev->of_node;
-       struct imx_media_subdev *csi[4];
+       struct imx_media_subdev *csi[4] = {0};
         struct imx_media_dev *imxmd;
         int ret;


Steve

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-12  3:22     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-12  3:22 UTC (permalink / raw)
  To: Tim Harvey, Steve Longerbeam
  Cc: Mark Rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Hans Verkuil, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, Russell King - ARM Linux, geert,
	linux-media, devicetree, Sascha Hauer, Arnd Bergmann, mchehab,
	bparrot, Rob Herring, Simon Horman, tiffany.lin,
	linux-arm-kernel, Niklas Söderlund, Greg Kroah-Hartman

Hi Tim,


On 01/11/2017 03:14 PM, Tim Harvey wrote:
>
> <snip>
>
> Hi Steve,
>
> I took a stab at testing this today on a gw51xx which has an adv7180
> hooked up as follows:
> - i2c3@0x20
> - 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
> (CSI0_IPU1)
> - PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
> - IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
> - all three analog inputs available to off-board connector
>
> My patch to the imx6qdl-gw51xx dtsi is:

As long as you used the patch to imx6qdl-sabreauto.dtsti that adds
the adv7180 support as a guide, you should be ok here.

> <snip>
>
>
>
> On an IMX6Q I'm getting the following when the adv7180 module loads:
> [   12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
> [   12.907767] imx-media: Registered subdev adv7180 2-0020
> [   12.907793] imx-media soc:media@0: Entity type for entity adv7180
> 2-0020 was not initialized!
> [   12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
> ipu1_csi0_mux:1
>
> Is the warning that adv7180 was not initialized expected and or an issue?

Yeah it's still a bug in the adv7180 driver, needs fixing.

>
> Now that your driver is hooking into the current media framework, I'm
> not at all clear on how to link and configure the media entities.

It's all documented at Documentation/media/v4l-drivers/imx.rst.
Follow the SabreAuto pipeline setup example.



> <snip>
>
>
> Additionally I've found that on an IMX6S/IMX6DL we crash while
> registering the media-ic subdev's:
> [    3.975473] imx-media: Registered subdev ipu1_csi1_mux
> [    3.980921] imx-media: Registered subdev ipu1_csi0_mux
> [    4.003205] imx-media: Registered subdev ipu1_ic_prpenc
> [    4.025373] imx-media: Registered subdev ipu1_ic_prpvf
> [    4.037944] ------------[ cut here ]------------
> [    4.042571] Kernel BUG at c06717dc [verbose debug info unavailable]
> [    4.048845] Internal error: Oops - BUG: 0 [#1] SMP ARM
> [    4.053990] Modules linked in:
> [    4.057076] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
> 4.9.0-rc6-00524-g84dad6e-dirty #446
> [    4.065260] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> ...
> [    4.296250] [<c0671780>] (v4l2_subdev_init) from [<c06fb02c>]
> (imx_ic_probe+0x94/0x1ac)
> [    4.304271] [<c06faf98>] (imx_ic_probe) from [<c05173d8>]
> (platform_drv_probe+0x54/0xb8)
> [    4.312373]  r9:c0d5e858 r8:00000000 r7:fffffdfb r6:c0e5dbf8
> r5:da603810 r4:c16738d8
> [    4.320129] [<c0517384>] (platform_drv_probe) from [<c0515978>]
> (driver_probe_device+0x20c/0x2c0)
> [    4.329010]  r7:c0e5dbf8 r6:00000000 r5:da603810 r4:c16738d8
> [    4.334681] [<c051576c>] (driver_probe_device) from [<c0515af4>]
> (__driver_attach+0xc8/0xcc)
> [    4.343129]  r9:c0d5e858 r8:00000000 r7:00000000 r6:da603844
> r5:c0e5dbf8 r4:da603810
> [    4.350889] [<c0515a2c>] (__driver_attach) from [<c0513adc>]
> (bus_for_each_dev+0x74/0xa8)
> [    4.359078]  r7:00000000 r6:c0515a2c r5:c0e5dbf8 r4:00000000
> [    4.364753] [<c0513a68>] (bus_for_each_dev) from [<c05151d4>]
> (driver_attach+0x20/0x28)
>
> I assume there is an iteration that needs a test on a missing pointer
> only available on chips with both IPU's or PRP

Yep, I only have quad boards here so I haven't gotten around to
testing on S/DL.

But it looks like I forgot to clear out the csi subdev pointer array before
passing it to imx_media_of_parse(). I think that might explain the OOPS
above. Try this patch:

diff --git a/drivers/staging/media/imx/imx-media-dev.c 
b/drivers/staging/media/imx/imx-media-dev.c
index 357654d..0cf2d61 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev)
  {
         struct device *dev = &pdev->dev;
         struct device_node *node = dev->of_node;
-       struct imx_media_subdev *csi[4];
+       struct imx_media_subdev *csi[4] = {0};
         struct imx_media_dev *imxmd;
         int ret;


Steve

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-12  3:22     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-12  3:22 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Tim,


On 01/11/2017 03:14 PM, Tim Harvey wrote:
>
> <snip>
>
> Hi Steve,
>
> I took a stab at testing this today on a gw51xx which has an adv7180
> hooked up as follows:
> - i2c3 at 0x20
> - 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
> (CSI0_IPU1)
> - PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
> - IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
> - all three analog inputs available to off-board connector
>
> My patch to the imx6qdl-gw51xx dtsi is:

As long as you used the patch to imx6qdl-sabreauto.dtsti that adds
the adv7180 support as a guide, you should be ok here.

> <snip>
>
>
>
> On an IMX6Q I'm getting the following when the adv7180 module loads:
> [   12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
> [   12.907767] imx-media: Registered subdev adv7180 2-0020
> [   12.907793] imx-media soc:media at 0: Entity type for entity adv7180
> 2-0020 was not initialized!
> [   12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
> ipu1_csi0_mux:1
>
> Is the warning that adv7180 was not initialized expected and or an issue?

Yeah it's still a bug in the adv7180 driver, needs fixing.

>
> Now that your driver is hooking into the current media framework, I'm
> not at all clear on how to link and configure the media entities.

It's all documented at Documentation/media/v4l-drivers/imx.rst.
Follow the SabreAuto pipeline setup example.



> <snip>
>
>
> Additionally I've found that on an IMX6S/IMX6DL we crash while
> registering the media-ic subdev's:
> [    3.975473] imx-media: Registered subdev ipu1_csi1_mux
> [    3.980921] imx-media: Registered subdev ipu1_csi0_mux
> [    4.003205] imx-media: Registered subdev ipu1_ic_prpenc
> [    4.025373] imx-media: Registered subdev ipu1_ic_prpvf
> [    4.037944] ------------[ cut here ]------------
> [    4.042571] Kernel BUG at c06717dc [verbose debug info unavailable]
> [    4.048845] Internal error: Oops - BUG: 0 [#1] SMP ARM
> [    4.053990] Modules linked in:
> [    4.057076] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
> 4.9.0-rc6-00524-g84dad6e-dirty #446
> [    4.065260] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> ...
> [    4.296250] [<c0671780>] (v4l2_subdev_init) from [<c06fb02c>]
> (imx_ic_probe+0x94/0x1ac)
> [    4.304271] [<c06faf98>] (imx_ic_probe) from [<c05173d8>]
> (platform_drv_probe+0x54/0xb8)
> [    4.312373]  r9:c0d5e858 r8:00000000 r7:fffffdfb r6:c0e5dbf8
> r5:da603810 r4:c16738d8
> [    4.320129] [<c0517384>] (platform_drv_probe) from [<c0515978>]
> (driver_probe_device+0x20c/0x2c0)
> [    4.329010]  r7:c0e5dbf8 r6:00000000 r5:da603810 r4:c16738d8
> [    4.334681] [<c051576c>] (driver_probe_device) from [<c0515af4>]
> (__driver_attach+0xc8/0xcc)
> [    4.343129]  r9:c0d5e858 r8:00000000 r7:00000000 r6:da603844
> r5:c0e5dbf8 r4:da603810
> [    4.350889] [<c0515a2c>] (__driver_attach) from [<c0513adc>]
> (bus_for_each_dev+0x74/0xa8)
> [    4.359078]  r7:00000000 r6:c0515a2c r5:c0e5dbf8 r4:00000000
> [    4.364753] [<c0513a68>] (bus_for_each_dev) from [<c05151d4>]
> (driver_attach+0x20/0x28)
>
> I assume there is an iteration that needs a test on a missing pointer
> only available on chips with both IPU's or PRP

Yep, I only have quad boards here so I haven't gotten around to
testing on S/DL.

But it looks like I forgot to clear out the csi subdev pointer array before
passing it to imx_media_of_parse(). I think that might explain the OOPS
above. Try this patch:

diff --git a/drivers/staging/media/imx/imx-media-dev.c 
b/drivers/staging/media/imx/imx-media-dev.c
index 357654d..0cf2d61 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev)
  {
         struct device *dev = &pdev->dev;
         struct device_node *node = dev->of_node;
-       struct imx_media_subdev *csi[4];
+       struct imx_media_subdev *csi[4] = {0};
         struct imx_media_dev *imxmd;
         int ret;


Steve

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-12 19:37     ` Tim Harvey
  -1 siblings, 0 replies; 549+ messages in thread
From: Tim Harvey @ 2017-01-12 19:37 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King - ARM Linux, mchehab, Hans Verkuil,
	Nick Dyer, markus.heiser, Philipp Zabel,
	laurent.pinchart+renesas, bparrot, geert, Arnd Bergmann,
	Sudip Mukherjee, minghsiu.tsai, Tiffany Lin,
	Jean-Christophe TROTIN, Simon Horman, Niklas Söderlund,
	robert.jarzmik, songjun.wu, andrew-ct.chen, Greg Kroah-Hartman,
	devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> Add pinctrl groups for both GPT input capture channels.
>
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> index 967c3b8..495709f 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> @@ -457,6 +457,18 @@
>                         >;
>                 };
>
> +               pinctrl_gpt_input_capture0: gptinputcapture0grp {
> +                       fsl,pins = <
> +                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
> +                       >;
> +               };
> +
> +               pinctrl_gpt_input_capture1: gptinputcapture1grp {
> +                       fsl,pins = <
> +                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
> +                       >;
> +               };
> +
>                 pinctrl_spdif: spdifgrp {
>                         fsl,pins = <
>                                 MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
> --

Steve,

These are not used anywhere.

Tim

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
@ 2017-01-12 19:37     ` Tim Harvey
  0 siblings, 0 replies; 549+ messages in thread
From: Tim Harvey @ 2017-01-12 19:37 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King - ARM Linux, mchehab, Hans Verkuil,
	Nick Dyer, markus.heiser, Philipp Zabel,
	laurent.pinchart+renesas, bparrot, geert, Arnd Bergmann,
	Sudip Mukherjee, minghsiu.tsai, Tiffany Lin,
	Jean-Christophe TROTIN, Simon Horman

On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> Add pinctrl groups for both GPT input capture channels.
>
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> index 967c3b8..495709f 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> @@ -457,6 +457,18 @@
>                         >;
>                 };
>
> +               pinctrl_gpt_input_capture0: gptinputcapture0grp {
> +                       fsl,pins = <
> +                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
> +                       >;
> +               };
> +
> +               pinctrl_gpt_input_capture1: gptinputcapture1grp {
> +                       fsl,pins = <
> +                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
> +                       >;
> +               };
> +
>                 pinctrl_spdif: spdifgrp {
>                         fsl,pins = <
>                                 MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
> --

Steve,

These are not used anywhere.

Tim

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
@ 2017-01-12 19:37     ` Tim Harvey
  0 siblings, 0 replies; 549+ messages in thread
From: Tim Harvey @ 2017-01-12 19:37 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> Add pinctrl groups for both GPT input capture channels.
>
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> index 967c3b8..495709f 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> @@ -457,6 +457,18 @@
>                         >;
>                 };
>
> +               pinctrl_gpt_input_capture0: gptinputcapture0grp {
> +                       fsl,pins = <
> +                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
> +                       >;
> +               };
> +
> +               pinctrl_gpt_input_capture1: gptinputcapture1grp {
> +                       fsl,pins = <
> +                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
> +                       >;
> +               };
> +
>                 pinctrl_spdif: spdifgrp {
>                         fsl,pins = <
>                                 MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
> --

Steve,

These are not used anywhere.

Tim

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-12  3:22     ` Steve Longerbeam
  (?)
  (?)
@ 2017-01-12 21:13     ` Tim Harvey
  2017-01-12 22:32       ` Steve Longerbeam
  -1 siblings, 1 reply; 549+ messages in thread
From: Tim Harvey @ 2017-01-12 21:13 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Steve Longerbeam, Shawn Guo, Sascha Hauer, Fabio Estevam,
	Hans Verkuil, Philipp Zabel, laurent.pinchart+renesas,
	linux-media

On Wed, Jan 11, 2017 at 7:22 PM, Steve Longerbeam
<steve_longerbeam@mentor.com> wrote:
> Hi Tim,
>
>
> On 01/11/2017 03:14 PM, Tim Harvey wrote:
>>
>>
>> <snip>
>>
>> Hi Steve,
>>
>> I took a stab at testing this today on a gw51xx which has an adv7180
>> hooked up as follows:
>> - i2c3@0x20
>> - 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
>> (CSI0_IPU1)
>> - PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
>> - IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
>> - all three analog inputs available to off-board connector
>>
>> My patch to the imx6qdl-gw51xx dtsi is:
>
>
> As long as you used the patch to imx6qdl-sabreauto.dtsti that adds
> the adv7180 support as a guide, you should be ok here.

yes - fairly straightforward

>
>> <snip>
>>
>>
>>
>> On an IMX6Q I'm getting the following when the adv7180 module loads:
>> [   12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
>> [   12.907767] imx-media: Registered subdev adv7180 2-0020
>> [   12.907793] imx-media soc:media@0: Entity type for entity adv7180
>> 2-0020 was not initialized!
>> [   12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
>> ipu1_csi0_mux:1
>>
>> Is the warning that adv7180 was not initialized expected and or an issue?
>
>
> Yeah it's still a bug in the adv7180 driver, needs fixing.
>

ok - ignoring

>>
>> Now that your driver is hooking into the current media framework, I'm
>> not at all clear on how to link and configure the media entities.
>
>
> It's all documented at Documentation/media/v4l-drivers/imx.rst.
> Follow the SabreAuto pipeline setup example.
>

ah yes... it helps to read your patches! You did a great job on the
documentation.

Regarding the The ipu1_csi0_mux/ipu2_csi1_mux entities which have 1
source and 2 sinks (which makes sense for a mux) how do you know which
sink pad you should use (in your adv7180 example you use the 2nd sink
pad vs the first)?

As my hardware is the same as the SabreAuto except that my adv7180 is
on i2c-2@0x20 I follow your example from
Documentation/media/v4l-drivers/imx.rst:

# Setup links
media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":1[1]'
media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'

# Configure pads
media-ctl -V "\"adv7180 2-0020\":0 [fmt:UYVY2X8/720x480]"
media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
# pad field types for camif can be any format prpvf supports
export outputfmt="UYVY2X8/720x480"
media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"

# select AIN1
v4l2-ctl -d0 -i0
Video input set to 0 (ADV7180 Composite on Ain1: ok)
v4l2-ctl -d0 --set-fmt-video=width=720,height=480,pixelformat=UYVY
# capture a single raw frame
v4l2-ctl -d0 --stream-mmap --stream-to=/x.raw --stream-count=1
[ 2092.056394] camif0: pipeline_set_stream failed with -32
VIDIOC_STREAMON: failed: Broken pipe

Enabling debug in drivers/media/media-entity.c I see:
[   38.870087] imx-media soc:media@0: link validation failed for
"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0, error -32

Looking at ipu1_smfc0 and ipu1_ic_prpvf with media-ctl I see:
- entity 12: ipu1_ic_prpvf (2 pads, 8 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev3
        pad0: Sink
                [fmt:UYVY2X8/720x480 field:alternate]
                <- "ipu1_csi0":1 []
                <- "ipu1_csi1":1 []
                <- "ipu1_smfc0":1 [ENABLED]
                <- "ipu1_smfc1":1 []
        pad1: Source
                [fmt:UYVY2X8/720x480 field:none]
                -> "camif0":0 [ENABLED]
                -> "camif1":0 []
                -> "ipu1_ic_pp0":0 []
                -> "ipu1_ic_pp1":0 []

- entity 45: ipu1_smfc0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev14
        pad0: Sink
                [fmt:UYVY2X8/720x480]
                <- "ipu1_csi0":1 [ENABLED]
        pad1: Source
                [fmt:UYVY2X8/720x480]
                -> "ipu1_ic_prpvf":0 [ENABLED]
                -> "ipu1_ic_pp0":0 []
                -> "camif0":0 []
                -> "camif1":0 []

Any ideas what is going wrong here? Seems like its perhaps a field
type mismatch. Is my outputfmt incorrect perhaps? I likely have
misunderstood the pad type comments in your documentation.

>
>
>> <snip>
>>
>>
>>
>> Additionally I've found that on an IMX6S/IMX6DL we crash while
>> registering the media-ic subdev's:
<snip>
>
> Yep, I only have quad boards here so I haven't gotten around to
> testing on S/DL.
>
> But it looks like I forgot to clear out the csi subdev pointer array before
> passing it to imx_media_of_parse(). I think that might explain the OOPS
> above. Try this patch:
>
> diff --git a/drivers/staging/media/imx/imx-media-dev.c
> b/drivers/staging/media/imx/imx-media-dev.c
> index 357654d..0cf2d61 100644
> --- a/drivers/staging/media/imx/imx-media-dev.c
> +++ b/drivers/staging/media/imx/imx-media-dev.c
> @@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev)
>  {
>         struct device *dev = &pdev->dev;
>         struct device_node *node = dev->of_node;
> -       struct imx_media_subdev *csi[4];
> +       struct imx_media_subdev *csi[4] = {0};
>         struct imx_media_dev *imxmd;
>         int ret;
>

This does resolves the crash on S/DL.

I do notice that the ipu1_csi*_mux entities on the S/DL have 3 more
sink pads compared to the D/Q which is from the additional ports
defined in the GPR nodes you add for mipi_vc1/vc2/vc3. Are there
really 3 more MIPI virtual channels on the S/DL vs the D/Q?

I get the same results on the S/DL as I do on D/Q as long as I adjust
the links to compensate for these additional sinks:
media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":4[1]'  # pad4
media-ctl -l '"ipu1_csi0_mux":5 -> "ipu1_csi0":0[1]' # pad5
...

This means link configuration must differ depending on S/DL vs D/Q
which is a bummer but I suppose this is the harsh reality as for
boards that use the EIM pads for IPU's they also will be using IPU2
for IMX6D/Q and IPU1 for IMX6S/DL.

Tim

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-12 21:13     ` Tim Harvey
@ 2017-01-12 22:32       ` Steve Longerbeam
  2017-01-13 21:04         ` Tim Harvey
  0 siblings, 1 reply; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-12 22:32 UTC (permalink / raw)
  To: Tim Harvey, Steve Longerbeam
  Cc: Shawn Guo, Sascha Hauer, Fabio Estevam, Hans Verkuil,
	Philipp Zabel, laurent.pinchart+renesas, linux-media

[-- Attachment #1: Type: text/plain, Size: 6656 bytes --]

Hi Tim,


On 01/12/2017 01:13 PM, Tim Harvey wrote:
>
>>> Now that your driver is hooking into the current media framework, I'm
>>> not at all clear on how to link and configure the media entities.
>>
>> It's all documented at Documentation/media/v4l-drivers/imx.rst.
>> Follow the SabreAuto pipeline setup example.
>>
> ah yes... it helps to read your patches! You did a great job on the
> documentation.
>
> Regarding the The ipu1_csi0_mux/ipu2_csi1_mux entities which have 1
> source and 2 sinks (which makes sense for a mux) how do you know which
> sink pad you should use (in your adv7180 example you use the 2nd sink
> pad vs the first)?

The adv7180 can only go to the parallel input pad (ipu1_csi0_mux:1
on quad). The other input pads select from the mipi csi-2 receiver virtual
channels.

Have you generated a dot graph? It makes it much easier to
visualize:

# media-ctl --print-dot > graph.dot

then on your host:

% dot -Tpng graph.dot > graph.png


>
> As my hardware is the same as the SabreAuto except that my adv7180 is
> on i2c-2@0x20 I follow your example from
> Documentation/media/v4l-drivers/imx.rst:
>
> # Setup links
> media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":1[1]'
> media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
> media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
> media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
>
> # Configure pads
> media-ctl -V "\"adv7180 2-0020\":0 [fmt:UYVY2X8/720x480]"
> media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
> media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
> media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
> media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
> media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
> media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
> media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
> # pad field types for camif can be any format prpvf supports
> export outputfmt="UYVY2X8/720x480"
> media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
> media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
> media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
>
> # select AIN1
> v4l2-ctl -d0 -i0
> Video input set to 0 (ADV7180 Composite on Ain1: ok)
> v4l2-ctl -d0 --set-fmt-video=width=720,height=480,pixelformat=UYVY
> # capture a single raw frame
> v4l2-ctl -d0 --stream-mmap --stream-to=/x.raw --stream-count=1
> [ 2092.056394] camif0: pipeline_set_stream failed with -32
> VIDIOC_STREAMON: failed: Broken pipe
>
> Enabling debug in drivers/media/media-entity.c I see:
> [   38.870087] imx-media soc:media@0: link validation failed for
> "ipu1_smfc0":1 -> "ipu1_ic_prpvf":0, error -32
>
> Looking at ipu1_smfc0 and ipu1_ic_prpvf with media-ctl I see:
> - entity 12: ipu1_ic_prpvf (2 pads, 8 links)
>               type V4L2 subdev subtype Unknown flags 0
>               device node name /dev/v4l-subdev3
>          pad0: Sink
>                  [fmt:UYVY2X8/720x480 field:alternate]
>                  <- "ipu1_csi0":1 []
>                  <- "ipu1_csi1":1 []
>                  <- "ipu1_smfc0":1 [ENABLED]
>                  <- "ipu1_smfc1":1 []
>          pad1: Source
>                  [fmt:UYVY2X8/720x480 field:none]
>                  -> "camif0":0 [ENABLED]
>                  -> "camif1":0 []
>                  -> "ipu1_ic_pp0":0 []
>                  -> "ipu1_ic_pp1":0 []
>
> - entity 45: ipu1_smfc0 (2 pads, 5 links)
>               type V4L2 subdev subtype Unknown flags 0
>               device node name /dev/v4l-subdev14
>          pad0: Sink
>                  [fmt:UYVY2X8/720x480]
>                  <- "ipu1_csi0":1 [ENABLED]
>          pad1: Source
>                  [fmt:UYVY2X8/720x480]
>                  -> "ipu1_ic_prpvf":0 [ENABLED]
>                  -> "ipu1_ic_pp0":0 []
>                  -> "camif0":0 []
>                  -> "camif1":0 []
>
> Any ideas what is going wrong here? Seems like its perhaps a field
> type mismatch.

Yes, exactly, you'll need to set the field types on every pad in your
pipeline.

>   Is my outputfmt incorrect perhaps? I likely have
> misunderstood the pad type comments in your documentation.

Attached is an update doc (from branch imx-media-staging-md-v7 on my fork).
I recently upgraded my v4l-utils package and media-ctl now supports 
specifying
the field type in the pad format strings. If you don't have the latest 
v4l-utils, it's
fairly straightforward to cross-build.

>
>>
>>> <snip>
>>>
>>>
>>>
>>> Additionally I've found that on an IMX6S/IMX6DL we crash while
>>> registering the media-ic subdev's:
> <snip>
>> Yep, I only have quad boards here so I haven't gotten around to
>> testing on S/DL.
>>
>> But it looks like I forgot to clear out the csi subdev pointer array before
>> passing it to imx_media_of_parse(). I think that might explain the OOPS
>> above. Try this patch:
>>
>> diff --git a/drivers/staging/media/imx/imx-media-dev.c
>> b/drivers/staging/media/imx/imx-media-dev.c
>> index 357654d..0cf2d61 100644
>> --- a/drivers/staging/media/imx/imx-media-dev.c
>> +++ b/drivers/staging/media/imx/imx-media-dev.c
>> @@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev)
>>   {
>>          struct device *dev = &pdev->dev;
>>          struct device_node *node = dev->of_node;
>> -       struct imx_media_subdev *csi[4];
>> +       struct imx_media_subdev *csi[4] = {0};
>>          struct imx_media_dev *imxmd;
>>          int ret;
>>
> This does resolves the crash on S/DL.

Cool thanks for verifying, I've applied this to the imx-media-staging-md-v7
branch.

>
> I do notice that the ipu1_csi*_mux entities on the S/DL have 3 more
> sink pads compared to the D/Q which is from the additional ports
> defined in the GPR nodes you add for mipi_vc1/vc2/vc3. Are there
> really 3 more MIPI virtual channels on the S/DL vs the D/Q?

Well, same number of virtual channels on quad vs S/DL. It's
just that the video mux on S/DL can select from all 4 virtual
channels, whereas quad's mux'es only select either vc0 or vc3.

>
> I get the same results on the S/DL as I do on D/Q as long as I adjust
> the links to compensate for these additional sinks:
> media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":4[1]'  # pad4
> media-ctl -l '"ipu1_csi0_mux":5 -> "ipu1_csi0":0[1]' # pad5
> ...
>
> This means link configuration must differ depending on S/DL vs D/Q
> which is a bummer but I suppose this is the harsh reality as for
> boards that use the EIM pads for IPU's they also will be using IPU2
> for IMX6D/Q and IPU1 for IMX6S/DL.

yeah, the links necessarily must be different between quad and S/DL.

Steve


[-- Attachment #2: imx.rst --]
[-- Type: text/plain, Size: 16978 bytes --]

i.MX Video Capture Driver
=========================

Introduction
------------

The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
handles the flow of image frames to and from capture devices and
display devices.

For image capture, the IPU contains the following internal subunits:

- Image DMA Controller (IDMAC)
- Camera Serial Interface (CSI)
- Image Converter (IC)
- Sensor Multi-FIFO Controller (SMFC)
- Image Rotator (IRT)
- Video De-Interlace Controller (VDIC)

The IDMAC is the DMA controller for transfer of image frames to and from
memory. Various dedicated DMA channels exist for both video capture and
display paths.

The CSI is the frontend capture unit that interfaces directly with
capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.

The IC handles color-space conversion, resizing, and rotation
operations. There are three independent "tasks" within the IC that can
carry out conversions concurrently: pre-processing encoding,
pre-processing preview, and post-processing.

The SMFC is composed of four independent channels that each can transfer
captured frames from sensors directly to memory concurrently.

The IRT carries out 90 and 270 degree image rotation operations.

The VDIC handles the conversion of interlaced video to progressive, with
support for different motion compensation modes (low, medium, and high
motion). The deinterlaced output frames from the VDIC can be sent to the
IC pre-process preview task for further conversions.

In addition to the IPU internal subunits, there are also two units
outside the IPU that are also involved in video capture on i.MX:

- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
  interface. This is a Synopsys DesignWare core.
- A video multiplexer for selecting among multiple sensor inputs to
  send to a CSI.

For more info, refer to the latest versions of the i.MX5/6 reference
manuals listed under References.


Features
--------

Some of the features of this driver include:

- Many different pipelines can be configured via media controller API,
  that correspond to the hardware video capture pipelines supported in
  the i.MX.

- Supports parallel, BT.565, and MIPI CSI-2 interfaces.

- Up to four concurrent sensor acquisitions, by configuring each
  sensor's pipeline using independent entities. This is currently
  demonstrated with the SabreSD and SabreLite reference boards with
  independent OV5642 and MIPI CSI-2 OV5640 sensor modules.

- Scaling, color-space conversion, and image rotation via IC task
  subdevs.

- Many pixel formats supported (RGB, packed and planar YUV, partial
  planar YUV).

- The IC pre-process preview subdev supports motion compensated
  de-interlacing using the VDIC, with three motion compensation modes:
  low, medium, and high motion. The mode is specified with a custom
  control. Pipelines are defined that allow sending frames to the
  preview subdev directly from the CSI or from the SMFC.

- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
  problems with the ADV718x video decoders. See below for a description
  of the FIM.


Capture Pipelines
-----------------

The following describe the various use-cases supported by the pipelines.

The links shown do not include the frontend sensor, video mux, or mipi
csi-2 receiver links. This depends on the type of sensor interface
(parallel or mipi csi-2). So in all cases, these pipelines begin with:

sensor -> ipu_csi_mux -> ipu_csi -> ...

for parallel sensors, or:

sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...

for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
to the video mux (ipu_csi_mux) before sending to the CSI, depending
on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
parenthesis.

Unprocessed Video Capture:
--------------------------

Send frames directly from sensor to camera interface, with no
conversions:

-> ipu_smfc -> camif

Note the ipu_smfc can do pixel reordering within the same colorspace.
For example, its sink pad can take UYVY2X8, but its source pad can
output YUYV2X8.

IC Direct Conversions:
----------------------

This pipeline uses the preprocess encode entity to route frames directly
from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
1024x1024 resolution, CSC, and image rotation:

-> ipu_ic_prpenc -> camif

This can be a useful capture pipeline for heavily loaded memory bus
traffic environments, since it has minimal IDMAC channel usage.

Post-Processing Conversions:
----------------------------

This pipeline routes frames from the SMFC to the post-processing
entity. In addition to CSC and rotation, this entity supports tiling
which allows scaled output beyond the 1024x1024 limitation of the IC
(up to 4096x4096 scaling output is supported):

-> ipu_smfc -> ipu_ic_pp -> camif

Motion Compensated De-interlace:
--------------------------------

This pipeline routes frames from the SMFC to the preprocess preview
entity to support motion-compensated de-interlacing using the VDIC,
scaling up to 1024x1024, and CSC:

-> ipu_smfc -> ipu_ic_prpvf -> camif

This pipeline also carries out the same conversions as above, but routes
frames directly from the CSI to the IC preprocess preview entity for
minimal memory bandwidth usage (note: this pipeline only works in
"high motion" mode):

-> ipu_ic_prpvf -> camif

This pipeline takes the motion-compensated de-interlaced frames and
sends them to the post-processor, to support motion-compensated
de-interlacing, scaling up to 4096x4096, CSC, and rotation:

-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif


Usage Notes
-----------

Many of the subdevs require information from the active sensor in the
current pipeline when configuring pad formats. Therefore the media links
should be established before configuring the media pad formats.

Similarly, the capture v4l2 interface subdev inherits controls from the
active subdevs in the current pipeline at link-setup time. Therefore the
capture links should be the last links established in order for capture
to "see" and inherit all possible controls.

The following are usage notes for Sabre- reference platforms:


SabreLite with OV5642 and OV5640
--------------------------------

This platform requires the OmniVision OV5642 module with a parallel
camera interface, and the OV5640 module with a MIPI CSI-2
interface. Both modules are available from Boundary Devices:

https://boundarydevices.com/products/nit6x_5mp
https://boundarydevices.com/product/nit6x_5mp_mipi

Note that if only one camera module is available, the other sensor
node can be disabled in the device tree.

The OV5642 module is connected to the parallel bus input on the i.MX
internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.

The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
receiver, and the four virtual channel outputs from the receiver are
routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
OV5640 must not share the same i2c slave address.

The following basic example configures unprocessed video capture
pipelines for both sensors. The OV5642 is routed to camif0
(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
virtual channel 1) is routed to camif1 (usually /dev/video1). Both
sensors are configured to output 640x480, the OV5642 outputs YUYV2X8,
the OV5640 UYVY2X8:

.. code-block:: none

   # Setup links for OV5642
   media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
   media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
   # Setup links for OV5640
   media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
   media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
   media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
   # Configure pads for OV5642 pipeline
   media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480 field:none]"
   # Configure pads for OV5640 pipeline
   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_smfc1\":0 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"ipu1_smfc1\":1 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480 field:none]"
   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480 field:none]"

Streaming can then begin independently on device nodes /dev/video0
and /dev/video1.

SabreAuto with ADV7180 decoder
------------------------------

On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
parallel bus input on the internal video mux to IPU1 CSI0.

The following example configures a pipeline to capture from the ADV7180
video decoder, assuming NTSC 720x480 input signals, with Motion
Compensated de-interlacing. Pad field types assume the adv7180 outputs
"alternate". $outputfmt can be any format supported by the
ipu1_ic_prpvf entity at its output pad:

.. code-block:: none

   # Setup links
   media-ctl -l '"adv7180 4-0021":0 -> "ipu1_csi0_mux":1[1]'
   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
   # Configure pads
   media-ctl -V "\"adv7180 4-0021\":0 [fmt:UYVY2X8/720x480]"
   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480 field:alternate]"
   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480 field:alternate]"
   media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480 field:alternate]"
   media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480 field:alternate]"
   media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480 field:alternate]"
   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480 field:alternate]"
   media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480 field:alternate]"
   media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt field:none]"
   media-ctl -V "\"camif0\":0 [fmt:$outputfmt field:none]"
   media-ctl -V "\"camif0\":1 [fmt:$outputfmt field:none]"

Streaming can then begin on /dev/video0.

This platform accepts Composite Video analog inputs to the ADV7180 on
Ain1 (connector J42) and Ain3 (connector J43).

To switch to Ain1:

.. code-block:: none

   # v4l2-ctl -i0

To switch to Ain3:

.. code-block:: none

   # v4l2-ctl -i1


Frame Interval Monitor
----------------------

The adv718x decoders can occasionally send corrupt fields during
NTSC/PAL signal re-sync (too little or too many video lines). When
this happens, the IPU triggers a mechanism to re-establish vertical
sync by adding 1 dummy line every frame, which causes a rolling effect
from image to image, and can last a long time before a stable image is
recovered. Or sometimes the mechanism doesn't work at all, causing a
permanent split image (one frame contains lines from two consecutive
captured images).

From experiment it was found that during image rolling, the frame
intervals (elapsed time between two EOF's) drop below the nominal
value for the current standard, by about one frame time (60 usec),
and remain at that value until rolling stops.

While the reason for this observation isn't known (the IPU dummy
line mechanism should show an increase in the intervals by 1 line
time every frame, not a fixed value), we can use it to detect the
corrupt fields using a frame interval monitor. If the FIM detects a
bad frame interval, a subdev event is sent. In response, userland can
issue a streaming restart to correct the rolling/split image.

The FIM is implemented in the imx-csi entity, and the entities that have
direct connections to the CSI call into the FIM to monitor the frame
intervals: ipu_smfc, ipu_ic_prpenc, and ipu_prpvf (when configured with
a direct link from ipu_csi). Userland can register with the FIM event
notifications on the imx-csi subdev device node
(V4L2_EVENT_IMX_FRAME_INTERVAL).

The imx-csi entity includes custom controls to tweak some dials for FIM.
If one of these controls is changed during streaming, the FIM will be
reset and will continue at the new settings.

- V4L2_CID_IMX_FIM_ENABLE

Enable/disable the FIM.

- V4L2_CID_IMX_FIM_NUM

How many frame interval errors to average before comparing against the
nominal frame interval reported by the sensor. This can reduce noise
from interrupt latency.

- V4L2_CID_IMX_FIM_TOLERANCE_MIN

If the averaged intervals fall outside nominal by this amount, in
microseconds, streaming will be restarted.

- V4L2_CID_IMX_FIM_TOLERANCE_MAX

If any interval errors are higher than this value, those error samples
are discarded and do not enter into the average. This can be used to
discard really high interval errors that might be due to very high
system load, causing excessive interrupt latencies.

- V4L2_CID_IMX_FIM_NUM_SKIP

How many frames to skip after a FIM reset or stream restart before
FIM begins to average intervals. It has been found that there can
be a few bad frame intervals after stream restart which are not
attributed to adv718x sending a corrupt field, so this is used to
skip those frames to prevent unnecessary restarts.


SabreSD with MIPI CSI-2 OV5640
------------------------------

Similarly to SabreLite, the SabreSD supports a parallel interface
OV5642 module on IPU1 CSI0, and a MIPI CSI-2 OV5640 module. The OV5642
connects to i2c bus 1 and the OV5640 to i2c bus 2.

The device tree for SabreSD includes OF graphs for both the parallel
OV5642 and the MIPI CSI-2 OV5640, but as of this writing only the MIPI
CSI-2 OV5640 has been tested, so the OV5642 node is currently disabled.
The OV5640 module connects to MIPI connector J5 (sorry I don't have the
compatible module part number or URL).

The following example configures a post-processing pipeline to capture
from the OV5640. $sensorfmt can be any format supported by the OV5640.
$outputfmt can be any format supported by the ipu1_ic_pp1 entity at its
output pad:

.. code-block:: none

   # Setup links
   media-ctl -l '"ov5640_mipi 1-003c":0 -> "imx-mipi-csi2":0[1]'
   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
   media-ctl -l '"ipu1_smfc1":1 -> "ipu1_ic_pp1":0[1]'
   media-ctl -l '"ipu1_ic_pp1":1 -> "camif0":0[1]'
   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
   # Configure pads
   media-ctl -V "\"ov5640_mipi 1-003c\":0 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"ipu1_csi1\":0 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"ipu1_csi1\":1 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"ipu1_smfc1\":0 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"ipu1_smfc1\":1 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"ipu1_ic_pp1\":0 [fmt:$sensorfmt field:none]"
   media-ctl -V "\"ipu1_ic_pp1\":1 [fmt:$outputfmt field:none]"
   media-ctl -V "\"camif0\":0 [fmt:$outputfmt field:none]"
   media-ctl -V "\"camif0\":1 [fmt:$outputfmt field:none]"

Streaming can then begin on /dev/video0.



Known Issues
------------

1. When using 90 or 270 degree rotation control at capture resolutions
   near the IC resizer limit of 1024x1024, and combined with planar
   pixel formats (YUV420, YUV422p), frame capture will often fail with
   no end-of-frame interrupts from the IDMAC channel. To work around
   this, use lower resolution and/or packed formats (YUYV, RGB3, etc.)
   when 90 or 270 rotations are needed.


File list
---------

drivers/staging/media/imx/
include/media/imx.h
include/uapi/media/imx.h

References
----------

[1] "i.MX 6Dual/6Quad Applications Processor Reference Manual"
[2] "i.MX 6Solo/6DualLite Applications Processor Reference Manual"


Author
------
Steve Longerbeam <steve_longerbeam@mentor.com>

Copyright (C) 2012-2016 Mentor Graphics Inc.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
  2017-01-12 19:37     ` Tim Harvey
  (?)
@ 2017-01-12 23:40       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-12 23:40 UTC (permalink / raw)
  To: Tim Harvey, Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King - ARM Linux, mchehab, Hans Verkuil,
	Nick Dyer, markus.heiser, Philipp Zabel,
	laurent.pinchart+renesas, bparrot, geert, Arnd Bergmann,
	Sudip Mukherjee, minghsiu.tsai, Tiffany Lin,
	Jean-Christophe TROTIN, Simon Horman, Niklas Söderlund,
	robert.jarzmik, songjun.wu, andrew-ct.chen, Greg Kroah-Hartman,
	devicetree, linux-kernel, linux-arm-kernel, linux-media, devel



On 01/12/2017 11:37 AM, Tim Harvey wrote:
> On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> Add pinctrl groups for both GPT input capture channels.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> index 967c3b8..495709f 100644
>> --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> @@ -457,6 +457,18 @@
>>                          >;
>>                  };
>>
>> +               pinctrl_gpt_input_capture0: gptinputcapture0grp {
>> +                       fsl,pins = <
>> +                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
>> +                       >;
>> +               };
>> +
>> +               pinctrl_gpt_input_capture1: gptinputcapture1grp {
>> +                       fsl,pins = <
>> +                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
>> +                       >;
>> +               };
>> +
>>                  pinctrl_spdif: spdifgrp {
>>                          fsl,pins = <
>>                                  MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
>> --
> Steve,
>
> These are not used anywhere.

Yes, maybe I should just remove this patch for now. I'm only keeping it
because eventually it will be needed to support i.MX6 input capture.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
@ 2017-01-12 23:40       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-12 23:40 UTC (permalink / raw)
  To: Tim Harvey, Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King - ARM Linux, mchehab, Hans Verkuil,
	Nick Dyer, markus.heiser, Philipp Zabel,
	laurent.pinchart+renesas, bparrot, geert, Arnd Bergmann,
	Sudip Mukherjee, minghsiu.tsai, Tiffany Lin,
	Jean-Christophe TROTIN, Simon Horman



On 01/12/2017 11:37 AM, Tim Harvey wrote:
> On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> Add pinctrl groups for both GPT input capture channels.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> index 967c3b8..495709f 100644
>> --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> @@ -457,6 +457,18 @@
>>                          >;
>>                  };
>>
>> +               pinctrl_gpt_input_capture0: gptinputcapture0grp {
>> +                       fsl,pins = <
>> +                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
>> +                       >;
>> +               };
>> +
>> +               pinctrl_gpt_input_capture1: gptinputcapture1grp {
>> +                       fsl,pins = <
>> +                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
>> +                       >;
>> +               };
>> +
>>                  pinctrl_spdif: spdifgrp {
>>                          fsl,pins = <
>>                                  MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
>> --
> Steve,
>
> These are not used anywhere.

Yes, maybe I should just remove this patch for now. I'm only keeping it
because eventually it will be needed to support i.MX6 input capture.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
@ 2017-01-12 23:40       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-12 23:40 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/12/2017 11:37 AM, Tim Harvey wrote:
> On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> Add pinctrl groups for both GPT input capture channels.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> index 967c3b8..495709f 100644
>> --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
>> @@ -457,6 +457,18 @@
>>                          >;
>>                  };
>>
>> +               pinctrl_gpt_input_capture0: gptinputcapture0grp {
>> +                       fsl,pins = <
>> +                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
>> +                       >;
>> +               };
>> +
>> +               pinctrl_gpt_input_capture1: gptinputcapture1grp {
>> +                       fsl,pins = <
>> +                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
>> +                       >;
>> +               };
>> +
>>                  pinctrl_spdif: spdifgrp {
>>                          fsl,pins = <
>>                                  MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
>> --
> Steve,
>
> These are not used anywhere.

Yes, maybe I should just remove this patch for now. I'm only keeping it
because eventually it will be needed to support i.MX6 input capture.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-13 11:55     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:55 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add bindings documentation for the i.MX media driver.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>  1 file changed, 57 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/imx.txt
> 
> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
> new file mode 100644
> index 0000000..254b64a
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/imx.txt
> @@ -0,0 +1,57 @@
> +Freescale i.MX Media Video Devices
> +
> +Video Media Controller node
> +---------------------------
> +
> +This is the parent media controller node for video capture support.
> +
> +Required properties:
> +- compatible : "fsl,imx-media";

Would you be opposed to calling this "capture-subsystem" instead of
"imx-media"? We already use "fsl,imx-display-subsystem" and
"fsl,imx-gpu-subsystem" for the display and GPU compound devices.

> +- ports      : Should contain a list of phandles pointing to camera
> +  	       sensor interface ports of IPU devices
> +
> +
> +fim child node
> +--------------
> +
> +This is an optional child node of the ipu_csi port nodes. If present and
> +available, it enables the Frame Interval Monitor. Its properties can be
> +used to modify the method in which the FIM measures frame intervals.
> +Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
> +Frame Interval Monitor.
> +
> +Optional properties:
> +- fsl,input-capture-channel: an input capture channel and channel flags,
> +			     specified as <chan flags>. The channel number
> +			     must be 0 or 1. The flags can be
> +			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
> +			     IRQ_TYPE_EDGE_BOTH, and specify which input
> +			     capture signal edge will trigger the input
> +			     capture event. If an input capture channel is
> +			     specified, the FIM will use this method to
> +			     measure frame intervals instead of via the EOF
> +			     interrupt. The input capture method is much
> +			     preferred over EOF as it is not subject to
> +			     interrupt latency errors. However it requires
> +			     routing the VSYNC or FIELD output signals of
> +			     the camera sensor to one of the i.MX input
> +			     capture pads (SD1_DAT0, SD1_DAT1), which also
> +			     gives up support for SD1.

This is a clever method to get better frame timestamps. Too bad about
the routing requirements. Can this be used on Nitrogen6X?

> +
> +mipi_csi2 node
> +--------------
> +
> +This is the device node for the MIPI CSI-2 Receiver, required for MIPI
> +CSI-2 sensors.
> +
> +Required properties:
> +- compatible	: "fsl,imx6-mipi-csi2";

I think this should get an additional "snps,dw-mipi-csi2" compatible,
since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

> +- reg           : physical base address and length of the register set;
> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
> +                  (the DPHY clock), video_27m, and eim_sel;

Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
shared gate bit that gates the HSI clocks as well as the MIPI
"ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
csi-2 core, but we are missing shared gate clocks in the clock tree for
these.
Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
video_27m seems fine.
But I don't get "dphy". Which input clock would that correspond to?
"pll_refclk?"
Also the pixel clock input is a gate after aclk_podf (which we call
eim_podf), not aclk_sel (eim_sel).

> +- clock-names	: must contain "dphy", "cfg", "pix";
> +
> +Optional properties:
> +- interrupts	: must contain two level-triggered interrupts,
> +                  in order: 100 and 101;

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-13 11:55     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:55 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add bindings documentation for the i.MX media driver.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>  1 file changed, 57 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/imx.txt
> 
> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
> new file mode 100644
> index 0000000..254b64a
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/imx.txt
> @@ -0,0 +1,57 @@
> +Freescale i.MX Media Video Devices
> +
> +Video Media Controller node
> +---------------------------
> +
> +This is the parent media controller node for video capture support.
> +
> +Required properties:
> +- compatible : "fsl,imx-media";

Would you be opposed to calling this "capture-subsystem" instead of
"imx-media"? We already use "fsl,imx-display-subsystem" and
"fsl,imx-gpu-subsystem" for the display and GPU compound devices.

> +- ports      : Should contain a list of phandles pointing to camera
> +  	       sensor interface ports of IPU devices
> +
> +
> +fim child node
> +--------------
> +
> +This is an optional child node of the ipu_csi port nodes. If present and
> +available, it enables the Frame Interval Monitor. Its properties can be
> +used to modify the method in which the FIM measures frame intervals.
> +Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
> +Frame Interval Monitor.
> +
> +Optional properties:
> +- fsl,input-capture-channel: an input capture channel and channel flags,
> +			     specified as <chan flags>. The channel number
> +			     must be 0 or 1. The flags can be
> +			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
> +			     IRQ_TYPE_EDGE_BOTH, and specify which input
> +			     capture signal edge will trigger the input
> +			     capture event. If an input capture channel is
> +			     specified, the FIM will use this method to
> +			     measure frame intervals instead of via the EOF
> +			     interrupt. The input capture method is much
> +			     preferred over EOF as it is not subject to
> +			     interrupt latency errors. However it requires
> +			     routing the VSYNC or FIELD output signals of
> +			     the camera sensor to one of the i.MX input
> +			     capture pads (SD1_DAT0, SD1_DAT1), which also
> +			     gives up support for SD1.

This is a clever method to get better frame timestamps. Too bad about
the routing requirements. Can this be used on Nitrogen6X?

> +
> +mipi_csi2 node
> +--------------
> +
> +This is the device node for the MIPI CSI-2 Receiver, required for MIPI
> +CSI-2 sensors.
> +
> +Required properties:
> +- compatible	: "fsl,imx6-mipi-csi2";

I think this should get an additional "snps,dw-mipi-csi2" compatible,
since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

> +- reg           : physical base address and length of the register set;
> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
> +                  (the DPHY clock), video_27m, and eim_sel;

Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
shared gate bit that gates the HSI clocks as well as the MIPI
"ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
csi-2 core, but we are missing shared gate clocks in the clock tree for
these.
Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
video_27m seems fine.
But I don't get "dphy". Which input clock would that correspond to?
"pll_refclk?"
Also the pixel clock input is a gate after aclk_podf (which we call
eim_podf), not aclk_sel (eim_sel).

> +- clock-names	: must contain "dphy", "cfg", "pix";
> +
> +Optional properties:
> +- interrupts	: must contain two level-triggered interrupts,
> +                  in order: 100 and 101;

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-13 11:55     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:55 UTC (permalink / raw)
  To: linux-arm-kernel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add bindings documentation for the i.MX media driver.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>  1 file changed, 57 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/imx.txt
> 
> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
> new file mode 100644
> index 0000000..254b64a
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/imx.txt
> @@ -0,0 +1,57 @@
> +Freescale i.MX Media Video Devices
> +
> +Video Media Controller node
> +---------------------------
> +
> +This is the parent media controller node for video capture support.
> +
> +Required properties:
> +- compatible : "fsl,imx-media";

Would you be opposed to calling this "capture-subsystem" instead of
"imx-media"? We already use "fsl,imx-display-subsystem" and
"fsl,imx-gpu-subsystem" for the display and GPU compound devices.

> +- ports      : Should contain a list of phandles pointing to camera
> +  	       sensor interface ports of IPU devices
> +
> +
> +fim child node
> +--------------
> +
> +This is an optional child node of the ipu_csi port nodes. If present and
> +available, it enables the Frame Interval Monitor. Its properties can be
> +used to modify the method in which the FIM measures frame intervals.
> +Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
> +Frame Interval Monitor.
> +
> +Optional properties:
> +- fsl,input-capture-channel: an input capture channel and channel flags,
> +			     specified as <chan flags>. The channel number
> +			     must be 0 or 1. The flags can be
> +			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
> +			     IRQ_TYPE_EDGE_BOTH, and specify which input
> +			     capture signal edge will trigger the input
> +			     capture event. If an input capture channel is
> +			     specified, the FIM will use this method to
> +			     measure frame intervals instead of via the EOF
> +			     interrupt. The input capture method is much
> +			     preferred over EOF as it is not subject to
> +			     interrupt latency errors. However it requires
> +			     routing the VSYNC or FIELD output signals of
> +			     the camera sensor to one of the i.MX input
> +			     capture pads (SD1_DAT0, SD1_DAT1), which also
> +			     gives up support for SD1.

This is a clever method to get better frame timestamps. Too bad about
the routing requirements. Can this be used on Nitrogen6X?

> +
> +mipi_csi2 node
> +--------------
> +
> +This is the device node for the MIPI CSI-2 Receiver, required for MIPI
> +CSI-2 sensors.
> +
> +Required properties:
> +- compatible	: "fsl,imx6-mipi-csi2";

I think this should get an additional "snps,dw-mipi-csi2" compatible,
since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

> +- reg           : physical base address and length of the register set;
> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
> +                  (the DPHY clock), video_27m, and eim_sel;

Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
shared gate bit that gates the HSI clocks as well as the MIPI
"ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
csi-2 core, but we are missing shared gate clocks in the clock tree for
these.
Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
video_27m seems fine.
But I don't get "dphy". Which input clock would that correspond to?
"pll_refclk?"
Also the pixel clock input is a gate after aclk_podf (which we call
eim_podf), not aclk_sel (eim_sel).

> +- clock-names	: must contain "dphy", "cfg", "pix";
> +
> +Optional properties:
> +- interrupts	: must contain two level-triggered interrupts,
> +                  in order: 100 and 101;

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-13 11:57     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:57 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
> clocks.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
> index 53e6e63..42926e9 100644
> --- a/arch/arm/boot/dts/imx6qdl.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
> @@ -1125,7 +1125,14 @@
>  			};
>  
>  			mipi_csi: mipi@021dc000 {
> +				compatible = "fsl,imx6-mipi-csi2";
>  				reg = <0x021dc000 0x4000>;
> +				interrupts = <0 100 0x04>, <0 101 0x04>;
> +				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
> +					 <&clks IMX6QDL_CLK_VIDEO_27M>,
> +					 <&clks IMX6QDL_CLK_EIM_SEL>;

I think the latter should be EIM_PODF

> +				clock-names = "dphy", "cfg", "pix";

and I'm not sure dphy is the right name for this one. Is that the pll
ref input?

> +				status = "disabled";
>  			};
>  
>  			mipi_dsi: mipi@021e0000 {

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
@ 2017-01-13 11:57     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:57 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
> clocks.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
> index 53e6e63..42926e9 100644
> --- a/arch/arm/boot/dts/imx6qdl.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
> @@ -1125,7 +1125,14 @@
>  			};
>  
>  			mipi_csi: mipi@021dc000 {
> +				compatible = "fsl,imx6-mipi-csi2";
>  				reg = <0x021dc000 0x4000>;
> +				interrupts = <0 100 0x04>, <0 101 0x04>;
> +				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
> +					 <&clks IMX6QDL_CLK_VIDEO_27M>,
> +					 <&clks IMX6QDL_CLK_EIM_SEL>;

I think the latter should be EIM_PODF

> +				clock-names = "dphy", "cfg", "pix";

and I'm not sure dphy is the right name for this one. Is that the pll
ref input?

> +				status = "disabled";
>  			};
>  
>  			mipi_dsi: mipi@021e0000 {

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
@ 2017-01-13 11:57     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:57 UTC (permalink / raw)
  To: linux-arm-kernel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
> clocks.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
> index 53e6e63..42926e9 100644
> --- a/arch/arm/boot/dts/imx6qdl.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
> @@ -1125,7 +1125,14 @@
>  			};
>  
>  			mipi_csi: mipi at 021dc000 {
> +				compatible = "fsl,imx6-mipi-csi2";
>  				reg = <0x021dc000 0x4000>;
> +				interrupts = <0 100 0x04>, <0 101 0x04>;
> +				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
> +					 <&clks IMX6QDL_CLK_VIDEO_27M>,
> +					 <&clks IMX6QDL_CLK_EIM_SEL>;

I think the latter should be EIM_PODF

> +				clock-names = "dphy", "cfg", "pix";

and I'm not sure dphy is the right name for this one. Is that the pll
ref input?

> +				status = "disabled";
>  			};
>  
>  			mipi_dsi: mipi at 021e0000 {

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 05/24] ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-13 11:59     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:59 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> There is a pin conflict with GPIO_6. This pin functions as a power
> input pin to the OV5642 camera sensor, but ENET uses it as the h/w
> workaround for erratum ERR006687, to wake-up the ARM cores on normal
> RX and TX packet done events. So we need to remove the h/w workaround
> to support the OV5642. The result is that the CPUidle driver will no
> longer allow entering the deep idle states on the sabrelite.
> 
> This is a partial revert of
> 
> commit 6261c4c8f13e ("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC
> 			interrupt.")
> commit a28eeb43ee57 ("ARM: dts: imx6: tag boards that have the HW workaround
> 			for ERR006687")
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Pity this has to be removed for all, Nitrogen6X should really use DT
overlays for the add-on boards / cameras.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 05/24] ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
@ 2017-01-13 11:59     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:59 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> There is a pin conflict with GPIO_6. This pin functions as a power
> input pin to the OV5642 camera sensor, but ENET uses it as the h/w
> workaround for erratum ERR006687, to wake-up the ARM cores on normal
> RX and TX packet done events. So we need to remove the h/w workaround
> to support the OV5642. The result is that the CPUidle driver will no
> longer allow entering the deep idle states on the sabrelite.
> 
> This is a partial revert of
> 
> commit 6261c4c8f13e ("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC
> 			interrupt.")
> commit a28eeb43ee57 ("ARM: dts: imx6: tag boards that have the HW workaround
> 			for ERR006687")
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Pity this has to be removed for all, Nitrogen6X should really use DT
overlays for the add-on boards / cameras.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 05/24] ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
@ 2017-01-13 11:59     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 11:59 UTC (permalink / raw)
  To: linux-arm-kernel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> There is a pin conflict with GPIO_6. This pin functions as a power
> input pin to the OV5642 camera sensor, but ENET uses it as the h/w
> workaround for erratum ERR006687, to wake-up the ARM cores on normal
> RX and TX packet done events. So we need to remove the h/w workaround
> to support the OV5642. The result is that the CPUidle driver will no
> longer allow entering the deep idle states on the sabrelite.
> 
> This is a partial revert of
> 
> commit 6261c4c8f13e ("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC
> 			interrupt.")
> commit a28eeb43ee57 ("ARM: dts: imx6: tag boards that have the HW workaround
> 			for ERR006687")
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Pity this has to be removed for all, Nitrogen6X should really use DT
overlays for the add-on boards / cameras.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-13 12:03     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 12:03 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> Both hang off the same i2c2 bus, so they require different (and non-
> default) i2c slave addresses.
> 
> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
> 
> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
>  arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
>  arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
>  3 files changed, 129 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> index 0f06ca5..fec2524 100644
> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> @@ -48,3 +48,8 @@
>  	model = "Freescale i.MX6 DualLite SABRE Lite Board";
>  	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
>  };
> +
> +&ipu1_csi1_from_ipu1_csi1_mux {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
> index 66d10d8..9e2d26d 100644
> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
> @@ -52,3 +52,9 @@
>  &sata {
>  	status = "okay";
>  };
> +
> +&ipu1_csi1_from_mipi_vc1 {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> +
> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> index 795b5a5..bca9fed 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> @@ -39,6 +39,8 @@
>   *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>   *     OTHER DEALINGS IN THE SOFTWARE.
>   */
> +
> +#include <dt-bindings/clock/imx6qdl-clock.h>
>  #include <dt-bindings/gpio/gpio.h>
>  #include <dt-bindings/input/input.h>
>  
> @@ -96,6 +98,15 @@
>  		};
>  	};
>  
> +	mipi_xclk: mipi_xclk {
> +		compatible = "pwm-clock";
> +		#clock-cells = <0>;
> +		clock-frequency = <22000000>;
> +		clock-output-names = "mipi_pwm3";
> +		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
> +		status = "okay";
> +	};
> +
>  	gpio-keys {
>  		compatible = "gpio-keys";
>  		pinctrl-names = "default";
> @@ -220,6 +231,22 @@
>  	};
>  };
>  
> +&ipu1_csi0_from_ipu1_csi0_mux {
> +	bus-width = <8>;
> +	data-shift = <12>; /* Lines 19:12 used */
> +	hsync-active = <1>;
> +	vync-active = <1>;
> +};
> +
> +&ipu1_csi0_mux_from_parallel_sensor {
> +	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
> +};
> +
> +&ipu1_csi0 {
> +	pinctrl-names = "default";
> +	pinctrl-0 = <&pinctrl_ipu1_csi0>;
> +};
> +
>  &audmux {
>  	pinctrl-names = "default";
>  	pinctrl-0 = <&pinctrl_audmux>;
> @@ -299,6 +326,52 @@
>  	pinctrl-names = "default";
>  	pinctrl-0 = <&pinctrl_i2c2>;
>  	status = "okay";
> +
> +	ov5640: camera@40 {
> +		compatible = "ovti,ov5640";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5640>;
> +		clocks = <&mipi_xclk>;
> +		clock-names = "xclk";
> +		reg = <0x40>;
> +		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

> +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> +
> +		port {
> +			#address-cells = <1>;
> +			#size-cells = <0>;
> +
> +			ov5640_to_mipi_csi: endpoint@1 {
> +				reg = <1>;
> +				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
> +				data-lanes = <0 1>;
> +				clock-lanes = <2>;
> +			};
> +		};
> +	};
> +
> +	ov5642: camera@42 {
> +		compatible = "ovti,ov5642";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5642>;
> +		clocks = <&clks IMX6QDL_CLK_CKO2>;
> +		clock-names = "xclk";
> +		reg = <0x42>;
> +		xclk = <24000000>;

Same here, use assigned-clock-rates on IMX6QDL_CLK_CKO2 if necessary.

> +		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
> +		pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
> +		gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
> +
> +		port {
> +			ov5642_to_ipu1_csi0_mux: endpoint {
> +				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
> +				bus-width = <8>;
> +				hsync-active = <1>;
> +				vsync-active = <1>;
> +			};
> +		};
> +	};
>  };
>  
>  &i2c3 {
> @@ -412,6 +485,23 @@
>  			>;
>  		};
>  
> +		pinctrl_ipu1_csi0: ipu1csi0grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
> +				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
> +				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
> +				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
> +				MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
> +			>;
> +		};
> +
>  		pinctrl_j15: j15grp {
>  			fsl,pins = <
>  				MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
> @@ -445,6 +535,22 @@
>  			>;
>  		};
>  
> +		pinctrl_ov5640: ov5640grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x000b0
> +				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
> +			>;
> +		};
> +
> +		pinctrl_ov5642: ov5642grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
> +				MX6QDL_PAD_GPIO_6__GPIO1_IO06   0x1b0b0
> +				MX6QDL_PAD_GPIO_8__GPIO1_IO08   0x130b0
> +				MX6QDL_PAD_GPIO_3__CCM_CLKO2    0x000b0
> +			>;
> +		};
> +
>  		pinctrl_pwm1: pwm1grp {
>  			fsl,pins = <
>  				MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
> @@ -601,3 +707,15 @@
>  	vmmc-supply = <&reg_3p3v>;
>  	status = "okay";
>  };
> +
> +&mipi_csi {
> +        status = "okay";
> +};
> +
> +/* Incoming port from sensor */
> +&mipi_csi_from_mipi_sensor {
> +        remote-endpoint = <&ov5640_to_mipi_csi>;
> +        data-lanes = <0 1>;
> +        clock-lanes = <2>;
> +};
> +

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-13 12:03     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 12:03 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> Both hang off the same i2c2 bus, so they require different (and non-
> default) i2c slave addresses.
> 
> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
> 
> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
>  arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
>  arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
>  3 files changed, 129 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> index 0f06ca5..fec2524 100644
> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> @@ -48,3 +48,8 @@
>  	model = "Freescale i.MX6 DualLite SABRE Lite Board";
>  	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
>  };
> +
> +&ipu1_csi1_from_ipu1_csi1_mux {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
> index 66d10d8..9e2d26d 100644
> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
> @@ -52,3 +52,9 @@
>  &sata {
>  	status = "okay";
>  };
> +
> +&ipu1_csi1_from_mipi_vc1 {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> +
> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> index 795b5a5..bca9fed 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> @@ -39,6 +39,8 @@
>   *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>   *     OTHER DEALINGS IN THE SOFTWARE.
>   */
> +
> +#include <dt-bindings/clock/imx6qdl-clock.h>
>  #include <dt-bindings/gpio/gpio.h>
>  #include <dt-bindings/input/input.h>
>  
> @@ -96,6 +98,15 @@
>  		};
>  	};
>  
> +	mipi_xclk: mipi_xclk {
> +		compatible = "pwm-clock";
> +		#clock-cells = <0>;
> +		clock-frequency = <22000000>;
> +		clock-output-names = "mipi_pwm3";
> +		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
> +		status = "okay";
> +	};
> +
>  	gpio-keys {
>  		compatible = "gpio-keys";
>  		pinctrl-names = "default";
> @@ -220,6 +231,22 @@
>  	};
>  };
>  
> +&ipu1_csi0_from_ipu1_csi0_mux {
> +	bus-width = <8>;
> +	data-shift = <12>; /* Lines 19:12 used */
> +	hsync-active = <1>;
> +	vync-active = <1>;
> +};
> +
> +&ipu1_csi0_mux_from_parallel_sensor {
> +	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
> +};
> +
> +&ipu1_csi0 {
> +	pinctrl-names = "default";
> +	pinctrl-0 = <&pinctrl_ipu1_csi0>;
> +};
> +
>  &audmux {
>  	pinctrl-names = "default";
>  	pinctrl-0 = <&pinctrl_audmux>;
> @@ -299,6 +326,52 @@
>  	pinctrl-names = "default";
>  	pinctrl-0 = <&pinctrl_i2c2>;
>  	status = "okay";
> +
> +	ov5640: camera@40 {
> +		compatible = "ovti,ov5640";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5640>;
> +		clocks = <&mipi_xclk>;
> +		clock-names = "xclk";
> +		reg = <0x40>;
> +		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

> +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> +
> +		port {
> +			#address-cells = <1>;
> +			#size-cells = <0>;
> +
> +			ov5640_to_mipi_csi: endpoint@1 {
> +				reg = <1>;
> +				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
> +				data-lanes = <0 1>;
> +				clock-lanes = <2>;
> +			};
> +		};
> +	};
> +
> +	ov5642: camera@42 {
> +		compatible = "ovti,ov5642";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5642>;
> +		clocks = <&clks IMX6QDL_CLK_CKO2>;
> +		clock-names = "xclk";
> +		reg = <0x42>;
> +		xclk = <24000000>;

Same here, use assigned-clock-rates on IMX6QDL_CLK_CKO2 if necessary.

> +		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
> +		pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
> +		gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
> +
> +		port {
> +			ov5642_to_ipu1_csi0_mux: endpoint {
> +				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
> +				bus-width = <8>;
> +				hsync-active = <1>;
> +				vsync-active = <1>;
> +			};
> +		};
> +	};
>  };
>  
>  &i2c3 {
> @@ -412,6 +485,23 @@
>  			>;
>  		};
>  
> +		pinctrl_ipu1_csi0: ipu1csi0grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
> +				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
> +				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
> +				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
> +				MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
> +			>;
> +		};
> +
>  		pinctrl_j15: j15grp {
>  			fsl,pins = <
>  				MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
> @@ -445,6 +535,22 @@
>  			>;
>  		};
>  
> +		pinctrl_ov5640: ov5640grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x000b0
> +				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
> +			>;
> +		};
> +
> +		pinctrl_ov5642: ov5642grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
> +				MX6QDL_PAD_GPIO_6__GPIO1_IO06   0x1b0b0
> +				MX6QDL_PAD_GPIO_8__GPIO1_IO08   0x130b0
> +				MX6QDL_PAD_GPIO_3__CCM_CLKO2    0x000b0
> +			>;
> +		};
> +
>  		pinctrl_pwm1: pwm1grp {
>  			fsl,pins = <
>  				MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
> @@ -601,3 +707,15 @@
>  	vmmc-supply = <&reg_3p3v>;
>  	status = "okay";
>  };
> +
> +&mipi_csi {
> +        status = "okay";
> +};
> +
> +/* Incoming port from sensor */
> +&mipi_csi_from_mipi_sensor {
> +        remote-endpoint = <&ov5640_to_mipi_csi>;
> +        data-lanes = <0 1>;
> +        clock-lanes = <2>;
> +};
> +

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-13 12:03     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 12:03 UTC (permalink / raw)
  To: linux-arm-kernel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> Both hang off the same i2c2 bus, so they require different (and non-
> default) i2c slave addresses.
> 
> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
> 
> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
>  arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
>  arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
>  3 files changed, 129 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> index 0f06ca5..fec2524 100644
> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> @@ -48,3 +48,8 @@
>  	model = "Freescale i.MX6 DualLite SABRE Lite Board";
>  	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
>  };
> +
> +&ipu1_csi1_from_ipu1_csi1_mux {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
> index 66d10d8..9e2d26d 100644
> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
> @@ -52,3 +52,9 @@
>  &sata {
>  	status = "okay";
>  };
> +
> +&ipu1_csi1_from_mipi_vc1 {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> +
> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> index 795b5a5..bca9fed 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> @@ -39,6 +39,8 @@
>   *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>   *     OTHER DEALINGS IN THE SOFTWARE.
>   */
> +
> +#include <dt-bindings/clock/imx6qdl-clock.h>
>  #include <dt-bindings/gpio/gpio.h>
>  #include <dt-bindings/input/input.h>
>  
> @@ -96,6 +98,15 @@
>  		};
>  	};
>  
> +	mipi_xclk: mipi_xclk {
> +		compatible = "pwm-clock";
> +		#clock-cells = <0>;
> +		clock-frequency = <22000000>;
> +		clock-output-names = "mipi_pwm3";
> +		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
> +		status = "okay";
> +	};
> +
>  	gpio-keys {
>  		compatible = "gpio-keys";
>  		pinctrl-names = "default";
> @@ -220,6 +231,22 @@
>  	};
>  };
>  
> +&ipu1_csi0_from_ipu1_csi0_mux {
> +	bus-width = <8>;
> +	data-shift = <12>; /* Lines 19:12 used */
> +	hsync-active = <1>;
> +	vync-active = <1>;
> +};
> +
> +&ipu1_csi0_mux_from_parallel_sensor {
> +	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
> +};
> +
> +&ipu1_csi0 {
> +	pinctrl-names = "default";
> +	pinctrl-0 = <&pinctrl_ipu1_csi0>;
> +};
> +
>  &audmux {
>  	pinctrl-names = "default";
>  	pinctrl-0 = <&pinctrl_audmux>;
> @@ -299,6 +326,52 @@
>  	pinctrl-names = "default";
>  	pinctrl-0 = <&pinctrl_i2c2>;
>  	status = "okay";
> +
> +	ov5640: camera at 40 {
> +		compatible = "ovti,ov5640";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5640>;
> +		clocks = <&mipi_xclk>;
> +		clock-names = "xclk";
> +		reg = <0x40>;
> +		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

> +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> +
> +		port {
> +			#address-cells = <1>;
> +			#size-cells = <0>;
> +
> +			ov5640_to_mipi_csi: endpoint at 1 {
> +				reg = <1>;
> +				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
> +				data-lanes = <0 1>;
> +				clock-lanes = <2>;
> +			};
> +		};
> +	};
> +
> +	ov5642: camera at 42 {
> +		compatible = "ovti,ov5642";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5642>;
> +		clocks = <&clks IMX6QDL_CLK_CKO2>;
> +		clock-names = "xclk";
> +		reg = <0x42>;
> +		xclk = <24000000>;

Same here, use assigned-clock-rates on IMX6QDL_CLK_CKO2 if necessary.

> +		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
> +		pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
> +		gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
> +
> +		port {
> +			ov5642_to_ipu1_csi0_mux: endpoint {
> +				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
> +				bus-width = <8>;
> +				hsync-active = <1>;
> +				vsync-active = <1>;
> +			};
> +		};
> +	};
>  };
>  
>  &i2c3 {
> @@ -412,6 +485,23 @@
>  			>;
>  		};
>  
> +		pinctrl_ipu1_csi0: ipu1csi0grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
> +				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
> +				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
> +				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
> +				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
> +				MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
> +			>;
> +		};
> +
>  		pinctrl_j15: j15grp {
>  			fsl,pins = <
>  				MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
> @@ -445,6 +535,22 @@
>  			>;
>  		};
>  
> +		pinctrl_ov5640: ov5640grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x000b0
> +				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
> +			>;
> +		};
> +
> +		pinctrl_ov5642: ov5642grp {
> +			fsl,pins = <
> +				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
> +				MX6QDL_PAD_GPIO_6__GPIO1_IO06   0x1b0b0
> +				MX6QDL_PAD_GPIO_8__GPIO1_IO08   0x130b0
> +				MX6QDL_PAD_GPIO_3__CCM_CLKO2    0x000b0
> +			>;
> +		};
> +
>  		pinctrl_pwm1: pwm1grp {
>  			fsl,pins = <
>  				MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
> @@ -601,3 +707,15 @@
>  	vmmc-supply = <&reg_3p3v>;
>  	status = "okay";
>  };
> +
> +&mipi_csi {
> +        status = "okay";
> +};
> +
> +/* Incoming port from sensor */
> +&mipi_csi_from_mipi_sensor {
> +        remote-endpoint = <&ov5640_to_mipi_csi>;
> +        data-lanes = <0 1>;
> +        clock-lanes = <2>;
> +};
> +

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 15/24] media: Add userspace header file for i.MX
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-13 12:05     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 12:05 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> This adds a header file for use by userspace programs wanting to interact
> with the i.MX media driver. It defines custom v4l2 controls and events
> generated by the i.MX v4l2 subdevices.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  include/uapi/media/Kbuild |  1 +
>  include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
>  2 files changed, 31 insertions(+)
>  create mode 100644 include/uapi/media/imx.h
> 
> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
> index aafaa5a..fa78958 100644
> --- a/include/uapi/media/Kbuild
> +++ b/include/uapi/media/Kbuild
> @@ -1 +1,2 @@
>  # UAPI Header export list
> +header-y += imx.h
> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
> new file mode 100644
> index 0000000..2421d9c
> --- /dev/null
> +++ b/include/uapi/media/imx.h
> @@ -0,0 +1,30 @@
> +/*
> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by the
> + * Free Software Foundation; either version 2 of the
> + * License, or (at your option) any later version
> + */
> +
> +#ifndef __UAPI_MEDIA_IMX_H__
> +#define __UAPI_MEDIA_IMX_H__
> +
> +/*
> + * events from the subdevs
> + */
> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)

Aren't these generic enough to warrant common events? I would think
there have to be other capture IP cores that can signal aborted frames
or frame timeouts.

> +
> +enum imx_ctrl_id {
> +	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
> +	V4L2_CID_IMX_FIM_ENABLE,
> +	V4L2_CID_IMX_FIM_NUM,
> +	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
> +	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
> +	V4L2_CID_IMX_FIM_NUM_SKIP,
> +};
> +
> +#endif

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-01-13 12:05     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 12:05 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> This adds a header file for use by userspace programs wanting to interact
> with the i.MX media driver. It defines custom v4l2 controls and events
> generated by the i.MX v4l2 subdevices.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  include/uapi/media/Kbuild |  1 +
>  include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
>  2 files changed, 31 insertions(+)
>  create mode 100644 include/uapi/media/imx.h
> 
> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
> index aafaa5a..fa78958 100644
> --- a/include/uapi/media/Kbuild
> +++ b/include/uapi/media/Kbuild
> @@ -1 +1,2 @@
>  # UAPI Header export list
> +header-y += imx.h
> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
> new file mode 100644
> index 0000000..2421d9c
> --- /dev/null
> +++ b/include/uapi/media/imx.h
> @@ -0,0 +1,30 @@
> +/*
> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by the
> + * Free Software Foundation; either version 2 of the
> + * License, or (at your option) any later version
> + */
> +
> +#ifndef __UAPI_MEDIA_IMX_H__
> +#define __UAPI_MEDIA_IMX_H__
> +
> +/*
> + * events from the subdevs
> + */
> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)

Aren't these generic enough to warrant common events? I would think
there have to be other capture IP cores that can signal aborted frames
or frame timeouts.

> +
> +enum imx_ctrl_id {
> +	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
> +	V4L2_CID_IMX_FIM_ENABLE,
> +	V4L2_CID_IMX_FIM_NUM,
> +	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
> +	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
> +	V4L2_CID_IMX_FIM_NUM_SKIP,
> +};
> +
> +#endif

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-01-13 12:05     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 12:05 UTC (permalink / raw)
  To: linux-arm-kernel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> This adds a header file for use by userspace programs wanting to interact
> with the i.MX media driver. It defines custom v4l2 controls and events
> generated by the i.MX v4l2 subdevices.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  include/uapi/media/Kbuild |  1 +
>  include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
>  2 files changed, 31 insertions(+)
>  create mode 100644 include/uapi/media/imx.h
> 
> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
> index aafaa5a..fa78958 100644
> --- a/include/uapi/media/Kbuild
> +++ b/include/uapi/media/Kbuild
> @@ -1 +1,2 @@
>  # UAPI Header export list
> +header-y += imx.h
> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
> new file mode 100644
> index 0000000..2421d9c
> --- /dev/null
> +++ b/include/uapi/media/imx.h
> @@ -0,0 +1,30 @@
> +/*
> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by the
> + * Free Software Foundation; either version 2 of the
> + * License, or (at your option) any later version
> + */
> +
> +#ifndef __UAPI_MEDIA_IMX_H__
> +#define __UAPI_MEDIA_IMX_H__
> +
> +/*
> + * events from the subdevs
> + */
> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)

Aren't these generic enough to warrant common events? I would think
there have to be other capture IP cores that can signal aborted frames
or frame timeouts.

> +
> +enum imx_ctrl_id {
> +	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
> +	V4L2_CID_IMX_FIM_ENABLE,
> +	V4L2_CID_IMX_FIM_NUM,
> +	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
> +	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
> +	V4L2_CID_IMX_FIM_NUM_SKIP,
> +};
> +
> +#endif

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-07  2:11 ` [PATCH v3 16/24] media: Add i.MX media core driver Steve Longerbeam
  2017-01-13 15:20     ` Philipp Zabel
@ 2017-01-13 15:20     ` Philipp Zabel
  2017-02-02 22:44     ` Russell King - ARM Linux
  2017-02-02 22:50     ` Russell King - ARM Linux
  3 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 15:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add the core media driver for i.MX SOC.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  Documentation/media/v4l-drivers/imx.rst           | 443 ++++++++++
>  drivers/staging/media/Kconfig                     |   2 +
>  drivers/staging/media/Makefile                    |   1 +
>  drivers/staging/media/imx/Kconfig                 |   8 +
>  drivers/staging/media/imx/Makefile                |   6 +
>  drivers/staging/media/imx/TODO                    |  22 +
>  drivers/staging/media/imx/imx-media-common.c      | 981 ++++++++++++++++++++++
>  drivers/staging/media/imx/imx-media-dev.c         | 486 +++++++++++
>  drivers/staging/media/imx/imx-media-fim.c         | 471 +++++++++++
>  drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
>  drivers/staging/media/imx/imx-media-of.c          | 289 +++++++
>  drivers/staging/media/imx/imx-media.h             | 310 +++++++
>  include/media/imx.h                               |  15 +
>  include/uapi/linux/v4l2-controls.h                |   4 +
>  14 files changed, 3495 insertions(+)
>  create mode 100644 Documentation/media/v4l-drivers/imx.rst
>  create mode 100644 drivers/staging/media/imx/Kconfig
>  create mode 100644 drivers/staging/media/imx/Makefile
>  create mode 100644 drivers/staging/media/imx/TODO
>  create mode 100644 drivers/staging/media/imx/imx-media-common.c
>  create mode 100644 drivers/staging/media/imx/imx-media-dev.c
>  create mode 100644 drivers/staging/media/imx/imx-media-fim.c
>  create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
>  create mode 100644 drivers/staging/media/imx/imx-media-of.c
>  create mode 100644 drivers/staging/media/imx/imx-media.h
>  create mode 100644 include/media/imx.h
> 
> diff --git a/Documentation/media/v4l-drivers/imx.rst b/Documentation/media/v4l-drivers/imx.rst
> new file mode 100644
> index 0000000..87b37b5
> --- /dev/null
> +++ b/Documentation/media/v4l-drivers/imx.rst
> @@ -0,0 +1,443 @@
> +i.MX Video Capture Driver
> +=========================
> +
> +Introduction
> +------------
> +
> +The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
> +handles the flow of image frames to and from capture devices and
> +display devices.
> +
> +For image capture, the IPU contains the following internal subunits:
> +
> +- Image DMA Controller (IDMAC)
> +- Camera Serial Interface (CSI)
> +- Image Converter (IC)
> +- Sensor Multi-FIFO Controller (SMFC)
> +- Image Rotator (IRT)
> +- Video De-Interlace Controller (VDIC)

Nitpick: Video De-Interlacing or Combining Block (VDIC)

> +
> +The IDMAC is the DMA controller for transfer of image frames to and from
> +memory. Various dedicated DMA channels exist for both video capture and
> +display paths.
> +
> +The CSI is the frontend capture unit that interfaces directly with
> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
> +
> +The IC handles color-space conversion, resizing, and rotation
> +operations. 

And horizontal flipping.

> There are three independent "tasks" within the IC that can
> +carry out conversions concurrently: pre-processing encoding,
> +pre-processing preview, and post-processing.

s/preview/viewfinder/ seems to be the commonly used name.

This paragraph could mention that a single hardware unit is used
transparently time multiplexed by the three tasks at different
granularity for the downsizing, main processing, and rotation sections.
The downscale unit switches between tasks at 8-pixel burst granularity,
the main processing unit at line granularity. The rotation units switch
only at frame granularity.

> +The SMFC is composed of four independent channels that each can transfer
> +captured frames from sensors directly to memory concurrently.
> +
> +The IRT carries out 90 and 270 degree image rotation operations.

... on 8x8 pixel blocks, supported by the IDMAC which handles block
transfers, block reordering, and vertical flipping.

> +The VDIC handles the conversion of interlaced video to progressive, with
> +support for different motion compensation modes (low, medium, and high
> +motion). The deinterlaced output frames from the VDIC can be sent to the
> +IC pre-process preview task for further conversions.
> +
> +In addition to the IPU internal subunits, there are also two units
> +outside the IPU that are also involved in video capture on i.MX:
> +
> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
> +  interface. This is a Synopsys DesignWare core.
> +- A video multiplexer for selecting among multiple sensor inputs to
> +  send to a CSI.

Two of them, actually.

> +For more info, refer to the latest versions of the i.MX5/6 reference
> +manuals listed under References.
> +
> +
> +Features
> +--------
> +
> +Some of the features of this driver include:
> +
> +- Many different pipelines can be configured via media controller API,
> +  that correspond to the hardware video capture pipelines supported in
> +  the i.MX.
> +
> +- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
> +
> +- Up to four concurrent sensor acquisitions, by configuring each
> +  sensor's pipeline using independent entities. This is currently
> +  demonstrated with the SabreSD and SabreLite reference boards with
> +  independent OV5642 and MIPI CSI-2 OV5640 sensor modules.
> +
> +- Scaling, color-space conversion, and image rotation via IC task
> +  subdevs.
> +
> +- Many pixel formats supported (RGB, packed and planar YUV, partial
> +  planar YUV).
> +
> +- The IC pre-process preview subdev supports motion compensated
> +  de-interlacing using the VDIC, with three motion compensation modes:
> +  low, medium, and high motion. The mode is specified with a custom
> +  control. Pipelines are defined that allow sending frames to the
> +  preview subdev directly from the CSI or from the SMFC.
> +
> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
> +  problems with the ADV718x video decoders. See below for a description
> +  of the FIM.

Could this also be used to calculate more precise capture timestamps?

> +Capture Pipelines
> +-----------------
> +
> +The following describe the various use-cases supported by the pipelines.
> +
> +The links shown do not include the frontend sensor, video mux, or mipi
> +csi-2 receiver links. This depends on the type of sensor interface
> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
> +
> +sensor -> ipu_csi_mux -> ipu_csi -> ...
> +
> +for parallel sensors, or:
> +
> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
> +
> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
> +parenthesis.
> +
> +Unprocessed Video Capture:
> +--------------------------
> +
> +Send frames directly from sensor to camera interface, with no
> +conversions:
> +
> +-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe
idmac if you want to mirror hardware names?

> +Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

> +For example, its sink pad can take UYVY2X8, but its source pad can
> +output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
in the CSI chapter, for 8-bit per component data, the internal format
between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
(or "bayer/generic data"). In either case, the internal format does not
change along the way.

> +IC Direct Conversions:
> +----------------------
> +
> +This pipeline uses the preprocess encode entity to route frames directly
> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
> +1024x1024 resolution, CSC, and image rotation:
> +
> +-> ipu_ic_prpenc -> camif
> +
> +This can be a useful capture pipeline for heavily loaded memory bus
> +traffic environments, since it has minimal IDMAC channel usage.

Note that if rotation is enabled, transfers between IC processing and
rotation still have to go through memory once.

> +Post-Processing Conversions:
> +----------------------------
> +
> +This pipeline routes frames from the SMFC to the post-processing
> +entity.

No, frames written by the CSI -> SMFC -> IDMAC path are read back into
the post-processing entity.

>  In addition to CSC and rotation, this entity supports tiling
> +which allows scaled output beyond the 1024x1024 limitation of the IC
> +(up to 4096x4096 scaling output is supported):
> +
> +-> ipu_smfc -> ipu_ic_pp -> camif
> +
> +Motion Compensated De-interlace:
> +--------------------------------
> +
> +This pipeline routes frames from the SMFC to the preprocess preview
> +entity to support motion-compensated de-interlacing using the VDIC,
> +scaling up to 1024x1024, and CSC:
> +
> +-> ipu_smfc -> ipu_ic_prpvf -> camif

Same as above.

> +This pipeline also carries out the same conversions as above, but routes
> +frames directly from the CSI to the IC preprocess preview entity for
> +minimal memory bandwidth usage (note: this pipeline only works in
> +"high motion" mode):
> +
> +-> ipu_ic_prpvf -> camif
> +
> +This pipeline takes the motion-compensated de-interlaced frames and
> +sends them to the post-processor, to support motion-compensated
> +de-interlacing, scaling up to 4096x4096, CSC, and rotation:
> +
> +-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif
> +
> +
> +Usage Notes
> +-----------
[...]
> +SabreLite with OV5642 and OV5640
> +--------------------------------
> +
> +This platform requires the OmniVision OV5642 module with a parallel
> +camera interface, and the OV5640 module with a MIPI CSI-2
> +interface. Both modules are available from Boundary Devices:
> +
> +https://boundarydevices.com/products/nit6x_5mp
> +https://boundarydevices.com/product/nit6x_5mp_mipi
> +
> +Note that if only one camera module is available, the other sensor
> +node can be disabled in the device tree.
> +
> +The OV5642 module is connected to the parallel bus input on the i.MX
> +internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
> +
> +The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
> +receiver, and the four virtual channel outputs from the receiver are
> +routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
> +vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
> +also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
> +OV5640 must not share the same i2c slave address.
> +
> +The following basic example configures unprocessed video capture
> +pipelines for both sensors. The OV5642 is routed to camif0
> +(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
> +virtual channel 1) is routed to camif1 (usually /dev/video1). Both
> +sensors are configured to output 640x480, UYVY (not shown: all pad
> +field types should be set to "NONE"):
> +
> +.. code-block:: none
> +
> +   # Setup links for OV5642
> +   media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> +   media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> +   # Setup links for OV5640
> +   media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
> +   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
> +   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
> +   media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
> +   media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
> +   # Configure pads for OV5642 pipeline
> +   media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"

I think the smfc entities should be dropped.

> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> +   # Configure pads for OV5640 pipeline
> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
[...]
> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the
csi1:1 and camif1:0 pads would be a 32-bit YUV format.
(MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format as that will allow
to chose the regular vs. companded packings in the future for formats
with more than 8 bits per component.

> +   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>+
> +Streaming can then begin independently on device nodes /dev/video0
> +and /dev/video1.
> +
> +SabreAuto with ADV7180 decoder
> +------------------------------
> +
> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
> +parallel bus input on the internal video mux to IPU1 CSI0.
> +
> +The following example configures a pipeline to capture from the ADV7180
> +video decoder, assuming NTSC 720x480 input signals, with Motion
> +Compensated de-interlacing (not shown: all pad field types should be set
> +as indicated). $outputfmt can be any format supported by the
> +ipu1_ic_prpvf entity at its output pad:
> +
> +.. code-block:: none
> +
> +   # Setup links
> +   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> +   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
> +   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> +   # Configure pads
> +   # pad field types for below pads must be an interlaced type
> +   # such as "ALTERNATE"

I think alternate should only extend as far as the CSI, since the CSI
can only capture NTSC/PAL fields in a fixed order.

> +   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"

>From here the interlaced field type should be sequential in the correct
order depending on NTSC/PAL.

> +   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
> +   # pad field types for below pads must be "NONE"
> +   media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
> +   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
> +   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
> +
> +Streaming can then begin on /dev/video0.
> +
> +This platform accepts Composite Video analog inputs to the ADV7180 on
> +Ain1 (connector J42) and Ain3 (connector J43).
> +
> +To switch to Ain1:
> +
> +.. code-block:: none
> +
> +   # v4l2-ctl -i0
> +
> +To switch to Ain3:
> +
> +.. code-block:: none
> +
> +   # v4l2-ctl -i1
> +
> +
> +Frame Interval Monitor
> +----------------------
> +
> +The adv718x decoders can occasionally send corrupt fields during
> +NTSC/PAL signal re-sync (too little or too many video lines). When
> +this happens, the IPU triggers a mechanism to re-establish vertical
> +sync by adding 1 dummy line every frame, which causes a rolling effect
> +from image to image, and can last a long time before a stable image is
> +recovered. Or sometimes the mechanism doesn't work at all, causing a
> +permanent split image (one frame contains lines from two consecutive
> +captured images).

Is it only SabreAuto on which the FIM mechanism can be used due to the
pad routing?

[...]
> +/*
> + * DMA buffer ring handling
> + */
> +struct imx_media_dma_buf_ring {
> +	struct imx_media_dev *imxmd;
> +
> +	/* the ring */
> +	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
> +	/* the scratch buffer for underruns */
> +	struct imx_media_dma_buf scratch;
> +
> +	/* buffer generator */
> +	struct media_entity *src;
> +	/* buffer receiver */
> +	struct media_entity *sink;
> +
> +	spinlock_t lock;
> +
> +	int num_bufs;
> +	unsigned long last_seq;
> +};

I don't think this belongs in the capture driver at all.
Memory-to-memory transfers should be handled at the videobuf2 level.

[...]
> +static struct imx_media_dma_buf *
> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
> +{
> +	struct imx_media_dma_buf *buf;
> +
> +	if (index >= ring->num_bufs)
> +		return ERR_PTR(-EINVAL);
> +
> +	buf = &ring->buf[index];
> +	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
> +		return ERR_PTR(-EINVAL);
> +
> +	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
> +	buf->seq = ring->last_seq++;
> +
> +	return buf;
> +}

Is this a whole software buffer queue implementation? I thought the
whole point of putting the custom mem2mem framework into the capture
driver was to use the hardware FSU channel linking?

> +int imx_media_dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
> +{
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	buf = __dma_buf_queue(ring, index);
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	if (IS_ERR(buf))
> +		return PTR_ERR(buf);
> +
> +	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued\n",
> +		index, ring->src->name, ring->sink->name);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue);
> +
> +int imx_media_dma_buf_queue_from_vb(struct imx_media_dma_buf_ring *ring,
> +				    struct vb2_buffer *vb)
> +{
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +	dma_addr_t phys;
> +	void *virt;
> +
> +	if (vb->index >= ring->num_bufs)
> +		return -EINVAL;
> +
> +	virt = vb2_plane_vaddr(vb, 0);
> +	phys = vb2_dma_contig_plane_dma_addr(vb, 0);
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	buf = __dma_buf_queue(ring, vb->index);
> +	if (IS_ERR(buf))
> +		goto err_unlock;
> +
> +	buf->virt = virt;
> +	buf->phys = phys;
> +	buf->vb = vb;
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued from vb\n",
> +		buf->index, ring->src->name, ring->sink->name);
> +
> +	return 0;
> +err_unlock:
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +	return PTR_ERR(buf);
> +}
> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue_from_vb);
> +
> +void imx_media_dma_buf_done(struct imx_media_dma_buf *buf,
> +			    enum imx_media_dma_buf_status status)
> +{
> +	struct imx_media_dma_buf_ring *ring = buf->ring;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_ACTIVE);
> +	buf->state = buf->status = status;
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	if (buf == &ring->scratch)
> +		dev_dbg(ring->imxmd->dev, "buf-scratch [%s -> %s] done\n",
> +			ring->src->name, ring->sink->name);
> +	else
> +		dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] done\n",
> +			buf->index, ring->src->name, ring->sink->name);
> +
> +	/* if the sink is a subdev, inform it that new buffers are available */
> +	if (is_media_entity_v4l2_subdev(ring->sink)) {
> +		struct v4l2_subdev *sd =
> +			media_entity_to_v4l2_subdev(ring->sink);
> +		v4l2_subdev_call(sd, core, ioctl, IMX_MEDIA_NEW_DMA_BUF, NULL);

What is the purpose of this if the sink should be triggered by the FSU?

[...]
> +/*
> + * The subdevs have to be powered on/off, and streaming
> + * enabled/disabled, in a specific sequence.
> + */
> +static const u32 stream_on_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI,
> +};
> +
> +static const u32 stream_off_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +};
> +
> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
> +
> +static const u32 power_on_seq[] = {
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +};
> +
> +static const u32 power_off_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +};

This seems somewhat arbitrary. Why is a power sequence needed?

[...]
> +/*
> + * Turn current pipeline power on/off starting from start_entity.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
> +				 struct media_entity_graph *graph,
> +				 struct media_entity *start_entity, bool on)
> +{
> +	struct media_entity *entity;
> +	struct v4l2_subdev *sd;
> +	int i, ret = 0;
> +	u32 id;
> +
> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
> +		id = on ? power_on_seq[i] : power_off_seq[i];
> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
> +		if (!entity)
> +			continue;
> +
> +		sd = media_entity_to_v4l2_subdev(entity);
> +
> +		ret = v4l2_subdev_call(sd, core, s_power, on);
> +		if (ret && ret != -ENOIOCTLCMD)
> +			break;
> +	}
> +
> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

> +/*
> + * Inherit the v4l2 controls from all entities in a pipeline
> + * to the given video device.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
> +			       struct video_device *vfd,
> +			       struct media_entity *start_entity)
> +{
> +	struct media_entity_graph graph;
> +	struct media_entity *entity;
> +	struct v4l2_subdev *sd;
> +	int ret;
> +
> +	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
> +	if (ret)
> +		return ret;
> +
> +	media_entity_graph_walk_start(&graph, start_entity);
> +
> +	while ((entity = media_entity_graph_walk_next(&graph))) {
> +		if (is_media_entity_v4l2_video_device(entity))
> +			continue;
> +
> +		sd = media_entity_to_v4l2_subdev(entity);
> +
> +		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
> +			__func__, sd->name);
> +
> +		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
> +					    sd->ctrl_handler,
> +					    NULL);
> +		if (ret)
> +			break;
> +	}
> +
> +	media_entity_graph_walk_cleanup(&graph);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
> +
> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
> new file mode 100644
> index 0000000..357654d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-dev.c

This file is full of code that should live in the v4l2 core.

[...]
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-internal-sd.c
[...]
> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> +				   struct imx_media_subdev *csi[4])
> +{
> +	int ret;
> +
> +	/* there must be at least one CSI in first IPU */

Why?

> +	if (!(csi[0] || csi[1]))
> +		return -EINVAL;
> +
> +	ret = add_ipu_internal_subdevs(imxmd, csi[0], csi[1], 0);
> +	if (ret)
> +		return ret;
> +
> +	if (csi[2] || csi[3])
> +		ret = add_ipu_internal_subdevs(imxmd, csi[2], csi[3], 1);
> +
> +	return ret;
> +}
> diff --git a/drivers/staging/media/imx/imx-media-of.c b/drivers/staging/media/imx/imx-media-of.c
> new file mode 100644
> index 0000000..a939c34
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-of.c
> @@ -0,0 +1,289 @@
> +/*
> + * Media driver for Freescale i.MX5/6 SOC
> + *
> + * Open Firmware parsing.
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/of_platform.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +static int of_add_pad_link(struct imx_media_dev *imxmd,
> +			   struct imx_media_pad *pad,
> +			   struct device_node *local_sd_node,
> +			   struct device_node *remote_sd_node,
> +			   int local_pad, int remote_pad)
> +{
> +	dev_dbg(imxmd->dev, "%s: adding %s:%d -> %s:%d\n", __func__,
> +		local_sd_node->name, local_pad,
> +		remote_sd_node->name, remote_pad);
> +
> +	return imx_media_add_pad_link(imxmd, pad, remote_sd_node, NULL,
> +				      local_pad, remote_pad);
> +}
> +
> +/* parse inputs property from a sensor node */
> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> +				   struct imx_media_subdev *sensor,
> +				   struct device_node *sensor_np)
> +{
> +	struct imx_media_sensor_input *sinput = &sensor->input;
> +	int ret, i;
> +
> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> +		const char *input_name;
> +		u32 val;
> +
> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> +		if (ret)
> +			break;
> +
> +		sinput->value[i] = val;
> +
> +		ret = of_property_read_string_index(sensor_np, "input-names",
> +						    i, &input_name);
> +		/*
> +		 * if input-names not provided, they will be set using
> +		 * the subdev name once the sensor is known during
> +		 * async bind
> +		 */
> +		if (!ret)
> +			strncpy(sinput->name[i], input_name,
> +				sizeof(sinput->name[i]));
> +	}
> +
> +	sinput->num = i;
> +
> +	/* if no inputs provided just assume a single input */
> +	if (sinput->num == 0)
> +		sinput->num = 1;
> +}

This should be parsed by the sensor driver, not imx-media.

> +static void of_parse_sensor(struct imx_media_dev *imxmd,
> +			    struct imx_media_subdev *sensor,
> +			    struct device_node *sensor_np)
> +{
> +	struct device_node *endpoint;
> +
> +	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
> +
> +	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
> +	if (endpoint) {
> +		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
> +		of_node_put(endpoint);
> +	}
> +}
> +
> +static int of_get_port_count(const struct device_node *np)
> +{
> +	struct device_node *child;
> +	int num = 0;
> +
> +	/* if this node is itself a port, return 1 */
> +	if (of_node_cmp(np->name, "port") == 0)
> +		return 1;
> +
> +	for_each_child_of_node(np, child)
> +		if (of_node_cmp(child->name, "port") == 0)
> +			num++;
> +
> +	return num;
> +}

If this is extended to handle the ports subnode properly, it could be
moved into drivers/of/base.c.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-13 15:20     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 15:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add the core media driver for i.MX SOC.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  Documentation/media/v4l-drivers/imx.rst           | 443 ++++++++++
>  drivers/staging/media/Kconfig                     |   2 +
>  drivers/staging/media/Makefile                    |   1 +
>  drivers/staging/media/imx/Kconfig                 |   8 +
>  drivers/staging/media/imx/Makefile                |   6 +
>  drivers/staging/media/imx/TODO                    |  22 +
>  drivers/staging/media/imx/imx-media-common.c      | 981 ++++++++++++++++++++++
>  drivers/staging/media/imx/imx-media-dev.c         | 486 +++++++++++
>  drivers/staging/media/imx/imx-media-fim.c         | 471 +++++++++++
>  drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
>  drivers/staging/media/imx/imx-media-of.c          | 289 +++++++
>  drivers/staging/media/imx/imx-media.h             | 310 +++++++
>  include/media/imx.h                               |  15 +
>  include/uapi/linux/v4l2-controls.h                |   4 +
>  14 files changed, 3495 insertions(+)
>  create mode 100644 Documentation/media/v4l-drivers/imx.rst
>  create mode 100644 drivers/staging/media/imx/Kconfig
>  create mode 100644 drivers/staging/media/imx/Makefile
>  create mode 100644 drivers/staging/media/imx/TODO
>  create mode 100644 drivers/staging/media/imx/imx-media-common.c
>  create mode 100644 drivers/staging/media/imx/imx-media-dev.c
>  create mode 100644 drivers/staging/media/imx/imx-media-fim.c
>  create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
>  create mode 100644 drivers/staging/media/imx/imx-media-of.c
>  create mode 100644 drivers/staging/media/imx/imx-media.h
>  create mode 100644 include/media/imx.h
> 
> diff --git a/Documentation/media/v4l-drivers/imx.rst b/Documentation/media/v4l-drivers/imx.rst
> new file mode 100644
> index 0000000..87b37b5
> --- /dev/null
> +++ b/Documentation/media/v4l-drivers/imx.rst
> @@ -0,0 +1,443 @@
> +i.MX Video Capture Driver
> +=========================
> +
> +Introduction
> +------------
> +
> +The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
> +handles the flow of image frames to and from capture devices and
> +display devices.
> +
> +For image capture, the IPU contains the following internal subunits:
> +
> +- Image DMA Controller (IDMAC)
> +- Camera Serial Interface (CSI)
> +- Image Converter (IC)
> +- Sensor Multi-FIFO Controller (SMFC)
> +- Image Rotator (IRT)
> +- Video De-Interlace Controller (VDIC)

Nitpick: Video De-Interlacing or Combining Block (VDIC)

> +
> +The IDMAC is the DMA controller for transfer of image frames to and from
> +memory. Various dedicated DMA channels exist for both video capture and
> +display paths.
> +
> +The CSI is the frontend capture unit that interfaces directly with
> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
> +
> +The IC handles color-space conversion, resizing, and rotation
> +operations. 

And horizontal flipping.

> There are three independent "tasks" within the IC that can
> +carry out conversions concurrently: pre-processing encoding,
> +pre-processing preview, and post-processing.

s/preview/viewfinder/ seems to be the commonly used name.

This paragraph could mention that a single hardware unit is used
transparently time multiplexed by the three tasks at different
granularity for the downsizing, main processing, and rotation sections.
The downscale unit switches between tasks at 8-pixel burst granularity,
the main processing unit at line granularity. The rotation units switch
only at frame granularity.

> +The SMFC is composed of four independent channels that each can transfer
> +captured frames from sensors directly to memory concurrently.
> +
> +The IRT carries out 90 and 270 degree image rotation operations.

... on 8x8 pixel blocks, supported by the IDMAC which handles block
transfers, block reordering, and vertical flipping.

> +The VDIC handles the conversion of interlaced video to progressive, with
> +support for different motion compensation modes (low, medium, and high
> +motion). The deinterlaced output frames from the VDIC can be sent to the
> +IC pre-process preview task for further conversions.
> +
> +In addition to the IPU internal subunits, there are also two units
> +outside the IPU that are also involved in video capture on i.MX:
> +
> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
> +  interface. This is a Synopsys DesignWare core.
> +- A video multiplexer for selecting among multiple sensor inputs to
> +  send to a CSI.

Two of them, actually.

> +For more info, refer to the latest versions of the i.MX5/6 reference
> +manuals listed under References.
> +
> +
> +Features
> +--------
> +
> +Some of the features of this driver include:
> +
> +- Many different pipelines can be configured via media controller API,
> +  that correspond to the hardware video capture pipelines supported in
> +  the i.MX.
> +
> +- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
> +
> +- Up to four concurrent sensor acquisitions, by configuring each
> +  sensor's pipeline using independent entities. This is currently
> +  demonstrated with the SabreSD and SabreLite reference boards with
> +  independent OV5642 and MIPI CSI-2 OV5640 sensor modules.
> +
> +- Scaling, color-space conversion, and image rotation via IC task
> +  subdevs.
> +
> +- Many pixel formats supported (RGB, packed and planar YUV, partial
> +  planar YUV).
> +
> +- The IC pre-process preview subdev supports motion compensated
> +  de-interlacing using the VDIC, with three motion compensation modes:
> +  low, medium, and high motion. The mode is specified with a custom
> +  control. Pipelines are defined that allow sending frames to the
> +  preview subdev directly from the CSI or from the SMFC.
> +
> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
> +  problems with the ADV718x video decoders. See below for a description
> +  of the FIM.

Could this also be used to calculate more precise capture timestamps?

> +Capture Pipelines
> +-----------------
> +
> +The following describe the various use-cases supported by the pipelines.
> +
> +The links shown do not include the frontend sensor, video mux, or mipi
> +csi-2 receiver links. This depends on the type of sensor interface
> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
> +
> +sensor -> ipu_csi_mux -> ipu_csi -> ...
> +
> +for parallel sensors, or:
> +
> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
> +
> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
> +parenthesis.
> +
> +Unprocessed Video Capture:
> +--------------------------
> +
> +Send frames directly from sensor to camera interface, with no
> +conversions:
> +
> +-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe
idmac if you want to mirror hardware names?

> +Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

> +For example, its sink pad can take UYVY2X8, but its source pad can
> +output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
in the CSI chapter, for 8-bit per component data, the internal format
between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
(or "bayer/generic data"). In either case, the internal format does not
change along the way.

> +IC Direct Conversions:
> +----------------------
> +
> +This pipeline uses the preprocess encode entity to route frames directly
> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
> +1024x1024 resolution, CSC, and image rotation:
> +
> +-> ipu_ic_prpenc -> camif
> +
> +This can be a useful capture pipeline for heavily loaded memory bus
> +traffic environments, since it has minimal IDMAC channel usage.

Note that if rotation is enabled, transfers between IC processing and
rotation still have to go through memory once.

> +Post-Processing Conversions:
> +----------------------------
> +
> +This pipeline routes frames from the SMFC to the post-processing
> +entity.

No, frames written by the CSI -> SMFC -> IDMAC path are read back into
the post-processing entity.

>  In addition to CSC and rotation, this entity supports tiling
> +which allows scaled output beyond the 1024x1024 limitation of the IC
> +(up to 4096x4096 scaling output is supported):
> +
> +-> ipu_smfc -> ipu_ic_pp -> camif
> +
> +Motion Compensated De-interlace:
> +--------------------------------
> +
> +This pipeline routes frames from the SMFC to the preprocess preview
> +entity to support motion-compensated de-interlacing using the VDIC,
> +scaling up to 1024x1024, and CSC:
> +
> +-> ipu_smfc -> ipu_ic_prpvf -> camif

Same as above.

> +This pipeline also carries out the same conversions as above, but routes
> +frames directly from the CSI to the IC preprocess preview entity for
> +minimal memory bandwidth usage (note: this pipeline only works in
> +"high motion" mode):
> +
> +-> ipu_ic_prpvf -> camif
> +
> +This pipeline takes the motion-compensated de-interlaced frames and
> +sends them to the post-processor, to support motion-compensated
> +de-interlacing, scaling up to 4096x4096, CSC, and rotation:
> +
> +-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif
> +
> +
> +Usage Notes
> +-----------
[...]
> +SabreLite with OV5642 and OV5640
> +--------------------------------
> +
> +This platform requires the OmniVision OV5642 module with a parallel
> +camera interface, and the OV5640 module with a MIPI CSI-2
> +interface. Both modules are available from Boundary Devices:
> +
> +https://boundarydevices.com/products/nit6x_5mp
> +https://boundarydevices.com/product/nit6x_5mp_mipi
> +
> +Note that if only one camera module is available, the other sensor
> +node can be disabled in the device tree.
> +
> +The OV5642 module is connected to the parallel bus input on the i.MX
> +internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
> +
> +The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
> +receiver, and the four virtual channel outputs from the receiver are
> +routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
> +vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
> +also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
> +OV5640 must not share the same i2c slave address.
> +
> +The following basic example configures unprocessed video capture
> +pipelines for both sensors. The OV5642 is routed to camif0
> +(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
> +virtual channel 1) is routed to camif1 (usually /dev/video1). Both
> +sensors are configured to output 640x480, UYVY (not shown: all pad
> +field types should be set to "NONE"):
> +
> +.. code-block:: none
> +
> +   # Setup links for OV5642
> +   media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> +   media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> +   # Setup links for OV5640
> +   media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
> +   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
> +   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
> +   media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
> +   media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
> +   # Configure pads for OV5642 pipeline
> +   media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"

I think the smfc entities should be dropped.

> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> +   # Configure pads for OV5640 pipeline
> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
[...]
> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the
csi1:1 and camif1:0 pads would be a 32-bit YUV format.
(MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format as that will allow
to chose the regular vs. companded packings in the future for formats
with more than 8 bits per component.

> +   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>+
> +Streaming can then begin independently on device nodes /dev/video0
> +and /dev/video1.
> +
> +SabreAuto with ADV7180 decoder
> +------------------------------
> +
> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
> +parallel bus input on the internal video mux to IPU1 CSI0.
> +
> +The following example configures a pipeline to capture from the ADV7180
> +video decoder, assuming NTSC 720x480 input signals, with Motion
> +Compensated de-interlacing (not shown: all pad field types should be set
> +as indicated). $outputfmt can be any format supported by the
> +ipu1_ic_prpvf entity at its output pad:
> +
> +.. code-block:: none
> +
> +   # Setup links
> +   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> +   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
> +   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> +   # Configure pads
> +   # pad field types for below pads must be an interlaced type
> +   # such as "ALTERNATE"

I think alternate should only extend as far as the CSI, since the CSI
can only capture NTSC/PAL fields in a fixed order.

> +   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"

>From here the interlaced field type should be sequential in the correct
order depending on NTSC/PAL.

> +   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
> +   # pad field types for below pads must be "NONE"
> +   media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
> +   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
> +   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
> +
> +Streaming can then begin on /dev/video0.
> +
> +This platform accepts Composite Video analog inputs to the ADV7180 on
> +Ain1 (connector J42) and Ain3 (connector J43).
> +
> +To switch to Ain1:
> +
> +.. code-block:: none
> +
> +   # v4l2-ctl -i0
> +
> +To switch to Ain3:
> +
> +.. code-block:: none
> +
> +   # v4l2-ctl -i1
> +
> +
> +Frame Interval Monitor
> +----------------------
> +
> +The adv718x decoders can occasionally send corrupt fields during
> +NTSC/PAL signal re-sync (too little or too many video lines). When
> +this happens, the IPU triggers a mechanism to re-establish vertical
> +sync by adding 1 dummy line every frame, which causes a rolling effect
> +from image to image, and can last a long time before a stable image is
> +recovered. Or sometimes the mechanism doesn't work at all, causing a
> +permanent split image (one frame contains lines from two consecutive
> +captured images).

Is it only SabreAuto on which the FIM mechanism can be used due to the
pad routing?

[...]
> +/*
> + * DMA buffer ring handling
> + */
> +struct imx_media_dma_buf_ring {
> +	struct imx_media_dev *imxmd;
> +
> +	/* the ring */
> +	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
> +	/* the scratch buffer for underruns */
> +	struct imx_media_dma_buf scratch;
> +
> +	/* buffer generator */
> +	struct media_entity *src;
> +	/* buffer receiver */
> +	struct media_entity *sink;
> +
> +	spinlock_t lock;
> +
> +	int num_bufs;
> +	unsigned long last_seq;
> +};

I don't think this belongs in the capture driver at all.
Memory-to-memory transfers should be handled at the videobuf2 level.

[...]
> +static struct imx_media_dma_buf *
> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
> +{
> +	struct imx_media_dma_buf *buf;
> +
> +	if (index >= ring->num_bufs)
> +		return ERR_PTR(-EINVAL);
> +
> +	buf = &ring->buf[index];
> +	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
> +		return ERR_PTR(-EINVAL);
> +
> +	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
> +	buf->seq = ring->last_seq++;
> +
> +	return buf;
> +}

Is this a whole software buffer queue implementation? I thought the
whole point of putting the custom mem2mem framework into the capture
driver was to use the hardware FSU channel linking?

> +int imx_media_dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
> +{
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	buf = __dma_buf_queue(ring, index);
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	if (IS_ERR(buf))
> +		return PTR_ERR(buf);
> +
> +	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued\n",
> +		index, ring->src->name, ring->sink->name);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue);
> +
> +int imx_media_dma_buf_queue_from_vb(struct imx_media_dma_buf_ring *ring,
> +				    struct vb2_buffer *vb)
> +{
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +	dma_addr_t phys;
> +	void *virt;
> +
> +	if (vb->index >= ring->num_bufs)
> +		return -EINVAL;
> +
> +	virt = vb2_plane_vaddr(vb, 0);
> +	phys = vb2_dma_contig_plane_dma_addr(vb, 0);
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	buf = __dma_buf_queue(ring, vb->index);
> +	if (IS_ERR(buf))
> +		goto err_unlock;
> +
> +	buf->virt = virt;
> +	buf->phys = phys;
> +	buf->vb = vb;
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued from vb\n",
> +		buf->index, ring->src->name, ring->sink->name);
> +
> +	return 0;
> +err_unlock:
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +	return PTR_ERR(buf);
> +}
> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue_from_vb);
> +
> +void imx_media_dma_buf_done(struct imx_media_dma_buf *buf,
> +			    enum imx_media_dma_buf_status status)
> +{
> +	struct imx_media_dma_buf_ring *ring = buf->ring;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_ACTIVE);
> +	buf->state = buf->status = status;
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	if (buf == &ring->scratch)
> +		dev_dbg(ring->imxmd->dev, "buf-scratch [%s -> %s] done\n",
> +			ring->src->name, ring->sink->name);
> +	else
> +		dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] done\n",
> +			buf->index, ring->src->name, ring->sink->name);
> +
> +	/* if the sink is a subdev, inform it that new buffers are available */
> +	if (is_media_entity_v4l2_subdev(ring->sink)) {
> +		struct v4l2_subdev *sd =
> +			media_entity_to_v4l2_subdev(ring->sink);
> +		v4l2_subdev_call(sd, core, ioctl, IMX_MEDIA_NEW_DMA_BUF, NULL);

What is the purpose of this if the sink should be triggered by the FSU?

[...]
> +/*
> + * The subdevs have to be powered on/off, and streaming
> + * enabled/disabled, in a specific sequence.
> + */
> +static const u32 stream_on_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI,
> +};
> +
> +static const u32 stream_off_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +};
> +
> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
> +
> +static const u32 power_on_seq[] = {
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +};
> +
> +static const u32 power_off_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +};

This seems somewhat arbitrary. Why is a power sequence needed?

[...]
> +/*
> + * Turn current pipeline power on/off starting from start_entity.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
> +				 struct media_entity_graph *graph,
> +				 struct media_entity *start_entity, bool on)
> +{
> +	struct media_entity *entity;
> +	struct v4l2_subdev *sd;
> +	int i, ret = 0;
> +	u32 id;
> +
> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
> +		id = on ? power_on_seq[i] : power_off_seq[i];
> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
> +		if (!entity)
> +			continue;
> +
> +		sd = media_entity_to_v4l2_subdev(entity);
> +
> +		ret = v4l2_subdev_call(sd, core, s_power, on);
> +		if (ret && ret != -ENOIOCTLCMD)
> +			break;
> +	}
> +
> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

> +/*
> + * Inherit the v4l2 controls from all entities in a pipeline
> + * to the given video device.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
> +			       struct video_device *vfd,
> +			       struct media_entity *start_entity)
> +{
> +	struct media_entity_graph graph;
> +	struct media_entity *entity;
> +	struct v4l2_subdev *sd;
> +	int ret;
> +
> +	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
> +	if (ret)
> +		return ret;
> +
> +	media_entity_graph_walk_start(&graph, start_entity);
> +
> +	while ((entity = media_entity_graph_walk_next(&graph))) {
> +		if (is_media_entity_v4l2_video_device(entity))
> +			continue;
> +
> +		sd = media_entity_to_v4l2_subdev(entity);
> +
> +		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
> +			__func__, sd->name);
> +
> +		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
> +					    sd->ctrl_handler,
> +					    NULL);
> +		if (ret)
> +			break;
> +	}
> +
> +	media_entity_graph_walk_cleanup(&graph);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
> +
> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
> new file mode 100644
> index 0000000..357654d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-dev.c

This file is full of code that should live in the v4l2 core.

[...]
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-internal-sd.c
[...]
> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> +				   struct imx_media_subdev *csi[4])
> +{
> +	int ret;
> +
> +	/* there must be at least one CSI in first IPU */

Why?

> +	if (!(csi[0] || csi[1]))
> +		return -EINVAL;
> +
> +	ret = add_ipu_internal_subdevs(imxmd, csi[0], csi[1], 0);
> +	if (ret)
> +		return ret;
> +
> +	if (csi[2] || csi[3])
> +		ret = add_ipu_internal_subdevs(imxmd, csi[2], csi[3], 1);
> +
> +	return ret;
> +}
> diff --git a/drivers/staging/media/imx/imx-media-of.c b/drivers/staging/media/imx/imx-media-of.c
> new file mode 100644
> index 0000000..a939c34
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-of.c
> @@ -0,0 +1,289 @@
> +/*
> + * Media driver for Freescale i.MX5/6 SOC
> + *
> + * Open Firmware parsing.
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/of_platform.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +static int of_add_pad_link(struct imx_media_dev *imxmd,
> +			   struct imx_media_pad *pad,
> +			   struct device_node *local_sd_node,
> +			   struct device_node *remote_sd_node,
> +			   int local_pad, int remote_pad)
> +{
> +	dev_dbg(imxmd->dev, "%s: adding %s:%d -> %s:%d\n", __func__,
> +		local_sd_node->name, local_pad,
> +		remote_sd_node->name, remote_pad);
> +
> +	return imx_media_add_pad_link(imxmd, pad, remote_sd_node, NULL,
> +				      local_pad, remote_pad);
> +}
> +
> +/* parse inputs property from a sensor node */
> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> +				   struct imx_media_subdev *sensor,
> +				   struct device_node *sensor_np)
> +{
> +	struct imx_media_sensor_input *sinput = &sensor->input;
> +	int ret, i;
> +
> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> +		const char *input_name;
> +		u32 val;
> +
> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> +		if (ret)
> +			break;
> +
> +		sinput->value[i] = val;
> +
> +		ret = of_property_read_string_index(sensor_np, "input-names",
> +						    i, &input_name);
> +		/*
> +		 * if input-names not provided, they will be set using
> +		 * the subdev name once the sensor is known during
> +		 * async bind
> +		 */
> +		if (!ret)
> +			strncpy(sinput->name[i], input_name,
> +				sizeof(sinput->name[i]));
> +	}
> +
> +	sinput->num = i;
> +
> +	/* if no inputs provided just assume a single input */
> +	if (sinput->num == 0)
> +		sinput->num = 1;
> +}

This should be parsed by the sensor driver, not imx-media.

> +static void of_parse_sensor(struct imx_media_dev *imxmd,
> +			    struct imx_media_subdev *sensor,
> +			    struct device_node *sensor_np)
> +{
> +	struct device_node *endpoint;
> +
> +	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
> +
> +	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
> +	if (endpoint) {
> +		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
> +		of_node_put(endpoint);
> +	}
> +}
> +
> +static int of_get_port_count(const struct device_node *np)
> +{
> +	struct device_node *child;
> +	int num = 0;
> +
> +	/* if this node is itself a port, return 1 */
> +	if (of_node_cmp(np->name, "port") == 0)
> +		return 1;
> +
> +	for_each_child_of_node(np, child)
> +		if (of_node_cmp(child->name, "port") == 0)
> +			num++;
> +
> +	return num;
> +}

If this is extended to handle the ports subnode properly, it could be
moved into drivers/of/base.c.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-13 15:20     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-13 15:20 UTC (permalink / raw)
  To: linux-arm-kernel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add the core media driver for i.MX SOC.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  Documentation/media/v4l-drivers/imx.rst           | 443 ++++++++++
>  drivers/staging/media/Kconfig                     |   2 +
>  drivers/staging/media/Makefile                    |   1 +
>  drivers/staging/media/imx/Kconfig                 |   8 +
>  drivers/staging/media/imx/Makefile                |   6 +
>  drivers/staging/media/imx/TODO                    |  22 +
>  drivers/staging/media/imx/imx-media-common.c      | 981 ++++++++++++++++++++++
>  drivers/staging/media/imx/imx-media-dev.c         | 486 +++++++++++
>  drivers/staging/media/imx/imx-media-fim.c         | 471 +++++++++++
>  drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
>  drivers/staging/media/imx/imx-media-of.c          | 289 +++++++
>  drivers/staging/media/imx/imx-media.h             | 310 +++++++
>  include/media/imx.h                               |  15 +
>  include/uapi/linux/v4l2-controls.h                |   4 +
>  14 files changed, 3495 insertions(+)
>  create mode 100644 Documentation/media/v4l-drivers/imx.rst
>  create mode 100644 drivers/staging/media/imx/Kconfig
>  create mode 100644 drivers/staging/media/imx/Makefile
>  create mode 100644 drivers/staging/media/imx/TODO
>  create mode 100644 drivers/staging/media/imx/imx-media-common.c
>  create mode 100644 drivers/staging/media/imx/imx-media-dev.c
>  create mode 100644 drivers/staging/media/imx/imx-media-fim.c
>  create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
>  create mode 100644 drivers/staging/media/imx/imx-media-of.c
>  create mode 100644 drivers/staging/media/imx/imx-media.h
>  create mode 100644 include/media/imx.h
> 
> diff --git a/Documentation/media/v4l-drivers/imx.rst b/Documentation/media/v4l-drivers/imx.rst
> new file mode 100644
> index 0000000..87b37b5
> --- /dev/null
> +++ b/Documentation/media/v4l-drivers/imx.rst
> @@ -0,0 +1,443 @@
> +i.MX Video Capture Driver
> +=========================
> +
> +Introduction
> +------------
> +
> +The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
> +handles the flow of image frames to and from capture devices and
> +display devices.
> +
> +For image capture, the IPU contains the following internal subunits:
> +
> +- Image DMA Controller (IDMAC)
> +- Camera Serial Interface (CSI)
> +- Image Converter (IC)
> +- Sensor Multi-FIFO Controller (SMFC)
> +- Image Rotator (IRT)
> +- Video De-Interlace Controller (VDIC)

Nitpick: Video De-Interlacing or Combining Block (VDIC)

> +
> +The IDMAC is the DMA controller for transfer of image frames to and from
> +memory. Various dedicated DMA channels exist for both video capture and
> +display paths.
> +
> +The CSI is the frontend capture unit that interfaces directly with
> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
> +
> +The IC handles color-space conversion, resizing, and rotation
> +operations. 

And horizontal flipping.

> There are three independent "tasks" within the IC that can
> +carry out conversions concurrently: pre-processing encoding,
> +pre-processing preview, and post-processing.

s/preview/viewfinder/ seems to be the commonly used name.

This paragraph could mention that a single hardware unit is used
transparently time multiplexed by the three tasks at different
granularity for the downsizing, main processing, and rotation sections.
The downscale unit switches between tasks at 8-pixel burst granularity,
the main processing unit at line granularity. The rotation units switch
only at frame granularity.

> +The SMFC is composed of four independent channels that each can transfer
> +captured frames from sensors directly to memory concurrently.
> +
> +The IRT carries out 90 and 270 degree image rotation operations.

... on 8x8 pixel blocks, supported by the IDMAC which handles block
transfers, block reordering, and vertical flipping.

> +The VDIC handles the conversion of interlaced video to progressive, with
> +support for different motion compensation modes (low, medium, and high
> +motion). The deinterlaced output frames from the VDIC can be sent to the
> +IC pre-process preview task for further conversions.
> +
> +In addition to the IPU internal subunits, there are also two units
> +outside the IPU that are also involved in video capture on i.MX:
> +
> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
> +  interface. This is a Synopsys DesignWare core.
> +- A video multiplexer for selecting among multiple sensor inputs to
> +  send to a CSI.

Two of them, actually.

> +For more info, refer to the latest versions of the i.MX5/6 reference
> +manuals listed under References.
> +
> +
> +Features
> +--------
> +
> +Some of the features of this driver include:
> +
> +- Many different pipelines can be configured via media controller API,
> +  that correspond to the hardware video capture pipelines supported in
> +  the i.MX.
> +
> +- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
> +
> +- Up to four concurrent sensor acquisitions, by configuring each
> +  sensor's pipeline using independent entities. This is currently
> +  demonstrated with the SabreSD and SabreLite reference boards with
> +  independent OV5642 and MIPI CSI-2 OV5640 sensor modules.
> +
> +- Scaling, color-space conversion, and image rotation via IC task
> +  subdevs.
> +
> +- Many pixel formats supported (RGB, packed and planar YUV, partial
> +  planar YUV).
> +
> +- The IC pre-process preview subdev supports motion compensated
> +  de-interlacing using the VDIC, with three motion compensation modes:
> +  low, medium, and high motion. The mode is specified with a custom
> +  control. Pipelines are defined that allow sending frames to the
> +  preview subdev directly from the CSI or from the SMFC.
> +
> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
> +  problems with the ADV718x video decoders. See below for a description
> +  of the FIM.

Could this also be used to calculate more precise capture timestamps?

> +Capture Pipelines
> +-----------------
> +
> +The following describe the various use-cases supported by the pipelines.
> +
> +The links shown do not include the frontend sensor, video mux, or mipi
> +csi-2 receiver links. This depends on the type of sensor interface
> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
> +
> +sensor -> ipu_csi_mux -> ipu_csi -> ...
> +
> +for parallel sensors, or:
> +
> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
> +
> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
> +parenthesis.
> +
> +Unprocessed Video Capture:
> +--------------------------
> +
> +Send frames directly from sensor to camera interface, with no
> +conversions:
> +
> +-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe
idmac if you want to mirror hardware names?

> +Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

> +For example, its sink pad can take UYVY2X8, but its source pad can
> +output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
in the CSI chapter, for 8-bit per component data, the internal format
between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
(or "bayer/generic data"). In either case, the internal format does not
change along the way.

> +IC Direct Conversions:
> +----------------------
> +
> +This pipeline uses the preprocess encode entity to route frames directly
> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
> +1024x1024 resolution, CSC, and image rotation:
> +
> +-> ipu_ic_prpenc -> camif
> +
> +This can be a useful capture pipeline for heavily loaded memory bus
> +traffic environments, since it has minimal IDMAC channel usage.

Note that if rotation is enabled, transfers between IC processing and
rotation still have to go through memory once.

> +Post-Processing Conversions:
> +----------------------------
> +
> +This pipeline routes frames from the SMFC to the post-processing
> +entity.

No, frames written by the CSI -> SMFC -> IDMAC path are read back into
the post-processing entity.

>  In addition to CSC and rotation, this entity supports tiling
> +which allows scaled output beyond the 1024x1024 limitation of the IC
> +(up to 4096x4096 scaling output is supported):
> +
> +-> ipu_smfc -> ipu_ic_pp -> camif
> +
> +Motion Compensated De-interlace:
> +--------------------------------
> +
> +This pipeline routes frames from the SMFC to the preprocess preview
> +entity to support motion-compensated de-interlacing using the VDIC,
> +scaling up to 1024x1024, and CSC:
> +
> +-> ipu_smfc -> ipu_ic_prpvf -> camif

Same as above.

> +This pipeline also carries out the same conversions as above, but routes
> +frames directly from the CSI to the IC preprocess preview entity for
> +minimal memory bandwidth usage (note: this pipeline only works in
> +"high motion" mode):
> +
> +-> ipu_ic_prpvf -> camif
> +
> +This pipeline takes the motion-compensated de-interlaced frames and
> +sends them to the post-processor, to support motion-compensated
> +de-interlacing, scaling up to 4096x4096, CSC, and rotation:
> +
> +-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif
> +
> +
> +Usage Notes
> +-----------
[...]
> +SabreLite with OV5642 and OV5640
> +--------------------------------
> +
> +This platform requires the OmniVision OV5642 module with a parallel
> +camera interface, and the OV5640 module with a MIPI CSI-2
> +interface. Both modules are available from Boundary Devices:
> +
> +https://boundarydevices.com/products/nit6x_5mp
> +https://boundarydevices.com/product/nit6x_5mp_mipi
> +
> +Note that if only one camera module is available, the other sensor
> +node can be disabled in the device tree.
> +
> +The OV5642 module is connected to the parallel bus input on the i.MX
> +internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
> +
> +The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
> +receiver, and the four virtual channel outputs from the receiver are
> +routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
> +vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
> +also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
> +OV5640 must not share the same i2c slave address.
> +
> +The following basic example configures unprocessed video capture
> +pipelines for both sensors. The OV5642 is routed to camif0
> +(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
> +virtual channel 1) is routed to camif1 (usually /dev/video1). Both
> +sensors are configured to output 640x480, UYVY (not shown: all pad
> +field types should be set to "NONE"):
> +
> +.. code-block:: none
> +
> +   # Setup links for OV5642
> +   media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> +   media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> +   # Setup links for OV5640
> +   media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
> +   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
> +   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
> +   media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
> +   media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
> +   # Configure pads for OV5642 pipeline
> +   media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"

I think the smfc entities should be dropped.

> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> +   # Configure pads for OV5640 pipeline
> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
[...]
> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the
csi1:1 and camif1:0 pads would be a 32-bit YUV format.
(MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format as that will allow
to chose the regular vs. companded packings in the future for formats
with more than 8 bits per component.

> +   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>+
> +Streaming can then begin independently on device nodes /dev/video0
> +and /dev/video1.
> +
> +SabreAuto with ADV7180 decoder
> +------------------------------
> +
> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
> +parallel bus input on the internal video mux to IPU1 CSI0.
> +
> +The following example configures a pipeline to capture from the ADV7180
> +video decoder, assuming NTSC 720x480 input signals, with Motion
> +Compensated de-interlacing (not shown: all pad field types should be set
> +as indicated). $outputfmt can be any format supported by the
> +ipu1_ic_prpvf entity at its output pad:
> +
> +.. code-block:: none
> +
> +   # Setup links
> +   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> +   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
> +   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> +   # Configure pads
> +   # pad field types for below pads must be an interlaced type
> +   # such as "ALTERNATE"

I think alternate should only extend as far as the CSI, since the CSI
can only capture NTSC/PAL fields in a fixed order.

> +   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"

>From here the interlaced field type should be sequential in the correct
order depending on NTSC/PAL.

> +   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
> +   media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
> +   # pad field types for below pads must be "NONE"
> +   media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
> +   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
> +   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
> +
> +Streaming can then begin on /dev/video0.
> +
> +This platform accepts Composite Video analog inputs to the ADV7180 on
> +Ain1 (connector J42) and Ain3 (connector J43).
> +
> +To switch to Ain1:
> +
> +.. code-block:: none
> +
> +   # v4l2-ctl -i0
> +
> +To switch to Ain3:
> +
> +.. code-block:: none
> +
> +   # v4l2-ctl -i1
> +
> +
> +Frame Interval Monitor
> +----------------------
> +
> +The adv718x decoders can occasionally send corrupt fields during
> +NTSC/PAL signal re-sync (too little or too many video lines). When
> +this happens, the IPU triggers a mechanism to re-establish vertical
> +sync by adding 1 dummy line every frame, which causes a rolling effect
> +from image to image, and can last a long time before a stable image is
> +recovered. Or sometimes the mechanism doesn't work at all, causing a
> +permanent split image (one frame contains lines from two consecutive
> +captured images).

Is it only SabreAuto on which the FIM mechanism can be used due to the
pad routing?

[...]
> +/*
> + * DMA buffer ring handling
> + */
> +struct imx_media_dma_buf_ring {
> +	struct imx_media_dev *imxmd;
> +
> +	/* the ring */
> +	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
> +	/* the scratch buffer for underruns */
> +	struct imx_media_dma_buf scratch;
> +
> +	/* buffer generator */
> +	struct media_entity *src;
> +	/* buffer receiver */
> +	struct media_entity *sink;
> +
> +	spinlock_t lock;
> +
> +	int num_bufs;
> +	unsigned long last_seq;
> +};

I don't think this belongs in the capture driver at all.
Memory-to-memory transfers should be handled at the videobuf2 level.

[...]
> +static struct imx_media_dma_buf *
> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
> +{
> +	struct imx_media_dma_buf *buf;
> +
> +	if (index >= ring->num_bufs)
> +		return ERR_PTR(-EINVAL);
> +
> +	buf = &ring->buf[index];
> +	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
> +		return ERR_PTR(-EINVAL);
> +
> +	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
> +	buf->seq = ring->last_seq++;
> +
> +	return buf;
> +}

Is this a whole software buffer queue implementation? I thought the
whole point of putting the custom mem2mem framework into the capture
driver was to use the hardware FSU channel linking?

> +int imx_media_dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
> +{
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	buf = __dma_buf_queue(ring, index);
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	if (IS_ERR(buf))
> +		return PTR_ERR(buf);
> +
> +	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued\n",
> +		index, ring->src->name, ring->sink->name);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue);
> +
> +int imx_media_dma_buf_queue_from_vb(struct imx_media_dma_buf_ring *ring,
> +				    struct vb2_buffer *vb)
> +{
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +	dma_addr_t phys;
> +	void *virt;
> +
> +	if (vb->index >= ring->num_bufs)
> +		return -EINVAL;
> +
> +	virt = vb2_plane_vaddr(vb, 0);
> +	phys = vb2_dma_contig_plane_dma_addr(vb, 0);
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	buf = __dma_buf_queue(ring, vb->index);
> +	if (IS_ERR(buf))
> +		goto err_unlock;
> +
> +	buf->virt = virt;
> +	buf->phys = phys;
> +	buf->vb = vb;
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued from vb\n",
> +		buf->index, ring->src->name, ring->sink->name);
> +
> +	return 0;
> +err_unlock:
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +	return PTR_ERR(buf);
> +}
> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue_from_vb);
> +
> +void imx_media_dma_buf_done(struct imx_media_dma_buf *buf,
> +			    enum imx_media_dma_buf_status status)
> +{
> +	struct imx_media_dma_buf_ring *ring = buf->ring;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&ring->lock, flags);
> +	WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_ACTIVE);
> +	buf->state = buf->status = status;
> +	spin_unlock_irqrestore(&ring->lock, flags);
> +
> +	if (buf == &ring->scratch)
> +		dev_dbg(ring->imxmd->dev, "buf-scratch [%s -> %s] done\n",
> +			ring->src->name, ring->sink->name);
> +	else
> +		dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] done\n",
> +			buf->index, ring->src->name, ring->sink->name);
> +
> +	/* if the sink is a subdev, inform it that new buffers are available */
> +	if (is_media_entity_v4l2_subdev(ring->sink)) {
> +		struct v4l2_subdev *sd =
> +			media_entity_to_v4l2_subdev(ring->sink);
> +		v4l2_subdev_call(sd, core, ioctl, IMX_MEDIA_NEW_DMA_BUF, NULL);

What is the purpose of this if the sink should be triggered by the FSU?

[...]
> +/*
> + * The subdevs have to be powered on/off, and streaming
> + * enabled/disabled, in a specific sequence.
> + */
> +static const u32 stream_on_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI,
> +};
> +
> +static const u32 stream_off_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +};
> +
> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
> +
> +static const u32 power_on_seq[] = {
> +	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +};
> +
> +static const u32 power_off_seq[] = {
> +	IMX_MEDIA_GRP_ID_IC_PP,
> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> +	IMX_MEDIA_GRP_ID_SMFC,
> +	IMX_MEDIA_GRP_ID_CSI,
> +	IMX_MEDIA_GRP_ID_VIDMUX,
> +	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
> +};

This seems somewhat arbitrary. Why is a power sequence needed?

[...]
> +/*
> + * Turn current pipeline power on/off starting from start_entity.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
> +				 struct media_entity_graph *graph,
> +				 struct media_entity *start_entity, bool on)
> +{
> +	struct media_entity *entity;
> +	struct v4l2_subdev *sd;
> +	int i, ret = 0;
> +	u32 id;
> +
> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
> +		id = on ? power_on_seq[i] : power_off_seq[i];
> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
> +		if (!entity)
> +			continue;
> +
> +		sd = media_entity_to_v4l2_subdev(entity);
> +
> +		ret = v4l2_subdev_call(sd, core, s_power, on);
> +		if (ret && ret != -ENOIOCTLCMD)
> +			break;
> +	}
> +
> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

> +/*
> + * Inherit the v4l2 controls from all entities in a pipeline
> + * to the given video device.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
> +			       struct video_device *vfd,
> +			       struct media_entity *start_entity)
> +{
> +	struct media_entity_graph graph;
> +	struct media_entity *entity;
> +	struct v4l2_subdev *sd;
> +	int ret;
> +
> +	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
> +	if (ret)
> +		return ret;
> +
> +	media_entity_graph_walk_start(&graph, start_entity);
> +
> +	while ((entity = media_entity_graph_walk_next(&graph))) {
> +		if (is_media_entity_v4l2_video_device(entity))
> +			continue;
> +
> +		sd = media_entity_to_v4l2_subdev(entity);
> +
> +		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
> +			__func__, sd->name);
> +
> +		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
> +					    sd->ctrl_handler,
> +					    NULL);
> +		if (ret)
> +			break;
> +	}
> +
> +	media_entity_graph_walk_cleanup(&graph);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
> +
> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
> new file mode 100644
> index 0000000..357654d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-dev.c

This file is full of code that should live in the v4l2 core.

[...]
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-internal-sd.c
[...]
> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> +				   struct imx_media_subdev *csi[4])
> +{
> +	int ret;
> +
> +	/* there must be at least one CSI in first IPU */

Why?

> +	if (!(csi[0] || csi[1]))
> +		return -EINVAL;
> +
> +	ret = add_ipu_internal_subdevs(imxmd, csi[0], csi[1], 0);
> +	if (ret)
> +		return ret;
> +
> +	if (csi[2] || csi[3])
> +		ret = add_ipu_internal_subdevs(imxmd, csi[2], csi[3], 1);
> +
> +	return ret;
> +}
> diff --git a/drivers/staging/media/imx/imx-media-of.c b/drivers/staging/media/imx/imx-media-of.c
> new file mode 100644
> index 0000000..a939c34
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-of.c
> @@ -0,0 +1,289 @@
> +/*
> + * Media driver for Freescale i.MX5/6 SOC
> + *
> + * Open Firmware parsing.
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/of_platform.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +static int of_add_pad_link(struct imx_media_dev *imxmd,
> +			   struct imx_media_pad *pad,
> +			   struct device_node *local_sd_node,
> +			   struct device_node *remote_sd_node,
> +			   int local_pad, int remote_pad)
> +{
> +	dev_dbg(imxmd->dev, "%s: adding %s:%d -> %s:%d\n", __func__,
> +		local_sd_node->name, local_pad,
> +		remote_sd_node->name, remote_pad);
> +
> +	return imx_media_add_pad_link(imxmd, pad, remote_sd_node, NULL,
> +				      local_pad, remote_pad);
> +}
> +
> +/* parse inputs property from a sensor node */
> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> +				   struct imx_media_subdev *sensor,
> +				   struct device_node *sensor_np)
> +{
> +	struct imx_media_sensor_input *sinput = &sensor->input;
> +	int ret, i;
> +
> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> +		const char *input_name;
> +		u32 val;
> +
> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> +		if (ret)
> +			break;
> +
> +		sinput->value[i] = val;
> +
> +		ret = of_property_read_string_index(sensor_np, "input-names",
> +						    i, &input_name);
> +		/*
> +		 * if input-names not provided, they will be set using
> +		 * the subdev name once the sensor is known during
> +		 * async bind
> +		 */
> +		if (!ret)
> +			strncpy(sinput->name[i], input_name,
> +				sizeof(sinput->name[i]));
> +	}
> +
> +	sinput->num = i;
> +
> +	/* if no inputs provided just assume a single input */
> +	if (sinput->num == 0)
> +		sinput->num = 1;
> +}

This should be parsed by the sensor driver, not imx-media.

> +static void of_parse_sensor(struct imx_media_dev *imxmd,
> +			    struct imx_media_subdev *sensor,
> +			    struct device_node *sensor_np)
> +{
> +	struct device_node *endpoint;
> +
> +	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
> +
> +	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
> +	if (endpoint) {
> +		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
> +		of_node_put(endpoint);
> +	}
> +}
> +
> +static int of_get_port_count(const struct device_node *np)
> +{
> +	struct device_node *child;
> +	int num = 0;
> +
> +	/* if this node is itself a port, return 1 */
> +	if (of_node_cmp(np->name, "port") == 0)
> +		return 1;
> +
> +	for_each_child_of_node(np, child)
> +		if (of_node_cmp(child->name, "port") == 0)
> +			num++;
> +
> +	return num;
> +}

If this is extended to handle the ports subnode properly, it could be
moved into drivers/of/base.c.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-13 19:03       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 19:03 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/13/2017 03:55 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Add bindings documentation for the i.MX media driver.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>>   1 file changed, 57 insertions(+)
>>   create mode 100644 Documentation/devicetree/bindings/media/imx.txt
>>
>> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
>> new file mode 100644
>> index 0000000..254b64a
>> --- /dev/null
>> +++ b/Documentation/devicetree/bindings/media/imx.txt
>> @@ -0,0 +1,57 @@
>> +Freescale i.MX Media Video Devices
>> +
>> +Video Media Controller node
>> +---------------------------
>> +
>> +This is the parent media controller node for video capture support.
>> +
>> +Required properties:
>> +- compatible : "fsl,imx-media";
> Would you be opposed to calling this "capture-subsystem" instead of
> "imx-media"? We already use "fsl,imx-display-subsystem" and
> "fsl,imx-gpu-subsystem" for the display and GPU compound devices.

sure. Some pie-in-the-sky day when DRM and media are unified,
there could be a single device that handles them all, but for now
I'm fine with "fsl,capture-subsystem".

>> +- ports      : Should contain a list of phandles pointing to camera
>> +  	       sensor interface ports of IPU devices
>> +
>> +
>> +fim child node
>> +--------------
>> +
>> +This is an optional child node of the ipu_csi port nodes. If present and
>> +available, it enables the Frame Interval Monitor. Its properties can be
>> +used to modify the method in which the FIM measures frame intervals.
>> +Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
>> +Frame Interval Monitor.
>> +
>> +Optional properties:
>> +- fsl,input-capture-channel: an input capture channel and channel flags,
>> +			     specified as <chan flags>. The channel number
>> +			     must be 0 or 1. The flags can be
>> +			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
>> +			     IRQ_TYPE_EDGE_BOTH, and specify which input
>> +			     capture signal edge will trigger the input
>> +			     capture event. If an input capture channel is
>> +			     specified, the FIM will use this method to
>> +			     measure frame intervals instead of via the EOF
>> +			     interrupt. The input capture method is much
>> +			     preferred over EOF as it is not subject to
>> +			     interrupt latency errors. However it requires
>> +			     routing the VSYNC or FIELD output signals of
>> +			     the camera sensor to one of the i.MX input
>> +			     capture pads (SD1_DAT0, SD1_DAT1), which also
>> +			     gives up support for SD1.
> This is a clever method to get better frame timestamps. Too bad about
> the routing requirements. Can this be used on Nitrogen6X?

Absolutely, this support just needs use of the input-capture channels in the
imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
input capture API, so at this point fsl,input-capture-channel has no effect,
but it does work (tested on SabreAuto).

>
>> +
>> +mipi_csi2 node
>> +--------------
>> +
>> +This is the device node for the MIPI CSI-2 Receiver, required for MIPI
>> +CSI-2 sensors.
>> +
>> +Required properties:
>> +- compatible	: "fsl,imx6-mipi-csi2";
> I think this should get an additional "snps,dw-mipi-csi2" compatible,
> since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
Or should wait until the day this subdev is exported for general use, after
pulling out the gasket specifics?


>
>> +- reg           : physical base address and length of the register set;
>> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
>> +                  (the DPHY clock), video_27m, and eim_sel;
> Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
> shared gate bit that gates the HSI clocks as well as the MIPI
> "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
> csi-2 core, but we are missing shared gate clocks in the clock tree for
> these.

Yes, so many clocks for the MIPI core. Why so many? I would think
there would need to be at most three: a clock for the MIPI CSI-2 core
and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
M-PHY if there is one). I have no clue what all these other clocks are.
But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

> Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
> video_27m seems fine.
> But I don't get "dphy".

I presume it's the clock for the D-PHY.

>   Which input clock would that correspond to?
> "pll_refclk?"

the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.


> Also the pixel clock input is a gate after aclk_podf (which we call
> eim_podf), not aclk_sel (eim_sel).

ok, I'll switch to eim_podf for the pixel clock.

Steve

>
>> +- clock-names	: must contain "dphy", "cfg", "pix";
>> +
>> +Optional properties:
>> +- interrupts	: must contain two level-triggered interrupts,
>> +                  in order: 100 and 101;
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-13 19:03       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 19:03 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/13/2017 03:55 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Add bindings documentation for the i.MX media driver.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
>> ---
>>   Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>>   1 file changed, 57 insertions(+)
>>   create mode 100644 Documentation/devicetree/bindings/media/imx.txt
>>
>> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
>> new file mode 100644
>> index 0000000..254b64a
>> --- /dev/null
>> +++ b/Documentation/devicetree/bindings/media/imx.txt
>> @@ -0,0 +1,57 @@
>> +Freescale i.MX Media Video Devices
>> +
>> +Video Media Controller node
>> +---------------------------
>> +
>> +This is the parent media controller node for video capture support.
>> +
>> +Required properties:
>> +- compatible : "fsl,imx-media";
> Would you be opposed to calling this "capture-subsystem" instead of
> "imx-media"? We already use "fsl,imx-display-subsystem" and
> "fsl,imx-gpu-subsystem" for the display and GPU compound devices.

sure. Some pie-in-the-sky day when DRM and media are unified,
there could be a single device that handles them all, but for now
I'm fine with "fsl,capture-subsystem".

>> +- ports      : Should contain a list of phandles pointing to camera
>> +  	       sensor interface ports of IPU devices
>> +
>> +
>> +fim child node
>> +--------------
>> +
>> +This is an optional child node of the ipu_csi port nodes. If present and
>> +available, it enables the Frame Interval Monitor. Its properties can be
>> +used to modify the method in which the FIM measures frame intervals.
>> +Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
>> +Frame Interval Monitor.
>> +
>> +Optional properties:
>> +- fsl,input-capture-channel: an input capture channel and channel flags,
>> +			     specified as <chan flags>. The channel number
>> +			     must be 0 or 1. The flags can be
>> +			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
>> +			     IRQ_TYPE_EDGE_BOTH, and specify which input
>> +			     capture signal edge will trigger the input
>> +			     capture event. If an input capture channel is
>> +			     specified, the FIM will use this method to
>> +			     measure frame intervals instead of via the EOF
>> +			     interrupt. The input capture method is much
>> +			     preferred over EOF as it is not subject to
>> +			     interrupt latency errors. However it requires
>> +			     routing the VSYNC or FIELD output signals of
>> +			     the camera sensor to one of the i.MX input
>> +			     capture pads (SD1_DAT0, SD1_DAT1), which also
>> +			     gives up support for SD1.
> This is a clever method to get better frame timestamps. Too bad about
> the routing requirements. Can this be used on Nitrogen6X?

Absolutely, this support just needs use of the input-capture channels in the
imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
input capture API, so at this point fsl,input-capture-channel has no effect,
but it does work (tested on SabreAuto).

>
>> +
>> +mipi_csi2 node
>> +--------------
>> +
>> +This is the device node for the MIPI CSI-2 Receiver, required for MIPI
>> +CSI-2 sensors.
>> +
>> +Required properties:
>> +- compatible	: "fsl,imx6-mipi-csi2";
> I think this should get an additional "snps,dw-mipi-csi2" compatible,
> since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
Or should wait until the day this subdev is exported for general use, after
pulling out the gasket specifics?


>
>> +- reg           : physical base address and length of the register set;
>> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
>> +                  (the DPHY clock), video_27m, and eim_sel;
> Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
> shared gate bit that gates the HSI clocks as well as the MIPI
> "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
> csi-2 core, but we are missing shared gate clocks in the clock tree for
> these.

Yes, so many clocks for the MIPI core. Why so many? I would think
there would need to be at most three: a clock for the MIPI CSI-2 core
and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
M-PHY if there is one). I have no clue what all these other clocks are.
But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

> Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
> video_27m seems fine.
> But I don't get "dphy".

I presume it's the clock for the D-PHY.

>   Which input clock would that correspond to?
> "pll_refclk?"

the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.


> Also the pixel clock input is a gate after aclk_podf (which we call
> eim_podf), not aclk_sel (eim_sel).

ok, I'll switch to eim_podf for the pixel clock.

Steve

>
>> +- clock-names	: must contain "dphy", "cfg", "pix";
>> +
>> +Optional properties:
>> +- interrupts	: must contain two level-triggered interrupts,
>> +                  in order: 100 and 101;
> regards
> Philipp
>

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-13 19:03       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 19:03 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/13/2017 03:55 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Add bindings documentation for the i.MX media driver.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>>   1 file changed, 57 insertions(+)
>>   create mode 100644 Documentation/devicetree/bindings/media/imx.txt
>>
>> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
>> new file mode 100644
>> index 0000000..254b64a
>> --- /dev/null
>> +++ b/Documentation/devicetree/bindings/media/imx.txt
>> @@ -0,0 +1,57 @@
>> +Freescale i.MX Media Video Devices
>> +
>> +Video Media Controller node
>> +---------------------------
>> +
>> +This is the parent media controller node for video capture support.
>> +
>> +Required properties:
>> +- compatible : "fsl,imx-media";
> Would you be opposed to calling this "capture-subsystem" instead of
> "imx-media"? We already use "fsl,imx-display-subsystem" and
> "fsl,imx-gpu-subsystem" for the display and GPU compound devices.

sure. Some pie-in-the-sky day when DRM and media are unified,
there could be a single device that handles them all, but for now
I'm fine with "fsl,capture-subsystem".

>> +- ports      : Should contain a list of phandles pointing to camera
>> +  	       sensor interface ports of IPU devices
>> +
>> +
>> +fim child node
>> +--------------
>> +
>> +This is an optional child node of the ipu_csi port nodes. If present and
>> +available, it enables the Frame Interval Monitor. Its properties can be
>> +used to modify the method in which the FIM measures frame intervals.
>> +Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
>> +Frame Interval Monitor.
>> +
>> +Optional properties:
>> +- fsl,input-capture-channel: an input capture channel and channel flags,
>> +			     specified as <chan flags>. The channel number
>> +			     must be 0 or 1. The flags can be
>> +			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
>> +			     IRQ_TYPE_EDGE_BOTH, and specify which input
>> +			     capture signal edge will trigger the input
>> +			     capture event. If an input capture channel is
>> +			     specified, the FIM will use this method to
>> +			     measure frame intervals instead of via the EOF
>> +			     interrupt. The input capture method is much
>> +			     preferred over EOF as it is not subject to
>> +			     interrupt latency errors. However it requires
>> +			     routing the VSYNC or FIELD output signals of
>> +			     the camera sensor to one of the i.MX input
>> +			     capture pads (SD1_DAT0, SD1_DAT1), which also
>> +			     gives up support for SD1.
> This is a clever method to get better frame timestamps. Too bad about
> the routing requirements. Can this be used on Nitrogen6X?

Absolutely, this support just needs use of the input-capture channels in the
imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
input capture API, so at this point fsl,input-capture-channel has no effect,
but it does work (tested on SabreAuto).

>
>> +
>> +mipi_csi2 node
>> +--------------
>> +
>> +This is the device node for the MIPI CSI-2 Receiver, required for MIPI
>> +CSI-2 sensors.
>> +
>> +Required properties:
>> +- compatible	: "fsl,imx6-mipi-csi2";
> I think this should get an additional "snps,dw-mipi-csi2" compatible,
> since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
Or should wait until the day this subdev is exported for general use, after
pulling out the gasket specifics?


>
>> +- reg           : physical base address and length of the register set;
>> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
>> +                  (the DPHY clock), video_27m, and eim_sel;
> Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
> shared gate bit that gates the HSI clocks as well as the MIPI
> "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
> csi-2 core, but we are missing shared gate clocks in the clock tree for
> these.

Yes, so many clocks for the MIPI core. Why so many? I would think
there would need to be at most three: a clock for the MIPI CSI-2 core
and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
M-PHY if there is one). I have no clue what all these other clocks are.
But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

> Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
> video_27m seems fine.
> But I don't get "dphy".

I presume it's the clock for the D-PHY.

>   Which input clock would that correspond to?
> "pll_refclk?"

the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.


> Also the pixel clock input is a gate after aclk_podf (which we call
> eim_podf), not aclk_sel (eim_sel).

ok, I'll switch to eim_podf for the pixel clock.

Steve

>
>> +- clock-names	: must contain "dphy", "cfg", "pix";
>> +
>> +Optional properties:
>> +- interrupts	: must contain two level-triggered interrupts,
>> +                  in order: 100 and 101;
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-12 22:32       ` Steve Longerbeam
@ 2017-01-13 21:04         ` Tim Harvey
  0 siblings, 0 replies; 549+ messages in thread
From: Tim Harvey @ 2017-01-13 21:04 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Steve Longerbeam, Shawn Guo, Sascha Hauer, Fabio Estevam,
	Hans Verkuil, Philipp Zabel, laurent.pinchart+renesas,
	linux-media

On Thu, Jan 12, 2017 at 2:32 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> Hi Tim,
>
>
> On 01/12/2017 01:13 PM, Tim Harvey wrote:
>>
>>
>>>> Now that your driver is hooking into the current media framework, I'm
>>>> not at all clear on how to link and configure the media entities.
>>>
>>>
>>> It's all documented at Documentation/media/v4l-drivers/imx.rst.
>>> Follow the SabreAuto pipeline setup example.
>>>
>> ah yes... it helps to read your patches! You did a great job on the
>> documentation.
>>
>> Regarding the The ipu1_csi0_mux/ipu2_csi1_mux entities which have 1
>> source and 2 sinks (which makes sense for a mux) how do you know which
>> sink pad you should use (in your adv7180 example you use the 2nd sink
>> pad vs the first)?
>
>
> The adv7180 can only go to the parallel input pad (ipu1_csi0_mux:1
> on quad). The other input pads select from the mipi csi-2 receiver virtual
> channels.

right - my question was how does the user know which pad is which. I
see that the imx6q.dtsi makes it clear that port0 (sink1) is the mipi
port and port1 is the parallel port (sink2). Do you know how a user
would determine this from runtime information (maybe something via
media-ctl or /sys/class/media that I haven't yet found) or perhaps
this is to be taken care of by documentation or referring to the dts?

>
> Have you generated a dot graph? It makes it much easier to
> visualize:
>
> # media-ctl --print-dot > graph.dot
>
> then on your host:
>
> % dot -Tpng graph.dot > graph.png
>

Yes - that makes it much easier to understand the possible links.

I notice 'media-ctl --print-topology' shows link and pad type fields,
but 'media-ctl --print-dot' does not include the pad type field (maybe
newer versions do or will in the future)

Do you know how one goes about determining the possible format types
possible for each pad?

>
>
>>
>> As my hardware is the same as the SabreAuto except that my adv7180 is
>> on i2c-2@0x20 I follow your example from
>> Documentation/media/v4l-drivers/imx.rst:
>>
>> # Setup links
>> media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":1[1]'
>> media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
>> media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
>> media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
>> media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
>> media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
>>
>> # Configure pads
>> media-ctl -V "\"adv7180 2-0020\":0 [fmt:UYVY2X8/720x480]"
>> media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
>> media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
>> media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
>> media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
>> media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
>> media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
>> media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
>> # pad field types for camif can be any format prpvf supports
>> export outputfmt="UYVY2X8/720x480"
>> media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
>> media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
>> media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
>>
>> # select AIN1
>> v4l2-ctl -d0 -i0
>> Video input set to 0 (ADV7180 Composite on Ain1: ok)
>> v4l2-ctl -d0 --set-fmt-video=width=720,height=480,pixelformat=UYVY
>> # capture a single raw frame
>> v4l2-ctl -d0 --stream-mmap --stream-to=/x.raw --stream-count=1
>> [ 2092.056394] camif0: pipeline_set_stream failed with -32
>> VIDIOC_STREAMON: failed: Broken pipe
>>
>> Enabling debug in drivers/media/media-entity.c I see:
>> [   38.870087] imx-media soc:media@0: link validation failed for
>> "ipu1_smfc0":1 -> "ipu1_ic_prpvf":0, error -32
>>
>> Looking at ipu1_smfc0 and ipu1_ic_prpvf with media-ctl I see:
>> - entity 12: ipu1_ic_prpvf (2 pads, 8 links)
>>               type V4L2 subdev subtype Unknown flags 0
>>               device node name /dev/v4l-subdev3
>>          pad0: Sink
>>                  [fmt:UYVY2X8/720x480 field:alternate]
>>                  <- "ipu1_csi0":1 []
>>                  <- "ipu1_csi1":1 []
>>                  <- "ipu1_smfc0":1 [ENABLED]
>>                  <- "ipu1_smfc1":1 []
>>          pad1: Source
>>                  [fmt:UYVY2X8/720x480 field:none]
>>                  -> "camif0":0 [ENABLED]
>>                  -> "camif1":0 []
>>                  -> "ipu1_ic_pp0":0 []
>>                  -> "ipu1_ic_pp1":0 []
>>
>> - entity 45: ipu1_smfc0 (2 pads, 5 links)
>>               type V4L2 subdev subtype Unknown flags 0
>>               device node name /dev/v4l-subdev14
>>          pad0: Sink
>>                  [fmt:UYVY2X8/720x480]
>>                  <- "ipu1_csi0":1 [ENABLED]
>>          pad1: Source
>>                  [fmt:UYVY2X8/720x480]
>>                  -> "ipu1_ic_prpvf":0 [ENABLED]
>>                  -> "ipu1_ic_pp0":0 []
>>                  -> "camif0":0 []
>>                  -> "camif1":0 []
>>
>> Any ideas what is going wrong here? Seems like its perhaps a field
>> type mismatch.
>
>
> Yes, exactly, you'll need to set the field types on every pad in your
> pipeline.
>
>>   Is my outputfmt incorrect perhaps? I likely have
>> misunderstood the pad type comments in your documentation.
>
>
> Attached is an update doc (from branch imx-media-staging-md-v7 on my fork).
> I recently upgraded my v4l-utils package and media-ctl now supports
> specifying
> the field type in the pad format strings. If you don't have the latest
> v4l-utils, it's
> fairly straightforward to cross-build.
>

Your updated imx.rst makes it very clear - I was misunderstanding the
pervious version and comments as it skipped setting the pad field
types.

The v4l-utils-1.10 from Ubuntu 16.04 allows setting field types with
the links so I didn't need to build a newer one and this did resolve
my issue.

>>
>>>
>>>> <snip>
>>>>
>>>>
>>>>
>>>> Additionally I've found that on an IMX6S/IMX6DL we crash while
>>>> registering the media-ic subdev's:
>>
>> <snip>
>>>
>>> Yep, I only have quad boards here so I haven't gotten around to
>>> testing on S/DL.
>>>
>>> But it looks like I forgot to clear out the csi subdev pointer array
>>> before
>>> passing it to imx_media_of_parse(). I think that might explain the OOPS
>>> above. Try this patch:
>>>
>>> diff --git a/drivers/staging/media/imx/imx-media-dev.c
>>> b/drivers/staging/media/imx/imx-media-dev.c
>>> index 357654d..0cf2d61 100644
>>> --- a/drivers/staging/media/imx/imx-media-dev.c
>>> +++ b/drivers/staging/media/imx/imx-media-dev.c
>>> @@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device
>>> *pdev)
>>>   {
>>>          struct device *dev = &pdev->dev;
>>>          struct device_node *node = dev->of_node;
>>> -       struct imx_media_subdev *csi[4];
>>> +       struct imx_media_subdev *csi[4] = {0};
>>>          struct imx_media_dev *imxmd;
>>>          int ret;
>>>
>> This does resolves the crash on S/DL.
>
>

I now have dts patches ready for the following which have an on-board
ADV7180 SD capture on:
arch/arm/boot/dts/imx6dl-gw52xx.dts
arch/arm/boot/dts/imx6dl-gw53xx.dts
arch/arm/boot/dts/imx6dl-gw54xx.dts
arch/arm/boot/dts/imx6q-gw52xx.dts
arch/arm/boot/dts/imx6q-gw53xx.dts
arch/arm/boot/dts/imx6q-gw54xx.dts
arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
arch/arm/boot/dts/imx6qdl-gw553x.dtsi

So for the above which I've tested with the 'sensor -> ipu_csi_mux ->
ipu_csi -> ipu_smfc -> ipu_ic_prpvf -> camif' pipeline
Tested-by: Tim Harvey <tharvey@gateworks.com>

Thanks for all your continued effort on these drivers!

Tim

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
  2017-01-13 11:57     ` Philipp Zabel
  (?)
@ 2017-01-13 22:40       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 22:40 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/13/2017 03:57 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
>> clocks.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
>> index 53e6e63..42926e9 100644
>> --- a/arch/arm/boot/dts/imx6qdl.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
>> @@ -1125,7 +1125,14 @@
>>   			};
>>   
>>   			mipi_csi: mipi@021dc000 {
>> +				compatible = "fsl,imx6-mipi-csi2";
>>   				reg = <0x021dc000 0x4000>;
>> +				interrupts = <0 100 0x04>, <0 101 0x04>;
>> +				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
>> +					 <&clks IMX6QDL_CLK_VIDEO_27M>,
>> +					 <&clks IMX6QDL_CLK_EIM_SEL>;
> I think the latter should be EIM_PODF

done.

>
>> +				clock-names = "dphy", "cfg", "pix";
> and I'm not sure dphy is the right name for this one. Is that the pll
> ref input?

I believe this naming came from FSL's mipi csi-2 driver. It is the "hsi_tx"
clock (presumably for the MIPI HSI controller) whose parents are selected
by the CDCDR register as PLL3_120M or PLL2_PFD2. I have no clue whether
this is indeed also used as the clock for the MIPI CSI-2 D-PHY, but 
according
to FSL naming convention it might be.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
@ 2017-01-13 22:40       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 22:40 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/13/2017 03:57 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
>> clocks.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
>> index 53e6e63..42926e9 100644
>> --- a/arch/arm/boot/dts/imx6qdl.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
>> @@ -1125,7 +1125,14 @@
>>   			};
>>   
>>   			mipi_csi: mipi@021dc000 {
>> +				compatible = "fsl,imx6-mipi-csi2";
>>   				reg = <0x021dc000 0x4000>;
>> +				interrupts = <0 100 0x04>, <0 101 0x04>;
>> +				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
>> +					 <&clks IMX6QDL_CLK_VIDEO_27M>,
>> +					 <&clks IMX6QDL_CLK_EIM_SEL>;
> I think the latter should be EIM_PODF

done.

>
>> +				clock-names = "dphy", "cfg", "pix";
> and I'm not sure dphy is the right name for this one. Is that the pll
> ref input?

I believe this naming came from FSL's mipi csi-2 driver. It is the "hsi_tx"
clock (presumably for the MIPI HSI controller) whose parents are selected
by the CDCDR register as PLL3_120M or PLL2_PFD2. I have no clue whether
this is indeed also used as the clock for the MIPI CSI-2 D-PHY, but 
according
to FSL naming convention it might be.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
@ 2017-01-13 22:40       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 22:40 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/13/2017 03:57 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
>> clocks.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
>> index 53e6e63..42926e9 100644
>> --- a/arch/arm/boot/dts/imx6qdl.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
>> @@ -1125,7 +1125,14 @@
>>   			};
>>   
>>   			mipi_csi: mipi at 021dc000 {
>> +				compatible = "fsl,imx6-mipi-csi2";
>>   				reg = <0x021dc000 0x4000>;
>> +				interrupts = <0 100 0x04>, <0 101 0x04>;
>> +				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
>> +					 <&clks IMX6QDL_CLK_VIDEO_27M>,
>> +					 <&clks IMX6QDL_CLK_EIM_SEL>;
> I think the latter should be EIM_PODF

done.

>
>> +				clock-names = "dphy", "cfg", "pix";
> and I'm not sure dphy is the right name for this one. Is that the pll
> ref input?

I believe this naming came from FSL's mipi csi-2 driver. It is the "hsi_tx"
clock (presumably for the MIPI HSI controller) whose parents are selected
by the CDCDR register as PLL3_120M or PLL2_PFD2. I have no clue whether
this is indeed also used as the clock for the MIPI CSI-2 D-PHY, but 
according
to FSL naming convention it might be.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  2017-01-13 12:03     ` Philipp Zabel
  (?)
@ 2017-01-13 23:04       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 23:04 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
>> Both hang off the same i2c2 bus, so they require different (and non-
>> default) i2c slave addresses.
>>
>> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
>>
>> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
>> mipi_csi. It is set to transmit over MIPI virtual channel 1.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
>>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
>>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
>>   3 files changed, 129 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> index 0f06ca5..fec2524 100644
>> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> @@ -48,3 +48,8 @@
>>   	model = "Freescale i.MX6 DualLite SABRE Lite Board";
>>   	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
>>   };
>> +
>> +&ipu1_csi1_from_ipu1_csi1_mux {
>> +	data-lanes = <0 1>;
>> +	clock-lanes = <2>;
>> +};
>> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
>> index 66d10d8..9e2d26d 100644
>> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
>> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
>> @@ -52,3 +52,9 @@
>>   &sata {
>>   	status = "okay";
>>   };
>> +
>> +&ipu1_csi1_from_mipi_vc1 {
>> +	data-lanes = <0 1>;
>> +	clock-lanes = <2>;
>> +};
>> +
>> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> index 795b5a5..bca9fed 100644
>> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> @@ -39,6 +39,8 @@
>>    *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>>    *     OTHER DEALINGS IN THE SOFTWARE.
>>    */
>> +
>> +#include <dt-bindings/clock/imx6qdl-clock.h>
>>   #include <dt-bindings/gpio/gpio.h>
>>   #include <dt-bindings/input/input.h>
>>   
>> @@ -96,6 +98,15 @@
>>   		};
>>   	};
>>   
>> +	mipi_xclk: mipi_xclk {
>> +		compatible = "pwm-clock";
>> +		#clock-cells = <0>;
>> +		clock-frequency = <22000000>;
>> +		clock-output-names = "mipi_pwm3";
>> +		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
>> +		status = "okay";
>> +	};
>> +
>>   	gpio-keys {
>>   		compatible = "gpio-keys";
>>   		pinctrl-names = "default";
>> @@ -220,6 +231,22 @@
>>   	};
>>   };
>>   
>> +&ipu1_csi0_from_ipu1_csi0_mux {
>> +	bus-width = <8>;
>> +	data-shift = <12>; /* Lines 19:12 used */
>> +	hsync-active = <1>;
>> +	vync-active = <1>;
>> +};
>> +
>> +&ipu1_csi0_mux_from_parallel_sensor {
>> +	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
>> +};
>> +
>> +&ipu1_csi0 {
>> +	pinctrl-names = "default";
>> +	pinctrl-0 = <&pinctrl_ipu1_csi0>;
>> +};
>> +
>>   &audmux {
>>   	pinctrl-names = "default";
>>   	pinctrl-0 = <&pinctrl_audmux>;
>> @@ -299,6 +326,52 @@
>>   	pinctrl-names = "default";
>>   	pinctrl-0 = <&pinctrl_i2c2>;
>>   	status = "okay";
>> +
>> +	ov5640: camera@40 {
>> +		compatible = "ovti,ov5640";
>> +		pinctrl-names = "default";
>> +		pinctrl-0 = <&pinctrl_ov5640>;
>> +		clocks = <&mipi_xclk>;
>> +		clock-names = "xclk";
>> +		reg = <0x40>;
>> +		xclk = <22000000>;
> This is superfluous, you can use clk_get_rate on mipi_xclk.

This property is actually there to tell the driver what to set the
rate to, with clk_set_rate(). So you are saying it would be better
to set the rate in the device tree and the driver should only
retrieve the rate?

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-13 23:04       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 23:04 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
>> Both hang off the same i2c2 bus, so they require different (and non-
>> default) i2c slave addresses.
>>
>> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
>>
>> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
>> mipi_csi. It is set to transmit over MIPI virtual channel 1.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
>>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
>>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
>>   3 files changed, 129 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> index 0f06ca5..fec2524 100644
>> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> @@ -48,3 +48,8 @@
>>   	model = "Freescale i.MX6 DualLite SABRE Lite Board";
>>   	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
>>   };
>> +
>> +&ipu1_csi1_from_ipu1_csi1_mux {
>> +	data-lanes = <0 1>;
>> +	clock-lanes = <2>;
>> +};
>> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
>> index 66d10d8..9e2d26d 100644
>> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
>> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
>> @@ -52,3 +52,9 @@
>>   &sata {
>>   	status = "okay";
>>   };
>> +
>> +&ipu1_csi1_from_mipi_vc1 {
>> +	data-lanes = <0 1>;
>> +	clock-lanes = <2>;
>> +};
>> +
>> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> index 795b5a5..bca9fed 100644
>> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> @@ -39,6 +39,8 @@
>>    *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>>    *     OTHER DEALINGS IN THE SOFTWARE.
>>    */
>> +
>> +#include <dt-bindings/clock/imx6qdl-clock.h>
>>   #include <dt-bindings/gpio/gpio.h>
>>   #include <dt-bindings/input/input.h>
>>   
>> @@ -96,6 +98,15 @@
>>   		};
>>   	};
>>   
>> +	mipi_xclk: mipi_xclk {
>> +		compatible = "pwm-clock";
>> +		#clock-cells = <0>;
>> +		clock-frequency = <22000000>;
>> +		clock-output-names = "mipi_pwm3";
>> +		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
>> +		status = "okay";
>> +	};
>> +
>>   	gpio-keys {
>>   		compatible = "gpio-keys";
>>   		pinctrl-names = "default";
>> @@ -220,6 +231,22 @@
>>   	};
>>   };
>>   
>> +&ipu1_csi0_from_ipu1_csi0_mux {
>> +	bus-width = <8>;
>> +	data-shift = <12>; /* Lines 19:12 used */
>> +	hsync-active = <1>;
>> +	vync-active = <1>;
>> +};
>> +
>> +&ipu1_csi0_mux_from_parallel_sensor {
>> +	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
>> +};
>> +
>> +&ipu1_csi0 {
>> +	pinctrl-names = "default";
>> +	pinctrl-0 = <&pinctrl_ipu1_csi0>;
>> +};
>> +
>>   &audmux {
>>   	pinctrl-names = "default";
>>   	pinctrl-0 = <&pinctrl_audmux>;
>> @@ -299,6 +326,52 @@
>>   	pinctrl-names = "default";
>>   	pinctrl-0 = <&pinctrl_i2c2>;
>>   	status = "okay";
>> +
>> +	ov5640: camera@40 {
>> +		compatible = "ovti,ov5640";
>> +		pinctrl-names = "default";
>> +		pinctrl-0 = <&pinctrl_ov5640>;
>> +		clocks = <&mipi_xclk>;
>> +		clock-names = "xclk";
>> +		reg = <0x40>;
>> +		xclk = <22000000>;
> This is superfluous, you can use clk_get_rate on mipi_xclk.

This property is actually there to tell the driver what to set the
rate to, with clk_set_rate(). So you are saying it would be better
to set the rate in the device tree and the driver should only
retrieve the rate?

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-13 23:04       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 23:04 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
>> Both hang off the same i2c2 bus, so they require different (and non-
>> default) i2c slave addresses.
>>
>> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
>>
>> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
>> mipi_csi. It is set to transmit over MIPI virtual channel 1.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
>>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
>>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
>>   3 files changed, 129 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> index 0f06ca5..fec2524 100644
>> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
>> @@ -48,3 +48,8 @@
>>   	model = "Freescale i.MX6 DualLite SABRE Lite Board";
>>   	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
>>   };
>> +
>> +&ipu1_csi1_from_ipu1_csi1_mux {
>> +	data-lanes = <0 1>;
>> +	clock-lanes = <2>;
>> +};
>> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
>> index 66d10d8..9e2d26d 100644
>> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
>> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
>> @@ -52,3 +52,9 @@
>>   &sata {
>>   	status = "okay";
>>   };
>> +
>> +&ipu1_csi1_from_mipi_vc1 {
>> +	data-lanes = <0 1>;
>> +	clock-lanes = <2>;
>> +};
>> +
>> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> index 795b5a5..bca9fed 100644
>> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
>> @@ -39,6 +39,8 @@
>>    *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>>    *     OTHER DEALINGS IN THE SOFTWARE.
>>    */
>> +
>> +#include <dt-bindings/clock/imx6qdl-clock.h>
>>   #include <dt-bindings/gpio/gpio.h>
>>   #include <dt-bindings/input/input.h>
>>   
>> @@ -96,6 +98,15 @@
>>   		};
>>   	};
>>   
>> +	mipi_xclk: mipi_xclk {
>> +		compatible = "pwm-clock";
>> +		#clock-cells = <0>;
>> +		clock-frequency = <22000000>;
>> +		clock-output-names = "mipi_pwm3";
>> +		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
>> +		status = "okay";
>> +	};
>> +
>>   	gpio-keys {
>>   		compatible = "gpio-keys";
>>   		pinctrl-names = "default";
>> @@ -220,6 +231,22 @@
>>   	};
>>   };
>>   
>> +&ipu1_csi0_from_ipu1_csi0_mux {
>> +	bus-width = <8>;
>> +	data-shift = <12>; /* Lines 19:12 used */
>> +	hsync-active = <1>;
>> +	vync-active = <1>;
>> +};
>> +
>> +&ipu1_csi0_mux_from_parallel_sensor {
>> +	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
>> +};
>> +
>> +&ipu1_csi0 {
>> +	pinctrl-names = "default";
>> +	pinctrl-0 = <&pinctrl_ipu1_csi0>;
>> +};
>> +
>>   &audmux {
>>   	pinctrl-names = "default";
>>   	pinctrl-0 = <&pinctrl_audmux>;
>> @@ -299,6 +326,52 @@
>>   	pinctrl-names = "default";
>>   	pinctrl-0 = <&pinctrl_i2c2>;
>>   	status = "okay";
>> +
>> +	ov5640: camera at 40 {
>> +		compatible = "ovti,ov5640";
>> +		pinctrl-names = "default";
>> +		pinctrl-0 = <&pinctrl_ov5640>;
>> +		clocks = <&mipi_xclk>;
>> +		clock-names = "xclk";
>> +		reg = <0x40>;
>> +		xclk = <22000000>;
> This is superfluous, you can use clk_get_rate on mipi_xclk.

This property is actually there to tell the driver what to set the
rate to, with clk_set_rate(). So you are saying it would be better
to set the rate in the device tree and the driver should only
retrieve the rate?

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 15/24] media: Add userspace header file for i.MX
  2017-01-13 12:05     ` Philipp Zabel
  (?)
@ 2017-01-13 23:13       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 23:13 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/13/2017 04:05 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> This adds a header file for use by userspace programs wanting to interact
>> with the i.MX media driver. It defines custom v4l2 controls and events
>> generated by the i.MX v4l2 subdevices.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   include/uapi/media/Kbuild |  1 +
>>   include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
>>   2 files changed, 31 insertions(+)
>>   create mode 100644 include/uapi/media/imx.h
>>
>> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
>> index aafaa5a..fa78958 100644
>> --- a/include/uapi/media/Kbuild
>> +++ b/include/uapi/media/Kbuild
>> @@ -1 +1,2 @@
>>   # UAPI Header export list
>> +header-y += imx.h
>> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
>> new file mode 100644
>> index 0000000..2421d9c
>> --- /dev/null
>> +++ b/include/uapi/media/imx.h
>> @@ -0,0 +1,30 @@
>> +/*
>> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by the
>> + * Free Software Foundation; either version 2 of the
>> + * License, or (at your option) any later version
>> + */
>> +
>> +#ifndef __UAPI_MEDIA_IMX_H__
>> +#define __UAPI_MEDIA_IMX_H__
>> +
>> +/*
>> + * events from the subdevs
>> + */
>> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
>> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
>> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
>> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
> Aren't these generic enough to warrant common events? I would think
> there have to be other capture IP cores that can signal aborted frames
> or frame timeouts.

Yes, agreed. A frame capture timeout, or frame interval error, are
both generic concepts. At some point it would be great to make the
Frame Interval Monitor generally available under v4l2-core. As for the
EOF timeout event, I'll look into moving that into a generic V4L2 event.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-01-13 23:13       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 23:13 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/13/2017 04:05 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> This adds a header file for use by userspace programs wanting to interact
>> with the i.MX media driver. It defines custom v4l2 controls and events
>> generated by the i.MX v4l2 subdevices.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   include/uapi/media/Kbuild |  1 +
>>   include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
>>   2 files changed, 31 insertions(+)
>>   create mode 100644 include/uapi/media/imx.h
>>
>> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
>> index aafaa5a..fa78958 100644
>> --- a/include/uapi/media/Kbuild
>> +++ b/include/uapi/media/Kbuild
>> @@ -1 +1,2 @@
>>   # UAPI Header export list
>> +header-y += imx.h
>> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
>> new file mode 100644
>> index 0000000..2421d9c
>> --- /dev/null
>> +++ b/include/uapi/media/imx.h
>> @@ -0,0 +1,30 @@
>> +/*
>> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by the
>> + * Free Software Foundation; either version 2 of the
>> + * License, or (at your option) any later version
>> + */
>> +
>> +#ifndef __UAPI_MEDIA_IMX_H__
>> +#define __UAPI_MEDIA_IMX_H__
>> +
>> +/*
>> + * events from the subdevs
>> + */
>> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
>> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
>> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
>> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
> Aren't these generic enough to warrant common events? I would think
> there have to be other capture IP cores that can signal aborted frames
> or frame timeouts.

Yes, agreed. A frame capture timeout, or frame interval error, are
both generic concepts. At some point it would be great to make the
Frame Interval Monitor generally available under v4l2-core. As for the
EOF timeout event, I'll look into moving that into a generic V4L2 event.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-01-13 23:13       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-13 23:13 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/13/2017 04:05 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> This adds a header file for use by userspace programs wanting to interact
>> with the i.MX media driver. It defines custom v4l2 controls and events
>> generated by the i.MX v4l2 subdevices.
>>
>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   include/uapi/media/Kbuild |  1 +
>>   include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
>>   2 files changed, 31 insertions(+)
>>   create mode 100644 include/uapi/media/imx.h
>>
>> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
>> index aafaa5a..fa78958 100644
>> --- a/include/uapi/media/Kbuild
>> +++ b/include/uapi/media/Kbuild
>> @@ -1 +1,2 @@
>>   # UAPI Header export list
>> +header-y += imx.h
>> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
>> new file mode 100644
>> index 0000000..2421d9c
>> --- /dev/null
>> +++ b/include/uapi/media/imx.h
>> @@ -0,0 +1,30 @@
>> +/*
>> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by the
>> + * Free Software Foundation; either version 2 of the
>> + * License, or (at your option) any later version
>> + */
>> +
>> +#ifndef __UAPI_MEDIA_IMX_H__
>> +#define __UAPI_MEDIA_IMX_H__
>> +
>> +/*
>> + * events from the subdevs
>> + */
>> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
>> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
>> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
>> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
> Aren't these generic enough to warrant common events? I would think
> there have to be other capture IP cores that can signal aborted frames
> or frame timeouts.

Yes, agreed. A frame capture timeout, or frame interval error, are
both generic concepts. At some point it would be great to make the
Frame Interval Monitor generally available under v4l2-core. As for the
EOF timeout event, I'll look into moving that into a generic V4L2 event.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-13 15:20     ` Philipp Zabel
  (?)
  (?)
@ 2017-01-14 22:42     ` Steve Longerbeam
  2017-01-19  1:44         ` Steve Longerbeam
  -1 siblings, 1 reply; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-14 22:42 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

[-- Attachment #1: Type: text/plain, Size: 20552 bytes --]



On 01/13/2017 07:20 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> +For image capture, the IPU contains the following internal subunits:
>> +
>> +- Image DMA Controller (IDMAC)
>> +- Camera Serial Interface (CSI)
>> +- Image Converter (IC)
>> +- Sensor Multi-FIFO Controller (SMFC)
>> +- Image Rotator (IRT)
>> +- Video De-Interlace Controller (VDIC)
> Nitpick: Video De-Interlacing or Combining Block (VDIC)

done.

>> +
>> +The IDMAC is the DMA controller for transfer of image frames to and from
>> +memory. Various dedicated DMA channels exist for both video capture and
>> +display paths.
>> +
>> +The CSI is the frontend capture unit that interfaces directly with
>> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
>> +
>> +The IC handles color-space conversion, resizing, and rotation
>> +operations.
> And horizontal flipping.

done.


>> There are three independent "tasks" within the IC that can
>> +carry out conversions concurrently: pre-processing encoding,
>> +pre-processing preview, and post-processing.
> s/preview/viewfinder/ seems to be the commonly used name.

replaced everywhere in the doc.

> This paragraph could mention that a single hardware unit is used
> transparently time multiplexed by the three tasks at different
> granularity for the downsizing, main processing, and rotation sections.
> The downscale unit switches between tasks at 8-pixel burst granularity,
> the main processing unit at line granularity. The rotation units switch
> only at frame granularity.

I've added that info.

>> +The SMFC is composed of four independent channels that each can transfer
>> +captured frames from sensors directly to memory concurrently.
>> +
>> +The IRT carries out 90 and 270 degree image rotation operations.
> ... on 8x8 pixel blocks, supported by the IDMAC which handles block
> transfers, block reordering, and vertical flipping.

done.

>> +The VDIC handles the conversion of interlaced video to progressive, with
>> +support for different motion compensation modes (low, medium, and high
>> +motion). The deinterlaced output frames from the VDIC can be sent to the
>> +IC pre-process preview task for further conversions.
>> +
>> +In addition to the IPU internal subunits, there are also two units
>> +outside the IPU that are also involved in video capture on i.MX:
>> +
>> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
>> +  interface. This is a Synopsys DesignWare core.
>> +- A video multiplexer for selecting among multiple sensor inputs to
>> +  send to a CSI.
> Two of them, actually.

done.

>> +
>> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
>> +  problems with the ADV718x video decoders. See below for a description
>> +  of the FIM.
> Could this also be used to calculate more precise capture timestamps?

An input capture function could do that, triggered off a VSYNC or FIELD
signal such as on the ADV718x. The FIM is only used to calculate
frame intervals at this point, but its input capture method could be
used to also record more accurate timestamps.


>> +Capture Pipelines
>> +-----------------
>> +
>> +The following describe the various use-cases supported by the pipelines.
>> +
>> +The links shown do not include the frontend sensor, video mux, or mipi
>> +csi-2 receiver links. This depends on the type of sensor interface
>> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
>> +
>> +sensor -> ipu_csi_mux -> ipu_csi -> ...
>> +
>> +for parallel sensors, or:
>> +
>> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
>> +
>> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
>> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
>> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
>> +parenthesis.
>> +
>> +Unprocessed Video Capture:
>> +--------------------------
>> +
>> +Send frames directly from sensor to camera interface, with no
>> +conversions:
>> +
>> +-> ipu_smfc -> camif
> I'd call this capture interface, this is not just for cameras. Or maybe
> idmac if you want to mirror hardware names?

Camif is so named because it is the V4L2 user interface for video
capture. I suppose it could be named "capif", but that doesn't role
off the tongue quite as well.

>> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

>> +For example, its sink pad can take UYVY2X8, but its source pad can
>> +output YUYV2X8.
> I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> in the CSI chapter, for 8-bit per component data, the internal format
> between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> (or "bayer/generic data"). In either case, the internal format does not
> change along the way.

these are pixels in memory buffers, not the IPU internal formats.


>> +IC Direct Conversions:
>> +----------------------
>> +
>> +This pipeline uses the preprocess encode entity to route frames directly
>> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
>> +1024x1024 resolution, CSC, and image rotation:
>> +
>> +-> ipu_ic_prpenc -> camif
>> +
>> +This can be a useful capture pipeline for heavily loaded memory bus
>> +traffic environments, since it has minimal IDMAC channel usage.
> Note that if rotation is enabled, transfers between IC processing and
> rotation still have to go through memory once.

yep.

>> +Post-Processing Conversions:
>> +----------------------------
>> +
>> +This pipeline routes frames from the SMFC to the post-processing
>> +entity.
> No, frames written by the CSI -> SMFC -> IDMAC path are read back into
> the post-processing entity.

that's true. The post-processing entity kicks off its read channels
to transfer those frames into the post-processor. Anyway this wording
will change after doing away with the SMFC entity.


>> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"
> I think the smfc entities should be dropped.

yes working on that.

>> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
>> +   # Configure pads for OV5640 pipeline
>> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> [...]
>> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> I agree this looks very intuitive, but technically correct for the
> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>
> I think it would be better to use the correct format

I'm not sure I follow you here.

>   as that will allow
> to chose the regular vs. companded packings in the future for formats
> with more than 8 bits per component.
>
>> +   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>> +
>> +Streaming can then begin independently on device nodes /dev/video0
>> +and /dev/video1.
>> +
>> +SabreAuto with ADV7180 decoder
>> +------------------------------
>> +
>> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
>> +parallel bus input on the internal video mux to IPU1 CSI0.
>> +
>> +The following example configures a pipeline to capture from the ADV7180
>> +video decoder, assuming NTSC 720x480 input signals, with Motion
>> +Compensated de-interlacing (not shown: all pad field types should be set
>> +as indicated). $outputfmt can be any format supported by the
>> +ipu1_ic_prpvf entity at its output pad:
>> +
>> +.. code-block:: none
>> +
>> +   # Setup links
>> +   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
>> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
>> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
>> +   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
>> +   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
>> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
>> +   # Configure pads
>> +   # pad field types for below pads must be an interlaced type
>> +   # such as "ALTERNATE"
> I think alternate should only extend as far as the CSI, since the CSI
> can only capture NTSC/PAL fields in a fixed order.

Agreed, I'm doing the translation from alternate to seq_bt/seq_tb depending
on sensor std in imx-vdic.c, but it should move upstream. Will fix.

>> +   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
>> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
>  From here the interlaced field type should be sequential in the correct
> order depending on NTSC/PAL.

right.

>> +
>> +Frame Interval Monitor
>> +----------------------
>> +
>> +The adv718x decoders can occasionally send corrupt fields during
>> +NTSC/PAL signal re-sync (too little or too many video lines). When
>> +this happens, the IPU triggers a mechanism to re-establish vertical
>> +sync by adding 1 dummy line every frame, which causes a rolling effect
>> +from image to image, and can last a long time before a stable image is
>> +recovered. Or sometimes the mechanism doesn't work at all, causing a
>> +permanent split image (one frame contains lines from two consecutive
>> +captured images).
> Is it only SabreAuto on which the FIM mechanism can be used due to the
> pad routing?

No, FIM can be used on any target, the fim child node of ipu_csi just
needs to be enabled via status property.

> [...]
>> +/*
>> + * DMA buffer ring handling
>> + */
>> +struct imx_media_dma_buf_ring {
>> +	struct imx_media_dev *imxmd;
>> +
>> +	/* the ring */
>> +	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
>> +	/* the scratch buffer for underruns */
>> +	struct imx_media_dma_buf scratch;
>> +
>> +	/* buffer generator */
>> +	struct media_entity *src;
>> +	/* buffer receiver */
>> +	struct media_entity *sink;
>> +
>> +	spinlock_t lock;
>> +
>> +	int num_bufs;
>> +	unsigned long last_seq;
>> +};
> I don't think this belongs in the capture driver at all.
> Memory-to-memory transfers should be handled at the videobuf2 level.

see below.

> [...]
>> +static struct imx_media_dma_buf *
>> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
>> +{
>> +	struct imx_media_dma_buf *buf;
>> +
>> +	if (index >= ring->num_bufs)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf = &ring->buf[index];
>> +	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
>> +	buf->seq = ring->last_seq++;
>> +
>> +	return buf;
>> +}
> Is this a whole software buffer queue implementation? I thought the
> whole point of putting the custom mem2mem framework into the capture
> driver was to use the hardware FSU channel linking?

  see below.

> What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries
in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

There does not appear to be support in the FSU for linking a write channel
to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
is support for the direct link from CSI (which I am using), but that's 
not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
But I think this links to channel 12, and not to channels 8,9,10 to the 
VDIC.
Or will it? It's worth experimenting. It would have helped if FSL listed 
which
IDMAC channels these FSU links correspond to, instead of making us guess
at it.

In any event, the docs are not clear enough to implement a real FSU
link to the VDIC read channels, if it's even possible. And trying to get
programming help from FSL can be difficult, and no coding examples
for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software
approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC
entity via a v4l2_subdev_ioctl() call that a buffer is available. The
VDIC then manually kicks off its read channels to bring that buffer
(and a previous buffer for F(n-1) field) into the VDIC.

There is a small amount of extra overhead going this route compared
to a FSU hardware link: there is the EOF irq latency (a few usec), and
the CPU overhead for the VDIC to manually start the read channels,
which is also a few usec at most (see prepare_vdi_in_buffers() in
imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
use plus irq latency) under normal system load.

Of course, in order to implement this software link, I had to implement
a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
at stream on, and the source requests a pointer to this ring in its own
stream on. Passing buffers from source to sink then follows a 
straightforward
FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
grabs queued buffers and makes them active, src signals completed
buffers to sink, sink dequeues buffers in response, and sink queues
buffers back when it is finished with them.


> [...]
>> +/*
>> + * The subdevs have to be powered on/off, and streaming
>> + * enabled/disabled, in a specific sequence.
>> + */
>> +static const u32 stream_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +};
>> +
>> +static const u32 stream_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +};
>> +
>> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
>> +
>> +static const u32 power_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +};
>> +
>> +static const u32 power_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +};
> This seems somewhat arbitrary. Why is a power sequence needed?

The CSI-2 receiver must be powered up before the sensor, that's the
only requirement IIRC. The others have no s_power requirement. So I
can probably change this to power up in the frontend -> backend order
(IC_PP to sensor). And vice-versa for power off.


> [...]
>> +/*
>> + * Turn current pipeline power on/off starting from start_entity.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
>> +				 struct media_entity_graph *graph,
>> +				 struct media_entity *start_entity, bool on)
>> +{
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int i, ret = 0;
>> +	u32 id;
>> +
>> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
>> +		id = on ? power_on_seq[i] : power_off_seq[i];
>> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
>> +		if (!entity)
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		ret = v4l2_subdev_call(sd, core, s_power, on);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			break;
>> +	}
>> +
>> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
doing some other stuff that bothered me, at least that's what I remember.
I will revisit this.

>> +/*
>> + * Inherit the v4l2 controls from all entities in a pipeline
>> + * to the given video device.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
>> +			       struct video_device *vfd,
>> +			       struct media_entity *start_entity)
>> +{
>> +	struct media_entity_graph graph;
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int ret;
>> +
>> +	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_entity_graph_walk_start(&graph, start_entity);
>> +
>> +	while ((entity = media_entity_graph_walk_next(&graph))) {
>> +		if (is_media_entity_v4l2_video_device(entity))
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
>> +			__func__, sd->name);
>> +
>> +		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
>> +					    sd->ctrl_handler,
>> +					    NULL);
>> +		if (ret)
>> +			break;
>> +	}
>> +
>> +	media_entity_graph_walk_cleanup(&graph);
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
>> +
>> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
>> new file mode 100644
>> index 0000000..357654d
>> --- /dev/null
>> +++ b/drivers/staging/media/imx/imx-media-dev.c
> This file is full of code that should live in the v4l2 core.

Agreed. Stuff like imx_media_inherit_controls() really should be widely
available.

>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *csi[4])
>> +{
>> +	int ret;
>> +
>> +	/* there must be at least one CSI in first IPU */
> Why?

Well yeah, imx-media doesn't necessarily need a CSI if things
like the VDIC or post-processor are being used by an output
overlay pipeline, for example. I'll fix this.

>> +
>> +/* parse inputs property from a sensor node */
>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *sensor,
>> +				   struct device_node *sensor_np)
>> +{
>> +	struct imx_media_sensor_input *sinput = &sensor->input;
>> +	int ret, i;
>> +
>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
>> +		const char *input_name;
>> +		u32 val;
>> +
>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
>> +		if (ret)
>> +			break;
>> +
>> +		sinput->value[i] = val;
>> +
>> +		ret = of_property_read_string_index(sensor_np, "input-names",
>> +						    i, &input_name);
>> +		/*
>> +		 * if input-names not provided, they will be set using
>> +		 * the subdev name once the sensor is known during
>> +		 * async bind
>> +		 */
>> +		if (!ret)
>> +			strncpy(sinput->name[i], input_name,
>> +				sizeof(sinput->name[i]));
>> +	}
>> +
>> +	sinput->num = i;
>> +
>> +	/* if no inputs provided just assume a single input */
>> +	if (sinput->num == 0)
>> +		sinput->num = 1;
>> +}
> This should be parsed by the sensor driver, not imx-media.

you're probably right. I'll submit a patch for adv7180.c.


>> +static void of_parse_sensor(struct imx_media_dev *imxmd,
>> +			    struct imx_media_subdev *sensor,
>> +			    struct device_node *sensor_np)
>> +{
>> +	struct device_node *endpoint;
>> +
>> +	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
>> +
>> +	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
>> +	if (endpoint) {
>> +		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
>> +		of_node_put(endpoint);
>> +	}
>> +}
>> +
>> +static int of_get_port_count(const struct device_node *np)
>> +{
>> +	struct device_node *child;
>> +	int num = 0;
>> +
>> +	/* if this node is itself a port, return 1 */
>> +	if (of_node_cmp(np->name, "port") == 0)
>> +		return 1;
>> +
>> +	for_each_child_of_node(np, child)
>> +		if (of_node_cmp(child->name, "port") == 0)
>> +			num++;
>> +
>> +	return num;
>> +}
> If this is extended to handle the ports subnode properly, it could be
> moved into drivers/of/base.c.

More that eventually needs exporting, agreed.


Steve


[-- Attachment #2: Type: text/html, Size: 27007 bytes --]

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-13 15:20     ` Philipp Zabel
  (?)
@ 2017-01-14 22:46       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-14 22:46 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

(sorry sending again w/o html)


On 01/13/2017 07:20 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> +For image capture, the IPU contains the following internal subunits:
>> +
>> +- Image DMA Controller (IDMAC)
>> +- Camera Serial Interface (CSI)
>> +- Image Converter (IC)
>> +- Sensor Multi-FIFO Controller (SMFC)
>> +- Image Rotator (IRT)
>> +- Video De-Interlace Controller (VDIC)
> Nitpick: Video De-Interlacing or Combining Block (VDIC)

done.

>> +
>> +The IDMAC is the DMA controller for transfer of image frames to and from
>> +memory. Various dedicated DMA channels exist for both video capture and
>> +display paths.
>> +
>> +The CSI is the frontend capture unit that interfaces directly with
>> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
>> +
>> +The IC handles color-space conversion, resizing, and rotation
>> +operations.
> And horizontal flipping.

done.


>> There are three independent "tasks" within the IC that can
>> +carry out conversions concurrently: pre-processing encoding,
>> +pre-processing preview, and post-processing.
> s/preview/viewfinder/ seems to be the commonly used name.

replaced everywhere in the doc.

> This paragraph could mention that a single hardware unit is used
> transparently time multiplexed by the three tasks at different
> granularity for the downsizing, main processing, and rotation sections.
> The downscale unit switches between tasks at 8-pixel burst granularity,
> the main processing unit at line granularity. The rotation units switch
> only at frame granularity.

I've added that info.

>> +The SMFC is composed of four independent channels that each can transfer
>> +captured frames from sensors directly to memory concurrently.
>> +
>> +The IRT carries out 90 and 270 degree image rotation operations.
> ... on 8x8 pixel blocks, supported by the IDMAC which handles block
> transfers, block reordering, and vertical flipping.

done.

>> +The VDIC handles the conversion of interlaced video to progressive, with
>> +support for different motion compensation modes (low, medium, and high
>> +motion). The deinterlaced output frames from the VDIC can be sent to the
>> +IC pre-process preview task for further conversions.
>> +
>> +In addition to the IPU internal subunits, there are also two units
>> +outside the IPU that are also involved in video capture on i.MX:
>> +
>> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
>> +  interface. This is a Synopsys DesignWare core.
>> +- A video multiplexer for selecting among multiple sensor inputs to
>> +  send to a CSI.
> Two of them, actually.

done.

>> +
>> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
>> +  problems with the ADV718x video decoders. See below for a description
>> +  of the FIM.
> Could this also be used to calculate more precise capture timestamps?

An input capture function could do that, triggered off a VSYNC or FIELD
signal such as on the ADV718x. The FIM is only used to calculate
frame intervals at this point, but its input capture method could be
used to also record more accurate timestamps.


>> +Capture Pipelines
>> +-----------------
>> +
>> +The following describe the various use-cases supported by the pipelines.
>> +
>> +The links shown do not include the frontend sensor, video mux, or mipi
>> +csi-2 receiver links. This depends on the type of sensor interface
>> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
>> +
>> +sensor -> ipu_csi_mux -> ipu_csi -> ...
>> +
>> +for parallel sensors, or:
>> +
>> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
>> +
>> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
>> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
>> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
>> +parenthesis.
>> +
>> +Unprocessed Video Capture:
>> +--------------------------
>> +
>> +Send frames directly from sensor to camera interface, with no
>> +conversions:
>> +
>> +-> ipu_smfc -> camif
> I'd call this capture interface, this is not just for cameras. Or maybe
> idmac if you want to mirror hardware names?

Camif is so named because it is the V4L2 user interface for video
capture. I suppose it could be named "capif", but that doesn't role
off the tongue quite as well.

>> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

>> +For example, its sink pad can take UYVY2X8, but its source pad can
>> +output YUYV2X8.
> I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> in the CSI chapter, for 8-bit per component data, the internal format
> between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> (or "bayer/generic data"). In either case, the internal format does not
> change along the way.

these are pixels in memory buffers, not the IPU internal formats.


>> +IC Direct Conversions:
>> +----------------------
>> +
>> +This pipeline uses the preprocess encode entity to route frames directly
>> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
>> +1024x1024 resolution, CSC, and image rotation:
>> +
>> +-> ipu_ic_prpenc -> camif
>> +
>> +This can be a useful capture pipeline for heavily loaded memory bus
>> +traffic environments, since it has minimal IDMAC channel usage.
> Note that if rotation is enabled, transfers between IC processing and
> rotation still have to go through memory once.

yep.

>> +Post-Processing Conversions:
>> +----------------------------
>> +
>> +This pipeline routes frames from the SMFC to the post-processing
>> +entity.
> No, frames written by the CSI -> SMFC -> IDMAC path are read back into
> the post-processing entity.

that's true. The post-processing entity kicks off its read channels
to transfer those frames into the post-processor. Anyway this wording
will change after doing away with the SMFC entity.


>> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"
> I think the smfc entities should be dropped.

yes working on that.

>> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
>> +   # Configure pads for OV5640 pipeline
>> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> [...]
>> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> I agree this looks very intuitive, but technically correct for the
> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>
> I think it would be better to use the correct format

I'm not sure I follow you here.

>   as that will allow
> to chose the regular vs. companded packings in the future for formats
> with more than 8 bits per component.
>
>> +   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>> +
>> +Streaming can then begin independently on device nodes /dev/video0
>> +and /dev/video1.
>> +
>> +SabreAuto with ADV7180 decoder
>> +------------------------------
>> +
>> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
>> +parallel bus input on the internal video mux to IPU1 CSI0.
>> +
>> +The following example configures a pipeline to capture from the ADV7180
>> +video decoder, assuming NTSC 720x480 input signals, with Motion
>> +Compensated de-interlacing (not shown: all pad field types should be set
>> +as indicated). $outputfmt can be any format supported by the
>> +ipu1_ic_prpvf entity at its output pad:
>> +
>> +.. code-block:: none
>> +
>> +   # Setup links
>> +   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
>> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
>> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
>> +   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
>> +   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
>> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
>> +   # Configure pads
>> +   # pad field types for below pads must be an interlaced type
>> +   # such as "ALTERNATE"
> I think alternate should only extend as far as the CSI, since the CSI
> can only capture NTSC/PAL fields in a fixed order.

Agreed, I'm doing the translation from alternate to seq_bt/seq_tb depending
on sensor std in imx-vdic.c, but it should move upstream. Will fix.

>> +   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
>> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
>  From here the interlaced field type should be sequential in the correct
> order depending on NTSC/PAL.

right.

>> +
>> +Frame Interval Monitor
>> +----------------------
>> +
>> +The adv718x decoders can occasionally send corrupt fields during
>> +NTSC/PAL signal re-sync (too little or too many video lines). When
>> +this happens, the IPU triggers a mechanism to re-establish vertical
>> +sync by adding 1 dummy line every frame, which causes a rolling effect
>> +from image to image, and can last a long time before a stable image is
>> +recovered. Or sometimes the mechanism doesn't work at all, causing a
>> +permanent split image (one frame contains lines from two consecutive
>> +captured images).
> Is it only SabreAuto on which the FIM mechanism can be used due to the
> pad routing?

No, FIM can be used on any target, the fim child node of ipu_csi just
needs to be enabled via status property.

> [...]
>> +/*
>> + * DMA buffer ring handling
>> + */
>> +struct imx_media_dma_buf_ring {
>> +	struct imx_media_dev *imxmd;
>> +
>> +	/* the ring */
>> +	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
>> +	/* the scratch buffer for underruns */
>> +	struct imx_media_dma_buf scratch;
>> +
>> +	/* buffer generator */
>> +	struct media_entity *src;
>> +	/* buffer receiver */
>> +	struct media_entity *sink;
>> +
>> +	spinlock_t lock;
>> +
>> +	int num_bufs;
>> +	unsigned long last_seq;
>> +};
> I don't think this belongs in the capture driver at all.
> Memory-to-memory transfers should be handled at the videobuf2 level.

see below.

> [...]
>> +static struct imx_media_dma_buf *
>> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
>> +{
>> +	struct imx_media_dma_buf *buf;
>> +
>> +	if (index >= ring->num_bufs)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf = &ring->buf[index];
>> +	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
>> +	buf->seq = ring->last_seq++;
>> +
>> +	return buf;
>> +}
> Is this a whole software buffer queue implementation? I thought the
> whole point of putting the custom mem2mem framework into the capture
> driver was to use the hardware FSU channel linking?

  see below.

> What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries
in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

There does not appear to be support in the FSU for linking a write channel
to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
is support for the direct link from CSI (which I am using), but that's 
not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
But I think this links to channel 12, and not to channels 8,9,10 to the 
VDIC.
Or will it? It's worth experimenting. It would have helped if FSL listed 
which
IDMAC channels these FSU links correspond to, instead of making us guess
at it.

In any event, the docs are not clear enough to implement a real FSU
link to the VDIC read channels, if it's even possible. And trying to get
programming help from FSL can be difficult, and no coding examples
for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software
approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC
entity via a v4l2_subdev_ioctl() call that a buffer is available. The
VDIC then manually kicks off its read channels to bring that buffer
(and a previous buffer for F(n-1) field) into the VDIC.

There is a small amount of extra overhead going this route compared
to a FSU hardware link: there is the EOF irq latency (a few usec), and
the CPU overhead for the VDIC to manually start the read channels,
which is also a few usec at most (see prepare_vdi_in_buffers() in
imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
use plus irq latency) under normal system load.

Of course, in order to implement this software link, I had to implement
a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
at stream on, and the source requests a pointer to this ring in its own
stream on. Passing buffers from source to sink then follows a 
straightforward
FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
grabs queued buffers and makes them active, src signals completed
buffers to sink, sink dequeues buffers in response, and sink queues
buffers back when it is finished with them.


> [...]
>> +/*
>> + * The subdevs have to be powered on/off, and streaming
>> + * enabled/disabled, in a specific sequence.
>> + */
>> +static const u32 stream_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +};
>> +
>> +static const u32 stream_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +};
>> +
>> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
>> +
>> +static const u32 power_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +};
>> +
>> +static const u32 power_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +};
> This seems somewhat arbitrary. Why is a power sequence needed?

The CSI-2 receiver must be powered up before the sensor, that's the
only requirement IIRC. The others have no s_power requirement. So I
can probably change this to power up in the frontend -> backend order
(IC_PP to sensor). And vice-versa for power off.


> [...]
>> +/*
>> + * Turn current pipeline power on/off starting from start_entity.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
>> +				 struct media_entity_graph *graph,
>> +				 struct media_entity *start_entity, bool on)
>> +{
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int i, ret = 0;
>> +	u32 id;
>> +
>> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
>> +		id = on ? power_on_seq[i] : power_off_seq[i];
>> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
>> +		if (!entity)
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		ret = v4l2_subdev_call(sd, core, s_power, on);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			break;
>> +	}
>> +
>> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
doing some other stuff that bothered me, at least that's what I remember.
I will revisit this.

>> +/*
>> + * Inherit the v4l2 controls from all entities in a pipeline
>> + * to the given video device.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
>> +			       struct video_device *vfd,
>> +			       struct media_entity *start_entity)
>> +{
>> +	struct media_entity_graph graph;
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int ret;
>> +
>> +	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_entity_graph_walk_start(&graph, start_entity);
>> +
>> +	while ((entity = media_entity_graph_walk_next(&graph))) {
>> +		if (is_media_entity_v4l2_video_device(entity))
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
>> +			__func__, sd->name);
>> +
>> +		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
>> +					    sd->ctrl_handler,
>> +					    NULL);
>> +		if (ret)
>> +			break;
>> +	}
>> +
>> +	media_entity_graph_walk_cleanup(&graph);
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
>> +
>> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
>> new file mode 100644
>> index 0000000..357654d
>> --- /dev/null
>> +++ b/drivers/staging/media/imx/imx-media-dev.c
> This file is full of code that should live in the v4l2 core.

Agreed. Stuff like imx_media_inherit_controls() really should be widely
available.

>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *csi[4])
>> +{
>> +	int ret;
>> +
>> +	/* there must be at least one CSI in first IPU */
> Why?

Well yeah, imx-media doesn't necessarily need a CSI if things
like the VDIC or post-processor are being used by an output
overlay pipeline, for example. I'll fix this.

>> +
>> +/* parse inputs property from a sensor node */
>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *sensor,
>> +				   struct device_node *sensor_np)
>> +{
>> +	struct imx_media_sensor_input *sinput = &sensor->input;
>> +	int ret, i;
>> +
>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
>> +		const char *input_name;
>> +		u32 val;
>> +
>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
>> +		if (ret)
>> +			break;
>> +
>> +		sinput->value[i] = val;
>> +
>> +		ret = of_property_read_string_index(sensor_np, "input-names",
>> +						    i, &input_name);
>> +		/*
>> +		 * if input-names not provided, they will be set using
>> +		 * the subdev name once the sensor is known during
>> +		 * async bind
>> +		 */
>> +		if (!ret)
>> +			strncpy(sinput->name[i], input_name,
>> +				sizeof(sinput->name[i]));
>> +	}
>> +
>> +	sinput->num = i;
>> +
>> +	/* if no inputs provided just assume a single input */
>> +	if (sinput->num == 0)
>> +		sinput->num = 1;
>> +}
> This should be parsed by the sensor driver, not imx-media.

you're probably right. I'll submit a patch for adv7180.c.


>> +static void of_parse_sensor(struct imx_media_dev *imxmd,
>> +			    struct imx_media_subdev *sensor,
>> +			    struct device_node *sensor_np)
>> +{
>> +	struct device_node *endpoint;
>> +
>> +	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
>> +
>> +	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
>> +	if (endpoint) {
>> +		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
>> +		of_node_put(endpoint);
>> +	}
>> +}
>> +
>> +static int of_get_port_count(const struct device_node *np)
>> +{
>> +	struct device_node *child;
>> +	int num = 0;
>> +
>> +	/* if this node is itself a port, return 1 */
>> +	if (of_node_cmp(np->name, "port") == 0)
>> +		return 1;
>> +
>> +	for_each_child_of_node(np, child)
>> +		if (of_node_cmp(child->name, "port") == 0)
>> +			num++;
>> +
>> +	return num;
>> +}
> If this is extended to handle the ports subnode properly, it could be
> moved into drivers/of/base.c.

More that eventually needs exporting, agreed.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-14 22:46       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-14 22:46 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

(sorry sending again w/o html)


On 01/13/2017 07:20 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> +For image capture, the IPU contains the following internal subunits:
>> +
>> +- Image DMA Controller (IDMAC)
>> +- Camera Serial Interface (CSI)
>> +- Image Converter (IC)
>> +- Sensor Multi-FIFO Controller (SMFC)
>> +- Image Rotator (IRT)
>> +- Video De-Interlace Controller (VDIC)
> Nitpick: Video De-Interlacing or Combining Block (VDIC)

done.

>> +
>> +The IDMAC is the DMA controller for transfer of image frames to and from
>> +memory. Various dedicated DMA channels exist for both video capture and
>> +display paths.
>> +
>> +The CSI is the frontend capture unit that interfaces directly with
>> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
>> +
>> +The IC handles color-space conversion, resizing, and rotation
>> +operations.
> And horizontal flipping.

done.


>> There are three independent "tasks" within the IC that can
>> +carry out conversions concurrently: pre-processing encoding,
>> +pre-processing preview, and post-processing.
> s/preview/viewfinder/ seems to be the commonly used name.

replaced everywhere in the doc.

> This paragraph could mention that a single hardware unit is used
> transparently time multiplexed by the three tasks at different
> granularity for the downsizing, main processing, and rotation sections.
> The downscale unit switches between tasks at 8-pixel burst granularity,
> the main processing unit at line granularity. The rotation units switch
> only at frame granularity.

I've added that info.

>> +The SMFC is composed of four independent channels that each can transfer
>> +captured frames from sensors directly to memory concurrently.
>> +
>> +The IRT carries out 90 and 270 degree image rotation operations.
> ... on 8x8 pixel blocks, supported by the IDMAC which handles block
> transfers, block reordering, and vertical flipping.

done.

>> +The VDIC handles the conversion of interlaced video to progressive, with
>> +support for different motion compensation modes (low, medium, and high
>> +motion). The deinterlaced output frames from the VDIC can be sent to the
>> +IC pre-process preview task for further conversions.
>> +
>> +In addition to the IPU internal subunits, there are also two units
>> +outside the IPU that are also involved in video capture on i.MX:
>> +
>> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
>> +  interface. This is a Synopsys DesignWare core.
>> +- A video multiplexer for selecting among multiple sensor inputs to
>> +  send to a CSI.
> Two of them, actually.

done.

>> +
>> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
>> +  problems with the ADV718x video decoders. See below for a description
>> +  of the FIM.
> Could this also be used to calculate more precise capture timestamps?

An input capture function could do that, triggered off a VSYNC or FIELD
signal such as on the ADV718x. The FIM is only used to calculate
frame intervals at this point, but its input capture method could be
used to also record more accurate timestamps.


>> +Capture Pipelines
>> +-----------------
>> +
>> +The following describe the various use-cases supported by the pipelines.
>> +
>> +The links shown do not include the frontend sensor, video mux, or mipi
>> +csi-2 receiver links. This depends on the type of sensor interface
>> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
>> +
>> +sensor -> ipu_csi_mux -> ipu_csi -> ...
>> +
>> +for parallel sensors, or:
>> +
>> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
>> +
>> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
>> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
>> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
>> +parenthesis.
>> +
>> +Unprocessed Video Capture:
>> +--------------------------
>> +
>> +Send frames directly from sensor to camera interface, with no
>> +conversions:
>> +
>> +-> ipu_smfc -> camif
> I'd call this capture interface, this is not just for cameras. Or maybe
> idmac if you want to mirror hardware names?

Camif is so named because it is the V4L2 user interface for video
capture. I suppose it could be named "capif", but that doesn't role
off the tongue quite as well.

>> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

>> +For example, its sink pad can take UYVY2X8, but its source pad can
>> +output YUYV2X8.
> I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> in the CSI chapter, for 8-bit per component data, the internal format
> between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> (or "bayer/generic data"). In either case, the internal format does not
> change along the way.

these are pixels in memory buffers, not the IPU internal formats.


>> +IC Direct Conversions:
>> +----------------------
>> +
>> +This pipeline uses the preprocess encode entity to route frames directly
>> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
>> +1024x1024 resolution, CSC, and image rotation:
>> +
>> +-> ipu_ic_prpenc -> camif
>> +
>> +This can be a useful capture pipeline for heavily loaded memory bus
>> +traffic environments, since it has minimal IDMAC channel usage.
> Note that if rotation is enabled, transfers between IC processing and
> rotation still have to go through memory once.

yep.

>> +Post-Processing Conversions:
>> +----------------------------
>> +
>> +This pipeline routes frames from the SMFC to the post-processing
>> +entity.
> No, frames written by the CSI -> SMFC -> IDMAC path are read back into
> the post-processing entity.

that's true. The post-processing entity kicks off its read channels
to transfer those frames into the post-processor. Anyway this wording
will change after doing away with the SMFC entity.


>> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"
> I think the smfc entities should be dropped.

yes working on that.

>> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
>> +   # Configure pads for OV5640 pipeline
>> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> [...]
>> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> I agree this looks very intuitive, but technically correct for the
> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>
> I think it would be better to use the correct format

I'm not sure I follow you here.

>   as that will allow
> to chose the regular vs. companded packings in the future for formats
> with more than 8 bits per component.
>
>> +   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>> +
>> +Streaming can then begin independently on device nodes /dev/video0
>> +and /dev/video1.
>> +
>> +SabreAuto with ADV7180 decoder
>> +------------------------------
>> +
>> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
>> +parallel bus input on the internal video mux to IPU1 CSI0.
>> +
>> +The following example configures a pipeline to capture from the ADV7180
>> +video decoder, assuming NTSC 720x480 input signals, with Motion
>> +Compensated de-interlacing (not shown: all pad field types should be set
>> +as indicated). $outputfmt can be any format supported by the
>> +ipu1_ic_prpvf entity at its output pad:
>> +
>> +.. code-block:: none
>> +
>> +   # Setup links
>> +   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
>> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
>> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
>> +   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
>> +   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
>> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
>> +   # Configure pads
>> +   # pad field types for below pads must be an interlaced type
>> +   # such as "ALTERNATE"
> I think alternate should only extend as far as the CSI, since the CSI
> can only capture NTSC/PAL fields in a fixed order.

Agreed, I'm doing the translation from alternate to seq_bt/seq_tb depending
on sensor std in imx-vdic.c, but it should move upstream. Will fix.

>> +   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
>> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
>  From here the interlaced field type should be sequential in the correct
> order depending on NTSC/PAL.

right.

>> +
>> +Frame Interval Monitor
>> +----------------------
>> +
>> +The adv718x decoders can occasionally send corrupt fields during
>> +NTSC/PAL signal re-sync (too little or too many video lines). When
>> +this happens, the IPU triggers a mechanism to re-establish vertical
>> +sync by adding 1 dummy line every frame, which causes a rolling effect
>> +from image to image, and can last a long time before a stable image is
>> +recovered. Or sometimes the mechanism doesn't work at all, causing a
>> +permanent split image (one frame contains lines from two consecutive
>> +captured images).
> Is it only SabreAuto on which the FIM mechanism can be used due to the
> pad routing?

No, FIM can be used on any target, the fim child node of ipu_csi just
needs to be enabled via status property.

> [...]
>> +/*
>> + * DMA buffer ring handling
>> + */
>> +struct imx_media_dma_buf_ring {
>> +	struct imx_media_dev *imxmd;
>> +
>> +	/* the ring */
>> +	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
>> +	/* the scratch buffer for underruns */
>> +	struct imx_media_dma_buf scratch;
>> +
>> +	/* buffer generator */
>> +	struct media_entity *src;
>> +	/* buffer receiver */
>> +	struct media_entity *sink;
>> +
>> +	spinlock_t lock;
>> +
>> +	int num_bufs;
>> +	unsigned long last_seq;
>> +};
> I don't think this belongs in the capture driver at all.
> Memory-to-memory transfers should be handled at the videobuf2 level.

see below.

> [...]
>> +static struct imx_media_dma_buf *
>> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
>> +{
>> +	struct imx_media_dma_buf *buf;
>> +
>> +	if (index >= ring->num_bufs)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf = &ring->buf[index];
>> +	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
>> +	buf->seq = ring->last_seq++;
>> +
>> +	return buf;
>> +}
> Is this a whole software buffer queue implementation? I thought the
> whole point of putting the custom mem2mem framework into the capture
> driver was to use the hardware FSU channel linking?

  see below.

> What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries
in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

There does not appear to be support in the FSU for linking a write channel
to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
is support for the direct link from CSI (which I am using), but that's 
not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
But I think this links to channel 12, and not to channels 8,9,10 to the 
VDIC.
Or will it? It's worth experimenting. It would have helped if FSL listed 
which
IDMAC channels these FSU links correspond to, instead of making us guess
at it.

In any event, the docs are not clear enough to implement a real FSU
link to the VDIC read channels, if it's even possible. And trying to get
programming help from FSL can be difficult, and no coding examples
for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software
approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC
entity via a v4l2_subdev_ioctl() call that a buffer is available. The
VDIC then manually kicks off its read channels to bring that buffer
(and a previous buffer for F(n-1) field) into the VDIC.

There is a small amount of extra overhead going this route compared
to a FSU hardware link: there is the EOF irq latency (a few usec), and
the CPU overhead for the VDIC to manually start the read channels,
which is also a few usec at most (see prepare_vdi_in_buffers() in
imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
use plus irq latency) under normal system load.

Of course, in order to implement this software link, I had to implement
a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
at stream on, and the source requests a pointer to this ring in its own
stream on. Passing buffers from source to sink then follows a 
straightforward
FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
grabs queued buffers and makes them active, src signals completed
buffers to sink, sink dequeues buffers in response, and sink queues
buffers back when it is finished with them.


> [...]
>> +/*
>> + * The subdevs have to be powered on/off, and streaming
>> + * enabled/disabled, in a specific sequence.
>> + */
>> +static const u32 stream_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +};
>> +
>> +static const u32 stream_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +};
>> +
>> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
>> +
>> +static const u32 power_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +};
>> +
>> +static const u32 power_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +};
> This seems somewhat arbitrary. Why is a power sequence needed?

The CSI-2 receiver must be powered up before the sensor, that's the
only requirement IIRC. The others have no s_power requirement. So I
can probably change this to power up in the frontend -> backend order
(IC_PP to sensor). And vice-versa for power off.


> [...]
>> +/*
>> + * Turn current pipeline power on/off starting from start_entity.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
>> +				 struct media_entity_graph *graph,
>> +				 struct media_entity *start_entity, bool on)
>> +{
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int i, ret = 0;
>> +	u32 id;
>> +
>> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
>> +		id = on ? power_on_seq[i] : power_off_seq[i];
>> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
>> +		if (!entity)
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		ret = v4l2_subdev_call(sd, core, s_power, on);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			break;
>> +	}
>> +
>> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
doing some other stuff that bothered me, at least that's what I remember.
I will revisit this.

>> +/*
>> + * Inherit the v4l2 controls from all entities in a pipeline
>> + * to the given video device.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
>> +			       struct video_device *vfd,
>> +			       struct media_entity *start_entity)
>> +{
>> +	struct media_entity_graph graph;
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int ret;
>> +
>> +	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_entity_graph_walk_start(&graph, start_entity);
>> +
>> +	while ((entity = media_entity_graph_walk_next(&graph))) {
>> +		if (is_media_entity_v4l2_video_device(entity))
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
>> +			__func__, sd->name);
>> +
>> +		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
>> +					    sd->ctrl_handler,
>> +					    NULL);
>> +		if (ret)
>> +			break;
>> +	}
>> +
>> +	media_entity_graph_walk_cleanup(&graph);
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
>> +
>> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
>> new file mode 100644
>> index 0000000..357654d
>> --- /dev/null
>> +++ b/drivers/staging/media/imx/imx-media-dev.c
> This file is full of code that should live in the v4l2 core.

Agreed. Stuff like imx_media_inherit_controls() really should be widely
available.

>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *csi[4])
>> +{
>> +	int ret;
>> +
>> +	/* there must be at least one CSI in first IPU */
> Why?

Well yeah, imx-media doesn't necessarily need a CSI if things
like the VDIC or post-processor are being used by an output
overlay pipeline, for example. I'll fix this.

>> +
>> +/* parse inputs property from a sensor node */
>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *sensor,
>> +				   struct device_node *sensor_np)
>> +{
>> +	struct imx_media_sensor_input *sinput = &sensor->input;
>> +	int ret, i;
>> +
>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
>> +		const char *input_name;
>> +		u32 val;
>> +
>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
>> +		if (ret)
>> +			break;
>> +
>> +		sinput->value[i] = val;
>> +
>> +		ret = of_property_read_string_index(sensor_np, "input-names",
>> +						    i, &input_name);
>> +		/*
>> +		 * if input-names not provided, they will be set using
>> +		 * the subdev name once the sensor is known during
>> +		 * async bind
>> +		 */
>> +		if (!ret)
>> +			strncpy(sinput->name[i], input_name,
>> +				sizeof(sinput->name[i]));
>> +	}
>> +
>> +	sinput->num = i;
>> +
>> +	/* if no inputs provided just assume a single input */
>> +	if (sinput->num == 0)
>> +		sinput->num = 1;
>> +}
> This should be parsed by the sensor driver, not imx-media.

you're probably right. I'll submit a patch for adv7180.c.


>> +static void of_parse_sensor(struct imx_media_dev *imxmd,
>> +			    struct imx_media_subdev *sensor,
>> +			    struct device_node *sensor_np)
>> +{
>> +	struct device_node *endpoint;
>> +
>> +	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
>> +
>> +	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
>> +	if (endpoint) {
>> +		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
>> +		of_node_put(endpoint);
>> +	}
>> +}
>> +
>> +static int of_get_port_count(const struct device_node *np)
>> +{
>> +	struct device_node *child;
>> +	int num = 0;
>> +
>> +	/* if this node is itself a port, return 1 */
>> +	if (of_node_cmp(np->name, "port") == 0)
>> +		return 1;
>> +
>> +	for_each_child_of_node(np, child)
>> +		if (of_node_cmp(child->name, "port") == 0)
>> +			num++;
>> +
>> +	return num;
>> +}
> If this is extended to handle the ports subnode properly, it could be
> moved into drivers/of/base.c.

More that eventually needs exporting, agreed.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-14 22:46       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-14 22:46 UTC (permalink / raw)
  To: linux-arm-kernel

(sorry sending again w/o html)


On 01/13/2017 07:20 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> +For image capture, the IPU contains the following internal subunits:
>> +
>> +- Image DMA Controller (IDMAC)
>> +- Camera Serial Interface (CSI)
>> +- Image Converter (IC)
>> +- Sensor Multi-FIFO Controller (SMFC)
>> +- Image Rotator (IRT)
>> +- Video De-Interlace Controller (VDIC)
> Nitpick: Video De-Interlacing or Combining Block (VDIC)

done.

>> +
>> +The IDMAC is the DMA controller for transfer of image frames to and from
>> +memory. Various dedicated DMA channels exist for both video capture and
>> +display paths.
>> +
>> +The CSI is the frontend capture unit that interfaces directly with
>> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
>> +
>> +The IC handles color-space conversion, resizing, and rotation
>> +operations.
> And horizontal flipping.

done.


>> There are three independent "tasks" within the IC that can
>> +carry out conversions concurrently: pre-processing encoding,
>> +pre-processing preview, and post-processing.
> s/preview/viewfinder/ seems to be the commonly used name.

replaced everywhere in the doc.

> This paragraph could mention that a single hardware unit is used
> transparently time multiplexed by the three tasks at different
> granularity for the downsizing, main processing, and rotation sections.
> The downscale unit switches between tasks at 8-pixel burst granularity,
> the main processing unit at line granularity. The rotation units switch
> only at frame granularity.

I've added that info.

>> +The SMFC is composed of four independent channels that each can transfer
>> +captured frames from sensors directly to memory concurrently.
>> +
>> +The IRT carries out 90 and 270 degree image rotation operations.
> ... on 8x8 pixel blocks, supported by the IDMAC which handles block
> transfers, block reordering, and vertical flipping.

done.

>> +The VDIC handles the conversion of interlaced video to progressive, with
>> +support for different motion compensation modes (low, medium, and high
>> +motion). The deinterlaced output frames from the VDIC can be sent to the
>> +IC pre-process preview task for further conversions.
>> +
>> +In addition to the IPU internal subunits, there are also two units
>> +outside the IPU that are also involved in video capture on i.MX:
>> +
>> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
>> +  interface. This is a Synopsys DesignWare core.
>> +- A video multiplexer for selecting among multiple sensor inputs to
>> +  send to a CSI.
> Two of them, actually.

done.

>> +
>> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
>> +  problems with the ADV718x video decoders. See below for a description
>> +  of the FIM.
> Could this also be used to calculate more precise capture timestamps?

An input capture function could do that, triggered off a VSYNC or FIELD
signal such as on the ADV718x. The FIM is only used to calculate
frame intervals at this point, but its input capture method could be
used to also record more accurate timestamps.


>> +Capture Pipelines
>> +-----------------
>> +
>> +The following describe the various use-cases supported by the pipelines.
>> +
>> +The links shown do not include the frontend sensor, video mux, or mipi
>> +csi-2 receiver links. This depends on the type of sensor interface
>> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
>> +
>> +sensor -> ipu_csi_mux -> ipu_csi -> ...
>> +
>> +for parallel sensors, or:
>> +
>> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
>> +
>> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
>> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
>> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
>> +parenthesis.
>> +
>> +Unprocessed Video Capture:
>> +--------------------------
>> +
>> +Send frames directly from sensor to camera interface, with no
>> +conversions:
>> +
>> +-> ipu_smfc -> camif
> I'd call this capture interface, this is not just for cameras. Or maybe
> idmac if you want to mirror hardware names?

Camif is so named because it is the V4L2 user interface for video
capture. I suppose it could be named "capif", but that doesn't role
off the tongue quite as well.

>> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

>> +For example, its sink pad can take UYVY2X8, but its source pad can
>> +output YUYV2X8.
> I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> in the CSI chapter, for 8-bit per component data, the internal format
> between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> (or "bayer/generic data"). In either case, the internal format does not
> change along the way.

these are pixels in memory buffers, not the IPU internal formats.


>> +IC Direct Conversions:
>> +----------------------
>> +
>> +This pipeline uses the preprocess encode entity to route frames directly
>> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
>> +1024x1024 resolution, CSC, and image rotation:
>> +
>> +-> ipu_ic_prpenc -> camif
>> +
>> +This can be a useful capture pipeline for heavily loaded memory bus
>> +traffic environments, since it has minimal IDMAC channel usage.
> Note that if rotation is enabled, transfers between IC processing and
> rotation still have to go through memory once.

yep.

>> +Post-Processing Conversions:
>> +----------------------------
>> +
>> +This pipeline routes frames from the SMFC to the post-processing
>> +entity.
> No, frames written by the CSI -> SMFC -> IDMAC path are read back into
> the post-processing entity.

that's true. The post-processing entity kicks off its read channels
to transfer those frames into the post-processor. Anyway this wording
will change after doing away with the SMFC entity.


>> +   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
>> +   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"
> I think the smfc entities should be dropped.

yes working on that.

>> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
>> +   # Configure pads for OV5640 pipeline
>> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
>> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> [...]
>> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> I agree this looks very intuitive, but technically correct for the
> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>
> I think it would be better to use the correct format

I'm not sure I follow you here.

>   as that will allow
> to chose the regular vs. companded packings in the future for formats
> with more than 8 bits per component.
>
>> +   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>> +
>> +Streaming can then begin independently on device nodes /dev/video0
>> +and /dev/video1.
>> +
>> +SabreAuto with ADV7180 decoder
>> +------------------------------
>> +
>> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
>> +parallel bus input on the internal video mux to IPU1 CSI0.
>> +
>> +The following example configures a pipeline to capture from the ADV7180
>> +video decoder, assuming NTSC 720x480 input signals, with Motion
>> +Compensated de-interlacing (not shown: all pad field types should be set
>> +as indicated). $outputfmt can be any format supported by the
>> +ipu1_ic_prpvf entity at its output pad:
>> +
>> +.. code-block:: none
>> +
>> +   # Setup links
>> +   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
>> +   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
>> +   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
>> +   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
>> +   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
>> +   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
>> +   # Configure pads
>> +   # pad field types for below pads must be an interlaced type
>> +   # such as "ALTERNATE"
> I think alternate should only extend as far as the CSI, since the CSI
> can only capture NTSC/PAL fields in a fixed order.

Agreed, I'm doing the translation from alternate to seq_bt/seq_tb depending
on sensor std in imx-vdic.c, but it should move upstream. Will fix.

>> +   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
>> +   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
>  From here the interlaced field type should be sequential in the correct
> order depending on NTSC/PAL.

right.

>> +
>> +Frame Interval Monitor
>> +----------------------
>> +
>> +The adv718x decoders can occasionally send corrupt fields during
>> +NTSC/PAL signal re-sync (too little or too many video lines). When
>> +this happens, the IPU triggers a mechanism to re-establish vertical
>> +sync by adding 1 dummy line every frame, which causes a rolling effect
>> +from image to image, and can last a long time before a stable image is
>> +recovered. Or sometimes the mechanism doesn't work at all, causing a
>> +permanent split image (one frame contains lines from two consecutive
>> +captured images).
> Is it only SabreAuto on which the FIM mechanism can be used due to the
> pad routing?

No, FIM can be used on any target, the fim child node of ipu_csi just
needs to be enabled via status property.

> [...]
>> +/*
>> + * DMA buffer ring handling
>> + */
>> +struct imx_media_dma_buf_ring {
>> +	struct imx_media_dev *imxmd;
>> +
>> +	/* the ring */
>> +	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
>> +	/* the scratch buffer for underruns */
>> +	struct imx_media_dma_buf scratch;
>> +
>> +	/* buffer generator */
>> +	struct media_entity *src;
>> +	/* buffer receiver */
>> +	struct media_entity *sink;
>> +
>> +	spinlock_t lock;
>> +
>> +	int num_bufs;
>> +	unsigned long last_seq;
>> +};
> I don't think this belongs in the capture driver at all.
> Memory-to-memory transfers should be handled at the videobuf2 level.

see below.

> [...]
>> +static struct imx_media_dma_buf *
>> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
>> +{
>> +	struct imx_media_dma_buf *buf;
>> +
>> +	if (index >= ring->num_bufs)
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf = &ring->buf[index];
>> +	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
>> +	buf->seq = ring->last_seq++;
>> +
>> +	return buf;
>> +}
> Is this a whole software buffer queue implementation? I thought the
> whole point of putting the custom mem2mem framework into the capture
> driver was to use the hardware FSU channel linking?

  see below.

> What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries
in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

There does not appear to be support in the FSU for linking a write channel
to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
is support for the direct link from CSI (which I am using), but that's 
not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
But I think this links to channel 12, and not to channels 8,9,10 to the 
VDIC.
Or will it? It's worth experimenting. It would have helped if FSL listed 
which
IDMAC channels these FSU links correspond to, instead of making us guess
at it.

In any event, the docs are not clear enough to implement a real FSU
link to the VDIC read channels, if it's even possible. And trying to get
programming help from FSL can be difficult, and no coding examples
for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software
approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC
entity via a v4l2_subdev_ioctl() call that a buffer is available. The
VDIC then manually kicks off its read channels to bring that buffer
(and a previous buffer for F(n-1) field) into the VDIC.

There is a small amount of extra overhead going this route compared
to a FSU hardware link: there is the EOF irq latency (a few usec), and
the CPU overhead for the VDIC to manually start the read channels,
which is also a few usec at most (see prepare_vdi_in_buffers() in
imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
use plus irq latency) under normal system load.

Of course, in order to implement this software link, I had to implement
a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
at stream on, and the source requests a pointer to this ring in its own
stream on. Passing buffers from source to sink then follows a 
straightforward
FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
grabs queued buffers and makes them active, src signals completed
buffers to sink, sink dequeues buffers in response, and sink queues
buffers back when it is finished with them.


> [...]
>> +/*
>> + * The subdevs have to be powered on/off, and streaming
>> + * enabled/disabled, in a specific sequence.
>> + */
>> +static const u32 stream_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +};
>> +
>> +static const u32 stream_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +};
>> +
>> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
>> +
>> +static const u32 power_on_seq[] = {
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +};
>> +
>> +static const u32 power_off_seq[] = {
>> +	IMX_MEDIA_GRP_ID_IC_PP,
>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>> +	IMX_MEDIA_GRP_ID_SMFC,
>> +	IMX_MEDIA_GRP_ID_CSI,
>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>> +	IMX_MEDIA_GRP_ID_SENSOR,
>> +	IMX_MEDIA_GRP_ID_CSI2,
>> +};
> This seems somewhat arbitrary. Why is a power sequence needed?

The CSI-2 receiver must be powered up before the sensor, that's the
only requirement IIRC. The others have no s_power requirement. So I
can probably change this to power up in the frontend -> backend order
(IC_PP to sensor). And vice-versa for power off.


> [...]
>> +/*
>> + * Turn current pipeline power on/off starting from start_entity.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
>> +				 struct media_entity_graph *graph,
>> +				 struct media_entity *start_entity, bool on)
>> +{
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int i, ret = 0;
>> +	u32 id;
>> +
>> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
>> +		id = on ? power_on_seq[i] : power_off_seq[i];
>> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
>> +		if (!entity)
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		ret = v4l2_subdev_call(sd, core, s_power, on);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			break;
>> +	}
>> +
>> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
doing some other stuff that bothered me, at least that's what I remember.
I will revisit this.

>> +/*
>> + * Inherit the v4l2 controls from all entities in a pipeline
>> + * to the given video device.
>> + * Must be called with mdev->graph_mutex held.
>> + */
>> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
>> +			       struct video_device *vfd,
>> +			       struct media_entity *start_entity)
>> +{
>> +	struct media_entity_graph graph;
>> +	struct media_entity *entity;
>> +	struct v4l2_subdev *sd;
>> +	int ret;
>> +
>> +	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_entity_graph_walk_start(&graph, start_entity);
>> +
>> +	while ((entity = media_entity_graph_walk_next(&graph))) {
>> +		if (is_media_entity_v4l2_video_device(entity))
>> +			continue;
>> +
>> +		sd = media_entity_to_v4l2_subdev(entity);
>> +
>> +		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
>> +			__func__, sd->name);
>> +
>> +		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
>> +					    sd->ctrl_handler,
>> +					    NULL);
>> +		if (ret)
>> +			break;
>> +	}
>> +
>> +	media_entity_graph_walk_cleanup(&graph);
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
>> +
>> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
>> new file mode 100644
>> index 0000000..357654d
>> --- /dev/null
>> +++ b/drivers/staging/media/imx/imx-media-dev.c
> This file is full of code that should live in the v4l2 core.

Agreed. Stuff like imx_media_inherit_controls() really should be widely
available.

>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *csi[4])
>> +{
>> +	int ret;
>> +
>> +	/* there must be at least one CSI in first IPU */
> Why?

Well yeah, imx-media doesn't necessarily need a CSI if things
like the VDIC or post-processor are being used by an output
overlay pipeline, for example. I'll fix this.

>> +
>> +/* parse inputs property from a sensor node */
>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
>> +				   struct imx_media_subdev *sensor,
>> +				   struct device_node *sensor_np)
>> +{
>> +	struct imx_media_sensor_input *sinput = &sensor->input;
>> +	int ret, i;
>> +
>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
>> +		const char *input_name;
>> +		u32 val;
>> +
>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
>> +		if (ret)
>> +			break;
>> +
>> +		sinput->value[i] = val;
>> +
>> +		ret = of_property_read_string_index(sensor_np, "input-names",
>> +						    i, &input_name);
>> +		/*
>> +		 * if input-names not provided, they will be set using
>> +		 * the subdev name once the sensor is known during
>> +		 * async bind
>> +		 */
>> +		if (!ret)
>> +			strncpy(sinput->name[i], input_name,
>> +				sizeof(sinput->name[i]));
>> +	}
>> +
>> +	sinput->num = i;
>> +
>> +	/* if no inputs provided just assume a single input */
>> +	if (sinput->num == 0)
>> +		sinput->num = 1;
>> +}
> This should be parsed by the sensor driver, not imx-media.

you're probably right. I'll submit a patch for adv7180.c.


>> +static void of_parse_sensor(struct imx_media_dev *imxmd,
>> +			    struct imx_media_subdev *sensor,
>> +			    struct device_node *sensor_np)
>> +{
>> +	struct device_node *endpoint;
>> +
>> +	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
>> +
>> +	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
>> +	if (endpoint) {
>> +		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
>> +		of_node_put(endpoint);
>> +	}
>> +}
>> +
>> +static int of_get_port_count(const struct device_node *np)
>> +{
>> +	struct device_node *child;
>> +	int num = 0;
>> +
>> +	/* if this node is itself a port, return 1 */
>> +	if (of_node_cmp(np->name, "port") == 0)
>> +		return 1;
>> +
>> +	for_each_child_of_node(np, child)
>> +		if (of_node_cmp(child->name, "port") == 0)
>> +			num++;
>> +
>> +	return num;
>> +}
> If this is extended to handle the ports subnode properly, it could be
> moved into drivers/of/base.c.

More that eventually needs exporting, agreed.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-16 12:09         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 12:09 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, 2017-01-13 at 11:03 -0800, Steve Longerbeam wrote:
> 
> On 01/13/2017 03:55 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> Add bindings documentation for the i.MX media driver.
> >>
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >>   Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
> >>   1 file changed, 57 insertions(+)
> >>   create mode 100644 Documentation/devicetree/bindings/media/imx.txt
> >>
> >> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
> >> new file mode 100644
> >> index 0000000..254b64a
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/media/imx.txt
> >> @@ -0,0 +1,57 @@
> >> +Freescale i.MX Media Video Devices
> >> +
> >> +Video Media Controller node
> >> +---------------------------
> >> +
> >> +This is the parent media controller node for video capture support.
> >> +
> >> +Required properties:
> >> +- compatible : "fsl,imx-media";
> > Would you be opposed to calling this "capture-subsystem" instead of
> > "imx-media"? We already use "fsl,imx-display-subsystem" and
> > "fsl,imx-gpu-subsystem" for the display and GPU compound devices.
> 
> sure. Some pie-in-the-sky day when DRM and media are unified,
> there could be a single device that handles them all,

Indeed :)

>  but for now
> I'm fine with "fsl,capture-subsystem".

Actually, I meant fsl,imx-capture-subsystem. fsl,imx-media-subsystem
would be fine, too. Either way, I'll be happy if it looks similar to the
other two.

[...]
> > This is a clever method to get better frame timestamps. Too bad about
> > the routing requirements. Can this be used on Nitrogen6X?
> 
> Absolutely, this support just needs use of the input-capture channels in the
> imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
> input capture API, so at this point fsl,input-capture-channel has no effect,
> but it does work (tested on SabreAuto).

Nice.

[...]
> >> +Required properties:
> >> +- compatible	: "fsl,imx6-mipi-csi2";
> > I think this should get an additional "snps,dw-mipi-csi2" compatible,
> > since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.
> 
> right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
> Or should wait until the day this subdev is exported for general use, after
> pulling out the gasket specifics?

It can be added right away.

> >> +- reg           : physical base address and length of the register set;
> >> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
> >> +                  (the DPHY clock), video_27m, and eim_sel;
> > Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
> > shared gate bit that gates the HSI clocks as well as the MIPI
> > "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
> > csi-2 core, but we are missing shared gate clocks in the clock tree for
> > these.
> 
> Yes, so many clocks for the MIPI core. Why so many? I would think
> there would need to be at most three: a clock for the MIPI CSI-2 core
> and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
> M-PHY if there is one). I have no clue what all these other clocks are.
> But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

I would imagine the CSI-2 core has a high-speed clock input from the
D-PHY for serial input, an APB clock for register access (ips_clk), and
a pixel clock input for the parallel output (pixel_clk), at least.
The D-PHY will have a PLL reference input (pll_refclk?) and probably its
own register clock (cfg_clk?).

I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is
used for DSI only.

> > Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
> > video_27m seems fine.
> > But I don't get "dphy".
> 
> I presume it's the clock for the D-PHY.
>
> >   Which input clock would that correspond to?
> > "pll_refclk?"
> 
> the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.

I think that makes sense.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-16 12:09         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 12:09 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, 2017-01-13 at 11:03 -0800, Steve Longerbeam wrote:
> 
> On 01/13/2017 03:55 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> Add bindings documentation for the i.MX media driver.
> >>
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
> >> ---
> >>   Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
> >>   1 file changed, 57 insertions(+)
> >>   create mode 100644 Documentation/devicetree/bindings/media/imx.txt
> >>
> >> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
> >> new file mode 100644
> >> index 0000000..254b64a
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/media/imx.txt
> >> @@ -0,0 +1,57 @@
> >> +Freescale i.MX Media Video Devices
> >> +
> >> +Video Media Controller node
> >> +---------------------------
> >> +
> >> +This is the parent media controller node for video capture support.
> >> +
> >> +Required properties:
> >> +- compatible : "fsl,imx-media";
> > Would you be opposed to calling this "capture-subsystem" instead of
> > "imx-media"? We already use "fsl,imx-display-subsystem" and
> > "fsl,imx-gpu-subsystem" for the display and GPU compound devices.
> 
> sure. Some pie-in-the-sky day when DRM and media are unified,
> there could be a single device that handles them all,

Indeed :)

>  but for now
> I'm fine with "fsl,capture-subsystem".

Actually, I meant fsl,imx-capture-subsystem. fsl,imx-media-subsystem
would be fine, too. Either way, I'll be happy if it looks similar to the
other two.

[...]
> > This is a clever method to get better frame timestamps. Too bad about
> > the routing requirements. Can this be used on Nitrogen6X?
> 
> Absolutely, this support just needs use of the input-capture channels in the
> imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
> input capture API, so at this point fsl,input-capture-channel has no effect,
> but it does work (tested on SabreAuto).

Nice.

[...]
> >> +Required properties:
> >> +- compatible	: "fsl,imx6-mipi-csi2";
> > I think this should get an additional "snps,dw-mipi-csi2" compatible,
> > since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.
> 
> right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
> Or should wait until the day this subdev is exported for general use, after
> pulling out the gasket specifics?

It can be added right away.

> >> +- reg           : physical base address and length of the register set;
> >> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
> >> +                  (the DPHY clock), video_27m, and eim_sel;
> > Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
> > shared gate bit that gates the HSI clocks as well as the MIPI
> > "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
> > csi-2 core, but we are missing shared gate clocks in the clock tree for
> > these.
> 
> Yes, so many clocks for the MIPI core. Why so many? I would think
> there would need to be at most three: a clock for the MIPI CSI-2 core
> and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
> M-PHY if there is one). I have no clue what all these other clocks are.
> But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

I would imagine the CSI-2 core has a high-speed clock input from the
D-PHY for serial input, an APB clock for register access (ips_clk), and
a pixel clock input for the parallel output (pixel_clk), at least.
The D-PHY will have a PLL reference input (pll_refclk?) and probably its
own register clock (cfg_clk?).

I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is
used for DSI only.

> > Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
> > video_27m seems fine.
> > But I don't get "dphy".
> 
> I presume it's the clock for the D-PHY.
>
> >   Which input clock would that correspond to?
> > "pll_refclk?"
> 
> the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.

I think that makes sense.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-16 12:09         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 12:09 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 2017-01-13 at 11:03 -0800, Steve Longerbeam wrote:
> 
> On 01/13/2017 03:55 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> Add bindings documentation for the i.MX media driver.
> >>
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >>   Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
> >>   1 file changed, 57 insertions(+)
> >>   create mode 100644 Documentation/devicetree/bindings/media/imx.txt
> >>
> >> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
> >> new file mode 100644
> >> index 0000000..254b64a
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/media/imx.txt
> >> @@ -0,0 +1,57 @@
> >> +Freescale i.MX Media Video Devices
> >> +
> >> +Video Media Controller node
> >> +---------------------------
> >> +
> >> +This is the parent media controller node for video capture support.
> >> +
> >> +Required properties:
> >> +- compatible : "fsl,imx-media";
> > Would you be opposed to calling this "capture-subsystem" instead of
> > "imx-media"? We already use "fsl,imx-display-subsystem" and
> > "fsl,imx-gpu-subsystem" for the display and GPU compound devices.
> 
> sure. Some pie-in-the-sky day when DRM and media are unified,
> there could be a single device that handles them all,

Indeed :)

>  but for now
> I'm fine with "fsl,capture-subsystem".

Actually, I meant fsl,imx-capture-subsystem. fsl,imx-media-subsystem
would be fine, too. Either way, I'll be happy if it looks similar to the
other two.

[...]
> > This is a clever method to get better frame timestamps. Too bad about
> > the routing requirements. Can this be used on Nitrogen6X?
> 
> Absolutely, this support just needs use of the input-capture channels in the
> imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
> input capture API, so at this point fsl,input-capture-channel has no effect,
> but it does work (tested on SabreAuto).

Nice.

[...]
> >> +Required properties:
> >> +- compatible	: "fsl,imx6-mipi-csi2";
> > I think this should get an additional "snps,dw-mipi-csi2" compatible,
> > since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.
> 
> right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
> Or should wait until the day this subdev is exported for general use, after
> pulling out the gasket specifics?

It can be added right away.

> >> +- reg           : physical base address and length of the register set;
> >> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
> >> +                  (the DPHY clock), video_27m, and eim_sel;
> > Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
> > shared gate bit that gates the HSI clocks as well as the MIPI
> > "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
> > csi-2 core, but we are missing shared gate clocks in the clock tree for
> > these.
> 
> Yes, so many clocks for the MIPI core. Why so many? I would think
> there would need to be at most three: a clock for the MIPI CSI-2 core
> and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
> M-PHY if there is one). I have no clue what all these other clocks are.
> But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

I would imagine the CSI-2 core has a high-speed clock input from the
D-PHY for serial input, an APB clock for register access (ips_clk), and
a pixel clock input for the parallel output (pixel_clk), at least.
The D-PHY will have a PLL reference input (pll_refclk?) and probably its
own register clock (cfg_clk?).

I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is
used for DSI only.

> > Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
> > video_27m seems fine.
> > But I don't get "dphy".
> 
> I presume it's the clock for the D-PHY.
>
> >   Which input clock would that correspond to?
> > "pll_refclk?"
> 
> the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.

I think that makes sense.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-16 12:55         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 12:55 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
>
> On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> >> Both hang off the same i2c2 bus, so they require different (and non-
> >> default) i2c slave addresses.
> >>
> >> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
> >>
> >> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> >> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> >>
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
> >>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
> >>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
> >>   3 files changed, 129 insertions(+)
> >>
> >> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> index 0f06ca5..fec2524 100644
> >> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
[...]
> >> @@ -299,6 +326,52 @@
> >>   	pinctrl-names = "default";
> >>   	pinctrl-0 = <&pinctrl_i2c2>;
> >>   	status = "okay";
> >> +
> >> +	ov5640: camera@40 {
> >> +		compatible = "ovti,ov5640";
> >> +		pinctrl-names = "default";
> >> +		pinctrl-0 = <&pinctrl_ov5640>;
> >> +		clocks = <&mipi_xclk>;
> >> +		clock-names = "xclk";
> >> +		reg = <0x40>;
> >> +		xclk = <22000000>;
> > This is superfluous, you can use clk_get_rate on mipi_xclk.
> 
> This property is actually there to tell the driver what to set the
> rate to, with clk_set_rate(). So you are saying it would be better
> to set the rate in the device tree and the driver should only
> retrieve the rate?

Yes. Given that this is a reference clock input that is constant on a
given board and never changes during runtime, I think this is the
correct way. The clock will be fixed rate on most boards, I assume.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-16 12:55         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 12:55 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
>
> On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> >> Both hang off the same i2c2 bus, so they require different (and non-
> >> default) i2c slave addresses.
> >>
> >> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
> >>
> >> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> >> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> >>
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
> >> ---
> >>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
> >>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
> >>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
> >>   3 files changed, 129 insertions(+)
> >>
> >> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> index 0f06ca5..fec2524 100644
> >> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
[...]
> >> @@ -299,6 +326,52 @@
> >>   	pinctrl-names = "default";
> >>   	pinctrl-0 = <&pinctrl_i2c2>;
> >>   	status = "okay";
> >> +
> >> +	ov5640: camera@40 {
> >> +		compatible = "ovti,ov5640";
> >> +		pinctrl-names = "default";
> >> +		pinctrl-0 = <&pinctrl_ov5640>;
> >> +		clocks = <&mipi_xclk>;
> >> +		clock-names = "xclk";
> >> +		reg = <0x40>;
> >> +		xclk = <22000000>;
> > This is superfluous, you can use clk_get_rate on mipi_xclk.
> 
> This property is actually there to tell the driver what to set the
> rate to, with clk_set_rate(). So you are saying it would be better
> to set the rate in the device tree and the driver should only
> retrieve the rate?

Yes. Given that this is a reference clock input that is constant on a
given board and never changes during runtime, I think this is the
correct way. The clock will be fixed rate on most boards, I assume.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-16 12:55         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 12:55 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
>
> On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> >> Both hang off the same i2c2 bus, so they require different (and non-
> >> default) i2c slave addresses.
> >>
> >> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
> >>
> >> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> >> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> >>
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
> >>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
> >>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
> >>   3 files changed, 129 insertions(+)
> >>
> >> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> index 0f06ca5..fec2524 100644
> >> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
[...]
> >> @@ -299,6 +326,52 @@
> >>   	pinctrl-names = "default";
> >>   	pinctrl-0 = <&pinctrl_i2c2>;
> >>   	status = "okay";
> >> +
> >> +	ov5640: camera at 40 {
> >> +		compatible = "ovti,ov5640";
> >> +		pinctrl-names = "default";
> >> +		pinctrl-0 = <&pinctrl_ov5640>;
> >> +		clocks = <&mipi_xclk>;
> >> +		clock-names = "xclk";
> >> +		reg = <0x40>;
> >> +		xclk = <22000000>;
> > This is superfluous, you can use clk_get_rate on mipi_xclk.
> 
> This property is actually there to tell the driver what to set the
> rate to, with clk_set_rate(). So you are saying it would be better
> to set the rate in the device tree and the driver should only
> retrieve the rate?

Yes. Given that this is a reference clock input that is constant on a
given board and never changes during runtime, I think this is the
correct way. The clock will be fixed rate on most boards, I assume.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-14 22:46       ` Steve Longerbeam
  (?)
@ 2017-01-16 13:47         ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 13:47 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]
> >> +Unprocessed Video Capture:
> >> +--------------------------
> >> +
> >> +Send frames directly from sensor to camera interface, with no
> >> +conversions:
> >> +
> >> +-> ipu_smfc -> camif
> > I'd call this capture interface, this is not just for cameras. Or maybe
> > idmac if you want to mirror hardware names?
> 
> Camif is so named because it is the V4L2 user interface for video
> capture. I suppose it could be named "capif", but that doesn't role
> off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because
Samsung S3C has a camera interface that is actually called "CAMIF".

> >> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> > That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).
> 
> yes, the doc is re-worded to make that more clear.
> 
> >> +For example, its sink pad can take UYVY2X8, but its source pad can
> >> +output YUYV2X8.
> > I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> > in the CSI chapter, for 8-bit per component data, the internal format
> > between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> > (or "bayer/generic data"). In either case, the internal format does not
> > change along the way.
> 
> these are pixels in memory buffers, not the IPU internal formats.

As long as we are talking about the CSI -> SMFC -> IDMAC path, these
should be IPU internal formats. How else would one choose between 8-bit
companded RGB, and 16-bit expanded RGB for a 10-bit per component input
signal? This is the same issue as in the next comment.

> >> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> >> +   # Configure pads for OV5640 pipeline
> >> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> > [...]
> >> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> > I agree this looks very intuitive, but technically correct for the
> > csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> > (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
> >
> > I think it would be better to use the correct format
> 
> I'm not sure I follow you here.

The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0
up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values
into the SMFC, so the CSI output pad and the IDMAC input pad should have
a YUV8_1X32 format.

Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
IDMAC chapter states that all internal submodules only work on 8-bit per
component formats with four components: YUVA or RGBA.

> > [...]
> > Is this a whole software buffer queue implementation? I thought the
> > whole point of putting the custom mem2mem framework into the capture
> > driver was to use the hardware FSU channel linking?
> 
>   see below.
> 
> > What is the purpose of this if the sink should be triggered by the FSU?
> 
> Ok, here is where I need to make an admission.
> 
> The only FSU links I have attempted (and which currently have entries
> in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

Which are not described as media entity links because the rotation units
do not have separate media entities. So me arguing against handling
mem2mem chaining via media entity links doesn't concern these implicit
links.

> There does not appear to be support in the FSU for linking a write channel
> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
> is support for the direct link from CSI (which I am using), but that's 
> not an
> IDMAC channel link.
> 
> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).

As I read it, that is 0 and 2 only, no idea why. But since there are
only 2 CSIs, that shouldn't be a problem.

> But I think this links to channel 12, and not to channels 8,9,10 to the 
> VDIC.
> Or will it? It's worth experimenting. It would have helped if FSL listed 
> which
> IDMAC channels these FSU links correspond to, instead of making us guess
> at it.

I would have assumed that the FSU triggering only works on 1:1 channels
and the VDIC with its three input channels is different. But then
there's the alleged VDOA link to ch8/9/10 ro ch9, depending on
VDI_MOT_SEL.

This makes me more convinced that the CSI -> VDIC link should only
describe the direct path (real-time mode, single field).

> In any event, the docs are not clear enough to implement a real FSU
> link to the VDIC read channels, if it's even possible. And trying to get
> programming help from FSL can be difficult, and no coding examples
> for this link AFAIK.
> 
> So I ended resorted to linking to VDIC channels 8,9,10 with a software
> approach, instead of attempting a hardware FSU link.
> 
> The EOF interrupt handler for the SMFC channels informs the VDIC
> entity via a v4l2_subdev_ioctl() call that a buffer is available. The
> VDIC then manually kicks off its read channels to bring that buffer
> (and a previous buffer for F(n-1) field) into the VDIC.
> 
> There is a small amount of extra overhead going this route compared
> to a FSU hardware link: there is the EOF irq latency (a few usec), and
> the CPU overhead for the VDIC to manually start the read channels,
> which is also a few usec at most (see prepare_vdi_in_buffers() in
> imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
> use plus irq latency) under normal system load.

That the same low overhead could be reached by linking videobuf2 queues
of different video devices, that would be a lot more flexible.

> Of course, in order to implement this software link, I had to implement
> a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
> at stream on, and the source requests a pointer to this ring in its own
> stream on. Passing buffers from source to sink then follows a 
> straightforward
> FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
> grabs queued buffers and makes them active, src signals completed
> buffers to sink, sink dequeues buffers in response, and sink queues
> buffers back when it is finished with them.

Thank you for the explanation.

[...]
> >> +static const u32 power_off_seq[] = {
> >> +	IMX_MEDIA_GRP_ID_IC_PP,
> >> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> >> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> >> +	IMX_MEDIA_GRP_ID_SMFC,
> >> +	IMX_MEDIA_GRP_ID_CSI,
> >> +	IMX_MEDIA_GRP_ID_VIDMUX,
> >> +	IMX_MEDIA_GRP_ID_SENSOR,
> >> +	IMX_MEDIA_GRP_ID_CSI2,
> >> +};
> > This seems somewhat arbitrary. Why is a power sequence needed?
> 
> The CSI-2 receiver must be powered up before the sensor, that's the
> only requirement IIRC. The others have no s_power requirement. So I
> can probably change this to power up in the frontend -> backend order
> (IC_PP to sensor). And vice-versa for power off.

Yes, I think that should work (see below).

> > [...]
> >> +/*
> >> + * Turn current pipeline power on/off starting from start_entity.
> >> + * Must be called with mdev->graph_mutex held.
> >> + */
> >> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
> >> +				 struct media_entity_graph *graph,
> >> +				 struct media_entity *start_entity, bool on)
> >> +{
> >> +	struct media_entity *entity;
> >> +	struct v4l2_subdev *sd;
> >> +	int i, ret = 0;
> >> +	u32 id;
> >> +
> >> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
> >> +		id = on ? power_on_seq[i] : power_off_seq[i];
> >> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
> >> +		if (!entity)
> >> +			continue;
> >> +
> >> +		sd = media_entity_to_v4l2_subdev(entity);
> >> +
> >> +		ret = v4l2_subdev_call(sd, core, s_power, on);
> >> +		if (ret && ret != -ENOIOCTLCMD)
> >> +			break;
> >> +	}
> >> +
> >> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> > This should really be handled by v4l2_pipeline_pm_use.
> 
> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
> doing some other stuff that bothered me, at least that's what I remember.
> I will revisit this.

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
any problems. It would be better to reuse and, if necessary, fix the
existing infrastructure where available.

> >> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> >> +				   struct imx_media_subdev *csi[4])
> >> +{
> >> +	int ret;
> >> +
> >> +	/* there must be at least one CSI in first IPU */
> > Why?
> 
> Well yeah, imx-media doesn't necessarily need a CSI if things
> like the VDIC or post-processor are being used by an output
> overlay pipeline, for example. I'll fix this.

I haven't even thought that far, but there could be boards with only a
parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
saving reasons.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-16 13:47         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 13:47 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]
> >> +Unprocessed Video Capture:
> >> +--------------------------
> >> +
> >> +Send frames directly from sensor to camera interface, with no
> >> +conversions:
> >> +
> >> +-> ipu_smfc -> camif
> > I'd call this capture interface, this is not just for cameras. Or maybe
> > idmac if you want to mirror hardware names?
> 
> Camif is so named because it is the V4L2 user interface for video
> capture. I suppose it could be named "capif", but that doesn't role
> off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because
Samsung S3C has a camera interface that is actually called "CAMIF".

> >> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> > That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).
> 
> yes, the doc is re-worded to make that more clear.
> 
> >> +For example, its sink pad can take UYVY2X8, but its source pad can
> >> +output YUYV2X8.
> > I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> > in the CSI chapter, for 8-bit per component data, the internal format
> > between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> > (or "bayer/generic data"). In either case, the internal format does not
> > change along the way.
> 
> these are pixels in memory buffers, not the IPU internal formats.

As long as we are talking about the CSI -> SMFC -> IDMAC path, these
should be IPU internal formats. How else would one choose between 8-bit
companded RGB, and 16-bit expanded RGB for a 10-bit per component input
signal? This is the same issue as in the next comment.

> >> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> >> +   # Configure pads for OV5640 pipeline
> >> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> > [...]
> >> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> > I agree this looks very intuitive, but technically correct for the
> > csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> > (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
> >
> > I think it would be better to use the correct format
> 
> I'm not sure I follow you here.

The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0
up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values
into the SMFC, so the CSI output pad and the IDMAC input pad should have
a YUV8_1X32 format.

Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
IDMAC chapter states that all internal submodules only work on 8-bit per
component formats with four components: YUVA or RGBA.

> > [...]
> > Is this a whole software buffer queue implementation? I thought the
> > whole point of putting the custom mem2mem framework into the capture
> > driver was to use the hardware FSU channel linking?
> 
>   see below.
> 
> > What is the purpose of this if the sink should be triggered by the FSU?
> 
> Ok, here is where I need to make an admission.
> 
> The only FSU links I have attempted (and which currently have entries
> in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

Which are not described as media entity links because the rotation units
do not have separate media entities. So me arguing against handling
mem2mem chaining via media entity links doesn't concern these implicit
links.

> There does not appear to be support in the FSU for linking a write channel
> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
> is support for the direct link from CSI (which I am using), but that's 
> not an
> IDMAC channel link.
> 
> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).

As I read it, that is 0 and 2 only, no idea why. But since there are
only 2 CSIs, that shouldn't be a problem.

> But I think this links to channel 12, and not to channels 8,9,10 to the 
> VDIC.
> Or will it? It's worth experimenting. It would have helped if FSL listed 
> which
> IDMAC channels these FSU links correspond to, instead of making us guess
> at it.

I would have assumed that the FSU triggering only works on 1:1 channels
and the VDIC with its three input channels is different. But then
there's the alleged VDOA link to ch8/9/10 ro ch9, depending on
VDI_MOT_SEL.

This makes me more convinced that the CSI -> VDIC link should only
describe the direct path (real-time mode, single field).

> In any event, the docs are not clear enough to implement a real FSU
> link to the VDIC read channels, if it's even possible. And trying to get
> programming help from FSL can be difficult, and no coding examples
> for this link AFAIK.
> 
> So I ended resorted to linking to VDIC channels 8,9,10 with a software
> approach, instead of attempting a hardware FSU link.
> 
> The EOF interrupt handler for the SMFC channels informs the VDIC
> entity via a v4l2_subdev_ioctl() call that a buffer is available. The
> VDIC then manually kicks off its read channels to bring that buffer
> (and a previous buffer for F(n-1) field) into the VDIC.
> 
> There is a small amount of extra overhead going this route compared
> to a FSU hardware link: there is the EOF irq latency (a few usec), and
> the CPU overhead for the VDIC to manually start the read channels,
> which is also a few usec at most (see prepare_vdi_in_buffers() in
> imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
> use plus irq latency) under normal system load.

That the same low overhead could be reached by linking videobuf2 queues
of different video devices, that would be a lot more flexible.

> Of course, in order to implement this software link, I had to implement
> a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
> at stream on, and the source requests a pointer to this ring in its own
> stream on. Passing buffers from source to sink then follows a 
> straightforward
> FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
> grabs queued buffers and makes them active, src signals completed
> buffers to sink, sink dequeues buffers in response, and sink queues
> buffers back when it is finished with them.

Thank you for the explanation.

[...]
> >> +static const u32 power_off_seq[] = {
> >> +	IMX_MEDIA_GRP_ID_IC_PP,
> >> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> >> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> >> +	IMX_MEDIA_GRP_ID_SMFC,
> >> +	IMX_MEDIA_GRP_ID_CSI,
> >> +	IMX_MEDIA_GRP_ID_VIDMUX,
> >> +	IMX_MEDIA_GRP_ID_SENSOR,
> >> +	IMX_MEDIA_GRP_ID_CSI2,
> >> +};
> > This seems somewhat arbitrary. Why is a power sequence needed?
> 
> The CSI-2 receiver must be powered up before the sensor, that's the
> only requirement IIRC. The others have no s_power requirement. So I
> can probably change this to power up in the frontend -> backend order
> (IC_PP to sensor). And vice-versa for power off.

Yes, I think that should work (see below).

> > [...]
> >> +/*
> >> + * Turn current pipeline power on/off starting from start_entity.
> >> + * Must be called with mdev->graph_mutex held.
> >> + */
> >> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
> >> +				 struct media_entity_graph *graph,
> >> +				 struct media_entity *start_entity, bool on)
> >> +{
> >> +	struct media_entity *entity;
> >> +	struct v4l2_subdev *sd;
> >> +	int i, ret = 0;
> >> +	u32 id;
> >> +
> >> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
> >> +		id = on ? power_on_seq[i] : power_off_seq[i];
> >> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
> >> +		if (!entity)
> >> +			continue;
> >> +
> >> +		sd = media_entity_to_v4l2_subdev(entity);
> >> +
> >> +		ret = v4l2_subdev_call(sd, core, s_power, on);
> >> +		if (ret && ret != -ENOIOCTLCMD)
> >> +			break;
> >> +	}
> >> +
> >> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> > This should really be handled by v4l2_pipeline_pm_use.
> 
> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
> doing some other stuff that bothered me, at least that's what I remember.
> I will revisit this.

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
any problems. It would be better to reuse and, if necessary, fix the
existing infrastructure where available.

> >> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> >> +				   struct imx_media_subdev *csi[4])
> >> +{
> >> +	int ret;
> >> +
> >> +	/* there must be at least one CSI in first IPU */
> > Why?
> 
> Well yeah, imx-media doesn't necessarily need a CSI if things
> like the VDIC or post-processor are being used by an output
> overlay pipeline, for example. I'll fix this.

I haven't even thought that far, but there could be boards with only a
parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
saving reasons.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-16 13:47         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 13:47 UTC (permalink / raw)
  To: linux-arm-kernel

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]
> >> +Unprocessed Video Capture:
> >> +--------------------------
> >> +
> >> +Send frames directly from sensor to camera interface, with no
> >> +conversions:
> >> +
> >> +-> ipu_smfc -> camif
> > I'd call this capture interface, this is not just for cameras. Or maybe
> > idmac if you want to mirror hardware names?
> 
> Camif is so named because it is the V4L2 user interface for video
> capture. I suppose it could be named "capif", but that doesn't role
> off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because
Samsung S3C has a camera interface that is actually called "CAMIF".

> >> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> > That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).
> 
> yes, the doc is re-worded to make that more clear.
> 
> >> +For example, its sink pad can take UYVY2X8, but its source pad can
> >> +output YUYV2X8.
> > I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> > in the CSI chapter, for 8-bit per component data, the internal format
> > between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> > (or "bayer/generic data"). In either case, the internal format does not
> > change along the way.
> 
> these are pixels in memory buffers, not the IPU internal formats.

As long as we are talking about the CSI -> SMFC -> IDMAC path, these
should be IPU internal formats. How else would one choose between 8-bit
companded RGB, and 16-bit expanded RGB for a 10-bit per component input
signal? This is the same issue as in the next comment.

> >> +   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> >> +   # Configure pads for OV5640 pipeline
> >> +   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> >> +   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> > [...]
> >> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> > I agree this looks very intuitive, but technically correct for the
> > csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> > (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
> >
> > I think it would be better to use the correct format
> 
> I'm not sure I follow you here.

The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0
up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values
into the SMFC, so the CSI output pad and the IDMAC input pad should have
a YUV8_1X32 format.

Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
IDMAC chapter states that all internal submodules only work on 8-bit per
component formats with four components: YUVA or RGBA.

> > [...]
> > Is this a whole software buffer queue implementation? I thought the
> > whole point of putting the custom mem2mem framework into the capture
> > driver was to use the hardware FSU channel linking?
> 
>   see below.
> 
> > What is the purpose of this if the sink should be triggered by the FSU?
> 
> Ok, here is where I need to make an admission.
> 
> The only FSU links I have attempted (and which currently have entries
> in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

Which are not described as media entity links because the rotation units
do not have separate media entities. So me arguing against handling
mem2mem chaining via media entity links doesn't concern these implicit
links.

> There does not appear to be support in the FSU for linking a write channel
> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
> is support for the direct link from CSI (which I am using), but that's 
> not an
> IDMAC channel link.
> 
> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).

As I read it, that is 0 and 2 only, no idea why. But since there are
only 2 CSIs, that shouldn't be a problem.

> But I think this links to channel 12, and not to channels 8,9,10 to the 
> VDIC.
> Or will it? It's worth experimenting. It would have helped if FSL listed 
> which
> IDMAC channels these FSU links correspond to, instead of making us guess
> at it.

I would have assumed that the FSU triggering only works on 1:1 channels
and the VDIC with its three input channels is different. But then
there's the alleged VDOA link to ch8/9/10 ro ch9, depending on
VDI_MOT_SEL.

This makes me more convinced that the CSI -> VDIC link should only
describe the direct path (real-time mode, single field).

> In any event, the docs are not clear enough to implement a real FSU
> link to the VDIC read channels, if it's even possible. And trying to get
> programming help from FSL can be difficult, and no coding examples
> for this link AFAIK.
> 
> So I ended resorted to linking to VDIC channels 8,9,10 with a software
> approach, instead of attempting a hardware FSU link.
> 
> The EOF interrupt handler for the SMFC channels informs the VDIC
> entity via a v4l2_subdev_ioctl() call that a buffer is available. The
> VDIC then manually kicks off its read channels to bring that buffer
> (and a previous buffer for F(n-1) field) into the VDIC.
> 
> There is a small amount of extra overhead going this route compared
> to a FSU hardware link: there is the EOF irq latency (a few usec), and
> the CPU overhead for the VDIC to manually start the read channels,
> which is also a few usec at most (see prepare_vdi_in_buffers() in
> imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
> use plus irq latency) under normal system load.

That the same low overhead could be reached by linking videobuf2 queues
of different video devices, that would be a lot more flexible.

> Of course, in order to implement this software link, I had to implement
> a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
> at stream on, and the source requests a pointer to this ring in its own
> stream on. Passing buffers from source to sink then follows a 
> straightforward
> FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
> grabs queued buffers and makes them active, src signals completed
> buffers to sink, sink dequeues buffers in response, and sink queues
> buffers back when it is finished with them.

Thank you for the explanation.

[...]
> >> +static const u32 power_off_seq[] = {
> >> +	IMX_MEDIA_GRP_ID_IC_PP,
> >> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> >> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> >> +	IMX_MEDIA_GRP_ID_SMFC,
> >> +	IMX_MEDIA_GRP_ID_CSI,
> >> +	IMX_MEDIA_GRP_ID_VIDMUX,
> >> +	IMX_MEDIA_GRP_ID_SENSOR,
> >> +	IMX_MEDIA_GRP_ID_CSI2,
> >> +};
> > This seems somewhat arbitrary. Why is a power sequence needed?
> 
> The CSI-2 receiver must be powered up before the sensor, that's the
> only requirement IIRC. The others have no s_power requirement. So I
> can probably change this to power up in the frontend -> backend order
> (IC_PP to sensor). And vice-versa for power off.

Yes, I think that should work (see below).

> > [...]
> >> +/*
> >> + * Turn current pipeline power on/off starting from start_entity.
> >> + * Must be called with mdev->graph_mutex held.
> >> + */
> >> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
> >> +				 struct media_entity_graph *graph,
> >> +				 struct media_entity *start_entity, bool on)
> >> +{
> >> +	struct media_entity *entity;
> >> +	struct v4l2_subdev *sd;
> >> +	int i, ret = 0;
> >> +	u32 id;
> >> +
> >> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
> >> +		id = on ? power_on_seq[i] : power_off_seq[i];
> >> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
> >> +		if (!entity)
> >> +			continue;
> >> +
> >> +		sd = media_entity_to_v4l2_subdev(entity);
> >> +
> >> +		ret = v4l2_subdev_call(sd, core, s_power, on);
> >> +		if (ret && ret != -ENOIOCTLCMD)
> >> +			break;
> >> +	}
> >> +
> >> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> > This should really be handled by v4l2_pipeline_pm_use.
> 
> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
> doing some other stuff that bothered me, at least that's what I remember.
> I will revisit this.

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
any problems. It would be better to reuse and, if necessary, fix the
existing infrastructure where available.

> >> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> >> +				   struct imx_media_subdev *csi[4])
> >> +{
> >> +	int ret;
> >> +
> >> +	/* there must be at least one CSI in first IPU */
> > Why?
> 
> Well yeah, imx-media doesn't necessarily need a CSI if things
> like the VDIC or post-processor are being used by an output
> overlay pipeline, for example. I'll fix this.

I haven't even thought that far, but there could be boards with only a
parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
saving reasons.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-16 15:03     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 15:03 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.

s/Serial/Sensor/

> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig   |  13 +
>  drivers/staging/media/imx/Makefile  |   2 +
>  drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 659 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-csi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
> index bfde58d..ce2d2c8 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
>  	  Say yes here to enable support for video4linux media controller
>  	  driver for the i.MX5/6 SOC.
>  
> +if VIDEO_IMX_MEDIA
> +menu "i.MX5/6 Media Sub devices"
> +
> +config VIDEO_IMX_CAMERA

s/CAMERA/CSI/ ?

> +	tristate "i.MX5/6 Camera driver"

i.MX5/6 Camera Sensor Interface driver

> +	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
> +	select VIDEOBUF2_DMA_CONTIG
> +	default y
> +	---help---
> +	  A video4linux camera capture driver for i.MX5/6.
> +
> +endmenu
> +endif
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index ef9f11b..133672a 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
>  
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
> +
> diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
> new file mode 100644
> index 0000000..64ef862
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-csi.c
> @@ -0,0 +1,644 @@
> +/*
> + * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +#define CSI_NUM_PADS 2
> +
> +struct csi_priv {
> +	struct device *dev;
> +	struct ipu_soc *ipu;
> +	struct imx_media_dev *md;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad[CSI_NUM_PADS];
> +	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
> +	struct v4l2_mbus_config sensor_mbus_cfg;
> +	struct v4l2_rect crop;
> +	struct ipu_csi *csi;
> +	int csi_id;
> +	int input_pad;
> +	int output_pad;
> +	bool power_on;  /* power is on */
> +	bool stream_on; /* streaming is on */
> +
> +	/* the sink for the captured frames */
> +	struct v4l2_subdev *sink_sd;
> +	enum ipu_csi_dest dest;
> +	struct v4l2_subdev *src_sd;

src_sd is not used except that its presence marks an enabled input link.
-> could be changed to bool.

> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	struct imx_media_fim *fim;
> +
> +	/* the attached sensor at stream on */
> +	struct imx_media_subdev *sensor;
> +};
> +
> +static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
> +{
> +	return container_of(sdev, struct csi_priv, sd);
> +}
> +
> +/* Update the CSI whole sensor and active windows */
> +static int csi_setup(struct csi_priv *priv)
> +{
> +	struct v4l2_mbus_framefmt infmt;
> +
> +	ipu_csi_set_window(priv->csi, &priv->crop);
> +
> +	/*
> +	 * the ipu-csi doesn't understand ALTERNATE, but it only
> +	 * needs to know whether the stream is interlaced, so set
> +	 * to INTERLACED if infmt field is ALTERNATE.
> +	 */
> +	infmt = priv->format_mbus[priv->input_pad];
> +	if (infmt.field == V4L2_FIELD_ALTERNATE)
> +		infmt.field = V4L2_FIELD_INTERLACED;

That should be SEQ_TB/BT depending on video standard.

> +	ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
> +
> +	ipu_csi_set_dest(priv->csi, priv->dest);
> +
> +	ipu_csi_dump(priv->csi);
> +
> +	return 0;
> +}
> +
> +static int csi_start(struct csi_priv *priv)
> +{
> +	int ret;
> +
> +	if (!priv->sensor) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = csi_setup(priv);
> +	if (ret)
> +		return ret;
> +
> +	/* start the frame interval monitor */
> +	if (priv->fim) {
> +		ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	ret = ipu_csi_enable(priv->csi);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
> +		goto fim_off;
> +	}
> +
> +	return 0;
> +
> +fim_off:
> +	if (priv->fim)
> +		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
> +	return ret;
> +}
> +
> +static void csi_stop(struct csi_priv *priv)
> +{
> +	/* stop the frame interval monitor */
> +	if (priv->fim)
> +		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
> +
> +	ipu_csi_disable(priv->csi);
> +}
> +
> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	int ret = 0;
> +
> +	if (!priv->src_sd || !priv->sink_sd)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

These could be silenced a bit.

[...]
> +static int csi_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	int ret = 0;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (priv->fim && on != priv->power_on)
> +		ret = imx_media_fim_set_power(priv->fim, on);
> +
> +	if (!ret)
> +		priv->power_on = on;
> +	return ret;
> +}

Is this called multiple times? I'd expect a poweron during open and a
poweroff during close, so no need for priv->power_on.

> +static int csi_link_setup(struct media_entity *entity,
> +			  const struct media_pad *local,
> +			  const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SINK) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->src_sd)
> +				return -EBUSY;
> +			priv->src_sd = remote_sd;
> +		} else {
> +			priv->src_sd = NULL;
> +		}
> +
> +		return 0;
> +	}
> +
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->sink_sd)
> +			return -EBUSY;
> +		priv->sink_sd = remote_sd;
> +	} else {
> +		priv->sink_sd = NULL;
> +		return 0;
> +	}
> +
> +	/* set CSI destination */
> +	switch (remote_sd->grp_id) {
> +	case IMX_MEDIA_GRP_ID_SMFC0:
> +	case IMX_MEDIA_GRP_ID_SMFC1:
> +	case IMX_MEDIA_GRP_ID_SMFC2:
> +	case IMX_MEDIA_GRP_ID_SMFC3:

With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1
to the SMFC2 channel.

[...]
> +static int csi_set_fmt(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       struct v4l2_subdev_format *sdformat)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	struct v4l2_rect crop;
> +	int ret;
> +
> +	if (sdformat->pad >= CSI_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.code = infmt->code;
> +		sdformat->format.field = infmt->field;
> +		crop.left = priv->crop.left;
> +		crop.top = priv->crop.top;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop);
> +		if (ret)
> +			return ret;
> +		sdformat->format.width = crop.width;
> +		sdformat->format.height = crop.height;
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {

Should there be some limitations on the format here?

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-16 15:03     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 15:03 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.

s/Serial/Sensor/

> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig   |  13 +
>  drivers/staging/media/imx/Makefile  |   2 +
>  drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 659 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-csi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
> index bfde58d..ce2d2c8 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
>  	  Say yes here to enable support for video4linux media controller
>  	  driver for the i.MX5/6 SOC.
>  
> +if VIDEO_IMX_MEDIA
> +menu "i.MX5/6 Media Sub devices"
> +
> +config VIDEO_IMX_CAMERA

s/CAMERA/CSI/ ?

> +	tristate "i.MX5/6 Camera driver"

i.MX5/6 Camera Sensor Interface driver

> +	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
> +	select VIDEOBUF2_DMA_CONTIG
> +	default y
> +	---help---
> +	  A video4linux camera capture driver for i.MX5/6.
> +
> +endmenu
> +endif
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index ef9f11b..133672a 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
>  
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
> +
> diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
> new file mode 100644
> index 0000000..64ef862
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-csi.c
> @@ -0,0 +1,644 @@
> +/*
> + * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +#define CSI_NUM_PADS 2
> +
> +struct csi_priv {
> +	struct device *dev;
> +	struct ipu_soc *ipu;
> +	struct imx_media_dev *md;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad[CSI_NUM_PADS];
> +	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
> +	struct v4l2_mbus_config sensor_mbus_cfg;
> +	struct v4l2_rect crop;
> +	struct ipu_csi *csi;
> +	int csi_id;
> +	int input_pad;
> +	int output_pad;
> +	bool power_on;  /* power is on */
> +	bool stream_on; /* streaming is on */
> +
> +	/* the sink for the captured frames */
> +	struct v4l2_subdev *sink_sd;
> +	enum ipu_csi_dest dest;
> +	struct v4l2_subdev *src_sd;

src_sd is not used except that its presence marks an enabled input link.
-> could be changed to bool.

> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	struct imx_media_fim *fim;
> +
> +	/* the attached sensor at stream on */
> +	struct imx_media_subdev *sensor;
> +};
> +
> +static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
> +{
> +	return container_of(sdev, struct csi_priv, sd);
> +}
> +
> +/* Update the CSI whole sensor and active windows */
> +static int csi_setup(struct csi_priv *priv)
> +{
> +	struct v4l2_mbus_framefmt infmt;
> +
> +	ipu_csi_set_window(priv->csi, &priv->crop);
> +
> +	/*
> +	 * the ipu-csi doesn't understand ALTERNATE, but it only
> +	 * needs to know whether the stream is interlaced, so set
> +	 * to INTERLACED if infmt field is ALTERNATE.
> +	 */
> +	infmt = priv->format_mbus[priv->input_pad];
> +	if (infmt.field == V4L2_FIELD_ALTERNATE)
> +		infmt.field = V4L2_FIELD_INTERLACED;

That should be SEQ_TB/BT depending on video standard.

> +	ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
> +
> +	ipu_csi_set_dest(priv->csi, priv->dest);
> +
> +	ipu_csi_dump(priv->csi);
> +
> +	return 0;
> +}
> +
> +static int csi_start(struct csi_priv *priv)
> +{
> +	int ret;
> +
> +	if (!priv->sensor) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = csi_setup(priv);
> +	if (ret)
> +		return ret;
> +
> +	/* start the frame interval monitor */
> +	if (priv->fim) {
> +		ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	ret = ipu_csi_enable(priv->csi);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
> +		goto fim_off;
> +	}
> +
> +	return 0;
> +
> +fim_off:
> +	if (priv->fim)
> +		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
> +	return ret;
> +}
> +
> +static void csi_stop(struct csi_priv *priv)
> +{
> +	/* stop the frame interval monitor */
> +	if (priv->fim)
> +		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
> +
> +	ipu_csi_disable(priv->csi);
> +}
> +
> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	int ret = 0;
> +
> +	if (!priv->src_sd || !priv->sink_sd)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

These could be silenced a bit.

[...]
> +static int csi_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	int ret = 0;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (priv->fim && on != priv->power_on)
> +		ret = imx_media_fim_set_power(priv->fim, on);
> +
> +	if (!ret)
> +		priv->power_on = on;
> +	return ret;
> +}

Is this called multiple times? I'd expect a poweron during open and a
poweroff during close, so no need for priv->power_on.

> +static int csi_link_setup(struct media_entity *entity,
> +			  const struct media_pad *local,
> +			  const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SINK) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->src_sd)
> +				return -EBUSY;
> +			priv->src_sd = remote_sd;
> +		} else {
> +			priv->src_sd = NULL;
> +		}
> +
> +		return 0;
> +	}
> +
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->sink_sd)
> +			return -EBUSY;
> +		priv->sink_sd = remote_sd;
> +	} else {
> +		priv->sink_sd = NULL;
> +		return 0;
> +	}
> +
> +	/* set CSI destination */
> +	switch (remote_sd->grp_id) {
> +	case IMX_MEDIA_GRP_ID_SMFC0:
> +	case IMX_MEDIA_GRP_ID_SMFC1:
> +	case IMX_MEDIA_GRP_ID_SMFC2:
> +	case IMX_MEDIA_GRP_ID_SMFC3:

With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1
to the SMFC2 channel.

[...]
> +static int csi_set_fmt(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       struct v4l2_subdev_format *sdformat)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	struct v4l2_rect crop;
> +	int ret;
> +
> +	if (sdformat->pad >= CSI_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.code = infmt->code;
> +		sdformat->format.field = infmt->field;
> +		crop.left = priv->crop.left;
> +		crop.top = priv->crop.top;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop);
> +		if (ret)
> +			return ret;
> +		sdformat->format.width = crop.width;
> +		sdformat->format.height = crop.height;
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {

Should there be some limitations on the format here?

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-16 15:03     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-16 15:03 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.

s/Serial/Sensor/

> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig   |  13 +
>  drivers/staging/media/imx/Makefile  |   2 +
>  drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 659 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-csi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
> index bfde58d..ce2d2c8 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
>  	  Say yes here to enable support for video4linux media controller
>  	  driver for the i.MX5/6 SOC.
>  
> +if VIDEO_IMX_MEDIA
> +menu "i.MX5/6 Media Sub devices"
> +
> +config VIDEO_IMX_CAMERA

s/CAMERA/CSI/ ?

> +	tristate "i.MX5/6 Camera driver"

i.MX5/6 Camera Sensor Interface driver

> +	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
> +	select VIDEOBUF2_DMA_CONTIG
> +	default y
> +	---help---
> +	  A video4linux camera capture driver for i.MX5/6.
> +
> +endmenu
> +endif
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index ef9f11b..133672a 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
>  
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
> +
> diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
> new file mode 100644
> index 0000000..64ef862
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-csi.c
> @@ -0,0 +1,644 @@
> +/*
> + * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +#define CSI_NUM_PADS 2
> +
> +struct csi_priv {
> +	struct device *dev;
> +	struct ipu_soc *ipu;
> +	struct imx_media_dev *md;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad[CSI_NUM_PADS];
> +	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
> +	struct v4l2_mbus_config sensor_mbus_cfg;
> +	struct v4l2_rect crop;
> +	struct ipu_csi *csi;
> +	int csi_id;
> +	int input_pad;
> +	int output_pad;
> +	bool power_on;  /* power is on */
> +	bool stream_on; /* streaming is on */
> +
> +	/* the sink for the captured frames */
> +	struct v4l2_subdev *sink_sd;
> +	enum ipu_csi_dest dest;
> +	struct v4l2_subdev *src_sd;

src_sd is not used except that its presence marks an enabled input link.
-> could be changed to bool.

> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	struct imx_media_fim *fim;
> +
> +	/* the attached sensor at stream on */
> +	struct imx_media_subdev *sensor;
> +};
> +
> +static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
> +{
> +	return container_of(sdev, struct csi_priv, sd);
> +}
> +
> +/* Update the CSI whole sensor and active windows */
> +static int csi_setup(struct csi_priv *priv)
> +{
> +	struct v4l2_mbus_framefmt infmt;
> +
> +	ipu_csi_set_window(priv->csi, &priv->crop);
> +
> +	/*
> +	 * the ipu-csi doesn't understand ALTERNATE, but it only
> +	 * needs to know whether the stream is interlaced, so set
> +	 * to INTERLACED if infmt field is ALTERNATE.
> +	 */
> +	infmt = priv->format_mbus[priv->input_pad];
> +	if (infmt.field == V4L2_FIELD_ALTERNATE)
> +		infmt.field = V4L2_FIELD_INTERLACED;

That should be SEQ_TB/BT depending on video standard.

> +	ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
> +
> +	ipu_csi_set_dest(priv->csi, priv->dest);
> +
> +	ipu_csi_dump(priv->csi);
> +
> +	return 0;
> +}
> +
> +static int csi_start(struct csi_priv *priv)
> +{
> +	int ret;
> +
> +	if (!priv->sensor) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = csi_setup(priv);
> +	if (ret)
> +		return ret;
> +
> +	/* start the frame interval monitor */
> +	if (priv->fim) {
> +		ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	ret = ipu_csi_enable(priv->csi);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
> +		goto fim_off;
> +	}
> +
> +	return 0;
> +
> +fim_off:
> +	if (priv->fim)
> +		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
> +	return ret;
> +}
> +
> +static void csi_stop(struct csi_priv *priv)
> +{
> +	/* stop the frame interval monitor */
> +	if (priv->fim)
> +		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
> +
> +	ipu_csi_disable(priv->csi);
> +}
> +
> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	int ret = 0;
> +
> +	if (!priv->src_sd || !priv->sink_sd)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

These could be silenced a bit.

[...]
> +static int csi_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	int ret = 0;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (priv->fim && on != priv->power_on)
> +		ret = imx_media_fim_set_power(priv->fim, on);
> +
> +	if (!ret)
> +		priv->power_on = on;
> +	return ret;
> +}

Is this called multiple times? I'd expect a poweron during open and a
poweroff during close, so no need for priv->power_on.

> +static int csi_link_setup(struct media_entity *entity,
> +			  const struct media_pad *local,
> +			  const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SINK) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->src_sd)
> +				return -EBUSY;
> +			priv->src_sd = remote_sd;
> +		} else {
> +			priv->src_sd = NULL;
> +		}
> +
> +		return 0;
> +	}
> +
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->sink_sd)
> +			return -EBUSY;
> +		priv->sink_sd = remote_sd;
> +	} else {
> +		priv->sink_sd = NULL;
> +		return 0;
> +	}
> +
> +	/* set CSI destination */
> +	switch (remote_sd->grp_id) {
> +	case IMX_MEDIA_GRP_ID_SMFC0:
> +	case IMX_MEDIA_GRP_ID_SMFC1:
> +	case IMX_MEDIA_GRP_ID_SMFC2:
> +	case IMX_MEDIA_GRP_ID_SMFC3:

With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1
to the SMFC2 channel.

[...]
> +static int csi_set_fmt(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       struct v4l2_subdev_format *sdformat)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	struct v4l2_rect crop;
> +	int ret;
> +
> +	if (sdformat->pad >= CSI_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.code = infmt->code;
> +		sdformat->format.field = infmt->field;
> +		crop.left = priv->crop.left;
> +		crop.top = priv->crop.top;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop);
> +		if (ret)
> +			return ret;
> +		sdformat->format.width = crop.width;
> +		sdformat->format.height = crop.height;
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {

Should there be some limitations on the format here?

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-16 17:13           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 17:13 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/16/2017 04:09 AM, Philipp Zabel wrote:
> On Fri, 2017-01-13 at 11:03 -0800, Steve Longerbeam wrote:
>> On 01/13/2017 03:55 AM, Philipp Zabel wrote:
>>> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>>>> Add bindings documentation for the i.MX media driver.
>>>>
>>>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>>>> ---
>>>>    Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>>>>    1 file changed, 57 insertions(+)
>>>>    create mode 100644 Documentation/devicetree/bindings/media/imx.txt
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
>>>> new file mode 100644
>>>> index 0000000..254b64a
>>>> --- /dev/null
>>>> +++ b/Documentation/devicetree/bindings/media/imx.txt
>>>> @@ -0,0 +1,57 @@
>>>> +Freescale i.MX Media Video Devices
>>>> +
>>>> +Video Media Controller node
>>>> +---------------------------
>>>> +
>>>> +This is the parent media controller node for video capture support.
>>>> +
>>>> +Required properties:
>>>> +- compatible : "fsl,imx-media";
>>> Would you be opposed to calling this "capture-subsystem" instead of
>>> "imx-media"? We already use "fsl,imx-display-subsystem" and
>>> "fsl,imx-gpu-subsystem" for the display and GPU compound devices.
>> sure. Some pie-in-the-sky day when DRM and media are unified,
>> there could be a single device that handles them all,
> Indeed :)
>
>>   but for now
>> I'm fine with "fsl,capture-subsystem".
> Actually, I meant fsl,imx-capture-subsystem.

right, I caught my error and that is the name chosen.

>   fsl,imx-media-subsystem
> would be fine, too. Either way, I'll be happy if it looks similar to the
> other two.
>
> [...]
>>> This is a clever method to get better frame timestamps. Too bad about
>>> the routing requirements. Can this be used on Nitrogen6X?
>> Absolutely, this support just needs use of the input-capture channels in the
>> imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
>> input capture API, so at this point fsl,input-capture-channel has no effect,
>> but it does work (tested on SabreAuto).
> Nice.
>
> [...]
>>>> +Required properties:
>>>> +- compatible	: "fsl,imx6-mipi-csi2";
>>> I think this should get an additional "snps,dw-mipi-csi2" compatible,
>>> since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.
>> right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
>> Or should wait until the day this subdev is exported for general use, after
>> pulling out the gasket specifics?
> It can be added right away.

ok, I will add.

Steve

>
>>>> +- reg           : physical base address and length of the register set;
>>>> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
>>>> +                  (the DPHY clock), video_27m, and eim_sel;
>>> Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
>>> shared gate bit that gates the HSI clocks as well as the MIPI
>>> "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
>>> csi-2 core, but we are missing shared gate clocks in the clock tree for
>>> these.
>> Yes, so many clocks for the MIPI core. Why so many? I would think
>> there would need to be at most three: a clock for the MIPI CSI-2 core
>> and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
>> M-PHY if there is one). I have no clue what all these other clocks are.
>> But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.
> I would imagine the CSI-2 core has a high-speed clock input from the
> D-PHY for serial input, an APB clock for register access (ips_clk), and
> a pixel clock input for the parallel output (pixel_clk), at least.
> The D-PHY will have a PLL reference input (pll_refclk?) and probably its
> own register clock (cfg_clk?).
>
> I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is
> used for DSI only.
>
>>> Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
>>> video_27m seems fine.
>>> But I don't get "dphy".
>> I presume it's the clock for the D-PHY.
>>
>>>    Which input clock would that correspond to?
>>> "pll_refclk?"
>> the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.
> I think that makes sense.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-16 17:13           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 17:13 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/16/2017 04:09 AM, Philipp Zabel wrote:
> On Fri, 2017-01-13 at 11:03 -0800, Steve Longerbeam wrote:
>> On 01/13/2017 03:55 AM, Philipp Zabel wrote:
>>> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>>>> Add bindings documentation for the i.MX media driver.
>>>>
>>>> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
>>>> ---
>>>>    Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>>>>    1 file changed, 57 insertions(+)
>>>>    create mode 100644 Documentation/devicetree/bindings/media/imx.txt
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
>>>> new file mode 100644
>>>> index 0000000..254b64a
>>>> --- /dev/null
>>>> +++ b/Documentation/devicetree/bindings/media/imx.txt
>>>> @@ -0,0 +1,57 @@
>>>> +Freescale i.MX Media Video Devices
>>>> +
>>>> +Video Media Controller node
>>>> +---------------------------
>>>> +
>>>> +This is the parent media controller node for video capture support.
>>>> +
>>>> +Required properties:
>>>> +- compatible : "fsl,imx-media";
>>> Would you be opposed to calling this "capture-subsystem" instead of
>>> "imx-media"? We already use "fsl,imx-display-subsystem" and
>>> "fsl,imx-gpu-subsystem" for the display and GPU compound devices.
>> sure. Some pie-in-the-sky day when DRM and media are unified,
>> there could be a single device that handles them all,
> Indeed :)
>
>>   but for now
>> I'm fine with "fsl,capture-subsystem".
> Actually, I meant fsl,imx-capture-subsystem.

right, I caught my error and that is the name chosen.

>   fsl,imx-media-subsystem
> would be fine, too. Either way, I'll be happy if it looks similar to the
> other two.
>
> [...]
>>> This is a clever method to get better frame timestamps. Too bad about
>>> the routing requirements. Can this be used on Nitrogen6X?
>> Absolutely, this support just needs use of the input-capture channels in the
>> imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
>> input capture API, so at this point fsl,input-capture-channel has no effect,
>> but it does work (tested on SabreAuto).
> Nice.
>
> [...]
>>>> +Required properties:
>>>> +- compatible	: "fsl,imx6-mipi-csi2";
>>> I think this should get an additional "snps,dw-mipi-csi2" compatible,
>>> since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.
>> right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
>> Or should wait until the day this subdev is exported for general use, after
>> pulling out the gasket specifics?
> It can be added right away.

ok, I will add.

Steve

>
>>>> +- reg           : physical base address and length of the register set;
>>>> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
>>>> +                  (the DPHY clock), video_27m, and eim_sel;
>>> Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
>>> shared gate bit that gates the HSI clocks as well as the MIPI
>>> "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
>>> csi-2 core, but we are missing shared gate clocks in the clock tree for
>>> these.
>> Yes, so many clocks for the MIPI core. Why so many? I would think
>> there would need to be at most three: a clock for the MIPI CSI-2 core
>> and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
>> M-PHY if there is one). I have no clue what all these other clocks are.
>> But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.
> I would imagine the CSI-2 core has a high-speed clock input from the
> D-PHY for serial input, an APB clock for register access (ips_clk), and
> a pixel clock input for the parallel output (pixel_clk), at least.
> The D-PHY will have a PLL reference input (pll_refclk?) and probably its
> own register clock (cfg_clk?).
>
> I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is
> used for DSI only.
>
>>> Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
>>> video_27m seems fine.
>>> But I don't get "dphy".
>> I presume it's the clock for the D-PHY.
>>
>>>    Which input clock would that correspond to?
>>> "pll_refclk?"
>> the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.
> I think that makes sense.
>
> regards
> Philipp
>

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver
@ 2017-01-16 17:13           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 17:13 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/16/2017 04:09 AM, Philipp Zabel wrote:
> On Fri, 2017-01-13 at 11:03 -0800, Steve Longerbeam wrote:
>> On 01/13/2017 03:55 AM, Philipp Zabel wrote:
>>> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>>>> Add bindings documentation for the i.MX media driver.
>>>>
>>>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>>>> ---
>>>>    Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
>>>>    1 file changed, 57 insertions(+)
>>>>    create mode 100644 Documentation/devicetree/bindings/media/imx.txt
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
>>>> new file mode 100644
>>>> index 0000000..254b64a
>>>> --- /dev/null
>>>> +++ b/Documentation/devicetree/bindings/media/imx.txt
>>>> @@ -0,0 +1,57 @@
>>>> +Freescale i.MX Media Video Devices
>>>> +
>>>> +Video Media Controller node
>>>> +---------------------------
>>>> +
>>>> +This is the parent media controller node for video capture support.
>>>> +
>>>> +Required properties:
>>>> +- compatible : "fsl,imx-media";
>>> Would you be opposed to calling this "capture-subsystem" instead of
>>> "imx-media"? We already use "fsl,imx-display-subsystem" and
>>> "fsl,imx-gpu-subsystem" for the display and GPU compound devices.
>> sure. Some pie-in-the-sky day when DRM and media are unified,
>> there could be a single device that handles them all,
> Indeed :)
>
>>   but for now
>> I'm fine with "fsl,capture-subsystem".
> Actually, I meant fsl,imx-capture-subsystem.

right, I caught my error and that is the name chosen.

>   fsl,imx-media-subsystem
> would be fine, too. Either way, I'll be happy if it looks similar to the
> other two.
>
> [...]
>>> This is a clever method to get better frame timestamps. Too bad about
>>> the routing requirements. Can this be used on Nitrogen6X?
>> Absolutely, this support just needs use of the input-capture channels in the
>> imx GPT. I still need to submit the patch to the imx-gpt driver that adds an
>> input capture API, so at this point fsl,input-capture-channel has no effect,
>> but it does work (tested on SabreAuto).
> Nice.
>
> [...]
>>>> +Required properties:
>>>> +- compatible	: "fsl,imx6-mipi-csi2";
>>> I think this should get an additional "snps,dw-mipi-csi2" compatible,
>>> since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.
>> right, minus the gasket it's a Synopsys core. I'll add that compatible flag.
>> Or should wait until the day this subdev is exported for general use, after
>> pulling out the gasket specifics?
> It can be added right away.

ok, I will add.

Steve

>
>>>> +- reg           : physical base address and length of the register set;
>>>> +- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
>>>> +                  (the DPHY clock), video_27m, and eim_sel;
>>> Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the
>>> shared gate bit that gates the HSI clocks as well as the MIPI
>>> "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi
>>> csi-2 core, but we are missing shared gate clocks in the clock tree for
>>> these.
>> Yes, so many clocks for the MIPI core. Why so many? I would think
>> there would need to be at most three: a clock for the MIPI CSI-2 core
>> and HSI core, and a clock for the D-PHY (oh and maybe a clock for an
>> M-PHY if there is one). I have no clue what all these other clocks are.
>> But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.
> I would imagine the CSI-2 core has a high-speed clock input from the
> D-PHY for serial input, an APB clock for register access (ips_clk), and
> a pixel clock input for the parallel output (pixel_clk), at least.
> The D-PHY will have a PLL reference input (pll_refclk?) and probably its
> own register clock (cfg_clk?).
>
> I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is
> used for DSI only.
>
>>> Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" ->
>>> video_27m seems fine.
>>> But I don't get "dphy".
>> I presume it's the clock for the D-PHY.
>>
>>>    Which input clock would that correspond to?
>>> "pll_refclk?"
>> the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.
> I think that makes sense.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-16 17:15           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 17:15 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/16/2017 04:55 AM, Philipp Zabel wrote:
> On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
>
>>>> @@ -299,6 +326,52 @@
>>>>    	pinctrl-names = "default";
>>>>    	pinctrl-0 = <&pinctrl_i2c2>;
>>>>    	status = "okay";
>>>> +
>>>> +	ov5640: camera@40 {
>>>> +		compatible = "ovti,ov5640";
>>>> +		pinctrl-names = "default";
>>>> +		pinctrl-0 = <&pinctrl_ov5640>;
>>>> +		clocks = <&mipi_xclk>;
>>>> +		clock-names = "xclk";
>>>> +		reg = <0x40>;
>>>> +		xclk = <22000000>;
>>> This is superfluous, you can use clk_get_rate on mipi_xclk.
>> This property is actually there to tell the driver what to set the
>> rate to, with clk_set_rate(). So you are saying it would be better
>> to set the rate in the device tree and the driver should only
>> retrieve the rate?
> Yes. Given that this is a reference clock input that is constant on a
> given board and never changes during runtime, I think this is the
> correct way. The clock will be fixed rate on most boards, I assume.

Ok, that makes sense, I'll make that change.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-16 17:15           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 17:15 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/16/2017 04:55 AM, Philipp Zabel wrote:
> On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
>
>>>> @@ -299,6 +326,52 @@
>>>>    	pinctrl-names = "default";
>>>>    	pinctrl-0 = <&pinctrl_i2c2>;
>>>>    	status = "okay";
>>>> +
>>>> +	ov5640: camera@40 {
>>>> +		compatible = "ovti,ov5640";
>>>> +		pinctrl-names = "default";
>>>> +		pinctrl-0 = <&pinctrl_ov5640>;
>>>> +		clocks = <&mipi_xclk>;
>>>> +		clock-names = "xclk";
>>>> +		reg = <0x40>;
>>>> +		xclk = <22000000>;
>>> This is superfluous, you can use clk_get_rate on mipi_xclk.
>> This property is actually there to tell the driver what to set the
>> rate to, with clk_set_rate(). So you are saying it would be better
>> to set the rate in the device tree and the driver should only
>> retrieve the rate?
> Yes. Given that this is a reference clock input that is constant on a
> given board and never changes during runtime, I think this is the
> correct way. The clock will be fixed rate on most boards, I assume.

Ok, that makes sense, I'll make that change.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-16 17:15           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 17:15 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/16/2017 04:55 AM, Philipp Zabel wrote:
> On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
>
>>>> @@ -299,6 +326,52 @@
>>>>    	pinctrl-names = "default";
>>>>    	pinctrl-0 = <&pinctrl_i2c2>;
>>>>    	status = "okay";
>>>> +
>>>> +	ov5640: camera at 40 {
>>>> +		compatible = "ovti,ov5640";
>>>> +		pinctrl-names = "default";
>>>> +		pinctrl-0 = <&pinctrl_ov5640>;
>>>> +		clocks = <&mipi_xclk>;
>>>> +		clock-names = "xclk";
>>>> +		reg = <0x40>;
>>>> +		xclk = <22000000>;
>>> This is superfluous, you can use clk_get_rate on mipi_xclk.
>> This property is actually there to tell the driver what to set the
>> rate to, with clk_set_rate(). So you are saying it would be better
>> to set the rate in the device tree and the driver should only
>> retrieve the rate?
> Yes. Given that this is a reference clock input that is constant on a
> given board and never changes during runtime, I think this is the
> correct way. The clock will be fixed rate on most boards, I assume.

Ok, that makes sense, I'll make that change.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
  2017-01-16 15:03     ` Philipp Zabel
  (?)
@ 2017-01-16 21:15       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 21:15 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/16/2017 07:03 AM, Philipp Zabel wrote:
> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
>> This is a media entity subdevice for the i.MX Camera
>> Serial Interface module.
> s/Serial/Sensor/

done.

>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   drivers/staging/media/imx/Kconfig   |  13 +
>>   drivers/staging/media/imx/Makefile  |   2 +
>>   drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
>>   3 files changed, 659 insertions(+)
>>   create mode 100644 drivers/staging/media/imx/imx-csi.c
>>
>> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
>> index bfde58d..ce2d2c8 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
>>   	  Say yes here to enable support for video4linux media controller
>>   	  driver for the i.MX5/6 SOC.
>>   
>> +if VIDEO_IMX_MEDIA
>> +menu "i.MX5/6 Media Sub devices"
>> +
>> +config VIDEO_IMX_CAMERA
> s/CAMERA/CSI/ ?

done.

>> +	tristate "i.MX5/6 Camera driver"
> i.MX5/6 Camera Sensor Interface driver

done.

>
>> +
>> +struct csi_priv {
>> +	struct device *dev;
>> +	struct ipu_soc *ipu;
>> +	struct imx_media_dev *md;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad[CSI_NUM_PADS];
>> +	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
>> +	struct v4l2_mbus_config sensor_mbus_cfg;
>> +	struct v4l2_rect crop;
>> +	struct ipu_csi *csi;
>> +	int csi_id;
>> +	int input_pad;
>> +	int output_pad;
>> +	bool power_on;  /* power is on */
>> +	bool stream_on; /* streaming is on */
>> +
>> +	/* the sink for the captured frames */
>> +	struct v4l2_subdev *sink_sd;
>> +	enum ipu_csi_dest dest;
>> +	struct v4l2_subdev *src_sd;
> src_sd is not used except that its presence marks an enabled input link.
> -> could be changed to bool.

For now I prefer to keep it a pointer to the src/sink subdevs.
At some point the CSI may have some reason to know the
identity of the source.

>
>> +	struct v4l2_ctrl_handler ctrl_hdlr;
>> +	struct imx_media_fim *fim;
>> +
>> +	/* the attached sensor at stream on */
>> +	struct imx_media_subdev *sensor;
>> +};
>> +
>> +static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
>> +{
>> +	return container_of(sdev, struct csi_priv, sd);
>> +}
>> +
>> +/* Update the CSI whole sensor and active windows */
>> +static int csi_setup(struct csi_priv *priv)
>> +{
>> +	struct v4l2_mbus_framefmt infmt;
>> +
>> +	ipu_csi_set_window(priv->csi, &priv->crop);
>> +
>> +	/*
>> +	 * the ipu-csi doesn't understand ALTERNATE, but it only
>> +	 * needs to know whether the stream is interlaced, so set
>> +	 * to INTERLACED if infmt field is ALTERNATE.
>> +	 */
>> +	infmt = priv->format_mbus[priv->input_pad];
>> +	if (infmt.field == V4L2_FIELD_ALTERNATE)
>> +		infmt.field = V4L2_FIELD_INTERLACED;
> That should be SEQ_TB/BT depending on video standard.

fixed.

>> +
>> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	int ret = 0;
>> +
>> +	if (!priv->src_sd || !priv->sink_sd)
>> +		return -EPIPE;
>> +
>> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> These could be silenced a bit.

yeah, I think it is time for that. I've silenced all the
v4l2_info()'s for stream on/off, power on/off, as well
as some others.

>
>> +static int csi_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	int ret = 0;
>> +
>> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> +	if (priv->fim && on != priv->power_on)
>> +		ret = imx_media_fim_set_power(priv->fim, on);
>> +
>> +	if (!ret)
>> +		priv->power_on = on;
>> +	return ret;
>> +}
> Is this called multiple times? I'd expect a poweron during open and a
> poweroff during close, so no need for priv->power_on.

It is actually called multiple times. The s_power subdev callbacks are
made every time there is a new link established, in the imx-media core's
link_notify(), in order to re-establish power to the active subdevs in the
new pipeline.

This might change after I look into using v4l2_pipeline_pm_use().

>
>> +static int csi_link_setup(struct media_entity *entity,
>> +			  const struct media_pad *local,
>> +			  const struct media_pad *remote, u32 flags)
>> +{
>> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	struct v4l2_subdev *remote_sd;
>> +
>> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
>> +		local->entity->name);
>> +
>> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
>> +
>> +	if (local->flags & MEDIA_PAD_FL_SINK) {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (priv->src_sd)
>> +				return -EBUSY;
>> +			priv->src_sd = remote_sd;
>> +		} else {
>> +			priv->src_sd = NULL;
>> +		}
>> +
>> +		return 0;
>> +	}
>> +
>> +	if (flags & MEDIA_LNK_FL_ENABLED) {
>> +		if (priv->sink_sd)
>> +			return -EBUSY;
>> +		priv->sink_sd = remote_sd;
>> +	} else {
>> +		priv->sink_sd = NULL;
>> +		return 0;
>> +	}
>> +
>> +	/* set CSI destination */
>> +	switch (remote_sd->grp_id) {
>> +	case IMX_MEDIA_GRP_ID_SMFC0:
>> +	case IMX_MEDIA_GRP_ID_SMFC1:
>> +	case IMX_MEDIA_GRP_ID_SMFC2:
>> +	case IMX_MEDIA_GRP_ID_SMFC3:
> With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1
> to the SMFC2 channel.

right, I'll do that.

>
> [...]
>> +static int csi_set_fmt(struct v4l2_subdev *sd,
>> +		       struct v4l2_subdev_pad_config *cfg,
>> +		       struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
>> +	struct v4l2_rect crop;
>> +	int ret;
>> +
>> +	if (sdformat->pad >= CSI_NUM_PADS)
>> +		return -EINVAL;
>> +
>> +	if (priv->stream_on)
>> +		return -EBUSY;
>> +
>> +	infmt = &priv->format_mbus[priv->input_pad];
>> +	outfmt = &priv->format_mbus[priv->output_pad];
>> +
>> +	if (sdformat->pad == priv->output_pad) {
>> +		sdformat->format.code = infmt->code;
>> +		sdformat->format.field = infmt->field;
>> +		crop.left = priv->crop.left;
>> +		crop.top = priv->crop.top;
>> +		crop.width = sdformat->format.width;
>> +		crop.height = sdformat->format.height;
>> +		ret = csi_try_crop(priv, &crop);
>> +		if (ret)
>> +			return ret;
>> +		sdformat->format.width = crop.width;
>> +		sdformat->format.height = crop.height;
>> +	}
>> +
>> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> Should there be some limitations on the format here?

done, I've added call to v4l_bound_align_image(), passing
it the min/max frame sizes. CSI's sensor/active frame size
register fields are 12 bits, so max is 4096 for both width and
height.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-16 21:15       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 21:15 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/16/2017 07:03 AM, Philipp Zabel wrote:
> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
>> This is a media entity subdevice for the i.MX Camera
>> Serial Interface module.
> s/Serial/Sensor/

done.

>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   drivers/staging/media/imx/Kconfig   |  13 +
>>   drivers/staging/media/imx/Makefile  |   2 +
>>   drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
>>   3 files changed, 659 insertions(+)
>>   create mode 100644 drivers/staging/media/imx/imx-csi.c
>>
>> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
>> index bfde58d..ce2d2c8 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
>>   	  Say yes here to enable support for video4linux media controller
>>   	  driver for the i.MX5/6 SOC.
>>   
>> +if VIDEO_IMX_MEDIA
>> +menu "i.MX5/6 Media Sub devices"
>> +
>> +config VIDEO_IMX_CAMERA
> s/CAMERA/CSI/ ?

done.

>> +	tristate "i.MX5/6 Camera driver"
> i.MX5/6 Camera Sensor Interface driver

done.

>
>> +
>> +struct csi_priv {
>> +	struct device *dev;
>> +	struct ipu_soc *ipu;
>> +	struct imx_media_dev *md;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad[CSI_NUM_PADS];
>> +	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
>> +	struct v4l2_mbus_config sensor_mbus_cfg;
>> +	struct v4l2_rect crop;
>> +	struct ipu_csi *csi;
>> +	int csi_id;
>> +	int input_pad;
>> +	int output_pad;
>> +	bool power_on;  /* power is on */
>> +	bool stream_on; /* streaming is on */
>> +
>> +	/* the sink for the captured frames */
>> +	struct v4l2_subdev *sink_sd;
>> +	enum ipu_csi_dest dest;
>> +	struct v4l2_subdev *src_sd;
> src_sd is not used except that its presence marks an enabled input link.
> -> could be changed to bool.

For now I prefer to keep it a pointer to the src/sink subdevs.
At some point the CSI may have some reason to know the
identity of the source.

>
>> +	struct v4l2_ctrl_handler ctrl_hdlr;
>> +	struct imx_media_fim *fim;
>> +
>> +	/* the attached sensor at stream on */
>> +	struct imx_media_subdev *sensor;
>> +};
>> +
>> +static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
>> +{
>> +	return container_of(sdev, struct csi_priv, sd);
>> +}
>> +
>> +/* Update the CSI whole sensor and active windows */
>> +static int csi_setup(struct csi_priv *priv)
>> +{
>> +	struct v4l2_mbus_framefmt infmt;
>> +
>> +	ipu_csi_set_window(priv->csi, &priv->crop);
>> +
>> +	/*
>> +	 * the ipu-csi doesn't understand ALTERNATE, but it only
>> +	 * needs to know whether the stream is interlaced, so set
>> +	 * to INTERLACED if infmt field is ALTERNATE.
>> +	 */
>> +	infmt = priv->format_mbus[priv->input_pad];
>> +	if (infmt.field == V4L2_FIELD_ALTERNATE)
>> +		infmt.field = V4L2_FIELD_INTERLACED;
> That should be SEQ_TB/BT depending on video standard.

fixed.

>> +
>> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	int ret = 0;
>> +
>> +	if (!priv->src_sd || !priv->sink_sd)
>> +		return -EPIPE;
>> +
>> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> These could be silenced a bit.

yeah, I think it is time for that. I've silenced all the
v4l2_info()'s for stream on/off, power on/off, as well
as some others.

>
>> +static int csi_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	int ret = 0;
>> +
>> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> +	if (priv->fim && on != priv->power_on)
>> +		ret = imx_media_fim_set_power(priv->fim, on);
>> +
>> +	if (!ret)
>> +		priv->power_on = on;
>> +	return ret;
>> +}
> Is this called multiple times? I'd expect a poweron during open and a
> poweroff during close, so no need for priv->power_on.

It is actually called multiple times. The s_power subdev callbacks are
made every time there is a new link established, in the imx-media core's
link_notify(), in order to re-establish power to the active subdevs in the
new pipeline.

This might change after I look into using v4l2_pipeline_pm_use().

>
>> +static int csi_link_setup(struct media_entity *entity,
>> +			  const struct media_pad *local,
>> +			  const struct media_pad *remote, u32 flags)
>> +{
>> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	struct v4l2_subdev *remote_sd;
>> +
>> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
>> +		local->entity->name);
>> +
>> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
>> +
>> +	if (local->flags & MEDIA_PAD_FL_SINK) {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (priv->src_sd)
>> +				return -EBUSY;
>> +			priv->src_sd = remote_sd;
>> +		} else {
>> +			priv->src_sd = NULL;
>> +		}
>> +
>> +		return 0;
>> +	}
>> +
>> +	if (flags & MEDIA_LNK_FL_ENABLED) {
>> +		if (priv->sink_sd)
>> +			return -EBUSY;
>> +		priv->sink_sd = remote_sd;
>> +	} else {
>> +		priv->sink_sd = NULL;
>> +		return 0;
>> +	}
>> +
>> +	/* set CSI destination */
>> +	switch (remote_sd->grp_id) {
>> +	case IMX_MEDIA_GRP_ID_SMFC0:
>> +	case IMX_MEDIA_GRP_ID_SMFC1:
>> +	case IMX_MEDIA_GRP_ID_SMFC2:
>> +	case IMX_MEDIA_GRP_ID_SMFC3:
> With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1
> to the SMFC2 channel.

right, I'll do that.

>
> [...]
>> +static int csi_set_fmt(struct v4l2_subdev *sd,
>> +		       struct v4l2_subdev_pad_config *cfg,
>> +		       struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
>> +	struct v4l2_rect crop;
>> +	int ret;
>> +
>> +	if (sdformat->pad >= CSI_NUM_PADS)
>> +		return -EINVAL;
>> +
>> +	if (priv->stream_on)
>> +		return -EBUSY;
>> +
>> +	infmt = &priv->format_mbus[priv->input_pad];
>> +	outfmt = &priv->format_mbus[priv->output_pad];
>> +
>> +	if (sdformat->pad == priv->output_pad) {
>> +		sdformat->format.code = infmt->code;
>> +		sdformat->format.field = infmt->field;
>> +		crop.left = priv->crop.left;
>> +		crop.top = priv->crop.top;
>> +		crop.width = sdformat->format.width;
>> +		crop.height = sdformat->format.height;
>> +		ret = csi_try_crop(priv, &crop);
>> +		if (ret)
>> +			return ret;
>> +		sdformat->format.width = crop.width;
>> +		sdformat->format.height = crop.height;
>> +	}
>> +
>> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> Should there be some limitations on the format here?

done, I've added call to v4l_bound_align_image(), passing
it the min/max frame sizes. CSI's sensor/active frame size
register fields are 12 bits, so max is 4096 for both width and
height.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-16 21:15       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-16 21:15 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/16/2017 07:03 AM, Philipp Zabel wrote:
> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
>> This is a media entity subdevice for the i.MX Camera
>> Serial Interface module.
> s/Serial/Sensor/

done.

>> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
>> ---
>>   drivers/staging/media/imx/Kconfig   |  13 +
>>   drivers/staging/media/imx/Makefile  |   2 +
>>   drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
>>   3 files changed, 659 insertions(+)
>>   create mode 100644 drivers/staging/media/imx/imx-csi.c
>>
>> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
>> index bfde58d..ce2d2c8 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
>>   	  Say yes here to enable support for video4linux media controller
>>   	  driver for the i.MX5/6 SOC.
>>   
>> +if VIDEO_IMX_MEDIA
>> +menu "i.MX5/6 Media Sub devices"
>> +
>> +config VIDEO_IMX_CAMERA
> s/CAMERA/CSI/ ?

done.

>> +	tristate "i.MX5/6 Camera driver"
> i.MX5/6 Camera Sensor Interface driver

done.

>
>> +
>> +struct csi_priv {
>> +	struct device *dev;
>> +	struct ipu_soc *ipu;
>> +	struct imx_media_dev *md;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad[CSI_NUM_PADS];
>> +	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
>> +	struct v4l2_mbus_config sensor_mbus_cfg;
>> +	struct v4l2_rect crop;
>> +	struct ipu_csi *csi;
>> +	int csi_id;
>> +	int input_pad;
>> +	int output_pad;
>> +	bool power_on;  /* power is on */
>> +	bool stream_on; /* streaming is on */
>> +
>> +	/* the sink for the captured frames */
>> +	struct v4l2_subdev *sink_sd;
>> +	enum ipu_csi_dest dest;
>> +	struct v4l2_subdev *src_sd;
> src_sd is not used except that its presence marks an enabled input link.
> -> could be changed to bool.

For now I prefer to keep it a pointer to the src/sink subdevs.
At some point the CSI may have some reason to know the
identity of the source.

>
>> +	struct v4l2_ctrl_handler ctrl_hdlr;
>> +	struct imx_media_fim *fim;
>> +
>> +	/* the attached sensor at stream on */
>> +	struct imx_media_subdev *sensor;
>> +};
>> +
>> +static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
>> +{
>> +	return container_of(sdev, struct csi_priv, sd);
>> +}
>> +
>> +/* Update the CSI whole sensor and active windows */
>> +static int csi_setup(struct csi_priv *priv)
>> +{
>> +	struct v4l2_mbus_framefmt infmt;
>> +
>> +	ipu_csi_set_window(priv->csi, &priv->crop);
>> +
>> +	/*
>> +	 * the ipu-csi doesn't understand ALTERNATE, but it only
>> +	 * needs to know whether the stream is interlaced, so set
>> +	 * to INTERLACED if infmt field is ALTERNATE.
>> +	 */
>> +	infmt = priv->format_mbus[priv->input_pad];
>> +	if (infmt.field == V4L2_FIELD_ALTERNATE)
>> +		infmt.field = V4L2_FIELD_INTERLACED;
> That should be SEQ_TB/BT depending on video standard.

fixed.

>> +
>> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	int ret = 0;
>> +
>> +	if (!priv->src_sd || !priv->sink_sd)
>> +		return -EPIPE;
>> +
>> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> These could be silenced a bit.

yeah, I think it is time for that. I've silenced all the
v4l2_info()'s for stream on/off, power on/off, as well
as some others.

>
>> +static int csi_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	int ret = 0;
>> +
>> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> +	if (priv->fim && on != priv->power_on)
>> +		ret = imx_media_fim_set_power(priv->fim, on);
>> +
>> +	if (!ret)
>> +		priv->power_on = on;
>> +	return ret;
>> +}
> Is this called multiple times? I'd expect a poweron during open and a
> poweroff during close, so no need for priv->power_on.

It is actually called multiple times. The s_power subdev callbacks are
made every time there is a new link established, in the imx-media core's
link_notify(), in order to re-establish power to the active subdevs in the
new pipeline.

This might change after I look into using v4l2_pipeline_pm_use().

>
>> +static int csi_link_setup(struct media_entity *entity,
>> +			  const struct media_pad *local,
>> +			  const struct media_pad *remote, u32 flags)
>> +{
>> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	struct v4l2_subdev *remote_sd;
>> +
>> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
>> +		local->entity->name);
>> +
>> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
>> +
>> +	if (local->flags & MEDIA_PAD_FL_SINK) {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (priv->src_sd)
>> +				return -EBUSY;
>> +			priv->src_sd = remote_sd;
>> +		} else {
>> +			priv->src_sd = NULL;
>> +		}
>> +
>> +		return 0;
>> +	}
>> +
>> +	if (flags & MEDIA_LNK_FL_ENABLED) {
>> +		if (priv->sink_sd)
>> +			return -EBUSY;
>> +		priv->sink_sd = remote_sd;
>> +	} else {
>> +		priv->sink_sd = NULL;
>> +		return 0;
>> +	}
>> +
>> +	/* set CSI destination */
>> +	switch (remote_sd->grp_id) {
>> +	case IMX_MEDIA_GRP_ID_SMFC0:
>> +	case IMX_MEDIA_GRP_ID_SMFC1:
>> +	case IMX_MEDIA_GRP_ID_SMFC2:
>> +	case IMX_MEDIA_GRP_ID_SMFC3:
> With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1
> to the SMFC2 channel.

right, I'll do that.

>
> [...]
>> +static int csi_set_fmt(struct v4l2_subdev *sd,
>> +		       struct v4l2_subdev_pad_config *cfg,
>> +		       struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
>> +	struct v4l2_rect crop;
>> +	int ret;
>> +
>> +	if (sdformat->pad >= CSI_NUM_PADS)
>> +		return -EINVAL;
>> +
>> +	if (priv->stream_on)
>> +		return -EBUSY;
>> +
>> +	infmt = &priv->format_mbus[priv->input_pad];
>> +	outfmt = &priv->format_mbus[priv->output_pad];
>> +
>> +	if (sdformat->pad == priv->output_pad) {
>> +		sdformat->format.code = infmt->code;
>> +		sdformat->format.field = infmt->field;
>> +		crop.left = priv->crop.left;
>> +		crop.top = priv->crop.top;
>> +		crop.width = sdformat->format.width;
>> +		crop.height = sdformat->format.height;
>> +		ret = csi_try_crop(priv, &crop);
>> +		if (ret)
>> +			return ret;
>> +		sdformat->format.width = crop.width;
>> +		sdformat->format.height = crop.height;
>> +	}
>> +
>> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> Should there be some limitations on the format here?

done, I've added call to v4l_bound_align_image(), passing
it the min/max frame sizes. CSI's sensor/active frame size
register fields are 12 bits, so max is 4096 for both width and
height.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-14 22:42     ` Steve Longerbeam
  2017-01-19  1:44         ` Steve Longerbeam
@ 2017-01-19  1:44         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-19  1:44 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/14/2017 02:42 PM, Steve Longerbeam wrote:
>
>>> +/* parse inputs property from a sensor node */
>>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
>>> +				   struct imx_media_subdev *sensor,
>>> +				   struct device_node *sensor_np)
>>> +{
>>> +	struct imx_media_sensor_input *sinput = &sensor->input;
>>> +	int ret, i;
>>> +
>>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
>>> +		const char *input_name;
>>> +		u32 val;
>>> +
>>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
>>> +		if (ret)
>>> +			break;
>>> +
>>> +		sinput->value[i] = val;
>>> +
>>> +		ret = of_property_read_string_index(sensor_np, "input-names",
>>> +						    i, &input_name);
>>> +		/*
>>> +		 * if input-names not provided, they will be set using
>>> +		 * the subdev name once the sensor is known during
>>> +		 * async bind
>>> +		 */
>>> +		if (!ret)
>>> +			strncpy(sinput->name[i], input_name,
>>> +				sizeof(sinput->name[i]));
>>> +	}
>>> +
>>> +	sinput->num = i;
>>> +
>>> +	/* if no inputs provided just assume a single input */
>>> +	if (sinput->num == 0)
>>> +		sinput->num = 1;
>>> +}
>> This should be parsed by the sensor driver, not imx-media.
>
> you're probably right. I'll submit a patch for adv7180.c.

Actually, the problem here is that this parses an input routing value to
pass to s_routing, and an input name string. There would need to be
another subdev callback, maybe enum_imput, that would return this
information for the bridge driver, if this info were to be parsed and
maintained by the sensor.

But this info should really be known and parsed by the bridge anyway,
because as the header for s_routing states,

"An i2c device shouldn't know about whether an input pin is connected
  to a Composite connector, because on another board or platform it
  might be connected to something else entirely. The calling driver is
  responsible for mapping a user-level input to the right pins on the i2c
  device."

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-19  1:44         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-19  1:44 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/14/2017 02:42 PM, Steve Longerbeam wrote:
>
>>> +/* parse inputs property from a sensor node */
>>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
>>> +				   struct imx_media_subdev *sensor,
>>> +				   struct device_node *sensor_np)
>>> +{
>>> +	struct imx_media_sensor_input *sinput = &sensor->input;
>>> +	int ret, i;
>>> +
>>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
>>> +		const char *input_name;
>>> +		u32 val;
>>> +
>>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
>>> +		if (ret)
>>> +			break;
>>> +
>>> +		sinput->value[i] = val;
>>> +
>>> +		ret = of_property_read_string_index(sensor_np, "input-names",
>>> +						    i, &input_name);
>>> +		/*
>>> +		 * if input-names not provided, they will be set using
>>> +		 * the subdev name once the sensor is known during
>>> +		 * async bind
>>> +		 */
>>> +		if (!ret)
>>> +			strncpy(sinput->name[i], input_name,
>>> +				sizeof(sinput->name[i]));
>>> +	}
>>> +
>>> +	sinput->num = i;
>>> +
>>> +	/* if no inputs provided just assume a single input */
>>> +	if (sinput->num == 0)
>>> +		sinput->num = 1;
>>> +}
>> This should be parsed by the sensor driver, not imx-media.
>
> you're probably right. I'll submit a patch for adv7180.c.

Actually, the problem here is that this parses an input routing value to
pass to s_routing, and an input name string. There would need to be
another subdev callback, maybe enum_imput, that would return this
information for the bridge driver, if this info were to be parsed and
maintained by the sensor.

But this info should really be known and parsed by the bridge anyway,
because as the header for s_routing states,

"An i2c device shouldn't know about whether an input pin is connected
  to a Composite connector, because on another board or platform it
  might be connected to something else entirely. The calling driver is
  responsible for mapping a user-level input to the right pins on the i2c
  device."

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-19  1:44         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-19  1:44 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/14/2017 02:42 PM, Steve Longerbeam wrote:
>
>>> +/* parse inputs property from a sensor node */
>>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
>>> +				   struct imx_media_subdev *sensor,
>>> +				   struct device_node *sensor_np)
>>> +{
>>> +	struct imx_media_sensor_input *sinput = &sensor->input;
>>> +	int ret, i;
>>> +
>>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
>>> +		const char *input_name;
>>> +		u32 val;
>>> +
>>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
>>> +		if (ret)
>>> +			break;
>>> +
>>> +		sinput->value[i] = val;
>>> +
>>> +		ret = of_property_read_string_index(sensor_np, "input-names",
>>> +						    i, &input_name);
>>> +		/*
>>> +		 * if input-names not provided, they will be set using
>>> +		 * the subdev name once the sensor is known during
>>> +		 * async bind
>>> +		 */
>>> +		if (!ret)
>>> +			strncpy(sinput->name[i], input_name,
>>> +				sizeof(sinput->name[i]));
>>> +	}
>>> +
>>> +	sinput->num = i;
>>> +
>>> +	/* if no inputs provided just assume a single input */
>>> +	if (sinput->num == 0)
>>> +		sinput->num = 1;
>>> +}
>> This should be parsed by the sensor driver, not imx-media.
>
> you're probably right. I'll submit a patch for adv7180.c.

Actually, the problem here is that this parses an input routing value to
pass to s_routing, and an input name string. There would need to be
another subdev callback, maybe enum_imput, that would return this
information for the bridge driver, if this info were to be parsed and
maintained by the sensor.

But this info should really be known and parsed by the bridge anyway,
because as the header for s_routing states,

"An i2c device shouldn't know about whether an input pin is connected
  to a Composite connector, because on another board or platform it
  might be connected to something else entirely. The calling driver is
  responsible for mapping a user-level input to the right pins on the i2c
  device."

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 13:52   ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 13:52 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Hi Steve, Philipp,

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> In version 3:
> 
> Changes suggested by Rob Herring <robh@kernel.org>:
> 
>   - prepended FIM node properties with vendor prefix "fsl,".
> 
>   - make mipi csi-2 receiver compatible string SoC specific:
>     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
> 
>   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
> 
>   - removed board-specific info from the media driver binding doc. These
>     were all related to sensor bindings, which already are (adv7180)
>     or will be (ov564x) covered in separate binding docs. All reference
>     board info not related to DT bindings has been moved to
>     Documentation/media/v4l-drivers/imx.rst.
> 
>   - removed "_mipi" from the OV5640 compatible string.
> 
> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
> 
>   Mostly cosmetic/non-functional changes which I won't list here, except
>   for the following:
> 
>   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
> 
>   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
> 
>   - check/handle return code from required reg property of CSI port nodes.
> 
>   - check/handle return code from clk_prepare_enable().
> 
> Changes suggested by Fabio Estevam <festevam@gmail.com>:
> 
>   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
> 
>   - finally got around to passing valid IOMUX pin config values to the
>     pin groups.
> 
> Other changes:
> 
>   - removed the FIM properties that overrided the v4l2 FIM control defaults
>     values. This was left-over from a requirement of a customer and is not
>     necessary here.
> 
>   - The FIM must be explicitly enabled in the fim child node under the CSI
>     port nodes, using the status property. If not enabled, FIM v4l2 controls
>     will not appear in the video capture driver.
> 
>   - brought in additional media types patch from Philipp Zabel. Use new
>     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
> 
>   - brought in latest platform generic video multiplexer subdevice driver
>     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
> 
>   - removed imx-media-of.h, moved those prototypes into imx-media.h.

Based on the discussion on the mailinglist it seems everyone agrees that this
is the preferred driver, correct?

There are a bunch of review comments, so I will wait for a v4. I plan to merge
that staging driver unless there are major issues with it.

I am not sure if I would merge those sensor drivers in staging, I'll have to
take a closer look at those once v4 is posted.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 13:52   ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 13:52 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r
  Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

Hi Steve, Philipp,

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> In version 3:
> 
> Changes suggested by Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> 
>   - prepended FIM node properties with vendor prefix "fsl,".
> 
>   - make mipi csi-2 receiver compatible string SoC specific:
>     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
> 
>   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
> 
>   - removed board-specific info from the media driver binding doc. These
>     were all related to sensor bindings, which already are (adv7180)
>     or will be (ov564x) covered in separate binding docs. All reference
>     board info not related to DT bindings has been moved to
>     Documentation/media/v4l-drivers/imx.rst.
> 
>   - removed "_mipi" from the OV5640 compatible string.
> 
> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>:
> 
>   Mostly cosmetic/non-functional changes which I won't list here, except
>   for the following:
> 
>   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
> 
>   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
> 
>   - check/handle return code from required reg property of CSI port nodes.
> 
>   - check/handle return code from clk_prepare_enable().
> 
> Changes suggested by Fabio Estevam <festevam-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 
>   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
> 
>   - finally got around to passing valid IOMUX pin config values to the
>     pin groups.
> 
> Other changes:
> 
>   - removed the FIM properties that overrided the v4l2 FIM control defaults
>     values. This was left-over from a requirement of a customer and is not
>     necessary here.
> 
>   - The FIM must be explicitly enabled in the fim child node under the CSI
>     port nodes, using the status property. If not enabled, FIM v4l2 controls
>     will not appear in the video capture driver.
> 
>   - brought in additional media types patch from Philipp Zabel. Use new
>     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
> 
>   - brought in latest platform generic video multiplexer subdevice driver
>     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
> 
>   - removed imx-media-of.h, moved those prototypes into imx-media.h.

Based on the discussion on the mailinglist it seems everyone agrees that this
is the preferred driver, correct?

There are a bunch of review comments, so I will wait for a v4. I plan to merge
that staging driver unless there are major issues with it.

I am not sure if I would merge those sensor drivers in staging, I'll have to
take a closer look at those once v4 is posted.

Regards,

	Hans
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 13:52   ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 13:52 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve, Philipp,

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> In version 3:
> 
> Changes suggested by Rob Herring <robh@kernel.org>:
> 
>   - prepended FIM node properties with vendor prefix "fsl,".
> 
>   - make mipi csi-2 receiver compatible string SoC specific:
>     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
> 
>   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
> 
>   - removed board-specific info from the media driver binding doc. These
>     were all related to sensor bindings, which already are (adv7180)
>     or will be (ov564x) covered in separate binding docs. All reference
>     board info not related to DT bindings has been moved to
>     Documentation/media/v4l-drivers/imx.rst.
> 
>   - removed "_mipi" from the OV5640 compatible string.
> 
> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
> 
>   Mostly cosmetic/non-functional changes which I won't list here, except
>   for the following:
> 
>   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
> 
>   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
> 
>   - check/handle return code from required reg property of CSI port nodes.
> 
>   - check/handle return code from clk_prepare_enable().
> 
> Changes suggested by Fabio Estevam <festevam@gmail.com>:
> 
>   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
> 
>   - finally got around to passing valid IOMUX pin config values to the
>     pin groups.
> 
> Other changes:
> 
>   - removed the FIM properties that overrided the v4l2 FIM control defaults
>     values. This was left-over from a requirement of a customer and is not
>     necessary here.
> 
>   - The FIM must be explicitly enabled in the fim child node under the CSI
>     port nodes, using the status property. If not enabled, FIM v4l2 controls
>     will not appear in the video capture driver.
> 
>   - brought in additional media types patch from Philipp Zabel. Use new
>     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
> 
>   - brought in latest platform generic video multiplexer subdevice driver
>     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
> 
>   - removed imx-media-of.h, moved those prototypes into imx-media.h.

Based on the discussion on the mailinglist it seems everyone agrees that this
is the preferred driver, correct?

There are a bunch of review comments, so I will wait for a v4. I plan to merge
that staging driver unless there are major issues with it.

I am not sure if I would merge those sensor drivers in staging, I'll have to
take a closer look at those once v4 is posted.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 12/24] add mux and video interface bridge entity functions
  2017-01-07  2:11   ` Steve Longerbeam
@ 2017-01-20 13:56     ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 13:56 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
>  include/uapi/linux/media.h                        |  6 ++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
> index 3e03dc2..023be29 100644
> --- a/Documentation/media/uapi/mediactl/media-types.rst
> +++ b/Documentation/media/uapi/mediactl/media-types.rst
> @@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
>  	  received on its sink pad and outputs the statistics data on
>  	  its source pad.
>  
> +    -  ..  row 29
> +
> +       ..  _MEDIA-ENT-F-MUX:
> +
> +       -  ``MEDIA_ENT_F_MUX``

I would prefer MEDIA_ENT_F_VID_MUX since this is video specific.

Regards,

	Hans

> +
> +       - Video multiplexer. An entity capable of multiplexing must have at
> +         least two sink pads and one source pad, and must pass the video
> +         frame(s) received from the active sink pad to the source pad. Video
> +         frame(s) from the inactive sink pads are discarded.
> +
> +    -  ..  row 30
> +
> +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> +
> +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> +
> +       - Video interface bridge. A video interface bridge entity must have at
> +         least one sink pad and one source pad. It receives video frame(s) on
> +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> +         converts them and outputs them on its source pad in another bus format
> +         (eDP, MIPI CSI-2, parallel, ...).
>  
>  ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
>  
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 4890787..08a8bfa 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -105,6 +105,12 @@ struct media_device_info {
>  #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
>  
>  /*
> + * Switch and bridge entitites
> + */
> +#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
> +#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
> +
> +/*
>   * Connectors
>   */
>  /* It is a responsibility of the entity drivers to add connectors and links */
> 

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 12/24] add mux and video interface bridge entity functions
@ 2017-01-20 13:56     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 13:56 UTC (permalink / raw)
  To: linux-arm-kernel

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
>  include/uapi/linux/media.h                        |  6 ++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
> index 3e03dc2..023be29 100644
> --- a/Documentation/media/uapi/mediactl/media-types.rst
> +++ b/Documentation/media/uapi/mediactl/media-types.rst
> @@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
>  	  received on its sink pad and outputs the statistics data on
>  	  its source pad.
>  
> +    -  ..  row 29
> +
> +       ..  _MEDIA-ENT-F-MUX:
> +
> +       -  ``MEDIA_ENT_F_MUX``

I would prefer MEDIA_ENT_F_VID_MUX since this is video specific.

Regards,

	Hans

> +
> +       - Video multiplexer. An entity capable of multiplexing must have at
> +         least two sink pads and one source pad, and must pass the video
> +         frame(s) received from the active sink pad to the source pad. Video
> +         frame(s) from the inactive sink pads are discarded.
> +
> +    -  ..  row 30
> +
> +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> +
> +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> +
> +       - Video interface bridge. A video interface bridge entity must have at
> +         least one sink pad and one source pad. It receives video frame(s) on
> +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> +         converts them and outputs them on its source pad in another bus format
> +         (eDP, MIPI CSI-2, parallel, ...).
>  
>  ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
>  
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 4890787..08a8bfa 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -105,6 +105,12 @@ struct media_device_info {
>  #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
>  
>  /*
> + * Switch and bridge entitites
> + */
> +#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
> +#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
> +
> +/*
>   * Connectors
>   */
>  /* It is a responsibility of the entity drivers to add connectors and links */
> 

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-20 14:03     ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:03 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Sascha Hauer, Steve Longerbeam

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"
> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.
> +
> +Example:
> +
> +syscon {
> +	compatible = "syscon", "simple-mfd";
> +
> +	mux {
> +		compatible = "video-multiplexer";
> +		/* Single bit (1 << 19) in syscon register 0x04: */
> +		reg = <0x04>;
> +		bit-mask = <1>;
> +		bit-shift = <19>;
> +		#address-cells = <1>;
> +		#size-cells = <0>;
> +
> +		port@0 {
> +			reg = <0>;
> +
> +			mux_in0: endpoint {
> +				remote-endpoint = <&video_source0_out>;
> +			};
> +		};
> +
> +		port@1 {
> +			reg = <1>;
> +
> +			mux_in1: endpoint {
> +				remote-endpoint = <&video_source1_out>;
> +			};
> +		};
> +
> +		port@2 {
> +			reg = <2>;
> +
> +			mux_out: endpoint {
> +				remote-endpoint = <&capture_interface_in>;
> +			};
> +		};
> +	};
> +};
> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> index d944421..65614b5 100644
> --- a/drivers/media/platform/Kconfig
> +++ b/drivers/media/platform/Kconfig
> @@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
>  	  To compile this driver as a module, choose M here: the
>  	  module will be called arv.
>  
> +config VIDEO_MULTIPLEXER
> +	tristate "Video Multiplexer"
> +	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> +	help
> +	  This driver provides support for SoC internal N:1 video bus
> +	  multiplexers controlled by register bitfields as well as external
> +	  2:1 video multiplexers controlled by a single GPIO.
> +
>  config VIDEO_OMAP3
>  	tristate "OMAP 3 Camera support"
>  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> index 5b3cb27..7cf0ee5 100644
> --- a/drivers/media/platform/Makefile
> +++ b/drivers/media/platform/Makefile
> @@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
>  
>  obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
>  
> +obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
> +
>  obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
>  obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
>  obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
> diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
> new file mode 100644
> index 0000000..48980c4
> --- /dev/null
> +++ b/drivers/media/platform/video-multiplexer.c
> @@ -0,0 +1,472 @@
> +/*
> + * video stream multiplexer controlled via gpio or syscon
> + *
> + * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
> + * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version 2
> + * of the License, or (at your option) any later version.
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <linux/err.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/mfd/syscon.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/of_graph.h>
> +#include <linux/platform_device.h>
> +#include <linux/regmap.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-of.h>
> +
> +struct vidsw {
> +	struct v4l2_subdev subdev;
> +	unsigned int num_pads;
> +	struct media_pad *pads;
> +	struct v4l2_mbus_framefmt *format_mbus;
> +	struct v4l2_fract timeperframe;
> +	struct v4l2_of_endpoint *endpoint;
> +	struct regmap_field *field;
> +	struct gpio_desc *gpio;
> +	int active;
> +};
> +
> +static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct vidsw, subdev);
> +}
> +
> +static void vidsw_set_active(struct vidsw *vidsw, int active)
> +{
> +	vidsw->active = active;
> +	if (active < 0)
> +		return;
> +
> +	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
> +
> +	if (vidsw->field)
> +		regmap_field_write(vidsw->field, active);
> +	else if (vidsw->gpio)
> +		gpiod_set_value(vidsw->gpio, active);
> +}
> +
> +static int vidsw_link_setup(struct media_entity *entity,
> +			    const struct media_pad *local,
> +			    const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	/* We have no limitations on enabling or disabling our output link */
> +	if (local->index == vidsw->num_pads - 1)
> +		return 0;
> +
> +	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
> +		if (local->index == vidsw->active) {
> +			dev_dbg(sd->dev, "going inactive\n");
> +			vidsw->active = -1;
> +		}
> +		return 0;
> +	}
> +
> +	if (vidsw->active >= 0) {
> +		struct media_pad *pad;
> +
> +		if (vidsw->active == local->index)
> +			return 0;
> +
> +		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> +		if (pad) {
> +			struct media_link *link;
> +			int ret;
> +
> +			link = media_entity_find_link(pad,
> +						&vidsw->pads[vidsw->active]);
> +			if (link) {
> +				ret = __media_entity_setup_link(link, 0);
> +				if (ret)
> +					return ret;
> +			}
> +		}
> +	}
> +
> +	vidsw_set_active(vidsw, local->index);
> +
> +	return 0;
> +}
> +
> +static struct media_entity_operations vidsw_ops = {
> +	.link_setup = vidsw_link_setup,
> +};
> +
> +static bool vidsw_endpoint_disabled(struct device_node *ep)
> +{
> +	struct device_node *rpp;
> +
> +	if (!of_device_is_available(ep))
> +		return true;
> +
> +	rpp = of_graph_get_remote_port_parent(ep);
> +	if (!rpp)
> +		return true;
> +
> +	return !of_device_is_available(rpp);
> +}
> +
> +static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
> +{
> +	struct device_node *ep;
> +	u32 portno;
> +	int numports;
> +	int ret;
> +	int i;
> +	bool active_link = false;
> +
> +	numports = vidsw->num_pads;
> +
> +	for (i = 0; i < numports - 1; i++)
> +		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
> +	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
> +
> +	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
> +	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
> +				     vidsw->pads);
> +	if (ret < 0)
> +		return ret;
> +
> +	vidsw->subdev.entity.ops = &vidsw_ops;
> +
> +	for_each_endpoint_of_node(node, ep) {
> +		struct v4l2_of_endpoint endpoint;
> +
> +		v4l2_of_parse_endpoint(ep, &endpoint);
> +
> +		portno = endpoint.base.port;
> +		if (portno >= numports - 1)
> +			continue;
> +
> +		if (vidsw_endpoint_disabled(ep)) {
> +			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
> +			continue;
> +		}
> +
> +		vidsw->endpoint[portno] = endpoint;
> +
> +		if (portno == vidsw->active)
> +			active_link = true;
> +	}
> +
> +	for (portno = 0; portno < numports - 1; portno++) {
> +		if (!vidsw->endpoint[portno].base.local_node)
> +			continue;
> +
> +		/* If the active input is not connected, use another */
> +		if (!active_link) {
> +			vidsw_set_active(vidsw, portno);
> +			active_link = true;
> +		}
> +	}
> +
> +	return v4l2_async_register_subdev(&vidsw->subdev);
> +}
> +
> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct media_pad *pad;
> +	int ret;
> +
> +	if (vidsw->active == -1) {
> +		dev_err(sd->dev, "no configuration for inactive mux\n");
> +		return -EINVAL;
> +	}
> +
> +	/*
> +	 * Retrieve media bus configuration from the entity connected to the
> +	 * active input
> +	 */
> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> +	if (pad) {
> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> +		if (ret == -ENOIOCTLCMD)
> +			pad = NULL;
> +		else if (ret < 0) {
> +			dev_err(sd->dev, "failed to get source configuration\n");
> +			return ret;
> +		}
> +	}
> +	if (!pad) {
> +		/* Mirror the input side on the output side */
> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> +		    cfg->type == V4L2_MBUS_BT656)
> +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
> +	}
> +
> +	return 0;
> +}

I am not certain this op is needed at all. In the current kernel this op is only
used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

> +
> +static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct v4l2_subdev *upstream_sd;
> +	struct media_pad *pad;
> +
> +	if (vidsw->active == -1) {
> +		dev_err(sd->dev, "Can not start streaming on inactive mux\n");
> +		return -EINVAL;
> +	}
> +
> +	pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
> +	if (!pad) {
> +		dev_err(sd->dev, "Failed to find remote source pad\n");
> +		return -ENOLINK;
> +	}
> +
> +	if (!is_media_entity_v4l2_subdev(pad->entity)) {
> +		dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
> +		return -ENODEV;
> +	}
> +
> +	upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
> +
> +	return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
> +}
> +
> +static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_frame_interval *fi)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	fi->interval = vidsw->timeperframe;
> +
> +	return 0;
> +}
> +
> +static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_frame_interval *fi)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	vidsw->timeperframe = fi->interval;
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
> +	.g_mbus_config = vidsw_g_mbus_config,
> +	.s_stream = vidsw_s_stream,
> +	.g_frame_interval = vidsw_g_frame_interval,
> +	.s_frame_interval = vidsw_s_frame_interval,
> +};
> +
> +static struct v4l2_mbus_framefmt *
> +__vidsw_get_pad_format(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       unsigned int pad, u32 which)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	switch (which) {
> +	case V4L2_SUBDEV_FORMAT_TRY:
> +		return v4l2_subdev_get_try_format(sd, cfg, pad);
> +	case V4L2_SUBDEV_FORMAT_ACTIVE:
> +		return &vidsw->format_mbus[pad];
> +	default:
> +		return NULL;
> +	}
> +}
> +
> +static int vidsw_get_format(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_pad_config *cfg,
> +			    struct v4l2_subdev_format *sdformat)
> +{
> +	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
> +						   sdformat->which);
> +	return 0;
> +}
> +
> +static int vidsw_set_format(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_pad_config *cfg,
> +			    struct v4l2_subdev_format *sdformat)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct v4l2_mbus_framefmt *mbusformat;
> +
> +	if (sdformat->pad >= vidsw->num_pads)
> +		return -EINVAL;
> +
> +	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
> +					    sdformat->which);
> +	if (!mbusformat)
> +		return -EINVAL;
> +
> +	/* Output pad mirrors active input pad, no limitations on input pads */
> +	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
> +		sdformat->format = vidsw->format_mbus[vidsw->active];
> +
> +	*mbusformat = sdformat->format;
> +
> +	return 0;
> +}
> +
> +static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
> +	.get_fmt = vidsw_get_format,
> +	.set_fmt = vidsw_set_format,
> +};
> +
> +static struct v4l2_subdev_ops vidsw_subdev_ops = {
> +	.pad = &vidsw_pad_ops,
> +	.video = &vidsw_subdev_video_ops,
> +};
> +
> +static int of_get_reg_field(struct device_node *node, struct reg_field *field)
> +{
> +	u32 bit_mask;
> +	int ret;
> +
> +	ret = of_property_read_u32(node, "reg", &field->reg);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
> +	if (ret < 0)
> +		return ret;
> +
> +	field->msb = field->lsb + fls(bit_mask) - 1;
> +
> +	return 0;
> +}
> +
> +static int vidsw_probe(struct platform_device *pdev)
> +{
> +	struct device_node *np = pdev->dev.of_node;
> +	struct of_endpoint endpoint;
> +	struct device_node *ep;
> +	struct reg_field field;
> +	struct vidsw *vidsw;
> +	struct regmap *map;
> +	unsigned int num_pads;
> +	int ret;
> +
> +	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
> +	if (!vidsw)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, vidsw);
> +
> +	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
> +	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
> +			np->name);
> +	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	vidsw->subdev.dev = &pdev->dev;
> +
> +	/*
> +	 * The largest numbered port is the output port. It determines
> +	 * total number of pads
> +	 */
> +	num_pads = 0;
> +	for_each_endpoint_of_node(np, ep) {
> +		of_graph_parse_endpoint(ep, &endpoint);
> +		num_pads = max(num_pads, endpoint.port + 1);
> +	}
> +
> +	if (num_pads < 2) {
> +		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
> +		return -EINVAL;
> +	}
> +
> +	ret = of_get_reg_field(np, &field);
> +	if (ret == 0) {
> +		map = syscon_node_to_regmap(np->parent);
> +		if (!map) {
> +			dev_err(&pdev->dev, "Failed to get syscon register map\n");
> +			return PTR_ERR(map);
> +		}
> +
> +		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
> +		if (IS_ERR(vidsw->field)) {
> +			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
> +			return PTR_ERR(vidsw->field);
> +		}
> +
> +		regmap_field_read(vidsw->field, &vidsw->active);
> +	} else {
> +		if (num_pads > 3) {
> +			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
> +			return -EINVAL;
> +		}
> +
> +		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
> +		if (IS_ERR(vidsw->gpio)) {
> +			dev_warn(&pdev->dev,
> +				 "could not request control gpio: %d\n", ret);
> +			vidsw->gpio = NULL;
> +		}
> +
> +		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
> +	}
> +
> +	vidsw->num_pads = num_pads;
> +	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
> +			GFP_KERNEL);
> +	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
> +			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
> +	vidsw->endpoint = devm_kzalloc(&pdev->dev,
> +			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
> +
> +	ret = vidsw_async_init(vidsw, np);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int vidsw_remove(struct platform_device *pdev)
> +{
> +	struct vidsw *vidsw = platform_get_drvdata(pdev);
> +	struct v4l2_subdev *sd = &vidsw->subdev;
> +
> +	v4l2_async_unregister_subdev(sd);
> +	media_entity_cleanup(&sd->entity);
> +	v4l2_device_unregister_subdev(sd);
> +
> +	return 0;
> +}
> +
> +static const struct of_device_id vidsw_dt_ids[] = {
> +	{ .compatible = "video-multiplexer", },
> +	{ /* sentinel */ }
> +};
> +
> +static struct platform_driver vidsw_driver = {
> +	.probe		= vidsw_probe,
> +	.remove		= vidsw_remove,
> +	.driver		= {
> +		.of_match_table = vidsw_dt_ids,
> +		.name = "video-multiplexer",
> +	},
> +};
> +
> +module_platform_driver(vidsw_driver);
> +
> +MODULE_DESCRIPTION("video stream multiplexer");
> +MODULE_AUTHOR("Sascha Hauer, Pengutronix");
> +MODULE_AUTHOR("Philipp Zabel, Pengutronix");
> +MODULE_LICENSE("GPL");
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-20 14:03     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:03 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, Sascha Hauer, linux-kernel,
	linux-arm-kernel, linux-media

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"
> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.
> +
> +Example:
> +
> +syscon {
> +	compatible = "syscon", "simple-mfd";
> +
> +	mux {
> +		compatible = "video-multiplexer";
> +		/* Single bit (1 << 19) in syscon register 0x04: */
> +		reg = <0x04>;
> +		bit-mask = <1>;
> +		bit-shift = <19>;
> +		#address-cells = <1>;
> +		#size-cells = <0>;
> +
> +		port@0 {
> +			reg = <0>;
> +
> +			mux_in0: endpoint {
> +				remote-endpoint = <&video_source0_out>;
> +			};
> +		};
> +
> +		port@1 {
> +			reg = <1>;
> +
> +			mux_in1: endpoint {
> +				remote-endpoint = <&video_source1_out>;
> +			};
> +		};
> +
> +		port@2 {
> +			reg = <2>;
> +
> +			mux_out: endpoint {
> +				remote-endpoint = <&capture_interface_in>;
> +			};
> +		};
> +	};
> +};
> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> index d944421..65614b5 100644
> --- a/drivers/media/platform/Kconfig
> +++ b/drivers/media/platform/Kconfig
> @@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
>  	  To compile this driver as a module, choose M here: the
>  	  module will be called arv.
>  
> +config VIDEO_MULTIPLEXER
> +	tristate "Video Multiplexer"
> +	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> +	help
> +	  This driver provides support for SoC internal N:1 video bus
> +	  multiplexers controlled by register bitfields as well as external
> +	  2:1 video multiplexers controlled by a single GPIO.
> +
>  config VIDEO_OMAP3
>  	tristate "OMAP 3 Camera support"
>  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> index 5b3cb27..7cf0ee5 100644
> --- a/drivers/media/platform/Makefile
> +++ b/drivers/media/platform/Makefile
> @@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
>  
>  obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
>  
> +obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
> +
>  obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
>  obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
>  obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
> diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
> new file mode 100644
> index 0000000..48980c4
> --- /dev/null
> +++ b/drivers/media/platform/video-multiplexer.c
> @@ -0,0 +1,472 @@
> +/*
> + * video stream multiplexer controlled via gpio or syscon
> + *
> + * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
> + * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version 2
> + * of the License, or (at your option) any later version.
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <linux/err.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/mfd/syscon.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/of_graph.h>
> +#include <linux/platform_device.h>
> +#include <linux/regmap.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-of.h>
> +
> +struct vidsw {
> +	struct v4l2_subdev subdev;
> +	unsigned int num_pads;
> +	struct media_pad *pads;
> +	struct v4l2_mbus_framefmt *format_mbus;
> +	struct v4l2_fract timeperframe;
> +	struct v4l2_of_endpoint *endpoint;
> +	struct regmap_field *field;
> +	struct gpio_desc *gpio;
> +	int active;
> +};
> +
> +static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct vidsw, subdev);
> +}
> +
> +static void vidsw_set_active(struct vidsw *vidsw, int active)
> +{
> +	vidsw->active = active;
> +	if (active < 0)
> +		return;
> +
> +	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
> +
> +	if (vidsw->field)
> +		regmap_field_write(vidsw->field, active);
> +	else if (vidsw->gpio)
> +		gpiod_set_value(vidsw->gpio, active);
> +}
> +
> +static int vidsw_link_setup(struct media_entity *entity,
> +			    const struct media_pad *local,
> +			    const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	/* We have no limitations on enabling or disabling our output link */
> +	if (local->index == vidsw->num_pads - 1)
> +		return 0;
> +
> +	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
> +		if (local->index == vidsw->active) {
> +			dev_dbg(sd->dev, "going inactive\n");
> +			vidsw->active = -1;
> +		}
> +		return 0;
> +	}
> +
> +	if (vidsw->active >= 0) {
> +		struct media_pad *pad;
> +
> +		if (vidsw->active == local->index)
> +			return 0;
> +
> +		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> +		if (pad) {
> +			struct media_link *link;
> +			int ret;
> +
> +			link = media_entity_find_link(pad,
> +						&vidsw->pads[vidsw->active]);
> +			if (link) {
> +				ret = __media_entity_setup_link(link, 0);
> +				if (ret)
> +					return ret;
> +			}
> +		}
> +	}
> +
> +	vidsw_set_active(vidsw, local->index);
> +
> +	return 0;
> +}
> +
> +static struct media_entity_operations vidsw_ops = {
> +	.link_setup = vidsw_link_setup,
> +};
> +
> +static bool vidsw_endpoint_disabled(struct device_node *ep)
> +{
> +	struct device_node *rpp;
> +
> +	if (!of_device_is_available(ep))
> +		return true;
> +
> +	rpp = of_graph_get_remote_port_parent(ep);
> +	if (!rpp)
> +		return true;
> +
> +	return !of_device_is_available(rpp);
> +}
> +
> +static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
> +{
> +	struct device_node *ep;
> +	u32 portno;
> +	int numports;
> +	int ret;
> +	int i;
> +	bool active_link = false;
> +
> +	numports = vidsw->num_pads;
> +
> +	for (i = 0; i < numports - 1; i++)
> +		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
> +	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
> +
> +	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
> +	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
> +				     vidsw->pads);
> +	if (ret < 0)
> +		return ret;
> +
> +	vidsw->subdev.entity.ops = &vidsw_ops;
> +
> +	for_each_endpoint_of_node(node, ep) {
> +		struct v4l2_of_endpoint endpoint;
> +
> +		v4l2_of_parse_endpoint(ep, &endpoint);
> +
> +		portno = endpoint.base.port;
> +		if (portno >= numports - 1)
> +			continue;
> +
> +		if (vidsw_endpoint_disabled(ep)) {
> +			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
> +			continue;
> +		}
> +
> +		vidsw->endpoint[portno] = endpoint;
> +
> +		if (portno == vidsw->active)
> +			active_link = true;
> +	}
> +
> +	for (portno = 0; portno < numports - 1; portno++) {
> +		if (!vidsw->endpoint[portno].base.local_node)
> +			continue;
> +
> +		/* If the active input is not connected, use another */
> +		if (!active_link) {
> +			vidsw_set_active(vidsw, portno);
> +			active_link = true;
> +		}
> +	}
> +
> +	return v4l2_async_register_subdev(&vidsw->subdev);
> +}
> +
> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct media_pad *pad;
> +	int ret;
> +
> +	if (vidsw->active == -1) {
> +		dev_err(sd->dev, "no configuration for inactive mux\n");
> +		return -EINVAL;
> +	}
> +
> +	/*
> +	 * Retrieve media bus configuration from the entity connected to the
> +	 * active input
> +	 */
> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> +	if (pad) {
> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> +		if (ret == -ENOIOCTLCMD)
> +			pad = NULL;
> +		else if (ret < 0) {
> +			dev_err(sd->dev, "failed to get source configuration\n");
> +			return ret;
> +		}
> +	}
> +	if (!pad) {
> +		/* Mirror the input side on the output side */
> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> +		    cfg->type == V4L2_MBUS_BT656)
> +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
> +	}
> +
> +	return 0;
> +}

I am not certain this op is needed at all. In the current kernel this op is only
used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

> +
> +static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct v4l2_subdev *upstream_sd;
> +	struct media_pad *pad;
> +
> +	if (vidsw->active == -1) {
> +		dev_err(sd->dev, "Can not start streaming on inactive mux\n");
> +		return -EINVAL;
> +	}
> +
> +	pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
> +	if (!pad) {
> +		dev_err(sd->dev, "Failed to find remote source pad\n");
> +		return -ENOLINK;
> +	}
> +
> +	if (!is_media_entity_v4l2_subdev(pad->entity)) {
> +		dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
> +		return -ENODEV;
> +	}
> +
> +	upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
> +
> +	return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
> +}
> +
> +static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_frame_interval *fi)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	fi->interval = vidsw->timeperframe;
> +
> +	return 0;
> +}
> +
> +static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_frame_interval *fi)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	vidsw->timeperframe = fi->interval;
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
> +	.g_mbus_config = vidsw_g_mbus_config,
> +	.s_stream = vidsw_s_stream,
> +	.g_frame_interval = vidsw_g_frame_interval,
> +	.s_frame_interval = vidsw_s_frame_interval,
> +};
> +
> +static struct v4l2_mbus_framefmt *
> +__vidsw_get_pad_format(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       unsigned int pad, u32 which)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	switch (which) {
> +	case V4L2_SUBDEV_FORMAT_TRY:
> +		return v4l2_subdev_get_try_format(sd, cfg, pad);
> +	case V4L2_SUBDEV_FORMAT_ACTIVE:
> +		return &vidsw->format_mbus[pad];
> +	default:
> +		return NULL;
> +	}
> +}
> +
> +static int vidsw_get_format(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_pad_config *cfg,
> +			    struct v4l2_subdev_format *sdformat)
> +{
> +	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
> +						   sdformat->which);
> +	return 0;
> +}
> +
> +static int vidsw_set_format(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_pad_config *cfg,
> +			    struct v4l2_subdev_format *sdformat)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct v4l2_mbus_framefmt *mbusformat;
> +
> +	if (sdformat->pad >= vidsw->num_pads)
> +		return -EINVAL;
> +
> +	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
> +					    sdformat->which);
> +	if (!mbusformat)
> +		return -EINVAL;
> +
> +	/* Output pad mirrors active input pad, no limitations on input pads */
> +	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
> +		sdformat->format = vidsw->format_mbus[vidsw->active];
> +
> +	*mbusformat = sdformat->format;
> +
> +	return 0;
> +}
> +
> +static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
> +	.get_fmt = vidsw_get_format,
> +	.set_fmt = vidsw_set_format,
> +};
> +
> +static struct v4l2_subdev_ops vidsw_subdev_ops = {
> +	.pad = &vidsw_pad_ops,
> +	.video = &vidsw_subdev_video_ops,
> +};
> +
> +static int of_get_reg_field(struct device_node *node, struct reg_field *field)
> +{
> +	u32 bit_mask;
> +	int ret;
> +
> +	ret = of_property_read_u32(node, "reg", &field->reg);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
> +	if (ret < 0)
> +		return ret;
> +
> +	field->msb = field->lsb + fls(bit_mask) - 1;
> +
> +	return 0;
> +}
> +
> +static int vidsw_probe(struct platform_device *pdev)
> +{
> +	struct device_node *np = pdev->dev.of_node;
> +	struct of_endpoint endpoint;
> +	struct device_node *ep;
> +	struct reg_field field;
> +	struct vidsw *vidsw;
> +	struct regmap *map;
> +	unsigned int num_pads;
> +	int ret;
> +
> +	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
> +	if (!vidsw)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, vidsw);
> +
> +	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
> +	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
> +			np->name);
> +	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	vidsw->subdev.dev = &pdev->dev;
> +
> +	/*
> +	 * The largest numbered port is the output port. It determines
> +	 * total number of pads
> +	 */
> +	num_pads = 0;
> +	for_each_endpoint_of_node(np, ep) {
> +		of_graph_parse_endpoint(ep, &endpoint);
> +		num_pads = max(num_pads, endpoint.port + 1);
> +	}
> +
> +	if (num_pads < 2) {
> +		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
> +		return -EINVAL;
> +	}
> +
> +	ret = of_get_reg_field(np, &field);
> +	if (ret == 0) {
> +		map = syscon_node_to_regmap(np->parent);
> +		if (!map) {
> +			dev_err(&pdev->dev, "Failed to get syscon register map\n");
> +			return PTR_ERR(map);
> +		}
> +
> +		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
> +		if (IS_ERR(vidsw->field)) {
> +			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
> +			return PTR_ERR(vidsw->field);
> +		}
> +
> +		regmap_field_read(vidsw->field, &vidsw->active);
> +	} else {
> +		if (num_pads > 3) {
> +			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
> +			return -EINVAL;
> +		}
> +
> +		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
> +		if (IS_ERR(vidsw->gpio)) {
> +			dev_warn(&pdev->dev,
> +				 "could not request control gpio: %d\n", ret);
> +			vidsw->gpio = NULL;
> +		}
> +
> +		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
> +	}
> +
> +	vidsw->num_pads = num_pads;
> +	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
> +			GFP_KERNEL);
> +	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
> +			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
> +	vidsw->endpoint = devm_kzalloc(&pdev->dev,
> +			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
> +
> +	ret = vidsw_async_init(vidsw, np);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int vidsw_remove(struct platform_device *pdev)
> +{
> +	struct vidsw *vidsw = platform_get_drvdata(pdev);
> +	struct v4l2_subdev *sd = &vidsw->subdev;
> +
> +	v4l2_async_unregister_subdev(sd);
> +	media_entity_cleanup(&sd->entity);
> +	v4l2_device_unregister_subdev(sd);
> +
> +	return 0;
> +}
> +
> +static const struct of_device_id vidsw_dt_ids[] = {
> +	{ .compatible = "video-multiplexer", },
> +	{ /* sentinel */ }
> +};
> +
> +static struct platform_driver vidsw_driver = {
> +	.probe		= vidsw_probe,
> +	.remove		= vidsw_remove,
> +	.driver		= {
> +		.of_match_table = vidsw_dt_ids,
> +		.name = "video-multiplexer",
> +	},
> +};
> +
> +module_platform_driver(vidsw_driver);
> +
> +MODULE_DESCRIPTION("video stream multiplexer");
> +MODULE_AUTHOR("Sascha Hauer, Pengutronix");
> +MODULE_AUTHOR("Philipp Zabel, Pengutronix");
> +MODULE_LICENSE("GPL");
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-20 14:03     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:03 UTC (permalink / raw)
  To: linux-arm-kernel

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"
> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.
> +
> +Example:
> +
> +syscon {
> +	compatible = "syscon", "simple-mfd";
> +
> +	mux {
> +		compatible = "video-multiplexer";
> +		/* Single bit (1 << 19) in syscon register 0x04: */
> +		reg = <0x04>;
> +		bit-mask = <1>;
> +		bit-shift = <19>;
> +		#address-cells = <1>;
> +		#size-cells = <0>;
> +
> +		port at 0 {
> +			reg = <0>;
> +
> +			mux_in0: endpoint {
> +				remote-endpoint = <&video_source0_out>;
> +			};
> +		};
> +
> +		port at 1 {
> +			reg = <1>;
> +
> +			mux_in1: endpoint {
> +				remote-endpoint = <&video_source1_out>;
> +			};
> +		};
> +
> +		port at 2 {
> +			reg = <2>;
> +
> +			mux_out: endpoint {
> +				remote-endpoint = <&capture_interface_in>;
> +			};
> +		};
> +	};
> +};
> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> index d944421..65614b5 100644
> --- a/drivers/media/platform/Kconfig
> +++ b/drivers/media/platform/Kconfig
> @@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
>  	  To compile this driver as a module, choose M here: the
>  	  module will be called arv.
>  
> +config VIDEO_MULTIPLEXER
> +	tristate "Video Multiplexer"
> +	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> +	help
> +	  This driver provides support for SoC internal N:1 video bus
> +	  multiplexers controlled by register bitfields as well as external
> +	  2:1 video multiplexers controlled by a single GPIO.
> +
>  config VIDEO_OMAP3
>  	tristate "OMAP 3 Camera support"
>  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> index 5b3cb27..7cf0ee5 100644
> --- a/drivers/media/platform/Makefile
> +++ b/drivers/media/platform/Makefile
> @@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
>  
>  obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
>  
> +obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
> +
>  obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
>  obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
>  obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
> diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
> new file mode 100644
> index 0000000..48980c4
> --- /dev/null
> +++ b/drivers/media/platform/video-multiplexer.c
> @@ -0,0 +1,472 @@
> +/*
> + * video stream multiplexer controlled via gpio or syscon
> + *
> + * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
> + * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version 2
> + * of the License, or (at your option) any later version.
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <linux/err.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/mfd/syscon.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/of_graph.h>
> +#include <linux/platform_device.h>
> +#include <linux/regmap.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-of.h>
> +
> +struct vidsw {
> +	struct v4l2_subdev subdev;
> +	unsigned int num_pads;
> +	struct media_pad *pads;
> +	struct v4l2_mbus_framefmt *format_mbus;
> +	struct v4l2_fract timeperframe;
> +	struct v4l2_of_endpoint *endpoint;
> +	struct regmap_field *field;
> +	struct gpio_desc *gpio;
> +	int active;
> +};
> +
> +static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct vidsw, subdev);
> +}
> +
> +static void vidsw_set_active(struct vidsw *vidsw, int active)
> +{
> +	vidsw->active = active;
> +	if (active < 0)
> +		return;
> +
> +	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
> +
> +	if (vidsw->field)
> +		regmap_field_write(vidsw->field, active);
> +	else if (vidsw->gpio)
> +		gpiod_set_value(vidsw->gpio, active);
> +}
> +
> +static int vidsw_link_setup(struct media_entity *entity,
> +			    const struct media_pad *local,
> +			    const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	/* We have no limitations on enabling or disabling our output link */
> +	if (local->index == vidsw->num_pads - 1)
> +		return 0;
> +
> +	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
> +		if (local->index == vidsw->active) {
> +			dev_dbg(sd->dev, "going inactive\n");
> +			vidsw->active = -1;
> +		}
> +		return 0;
> +	}
> +
> +	if (vidsw->active >= 0) {
> +		struct media_pad *pad;
> +
> +		if (vidsw->active == local->index)
> +			return 0;
> +
> +		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> +		if (pad) {
> +			struct media_link *link;
> +			int ret;
> +
> +			link = media_entity_find_link(pad,
> +						&vidsw->pads[vidsw->active]);
> +			if (link) {
> +				ret = __media_entity_setup_link(link, 0);
> +				if (ret)
> +					return ret;
> +			}
> +		}
> +	}
> +
> +	vidsw_set_active(vidsw, local->index);
> +
> +	return 0;
> +}
> +
> +static struct media_entity_operations vidsw_ops = {
> +	.link_setup = vidsw_link_setup,
> +};
> +
> +static bool vidsw_endpoint_disabled(struct device_node *ep)
> +{
> +	struct device_node *rpp;
> +
> +	if (!of_device_is_available(ep))
> +		return true;
> +
> +	rpp = of_graph_get_remote_port_parent(ep);
> +	if (!rpp)
> +		return true;
> +
> +	return !of_device_is_available(rpp);
> +}
> +
> +static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
> +{
> +	struct device_node *ep;
> +	u32 portno;
> +	int numports;
> +	int ret;
> +	int i;
> +	bool active_link = false;
> +
> +	numports = vidsw->num_pads;
> +
> +	for (i = 0; i < numports - 1; i++)
> +		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
> +	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
> +
> +	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
> +	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
> +				     vidsw->pads);
> +	if (ret < 0)
> +		return ret;
> +
> +	vidsw->subdev.entity.ops = &vidsw_ops;
> +
> +	for_each_endpoint_of_node(node, ep) {
> +		struct v4l2_of_endpoint endpoint;
> +
> +		v4l2_of_parse_endpoint(ep, &endpoint);
> +
> +		portno = endpoint.base.port;
> +		if (portno >= numports - 1)
> +			continue;
> +
> +		if (vidsw_endpoint_disabled(ep)) {
> +			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
> +			continue;
> +		}
> +
> +		vidsw->endpoint[portno] = endpoint;
> +
> +		if (portno == vidsw->active)
> +			active_link = true;
> +	}
> +
> +	for (portno = 0; portno < numports - 1; portno++) {
> +		if (!vidsw->endpoint[portno].base.local_node)
> +			continue;
> +
> +		/* If the active input is not connected, use another */
> +		if (!active_link) {
> +			vidsw_set_active(vidsw, portno);
> +			active_link = true;
> +		}
> +	}
> +
> +	return v4l2_async_register_subdev(&vidsw->subdev);
> +}
> +
> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct media_pad *pad;
> +	int ret;
> +
> +	if (vidsw->active == -1) {
> +		dev_err(sd->dev, "no configuration for inactive mux\n");
> +		return -EINVAL;
> +	}
> +
> +	/*
> +	 * Retrieve media bus configuration from the entity connected to the
> +	 * active input
> +	 */
> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> +	if (pad) {
> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> +		if (ret == -ENOIOCTLCMD)
> +			pad = NULL;
> +		else if (ret < 0) {
> +			dev_err(sd->dev, "failed to get source configuration\n");
> +			return ret;
> +		}
> +	}
> +	if (!pad) {
> +		/* Mirror the input side on the output side */
> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> +		    cfg->type == V4L2_MBUS_BT656)
> +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
> +	}
> +
> +	return 0;
> +}

I am not certain this op is needed at all. In the current kernel this op is only
used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

> +
> +static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct v4l2_subdev *upstream_sd;
> +	struct media_pad *pad;
> +
> +	if (vidsw->active == -1) {
> +		dev_err(sd->dev, "Can not start streaming on inactive mux\n");
> +		return -EINVAL;
> +	}
> +
> +	pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
> +	if (!pad) {
> +		dev_err(sd->dev, "Failed to find remote source pad\n");
> +		return -ENOLINK;
> +	}
> +
> +	if (!is_media_entity_v4l2_subdev(pad->entity)) {
> +		dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
> +		return -ENODEV;
> +	}
> +
> +	upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
> +
> +	return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
> +}
> +
> +static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_frame_interval *fi)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	fi->interval = vidsw->timeperframe;
> +
> +	return 0;
> +}
> +
> +static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_frame_interval *fi)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	vidsw->timeperframe = fi->interval;
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
> +	.g_mbus_config = vidsw_g_mbus_config,
> +	.s_stream = vidsw_s_stream,
> +	.g_frame_interval = vidsw_g_frame_interval,
> +	.s_frame_interval = vidsw_s_frame_interval,
> +};
> +
> +static struct v4l2_mbus_framefmt *
> +__vidsw_get_pad_format(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       unsigned int pad, u32 which)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +
> +	switch (which) {
> +	case V4L2_SUBDEV_FORMAT_TRY:
> +		return v4l2_subdev_get_try_format(sd, cfg, pad);
> +	case V4L2_SUBDEV_FORMAT_ACTIVE:
> +		return &vidsw->format_mbus[pad];
> +	default:
> +		return NULL;
> +	}
> +}
> +
> +static int vidsw_get_format(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_pad_config *cfg,
> +			    struct v4l2_subdev_format *sdformat)
> +{
> +	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
> +						   sdformat->which);
> +	return 0;
> +}
> +
> +static int vidsw_set_format(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_pad_config *cfg,
> +			    struct v4l2_subdev_format *sdformat)
> +{
> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> +	struct v4l2_mbus_framefmt *mbusformat;
> +
> +	if (sdformat->pad >= vidsw->num_pads)
> +		return -EINVAL;
> +
> +	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
> +					    sdformat->which);
> +	if (!mbusformat)
> +		return -EINVAL;
> +
> +	/* Output pad mirrors active input pad, no limitations on input pads */
> +	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
> +		sdformat->format = vidsw->format_mbus[vidsw->active];
> +
> +	*mbusformat = sdformat->format;
> +
> +	return 0;
> +}
> +
> +static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
> +	.get_fmt = vidsw_get_format,
> +	.set_fmt = vidsw_set_format,
> +};
> +
> +static struct v4l2_subdev_ops vidsw_subdev_ops = {
> +	.pad = &vidsw_pad_ops,
> +	.video = &vidsw_subdev_video_ops,
> +};
> +
> +static int of_get_reg_field(struct device_node *node, struct reg_field *field)
> +{
> +	u32 bit_mask;
> +	int ret;
> +
> +	ret = of_property_read_u32(node, "reg", &field->reg);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
> +	if (ret < 0)
> +		return ret;
> +
> +	field->msb = field->lsb + fls(bit_mask) - 1;
> +
> +	return 0;
> +}
> +
> +static int vidsw_probe(struct platform_device *pdev)
> +{
> +	struct device_node *np = pdev->dev.of_node;
> +	struct of_endpoint endpoint;
> +	struct device_node *ep;
> +	struct reg_field field;
> +	struct vidsw *vidsw;
> +	struct regmap *map;
> +	unsigned int num_pads;
> +	int ret;
> +
> +	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
> +	if (!vidsw)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, vidsw);
> +
> +	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
> +	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
> +			np->name);
> +	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	vidsw->subdev.dev = &pdev->dev;
> +
> +	/*
> +	 * The largest numbered port is the output port. It determines
> +	 * total number of pads
> +	 */
> +	num_pads = 0;
> +	for_each_endpoint_of_node(np, ep) {
> +		of_graph_parse_endpoint(ep, &endpoint);
> +		num_pads = max(num_pads, endpoint.port + 1);
> +	}
> +
> +	if (num_pads < 2) {
> +		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
> +		return -EINVAL;
> +	}
> +
> +	ret = of_get_reg_field(np, &field);
> +	if (ret == 0) {
> +		map = syscon_node_to_regmap(np->parent);
> +		if (!map) {
> +			dev_err(&pdev->dev, "Failed to get syscon register map\n");
> +			return PTR_ERR(map);
> +		}
> +
> +		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
> +		if (IS_ERR(vidsw->field)) {
> +			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
> +			return PTR_ERR(vidsw->field);
> +		}
> +
> +		regmap_field_read(vidsw->field, &vidsw->active);
> +	} else {
> +		if (num_pads > 3) {
> +			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
> +			return -EINVAL;
> +		}
> +
> +		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
> +		if (IS_ERR(vidsw->gpio)) {
> +			dev_warn(&pdev->dev,
> +				 "could not request control gpio: %d\n", ret);
> +			vidsw->gpio = NULL;
> +		}
> +
> +		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
> +	}
> +
> +	vidsw->num_pads = num_pads;
> +	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
> +			GFP_KERNEL);
> +	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
> +			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
> +	vidsw->endpoint = devm_kzalloc(&pdev->dev,
> +			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
> +
> +	ret = vidsw_async_init(vidsw, np);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int vidsw_remove(struct platform_device *pdev)
> +{
> +	struct vidsw *vidsw = platform_get_drvdata(pdev);
> +	struct v4l2_subdev *sd = &vidsw->subdev;
> +
> +	v4l2_async_unregister_subdev(sd);
> +	media_entity_cleanup(&sd->entity);
> +	v4l2_device_unregister_subdev(sd);
> +
> +	return 0;
> +}
> +
> +static const struct of_device_id vidsw_dt_ids[] = {
> +	{ .compatible = "video-multiplexer", },
> +	{ /* sentinel */ }
> +};
> +
> +static struct platform_driver vidsw_driver = {
> +	.probe		= vidsw_probe,
> +	.remove		= vidsw_remove,
> +	.driver		= {
> +		.of_match_table = vidsw_dt_ids,
> +		.name = "video-multiplexer",
> +	},
> +};
> +
> +module_platform_driver(vidsw_driver);
> +
> +MODULE_DESCRIPTION("video stream multiplexer");
> +MODULE_AUTHOR("Sascha Hauer, Pengutronix");
> +MODULE_AUTHOR("Philipp Zabel, Pengutronix");
> +MODULE_LICENSE("GPL");
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-20 14:29     ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:29 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This is a set of three media entity subdevice drivers for the i.MX
> Image Converter. The i.MX IC module contains three independent
> "tasks":
> 
> - Pre-processing Encode task: video frames are routed directly from
>   the CSI and can be scaled, color-space converted, and rotated.
>   Scaled output is limited to 1024x1024 resolution. Output frames
>   are routed to the camera interface entities (camif).
> 
> - Pre-processing Viewfinder task: this task can perform the same
>   conversions as the pre-process encode task, but in addition can
>   be used for hardware motion compensated deinterlacing. Frames can
>   come either directly from the CSI or from the SMFC entities (memory
>   buffers via IDMAC channels). Scaled output is limited to 1024x1024
>   resolution. Output frames can be routed to various sinks including
>   the post-processing task entities.
> 
> - Post-processing task: same conversions as pre-process encode. However
>   this entity sends frames to the i.MX IPU image converter which supports
>   image tiling, which allows scaled output up to 4096x4096 resolution.
>   Output frames can be routed to the camera interfaces.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile        |    2 +
>  drivers/staging/media/imx/imx-ic-common.c |  109 +++
>  drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
>  drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
>  drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
>  drivers/staging/media/imx/imx-ic.h        |   38 +
>  6 files changed, 2997 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-ic-common.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
>  create mode 100644 drivers/staging/media/imx/imx-ic.h
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index 3559d7b..d2a962c 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -1,8 +1,10 @@
>  imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
>  	imx-media-of.o
> +imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
>  
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
> +obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
> new file mode 100644
> index 0000000..45706ca
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-common.c
> @@ -0,0 +1,109 @@
> +/*
> + * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
> +	[IC_TASK_ENCODER]        = &imx_ic_prpenc_ops,
> +	[IC_TASK_VIEWFINDER]     = &imx_ic_prpvf_ops,
> +	[IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
> +};
> +
> +static int imx_ic_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct imx_ic_priv *priv;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, &priv->sd);
> +	priv->dev = &pdev->dev;
> +
> +	/* get our ipu_id, grp_id and IC task id */
> +	pdata = priv->dev->platform_data;
> +	priv->ipu_id = pdata->ipu_id;
> +	switch (pdata->grp_id) {
> +	case IMX_MEDIA_GRP_ID_IC_PRPENC:
> +		priv->task_id = IC_TASK_ENCODER;
> +		break;
> +	case IMX_MEDIA_GRP_ID_IC_PRPVF:
> +		priv->task_id = IC_TASK_VIEWFINDER;
> +		break;
> +	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
> +		priv->task_id = IC_TASK_POST_PROCESSOR;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
> +	v4l2_set_subdevdata(&priv->sd, priv);
> +	priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
> +	priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
> +	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> +	priv->sd.dev = &pdev->dev;
> +	priv->sd.owner = THIS_MODULE;
> +	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +	priv->sd.grp_id = pdata->grp_id;
> +	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
> +
> +	ret = ic_ops[priv->task_id]->init(priv);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l2_async_register_subdev(&priv->sd);
> +	if (ret)
> +		ic_ops[priv->task_id]->remove(priv);
> +
> +	return ret;
> +}
> +
> +static int imx_ic_remove(struct platform_device *pdev)
> +{
> +	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
> +	struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
> +
> +	ic_ops[priv->task_id]->remove(priv);
> +
> +	v4l2_async_unregister_subdev(&priv->sd);
> +	media_entity_cleanup(&priv->sd.entity);
> +	v4l2_device_unregister_subdev(sd);
> +
> +	return 0;
> +}
> +
> +static const struct platform_device_id imx_ic_ids[] = {
> +	{ .name = "imx-ipuv3-ic" },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(platform, imx_ic_ids);
> +
> +static struct platform_driver imx_ic_driver = {
> +	.probe = imx_ic_probe,
> +	.remove = imx_ic_remove,
> +	.id_table = imx_ic_ids,
> +	.driver = {
> +		.name = "imx-ipuv3-ic",
> +	},
> +};
> +module_platform_driver(imx_ic_driver);
> +
> +MODULE_DESCRIPTION("i.MX IC subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> +MODULE_ALIAS("platform:imx-ipuv3-ic");
> diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
> new file mode 100644
> index 0000000..1f75616
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-pp.c
> @@ -0,0 +1,636 @@
> +/*
> + * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-image-convert.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +#define PP_NUM_PADS 2
> +
> +struct pp_priv {
> +	struct imx_media_dev *md;
> +	struct imx_ic_priv *ic_priv;
> +	int pp_id;
> +
> +	struct ipu_soc *ipu;
> +	struct ipu_image_convert_ctx *ic_ctx;
> +
> +	struct media_pad pad[PP_NUM_PADS];
> +	int input_pad;
> +	int output_pad;
> +
> +	/* our dma buffer sink ring */
> +	struct imx_media_dma_buf_ring *in_ring;
> +	/* the dma buffer ring we send to sink */
> +	struct imx_media_dma_buf_ring *out_ring;
> +	struct ipu_image_convert_run *out_run;
> +
> +	struct imx_media_dma_buf *inbuf; /* last input buffer */
> +
> +	bool stream_on;    /* streaming is on */
> +	bool stop;         /* streaming is stopping */
> +	spinlock_t irqlock;
> +
> +	struct v4l2_subdev *src_sd;
> +	struct v4l2_subdev *sink_sd;
> +
> +	struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[PP_NUM_PADS];
> +
> +	/* motion select control */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	int  rotation; /* degrees */
> +	bool hflip;
> +	bool vflip;
> +
> +	/* derived from rotation, hflip, vflip controls */
> +	enum ipu_rotate_mode rot_mode;
> +};
> +
> +static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +
> +	return ic_priv->task_priv;
> +}
> +
> +static void pp_convert_complete(struct ipu_image_convert_run *run,
> +				void *data)
> +{
> +	struct pp_priv *priv = data;
> +	struct imx_media_dma_buf *done;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +
> +	done = imx_media_dma_buf_get_active(priv->out_ring);
> +	/* give the completed buffer to the sink */
> +	if (!WARN_ON(!done))
> +		imx_media_dma_buf_done(done, run->status ?
> +				       IMX_MEDIA_BUF_STATUS_ERROR :
> +				       IMX_MEDIA_BUF_STATUS_DONE);
> +
> +	/* we're done with the inbuf, queue it back */
> +	imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
> +
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +}
> +
> +static void pp_queue_conversion(struct pp_priv *priv,
> +				struct imx_media_dma_buf *inbuf)
> +{
> +	struct ipu_image_convert_run *run;
> +	struct imx_media_dma_buf *outbuf;
> +
> +	/* get next queued buffer and make it active */
> +	outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(outbuf);
> +	priv->inbuf = inbuf;
> +
> +	run = &priv->out_run[outbuf->index];
> +	run->ctx = priv->ic_ctx;
> +	run->in_phys = inbuf->phys;
> +	run->out_phys = outbuf->phys;
> +	ipu_image_convert_queue(run);
> +}
> +
> +static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct imx_media_dma_buf_ring **ring;
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +
> +	switch (cmd) {
> +	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
> +		/* src asks for a buffer ring */
> +		if (!priv->in_ring)
> +			return -EINVAL;
> +		ring = (struct imx_media_dma_buf_ring **)arg;
> +		*ring = priv->in_ring;
> +		break;
> +	case IMX_MEDIA_NEW_DMA_BUF:
> +		/* src hands us a new buffer */
> +		spin_lock_irqsave(&priv->irqlock, flags);
> +		if (!priv->stop &&
> +		    !imx_media_dma_buf_get_active(priv->out_ring)) {
> +			buf = imx_media_dma_buf_dequeue(priv->in_ring);
> +			if (buf)
> +				pp_queue_conversion(priv, buf);
> +		}
> +		spin_unlock_irqrestore(&priv->irqlock, flags);
> +		break;
> +	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
> +		/* src indicates sink buffer ring can be freed */
> +		if (!priv->in_ring)
> +			return 0;
> +		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_start(struct pp_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct ipu_image image_in, image_out;
> +	const struct imx_media_pixfmt *incc;
> +	struct v4l2_mbus_framefmt *infmt;
> +	int i, in_size, ret;
> +
> +	/* ask the sink for the buffer ring */
> +	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
> +			       &priv->out_ring);
> +	if (ret)
> +		return ret;
> +
> +	imx_media_mbus_fmt_to_ipu_image(&image_in,
> +					&priv->format_mbus[priv->input_pad]);
> +	imx_media_mbus_fmt_to_ipu_image(&image_out,
> +					&priv->format_mbus[priv->output_pad]);
> +
> +	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
> +	priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
> +						 IC_TASK_POST_PROCESSOR,
> +						 &image_in, &image_out,
> +						 priv->rot_mode,
> +						 pp_convert_complete, priv);
> +	if (IS_ERR(priv->ic_ctx))
> +		return PTR_ERR(priv->ic_ctx);
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	incc = priv->cc[priv->input_pad];
> +	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
> +
> +	if (priv->in_ring) {
> +		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
> +			  __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +	}
> +
> +	priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
> +						     &priv->src_sd->entity,
> +						     &ic_priv->sd.entity,
> +						     in_size,
> +						     IMX_MEDIA_MIN_RING_BUFS,
> +						     true);
> +	if (IS_ERR(priv->in_ring)) {
> +		v4l2_err(&ic_priv->sd,
> +			 "failed to alloc dma-buf ring\n");
> +		ret = PTR_ERR(priv->in_ring);
> +		priv->in_ring = NULL;
> +		goto out_unprep;
> +	}
> +
> +	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
> +		imx_media_dma_buf_queue(priv->in_ring, i);
> +
> +	priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
> +				sizeof(*priv->out_run), GFP_KERNEL);
> +	if (!priv->out_run) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
> +		ret = -ENOMEM;
> +		goto out_free_ring;
> +	}
> +
> +	priv->stop = false;
> +
> +	return 0;
> +
> +out_free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +out_unprep:
> +	ipu_image_convert_unprepare(priv->ic_ctx);
> +	return ret;
> +}
> +
> +static void pp_stop(struct pp_priv *priv)
> +{
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +	priv->stop = true;
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +
> +	ipu_image_convert_unprepare(priv->ic_ctx);
> +	kfree(priv->out_run);
> +
> +	priv->out_ring = NULL;
> +
> +	/* inform sink that its sink buffer ring can now be freed */
> +	v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
> +}
> +
> +static int pp_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	int ret = 0;
> +
> +	if (!priv->src_sd || !priv->sink_sd)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	if (enable && !priv->stream_on)
> +		ret = pp_start(priv);
> +	else if (!enable && priv->stream_on)
> +		pp_stop(priv);
> +
> +	if (!ret)
> +		priv->stream_on = enable;
> +	return ret;
> +}
> +
> +static int pp_enum_mbus_code(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_pad_config *cfg,
> +			     struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	const struct imx_media_pixfmt *cc;
> +	u32 fourcc;
> +	int ret;
> +
> +	if (code->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	ret = ipu_image_convert_enum_format(code->index, &fourcc);
> +	if (ret)
> +		return ret;
> +
> +	/* convert returned fourcc to mbus code */
> +	cc = imx_media_find_format(fourcc, 0, true, true);
> +	if (WARN_ON(!cc))
> +		return -EINVAL;
> +
> +	code->code = cc->codes[0];
> +	return 0;
> +}
> +
> +static int pp_get_fmt(struct v4l2_subdev *sd,
> +		      struct v4l2_subdev_pad_config *cfg,
> +		      struct v4l2_subdev_format *sdformat)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +
> +	if (sdformat->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int pp_set_fmt(struct v4l2_subdev *sd,
> +		      struct v4l2_subdev_pad_config *cfg,
> +		      struct v4l2_subdev_format *sdformat)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *cc;
> +	struct ipu_image test_in, test_out;
> +	u32 code;
> +
> +	if (sdformat->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	cc = imx_media_find_format(0, sdformat->format.code, true, true);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, true);
> +		cc = imx_media_find_format(0, code, true, true);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
> +		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
> +		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
> +	} else {
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
> +		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
> +		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		if (sdformat->pad == priv->output_pad) {
> +			*outfmt = sdformat->format;
> +			imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
> +		} else {
> +			*infmt = sdformat->format;
> +			imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
> +		}
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_link_setup(struct media_entity *entity,
> +			 const struct media_pad *local,
> +			 const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct pp_priv *priv = ic_priv->task_priv;
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->sink_sd)
> +				return -EBUSY;
> +			priv->sink_sd = remote_sd;
> +		} else {
> +			priv->sink_sd = NULL;
> +		}
> +	} else {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->src_sd)
> +				return -EBUSY;
> +			priv->src_sd = remote_sd;
> +		} else {
> +			priv->src_sd = NULL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct pp_priv *priv = container_of(ctrl->handler,
> +					       struct pp_priv, ctrl_hdlr);
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	enum ipu_rotate_mode rot_mode;
> +	bool hflip, vflip;
> +	int rotation, ret;
> +
> +	rotation = priv->rotation;
> +	hflip = priv->hflip;
> +	vflip = priv->vflip;
> +
> +	switch (ctrl->id) {
> +	case V4L2_CID_HFLIP:
> +		hflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_VFLIP:
> +		vflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_ROTATE:
> +		rotation = ctrl->val;
> +		break;
> +	default:
> +		v4l2_err(&ic_priv->sd, "Invalid control\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
> +	if (ret)
> +		return ret;
> +
> +	if (rot_mode != priv->rot_mode) {
> +		struct v4l2_mbus_framefmt *infmt, *outfmt;
> +		struct ipu_image test_in, test_out;
> +
> +		/* can't change rotation mid-streaming */
> +		if (priv->stream_on)
> +			return -EBUSY;
> +
> +		/*
> +		 * make sure this rotation will work with current input/output
> +		 * formats before setting
> +		 */
> +		infmt = &priv->format_mbus[priv->input_pad];
> +		outfmt = &priv->format_mbus[priv->output_pad];
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
> +
> +		ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);
> +		if (ret)
> +			return ret;
> +
> +		priv->rot_mode = rot_mode;
> +		priv->rotation = rotation;
> +		priv->hflip = hflip;
> +		priv->vflip = vflip;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ctrl_ops pp_ctrl_ops = {
> +	.s_ctrl = pp_s_ctrl,
> +};
> +
> +static const struct v4l2_ctrl_config pp_std_ctrl[] = {
> +	{
> +		.id = V4L2_CID_HFLIP,
> +		.name = "Horizontal Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_VFLIP,
> +		.name = "Vertical Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_ROTATE,
> +		.name = "Rotation",
> +		.type = V4L2_CTRL_TYPE_INTEGER,
> +		.def =   0,
> +		.min =   0,
> +		.max = 270,
> +		.step = 90,
> +	},
> +};
> +
> +#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
> +
> +static int pp_init_controls(struct pp_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
> +	const struct v4l2_ctrl_config *c;
> +	int i, ret;
> +
> +	v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
> +
> +	for (i = 0; i < PP_NUM_CONTROLS; i++) {
> +		c = &pp_std_ctrl[i];
> +		v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
> +				  c->id, c->min, c->max, c->step, c->def);
> +	}
> +
> +	ic_priv->sd.ctrl_handler = hdlr;
> +
> +	if (hdlr->error) {
> +		ret = hdlr->error;
> +		v4l2_ctrl_handler_free(hdlr);
> +		return ret;
> +	}
> +
> +	v4l2_ctrl_handler_setup(hdlr);
> +
> +	return 0;
> +}
> +
> +/*
> + * retrieve our pads parsed from the OF graph by the media device
> + */
> +static int pp_registered(struct v4l2_subdev *sd)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct imx_media_subdev *imxsd;
> +	struct imx_media_pad *pad;
> +	int i, ret;
> +
> +	/* get media device */
> +	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
> +
> +	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
> +	if (IS_ERR(imxsd))
> +		return PTR_ERR(imxsd);
> +
> +	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
> +		return -EINVAL;
> +
> +	for (i = 0; i < PP_NUM_PADS; i++) {
> +		pad = &imxsd->pad[i];
> +		priv->pad[i] = pad->pad;
> +		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
> +			priv->input_pad = i;
> +		else
> +			priv->output_pad = i;
> +
> +		/* set a default mbus format  */
> +		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
> +					      640, 480, 0, V4L2_FIELD_NONE,
> +					      &priv->cc[i]);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	ret = pp_init_controls(priv);
> +	if (ret)
> +		return ret;
> +
> +	ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +	return ret;
> +}
> +
> +static struct v4l2_subdev_pad_ops pp_pad_ops = {
> +	.enum_mbus_code = pp_enum_mbus_code,
> +	.get_fmt = pp_get_fmt,
> +	.set_fmt = pp_set_fmt,
> +};
> +
> +static struct v4l2_subdev_video_ops pp_video_ops = {
> +	.s_stream = pp_s_stream,
> +};
> +
> +static struct v4l2_subdev_core_ops pp_core_ops = {
> +	.ioctl = pp_ioctl,
> +};
> +
> +static struct media_entity_operations pp_entity_ops = {
> +	.link_setup = pp_link_setup,
> +	.link_validate = v4l2_subdev_link_validate,
> +};
> +
> +static struct v4l2_subdev_ops pp_subdev_ops = {
> +	.video = &pp_video_ops,
> +	.pad = &pp_pad_ops,
> +	.core = &pp_core_ops,
> +};
> +
> +static struct v4l2_subdev_internal_ops pp_internal_ops = {
> +	.registered = pp_registered,
> +};
> +
> +static int pp_init(struct imx_ic_priv *ic_priv)
> +{
> +	struct pp_priv *priv;
> +
> +	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	ic_priv->task_priv = priv;
> +	priv->ic_priv = ic_priv;
> +	spin_lock_init(&priv->irqlock);
> +
> +	/* get our PP id */
> +	priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
> +
> +	return 0;
> +}
> +
> +static void pp_remove(struct imx_ic_priv *ic_priv)
> +{
> +	struct pp_priv *priv = ic_priv->task_priv;
> +
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +}
> +
> +struct imx_ic_ops imx_ic_pp_ops = {
> +	.subdev_ops = &pp_subdev_ops,
> +	.internal_ops = &pp_internal_ops,
> +	.entity_ops = &pp_entity_ops,
> +	.init = pp_init,
> +	.remove = pp_remove,
> +};
> diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
> new file mode 100644
> index 0000000..3d85a82
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-prpenc.c
> @@ -0,0 +1,1033 @@
> +/*
> + * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
> + *
> + * This subdevice handles capture of video frames from the CSI, which
> + * are routed directly to the Image Converter preprocess encode task,
> + * for resizing, colorspace conversion, and rotation.
> + *
> + * Copyright (c) 2012-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +#define PRPENC_NUM_PADS 2
> +
> +#define MAX_W_IC   1024
> +#define MAX_H_IC   1024
> +#define MAX_W_SINK 4096
> +#define MAX_H_SINK 4096
> +
> +struct prpenc_priv {
> +	struct imx_media_dev *md;
> +	struct imx_ic_priv *ic_priv;
> +
> +	/* IPU units we require */
> +	struct ipu_soc *ipu;
> +	struct ipu_ic *ic_enc;
> +
> +	struct media_pad pad[PRPENC_NUM_PADS];
> +	int input_pad;
> +	int output_pad;
> +
> +	struct ipuv3_channel *enc_ch;
> +	struct ipuv3_channel *enc_rot_in_ch;
> +	struct ipuv3_channel *enc_rot_out_ch;
> +
> +	/* the dma buffer ring to send to sink */
> +	struct imx_media_dma_buf_ring *out_ring;
> +	struct imx_media_dma_buf *next;
> +
> +	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
> +
> +	struct v4l2_subdev *src_sd;
> +	struct v4l2_subdev *sink_sd;
> +
> +	/* the CSI id at link validate */
> +	int csi_id;
> +
> +	/* the attached sensor at stream on */
> +	struct imx_media_subdev *sensor;
> +
> +	struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
> +
> +	struct imx_media_dma_buf rot_buf[2];
> +
> +	/* controls */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	int  rotation; /* degrees */
> +	bool hflip;
> +	bool vflip;
> +
> +	/* derived from rotation, hflip, vflip controls */
> +	enum ipu_rotate_mode rot_mode;
> +
> +	spinlock_t irqlock;
> +
> +	struct timer_list eof_timeout_timer;
> +	int eof_irq;
> +	int nfb4eof_irq;
> +
> +	bool stream_on; /* streaming is on */
> +	bool last_eof;  /* waiting for last EOF at stream off */
> +	struct completion last_eof_comp;
> +};
> +
> +static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +
> +	return ic_priv->task_priv;
> +}
> +
> +static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
> +{
> +	if (!IS_ERR_OR_NULL(priv->ic_enc))
> +		ipu_ic_put(priv->ic_enc);
> +	priv->ic_enc = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_ch))
> +		ipu_idmac_put(priv->enc_ch);
> +	priv->enc_ch = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
> +		ipu_idmac_put(priv->enc_rot_in_ch);
> +	priv->enc_rot_in_ch = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
> +		ipu_idmac_put(priv->enc_rot_out_ch);
> +	priv->enc_rot_out_ch = NULL;
> +}
> +
> +static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	int ret;
> +
> +	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
> +
> +	priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
> +	if (IS_ERR(priv->ic_enc)) {
> +		v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
> +		ret = PTR_ERR(priv->ic_enc);
> +		goto out;
> +	}
> +
> +	priv->enc_ch = ipu_idmac_get(priv->ipu,
> +				     IPUV3_CHANNEL_IC_PRP_ENC_MEM);
> +	if (IS_ERR(priv->enc_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_IC_PRP_ENC_MEM);
> +		ret = PTR_ERR(priv->enc_ch);
> +		goto out;
> +	}
> +
> +	priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
> +					    IPUV3_CHANNEL_MEM_ROT_ENC);
> +	if (IS_ERR(priv->enc_rot_in_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_MEM_ROT_ENC);
> +		ret = PTR_ERR(priv->enc_rot_in_ch);
> +		goto out;
> +	}
> +
> +	priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
> +					     IPUV3_CHANNEL_ROT_ENC_MEM);
> +	if (IS_ERR(priv->enc_rot_out_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_ROT_ENC_MEM);
> +		ret = PTR_ERR(priv->enc_rot_out_ch);
> +		goto out;
> +	}
> +
> +	return 0;
> +out:
> +	prpenc_put_ipu_resources(priv);
> +	return ret;
> +}
> +
> +static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
> +{
> +	struct prpenc_priv *priv = dev_id;
> +	struct imx_media_dma_buf *done, *next;
> +	struct ipuv3_channel *channel;
> +
> +	spin_lock(&priv->irqlock);
> +
> +	if (priv->last_eof) {
> +		complete(&priv->last_eof_comp);
> +		priv->last_eof = false;
> +		goto unlock;
> +	}
> +
> +	/* inform CSI of this EOF so it can monitor frame intervals */
> +	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
> +			 0, NULL);
> +
> +	channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
> +		priv->enc_rot_out_ch : priv->enc_ch;
> +
> +	done = imx_media_dma_buf_get_active(priv->out_ring);
> +	/* give the completed buffer to the sink  */
> +	if (!WARN_ON(!done))
> +		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
> +
> +	/* priv->next buffer is now the active one */
> +	imx_media_dma_buf_set_active(priv->next);
> +
> +	/* bump the EOF timeout timer */
> +	mod_timer(&priv->eof_timeout_timer,
> +		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +
> +	if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
> +		ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);
> +
> +	/* get next queued buffer */
> +	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +
> +	ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
> +	ipu_idmac_select_buffer(channel, priv->ipu_buf_num);
> +
> +	/* toggle IPU double-buffer index */
> +	priv->ipu_buf_num ^= 1;
> +	priv->next = next;
> +
> +unlock:
> +	spin_unlock(&priv->irqlock);
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
> +{
> +	struct prpenc_priv *priv = dev_id;
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	static const struct v4l2_event ev = {
> +		.type = V4L2_EVENT_IMX_NFB4EOF,
> +	};
> +
> +	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
> +
> +	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
> +
> +	return IRQ_HANDLED;
> +}
> +
> +/*
> + * EOF timeout timer function.
> + */
> +static void prpenc_eof_timeout(unsigned long data)
> +{
> +	struct prpenc_priv *priv = (struct prpenc_priv *)data;
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	static const struct v4l2_event ev = {
> +		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
> +	};
> +
> +	v4l2_err(&ic_priv->sd, "EOF timeout\n");
> +
> +	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
> +}
> +
> +static void prpenc_setup_channel(struct prpenc_priv *priv,
> +				 struct ipuv3_channel *channel,
> +				 enum ipu_rotate_mode rot_mode,
> +				 dma_addr_t addr0, dma_addr_t addr1,
> +				 bool rot_swap_width_height)
> +{
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	unsigned int burst_size;
> +	struct ipu_image image;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (rot_swap_width_height)
> +		swap(outfmt->width, outfmt->height);
> +
> +	ipu_cpmem_zero(channel);
> +
> +	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
> +
> +	image.phys0 = addr0;
> +	image.phys1 = addr1;
> +	ipu_cpmem_set_image(channel, &image);
> +
> +	if (channel == priv->enc_rot_in_ch ||
> +	    channel == priv->enc_rot_out_ch) {
> +		burst_size = 8;
> +		ipu_cpmem_set_block_mode(channel);
> +	} else {
> +		burst_size = (outfmt->width & 0xf) ? 8 : 16;
> +	}
> +
> +	ipu_cpmem_set_burstsize(channel, burst_size);
> +
> +	if (rot_mode)
> +		ipu_cpmem_set_rotation(channel, rot_mode);
> +
> +	if (outfmt->field == V4L2_FIELD_NONE &&
> +	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
> +	     infmt->field == V4L2_FIELD_ALTERNATE) &&
> +	    channel == priv->enc_ch)
> +		ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
> +
> +	ipu_ic_task_idma_init(priv->ic_enc, channel,
> +			      outfmt->width, outfmt->height,
> +			      burst_size, rot_mode);
> +	ipu_cpmem_set_axi_id(channel, 1);
> +
> +	ipu_idmac_set_double_buffer(channel, true);
> +
> +	if (rot_swap_width_height)
> +		swap(outfmt->width, outfmt->height);
> +}
> +
> +static int prpenc_setup_rotation(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *outcc, *incc;
> +	struct imx_media_dma_buf *buf0, *buf1;
> +	int out_size, ret;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	incc = priv->cc[priv->input_pad];
> +	outcc = priv->cc[priv->output_pad];
> +
> +	out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
> +
> +	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
> +		return ret;
> +	}
> +	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
> +		goto free_rot0;
> +	}
> +
> +	ret = ipu_ic_task_init(priv->ic_enc,
> +			       infmt->width, infmt->height,
> +			       outfmt->height, outfmt->width,
> +			       incc->cs, outcc->cs);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
> +		goto free_rot1;
> +	}
> +
> +	/* init the IC ENC-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_ch,
> +			     IPU_ROTATE_NONE,
> +			     priv->rot_buf[0].phys,
> +			     priv->rot_buf[1].phys,
> +			     true);
> +
> +	/* init the MEM-->IC ENC ROT IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_rot_in_ch,
> +			     priv->rot_mode,
> +			     priv->rot_buf[0].phys,
> +			     priv->rot_buf[1].phys,
> +			     true);
> +
> +	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(buf0);
> +	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	priv->next = buf1;
> +
> +	/* init the destination IC ENC ROT-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_rot_out_ch,
> +			     IPU_ROTATE_NONE,
> +			     buf0->phys, buf1->phys,
> +			     false);
> +
> +	/* now link IC ENC-->MEM to MEM-->IC ENC ROT */
> +	ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
> +
> +	/* enable the IC */
> +	ipu_ic_enable(priv->ic_enc);
> +
> +	/* set buffers ready */
> +	ipu_idmac_select_buffer(priv->enc_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_ch, 1);
> +	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);
> +
> +	/* enable the channels */
> +	ipu_idmac_enable_channel(priv->enc_ch);
> +	ipu_idmac_enable_channel(priv->enc_rot_in_ch);
> +	ipu_idmac_enable_channel(priv->enc_rot_out_ch);
> +
> +	/* and finally enable the IC PRPENC task */
> +	ipu_ic_task_enable(priv->ic_enc);
> +
> +	return 0;
> +
> +free_rot1:
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
> +free_rot0:
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
> +	return ret;
> +}
> +
> +static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
> +{
> +	ipu_ic_task_disable(priv->ic_enc);
> +
> +	ipu_idmac_disable_channel(priv->enc_ch);
> +	ipu_idmac_disable_channel(priv->enc_rot_in_ch);
> +	ipu_idmac_disable_channel(priv->enc_rot_out_ch);
> +
> +	ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
> +
> +	ipu_ic_disable(priv->ic_enc);
> +
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
> +}
> +
> +static int prpenc_setup_norotation(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *outcc, *incc;
> +	struct imx_media_dma_buf *buf0, *buf1;
> +	int ret;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	incc = priv->cc[priv->input_pad];
> +	outcc = priv->cc[priv->output_pad];
> +
> +	ret = ipu_ic_task_init(priv->ic_enc,
> +			       infmt->width, infmt->height,
> +			       outfmt->width, outfmt->height,
> +			       incc->cs, outcc->cs);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
> +		return ret;
> +	}
> +
> +	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(buf0);
> +	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	priv->next = buf1;
> +
> +	/* init the IC PRP-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
> +			     buf0->phys, buf1->phys,
> +			     false);
> +
> +	ipu_cpmem_dump(priv->enc_ch);
> +	ipu_ic_dump(priv->ic_enc);
> +	ipu_dump(priv->ipu);
> +
> +	ipu_ic_enable(priv->ic_enc);
> +
> +	/* set buffers ready */
> +	ipu_idmac_select_buffer(priv->enc_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_ch, 1);
> +
> +	/* enable the channels */
> +	ipu_idmac_enable_channel(priv->enc_ch);
> +
> +	/* enable the IC ENCODE task */
> +	ipu_ic_task_enable(priv->ic_enc);
> +
> +	return 0;
> +}
> +
> +static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
> +{
> +	ipu_ic_task_disable(priv->ic_enc);
> +	ipu_idmac_disable_channel(priv->enc_ch);
> +	ipu_ic_disable(priv->ic_enc);
> +}
> +
> +static int prpenc_start(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	int ret;
> +
> +	if (!priv->sensor) {
> +		v4l2_err(&ic_priv->sd, "no sensor attached\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = prpenc_get_ipu_resources(priv);
> +	if (ret)
> +		return ret;
> +
> +	/* set IC to receive from CSI */
> +	ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
> +
> +	/* ask the sink for the buffer ring */
> +	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
> +			       &priv->out_ring);
> +	if (ret)
> +		goto out_put_ipu;
> +
> +	priv->ipu_buf_num = 0;
> +
> +	/* init EOF completion waitq */
> +	init_completion(&priv->last_eof_comp);
> +	priv->last_eof = false;
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		ret = prpenc_setup_rotation(priv);
> +	else
> +		ret = prpenc_setup_norotation(priv);
> +	if (ret)
> +		goto out_put_ipu;
> +
> +	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
> +						  priv->enc_ch,
> +						  IPU_IRQ_NFB4EOF);
> +	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
> +			       prpenc_nfb4eof_interrupt, 0,
> +			       "imx-ic-prpenc-nfb4eof", priv);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd,
> +			 "Error registering NFB4EOF irq: %d\n", ret);
> +		goto out_unsetup;
> +	}
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		priv->eof_irq = ipu_idmac_channel_irq(
> +			priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
> +	else
> +		priv->eof_irq = ipu_idmac_channel_irq(
> +			priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
> +
> +	ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
> +			       prpenc_eof_interrupt, 0,
> +			       "imx-ic-prpenc-eof", priv);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd,
> +			 "Error registering eof irq: %d\n", ret);
> +		goto out_free_nfb4eof_irq;
> +	}
> +
> +	/* start the EOF timeout timer */
> +	mod_timer(&priv->eof_timeout_timer,
> +		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +
> +	return 0;
> +
> +out_free_nfb4eof_irq:
> +	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
> +out_unsetup:
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		prpenc_unsetup_rotation(priv);
> +	else
> +		prpenc_unsetup_norotation(priv);
> +out_put_ipu:
> +	prpenc_put_ipu_resources(priv);
> +	return ret;
> +}
> +
> +static void prpenc_stop(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	unsigned long flags;
> +	int ret;
> +
> +	/* mark next EOF interrupt as the last before stream off */
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +	priv->last_eof = true;
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +
> +	/*
> +	 * and then wait for interrupt handler to mark completion.
> +	 */
> +	ret = wait_for_completion_timeout(
> +		&priv->last_eof_comp,
> +		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +	if (ret == 0)
> +		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
> +
> +	devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
> +	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		prpenc_unsetup_rotation(priv);
> +	else
> +		prpenc_unsetup_norotation(priv);
> +
> +	prpenc_put_ipu_resources(priv);
> +
> +	/* cancel the EOF timeout timer */
> +	del_timer_sync(&priv->eof_timeout_timer);
> +
> +	priv->out_ring = NULL;
> +
> +	/* inform sink that the buffer ring can now be freed */
> +	v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
> +}
> +
> +static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +	bool allow_planar;
> +
> +	if (code->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	allow_planar = (code->pad == priv->output_pad);
> +
> +	return imx_media_enum_format(&code->code, code->index,
> +				     true, allow_planar);
> +}
> +
> +static int prpenc_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *sdformat)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +
> +	if (sdformat->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int prpenc_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *sdformat)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *cc;
> +	bool allow_planar;
> +	u32 code;
> +
> +	if (sdformat->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	allow_planar = (sdformat->pad == priv->output_pad);
> +
> +	cc = imx_media_find_format(0, sdformat->format.code,
> +				   true, allow_planar);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, false);
> +		cc = imx_media_find_format(0, code, true, false);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.width = min_t(__u32,
> +					       sdformat->format.width,
> +					       MAX_W_IC);
> +		sdformat->format.height = min_t(__u32,
> +						sdformat->format.height,
> +						MAX_H_IC);
> +
> +		if (sdformat->format.field != V4L2_FIELD_NONE)
> +			sdformat->format.field = infmt->field;
> +
> +		/* IC resizer cannot downsize more than 4:1 */
> +		if (ipu_rot_mode_is_irt(priv->rot_mode)) {
> +			sdformat->format.width = max_t(__u32,
> +						       sdformat->format.width,
> +						       infmt->height / 4);
> +			sdformat->format.height = max_t(__u32,
> +							sdformat->format.height,
> +							infmt->width / 4);
> +		} else {
> +			sdformat->format.width = max_t(__u32,
> +						       sdformat->format.width,
> +						       infmt->width / 4);
> +			sdformat->format.height = max_t(__u32,
> +							sdformat->format.height,
> +							infmt->height / 4);
> +		}
> +	} else {
> +		sdformat->format.width = min_t(__u32,
> +					       sdformat->format.width,
> +					       MAX_W_SINK);
> +		sdformat->format.height = min_t(__u32,
> +						sdformat->format.height,
> +						MAX_H_SINK);
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_link_setup(struct media_entity *entity,
> +			     const struct media_pad *local,
> +			     const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct prpenc_priv *priv = ic_priv->task_priv;
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->sink_sd)
> +				return -EBUSY;
> +			priv->sink_sd = remote_sd;
> +		} else {
> +			priv->sink_sd = NULL;
> +		}
> +
> +		return 0;
> +	}
> +
> +	/* this is sink pad */
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->src_sd)
> +			return -EBUSY;
> +		priv->src_sd = remote_sd;
> +	} else {
> +		priv->src_sd = NULL;
> +		return 0;
> +	}
> +
> +	switch (remote_sd->grp_id) {
> +	case IMX_MEDIA_GRP_ID_CSI0:
> +		priv->csi_id = 0;
> +		break;
> +	case IMX_MEDIA_GRP_ID_CSI1:
> +		priv->csi_id = 1;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_link_validate(struct v4l2_subdev *sd,
> +				struct media_link *link,
> +				struct v4l2_subdev_format *source_fmt,
> +				struct v4l2_subdev_format *sink_fmt)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct prpenc_priv *priv = ic_priv->task_priv;
> +	struct v4l2_mbus_config sensor_mbus_cfg;
> +	int ret;
> +
> +	ret = v4l2_subdev_link_validate_default(sd, link,
> +						source_fmt, sink_fmt);
> +	if (ret)
> +		return ret;
> +
> +	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
> +	if (IS_ERR(priv->sensor)) {
> +		v4l2_err(&ic_priv->sd, "no sensor attached\n");
> +		ret = PTR_ERR(priv->sensor);
> +		priv->sensor = NULL;
> +		return ret;
> +	}
> +
> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> +			       &sensor_mbus_cfg);
> +	if (ret)
> +		return ret;
> +
> +	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
> +		int vc_num = 0;
> +		/* see NOTE in imx-csi.c */
> +#if 0
> +		vc_num = imx_media_find_mipi_csi2_channel(
> +			priv->md, &ic_priv->sd.entity);
> +		if (vc_num < 0)
> +			return vc_num;
> +#endif
> +		/* only virtual channel 0 can be sent to IC */
> +		if (vc_num != 0)
> +			return -EINVAL;
> +	} else {
> +		/*
> +		 * only 8-bit pixels can be sent to IC for parallel
> +		 * busses
> +		 */
> +		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct prpenc_priv *priv = container_of(ctrl->handler,
> +					       struct prpenc_priv, ctrl_hdlr);
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	enum ipu_rotate_mode rot_mode;
> +	bool hflip, vflip;
> +	int rotation, ret;
> +
> +	rotation = priv->rotation;
> +	hflip = priv->hflip;
> +	vflip = priv->vflip;
> +
> +	switch (ctrl->id) {
> +	case V4L2_CID_HFLIP:
> +		hflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_VFLIP:
> +		vflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_ROTATE:
> +		rotation = ctrl->val;
> +		break;
> +	default:
> +		v4l2_err(&ic_priv->sd, "Invalid control\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
> +	if (ret)
> +		return ret;
> +
> +	if (rot_mode != priv->rot_mode) {
> +		/* can't change rotation mid-streaming */
> +		if (priv->stream_on)
> +			return -EBUSY;
> +
> +		priv->rot_mode = rot_mode;
> +		priv->rotation = rotation;
> +		priv->hflip = hflip;
> +		priv->vflip = vflip;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
> +	.s_ctrl = prpenc_s_ctrl,
> +};
> +
> +static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
> +	{
> +		.id = V4L2_CID_HFLIP,
> +		.name = "Horizontal Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_VFLIP,
> +		.name = "Vertical Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_ROTATE,
> +		.name = "Rotation",
> +		.type = V4L2_CTRL_TYPE_INTEGER,
> +		.def =   0,
> +		.min =   0,
> +		.max = 270,
> +		.step = 90,
> +	},
> +};

Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information
like the name and type.

If this is also done elsewhere, then it should be changed there as well.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-20 14:29     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:29 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This is a set of three media entity subdevice drivers for the i.MX
> Image Converter. The i.MX IC module contains three independent
> "tasks":
> 
> - Pre-processing Encode task: video frames are routed directly from
>   the CSI and can be scaled, color-space converted, and rotated.
>   Scaled output is limited to 1024x1024 resolution. Output frames
>   are routed to the camera interface entities (camif).
> 
> - Pre-processing Viewfinder task: this task can perform the same
>   conversions as the pre-process encode task, but in addition can
>   be used for hardware motion compensated deinterlacing. Frames can
>   come either directly from the CSI or from the SMFC entities (memory
>   buffers via IDMAC channels). Scaled output is limited to 1024x1024
>   resolution. Output frames can be routed to various sinks including
>   the post-processing task entities.
> 
> - Post-processing task: same conversions as pre-process encode. However
>   this entity sends frames to the i.MX IPU image converter which supports
>   image tiling, which allows scaled output up to 4096x4096 resolution.
>   Output frames can be routed to the camera interfaces.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile        |    2 +
>  drivers/staging/media/imx/imx-ic-common.c |  109 +++
>  drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
>  drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
>  drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
>  drivers/staging/media/imx/imx-ic.h        |   38 +
>  6 files changed, 2997 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-ic-common.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
>  create mode 100644 drivers/staging/media/imx/imx-ic.h
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index 3559d7b..d2a962c 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -1,8 +1,10 @@
>  imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
>  	imx-media-of.o
> +imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
>  
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
> +obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
> new file mode 100644
> index 0000000..45706ca
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-common.c
> @@ -0,0 +1,109 @@
> +/*
> + * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
> +	[IC_TASK_ENCODER]        = &imx_ic_prpenc_ops,
> +	[IC_TASK_VIEWFINDER]     = &imx_ic_prpvf_ops,
> +	[IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
> +};
> +
> +static int imx_ic_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct imx_ic_priv *priv;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, &priv->sd);
> +	priv->dev = &pdev->dev;
> +
> +	/* get our ipu_id, grp_id and IC task id */
> +	pdata = priv->dev->platform_data;
> +	priv->ipu_id = pdata->ipu_id;
> +	switch (pdata->grp_id) {
> +	case IMX_MEDIA_GRP_ID_IC_PRPENC:
> +		priv->task_id = IC_TASK_ENCODER;
> +		break;
> +	case IMX_MEDIA_GRP_ID_IC_PRPVF:
> +		priv->task_id = IC_TASK_VIEWFINDER;
> +		break;
> +	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
> +		priv->task_id = IC_TASK_POST_PROCESSOR;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
> +	v4l2_set_subdevdata(&priv->sd, priv);
> +	priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
> +	priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
> +	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> +	priv->sd.dev = &pdev->dev;
> +	priv->sd.owner = THIS_MODULE;
> +	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +	priv->sd.grp_id = pdata->grp_id;
> +	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
> +
> +	ret = ic_ops[priv->task_id]->init(priv);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l2_async_register_subdev(&priv->sd);
> +	if (ret)
> +		ic_ops[priv->task_id]->remove(priv);
> +
> +	return ret;
> +}
> +
> +static int imx_ic_remove(struct platform_device *pdev)
> +{
> +	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
> +	struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
> +
> +	ic_ops[priv->task_id]->remove(priv);
> +
> +	v4l2_async_unregister_subdev(&priv->sd);
> +	media_entity_cleanup(&priv->sd.entity);
> +	v4l2_device_unregister_subdev(sd);
> +
> +	return 0;
> +}
> +
> +static const struct platform_device_id imx_ic_ids[] = {
> +	{ .name = "imx-ipuv3-ic" },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(platform, imx_ic_ids);
> +
> +static struct platform_driver imx_ic_driver = {
> +	.probe = imx_ic_probe,
> +	.remove = imx_ic_remove,
> +	.id_table = imx_ic_ids,
> +	.driver = {
> +		.name = "imx-ipuv3-ic",
> +	},
> +};
> +module_platform_driver(imx_ic_driver);
> +
> +MODULE_DESCRIPTION("i.MX IC subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> +MODULE_ALIAS("platform:imx-ipuv3-ic");
> diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
> new file mode 100644
> index 0000000..1f75616
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-pp.c
> @@ -0,0 +1,636 @@
> +/*
> + * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-image-convert.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +#define PP_NUM_PADS 2
> +
> +struct pp_priv {
> +	struct imx_media_dev *md;
> +	struct imx_ic_priv *ic_priv;
> +	int pp_id;
> +
> +	struct ipu_soc *ipu;
> +	struct ipu_image_convert_ctx *ic_ctx;
> +
> +	struct media_pad pad[PP_NUM_PADS];
> +	int input_pad;
> +	int output_pad;
> +
> +	/* our dma buffer sink ring */
> +	struct imx_media_dma_buf_ring *in_ring;
> +	/* the dma buffer ring we send to sink */
> +	struct imx_media_dma_buf_ring *out_ring;
> +	struct ipu_image_convert_run *out_run;
> +
> +	struct imx_media_dma_buf *inbuf; /* last input buffer */
> +
> +	bool stream_on;    /* streaming is on */
> +	bool stop;         /* streaming is stopping */
> +	spinlock_t irqlock;
> +
> +	struct v4l2_subdev *src_sd;
> +	struct v4l2_subdev *sink_sd;
> +
> +	struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[PP_NUM_PADS];
> +
> +	/* motion select control */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	int  rotation; /* degrees */
> +	bool hflip;
> +	bool vflip;
> +
> +	/* derived from rotation, hflip, vflip controls */
> +	enum ipu_rotate_mode rot_mode;
> +};
> +
> +static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +
> +	return ic_priv->task_priv;
> +}
> +
> +static void pp_convert_complete(struct ipu_image_convert_run *run,
> +				void *data)
> +{
> +	struct pp_priv *priv = data;
> +	struct imx_media_dma_buf *done;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +
> +	done = imx_media_dma_buf_get_active(priv->out_ring);
> +	/* give the completed buffer to the sink */
> +	if (!WARN_ON(!done))
> +		imx_media_dma_buf_done(done, run->status ?
> +				       IMX_MEDIA_BUF_STATUS_ERROR :
> +				       IMX_MEDIA_BUF_STATUS_DONE);
> +
> +	/* we're done with the inbuf, queue it back */
> +	imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
> +
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +}
> +
> +static void pp_queue_conversion(struct pp_priv *priv,
> +				struct imx_media_dma_buf *inbuf)
> +{
> +	struct ipu_image_convert_run *run;
> +	struct imx_media_dma_buf *outbuf;
> +
> +	/* get next queued buffer and make it active */
> +	outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(outbuf);
> +	priv->inbuf = inbuf;
> +
> +	run = &priv->out_run[outbuf->index];
> +	run->ctx = priv->ic_ctx;
> +	run->in_phys = inbuf->phys;
> +	run->out_phys = outbuf->phys;
> +	ipu_image_convert_queue(run);
> +}
> +
> +static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct imx_media_dma_buf_ring **ring;
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +
> +	switch (cmd) {
> +	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
> +		/* src asks for a buffer ring */
> +		if (!priv->in_ring)
> +			return -EINVAL;
> +		ring = (struct imx_media_dma_buf_ring **)arg;
> +		*ring = priv->in_ring;
> +		break;
> +	case IMX_MEDIA_NEW_DMA_BUF:
> +		/* src hands us a new buffer */
> +		spin_lock_irqsave(&priv->irqlock, flags);
> +		if (!priv->stop &&
> +		    !imx_media_dma_buf_get_active(priv->out_ring)) {
> +			buf = imx_media_dma_buf_dequeue(priv->in_ring);
> +			if (buf)
> +				pp_queue_conversion(priv, buf);
> +		}
> +		spin_unlock_irqrestore(&priv->irqlock, flags);
> +		break;
> +	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
> +		/* src indicates sink buffer ring can be freed */
> +		if (!priv->in_ring)
> +			return 0;
> +		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_start(struct pp_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct ipu_image image_in, image_out;
> +	const struct imx_media_pixfmt *incc;
> +	struct v4l2_mbus_framefmt *infmt;
> +	int i, in_size, ret;
> +
> +	/* ask the sink for the buffer ring */
> +	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
> +			       &priv->out_ring);
> +	if (ret)
> +		return ret;
> +
> +	imx_media_mbus_fmt_to_ipu_image(&image_in,
> +					&priv->format_mbus[priv->input_pad]);
> +	imx_media_mbus_fmt_to_ipu_image(&image_out,
> +					&priv->format_mbus[priv->output_pad]);
> +
> +	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
> +	priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
> +						 IC_TASK_POST_PROCESSOR,
> +						 &image_in, &image_out,
> +						 priv->rot_mode,
> +						 pp_convert_complete, priv);
> +	if (IS_ERR(priv->ic_ctx))
> +		return PTR_ERR(priv->ic_ctx);
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	incc = priv->cc[priv->input_pad];
> +	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
> +
> +	if (priv->in_ring) {
> +		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
> +			  __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +	}
> +
> +	priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
> +						     &priv->src_sd->entity,
> +						     &ic_priv->sd.entity,
> +						     in_size,
> +						     IMX_MEDIA_MIN_RING_BUFS,
> +						     true);
> +	if (IS_ERR(priv->in_ring)) {
> +		v4l2_err(&ic_priv->sd,
> +			 "failed to alloc dma-buf ring\n");
> +		ret = PTR_ERR(priv->in_ring);
> +		priv->in_ring = NULL;
> +		goto out_unprep;
> +	}
> +
> +	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
> +		imx_media_dma_buf_queue(priv->in_ring, i);
> +
> +	priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
> +				sizeof(*priv->out_run), GFP_KERNEL);
> +	if (!priv->out_run) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
> +		ret = -ENOMEM;
> +		goto out_free_ring;
> +	}
> +
> +	priv->stop = false;
> +
> +	return 0;
> +
> +out_free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +out_unprep:
> +	ipu_image_convert_unprepare(priv->ic_ctx);
> +	return ret;
> +}
> +
> +static void pp_stop(struct pp_priv *priv)
> +{
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +	priv->stop = true;
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +
> +	ipu_image_convert_unprepare(priv->ic_ctx);
> +	kfree(priv->out_run);
> +
> +	priv->out_ring = NULL;
> +
> +	/* inform sink that its sink buffer ring can now be freed */
> +	v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
> +}
> +
> +static int pp_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	int ret = 0;
> +
> +	if (!priv->src_sd || !priv->sink_sd)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	if (enable && !priv->stream_on)
> +		ret = pp_start(priv);
> +	else if (!enable && priv->stream_on)
> +		pp_stop(priv);
> +
> +	if (!ret)
> +		priv->stream_on = enable;
> +	return ret;
> +}
> +
> +static int pp_enum_mbus_code(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_pad_config *cfg,
> +			     struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	const struct imx_media_pixfmt *cc;
> +	u32 fourcc;
> +	int ret;
> +
> +	if (code->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	ret = ipu_image_convert_enum_format(code->index, &fourcc);
> +	if (ret)
> +		return ret;
> +
> +	/* convert returned fourcc to mbus code */
> +	cc = imx_media_find_format(fourcc, 0, true, true);
> +	if (WARN_ON(!cc))
> +		return -EINVAL;
> +
> +	code->code = cc->codes[0];
> +	return 0;
> +}
> +
> +static int pp_get_fmt(struct v4l2_subdev *sd,
> +		      struct v4l2_subdev_pad_config *cfg,
> +		      struct v4l2_subdev_format *sdformat)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +
> +	if (sdformat->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int pp_set_fmt(struct v4l2_subdev *sd,
> +		      struct v4l2_subdev_pad_config *cfg,
> +		      struct v4l2_subdev_format *sdformat)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *cc;
> +	struct ipu_image test_in, test_out;
> +	u32 code;
> +
> +	if (sdformat->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	cc = imx_media_find_format(0, sdformat->format.code, true, true);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, true);
> +		cc = imx_media_find_format(0, code, true, true);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
> +		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
> +		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
> +	} else {
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
> +		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
> +		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		if (sdformat->pad == priv->output_pad) {
> +			*outfmt = sdformat->format;
> +			imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
> +		} else {
> +			*infmt = sdformat->format;
> +			imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
> +		}
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_link_setup(struct media_entity *entity,
> +			 const struct media_pad *local,
> +			 const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct pp_priv *priv = ic_priv->task_priv;
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->sink_sd)
> +				return -EBUSY;
> +			priv->sink_sd = remote_sd;
> +		} else {
> +			priv->sink_sd = NULL;
> +		}
> +	} else {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->src_sd)
> +				return -EBUSY;
> +			priv->src_sd = remote_sd;
> +		} else {
> +			priv->src_sd = NULL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct pp_priv *priv = container_of(ctrl->handler,
> +					       struct pp_priv, ctrl_hdlr);
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	enum ipu_rotate_mode rot_mode;
> +	bool hflip, vflip;
> +	int rotation, ret;
> +
> +	rotation = priv->rotation;
> +	hflip = priv->hflip;
> +	vflip = priv->vflip;
> +
> +	switch (ctrl->id) {
> +	case V4L2_CID_HFLIP:
> +		hflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_VFLIP:
> +		vflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_ROTATE:
> +		rotation = ctrl->val;
> +		break;
> +	default:
> +		v4l2_err(&ic_priv->sd, "Invalid control\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
> +	if (ret)
> +		return ret;
> +
> +	if (rot_mode != priv->rot_mode) {
> +		struct v4l2_mbus_framefmt *infmt, *outfmt;
> +		struct ipu_image test_in, test_out;
> +
> +		/* can't change rotation mid-streaming */
> +		if (priv->stream_on)
> +			return -EBUSY;
> +
> +		/*
> +		 * make sure this rotation will work with current input/output
> +		 * formats before setting
> +		 */
> +		infmt = &priv->format_mbus[priv->input_pad];
> +		outfmt = &priv->format_mbus[priv->output_pad];
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
> +
> +		ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);
> +		if (ret)
> +			return ret;
> +
> +		priv->rot_mode = rot_mode;
> +		priv->rotation = rotation;
> +		priv->hflip = hflip;
> +		priv->vflip = vflip;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ctrl_ops pp_ctrl_ops = {
> +	.s_ctrl = pp_s_ctrl,
> +};
> +
> +static const struct v4l2_ctrl_config pp_std_ctrl[] = {
> +	{
> +		.id = V4L2_CID_HFLIP,
> +		.name = "Horizontal Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_VFLIP,
> +		.name = "Vertical Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_ROTATE,
> +		.name = "Rotation",
> +		.type = V4L2_CTRL_TYPE_INTEGER,
> +		.def =   0,
> +		.min =   0,
> +		.max = 270,
> +		.step = 90,
> +	},
> +};
> +
> +#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
> +
> +static int pp_init_controls(struct pp_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
> +	const struct v4l2_ctrl_config *c;
> +	int i, ret;
> +
> +	v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
> +
> +	for (i = 0; i < PP_NUM_CONTROLS; i++) {
> +		c = &pp_std_ctrl[i];
> +		v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
> +				  c->id, c->min, c->max, c->step, c->def);
> +	}
> +
> +	ic_priv->sd.ctrl_handler = hdlr;
> +
> +	if (hdlr->error) {
> +		ret = hdlr->error;
> +		v4l2_ctrl_handler_free(hdlr);
> +		return ret;
> +	}
> +
> +	v4l2_ctrl_handler_setup(hdlr);
> +
> +	return 0;
> +}
> +
> +/*
> + * retrieve our pads parsed from the OF graph by the media device
> + */
> +static int pp_registered(struct v4l2_subdev *sd)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct imx_media_subdev *imxsd;
> +	struct imx_media_pad *pad;
> +	int i, ret;
> +
> +	/* get media device */
> +	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
> +
> +	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
> +	if (IS_ERR(imxsd))
> +		return PTR_ERR(imxsd);
> +
> +	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
> +		return -EINVAL;
> +
> +	for (i = 0; i < PP_NUM_PADS; i++) {
> +		pad = &imxsd->pad[i];
> +		priv->pad[i] = pad->pad;
> +		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
> +			priv->input_pad = i;
> +		else
> +			priv->output_pad = i;
> +
> +		/* set a default mbus format  */
> +		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
> +					      640, 480, 0, V4L2_FIELD_NONE,
> +					      &priv->cc[i]);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	ret = pp_init_controls(priv);
> +	if (ret)
> +		return ret;
> +
> +	ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +	return ret;
> +}
> +
> +static struct v4l2_subdev_pad_ops pp_pad_ops = {
> +	.enum_mbus_code = pp_enum_mbus_code,
> +	.get_fmt = pp_get_fmt,
> +	.set_fmt = pp_set_fmt,
> +};
> +
> +static struct v4l2_subdev_video_ops pp_video_ops = {
> +	.s_stream = pp_s_stream,
> +};
> +
> +static struct v4l2_subdev_core_ops pp_core_ops = {
> +	.ioctl = pp_ioctl,
> +};
> +
> +static struct media_entity_operations pp_entity_ops = {
> +	.link_setup = pp_link_setup,
> +	.link_validate = v4l2_subdev_link_validate,
> +};
> +
> +static struct v4l2_subdev_ops pp_subdev_ops = {
> +	.video = &pp_video_ops,
> +	.pad = &pp_pad_ops,
> +	.core = &pp_core_ops,
> +};
> +
> +static struct v4l2_subdev_internal_ops pp_internal_ops = {
> +	.registered = pp_registered,
> +};
> +
> +static int pp_init(struct imx_ic_priv *ic_priv)
> +{
> +	struct pp_priv *priv;
> +
> +	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	ic_priv->task_priv = priv;
> +	priv->ic_priv = ic_priv;
> +	spin_lock_init(&priv->irqlock);
> +
> +	/* get our PP id */
> +	priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
> +
> +	return 0;
> +}
> +
> +static void pp_remove(struct imx_ic_priv *ic_priv)
> +{
> +	struct pp_priv *priv = ic_priv->task_priv;
> +
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +}
> +
> +struct imx_ic_ops imx_ic_pp_ops = {
> +	.subdev_ops = &pp_subdev_ops,
> +	.internal_ops = &pp_internal_ops,
> +	.entity_ops = &pp_entity_ops,
> +	.init = pp_init,
> +	.remove = pp_remove,
> +};
> diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
> new file mode 100644
> index 0000000..3d85a82
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-prpenc.c
> @@ -0,0 +1,1033 @@
> +/*
> + * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
> + *
> + * This subdevice handles capture of video frames from the CSI, which
> + * are routed directly to the Image Converter preprocess encode task,
> + * for resizing, colorspace conversion, and rotation.
> + *
> + * Copyright (c) 2012-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +#define PRPENC_NUM_PADS 2
> +
> +#define MAX_W_IC   1024
> +#define MAX_H_IC   1024
> +#define MAX_W_SINK 4096
> +#define MAX_H_SINK 4096
> +
> +struct prpenc_priv {
> +	struct imx_media_dev *md;
> +	struct imx_ic_priv *ic_priv;
> +
> +	/* IPU units we require */
> +	struct ipu_soc *ipu;
> +	struct ipu_ic *ic_enc;
> +
> +	struct media_pad pad[PRPENC_NUM_PADS];
> +	int input_pad;
> +	int output_pad;
> +
> +	struct ipuv3_channel *enc_ch;
> +	struct ipuv3_channel *enc_rot_in_ch;
> +	struct ipuv3_channel *enc_rot_out_ch;
> +
> +	/* the dma buffer ring to send to sink */
> +	struct imx_media_dma_buf_ring *out_ring;
> +	struct imx_media_dma_buf *next;
> +
> +	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
> +
> +	struct v4l2_subdev *src_sd;
> +	struct v4l2_subdev *sink_sd;
> +
> +	/* the CSI id at link validate */
> +	int csi_id;
> +
> +	/* the attached sensor at stream on */
> +	struct imx_media_subdev *sensor;
> +
> +	struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
> +
> +	struct imx_media_dma_buf rot_buf[2];
> +
> +	/* controls */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	int  rotation; /* degrees */
> +	bool hflip;
> +	bool vflip;
> +
> +	/* derived from rotation, hflip, vflip controls */
> +	enum ipu_rotate_mode rot_mode;
> +
> +	spinlock_t irqlock;
> +
> +	struct timer_list eof_timeout_timer;
> +	int eof_irq;
> +	int nfb4eof_irq;
> +
> +	bool stream_on; /* streaming is on */
> +	bool last_eof;  /* waiting for last EOF at stream off */
> +	struct completion last_eof_comp;
> +};
> +
> +static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +
> +	return ic_priv->task_priv;
> +}
> +
> +static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
> +{
> +	if (!IS_ERR_OR_NULL(priv->ic_enc))
> +		ipu_ic_put(priv->ic_enc);
> +	priv->ic_enc = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_ch))
> +		ipu_idmac_put(priv->enc_ch);
> +	priv->enc_ch = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
> +		ipu_idmac_put(priv->enc_rot_in_ch);
> +	priv->enc_rot_in_ch = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
> +		ipu_idmac_put(priv->enc_rot_out_ch);
> +	priv->enc_rot_out_ch = NULL;
> +}
> +
> +static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	int ret;
> +
> +	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
> +
> +	priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
> +	if (IS_ERR(priv->ic_enc)) {
> +		v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
> +		ret = PTR_ERR(priv->ic_enc);
> +		goto out;
> +	}
> +
> +	priv->enc_ch = ipu_idmac_get(priv->ipu,
> +				     IPUV3_CHANNEL_IC_PRP_ENC_MEM);
> +	if (IS_ERR(priv->enc_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_IC_PRP_ENC_MEM);
> +		ret = PTR_ERR(priv->enc_ch);
> +		goto out;
> +	}
> +
> +	priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
> +					    IPUV3_CHANNEL_MEM_ROT_ENC);
> +	if (IS_ERR(priv->enc_rot_in_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_MEM_ROT_ENC);
> +		ret = PTR_ERR(priv->enc_rot_in_ch);
> +		goto out;
> +	}
> +
> +	priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
> +					     IPUV3_CHANNEL_ROT_ENC_MEM);
> +	if (IS_ERR(priv->enc_rot_out_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_ROT_ENC_MEM);
> +		ret = PTR_ERR(priv->enc_rot_out_ch);
> +		goto out;
> +	}
> +
> +	return 0;
> +out:
> +	prpenc_put_ipu_resources(priv);
> +	return ret;
> +}
> +
> +static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
> +{
> +	struct prpenc_priv *priv = dev_id;
> +	struct imx_media_dma_buf *done, *next;
> +	struct ipuv3_channel *channel;
> +
> +	spin_lock(&priv->irqlock);
> +
> +	if (priv->last_eof) {
> +		complete(&priv->last_eof_comp);
> +		priv->last_eof = false;
> +		goto unlock;
> +	}
> +
> +	/* inform CSI of this EOF so it can monitor frame intervals */
> +	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
> +			 0, NULL);
> +
> +	channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
> +		priv->enc_rot_out_ch : priv->enc_ch;
> +
> +	done = imx_media_dma_buf_get_active(priv->out_ring);
> +	/* give the completed buffer to the sink  */
> +	if (!WARN_ON(!done))
> +		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
> +
> +	/* priv->next buffer is now the active one */
> +	imx_media_dma_buf_set_active(priv->next);
> +
> +	/* bump the EOF timeout timer */
> +	mod_timer(&priv->eof_timeout_timer,
> +		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +
> +	if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
> +		ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);
> +
> +	/* get next queued buffer */
> +	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +
> +	ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
> +	ipu_idmac_select_buffer(channel, priv->ipu_buf_num);
> +
> +	/* toggle IPU double-buffer index */
> +	priv->ipu_buf_num ^= 1;
> +	priv->next = next;
> +
> +unlock:
> +	spin_unlock(&priv->irqlock);
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
> +{
> +	struct prpenc_priv *priv = dev_id;
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	static const struct v4l2_event ev = {
> +		.type = V4L2_EVENT_IMX_NFB4EOF,
> +	};
> +
> +	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
> +
> +	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
> +
> +	return IRQ_HANDLED;
> +}
> +
> +/*
> + * EOF timeout timer function.
> + */
> +static void prpenc_eof_timeout(unsigned long data)
> +{
> +	struct prpenc_priv *priv = (struct prpenc_priv *)data;
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	static const struct v4l2_event ev = {
> +		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
> +	};
> +
> +	v4l2_err(&ic_priv->sd, "EOF timeout\n");
> +
> +	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
> +}
> +
> +static void prpenc_setup_channel(struct prpenc_priv *priv,
> +				 struct ipuv3_channel *channel,
> +				 enum ipu_rotate_mode rot_mode,
> +				 dma_addr_t addr0, dma_addr_t addr1,
> +				 bool rot_swap_width_height)
> +{
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	unsigned int burst_size;
> +	struct ipu_image image;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (rot_swap_width_height)
> +		swap(outfmt->width, outfmt->height);
> +
> +	ipu_cpmem_zero(channel);
> +
> +	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
> +
> +	image.phys0 = addr0;
> +	image.phys1 = addr1;
> +	ipu_cpmem_set_image(channel, &image);
> +
> +	if (channel == priv->enc_rot_in_ch ||
> +	    channel == priv->enc_rot_out_ch) {
> +		burst_size = 8;
> +		ipu_cpmem_set_block_mode(channel);
> +	} else {
> +		burst_size = (outfmt->width & 0xf) ? 8 : 16;
> +	}
> +
> +	ipu_cpmem_set_burstsize(channel, burst_size);
> +
> +	if (rot_mode)
> +		ipu_cpmem_set_rotation(channel, rot_mode);
> +
> +	if (outfmt->field == V4L2_FIELD_NONE &&
> +	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
> +	     infmt->field == V4L2_FIELD_ALTERNATE) &&
> +	    channel == priv->enc_ch)
> +		ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
> +
> +	ipu_ic_task_idma_init(priv->ic_enc, channel,
> +			      outfmt->width, outfmt->height,
> +			      burst_size, rot_mode);
> +	ipu_cpmem_set_axi_id(channel, 1);
> +
> +	ipu_idmac_set_double_buffer(channel, true);
> +
> +	if (rot_swap_width_height)
> +		swap(outfmt->width, outfmt->height);
> +}
> +
> +static int prpenc_setup_rotation(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *outcc, *incc;
> +	struct imx_media_dma_buf *buf0, *buf1;
> +	int out_size, ret;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	incc = priv->cc[priv->input_pad];
> +	outcc = priv->cc[priv->output_pad];
> +
> +	out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
> +
> +	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
> +		return ret;
> +	}
> +	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
> +		goto free_rot0;
> +	}
> +
> +	ret = ipu_ic_task_init(priv->ic_enc,
> +			       infmt->width, infmt->height,
> +			       outfmt->height, outfmt->width,
> +			       incc->cs, outcc->cs);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
> +		goto free_rot1;
> +	}
> +
> +	/* init the IC ENC-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_ch,
> +			     IPU_ROTATE_NONE,
> +			     priv->rot_buf[0].phys,
> +			     priv->rot_buf[1].phys,
> +			     true);
> +
> +	/* init the MEM-->IC ENC ROT IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_rot_in_ch,
> +			     priv->rot_mode,
> +			     priv->rot_buf[0].phys,
> +			     priv->rot_buf[1].phys,
> +			     true);
> +
> +	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(buf0);
> +	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	priv->next = buf1;
> +
> +	/* init the destination IC ENC ROT-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_rot_out_ch,
> +			     IPU_ROTATE_NONE,
> +			     buf0->phys, buf1->phys,
> +			     false);
> +
> +	/* now link IC ENC-->MEM to MEM-->IC ENC ROT */
> +	ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
> +
> +	/* enable the IC */
> +	ipu_ic_enable(priv->ic_enc);
> +
> +	/* set buffers ready */
> +	ipu_idmac_select_buffer(priv->enc_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_ch, 1);
> +	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);
> +
> +	/* enable the channels */
> +	ipu_idmac_enable_channel(priv->enc_ch);
> +	ipu_idmac_enable_channel(priv->enc_rot_in_ch);
> +	ipu_idmac_enable_channel(priv->enc_rot_out_ch);
> +
> +	/* and finally enable the IC PRPENC task */
> +	ipu_ic_task_enable(priv->ic_enc);
> +
> +	return 0;
> +
> +free_rot1:
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
> +free_rot0:
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
> +	return ret;
> +}
> +
> +static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
> +{
> +	ipu_ic_task_disable(priv->ic_enc);
> +
> +	ipu_idmac_disable_channel(priv->enc_ch);
> +	ipu_idmac_disable_channel(priv->enc_rot_in_ch);
> +	ipu_idmac_disable_channel(priv->enc_rot_out_ch);
> +
> +	ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
> +
> +	ipu_ic_disable(priv->ic_enc);
> +
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
> +}
> +
> +static int prpenc_setup_norotation(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *outcc, *incc;
> +	struct imx_media_dma_buf *buf0, *buf1;
> +	int ret;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	incc = priv->cc[priv->input_pad];
> +	outcc = priv->cc[priv->output_pad];
> +
> +	ret = ipu_ic_task_init(priv->ic_enc,
> +			       infmt->width, infmt->height,
> +			       outfmt->width, outfmt->height,
> +			       incc->cs, outcc->cs);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
> +		return ret;
> +	}
> +
> +	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(buf0);
> +	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	priv->next = buf1;
> +
> +	/* init the IC PRP-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
> +			     buf0->phys, buf1->phys,
> +			     false);
> +
> +	ipu_cpmem_dump(priv->enc_ch);
> +	ipu_ic_dump(priv->ic_enc);
> +	ipu_dump(priv->ipu);
> +
> +	ipu_ic_enable(priv->ic_enc);
> +
> +	/* set buffers ready */
> +	ipu_idmac_select_buffer(priv->enc_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_ch, 1);
> +
> +	/* enable the channels */
> +	ipu_idmac_enable_channel(priv->enc_ch);
> +
> +	/* enable the IC ENCODE task */
> +	ipu_ic_task_enable(priv->ic_enc);
> +
> +	return 0;
> +}
> +
> +static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
> +{
> +	ipu_ic_task_disable(priv->ic_enc);
> +	ipu_idmac_disable_channel(priv->enc_ch);
> +	ipu_ic_disable(priv->ic_enc);
> +}
> +
> +static int prpenc_start(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	int ret;
> +
> +	if (!priv->sensor) {
> +		v4l2_err(&ic_priv->sd, "no sensor attached\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = prpenc_get_ipu_resources(priv);
> +	if (ret)
> +		return ret;
> +
> +	/* set IC to receive from CSI */
> +	ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
> +
> +	/* ask the sink for the buffer ring */
> +	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
> +			       &priv->out_ring);
> +	if (ret)
> +		goto out_put_ipu;
> +
> +	priv->ipu_buf_num = 0;
> +
> +	/* init EOF completion waitq */
> +	init_completion(&priv->last_eof_comp);
> +	priv->last_eof = false;
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		ret = prpenc_setup_rotation(priv);
> +	else
> +		ret = prpenc_setup_norotation(priv);
> +	if (ret)
> +		goto out_put_ipu;
> +
> +	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
> +						  priv->enc_ch,
> +						  IPU_IRQ_NFB4EOF);
> +	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
> +			       prpenc_nfb4eof_interrupt, 0,
> +			       "imx-ic-prpenc-nfb4eof", priv);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd,
> +			 "Error registering NFB4EOF irq: %d\n", ret);
> +		goto out_unsetup;
> +	}
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		priv->eof_irq = ipu_idmac_channel_irq(
> +			priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
> +	else
> +		priv->eof_irq = ipu_idmac_channel_irq(
> +			priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
> +
> +	ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
> +			       prpenc_eof_interrupt, 0,
> +			       "imx-ic-prpenc-eof", priv);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd,
> +			 "Error registering eof irq: %d\n", ret);
> +		goto out_free_nfb4eof_irq;
> +	}
> +
> +	/* start the EOF timeout timer */
> +	mod_timer(&priv->eof_timeout_timer,
> +		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +
> +	return 0;
> +
> +out_free_nfb4eof_irq:
> +	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
> +out_unsetup:
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		prpenc_unsetup_rotation(priv);
> +	else
> +		prpenc_unsetup_norotation(priv);
> +out_put_ipu:
> +	prpenc_put_ipu_resources(priv);
> +	return ret;
> +}
> +
> +static void prpenc_stop(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	unsigned long flags;
> +	int ret;
> +
> +	/* mark next EOF interrupt as the last before stream off */
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +	priv->last_eof = true;
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +
> +	/*
> +	 * and then wait for interrupt handler to mark completion.
> +	 */
> +	ret = wait_for_completion_timeout(
> +		&priv->last_eof_comp,
> +		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +	if (ret == 0)
> +		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
> +
> +	devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
> +	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		prpenc_unsetup_rotation(priv);
> +	else
> +		prpenc_unsetup_norotation(priv);
> +
> +	prpenc_put_ipu_resources(priv);
> +
> +	/* cancel the EOF timeout timer */
> +	del_timer_sync(&priv->eof_timeout_timer);
> +
> +	priv->out_ring = NULL;
> +
> +	/* inform sink that the buffer ring can now be freed */
> +	v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
> +}
> +
> +static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +	bool allow_planar;
> +
> +	if (code->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	allow_planar = (code->pad == priv->output_pad);
> +
> +	return imx_media_enum_format(&code->code, code->index,
> +				     true, allow_planar);
> +}
> +
> +static int prpenc_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *sdformat)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +
> +	if (sdformat->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int prpenc_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *sdformat)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *cc;
> +	bool allow_planar;
> +	u32 code;
> +
> +	if (sdformat->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	allow_planar = (sdformat->pad == priv->output_pad);
> +
> +	cc = imx_media_find_format(0, sdformat->format.code,
> +				   true, allow_planar);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, false);
> +		cc = imx_media_find_format(0, code, true, false);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.width = min_t(__u32,
> +					       sdformat->format.width,
> +					       MAX_W_IC);
> +		sdformat->format.height = min_t(__u32,
> +						sdformat->format.height,
> +						MAX_H_IC);
> +
> +		if (sdformat->format.field != V4L2_FIELD_NONE)
> +			sdformat->format.field = infmt->field;
> +
> +		/* IC resizer cannot downsize more than 4:1 */
> +		if (ipu_rot_mode_is_irt(priv->rot_mode)) {
> +			sdformat->format.width = max_t(__u32,
> +						       sdformat->format.width,
> +						       infmt->height / 4);
> +			sdformat->format.height = max_t(__u32,
> +							sdformat->format.height,
> +							infmt->width / 4);
> +		} else {
> +			sdformat->format.width = max_t(__u32,
> +						       sdformat->format.width,
> +						       infmt->width / 4);
> +			sdformat->format.height = max_t(__u32,
> +							sdformat->format.height,
> +							infmt->height / 4);
> +		}
> +	} else {
> +		sdformat->format.width = min_t(__u32,
> +					       sdformat->format.width,
> +					       MAX_W_SINK);
> +		sdformat->format.height = min_t(__u32,
> +						sdformat->format.height,
> +						MAX_H_SINK);
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_link_setup(struct media_entity *entity,
> +			     const struct media_pad *local,
> +			     const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct prpenc_priv *priv = ic_priv->task_priv;
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->sink_sd)
> +				return -EBUSY;
> +			priv->sink_sd = remote_sd;
> +		} else {
> +			priv->sink_sd = NULL;
> +		}
> +
> +		return 0;
> +	}
> +
> +	/* this is sink pad */
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->src_sd)
> +			return -EBUSY;
> +		priv->src_sd = remote_sd;
> +	} else {
> +		priv->src_sd = NULL;
> +		return 0;
> +	}
> +
> +	switch (remote_sd->grp_id) {
> +	case IMX_MEDIA_GRP_ID_CSI0:
> +		priv->csi_id = 0;
> +		break;
> +	case IMX_MEDIA_GRP_ID_CSI1:
> +		priv->csi_id = 1;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_link_validate(struct v4l2_subdev *sd,
> +				struct media_link *link,
> +				struct v4l2_subdev_format *source_fmt,
> +				struct v4l2_subdev_format *sink_fmt)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct prpenc_priv *priv = ic_priv->task_priv;
> +	struct v4l2_mbus_config sensor_mbus_cfg;
> +	int ret;
> +
> +	ret = v4l2_subdev_link_validate_default(sd, link,
> +						source_fmt, sink_fmt);
> +	if (ret)
> +		return ret;
> +
> +	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
> +	if (IS_ERR(priv->sensor)) {
> +		v4l2_err(&ic_priv->sd, "no sensor attached\n");
> +		ret = PTR_ERR(priv->sensor);
> +		priv->sensor = NULL;
> +		return ret;
> +	}
> +
> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> +			       &sensor_mbus_cfg);
> +	if (ret)
> +		return ret;
> +
> +	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
> +		int vc_num = 0;
> +		/* see NOTE in imx-csi.c */
> +#if 0
> +		vc_num = imx_media_find_mipi_csi2_channel(
> +			priv->md, &ic_priv->sd.entity);
> +		if (vc_num < 0)
> +			return vc_num;
> +#endif
> +		/* only virtual channel 0 can be sent to IC */
> +		if (vc_num != 0)
> +			return -EINVAL;
> +	} else {
> +		/*
> +		 * only 8-bit pixels can be sent to IC for parallel
> +		 * busses
> +		 */
> +		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct prpenc_priv *priv = container_of(ctrl->handler,
> +					       struct prpenc_priv, ctrl_hdlr);
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	enum ipu_rotate_mode rot_mode;
> +	bool hflip, vflip;
> +	int rotation, ret;
> +
> +	rotation = priv->rotation;
> +	hflip = priv->hflip;
> +	vflip = priv->vflip;
> +
> +	switch (ctrl->id) {
> +	case V4L2_CID_HFLIP:
> +		hflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_VFLIP:
> +		vflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_ROTATE:
> +		rotation = ctrl->val;
> +		break;
> +	default:
> +		v4l2_err(&ic_priv->sd, "Invalid control\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
> +	if (ret)
> +		return ret;
> +
> +	if (rot_mode != priv->rot_mode) {
> +		/* can't change rotation mid-streaming */
> +		if (priv->stream_on)
> +			return -EBUSY;
> +
> +		priv->rot_mode = rot_mode;
> +		priv->rotation = rotation;
> +		priv->hflip = hflip;
> +		priv->vflip = vflip;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
> +	.s_ctrl = prpenc_s_ctrl,
> +};
> +
> +static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
> +	{
> +		.id = V4L2_CID_HFLIP,
> +		.name = "Horizontal Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_VFLIP,
> +		.name = "Vertical Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_ROTATE,
> +		.name = "Rotation",
> +		.type = V4L2_CTRL_TYPE_INTEGER,
> +		.def =   0,
> +		.min =   0,
> +		.max = 270,
> +		.step = 90,
> +	},
> +};

Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information
like the name and type.

If this is also done elsewhere, then it should be changed there as well.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-20 14:29     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:29 UTC (permalink / raw)
  To: linux-arm-kernel

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This is a set of three media entity subdevice drivers for the i.MX
> Image Converter. The i.MX IC module contains three independent
> "tasks":
> 
> - Pre-processing Encode task: video frames are routed directly from
>   the CSI and can be scaled, color-space converted, and rotated.
>   Scaled output is limited to 1024x1024 resolution. Output frames
>   are routed to the camera interface entities (camif).
> 
> - Pre-processing Viewfinder task: this task can perform the same
>   conversions as the pre-process encode task, but in addition can
>   be used for hardware motion compensated deinterlacing. Frames can
>   come either directly from the CSI or from the SMFC entities (memory
>   buffers via IDMAC channels). Scaled output is limited to 1024x1024
>   resolution. Output frames can be routed to various sinks including
>   the post-processing task entities.
> 
> - Post-processing task: same conversions as pre-process encode. However
>   this entity sends frames to the i.MX IPU image converter which supports
>   image tiling, which allows scaled output up to 4096x4096 resolution.
>   Output frames can be routed to the camera interfaces.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile        |    2 +
>  drivers/staging/media/imx/imx-ic-common.c |  109 +++
>  drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
>  drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
>  drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
>  drivers/staging/media/imx/imx-ic.h        |   38 +
>  6 files changed, 2997 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-ic-common.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
>  create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
>  create mode 100644 drivers/staging/media/imx/imx-ic.h
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index 3559d7b..d2a962c 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -1,8 +1,10 @@
>  imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
>  	imx-media-of.o
> +imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
>  
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
>  obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
> +obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
> new file mode 100644
> index 0000000..45706ca
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-common.c
> @@ -0,0 +1,109 @@
> +/*
> + * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
> +	[IC_TASK_ENCODER]        = &imx_ic_prpenc_ops,
> +	[IC_TASK_VIEWFINDER]     = &imx_ic_prpvf_ops,
> +	[IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
> +};
> +
> +static int imx_ic_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct imx_ic_priv *priv;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, &priv->sd);
> +	priv->dev = &pdev->dev;
> +
> +	/* get our ipu_id, grp_id and IC task id */
> +	pdata = priv->dev->platform_data;
> +	priv->ipu_id = pdata->ipu_id;
> +	switch (pdata->grp_id) {
> +	case IMX_MEDIA_GRP_ID_IC_PRPENC:
> +		priv->task_id = IC_TASK_ENCODER;
> +		break;
> +	case IMX_MEDIA_GRP_ID_IC_PRPVF:
> +		priv->task_id = IC_TASK_VIEWFINDER;
> +		break;
> +	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
> +		priv->task_id = IC_TASK_POST_PROCESSOR;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
> +	v4l2_set_subdevdata(&priv->sd, priv);
> +	priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
> +	priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
> +	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> +	priv->sd.dev = &pdev->dev;
> +	priv->sd.owner = THIS_MODULE;
> +	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +	priv->sd.grp_id = pdata->grp_id;
> +	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
> +
> +	ret = ic_ops[priv->task_id]->init(priv);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l2_async_register_subdev(&priv->sd);
> +	if (ret)
> +		ic_ops[priv->task_id]->remove(priv);
> +
> +	return ret;
> +}
> +
> +static int imx_ic_remove(struct platform_device *pdev)
> +{
> +	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
> +	struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
> +
> +	ic_ops[priv->task_id]->remove(priv);
> +
> +	v4l2_async_unregister_subdev(&priv->sd);
> +	media_entity_cleanup(&priv->sd.entity);
> +	v4l2_device_unregister_subdev(sd);
> +
> +	return 0;
> +}
> +
> +static const struct platform_device_id imx_ic_ids[] = {
> +	{ .name = "imx-ipuv3-ic" },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(platform, imx_ic_ids);
> +
> +static struct platform_driver imx_ic_driver = {
> +	.probe = imx_ic_probe,
> +	.remove = imx_ic_remove,
> +	.id_table = imx_ic_ids,
> +	.driver = {
> +		.name = "imx-ipuv3-ic",
> +	},
> +};
> +module_platform_driver(imx_ic_driver);
> +
> +MODULE_DESCRIPTION("i.MX IC subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> +MODULE_ALIAS("platform:imx-ipuv3-ic");
> diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
> new file mode 100644
> index 0000000..1f75616
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-pp.c
> @@ -0,0 +1,636 @@
> +/*
> + * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-image-convert.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +#define PP_NUM_PADS 2
> +
> +struct pp_priv {
> +	struct imx_media_dev *md;
> +	struct imx_ic_priv *ic_priv;
> +	int pp_id;
> +
> +	struct ipu_soc *ipu;
> +	struct ipu_image_convert_ctx *ic_ctx;
> +
> +	struct media_pad pad[PP_NUM_PADS];
> +	int input_pad;
> +	int output_pad;
> +
> +	/* our dma buffer sink ring */
> +	struct imx_media_dma_buf_ring *in_ring;
> +	/* the dma buffer ring we send to sink */
> +	struct imx_media_dma_buf_ring *out_ring;
> +	struct ipu_image_convert_run *out_run;
> +
> +	struct imx_media_dma_buf *inbuf; /* last input buffer */
> +
> +	bool stream_on;    /* streaming is on */
> +	bool stop;         /* streaming is stopping */
> +	spinlock_t irqlock;
> +
> +	struct v4l2_subdev *src_sd;
> +	struct v4l2_subdev *sink_sd;
> +
> +	struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[PP_NUM_PADS];
> +
> +	/* motion select control */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	int  rotation; /* degrees */
> +	bool hflip;
> +	bool vflip;
> +
> +	/* derived from rotation, hflip, vflip controls */
> +	enum ipu_rotate_mode rot_mode;
> +};
> +
> +static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +
> +	return ic_priv->task_priv;
> +}
> +
> +static void pp_convert_complete(struct ipu_image_convert_run *run,
> +				void *data)
> +{
> +	struct pp_priv *priv = data;
> +	struct imx_media_dma_buf *done;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +
> +	done = imx_media_dma_buf_get_active(priv->out_ring);
> +	/* give the completed buffer to the sink */
> +	if (!WARN_ON(!done))
> +		imx_media_dma_buf_done(done, run->status ?
> +				       IMX_MEDIA_BUF_STATUS_ERROR :
> +				       IMX_MEDIA_BUF_STATUS_DONE);
> +
> +	/* we're done with the inbuf, queue it back */
> +	imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
> +
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +}
> +
> +static void pp_queue_conversion(struct pp_priv *priv,
> +				struct imx_media_dma_buf *inbuf)
> +{
> +	struct ipu_image_convert_run *run;
> +	struct imx_media_dma_buf *outbuf;
> +
> +	/* get next queued buffer and make it active */
> +	outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(outbuf);
> +	priv->inbuf = inbuf;
> +
> +	run = &priv->out_run[outbuf->index];
> +	run->ctx = priv->ic_ctx;
> +	run->in_phys = inbuf->phys;
> +	run->out_phys = outbuf->phys;
> +	ipu_image_convert_queue(run);
> +}
> +
> +static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct imx_media_dma_buf_ring **ring;
> +	struct imx_media_dma_buf *buf;
> +	unsigned long flags;
> +
> +	switch (cmd) {
> +	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
> +		/* src asks for a buffer ring */
> +		if (!priv->in_ring)
> +			return -EINVAL;
> +		ring = (struct imx_media_dma_buf_ring **)arg;
> +		*ring = priv->in_ring;
> +		break;
> +	case IMX_MEDIA_NEW_DMA_BUF:
> +		/* src hands us a new buffer */
> +		spin_lock_irqsave(&priv->irqlock, flags);
> +		if (!priv->stop &&
> +		    !imx_media_dma_buf_get_active(priv->out_ring)) {
> +			buf = imx_media_dma_buf_dequeue(priv->in_ring);
> +			if (buf)
> +				pp_queue_conversion(priv, buf);
> +		}
> +		spin_unlock_irqrestore(&priv->irqlock, flags);
> +		break;
> +	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
> +		/* src indicates sink buffer ring can be freed */
> +		if (!priv->in_ring)
> +			return 0;
> +		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_start(struct pp_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct ipu_image image_in, image_out;
> +	const struct imx_media_pixfmt *incc;
> +	struct v4l2_mbus_framefmt *infmt;
> +	int i, in_size, ret;
> +
> +	/* ask the sink for the buffer ring */
> +	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
> +			       &priv->out_ring);
> +	if (ret)
> +		return ret;
> +
> +	imx_media_mbus_fmt_to_ipu_image(&image_in,
> +					&priv->format_mbus[priv->input_pad]);
> +	imx_media_mbus_fmt_to_ipu_image(&image_out,
> +					&priv->format_mbus[priv->output_pad]);
> +
> +	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
> +	priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
> +						 IC_TASK_POST_PROCESSOR,
> +						 &image_in, &image_out,
> +						 priv->rot_mode,
> +						 pp_convert_complete, priv);
> +	if (IS_ERR(priv->ic_ctx))
> +		return PTR_ERR(priv->ic_ctx);
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	incc = priv->cc[priv->input_pad];
> +	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
> +
> +	if (priv->in_ring) {
> +		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
> +			  __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +	}
> +
> +	priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
> +						     &priv->src_sd->entity,
> +						     &ic_priv->sd.entity,
> +						     in_size,
> +						     IMX_MEDIA_MIN_RING_BUFS,
> +						     true);
> +	if (IS_ERR(priv->in_ring)) {
> +		v4l2_err(&ic_priv->sd,
> +			 "failed to alloc dma-buf ring\n");
> +		ret = PTR_ERR(priv->in_ring);
> +		priv->in_ring = NULL;
> +		goto out_unprep;
> +	}
> +
> +	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
> +		imx_media_dma_buf_queue(priv->in_ring, i);
> +
> +	priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
> +				sizeof(*priv->out_run), GFP_KERNEL);
> +	if (!priv->out_run) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
> +		ret = -ENOMEM;
> +		goto out_free_ring;
> +	}
> +
> +	priv->stop = false;
> +
> +	return 0;
> +
> +out_free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +out_unprep:
> +	ipu_image_convert_unprepare(priv->ic_ctx);
> +	return ret;
> +}
> +
> +static void pp_stop(struct pp_priv *priv)
> +{
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +	priv->stop = true;
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +
> +	ipu_image_convert_unprepare(priv->ic_ctx);
> +	kfree(priv->out_run);
> +
> +	priv->out_ring = NULL;
> +
> +	/* inform sink that its sink buffer ring can now be freed */
> +	v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
> +}
> +
> +static int pp_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	int ret = 0;
> +
> +	if (!priv->src_sd || !priv->sink_sd)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	if (enable && !priv->stream_on)
> +		ret = pp_start(priv);
> +	else if (!enable && priv->stream_on)
> +		pp_stop(priv);
> +
> +	if (!ret)
> +		priv->stream_on = enable;
> +	return ret;
> +}
> +
> +static int pp_enum_mbus_code(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_pad_config *cfg,
> +			     struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	const struct imx_media_pixfmt *cc;
> +	u32 fourcc;
> +	int ret;
> +
> +	if (code->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	ret = ipu_image_convert_enum_format(code->index, &fourcc);
> +	if (ret)
> +		return ret;
> +
> +	/* convert returned fourcc to mbus code */
> +	cc = imx_media_find_format(fourcc, 0, true, true);
> +	if (WARN_ON(!cc))
> +		return -EINVAL;
> +
> +	code->code = cc->codes[0];
> +	return 0;
> +}
> +
> +static int pp_get_fmt(struct v4l2_subdev *sd,
> +		      struct v4l2_subdev_pad_config *cfg,
> +		      struct v4l2_subdev_format *sdformat)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +
> +	if (sdformat->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int pp_set_fmt(struct v4l2_subdev *sd,
> +		      struct v4l2_subdev_pad_config *cfg,
> +		      struct v4l2_subdev_format *sdformat)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *cc;
> +	struct ipu_image test_in, test_out;
> +	u32 code;
> +
> +	if (sdformat->pad >= PP_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	cc = imx_media_find_format(0, sdformat->format.code, true, true);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, true);
> +		cc = imx_media_find_format(0, code, true, true);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
> +		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
> +		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
> +	} else {
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
> +		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
> +		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		if (sdformat->pad == priv->output_pad) {
> +			*outfmt = sdformat->format;
> +			imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
> +		} else {
> +			*infmt = sdformat->format;
> +			imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
> +		}
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_link_setup(struct media_entity *entity,
> +			 const struct media_pad *local,
> +			 const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct pp_priv *priv = ic_priv->task_priv;
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->sink_sd)
> +				return -EBUSY;
> +			priv->sink_sd = remote_sd;
> +		} else {
> +			priv->sink_sd = NULL;
> +		}
> +	} else {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->src_sd)
> +				return -EBUSY;
> +			priv->src_sd = remote_sd;
> +		} else {
> +			priv->src_sd = NULL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct pp_priv *priv = container_of(ctrl->handler,
> +					       struct pp_priv, ctrl_hdlr);
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	enum ipu_rotate_mode rot_mode;
> +	bool hflip, vflip;
> +	int rotation, ret;
> +
> +	rotation = priv->rotation;
> +	hflip = priv->hflip;
> +	vflip = priv->vflip;
> +
> +	switch (ctrl->id) {
> +	case V4L2_CID_HFLIP:
> +		hflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_VFLIP:
> +		vflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_ROTATE:
> +		rotation = ctrl->val;
> +		break;
> +	default:
> +		v4l2_err(&ic_priv->sd, "Invalid control\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
> +	if (ret)
> +		return ret;
> +
> +	if (rot_mode != priv->rot_mode) {
> +		struct v4l2_mbus_framefmt *infmt, *outfmt;
> +		struct ipu_image test_in, test_out;
> +
> +		/* can't change rotation mid-streaming */
> +		if (priv->stream_on)
> +			return -EBUSY;
> +
> +		/*
> +		 * make sure this rotation will work with current input/output
> +		 * formats before setting
> +		 */
> +		infmt = &priv->format_mbus[priv->input_pad];
> +		outfmt = &priv->format_mbus[priv->output_pad];
> +		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
> +		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
> +
> +		ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);
> +		if (ret)
> +			return ret;
> +
> +		priv->rot_mode = rot_mode;
> +		priv->rotation = rotation;
> +		priv->hflip = hflip;
> +		priv->vflip = vflip;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ctrl_ops pp_ctrl_ops = {
> +	.s_ctrl = pp_s_ctrl,
> +};
> +
> +static const struct v4l2_ctrl_config pp_std_ctrl[] = {
> +	{
> +		.id = V4L2_CID_HFLIP,
> +		.name = "Horizontal Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_VFLIP,
> +		.name = "Vertical Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_ROTATE,
> +		.name = "Rotation",
> +		.type = V4L2_CTRL_TYPE_INTEGER,
> +		.def =   0,
> +		.min =   0,
> +		.max = 270,
> +		.step = 90,
> +	},
> +};
> +
> +#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
> +
> +static int pp_init_controls(struct pp_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
> +	const struct v4l2_ctrl_config *c;
> +	int i, ret;
> +
> +	v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
> +
> +	for (i = 0; i < PP_NUM_CONTROLS; i++) {
> +		c = &pp_std_ctrl[i];
> +		v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
> +				  c->id, c->min, c->max, c->step, c->def);
> +	}
> +
> +	ic_priv->sd.ctrl_handler = hdlr;
> +
> +	if (hdlr->error) {
> +		ret = hdlr->error;
> +		v4l2_ctrl_handler_free(hdlr);
> +		return ret;
> +	}
> +
> +	v4l2_ctrl_handler_setup(hdlr);
> +
> +	return 0;
> +}
> +
> +/*
> + * retrieve our pads parsed from the OF graph by the media device
> + */
> +static int pp_registered(struct v4l2_subdev *sd)
> +{
> +	struct pp_priv *priv = sd_to_priv(sd);
> +	struct imx_media_subdev *imxsd;
> +	struct imx_media_pad *pad;
> +	int i, ret;
> +
> +	/* get media device */
> +	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
> +
> +	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
> +	if (IS_ERR(imxsd))
> +		return PTR_ERR(imxsd);
> +
> +	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
> +		return -EINVAL;
> +
> +	for (i = 0; i < PP_NUM_PADS; i++) {
> +		pad = &imxsd->pad[i];
> +		priv->pad[i] = pad->pad;
> +		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
> +			priv->input_pad = i;
> +		else
> +			priv->output_pad = i;
> +
> +		/* set a default mbus format  */
> +		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
> +					      640, 480, 0, V4L2_FIELD_NONE,
> +					      &priv->cc[i]);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	ret = pp_init_controls(priv);
> +	if (ret)
> +		return ret;
> +
> +	ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +	return ret;
> +}
> +
> +static struct v4l2_subdev_pad_ops pp_pad_ops = {
> +	.enum_mbus_code = pp_enum_mbus_code,
> +	.get_fmt = pp_get_fmt,
> +	.set_fmt = pp_set_fmt,
> +};
> +
> +static struct v4l2_subdev_video_ops pp_video_ops = {
> +	.s_stream = pp_s_stream,
> +};
> +
> +static struct v4l2_subdev_core_ops pp_core_ops = {
> +	.ioctl = pp_ioctl,
> +};
> +
> +static struct media_entity_operations pp_entity_ops = {
> +	.link_setup = pp_link_setup,
> +	.link_validate = v4l2_subdev_link_validate,
> +};
> +
> +static struct v4l2_subdev_ops pp_subdev_ops = {
> +	.video = &pp_video_ops,
> +	.pad = &pp_pad_ops,
> +	.core = &pp_core_ops,
> +};
> +
> +static struct v4l2_subdev_internal_ops pp_internal_ops = {
> +	.registered = pp_registered,
> +};
> +
> +static int pp_init(struct imx_ic_priv *ic_priv)
> +{
> +	struct pp_priv *priv;
> +
> +	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	ic_priv->task_priv = priv;
> +	priv->ic_priv = ic_priv;
> +	spin_lock_init(&priv->irqlock);
> +
> +	/* get our PP id */
> +	priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
> +
> +	return 0;
> +}
> +
> +static void pp_remove(struct imx_ic_priv *ic_priv)
> +{
> +	struct pp_priv *priv = ic_priv->task_priv;
> +
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +}
> +
> +struct imx_ic_ops imx_ic_pp_ops = {
> +	.subdev_ops = &pp_subdev_ops,
> +	.internal_ops = &pp_internal_ops,
> +	.entity_ops = &pp_entity_ops,
> +	.init = pp_init,
> +	.remove = pp_remove,
> +};
> diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
> new file mode 100644
> index 0000000..3d85a82
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic-prpenc.c
> @@ -0,0 +1,1033 @@
> +/*
> + * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
> + *
> + * This subdevice handles capture of video frames from the CSI, which
> + * are routed directly to the Image Converter preprocess encode task,
> + * for resizing, colorspace conversion, and rotation.
> + *
> + * Copyright (c) 2012-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +#define PRPENC_NUM_PADS 2
> +
> +#define MAX_W_IC   1024
> +#define MAX_H_IC   1024
> +#define MAX_W_SINK 4096
> +#define MAX_H_SINK 4096
> +
> +struct prpenc_priv {
> +	struct imx_media_dev *md;
> +	struct imx_ic_priv *ic_priv;
> +
> +	/* IPU units we require */
> +	struct ipu_soc *ipu;
> +	struct ipu_ic *ic_enc;
> +
> +	struct media_pad pad[PRPENC_NUM_PADS];
> +	int input_pad;
> +	int output_pad;
> +
> +	struct ipuv3_channel *enc_ch;
> +	struct ipuv3_channel *enc_rot_in_ch;
> +	struct ipuv3_channel *enc_rot_out_ch;
> +
> +	/* the dma buffer ring to send to sink */
> +	struct imx_media_dma_buf_ring *out_ring;
> +	struct imx_media_dma_buf *next;
> +
> +	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
> +
> +	struct v4l2_subdev *src_sd;
> +	struct v4l2_subdev *sink_sd;
> +
> +	/* the CSI id at link validate */
> +	int csi_id;
> +
> +	/* the attached sensor at stream on */
> +	struct imx_media_subdev *sensor;
> +
> +	struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
> +
> +	struct imx_media_dma_buf rot_buf[2];
> +
> +	/* controls */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +	int  rotation; /* degrees */
> +	bool hflip;
> +	bool vflip;
> +
> +	/* derived from rotation, hflip, vflip controls */
> +	enum ipu_rotate_mode rot_mode;
> +
> +	spinlock_t irqlock;
> +
> +	struct timer_list eof_timeout_timer;
> +	int eof_irq;
> +	int nfb4eof_irq;
> +
> +	bool stream_on; /* streaming is on */
> +	bool last_eof;  /* waiting for last EOF at stream off */
> +	struct completion last_eof_comp;
> +};
> +
> +static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +
> +	return ic_priv->task_priv;
> +}
> +
> +static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
> +{
> +	if (!IS_ERR_OR_NULL(priv->ic_enc))
> +		ipu_ic_put(priv->ic_enc);
> +	priv->ic_enc = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_ch))
> +		ipu_idmac_put(priv->enc_ch);
> +	priv->enc_ch = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
> +		ipu_idmac_put(priv->enc_rot_in_ch);
> +	priv->enc_rot_in_ch = NULL;
> +
> +	if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
> +		ipu_idmac_put(priv->enc_rot_out_ch);
> +	priv->enc_rot_out_ch = NULL;
> +}
> +
> +static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	int ret;
> +
> +	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
> +
> +	priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
> +	if (IS_ERR(priv->ic_enc)) {
> +		v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
> +		ret = PTR_ERR(priv->ic_enc);
> +		goto out;
> +	}
> +
> +	priv->enc_ch = ipu_idmac_get(priv->ipu,
> +				     IPUV3_CHANNEL_IC_PRP_ENC_MEM);
> +	if (IS_ERR(priv->enc_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_IC_PRP_ENC_MEM);
> +		ret = PTR_ERR(priv->enc_ch);
> +		goto out;
> +	}
> +
> +	priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
> +					    IPUV3_CHANNEL_MEM_ROT_ENC);
> +	if (IS_ERR(priv->enc_rot_in_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_MEM_ROT_ENC);
> +		ret = PTR_ERR(priv->enc_rot_in_ch);
> +		goto out;
> +	}
> +
> +	priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
> +					     IPUV3_CHANNEL_ROT_ENC_MEM);
> +	if (IS_ERR(priv->enc_rot_out_ch)) {
> +		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
> +			 IPUV3_CHANNEL_ROT_ENC_MEM);
> +		ret = PTR_ERR(priv->enc_rot_out_ch);
> +		goto out;
> +	}
> +
> +	return 0;
> +out:
> +	prpenc_put_ipu_resources(priv);
> +	return ret;
> +}
> +
> +static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
> +{
> +	struct prpenc_priv *priv = dev_id;
> +	struct imx_media_dma_buf *done, *next;
> +	struct ipuv3_channel *channel;
> +
> +	spin_lock(&priv->irqlock);
> +
> +	if (priv->last_eof) {
> +		complete(&priv->last_eof_comp);
> +		priv->last_eof = false;
> +		goto unlock;
> +	}
> +
> +	/* inform CSI of this EOF so it can monitor frame intervals */
> +	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
> +			 0, NULL);
> +
> +	channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
> +		priv->enc_rot_out_ch : priv->enc_ch;
> +
> +	done = imx_media_dma_buf_get_active(priv->out_ring);
> +	/* give the completed buffer to the sink  */
> +	if (!WARN_ON(!done))
> +		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
> +
> +	/* priv->next buffer is now the active one */
> +	imx_media_dma_buf_set_active(priv->next);
> +
> +	/* bump the EOF timeout timer */
> +	mod_timer(&priv->eof_timeout_timer,
> +		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +
> +	if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
> +		ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);
> +
> +	/* get next queued buffer */
> +	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +
> +	ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
> +	ipu_idmac_select_buffer(channel, priv->ipu_buf_num);
> +
> +	/* toggle IPU double-buffer index */
> +	priv->ipu_buf_num ^= 1;
> +	priv->next = next;
> +
> +unlock:
> +	spin_unlock(&priv->irqlock);
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
> +{
> +	struct prpenc_priv *priv = dev_id;
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	static const struct v4l2_event ev = {
> +		.type = V4L2_EVENT_IMX_NFB4EOF,
> +	};
> +
> +	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
> +
> +	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
> +
> +	return IRQ_HANDLED;
> +}
> +
> +/*
> + * EOF timeout timer function.
> + */
> +static void prpenc_eof_timeout(unsigned long data)
> +{
> +	struct prpenc_priv *priv = (struct prpenc_priv *)data;
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	static const struct v4l2_event ev = {
> +		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
> +	};
> +
> +	v4l2_err(&ic_priv->sd, "EOF timeout\n");
> +
> +	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
> +}
> +
> +static void prpenc_setup_channel(struct prpenc_priv *priv,
> +				 struct ipuv3_channel *channel,
> +				 enum ipu_rotate_mode rot_mode,
> +				 dma_addr_t addr0, dma_addr_t addr1,
> +				 bool rot_swap_width_height)
> +{
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	unsigned int burst_size;
> +	struct ipu_image image;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (rot_swap_width_height)
> +		swap(outfmt->width, outfmt->height);
> +
> +	ipu_cpmem_zero(channel);
> +
> +	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
> +
> +	image.phys0 = addr0;
> +	image.phys1 = addr1;
> +	ipu_cpmem_set_image(channel, &image);
> +
> +	if (channel == priv->enc_rot_in_ch ||
> +	    channel == priv->enc_rot_out_ch) {
> +		burst_size = 8;
> +		ipu_cpmem_set_block_mode(channel);
> +	} else {
> +		burst_size = (outfmt->width & 0xf) ? 8 : 16;
> +	}
> +
> +	ipu_cpmem_set_burstsize(channel, burst_size);
> +
> +	if (rot_mode)
> +		ipu_cpmem_set_rotation(channel, rot_mode);
> +
> +	if (outfmt->field == V4L2_FIELD_NONE &&
> +	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
> +	     infmt->field == V4L2_FIELD_ALTERNATE) &&
> +	    channel == priv->enc_ch)
> +		ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
> +
> +	ipu_ic_task_idma_init(priv->ic_enc, channel,
> +			      outfmt->width, outfmt->height,
> +			      burst_size, rot_mode);
> +	ipu_cpmem_set_axi_id(channel, 1);
> +
> +	ipu_idmac_set_double_buffer(channel, true);
> +
> +	if (rot_swap_width_height)
> +		swap(outfmt->width, outfmt->height);
> +}
> +
> +static int prpenc_setup_rotation(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *outcc, *incc;
> +	struct imx_media_dma_buf *buf0, *buf1;
> +	int out_size, ret;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	incc = priv->cc[priv->input_pad];
> +	outcc = priv->cc[priv->output_pad];
> +
> +	out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
> +
> +	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
> +		return ret;
> +	}
> +	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
> +		goto free_rot0;
> +	}
> +
> +	ret = ipu_ic_task_init(priv->ic_enc,
> +			       infmt->width, infmt->height,
> +			       outfmt->height, outfmt->width,
> +			       incc->cs, outcc->cs);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
> +		goto free_rot1;
> +	}
> +
> +	/* init the IC ENC-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_ch,
> +			     IPU_ROTATE_NONE,
> +			     priv->rot_buf[0].phys,
> +			     priv->rot_buf[1].phys,
> +			     true);
> +
> +	/* init the MEM-->IC ENC ROT IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_rot_in_ch,
> +			     priv->rot_mode,
> +			     priv->rot_buf[0].phys,
> +			     priv->rot_buf[1].phys,
> +			     true);
> +
> +	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(buf0);
> +	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	priv->next = buf1;
> +
> +	/* init the destination IC ENC ROT-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_rot_out_ch,
> +			     IPU_ROTATE_NONE,
> +			     buf0->phys, buf1->phys,
> +			     false);
> +
> +	/* now link IC ENC-->MEM to MEM-->IC ENC ROT */
> +	ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
> +
> +	/* enable the IC */
> +	ipu_ic_enable(priv->ic_enc);
> +
> +	/* set buffers ready */
> +	ipu_idmac_select_buffer(priv->enc_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_ch, 1);
> +	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);
> +
> +	/* enable the channels */
> +	ipu_idmac_enable_channel(priv->enc_ch);
> +	ipu_idmac_enable_channel(priv->enc_rot_in_ch);
> +	ipu_idmac_enable_channel(priv->enc_rot_out_ch);
> +
> +	/* and finally enable the IC PRPENC task */
> +	ipu_ic_task_enable(priv->ic_enc);
> +
> +	return 0;
> +
> +free_rot1:
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
> +free_rot0:
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
> +	return ret;
> +}
> +
> +static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
> +{
> +	ipu_ic_task_disable(priv->ic_enc);
> +
> +	ipu_idmac_disable_channel(priv->enc_ch);
> +	ipu_idmac_disable_channel(priv->enc_rot_in_ch);
> +	ipu_idmac_disable_channel(priv->enc_rot_out_ch);
> +
> +	ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
> +
> +	ipu_ic_disable(priv->ic_enc);
> +
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
> +	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
> +}
> +
> +static int prpenc_setup_norotation(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *outcc, *incc;
> +	struct imx_media_dma_buf *buf0, *buf1;
> +	int ret;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	incc = priv->cc[priv->input_pad];
> +	outcc = priv->cc[priv->output_pad];
> +
> +	ret = ipu_ic_task_init(priv->ic_enc,
> +			       infmt->width, infmt->height,
> +			       outfmt->width, outfmt->height,
> +			       incc->cs, outcc->cs);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
> +		return ret;
> +	}
> +
> +	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	imx_media_dma_buf_set_active(buf0);
> +	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
> +	priv->next = buf1;
> +
> +	/* init the IC PRP-->MEM IDMAC channel */
> +	prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
> +			     buf0->phys, buf1->phys,
> +			     false);
> +
> +	ipu_cpmem_dump(priv->enc_ch);
> +	ipu_ic_dump(priv->ic_enc);
> +	ipu_dump(priv->ipu);
> +
> +	ipu_ic_enable(priv->ic_enc);
> +
> +	/* set buffers ready */
> +	ipu_idmac_select_buffer(priv->enc_ch, 0);
> +	ipu_idmac_select_buffer(priv->enc_ch, 1);
> +
> +	/* enable the channels */
> +	ipu_idmac_enable_channel(priv->enc_ch);
> +
> +	/* enable the IC ENCODE task */
> +	ipu_ic_task_enable(priv->ic_enc);
> +
> +	return 0;
> +}
> +
> +static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
> +{
> +	ipu_ic_task_disable(priv->ic_enc);
> +	ipu_idmac_disable_channel(priv->enc_ch);
> +	ipu_ic_disable(priv->ic_enc);
> +}
> +
> +static int prpenc_start(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	int ret;
> +
> +	if (!priv->sensor) {
> +		v4l2_err(&ic_priv->sd, "no sensor attached\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = prpenc_get_ipu_resources(priv);
> +	if (ret)
> +		return ret;
> +
> +	/* set IC to receive from CSI */
> +	ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
> +
> +	/* ask the sink for the buffer ring */
> +	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
> +			       &priv->out_ring);
> +	if (ret)
> +		goto out_put_ipu;
> +
> +	priv->ipu_buf_num = 0;
> +
> +	/* init EOF completion waitq */
> +	init_completion(&priv->last_eof_comp);
> +	priv->last_eof = false;
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		ret = prpenc_setup_rotation(priv);
> +	else
> +		ret = prpenc_setup_norotation(priv);
> +	if (ret)
> +		goto out_put_ipu;
> +
> +	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
> +						  priv->enc_ch,
> +						  IPU_IRQ_NFB4EOF);
> +	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
> +			       prpenc_nfb4eof_interrupt, 0,
> +			       "imx-ic-prpenc-nfb4eof", priv);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd,
> +			 "Error registering NFB4EOF irq: %d\n", ret);
> +		goto out_unsetup;
> +	}
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		priv->eof_irq = ipu_idmac_channel_irq(
> +			priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
> +	else
> +		priv->eof_irq = ipu_idmac_channel_irq(
> +			priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
> +
> +	ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
> +			       prpenc_eof_interrupt, 0,
> +			       "imx-ic-prpenc-eof", priv);
> +	if (ret) {
> +		v4l2_err(&ic_priv->sd,
> +			 "Error registering eof irq: %d\n", ret);
> +		goto out_free_nfb4eof_irq;
> +	}
> +
> +	/* start the EOF timeout timer */
> +	mod_timer(&priv->eof_timeout_timer,
> +		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +
> +	return 0;
> +
> +out_free_nfb4eof_irq:
> +	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
> +out_unsetup:
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		prpenc_unsetup_rotation(priv);
> +	else
> +		prpenc_unsetup_norotation(priv);
> +out_put_ipu:
> +	prpenc_put_ipu_resources(priv);
> +	return ret;
> +}
> +
> +static void prpenc_stop(struct prpenc_priv *priv)
> +{
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	unsigned long flags;
> +	int ret;
> +
> +	/* mark next EOF interrupt as the last before stream off */
> +	spin_lock_irqsave(&priv->irqlock, flags);
> +	priv->last_eof = true;
> +	spin_unlock_irqrestore(&priv->irqlock, flags);
> +
> +	/*
> +	 * and then wait for interrupt handler to mark completion.
> +	 */
> +	ret = wait_for_completion_timeout(
> +		&priv->last_eof_comp,
> +		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
> +	if (ret == 0)
> +		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
> +
> +	devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
> +	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
> +
> +	if (ipu_rot_mode_is_irt(priv->rot_mode))
> +		prpenc_unsetup_rotation(priv);
> +	else
> +		prpenc_unsetup_norotation(priv);
> +
> +	prpenc_put_ipu_resources(priv);
> +
> +	/* cancel the EOF timeout timer */
> +	del_timer_sync(&priv->eof_timeout_timer);
> +
> +	priv->out_ring = NULL;
> +
> +	/* inform sink that the buffer ring can now be freed */
> +	v4l2_subdev_call(priv->sink_sd, core, ioctl,
> +			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
> +}
> +
> +static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +	bool allow_planar;
> +
> +	if (code->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	allow_planar = (code->pad == priv->output_pad);
> +
> +	return imx_media_enum_format(&code->code, code->index,
> +				     true, allow_planar);
> +}
> +
> +static int prpenc_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *sdformat)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +
> +	if (sdformat->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int prpenc_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *sdformat)
> +{
> +	struct prpenc_priv *priv = sd_to_priv(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	const struct imx_media_pixfmt *cc;
> +	bool allow_planar;
> +	u32 code;
> +
> +	if (sdformat->pad >= PRPENC_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	allow_planar = (sdformat->pad == priv->output_pad);
> +
> +	cc = imx_media_find_format(0, sdformat->format.code,
> +				   true, allow_planar);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, false);
> +		cc = imx_media_find_format(0, code, true, false);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.width = min_t(__u32,
> +					       sdformat->format.width,
> +					       MAX_W_IC);
> +		sdformat->format.height = min_t(__u32,
> +						sdformat->format.height,
> +						MAX_H_IC);
> +
> +		if (sdformat->format.field != V4L2_FIELD_NONE)
> +			sdformat->format.field = infmt->field;
> +
> +		/* IC resizer cannot downsize more than 4:1 */
> +		if (ipu_rot_mode_is_irt(priv->rot_mode)) {
> +			sdformat->format.width = max_t(__u32,
> +						       sdformat->format.width,
> +						       infmt->height / 4);
> +			sdformat->format.height = max_t(__u32,
> +							sdformat->format.height,
> +							infmt->width / 4);
> +		} else {
> +			sdformat->format.width = max_t(__u32,
> +						       sdformat->format.width,
> +						       infmt->width / 4);
> +			sdformat->format.height = max_t(__u32,
> +							sdformat->format.height,
> +							infmt->height / 4);
> +		}
> +	} else {
> +		sdformat->format.width = min_t(__u32,
> +					       sdformat->format.width,
> +					       MAX_W_SINK);
> +		sdformat->format.height = min_t(__u32,
> +						sdformat->format.height,
> +						MAX_H_SINK);
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_link_setup(struct media_entity *entity,
> +			     const struct media_pad *local,
> +			     const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct prpenc_priv *priv = ic_priv->task_priv;
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (priv->sink_sd)
> +				return -EBUSY;
> +			priv->sink_sd = remote_sd;
> +		} else {
> +			priv->sink_sd = NULL;
> +		}
> +
> +		return 0;
> +	}
> +
> +	/* this is sink pad */
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->src_sd)
> +			return -EBUSY;
> +		priv->src_sd = remote_sd;
> +	} else {
> +		priv->src_sd = NULL;
> +		return 0;
> +	}
> +
> +	switch (remote_sd->grp_id) {
> +	case IMX_MEDIA_GRP_ID_CSI0:
> +		priv->csi_id = 0;
> +		break;
> +	case IMX_MEDIA_GRP_ID_CSI1:
> +		priv->csi_id = 1;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_link_validate(struct v4l2_subdev *sd,
> +				struct media_link *link,
> +				struct v4l2_subdev_format *source_fmt,
> +				struct v4l2_subdev_format *sink_fmt)
> +{
> +	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
> +	struct prpenc_priv *priv = ic_priv->task_priv;
> +	struct v4l2_mbus_config sensor_mbus_cfg;
> +	int ret;
> +
> +	ret = v4l2_subdev_link_validate_default(sd, link,
> +						source_fmt, sink_fmt);
> +	if (ret)
> +		return ret;
> +
> +	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
> +	if (IS_ERR(priv->sensor)) {
> +		v4l2_err(&ic_priv->sd, "no sensor attached\n");
> +		ret = PTR_ERR(priv->sensor);
> +		priv->sensor = NULL;
> +		return ret;
> +	}
> +
> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> +			       &sensor_mbus_cfg);
> +	if (ret)
> +		return ret;
> +
> +	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
> +		int vc_num = 0;
> +		/* see NOTE in imx-csi.c */
> +#if 0
> +		vc_num = imx_media_find_mipi_csi2_channel(
> +			priv->md, &ic_priv->sd.entity);
> +		if (vc_num < 0)
> +			return vc_num;
> +#endif
> +		/* only virtual channel 0 can be sent to IC */
> +		if (vc_num != 0)
> +			return -EINVAL;
> +	} else {
> +		/*
> +		 * only 8-bit pixels can be sent to IC for parallel
> +		 * busses
> +		 */
> +		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct prpenc_priv *priv = container_of(ctrl->handler,
> +					       struct prpenc_priv, ctrl_hdlr);
> +	struct imx_ic_priv *ic_priv = priv->ic_priv;
> +	enum ipu_rotate_mode rot_mode;
> +	bool hflip, vflip;
> +	int rotation, ret;
> +
> +	rotation = priv->rotation;
> +	hflip = priv->hflip;
> +	vflip = priv->vflip;
> +
> +	switch (ctrl->id) {
> +	case V4L2_CID_HFLIP:
> +		hflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_VFLIP:
> +		vflip = (ctrl->val == 1);
> +		break;
> +	case V4L2_CID_ROTATE:
> +		rotation = ctrl->val;
> +		break;
> +	default:
> +		v4l2_err(&ic_priv->sd, "Invalid control\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
> +	if (ret)
> +		return ret;
> +
> +	if (rot_mode != priv->rot_mode) {
> +		/* can't change rotation mid-streaming */
> +		if (priv->stream_on)
> +			return -EBUSY;
> +
> +		priv->rot_mode = rot_mode;
> +		priv->rotation = rotation;
> +		priv->hflip = hflip;
> +		priv->vflip = vflip;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
> +	.s_ctrl = prpenc_s_ctrl,
> +};
> +
> +static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
> +	{
> +		.id = V4L2_CID_HFLIP,
> +		.name = "Horizontal Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_VFLIP,
> +		.name = "Vertical Flip",
> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		.def =  0,
> +		.min =  0,
> +		.max =  1,
> +		.step = 1,
> +	}, {
> +		.id = V4L2_CID_ROTATE,
> +		.name = "Rotation",
> +		.type = V4L2_CTRL_TYPE_INTEGER,
> +		.def =   0,
> +		.min =   0,
> +		.max = 270,
> +		.step = 90,
> +	},
> +};

Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information
like the name and type.

If this is also done elsewhere, then it should be changed there as well.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-20 14:38     ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:38 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This is the camera interface driver that provides the v4l2
> user interface. Frames can be received from various sources:
> 
> - directly from SMFC for capturing unconverted images directly from
>   camera sensors.
> 
> - from the IC pre-process encode task.
> 
> - from the IC pre-process viewfinder task.
> 
> - from the IC post-process task.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile    |    2 +-
>  drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
>  2 files changed, 1001 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/staging/media/imx/imx-camif.c
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index d2a962c..fe9e992 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> -
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
> new file mode 100644
> index 0000000..404f724
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-camif.c
> @@ -0,0 +1,1000 @@
> +/*
> + * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2012-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/module.h>
> +#include <linux/of_platform.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +
> +#define CAMIF_NUM_PADS 2
> +
> +struct camif_priv {
> +	struct device         *dev;
> +	struct video_device    vfd;
> +	struct media_pipeline  mp;
> +	struct imx_media_dev  *md;
> +	struct v4l2_subdev     sd;
> +	struct media_pad       pad[CAMIF_NUM_PADS];
> +	struct media_pad       vd_pad;
> +	int id;
> +	int input_pad;
> +	int output_pad;
> +
> +	struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
> +
> +	/* dma buffer ring */
> +	struct imx_media_dma_buf_ring *in_ring;
> +	struct v4l2_subdev     *src_sd;
> +
> +	struct mutex           mutex;       /* capture device mutex */
> +	spinlock_t             q_lock;      /* protect ready_q */
> +
> +	/* buffer queue used in videobuf2 */
> +	struct vb2_queue       buffer_queue;
> +
> +	/* streaming buffer queue */
> +	struct list_head       ready_q;
> +
> +	/* controls inherited from subdevs */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +
> +	/* misc status */
> +	int                    current_input; /* the current input */
> +	v4l2_std_id            current_std;   /* current standard */
> +	bool                   stop;          /* streaming is stopping */
> +};
> +
> +/* In bytes, per queue */
> +#define VID_MEM_LIMIT	SZ_64M
> +
> +static struct vb2_ops camif_qops;
> +
> +/*
> + * Video ioctls follow
> + */
> +
> +static int vidioc_querycap(struct file *file, void *fh,
> +			   struct v4l2_capability *cap)
> +{
> +	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
> +	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
> +	cap->bus_info[0] = 0;

Should be set to something like 'platform:imx-media-camif'. v4l2-compliance should
complain about this.

> +	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
> +	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

Set device_caps in struct video_device, then drop these two lines since
the core will set these up based on the device_caps field in struct video_device.

> +
> +	return 0;
> +}
> +
> +static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
> +				  struct v4l2_fmtdesc *f)
> +{
> +	const struct imx_media_pixfmt *cc;
> +	u32 code;
> +	int ret;
> +
> +	ret = imx_media_enum_format(&code, f->index, true, true);
> +	if (ret)
> +		return ret;
> +	cc = imx_media_find_format(0, code, true, true);
> +	if (!cc)
> +		return -EINVAL;
> +
> +	f->pixelformat = cc->fourcc;
> +
> +	return 0;
> +}
> +
> +static int camif_g_fmt_vid_cap(struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct v4l2_mbus_framefmt *outfmt;
> +
> +	/* user format is the same as the format from output pad */
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
> +}
> +
> +static int camif_try_fmt_vid_cap(struct file *file, void *fh,
> +				 struct v4l2_format *f)
> +{
> +	return camif_g_fmt_vid_cap(file, fh, f);
> +}
> +
> +static int camif_s_fmt_vid_cap(struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	if (vb2_is_busy(&priv->buffer_queue)) {
> +		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
> +		return -EBUSY;
> +	}
> +
> +	return camif_try_fmt_vid_cap(file, priv, f);
> +}
> +
> +static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	return v4l2_subdev_call(sensor->sd, video, querystd, std);
> +}
> +
> +static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	*std = priv->current_std;
> +	return 0;
> +}
> +
> +static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int ret;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (vb2_is_busy(&priv->buffer_queue))
> +		return -EBUSY;
> +
> +	ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
> +	if (ret < 0)
> +		return ret;
> +
> +	priv->current_std = std;
> +	return 0;
> +}
> +
> +static int camif_enum_input(struct file *file, void *fh,
> +			    struct v4l2_input *input)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int index = input->index;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (index >= sensor->input.num)
> +		return -EINVAL;
> +
> +	input->type = V4L2_INPUT_TYPE_CAMERA;
> +	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
> +
> +	if (index == priv->current_input) {
> +		v4l2_subdev_call(sensor->sd, video, g_input_status,
> +				 &input->status);
> +		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);

Wrong op, use g_tvnorms instead.

> +	} else {
> +		input->status = V4L2_IN_ST_NO_SIGNAL;
> +		input->std = V4L2_STD_UNKNOWN;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_g_input(struct file *file, void *fh, unsigned int *index)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	*index = priv->current_input;
> +	return 0;
> +}
> +
> +static int camif_s_input(struct file *file, void *fh, unsigned int index)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int ret;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (index >= sensor->input.num)
> +		return -EINVAL;
> +
> +	if (index == priv->current_input)
> +		return 0;
> +
> +	/* select the sensor's input */
> +	ret = v4l2_subdev_call(sensor->sd, video, s_routing,
> +			       sensor->input.value[index], 0, 0);
> +	if (!ret)
> +		priv->current_input = index;
> +
> +	return ret;
> +}
> +
> +static int camif_g_parm(struct file *file, void *fh,
> +			struct v4l2_streamparm *a)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	return v4l2_subdev_call(sensor->sd, video, g_parm, a);
> +}
> +
> +static int camif_s_parm(struct file *file, void *fh,
> +			struct v4l2_streamparm *a)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	return v4l2_subdev_call(sensor->sd, video, s_parm, a);
> +}
> +
> +static const struct v4l2_ioctl_ops camif_ioctl_ops = {
> +	.vidioc_querycap	= vidioc_querycap,
> +
> +	.vidioc_enum_fmt_vid_cap        = camif_enum_fmt_vid_cap,
> +	.vidioc_g_fmt_vid_cap           = camif_g_fmt_vid_cap,
> +	.vidioc_try_fmt_vid_cap         = camif_try_fmt_vid_cap,
> +	.vidioc_s_fmt_vid_cap           = camif_s_fmt_vid_cap,
> +
> +	.vidioc_querystd        = camif_querystd,
> +	.vidioc_g_std           = camif_g_std,
> +	.vidioc_s_std           = camif_s_std,
> +
> +	.vidioc_enum_input      = camif_enum_input,
> +	.vidioc_g_input         = camif_g_input,
> +	.vidioc_s_input         = camif_s_input,
> +
> +	.vidioc_g_parm          = camif_g_parm,
> +	.vidioc_s_parm          = camif_s_parm,
> +
> +	.vidioc_reqbufs		= vb2_ioctl_reqbufs,
> +	.vidioc_create_bufs     = vb2_ioctl_create_bufs,
> +	.vidioc_prepare_buf     = vb2_ioctl_prepare_buf,
> +	.vidioc_querybuf	= vb2_ioctl_querybuf,
> +	.vidioc_qbuf		= vb2_ioctl_qbuf,
> +	.vidioc_dqbuf		= vb2_ioctl_dqbuf,
> +	.vidioc_expbuf		= vb2_ioctl_expbuf,
> +	.vidioc_streamon	= vb2_ioctl_streamon,
> +	.vidioc_streamoff	= vb2_ioctl_streamoff,
> +};
> +
> +/*
> + * Queue operations
> + */
> +
> +static u32 camif_get_sizeimage(struct camif_priv *priv)
> +{
> +	struct v4l2_mbus_framefmt *outfmt;
> +	const struct imx_media_pixfmt *outcc;
> +
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	outcc = priv->cc[priv->output_pad];
> +	return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
> +}
> +
> +static int camif_queue_setup(struct vb2_queue *vq,
> +			     unsigned int *nbuffers, unsigned int *nplanes,
> +			     unsigned int sizes[], struct device *alloc_devs[])
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	unsigned int count = *nbuffers;
> +	u32 sizeimage;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	sizeimage = camif_get_sizeimage(priv);
> +
> +	count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
> +	count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
> +
> +	while (sizeimage * count > VID_MEM_LIMIT)
> +		count--;
> +
> +	if (count < IMX_MEDIA_MIN_RING_BUFS)
> +		return -EINVAL;
> +
> +	*nplanes = 1;
> +	*nbuffers = count;
> +	sizes[0] = sizeimage;
> +
> +	return 0;
> +}
> +
> +static int camif_buf_init(struct vb2_buffer *vb)
> +{
> +	struct vb2_queue *vq = vb->vb2_queue;
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	struct imx_media_buffer *buf = to_imx_media_vb(vb);
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	INIT_LIST_HEAD(&buf->list);
> +
> +	return 0;
> +}
> +
> +static int camif_buf_prepare(struct vb2_buffer *vb)
> +{
> +	struct vb2_queue *vq = vb->vb2_queue;
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	u32 sizeimage = camif_get_sizeimage(priv);
> +	int ret;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (vb2_plane_size(vb, 0) < sizeimage) {
> +		v4l2_err(&priv->sd,
> +			 "data will not fit into plane (%lu < %lu)\n",
> +			 vb2_plane_size(vb, 0), (long)sizeimage);
> +		return -EINVAL;
> +	}
> +
> +	vb2_set_plane_payload(vb, 0, sizeimage);
> +
> +	if (!priv->in_ring) {
> +		priv->in_ring = imx_media_alloc_dma_buf_ring(
> +			priv->md, &priv->src_sd->entity, &priv->sd.entity,
> +			sizeimage, vq->num_buffers, false);
> +		if (IS_ERR(priv->in_ring)) {
> +			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
> +			ret = PTR_ERR(priv->in_ring);
> +			priv->in_ring = NULL;
> +			return ret;
> +		}
> +	}
> +
> +	ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
> +	if (ret)
> +		goto free_ring;
> +
> +	return 0;
> +
> +free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +	return ret;
> +}
> +
> +static void camif_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
> +	struct imx_media_buffer *buf = to_imx_media_vb(vb);
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +
> +	list_add_tail(&buf->list, &priv->ready_q);
> +
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	u32 sizeimage = camif_get_sizeimage(priv);
> +	struct imx_media_buffer *buf, *tmp;
> +	unsigned long flags;
> +	int ret;
> +
> +	if (vb2_is_streaming(vq))
> +		return 0;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (!priv->in_ring) {
> +		priv->in_ring = imx_media_alloc_dma_buf_ring(
> +			priv->md, &priv->src_sd->entity, &priv->sd.entity,
> +			sizeimage, vq->num_buffers, false);
> +		if (IS_ERR(priv->in_ring)) {
> +			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
> +			ret = PTR_ERR(priv->in_ring);
> +			priv->in_ring = NULL;
> +			goto return_bufs;
> +		}
> +	}
> +
> +	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
> +					    &priv->mp, true);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
> +			 ret);
> +		goto free_ring;
> +	}
> +
> +	priv->stop = false;
> +
> +	return 0;
> +
> +free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +return_bufs:
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
> +		list_del(&buf->list);
> +		vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
> +	}
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +	return ret;
> +}
> +
> +static void camif_stop_streaming(struct vb2_queue *vq)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	struct imx_media_buffer *frame;
> +	unsigned long flags;
> +	int ret;
> +
> +	if (!vb2_is_streaming(vq))
> +		return;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	priv->stop = true;
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +
> +	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
> +					    &priv->mp, false);
> +	if (ret)
> +		v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
> +			  ret);
> +
> +	if (priv->in_ring) {
> +		v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
> +			  __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +	}
> +
> +	/* release all active buffers */
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	while (!list_empty(&priv->ready_q)) {
> +		frame = list_entry(priv->ready_q.next,
> +				   struct imx_media_buffer, list);
> +		list_del(&frame->list);
> +		vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
> +	}
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static struct vb2_ops camif_qops = {
> +	.queue_setup	 = camif_queue_setup,
> +	.buf_init        = camif_buf_init,
> +	.buf_prepare	 = camif_buf_prepare,
> +	.buf_queue	 = camif_buf_queue,
> +	.wait_prepare	 = vb2_ops_wait_prepare,
> +	.wait_finish	 = vb2_ops_wait_finish,
> +	.start_streaming = camif_start_streaming,
> +	.stop_streaming  = camif_stop_streaming,
> +};
> +
> +/*
> + * File operations
> + */
> +static int camif_open(struct file *file)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	int ret;
> +
> +	if (mutex_lock_interruptible(&priv->mutex))
> +		return -ERESTARTSYS;
> +
> +	ret = v4l2_fh_open(file);
> +	if (ret)
> +		v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
> +
> +	mutex_unlock(&priv->mutex);
> +	return ret;
> +}
> +
> +static int camif_release(struct file *file)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct vb2_queue *vq = &priv->buffer_queue;
> +	int ret = 0;
> +
> +	mutex_lock(&priv->mutex);
> +
> +	if (file->private_data == vq->owner) {
> +		vb2_queue_release(vq);
> +		vq->owner = NULL;
> +	}
> +
> +	v4l2_fh_release(file);
> +	mutex_unlock(&priv->mutex);
> +	return ret;
> +}
> +
> +static const struct v4l2_file_operations camif_fops = {
> +	.owner		= THIS_MODULE,
> +	.open		= camif_open,
> +	.release	= camif_release,
> +	.poll		= vb2_fop_poll,
> +	.unlocked_ioctl	= video_ioctl2,
> +	.mmap		= vb2_fop_mmap,
> +};
> +
> +static struct video_device camif_videodev = {
> +	.fops		= &camif_fops,
> +	.ioctl_ops	= &camif_ioctl_ops,
> +	.minor		= -1,
> +	.release	= video_device_release,
> +	.vfl_dir	= VFL_DIR_RX,
> +	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
> +};
> +
> +/*
> + * Subdev and media entity operations
> + */
> +
> +static void camif_new_dma_buf(struct camif_priv *priv)
> +{
> +	struct imx_media_dma_buf *dmabuf;
> +	struct imx_media_buffer *buf;
> +	enum vb2_buffer_state state;
> +	struct vb2_buffer *vb;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +
> +	if (priv->stop || list_empty(&priv->ready_q))
> +		goto unlock;
> +
> +	dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
> +	if (!dmabuf)
> +		goto unlock;
> +
> +	vb = dmabuf->vb;
> +	buf = to_imx_media_vb(vb);
> +	if (list_empty(&buf->list)) {
> +		dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
> +			vb->index);
> +		goto unlock;
> +	}
> +
> +	dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
> +	vb->timestamp = ktime_get_ns();
> +	list_del_init(&buf->list);
> +	state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
> +		VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
> +	vb2_buffer_done(vb, state);
> +unlock:
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct imx_media_dma_buf_ring **ring;
> +
> +	switch (cmd) {
> +	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
> +		if (!priv->in_ring)
> +			return -EINVAL;
> +		ring = (struct imx_media_dma_buf_ring **)arg;
> +		*ring = priv->in_ring;
> +		break;
> +	case IMX_MEDIA_NEW_DMA_BUF:
> +		camif_new_dma_buf(priv);
> +		break;
> +	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
> +		/* src indicates sink buffer ring can be freed */
> +		if (!priv->in_ring)
> +			return 0;
> +		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_link_setup(struct media_entity *entity,
> +			    const struct media_pad *local,
> +			    const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct video_device *vfd = &priv->vfd;
> +	struct v4l2_subdev *remote_sd;
> +	int ret = 0;
> +
> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	if (is_media_entity_v4l2_video_device(remote->entity))
> +		return 0;
> +
> +	WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	/* reset controls to refresh with inherited from subdevs */
> +	v4l2_ctrl_handler_free(vfd->ctrl_handler);
> +	v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
> +
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->src_sd)
> +			return -EBUSY;
> +		priv->src_sd = remote_sd;
> +		ret = imx_media_inherit_controls(priv->md, vfd,
> +						 &priv->src_sd->entity);
> +	} else {
> +		priv->src_sd = NULL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int camif_enum_mbus_code(struct v4l2_subdev *sd,
> +				struct v4l2_subdev_pad_config *cfg,
> +				struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	if (code->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	return imx_media_enum_format(&code->code, code->index, true, true);
> +}
> +
> +static int camif_get_fmt(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_pad_config *cfg,
> +			 struct v4l2_subdev_format *sdformat)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +
> +	if (sdformat->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int camif_set_fmt(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_pad_config *cfg,
> +			 struct v4l2_subdev_format *sdformat)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	const struct imx_media_pixfmt *cc;
> +	u32 code;
> +
> +	if (sdformat->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (vb2_is_busy(&priv->buffer_queue)) {
> +		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
> +		return -EBUSY;
> +	}
> +
> +	cc = imx_media_find_format(0, sdformat->format.code, true, true);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, true);
> +		cc = imx_media_find_format(0, code, true, true);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	/* Output pad mirrors input pad, no limitations on input pads */
> +	if (sdformat->pad == priv->output_pad)
> +		sdformat->format = priv->format_mbus[priv->input_pad];
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_init_pads(struct camif_priv *priv)
> +{
> +	struct video_device *vfd = &priv->vfd;
> +	struct imx_media_subdev *imxsd;
> +	struct imx_media_pad *pad;
> +	int i, ret;
> +
> +	imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
> +	if (IS_ERR(imxsd))
> +		return PTR_ERR(imxsd);
> +
> +	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
> +		v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
> +			 imxsd->num_sink_pads, imxsd->num_src_pads);
> +		return -EINVAL;
> +	}
> +
> +	priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
> +	ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "failed to init device node pad\n");
> +		return ret;
> +	}
> +
> +	for (i = 0; i < CAMIF_NUM_PADS; i++) {
> +		pad = &imxsd->pad[i];
> +		priv->pad[i] = pad->pad;
> +		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
> +			priv->input_pad = i;
> +		else
> +			priv->output_pad = i;
> +
> +		/* set a default mbus format  */
> +		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
> +					      640, 480, 0, V4L2_FIELD_NONE,
> +					      &priv->cc[i]);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
> +				      priv->pad);
> +}
> +
> +static int camif_registered(struct v4l2_subdev *sd)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct vb2_queue *vq = &priv->buffer_queue;
> +	struct video_device *vfd = &priv->vfd;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	int ret;
> +
> +	/* get media device */
> +	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
> +
> +	vfd->v4l2_dev = sd->v4l2_dev;
> +
> +	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
> +	if (ret) {
> +		v4l2_err(sd, "Failed to register video device\n");
> +		return ret;
> +	}
> +
> +	vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	vq->io_modes = VB2_MMAP | VB2_DMABUF;
> +	vq->drv_priv = priv;
> +	vq->buf_struct_size = sizeof(struct imx_media_buffer);
> +	vq->ops = &camif_qops;
> +	vq->mem_ops = &vb2_dma_contig_memops;
> +	vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +	vq->lock = &priv->mutex;
> +	vq->min_buffers_needed = 2;
> +	vq->dev = priv->dev;
> +
> +	ret = vb2_queue_init(vq);
> +	if (ret) {
> +		v4l2_err(sd, "vb2_queue_init failed\n");
> +		goto unreg;
> +	}
> +
> +	INIT_LIST_HEAD(&priv->ready_q);
> +
> +	ret = camif_init_pads(priv);
> +	if (ret) {
> +		v4l2_err(sd, "camif_init_pads failed\n");
> +		goto unreg;
> +	}
> +
> +	/* create the link to our device node */
> +	ret = media_create_pad_link(&sd->entity, priv->output_pad,
> +				    &vfd->entity, 0,
> +				    MEDIA_LNK_FL_IMMUTABLE |
> +				    MEDIA_LNK_FL_ENABLED);
> +	if (ret) {
> +		v4l2_err(sd, "failed to create link to device node\n");
> +		goto unreg;
> +	}
> +
> +	/* setup default pad formats */
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
> +				      &priv->cc[priv->output_pad]);
> +	if (ret)
> +		goto unreg;
> +
> +	*infmt = *outfmt;
> +	priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
> +
> +	priv->current_std = V4L2_STD_UNKNOWN;
> +
> +	v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
> +		  video_device_node_name(vfd));
> +
> +	vfd->ctrl_handler = &priv->ctrl_hdlr;
> +
> +	return 0;
> +unreg:
> +	video_unregister_device(vfd);
> +	return ret;
> +}
> +
> +static void camif_unregistered(struct v4l2_subdev *sd)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct video_device *vfd = &priv->vfd;
> +
> +	mutex_lock(&priv->mutex);
> +
> +	if (video_is_registered(vfd)) {
> +		video_unregister_device(vfd);
> +		media_entity_cleanup(&vfd->entity);
> +	}
> +
> +	mutex_unlock(&priv->mutex);
> +}
> +
> +static const struct v4l2_subdev_internal_ops camif_internal_ops = {
> +	.registered = camif_registered,
> +	.unregistered = camif_unregistered,
> +};
> +
> +static struct v4l2_subdev_core_ops camif_core_ops = {
> +	.ioctl = camif_ioctl,
> +};
> +
> +static struct media_entity_operations camif_entity_ops = {
> +	.link_setup = camif_link_setup,
> +	.link_validate = v4l2_subdev_link_validate,
> +};
> +
> +static struct v4l2_subdev_pad_ops camif_pad_ops = {
> +	.enum_mbus_code = camif_enum_mbus_code,
> +	.get_fmt = camif_get_fmt,
> +	.set_fmt = camif_set_fmt,
> +};
> +
> +static struct v4l2_subdev_ops camif_subdev_ops = {
> +	.pad = &camif_pad_ops,
> +	.core = &camif_core_ops,
> +};
> +
> +static int camif_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct camif_priv *priv;
> +	struct video_device *vfd;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, priv);
> +	priv->dev = &pdev->dev;
> +
> +	pdata = priv->dev->platform_data;
> +
> +	mutex_init(&priv->mutex);
> +	spin_lock_init(&priv->q_lock);
> +
> +	v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
> +	v4l2_set_subdevdata(&priv->sd, priv);
> +	priv->sd.internal_ops = &camif_internal_ops;
> +	priv->sd.entity.ops = &camif_entity_ops;
> +	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> +	priv->sd.dev = &pdev->dev;
> +	priv->sd.owner = THIS_MODULE;
> +	/* get our group id and camif id */
> +	priv->sd.grp_id = pdata->grp_id;
> +	priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
> +	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
> +	snprintf(camif_videodev.name, sizeof(camif_videodev.name),
> +		 "%s devnode", pdata->sd_name);
> +
> +	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +
> +	vfd = &priv->vfd;
> +	*vfd = camif_videodev;
> +	vfd->lock = &priv->mutex;
> +	vfd->queue = &priv->buffer_queue;
> +
> +	video_set_drvdata(vfd, priv);
> +
> +	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
> +
> +	ret = v4l2_async_register_subdev(&priv->sd);
> +	if (ret)
> +		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +
> +	return ret;
> +}
> +
> +static int camif_remove(struct platform_device *pdev)
> +{
> +	struct camif_priv *priv =
> +		(struct camif_priv *)platform_get_drvdata(pdev);
> +
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +	v4l2_async_unregister_subdev(&priv->sd);
> +	media_entity_cleanup(&priv->sd.entity);
> +	v4l2_device_unregister_subdev(&priv->sd);
> +
> +	return 0;
> +}
> +
> +static const struct platform_device_id camif_ids[] = {
> +	{ .name = "imx-media-camif" },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(platform, camif_ids);
> +
> +static struct platform_driver imx_camif_driver = {
> +	.probe		= camif_probe,
> +	.remove		= camif_remove,
> +	.driver		= {
> +		.name	= "imx-media-camif",
> +	},
> +};
> +
> +module_platform_driver(imx_camif_driver);
> +
> +MODULE_DESCRIPTION("i.MX camera interface subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-20 14:38     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:38 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devel, devicetree, Steve Longerbeam, linux-kernel,
	linux-arm-kernel, linux-media

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This is the camera interface driver that provides the v4l2
> user interface. Frames can be received from various sources:
> 
> - directly from SMFC for capturing unconverted images directly from
>   camera sensors.
> 
> - from the IC pre-process encode task.
> 
> - from the IC pre-process viewfinder task.
> 
> - from the IC post-process task.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile    |    2 +-
>  drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
>  2 files changed, 1001 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/staging/media/imx/imx-camif.c
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index d2a962c..fe9e992 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> -
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
> new file mode 100644
> index 0000000..404f724
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-camif.c
> @@ -0,0 +1,1000 @@
> +/*
> + * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2012-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/module.h>
> +#include <linux/of_platform.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +
> +#define CAMIF_NUM_PADS 2
> +
> +struct camif_priv {
> +	struct device         *dev;
> +	struct video_device    vfd;
> +	struct media_pipeline  mp;
> +	struct imx_media_dev  *md;
> +	struct v4l2_subdev     sd;
> +	struct media_pad       pad[CAMIF_NUM_PADS];
> +	struct media_pad       vd_pad;
> +	int id;
> +	int input_pad;
> +	int output_pad;
> +
> +	struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
> +
> +	/* dma buffer ring */
> +	struct imx_media_dma_buf_ring *in_ring;
> +	struct v4l2_subdev     *src_sd;
> +
> +	struct mutex           mutex;       /* capture device mutex */
> +	spinlock_t             q_lock;      /* protect ready_q */
> +
> +	/* buffer queue used in videobuf2 */
> +	struct vb2_queue       buffer_queue;
> +
> +	/* streaming buffer queue */
> +	struct list_head       ready_q;
> +
> +	/* controls inherited from subdevs */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +
> +	/* misc status */
> +	int                    current_input; /* the current input */
> +	v4l2_std_id            current_std;   /* current standard */
> +	bool                   stop;          /* streaming is stopping */
> +};
> +
> +/* In bytes, per queue */
> +#define VID_MEM_LIMIT	SZ_64M
> +
> +static struct vb2_ops camif_qops;
> +
> +/*
> + * Video ioctls follow
> + */
> +
> +static int vidioc_querycap(struct file *file, void *fh,
> +			   struct v4l2_capability *cap)
> +{
> +	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
> +	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
> +	cap->bus_info[0] = 0;

Should be set to something like 'platform:imx-media-camif'. v4l2-compliance should
complain about this.

> +	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
> +	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

Set device_caps in struct video_device, then drop these two lines since
the core will set these up based on the device_caps field in struct video_device.

> +
> +	return 0;
> +}
> +
> +static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
> +				  struct v4l2_fmtdesc *f)
> +{
> +	const struct imx_media_pixfmt *cc;
> +	u32 code;
> +	int ret;
> +
> +	ret = imx_media_enum_format(&code, f->index, true, true);
> +	if (ret)
> +		return ret;
> +	cc = imx_media_find_format(0, code, true, true);
> +	if (!cc)
> +		return -EINVAL;
> +
> +	f->pixelformat = cc->fourcc;
> +
> +	return 0;
> +}
> +
> +static int camif_g_fmt_vid_cap(struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct v4l2_mbus_framefmt *outfmt;
> +
> +	/* user format is the same as the format from output pad */
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
> +}
> +
> +static int camif_try_fmt_vid_cap(struct file *file, void *fh,
> +				 struct v4l2_format *f)
> +{
> +	return camif_g_fmt_vid_cap(file, fh, f);
> +}
> +
> +static int camif_s_fmt_vid_cap(struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	if (vb2_is_busy(&priv->buffer_queue)) {
> +		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
> +		return -EBUSY;
> +	}
> +
> +	return camif_try_fmt_vid_cap(file, priv, f);
> +}
> +
> +static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	return v4l2_subdev_call(sensor->sd, video, querystd, std);
> +}
> +
> +static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	*std = priv->current_std;
> +	return 0;
> +}
> +
> +static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int ret;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (vb2_is_busy(&priv->buffer_queue))
> +		return -EBUSY;
> +
> +	ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
> +	if (ret < 0)
> +		return ret;
> +
> +	priv->current_std = std;
> +	return 0;
> +}
> +
> +static int camif_enum_input(struct file *file, void *fh,
> +			    struct v4l2_input *input)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int index = input->index;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (index >= sensor->input.num)
> +		return -EINVAL;
> +
> +	input->type = V4L2_INPUT_TYPE_CAMERA;
> +	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
> +
> +	if (index == priv->current_input) {
> +		v4l2_subdev_call(sensor->sd, video, g_input_status,
> +				 &input->status);
> +		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);

Wrong op, use g_tvnorms instead.

> +	} else {
> +		input->status = V4L2_IN_ST_NO_SIGNAL;
> +		input->std = V4L2_STD_UNKNOWN;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_g_input(struct file *file, void *fh, unsigned int *index)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	*index = priv->current_input;
> +	return 0;
> +}
> +
> +static int camif_s_input(struct file *file, void *fh, unsigned int index)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int ret;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (index >= sensor->input.num)
> +		return -EINVAL;
> +
> +	if (index == priv->current_input)
> +		return 0;
> +
> +	/* select the sensor's input */
> +	ret = v4l2_subdev_call(sensor->sd, video, s_routing,
> +			       sensor->input.value[index], 0, 0);
> +	if (!ret)
> +		priv->current_input = index;
> +
> +	return ret;
> +}
> +
> +static int camif_g_parm(struct file *file, void *fh,
> +			struct v4l2_streamparm *a)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	return v4l2_subdev_call(sensor->sd, video, g_parm, a);
> +}
> +
> +static int camif_s_parm(struct file *file, void *fh,
> +			struct v4l2_streamparm *a)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	return v4l2_subdev_call(sensor->sd, video, s_parm, a);
> +}
> +
> +static const struct v4l2_ioctl_ops camif_ioctl_ops = {
> +	.vidioc_querycap	= vidioc_querycap,
> +
> +	.vidioc_enum_fmt_vid_cap        = camif_enum_fmt_vid_cap,
> +	.vidioc_g_fmt_vid_cap           = camif_g_fmt_vid_cap,
> +	.vidioc_try_fmt_vid_cap         = camif_try_fmt_vid_cap,
> +	.vidioc_s_fmt_vid_cap           = camif_s_fmt_vid_cap,
> +
> +	.vidioc_querystd        = camif_querystd,
> +	.vidioc_g_std           = camif_g_std,
> +	.vidioc_s_std           = camif_s_std,
> +
> +	.vidioc_enum_input      = camif_enum_input,
> +	.vidioc_g_input         = camif_g_input,
> +	.vidioc_s_input         = camif_s_input,
> +
> +	.vidioc_g_parm          = camif_g_parm,
> +	.vidioc_s_parm          = camif_s_parm,
> +
> +	.vidioc_reqbufs		= vb2_ioctl_reqbufs,
> +	.vidioc_create_bufs     = vb2_ioctl_create_bufs,
> +	.vidioc_prepare_buf     = vb2_ioctl_prepare_buf,
> +	.vidioc_querybuf	= vb2_ioctl_querybuf,
> +	.vidioc_qbuf		= vb2_ioctl_qbuf,
> +	.vidioc_dqbuf		= vb2_ioctl_dqbuf,
> +	.vidioc_expbuf		= vb2_ioctl_expbuf,
> +	.vidioc_streamon	= vb2_ioctl_streamon,
> +	.vidioc_streamoff	= vb2_ioctl_streamoff,
> +};
> +
> +/*
> + * Queue operations
> + */
> +
> +static u32 camif_get_sizeimage(struct camif_priv *priv)
> +{
> +	struct v4l2_mbus_framefmt *outfmt;
> +	const struct imx_media_pixfmt *outcc;
> +
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	outcc = priv->cc[priv->output_pad];
> +	return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
> +}
> +
> +static int camif_queue_setup(struct vb2_queue *vq,
> +			     unsigned int *nbuffers, unsigned int *nplanes,
> +			     unsigned int sizes[], struct device *alloc_devs[])
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	unsigned int count = *nbuffers;
> +	u32 sizeimage;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	sizeimage = camif_get_sizeimage(priv);
> +
> +	count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
> +	count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
> +
> +	while (sizeimage * count > VID_MEM_LIMIT)
> +		count--;
> +
> +	if (count < IMX_MEDIA_MIN_RING_BUFS)
> +		return -EINVAL;
> +
> +	*nplanes = 1;
> +	*nbuffers = count;
> +	sizes[0] = sizeimage;
> +
> +	return 0;
> +}
> +
> +static int camif_buf_init(struct vb2_buffer *vb)
> +{
> +	struct vb2_queue *vq = vb->vb2_queue;
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	struct imx_media_buffer *buf = to_imx_media_vb(vb);
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	INIT_LIST_HEAD(&buf->list);
> +
> +	return 0;
> +}
> +
> +static int camif_buf_prepare(struct vb2_buffer *vb)
> +{
> +	struct vb2_queue *vq = vb->vb2_queue;
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	u32 sizeimage = camif_get_sizeimage(priv);
> +	int ret;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (vb2_plane_size(vb, 0) < sizeimage) {
> +		v4l2_err(&priv->sd,
> +			 "data will not fit into plane (%lu < %lu)\n",
> +			 vb2_plane_size(vb, 0), (long)sizeimage);
> +		return -EINVAL;
> +	}
> +
> +	vb2_set_plane_payload(vb, 0, sizeimage);
> +
> +	if (!priv->in_ring) {
> +		priv->in_ring = imx_media_alloc_dma_buf_ring(
> +			priv->md, &priv->src_sd->entity, &priv->sd.entity,
> +			sizeimage, vq->num_buffers, false);
> +		if (IS_ERR(priv->in_ring)) {
> +			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
> +			ret = PTR_ERR(priv->in_ring);
> +			priv->in_ring = NULL;
> +			return ret;
> +		}
> +	}
> +
> +	ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
> +	if (ret)
> +		goto free_ring;
> +
> +	return 0;
> +
> +free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +	return ret;
> +}
> +
> +static void camif_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
> +	struct imx_media_buffer *buf = to_imx_media_vb(vb);
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +
> +	list_add_tail(&buf->list, &priv->ready_q);
> +
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	u32 sizeimage = camif_get_sizeimage(priv);
> +	struct imx_media_buffer *buf, *tmp;
> +	unsigned long flags;
> +	int ret;
> +
> +	if (vb2_is_streaming(vq))
> +		return 0;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (!priv->in_ring) {
> +		priv->in_ring = imx_media_alloc_dma_buf_ring(
> +			priv->md, &priv->src_sd->entity, &priv->sd.entity,
> +			sizeimage, vq->num_buffers, false);
> +		if (IS_ERR(priv->in_ring)) {
> +			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
> +			ret = PTR_ERR(priv->in_ring);
> +			priv->in_ring = NULL;
> +			goto return_bufs;
> +		}
> +	}
> +
> +	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
> +					    &priv->mp, true);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
> +			 ret);
> +		goto free_ring;
> +	}
> +
> +	priv->stop = false;
> +
> +	return 0;
> +
> +free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +return_bufs:
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
> +		list_del(&buf->list);
> +		vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
> +	}
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +	return ret;
> +}
> +
> +static void camif_stop_streaming(struct vb2_queue *vq)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	struct imx_media_buffer *frame;
> +	unsigned long flags;
> +	int ret;
> +
> +	if (!vb2_is_streaming(vq))
> +		return;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	priv->stop = true;
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +
> +	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
> +					    &priv->mp, false);
> +	if (ret)
> +		v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
> +			  ret);
> +
> +	if (priv->in_ring) {
> +		v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
> +			  __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +	}
> +
> +	/* release all active buffers */
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	while (!list_empty(&priv->ready_q)) {
> +		frame = list_entry(priv->ready_q.next,
> +				   struct imx_media_buffer, list);
> +		list_del(&frame->list);
> +		vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
> +	}
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static struct vb2_ops camif_qops = {
> +	.queue_setup	 = camif_queue_setup,
> +	.buf_init        = camif_buf_init,
> +	.buf_prepare	 = camif_buf_prepare,
> +	.buf_queue	 = camif_buf_queue,
> +	.wait_prepare	 = vb2_ops_wait_prepare,
> +	.wait_finish	 = vb2_ops_wait_finish,
> +	.start_streaming = camif_start_streaming,
> +	.stop_streaming  = camif_stop_streaming,
> +};
> +
> +/*
> + * File operations
> + */
> +static int camif_open(struct file *file)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	int ret;
> +
> +	if (mutex_lock_interruptible(&priv->mutex))
> +		return -ERESTARTSYS;
> +
> +	ret = v4l2_fh_open(file);
> +	if (ret)
> +		v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
> +
> +	mutex_unlock(&priv->mutex);
> +	return ret;
> +}
> +
> +static int camif_release(struct file *file)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct vb2_queue *vq = &priv->buffer_queue;
> +	int ret = 0;
> +
> +	mutex_lock(&priv->mutex);
> +
> +	if (file->private_data == vq->owner) {
> +		vb2_queue_release(vq);
> +		vq->owner = NULL;
> +	}
> +
> +	v4l2_fh_release(file);
> +	mutex_unlock(&priv->mutex);
> +	return ret;
> +}
> +
> +static const struct v4l2_file_operations camif_fops = {
> +	.owner		= THIS_MODULE,
> +	.open		= camif_open,
> +	.release	= camif_release,
> +	.poll		= vb2_fop_poll,
> +	.unlocked_ioctl	= video_ioctl2,
> +	.mmap		= vb2_fop_mmap,
> +};
> +
> +static struct video_device camif_videodev = {
> +	.fops		= &camif_fops,
> +	.ioctl_ops	= &camif_ioctl_ops,
> +	.minor		= -1,
> +	.release	= video_device_release,
> +	.vfl_dir	= VFL_DIR_RX,
> +	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
> +};
> +
> +/*
> + * Subdev and media entity operations
> + */
> +
> +static void camif_new_dma_buf(struct camif_priv *priv)
> +{
> +	struct imx_media_dma_buf *dmabuf;
> +	struct imx_media_buffer *buf;
> +	enum vb2_buffer_state state;
> +	struct vb2_buffer *vb;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +
> +	if (priv->stop || list_empty(&priv->ready_q))
> +		goto unlock;
> +
> +	dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
> +	if (!dmabuf)
> +		goto unlock;
> +
> +	vb = dmabuf->vb;
> +	buf = to_imx_media_vb(vb);
> +	if (list_empty(&buf->list)) {
> +		dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
> +			vb->index);
> +		goto unlock;
> +	}
> +
> +	dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
> +	vb->timestamp = ktime_get_ns();
> +	list_del_init(&buf->list);
> +	state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
> +		VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
> +	vb2_buffer_done(vb, state);
> +unlock:
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct imx_media_dma_buf_ring **ring;
> +
> +	switch (cmd) {
> +	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
> +		if (!priv->in_ring)
> +			return -EINVAL;
> +		ring = (struct imx_media_dma_buf_ring **)arg;
> +		*ring = priv->in_ring;
> +		break;
> +	case IMX_MEDIA_NEW_DMA_BUF:
> +		camif_new_dma_buf(priv);
> +		break;
> +	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
> +		/* src indicates sink buffer ring can be freed */
> +		if (!priv->in_ring)
> +			return 0;
> +		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_link_setup(struct media_entity *entity,
> +			    const struct media_pad *local,
> +			    const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct video_device *vfd = &priv->vfd;
> +	struct v4l2_subdev *remote_sd;
> +	int ret = 0;
> +
> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	if (is_media_entity_v4l2_video_device(remote->entity))
> +		return 0;
> +
> +	WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	/* reset controls to refresh with inherited from subdevs */
> +	v4l2_ctrl_handler_free(vfd->ctrl_handler);
> +	v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
> +
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->src_sd)
> +			return -EBUSY;
> +		priv->src_sd = remote_sd;
> +		ret = imx_media_inherit_controls(priv->md, vfd,
> +						 &priv->src_sd->entity);
> +	} else {
> +		priv->src_sd = NULL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int camif_enum_mbus_code(struct v4l2_subdev *sd,
> +				struct v4l2_subdev_pad_config *cfg,
> +				struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	if (code->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	return imx_media_enum_format(&code->code, code->index, true, true);
> +}
> +
> +static int camif_get_fmt(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_pad_config *cfg,
> +			 struct v4l2_subdev_format *sdformat)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +
> +	if (sdformat->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int camif_set_fmt(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_pad_config *cfg,
> +			 struct v4l2_subdev_format *sdformat)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	const struct imx_media_pixfmt *cc;
> +	u32 code;
> +
> +	if (sdformat->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (vb2_is_busy(&priv->buffer_queue)) {
> +		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
> +		return -EBUSY;
> +	}
> +
> +	cc = imx_media_find_format(0, sdformat->format.code, true, true);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, true);
> +		cc = imx_media_find_format(0, code, true, true);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	/* Output pad mirrors input pad, no limitations on input pads */
> +	if (sdformat->pad == priv->output_pad)
> +		sdformat->format = priv->format_mbus[priv->input_pad];
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_init_pads(struct camif_priv *priv)
> +{
> +	struct video_device *vfd = &priv->vfd;
> +	struct imx_media_subdev *imxsd;
> +	struct imx_media_pad *pad;
> +	int i, ret;
> +
> +	imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
> +	if (IS_ERR(imxsd))
> +		return PTR_ERR(imxsd);
> +
> +	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
> +		v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
> +			 imxsd->num_sink_pads, imxsd->num_src_pads);
> +		return -EINVAL;
> +	}
> +
> +	priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
> +	ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "failed to init device node pad\n");
> +		return ret;
> +	}
> +
> +	for (i = 0; i < CAMIF_NUM_PADS; i++) {
> +		pad = &imxsd->pad[i];
> +		priv->pad[i] = pad->pad;
> +		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
> +			priv->input_pad = i;
> +		else
> +			priv->output_pad = i;
> +
> +		/* set a default mbus format  */
> +		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
> +					      640, 480, 0, V4L2_FIELD_NONE,
> +					      &priv->cc[i]);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
> +				      priv->pad);
> +}
> +
> +static int camif_registered(struct v4l2_subdev *sd)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct vb2_queue *vq = &priv->buffer_queue;
> +	struct video_device *vfd = &priv->vfd;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	int ret;
> +
> +	/* get media device */
> +	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
> +
> +	vfd->v4l2_dev = sd->v4l2_dev;
> +
> +	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
> +	if (ret) {
> +		v4l2_err(sd, "Failed to register video device\n");
> +		return ret;
> +	}
> +
> +	vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	vq->io_modes = VB2_MMAP | VB2_DMABUF;
> +	vq->drv_priv = priv;
> +	vq->buf_struct_size = sizeof(struct imx_media_buffer);
> +	vq->ops = &camif_qops;
> +	vq->mem_ops = &vb2_dma_contig_memops;
> +	vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +	vq->lock = &priv->mutex;
> +	vq->min_buffers_needed = 2;
> +	vq->dev = priv->dev;
> +
> +	ret = vb2_queue_init(vq);
> +	if (ret) {
> +		v4l2_err(sd, "vb2_queue_init failed\n");
> +		goto unreg;
> +	}
> +
> +	INIT_LIST_HEAD(&priv->ready_q);
> +
> +	ret = camif_init_pads(priv);
> +	if (ret) {
> +		v4l2_err(sd, "camif_init_pads failed\n");
> +		goto unreg;
> +	}
> +
> +	/* create the link to our device node */
> +	ret = media_create_pad_link(&sd->entity, priv->output_pad,
> +				    &vfd->entity, 0,
> +				    MEDIA_LNK_FL_IMMUTABLE |
> +				    MEDIA_LNK_FL_ENABLED);
> +	if (ret) {
> +		v4l2_err(sd, "failed to create link to device node\n");
> +		goto unreg;
> +	}
> +
> +	/* setup default pad formats */
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
> +				      &priv->cc[priv->output_pad]);
> +	if (ret)
> +		goto unreg;
> +
> +	*infmt = *outfmt;
> +	priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
> +
> +	priv->current_std = V4L2_STD_UNKNOWN;
> +
> +	v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
> +		  video_device_node_name(vfd));
> +
> +	vfd->ctrl_handler = &priv->ctrl_hdlr;
> +
> +	return 0;
> +unreg:
> +	video_unregister_device(vfd);
> +	return ret;
> +}
> +
> +static void camif_unregistered(struct v4l2_subdev *sd)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct video_device *vfd = &priv->vfd;
> +
> +	mutex_lock(&priv->mutex);
> +
> +	if (video_is_registered(vfd)) {
> +		video_unregister_device(vfd);
> +		media_entity_cleanup(&vfd->entity);
> +	}
> +
> +	mutex_unlock(&priv->mutex);
> +}
> +
> +static const struct v4l2_subdev_internal_ops camif_internal_ops = {
> +	.registered = camif_registered,
> +	.unregistered = camif_unregistered,
> +};
> +
> +static struct v4l2_subdev_core_ops camif_core_ops = {
> +	.ioctl = camif_ioctl,
> +};
> +
> +static struct media_entity_operations camif_entity_ops = {
> +	.link_setup = camif_link_setup,
> +	.link_validate = v4l2_subdev_link_validate,
> +};
> +
> +static struct v4l2_subdev_pad_ops camif_pad_ops = {
> +	.enum_mbus_code = camif_enum_mbus_code,
> +	.get_fmt = camif_get_fmt,
> +	.set_fmt = camif_set_fmt,
> +};
> +
> +static struct v4l2_subdev_ops camif_subdev_ops = {
> +	.pad = &camif_pad_ops,
> +	.core = &camif_core_ops,
> +};
> +
> +static int camif_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct camif_priv *priv;
> +	struct video_device *vfd;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, priv);
> +	priv->dev = &pdev->dev;
> +
> +	pdata = priv->dev->platform_data;
> +
> +	mutex_init(&priv->mutex);
> +	spin_lock_init(&priv->q_lock);
> +
> +	v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
> +	v4l2_set_subdevdata(&priv->sd, priv);
> +	priv->sd.internal_ops = &camif_internal_ops;
> +	priv->sd.entity.ops = &camif_entity_ops;
> +	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> +	priv->sd.dev = &pdev->dev;
> +	priv->sd.owner = THIS_MODULE;
> +	/* get our group id and camif id */
> +	priv->sd.grp_id = pdata->grp_id;
> +	priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
> +	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
> +	snprintf(camif_videodev.name, sizeof(camif_videodev.name),
> +		 "%s devnode", pdata->sd_name);
> +
> +	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +
> +	vfd = &priv->vfd;
> +	*vfd = camif_videodev;
> +	vfd->lock = &priv->mutex;
> +	vfd->queue = &priv->buffer_queue;
> +
> +	video_set_drvdata(vfd, priv);
> +
> +	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
> +
> +	ret = v4l2_async_register_subdev(&priv->sd);
> +	if (ret)
> +		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +
> +	return ret;
> +}
> +
> +static int camif_remove(struct platform_device *pdev)
> +{
> +	struct camif_priv *priv =
> +		(struct camif_priv *)platform_get_drvdata(pdev);
> +
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +	v4l2_async_unregister_subdev(&priv->sd);
> +	media_entity_cleanup(&priv->sd.entity);
> +	v4l2_device_unregister_subdev(&priv->sd);
> +
> +	return 0;
> +}
> +
> +static const struct platform_device_id camif_ids[] = {
> +	{ .name = "imx-media-camif" },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(platform, camif_ids);
> +
> +static struct platform_driver imx_camif_driver = {
> +	.probe		= camif_probe,
> +	.remove		= camif_remove,
> +	.driver		= {
> +		.name	= "imx-media-camif",
> +	},
> +};
> +
> +module_platform_driver(imx_camif_driver);
> +
> +MODULE_DESCRIPTION("i.MX camera interface subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-20 14:38     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:38 UTC (permalink / raw)
  To: linux-arm-kernel

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This is the camera interface driver that provides the v4l2
> user interface. Frames can be received from various sources:
> 
> - directly from SMFC for capturing unconverted images directly from
>   camera sensors.
> 
> - from the IC pre-process encode task.
> 
> - from the IC pre-process viewfinder task.
> 
> - from the IC post-process task.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile    |    2 +-
>  drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
>  2 files changed, 1001 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/staging/media/imx/imx-camif.c
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index d2a962c..fe9e992 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> -
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
> new file mode 100644
> index 0000000..404f724
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-camif.c
> @@ -0,0 +1,1000 @@
> +/*
> + * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2012-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/module.h>
> +#include <linux/of_platform.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/timer.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +
> +#define CAMIF_NUM_PADS 2
> +
> +struct camif_priv {
> +	struct device         *dev;
> +	struct video_device    vfd;
> +	struct media_pipeline  mp;
> +	struct imx_media_dev  *md;
> +	struct v4l2_subdev     sd;
> +	struct media_pad       pad[CAMIF_NUM_PADS];
> +	struct media_pad       vd_pad;
> +	int id;
> +	int input_pad;
> +	int output_pad;
> +
> +	struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
> +	const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
> +
> +	/* dma buffer ring */
> +	struct imx_media_dma_buf_ring *in_ring;
> +	struct v4l2_subdev     *src_sd;
> +
> +	struct mutex           mutex;       /* capture device mutex */
> +	spinlock_t             q_lock;      /* protect ready_q */
> +
> +	/* buffer queue used in videobuf2 */
> +	struct vb2_queue       buffer_queue;
> +
> +	/* streaming buffer queue */
> +	struct list_head       ready_q;
> +
> +	/* controls inherited from subdevs */
> +	struct v4l2_ctrl_handler ctrl_hdlr;
> +
> +	/* misc status */
> +	int                    current_input; /* the current input */
> +	v4l2_std_id            current_std;   /* current standard */
> +	bool                   stop;          /* streaming is stopping */
> +};
> +
> +/* In bytes, per queue */
> +#define VID_MEM_LIMIT	SZ_64M
> +
> +static struct vb2_ops camif_qops;
> +
> +/*
> + * Video ioctls follow
> + */
> +
> +static int vidioc_querycap(struct file *file, void *fh,
> +			   struct v4l2_capability *cap)
> +{
> +	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
> +	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
> +	cap->bus_info[0] = 0;

Should be set to something like 'platform:imx-media-camif'. v4l2-compliance should
complain about this.

> +	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
> +	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

Set device_caps in struct video_device, then drop these two lines since
the core will set these up based on the device_caps field in struct video_device.

> +
> +	return 0;
> +}
> +
> +static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
> +				  struct v4l2_fmtdesc *f)
> +{
> +	const struct imx_media_pixfmt *cc;
> +	u32 code;
> +	int ret;
> +
> +	ret = imx_media_enum_format(&code, f->index, true, true);
> +	if (ret)
> +		return ret;
> +	cc = imx_media_find_format(0, code, true, true);
> +	if (!cc)
> +		return -EINVAL;
> +
> +	f->pixelformat = cc->fourcc;
> +
> +	return 0;
> +}
> +
> +static int camif_g_fmt_vid_cap(struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct v4l2_mbus_framefmt *outfmt;
> +
> +	/* user format is the same as the format from output pad */
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
> +}
> +
> +static int camif_try_fmt_vid_cap(struct file *file, void *fh,
> +				 struct v4l2_format *f)
> +{
> +	return camif_g_fmt_vid_cap(file, fh, f);
> +}
> +
> +static int camif_s_fmt_vid_cap(struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	if (vb2_is_busy(&priv->buffer_queue)) {
> +		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
> +		return -EBUSY;
> +	}
> +
> +	return camif_try_fmt_vid_cap(file, priv, f);
> +}
> +
> +static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	return v4l2_subdev_call(sensor->sd, video, querystd, std);
> +}
> +
> +static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	*std = priv->current_std;
> +	return 0;
> +}
> +
> +static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int ret;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (vb2_is_busy(&priv->buffer_queue))
> +		return -EBUSY;
> +
> +	ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
> +	if (ret < 0)
> +		return ret;
> +
> +	priv->current_std = std;
> +	return 0;
> +}
> +
> +static int camif_enum_input(struct file *file, void *fh,
> +			    struct v4l2_input *input)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int index = input->index;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (index >= sensor->input.num)
> +		return -EINVAL;
> +
> +	input->type = V4L2_INPUT_TYPE_CAMERA;
> +	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
> +
> +	if (index == priv->current_input) {
> +		v4l2_subdev_call(sensor->sd, video, g_input_status,
> +				 &input->status);
> +		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);

Wrong op, use g_tvnorms instead.

> +	} else {
> +		input->status = V4L2_IN_ST_NO_SIGNAL;
> +		input->std = V4L2_STD_UNKNOWN;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_g_input(struct file *file, void *fh, unsigned int *index)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +
> +	*index = priv->current_input;
> +	return 0;
> +}
> +
> +static int camif_s_input(struct file *file, void *fh, unsigned int index)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +	int ret;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (index >= sensor->input.num)
> +		return -EINVAL;
> +
> +	if (index == priv->current_input)
> +		return 0;
> +
> +	/* select the sensor's input */
> +	ret = v4l2_subdev_call(sensor->sd, video, s_routing,
> +			       sensor->input.value[index], 0, 0);
> +	if (!ret)
> +		priv->current_input = index;
> +
> +	return ret;
> +}
> +
> +static int camif_g_parm(struct file *file, void *fh,
> +			struct v4l2_streamparm *a)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	return v4l2_subdev_call(sensor->sd, video, g_parm, a);
> +}
> +
> +static int camif_s_parm(struct file *file, void *fh,
> +			struct v4l2_streamparm *a)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct imx_media_subdev *sensor;
> +
> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		return PTR_ERR(sensor);
> +	}
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	return v4l2_subdev_call(sensor->sd, video, s_parm, a);
> +}
> +
> +static const struct v4l2_ioctl_ops camif_ioctl_ops = {
> +	.vidioc_querycap	= vidioc_querycap,
> +
> +	.vidioc_enum_fmt_vid_cap        = camif_enum_fmt_vid_cap,
> +	.vidioc_g_fmt_vid_cap           = camif_g_fmt_vid_cap,
> +	.vidioc_try_fmt_vid_cap         = camif_try_fmt_vid_cap,
> +	.vidioc_s_fmt_vid_cap           = camif_s_fmt_vid_cap,
> +
> +	.vidioc_querystd        = camif_querystd,
> +	.vidioc_g_std           = camif_g_std,
> +	.vidioc_s_std           = camif_s_std,
> +
> +	.vidioc_enum_input      = camif_enum_input,
> +	.vidioc_g_input         = camif_g_input,
> +	.vidioc_s_input         = camif_s_input,
> +
> +	.vidioc_g_parm          = camif_g_parm,
> +	.vidioc_s_parm          = camif_s_parm,
> +
> +	.vidioc_reqbufs		= vb2_ioctl_reqbufs,
> +	.vidioc_create_bufs     = vb2_ioctl_create_bufs,
> +	.vidioc_prepare_buf     = vb2_ioctl_prepare_buf,
> +	.vidioc_querybuf	= vb2_ioctl_querybuf,
> +	.vidioc_qbuf		= vb2_ioctl_qbuf,
> +	.vidioc_dqbuf		= vb2_ioctl_dqbuf,
> +	.vidioc_expbuf		= vb2_ioctl_expbuf,
> +	.vidioc_streamon	= vb2_ioctl_streamon,
> +	.vidioc_streamoff	= vb2_ioctl_streamoff,
> +};
> +
> +/*
> + * Queue operations
> + */
> +
> +static u32 camif_get_sizeimage(struct camif_priv *priv)
> +{
> +	struct v4l2_mbus_framefmt *outfmt;
> +	const struct imx_media_pixfmt *outcc;
> +
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	outcc = priv->cc[priv->output_pad];
> +	return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
> +}
> +
> +static int camif_queue_setup(struct vb2_queue *vq,
> +			     unsigned int *nbuffers, unsigned int *nplanes,
> +			     unsigned int sizes[], struct device *alloc_devs[])
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	unsigned int count = *nbuffers;
> +	u32 sizeimage;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	sizeimage = camif_get_sizeimage(priv);
> +
> +	count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
> +	count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
> +
> +	while (sizeimage * count > VID_MEM_LIMIT)
> +		count--;
> +
> +	if (count < IMX_MEDIA_MIN_RING_BUFS)
> +		return -EINVAL;
> +
> +	*nplanes = 1;
> +	*nbuffers = count;
> +	sizes[0] = sizeimage;
> +
> +	return 0;
> +}
> +
> +static int camif_buf_init(struct vb2_buffer *vb)
> +{
> +	struct vb2_queue *vq = vb->vb2_queue;
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	struct imx_media_buffer *buf = to_imx_media_vb(vb);
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	INIT_LIST_HEAD(&buf->list);
> +
> +	return 0;
> +}
> +
> +static int camif_buf_prepare(struct vb2_buffer *vb)
> +{
> +	struct vb2_queue *vq = vb->vb2_queue;
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	u32 sizeimage = camif_get_sizeimage(priv);
> +	int ret;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (vb2_plane_size(vb, 0) < sizeimage) {
> +		v4l2_err(&priv->sd,
> +			 "data will not fit into plane (%lu < %lu)\n",
> +			 vb2_plane_size(vb, 0), (long)sizeimage);
> +		return -EINVAL;
> +	}
> +
> +	vb2_set_plane_payload(vb, 0, sizeimage);
> +
> +	if (!priv->in_ring) {
> +		priv->in_ring = imx_media_alloc_dma_buf_ring(
> +			priv->md, &priv->src_sd->entity, &priv->sd.entity,
> +			sizeimage, vq->num_buffers, false);
> +		if (IS_ERR(priv->in_ring)) {
> +			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
> +			ret = PTR_ERR(priv->in_ring);
> +			priv->in_ring = NULL;
> +			return ret;
> +		}
> +	}
> +
> +	ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
> +	if (ret)
> +		goto free_ring;
> +
> +	return 0;
> +
> +free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +	return ret;
> +}
> +
> +static void camif_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
> +	struct imx_media_buffer *buf = to_imx_media_vb(vb);
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +
> +	list_add_tail(&buf->list, &priv->ready_q);
> +
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	u32 sizeimage = camif_get_sizeimage(priv);
> +	struct imx_media_buffer *buf, *tmp;
> +	unsigned long flags;
> +	int ret;
> +
> +	if (vb2_is_streaming(vq))
> +		return 0;
> +
> +	if (!priv->src_sd)
> +		return -EPIPE;
> +
> +	if (!priv->in_ring) {
> +		priv->in_ring = imx_media_alloc_dma_buf_ring(
> +			priv->md, &priv->src_sd->entity, &priv->sd.entity,
> +			sizeimage, vq->num_buffers, false);
> +		if (IS_ERR(priv->in_ring)) {
> +			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
> +			ret = PTR_ERR(priv->in_ring);
> +			priv->in_ring = NULL;
> +			goto return_bufs;
> +		}
> +	}
> +
> +	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
> +					    &priv->mp, true);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
> +			 ret);
> +		goto free_ring;
> +	}
> +
> +	priv->stop = false;
> +
> +	return 0;
> +
> +free_ring:
> +	imx_media_free_dma_buf_ring(priv->in_ring);
> +	priv->in_ring = NULL;
> +return_bufs:
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
> +		list_del(&buf->list);
> +		vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
> +	}
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +	return ret;
> +}
> +
> +static void camif_stop_streaming(struct vb2_queue *vq)
> +{
> +	struct camif_priv *priv = vb2_get_drv_priv(vq);
> +	struct imx_media_buffer *frame;
> +	unsigned long flags;
> +	int ret;
> +
> +	if (!vb2_is_streaming(vq))
> +		return;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	priv->stop = true;
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +
> +	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
> +					    &priv->mp, false);
> +	if (ret)
> +		v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
> +			  ret);
> +
> +	if (priv->in_ring) {
> +		v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
> +			  __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +	}
> +
> +	/* release all active buffers */
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +	while (!list_empty(&priv->ready_q)) {
> +		frame = list_entry(priv->ready_q.next,
> +				   struct imx_media_buffer, list);
> +		list_del(&frame->list);
> +		vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
> +	}
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static struct vb2_ops camif_qops = {
> +	.queue_setup	 = camif_queue_setup,
> +	.buf_init        = camif_buf_init,
> +	.buf_prepare	 = camif_buf_prepare,
> +	.buf_queue	 = camif_buf_queue,
> +	.wait_prepare	 = vb2_ops_wait_prepare,
> +	.wait_finish	 = vb2_ops_wait_finish,
> +	.start_streaming = camif_start_streaming,
> +	.stop_streaming  = camif_stop_streaming,
> +};
> +
> +/*
> + * File operations
> + */
> +static int camif_open(struct file *file)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	int ret;
> +
> +	if (mutex_lock_interruptible(&priv->mutex))
> +		return -ERESTARTSYS;
> +
> +	ret = v4l2_fh_open(file);
> +	if (ret)
> +		v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
> +
> +	mutex_unlock(&priv->mutex);
> +	return ret;
> +}
> +
> +static int camif_release(struct file *file)
> +{
> +	struct camif_priv *priv = video_drvdata(file);
> +	struct vb2_queue *vq = &priv->buffer_queue;
> +	int ret = 0;
> +
> +	mutex_lock(&priv->mutex);
> +
> +	if (file->private_data == vq->owner) {
> +		vb2_queue_release(vq);
> +		vq->owner = NULL;
> +	}
> +
> +	v4l2_fh_release(file);
> +	mutex_unlock(&priv->mutex);
> +	return ret;
> +}
> +
> +static const struct v4l2_file_operations camif_fops = {
> +	.owner		= THIS_MODULE,
> +	.open		= camif_open,
> +	.release	= camif_release,
> +	.poll		= vb2_fop_poll,
> +	.unlocked_ioctl	= video_ioctl2,
> +	.mmap		= vb2_fop_mmap,
> +};
> +
> +static struct video_device camif_videodev = {
> +	.fops		= &camif_fops,
> +	.ioctl_ops	= &camif_ioctl_ops,
> +	.minor		= -1,
> +	.release	= video_device_release,
> +	.vfl_dir	= VFL_DIR_RX,
> +	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
> +};
> +
> +/*
> + * Subdev and media entity operations
> + */
> +
> +static void camif_new_dma_buf(struct camif_priv *priv)
> +{
> +	struct imx_media_dma_buf *dmabuf;
> +	struct imx_media_buffer *buf;
> +	enum vb2_buffer_state state;
> +	struct vb2_buffer *vb;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&priv->q_lock, flags);
> +
> +	if (priv->stop || list_empty(&priv->ready_q))
> +		goto unlock;
> +
> +	dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
> +	if (!dmabuf)
> +		goto unlock;
> +
> +	vb = dmabuf->vb;
> +	buf = to_imx_media_vb(vb);
> +	if (list_empty(&buf->list)) {
> +		dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
> +			vb->index);
> +		goto unlock;
> +	}
> +
> +	dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
> +	vb->timestamp = ktime_get_ns();
> +	list_del_init(&buf->list);
> +	state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
> +		VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
> +	vb2_buffer_done(vb, state);
> +unlock:
> +	spin_unlock_irqrestore(&priv->q_lock, flags);
> +}
> +
> +static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct imx_media_dma_buf_ring **ring;
> +
> +	switch (cmd) {
> +	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
> +		if (!priv->in_ring)
> +			return -EINVAL;
> +		ring = (struct imx_media_dma_buf_ring **)arg;
> +		*ring = priv->in_ring;
> +		break;
> +	case IMX_MEDIA_NEW_DMA_BUF:
> +		camif_new_dma_buf(priv);
> +		break;
> +	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
> +		/* src indicates sink buffer ring can be freed */
> +		if (!priv->in_ring)
> +			return 0;
> +		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
> +		imx_media_free_dma_buf_ring(priv->in_ring);
> +		priv->in_ring = NULL;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_link_setup(struct media_entity *entity,
> +			    const struct media_pad *local,
> +			    const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct video_device *vfd = &priv->vfd;
> +	struct v4l2_subdev *remote_sd;
> +	int ret = 0;
> +
> +	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	if (is_media_entity_v4l2_video_device(remote->entity))
> +		return 0;
> +
> +	WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	/* reset controls to refresh with inherited from subdevs */
> +	v4l2_ctrl_handler_free(vfd->ctrl_handler);
> +	v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
> +
> +	if (flags & MEDIA_LNK_FL_ENABLED) {
> +		if (priv->src_sd)
> +			return -EBUSY;
> +		priv->src_sd = remote_sd;
> +		ret = imx_media_inherit_controls(priv->md, vfd,
> +						 &priv->src_sd->entity);
> +	} else {
> +		priv->src_sd = NULL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int camif_enum_mbus_code(struct v4l2_subdev *sd,
> +				struct v4l2_subdev_pad_config *cfg,
> +				struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	if (code->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	return imx_media_enum_format(&code->code, code->index, true, true);
> +}
> +
> +static int camif_get_fmt(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_pad_config *cfg,
> +			 struct v4l2_subdev_format *sdformat)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +
> +	if (sdformat->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	sdformat->format = priv->format_mbus[sdformat->pad];
> +
> +	return 0;
> +}
> +
> +static int camif_set_fmt(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_pad_config *cfg,
> +			 struct v4l2_subdev_format *sdformat)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	const struct imx_media_pixfmt *cc;
> +	u32 code;
> +
> +	if (sdformat->pad >= CAMIF_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (vb2_is_busy(&priv->buffer_queue)) {
> +		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
> +		return -EBUSY;
> +	}
> +
> +	cc = imx_media_find_format(0, sdformat->format.code, true, true);
> +	if (!cc) {
> +		imx_media_enum_format(&code, 0, true, true);
> +		cc = imx_media_find_format(0, code, true, true);
> +		sdformat->format.code = cc->codes[0];
> +	}
> +
> +	/* Output pad mirrors input pad, no limitations on input pads */
> +	if (sdformat->pad == priv->output_pad)
> +		sdformat->format = priv->format_mbus[priv->input_pad];
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		priv->cc[sdformat->pad] = cc;
> +	}
> +
> +	return 0;
> +}
> +
> +static int camif_init_pads(struct camif_priv *priv)
> +{
> +	struct video_device *vfd = &priv->vfd;
> +	struct imx_media_subdev *imxsd;
> +	struct imx_media_pad *pad;
> +	int i, ret;
> +
> +	imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
> +	if (IS_ERR(imxsd))
> +		return PTR_ERR(imxsd);
> +
> +	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
> +		v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
> +			 imxsd->num_sink_pads, imxsd->num_src_pads);
> +		return -EINVAL;
> +	}
> +
> +	priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
> +	ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
> +	if (ret) {
> +		v4l2_err(&priv->sd, "failed to init device node pad\n");
> +		return ret;
> +	}
> +
> +	for (i = 0; i < CAMIF_NUM_PADS; i++) {
> +		pad = &imxsd->pad[i];
> +		priv->pad[i] = pad->pad;
> +		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
> +			priv->input_pad = i;
> +		else
> +			priv->output_pad = i;
> +
> +		/* set a default mbus format  */
> +		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
> +					      640, 480, 0, V4L2_FIELD_NONE,
> +					      &priv->cc[i]);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
> +				      priv->pad);
> +}
> +
> +static int camif_registered(struct v4l2_subdev *sd)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct vb2_queue *vq = &priv->buffer_queue;
> +	struct video_device *vfd = &priv->vfd;
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	int ret;
> +
> +	/* get media device */
> +	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
> +
> +	vfd->v4l2_dev = sd->v4l2_dev;
> +
> +	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
> +	if (ret) {
> +		v4l2_err(sd, "Failed to register video device\n");
> +		return ret;
> +	}
> +
> +	vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	vq->io_modes = VB2_MMAP | VB2_DMABUF;
> +	vq->drv_priv = priv;
> +	vq->buf_struct_size = sizeof(struct imx_media_buffer);
> +	vq->ops = &camif_qops;
> +	vq->mem_ops = &vb2_dma_contig_memops;
> +	vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +	vq->lock = &priv->mutex;
> +	vq->min_buffers_needed = 2;
> +	vq->dev = priv->dev;
> +
> +	ret = vb2_queue_init(vq);
> +	if (ret) {
> +		v4l2_err(sd, "vb2_queue_init failed\n");
> +		goto unreg;
> +	}
> +
> +	INIT_LIST_HEAD(&priv->ready_q);
> +
> +	ret = camif_init_pads(priv);
> +	if (ret) {
> +		v4l2_err(sd, "camif_init_pads failed\n");
> +		goto unreg;
> +	}
> +
> +	/* create the link to our device node */
> +	ret = media_create_pad_link(&sd->entity, priv->output_pad,
> +				    &vfd->entity, 0,
> +				    MEDIA_LNK_FL_IMMUTABLE |
> +				    MEDIA_LNK_FL_ENABLED);
> +	if (ret) {
> +		v4l2_err(sd, "failed to create link to device node\n");
> +		goto unreg;
> +	}
> +
> +	/* setup default pad formats */
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +	ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
> +				      &priv->cc[priv->output_pad]);
> +	if (ret)
> +		goto unreg;
> +
> +	*infmt = *outfmt;
> +	priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
> +
> +	priv->current_std = V4L2_STD_UNKNOWN;
> +
> +	v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
> +		  video_device_node_name(vfd));
> +
> +	vfd->ctrl_handler = &priv->ctrl_hdlr;
> +
> +	return 0;
> +unreg:
> +	video_unregister_device(vfd);
> +	return ret;
> +}
> +
> +static void camif_unregistered(struct v4l2_subdev *sd)
> +{
> +	struct camif_priv *priv = v4l2_get_subdevdata(sd);
> +	struct video_device *vfd = &priv->vfd;
> +
> +	mutex_lock(&priv->mutex);
> +
> +	if (video_is_registered(vfd)) {
> +		video_unregister_device(vfd);
> +		media_entity_cleanup(&vfd->entity);
> +	}
> +
> +	mutex_unlock(&priv->mutex);
> +}
> +
> +static const struct v4l2_subdev_internal_ops camif_internal_ops = {
> +	.registered = camif_registered,
> +	.unregistered = camif_unregistered,
> +};
> +
> +static struct v4l2_subdev_core_ops camif_core_ops = {
> +	.ioctl = camif_ioctl,
> +};
> +
> +static struct media_entity_operations camif_entity_ops = {
> +	.link_setup = camif_link_setup,
> +	.link_validate = v4l2_subdev_link_validate,
> +};
> +
> +static struct v4l2_subdev_pad_ops camif_pad_ops = {
> +	.enum_mbus_code = camif_enum_mbus_code,
> +	.get_fmt = camif_get_fmt,
> +	.set_fmt = camif_set_fmt,
> +};
> +
> +static struct v4l2_subdev_ops camif_subdev_ops = {
> +	.pad = &camif_pad_ops,
> +	.core = &camif_core_ops,
> +};
> +
> +static int camif_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct camif_priv *priv;
> +	struct video_device *vfd;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	platform_set_drvdata(pdev, priv);
> +	priv->dev = &pdev->dev;
> +
> +	pdata = priv->dev->platform_data;
> +
> +	mutex_init(&priv->mutex);
> +	spin_lock_init(&priv->q_lock);
> +
> +	v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
> +	v4l2_set_subdevdata(&priv->sd, priv);
> +	priv->sd.internal_ops = &camif_internal_ops;
> +	priv->sd.entity.ops = &camif_entity_ops;
> +	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> +	priv->sd.dev = &pdev->dev;
> +	priv->sd.owner = THIS_MODULE;
> +	/* get our group id and camif id */
> +	priv->sd.grp_id = pdata->grp_id;
> +	priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
> +	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
> +	snprintf(camif_videodev.name, sizeof(camif_videodev.name),
> +		 "%s devnode", pdata->sd_name);
> +
> +	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +
> +	vfd = &priv->vfd;
> +	*vfd = camif_videodev;
> +	vfd->lock = &priv->mutex;
> +	vfd->queue = &priv->buffer_queue;
> +
> +	video_set_drvdata(vfd, priv);
> +
> +	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
> +
> +	ret = v4l2_async_register_subdev(&priv->sd);
> +	if (ret)
> +		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +
> +	return ret;
> +}
> +
> +static int camif_remove(struct platform_device *pdev)
> +{
> +	struct camif_priv *priv =
> +		(struct camif_priv *)platform_get_drvdata(pdev);
> +
> +	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
> +	v4l2_async_unregister_subdev(&priv->sd);
> +	media_entity_cleanup(&priv->sd.entity);
> +	v4l2_device_unregister_subdev(&priv->sd);
> +
> +	return 0;
> +}
> +
> +static const struct platform_device_id camif_ids[] = {
> +	{ .name = "imx-media-camif" },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(platform, camif_ids);
> +
> +static struct platform_driver imx_camif_driver = {
> +	.probe		= camif_probe,
> +	.remove		= camif_remove,
> +	.driver		= {
> +		.name	= "imx-media-camif",
> +	},
> +};
> +
> +module_platform_driver(imx_camif_driver);
> +
> +MODULE_DESCRIPTION("i.MX camera interface subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_LICENSE("GPL");
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  2017-01-07  2:11   ` Steve Longerbeam
@ 2017-01-20 14:48     ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:48 UTC (permalink / raw)
  To: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
> branch, modified heavily to bring forward to latest interfaces and code
> cleanup.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig       |    8 +
>  drivers/staging/media/imx/Makefile      |    2 +
>  drivers/staging/media/imx/ov5640-mipi.c | 2348 +++++++++++++++++++++++++++++++
>  3 files changed, 2358 insertions(+)
>  create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
> index ce2d2c8..09f373d 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>  	---help---
>  	  A video4linux camera capture driver for i.MX5/6.
>  
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> +       select IMX_MIPI_CSI2
> +       default y
> +       ---help---
> +         MIPI CSI-2 OV5640 Camera support.
> +
>  endmenu
>  endif
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index 0decef7..aa954c1 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -10,3 +10,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
> +
> +obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
> diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
> new file mode 100644
> index 0000000..54647a7
> --- /dev/null
> +++ b/drivers/staging/media/imx/ov5640-mipi.c
> @@ -0,0 +1,2348 @@
> +/*
> + * Copyright (c) 2014 Mentor Graphics Inc.
> + * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/init.h>
> +#include <linux/slab.h>
> +#include <linux/ctype.h>
> +#include <linux/types.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/i2c.h>
> +#include <linux/of_device.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/regulator/consumer.h>
> +#include <linux/clk.h>
> +#include <linux/clk-provider.h>
> +#include <linux/clkdev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>
> +
> +#define OV5640_VOLTAGE_ANALOG               2800000
> +#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
> +#define OV5640_VOLTAGE_DIGITAL_IO           1800000
> +
> +#define MIN_FPS 15
> +#define MAX_FPS 30
> +#define DEFAULT_FPS 30
> +
> +/* min/typical/max system clock (xclk) frequencies */
> +#define OV5640_XCLK_MIN  6000000
> +#define OV5640_XCLK_TYP 24000000
> +#define OV5640_XCLK_MAX 54000000
> +
> +/* min/typical/max pixel clock (mclk) frequencies */
> +#define OV5640_MCLK_MIN 48000000
> +#define OV5640_MCLK_TYP 48000000
> +#define OV5640_MCLK_MAX 96000000
> +
> +#define OV5640_CHIP_ID  0x300A
> +#define OV5640_SLAVE_ID 0x3100
> +#define OV5640_DEFAULT_SLAVE_ID 0x3c
> +
> +#define OV5640_MAX_CONTROLS 64
> +
> +enum ov5640_mode {
> +	ov5640_mode_MIN = 0,
> +	ov5640_mode_QCIF_176_144 = 0,
> +	ov5640_mode_QVGA_320_240,
> +	ov5640_mode_VGA_640_480,
> +	ov5640_mode_NTSC_720_480,
> +	ov5640_mode_PAL_720_576,
> +	ov5640_mode_XGA_1024_768,
> +	ov5640_mode_720P_1280_720,
> +	ov5640_mode_1080P_1920_1080,
> +	ov5640_mode_QSXGA_2592_1944,
> +	ov5640_num_modes,
> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/
> +};
> +
> +enum ov5640_frame_rate {
> +	ov5640_15_fps,
> +	ov5640_30_fps
> +};
> +
> +static int ov5640_framerates[] = {
> +	[ov5640_15_fps] = 15,
> +	[ov5640_30_fps] = 30,
> +};
> +#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
> +
> +/* image size under 1280 * 960 are SUBSAMPLING
> + * image size upper 1280 * 960 are SCALING
> + */
> +enum ov5640_downsize_mode {
> +	SUBSAMPLING,
> +	SCALING,
> +};
> +
> +struct reg_value {
> +	u16 reg_addr;
> +	u8 val;
> +	u8 mask;
> +	u32 delay_ms;
> +};
> +
> +struct ov5640_mode_info {
> +	enum ov5640_mode mode;
> +	enum ov5640_downsize_mode dn_mode;
> +	u32 width;
> +	u32 height;
> +	struct reg_value *init_data_ptr;
> +	u32 init_data_size;
> +};
> +
> +struct ov5640_dev {
> +	struct i2c_client *i2c_client;
> +	struct device *dev;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad;
> +	struct v4l2_ctrl_handler ctrl_hdl;
> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
> +	struct v4l2_mbus_framefmt fmt;
> +	struct v4l2_captureparm streamcap;
> +	struct clk *xclk; /* system clock to OV5640 */
> +	int xclk_freq;    /* requested xclk freq from devicetree */
> +
> +	enum ov5640_mode current_mode;
> +	enum ov5640_frame_rate current_fr;
> +
> +	bool on;
> +	bool awb_on;
> +	bool agc_on;
> +
> +	/* cached control settings */
> +	int ctrl_cache[OV5640_MAX_CONTROLS];

This is just duplicating the cached value in the control framework. I think this can be dropped.

> +
> +	struct gpio_desc *reset_gpio;
> +	struct gpio_desc *pwdn_gpio;
> +	struct gpio_desc *gp_gpio;
> +
> +	int prev_sysclk, prev_hts;
> +	int ae_low, ae_high, ae_target;
> +
> +	struct regulator *io_regulator;
> +	struct regulator *core_regulator;
> +	struct regulator *analog_regulator;
> +	struct regulator *gpo_regulator;
> +};
> +
> +static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct ov5640_dev, sd);
> +}
> +
> +static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
> +{
> +	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
> +}
> +
> +struct ov5640_control {
> +	struct v4l2_queryctrl ctrl;
> +	int (*set)(struct ov5640_dev *sensor, int value);
> +};
> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
> +static void ov5640_reset(struct ov5640_dev *sensor);
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_gain(struct ov5640_dev *sensor);
> +
> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
> +
> +	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
> +	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
> +	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
> +	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
> +	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
> +	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
> +	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
> +	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
> +	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
> +	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
> +	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
> +	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
> +	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
> +	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
> +	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
> +	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
> +	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
> +	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
> +	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
> +	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
> +	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
> +	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
> +	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
> +	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
> +	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
> +	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
> +	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
> +	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
> +	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
> +	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
> +	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
> +	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
> +	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
> +	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
> +	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
> +	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
> +	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
> +	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
> +	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
> +	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
> +	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
> +	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
> +	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
> +	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
> +	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
> +	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
> +	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
> +	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
> +	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
> +	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
> +	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
> +	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
> +	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
> +	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
> +	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
> +	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
> +	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
> +	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
> +	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
> +	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
> +	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
> +	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
> +	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
> +	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
> +	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
> +	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
> +	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_VGA_640_480[] = {
> +
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_VGA_640_480[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_XGA_1024_768[] = {
> +
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
> +	{0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
> +	{0x380b, 0x00, 0, 0}, {0x3035, 0x12, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_XGA_1024_768[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3808, 0x04, 0, 0},
> +	{0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0}, {0x380b, 0x00, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_QVGA_320_240[] = {
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_QVGA_320_240[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_QCIF_176_144[] = {
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +static struct reg_value ov5640_setting_15fps_QCIF_176_144[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_NTSC_720_480[] = {
> +	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_NTSC_720_480[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_PAL_720_576[] = {
> +	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_PAL_720_576[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_720P_1280_720[] = {
> +	{0x3008, 0x42, 0, 0},
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
> +	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
> +	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
> +	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
> +	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
> +	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
> +	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0}, {0x4005, 0x1a, 0, 0},
> +	{0x3008, 0x02, 0, 0}, {0x3503, 0,    0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_720P_1280_720[] = {
> +	{0x3035, 0x41, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
> +	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
> +	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
> +	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
> +	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
> +	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
> +	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_1080P_1920_1080[] = {
> +	{0x3008, 0x42, 0, 0},
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x11, 0, 0},
> +	{0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
> +	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
> +	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
> +	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
> +	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
> +	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
> +	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
> +	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
> +	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
> +	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0},
> +	{0x3503, 0, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_1080P_1920_1080[] = {
> +	{0x3008, 0x42, 0, 0},
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x21, 0, 0},
> +	{0x3036, 0x54, 0, 1}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
> +	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
> +	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
> +	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
> +	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
> +	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
> +	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
> +	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
> +	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
> +	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3503, 0, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
> +};
> +
> +static struct ov5640_mode_info
> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
> +	{
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_15fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_15fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_15fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_15fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_15fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_15fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_15fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_15fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
> +		 ov5640_setting_15fps_QSXGA_2592_1944,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
> +	}, {
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_30fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_30fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_30fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_30fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_30fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_30fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_30fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_30fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
> +	},
> +};
> +
> +static int ov5640_probe(struct i2c_client *adapter,
> +			const struct i2c_device_id *device_id);
> +static int ov5640_remove(struct i2c_client *client);
> +
> +static int ov5640_init_slave_id(struct ov5640_dev *sensor)
> +{
> +	struct i2c_msg msg;
> +	u8 buf[4];
> +	int ret;
> +
> +	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
> +		return 0;
> +
> +	buf[0] = OV5640_SLAVE_ID >> 8;
> +	buf[1] = OV5640_SLAVE_ID & 0xff;
> +	buf[2] = sensor->i2c_client->addr << 1;
> +	msg.addr = OV5640_DEFAULT_SLAVE_ID;
> +	msg.flags = 0;
> +	msg.len = 3;
> +	msg.buf = buf;
> +
> +	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
> +	if (ret < 0) {
> +		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
> +{
> +	u8 buf[3] = {0};
> +	int ret;
> +
> +	buf[0] = reg >> 8;
> +	buf[1] = reg & 0xff;
> +	buf[2] = val;
> +
> +	ret = i2c_master_send(sensor->i2c_client, buf, 3);
> +	if (ret < 0) {
> +		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
> +			__func__, reg, val);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
> +{
> +	u8 reg_buf[2] = {0};
> +	u8 read_val = 0;
> +
> +	reg_buf[0] = reg >> 8;
> +	reg_buf[1] = reg & 0xff;
> +
> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
> +			__func__, reg);
> +		return -EIO;
> +	}
> +
> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
> +			__func__, reg, read_val);
> +		return -EIO;
> +	}
> +
> +	*val = read_val;
> +	return 0;
> +}
> +
> +#define OV5640_READ_REG(s, r, v) {				\
> +		ret = ov5640_read_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +#define OV5640_WRITE_REG(s, r, v) {				\
> +		ret = ov5640_write_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
> +{
> +	u8 hi, lo;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &hi);
> +	OV5640_READ_REG(sensor, reg+1, &lo);
> +
> +	*val = ((u16)hi << 8) | (u16)lo;
> +	return 0;
> +}
> +#define OV5640_READ_REG16(s, r, v) {				\
> +		ret = ov5640_read_reg16((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, reg, val >> 8);
> +	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
> +	return 0;
> +}
> +#define OV5640_WRITE_REG16(s, r, v) {				\
> +		ret = ov5640_write_reg16((s), (r), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
> +			  u8 mask, u8 val)
> +{
> +	u8 readval;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &readval);
> +
> +	readval &= ~mask;
> +	val &= mask;
> +	val |= readval;
> +
> +	OV5640_WRITE_REG(sensor, reg, val);
> +	return 0;
> +}
> +#define OV5640_MOD_REG(s, r, m, v) {				\
> +		ret = ov5640_mod_reg((s), (r), (m), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +/* download ov5640 settings to sensor through i2c */
> +static int ov5640_load_regs(struct ov5640_dev *sensor,
> +			    struct reg_value *regs,
> +			    int size)
> +{
> +	register u32 delay_ms = 0;
> +	register u16 reg_addr = 0;
> +	register u8 mask = 0;
> +	register u8 val = 0;
> +	int i, ret;
> +
> +	for (i = 0; i < size; ++i, ++regs) {
> +		delay_ms = regs->delay_ms;
> +		reg_addr = regs->reg_addr;
> +		val = regs->val;
> +		mask = regs->mask;
> +
> +		if (mask) {
> +			OV5640_MOD_REG(sensor, reg_addr, mask, val);
> +		} else {
> +			OV5640_WRITE_REG(sensor, reg_addr, val);
> +		}
> +		if (delay_ms)
> +			usleep_range(1000*delay_ms, 1000*delay_ms+100);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
> +	return 0;
> +}
> +
> +static int ov5640_get_sysclk(struct ov5640_dev *sensor)
> +{
> +	 /* calculate sysclk */
> +	int xvclk = sensor->xclk_freq / 10000;
> +	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
> +	int sclk_rdiv_map[] = {1, 2, 4, 8};
> +	int bit_div2x = 1, sclk_rdiv, sysclk;
> +	u8 temp1, temp2;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3034, &temp1);
> +	temp2 = temp1 & 0x0f;
> +	if (temp2 == 8 || temp2 == 10)
> +		bit_div2x = temp2 / 2;
> +
> +	OV5640_READ_REG(sensor, 0x3035, &temp1);
> +	sysdiv = temp1>>4;
> +	if (sysdiv == 0)
> +		sysdiv = 16;
> +
> +	OV5640_READ_REG(sensor, 0x3036, &temp1);
> +	multiplier = temp1;
> +
> +	OV5640_READ_REG(sensor, 0x3037, &temp1);
> +	prediv = temp1 & 0x0f;
> +	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
> +
> +	OV5640_READ_REG(sensor, 0x3108, &temp1);
> +	temp2 = temp1 & 0x03;
> +	sclk_rdiv = sclk_rdiv_map[temp2];
> +
> +	VCO = xvclk * multiplier / prediv;
> +
> +	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
> +
> +	return sysclk;
> +}
> +
> +static int ov5640_set_night_mode(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u8 mode;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3a00, &mode);
> +	mode &= 0xfb;
> +	OV5640_WRITE_REG(sensor, 0x3a00, mode);
> +	return 0;
> +}
> +
> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u16 HTS;
> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380c, &HTS);
> +	return HTS;
> +}
> +
> +static int ov5640_get_VTS(struct ov5640_dev *sensor)
> +{
> +	u16 VTS;
> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380e, &VTS);
> +	return VTS;
> +}
> +
> +static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
> +	return 0;
> +}
> +
> +static int ov5640_get_light_freq(struct ov5640_dev *sensor)
> +{
> +	/* get banding filter value */
> +	u8 temp, temp1;
> +	int light_freq = 0;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3c01, &temp);
> +
> +	if (temp & 0x80) {
> +		/* manual */
> +		OV5640_READ_REG(sensor, 0x3c00, &temp1);
> +		if (temp1 & 0x04) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +			light_freq = 60;
> +		}
> +	} else {
> +		/* auto */
> +		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
> +		if (temp1 & 0x01) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +		}
> +	}
> +
> +	return light_freq;
> +}
> +
> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
> +{
> +	int prev_vts;
> +	int band_step60, max_band60, band_step50, max_band50;
> +	int ret;
> +
> +	/* read preview PCLK */
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_sysclk = ret;
> +	/* read preview HTS */
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_hts = ret;
> +
> +	/* read preview VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_vts = ret;
> +
> +	/* calculate banding filter */
> +	/* 60Hz */
> +	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
> +	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
> +
> +	max_band60 = (int)((prev_vts-4)/band_step60);
> +	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
> +
> +	/* 50Hz */
> +	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
> +	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
> +
> +	max_band50 = (int)((prev_vts-4)/band_step50);
> +	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
> +{
> +	/* stable in high */
> +	int fast_high, fast_low;
> +	int ret;
> +
> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
> +
> +	fast_high = sensor->ae_high<<1;
> +	if (fast_high > 255)
> +		fast_high = 255;
> +
> +	fast_low = sensor->ae_low >> 1;
> +
> +	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
> +
> +	return 0;
> +}
> +
> +static int ov5640_binning_on(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3821, &temp);
> +	temp &= 0xfe;
> +
> +	return temp ? 1 : 0;
> +}
> +
> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
> +{
> +	u8 temp, channel = sensor->ep.base.id;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x4814, &temp);
> +	temp &= ~(3 << 6);
> +	temp |= (channel << 6);
> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
> +
> +	return 0;
> +}
> +
> +static enum ov5640_mode
> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
> +			 int width, int height)
> +{
> +	int i;
> +
> +	for (i = ov5640_num_modes - 1; i >= 0; i--) {
> +		if (ov5640_mode_info_data[0][i].width <= width &&
> +		    ov5640_mode_info_data[0][i].height <= height)
> +			break;
> +	}
> +
> +	if (i < 0)
> +		i = 0;
> +
> +	return (enum ov5640_mode)i;
> +}
> +
> +/*
> + * sensor changes between scaling and subsampling, go through
> + * exposure calculation
> + */
> +static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
> +					    enum ov5640_frame_rate frame_rate,
> +					    enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	u8 average;
> +	int prev_shutter, prev_gain16;
> +	int cap_shutter, cap_gain16;
> +	int cap_sysclk, cap_hts, cap_vts;
> +	int light_freq, cap_bandfilt, cap_maxband;
> +	long cap_gain16_shutter;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* auto focus */
> +	/* ov5640_auto_focus();//if no af function, just skip it */
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read preview shutter */
> +	ret = ov5640_get_exposure(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_shutter = ret;
> +	ret = ov5640_binning_on(sensor);
> +	if (ret < 0)
> +		return ret;
> +	if (ret && mode != ov5640_mode_720P_1280_720 &&
> +	    mode != ov5640_mode_1080P_1920_1080)
> +		prev_shutter *= 2;
> +
> +	/* read preview gain */
> +	ret = ov5640_get_gain(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_gain16 = ret;
> +
> +	/* get average */
> +	OV5640_READ_REG(sensor, 0x56a1, &average);
> +
> +	/* turn off night mode for capture */
> +	ret = ov5640_set_night_mode(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* turn off overlay */
> +	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
> +	   just skip it */
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read capture VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_vts = ret;
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_hts = ret;
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_sysclk = ret;
> +
> +	/* calculate capture banding filter */
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	light_freq = ret;
> +
> +	if (light_freq == 60) {
> +		/* 60Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
> +	} else {
> +		/* 50Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts;
> +	}
> +	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
> +
> +	/* calculate capture shutter/gain16 */
> +	if (average > sensor->ae_low && average < sensor->ae_high) {
> +		/* in stable range */
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts *
> +			sensor->ae_target / average;
> +	} else {
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts;
> +	}
> +
> +	/* gain to shutter */
> +	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
> +		/* shutter < 1/100 */
> +		cap_shutter = cap_gain16_shutter / 16;
> +		if (cap_shutter < 1)
> +			cap_shutter = 1;
> +
> +		cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		if (cap_gain16 < 16)
> +			cap_gain16 = 16;
> +	} else {
> +		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
> +			/* exposure reach max */
> +			cap_shutter = cap_bandfilt * cap_maxband;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		} else {
> +			/* 1/100 < (cap_shutter = n/100) =< max */
> +			cap_shutter =
> +				((int)(cap_gain16_shutter / 16 / cap_bandfilt))
> +				* cap_bandfilt;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		}
> +	}
> +
> +	/* write capture gain */
> +	ret = ov5640_set_gain(sensor, cap_gain16);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* write capture shutter */
> +	if (cap_shutter > (cap_vts - 4)) {
> +		cap_vts = cap_shutter + 4;
> +		ret = ov5640_set_VTS(sensor, cap_vts);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	ret = ov5640_set_exposure(sensor, cap_shutter);
> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_stream(sensor, true);
> +}
> +
> +/*
> + * if sensor changes inside scaling or subsampling
> + * change mode directly
> + */
> +static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
> +				     enum ov5640_frame_rate frame_rate,
> +				     enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, true);
> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_agc(sensor, true);
> +}
> +
> +static int ov5640_change_mode(struct ov5640_dev *sensor,
> +			      enum ov5640_frame_rate frame_rate,
> +			      enum ov5640_mode mode,
> +			      enum ov5640_mode orig_mode)
> +{
> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;
> +
> +	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
> +	    mode != ov5640_mode_INIT) {
> +		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
> +		return -EINVAL;
> +	}
> +
> +	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
> +	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
> +	if (mode == ov5640_mode_INIT) {
> +		mode_data = ov5640_init_setting_30fps_VGA;
> +		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
> +
> +		sensor->fmt.width = 640;
> +		sensor->fmt.height = 480;
> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +		if (ret < 0)
> +			return ret;
> +
> +		mode_data = ov5640_setting_30fps_VGA_640_480;
> +		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
> +			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
> +		/* change between subsampling and scaling
> +		 * go through exposure calucation */
> +		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
> +							  mode);
> +	} else {
> +		/* change inside subsampling or scaling
> +		 * download firmware directly */
> +		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
> +	}
> +
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_bandingfilter(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_virtual_channel(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* restore controls */
> +	ov5640_restore_ctrls(sensor);
> +
> +	if (ret >= 0 && mode != ov5640_mode_INIT) {
> +		sensor->current_mode = mode;
> +		sensor->current_fr = frame_rate;
> +	}
> +
> +	return 0;
> +}
> +
> +/* restore the last set video mode after chip power-on */
> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
> +{
> +	int ret = 0;
> +
> +	/* first we need to set some initial register values */
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				    ov5640_mode_INIT, ov5640_mode_INIT);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* now restore the last capture mode */
> +	return ov5640_change_mode(sensor,
> +				  sensor->current_fr,
> +				  sensor->current_mode,
> +				  ov5640_mode_VGA_640_480);
> +}
> +
> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
> +{
> +	int ret;
> +
> +	if (sensor->io_regulator) {
> +		ret = regulator_enable(sensor->io_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->core_regulator) {
> +		ret = regulator_enable(sensor->core_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->gpo_regulator) {
> +		ret = regulator_enable(sensor->gpo_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->analog_regulator) {
> +		ret = regulator_enable(sensor->analog_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static void ov5640_regulators_off(struct ov5640_dev *sensor)
> +{
> +	if (sensor->analog_regulator)
> +		regulator_disable(sensor->analog_regulator);
> +	if (sensor->core_regulator)
> +		regulator_disable(sensor->core_regulator);
> +	if (sensor->io_regulator)
> +		regulator_disable(sensor->io_regulator);
> +	if (sensor->gpo_regulator)
> +		regulator_disable(sensor->gpo_regulator);
> +}
> +
> +/* --------------- Subdev Operations --------------- */
> +
> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	int ret;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (on && !sensor->on) {
> +		if (sensor->xclk)
> +			clk_prepare_enable(sensor->xclk);
> +
> +		ret = ov5640_regulators_on(sensor);
> +		if (ret)
> +			return ret;
> +
> +		ov5640_reset(sensor);
> +		ov5640_power(sensor, true);
> +
> +		ret = ov5640_init_slave_id(sensor);
> +		if (ret)
> +			return ret;
> +
> +		ret = ov5640_restore_mode(sensor);
> +		if (ret)
> +			return ret;
> +
> +		/*
> +		 * NOTE: Freescale adds a long delay (600 msec) after
> +		 * powering up and programming a mode on the ov5640-mipi
> +		 * camera (search for "msec_wait4stable" in FSL's
> +		 * ov5640_mipi.c), which equivalently would need to go
> +		 * right here. If we run into MIPI CSI-2 receiver dphy
> +		 * ready timeouts, it might be a clue to add that delay
> +		 * here.
> +		 */
> +	} else if (!on && sensor->on) {
> +		ov5640_power(sensor, false);
> +
> +		ov5640_regulators_off(sensor);
> +
> +		if (sensor->xclk)
> +			clk_disable_unprepare(sensor->xclk);
> +	}
> +
> +	sensor->on = on;
> +
> +	return 0;
> +}
> +
> +static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_captureparm *cparm = &a->parm.capture;
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* This is the only case currently handled. */
> +	memset(a, 0, sizeof(*a));
> +	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	cparm->capability = sensor->streamcap.capability;
> +	cparm->timeperframe = sensor->streamcap.timeperframe;
> +	cparm->capturemode = sensor->streamcap.capturemode;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
> +	enum ov5640_frame_rate frame_rate;
> +	u32 tgt_fps;	/* target frames per secound */
> +	int ret = 0;
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* Check that the new frame rate is allowed. */
> +	if ((timeperframe->numerator == 0) ||
> +	    (timeperframe->denominator == 0)) {
> +		timeperframe->denominator = DEFAULT_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps > MAX_FPS) {
> +		timeperframe->denominator = MAX_FPS;
> +		timeperframe->numerator = 1;
> +	} else if (tgt_fps < MIN_FPS) {
> +		timeperframe->denominator = MIN_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	/* Actual frame rate we use */
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps == 15)
> +		frame_rate = ov5640_15_fps;
> +	else if (tgt_fps == 30)
> +		frame_rate = ov5640_30_fps;
> +	else {
> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
> +			 tgt_fps);
> +		return -EINVAL;
> +	}
> +
> +	ret = ov5640_change_mode(sensor, frame_rate,
> +				 sensor->current_mode,
> +				 sensor->current_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	sensor->streamcap.timeperframe = *timeperframe;
> +
> +	return 0;
> +}
> +
> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	format->format = sensor->fmt;
> +
> +	return 0;
> +}
> +
> +static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
> +				   struct v4l2_mbus_framefmt *fmt,
> +				   enum ov5640_mode *new_mode)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
> +
> +	fmt->width = ov5640_mode_info_data[0][mode].width;
> +	fmt->height = ov5640_mode_info_data[0][mode].height;
> +	fmt->code = sensor->fmt.code;
> +
> +	if (new_mode)
> +		*new_mode = mode;
> +	return 0;
> +}
> +
> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode new_mode;
> +	int ret;
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
> +		if (ret)
> +			return ret;
> +		cfg->try_fmt = format->format;
> +		return 0;
> +	}
> +
> +	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
> +	if (ret)
> +		return ret;
> +
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				 new_mode, sensor->current_mode);
> +	if (ret >= 0)
> +		sensor->fmt = format->format;
> +
> +	return ret;
> +}
> +
> +
> +/*
> + * Sensor Controls.
> + */
> +
> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
> +		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
> +		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	sensor->awb_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
> +	return 0;
> +}
> +
> +static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +#if 0
> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +#endif
> +
> +static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
> +{
> +	u16 max_exp = 0;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
> +	if (value < max_exp) {
> +		u32 exp = value << 4;
> +
> +		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
> +	}
> +
> +	return 0;
> +}
> +
> +/* read exposure, in number of line periods */
> +static int ov5640_get_exposure(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int exp, ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG(sensor, 0x3500, &temp);
> +	exp = ((int)temp & 0x0f) << 16;
> +	OV5640_READ_REG(sensor, 0x3501, &temp);
> +	exp |= ((int)temp << 8);
> +	OV5640_READ_REG(sensor, 0x3502, &temp);
> +	exp |= (int)temp;
> +
> +	return exp >> 4;
> +}
> +
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	/* this enables/disables both AEC and AGC */
> +	sensor->agc_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
> +	return 0;
> +}
> +
> +static int ov5640_get_gain(struct ov5640_dev *sensor)
> +{
> +	u16 gain;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
> +
> +	return gain & 0x3ff;
> +}
> +
> +#if 0
> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
> +	return 0;
> +}
> +#endif
> +
> +static struct ov5640_control ov5640_ctrls[] = {
> +	{
> +		.set = ov5640_set_agc,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTOGAIN,
> +			.name = "Auto Gain/Exposure Control",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_exposure,
> +		.ctrl = {
> +			.id = V4L2_CID_EXPOSURE,
> +			.name = "Exposure",
> +			.minimum = 0,
> +			.maximum = 65535,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_gain,
> +		.ctrl = {
> +			.id = V4L2_CID_GAIN,
> +			.name = "Gain",
> +			.minimum = 0,
> +			.maximum = 1023,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_hue,
> +		.ctrl = {
> +			.id = V4L2_CID_HUE,
> +			.name = "Hue",
> +			.minimum = 0,
> +			.maximum = 359,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_contrast,
> +		.ctrl = {
> +			.id = V4L2_CID_CONTRAST,
> +			.name = "Contrast",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_saturation,
> +		.ctrl = {
> +			.id = V4L2_CID_SATURATION,
> +			.name = "Saturation",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 64,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_awb,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTO_WHITE_BALANCE,
> +			.name = "Auto White Balance",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_red_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_RED_BALANCE,
> +			.name = "Red Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_blue_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_BLUE_BALANCE,
> +			.name = "Blue Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	},
> +};
> +#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)

This should use v4l2_ctrl_new_std() instead of this array.
Just put a switch on ctrl->id in s_ctrl, and each case calls the corresponding
set function.

> +
> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
> +{
> +	struct ov5640_control *ret = NULL;
> +	int i;
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		if (id == ov5640_ctrls[i].ctrl.id) {
> +			ret = &ov5640_ctrls[i];
> +			break;
> +		}
> +	}
> +
> +	if (ret && index)
> +		*index = i;
> +	return ret;
> +}
> +
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;
> +	int i;
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +		c->set(sensor, sensor->ctrl_cache[i]);
> +	}
> +
> +	return 0;
> +}

This does the same as v4l2_ctrl_handler_setup() if I understand the code correctly.

> +
> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
> +	struct ov5640_control *c;
> +	int ret = 0;
> +	int i;
> +
> +	c = ov5640_get_ctrl(ctrl->id, &i);
> +	if (!c)
> +		return -EINVAL;
> +
> +	ret = c->set(sensor, ctrl->val);
> +	/* update cached value if no error */
> +	if (!ret)
> +		sensor->ctrl_cache[i] = ctrl->val;
> +
> +	return ret;
> +}
> +
> +static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
> +	.s_ctrl = ov5640_s_ctrl,
> +};
> +
> +static int ov5640_init_controls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;
> +	int i;
> +
> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +
> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
> +				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
> +				  c->ctrl.step, c->ctrl.default_value);
> +	}

As mentioned, just drop the ov5640_ctrls array and call v4l2_ctr_new_std for each
control you're adding.

> +
> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
> +	if (sensor->ctrl_hdl.error) {
> +		int err = sensor->ctrl_hdl.error;
> +
> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
> +		return err;
> +	}
> +	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);
> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_frame_size_enum *fse)
> +{
> +	if (fse->pad != 0)
> +		return -EINVAL;
> +	if (fse->index >= ov5640_num_modes)
> +		return -EINVAL;
> +
> +	fse->min_width = fse->max_width =
> +		ov5640_mode_info_data[0][fse->index].width;
> +	fse->min_height = fse->max_height =
> +		ov5640_mode_info_data[0][fse->index].height;
> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_interval(
> +	struct v4l2_subdev *sd,
> +	struct v4l2_subdev_pad_config *cfg,
> +	struct v4l2_subdev_frame_interval_enum *fie)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	if (fie->pad != 0)
> +		return -EINVAL;
> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
> +		return -EINVAL;
> +
> +	if (fie->width == 0 || fie->height == 0)
> +		return -EINVAL;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
> +
> +	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
> +		return -EINVAL;
> +
> +	fie->interval.numerator = 1;
> +	fie->interval.denominator = ov5640_framerates[fie->index];
> +
> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
> +		fie->width, fie->height, fie->index, fie->interval.denominator);
> +	return 0;
> +}
> +
> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
> +			    u32 output, u32 config)
> +{
> +	return (input != 0) ? -EINVAL : 0;
> +}
> +
> +static int ov5640_enum_mbus_code(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	if (code->pad != 0)
> +		return -EINVAL;
> +	if (code->index != 0)
> +		return -EINVAL;
> +
> +	code->code = sensor->fmt.code;
> +
> +	return 0;
> +}
> +
> +static int ov5640_g_mbus_config(struct v4l2_subdev *sd,
> +				struct v4l2_mbus_config *cfg)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	cfg->type = V4L2_MBUS_CSI2;
> +	cfg->flags = sensor->ep.bus.mipi_csi2.flags;
> +	cfg->flags |= (1 << (sensor->ep.bus.mipi_csi2.num_data_lanes - 1));
> +	cfg->flags |= V4L2_MBUS_CSI2_CHANNEL_0;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	return ov5640_set_stream(sensor, enable);
> +}
> +
> +static struct v4l2_subdev_core_ops ov5640_core_ops = {
> +	.s_power = ov5640_s_power,
> +};
> +
> +static struct v4l2_subdev_video_ops ov5640_video_ops = {
> +	.s_parm = ov5640_s_parm,
> +	.g_parm = ov5640_g_parm,
> +	.g_input_status = ov5640_g_input_status,
> +	.s_routing = ov5640_s_routing,
> +	.g_mbus_config  = ov5640_g_mbus_config,
> +	.s_stream = ov5640_s_stream,
> +};
> +
> +static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
> +	.enum_mbus_code = ov5640_enum_mbus_code,
> +	.get_fmt = ov5640_get_fmt,
> +	.set_fmt = ov5640_set_fmt,
> +	.enum_frame_size = ov5640_enum_frame_size,
> +	.enum_frame_interval = ov5640_enum_frame_interval,
> +};
> +
> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
> +	.core = &ov5640_core_ops,
> +	.video = &ov5640_video_ops,
> +	.pad = &ov5640_pad_ops,
> +};
> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
> +{
> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
> +}
> +
> +static void ov5640_reset(struct ov5640_dev *sensor)
> +{
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +
> +	/* camera power cycle */
> +	ov5640_power(sensor, false);
> +	usleep_range(5000, 10000);
> +	ov5640_power(sensor, true);
> +	usleep_range(5000, 10000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 1);
> +	usleep_range(1000, 2000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +	usleep_range(5000, 10000);
> +}
> +
> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
> +{
> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
> +	if (!IS_ERR(sensor->io_regulator)) {
> +		regulator_set_voltage(sensor->io_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_IO,
> +				      OV5640_VOLTAGE_DIGITAL_IO);
> +	} else {
> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
> +			__func__);
> +		sensor->io_regulator = NULL;
> +	}
> +
> +	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
> +	if (!IS_ERR(sensor->core_regulator)) {
> +		regulator_set_voltage(sensor->core_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_CORE,
> +				      OV5640_VOLTAGE_DIGITAL_CORE);
> +	} else {
> +		sensor->core_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
> +			__func__);
> +	}
> +
> +	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
> +	if (!IS_ERR(sensor->analog_regulator)) {
> +		regulator_set_voltage(sensor->analog_regulator,
> +				      OV5640_VOLTAGE_ANALOG,
> +				      OV5640_VOLTAGE_ANALOG);
> +	} else {
> +		sensor->analog_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
> +			__func__);
> +	}
> +}
> +
> +static int ov5640_probe(struct i2c_client *client,
> +			const struct i2c_device_id *id)
> +{
> +	struct device *dev = &client->dev;
> +	struct device_node *endpoint;
> +	struct ov5640_dev *sensor;
> +	int i, xclk, ret;
> +
> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
> +	if (!sensor)
> +		return -ENOMEM;
> +
> +	sensor->i2c_client = client;
> +	sensor->dev = dev;
> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
> +	sensor->fmt.width = 640;
> +	sensor->fmt.height = 480;
> +	sensor->fmt.field = V4L2_FIELD_NONE;
> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
> +					   V4L2_CAP_TIMEPERFRAME;
> +	sensor->streamcap.capturemode = 0;
> +	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
> +	sensor->streamcap.timeperframe.numerator = 1;
> +
> +	sensor->current_mode = ov5640_mode_VGA_640_480;
> +	sensor->current_fr = ov5640_30_fps;
> +
> +	sensor->ae_target = 52;
> +
> +	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
> +	if (!endpoint) {
> +		dev_err(dev, "endpoint node not found\n");
> +		return -EINVAL;
> +	}
> +
> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
> +		return -EINVAL;
> +	}
> +	of_node_put(endpoint);
> +
> +	/* get system clock (xclk) frequency */
> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
> +	if (!ret) {
> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
> +			dev_err(dev, "invalid xclk frequency\n");
> +			return -EINVAL;
> +		}
> +		sensor->xclk_freq = xclk;
> +	}
> +
> +	/* get system clock (xclk) */
> +	sensor->xclk = devm_clk_get(dev, "xclk");
> +	if (!IS_ERR(sensor->xclk)) {
> +		if (!sensor->xclk_freq) {
> +			dev_err(dev, "xclk requires xclk frequency!\n");
> +			return -EINVAL;
> +		}
> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
> +	} else {
> +		/* assume system clock enabled by default */
> +		sensor->xclk = NULL;
> +	}
> +
> +	/* request power down pin */
> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
> +	if (IS_ERR(sensor->pwdn_gpio)) {
> +		dev_err(dev, "request for power down gpio failed\n");
> +		return PTR_ERR(sensor->pwdn_gpio);
> +	}
> +
> +	/* request reset pin */
> +	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
> +	if (IS_ERR(sensor->reset_gpio)) {
> +		dev_err(dev, "request for reset gpio failed\n");
> +		return PTR_ERR(sensor->reset_gpio);
> +	}
> +
> +	/* initialize the cached controls to their defaults */
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		struct ov5640_control *c = &ov5640_ctrls[i];
> +
> +		sensor->ctrl_cache[i] = c->ctrl.default_value;
> +	}
> +	sensor->awb_on = sensor->agc_on = true;
> +
> +	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
> +
> +	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
> +	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> +	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
> +	if (ret)
> +		return ret;
> +
> +	ov5640_get_regulators(sensor);
> +
> +	ret = ov5640_s_power(&sensor->sd, 1);
> +	if (ret)
> +		goto entity_cleanup;
> +	ret = ov5640_init_controls(sensor);
> +	if (ret)
> +		goto power_off;
> +
> +	ret = ov5640_s_power(&sensor->sd, 0);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	ret = v4l2_async_register_subdev(&sensor->sd);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +power_off:
> +	ov5640_s_power(&sensor->sd, 0);
> +entity_cleanup:
> +	media_entity_cleanup(&sensor->sd.entity);
> +	ov5640_regulators_off(sensor);
> +	return ret;
> +}
> +
> +/*!
> + * ov5640 I2C detach function
> + *
> + * @param client            struct i2c_client *
> + * @return  Error code indicating success or failure
> + */
> +static int ov5640_remove(struct i2c_client *client)
> +{
> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	ov5640_regulators_off(sensor);
> +
> +	v4l2_async_unregister_subdev(&sensor->sd);
> +	media_entity_cleanup(&sensor->sd.entity);
> +	v4l2_device_unregister_subdev(sd);
> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +	return 0;
> +}
> +
> +static const struct i2c_device_id ov5640_id[] = {
> +	{"ov5640", 0},
> +	{},
> +};
> +MODULE_DEVICE_TABLE(i2c, ov5640_id);
> +
> +static const struct of_device_id ov5640_dt_ids[] = {
> +	{ .compatible = "ovti,ov5640" },
> +	{ /* sentinel */ }
> +};
> +MODULE_DEVICE_TABLE(of, ov5640_dt_ids);
> +
> +static struct i2c_driver ov5640_i2c_driver = {
> +	.driver = {
> +		.name  = "ov5640",
> +		.of_match_table	= ov5640_dt_ids,
> +	},
> +	.id_table = ov5640_id,
> +	.probe    = ov5640_probe,
> +	.remove   = ov5640_remove,
> +};
> +
> +module_i2c_driver(ov5640_i2c_driver);
> +
> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
> +MODULE_LICENSE("GPL");
> +MODULE_VERSION("1.0");
> 

Same comments apply to the next patch, so I won't repeat them.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-20 14:48     ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 14:48 UTC (permalink / raw)
  To: linux-arm-kernel

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
> branch, modified heavily to bring forward to latest interfaces and code
> cleanup.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig       |    8 +
>  drivers/staging/media/imx/Makefile      |    2 +
>  drivers/staging/media/imx/ov5640-mipi.c | 2348 +++++++++++++++++++++++++++++++
>  3 files changed, 2358 insertions(+)
>  create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
> index ce2d2c8..09f373d 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>  	---help---
>  	  A video4linux camera capture driver for i.MX5/6.
>  
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> +       select IMX_MIPI_CSI2
> +       default y
> +       ---help---
> +         MIPI CSI-2 OV5640 Camera support.
> +
>  endmenu
>  endif
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index 0decef7..aa954c1 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -10,3 +10,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
> +
> +obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
> diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
> new file mode 100644
> index 0000000..54647a7
> --- /dev/null
> +++ b/drivers/staging/media/imx/ov5640-mipi.c
> @@ -0,0 +1,2348 @@
> +/*
> + * Copyright (c) 2014 Mentor Graphics Inc.
> + * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/init.h>
> +#include <linux/slab.h>
> +#include <linux/ctype.h>
> +#include <linux/types.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/i2c.h>
> +#include <linux/of_device.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/regulator/consumer.h>
> +#include <linux/clk.h>
> +#include <linux/clk-provider.h>
> +#include <linux/clkdev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>
> +
> +#define OV5640_VOLTAGE_ANALOG               2800000
> +#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
> +#define OV5640_VOLTAGE_DIGITAL_IO           1800000
> +
> +#define MIN_FPS 15
> +#define MAX_FPS 30
> +#define DEFAULT_FPS 30
> +
> +/* min/typical/max system clock (xclk) frequencies */
> +#define OV5640_XCLK_MIN  6000000
> +#define OV5640_XCLK_TYP 24000000
> +#define OV5640_XCLK_MAX 54000000
> +
> +/* min/typical/max pixel clock (mclk) frequencies */
> +#define OV5640_MCLK_MIN 48000000
> +#define OV5640_MCLK_TYP 48000000
> +#define OV5640_MCLK_MAX 96000000
> +
> +#define OV5640_CHIP_ID  0x300A
> +#define OV5640_SLAVE_ID 0x3100
> +#define OV5640_DEFAULT_SLAVE_ID 0x3c
> +
> +#define OV5640_MAX_CONTROLS 64
> +
> +enum ov5640_mode {
> +	ov5640_mode_MIN = 0,
> +	ov5640_mode_QCIF_176_144 = 0,
> +	ov5640_mode_QVGA_320_240,
> +	ov5640_mode_VGA_640_480,
> +	ov5640_mode_NTSC_720_480,
> +	ov5640_mode_PAL_720_576,
> +	ov5640_mode_XGA_1024_768,
> +	ov5640_mode_720P_1280_720,
> +	ov5640_mode_1080P_1920_1080,
> +	ov5640_mode_QSXGA_2592_1944,
> +	ov5640_num_modes,
> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/
> +};
> +
> +enum ov5640_frame_rate {
> +	ov5640_15_fps,
> +	ov5640_30_fps
> +};
> +
> +static int ov5640_framerates[] = {
> +	[ov5640_15_fps] = 15,
> +	[ov5640_30_fps] = 30,
> +};
> +#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
> +
> +/* image size under 1280 * 960 are SUBSAMPLING
> + * image size upper 1280 * 960 are SCALING
> + */
> +enum ov5640_downsize_mode {
> +	SUBSAMPLING,
> +	SCALING,
> +};
> +
> +struct reg_value {
> +	u16 reg_addr;
> +	u8 val;
> +	u8 mask;
> +	u32 delay_ms;
> +};
> +
> +struct ov5640_mode_info {
> +	enum ov5640_mode mode;
> +	enum ov5640_downsize_mode dn_mode;
> +	u32 width;
> +	u32 height;
> +	struct reg_value *init_data_ptr;
> +	u32 init_data_size;
> +};
> +
> +struct ov5640_dev {
> +	struct i2c_client *i2c_client;
> +	struct device *dev;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad;
> +	struct v4l2_ctrl_handler ctrl_hdl;
> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
> +	struct v4l2_mbus_framefmt fmt;
> +	struct v4l2_captureparm streamcap;
> +	struct clk *xclk; /* system clock to OV5640 */
> +	int xclk_freq;    /* requested xclk freq from devicetree */
> +
> +	enum ov5640_mode current_mode;
> +	enum ov5640_frame_rate current_fr;
> +
> +	bool on;
> +	bool awb_on;
> +	bool agc_on;
> +
> +	/* cached control settings */
> +	int ctrl_cache[OV5640_MAX_CONTROLS];

This is just duplicating the cached value in the control framework. I think this can be dropped.

> +
> +	struct gpio_desc *reset_gpio;
> +	struct gpio_desc *pwdn_gpio;
> +	struct gpio_desc *gp_gpio;
> +
> +	int prev_sysclk, prev_hts;
> +	int ae_low, ae_high, ae_target;
> +
> +	struct regulator *io_regulator;
> +	struct regulator *core_regulator;
> +	struct regulator *analog_regulator;
> +	struct regulator *gpo_regulator;
> +};
> +
> +static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct ov5640_dev, sd);
> +}
> +
> +static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
> +{
> +	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
> +}
> +
> +struct ov5640_control {
> +	struct v4l2_queryctrl ctrl;
> +	int (*set)(struct ov5640_dev *sensor, int value);
> +};
> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
> +static void ov5640_reset(struct ov5640_dev *sensor);
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_gain(struct ov5640_dev *sensor);
> +
> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
> +
> +	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
> +	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
> +	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
> +	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
> +	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
> +	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
> +	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
> +	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
> +	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
> +	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
> +	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
> +	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
> +	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
> +	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
> +	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
> +	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
> +	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
> +	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
> +	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
> +	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
> +	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
> +	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
> +	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
> +	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
> +	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
> +	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
> +	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
> +	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
> +	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
> +	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
> +	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
> +	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
> +	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
> +	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
> +	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
> +	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
> +	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
> +	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
> +	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
> +	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
> +	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
> +	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
> +	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
> +	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
> +	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
> +	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
> +	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
> +	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
> +	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
> +	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
> +	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
> +	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
> +	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
> +	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
> +	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
> +	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
> +	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
> +	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
> +	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
> +	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
> +	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
> +	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
> +	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
> +	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
> +	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
> +	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
> +	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_VGA_640_480[] = {
> +
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_VGA_640_480[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_XGA_1024_768[] = {
> +
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
> +	{0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
> +	{0x380b, 0x00, 0, 0}, {0x3035, 0x12, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_XGA_1024_768[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3808, 0x04, 0, 0},
> +	{0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0}, {0x380b, 0x00, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_QVGA_320_240[] = {
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_QVGA_320_240[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_QCIF_176_144[] = {
> +	{0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +static struct reg_value ov5640_setting_15fps_QCIF_176_144[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
> +	{0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_NTSC_720_480[] = {
> +	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_NTSC_720_480[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_PAL_720_576[] = {
> +	{0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_PAL_720_576[] = {
> +	{0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_720P_1280_720[] = {
> +	{0x3008, 0x42, 0, 0},
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
> +	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
> +	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
> +	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
> +	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
> +	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
> +	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0}, {0x4005, 0x1a, 0, 0},
> +	{0x3008, 0x02, 0, 0}, {0x3503, 0,    0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_720P_1280_720[] = {
> +	{0x3035, 0x41, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
> +	{0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
> +	{0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
> +	{0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
> +	{0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
> +	{0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
> +	{0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
> +	{0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_30fps_1080P_1920_1080[] = {
> +	{0x3008, 0x42, 0, 0},
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x11, 0, 0},
> +	{0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
> +	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
> +	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
> +	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
> +	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
> +	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
> +	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
> +	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
> +	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
> +	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0},
> +	{0x3503, 0, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_1080P_1920_1080[] = {
> +	{0x3008, 0x42, 0, 0},
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x21, 0, 0},
> +	{0x3036, 0x54, 0, 1}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
> +	{0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
> +	{0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
> +	{0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
> +	{0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
> +	{0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
> +	{0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
> +	{0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
> +	{0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
> +	{0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3503, 0, 0, 0},
> +};
> +
> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
> +};
> +
> +static struct ov5640_mode_info
> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
> +	{
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_15fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_15fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_15fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_15fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_15fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_15fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_15fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_15fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
> +		 ov5640_setting_15fps_QSXGA_2592_1944,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
> +	}, {
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_30fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_30fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_30fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_30fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_30fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_30fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_30fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_30fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
> +	},
> +};
> +
> +static int ov5640_probe(struct i2c_client *adapter,
> +			const struct i2c_device_id *device_id);
> +static int ov5640_remove(struct i2c_client *client);
> +
> +static int ov5640_init_slave_id(struct ov5640_dev *sensor)
> +{
> +	struct i2c_msg msg;
> +	u8 buf[4];
> +	int ret;
> +
> +	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
> +		return 0;
> +
> +	buf[0] = OV5640_SLAVE_ID >> 8;
> +	buf[1] = OV5640_SLAVE_ID & 0xff;
> +	buf[2] = sensor->i2c_client->addr << 1;
> +	msg.addr = OV5640_DEFAULT_SLAVE_ID;
> +	msg.flags = 0;
> +	msg.len = 3;
> +	msg.buf = buf;
> +
> +	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
> +	if (ret < 0) {
> +		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
> +{
> +	u8 buf[3] = {0};
> +	int ret;
> +
> +	buf[0] = reg >> 8;
> +	buf[1] = reg & 0xff;
> +	buf[2] = val;
> +
> +	ret = i2c_master_send(sensor->i2c_client, buf, 3);
> +	if (ret < 0) {
> +		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
> +			__func__, reg, val);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
> +{
> +	u8 reg_buf[2] = {0};
> +	u8 read_val = 0;
> +
> +	reg_buf[0] = reg >> 8;
> +	reg_buf[1] = reg & 0xff;
> +
> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
> +			__func__, reg);
> +		return -EIO;
> +	}
> +
> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
> +			__func__, reg, read_val);
> +		return -EIO;
> +	}
> +
> +	*val = read_val;
> +	return 0;
> +}
> +
> +#define OV5640_READ_REG(s, r, v) {				\
> +		ret = ov5640_read_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +#define OV5640_WRITE_REG(s, r, v) {				\
> +		ret = ov5640_write_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
> +{
> +	u8 hi, lo;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &hi);
> +	OV5640_READ_REG(sensor, reg+1, &lo);
> +
> +	*val = ((u16)hi << 8) | (u16)lo;
> +	return 0;
> +}
> +#define OV5640_READ_REG16(s, r, v) {				\
> +		ret = ov5640_read_reg16((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, reg, val >> 8);
> +	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
> +	return 0;
> +}
> +#define OV5640_WRITE_REG16(s, r, v) {				\
> +		ret = ov5640_write_reg16((s), (r), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
> +			  u8 mask, u8 val)
> +{
> +	u8 readval;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &readval);
> +
> +	readval &= ~mask;
> +	val &= mask;
> +	val |= readval;
> +
> +	OV5640_WRITE_REG(sensor, reg, val);
> +	return 0;
> +}
> +#define OV5640_MOD_REG(s, r, m, v) {				\
> +		ret = ov5640_mod_reg((s), (r), (m), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +/* download ov5640 settings to sensor through i2c */
> +static int ov5640_load_regs(struct ov5640_dev *sensor,
> +			    struct reg_value *regs,
> +			    int size)
> +{
> +	register u32 delay_ms = 0;
> +	register u16 reg_addr = 0;
> +	register u8 mask = 0;
> +	register u8 val = 0;
> +	int i, ret;
> +
> +	for (i = 0; i < size; ++i, ++regs) {
> +		delay_ms = regs->delay_ms;
> +		reg_addr = regs->reg_addr;
> +		val = regs->val;
> +		mask = regs->mask;
> +
> +		if (mask) {
> +			OV5640_MOD_REG(sensor, reg_addr, mask, val);
> +		} else {
> +			OV5640_WRITE_REG(sensor, reg_addr, val);
> +		}
> +		if (delay_ms)
> +			usleep_range(1000*delay_ms, 1000*delay_ms+100);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
> +	return 0;
> +}
> +
> +static int ov5640_get_sysclk(struct ov5640_dev *sensor)
> +{
> +	 /* calculate sysclk */
> +	int xvclk = sensor->xclk_freq / 10000;
> +	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
> +	int sclk_rdiv_map[] = {1, 2, 4, 8};
> +	int bit_div2x = 1, sclk_rdiv, sysclk;
> +	u8 temp1, temp2;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3034, &temp1);
> +	temp2 = temp1 & 0x0f;
> +	if (temp2 == 8 || temp2 == 10)
> +		bit_div2x = temp2 / 2;
> +
> +	OV5640_READ_REG(sensor, 0x3035, &temp1);
> +	sysdiv = temp1>>4;
> +	if (sysdiv == 0)
> +		sysdiv = 16;
> +
> +	OV5640_READ_REG(sensor, 0x3036, &temp1);
> +	multiplier = temp1;
> +
> +	OV5640_READ_REG(sensor, 0x3037, &temp1);
> +	prediv = temp1 & 0x0f;
> +	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
> +
> +	OV5640_READ_REG(sensor, 0x3108, &temp1);
> +	temp2 = temp1 & 0x03;
> +	sclk_rdiv = sclk_rdiv_map[temp2];
> +
> +	VCO = xvclk * multiplier / prediv;
> +
> +	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
> +
> +	return sysclk;
> +}
> +
> +static int ov5640_set_night_mode(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u8 mode;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3a00, &mode);
> +	mode &= 0xfb;
> +	OV5640_WRITE_REG(sensor, 0x3a00, mode);
> +	return 0;
> +}
> +
> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u16 HTS;
> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380c, &HTS);
> +	return HTS;
> +}
> +
> +static int ov5640_get_VTS(struct ov5640_dev *sensor)
> +{
> +	u16 VTS;
> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380e, &VTS);
> +	return VTS;
> +}
> +
> +static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
> +	return 0;
> +}
> +
> +static int ov5640_get_light_freq(struct ov5640_dev *sensor)
> +{
> +	/* get banding filter value */
> +	u8 temp, temp1;
> +	int light_freq = 0;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3c01, &temp);
> +
> +	if (temp & 0x80) {
> +		/* manual */
> +		OV5640_READ_REG(sensor, 0x3c00, &temp1);
> +		if (temp1 & 0x04) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +			light_freq = 60;
> +		}
> +	} else {
> +		/* auto */
> +		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
> +		if (temp1 & 0x01) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +		}
> +	}
> +
> +	return light_freq;
> +}
> +
> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
> +{
> +	int prev_vts;
> +	int band_step60, max_band60, band_step50, max_band50;
> +	int ret;
> +
> +	/* read preview PCLK */
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_sysclk = ret;
> +	/* read preview HTS */
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_hts = ret;
> +
> +	/* read preview VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_vts = ret;
> +
> +	/* calculate banding filter */
> +	/* 60Hz */
> +	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
> +	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
> +
> +	max_band60 = (int)((prev_vts-4)/band_step60);
> +	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
> +
> +	/* 50Hz */
> +	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
> +	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
> +
> +	max_band50 = (int)((prev_vts-4)/band_step50);
> +	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
> +{
> +	/* stable in high */
> +	int fast_high, fast_low;
> +	int ret;
> +
> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
> +
> +	fast_high = sensor->ae_high<<1;
> +	if (fast_high > 255)
> +		fast_high = 255;
> +
> +	fast_low = sensor->ae_low >> 1;
> +
> +	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
> +
> +	return 0;
> +}
> +
> +static int ov5640_binning_on(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3821, &temp);
> +	temp &= 0xfe;
> +
> +	return temp ? 1 : 0;
> +}
> +
> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
> +{
> +	u8 temp, channel = sensor->ep.base.id;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x4814, &temp);
> +	temp &= ~(3 << 6);
> +	temp |= (channel << 6);
> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
> +
> +	return 0;
> +}
> +
> +static enum ov5640_mode
> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
> +			 int width, int height)
> +{
> +	int i;
> +
> +	for (i = ov5640_num_modes - 1; i >= 0; i--) {
> +		if (ov5640_mode_info_data[0][i].width <= width &&
> +		    ov5640_mode_info_data[0][i].height <= height)
> +			break;
> +	}
> +
> +	if (i < 0)
> +		i = 0;
> +
> +	return (enum ov5640_mode)i;
> +}
> +
> +/*
> + * sensor changes between scaling and subsampling, go through
> + * exposure calculation
> + */
> +static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
> +					    enum ov5640_frame_rate frame_rate,
> +					    enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	u8 average;
> +	int prev_shutter, prev_gain16;
> +	int cap_shutter, cap_gain16;
> +	int cap_sysclk, cap_hts, cap_vts;
> +	int light_freq, cap_bandfilt, cap_maxband;
> +	long cap_gain16_shutter;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* auto focus */
> +	/* ov5640_auto_focus();//if no af function, just skip it */
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read preview shutter */
> +	ret = ov5640_get_exposure(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_shutter = ret;
> +	ret = ov5640_binning_on(sensor);
> +	if (ret < 0)
> +		return ret;
> +	if (ret && mode != ov5640_mode_720P_1280_720 &&
> +	    mode != ov5640_mode_1080P_1920_1080)
> +		prev_shutter *= 2;
> +
> +	/* read preview gain */
> +	ret = ov5640_get_gain(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_gain16 = ret;
> +
> +	/* get average */
> +	OV5640_READ_REG(sensor, 0x56a1, &average);
> +
> +	/* turn off night mode for capture */
> +	ret = ov5640_set_night_mode(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* turn off overlay */
> +	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
> +	   just skip it */
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read capture VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_vts = ret;
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_hts = ret;
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_sysclk = ret;
> +
> +	/* calculate capture banding filter */
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	light_freq = ret;
> +
> +	if (light_freq == 60) {
> +		/* 60Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
> +	} else {
> +		/* 50Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts;
> +	}
> +	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
> +
> +	/* calculate capture shutter/gain16 */
> +	if (average > sensor->ae_low && average < sensor->ae_high) {
> +		/* in stable range */
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts *
> +			sensor->ae_target / average;
> +	} else {
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts;
> +	}
> +
> +	/* gain to shutter */
> +	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
> +		/* shutter < 1/100 */
> +		cap_shutter = cap_gain16_shutter / 16;
> +		if (cap_shutter < 1)
> +			cap_shutter = 1;
> +
> +		cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		if (cap_gain16 < 16)
> +			cap_gain16 = 16;
> +	} else {
> +		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
> +			/* exposure reach max */
> +			cap_shutter = cap_bandfilt * cap_maxband;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		} else {
> +			/* 1/100 < (cap_shutter = n/100) =< max */
> +			cap_shutter =
> +				((int)(cap_gain16_shutter / 16 / cap_bandfilt))
> +				* cap_bandfilt;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		}
> +	}
> +
> +	/* write capture gain */
> +	ret = ov5640_set_gain(sensor, cap_gain16);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* write capture shutter */
> +	if (cap_shutter > (cap_vts - 4)) {
> +		cap_vts = cap_shutter + 4;
> +		ret = ov5640_set_VTS(sensor, cap_vts);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	ret = ov5640_set_exposure(sensor, cap_shutter);
> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_stream(sensor, true);
> +}
> +
> +/*
> + * if sensor changes inside scaling or subsampling
> + * change mode directly
> + */
> +static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
> +				     enum ov5640_frame_rate frame_rate,
> +				     enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, true);
> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_agc(sensor, true);
> +}
> +
> +static int ov5640_change_mode(struct ov5640_dev *sensor,
> +			      enum ov5640_frame_rate frame_rate,
> +			      enum ov5640_mode mode,
> +			      enum ov5640_mode orig_mode)
> +{
> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;
> +
> +	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
> +	    mode != ov5640_mode_INIT) {
> +		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
> +		return -EINVAL;
> +	}
> +
> +	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
> +	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
> +	if (mode == ov5640_mode_INIT) {
> +		mode_data = ov5640_init_setting_30fps_VGA;
> +		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
> +
> +		sensor->fmt.width = 640;
> +		sensor->fmt.height = 480;
> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +		if (ret < 0)
> +			return ret;
> +
> +		mode_data = ov5640_setting_30fps_VGA_640_480;
> +		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
> +			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
> +		/* change between subsampling and scaling
> +		 * go through exposure calucation */
> +		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
> +							  mode);
> +	} else {
> +		/* change inside subsampling or scaling
> +		 * download firmware directly */
> +		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
> +	}
> +
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_bandingfilter(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_virtual_channel(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* restore controls */
> +	ov5640_restore_ctrls(sensor);
> +
> +	if (ret >= 0 && mode != ov5640_mode_INIT) {
> +		sensor->current_mode = mode;
> +		sensor->current_fr = frame_rate;
> +	}
> +
> +	return 0;
> +}
> +
> +/* restore the last set video mode after chip power-on */
> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
> +{
> +	int ret = 0;
> +
> +	/* first we need to set some initial register values */
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				    ov5640_mode_INIT, ov5640_mode_INIT);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* now restore the last capture mode */
> +	return ov5640_change_mode(sensor,
> +				  sensor->current_fr,
> +				  sensor->current_mode,
> +				  ov5640_mode_VGA_640_480);
> +}
> +
> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
> +{
> +	int ret;
> +
> +	if (sensor->io_regulator) {
> +		ret = regulator_enable(sensor->io_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->core_regulator) {
> +		ret = regulator_enable(sensor->core_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->gpo_regulator) {
> +		ret = regulator_enable(sensor->gpo_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->analog_regulator) {
> +		ret = regulator_enable(sensor->analog_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static void ov5640_regulators_off(struct ov5640_dev *sensor)
> +{
> +	if (sensor->analog_regulator)
> +		regulator_disable(sensor->analog_regulator);
> +	if (sensor->core_regulator)
> +		regulator_disable(sensor->core_regulator);
> +	if (sensor->io_regulator)
> +		regulator_disable(sensor->io_regulator);
> +	if (sensor->gpo_regulator)
> +		regulator_disable(sensor->gpo_regulator);
> +}
> +
> +/* --------------- Subdev Operations --------------- */
> +
> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	int ret;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (on && !sensor->on) {
> +		if (sensor->xclk)
> +			clk_prepare_enable(sensor->xclk);
> +
> +		ret = ov5640_regulators_on(sensor);
> +		if (ret)
> +			return ret;
> +
> +		ov5640_reset(sensor);
> +		ov5640_power(sensor, true);
> +
> +		ret = ov5640_init_slave_id(sensor);
> +		if (ret)
> +			return ret;
> +
> +		ret = ov5640_restore_mode(sensor);
> +		if (ret)
> +			return ret;
> +
> +		/*
> +		 * NOTE: Freescale adds a long delay (600 msec) after
> +		 * powering up and programming a mode on the ov5640-mipi
> +		 * camera (search for "msec_wait4stable" in FSL's
> +		 * ov5640_mipi.c), which equivalently would need to go
> +		 * right here. If we run into MIPI CSI-2 receiver dphy
> +		 * ready timeouts, it might be a clue to add that delay
> +		 * here.
> +		 */
> +	} else if (!on && sensor->on) {
> +		ov5640_power(sensor, false);
> +
> +		ov5640_regulators_off(sensor);
> +
> +		if (sensor->xclk)
> +			clk_disable_unprepare(sensor->xclk);
> +	}
> +
> +	sensor->on = on;
> +
> +	return 0;
> +}
> +
> +static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_captureparm *cparm = &a->parm.capture;
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* This is the only case currently handled. */
> +	memset(a, 0, sizeof(*a));
> +	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	cparm->capability = sensor->streamcap.capability;
> +	cparm->timeperframe = sensor->streamcap.timeperframe;
> +	cparm->capturemode = sensor->streamcap.capturemode;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
> +	enum ov5640_frame_rate frame_rate;
> +	u32 tgt_fps;	/* target frames per secound */
> +	int ret = 0;
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* Check that the new frame rate is allowed. */
> +	if ((timeperframe->numerator == 0) ||
> +	    (timeperframe->denominator == 0)) {
> +		timeperframe->denominator = DEFAULT_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps > MAX_FPS) {
> +		timeperframe->denominator = MAX_FPS;
> +		timeperframe->numerator = 1;
> +	} else if (tgt_fps < MIN_FPS) {
> +		timeperframe->denominator = MIN_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	/* Actual frame rate we use */
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps == 15)
> +		frame_rate = ov5640_15_fps;
> +	else if (tgt_fps == 30)
> +		frame_rate = ov5640_30_fps;
> +	else {
> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
> +			 tgt_fps);
> +		return -EINVAL;
> +	}
> +
> +	ret = ov5640_change_mode(sensor, frame_rate,
> +				 sensor->current_mode,
> +				 sensor->current_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	sensor->streamcap.timeperframe = *timeperframe;
> +
> +	return 0;
> +}
> +
> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	format->format = sensor->fmt;
> +
> +	return 0;
> +}
> +
> +static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
> +				   struct v4l2_mbus_framefmt *fmt,
> +				   enum ov5640_mode *new_mode)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
> +
> +	fmt->width = ov5640_mode_info_data[0][mode].width;
> +	fmt->height = ov5640_mode_info_data[0][mode].height;
> +	fmt->code = sensor->fmt.code;
> +
> +	if (new_mode)
> +		*new_mode = mode;
> +	return 0;
> +}
> +
> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode new_mode;
> +	int ret;
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
> +		if (ret)
> +			return ret;
> +		cfg->try_fmt = format->format;
> +		return 0;
> +	}
> +
> +	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
> +	if (ret)
> +		return ret;
> +
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				 new_mode, sensor->current_mode);
> +	if (ret >= 0)
> +		sensor->fmt = format->format;
> +
> +	return ret;
> +}
> +
> +
> +/*
> + * Sensor Controls.
> + */
> +
> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
> +		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
> +		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	sensor->awb_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
> +	return 0;
> +}
> +
> +static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +#if 0
> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +#endif
> +
> +static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
> +{
> +	u16 max_exp = 0;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
> +	if (value < max_exp) {
> +		u32 exp = value << 4;
> +
> +		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
> +	}
> +
> +	return 0;
> +}
> +
> +/* read exposure, in number of line periods */
> +static int ov5640_get_exposure(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int exp, ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG(sensor, 0x3500, &temp);
> +	exp = ((int)temp & 0x0f) << 16;
> +	OV5640_READ_REG(sensor, 0x3501, &temp);
> +	exp |= ((int)temp << 8);
> +	OV5640_READ_REG(sensor, 0x3502, &temp);
> +	exp |= (int)temp;
> +
> +	return exp >> 4;
> +}
> +
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	/* this enables/disables both AEC and AGC */
> +	sensor->agc_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
> +	return 0;
> +}
> +
> +static int ov5640_get_gain(struct ov5640_dev *sensor)
> +{
> +	u16 gain;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
> +
> +	return gain & 0x3ff;
> +}
> +
> +#if 0
> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
> +	return 0;
> +}
> +#endif
> +
> +static struct ov5640_control ov5640_ctrls[] = {
> +	{
> +		.set = ov5640_set_agc,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTOGAIN,
> +			.name = "Auto Gain/Exposure Control",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_exposure,
> +		.ctrl = {
> +			.id = V4L2_CID_EXPOSURE,
> +			.name = "Exposure",
> +			.minimum = 0,
> +			.maximum = 65535,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_gain,
> +		.ctrl = {
> +			.id = V4L2_CID_GAIN,
> +			.name = "Gain",
> +			.minimum = 0,
> +			.maximum = 1023,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_hue,
> +		.ctrl = {
> +			.id = V4L2_CID_HUE,
> +			.name = "Hue",
> +			.minimum = 0,
> +			.maximum = 359,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_contrast,
> +		.ctrl = {
> +			.id = V4L2_CID_CONTRAST,
> +			.name = "Contrast",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_saturation,
> +		.ctrl = {
> +			.id = V4L2_CID_SATURATION,
> +			.name = "Saturation",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 64,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_awb,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTO_WHITE_BALANCE,
> +			.name = "Auto White Balance",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_red_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_RED_BALANCE,
> +			.name = "Red Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_blue_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_BLUE_BALANCE,
> +			.name = "Blue Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	},
> +};
> +#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)

This should use v4l2_ctrl_new_std() instead of this array.
Just put a switch on ctrl->id in s_ctrl, and each case calls the corresponding
set function.

> +
> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
> +{
> +	struct ov5640_control *ret = NULL;
> +	int i;
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		if (id == ov5640_ctrls[i].ctrl.id) {
> +			ret = &ov5640_ctrls[i];
> +			break;
> +		}
> +	}
> +
> +	if (ret && index)
> +		*index = i;
> +	return ret;
> +}
> +
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;
> +	int i;
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +		c->set(sensor, sensor->ctrl_cache[i]);
> +	}
> +
> +	return 0;
> +}

This does the same as v4l2_ctrl_handler_setup() if I understand the code correctly.

> +
> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
> +	struct ov5640_control *c;
> +	int ret = 0;
> +	int i;
> +
> +	c = ov5640_get_ctrl(ctrl->id, &i);
> +	if (!c)
> +		return -EINVAL;
> +
> +	ret = c->set(sensor, ctrl->val);
> +	/* update cached value if no error */
> +	if (!ret)
> +		sensor->ctrl_cache[i] = ctrl->val;
> +
> +	return ret;
> +}
> +
> +static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
> +	.s_ctrl = ov5640_s_ctrl,
> +};
> +
> +static int ov5640_init_controls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;
> +	int i;
> +
> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +
> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
> +				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
> +				  c->ctrl.step, c->ctrl.default_value);
> +	}

As mentioned, just drop the ov5640_ctrls array and call v4l2_ctr_new_std for each
control you're adding.

> +
> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
> +	if (sensor->ctrl_hdl.error) {
> +		int err = sensor->ctrl_hdl.error;
> +
> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
> +		return err;
> +	}
> +	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);
> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_frame_size_enum *fse)
> +{
> +	if (fse->pad != 0)
> +		return -EINVAL;
> +	if (fse->index >= ov5640_num_modes)
> +		return -EINVAL;
> +
> +	fse->min_width = fse->max_width =
> +		ov5640_mode_info_data[0][fse->index].width;
> +	fse->min_height = fse->max_height =
> +		ov5640_mode_info_data[0][fse->index].height;
> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_interval(
> +	struct v4l2_subdev *sd,
> +	struct v4l2_subdev_pad_config *cfg,
> +	struct v4l2_subdev_frame_interval_enum *fie)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	if (fie->pad != 0)
> +		return -EINVAL;
> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
> +		return -EINVAL;
> +
> +	if (fie->width == 0 || fie->height == 0)
> +		return -EINVAL;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
> +
> +	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
> +		return -EINVAL;
> +
> +	fie->interval.numerator = 1;
> +	fie->interval.denominator = ov5640_framerates[fie->index];
> +
> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
> +		fie->width, fie->height, fie->index, fie->interval.denominator);
> +	return 0;
> +}
> +
> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
> +			    u32 output, u32 config)
> +{
> +	return (input != 0) ? -EINVAL : 0;
> +}
> +
> +static int ov5640_enum_mbus_code(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	if (code->pad != 0)
> +		return -EINVAL;
> +	if (code->index != 0)
> +		return -EINVAL;
> +
> +	code->code = sensor->fmt.code;
> +
> +	return 0;
> +}
> +
> +static int ov5640_g_mbus_config(struct v4l2_subdev *sd,
> +				struct v4l2_mbus_config *cfg)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	cfg->type = V4L2_MBUS_CSI2;
> +	cfg->flags = sensor->ep.bus.mipi_csi2.flags;
> +	cfg->flags |= (1 << (sensor->ep.bus.mipi_csi2.num_data_lanes - 1));
> +	cfg->flags |= V4L2_MBUS_CSI2_CHANNEL_0;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	return ov5640_set_stream(sensor, enable);
> +}
> +
> +static struct v4l2_subdev_core_ops ov5640_core_ops = {
> +	.s_power = ov5640_s_power,
> +};
> +
> +static struct v4l2_subdev_video_ops ov5640_video_ops = {
> +	.s_parm = ov5640_s_parm,
> +	.g_parm = ov5640_g_parm,
> +	.g_input_status = ov5640_g_input_status,
> +	.s_routing = ov5640_s_routing,
> +	.g_mbus_config  = ov5640_g_mbus_config,
> +	.s_stream = ov5640_s_stream,
> +};
> +
> +static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
> +	.enum_mbus_code = ov5640_enum_mbus_code,
> +	.get_fmt = ov5640_get_fmt,
> +	.set_fmt = ov5640_set_fmt,
> +	.enum_frame_size = ov5640_enum_frame_size,
> +	.enum_frame_interval = ov5640_enum_frame_interval,
> +};
> +
> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
> +	.core = &ov5640_core_ops,
> +	.video = &ov5640_video_ops,
> +	.pad = &ov5640_pad_ops,
> +};
> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
> +{
> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
> +}
> +
> +static void ov5640_reset(struct ov5640_dev *sensor)
> +{
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +
> +	/* camera power cycle */
> +	ov5640_power(sensor, false);
> +	usleep_range(5000, 10000);
> +	ov5640_power(sensor, true);
> +	usleep_range(5000, 10000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 1);
> +	usleep_range(1000, 2000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +	usleep_range(5000, 10000);
> +}
> +
> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
> +{
> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
> +	if (!IS_ERR(sensor->io_regulator)) {
> +		regulator_set_voltage(sensor->io_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_IO,
> +				      OV5640_VOLTAGE_DIGITAL_IO);
> +	} else {
> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
> +			__func__);
> +		sensor->io_regulator = NULL;
> +	}
> +
> +	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
> +	if (!IS_ERR(sensor->core_regulator)) {
> +		regulator_set_voltage(sensor->core_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_CORE,
> +				      OV5640_VOLTAGE_DIGITAL_CORE);
> +	} else {
> +		sensor->core_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
> +			__func__);
> +	}
> +
> +	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
> +	if (!IS_ERR(sensor->analog_regulator)) {
> +		regulator_set_voltage(sensor->analog_regulator,
> +				      OV5640_VOLTAGE_ANALOG,
> +				      OV5640_VOLTAGE_ANALOG);
> +	} else {
> +		sensor->analog_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
> +			__func__);
> +	}
> +}
> +
> +static int ov5640_probe(struct i2c_client *client,
> +			const struct i2c_device_id *id)
> +{
> +	struct device *dev = &client->dev;
> +	struct device_node *endpoint;
> +	struct ov5640_dev *sensor;
> +	int i, xclk, ret;
> +
> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
> +	if (!sensor)
> +		return -ENOMEM;
> +
> +	sensor->i2c_client = client;
> +	sensor->dev = dev;
> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
> +	sensor->fmt.width = 640;
> +	sensor->fmt.height = 480;
> +	sensor->fmt.field = V4L2_FIELD_NONE;
> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
> +					   V4L2_CAP_TIMEPERFRAME;
> +	sensor->streamcap.capturemode = 0;
> +	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
> +	sensor->streamcap.timeperframe.numerator = 1;
> +
> +	sensor->current_mode = ov5640_mode_VGA_640_480;
> +	sensor->current_fr = ov5640_30_fps;
> +
> +	sensor->ae_target = 52;
> +
> +	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
> +	if (!endpoint) {
> +		dev_err(dev, "endpoint node not found\n");
> +		return -EINVAL;
> +	}
> +
> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
> +		return -EINVAL;
> +	}
> +	of_node_put(endpoint);
> +
> +	/* get system clock (xclk) frequency */
> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
> +	if (!ret) {
> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
> +			dev_err(dev, "invalid xclk frequency\n");
> +			return -EINVAL;
> +		}
> +		sensor->xclk_freq = xclk;
> +	}
> +
> +	/* get system clock (xclk) */
> +	sensor->xclk = devm_clk_get(dev, "xclk");
> +	if (!IS_ERR(sensor->xclk)) {
> +		if (!sensor->xclk_freq) {
> +			dev_err(dev, "xclk requires xclk frequency!\n");
> +			return -EINVAL;
> +		}
> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
> +	} else {
> +		/* assume system clock enabled by default */
> +		sensor->xclk = NULL;
> +	}
> +
> +	/* request power down pin */
> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
> +	if (IS_ERR(sensor->pwdn_gpio)) {
> +		dev_err(dev, "request for power down gpio failed\n");
> +		return PTR_ERR(sensor->pwdn_gpio);
> +	}
> +
> +	/* request reset pin */
> +	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
> +	if (IS_ERR(sensor->reset_gpio)) {
> +		dev_err(dev, "request for reset gpio failed\n");
> +		return PTR_ERR(sensor->reset_gpio);
> +	}
> +
> +	/* initialize the cached controls to their defaults */
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		struct ov5640_control *c = &ov5640_ctrls[i];
> +
> +		sensor->ctrl_cache[i] = c->ctrl.default_value;
> +	}
> +	sensor->awb_on = sensor->agc_on = true;
> +
> +	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
> +
> +	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
> +	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> +	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
> +	if (ret)
> +		return ret;
> +
> +	ov5640_get_regulators(sensor);
> +
> +	ret = ov5640_s_power(&sensor->sd, 1);
> +	if (ret)
> +		goto entity_cleanup;
> +	ret = ov5640_init_controls(sensor);
> +	if (ret)
> +		goto power_off;
> +
> +	ret = ov5640_s_power(&sensor->sd, 0);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	ret = v4l2_async_register_subdev(&sensor->sd);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +power_off:
> +	ov5640_s_power(&sensor->sd, 0);
> +entity_cleanup:
> +	media_entity_cleanup(&sensor->sd.entity);
> +	ov5640_regulators_off(sensor);
> +	return ret;
> +}
> +
> +/*!
> + * ov5640 I2C detach function
> + *
> + * @param client            struct i2c_client *
> + * @return  Error code indicating success or failure
> + */
> +static int ov5640_remove(struct i2c_client *client)
> +{
> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	ov5640_regulators_off(sensor);
> +
> +	v4l2_async_unregister_subdev(&sensor->sd);
> +	media_entity_cleanup(&sensor->sd.entity);
> +	v4l2_device_unregister_subdev(sd);
> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +	return 0;
> +}
> +
> +static const struct i2c_device_id ov5640_id[] = {
> +	{"ov5640", 0},
> +	{},
> +};
> +MODULE_DEVICE_TABLE(i2c, ov5640_id);
> +
> +static const struct of_device_id ov5640_dt_ids[] = {
> +	{ .compatible = "ovti,ov5640" },
> +	{ /* sentinel */ }
> +};
> +MODULE_DEVICE_TABLE(of, ov5640_dt_ids);
> +
> +static struct i2c_driver ov5640_i2c_driver = {
> +	.driver = {
> +		.name  = "ov5640",
> +		.of_match_table	= ov5640_dt_ids,
> +	},
> +	.id_table = ov5640_id,
> +	.probe    = ov5640_probe,
> +	.remove   = ov5640_remove,
> +};
> +
> +module_i2c_driver(ov5640_i2c_driver);
> +
> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
> +MODULE_LICENSE("GPL");
> +MODULE_VERSION("1.0");
> 

Same comments apply to the next patch, so I won't repeat them.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-20 13:52   ` Hans Verkuil
  (?)
@ 2017-01-20 16:31     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-20 16:31 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Hans,

On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
> Hi Steve, Philipp,
> 
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> > In version 3:
> > 
> > Changes suggested by Rob Herring <robh@kernel.org>:
> > 
> >   - prepended FIM node properties with vendor prefix "fsl,".
> > 
> >   - make mipi csi-2 receiver compatible string SoC specific:
> >     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
> > 
> >   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
> > 
> >   - removed board-specific info from the media driver binding doc. These
> >     were all related to sensor bindings, which already are (adv7180)
> >     or will be (ov564x) covered in separate binding docs. All reference
> >     board info not related to DT bindings has been moved to
> >     Documentation/media/v4l-drivers/imx.rst.
> > 
> >   - removed "_mipi" from the OV5640 compatible string.
> > 
> > Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
> > 
> >   Mostly cosmetic/non-functional changes which I won't list here, except
> >   for the following:
> > 
> >   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
> > 
> >   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
> > 
> >   - check/handle return code from required reg property of CSI port nodes.
> > 
> >   - check/handle return code from clk_prepare_enable().
> > 
> > Changes suggested by Fabio Estevam <festevam@gmail.com>:
> > 
> >   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
> > 
> >   - finally got around to passing valid IOMUX pin config values to the
> >     pin groups.
> > 
> > Other changes:
> > 
> >   - removed the FIM properties that overrided the v4l2 FIM control defaults
> >     values. This was left-over from a requirement of a customer and is not
> >     necessary here.
> > 
> >   - The FIM must be explicitly enabled in the fim child node under the CSI
> >     port nodes, using the status property. If not enabled, FIM v4l2 controls
> >     will not appear in the video capture driver.
> > 
> >   - brought in additional media types patch from Philipp Zabel. Use new
> >     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
> > 
> >   - brought in latest platform generic video multiplexer subdevice driver
> >     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
> > 
> >   - removed imx-media-of.h, moved those prototypes into imx-media.h.
> 
> Based on the discussion on the mailinglist it seems everyone agrees that this
> is the preferred driver, correct?

No. I have some major reservations against the custom mem2mem framework
embedded in Steve's driver.
I think it is a misuse of the media entity links (which should describe
hardware connections) for something that should be done at the vb2 level
(letting one device's capture EOF interrupt trigger the next device's
m2m device_run without going through userspace).
Steve and I disagree on that point, so we'd appreciate if we could get
some more eyes on the above issue.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 16:31     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-20 16:31 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, Steve Longerbeam,
	linux-media, devicetree, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

Hi Hans,

On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
> Hi Steve, Philipp,
> 
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> > In version 3:
> > 
> > Changes suggested by Rob Herring <robh@kernel.org>:
> > 
> >   - prepended FIM node properties with vendor prefix "fsl,".
> > 
> >   - make mipi csi-2 receiver compatible string SoC specific:
> >     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
> > 
> >   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
> > 
> >   - removed board-specific info from the media driver binding doc. These
> >     were all related to sensor bindings, which already are (adv7180)
> >     or will be (ov564x) covered in separate binding docs. All reference
> >     board info not related to DT bindings has been moved to
> >     Documentation/media/v4l-drivers/imx.rst.
> > 
> >   - removed "_mipi" from the OV5640 compatible string.
> > 
> > Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
> > 
> >   Mostly cosmetic/non-functional changes which I won't list here, except
> >   for the following:
> > 
> >   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
> > 
> >   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
> > 
> >   - check/handle return code from required reg property of CSI port nodes.
> > 
> >   - check/handle return code from clk_prepare_enable().
> > 
> > Changes suggested by Fabio Estevam <festevam@gmail.com>:
> > 
> >   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
> > 
> >   - finally got around to passing valid IOMUX pin config values to the
> >     pin groups.
> > 
> > Other changes:
> > 
> >   - removed the FIM properties that overrided the v4l2 FIM control defaults
> >     values. This was left-over from a requirement of a customer and is not
> >     necessary here.
> > 
> >   - The FIM must be explicitly enabled in the fim child node under the CSI
> >     port nodes, using the status property. If not enabled, FIM v4l2 controls
> >     will not appear in the video capture driver.
> > 
> >   - brought in additional media types patch from Philipp Zabel. Use new
> >     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
> > 
> >   - brought in latest platform generic video multiplexer subdevice driver
> >     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
> > 
> >   - removed imx-media-of.h, moved those prototypes into imx-media.h.
> 
> Based on the discussion on the mailinglist it seems everyone agrees that this
> is the preferred driver, correct?

No. I have some major reservations against the custom mem2mem framework
embedded in Steve's driver.
I think it is a misuse of the media entity links (which should describe
hardware connections) for something that should be done at the vb2 level
(letting one device's capture EOF interrupt trigger the next device's
m2m device_run without going through userspace).
Steve and I disagree on that point, so we'd appreciate if we could get
some more eyes on the above issue.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 16:31     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-20 16:31 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Hans,

On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
> Hi Steve, Philipp,
> 
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> > In version 3:
> > 
> > Changes suggested by Rob Herring <robh@kernel.org>:
> > 
> >   - prepended FIM node properties with vendor prefix "fsl,".
> > 
> >   - make mipi csi-2 receiver compatible string SoC specific:
> >     "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
> > 
> >   - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
> > 
> >   - removed board-specific info from the media driver binding doc. These
> >     were all related to sensor bindings, which already are (adv7180)
> >     or will be (ov564x) covered in separate binding docs. All reference
> >     board info not related to DT bindings has been moved to
> >     Documentation/media/v4l-drivers/imx.rst.
> > 
> >   - removed "_mipi" from the OV5640 compatible string.
> > 
> > Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
> > 
> >   Mostly cosmetic/non-functional changes which I won't list here, except
> >   for the following:
> > 
> >   - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
> > 
> >   - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
> > 
> >   - check/handle return code from required reg property of CSI port nodes.
> > 
> >   - check/handle return code from clk_prepare_enable().
> > 
> > Changes suggested by Fabio Estevam <festevam@gmail.com>:
> > 
> >   - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
> > 
> >   - finally got around to passing valid IOMUX pin config values to the
> >     pin groups.
> > 
> > Other changes:
> > 
> >   - removed the FIM properties that overrided the v4l2 FIM control defaults
> >     values. This was left-over from a requirement of a customer and is not
> >     necessary here.
> > 
> >   - The FIM must be explicitly enabled in the fim child node under the CSI
> >     port nodes, using the status property. If not enabled, FIM v4l2 controls
> >     will not appear in the video capture driver.
> > 
> >   - brought in additional media types patch from Philipp Zabel. Use new
> >     MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
> > 
> >   - brought in latest platform generic video multiplexer subdevice driver
> >     from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
> > 
> >   - removed imx-media-of.h, moved those prototypes into imx-media.h.
> 
> Based on the discussion on the mailinglist it seems everyone agrees that this
> is the preferred driver, correct?

No. I have some major reservations against the custom mem2mem framework
embedded in Steve's driver.
I think it is a misuse of the media entity links (which should describe
hardware connections) for something that should be done at the vb2 level
(letting one device's capture EOF interrupt trigger the next device's
m2m device_run without going through userspace).
Steve and I disagree on that point, so we'd appreciate if we could get
some more eyes on the above issue.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-20 16:31     ` Philipp Zabel
@ 2017-01-20 18:40       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-20 18:40 UTC (permalink / raw)
  To: Philipp Zabel, Hans Verkuil
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, nick, markus.heiser, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

Hi Hans, Philipp,


On 01/20/2017 08:31 AM, Philipp Zabel wrote:
> Hi Hans,
>
> On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
>> Hi Steve, Philipp,
>>
>> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>>> In version 3:
>>>
>>> Changes suggested by Rob Herring <robh@kernel.org>:
>>>
>>>    - prepended FIM node properties with vendor prefix "fsl,".
>>>
>>>    - make mipi csi-2 receiver compatible string SoC specific:
>>>      "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>>>
>>>    - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>>>
>>>    - removed board-specific info from the media driver binding doc. These
>>>      were all related to sensor bindings, which already are (adv7180)
>>>      or will be (ov564x) covered in separate binding docs. All reference
>>>      board info not related to DT bindings has been moved to
>>>      Documentation/media/v4l-drivers/imx.rst.
>>>
>>>    - removed "_mipi" from the OV5640 compatible string.
>>>
>>> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>>>
>>>    Mostly cosmetic/non-functional changes which I won't list here, except
>>>    for the following:
>>>
>>>    - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>>>
>>>    - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>>>
>>>    - check/handle return code from required reg property of CSI port nodes.
>>>
>>>    - check/handle return code from clk_prepare_enable().
>>>
>>> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>>>
>>>    - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>>>
>>>    - finally got around to passing valid IOMUX pin config values to the
>>>      pin groups.
>>>
>>> Other changes:
>>>
>>>    - removed the FIM properties that overrided the v4l2 FIM control defaults
>>>      values. This was left-over from a requirement of a customer and is not
>>>      necessary here.
>>>
>>>    - The FIM must be explicitly enabled in the fim child node under the CSI
>>>      port nodes, using the status property. If not enabled, FIM v4l2 controls
>>>      will not appear in the video capture driver.
>>>
>>>    - brought in additional media types patch from Philipp Zabel. Use new
>>>      MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>>>
>>>    - brought in latest platform generic video multiplexer subdevice driver
>>>      from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>>>
>>>    - removed imx-media-of.h, moved those prototypes into imx-media.h.
>> Based on the discussion on the mailinglist it seems everyone agrees that this
>> is the preferred driver, correct?
> No. I have some major reservations against the custom mem2mem framework
> embedded in Steve's driver.
> I think it is a misuse of the media entity links (which should describe
> hardware connections) for something that should be done at the vb2 level
> (letting one device's capture EOF interrupt trigger the next device's
> m2m device_run without going through userspace).
> Steve and I disagree on that point, so we'd appreciate if we could get
> some more eyes on the above issue.

This needs some background first, so let me first describe one example
pipeline in this driver.

There is a VDIC entity in the i.MX IPU that performs de-interlacing with
hardware filters for motion compensation. Some of the motion compensation
modes ("low" and "medium" motion) require that the VDIC receive video
frame fields from memory buffers (dedicated dma channels in the
IPU are used to transfer those buffers into the VDIC).

So one option to support those modes would be to pass the raw buffers
from a camera sensor up to userspace to a capture device, and then pass
them back to the VDIC for de-interlacing using a mem2mem device.

Philipp and I are both in agreement that, since userland is not interested
in the intermediate interlaced buffers in this case, but only the final
result (motion compensated, de-interlaced frames), it is more efficient
to provide a media link that allows passing those intermediate frames
directly from a camera source pad to VDIC sink pad, without having
to route them through userspace.

So in order to support that, I've implemented a simple FIFO dma buffer
queue in the driver to allow passing video buffers directly from a source
to a sink. It is modeled loosely off the vb2 state machine and API, but
simpler (for instance it only allows contiguous, cache-coherent buffers).

This is where Philipp has an argument, that this should be done with a
new API in videobuf2.

And I'm actually in total agreement with that. I definitely agree that there
should be a mechanism in the media framework that allows passing video
buffers from a source pad to a sink pad using a software queue, with no
involvement from userland.

My only disagreement is when this should be implemented. I think it is
fine to keep my custom implementation of this in the driver for now. Once
an extension of vb2 is ready to support this feature, it would be fairly
straightforward to strip out my custom implementation and go with the
new API.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 18:40       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-20 18:40 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Hans, Philipp,


On 01/20/2017 08:31 AM, Philipp Zabel wrote:
> Hi Hans,
>
> On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
>> Hi Steve, Philipp,
>>
>> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>>> In version 3:
>>>
>>> Changes suggested by Rob Herring <robh@kernel.org>:
>>>
>>>    - prepended FIM node properties with vendor prefix "fsl,".
>>>
>>>    - make mipi csi-2 receiver compatible string SoC specific:
>>>      "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>>>
>>>    - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>>>
>>>    - removed board-specific info from the media driver binding doc. These
>>>      were all related to sensor bindings, which already are (adv7180)
>>>      or will be (ov564x) covered in separate binding docs. All reference
>>>      board info not related to DT bindings has been moved to
>>>      Documentation/media/v4l-drivers/imx.rst.
>>>
>>>    - removed "_mipi" from the OV5640 compatible string.
>>>
>>> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>>>
>>>    Mostly cosmetic/non-functional changes which I won't list here, except
>>>    for the following:
>>>
>>>    - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>>>
>>>    - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>>>
>>>    - check/handle return code from required reg property of CSI port nodes.
>>>
>>>    - check/handle return code from clk_prepare_enable().
>>>
>>> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>>>
>>>    - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>>>
>>>    - finally got around to passing valid IOMUX pin config values to the
>>>      pin groups.
>>>
>>> Other changes:
>>>
>>>    - removed the FIM properties that overrided the v4l2 FIM control defaults
>>>      values. This was left-over from a requirement of a customer and is not
>>>      necessary here.
>>>
>>>    - The FIM must be explicitly enabled in the fim child node under the CSI
>>>      port nodes, using the status property. If not enabled, FIM v4l2 controls
>>>      will not appear in the video capture driver.
>>>
>>>    - brought in additional media types patch from Philipp Zabel. Use new
>>>      MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>>>
>>>    - brought in latest platform generic video multiplexer subdevice driver
>>>      from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>>>
>>>    - removed imx-media-of.h, moved those prototypes into imx-media.h.
>> Based on the discussion on the mailinglist it seems everyone agrees that this
>> is the preferred driver, correct?
> No. I have some major reservations against the custom mem2mem framework
> embedded in Steve's driver.
> I think it is a misuse of the media entity links (which should describe
> hardware connections) for something that should be done at the vb2 level
> (letting one device's capture EOF interrupt trigger the next device's
> m2m device_run without going through userspace).
> Steve and I disagree on that point, so we'd appreciate if we could get
> some more eyes on the above issue.

This needs some background first, so let me first describe one example
pipeline in this driver.

There is a VDIC entity in the i.MX IPU that performs de-interlacing with
hardware filters for motion compensation. Some of the motion compensation
modes ("low" and "medium" motion) require that the VDIC receive video
frame fields from memory buffers (dedicated dma channels in the
IPU are used to transfer those buffers into the VDIC).

So one option to support those modes would be to pass the raw buffers
from a camera sensor up to userspace to a capture device, and then pass
them back to the VDIC for de-interlacing using a mem2mem device.

Philipp and I are both in agreement that, since userland is not interested
in the intermediate interlaced buffers in this case, but only the final
result (motion compensated, de-interlaced frames), it is more efficient
to provide a media link that allows passing those intermediate frames
directly from a camera source pad to VDIC sink pad, without having
to route them through userspace.

So in order to support that, I've implemented a simple FIFO dma buffer
queue in the driver to allow passing video buffers directly from a source
to a sink. It is modeled loosely off the vb2 state machine and API, but
simpler (for instance it only allows contiguous, cache-coherent buffers).

This is where Philipp has an argument, that this should be done with a
new API in videobuf2.

And I'm actually in total agreement with that. I definitely agree that there
should be a mechanism in the media framework that allows passing video
buffers from a source pad to a sink pad using a software queue, with no
involvement from userland.

My only disagreement is when this should be implemented. I think it is
fine to keep my custom implementation of this in the driver for now. Once
an extension of vb2 is ready to support this feature, it would be fairly
straightforward to strip out my custom implementation and go with the
new API.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-20 18:40       ` Steve Longerbeam
  (?)
@ 2017-01-20 20:39         ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 20:39 UTC (permalink / raw)
  To: Steve Longerbeam, Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, nick, markus.heiser, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam

On 01/20/2017 07:40 PM, Steve Longerbeam wrote:
> Hi Hans, Philipp,
> 
> 
> On 01/20/2017 08:31 AM, Philipp Zabel wrote:
>> Hi Hans,
>>
>> On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
>>> Hi Steve, Philipp,
>>>
>>> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>>>> In version 3:
>>>>
>>>> Changes suggested by Rob Herring <robh@kernel.org>:
>>>>
>>>>    - prepended FIM node properties with vendor prefix "fsl,".
>>>>
>>>>    - make mipi csi-2 receiver compatible string SoC specific:
>>>>      "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>>>>
>>>>    - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>>>>
>>>>    - removed board-specific info from the media driver binding doc. These
>>>>      were all related to sensor bindings, which already are (adv7180)
>>>>      or will be (ov564x) covered in separate binding docs. All reference
>>>>      board info not related to DT bindings has been moved to
>>>>      Documentation/media/v4l-drivers/imx.rst.
>>>>
>>>>    - removed "_mipi" from the OV5640 compatible string.
>>>>
>>>> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>>>>
>>>>    Mostly cosmetic/non-functional changes which I won't list here, except
>>>>    for the following:
>>>>
>>>>    - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>>>>
>>>>    - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>>>>
>>>>    - check/handle return code from required reg property of CSI port nodes.
>>>>
>>>>    - check/handle return code from clk_prepare_enable().
>>>>
>>>> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>>>>
>>>>    - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>>>>
>>>>    - finally got around to passing valid IOMUX pin config values to the
>>>>      pin groups.
>>>>
>>>> Other changes:
>>>>
>>>>    - removed the FIM properties that overrided the v4l2 FIM control defaults
>>>>      values. This was left-over from a requirement of a customer and is not
>>>>      necessary here.
>>>>
>>>>    - The FIM must be explicitly enabled in the fim child node under the CSI
>>>>      port nodes, using the status property. If not enabled, FIM v4l2 controls
>>>>      will not appear in the video capture driver.
>>>>
>>>>    - brought in additional media types patch from Philipp Zabel. Use new
>>>>      MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>>>>
>>>>    - brought in latest platform generic video multiplexer subdevice driver
>>>>      from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>>>>
>>>>    - removed imx-media-of.h, moved those prototypes into imx-media.h.
>>> Based on the discussion on the mailinglist it seems everyone agrees that this
>>> is the preferred driver, correct?
>> No. I have some major reservations against the custom mem2mem framework
>> embedded in Steve's driver.
>> I think it is a misuse of the media entity links (which should describe
>> hardware connections) for something that should be done at the vb2 level
>> (letting one device's capture EOF interrupt trigger the next device's
>> m2m device_run without going through userspace).
>> Steve and I disagree on that point, so we'd appreciate if we could get
>> some more eyes on the above issue.
> 
> This needs some background first, so let me first describe one example
> pipeline in this driver.
> 
> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
> hardware filters for motion compensation. Some of the motion compensation
> modes ("low" and "medium" motion) require that the VDIC receive video
> frame fields from memory buffers (dedicated dma channels in the
> IPU are used to transfer those buffers into the VDIC).
> 
> So one option to support those modes would be to pass the raw buffers
> from a camera sensor up to userspace to a capture device, and then pass
> them back to the VDIC for de-interlacing using a mem2mem device.
> 
> Philipp and I are both in agreement that, since userland is not interested
> in the intermediate interlaced buffers in this case, but only the final
> result (motion compensated, de-interlaced frames), it is more efficient
> to provide a media link that allows passing those intermediate frames
> directly from a camera source pad to VDIC sink pad, without having
> to route them through userspace.
> 
> So in order to support that, I've implemented a simple FIFO dma buffer
> queue in the driver to allow passing video buffers directly from a source
> to a sink. It is modeled loosely off the vb2 state machine and API, but
> simpler (for instance it only allows contiguous, cache-coherent buffers).
> 
> This is where Philipp has an argument, that this should be done with a
> new API in videobuf2.
> 
> And I'm actually in total agreement with that. I definitely agree that there
> should be a mechanism in the media framework that allows passing video
> buffers from a source pad to a sink pad using a software queue, with no
> involvement from userland.
> 
> My only disagreement is when this should be implemented. I think it is
> fine to keep my custom implementation of this in the driver for now. Once
> an extension of vb2 is ready to support this feature, it would be fairly
> straightforward to strip out my custom implementation and go with the
> new API.

For a staging driver this isn't necessary, as long as it is documented in
the TODO file that this needs to be fixed before it can be moved out of
staging. The whole point of staging is that there is still work to be
done in the driver, after all :-)

BTW, did you look at the vb2_thread_* functions in videobuf2-core.c? A lot
of what you need is already there. Making a new version that has producer
and consumer queues shouldn't be hard given that code.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 20:39         ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 20:39 UTC (permalink / raw)
  To: Steve Longerbeam, Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On 01/20/2017 07:40 PM, Steve Longerbeam wrote:
> Hi Hans, Philipp,
> 
> 
> On 01/20/2017 08:31 AM, Philipp Zabel wrote:
>> Hi Hans,
>>
>> On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
>>> Hi Steve, Philipp,
>>>
>>> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>>>> In version 3:
>>>>
>>>> Changes suggested by Rob Herring <robh@kernel.org>:
>>>>
>>>>    - prepended FIM node properties with vendor prefix "fsl,".
>>>>
>>>>    - make mipi csi-2 receiver compatible string SoC specific:
>>>>      "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>>>>
>>>>    - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>>>>
>>>>    - removed board-specific info from the media driver binding doc. These
>>>>      were all related to sensor bindings, which already are (adv7180)
>>>>      or will be (ov564x) covered in separate binding docs. All reference
>>>>      board info not related to DT bindings has been moved to
>>>>      Documentation/media/v4l-drivers/imx.rst.
>>>>
>>>>    - removed "_mipi" from the OV5640 compatible string.
>>>>
>>>> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>>>>
>>>>    Mostly cosmetic/non-functional changes which I won't list here, except
>>>>    for the following:
>>>>
>>>>    - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>>>>
>>>>    - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>>>>
>>>>    - check/handle return code from required reg property of CSI port nodes.
>>>>
>>>>    - check/handle return code from clk_prepare_enable().
>>>>
>>>> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>>>>
>>>>    - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>>>>
>>>>    - finally got around to passing valid IOMUX pin config values to the
>>>>      pin groups.
>>>>
>>>> Other changes:
>>>>
>>>>    - removed the FIM properties that overrided the v4l2 FIM control defaults
>>>>      values. This was left-over from a requirement of a customer and is not
>>>>      necessary here.
>>>>
>>>>    - The FIM must be explicitly enabled in the fim child node under the CSI
>>>>      port nodes, using the status property. If not enabled, FIM v4l2 controls
>>>>      will not appear in the video capture driver.
>>>>
>>>>    - brought in additional media types patch from Philipp Zabel. Use new
>>>>      MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>>>>
>>>>    - brought in latest platform generic video multiplexer subdevice driver
>>>>      from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>>>>
>>>>    - removed imx-media-of.h, moved those prototypes into imx-media.h.
>>> Based on the discussion on the mailinglist it seems everyone agrees that this
>>> is the preferred driver, correct?
>> No. I have some major reservations against the custom mem2mem framework
>> embedded in Steve's driver.
>> I think it is a misuse of the media entity links (which should describe
>> hardware connections) for something that should be done at the vb2 level
>> (letting one device's capture EOF interrupt trigger the next device's
>> m2m device_run without going through userspace).
>> Steve and I disagree on that point, so we'd appreciate if we could get
>> some more eyes on the above issue.
> 
> This needs some background first, so let me first describe one example
> pipeline in this driver.
> 
> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
> hardware filters for motion compensation. Some of the motion compensation
> modes ("low" and "medium" motion) require that the VDIC receive video
> frame fields from memory buffers (dedicated dma channels in the
> IPU are used to transfer those buffers into the VDIC).
> 
> So one option to support those modes would be to pass the raw buffers
> from a camera sensor up to userspace to a capture device, and then pass
> them back to the VDIC for de-interlacing using a mem2mem device.
> 
> Philipp and I are both in agreement that, since userland is not interested
> in the intermediate interlaced buffers in this case, but only the final
> result (motion compensated, de-interlaced frames), it is more efficient
> to provide a media link that allows passing those intermediate frames
> directly from a camera source pad to VDIC sink pad, without having
> to route them through userspace.
> 
> So in order to support that, I've implemented a simple FIFO dma buffer
> queue in the driver to allow passing video buffers directly from a source
> to a sink. It is modeled loosely off the vb2 state machine and API, but
> simpler (for instance it only allows contiguous, cache-coherent buffers).
> 
> This is where Philipp has an argument, that this should be done with a
> new API in videobuf2.
> 
> And I'm actually in total agreement with that. I definitely agree that there
> should be a mechanism in the media framework that allows passing video
> buffers from a source pad to a sink pad using a software queue, with no
> involvement from userland.
> 
> My only disagreement is when this should be implemented. I think it is
> fine to keep my custom implementation of this in the driver for now. Once
> an extension of vb2 is ready to support this feature, it would be fairly
> straightforward to strip out my custom implementation and go with the
> new API.

For a staging driver this isn't necessary, as long as it is documented in
the TODO file that this needs to be fixed before it can be moved out of
staging. The whole point of staging is that there is still work to be
done in the driver, after all :-)

BTW, did you look at the vb2_thread_* functions in videobuf2-core.c? A lot
of what you need is already there. Making a new version that has producer
and consumer queues shouldn't be hard given that code.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-20 20:39         ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-20 20:39 UTC (permalink / raw)
  To: linux-arm-kernel

On 01/20/2017 07:40 PM, Steve Longerbeam wrote:
> Hi Hans, Philipp,
> 
> 
> On 01/20/2017 08:31 AM, Philipp Zabel wrote:
>> Hi Hans,
>>
>> On Fri, 2017-01-20 at 14:52 +0100, Hans Verkuil wrote:
>>> Hi Steve, Philipp,
>>>
>>> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>>>> In version 3:
>>>>
>>>> Changes suggested by Rob Herring <robh@kernel.org>:
>>>>
>>>>    - prepended FIM node properties with vendor prefix "fsl,".
>>>>
>>>>    - make mipi csi-2 receiver compatible string SoC specific:
>>>>      "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".
>>>>
>>>>    - redundant "_clk" removed from mipi csi-2 receiver clock-names property.
>>>>
>>>>    - removed board-specific info from the media driver binding doc. These
>>>>      were all related to sensor bindings, which already are (adv7180)
>>>>      or will be (ov564x) covered in separate binding docs. All reference
>>>>      board info not related to DT bindings has been moved to
>>>>      Documentation/media/v4l-drivers/imx.rst.
>>>>
>>>>    - removed "_mipi" from the OV5640 compatible string.
>>>>
>>>> Changes suggested by Vladimir Zapolskiy <vladimir_zapolskiy@mentor.com>:
>>>>
>>>>    Mostly cosmetic/non-functional changes which I won't list here, except
>>>>    for the following:
>>>>
>>>>    - spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.
>>>>
>>>>    - fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.
>>>>
>>>>    - check/handle return code from required reg property of CSI port nodes.
>>>>
>>>>    - check/handle return code from clk_prepare_enable().
>>>>
>>>> Changes suggested by Fabio Estevam <festevam@gmail.com>:
>>>>
>>>>    - switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.
>>>>
>>>>    - finally got around to passing valid IOMUX pin config values to the
>>>>      pin groups.
>>>>
>>>> Other changes:
>>>>
>>>>    - removed the FIM properties that overrided the v4l2 FIM control defaults
>>>>      values. This was left-over from a requirement of a customer and is not
>>>>      necessary here.
>>>>
>>>>    - The FIM must be explicitly enabled in the fim child node under the CSI
>>>>      port nodes, using the status property. If not enabled, FIM v4l2 controls
>>>>      will not appear in the video capture driver.
>>>>
>>>>    - brought in additional media types patch from Philipp Zabel. Use new
>>>>      MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.
>>>>
>>>>    - brought in latest platform generic video multiplexer subdevice driver
>>>>      from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).
>>>>
>>>>    - removed imx-media-of.h, moved those prototypes into imx-media.h.
>>> Based on the discussion on the mailinglist it seems everyone agrees that this
>>> is the preferred driver, correct?
>> No. I have some major reservations against the custom mem2mem framework
>> embedded in Steve's driver.
>> I think it is a misuse of the media entity links (which should describe
>> hardware connections) for something that should be done at the vb2 level
>> (letting one device's capture EOF interrupt trigger the next device's
>> m2m device_run without going through userspace).
>> Steve and I disagree on that point, so we'd appreciate if we could get
>> some more eyes on the above issue.
> 
> This needs some background first, so let me first describe one example
> pipeline in this driver.
> 
> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
> hardware filters for motion compensation. Some of the motion compensation
> modes ("low" and "medium" motion) require that the VDIC receive video
> frame fields from memory buffers (dedicated dma channels in the
> IPU are used to transfer those buffers into the VDIC).
> 
> So one option to support those modes would be to pass the raw buffers
> from a camera sensor up to userspace to a capture device, and then pass
> them back to the VDIC for de-interlacing using a mem2mem device.
> 
> Philipp and I are both in agreement that, since userland is not interested
> in the intermediate interlaced buffers in this case, but only the final
> result (motion compensated, de-interlaced frames), it is more efficient
> to provide a media link that allows passing those intermediate frames
> directly from a camera source pad to VDIC sink pad, without having
> to route them through userspace.
> 
> So in order to support that, I've implemented a simple FIFO dma buffer
> queue in the driver to allow passing video buffers directly from a source
> to a sink. It is modeled loosely off the vb2 state machine and API, but
> simpler (for instance it only allows contiguous, cache-coherent buffers).
> 
> This is where Philipp has an argument, that this should be done with a
> new API in videobuf2.
> 
> And I'm actually in total agreement with that. I definitely agree that there
> should be a mechanism in the media framework that allows passing video
> buffers from a source pad to a sink pad using a software queue, with no
> involvement from userland.
> 
> My only disagreement is when this should be implemented. I think it is
> fine to keep my custom implementation of this in the driver for now. Once
> an extension of vb2 is ready to support this feature, it would be fairly
> straightforward to strip out my custom implementation and go with the
> new API.

For a staging driver this isn't necessary, as long as it is documented in
the TODO file that this needs to be fixed before it can be moved out of
staging. The whole point of staging is that there is still work to be
done in the driver, after all :-)

BTW, did you look at the vb2_thread_* functions in videobuf2-core.c? A lot
of what you need is already there. Making a new version that has producer
and consumer queues shouldn't be hard given that code.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-23  2:31           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-23  2:31 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/16/2017 05:47 AM, Philipp Zabel wrote:
> On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
> [...]
>>>> +Unprocessed Video Capture:
>>>> +--------------------------
>>>> +
>>>> +Send frames directly from sensor to camera interface, with no
>>>> +conversions:
>>>> +
>>>> +-> ipu_smfc -> camif
>>> I'd call this capture interface, this is not just for cameras. Or maybe
>>> idmac if you want to mirror hardware names?
>> Camif is so named because it is the V4L2 user interface for video
>> capture. I suppose it could be named "capif", but that doesn't role
>> off the tongue quite as well.
> Agreed, capif sounds weird. I find camif a bit confusing though, because
> Samsung S3C has a camera interface that is actually called "CAMIF".

how about simply "capture" ?


<snip>

>>> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
>> I agree this looks very intuitive, but technically correct for the
>> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
>> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>>
>> I think it would be better to use the correct format
>> I'm not sure I follow you here.
> The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0
> up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values
> into the SMFC, so the CSI output pad and the IDMAC input pad should have
> a YUV8_1X32 format.
>
> Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
> IDMAC chapter states that all internal submodules only work on 8-bit per
> component formats with four components: YUVA or RGBA.

Right, the "direct" IPU internal (that do not transfer buffers to/from
memory via IDMAC channels) should only allow the IPU internal
formats YUVA and RGBA. I get you now.

The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and
MEDIA_BUS_FMT_ARGB8888_1X32.

Those pads are:

ipu_csi:1
ipu_vdic:1
ipu_ic_prp:0
ipu_ic_prp:1
ipu_ic_prpenc:0
ipu_ic_prpenc:1
ipu_ic_prpvf:0
ipu_ic_prpvf:1


<snip>
>
>> There does not appear to be support in the FSU for linking a write channel
>> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
>> is support for the direct link from CSI (which I am using), but that's
>> not an
>> IDMAC channel link.
>>
>> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
>> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
> As I read it, that is 0 and 2 only, no idea why. But since there are
> only 2 CSIs, that shouldn't be a problem.

ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still
at channel 0).

<snip>

>>>> +static const u32 power_off_seq[] = {
>>>> +	IMX_MEDIA_GRP_ID_IC_PP,
>>>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>>>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>>>> +	IMX_MEDIA_GRP_ID_SMFC,
>>>> +	IMX_MEDIA_GRP_ID_CSI,
>>>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>>>> +	IMX_MEDIA_GRP_ID_SENSOR,
>>>> +	IMX_MEDIA_GRP_ID_CSI2,
>>>> +};
>>> This seems somewhat arbitrary. Why is a power sequence needed?
>> The CSI-2 receiver must be powered up before the sensor, that's the
>> only requirement IIRC. The others have no s_power requirement. So I
>> can probably change this to power up in the frontend -> backend order
>> (IC_PP to sensor). And vice-versa for power off.
> Yes, I think that should work (see below).

Actually there are problems using this, see below.

>>> [...]
>>>> +/*
>>>> + * Turn current pipeline power on/off starting from start_entity.
>>>> + * Must be called with mdev->graph_mutex held.
>>>> + */
>>>> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
>>>> +				 struct media_entity_graph *graph,
>>>> +				 struct media_entity *start_entity, bool on)
>>>> +{
>>>> +	struct media_entity *entity;
>>>> +	struct v4l2_subdev *sd;
>>>> +	int i, ret = 0;
>>>> +	u32 id;
>>>> +
>>>> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
>>>> +		id = on ? power_on_seq[i] : power_off_seq[i];
>>>> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
>>>> +		if (!entity)
>>>> +			continue;
>>>> +
>>>> +		sd = media_entity_to_v4l2_subdev(entity);
>>>> +
>>>> +		ret = v4l2_subdev_call(sd, core, s_power, on);
>>>> +		if (ret && ret != -ENOIOCTLCMD)
>>>> +			break;
>>>> +	}
>>>> +
>>>> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
>>> This should really be handled by v4l2_pipeline_pm_use.
>> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
>> doing some other stuff that bothered me, at least that's what I remember.
>> I will revisit this.
> I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
> any problems. It would be better to reuse and, if necessary, fix the
> existing infrastructure where available.

I tried this API, by switching to v4l2_pipeline_pm_use() in camif 
open/release,
and switched to v4l2_pipeline_link_notify() instead of 
imx_media_link_notify()
in the media driver's media_device_ops.

This API assumes the video device has an open file handle while the media
links are being established. This doesn't work for me, I want to be able to
establish the links using 'media-ctl -l', and that won't work unless 
there is an
open file handle on the video capture device node.

Also, I looked into calling v4l2_pipeline_pm_use() during 
imx_media_link_notify(),
instead of imx_media_pipeline_set_power(). Again there are problems with 
that.

First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be 
called inside
link_notify which already acquires that lock. The header for this 
function also
clearly states it should only be called in open/release.

Second, ignoring the above locking issue for a moment, 
v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
wrong order.
In my version which enforces the correct power on order, the mipi csi-2 
s_power
is called first in that link setup, followed by the sensor.



>>>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>>>> +				   struct imx_media_subdev *csi[4])
>>>> +{
>>>> +	int ret;
>>>> +
>>>> +	/* there must be at least one CSI in first IPU */
>>> Why?
>> Well yeah, imx-media doesn't necessarily need a CSI if things
>> like the VDIC or post-processor are being used by an output
>> overlay pipeline, for example. I'll fix this.
> I haven't even thought that far, but there could be boards with only a
> parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
> saving reasons.

done! A very simple change to imx_media_add_internal_subdevs(),
and now no CSI's are necessary, but the remaining IPU internal entities
are loaded and linked just fine without them, so for example with no
CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and
could still be used in the future by a mem2mem device for instance.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-23  2:31           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-23  2:31 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/16/2017 05:47 AM, Philipp Zabel wrote:
> On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
> [...]
>>>> +Unprocessed Video Capture:
>>>> +--------------------------
>>>> +
>>>> +Send frames directly from sensor to camera interface, with no
>>>> +conversions:
>>>> +
>>>> +-> ipu_smfc -> camif
>>> I'd call this capture interface, this is not just for cameras. Or maybe
>>> idmac if you want to mirror hardware names?
>> Camif is so named because it is the V4L2 user interface for video
>> capture. I suppose it could be named "capif", but that doesn't role
>> off the tongue quite as well.
> Agreed, capif sounds weird. I find camif a bit confusing though, because
> Samsung S3C has a camera interface that is actually called "CAMIF".

how about simply "capture" ?


<snip>

>>> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
>> I agree this looks very intuitive, but technically correct for the
>> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
>> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>>
>> I think it would be better to use the correct format
>> I'm not sure I follow you here.
> The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0
> up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values
> into the SMFC, so the CSI output pad and the IDMAC input pad should have
> a YUV8_1X32 format.
>
> Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
> IDMAC chapter states that all internal submodules only work on 8-bit per
> component formats with four components: YUVA or RGBA.

Right, the "direct" IPU internal (that do not transfer buffers to/from
memory via IDMAC channels) should only allow the IPU internal
formats YUVA and RGBA. I get you now.

The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and
MEDIA_BUS_FMT_ARGB8888_1X32.

Those pads are:

ipu_csi:1
ipu_vdic:1
ipu_ic_prp:0
ipu_ic_prp:1
ipu_ic_prpenc:0
ipu_ic_prpenc:1
ipu_ic_prpvf:0
ipu_ic_prpvf:1


<snip>
>
>> There does not appear to be support in the FSU for linking a write channel
>> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
>> is support for the direct link from CSI (which I am using), but that's
>> not an
>> IDMAC channel link.
>>
>> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
>> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
> As I read it, that is 0 and 2 only, no idea why. But since there are
> only 2 CSIs, that shouldn't be a problem.

ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still
at channel 0).

<snip>

>>>> +static const u32 power_off_seq[] = {
>>>> +	IMX_MEDIA_GRP_ID_IC_PP,
>>>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>>>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>>>> +	IMX_MEDIA_GRP_ID_SMFC,
>>>> +	IMX_MEDIA_GRP_ID_CSI,
>>>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>>>> +	IMX_MEDIA_GRP_ID_SENSOR,
>>>> +	IMX_MEDIA_GRP_ID_CSI2,
>>>> +};
>>> This seems somewhat arbitrary. Why is a power sequence needed?
>> The CSI-2 receiver must be powered up before the sensor, that's the
>> only requirement IIRC. The others have no s_power requirement. So I
>> can probably change this to power up in the frontend -> backend order
>> (IC_PP to sensor). And vice-versa for power off.
> Yes, I think that should work (see below).

Actually there are problems using this, see below.

>>> [...]
>>>> +/*
>>>> + * Turn current pipeline power on/off starting from start_entity.
>>>> + * Must be called with mdev->graph_mutex held.
>>>> + */
>>>> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
>>>> +				 struct media_entity_graph *graph,
>>>> +				 struct media_entity *start_entity, bool on)
>>>> +{
>>>> +	struct media_entity *entity;
>>>> +	struct v4l2_subdev *sd;
>>>> +	int i, ret = 0;
>>>> +	u32 id;
>>>> +
>>>> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
>>>> +		id = on ? power_on_seq[i] : power_off_seq[i];
>>>> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
>>>> +		if (!entity)
>>>> +			continue;
>>>> +
>>>> +		sd = media_entity_to_v4l2_subdev(entity);
>>>> +
>>>> +		ret = v4l2_subdev_call(sd, core, s_power, on);
>>>> +		if (ret && ret != -ENOIOCTLCMD)
>>>> +			break;
>>>> +	}
>>>> +
>>>> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
>>> This should really be handled by v4l2_pipeline_pm_use.
>> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
>> doing some other stuff that bothered me, at least that's what I remember.
>> I will revisit this.
> I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
> any problems. It would be better to reuse and, if necessary, fix the
> existing infrastructure where available.

I tried this API, by switching to v4l2_pipeline_pm_use() in camif 
open/release,
and switched to v4l2_pipeline_link_notify() instead of 
imx_media_link_notify()
in the media driver's media_device_ops.

This API assumes the video device has an open file handle while the media
links are being established. This doesn't work for me, I want to be able to
establish the links using 'media-ctl -l', and that won't work unless 
there is an
open file handle on the video capture device node.

Also, I looked into calling v4l2_pipeline_pm_use() during 
imx_media_link_notify(),
instead of imx_media_pipeline_set_power(). Again there are problems with 
that.

First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be 
called inside
link_notify which already acquires that lock. The header for this 
function also
clearly states it should only be called in open/release.

Second, ignoring the above locking issue for a moment, 
v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
wrong order.
In my version which enforces the correct power on order, the mipi csi-2 
s_power
is called first in that link setup, followed by the sensor.



>>>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>>>> +				   struct imx_media_subdev *csi[4])
>>>> +{
>>>> +	int ret;
>>>> +
>>>> +	/* there must be at least one CSI in first IPU */
>>> Why?
>> Well yeah, imx-media doesn't necessarily need a CSI if things
>> like the VDIC or post-processor are being used by an output
>> overlay pipeline, for example. I'll fix this.
> I haven't even thought that far, but there could be boards with only a
> parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
> saving reasons.

done! A very simple change to imx_media_add_internal_subdevs(),
and now no CSI's are necessary, but the remaining IPU internal entities
are loaded and linked just fine without them, so for example with no
CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and
could still be used in the future by a mem2mem device for instance.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-23  2:31           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-23  2:31 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/16/2017 05:47 AM, Philipp Zabel wrote:
> On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
> [...]
>>>> +Unprocessed Video Capture:
>>>> +--------------------------
>>>> +
>>>> +Send frames directly from sensor to camera interface, with no
>>>> +conversions:
>>>> +
>>>> +-> ipu_smfc -> camif
>>> I'd call this capture interface, this is not just for cameras. Or maybe
>>> idmac if you want to mirror hardware names?
>> Camif is so named because it is the V4L2 user interface for video
>> capture. I suppose it could be named "capif", but that doesn't role
>> off the tongue quite as well.
> Agreed, capif sounds weird. I find camif a bit confusing though, because
> Samsung S3C has a camera interface that is actually called "CAMIF".

how about simply "capture" ?


<snip>

>>> +   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
>> I agree this looks very intuitive, but technically correct for the
>> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
>> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>>
>> I think it would be better to use the correct format
>> I'm not sure I follow you here.
> The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0
> up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values
> into the SMFC, so the CSI output pad and the IDMAC input pad should have
> a YUV8_1X32 format.
>
> Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
> IDMAC chapter states that all internal submodules only work on 8-bit per
> component formats with four components: YUVA or RGBA.

Right, the "direct" IPU internal (that do not transfer buffers to/from
memory via IDMAC channels) should only allow the IPU internal
formats YUVA and RGBA. I get you now.

The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and
MEDIA_BUS_FMT_ARGB8888_1X32.

Those pads are:

ipu_csi:1
ipu_vdic:1
ipu_ic_prp:0
ipu_ic_prp:1
ipu_ic_prpenc:0
ipu_ic_prpenc:1
ipu_ic_prpvf:0
ipu_ic_prpvf:1


<snip>
>
>> There does not appear to be support in the FSU for linking a write channel
>> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
>> is support for the direct link from CSI (which I am using), but that's
>> not an
>> IDMAC channel link.
>>
>> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
>> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
> As I read it, that is 0 and 2 only, no idea why. But since there are
> only 2 CSIs, that shouldn't be a problem.

ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still
at channel 0).

<snip>

>>>> +static const u32 power_off_seq[] = {
>>>> +	IMX_MEDIA_GRP_ID_IC_PP,
>>>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
>>>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
>>>> +	IMX_MEDIA_GRP_ID_SMFC,
>>>> +	IMX_MEDIA_GRP_ID_CSI,
>>>> +	IMX_MEDIA_GRP_ID_VIDMUX,
>>>> +	IMX_MEDIA_GRP_ID_SENSOR,
>>>> +	IMX_MEDIA_GRP_ID_CSI2,
>>>> +};
>>> This seems somewhat arbitrary. Why is a power sequence needed?
>> The CSI-2 receiver must be powered up before the sensor, that's the
>> only requirement IIRC. The others have no s_power requirement. So I
>> can probably change this to power up in the frontend -> backend order
>> (IC_PP to sensor). And vice-versa for power off.
> Yes, I think that should work (see below).

Actually there are problems using this, see below.

>>> [...]
>>>> +/*
>>>> + * Turn current pipeline power on/off starting from start_entity.
>>>> + * Must be called with mdev->graph_mutex held.
>>>> + */
>>>> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
>>>> +				 struct media_entity_graph *graph,
>>>> +				 struct media_entity *start_entity, bool on)
>>>> +{
>>>> +	struct media_entity *entity;
>>>> +	struct v4l2_subdev *sd;
>>>> +	int i, ret = 0;
>>>> +	u32 id;
>>>> +
>>>> +	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
>>>> +		id = on ? power_on_seq[i] : power_off_seq[i];
>>>> +		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
>>>> +		if (!entity)
>>>> +			continue;
>>>> +
>>>> +		sd = media_entity_to_v4l2_subdev(entity);
>>>> +
>>>> +		ret = v4l2_subdev_call(sd, core, s_power, on);
>>>> +		if (ret && ret != -ENOIOCTLCMD)
>>>> +			break;
>>>> +	}
>>>> +
>>>> +	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
>>> This should really be handled by v4l2_pipeline_pm_use.
>> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
>> doing some other stuff that bothered me, at least that's what I remember.
>> I will revisit this.
> I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
> any problems. It would be better to reuse and, if necessary, fix the
> existing infrastructure where available.

I tried this API, by switching to v4l2_pipeline_pm_use() in camif 
open/release,
and switched to v4l2_pipeline_link_notify() instead of 
imx_media_link_notify()
in the media driver's media_device_ops.

This API assumes the video device has an open file handle while the media
links are being established. This doesn't work for me, I want to be able to
establish the links using 'media-ctl -l', and that won't work unless 
there is an
open file handle on the video capture device node.

Also, I looked into calling v4l2_pipeline_pm_use() during 
imx_media_link_notify(),
instead of imx_media_pipeline_set_power(). Again there are problems with 
that.

First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be 
called inside
link_notify which already acquires that lock. The header for this 
function also
clearly states it should only be called in open/release.

Second, ignoring the above locking issue for a moment, 
v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
wrong order.
In my version which enforces the correct power on order, the mipi csi-2 
s_power
is called first in that link setup, followed by the sensor.



>>>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>>>> +				   struct imx_media_subdev *csi[4])
>>>> +{
>>>> +	int ret;
>>>> +
>>>> +	/* there must be at least one CSI in first IPU */
>>> Why?
>> Well yeah, imx-media doesn't necessarily need a CSI if things
>> like the VDIC or post-processor are being used by an output
>> overlay pipeline, for example. I'll fix this.
> I haven't even thought that far, but there could be boards with only a
> parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
> saving reasons.

done! A very simple change to imx_media_add_internal_subdevs(),
and now no CSI's are necessary, but the remaining IPU internal entities
are loaded and linked just fine without them, so for example with no
CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and
could still be used in the future by a mem2mem device for instance.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-20 20:39         ` Hans Verkuil
  (?)
@ 2017-01-23 11:00           ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-23 11:00 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
[...]
> > There is a VDIC entity in the i.MX IPU that performs de-interlacing with
> > hardware filters for motion compensation. Some of the motion compensation
> > modes ("low" and "medium" motion) require that the VDIC receive video
> > frame fields from memory buffers (dedicated dma channels in the
> > IPU are used to transfer those buffers into the VDIC).
> > 
> > So one option to support those modes would be to pass the raw buffers
> > from a camera sensor up to userspace to a capture device, and then pass
> > them back to the VDIC for de-interlacing using a mem2mem device.
> > 
> > Philipp and I are both in agreement that, since userland is not interested
> > in the intermediate interlaced buffers in this case, but only the final
> > result (motion compensated, de-interlaced frames), it is more efficient
> > to provide a media link that allows passing those intermediate frames
> > directly from a camera source pad to VDIC sink pad, without having
> > to route them through userspace.
> > 
> > So in order to support that, I've implemented a simple FIFO dma buffer
> > queue in the driver to allow passing video buffers directly from a source
> > to a sink. It is modeled loosely off the vb2 state machine and API, but
> > simpler (for instance it only allows contiguous, cache-coherent buffers).
> > 
> > This is where Philipp has an argument, that this should be done with a
> > new API in videobuf2.

That is one part of the argument. I'm glad to understand now that we
agree about this.

> > And I'm actually in total agreement with that. I definitely agree that there
> > should be a mechanism in the media framework that allows passing video
> > buffers from a source pad to a sink pad using a software queue, with no
> > involvement from userland.

That is the other part of the argument. I do not agree that these
software queue "links" should be presented to userspace as media pad
links between two entities of a media device. 

First, that would limit the links to subdevices contained in the same
media graph, while this should work between any two capture and output
queues of different devices.
Assume for example, we want to encode the captured, deinterlaced video
to h.264 with the coda VPU driver. A software queue link could be
established between the CSI capture and the VDIC deinterlacer input,
just as between the VDIC deinterlacer output and the coda VPU input.
Technically, there would be no difference between those two linked
capture/output queue pairs. But the coda driver is a completely separate
mem2mem device. And since it is not part of the i.MX media graph, there
is no entity pad to link to.
Or assume there is an USB analog capture device that produces interlaced
frames. I think it should be possible to connect its capture queue to
the VDIC deinterlacer output queue just the same way as linking the CSI
to the VDIC (in software queue mode).

Second, the subdevice pad formats describe wire formats, not memory
formats. The user might want to choose between 4:2:2 and 4:2:0
subsampled YUV formats for the intermediate buffer, for example,
depending on memory bandwidth constraints and quality requirements. This
is impossible with the media entity / subdevice pad links.

I think an interface where userspace configures the capture and output
queues via v4l2 API, passes dma buffers around from one to the other
queue, and then puts both queues into a free running mode would be a
much better fit for this mechanism.

> > My only disagreement is when this should be implemented. I think it is
> > fine to keep my custom implementation of this in the driver for now. Once
> > an extension of vb2 is ready to support this feature, it would be fairly
> > straightforward to strip out my custom implementation and go with the
> > new API.
> 
> For a staging driver this isn't necessary, as long as it is documented in
> the TODO file that this needs to be fixed before it can be moved out of
> staging. The whole point of staging is that there is still work to be
> done in the driver, after all :-)

Absolutely. The reason I am arguing against merging the mem2mem media
control links so vehemently is that I am convinced the userspace
interface is wrong, and I am afraid that even though in staging, it
might become established.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-23 11:00           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-23 11:00 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
[...]
> > There is a VDIC entity in the i.MX IPU that performs de-interlacing with
> > hardware filters for motion compensation. Some of the motion compensation
> > modes ("low" and "medium" motion) require that the VDIC receive video
> > frame fields from memory buffers (dedicated dma channels in the
> > IPU are used to transfer those buffers into the VDIC).
> > 
> > So one option to support those modes would be to pass the raw buffers
> > from a camera sensor up to userspace to a capture device, and then pass
> > them back to the VDIC for de-interlacing using a mem2mem device.
> > 
> > Philipp and I are both in agreement that, since userland is not interested
> > in the intermediate interlaced buffers in this case, but only the final
> > result (motion compensated, de-interlaced frames), it is more efficient
> > to provide a media link that allows passing those intermediate frames
> > directly from a camera source pad to VDIC sink pad, without having
> > to route them through userspace.
> > 
> > So in order to support that, I've implemented a simple FIFO dma buffer
> > queue in the driver to allow passing video buffers directly from a source
> > to a sink. It is modeled loosely off the vb2 state machine and API, but
> > simpler (for instance it only allows contiguous, cache-coherent buffers).
> > 
> > This is where Philipp has an argument, that this should be done with a
> > new API in videobuf2.

That is one part of the argument. I'm glad to understand now that we
agree about this.

> > And I'm actually in total agreement with that. I definitely agree that there
> > should be a mechanism in the media framework that allows passing video
> > buffers from a source pad to a sink pad using a software queue, with no
> > involvement from userland.

That is the other part of the argument. I do not agree that these
software queue "links" should be presented to userspace as media pad
links between two entities of a media device. 

First, that would limit the links to subdevices contained in the same
media graph, while this should work between any two capture and output
queues of different devices.
Assume for example, we want to encode the captured, deinterlaced video
to h.264 with the coda VPU driver. A software queue link could be
established between the CSI capture and the VDIC deinterlacer input,
just as between the VDIC deinterlacer output and the coda VPU input.
Technically, there would be no difference between those two linked
capture/output queue pairs. But the coda driver is a completely separate
mem2mem device. And since it is not part of the i.MX media graph, there
is no entity pad to link to.
Or assume there is an USB analog capture device that produces interlaced
frames. I think it should be possible to connect its capture queue to
the VDIC deinterlacer output queue just the same way as linking the CSI
to the VDIC (in software queue mode).

Second, the subdevice pad formats describe wire formats, not memory
formats. The user might want to choose between 4:2:2 and 4:2:0
subsampled YUV formats for the intermediate buffer, for example,
depending on memory bandwidth constraints and quality requirements. This
is impossible with the media entity / subdevice pad links.

I think an interface where userspace configures the capture and output
queues via v4l2 API, passes dma buffers around from one to the other
queue, and then puts both queues into a free running mode would be a
much better fit for this mechanism.

> > My only disagreement is when this should be implemented. I think it is
> > fine to keep my custom implementation of this in the driver for now. Once
> > an extension of vb2 is ready to support this feature, it would be fairly
> > straightforward to strip out my custom implementation and go with the
> > new API.
> 
> For a staging driver this isn't necessary, as long as it is documented in
> the TODO file that this needs to be fixed before it can be moved out of
> staging. The whole point of staging is that there is still work to be
> done in the driver, after all :-)

Absolutely. The reason I am arguing against merging the mem2mem media
control links so vehemently is that I am convinced the userspace
interface is wrong, and I am afraid that even though in staging, it
might become established.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-23 11:00           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-23 11:00 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
[...]
> > There is a VDIC entity in the i.MX IPU that performs de-interlacing with
> > hardware filters for motion compensation. Some of the motion compensation
> > modes ("low" and "medium" motion) require that the VDIC receive video
> > frame fields from memory buffers (dedicated dma channels in the
> > IPU are used to transfer those buffers into the VDIC).
> > 
> > So one option to support those modes would be to pass the raw buffers
> > from a camera sensor up to userspace to a capture device, and then pass
> > them back to the VDIC for de-interlacing using a mem2mem device.
> > 
> > Philipp and I are both in agreement that, since userland is not interested
> > in the intermediate interlaced buffers in this case, but only the final
> > result (motion compensated, de-interlaced frames), it is more efficient
> > to provide a media link that allows passing those intermediate frames
> > directly from a camera source pad to VDIC sink pad, without having
> > to route them through userspace.
> > 
> > So in order to support that, I've implemented a simple FIFO dma buffer
> > queue in the driver to allow passing video buffers directly from a source
> > to a sink. It is modeled loosely off the vb2 state machine and API, but
> > simpler (for instance it only allows contiguous, cache-coherent buffers).
> > 
> > This is where Philipp has an argument, that this should be done with a
> > new API in videobuf2.

That is one part of the argument. I'm glad to understand now that we
agree about this.

> > And I'm actually in total agreement with that. I definitely agree that there
> > should be a mechanism in the media framework that allows passing video
> > buffers from a source pad to a sink pad using a software queue, with no
> > involvement from userland.

That is the other part of the argument. I do not agree that these
software queue "links" should be presented to userspace as media pad
links between two entities of a media device. 

First, that would limit the links to subdevices contained in the same
media graph, while this should work between any two capture and output
queues of different devices.
Assume for example, we want to encode the captured, deinterlaced video
to h.264 with the coda VPU driver. A software queue link could be
established between the CSI capture and the VDIC deinterlacer input,
just as between the VDIC deinterlacer output and the coda VPU input.
Technically, there would be no difference between those two linked
capture/output queue pairs. But the coda driver is a completely separate
mem2mem device. And since it is not part of the i.MX media graph, there
is no entity pad to link to.
Or assume there is an USB analog capture device that produces interlaced
frames. I think it should be possible to connect its capture queue to
the VDIC deinterlacer output queue just the same way as linking the CSI
to the VDIC (in software queue mode).

Second, the subdevice pad formats describe wire formats, not memory
formats. The user might want to choose between 4:2:2 and 4:2:0
subsampled YUV formats for the intermediate buffer, for example,
depending on memory bandwidth constraints and quality requirements. This
is impossible with the media entity / subdevice pad links.

I think an interface where userspace configures the capture and output
queues via v4l2 API, passes dma buffers around from one to the other
queue, and then puts both queues into a free running mode would be a
much better fit for this mechanism.

> > My only disagreement is when this should be implemented. I think it is
> > fine to keep my custom implementation of this in the driver for now. Once
> > an extension of vb2 is ready to support this feature, it would be fairly
> > straightforward to strip out my custom implementation and go with the
> > new API.
> 
> For a staging driver this isn't necessary, as long as it is documented in
> the TODO file that this needs to be fixed before it can be moved out of
> staging. The whole point of staging is that there is still work to be
> done in the driver, after all :-)

Absolutely. The reason I am arguing against merging the mem2mem media
control links so vehemently is that I am convinced the userspace
interface is wrong, and I am afraid that even though in staging, it
might become established.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-23 11:00           ` Philipp Zabel
  (?)
@ 2017-01-23 11:08             ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-23 11:08 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On 01/23/2017 12:00 PM, Philipp Zabel wrote:
> On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
> [...]
>>> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
>>> hardware filters for motion compensation. Some of the motion compensation
>>> modes ("low" and "medium" motion) require that the VDIC receive video
>>> frame fields from memory buffers (dedicated dma channels in the
>>> IPU are used to transfer those buffers into the VDIC).
>>>
>>> So one option to support those modes would be to pass the raw buffers
>>> from a camera sensor up to userspace to a capture device, and then pass
>>> them back to the VDIC for de-interlacing using a mem2mem device.
>>>
>>> Philipp and I are both in agreement that, since userland is not interested
>>> in the intermediate interlaced buffers in this case, but only the final
>>> result (motion compensated, de-interlaced frames), it is more efficient
>>> to provide a media link that allows passing those intermediate frames
>>> directly from a camera source pad to VDIC sink pad, without having
>>> to route them through userspace.
>>>
>>> So in order to support that, I've implemented a simple FIFO dma buffer
>>> queue in the driver to allow passing video buffers directly from a source
>>> to a sink. It is modeled loosely off the vb2 state machine and API, but
>>> simpler (for instance it only allows contiguous, cache-coherent buffers).
>>>
>>> This is where Philipp has an argument, that this should be done with a
>>> new API in videobuf2.
> 
> That is one part of the argument. I'm glad to understand now that we
> agree about this.
> 
>>> And I'm actually in total agreement with that. I definitely agree that there
>>> should be a mechanism in the media framework that allows passing video
>>> buffers from a source pad to a sink pad using a software queue, with no
>>> involvement from userland.
> 
> That is the other part of the argument. I do not agree that these
> software queue "links" should be presented to userspace as media pad
> links between two entities of a media device. 
> 
> First, that would limit the links to subdevices contained in the same
> media graph, while this should work between any two capture and output
> queues of different devices.
> Assume for example, we want to encode the captured, deinterlaced video
> to h.264 with the coda VPU driver. A software queue link could be
> established between the CSI capture and the VDIC deinterlacer input,
> just as between the VDIC deinterlacer output and the coda VPU input.
> Technically, there would be no difference between those two linked
> capture/output queue pairs. But the coda driver is a completely separate
> mem2mem device. And since it is not part of the i.MX media graph, there
> is no entity pad to link to.
> Or assume there is an USB analog capture device that produces interlaced
> frames. I think it should be possible to connect its capture queue to
> the VDIC deinterlacer output queue just the same way as linking the CSI
> to the VDIC (in software queue mode).
> 
> Second, the subdevice pad formats describe wire formats, not memory
> formats. The user might want to choose between 4:2:2 and 4:2:0
> subsampled YUV formats for the intermediate buffer, for example,
> depending on memory bandwidth constraints and quality requirements. This
> is impossible with the media entity / subdevice pad links.
> 
> I think an interface where userspace configures the capture and output
> queues via v4l2 API, passes dma buffers around from one to the other
> queue, and then puts both queues into a free running mode would be a
> much better fit for this mechanism.
> 
>>> My only disagreement is when this should be implemented. I think it is
>>> fine to keep my custom implementation of this in the driver for now. Once
>>> an extension of vb2 is ready to support this feature, it would be fairly
>>> straightforward to strip out my custom implementation and go with the
>>> new API.
>>
>> For a staging driver this isn't necessary, as long as it is documented in
>> the TODO file that this needs to be fixed before it can be moved out of
>> staging. The whole point of staging is that there is still work to be
>> done in the driver, after all :-)
> 
> Absolutely. The reason I am arguing against merging the mem2mem media
> control links so vehemently is that I am convinced the userspace
> interface is wrong, and I am afraid that even though in staging, it
> might become established.

As long as it is mentioned in the TODO, and ideally in the Kconfig as well,
then I'm fine with it.

The big advantage of being in the kernel is that it is much easier to start
providing fixes, improvements, etc. If you use a staging driver you know
that there is no guarantee whatsoever with respect to stable ABI/APIs.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-23 11:08             ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-23 11:08 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

On 01/23/2017 12:00 PM, Philipp Zabel wrote:
> On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
> [...]
>>> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
>>> hardware filters for motion compensation. Some of the motion compensation
>>> modes ("low" and "medium" motion) require that the VDIC receive video
>>> frame fields from memory buffers (dedicated dma channels in the
>>> IPU are used to transfer those buffers into the VDIC).
>>>
>>> So one option to support those modes would be to pass the raw buffers
>>> from a camera sensor up to userspace to a capture device, and then pass
>>> them back to the VDIC for de-interlacing using a mem2mem device.
>>>
>>> Philipp and I are both in agreement that, since userland is not interested
>>> in the intermediate interlaced buffers in this case, but only the final
>>> result (motion compensated, de-interlaced frames), it is more efficient
>>> to provide a media link that allows passing those intermediate frames
>>> directly from a camera source pad to VDIC sink pad, without having
>>> to route them through userspace.
>>>
>>> So in order to support that, I've implemented a simple FIFO dma buffer
>>> queue in the driver to allow passing video buffers directly from a source
>>> to a sink. It is modeled loosely off the vb2 state machine and API, but
>>> simpler (for instance it only allows contiguous, cache-coherent buffers).
>>>
>>> This is where Philipp has an argument, that this should be done with a
>>> new API in videobuf2.
> 
> That is one part of the argument. I'm glad to understand now that we
> agree about this.
> 
>>> And I'm actually in total agreement with that. I definitely agree that there
>>> should be a mechanism in the media framework that allows passing video
>>> buffers from a source pad to a sink pad using a software queue, with no
>>> involvement from userland.
> 
> That is the other part of the argument. I do not agree that these
> software queue "links" should be presented to userspace as media pad
> links between two entities of a media device. 
> 
> First, that would limit the links to subdevices contained in the same
> media graph, while this should work between any two capture and output
> queues of different devices.
> Assume for example, we want to encode the captured, deinterlaced video
> to h.264 with the coda VPU driver. A software queue link could be
> established between the CSI capture and the VDIC deinterlacer input,
> just as between the VDIC deinterlacer output and the coda VPU input.
> Technically, there would be no difference between those two linked
> capture/output queue pairs. But the coda driver is a completely separate
> mem2mem device. And since it is not part of the i.MX media graph, there
> is no entity pad to link to.
> Or assume there is an USB analog capture device that produces interlaced
> frames. I think it should be possible to connect its capture queue to
> the VDIC deinterlacer output queue just the same way as linking the CSI
> to the VDIC (in software queue mode).
> 
> Second, the subdevice pad formats describe wire formats, not memory
> formats. The user might want to choose between 4:2:2 and 4:2:0
> subsampled YUV formats for the intermediate buffer, for example,
> depending on memory bandwidth constraints and quality requirements. This
> is impossible with the media entity / subdevice pad links.
> 
> I think an interface where userspace configures the capture and output
> queues via v4l2 API, passes dma buffers around from one to the other
> queue, and then puts both queues into a free running mode would be a
> much better fit for this mechanism.
> 
>>> My only disagreement is when this should be implemented. I think it is
>>> fine to keep my custom implementation of this in the driver for now. Once
>>> an extension of vb2 is ready to support this feature, it would be fairly
>>> straightforward to strip out my custom implementation and go with the
>>> new API.
>>
>> For a staging driver this isn't necessary, as long as it is documented in
>> the TODO file that this needs to be fixed before it can be moved out of
>> staging. The whole point of staging is that there is still work to be
>> done in the driver, after all :-)
> 
> Absolutely. The reason I am arguing against merging the mem2mem media
> control links so vehemently is that I am convinced the userspace
> interface is wrong, and I am afraid that even though in staging, it
> might become established.

As long as it is mentioned in the TODO, and ideally in the Kconfig as well,
then I'm fine with it.

The big advantage of being in the kernel is that it is much easier to start
providing fixes, improvements, etc. If you use a staging driver you know
that there is no guarantee whatsoever with respect to stable ABI/APIs.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-23 11:08             ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-01-23 11:08 UTC (permalink / raw)
  To: linux-arm-kernel

On 01/23/2017 12:00 PM, Philipp Zabel wrote:
> On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
> [...]
>>> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
>>> hardware filters for motion compensation. Some of the motion compensation
>>> modes ("low" and "medium" motion) require that the VDIC receive video
>>> frame fields from memory buffers (dedicated dma channels in the
>>> IPU are used to transfer those buffers into the VDIC).
>>>
>>> So one option to support those modes would be to pass the raw buffers
>>> from a camera sensor up to userspace to a capture device, and then pass
>>> them back to the VDIC for de-interlacing using a mem2mem device.
>>>
>>> Philipp and I are both in agreement that, since userland is not interested
>>> in the intermediate interlaced buffers in this case, but only the final
>>> result (motion compensated, de-interlaced frames), it is more efficient
>>> to provide a media link that allows passing those intermediate frames
>>> directly from a camera source pad to VDIC sink pad, without having
>>> to route them through userspace.
>>>
>>> So in order to support that, I've implemented a simple FIFO dma buffer
>>> queue in the driver to allow passing video buffers directly from a source
>>> to a sink. It is modeled loosely off the vb2 state machine and API, but
>>> simpler (for instance it only allows contiguous, cache-coherent buffers).
>>>
>>> This is where Philipp has an argument, that this should be done with a
>>> new API in videobuf2.
> 
> That is one part of the argument. I'm glad to understand now that we
> agree about this.
> 
>>> And I'm actually in total agreement with that. I definitely agree that there
>>> should be a mechanism in the media framework that allows passing video
>>> buffers from a source pad to a sink pad using a software queue, with no
>>> involvement from userland.
> 
> That is the other part of the argument. I do not agree that these
> software queue "links" should be presented to userspace as media pad
> links between two entities of a media device. 
> 
> First, that would limit the links to subdevices contained in the same
> media graph, while this should work between any two capture and output
> queues of different devices.
> Assume for example, we want to encode the captured, deinterlaced video
> to h.264 with the coda VPU driver. A software queue link could be
> established between the CSI capture and the VDIC deinterlacer input,
> just as between the VDIC deinterlacer output and the coda VPU input.
> Technically, there would be no difference between those two linked
> capture/output queue pairs. But the coda driver is a completely separate
> mem2mem device. And since it is not part of the i.MX media graph, there
> is no entity pad to link to.
> Or assume there is an USB analog capture device that produces interlaced
> frames. I think it should be possible to connect its capture queue to
> the VDIC deinterlacer output queue just the same way as linking the CSI
> to the VDIC (in software queue mode).
> 
> Second, the subdevice pad formats describe wire formats, not memory
> formats. The user might want to choose between 4:2:2 and 4:2:0
> subsampled YUV formats for the intermediate buffer, for example,
> depending on memory bandwidth constraints and quality requirements. This
> is impossible with the media entity / subdevice pad links.
> 
> I think an interface where userspace configures the capture and output
> queues via v4l2 API, passes dma buffers around from one to the other
> queue, and then puts both queues into a free running mode would be a
> much better fit for this mechanism.
> 
>>> My only disagreement is when this should be implemented. I think it is
>>> fine to keep my custom implementation of this in the driver for now. Once
>>> an extension of vb2 is ready to support this feature, it would be fairly
>>> straightforward to strip out my custom implementation and go with the
>>> new API.
>>
>> For a staging driver this isn't necessary, as long as it is documented in
>> the TODO file that this needs to be fixed before it can be moved out of
>> staging. The whole point of staging is that there is still work to be
>> done in the driver, after all :-)
> 
> Absolutely. The reason I am arguing against merging the mem2mem media
> control links so vehemently is that I am convinced the userspace
> interface is wrong, and I am afraid that even though in staging, it
> might become established.

As long as it is mentioned in the TODO, and ideally in the Kconfig as well,
then I'm fine with it.

The big advantage of being in the kernel is that it is much easier to start
providing fixes, improvements, etc. If you use a staging driver you know
that there is no guarantee whatsoever with respect to stable ABI/APIs.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-23  2:31           ` Steve Longerbeam
  (?)
@ 2017-01-23 11:13             ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-23 11:13 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
> 
> On 01/16/2017 05:47 AM, Philipp Zabel wrote:
> > On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
> > [...]
> >>>> +Unprocessed Video Capture:
> >>>> +--------------------------
> >>>> +
> >>>> +Send frames directly from sensor to camera interface, with no
> >>>> +conversions:
> >>>> +
> >>>> +-> ipu_smfc -> camif
> >>> I'd call this capture interface, this is not just for cameras. Or maybe
> >>> idmac if you want to mirror hardware names?
> >> Camif is so named because it is the V4L2 user interface for video
> >> capture. I suppose it could be named "capif", but that doesn't role
> >> off the tongue quite as well.
> > Agreed, capif sounds weird. I find camif a bit confusing though, because
> > Samsung S3C has a camera interface that is actually called "CAMIF".
> 
> how about simply "capture" ?

That sounds good to me.

[...]
> > Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
> > IDMAC chapter states that all internal submodules only work on 8-bit per
> > component formats with four components: YUVA or RGBA.
> 
> Right, the "direct" IPU internal (that do not transfer buffers to/from
> memory via IDMAC channels) should only allow the IPU internal
> formats YUVA and RGBA. I get you now.
> 
> The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and
> MEDIA_BUS_FMT_ARGB8888_1X32.
> 
> Those pads are:
> 
> ipu_csi:1
> ipu_vdic:1
> ipu_ic_prp:0
> ipu_ic_prp:1
> ipu_ic_prpenc:0
> ipu_ic_prpenc:1
> ipu_ic_prpvf:0
> ipu_ic_prpvf:1

Yes, that is what I meant. The csi:1 can then be extended to support
additional expanded/packed/raw formats for the SMFC->memory path.

> >> There does not appear to be support in the FSU for linking a write channel
> >> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
> >> is support for the direct link from CSI (which I am using), but that's
> >> not an
> >> IDMAC channel link.
> >>
> >> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
> >> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
> > As I read it, that is 0 and 2 only, no idea why. But since there are
> > only 2 CSIs, that shouldn't be a problem.
> 
> ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still
> at channel 0).

Ok.

> >>>> +static const u32 power_off_seq[] = {
> >>>> +	IMX_MEDIA_GRP_ID_IC_PP,
> >>>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> >>>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> >>>> +	IMX_MEDIA_GRP_ID_SMFC,
> >>>> +	IMX_MEDIA_GRP_ID_CSI,
> >>>> +	IMX_MEDIA_GRP_ID_VIDMUX,
> >>>> +	IMX_MEDIA_GRP_ID_SENSOR,
> >>>> +	IMX_MEDIA_GRP_ID_CSI2,
> >>>> +};
> >>> This seems somewhat arbitrary. Why is a power sequence needed?
> >> The CSI-2 receiver must be powered up before the sensor, that's the
> >> only requirement IIRC. The others have no s_power requirement. So I
> >> can probably change this to power up in the frontend -> backend order
> >> (IC_PP to sensor). And vice-versa for power off.
> > Yes, I think that should work (see below).
> 
> Actually there are problems using this, see below.
[...]
> >>>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> >>> This should really be handled by v4l2_pipeline_pm_use.
> >> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
> >> doing some other stuff that bothered me, at least that's what I remember.
> >> I will revisit this.
> > I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
> > any problems. It would be better to reuse and, if necessary, fix the
> > existing infrastructure where available.
> 
> I tried this API, by switching to v4l2_pipeline_pm_use() in camif 
> open/release,
> and switched to v4l2_pipeline_link_notify() instead of 
> imx_media_link_notify()
> in the media driver's media_device_ops.
> 
> This API assumes the video device has an open file handle while the media
> links are being established. This doesn't work for me, I want to be able to
> establish the links using 'media-ctl -l', and that won't work unless 
> there is an
> open file handle on the video capture device node.
> 
> Also, I looked into calling v4l2_pipeline_pm_use() during 
> imx_media_link_notify(),
> instead of imx_media_pipeline_set_power(). Again there are problems with 
> that.
> 
> First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be 
> called inside
> link_notify which already acquires that lock. The header for this 
> function also
> clearly states it should only be called in open/release.

So why not call it in open/release then?

> Second, ignoring the above locking issue for a moment, 
> v4l2_pipeline_pm_use()
> will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
> when executing
> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
> wrong order.
> In my version which enforces the correct power on order, the mipi csi-2 
> s_power
> is called first in that link setup, followed by the sensor.

I don't understand why you want to power up subdevs as soon as the links
are established. Shouldn't that rather be done for all subdevices in the
pipeline when the corresponding capture device is opened?
It seems to me that powering up the pipeline should be the last step
before userspace actually starts the capture.

[...]
> >>>> +	/* there must be at least one CSI in first IPU */
> >>> Why?
> >> Well yeah, imx-media doesn't necessarily need a CSI if things
> >> like the VDIC or post-processor are being used by an output
> >> overlay pipeline, for example. I'll fix this.
> > I haven't even thought that far, but there could be boards with only a
> > parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
> > saving reasons.
> 
> done! A very simple change to imx_media_add_internal_subdevs(),
> and now no CSI's are necessary, but the remaining IPU internal entities
> are loaded and linked just fine without them, so for example with no
> CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and
> could still be used in the future by a mem2mem device for instance.

Excellent, thanks.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-23 11:13             ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-23 11:13 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
> 
> On 01/16/2017 05:47 AM, Philipp Zabel wrote:
> > On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
> > [...]
> >>>> +Unprocessed Video Capture:
> >>>> +--------------------------
> >>>> +
> >>>> +Send frames directly from sensor to camera interface, with no
> >>>> +conversions:
> >>>> +
> >>>> +-> ipu_smfc -> camif
> >>> I'd call this capture interface, this is not just for cameras. Or maybe
> >>> idmac if you want to mirror hardware names?
> >> Camif is so named because it is the V4L2 user interface for video
> >> capture. I suppose it could be named "capif", but that doesn't role
> >> off the tongue quite as well.
> > Agreed, capif sounds weird. I find camif a bit confusing though, because
> > Samsung S3C has a camera interface that is actually called "CAMIF".
> 
> how about simply "capture" ?

That sounds good to me.

[...]
> > Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
> > IDMAC chapter states that all internal submodules only work on 8-bit per
> > component formats with four components: YUVA or RGBA.
> 
> Right, the "direct" IPU internal (that do not transfer buffers to/from
> memory via IDMAC channels) should only allow the IPU internal
> formats YUVA and RGBA. I get you now.
> 
> The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and
> MEDIA_BUS_FMT_ARGB8888_1X32.
> 
> Those pads are:
> 
> ipu_csi:1
> ipu_vdic:1
> ipu_ic_prp:0
> ipu_ic_prp:1
> ipu_ic_prpenc:0
> ipu_ic_prpenc:1
> ipu_ic_prpvf:0
> ipu_ic_prpvf:1

Yes, that is what I meant. The csi:1 can then be extended to support
additional expanded/packed/raw formats for the SMFC->memory path.

> >> There does not appear to be support in the FSU for linking a write channel
> >> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
> >> is support for the direct link from CSI (which I am using), but that's
> >> not an
> >> IDMAC channel link.
> >>
> >> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
> >> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
> > As I read it, that is 0 and 2 only, no idea why. But since there are
> > only 2 CSIs, that shouldn't be a problem.
> 
> ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still
> at channel 0).

Ok.

> >>>> +static const u32 power_off_seq[] = {
> >>>> +	IMX_MEDIA_GRP_ID_IC_PP,
> >>>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> >>>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> >>>> +	IMX_MEDIA_GRP_ID_SMFC,
> >>>> +	IMX_MEDIA_GRP_ID_CSI,
> >>>> +	IMX_MEDIA_GRP_ID_VIDMUX,
> >>>> +	IMX_MEDIA_GRP_ID_SENSOR,
> >>>> +	IMX_MEDIA_GRP_ID_CSI2,
> >>>> +};
> >>> This seems somewhat arbitrary. Why is a power sequence needed?
> >> The CSI-2 receiver must be powered up before the sensor, that's the
> >> only requirement IIRC. The others have no s_power requirement. So I
> >> can probably change this to power up in the frontend -> backend order
> >> (IC_PP to sensor). And vice-versa for power off.
> > Yes, I think that should work (see below).
> 
> Actually there are problems using this, see below.
[...]
> >>>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> >>> This should really be handled by v4l2_pipeline_pm_use.
> >> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
> >> doing some other stuff that bothered me, at least that's what I remember.
> >> I will revisit this.
> > I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
> > any problems. It would be better to reuse and, if necessary, fix the
> > existing infrastructure where available.
> 
> I tried this API, by switching to v4l2_pipeline_pm_use() in camif 
> open/release,
> and switched to v4l2_pipeline_link_notify() instead of 
> imx_media_link_notify()
> in the media driver's media_device_ops.
> 
> This API assumes the video device has an open file handle while the media
> links are being established. This doesn't work for me, I want to be able to
> establish the links using 'media-ctl -l', and that won't work unless 
> there is an
> open file handle on the video capture device node.
> 
> Also, I looked into calling v4l2_pipeline_pm_use() during 
> imx_media_link_notify(),
> instead of imx_media_pipeline_set_power(). Again there are problems with 
> that.
> 
> First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be 
> called inside
> link_notify which already acquires that lock. The header for this 
> function also
> clearly states it should only be called in open/release.

So why not call it in open/release then?

> Second, ignoring the above locking issue for a moment, 
> v4l2_pipeline_pm_use()
> will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
> when executing
> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
> wrong order.
> In my version which enforces the correct power on order, the mipi csi-2 
> s_power
> is called first in that link setup, followed by the sensor.

I don't understand why you want to power up subdevs as soon as the links
are established. Shouldn't that rather be done for all subdevices in the
pipeline when the corresponding capture device is opened?
It seems to me that powering up the pipeline should be the last step
before userspace actually starts the capture.

[...]
> >>>> +	/* there must be at least one CSI in first IPU */
> >>> Why?
> >> Well yeah, imx-media doesn't necessarily need a CSI if things
> >> like the VDIC or post-processor are being used by an output
> >> overlay pipeline, for example. I'll fix this.
> > I haven't even thought that far, but there could be boards with only a
> > parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
> > saving reasons.
> 
> done! A very simple change to imx_media_add_internal_subdevs(),
> and now no CSI's are necessary, but the remaining IPU internal entities
> are loaded and linked just fine without them, so for example with no
> CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and
> could still be used in the future by a mem2mem device for instance.

Excellent, thanks.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-23 11:13             ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-23 11:13 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
> 
> On 01/16/2017 05:47 AM, Philipp Zabel wrote:
> > On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
> > [...]
> >>>> +Unprocessed Video Capture:
> >>>> +--------------------------
> >>>> +
> >>>> +Send frames directly from sensor to camera interface, with no
> >>>> +conversions:
> >>>> +
> >>>> +-> ipu_smfc -> camif
> >>> I'd call this capture interface, this is not just for cameras. Or maybe
> >>> idmac if you want to mirror hardware names?
> >> Camif is so named because it is the V4L2 user interface for video
> >> capture. I suppose it could be named "capif", but that doesn't role
> >> off the tongue quite as well.
> > Agreed, capif sounds weird. I find camif a bit confusing though, because
> > Samsung S3C has a camera interface that is actually called "CAMIF".
> 
> how about simply "capture" ?

That sounds good to me.

[...]
> > Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
> > IDMAC chapter states that all internal submodules only work on 8-bit per
> > component formats with four components: YUVA or RGBA.
> 
> Right, the "direct" IPU internal (that do not transfer buffers to/from
> memory via IDMAC channels) should only allow the IPU internal
> formats YUVA and RGBA. I get you now.
> 
> The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and
> MEDIA_BUS_FMT_ARGB8888_1X32.
> 
> Those pads are:
> 
> ipu_csi:1
> ipu_vdic:1
> ipu_ic_prp:0
> ipu_ic_prp:1
> ipu_ic_prpenc:0
> ipu_ic_prpenc:1
> ipu_ic_prpvf:0
> ipu_ic_prpvf:1

Yes, that is what I meant. The csi:1 can then be extended to support
additional expanded/packed/raw formats for the SMFC->memory path.

> >> There does not appear to be support in the FSU for linking a write channel
> >> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
> >> is support for the direct link from CSI (which I am using), but that's
> >> not an
> >> IDMAC channel link.
> >>
> >> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
> >> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
> > As I read it, that is 0 and 2 only, no idea why. But since there are
> > only 2 CSIs, that shouldn't be a problem.
> 
> ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still
> at channel 0).

Ok.

> >>>> +static const u32 power_off_seq[] = {
> >>>> +	IMX_MEDIA_GRP_ID_IC_PP,
> >>>> +	IMX_MEDIA_GRP_ID_IC_PRPVF,
> >>>> +	IMX_MEDIA_GRP_ID_IC_PRPENC,
> >>>> +	IMX_MEDIA_GRP_ID_SMFC,
> >>>> +	IMX_MEDIA_GRP_ID_CSI,
> >>>> +	IMX_MEDIA_GRP_ID_VIDMUX,
> >>>> +	IMX_MEDIA_GRP_ID_SENSOR,
> >>>> +	IMX_MEDIA_GRP_ID_CSI2,
> >>>> +};
> >>> This seems somewhat arbitrary. Why is a power sequence needed?
> >> The CSI-2 receiver must be powered up before the sensor, that's the
> >> only requirement IIRC. The others have no s_power requirement. So I
> >> can probably change this to power up in the frontend -> backend order
> >> (IC_PP to sensor). And vice-versa for power off.
> > Yes, I think that should work (see below).
> 
> Actually there are problems using this, see below.
[...]
> >>>> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);
> >>> This should really be handled by v4l2_pipeline_pm_use.
> >> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
> >> doing some other stuff that bothered me, at least that's what I remember.
> >> I will revisit this.
> > I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
> > any problems. It would be better to reuse and, if necessary, fix the
> > existing infrastructure where available.
> 
> I tried this API, by switching to v4l2_pipeline_pm_use() in camif 
> open/release,
> and switched to v4l2_pipeline_link_notify() instead of 
> imx_media_link_notify()
> in the media driver's media_device_ops.
> 
> This API assumes the video device has an open file handle while the media
> links are being established. This doesn't work for me, I want to be able to
> establish the links using 'media-ctl -l', and that won't work unless 
> there is an
> open file handle on the video capture device node.
> 
> Also, I looked into calling v4l2_pipeline_pm_use() during 
> imx_media_link_notify(),
> instead of imx_media_pipeline_set_power(). Again there are problems with 
> that.
> 
> First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be 
> called inside
> link_notify which already acquires that lock. The header for this 
> function also
> clearly states it should only be called in open/release.

So why not call it in open/release then?

> Second, ignoring the above locking issue for a moment, 
> v4l2_pipeline_pm_use()
> will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
> when executing
> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
> wrong order.
> In my version which enforces the correct power on order, the mipi csi-2 
> s_power
> is called first in that link setup, followed by the sensor.

I don't understand why you want to power up subdevs as soon as the links
are established. Shouldn't that rather be done for all subdevices in the
pipeline when the corresponding capture device is opened?
It seems to me that powering up the pipeline should be the last step
before userspace actually starts the capture.

[...]
> >>>> +	/* there must be at least one CSI in first IPU */
> >>> Why?
> >> Well yeah, imx-media doesn't necessarily need a CSI if things
> >> like the VDIC or post-processor are being used by an output
> >> overlay pipeline, for example. I'll fix this.
> > I haven't even thought that far, but there could be boards with only a
> > parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
> > saving reasons.
> 
> done! A very simple change to imx_media_add_internal_subdevs(),
> and now no CSI's are necessary, but the remaining IPU internal entities
> are loaded and linked just fine without them, so for example with no
> CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and
> could still be used in the future by a mem2mem device for instance.

Excellent, thanks.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-23 11:00           ` Philipp Zabel
@ 2017-01-23 23:08             ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-23 23:08 UTC (permalink / raw)
  To: Philipp Zabel, Hans Verkuil
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, nick, markus.heiser, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam



On 01/23/2017 03:00 AM, Philipp Zabel wrote:
> On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
> [...]
>>> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
>>> hardware filters for motion compensation. Some of the motion compensation
>>> modes ("low" and "medium" motion) require that the VDIC receive video
>>> frame fields from memory buffers (dedicated dma channels in the
>>> IPU are used to transfer those buffers into the VDIC).
>>>
>>> So one option to support those modes would be to pass the raw buffers
>>> from a camera sensor up to userspace to a capture device, and then pass
>>> them back to the VDIC for de-interlacing using a mem2mem device.
>>>
>>> Philipp and I are both in agreement that, since userland is not interested
>>> in the intermediate interlaced buffers in this case, but only the final
>>> result (motion compensated, de-interlaced frames), it is more efficient
>>> to provide a media link that allows passing those intermediate frames
>>> directly from a camera source pad to VDIC sink pad, without having
>>> to route them through userspace.
>>>
>>> So in order to support that, I've implemented a simple FIFO dma buffer
>>> queue in the driver to allow passing video buffers directly from a source
>>> to a sink. It is modeled loosely off the vb2 state machine and API, but
>>> simpler (for instance it only allows contiguous, cache-coherent buffers).
>>>
>>> This is where Philipp has an argument, that this should be done with a
>>> new API in videobuf2.
> That is one part of the argument. I'm glad to understand now that we
> agree about this.
>
>>> And I'm actually in total agreement with that. I definitely agree that there
>>> should be a mechanism in the media framework that allows passing video
>>> buffers from a source pad to a sink pad using a software queue, with no
>>> involvement from userland.
> That is the other part of the argument. I do not agree that these
> software queue "links" should be presented to userspace as media pad
> links between two entities of a media device.
> First, that would limit the links to subdevices contained in the same
> media graph, while this should work between any two capture and output
> queues of different devices.

It sounds like we are talking about two different new proposed features.

My proposal is to implement a software buffer queue between pads.
Beyond enabling the link between pads using the existing media controller
API, userspace is not involved after that. The fact that this link is 
accomplished
with a software buffer queue is not known, and doesn't need to be known,
by userspace.

Your proposal, if I have it right, is to allow linking two v4l2 device 
vb2 queues
(i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
mode such that v4l2 buffers get passed from one device's vb2 queue to the
other without requiring the v4l2 user program to actively forward those 
buffers.

There isn't anything that would preclude one from the other, they can
both exist. But they are different ideas. One implements software queues
at the _pad level_ and is opaque to userspace, the other links queues
at the _device level_ using a new user API, but once the link is 
established,
also does not require any involvement from userspace.

What I'm saying is we can do _both_.


> Assume for example, we want to encode the captured, deinterlaced video
> to h.264 with the coda VPU driver. A software queue link could be
> established between the CSI capture and the VDIC deinterlacer input,

That's already available in the media graph. By linking CSI and
VDIC entities. The capture device will then already be providing
de-interlaced video, and ...

> just as between the VDIC deinterlacer output and the coda VPU input.
> Technically, there would be no difference between those two linked
> capture/output queue pairs. But the coda driver is a completely separate
> mem2mem device. And since it is not part of the i.MX media graph, there
> is no entity pad to link to.

your free-run queue linking could then be used to link the (already)
de-interlaced stream to the coda device for h.264 encode.

The other idea would be to eventually make the coda device part of
the media graph as an entity. Then this link would instead be via pads.

> Or assume there is an USB analog capture device that produces interlaced
> frames. I think it should be possible to connect its capture queue to
> the VDIC deinterlacer output queue just the same way as linking the CSI
> to the VDIC (in software queue mode).

Right, for devices that are outside the i.MX media graph, such as a USB
capture device (or coda), access to the i.MX entities such as the VDIC would
require an i.MX mem2mem device with media links to the VDIC. The USB
capture device would forward its captured frames to mem2mem (maybe
using your free-run vb2 queue linking idea):

usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device



> Second, the subdevice pad formats describe wire formats, not memory
> formats. The user might want to choose between 4:2:2 and 4:2:0
> subsampled YUV formats for the intermediate buffer, for example,
> depending on memory bandwidth constraints and quality requirements. This
> is impossible with the media entity / subdevice pad links.

It's true that there are currently no defined planar media bus
pixel formats. We just need to add new definitions for them. Once
that is done, the media driver will support planar YUV formats
simply by adding the new codes to imx_media_formats[].

Perhaps this gets to the root of the issue.

Is the media bus concept an abstract one, or is the media bus
intended to represent actual physical buses (as the lack of planar
media bus formats would imply)?

Can we break with the physical-bus-only idea if that is the case, and
loosen the definition of a media bus to mean the passage of media
data from one pad to another by whatever means?

In my view the idea of a physical bus at the sensor makes sense, but
beyond that, keeping that restriction limits how data can pass between
pads.

Hans, any input here?

If this is really anathema, then I'm willing to remove the software queues
between pads, but it will be giving up some functionality in the media 
driver.
It would also mean splitting the VDIC in two. The VDIC entity would be 
limited
to only one motion compensation mode, and the full functionality would have
to be added somewhere else. Currently all functionality of the VDIC is
implemented in a single media entity.

> I think an interface where userspace configures the capture and output
> queues via v4l2 API, passes dma buffers around from one to the other
> queue, and then puts both queues into a free running mode would be a
> much better fit for this mechanism.

As I said, I see these as two different ideas that can both be
implemented.

>>> My only disagreement is when this should be implemented. I think it is
>>> fine to keep my custom implementation of this in the driver for now. Once
>>> an extension of vb2 is ready to support this feature, it would be fairly
>>> straightforward to strip out my custom implementation and go with the
>>> new API.
>> For a staging driver this isn't necessary, as long as it is documented in
>> the TODO file that this needs to be fixed before it can be moved out of
>> staging. The whole point of staging is that there is still work to be
>> done in the driver, after all :-)
> Absolutely. The reason I am arguing against merging the mem2mem media
> control links so vehemently is that I am convinced the userspace
> interface is wrong, and I am afraid that even though in staging, it
> might become established.

I don't believe there is anything wrong with the userspace interface,
In fact it hasn't even changed. The fact two pads are passing memory
buffers is "under the hood".

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-23 23:08             ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-23 23:08 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/23/2017 03:00 AM, Philipp Zabel wrote:
> On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
> [...]
>>> There is a VDIC entity in the i.MX IPU that performs de-interlacing with
>>> hardware filters for motion compensation. Some of the motion compensation
>>> modes ("low" and "medium" motion) require that the VDIC receive video
>>> frame fields from memory buffers (dedicated dma channels in the
>>> IPU are used to transfer those buffers into the VDIC).
>>>
>>> So one option to support those modes would be to pass the raw buffers
>>> from a camera sensor up to userspace to a capture device, and then pass
>>> them back to the VDIC for de-interlacing using a mem2mem device.
>>>
>>> Philipp and I are both in agreement that, since userland is not interested
>>> in the intermediate interlaced buffers in this case, but only the final
>>> result (motion compensated, de-interlaced frames), it is more efficient
>>> to provide a media link that allows passing those intermediate frames
>>> directly from a camera source pad to VDIC sink pad, without having
>>> to route them through userspace.
>>>
>>> So in order to support that, I've implemented a simple FIFO dma buffer
>>> queue in the driver to allow passing video buffers directly from a source
>>> to a sink. It is modeled loosely off the vb2 state machine and API, but
>>> simpler (for instance it only allows contiguous, cache-coherent buffers).
>>>
>>> This is where Philipp has an argument, that this should be done with a
>>> new API in videobuf2.
> That is one part of the argument. I'm glad to understand now that we
> agree about this.
>
>>> And I'm actually in total agreement with that. I definitely agree that there
>>> should be a mechanism in the media framework that allows passing video
>>> buffers from a source pad to a sink pad using a software queue, with no
>>> involvement from userland.
> That is the other part of the argument. I do not agree that these
> software queue "links" should be presented to userspace as media pad
> links between two entities of a media device.
> First, that would limit the links to subdevices contained in the same
> media graph, while this should work between any two capture and output
> queues of different devices.

It sounds like we are talking about two different new proposed features.

My proposal is to implement a software buffer queue between pads.
Beyond enabling the link between pads using the existing media controller
API, userspace is not involved after that. The fact that this link is 
accomplished
with a software buffer queue is not known, and doesn't need to be known,
by userspace.

Your proposal, if I have it right, is to allow linking two v4l2 device 
vb2 queues
(i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
mode such that v4l2 buffers get passed from one device's vb2 queue to the
other without requiring the v4l2 user program to actively forward those 
buffers.

There isn't anything that would preclude one from the other, they can
both exist. But they are different ideas. One implements software queues
at the _pad level_ and is opaque to userspace, the other links queues
at the _device level_ using a new user API, but once the link is 
established,
also does not require any involvement from userspace.

What I'm saying is we can do _both_.


> Assume for example, we want to encode the captured, deinterlaced video
> to h.264 with the coda VPU driver. A software queue link could be
> established between the CSI capture and the VDIC deinterlacer input,

That's already available in the media graph. By linking CSI and
VDIC entities. The capture device will then already be providing
de-interlaced video, and ...

> just as between the VDIC deinterlacer output and the coda VPU input.
> Technically, there would be no difference between those two linked
> capture/output queue pairs. But the coda driver is a completely separate
> mem2mem device. And since it is not part of the i.MX media graph, there
> is no entity pad to link to.

your free-run queue linking could then be used to link the (already)
de-interlaced stream to the coda device for h.264 encode.

The other idea would be to eventually make the coda device part of
the media graph as an entity. Then this link would instead be via pads.

> Or assume there is an USB analog capture device that produces interlaced
> frames. I think it should be possible to connect its capture queue to
> the VDIC deinterlacer output queue just the same way as linking the CSI
> to the VDIC (in software queue mode).

Right, for devices that are outside the i.MX media graph, such as a USB
capture device (or coda), access to the i.MX entities such as the VDIC would
require an i.MX mem2mem device with media links to the VDIC. The USB
capture device would forward its captured frames to mem2mem (maybe
using your free-run vb2 queue linking idea):

usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device



> Second, the subdevice pad formats describe wire formats, not memory
> formats. The user might want to choose between 4:2:2 and 4:2:0
> subsampled YUV formats for the intermediate buffer, for example,
> depending on memory bandwidth constraints and quality requirements. This
> is impossible with the media entity / subdevice pad links.

It's true that there are currently no defined planar media bus
pixel formats. We just need to add new definitions for them. Once
that is done, the media driver will support planar YUV formats
simply by adding the new codes to imx_media_formats[].

Perhaps this gets to the root of the issue.

Is the media bus concept an abstract one, or is the media bus
intended to represent actual physical buses (as the lack of planar
media bus formats would imply)?

Can we break with the physical-bus-only idea if that is the case, and
loosen the definition of a media bus to mean the passage of media
data from one pad to another by whatever means?

In my view the idea of a physical bus at the sensor makes sense, but
beyond that, keeping that restriction limits how data can pass between
pads.

Hans, any input here?

If this is really anathema, then I'm willing to remove the software queues
between pads, but it will be giving up some functionality in the media 
driver.
It would also mean splitting the VDIC in two. The VDIC entity would be 
limited
to only one motion compensation mode, and the full functionality would have
to be added somewhere else. Currently all functionality of the VDIC is
implemented in a single media entity.

> I think an interface where userspace configures the capture and output
> queues via v4l2 API, passes dma buffers around from one to the other
> queue, and then puts both queues into a free running mode would be a
> much better fit for this mechanism.

As I said, I see these as two different ideas that can both be
implemented.

>>> My only disagreement is when this should be implemented. I think it is
>>> fine to keep my custom implementation of this in the driver for now. Once
>>> an extension of vb2 is ready to support this feature, it would be fairly
>>> straightforward to strip out my custom implementation and go with the
>>> new API.
>> For a staging driver this isn't necessary, as long as it is documented in
>> the TODO file that this needs to be fixed before it can be moved out of
>> staging. The whole point of staging is that there is still work to be
>> done in the driver, after all :-)
> Absolutely. The reason I am arguing against merging the mem2mem media
> control links so vehemently is that I am convinced the userspace
> interface is wrong, and I am afraid that even though in staging, it
> might become established.

I don't believe there is anything wrong with the userspace interface,
In fact it hasn't even changed. The fact two pads are passing memory
buffers is "under the hood".

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24  1:38               ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  1:38 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/23/2017 03:13 AM, Philipp Zabel wrote:
> Hi Steve,
>
> On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
>> On 01/16/2017 05:47 AM, Philipp Zabel wrote:
>>> On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
>>> [...]
>>>>>> +Unprocessed Video Capture:
>>>>>> +--------------------------
>>>>>> +
>>>>>> +Send frames directly from sensor to camera interface, with no
>>>>>> +conversions:
>>>>>> +
>>>>>> +-> ipu_smfc -> camif
>>>>> I'd call this capture interface, this is not just for cameras. Or maybe
>>>>> idmac if you want to mirror hardware names?
>>>> Camif is so named because it is the V4L2 user interface for video
>>>> capture. I suppose it could be named "capif", but that doesn't role
>>>> off the tongue quite as well.
>>> Agreed, capif sounds weird. I find camif a bit confusing though, because
>>> Samsung S3C has a camera interface that is actually called "CAMIF".
>> how about simply "capture" ?
> That sounds good to me.

done.

>
>>>>> This should really be handled by v4l2_pipeline_pm_use.
>>>> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
>>>> doing some other stuff that bothered me, at least that's what I remember.
>>>> I will revisit this.
>>> I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
>>> any problems. It would be better to reuse and, if necessary, fix the
>>> existing infrastructure where available.
>> I tried this API, by switching to v4l2_pipeline_pm_use() in camif
>> open/release,
>> and switched to v4l2_pipeline_link_notify() instead of
>> imx_media_link_notify()
>> in the media driver's media_device_ops.
>>
>> This API assumes the video device has an open file handle while the media
>> links are being established. This doesn't work for me, I want to be able to
>> establish the links using 'media-ctl -l', and that won't work unless
>> there is an
>> open file handle on the video capture device node.
>>
>> Also, I looked into calling v4l2_pipeline_pm_use() during
>> imx_media_link_notify(),
>> instead of imx_media_pipeline_set_power(). Again there are problems with
>> that.
>>
>> First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be
>> called inside
>> link_notify which already acquires that lock. The header for this
>> function also
>> clearly states it should only be called in open/release.
> So why not call it in open/release then?

er, see above (?)

>
>> Second, ignoring the above locking issue for a moment,
>> v4l2_pipeline_pm_use()
>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
>> when executing
>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
>> wrong order.
>> In my version which enforces the correct power on order, the mipi csi-2
>> s_power
>> is called first in that link setup, followed by the sensor.
> I don't understand why you want to power up subdevs as soon as the links
> are established.

Because that is the precedence, all other media drivers do pipeline
power on/off at link_notify. And v4l2_pipeline_link_notify() was written
as a link_notify method.

>   Shouldn't that rather be done for all subdevices in the
> pipeline when the corresponding capture device is opened?

that won't work. There's no guarantee the links will be established
at capture device open time.

> It seems to me that powering up the pipeline should be the last step
> before userspace actually starts the capture.

Well, I'm ok with moving pipeline power on/off to start/stop streaming.
I would actually prefer to do it then, I only chose at link_notify because
of precedence. I'll look into it.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24  1:38               ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  1:38 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/23/2017 03:13 AM, Philipp Zabel wrote:
> Hi Steve,
>
> On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
>> On 01/16/2017 05:47 AM, Philipp Zabel wrote:
>>> On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
>>> [...]
>>>>>> +Unprocessed Video Capture:
>>>>>> +--------------------------
>>>>>> +
>>>>>> +Send frames directly from sensor to camera interface, with no
>>>>>> +conversions:
>>>>>> +
>>>>>> +-> ipu_smfc -> camif
>>>>> I'd call this capture interface, this is not just for cameras. Or maybe
>>>>> idmac if you want to mirror hardware names?
>>>> Camif is so named because it is the V4L2 user interface for video
>>>> capture. I suppose it could be named "capif", but that doesn't role
>>>> off the tongue quite as well.
>>> Agreed, capif sounds weird. I find camif a bit confusing though, because
>>> Samsung S3C has a camera interface that is actually called "CAMIF".
>> how about simply "capture" ?
> That sounds good to me.

done.

>
>>>>> This should really be handled by v4l2_pipeline_pm_use.
>>>> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
>>>> doing some other stuff that bothered me, at least that's what I remember.
>>>> I will revisit this.
>>> I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
>>> any problems. It would be better to reuse and, if necessary, fix the
>>> existing infrastructure where available.
>> I tried this API, by switching to v4l2_pipeline_pm_use() in camif
>> open/release,
>> and switched to v4l2_pipeline_link_notify() instead of
>> imx_media_link_notify()
>> in the media driver's media_device_ops.
>>
>> This API assumes the video device has an open file handle while the media
>> links are being established. This doesn't work for me, I want to be able to
>> establish the links using 'media-ctl -l', and that won't work unless
>> there is an
>> open file handle on the video capture device node.
>>
>> Also, I looked into calling v4l2_pipeline_pm_use() during
>> imx_media_link_notify(),
>> instead of imx_media_pipeline_set_power(). Again there are problems with
>> that.
>>
>> First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be
>> called inside
>> link_notify which already acquires that lock. The header for this
>> function also
>> clearly states it should only be called in open/release.
> So why not call it in open/release then?

er, see above (?)

>
>> Second, ignoring the above locking issue for a moment,
>> v4l2_pipeline_pm_use()
>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
>> when executing
>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
>> wrong order.
>> In my version which enforces the correct power on order, the mipi csi-2
>> s_power
>> is called first in that link setup, followed by the sensor.
> I don't understand why you want to power up subdevs as soon as the links
> are established.

Because that is the precedence, all other media drivers do pipeline
power on/off at link_notify. And v4l2_pipeline_link_notify() was written
as a link_notify method.

>   Shouldn't that rather be done for all subdevices in the
> pipeline when the corresponding capture device is opened?

that won't work. There's no guarantee the links will be established
at capture device open time.

> It seems to me that powering up the pipeline should be the last step
> before userspace actually starts the capture.

Well, I'm ok with moving pipeline power on/off to start/stop streaming.
I would actually prefer to do it then, I only chose at link_notify because
of precedence. I'll look into it.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24  1:38               ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  1:38 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/23/2017 03:13 AM, Philipp Zabel wrote:
> Hi Steve,
>
> On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
>> On 01/16/2017 05:47 AM, Philipp Zabel wrote:
>>> On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
>>> [...]
>>>>>> +Unprocessed Video Capture:
>>>>>> +--------------------------
>>>>>> +
>>>>>> +Send frames directly from sensor to camera interface, with no
>>>>>> +conversions:
>>>>>> +
>>>>>> +-> ipu_smfc -> camif
>>>>> I'd call this capture interface, this is not just for cameras. Or maybe
>>>>> idmac if you want to mirror hardware names?
>>>> Camif is so named because it is the V4L2 user interface for video
>>>> capture. I suppose it could be named "capif", but that doesn't role
>>>> off the tongue quite as well.
>>> Agreed, capif sounds weird. I find camif a bit confusing though, because
>>> Samsung S3C has a camera interface that is actually called "CAMIF".
>> how about simply "capture" ?
> That sounds good to me.

done.

>
>>>>> This should really be handled by v4l2_pipeline_pm_use.
>>>> I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
>>>> doing some other stuff that bothered me, at least that's what I remember.
>>>> I will revisit this.
>>> I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
>>> any problems. It would be better to reuse and, if necessary, fix the
>>> existing infrastructure where available.
>> I tried this API, by switching to v4l2_pipeline_pm_use() in camif
>> open/release,
>> and switched to v4l2_pipeline_link_notify() instead of
>> imx_media_link_notify()
>> in the media driver's media_device_ops.
>>
>> This API assumes the video device has an open file handle while the media
>> links are being established. This doesn't work for me, I want to be able to
>> establish the links using 'media-ctl -l', and that won't work unless
>> there is an
>> open file handle on the video capture device node.
>>
>> Also, I looked into calling v4l2_pipeline_pm_use() during
>> imx_media_link_notify(),
>> instead of imx_media_pipeline_set_power(). Again there are problems with
>> that.
>>
>> First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be
>> called inside
>> link_notify which already acquires that lock. The header for this
>> function also
>> clearly states it should only be called in open/release.
> So why not call it in open/release then?

er, see above (?)

>
>> Second, ignoring the above locking issue for a moment,
>> v4l2_pipeline_pm_use()
>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
>> when executing
>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
>> wrong order.
>> In my version which enforces the correct power on order, the mipi csi-2
>> s_power
>> is called first in that link setup, followed by the sensor.
> I don't understand why you want to power up subdevs as soon as the links
> are established.

Because that is the precedence, all other media drivers do pipeline
power on/off at link_notify. And v4l2_pipeline_link_notify() was written
as a link_notify method.

>   Shouldn't that rather be done for all subdevices in the
> pipeline when the corresponding capture device is opened?

that won't work. There's no guarantee the links will be established
at capture device open time.

> It seems to me that powering up the pipeline should be the last step
> before userspace actually starts the capture.

Well, I'm ok with moving pipeline power on/off to start/stop streaming.
I would actually prefer to do it then, I only chose at link_notify because
of precedence. I'll look into it.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-24  1:38               ` Steve Longerbeam
  (?)
@ 2017-01-24  1:45                 ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  1:45 UTC (permalink / raw)
  To: Steve Longerbeam, Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel



On 01/23/2017 05:38 PM, Steve Longerbeam wrote:
>
>>
>>> Second, ignoring the above locking issue for a moment,
>>> v4l2_pipeline_pm_use()
>>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
>>> when executing
>>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
>>> wrong order.
>>> In my version which enforces the correct power on order, the mipi csi-2
>>> s_power
>>> is called first in that link setup, followed by the sensor.
>> I don't understand why you want to power up subdevs as soon as the links
>> are established.
>
> Because that is the precedence, all other media drivers do pipeline
> power on/off at link_notify. And v4l2_pipeline_link_notify() was written
> as a link_notify method.
>
>>   Shouldn't that rather be done for all subdevices in the
>> pipeline when the corresponding capture device is opened?
>
> that won't work. There's no guarantee the links will be established
> at capture device open time.

ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are
no links yet, it would basically be a no-op. And stream on requires 
opening the
device, and the pipeline links should be established by then, so this 
might be
fine, looking into this too.

Steve

>
>> It seems to me that powering up the pipeline should be the last step
>> before userspace actually starts the capture.
>
> Well, I'm ok with moving pipeline power on/off to start/stop streaming.
> I would actually prefer to do it then, I only chose at link_notify 
> because
> of precedence. I'll look into it.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24  1:45                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  1:45 UTC (permalink / raw)
  To: Steve Longerbeam, Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media



On 01/23/2017 05:38 PM, Steve Longerbeam wrote:
>
>>
>>> Second, ignoring the above locking issue for a moment,
>>> v4l2_pipeline_pm_use()
>>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
>>> when executing
>>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
>>> wrong order.
>>> In my version which enforces the correct power on order, the mipi csi-2
>>> s_power
>>> is called first in that link setup, followed by the sensor.
>> I don't understand why you want to power up subdevs as soon as the links
>> are established.
>
> Because that is the precedence, all other media drivers do pipeline
> power on/off at link_notify. And v4l2_pipeline_link_notify() was written
> as a link_notify method.
>
>>   Shouldn't that rather be done for all subdevices in the
>> pipeline when the corresponding capture device is opened?
>
> that won't work. There's no guarantee the links will be established
> at capture device open time.

ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are
no links yet, it would basically be a no-op. And stream on requires 
opening the
device, and the pipeline links should be established by then, so this 
might be
fine, looking into this too.

Steve

>
>> It seems to me that powering up the pipeline should be the last step
>> before userspace actually starts the capture.
>
> Well, I'm ok with moving pipeline power on/off to start/stop streaming.
> I would actually prefer to do it then, I only chose at link_notify 
> because
> of precedence. I'll look into it.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24  1:45                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  1:45 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/23/2017 05:38 PM, Steve Longerbeam wrote:
>
>>
>>> Second, ignoring the above locking issue for a moment,
>>> v4l2_pipeline_pm_use()
>>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
>>> when executing
>>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
>>> wrong order.
>>> In my version which enforces the correct power on order, the mipi csi-2
>>> s_power
>>> is called first in that link setup, followed by the sensor.
>> I don't understand why you want to power up subdevs as soon as the links
>> are established.
>
> Because that is the precedence, all other media drivers do pipeline
> power on/off at link_notify. And v4l2_pipeline_link_notify() was written
> as a link_notify method.
>
>>   Shouldn't that rather be done for all subdevices in the
>> pipeline when the corresponding capture device is opened?
>
> that won't work. There's no guarantee the links will be established
> at capture device open time.

ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are
no links yet, it would basically be a no-op. And stream on requires 
opening the
device, and the pipeline links should be established by then, so this 
might be
fine, looking into this too.

Steve

>
>> It seems to me that powering up the pipeline should be the last step
>> before userspace actually starts the capture.
>
> Well, I'm ok with moving pipeline power on/off to start/stop streaming.
> I would actually prefer to do it then, I only chose at link_notify 
> because
> of precedence. I'll look into it.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-20 14:38     ` Hans Verkuil
@ 2017-01-24  2:15       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  2:15 UTC (permalink / raw)
  To: Hans Verkuil, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam



On 01/20/2017 06:38 AM, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>> +static int vidioc_querycap(struct file *file, void *fh,
>> +			   struct v4l2_capability *cap)
>> +{
>> +	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
>> +	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
>> +	cap->bus_info[0] = 0;
> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance should
> complain about this.

Right, I've fixed this already as part of v4l2-compliance testing.

>
>> +	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
>> +	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
> Set device_caps in struct video_device, then drop these two lines since
> the core will set these up based on the device_caps field in struct video_device.

done.

>
>> +
>> +static int camif_enum_input(struct file *file, void *fh,
>> +			    struct v4l2_input *input)
>> +{
>> +	struct camif_priv *priv = video_drvdata(file);
>> +	struct imx_media_subdev *sensor;
>> +	int index = input->index;
>> +
>> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
>> +	if (IS_ERR(sensor)) {
>> +		v4l2_err(&priv->sd, "no sensor attached\n");
>> +		return PTR_ERR(sensor);
>> +	}
>> +
>> +	if (index >= sensor->input.num)
>> +		return -EINVAL;
>> +
>> +	input->type = V4L2_INPUT_TYPE_CAMERA;
>> +	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
>> +
>> +	if (index == priv->current_input) {
>> +		v4l2_subdev_call(sensor->sd, video, g_input_status,
>> +				 &input->status);
>> +		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
> Wrong op, use g_tvnorms instead.

done.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-24  2:15       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-24  2:15 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/20/2017 06:38 AM, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>> +static int vidioc_querycap(struct file *file, void *fh,
>> +			   struct v4l2_capability *cap)
>> +{
>> +	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
>> +	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
>> +	cap->bus_info[0] = 0;
> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance should
> complain about this.

Right, I've fixed this already as part of v4l2-compliance testing.

>
>> +	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
>> +	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
> Set device_caps in struct video_device, then drop these two lines since
> the core will set these up based on the device_caps field in struct video_device.

done.

>
>> +
>> +static int camif_enum_input(struct file *file, void *fh,
>> +			    struct v4l2_input *input)
>> +{
>> +	struct camif_priv *priv = video_drvdata(file);
>> +	struct imx_media_subdev *sensor;
>> +	int index = input->index;
>> +
>> +	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
>> +	if (IS_ERR(sensor)) {
>> +		v4l2_err(&priv->sd, "no sensor attached\n");
>> +		return PTR_ERR(sensor);
>> +	}
>> +
>> +	if (index >= sensor->input.num)
>> +		return -EINVAL;
>> +
>> +	input->type = V4L2_INPUT_TYPE_CAMERA;
>> +	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
>> +
>> +	if (index == priv->current_input) {
>> +		v4l2_subdev_call(sensor->sd, video, g_input_status,
>> +				 &input->status);
>> +		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
> Wrong op, use g_tvnorms instead.

done.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-24 11:27               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:27 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Hans Verkuil, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam,
	Laurent Pinchart

Hi Steve, Hans,

[added Laurent to Cc: who I believe might have an opinion on the media
bus formats, too. Sorry for the wall of text, I have put a marker where
the MEDIA_BUS argument starts]

The central issue seems to be that I think media pad links / media bus
formats should describe physical links, such as parallel or serial
buses, and the formats of pixels flowing through them, whereas Steve
would like to extend them to describe software transports and in-memory
formats.

On Mon, 2017-01-23 at 15:08 -0800, Steve Longerbeam wrote:
[...]
> >>> And I'm actually in total agreement with that. I definitely agree that there
> >>> should be a mechanism in the media framework that allows passing video
> >>> buffers from a source pad to a sink pad using a software queue, with no
> >>> involvement from userland.
> > That is the other part of the argument. I do not agree that these
> > software queue "links" should be presented to userspace as media pad
> > links between two entities of a media device.
> > First, that would limit the links to subdevices contained in the same
> > media graph, while this should work between any two capture and output
> > queues of different devices.
> 
> It sounds like we are talking about two different new proposed features.

We are talking about the same thing, but we both want a different user
interface.
Technically, the issue is to trigger the DMA read channel of a mem2mem
device automatically whenever another capture device's DMA write channel
signals a finished frame. Where we disagree is how to present this to
userspace.

You represent the capture DMA write channel and mem2mem DMA read channel
as pads on media entites and configure the in-kernel software queue
between the two using a media pad link. At the same time a different
representation of the same DMA write and read channels (the capture
vb2_queue of the capture device and the output vb2_queue of the mem2mem
device) would be used for operation in the classic, userspace controlled
mode via dmabuf passing.

I don't want the software-only link in the media graph, but instead use
the vb2_queue representation for both cases, and implement the in-kernel
queue link on top of the vb2_queue interface. This would allow userspace
to have control over buffer allocations and format, and thus avoid
unexpected performance implications: it is impossible for userspace to
understand which media entity link, when enabled, will cause a
significant increase in memory bandwidth usage, or even how much.
Also the same mechanism could then be used to link any two devices in a
generic manner, instead of special casing the software queue link for
two devices that happen to be part of the same media graph.

> My proposal is to implement a software buffer queue between pads.
> Beyond enabling the link between pads using the existing media controller
> API, userspace is not involved after that. The fact that this link is 
> accomplished with a software buffer queue is not known, and doesn't
> need to be known, by userspace.

I don't think this is a good thing for the reasons stated above and
below:
Since the software buffer queue is opaque to userspace, it is completely
out of userspace control which format is chosen and how the buffers are
allocated.
By using media bus formats to configure software links the kernel
pretends to userspace that there is a physical connection where there
isn't one.
Also, the media entity graph would quickly become very unreadable if we
were to add all devices to it that could reasonably be linked with
software queues.

> Your proposal, if I have it right, is to allow linking two v4l2 device 
> vb2 queues
> (i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
> mode such that v4l2 buffers get passed from one device's vb2 queue to the
> other without requiring the v4l2 user program to actively forward those 
> buffers.

Yes.

> There isn't anything that would preclude one from the other, they can
> both exist. But they are different ideas. One implements software queues
> at the _pad level_ and is opaque to userspace, the other links queues
> at the _device level_ using a new user API, but once the link is 
> established, also does not require any involvement from userspace.

Well, they are different ideas of how the userspace interface _for the
same thing_ should look like.

> What I'm saying is we can do _both_.

What I am saying is we shouldn't do the pad link interface for the
software queues. In my opinion it is the wrong abstraction, and apart
from the convenience of being able to switch the links on with a single
media-ctl invocation, I see too many downsides.

> > Assume for example, we want to encode the captured, deinterlaced video
> > to h.264 with the coda VPU driver. A software queue link could be
> > established between the CSI capture and the VDIC deinterlacer input,
> 
> That's already available in the media graph. By linking CSI and
> VDIC entities. The capture device will then already be providing
> de-interlaced video, and ...

I know it is in your code. That is the cause for my concern. The link
between CSI and VDIC entity should only describe the direct physical
connection through the VDIC FIFO1 in my opinion.
For the indirect CSI -> SMFC -> IDMAC -> RAM, RAM -> IDMAC -> VDIC
software queue, I would strongly prefer to use linked vb2_queues instead
of the media entity link.

> > just as between the VDIC deinterlacer output and the coda VPU input.
> > Technically, there would be no difference between those two linked
> > capture/output queue pairs. But the coda driver is a completely separate
> > mem2mem device. And since it is not part of the i.MX media graph, there
> > is no entity pad to link to.
> 
> your free-run queue linking could then be used to link the (already)
> de-interlaced stream to the coda device for h.264 encode.

Yes, and I see no reason why that should use a different interface than
what is exactly the same process between CSI and VDIC.

> The other idea would be to eventually make the coda device part of
> the media graph as an entity. Then this link would instead be via pads.

I would only want to do this if there was a direct connection between
the IPU FIFOs and the coda VPU device somehow. But since the devices are
completely separate, they should be described as such.

> > Or assume there is an USB analog capture device that produces interlaced
> > frames. I think it should be possible to connect its capture queue to
> > the VDIC deinterlacer output queue just the same way as linking the CSI
> > to the VDIC (in software queue mode).
> 
> Right, for devices that are outside the i.MX media graph, such as a USB
> capture device (or coda), access to the i.MX entities such as the VDIC would
> require an i.MX mem2mem device with media links to the VDIC. The USB
> capture device would forward its captured frames to mem2mem (maybe
> using your free-run vb2 queue linking idea):
> 
> usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device

The VDIC doesn't have a direct to memory channel, so that would be
mem2mem -> VDIC -> IC -> mem2mem, I think?

========== MEDIA_BUS formats below =====================================

> > Second, the subdevice pad formats describe wire formats, not memory
> > formats. The user might want to choose between 4:2:2 and 4:2:0
> > subsampled YUV formats for the intermediate buffer, for example,
> > depending on memory bandwidth constraints and quality requirements. This
> > is impossible with the media entity / subdevice pad links.
> 
> It's true that there are currently no defined planar media bus
> pixel formats. We just need to add new definitions for them. Once
> that is done, the media driver will support planar YUV formats
> simply by adding the new codes to imx_media_formats[].

I am not comfortable with starting to mix MEDIA_BUS_FMT and V4L2_PIX_FMT
this way.

> Perhaps this gets to the root of the issue.
> 
> Is the media bus concept an abstract one, or is the media bus
> intended to represent actual physical buses (as the lack of planar
> media bus formats would imply)?

Yes, maybe that is the root of our disconnect. As I understand it, the
media bus formats are describing "image formats as flowing over physical
busses" [1].

[1] Linux Media Subsystem Documentation, Chapter 4.15.3.4.1.1. Media Bus Pixel Codes
    https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/subdev-formats.html?highlight=media%20bus#v4l2-mbus-pixelcode

I would like to keep it that way and not soften up that description.

> Can we break with the physical-bus-only idea if that is the case, and
> loosen the definition of a media bus to mean the passage of media
> data from one pad to another by whatever means?
> 
> In my view the idea of a physical bus at the sensor makes sense, but
> beyond that, keeping that restriction limits how data can pass between
> pads.
> 
> Hans, any input here?
> 
> If this is really anathema, then I'm willing to remove the software queues
> between pads, but it will be giving up some functionality in the media 
> driver.

So far I am the only one arguing against this, and I haven't yet heard
anything yet that would have convinced me otherwise. That's why I'd too
like some more input on this issue.

Certainly removing the controversial pad link controlled software queues
would remove the point of contention. I think losing some functionality
(in this case "higher quality" deinterlacing without userspace
intervention) for now would be worth it to achieve consensus, and also
reduce the list of things that have to be done for this driver to leave
staging, but that is of course from the point of view of the guy arguing
against that interface.

> It would also mean splitting the VDIC in two. The VDIC entity would be
> limited to only one motion compensation mode,

Currently, yes. That direct mode is the only one that should be
described by the media pad link between CSI and VDIC in my opinion.

> and the full functionality would have
> to be added somewhere else.

I don't understand why the other functionality would necessarily have to
live somewere else, but I can see that it might make sense to do so. In
any case, the separate control of the VDIC/IC via a mem2mem video device
will be needed anyway, as pointed out in the USB example above, or to
deinterlace streams received via network or played back from files.

>  Currently all functionality of the VDIC is implemented in a single media entity.

Yes, at least the mem2mem part should move into its own mem2mem video
device.

> > I think an interface where userspace configures the capture and output
> > queues via v4l2 API, passes dma buffers around from one to the other
> > queue, and then puts both queues into a free running mode would be a
> > much better fit for this mechanism.
> 
> As I said, I see these as two different ideas that can both be
> implemented.

I think we should have cleared up now where we disagree. These are two
different ideas for userspace interfaces for the same functionality. My
opinion is that both should not be implemented

> >>> My only disagreement is when this should be implemented. I think it is
> >>> fine to keep my custom implementation of this in the driver for now. Once
> >>> an extension of vb2 is ready to support this feature, it would be fairly
> >>> straightforward to strip out my custom implementation and go with the
> >>> new API.
> >> For a staging driver this isn't necessary, as long as it is documented in
> >> the TODO file that this needs to be fixed before it can be moved out of
> >> staging. The whole point of staging is that there is still work to be
> >> done in the driver, after all :-)
> > Absolutely. The reason I am arguing against merging the mem2mem media
> > control links so vehemently is that I am convinced the userspace
> > interface is wrong, and I am afraid that even though in staging, it
> > might become established.
> 
> I don't believe there is anything wrong with the userspace interface,
> In fact it hasn't even changed. The fact two pads are passing memory
> buffers is "under the hood".

Which I disagree with. Doing things like this under the hood would be
fine only if they were properly introspectable and if the interface
wouldn't break assumptions that I believe to be there, such as media
links describing physical connections between hardware entities, and
media bus formats describing the image format on a physical bus.
Also with videobuf2 we already have a userspace interface for DMA read
and write queues, and I'd prefer to extend and improve that instead of
reimplementing the same functionality, even simplified, under the hood.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-24 11:27               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:27 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

Hi Steve, Hans,

[added Laurent to Cc: who I believe might have an opinion on the media
bus formats, too. Sorry for the wall of text, I have put a marker where
the MEDIA_BUS argument starts]

The central issue seems to be that I think media pad links / media bus
formats should describe physical links, such as parallel or serial
buses, and the formats of pixels flowing through them, whereas Steve
would like to extend them to describe software transports and in-memory
formats.

On Mon, 2017-01-23 at 15:08 -0800, Steve Longerbeam wrote:
[...]
> >>> And I'm actually in total agreement with that. I definitely agree that there
> >>> should be a mechanism in the media framework that allows passing video
> >>> buffers from a source pad to a sink pad using a software queue, with no
> >>> involvement from userland.
> > That is the other part of the argument. I do not agree that these
> > software queue "links" should be presented to userspace as media pad
> > links between two entities of a media device.
> > First, that would limit the links to subdevices contained in the same
> > media graph, while this should work between any two capture and output
> > queues of different devices.
> 
> It sounds like we are talking about two different new proposed features.

We are talking about the same thing, but we both want a different user
interface.
Technically, the issue is to trigger the DMA read channel of a mem2mem
device automatically whenever another capture device's DMA write channel
signals a finished frame. Where we disagree is how to present this to
userspace.

You represent the capture DMA write channel and mem2mem DMA read channel
as pads on media entites and configure the in-kernel software queue
between the two using a media pad link. At the same time a different
representation of the same DMA write and read channels (the capture
vb2_queue of the capture device and the output vb2_queue of the mem2mem
device) would be used for operation in the classic, userspace controlled
mode via dmabuf passing.

I don't want the software-only link in the media graph, but instead use
the vb2_queue representation for both cases, and implement the in-kernel
queue link on top of the vb2_queue interface. This would allow userspace
to have control over buffer allocations and format, and thus avoid
unexpected performance implications: it is impossible for userspace to
understand which media entity link, when enabled, will cause a
significant increase in memory bandwidth usage, or even how much.
Also the same mechanism could then be used to link any two devices in a
generic manner, instead of special casing the software queue link for
two devices that happen to be part of the same media graph.

> My proposal is to implement a software buffer queue between pads.
> Beyond enabling the link between pads using the existing media controller
> API, userspace is not involved after that. The fact that this link is 
> accomplished with a software buffer queue is not known, and doesn't
> need to be known, by userspace.

I don't think this is a good thing for the reasons stated above and
below:
Since the software buffer queue is opaque to userspace, it is completely
out of userspace control which format is chosen and how the buffers are
allocated.
By using media bus formats to configure software links the kernel
pretends to userspace that there is a physical connection where there
isn't one.
Also, the media entity graph would quickly become very unreadable if we
were to add all devices to it that could reasonably be linked with
software queues.

> Your proposal, if I have it right, is to allow linking two v4l2 device 
> vb2 queues
> (i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
> mode such that v4l2 buffers get passed from one device's vb2 queue to the
> other without requiring the v4l2 user program to actively forward those 
> buffers.

Yes.

> There isn't anything that would preclude one from the other, they can
> both exist. But they are different ideas. One implements software queues
> at the _pad level_ and is opaque to userspace, the other links queues
> at the _device level_ using a new user API, but once the link is 
> established, also does not require any involvement from userspace.

Well, they are different ideas of how the userspace interface _for the
same thing_ should look like.

> What I'm saying is we can do _both_.

What I am saying is we shouldn't do the pad link interface for the
software queues. In my opinion it is the wrong abstraction, and apart
from the convenience of being able to switch the links on with a single
media-ctl invocation, I see too many downsides.

> > Assume for example, we want to encode the captured, deinterlaced video
> > to h.264 with the coda VPU driver. A software queue link could be
> > established between the CSI capture and the VDIC deinterlacer input,
> 
> That's already available in the media graph. By linking CSI and
> VDIC entities. The capture device will then already be providing
> de-interlaced video, and ...

I know it is in your code. That is the cause for my concern. The link
between CSI and VDIC entity should only describe the direct physical
connection through the VDIC FIFO1 in my opinion.
For the indirect CSI -> SMFC -> IDMAC -> RAM, RAM -> IDMAC -> VDIC
software queue, I would strongly prefer to use linked vb2_queues instead
of the media entity link.

> > just as between the VDIC deinterlacer output and the coda VPU input.
> > Technically, there would be no difference between those two linked
> > capture/output queue pairs. But the coda driver is a completely separate
> > mem2mem device. And since it is not part of the i.MX media graph, there
> > is no entity pad to link to.
> 
> your free-run queue linking could then be used to link the (already)
> de-interlaced stream to the coda device for h.264 encode.

Yes, and I see no reason why that should use a different interface than
what is exactly the same process between CSI and VDIC.

> The other idea would be to eventually make the coda device part of
> the media graph as an entity. Then this link would instead be via pads.

I would only want to do this if there was a direct connection between
the IPU FIFOs and the coda VPU device somehow. But since the devices are
completely separate, they should be described as such.

> > Or assume there is an USB analog capture device that produces interlaced
> > frames. I think it should be possible to connect its capture queue to
> > the VDIC deinterlacer output queue just the same way as linking the CSI
> > to the VDIC (in software queue mode).
> 
> Right, for devices that are outside the i.MX media graph, such as a USB
> capture device (or coda), access to the i.MX entities such as the VDIC would
> require an i.MX mem2mem device with media links to the VDIC. The USB
> capture device would forward its captured frames to mem2mem (maybe
> using your free-run vb2 queue linking idea):
> 
> usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device

The VDIC doesn't have a direct to memory channel, so that would be
mem2mem -> VDIC -> IC -> mem2mem, I think?

========== MEDIA_BUS formats below =====================================

> > Second, the subdevice pad formats describe wire formats, not memory
> > formats. The user might want to choose between 4:2:2 and 4:2:0
> > subsampled YUV formats for the intermediate buffer, for example,
> > depending on memory bandwidth constraints and quality requirements. This
> > is impossible with the media entity / subdevice pad links.
> 
> It's true that there are currently no defined planar media bus
> pixel formats. We just need to add new definitions for them. Once
> that is done, the media driver will support planar YUV formats
> simply by adding the new codes to imx_media_formats[].

I am not comfortable with starting to mix MEDIA_BUS_FMT and V4L2_PIX_FMT
this way.

> Perhaps this gets to the root of the issue.
> 
> Is the media bus concept an abstract one, or is the media bus
> intended to represent actual physical buses (as the lack of planar
> media bus formats would imply)?

Yes, maybe that is the root of our disconnect. As I understand it, the
media bus formats are describing "image formats as flowing over physical
busses" [1].

[1] Linux Media Subsystem Documentation, Chapter 4.15.3.4.1.1. Media Bus Pixel Codes
    https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/subdev-formats.html?highlight=media%20bus#v4l2-mbus-pixelcode

I would like to keep it that way and not soften up that description.

> Can we break with the physical-bus-only idea if that is the case, and
> loosen the definition of a media bus to mean the passage of media
> data from one pad to another by whatever means?
> 
> In my view the idea of a physical bus at the sensor makes sense, but
> beyond that, keeping that restriction limits how data can pass between
> pads.
> 
> Hans, any input here?
> 
> If this is really anathema, then I'm willing to remove the software queues
> between pads, but it will be giving up some functionality in the media 
> driver.

So far I am the only one arguing against this, and I haven't yet heard
anything yet that would have convinced me otherwise. That's why I'd too
like some more input on this issue.

Certainly removing the controversial pad link controlled software queues
would remove the point of contention. I think losing some functionality
(in this case "higher quality" deinterlacing without userspace
intervention) for now would be worth it to achieve consensus, and also
reduce the list of things that have to be done for this driver to leave
staging, but that is of course from the point of view of the guy arguing
against that interface.

> It would also mean splitting the VDIC in two. The VDIC entity would be
> limited to only one motion compensation mode,

Currently, yes. That direct mode is the only one that should be
described by the media pad link between CSI and VDIC in my opinion.

> and the full functionality would have
> to be added somewhere else.

I don't understand why the other functionality would necessarily have to
live somewere else, but I can see that it might make sense to do so. In
any case, the separate control of the VDIC/IC via a mem2mem video device
will be needed anyway, as pointed out in the USB example above, or to
deinterlace streams received via network or played back from files.

>  Currently all functionality of the VDIC is implemented in a single media entity.

Yes, at least the mem2mem part should move into its own mem2mem video
device.

> > I think an interface where userspace configures the capture and output
> > queues via v4l2 API, passes dma buffers around from one to the other
> > queue, and then puts both queues into a free running mode would be a
> > much better fit for this mechanism.
> 
> As I said, I see these as two different ideas that can both be
> implemented.

I think we should have cleared up now where we disagree. These are two
different ideas for userspace interfaces for the same functionality. My
opinion is that both should not be implemented

> >>> My only disagreement is when this should be implemented. I think it is
> >>> fine to keep my custom implementation of this in the driver for now. Once
> >>> an extension of vb2 is ready to support this feature, it would be fairly
> >>> straightforward to strip out my custom implementation and go with the
> >>> new API.
> >> For a staging driver this isn't necessary, as long as it is documented in
> >> the TODO file that this needs to be fixed before it can be moved out of
> >> staging. The whole point of staging is that there is still work to be
> >> done in the driver, after all :-)
> > Absolutely. The reason I am arguing against merging the mem2mem media
> > control links so vehemently is that I am convinced the userspace
> > interface is wrong, and I am afraid that even though in staging, it
> > might become established.
> 
> I don't believe there is anything wrong with the userspace interface,
> In fact it hasn't even changed. The fact two pads are passing memory
> buffers is "under the hood".

Which I disagree with. Doing things like this under the hood would be
fine only if they were properly introspectable and if the interface
wouldn't break assumptions that I believe to be there, such as media
links describing physical connections between hardware entities, and
media bus formats describing the image format on a physical bus.
Also with videobuf2 we already have a userspace interface for DMA read
and write queues, and I'd prefer to extend and improve that instead of
reimplementing the same functionality, even simplified, under the hood.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-24 11:27               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:27 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve, Hans,

[added Laurent to Cc: who I believe might have an opinion on the media
bus formats, too. Sorry for the wall of text, I have put a marker where
the MEDIA_BUS argument starts]

The central issue seems to be that I think media pad links / media bus
formats should describe physical links, such as parallel or serial
buses, and the formats of pixels flowing through them, whereas Steve
would like to extend them to describe software transports and in-memory
formats.

On Mon, 2017-01-23 at 15:08 -0800, Steve Longerbeam wrote:
[...]
> >>> And I'm actually in total agreement with that. I definitely agree that there
> >>> should be a mechanism in the media framework that allows passing video
> >>> buffers from a source pad to a sink pad using a software queue, with no
> >>> involvement from userland.
> > That is the other part of the argument. I do not agree that these
> > software queue "links" should be presented to userspace as media pad
> > links between two entities of a media device.
> > First, that would limit the links to subdevices contained in the same
> > media graph, while this should work between any two capture and output
> > queues of different devices.
> 
> It sounds like we are talking about two different new proposed features.

We are talking about the same thing, but we both want a different user
interface.
Technically, the issue is to trigger the DMA read channel of a mem2mem
device automatically whenever another capture device's DMA write channel
signals a finished frame. Where we disagree is how to present this to
userspace.

You represent the capture DMA write channel and mem2mem DMA read channel
as pads on media entites and configure the in-kernel software queue
between the two using a media pad link. At the same time a different
representation of the same DMA write and read channels (the capture
vb2_queue of the capture device and the output vb2_queue of the mem2mem
device) would be used for operation in the classic, userspace controlled
mode via dmabuf passing.

I don't want the software-only link in the media graph, but instead use
the vb2_queue representation for both cases, and implement the in-kernel
queue link on top of the vb2_queue interface. This would allow userspace
to have control over buffer allocations and format, and thus avoid
unexpected performance implications: it is impossible for userspace to
understand which media entity link, when enabled, will cause a
significant increase in memory bandwidth usage, or even how much.
Also the same mechanism could then be used to link any two devices in a
generic manner, instead of special casing the software queue link for
two devices that happen to be part of the same media graph.

> My proposal is to implement a software buffer queue between pads.
> Beyond enabling the link between pads using the existing media controller
> API, userspace is not involved after that. The fact that this link is 
> accomplished with a software buffer queue is not known, and doesn't
> need to be known, by userspace.

I don't think this is a good thing for the reasons stated above and
below:
Since the software buffer queue is opaque to userspace, it is completely
out of userspace control which format is chosen and how the buffers are
allocated.
By using media bus formats to configure software links the kernel
pretends to userspace that there is a physical connection where there
isn't one.
Also, the media entity graph would quickly become very unreadable if we
were to add all devices to it that could reasonably be linked with
software queues.

> Your proposal, if I have it right, is to allow linking two v4l2 device 
> vb2 queues
> (i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
> mode such that v4l2 buffers get passed from one device's vb2 queue to the
> other without requiring the v4l2 user program to actively forward those 
> buffers.

Yes.

> There isn't anything that would preclude one from the other, they can
> both exist. But they are different ideas. One implements software queues
> at the _pad level_ and is opaque to userspace, the other links queues
> at the _device level_ using a new user API, but once the link is 
> established, also does not require any involvement from userspace.

Well, they are different ideas of how the userspace interface _for the
same thing_ should look like.

> What I'm saying is we can do _both_.

What I am saying is we shouldn't do the pad link interface for the
software queues. In my opinion it is the wrong abstraction, and apart
from the convenience of being able to switch the links on with a single
media-ctl invocation, I see too many downsides.

> > Assume for example, we want to encode the captured, deinterlaced video
> > to h.264 with the coda VPU driver. A software queue link could be
> > established between the CSI capture and the VDIC deinterlacer input,
> 
> That's already available in the media graph. By linking CSI and
> VDIC entities. The capture device will then already be providing
> de-interlaced video, and ...

I know it is in your code. That is the cause for my concern. The link
between CSI and VDIC entity should only describe the direct physical
connection through the VDIC FIFO1 in my opinion.
For the indirect CSI -> SMFC -> IDMAC -> RAM, RAM -> IDMAC -> VDIC
software queue, I would strongly prefer to use linked vb2_queues instead
of the media entity link.

> > just as between the VDIC deinterlacer output and the coda VPU input.
> > Technically, there would be no difference between those two linked
> > capture/output queue pairs. But the coda driver is a completely separate
> > mem2mem device. And since it is not part of the i.MX media graph, there
> > is no entity pad to link to.
> 
> your free-run queue linking could then be used to link the (already)
> de-interlaced stream to the coda device for h.264 encode.

Yes, and I see no reason why that should use a different interface than
what is exactly the same process between CSI and VDIC.

> The other idea would be to eventually make the coda device part of
> the media graph as an entity. Then this link would instead be via pads.

I would only want to do this if there was a direct connection between
the IPU FIFOs and the coda VPU device somehow. But since the devices are
completely separate, they should be described as such.

> > Or assume there is an USB analog capture device that produces interlaced
> > frames. I think it should be possible to connect its capture queue to
> > the VDIC deinterlacer output queue just the same way as linking the CSI
> > to the VDIC (in software queue mode).
> 
> Right, for devices that are outside the i.MX media graph, such as a USB
> capture device (or coda), access to the i.MX entities such as the VDIC would
> require an i.MX mem2mem device with media links to the VDIC. The USB
> capture device would forward its captured frames to mem2mem (maybe
> using your free-run vb2 queue linking idea):
> 
> usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device

The VDIC doesn't have a direct to memory channel, so that would be
mem2mem -> VDIC -> IC -> mem2mem, I think?

========== MEDIA_BUS formats below =====================================

> > Second, the subdevice pad formats describe wire formats, not memory
> > formats. The user might want to choose between 4:2:2 and 4:2:0
> > subsampled YUV formats for the intermediate buffer, for example,
> > depending on memory bandwidth constraints and quality requirements. This
> > is impossible with the media entity / subdevice pad links.
> 
> It's true that there are currently no defined planar media bus
> pixel formats. We just need to add new definitions for them. Once
> that is done, the media driver will support planar YUV formats
> simply by adding the new codes to imx_media_formats[].

I am not comfortable with starting to mix MEDIA_BUS_FMT and V4L2_PIX_FMT
this way.

> Perhaps this gets to the root of the issue.
> 
> Is the media bus concept an abstract one, or is the media bus
> intended to represent actual physical buses (as the lack of planar
> media bus formats would imply)?

Yes, maybe that is the root of our disconnect. As I understand it, the
media bus formats are describing "image formats as flowing over physical
busses" [1].

[1] Linux Media Subsystem Documentation, Chapter 4.15.3.4.1.1. Media Bus Pixel Codes
    https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/subdev-formats.html?highlight=media%20bus#v4l2-mbus-pixelcode

I would like to keep it that way and not soften up that description.

> Can we break with the physical-bus-only idea if that is the case, and
> loosen the definition of a media bus to mean the passage of media
> data from one pad to another by whatever means?
> 
> In my view the idea of a physical bus at the sensor makes sense, but
> beyond that, keeping that restriction limits how data can pass between
> pads.
> 
> Hans, any input here?
> 
> If this is really anathema, then I'm willing to remove the software queues
> between pads, but it will be giving up some functionality in the media 
> driver.

So far I am the only one arguing against this, and I haven't yet heard
anything yet that would have convinced me otherwise. That's why I'd too
like some more input on this issue.

Certainly removing the controversial pad link controlled software queues
would remove the point of contention. I think losing some functionality
(in this case "higher quality" deinterlacing without userspace
intervention) for now would be worth it to achieve consensus, and also
reduce the list of things that have to be done for this driver to leave
staging, but that is of course from the point of view of the guy arguing
against that interface.

> It would also mean splitting the VDIC in two. The VDIC entity would be
> limited to only one motion compensation mode,

Currently, yes. That direct mode is the only one that should be
described by the media pad link between CSI and VDIC in my opinion.

> and the full functionality would have
> to be added somewhere else.

I don't understand why the other functionality would necessarily have to
live somewere else, but I can see that it might make sense to do so. In
any case, the separate control of the VDIC/IC via a mem2mem video device
will be needed anyway, as pointed out in the USB example above, or to
deinterlace streams received via network or played back from files.

>  Currently all functionality of the VDIC is implemented in a single media entity.

Yes, at least the mem2mem part should move into its own mem2mem video
device.

> > I think an interface where userspace configures the capture and output
> > queues via v4l2 API, passes dma buffers around from one to the other
> > queue, and then puts both queues into a free running mode would be a
> > much better fit for this mechanism.
> 
> As I said, I see these as two different ideas that can both be
> implemented.

I think we should have cleared up now where we disagree. These are two
different ideas for userspace interfaces for the same functionality. My
opinion is that both should not be implemented

> >>> My only disagreement is when this should be implemented. I think it is
> >>> fine to keep my custom implementation of this in the driver for now. Once
> >>> an extension of vb2 is ready to support this feature, it would be fairly
> >>> straightforward to strip out my custom implementation and go with the
> >>> new API.
> >> For a staging driver this isn't necessary, as long as it is documented in
> >> the TODO file that this needs to be fixed before it can be moved out of
> >> staging. The whole point of staging is that there is still work to be
> >> done in the driver, after all :-)
> > Absolutely. The reason I am arguing against merging the mem2mem media
> > control links so vehemently is that I am convinced the userspace
> > interface is wrong, and I am afraid that even though in staging, it
> > might become established.
> 
> I don't believe there is anything wrong with the userspace interface,
> In fact it hasn't even changed. The fact two pads are passing memory
> buffers is "under the hood".

Which I disagree with. Doing things like this under the hood would be
fine only if they were properly introspectable and if the interface
wouldn't break assumptions that I believe to be there, such as media
links describing physical connections between hardware entities, and
media bus formats describing the image format on a physical bus.
Also with videobuf2 we already have a userspace interface for DMA read
and write queues, and I'd prefer to extend and improve that instead of
reimplementing the same functionality, even simplified, under the hood.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-23 11:08             ` Hans Verkuil
  (?)
@ 2017-01-24 11:28               ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:28 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Mon, 2017-01-23 at 12:08 +0100, Hans Verkuil wrote:
> On 01/23/2017 12:00 PM, Philipp Zabel wrote:
> > On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
[...]
> As long as it is mentioned in the TODO, and ideally in the Kconfig as well,
> then I'm fine with it.
>
> The big advantage of being in the kernel is that it is much easier to start
> providing fixes, improvements, etc. If you use a staging driver you know
> that there is no guarantee whatsoever with respect to stable ABI/APIs.

Of course, but there should be a clear way how to progress on those
issues that are documented as blockers, otherwise the driver will linger
in staging.
Worse, currently we are not even in agreement what to put into the TODO.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-24 11:28               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:28 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, Steve Longerbeam,
	linux-media, devicetree, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Mon, 2017-01-23 at 12:08 +0100, Hans Verkuil wrote:
> On 01/23/2017 12:00 PM, Philipp Zabel wrote:
> > On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
[...]
> As long as it is mentioned in the TODO, and ideally in the Kconfig as well,
> then I'm fine with it.
>
> The big advantage of being in the kernel is that it is much easier to start
> providing fixes, improvements, etc. If you use a staging driver you know
> that there is no guarantee whatsoever with respect to stable ABI/APIs.

Of course, but there should be a clear way how to progress on those
issues that are documented as blockers, otherwise the driver will linger
in staging.
Worse, currently we are not even in agreement what to put into the TODO.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-24 11:28               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:28 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, 2017-01-23 at 12:08 +0100, Hans Verkuil wrote:
> On 01/23/2017 12:00 PM, Philipp Zabel wrote:
> > On Fri, 2017-01-20 at 21:39 +0100, Hans Verkuil wrote:
[...]
> As long as it is mentioned in the TODO, and ideally in the Kconfig as well,
> then I'm fine with it.
>
> The big advantage of being in the kernel is that it is much easier to start
> providing fixes, improvements, etc. If you use a staging driver you know
> that there is no guarantee whatsoever with respect to stable ABI/APIs.

Of course, but there should be a clear way how to progress on those
issues that are documented as blockers, otherwise the driver will linger
in staging.
Worse, currently we are not even in agreement what to put into the TODO.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24 11:37                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:37 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

Hi Steve,

On Mon, 2017-01-23 at 17:45 -0800, Steve Longerbeam wrote:
> 
> On 01/23/2017 05:38 PM, Steve Longerbeam wrote:
> >
> >>
> >>> Second, ignoring the above locking issue for a moment,
> >>> v4l2_pipeline_pm_use()
> >>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
> >>> when executing
> >>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
> >>> wrong order.
> >>> In my version which enforces the correct power on order, the mipi csi-2
> >>> s_power
> >>> is called first in that link setup, followed by the sensor.
> >> I don't understand why you want to power up subdevs as soon as the links
> >> are established.
> >
> > Because that is the precedence, all other media drivers do pipeline
> > power on/off at link_notify. And v4l2_pipeline_link_notify() was written
> > as a link_notify method.
> >
> >>   Shouldn't that rather be done for all subdevices in the
> >> pipeline when the corresponding capture device is opened?
> >
> > that won't work. There's no guarantee the links will be established
> > at capture device open time.

If the device is opened before the links are established, it won't be
usable anyway. And I think the connected pipeline should be locked in
place while the video device is opened. Is there any reason to ever open
the video device and only then start linking entities?

> ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are
> no links yet, it would basically be a no-op. And stream on requires 
> opening the
> device, and the pipeline links should be established by then, so this 
> might be
> fine, looking into this too.

Thanks for looking into it, at least I had that working for the
TC358743->MIPI-CSI2 link in my driver.

> >> It seems to me that powering up the pipeline should be the last step
> >> before userspace actually starts the capture.
> >
> > Well, I'm ok with moving pipeline power on/off to start/stop streaming.
> > I would actually prefer to do it then, I only chose at link_notify 
> > because of precedence. I'll look into it.

That might be too late, though. I would expect STREAMON/STREAMOFF to be
a rather fast operation as all the slow preparation could be at open /
REQBUFS time. Also, there might be sensors that need to be powered on to
handle the v4l2_ctrl passthrough?

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24 11:37                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:37 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

Hi Steve,

On Mon, 2017-01-23 at 17:45 -0800, Steve Longerbeam wrote:
> 
> On 01/23/2017 05:38 PM, Steve Longerbeam wrote:
> >
> >>
> >>> Second, ignoring the above locking issue for a moment,
> >>> v4l2_pipeline_pm_use()
> >>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
> >>> when executing
> >>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
> >>> wrong order.
> >>> In my version which enforces the correct power on order, the mipi csi-2
> >>> s_power
> >>> is called first in that link setup, followed by the sensor.
> >> I don't understand why you want to power up subdevs as soon as the links
> >> are established.
> >
> > Because that is the precedence, all other media drivers do pipeline
> > power on/off at link_notify. And v4l2_pipeline_link_notify() was written
> > as a link_notify method.
> >
> >>   Shouldn't that rather be done for all subdevices in the
> >> pipeline when the corresponding capture device is opened?
> >
> > that won't work. There's no guarantee the links will be established
> > at capture device open time.

If the device is opened before the links are established, it won't be
usable anyway. And I think the connected pipeline should be locked in
place while the video device is opened. Is there any reason to ever open
the video device and only then start linking entities?

> ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are
> no links yet, it would basically be a no-op. And stream on requires 
> opening the
> device, and the pipeline links should be established by then, so this 
> might be
> fine, looking into this too.

Thanks for looking into it, at least I had that working for the
TC358743->MIPI-CSI2 link in my driver.

> >> It seems to me that powering up the pipeline should be the last step
> >> before userspace actually starts the capture.
> >
> > Well, I'm ok with moving pipeline power on/off to start/stop streaming.
> > I would actually prefer to do it then, I only chose at link_notify 
> > because of precedence. I'll look into it.

That might be too late, though. I would expect STREAMON/STREAMOFF to be
a rather fast operation as all the slow preparation could be at open /
REQBUFS time. Also, there might be sensors that need to be powered on to
handle the v4l2_ctrl passthrough?

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24 11:37                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:37 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Mon, 2017-01-23 at 17:45 -0800, Steve Longerbeam wrote:
> 
> On 01/23/2017 05:38 PM, Steve Longerbeam wrote:
> >
> >>
> >>> Second, ignoring the above locking issue for a moment,
> >>> v4l2_pipeline_pm_use()
> >>> will call s_power on the sensor _first_, then the mipi csi-2 s_power,
> >>> when executing
> >>> media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the
> >>> wrong order.
> >>> In my version which enforces the correct power on order, the mipi csi-2
> >>> s_power
> >>> is called first in that link setup, followed by the sensor.
> >> I don't understand why you want to power up subdevs as soon as the links
> >> are established.
> >
> > Because that is the precedence, all other media drivers do pipeline
> > power on/off at link_notify. And v4l2_pipeline_link_notify() was written
> > as a link_notify method.
> >
> >>   Shouldn't that rather be done for all subdevices in the
> >> pipeline when the corresponding capture device is opened?
> >
> > that won't work. There's no guarantee the links will be established
> > at capture device open time.

If the device is opened before the links are established, it won't be
usable anyway. And I think the connected pipeline should be locked in
place while the video device is opened. Is there any reason to ever open
the video device and only then start linking entities?

> ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are
> no links yet, it would basically be a no-op. And stream on requires 
> opening the
> device, and the pipeline links should be established by then, so this 
> might be
> fine, looking into this too.

Thanks for looking into it, at least I had that working for the
TC358743->MIPI-CSI2 link in my driver.

> >> It seems to me that powering up the pipeline should be the last step
> >> before userspace actually starts the capture.
> >
> > Well, I'm ok with moving pipeline power on/off to start/stop streaming.
> > I would actually prefer to do it then, I only chose at link_notify 
> > because of precedence. I'll look into it.

That might be too late, though. I would expect STREAMON/STREAMOFF to be
a rather fast operation as all the slow preparation could be at open /
REQBUFS time. Also, there might be sensors that need to be powered on to
handle the v4l2_ctrl passthrough?

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24 11:39           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:39 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Wed, 2017-01-18 at 17:44 -0800, Steve Longerbeam wrote:
> 
> On 01/14/2017 02:42 PM, Steve Longerbeam wrote:
> >
> >>> +/* parse inputs property from a sensor node */
> >>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> >>> +				   struct imx_media_subdev *sensor,
> >>> +				   struct device_node *sensor_np)
> >>> +{
> >>> +	struct imx_media_sensor_input *sinput = &sensor->input;
> >>> +	int ret, i;
> >>> +
> >>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> >>> +		const char *input_name;
> >>> +		u32 val;
> >>> +
> >>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> >>> +		if (ret)
> >>> +			break;
> >>> +
> >>> +		sinput->value[i] = val;
> >>> +
> >>> +		ret = of_property_read_string_index(sensor_np, "input-names",
> >>> +						    i, &input_name);
> >>> +		/*
> >>> +		 * if input-names not provided, they will be set using
> >>> +		 * the subdev name once the sensor is known during
> >>> +		 * async bind
> >>> +		 */
> >>> +		if (!ret)
> >>> +			strncpy(sinput->name[i], input_name,
> >>> +				sizeof(sinput->name[i]));
> >>> +	}
> >>> +
> >>> +	sinput->num = i;
> >>> +
> >>> +	/* if no inputs provided just assume a single input */
> >>> +	if (sinput->num == 0)
> >>> +		sinput->num = 1;
> >>> +}
> >> This should be parsed by the sensor driver, not imx-media.
> >
> > you're probably right. I'll submit a patch for adv7180.c.
> 
> Actually, the problem here is that this parses an input routing value to
> pass to s_routing, and an input name string. There would need to be
> another subdev callback, maybe enum_imput, that would return this
> information for the bridge driver, if this info were to be parsed and
> maintained by the sensor.
> 
> But this info should really be known and parsed by the bridge anyway,
> because as the header for s_routing states,
> 
> "An i2c device shouldn't know about whether an input pin is connected
>   to a Composite connector, because on another board or platform it
>   might be connected to something else entirely. The calling driver is
>   responsible for mapping a user-level input to the right pins on the i2c
>   device."

I think this description is for non-DT only, as parsing the DT bindings
is the obligation of the bound driver, and with that it information it
can very well know its connections.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24 11:39           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:39 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Wed, 2017-01-18 at 17:44 -0800, Steve Longerbeam wrote:
> 
> On 01/14/2017 02:42 PM, Steve Longerbeam wrote:
> >
> >>> +/* parse inputs property from a sensor node */
> >>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> >>> +				   struct imx_media_subdev *sensor,
> >>> +				   struct device_node *sensor_np)
> >>> +{
> >>> +	struct imx_media_sensor_input *sinput = &sensor->input;
> >>> +	int ret, i;
> >>> +
> >>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> >>> +		const char *input_name;
> >>> +		u32 val;
> >>> +
> >>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> >>> +		if (ret)
> >>> +			break;
> >>> +
> >>> +		sinput->value[i] = val;
> >>> +
> >>> +		ret = of_property_read_string_index(sensor_np, "input-names",
> >>> +						    i, &input_name);
> >>> +		/*
> >>> +		 * if input-names not provided, they will be set using
> >>> +		 * the subdev name once the sensor is known during
> >>> +		 * async bind
> >>> +		 */
> >>> +		if (!ret)
> >>> +			strncpy(sinput->name[i], input_name,
> >>> +				sizeof(sinput->name[i]));
> >>> +	}
> >>> +
> >>> +	sinput->num = i;
> >>> +
> >>> +	/* if no inputs provided just assume a single input */
> >>> +	if (sinput->num == 0)
> >>> +		sinput->num = 1;
> >>> +}
> >> This should be parsed by the sensor driver, not imx-media.
> >
> > you're probably right. I'll submit a patch for adv7180.c.
> 
> Actually, the problem here is that this parses an input routing value to
> pass to s_routing, and an input name string. There would need to be
> another subdev callback, maybe enum_imput, that would return this
> information for the bridge driver, if this info were to be parsed and
> maintained by the sensor.
> 
> But this info should really be known and parsed by the bridge anyway,
> because as the header for s_routing states,
> 
> "An i2c device shouldn't know about whether an input pin is connected
>   to a Composite connector, because on another board or platform it
>   might be connected to something else entirely. The calling driver is
>   responsible for mapping a user-level input to the right pins on the i2c
>   device."

I think this description is for non-DT only, as parsing the DT bindings
is the obligation of the bound driver, and with that it information it
can very well know its connections.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-24 11:39           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 11:39 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, 2017-01-18 at 17:44 -0800, Steve Longerbeam wrote:
> 
> On 01/14/2017 02:42 PM, Steve Longerbeam wrote:
> >
> >>> +/* parse inputs property from a sensor node */
> >>> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> >>> +				   struct imx_media_subdev *sensor,
> >>> +				   struct device_node *sensor_np)
> >>> +{
> >>> +	struct imx_media_sensor_input *sinput = &sensor->input;
> >>> +	int ret, i;
> >>> +
> >>> +	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> >>> +		const char *input_name;
> >>> +		u32 val;
> >>> +
> >>> +		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> >>> +		if (ret)
> >>> +			break;
> >>> +
> >>> +		sinput->value[i] = val;
> >>> +
> >>> +		ret = of_property_read_string_index(sensor_np, "input-names",
> >>> +						    i, &input_name);
> >>> +		/*
> >>> +		 * if input-names not provided, they will be set using
> >>> +		 * the subdev name once the sensor is known during
> >>> +		 * async bind
> >>> +		 */
> >>> +		if (!ret)
> >>> +			strncpy(sinput->name[i], input_name,
> >>> +				sizeof(sinput->name[i]));
> >>> +	}
> >>> +
> >>> +	sinput->num = i;
> >>> +
> >>> +	/* if no inputs provided just assume a single input */
> >>> +	if (sinput->num == 0)
> >>> +		sinput->num = 1;
> >>> +}
> >> This should be parsed by the sensor driver, not imx-media.
> >
> > you're probably right. I'll submit a patch for adv7180.c.
> 
> Actually, the problem here is that this parses an input routing value to
> pass to s_routing, and an input name string. There would need to be
> another subdev callback, maybe enum_imput, that would return this
> information for the bridge driver, if this info were to be parsed and
> maintained by the sensor.
> 
> But this info should really be known and parsed by the bridge anyway,
> because as the header for s_routing states,
> 
> "An i2c device shouldn't know about whether an input pin is connected
>   to a Composite connector, because on another board or platform it
>   might be connected to something else entirely. The calling driver is
>   responsible for mapping a user-level input to the right pins on the i2c
>   device."

I think this description is for non-DT only, as parsing the DT bindings
is the obligation of the bound driver, and with that it information it
can very well know its connections.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-24 12:02       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 12:02 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam

Hi Hans,

On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > This driver can handle SoC internal and external video bus multiplexers,
> > controlled either by register bit fields or by a GPIO. The subdevice
> > passes through frame interval and mbus configuration of the active input
> > to the output side.
> > 
> > Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > --
> > 
> > - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
> >   should be unregister.
> > 
> > - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >   to vidsw_remove().
> > 
> > - there was a line left over from a previous iteration that negated
> >   the new way of determining the pad count just before it which
> >   has been removed (num_pads = of_get_child_count(np)).
> > 
> > - Philipp Zabel has developed a set of patches that allow adding
> >   to the subdev async notifier waiting list using a chaining method
> >   from the async registered callbacks (v4l2_of_subdev_registered()
> >   and the prep patches for that). For now, I've removed the use of
> >   v4l2_of_subdev_registered() for the vidmux driver's registered
> >   callback. This doesn't affect the functionality of this driver,
> >   but allows for it to be merged now, before adding the chaining
> >   support.
> > 
> > Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> > ---
> >  .../bindings/media/video-multiplexer.txt           |  59 +++
> >  drivers/media/platform/Kconfig                     |   8 +
> >  drivers/media/platform/Makefile                    |   2 +
> >  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
> >  4 files changed, 541 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
> >  create mode 100644 drivers/media/platform/video-multiplexer.c
> > 
> > diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > new file mode 100644
> > index 0000000..9d133d9
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > @@ -0,0 +1,59 @@
> > +Video Multiplexer
> > +=================
> > +
> > +Video multiplexers allow to select between multiple input ports. Video received
> > +on the active input port is passed through to the output port. Muxes described
> > +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> > +
> > +Required properties:
> > +- compatible : should be "video-multiplexer"
> > +- reg: should be register base of the register containing the control bitfield
> > +- bit-mask: bitmask of the control bitfield in the control register
> > +- bit-shift: bit offset of the control bitfield in the control register
> > +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> > +  may be given to switch between two inputs
> > +- #address-cells: should be <1>
> > +- #size-cells: should be <0>
> > +- port@*: at least three port nodes containing endpoints connecting to the
> > +  source and sink devices according to of_graph bindings. The last port is
> > +  the output port, all others are inputs.
> > +
> > +Example:
> > +
> > +syscon {
> > +	compatible = "syscon", "simple-mfd";
> > +
> > +	mux {
> > +		compatible = "video-multiplexer";
> > +		/* Single bit (1 << 19) in syscon register 0x04: */
> > +		reg = <0x04>;
> > +		bit-mask = <1>;
> > +		bit-shift = <19>;
> > +		#address-cells = <1>;
> > +		#size-cells = <0>;
> > +
> > +		port@0 {
> > +			reg = <0>;
> > +
> > +			mux_in0: endpoint {
> > +				remote-endpoint = <&video_source0_out>;
> > +			};
> > +		};
> > +
> > +		port@1 {
> > +			reg = <1>;
> > +
> > +			mux_in1: endpoint {
> > +				remote-endpoint = <&video_source1_out>;
> > +			};
> > +		};
> > +
> > +		port@2 {
> > +			reg = <2>;
> > +
> > +			mux_out: endpoint {
> > +				remote-endpoint = <&capture_interface_in>;
> > +			};
> > +		};
> > +	};
> > +};
> > diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> > index d944421..65614b5 100644
> > --- a/drivers/media/platform/Kconfig
> > +++ b/drivers/media/platform/Kconfig
> > @@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
> >  	  To compile this driver as a module, choose M here: the
> >  	  module will be called arv.
> >  
> > +config VIDEO_MULTIPLEXER
> > +	tristate "Video Multiplexer"
> > +	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> > +	help
> > +	  This driver provides support for SoC internal N:1 video bus
> > +	  multiplexers controlled by register bitfields as well as external
> > +	  2:1 video multiplexers controlled by a single GPIO.
> > +
> >  config VIDEO_OMAP3
> >  	tristate "OMAP 3 Camera support"
> >  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
> > diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> > index 5b3cb27..7cf0ee5 100644
> > --- a/drivers/media/platform/Makefile
> > +++ b/drivers/media/platform/Makefile
> > @@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
> >  
> >  obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
> >  
> > +obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
> > +
> >  obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
> >  obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
> >  obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
> > diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
> > new file mode 100644
> > index 0000000..48980c4
> > --- /dev/null
> > +++ b/drivers/media/platform/video-multiplexer.c
> > @@ -0,0 +1,472 @@
> > +/*
> > + * video stream multiplexer controlled via gpio or syscon
> > + *
> > + * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
> > + * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
> > + *
> > + * This program is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU General Public License
> > + * as published by the Free Software Foundation; either version 2
> > + * of the License, or (at your option) any later version.
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <linux/err.h>
> > +#include <linux/gpio/consumer.h>
> > +#include <linux/mfd/syscon.h>
> > +#include <linux/module.h>
> > +#include <linux/of.h>
> > +#include <linux/of_graph.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/regmap.h>
> > +#include <media/v4l2-async.h>
> > +#include <media/v4l2-device.h>
> > +#include <media/v4l2-subdev.h>
> > +#include <media/v4l2-of.h>
> > +
> > +struct vidsw {
> > +	struct v4l2_subdev subdev;
> > +	unsigned int num_pads;
> > +	struct media_pad *pads;
> > +	struct v4l2_mbus_framefmt *format_mbus;
> > +	struct v4l2_fract timeperframe;
> > +	struct v4l2_of_endpoint *endpoint;
> > +	struct regmap_field *field;
> > +	struct gpio_desc *gpio;
> > +	int active;
> > +};
> > +
> > +static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
> > +{
> > +	return container_of(sd, struct vidsw, subdev);
> > +}
> > +
> > +static void vidsw_set_active(struct vidsw *vidsw, int active)
> > +{
> > +	vidsw->active = active;
> > +	if (active < 0)
> > +		return;
> > +
> > +	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
> > +
> > +	if (vidsw->field)
> > +		regmap_field_write(vidsw->field, active);
> > +	else if (vidsw->gpio)
> > +		gpiod_set_value(vidsw->gpio, active);
> > +}
> > +
> > +static int vidsw_link_setup(struct media_entity *entity,
> > +			    const struct media_pad *local,
> > +			    const struct media_pad *remote, u32 flags)
> > +{
> > +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> > +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> > +
> > +	/* We have no limitations on enabling or disabling our output link */
> > +	if (local->index == vidsw->num_pads - 1)
> > +		return 0;
> > +
> > +	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
> > +		local->entity->name);
> > +
> > +	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
> > +		if (local->index == vidsw->active) {
> > +			dev_dbg(sd->dev, "going inactive\n");
> > +			vidsw->active = -1;
> > +		}
> > +		return 0;
> > +	}
> > +
> > +	if (vidsw->active >= 0) {
> > +		struct media_pad *pad;
> > +
> > +		if (vidsw->active == local->index)
> > +			return 0;
> > +
> > +		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> > +		if (pad) {
> > +			struct media_link *link;
> > +			int ret;
> > +
> > +			link = media_entity_find_link(pad,
> > +						&vidsw->pads[vidsw->active]);
> > +			if (link) {
> > +				ret = __media_entity_setup_link(link, 0);
> > +				if (ret)
> > +					return ret;
> > +			}
> > +		}
> > +	}
> > +
> > +	vidsw_set_active(vidsw, local->index);
> > +
> > +	return 0;
> > +}
> > +
> > +static struct media_entity_operations vidsw_ops = {
> > +	.link_setup = vidsw_link_setup,
> > +};
> > +
> > +static bool vidsw_endpoint_disabled(struct device_node *ep)
> > +{
> > +	struct device_node *rpp;
> > +
> > +	if (!of_device_is_available(ep))
> > +		return true;
> > +
> > +	rpp = of_graph_get_remote_port_parent(ep);
> > +	if (!rpp)
> > +		return true;
> > +
> > +	return !of_device_is_available(rpp);
> > +}
> > +
> > +static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
> > +{
> > +	struct device_node *ep;
> > +	u32 portno;
> > +	int numports;
> > +	int ret;
> > +	int i;
> > +	bool active_link = false;
> > +
> > +	numports = vidsw->num_pads;
> > +
> > +	for (i = 0; i < numports - 1; i++)
> > +		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
> > +	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
> > +
> > +	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
> > +	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
> > +				     vidsw->pads);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	vidsw->subdev.entity.ops = &vidsw_ops;
> > +
> > +	for_each_endpoint_of_node(node, ep) {
> > +		struct v4l2_of_endpoint endpoint;
> > +
> > +		v4l2_of_parse_endpoint(ep, &endpoint);
> > +
> > +		portno = endpoint.base.port;
> > +		if (portno >= numports - 1)
> > +			continue;
> > +
> > +		if (vidsw_endpoint_disabled(ep)) {
> > +			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
> > +			continue;
> > +		}
> > +
> > +		vidsw->endpoint[portno] = endpoint;
> > +
> > +		if (portno == vidsw->active)
> > +			active_link = true;
> > +	}
> > +
> > +	for (portno = 0; portno < numports - 1; portno++) {
> > +		if (!vidsw->endpoint[portno].base.local_node)
> > +			continue;
> > +
> > +		/* If the active input is not connected, use another */
> > +		if (!active_link) {
> > +			vidsw_set_active(vidsw, portno);
> > +			active_link = true;
> > +		}
> > +	}
> > +
> > +	return v4l2_async_register_subdev(&vidsw->subdev);
> > +}
> > +
> > +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
> > +{
> > +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> > +	struct media_pad *pad;
> > +	int ret;
> > +
> > +	if (vidsw->active == -1) {
> > +		dev_err(sd->dev, "no configuration for inactive mux\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	/*
> > +	 * Retrieve media bus configuration from the entity connected to the
> > +	 * active input
> > +	 */
> > +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> > +	if (pad) {
> > +		sd = media_entity_to_v4l2_subdev(pad->entity);
> > +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> > +		if (ret == -ENOIOCTLCMD)
> > +			pad = NULL;
> > +		else if (ret < 0) {
> > +			dev_err(sd->dev, "failed to get source configuration\n");
> > +			return ret;
> > +		}
> > +	}
> > +	if (!pad) {
> > +		/* Mirror the input side on the output side */
> > +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> > +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> > +		    cfg->type == V4L2_MBUS_BT656)
> > +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
> > +	}
> > +
> > +	return 0;
> > +}
> 
> I am not certain this op is needed at all. In the current kernel this op is only
> used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
> information should come from the device tree and there should be no need for this op.
> 
> My (tentative) long-term plan was to get rid of this op.
> 
> If you don't need it, then I recommend it is removed.

We currently use this to make the CSI capture interface understand
whether its source from the MIPI CSI-2 or from the parallel bus. That is
probably something that should be fixed, but I'm not quite sure how.

The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
MIPI specific metadata (virtual channel, data type).

Then the CSI2IPU gasket turns this input bus into four separate parallel
16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
carries the MIPI metadata. The incoming data is split into the four
outputs according to the MIPI virtual channel.

Two of these 16-bit + 8-bit parallel buses are routed through a
multiplexer before finally arriving at the CSI on the other side.

We need to configure the CSI to either use or ignore the data from the
8-bit mct_di bus depending on whether the source of the mux is
configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
input.

Currently we let g_mbus_config pretend that even the internal 32-bit +
metadata and 16-bit + 8-bit metadata parallel buses are of type
V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
CSI-2 receiver, if connected.

Without g_mbus_config we'd need to get that information from somewhere
else. One possibility would be to extend MEDIA_BUS formats to describe
these "parallelized MIPI data" buses separately.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-24 12:02       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 12:02 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

Hi Hans,

On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> > 
> > This driver can handle SoC internal and external video bus multiplexers,
> > controlled either by register bit fields or by a GPIO. The subdevice
> > passes through frame interval and mbus configuration of the active input
> > to the output side.
> > 
> > Signed-off-by: Sascha Hauer <s.hauer-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> > Signed-off-by: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> > 
> > --
> > 
> > - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
> >   should be unregister.
> > 
> > - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >   to vidsw_remove().
> > 
> > - there was a line left over from a previous iteration that negated
> >   the new way of determining the pad count just before it which
> >   has been removed (num_pads = of_get_child_count(np)).
> > 
> > - Philipp Zabel has developed a set of patches that allow adding
> >   to the subdev async notifier waiting list using a chaining method
> >   from the async registered callbacks (v4l2_of_subdev_registered()
> >   and the prep patches for that). For now, I've removed the use of
> >   v4l2_of_subdev_registered() for the vidmux driver's registered
> >   callback. This doesn't affect the functionality of this driver,
> >   but allows for it to be merged now, before adding the chaining
> >   support.
> > 
> > Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
> > ---
> >  .../bindings/media/video-multiplexer.txt           |  59 +++
> >  drivers/media/platform/Kconfig                     |   8 +
> >  drivers/media/platform/Makefile                    |   2 +
> >  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
> >  4 files changed, 541 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
> >  create mode 100644 drivers/media/platform/video-multiplexer.c
> > 
> > diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > new file mode 100644
> > index 0000000..9d133d9
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > @@ -0,0 +1,59 @@
> > +Video Multiplexer
> > +=================
> > +
> > +Video multiplexers allow to select between multiple input ports. Video received
> > +on the active input port is passed through to the output port. Muxes described
> > +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> > +
> > +Required properties:
> > +- compatible : should be "video-multiplexer"
> > +- reg: should be register base of the register containing the control bitfield
> > +- bit-mask: bitmask of the control bitfield in the control register
> > +- bit-shift: bit offset of the control bitfield in the control register
> > +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> > +  may be given to switch between two inputs
> > +- #address-cells: should be <1>
> > +- #size-cells: should be <0>
> > +- port@*: at least three port nodes containing endpoints connecting to the
> > +  source and sink devices according to of_graph bindings. The last port is
> > +  the output port, all others are inputs.
> > +
> > +Example:
> > +
> > +syscon {
> > +	compatible = "syscon", "simple-mfd";
> > +
> > +	mux {
> > +		compatible = "video-multiplexer";
> > +		/* Single bit (1 << 19) in syscon register 0x04: */
> > +		reg = <0x04>;
> > +		bit-mask = <1>;
> > +		bit-shift = <19>;
> > +		#address-cells = <1>;
> > +		#size-cells = <0>;
> > +
> > +		port@0 {
> > +			reg = <0>;
> > +
> > +			mux_in0: endpoint {
> > +				remote-endpoint = <&video_source0_out>;
> > +			};
> > +		};
> > +
> > +		port@1 {
> > +			reg = <1>;
> > +
> > +			mux_in1: endpoint {
> > +				remote-endpoint = <&video_source1_out>;
> > +			};
> > +		};
> > +
> > +		port@2 {
> > +			reg = <2>;
> > +
> > +			mux_out: endpoint {
> > +				remote-endpoint = <&capture_interface_in>;
> > +			};
> > +		};
> > +	};
> > +};
> > diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> > index d944421..65614b5 100644
> > --- a/drivers/media/platform/Kconfig
> > +++ b/drivers/media/platform/Kconfig
> > @@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
> >  	  To compile this driver as a module, choose M here: the
> >  	  module will be called arv.
> >  
> > +config VIDEO_MULTIPLEXER
> > +	tristate "Video Multiplexer"
> > +	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> > +	help
> > +	  This driver provides support for SoC internal N:1 video bus
> > +	  multiplexers controlled by register bitfields as well as external
> > +	  2:1 video multiplexers controlled by a single GPIO.
> > +
> >  config VIDEO_OMAP3
> >  	tristate "OMAP 3 Camera support"
> >  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
> > diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> > index 5b3cb27..7cf0ee5 100644
> > --- a/drivers/media/platform/Makefile
> > +++ b/drivers/media/platform/Makefile
> > @@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
> >  
> >  obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
> >  
> > +obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
> > +
> >  obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
> >  obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
> >  obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
> > diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
> > new file mode 100644
> > index 0000000..48980c4
> > --- /dev/null
> > +++ b/drivers/media/platform/video-multiplexer.c
> > @@ -0,0 +1,472 @@
> > +/*
> > + * video stream multiplexer controlled via gpio or syscon
> > + *
> > + * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> > + * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> > + *
> > + * This program is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU General Public License
> > + * as published by the Free Software Foundation; either version 2
> > + * of the License, or (at your option) any later version.
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <linux/err.h>
> > +#include <linux/gpio/consumer.h>
> > +#include <linux/mfd/syscon.h>
> > +#include <linux/module.h>
> > +#include <linux/of.h>
> > +#include <linux/of_graph.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/regmap.h>
> > +#include <media/v4l2-async.h>
> > +#include <media/v4l2-device.h>
> > +#include <media/v4l2-subdev.h>
> > +#include <media/v4l2-of.h>
> > +
> > +struct vidsw {
> > +	struct v4l2_subdev subdev;
> > +	unsigned int num_pads;
> > +	struct media_pad *pads;
> > +	struct v4l2_mbus_framefmt *format_mbus;
> > +	struct v4l2_fract timeperframe;
> > +	struct v4l2_of_endpoint *endpoint;
> > +	struct regmap_field *field;
> > +	struct gpio_desc *gpio;
> > +	int active;
> > +};
> > +
> > +static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
> > +{
> > +	return container_of(sd, struct vidsw, subdev);
> > +}
> > +
> > +static void vidsw_set_active(struct vidsw *vidsw, int active)
> > +{
> > +	vidsw->active = active;
> > +	if (active < 0)
> > +		return;
> > +
> > +	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
> > +
> > +	if (vidsw->field)
> > +		regmap_field_write(vidsw->field, active);
> > +	else if (vidsw->gpio)
> > +		gpiod_set_value(vidsw->gpio, active);
> > +}
> > +
> > +static int vidsw_link_setup(struct media_entity *entity,
> > +			    const struct media_pad *local,
> > +			    const struct media_pad *remote, u32 flags)
> > +{
> > +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> > +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> > +
> > +	/* We have no limitations on enabling or disabling our output link */
> > +	if (local->index == vidsw->num_pads - 1)
> > +		return 0;
> > +
> > +	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
> > +		local->entity->name);
> > +
> > +	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
> > +		if (local->index == vidsw->active) {
> > +			dev_dbg(sd->dev, "going inactive\n");
> > +			vidsw->active = -1;
> > +		}
> > +		return 0;
> > +	}
> > +
> > +	if (vidsw->active >= 0) {
> > +		struct media_pad *pad;
> > +
> > +		if (vidsw->active == local->index)
> > +			return 0;
> > +
> > +		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> > +		if (pad) {
> > +			struct media_link *link;
> > +			int ret;
> > +
> > +			link = media_entity_find_link(pad,
> > +						&vidsw->pads[vidsw->active]);
> > +			if (link) {
> > +				ret = __media_entity_setup_link(link, 0);
> > +				if (ret)
> > +					return ret;
> > +			}
> > +		}
> > +	}
> > +
> > +	vidsw_set_active(vidsw, local->index);
> > +
> > +	return 0;
> > +}
> > +
> > +static struct media_entity_operations vidsw_ops = {
> > +	.link_setup = vidsw_link_setup,
> > +};
> > +
> > +static bool vidsw_endpoint_disabled(struct device_node *ep)
> > +{
> > +	struct device_node *rpp;
> > +
> > +	if (!of_device_is_available(ep))
> > +		return true;
> > +
> > +	rpp = of_graph_get_remote_port_parent(ep);
> > +	if (!rpp)
> > +		return true;
> > +
> > +	return !of_device_is_available(rpp);
> > +}
> > +
> > +static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
> > +{
> > +	struct device_node *ep;
> > +	u32 portno;
> > +	int numports;
> > +	int ret;
> > +	int i;
> > +	bool active_link = false;
> > +
> > +	numports = vidsw->num_pads;
> > +
> > +	for (i = 0; i < numports - 1; i++)
> > +		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
> > +	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
> > +
> > +	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
> > +	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
> > +				     vidsw->pads);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	vidsw->subdev.entity.ops = &vidsw_ops;
> > +
> > +	for_each_endpoint_of_node(node, ep) {
> > +		struct v4l2_of_endpoint endpoint;
> > +
> > +		v4l2_of_parse_endpoint(ep, &endpoint);
> > +
> > +		portno = endpoint.base.port;
> > +		if (portno >= numports - 1)
> > +			continue;
> > +
> > +		if (vidsw_endpoint_disabled(ep)) {
> > +			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
> > +			continue;
> > +		}
> > +
> > +		vidsw->endpoint[portno] = endpoint;
> > +
> > +		if (portno == vidsw->active)
> > +			active_link = true;
> > +	}
> > +
> > +	for (portno = 0; portno < numports - 1; portno++) {
> > +		if (!vidsw->endpoint[portno].base.local_node)
> > +			continue;
> > +
> > +		/* If the active input is not connected, use another */
> > +		if (!active_link) {
> > +			vidsw_set_active(vidsw, portno);
> > +			active_link = true;
> > +		}
> > +	}
> > +
> > +	return v4l2_async_register_subdev(&vidsw->subdev);
> > +}
> > +
> > +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
> > +{
> > +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> > +	struct media_pad *pad;
> > +	int ret;
> > +
> > +	if (vidsw->active == -1) {
> > +		dev_err(sd->dev, "no configuration for inactive mux\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	/*
> > +	 * Retrieve media bus configuration from the entity connected to the
> > +	 * active input
> > +	 */
> > +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> > +	if (pad) {
> > +		sd = media_entity_to_v4l2_subdev(pad->entity);
> > +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> > +		if (ret == -ENOIOCTLCMD)
> > +			pad = NULL;
> > +		else if (ret < 0) {
> > +			dev_err(sd->dev, "failed to get source configuration\n");
> > +			return ret;
> > +		}
> > +	}
> > +	if (!pad) {
> > +		/* Mirror the input side on the output side */
> > +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> > +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> > +		    cfg->type == V4L2_MBUS_BT656)
> > +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
> > +	}
> > +
> > +	return 0;
> > +}
> 
> I am not certain this op is needed at all. In the current kernel this op is only
> used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
> information should come from the device tree and there should be no need for this op.
> 
> My (tentative) long-term plan was to get rid of this op.
> 
> If you don't need it, then I recommend it is removed.

We currently use this to make the CSI capture interface understand
whether its source from the MIPI CSI-2 or from the parallel bus. That is
probably something that should be fixed, but I'm not quite sure how.

The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
MIPI specific metadata (virtual channel, data type).

Then the CSI2IPU gasket turns this input bus into four separate parallel
16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
carries the MIPI metadata. The incoming data is split into the four
outputs according to the MIPI virtual channel.

Two of these 16-bit + 8-bit parallel buses are routed through a
multiplexer before finally arriving at the CSI on the other side.

We need to configure the CSI to either use or ignore the data from the
8-bit mct_di bus depending on whether the source of the mux is
configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
input.

Currently we let g_mbus_config pretend that even the internal 32-bit +
metadata and 16-bit + 8-bit metadata parallel buses are of type
V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
CSI-2 receiver, if connected.

Without g_mbus_config we'd need to get that information from somewhere
else. One possibility would be to extend MEDIA_BUS formats to describe
these "parallelized MIPI data" buses separately.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-24 12:02       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-24 12:02 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Hans,

On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > This driver can handle SoC internal and external video bus multiplexers,
> > controlled either by register bit fields or by a GPIO. The subdevice
> > passes through frame interval and mbus configuration of the active input
> > to the output side.
> > 
> > Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > --
> > 
> > - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
> >   should be unregister.
> > 
> > - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >   to vidsw_remove().
> > 
> > - there was a line left over from a previous iteration that negated
> >   the new way of determining the pad count just before it which
> >   has been removed (num_pads = of_get_child_count(np)).
> > 
> > - Philipp Zabel has developed a set of patches that allow adding
> >   to the subdev async notifier waiting list using a chaining method
> >   from the async registered callbacks (v4l2_of_subdev_registered()
> >   and the prep patches for that). For now, I've removed the use of
> >   v4l2_of_subdev_registered() for the vidmux driver's registered
> >   callback. This doesn't affect the functionality of this driver,
> >   but allows for it to be merged now, before adding the chaining
> >   support.
> > 
> > Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> > ---
> >  .../bindings/media/video-multiplexer.txt           |  59 +++
> >  drivers/media/platform/Kconfig                     |   8 +
> >  drivers/media/platform/Makefile                    |   2 +
> >  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
> >  4 files changed, 541 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
> >  create mode 100644 drivers/media/platform/video-multiplexer.c
> > 
> > diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > new file mode 100644
> > index 0000000..9d133d9
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > @@ -0,0 +1,59 @@
> > +Video Multiplexer
> > +=================
> > +
> > +Video multiplexers allow to select between multiple input ports. Video received
> > +on the active input port is passed through to the output port. Muxes described
> > +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> > +
> > +Required properties:
> > +- compatible : should be "video-multiplexer"
> > +- reg: should be register base of the register containing the control bitfield
> > +- bit-mask: bitmask of the control bitfield in the control register
> > +- bit-shift: bit offset of the control bitfield in the control register
> > +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> > +  may be given to switch between two inputs
> > +- #address-cells: should be <1>
> > +- #size-cells: should be <0>
> > +- port@*: at least three port nodes containing endpoints connecting to the
> > +  source and sink devices according to of_graph bindings. The last port is
> > +  the output port, all others are inputs.
> > +
> > +Example:
> > +
> > +syscon {
> > +	compatible = "syscon", "simple-mfd";
> > +
> > +	mux {
> > +		compatible = "video-multiplexer";
> > +		/* Single bit (1 << 19) in syscon register 0x04: */
> > +		reg = <0x04>;
> > +		bit-mask = <1>;
> > +		bit-shift = <19>;
> > +		#address-cells = <1>;
> > +		#size-cells = <0>;
> > +
> > +		port at 0 {
> > +			reg = <0>;
> > +
> > +			mux_in0: endpoint {
> > +				remote-endpoint = <&video_source0_out>;
> > +			};
> > +		};
> > +
> > +		port at 1 {
> > +			reg = <1>;
> > +
> > +			mux_in1: endpoint {
> > +				remote-endpoint = <&video_source1_out>;
> > +			};
> > +		};
> > +
> > +		port at 2 {
> > +			reg = <2>;
> > +
> > +			mux_out: endpoint {
> > +				remote-endpoint = <&capture_interface_in>;
> > +			};
> > +		};
> > +	};
> > +};
> > diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> > index d944421..65614b5 100644
> > --- a/drivers/media/platform/Kconfig
> > +++ b/drivers/media/platform/Kconfig
> > @@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
> >  	  To compile this driver as a module, choose M here: the
> >  	  module will be called arv.
> >  
> > +config VIDEO_MULTIPLEXER
> > +	tristate "Video Multiplexer"
> > +	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> > +	help
> > +	  This driver provides support for SoC internal N:1 video bus
> > +	  multiplexers controlled by register bitfields as well as external
> > +	  2:1 video multiplexers controlled by a single GPIO.
> > +
> >  config VIDEO_OMAP3
> >  	tristate "OMAP 3 Camera support"
> >  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
> > diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> > index 5b3cb27..7cf0ee5 100644
> > --- a/drivers/media/platform/Makefile
> > +++ b/drivers/media/platform/Makefile
> > @@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
> >  
> >  obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
> >  
> > +obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
> > +
> >  obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
> >  obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
> >  obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
> > diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
> > new file mode 100644
> > index 0000000..48980c4
> > --- /dev/null
> > +++ b/drivers/media/platform/video-multiplexer.c
> > @@ -0,0 +1,472 @@
> > +/*
> > + * video stream multiplexer controlled via gpio or syscon
> > + *
> > + * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
> > + * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
> > + *
> > + * This program is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU General Public License
> > + * as published by the Free Software Foundation; either version 2
> > + * of the License, or (at your option) any later version.
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <linux/err.h>
> > +#include <linux/gpio/consumer.h>
> > +#include <linux/mfd/syscon.h>
> > +#include <linux/module.h>
> > +#include <linux/of.h>
> > +#include <linux/of_graph.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/regmap.h>
> > +#include <media/v4l2-async.h>
> > +#include <media/v4l2-device.h>
> > +#include <media/v4l2-subdev.h>
> > +#include <media/v4l2-of.h>
> > +
> > +struct vidsw {
> > +	struct v4l2_subdev subdev;
> > +	unsigned int num_pads;
> > +	struct media_pad *pads;
> > +	struct v4l2_mbus_framefmt *format_mbus;
> > +	struct v4l2_fract timeperframe;
> > +	struct v4l2_of_endpoint *endpoint;
> > +	struct regmap_field *field;
> > +	struct gpio_desc *gpio;
> > +	int active;
> > +};
> > +
> > +static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
> > +{
> > +	return container_of(sd, struct vidsw, subdev);
> > +}
> > +
> > +static void vidsw_set_active(struct vidsw *vidsw, int active)
> > +{
> > +	vidsw->active = active;
> > +	if (active < 0)
> > +		return;
> > +
> > +	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
> > +
> > +	if (vidsw->field)
> > +		regmap_field_write(vidsw->field, active);
> > +	else if (vidsw->gpio)
> > +		gpiod_set_value(vidsw->gpio, active);
> > +}
> > +
> > +static int vidsw_link_setup(struct media_entity *entity,
> > +			    const struct media_pad *local,
> > +			    const struct media_pad *remote, u32 flags)
> > +{
> > +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> > +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> > +
> > +	/* We have no limitations on enabling or disabling our output link */
> > +	if (local->index == vidsw->num_pads - 1)
> > +		return 0;
> > +
> > +	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
> > +		local->entity->name);
> > +
> > +	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
> > +		if (local->index == vidsw->active) {
> > +			dev_dbg(sd->dev, "going inactive\n");
> > +			vidsw->active = -1;
> > +		}
> > +		return 0;
> > +	}
> > +
> > +	if (vidsw->active >= 0) {
> > +		struct media_pad *pad;
> > +
> > +		if (vidsw->active == local->index)
> > +			return 0;
> > +
> > +		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> > +		if (pad) {
> > +			struct media_link *link;
> > +			int ret;
> > +
> > +			link = media_entity_find_link(pad,
> > +						&vidsw->pads[vidsw->active]);
> > +			if (link) {
> > +				ret = __media_entity_setup_link(link, 0);
> > +				if (ret)
> > +					return ret;
> > +			}
> > +		}
> > +	}
> > +
> > +	vidsw_set_active(vidsw, local->index);
> > +
> > +	return 0;
> > +}
> > +
> > +static struct media_entity_operations vidsw_ops = {
> > +	.link_setup = vidsw_link_setup,
> > +};
> > +
> > +static bool vidsw_endpoint_disabled(struct device_node *ep)
> > +{
> > +	struct device_node *rpp;
> > +
> > +	if (!of_device_is_available(ep))
> > +		return true;
> > +
> > +	rpp = of_graph_get_remote_port_parent(ep);
> > +	if (!rpp)
> > +		return true;
> > +
> > +	return !of_device_is_available(rpp);
> > +}
> > +
> > +static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
> > +{
> > +	struct device_node *ep;
> > +	u32 portno;
> > +	int numports;
> > +	int ret;
> > +	int i;
> > +	bool active_link = false;
> > +
> > +	numports = vidsw->num_pads;
> > +
> > +	for (i = 0; i < numports - 1; i++)
> > +		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
> > +	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
> > +
> > +	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
> > +	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
> > +				     vidsw->pads);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	vidsw->subdev.entity.ops = &vidsw_ops;
> > +
> > +	for_each_endpoint_of_node(node, ep) {
> > +		struct v4l2_of_endpoint endpoint;
> > +
> > +		v4l2_of_parse_endpoint(ep, &endpoint);
> > +
> > +		portno = endpoint.base.port;
> > +		if (portno >= numports - 1)
> > +			continue;
> > +
> > +		if (vidsw_endpoint_disabled(ep)) {
> > +			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
> > +			continue;
> > +		}
> > +
> > +		vidsw->endpoint[portno] = endpoint;
> > +
> > +		if (portno == vidsw->active)
> > +			active_link = true;
> > +	}
> > +
> > +	for (portno = 0; portno < numports - 1; portno++) {
> > +		if (!vidsw->endpoint[portno].base.local_node)
> > +			continue;
> > +
> > +		/* If the active input is not connected, use another */
> > +		if (!active_link) {
> > +			vidsw_set_active(vidsw, portno);
> > +			active_link = true;
> > +		}
> > +	}
> > +
> > +	return v4l2_async_register_subdev(&vidsw->subdev);
> > +}
> > +
> > +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
> > +{
> > +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> > +	struct media_pad *pad;
> > +	int ret;
> > +
> > +	if (vidsw->active == -1) {
> > +		dev_err(sd->dev, "no configuration for inactive mux\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	/*
> > +	 * Retrieve media bus configuration from the entity connected to the
> > +	 * active input
> > +	 */
> > +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> > +	if (pad) {
> > +		sd = media_entity_to_v4l2_subdev(pad->entity);
> > +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> > +		if (ret == -ENOIOCTLCMD)
> > +			pad = NULL;
> > +		else if (ret < 0) {
> > +			dev_err(sd->dev, "failed to get source configuration\n");
> > +			return ret;
> > +		}
> > +	}
> > +	if (!pad) {
> > +		/* Mirror the input side on the output side */
> > +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> > +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> > +		    cfg->type == V4L2_MBUS_BT656)
> > +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
> > +	}
> > +
> > +	return 0;
> > +}
> 
> I am not certain this op is needed at all. In the current kernel this op is only
> used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
> information should come from the device tree and there should be no need for this op.
> 
> My (tentative) long-term plan was to get rid of this op.
> 
> If you don't need it, then I recommend it is removed.

We currently use this to make the CSI capture interface understand
whether its source from the MIPI CSI-2 or from the parallel bus. That is
probably something that should be fixed, but I'm not quite sure how.

The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
MIPI specific metadata (virtual channel, data type).

Then the CSI2IPU gasket turns this input bus into four separate parallel
16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
carries the MIPI metadata. The incoming data is split into the four
outputs according to the MIPI virtual channel.

Two of these 16-bit + 8-bit parallel buses are routed through a
multiplexer before finally arriving at the CSI on the other side.

We need to configure the CSI to either use or ignore the data from the
8-bit mct_di bus depending on whether the source of the mux is
configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
input.

Currently we let g_mbus_config pretend that even the internal 32-bit +
metadata and 16-bit + 8-bit metadata parallel buses are of type
V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
CSI-2 receiver, if connected.

Without g_mbus_config we'd need to get that information from somewhere
else. One possibility would be to extend MEDIA_BUS formats to describe
these "parallelized MIPI data" buses separately.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
  (?)
@ 2017-01-24 12:44     ` Javier Martinez Canillas
  -1 siblings, 0 replies; 549+ messages in thread
From: Javier Martinez Canillas @ 2017-01-24 12:44 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King, Mauro Carvalho Chehab, Hans Verkuil,
	nick, Markus Heiser, Philipp Zabel, Laurent Pinchart,
	Benoit Parrot, Geert Uytterhoeven, Arnd Bergmann,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	Jean-Christophe Trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, Greg Kroah-Hartman,
	devel, devicetree, Steve Longerbeam, Sascha Hauer, Linux Kernel,
	linux-arm-kernel, Linux Media Mailing List

Hello Steve,

On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>

[snip]

>
> +config VIDEO_MULTIPLEXER
> +       tristate "Video Multiplexer"
> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER

The driver can be build as a module...

> +
> +static const struct of_device_id vidsw_dt_ids[] = {
> +       { .compatible = "video-multiplexer", },
> +       { /* sentinel */ }
> +};
> +

... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
otherwise module autoloading won't work.

Best regards,
Javier

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-24 12:44     ` Javier Martinez Canillas
  0 siblings, 0 replies; 549+ messages in thread
From: Javier Martinez Canillas @ 2017-01-24 12:44 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King, Mauro Carvalho Chehab, Hans Verkuil,
	nick, Markus Heiser, Philipp Zabel, Laurent Pinchart,
	Benoit Parrot, Geert Uytterhoeven, Arnd Bergmann,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	Jean-Christophe Trotin, horms+renesas, niklas.soderlund+renesas

Hello Steve,

On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>

[snip]

>
> +config VIDEO_MULTIPLEXER
> +       tristate "Video Multiplexer"
> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER

The driver can be build as a module...

> +
> +static const struct of_device_id vidsw_dt_ids[] = {
> +       { .compatible = "video-multiplexer", },
> +       { /* sentinel */ }
> +};
> +

... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
otherwise module autoloading won't work.

Best regards,
Javier

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-24 12:44     ` Javier Martinez Canillas
  0 siblings, 0 replies; 549+ messages in thread
From: Javier Martinez Canillas @ 2017-01-24 12:44 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King, Mauro Carvalho Chehab, Hans Verkuil,
	nick, Markus Heiser, Philipp Zabel, Laurent Pinchart,
	Benoit Parrot, Geert Uytterhoeven, Arnd Bergmann,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	Jean-Christophe Trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, Greg Kroah-Hartman,
	devel, devicetree, Steve Longerbeam, Sascha Hauer, Linux Kernel,
	linux-arm-kernel, Linux Media Mailing List

Hello Steve,

On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>

[snip]

>
> +config VIDEO_MULTIPLEXER
> +       tristate "Video Multiplexer"
> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER

The driver can be build as a module...

> +
> +static const struct of_device_id vidsw_dt_ids[] = {
> +       { .compatible = "video-multiplexer", },
> +       { /* sentinel */ }
> +};
> +

... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
otherwise module autoloading won't work.

Best regards,
Javier

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-24 12:44     ` Javier Martinez Canillas
  0 siblings, 0 replies; 549+ messages in thread
From: Javier Martinez Canillas @ 2017-01-24 12:44 UTC (permalink / raw)
  To: linux-arm-kernel

Hello Steve,

On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>

[snip]

>
> +config VIDEO_MULTIPLEXER
> +       tristate "Video Multiplexer"
> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER

The driver can be build as a module...

> +
> +static const struct of_device_id vidsw_dt_ids[] = {
> +       { .compatible = "video-multiplexer", },
> +       { /* sentinel */ }
> +};
> +

... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
otherwise module autoloading won't work.

Best regards,
Javier

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-01-24 12:02       ` Philipp Zabel
  (?)
@ 2017-01-25  2:07         ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25  2:07 UTC (permalink / raw)
  To: Philipp Zabel, Hans Verkuil
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, nick, markus.heiser, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam



On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> Hi Hans,
>
> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>> <snip>
>>> +
>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
>>> +{
>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>> +	struct media_pad *pad;
>>> +	int ret;
>>> +
>>> +	if (vidsw->active == -1) {
>>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	/*
>>> +	 * Retrieve media bus configuration from the entity connected to the
>>> +	 * active input
>>> +	 */
>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>> +	if (pad) {
>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>> +		if (ret == -ENOIOCTLCMD)
>>> +			pad = NULL;
>>> +		else if (ret < 0) {
>>> +			dev_err(sd->dev, "failed to get source configuration\n");
>>> +			return ret;
>>> +		}
>>> +	}
>>> +	if (!pad) {
>>> +		/* Mirror the input side on the output side */
>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>> +		    cfg->type == V4L2_MBUS_BT656)
>>> +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>> I am not certain this op is needed at all. In the current kernel this op is only
>> used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
>> information should come from the device tree and there should be no need for this op.
>>
>> My (tentative) long-term plan was to get rid of this op.
>>
>> If you don't need it, then I recommend it is removed.

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
sensor, and it was doing that to determine the sensor's bus type. This info
was already available from parsing a v4l2_of_endpoint from the sensor node.
So it was simple to remove the g_mbus_config calls, and instead rely on the
parsed sensor v4l2_of_endpoint.

> We currently use this to make the CSI capture interface understand
> whether its source from the MIPI CSI-2 or from the parallel bus. That is
> probably something that should be fixed, but I'm not quite sure how.
>
> The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
> CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
> MIPI specific metadata (virtual channel, data type).
>
> Then the CSI2IPU gasket turns this input bus into four separate parallel
> 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
> carries the MIPI metadata. The incoming data is split into the four
> outputs according to the MIPI virtual channel.
>
> Two of these 16-bit + 8-bit parallel buses are routed through a
> multiplexer before finally arriving at the CSI on the other side.
>
> We need to configure the CSI to either use or ignore the data from the
> 8-bit mct_di bus depending on whether the source of the mux is
> configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
> input.

Philipp, from my experience, the CSI_MIPI_DI register (configured
by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0,
otherwise no data is received from the MIPI CSI-2 sensor, regardless
of the virtual channel the sensor is transmitting over.

So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU
gasket are ignored by the CSI's, at least the virtual channel number is
ignored.

For example, if the sensor is transmitting on vc 1, the gasket routes
the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1
is written with vc 1, no data is received.

Steve

>
> Currently we let g_mbus_config pretend that even the internal 32-bit +
> metadata and 16-bit + 8-bit metadata parallel buses are of type
> V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
> CSI-2 receiver, if connected.
>
> Without g_mbus_config we'd need to get that information from somewhere
> else. One possibility would be to extend MEDIA_BUS formats to describe
> these "parallelized MIPI data" buses separately.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-25  2:07         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25  2:07 UTC (permalink / raw)
  To: Philipp Zabel, Hans Verkuil
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, nick, markus.heiser, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve



On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> Hi Hans,
>
> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>> <snip>
>>> +
>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
>>> +{
>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>> +	struct media_pad *pad;
>>> +	int ret;
>>> +
>>> +	if (vidsw->active == -1) {
>>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	/*
>>> +	 * Retrieve media bus configuration from the entity connected to the
>>> +	 * active input
>>> +	 */
>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>> +	if (pad) {
>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>> +		if (ret == -ENOIOCTLCMD)
>>> +			pad = NULL;
>>> +		else if (ret < 0) {
>>> +			dev_err(sd->dev, "failed to get source configuration\n");
>>> +			return ret;
>>> +		}
>>> +	}
>>> +	if (!pad) {
>>> +		/* Mirror the input side on the output side */
>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>> +		    cfg->type == V4L2_MBUS_BT656)
>>> +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>> I am not certain this op is needed at all. In the current kernel this op is only
>> used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
>> information should come from the device tree and there should be no need for this op.
>>
>> My (tentative) long-term plan was to get rid of this op.
>>
>> If you don't need it, then I recommend it is removed.

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
sensor, and it was doing that to determine the sensor's bus type. This info
was already available from parsing a v4l2_of_endpoint from the sensor node.
So it was simple to remove the g_mbus_config calls, and instead rely on the
parsed sensor v4l2_of_endpoint.

> We currently use this to make the CSI capture interface understand
> whether its source from the MIPI CSI-2 or from the parallel bus. That is
> probably something that should be fixed, but I'm not quite sure how.
>
> The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
> CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
> MIPI specific metadata (virtual channel, data type).
>
> Then the CSI2IPU gasket turns this input bus into four separate parallel
> 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
> carries the MIPI metadata. The incoming data is split into the four
> outputs according to the MIPI virtual channel.
>
> Two of these 16-bit + 8-bit parallel buses are routed through a
> multiplexer before finally arriving at the CSI on the other side.
>
> We need to configure the CSI to either use or ignore the data from the
> 8-bit mct_di bus depending on whether the source of the mux is
> configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
> input.

Philipp, from my experience, the CSI_MIPI_DI register (configured
by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0,
otherwise no data is received from the MIPI CSI-2 sensor, regardless
of the virtual channel the sensor is transmitting over.

So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU
gasket are ignored by the CSI's, at least the virtual channel number is
ignored.

For example, if the sensor is transmitting on vc 1, the gasket routes
the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1
is written with vc 1, no data is received.

Steve

>
> Currently we let g_mbus_config pretend that even the internal 32-bit +
> metadata and 16-bit + 8-bit metadata parallel buses are of type
> V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
> CSI-2 receiver, if connected.
>
> Without g_mbus_config we'd need to get that information from somewhere
> else. One possibility would be to extend MEDIA_BUS formats to describe
> these "parallelized MIPI data" buses separately.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-25  2:07         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25  2:07 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> Hi Hans,
>
> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>> <snip>
>>> +
>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
>>> +{
>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>> +	struct media_pad *pad;
>>> +	int ret;
>>> +
>>> +	if (vidsw->active == -1) {
>>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	/*
>>> +	 * Retrieve media bus configuration from the entity connected to the
>>> +	 * active input
>>> +	 */
>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>> +	if (pad) {
>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>> +		if (ret == -ENOIOCTLCMD)
>>> +			pad = NULL;
>>> +		else if (ret < 0) {
>>> +			dev_err(sd->dev, "failed to get source configuration\n");
>>> +			return ret;
>>> +		}
>>> +	}
>>> +	if (!pad) {
>>> +		/* Mirror the input side on the output side */
>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>> +		    cfg->type == V4L2_MBUS_BT656)
>>> +			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>> I am not certain this op is needed at all. In the current kernel this op is only
>> used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this
>> information should come from the device tree and there should be no need for this op.
>>
>> My (tentative) long-term plan was to get rid of this op.
>>
>> If you don't need it, then I recommend it is removed.

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
sensor, and it was doing that to determine the sensor's bus type. This info
was already available from parsing a v4l2_of_endpoint from the sensor node.
So it was simple to remove the g_mbus_config calls, and instead rely on the
parsed sensor v4l2_of_endpoint.

> We currently use this to make the CSI capture interface understand
> whether its source from the MIPI CSI-2 or from the parallel bus. That is
> probably something that should be fixed, but I'm not quite sure how.
>
> The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
> CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
> MIPI specific metadata (virtual channel, data type).
>
> Then the CSI2IPU gasket turns this input bus into four separate parallel
> 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
> carries the MIPI metadata. The incoming data is split into the four
> outputs according to the MIPI virtual channel.
>
> Two of these 16-bit + 8-bit parallel buses are routed through a
> multiplexer before finally arriving at the CSI on the other side.
>
> We need to configure the CSI to either use or ignore the data from the
> 8-bit mct_di bus depending on whether the source of the mux is
> configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
> input.

Philipp, from my experience, the CSI_MIPI_DI register (configured
by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0,
otherwise no data is received from the MIPI CSI-2 sensor, regardless
of the virtual channel the sensor is transmitting over.

So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU
gasket are ignored by the CSI's, at least the virtual channel number is
ignored.

For example, if the sensor is transmitting on vc 1, the gasket routes
the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1
is written with vc 1, no data is received.

Steve

>
> Currently we let g_mbus_config pretend that even the internal 32-bit +
> metadata and 16-bit + 8-bit metadata parallel buses are of type
> V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
> CSI-2 receiver, if connected.
>
> Without g_mbus_config we'd need to get that information from somewhere
> else. One possibility would be to extend MEDIA_BUS formats to describe
> these "parallelized MIPI data" buses separately.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-25  2:39       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25  2:39 UTC (permalink / raw)
  To: Hans Verkuil, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam



On 01/20/2017 06:29 AM, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>> +
>> +static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
>> +	{
>> +		.id = V4L2_CID_HFLIP,
>> +		.name = "Horizontal Flip",
>> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		.def =  0,
>> +		.min =  0,
>> +		.max =  1,
>> +		.step = 1,
>> +	}, {
>> +		.id = V4L2_CID_VFLIP,
>> +		.name = "Vertical Flip",
>> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		.def =  0,
>> +		.min =  0,
>> +		.max =  1,
>> +		.step = 1,
>> +	}, {
>> +		.id = V4L2_CID_ROTATE,
>> +		.name = "Rotation",
>> +		.type = V4L2_CTRL_TYPE_INTEGER,
>> +		.def =   0,
>> +		.min =   0,
>> +		.max = 270,
>> +		.step = 90,
>> +	},
>> +};
> Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information
> like the name and type.
>
> If this is also done elsewhere, then it should be changed there as well.

done.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-25  2:39       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25  2:39 UTC (permalink / raw)
  To: Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r
  Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/20/2017 06:29 AM, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>> +
>> +static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
>> +	{
>> +		.id = V4L2_CID_HFLIP,
>> +		.name = "Horizontal Flip",
>> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		.def =  0,
>> +		.min =  0,
>> +		.max =  1,
>> +		.step = 1,
>> +	}, {
>> +		.id = V4L2_CID_VFLIP,
>> +		.name = "Vertical Flip",
>> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		.def =  0,
>> +		.min =  0,
>> +		.max =  1,
>> +		.step = 1,
>> +	}, {
>> +		.id = V4L2_CID_ROTATE,
>> +		.name = "Rotation",
>> +		.type = V4L2_CTRL_TYPE_INTEGER,
>> +		.def =   0,
>> +		.min =   0,
>> +		.max = 270,
>> +		.step = 90,
>> +	},
>> +};
> Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information
> like the name and type.
>
> If this is also done elsewhere, then it should be changed there as well.

done.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-25  2:39       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25  2:39 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/20/2017 06:29 AM, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>> +
>> +static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
>> +	{
>> +		.id = V4L2_CID_HFLIP,
>> +		.name = "Horizontal Flip",
>> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		.def =  0,
>> +		.min =  0,
>> +		.max =  1,
>> +		.step = 1,
>> +	}, {
>> +		.id = V4L2_CID_VFLIP,
>> +		.name = "Vertical Flip",
>> +		.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		.def =  0,
>> +		.min =  0,
>> +		.max =  1,
>> +		.step = 1,
>> +	}, {
>> +		.id = V4L2_CID_ROTATE,
>> +		.name = "Rotation",
>> +		.type = V4L2_CTRL_TYPE_INTEGER,
>> +		.def =   0,
>> +		.min =   0,
>> +		.max = 270,
>> +		.step = 90,
>> +	},
>> +};
> Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information
> like the name and type.
>
> If this is also done elsewhere, then it should be changed there as well.

done.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  2017-01-20 14:48     ` Hans Verkuil
@ 2017-01-25 19:10       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25 19:10 UTC (permalink / raw)
  To: Hans Verkuil, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh
  Cc: devicetree, linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam



On 01/20/2017 06:48 AM, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>> +
>> +	/* cached control settings */
>> +	int ctrl_cache[OV5640_MAX_CONTROLS];
> This is just duplicating the cached value in the control framework. I think this can be dropped.

done, see below.

>
>> +
>> +static struct ov5640_control ov5640_ctrls[] = {
>> +	{
>> +		.set = ov5640_set_agc,
>> +		.ctrl = {
>> +			.id = V4L2_CID_AUTOGAIN,
>> +			.name = "Auto Gain/Exposure Control",
>> +			.minimum = 0,
>> +			.maximum = 1,
>> +			.step = 1,
>> +			.default_value = 1,
>> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_exposure,
>> +		.ctrl = {
>> +			.id = V4L2_CID_EXPOSURE,
>> +			.name = "Exposure",
>> +			.minimum = 0,
>> +			.maximum = 65535,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_gain,
>> +		.ctrl = {
>> +			.id = V4L2_CID_GAIN,
>> +			.name = "Gain",
>> +			.minimum = 0,
>> +			.maximum = 1023,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_hue,
>> +		.ctrl = {
>> +			.id = V4L2_CID_HUE,
>> +			.name = "Hue",
>> +			.minimum = 0,
>> +			.maximum = 359,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_contrast,
>> +		.ctrl = {
>> +			.id = V4L2_CID_CONTRAST,
>> +			.name = "Contrast",
>> +			.minimum = 0,
>> +			.maximum = 255,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_saturation,
>> +		.ctrl = {
>> +			.id = V4L2_CID_SATURATION,
>> +			.name = "Saturation",
>> +			.minimum = 0,
>> +			.maximum = 255,
>> +			.step = 1,
>> +			.default_value = 64,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_awb,
>> +		.ctrl = {
>> +			.id = V4L2_CID_AUTO_WHITE_BALANCE,
>> +			.name = "Auto White Balance",
>> +			.minimum = 0,
>> +			.maximum = 1,
>> +			.step = 1,
>> +			.default_value = 1,
>> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_red_balance,
>> +		.ctrl = {
>> +			.id = V4L2_CID_RED_BALANCE,
>> +			.name = "Red Balance",
>> +			.minimum = 0,
>> +			.maximum = 4095,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_blue_balance,
>> +		.ctrl = {
>> +			.id = V4L2_CID_BLUE_BALANCE,
>> +			.name = "Blue Balance",
>> +			.minimum = 0,
>> +			.maximum = 4095,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	},
>> +};
>> +#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
> This should use v4l2_ctrl_new_std() instead of this array.
> Just put a switch on ctrl->id in s_ctrl, and each case calls the corresponding
> set function.

In this case, because there are lots of controls, my preference is to use
table lookup rather than a large switch statement. However I did remove
.name and .type from the table entries, leaving only .def, .min, .max, .step
as required to pass to v4l2_ctrl_new_std(). Also converted to 'struct 
v4l2_ctrl_config'
in the table.


>
>> +
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +		c->set(sensor, sensor->ctrl_cache[i]);
>> +	}
>> +
>> +	return 0;
>> +}
> This does the same as v4l2_ctrl_handler_setup() if I understand the code correctly.

yes thanks. I remember looking at this and thinking 
v4l2_ctrl_handler_setup()
was setting up the default values rather than the current values, but 
after a
re-read it does look to be restoring the current values, which is 
exactly what
is needed here.

>> +
>> +static int ov5640_init_controls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
>> +
>> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +
>> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
>> +				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
>> +				  c->ctrl.step, c->ctrl.default_value);
>> +	}
> As mentioned, just drop the ov5640_ctrls array and call v4l2_ctr_new_std for each
> control you're adding.

if really pressed I can be persuaded to use a switch statement and call
v4l2_ctrl_new_std() multiple times, but I don't any problem with using
a table.

>> +
>> +module_i2c_driver(ov5640_i2c_driver);
>> +
>> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
>> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
>> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
>> +MODULE_LICENSE("GPL");
>> +MODULE_VERSION("1.0");
>>
> Same comments apply to the next patch, so I won't repeat them.

done, I've made the same mods to the ov5642 subdev.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-25 19:10       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-25 19:10 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/20/2017 06:48 AM, Hans Verkuil wrote:
> On 01/07/2017 03:11 AM, Steve Longerbeam wrote:
>> +
>> +	/* cached control settings */
>> +	int ctrl_cache[OV5640_MAX_CONTROLS];
> This is just duplicating the cached value in the control framework. I think this can be dropped.

done, see below.

>
>> +
>> +static struct ov5640_control ov5640_ctrls[] = {
>> +	{
>> +		.set = ov5640_set_agc,
>> +		.ctrl = {
>> +			.id = V4L2_CID_AUTOGAIN,
>> +			.name = "Auto Gain/Exposure Control",
>> +			.minimum = 0,
>> +			.maximum = 1,
>> +			.step = 1,
>> +			.default_value = 1,
>> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_exposure,
>> +		.ctrl = {
>> +			.id = V4L2_CID_EXPOSURE,
>> +			.name = "Exposure",
>> +			.minimum = 0,
>> +			.maximum = 65535,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_gain,
>> +		.ctrl = {
>> +			.id = V4L2_CID_GAIN,
>> +			.name = "Gain",
>> +			.minimum = 0,
>> +			.maximum = 1023,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_hue,
>> +		.ctrl = {
>> +			.id = V4L2_CID_HUE,
>> +			.name = "Hue",
>> +			.minimum = 0,
>> +			.maximum = 359,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_contrast,
>> +		.ctrl = {
>> +			.id = V4L2_CID_CONTRAST,
>> +			.name = "Contrast",
>> +			.minimum = 0,
>> +			.maximum = 255,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_saturation,
>> +		.ctrl = {
>> +			.id = V4L2_CID_SATURATION,
>> +			.name = "Saturation",
>> +			.minimum = 0,
>> +			.maximum = 255,
>> +			.step = 1,
>> +			.default_value = 64,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_awb,
>> +		.ctrl = {
>> +			.id = V4L2_CID_AUTO_WHITE_BALANCE,
>> +			.name = "Auto White Balance",
>> +			.minimum = 0,
>> +			.maximum = 1,
>> +			.step = 1,
>> +			.default_value = 1,
>> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_red_balance,
>> +		.ctrl = {
>> +			.id = V4L2_CID_RED_BALANCE,
>> +			.name = "Red Balance",
>> +			.minimum = 0,
>> +			.maximum = 4095,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	}, {
>> +		.set = ov5640_set_blue_balance,
>> +		.ctrl = {
>> +			.id = V4L2_CID_BLUE_BALANCE,
>> +			.name = "Blue Balance",
>> +			.minimum = 0,
>> +			.maximum = 4095,
>> +			.step = 1,
>> +			.default_value = 0,
>> +			.type = V4L2_CTRL_TYPE_INTEGER,
>> +		},
>> +	},
>> +};
>> +#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
> This should use v4l2_ctrl_new_std() instead of this array.
> Just put a switch on ctrl->id in s_ctrl, and each case calls the corresponding
> set function.

In this case, because there are lots of controls, my preference is to use
table lookup rather than a large switch statement. However I did remove
.name and .type from the table entries, leaving only .def, .min, .max, .step
as required to pass to v4l2_ctrl_new_std(). Also converted to 'struct 
v4l2_ctrl_config'
in the table.


>
>> +
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +		c->set(sensor, sensor->ctrl_cache[i]);
>> +	}
>> +
>> +	return 0;
>> +}
> This does the same as v4l2_ctrl_handler_setup() if I understand the code correctly.

yes thanks. I remember looking at this and thinking 
v4l2_ctrl_handler_setup()
was setting up the default values rather than the current values, but 
after a
re-read it does look to be restoring the current values, which is 
exactly what
is needed here.

>> +
>> +static int ov5640_init_controls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
>> +
>> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +
>> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
>> +				  c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
>> +				  c->ctrl.step, c->ctrl.default_value);
>> +	}
> As mentioned, just drop the ov5640_ctrls array and call v4l2_ctr_new_std for each
> control you're adding.

if really pressed I can be persuaded to use a switch statement and call
v4l2_ctrl_new_std() multiple times, but I don't any problem with using
a table.

>> +
>> +module_i2c_driver(ov5640_i2c_driver);
>> +
>> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
>> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
>> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
>> +MODULE_LICENSE("GPL");
>> +MODULE_VERSION("1.0");
>>
> Same comments apply to the next patch, so I won't repeat them.

done, I've made the same mods to the ov5642 subdev.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-26  1:22       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-26  1:22 UTC (permalink / raw)
  To: Javier Martinez Canillas
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King, Mauro Carvalho Chehab, Hans Verkuil,
	nick, Markus Heiser, Philipp Zabel, Laurent Pinchart,
	Benoit Parrot, Geert Uytterhoeven, Arnd Bergmann,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	Jean-Christophe Trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, Greg Kroah-Hartman,
	devel, devicetree, Steve Longerbeam, Sascha Hauer, Linux Kernel,
	linux-arm-kernel, Linux Media Mailing List



On 01/24/2017 04:44 AM, Javier Martinez Canillas wrote:
> Hello Steve,
>
> On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> From: Philipp Zabel <p.zabel@pengutronix.de>
> [snip]
>
>> +config VIDEO_MULTIPLEXER
>> +       tristate "Video Multiplexer"
>> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> The driver can be build as a module...
>
>> +
>> +static const struct of_device_id vidsw_dt_ids[] = {
>> +       { .compatible = "video-multiplexer", },
>> +       { /* sentinel */ }
>> +};
>> +
> ... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
> otherwise module autoloading won't work.

Hi Javier, thanks for catching, done.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-26  1:22       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-26  1:22 UTC (permalink / raw)
  To: Javier Martinez Canillas
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King, Mauro Carvalho Chehab, Hans Verkuil,
	nick-gcszYUEDH4VrovVCs/uTlw, Markus Heiser, Philipp Zabel,
	Laurent Pinchart, Benoit Parrot, Geert Uytterhoeven,
	Arnd Bergmann, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w, Jean-Christophe Trotin,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ, niklas.soderlund+renesas



On 01/24/2017 04:44 AM, Javier Martinez Canillas wrote:
> Hello Steve,
>
> On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> From: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> [snip]
>
>> +config VIDEO_MULTIPLEXER
>> +       tristate "Video Multiplexer"
>> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> The driver can be build as a module...
>
>> +
>> +static const struct of_device_id vidsw_dt_ids[] = {
>> +       { .compatible = "video-multiplexer", },
>> +       { /* sentinel */ }
>> +};
>> +
> ... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
> otherwise module autoloading won't work.

Hi Javier, thanks for catching, done.

Steve


--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-26  1:22       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-26  1:22 UTC (permalink / raw)
  To: Javier Martinez Canillas
  Cc: Rob Herring, Mark Rutland, Shawn Guo, Sascha Hauer,
	Fabio Estevam, Russell King, Mauro Carvalho Chehab, Hans Verkuil,
	nick, Markus Heiser, Philipp Zabel, Laurent Pinchart,
	Benoit Parrot, Geert Uytterhoeven, Arnd Bergmann,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	Jean-Christophe Trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, Greg Kroah-Hartman,
	devel, devicetree, Steve Longerbeam, Sascha Hauer, Linux Kernel,
	linux-arm-kernel, Linux Media Mailing List



On 01/24/2017 04:44 AM, Javier Martinez Canillas wrote:
> Hello Steve,
>
> On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> From: Philipp Zabel <p.zabel@pengutronix.de>
> [snip]
>
>> +config VIDEO_MULTIPLEXER
>> +       tristate "Video Multiplexer"
>> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> The driver can be build as a module...
>
>> +
>> +static const struct of_device_id vidsw_dt_ids[] = {
>> +       { .compatible = "video-multiplexer", },
>> +       { /* sentinel */ }
>> +};
>> +
> ... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
> otherwise module autoloading won't work.

Hi Javier, thanks for catching, done.

Steve



^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-01-26  1:22       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-26  1:22 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/24/2017 04:44 AM, Javier Martinez Canillas wrote:
> Hello Steve,
>
> On Fri, Jan 6, 2017 at 11:11 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> From: Philipp Zabel <p.zabel@pengutronix.de>
> [snip]
>
>> +config VIDEO_MULTIPLEXER
>> +       tristate "Video Multiplexer"
>> +       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
> The driver can be build as a module...
>
>> +
>> +static const struct of_device_id vidsw_dt_ids[] = {
>> +       { .compatible = "video-multiplexer", },
>> +       { /* sentinel */ }
>> +};
>> +
> ... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or
> otherwise module autoloading won't work.

Hi Javier, thanks for catching, done.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-28 19:27                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-28 19:27 UTC (permalink / raw)
  To: Philipp Zabel, Hans Verkuil
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, nick, markus.heiser, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam, Laurent Pinchart



On 01/24/2017 03:27 AM, Philipp Zabel wrote:
> Hi Steve, Hans,
>
> [added Laurent to Cc: who I believe might have an opinion on the media
> bus formats, too. Sorry for the wall of text, I have put a marker where
> the MEDIA_BUS argument starts]
>
> The central issue seems to be that I think media pad links / media bus
> formats should describe physical links, such as parallel or serial
> buses, and the formats of pixels flowing through them, whereas Steve
> would like to extend them to describe software transports and in-memory
> formats.

Hi Philipp, Hans,

I've decided to pull the dma read/write channel linking between
pads. Although I haven't heard any other feedback yet, I agree it is
controversial and it is fairly clear it violates the current media bus
concept that describes physical links.

So the VDIC entity will support only the high motion mode, and
the post-processor entities will be removed. We've talked in the
past of adding full VDIC support to the IPU image conversion API
in the IPUv3 driver, so the complete motion compensation modes
can live there eventually, accessed from a mem2mem device. I'm
also considering leaving the low/medium motion support in the VDIC
entity, but accessed eventually from a separate device node sink pad
(from a future output device), instead of from a subdev pad.

So now there will be four capture device nodes per IPU: two linked
directly from each CSI IDMAC source pad, one from the prp-encode
source pad, and one from the prp-viewfinder source pad.

I will post version 4 in a couple days.

Steve


> On Mon, 2017-01-23 at 15:08 -0800, Steve Longerbeam wrote:
> [...]
>>>>> And I'm actually in total agreement with that. I definitely agree that there
>>>>> should be a mechanism in the media framework that allows passing video
>>>>> buffers from a source pad to a sink pad using a software queue, with no
>>>>> involvement from userland.
>>> That is the other part of the argument. I do not agree that these
>>> software queue "links" should be presented to userspace as media pad
>>> links between two entities of a media device.
>>> First, that would limit the links to subdevices contained in the same
>>> media graph, while this should work between any two capture and output
>>> queues of different devices.
>> It sounds like we are talking about two different new proposed features.
> We are talking about the same thing, but we both want a different user
> interface.
> Technically, the issue is to trigger the DMA read channel of a mem2mem
> device automatically whenever another capture device's DMA write channel
> signals a finished frame. Where we disagree is how to present this to
> userspace.
>
> You represent the capture DMA write channel and mem2mem DMA read channel
> as pads on media entites and configure the in-kernel software queue
> between the two using a media pad link. At the same time a different
> representation of the same DMA write and read channels (the capture
> vb2_queue of the capture device and the output vb2_queue of the mem2mem
> device) would be used for operation in the classic, userspace controlled
> mode via dmabuf passing.
>
> I don't want the software-only link in the media graph, but instead use
> the vb2_queue representation for both cases, and implement the in-kernel
> queue link on top of the vb2_queue interface. This would allow userspace
> to have control over buffer allocations and format, and thus avoid
> unexpected performance implications: it is impossible for userspace to
> understand which media entity link, when enabled, will cause a
> significant increase in memory bandwidth usage, or even how much.
> Also the same mechanism could then be used to link any two devices in a
> generic manner, instead of special casing the software queue link for
> two devices that happen to be part of the same media graph.
>
>> My proposal is to implement a software buffer queue between pads.
>> Beyond enabling the link between pads using the existing media controller
>> API, userspace is not involved after that. The fact that this link is
>> accomplished with a software buffer queue is not known, and doesn't
>> need to be known, by userspace.
> I don't think this is a good thing for the reasons stated above and
> below:
> Since the software buffer queue is opaque to userspace, it is completely
> out of userspace control which format is chosen and how the buffers are
> allocated.
> By using media bus formats to configure software links the kernel
> pretends to userspace that there is a physical connection where there
> isn't one.
> Also, the media entity graph would quickly become very unreadable if we
> were to add all devices to it that could reasonably be linked with
> software queues.
>
>> Your proposal, if I have it right, is to allow linking two v4l2 device
>> vb2 queues
>> (i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
>> mode such that v4l2 buffers get passed from one device's vb2 queue to the
>> other without requiring the v4l2 user program to actively forward those
>> buffers.
> Yes.
>
>> There isn't anything that would preclude one from the other, they can
>> both exist. But they are different ideas. One implements software queues
>> at the _pad level_ and is opaque to userspace, the other links queues
>> at the _device level_ using a new user API, but once the link is
>> established, also does not require any involvement from userspace.
> Well, they are different ideas of how the userspace interface _for the
> same thing_ should look like.
>
>> What I'm saying is we can do _both_.
> What I am saying is we shouldn't do the pad link interface for the
> software queues. In my opinion it is the wrong abstraction, and apart
> from the convenience of being able to switch the links on with a single
> media-ctl invocation, I see too many downsides.
>
>>> Assume for example, we want to encode the captured, deinterlaced video
>>> to h.264 with the coda VPU driver. A software queue link could be
>>> established between the CSI capture and the VDIC deinterlacer input,
>> That's already available in the media graph. By linking CSI and
>> VDIC entities. The capture device will then already be providing
>> de-interlaced video, and ...
> I know it is in your code. That is the cause for my concern. The link
> between CSI and VDIC entity should only describe the direct physical
> connection through the VDIC FIFO1 in my opinion.
> For the indirect CSI -> SMFC -> IDMAC -> RAM, RAM -> IDMAC -> VDIC
> software queue, I would strongly prefer to use linked vb2_queues instead
> of the media entity link.
>
>>> just as between the VDIC deinterlacer output and the coda VPU input.
>>> Technically, there would be no difference between those two linked
>>> capture/output queue pairs. But the coda driver is a completely separate
>>> mem2mem device. And since it is not part of the i.MX media graph, there
>>> is no entity pad to link to.
>> your free-run queue linking could then be used to link the (already)
>> de-interlaced stream to the coda device for h.264 encode.
> Yes, and I see no reason why that should use a different interface than
> what is exactly the same process between CSI and VDIC.
>
>> The other idea would be to eventually make the coda device part of
>> the media graph as an entity. Then this link would instead be via pads.
> I would only want to do this if there was a direct connection between
> the IPU FIFOs and the coda VPU device somehow. But since the devices are
> completely separate, they should be described as such.
>
>>> Or assume there is an USB analog capture device that produces interlaced
>>> frames. I think it should be possible to connect its capture queue to
>>> the VDIC deinterlacer output queue just the same way as linking the CSI
>>> to the VDIC (in software queue mode).
>> Right, for devices that are outside the i.MX media graph, such as a USB
>> capture device (or coda), access to the i.MX entities such as the VDIC would
>> require an i.MX mem2mem device with media links to the VDIC. The USB
>> capture device would forward its captured frames to mem2mem (maybe
>> using your free-run vb2 queue linking idea):
>>
>> usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device
> The VDIC doesn't have a direct to memory channel, so that would be
> mem2mem -> VDIC -> IC -> mem2mem, I think?
>
> ========== MEDIA_BUS formats below =====================================
>
>>> Second, the subdevice pad formats describe wire formats, not memory
>>> formats. The user might want to choose between 4:2:2 and 4:2:0
>>> subsampled YUV formats for the intermediate buffer, for example,
>>> depending on memory bandwidth constraints and quality requirements. This
>>> is impossible with the media entity / subdevice pad links.
>> It's true that there are currently no defined planar media bus
>> pixel formats. We just need to add new definitions for them. Once
>> that is done, the media driver will support planar YUV formats
>> simply by adding the new codes to imx_media_formats[].
> I am not comfortable with starting to mix MEDIA_BUS_FMT and V4L2_PIX_FMT
> this way.
>
>> Perhaps this gets to the root of the issue.
>>
>> Is the media bus concept an abstract one, or is the media bus
>> intended to represent actual physical buses (as the lack of planar
>> media bus formats would imply)?
> Yes, maybe that is the root of our disconnect. As I understand it, the
> media bus formats are describing "image formats as flowing over physical
> busses" [1].
>
> [1] Linux Media Subsystem Documentation, Chapter 4.15.3.4.1.1. Media Bus Pixel Codes
>      https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/subdev-formats.html?highlight=media%20bus#v4l2-mbus-pixelcode
>
> I would like to keep it that way and not soften up that description.
>
>> Can we break with the physical-bus-only idea if that is the case, and
>> loosen the definition of a media bus to mean the passage of media
>> data from one pad to another by whatever means?
>>
>> In my view the idea of a physical bus at the sensor makes sense, but
>> beyond that, keeping that restriction limits how data can pass between
>> pads.
>>
>> Hans, any input here?
>>
>> If this is really anathema, then I'm willing to remove the software queues
>> between pads, but it will be giving up some functionality in the media
>> driver.
> So far I am the only one arguing against this, and I haven't yet heard
> anything yet that would have convinced me otherwise. That's why I'd too
> like some more input on this issue.
>
> Certainly removing the controversial pad link controlled software queues
> would remove the point of contention. I think losing some functionality
> (in this case "higher quality" deinterlacing without userspace
> intervention) for now would be worth it to achieve consensus, and also
> reduce the list of things that have to be done for this driver to leave
> staging, but that is of course from the point of view of the guy arguing
> against that interface.
>
>> It would also mean splitting the VDIC in two. The VDIC entity would be
>> limited to only one motion compensation mode,
> Currently, yes. That direct mode is the only one that should be
> described by the media pad link between CSI and VDIC in my opinion.
>
>> and the full functionality would have
>> to be added somewhere else.
> I don't understand why the other functionality would necessarily have to
> live somewere else, but I can see that it might make sense to do so. In
> any case, the separate control of the VDIC/IC via a mem2mem video device
> will be needed anyway, as pointed out in the USB example above, or to
> deinterlace streams received via network or played back from files.
>
>>   Currently all functionality of the VDIC is implemented in a single media entity.
> Yes, at least the mem2mem part should move into its own mem2mem video
> device.
>
>>> I think an interface where userspace configures the capture and output
>>> queues via v4l2 API, passes dma buffers around from one to the other
>>> queue, and then puts both queues into a free running mode would be a
>>> much better fit for this mechanism.
>> As I said, I see these as two different ideas that can both be
>> implemented.
> I think we should have cleared up now where we disagree. These are two
> different ideas for userspace interfaces for the same functionality. My
> opinion is that both should not be implemented
>
>>>>> My only disagreement is when this should be implemented. I think it is
>>>>> fine to keep my custom implementation of this in the driver for now. Once
>>>>> an extension of vb2 is ready to support this feature, it would be fairly
>>>>> straightforward to strip out my custom implementation and go with the
>>>>> new API.
>>>> For a staging driver this isn't necessary, as long as it is documented in
>>>> the TODO file that this needs to be fixed before it can be moved out of
>>>> staging. The whole point of staging is that there is still work to be
>>>> done in the driver, after all :-)
>>> Absolutely. The reason I am arguing against merging the mem2mem media
>>> control links so vehemently is that I am convinced the userspace
>>> interface is wrong, and I am afraid that even though in staging, it
>>> might become established.
>> I don't believe there is anything wrong with the userspace interface,
>> In fact it hasn't even changed. The fact two pads are passing memory
>> buffers is "under the hood".
> Which I disagree with. Doing things like this under the hood would be
> fine only if they were properly introspectable and if the interface
> wouldn't break assumptions that I believe to be there, such as media
> links describing physical connections between hardware entities, and
> media bus formats describing the image format on a physical bus.
> Also with videobuf2 we already have a userspace interface for DMA read
> and write queues, and I'd prefer to extend and improve that instead of
> reimplementing the same functionality, even simplified, under the hood.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-28 19:27                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-28 19:27 UTC (permalink / raw)
  To: Philipp Zabel, Hans Verkuil
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/24/2017 03:27 AM, Philipp Zabel wrote:
> Hi Steve, Hans,
>
> [added Laurent to Cc: who I believe might have an opinion on the media
> bus formats, too. Sorry for the wall of text, I have put a marker where
> the MEDIA_BUS argument starts]
>
> The central issue seems to be that I think media pad links / media bus
> formats should describe physical links, such as parallel or serial
> buses, and the formats of pixels flowing through them, whereas Steve
> would like to extend them to describe software transports and in-memory
> formats.

Hi Philipp, Hans,

I've decided to pull the dma read/write channel linking between
pads. Although I haven't heard any other feedback yet, I agree it is
controversial and it is fairly clear it violates the current media bus
concept that describes physical links.

So the VDIC entity will support only the high motion mode, and
the post-processor entities will be removed. We've talked in the
past of adding full VDIC support to the IPU image conversion API
in the IPUv3 driver, so the complete motion compensation modes
can live there eventually, accessed from a mem2mem device. I'm
also considering leaving the low/medium motion support in the VDIC
entity, but accessed eventually from a separate device node sink pad
(from a future output device), instead of from a subdev pad.

So now there will be four capture device nodes per IPU: two linked
directly from each CSI IDMAC source pad, one from the prp-encode
source pad, and one from the prp-viewfinder source pad.

I will post version 4 in a couple days.

Steve


> On Mon, 2017-01-23 at 15:08 -0800, Steve Longerbeam wrote:
> [...]
>>>>> And I'm actually in total agreement with that. I definitely agree that there
>>>>> should be a mechanism in the media framework that allows passing video
>>>>> buffers from a source pad to a sink pad using a software queue, with no
>>>>> involvement from userland.
>>> That is the other part of the argument. I do not agree that these
>>> software queue "links" should be presented to userspace as media pad
>>> links between two entities of a media device.
>>> First, that would limit the links to subdevices contained in the same
>>> media graph, while this should work between any two capture and output
>>> queues of different devices.
>> It sounds like we are talking about two different new proposed features.
> We are talking about the same thing, but we both want a different user
> interface.
> Technically, the issue is to trigger the DMA read channel of a mem2mem
> device automatically whenever another capture device's DMA write channel
> signals a finished frame. Where we disagree is how to present this to
> userspace.
>
> You represent the capture DMA write channel and mem2mem DMA read channel
> as pads on media entites and configure the in-kernel software queue
> between the two using a media pad link. At the same time a different
> representation of the same DMA write and read channels (the capture
> vb2_queue of the capture device and the output vb2_queue of the mem2mem
> device) would be used for operation in the classic, userspace controlled
> mode via dmabuf passing.
>
> I don't want the software-only link in the media graph, but instead use
> the vb2_queue representation for both cases, and implement the in-kernel
> queue link on top of the vb2_queue interface. This would allow userspace
> to have control over buffer allocations and format, and thus avoid
> unexpected performance implications: it is impossible for userspace to
> understand which media entity link, when enabled, will cause a
> significant increase in memory bandwidth usage, or even how much.
> Also the same mechanism could then be used to link any two devices in a
> generic manner, instead of special casing the software queue link for
> two devices that happen to be part of the same media graph.
>
>> My proposal is to implement a software buffer queue between pads.
>> Beyond enabling the link between pads using the existing media controller
>> API, userspace is not involved after that. The fact that this link is
>> accomplished with a software buffer queue is not known, and doesn't
>> need to be known, by userspace.
> I don't think this is a good thing for the reasons stated above and
> below:
> Since the software buffer queue is opaque to userspace, it is completely
> out of userspace control which format is chosen and how the buffers are
> allocated.
> By using media bus formats to configure software links the kernel
> pretends to userspace that there is a physical connection where there
> isn't one.
> Also, the media entity graph would quickly become very unreadable if we
> were to add all devices to it that could reasonably be linked with
> software queues.
>
>> Your proposal, if I have it right, is to allow linking two v4l2 device
>> vb2 queues
>> (i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
>> mode such that v4l2 buffers get passed from one device's vb2 queue to the
>> other without requiring the v4l2 user program to actively forward those
>> buffers.
> Yes.
>
>> There isn't anything that would preclude one from the other, they can
>> both exist. But they are different ideas. One implements software queues
>> at the _pad level_ and is opaque to userspace, the other links queues
>> at the _device level_ using a new user API, but once the link is
>> established, also does not require any involvement from userspace.
> Well, they are different ideas of how the userspace interface _for the
> same thing_ should look like.
>
>> What I'm saying is we can do _both_.
> What I am saying is we shouldn't do the pad link interface for the
> software queues. In my opinion it is the wrong abstraction, and apart
> from the convenience of being able to switch the links on with a single
> media-ctl invocation, I see too many downsides.
>
>>> Assume for example, we want to encode the captured, deinterlaced video
>>> to h.264 with the coda VPU driver. A software queue link could be
>>> established between the CSI capture and the VDIC deinterlacer input,
>> That's already available in the media graph. By linking CSI and
>> VDIC entities. The capture device will then already be providing
>> de-interlaced video, and ...
> I know it is in your code. That is the cause for my concern. The link
> between CSI and VDIC entity should only describe the direct physical
> connection through the VDIC FIFO1 in my opinion.
> For the indirect CSI -> SMFC -> IDMAC -> RAM, RAM -> IDMAC -> VDIC
> software queue, I would strongly prefer to use linked vb2_queues instead
> of the media entity link.
>
>>> just as between the VDIC deinterlacer output and the coda VPU input.
>>> Technically, there would be no difference between those two linked
>>> capture/output queue pairs. But the coda driver is a completely separate
>>> mem2mem device. And since it is not part of the i.MX media graph, there
>>> is no entity pad to link to.
>> your free-run queue linking could then be used to link the (already)
>> de-interlaced stream to the coda device for h.264 encode.
> Yes, and I see no reason why that should use a different interface than
> what is exactly the same process between CSI and VDIC.
>
>> The other idea would be to eventually make the coda device part of
>> the media graph as an entity. Then this link would instead be via pads.
> I would only want to do this if there was a direct connection between
> the IPU FIFOs and the coda VPU device somehow. But since the devices are
> completely separate, they should be described as such.
>
>>> Or assume there is an USB analog capture device that produces interlaced
>>> frames. I think it should be possible to connect its capture queue to
>>> the VDIC deinterlacer output queue just the same way as linking the CSI
>>> to the VDIC (in software queue mode).
>> Right, for devices that are outside the i.MX media graph, such as a USB
>> capture device (or coda), access to the i.MX entities such as the VDIC would
>> require an i.MX mem2mem device with media links to the VDIC. The USB
>> capture device would forward its captured frames to mem2mem (maybe
>> using your free-run vb2 queue linking idea):
>>
>> usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device
> The VDIC doesn't have a direct to memory channel, so that would be
> mem2mem -> VDIC -> IC -> mem2mem, I think?
>
> ========== MEDIA_BUS formats below =====================================
>
>>> Second, the subdevice pad formats describe wire formats, not memory
>>> formats. The user might want to choose between 4:2:2 and 4:2:0
>>> subsampled YUV formats for the intermediate buffer, for example,
>>> depending on memory bandwidth constraints and quality requirements. This
>>> is impossible with the media entity / subdevice pad links.
>> It's true that there are currently no defined planar media bus
>> pixel formats. We just need to add new definitions for them. Once
>> that is done, the media driver will support planar YUV formats
>> simply by adding the new codes to imx_media_formats[].
> I am not comfortable with starting to mix MEDIA_BUS_FMT and V4L2_PIX_FMT
> this way.
>
>> Perhaps this gets to the root of the issue.
>>
>> Is the media bus concept an abstract one, or is the media bus
>> intended to represent actual physical buses (as the lack of planar
>> media bus formats would imply)?
> Yes, maybe that is the root of our disconnect. As I understand it, the
> media bus formats are describing "image formats as flowing over physical
> busses" [1].
>
> [1] Linux Media Subsystem Documentation, Chapter 4.15.3.4.1.1. Media Bus Pixel Codes
>      https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/subdev-formats.html?highlight=media%20bus#v4l2-mbus-pixelcode
>
> I would like to keep it that way and not soften up that description.
>
>> Can we break with the physical-bus-only idea if that is the case, and
>> loosen the definition of a media bus to mean the passage of media
>> data from one pad to another by whatever means?
>>
>> In my view the idea of a physical bus at the sensor makes sense, but
>> beyond that, keeping that restriction limits how data can pass between
>> pads.
>>
>> Hans, any input here?
>>
>> If this is really anathema, then I'm willing to remove the software queues
>> between pads, but it will be giving up some functionality in the media
>> driver.
> So far I am the only one arguing against this, and I haven't yet heard
> anything yet that would have convinced me otherwise. That's why I'd too
> like some more input on this issue.
>
> Certainly removing the controversial pad link controlled software queues
> would remove the point of contention. I think losing some functionality
> (in this case "higher quality" deinterlacing without userspace
> intervention) for now would be worth it to achieve consensus, and also
> reduce the list of things that have to be done for this driver to leave
> staging, but that is of course from the point of view of the guy arguing
> against that interface.
>
>> It would also mean splitting the VDIC in two. The VDIC entity would be
>> limited to only one motion compensation mode,
> Currently, yes. That direct mode is the only one that should be
> described by the media pad link between CSI and VDIC in my opinion.
>
>> and the full functionality would have
>> to be added somewhere else.
> I don't understand why the other functionality would necessarily have to
> live somewere else, but I can see that it might make sense to do so. In
> any case, the separate control of the VDIC/IC via a mem2mem video device
> will be needed anyway, as pointed out in the USB example above, or to
> deinterlace streams received via network or played back from files.
>
>>   Currently all functionality of the VDIC is implemented in a single media entity.
> Yes, at least the mem2mem part should move into its own mem2mem video
> device.
>
>>> I think an interface where userspace configures the capture and output
>>> queues via v4l2 API, passes dma buffers around from one to the other
>>> queue, and then puts both queues into a free running mode would be a
>>> much better fit for this mechanism.
>> As I said, I see these as two different ideas that can both be
>> implemented.
> I think we should have cleared up now where we disagree. These are two
> different ideas for userspace interfaces for the same functionality. My
> opinion is that both should not be implemented
>
>>>>> My only disagreement is when this should be implemented. I think it is
>>>>> fine to keep my custom implementation of this in the driver for now. Once
>>>>> an extension of vb2 is ready to support this feature, it would be fairly
>>>>> straightforward to strip out my custom implementation and go with the
>>>>> new API.
>>>> For a staging driver this isn't necessary, as long as it is documented in
>>>> the TODO file that this needs to be fixed before it can be moved out of
>>>> staging. The whole point of staging is that there is still work to be
>>>> done in the driver, after all :-)
>>> Absolutely. The reason I am arguing against merging the mem2mem media
>>> control links so vehemently is that I am convinced the userspace
>>> interface is wrong, and I am afraid that even though in staging, it
>>> might become established.
>> I don't believe there is anything wrong with the userspace interface,
>> In fact it hasn't even changed. The fact two pads are passing memory
>> buffers is "under the hood".
> Which I disagree with. Doing things like this under the hood would be
> fine only if they were properly introspectable and if the interface
> wouldn't break assumptions that I believe to be there, such as media
> links describing physical connections between hardware entities, and
> media bus formats describing the image format on a physical bus.
> Also with videobuf2 we already have a userspace interface for DMA read
> and write queues, and I'd prefer to extend and improve that instead of
> reimplementing the same functionality, even simplified, under the hood.
>
> regards
> Philipp
>

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-28 19:27                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-28 19:27 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/24/2017 03:27 AM, Philipp Zabel wrote:
> Hi Steve, Hans,
>
> [added Laurent to Cc: who I believe might have an opinion on the media
> bus formats, too. Sorry for the wall of text, I have put a marker where
> the MEDIA_BUS argument starts]
>
> The central issue seems to be that I think media pad links / media bus
> formats should describe physical links, such as parallel or serial
> buses, and the formats of pixels flowing through them, whereas Steve
> would like to extend them to describe software transports and in-memory
> formats.

Hi Philipp, Hans,

I've decided to pull the dma read/write channel linking between
pads. Although I haven't heard any other feedback yet, I agree it is
controversial and it is fairly clear it violates the current media bus
concept that describes physical links.

So the VDIC entity will support only the high motion mode, and
the post-processor entities will be removed. We've talked in the
past of adding full VDIC support to the IPU image conversion API
in the IPUv3 driver, so the complete motion compensation modes
can live there eventually, accessed from a mem2mem device. I'm
also considering leaving the low/medium motion support in the VDIC
entity, but accessed eventually from a separate device node sink pad
(from a future output device), instead of from a subdev pad.

So now there will be four capture device nodes per IPU: two linked
directly from each CSI IDMAC source pad, one from the prp-encode
source pad, and one from the prp-viewfinder source pad.

I will post version 4 in a couple days.

Steve


> On Mon, 2017-01-23 at 15:08 -0800, Steve Longerbeam wrote:
> [...]
>>>>> And I'm actually in total agreement with that. I definitely agree that there
>>>>> should be a mechanism in the media framework that allows passing video
>>>>> buffers from a source pad to a sink pad using a software queue, with no
>>>>> involvement from userland.
>>> That is the other part of the argument. I do not agree that these
>>> software queue "links" should be presented to userspace as media pad
>>> links between two entities of a media device.
>>> First, that would limit the links to subdevices contained in the same
>>> media graph, while this should work between any two capture and output
>>> queues of different devices.
>> It sounds like we are talking about two different new proposed features.
> We are talking about the same thing, but we both want a different user
> interface.
> Technically, the issue is to trigger the DMA read channel of a mem2mem
> device automatically whenever another capture device's DMA write channel
> signals a finished frame. Where we disagree is how to present this to
> userspace.
>
> You represent the capture DMA write channel and mem2mem DMA read channel
> as pads on media entites and configure the in-kernel software queue
> between the two using a media pad link. At the same time a different
> representation of the same DMA write and read channels (the capture
> vb2_queue of the capture device and the output vb2_queue of the mem2mem
> device) would be used for operation in the classic, userspace controlled
> mode via dmabuf passing.
>
> I don't want the software-only link in the media graph, but instead use
> the vb2_queue representation for both cases, and implement the in-kernel
> queue link on top of the vb2_queue interface. This would allow userspace
> to have control over buffer allocations and format, and thus avoid
> unexpected performance implications: it is impossible for userspace to
> understand which media entity link, when enabled, will cause a
> significant increase in memory bandwidth usage, or even how much.
> Also the same mechanism could then be used to link any two devices in a
> generic manner, instead of special casing the software queue link for
> two devices that happen to be part of the same media graph.
>
>> My proposal is to implement a software buffer queue between pads.
>> Beyond enabling the link between pads using the existing media controller
>> API, userspace is not involved after that. The fact that this link is
>> accomplished with a software buffer queue is not known, and doesn't
>> need to be known, by userspace.
> I don't think this is a good thing for the reasons stated above and
> below:
> Since the software buffer queue is opaque to userspace, it is completely
> out of userspace control which format is chosen and how the buffers are
> allocated.
> By using media bus formats to configure software links the kernel
> pretends to userspace that there is a physical connection where there
> isn't one.
> Also, the media entity graph would quickly become very unreadable if we
> were to add all devices to it that could reasonably be linked with
> software queues.
>
>> Your proposal, if I have it right, is to allow linking two v4l2 device
>> vb2 queues
>> (i.e. /dev/videoX -> /dev/videoY), using a new user level API, in a free-run
>> mode such that v4l2 buffers get passed from one device's vb2 queue to the
>> other without requiring the v4l2 user program to actively forward those
>> buffers.
> Yes.
>
>> There isn't anything that would preclude one from the other, they can
>> both exist. But they are different ideas. One implements software queues
>> at the _pad level_ and is opaque to userspace, the other links queues
>> at the _device level_ using a new user API, but once the link is
>> established, also does not require any involvement from userspace.
> Well, they are different ideas of how the userspace interface _for the
> same thing_ should look like.
>
>> What I'm saying is we can do _both_.
> What I am saying is we shouldn't do the pad link interface for the
> software queues. In my opinion it is the wrong abstraction, and apart
> from the convenience of being able to switch the links on with a single
> media-ctl invocation, I see too many downsides.
>
>>> Assume for example, we want to encode the captured, deinterlaced video
>>> to h.264 with the coda VPU driver. A software queue link could be
>>> established between the CSI capture and the VDIC deinterlacer input,
>> That's already available in the media graph. By linking CSI and
>> VDIC entities. The capture device will then already be providing
>> de-interlaced video, and ...
> I know it is in your code. That is the cause for my concern. The link
> between CSI and VDIC entity should only describe the direct physical
> connection through the VDIC FIFO1 in my opinion.
> For the indirect CSI -> SMFC -> IDMAC -> RAM, RAM -> IDMAC -> VDIC
> software queue, I would strongly prefer to use linked vb2_queues instead
> of the media entity link.
>
>>> just as between the VDIC deinterlacer output and the coda VPU input.
>>> Technically, there would be no difference between those two linked
>>> capture/output queue pairs. But the coda driver is a completely separate
>>> mem2mem device. And since it is not part of the i.MX media graph, there
>>> is no entity pad to link to.
>> your free-run queue linking could then be used to link the (already)
>> de-interlaced stream to the coda device for h.264 encode.
> Yes, and I see no reason why that should use a different interface than
> what is exactly the same process between CSI and VDIC.
>
>> The other idea would be to eventually make the coda device part of
>> the media graph as an entity. Then this link would instead be via pads.
> I would only want to do this if there was a direct connection between
> the IPU FIFOs and the coda VPU device somehow. But since the devices are
> completely separate, they should be described as such.
>
>>> Or assume there is an USB analog capture device that produces interlaced
>>> frames. I think it should be possible to connect its capture queue to
>>> the VDIC deinterlacer output queue just the same way as linking the CSI
>>> to the VDIC (in software queue mode).
>> Right, for devices that are outside the i.MX media graph, such as a USB
>> capture device (or coda), access to the i.MX entities such as the VDIC would
>> require an i.MX mem2mem device with media links to the VDIC. The USB
>> capture device would forward its captured frames to mem2mem (maybe
>> using your free-run vb2 queue linking idea):
>>
>> usb device -> i.mx mem2mem device -> VDIC entity -> i.mx mem2mem device
> The VDIC doesn't have a direct to memory channel, so that would be
> mem2mem -> VDIC -> IC -> mem2mem, I think?
>
> ========== MEDIA_BUS formats below =====================================
>
>>> Second, the subdevice pad formats describe wire formats, not memory
>>> formats. The user might want to choose between 4:2:2 and 4:2:0
>>> subsampled YUV formats for the intermediate buffer, for example,
>>> depending on memory bandwidth constraints and quality requirements. This
>>> is impossible with the media entity / subdevice pad links.
>> It's true that there are currently no defined planar media bus
>> pixel formats. We just need to add new definitions for them. Once
>> that is done, the media driver will support planar YUV formats
>> simply by adding the new codes to imx_media_formats[].
> I am not comfortable with starting to mix MEDIA_BUS_FMT and V4L2_PIX_FMT
> this way.
>
>> Perhaps this gets to the root of the issue.
>>
>> Is the media bus concept an abstract one, or is the media bus
>> intended to represent actual physical buses (as the lack of planar
>> media bus formats would imply)?
> Yes, maybe that is the root of our disconnect. As I understand it, the
> media bus formats are describing "image formats as flowing over physical
> busses" [1].
>
> [1] Linux Media Subsystem Documentation, Chapter 4.15.3.4.1.1. Media Bus Pixel Codes
>      https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/subdev-formats.html?highlight=media%20bus#v4l2-mbus-pixelcode
>
> I would like to keep it that way and not soften up that description.
>
>> Can we break with the physical-bus-only idea if that is the case, and
>> loosen the definition of a media bus to mean the passage of media
>> data from one pad to another by whatever means?
>>
>> In my view the idea of a physical bus at the sensor makes sense, but
>> beyond that, keeping that restriction limits how data can pass between
>> pads.
>>
>> Hans, any input here?
>>
>> If this is really anathema, then I'm willing to remove the software queues
>> between pads, but it will be giving up some functionality in the media
>> driver.
> So far I am the only one arguing against this, and I haven't yet heard
> anything yet that would have convinced me otherwise. That's why I'd too
> like some more input on this issue.
>
> Certainly removing the controversial pad link controlled software queues
> would remove the point of contention. I think losing some functionality
> (in this case "higher quality" deinterlacing without userspace
> intervention) for now would be worth it to achieve consensus, and also
> reduce the list of things that have to be done for this driver to leave
> staging, but that is of course from the point of view of the guy arguing
> against that interface.
>
>> It would also mean splitting the VDIC in two. The VDIC entity would be
>> limited to only one motion compensation mode,
> Currently, yes. That direct mode is the only one that should be
> described by the media pad link between CSI and VDIC in my opinion.
>
>> and the full functionality would have
>> to be added somewhere else.
> I don't understand why the other functionality would necessarily have to
> live somewere else, but I can see that it might make sense to do so. In
> any case, the separate control of the VDIC/IC via a mem2mem video device
> will be needed anyway, as pointed out in the USB example above, or to
> deinterlace streams received via network or played back from files.
>
>>   Currently all functionality of the VDIC is implemented in a single media entity.
> Yes, at least the mem2mem part should move into its own mem2mem video
> device.
>
>>> I think an interface where userspace configures the capture and output
>>> queues via v4l2 API, passes dma buffers around from one to the other
>>> queue, and then puts both queues into a free running mode would be a
>>> much better fit for this mechanism.
>> As I said, I see these as two different ideas that can both be
>> implemented.
> I think we should have cleared up now where we disagree. These are two
> different ideas for userspace interfaces for the same functionality. My
> opinion is that both should not be implemented
>
>>>>> My only disagreement is when this should be implemented. I think it is
>>>>> fine to keep my custom implementation of this in the driver for now. Once
>>>>> an extension of vb2 is ready to support this feature, it would be fairly
>>>>> straightforward to strip out my custom implementation and go with the
>>>>> new API.
>>>> For a staging driver this isn't necessary, as long as it is documented in
>>>> the TODO file that this needs to be fixed before it can be moved out of
>>>> staging. The whole point of staging is that there is still work to be
>>>> done in the driver, after all :-)
>>> Absolutely. The reason I am arguing against merging the mem2mem media
>>> control links so vehemently is that I am convinced the userspace
>>> interface is wrong, and I am afraid that even though in staging, it
>>> might become established.
>> I don't believe there is anything wrong with the userspace interface,
>> In fact it hasn't even changed. The fact two pads are passing memory
>> buffers is "under the hood".
> Which I disagree with. Doing things like this under the hood would be
> fine only if they were properly introspectable and if the interface
> wouldn't break assumptions that I believe to be there, such as media
> links describing physical connections between hardware entities, and
> media bus formats describing the image format on a physical bus.
> Also with videobuf2 we already have a userspace interface for DMA read
> and write queues, and I'd prefer to extend and improve that instead of
> reimplementing the same functionality, even simplified, under the hood.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-30 13:06                 ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 13:06 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam,
	Laurent Pinchart

> The central issue seems to be that I think media pad links / media bus
> formats should describe physical links, such as parallel or serial
> buses, and the formats of pixels flowing through them, whereas Steve
> would like to extend them to describe software transports and in-memory
> formats.

This probably isn't the right place to attach this comment in this
thread, but... the issue of media bus formats matching physical buses
is an argument that I think is already lost.

For example, take the 10-bit bayer formats:

#define MEDIA_BUS_FMT_SBGGR10_1X10              0x3007
#define MEDIA_BUS_FMT_SGBRG10_1X10              0x300e
#define MEDIA_BUS_FMT_SGRBG10_1X10              0x300a
#define MEDIA_BUS_FMT_SRGGB10_1X10              0x300f

These are commonly used on CSI serial buses (see the smiapp driver for
example).  From the description at the top of the file, it says the
1X10 means that one pixel is transferred as one 10-bit sample.

However, the format on wire is somewhat different - four pixels are
transmitted over five bytes:

	P0	P1	P2	P3	P0	P1	P2	P3
	8-bit	8-bit	8-bit	8-bit	2-bit	2-bit	2-bit	2-bit

This gives two problems:
1) it doesn't fit in any sensible kind of "one pixel transferred as
   N M-bit samples" description because the pixel/sample values
   (depending how you look at them) are broken up.

2) changing this will probably be a user visible change, as things
   like smiapp are already in use.

So, I think what we actually have is the media bus formats describing
the _logical_ bus format.  Yes, one pixel is transferred as one 10-bit
sample in this case.

To help illustrate my point, consider the difference between
MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
16-bit wide bus (if it's not 16-bit, then it has to be broken up into
separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.

So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
interesting:

	first byte	2nd	3rd
lane 1	P0 9:2		S0	P7 9:2
lane 2	P1 9:2		P4 9:2	S1
lane 3	P2 9:2		P5 9:2	P8 9:2
lane 4	P3 9:2		P6 9:2	P9 9:2

S0 = P0/P1/P2/P3 least significant two bits
S1 = P4/P5/P6/P7 least significant two bits

or 2 lane CSI:
	first byte	2nd	3rd	4th	5th
lane 1	P0 9:2		P2	S0	P5	P7
lane 2	P1 9:2		P3	P4	P6	S1

or 1 lane CSI:
lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...

etc.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-30 13:06                 ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 13:06 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

> The central issue seems to be that I think media pad links / media bus
> formats should describe physical links, such as parallel or serial
> buses, and the formats of pixels flowing through them, whereas Steve
> would like to extend them to describe software transports and in-memory
> formats.

This probably isn't the right place to attach this comment in this
thread, but... the issue of media bus formats matching physical buses
is an argument that I think is already lost.

For example, take the 10-bit bayer formats:

#define MEDIA_BUS_FMT_SBGGR10_1X10              0x3007
#define MEDIA_BUS_FMT_SGBRG10_1X10              0x300e
#define MEDIA_BUS_FMT_SGRBG10_1X10              0x300a
#define MEDIA_BUS_FMT_SRGGB10_1X10              0x300f

These are commonly used on CSI serial buses (see the smiapp driver for
example).  From the description at the top of the file, it says the
1X10 means that one pixel is transferred as one 10-bit sample.

However, the format on wire is somewhat different - four pixels are
transmitted over five bytes:

	P0	P1	P2	P3	P0	P1	P2	P3
	8-bit	8-bit	8-bit	8-bit	2-bit	2-bit	2-bit	2-bit

This gives two problems:
1) it doesn't fit in any sensible kind of "one pixel transferred as
   N M-bit samples" description because the pixel/sample values
   (depending how you look at them) are broken up.

2) changing this will probably be a user visible change, as things
   like smiapp are already in use.

So, I think what we actually have is the media bus formats describing
the _logical_ bus format.  Yes, one pixel is transferred as one 10-bit
sample in this case.

To help illustrate my point, consider the difference between
MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
16-bit wide bus (if it's not 16-bit, then it has to be broken up into
separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.

So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
interesting:

	first byte	2nd	3rd
lane 1	P0 9:2		S0	P7 9:2
lane 2	P1 9:2		P4 9:2	S1
lane 3	P2 9:2		P5 9:2	P8 9:2
lane 4	P3 9:2		P6 9:2	P9 9:2

S0 = P0/P1/P2/P3 least significant two bits
S1 = P4/P5/P6/P7 least significant two bits

or 2 lane CSI:
	first byte	2nd	3rd	4th	5th
lane 1	P0 9:2		P2	S0	P5	P7
lane 2	P1 9:2		P3	P4	P6	S1

or 1 lane CSI:
lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...

etc.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-30 13:06                 ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 13:06 UTC (permalink / raw)
  To: linux-arm-kernel

> The central issue seems to be that I think media pad links / media bus
> formats should describe physical links, such as parallel or serial
> buses, and the formats of pixels flowing through them, whereas Steve
> would like to extend them to describe software transports and in-memory
> formats.

This probably isn't the right place to attach this comment in this
thread, but... the issue of media bus formats matching physical buses
is an argument that I think is already lost.

For example, take the 10-bit bayer formats:

#define MEDIA_BUS_FMT_SBGGR10_1X10              0x3007
#define MEDIA_BUS_FMT_SGBRG10_1X10              0x300e
#define MEDIA_BUS_FMT_SGRBG10_1X10              0x300a
#define MEDIA_BUS_FMT_SRGGB10_1X10              0x300f

These are commonly used on CSI serial buses (see the smiapp driver for
example).  From the description at the top of the file, it says the
1X10 means that one pixel is transferred as one 10-bit sample.

However, the format on wire is somewhat different - four pixels are
transmitted over five bytes:

	P0	P1	P2	P3	P0	P1	P2	P3
	8-bit	8-bit	8-bit	8-bit	2-bit	2-bit	2-bit	2-bit

This gives two problems:
1) it doesn't fit in any sensible kind of "one pixel transferred as
   N M-bit samples" description because the pixel/sample values
   (depending how you look at them) are broken up.

2) changing this will probably be a user visible change, as things
   like smiapp are already in use.

So, I think what we actually have is the media bus formats describing
the _logical_ bus format.  Yes, one pixel is transferred as one 10-bit
sample in this case.

To help illustrate my point, consider the difference between
MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
16-bit wide bus (if it's not 16-bit, then it has to be broken up into
separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.

So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
interesting:

	first byte	2nd	3rd
lane 1	P0 9:2		S0	P7 9:2
lane 2	P1 9:2		P4 9:2	S1
lane 3	P2 9:2		P5 9:2	P8 9:2
lane 4	P3 9:2		P6 9:2	P9 9:2

S0 = P0/P1/P2/P3 least significant two bits
S1 = P4/P5/P6/P7 least significant two bits

or 2 lane CSI:
	first byte	2nd	3rd	4th	5th
lane 1	P0 9:2		P2	S0	P5	P7
lane 2	P1 9:2		P3	P4	P6	S1

or 1 lane CSI:
lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...

etc.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-23 11:13             ` Philipp Zabel
  (?)
@ 2017-01-30 15:28               ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 15:28 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Mon, Jan 23, 2017 at 12:13:26PM +0100, Philipp Zabel wrote:
> Hi Steve,
> 
> On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
> > Second, ignoring the above locking issue for a moment, 
> > v4l2_pipeline_pm_use()
> > will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
> > when executing
> > media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
> > wrong order.
> > In my version which enforces the correct power on order, the mipi csi-2 
> > s_power
> > is called first in that link setup, followed by the sensor.
> 
> I don't understand why you want to power up subdevs as soon as the links
> are established. Shouldn't that rather be done for all subdevices in the
> pipeline when the corresponding capture device is opened?
> It seems to me that powering up the pipeline should be the last step
> before userspace actually starts the capture.

I agree with Philipp here - configuration of the software pipeline
shouldn't result in hardware being forced to be powered up.  That's
more of a decision for the individual sub-driver than for core.

Executing media-ctl to enable a link between two sub-device endpoints
should really be a matter of setting the software state, and when the
video device is opened for streaming, surely that's when the hardware
in the chain between the source and the capture device should be
powered up and programmed.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-30 15:28               ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 15:28 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, Steve Longerbeam, linux-media,
	devicetree, arnd, mchehab, bparrot, robh+dt, horms+renesas,
	tiffany.lin, linux-arm-kernel, niklas.soderlund+renesas, gregkh,
	linux-kernel, jean-christophe.trotin, kernel, fabio.estevam,
	shawnguo, sudipm.mukherjee

On Mon, Jan 23, 2017 at 12:13:26PM +0100, Philipp Zabel wrote:
> Hi Steve,
> 
> On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
> > Second, ignoring the above locking issue for a moment, 
> > v4l2_pipeline_pm_use()
> > will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
> > when executing
> > media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
> > wrong order.
> > In my version which enforces the correct power on order, the mipi csi-2 
> > s_power
> > is called first in that link setup, followed by the sensor.
> 
> I don't understand why you want to power up subdevs as soon as the links
> are established. Shouldn't that rather be done for all subdevices in the
> pipeline when the corresponding capture device is opened?
> It seems to me that powering up the pipeline should be the last step
> before userspace actually starts the capture.

I agree with Philipp here - configuration of the software pipeline
shouldn't result in hardware being forced to be powered up.  That's
more of a decision for the individual sub-driver than for core.

Executing media-ctl to enable a link between two sub-device endpoints
should really be a matter of setting the software state, and when the
video device is opened for streaming, surely that's when the hardware
in the chain between the source and the capture device should be
powered up and programmed.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-30 15:28               ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 15:28 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, Jan 23, 2017 at 12:13:26PM +0100, Philipp Zabel wrote:
> Hi Steve,
> 
> On Sun, 2017-01-22 at 18:31 -0800, Steve Longerbeam wrote:
> > Second, ignoring the above locking issue for a moment, 
> > v4l2_pipeline_pm_use()
> > will call s_power on the sensor _first_, then the mipi csi-2 s_power, 
> > when executing
> > media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the 
> > wrong order.
> > In my version which enforces the correct power on order, the mipi csi-2 
> > s_power
> > is called first in that link setup, followed by the sensor.
> 
> I don't understand why you want to power up subdevs as soon as the links
> are established. Shouldn't that rather be done for all subdevices in the
> pipeline when the corresponding capture device is opened?
> It seems to me that powering up the pipeline should be the last step
> before userspace actually starts the capture.

I agree with Philipp here - configuration of the software pipeline
shouldn't result in hardware being forced to be powered up.  That's
more of a decision for the individual sub-driver than for core.

Executing media-ctl to enable a link between two sub-device endpoints
should really be a matter of setting the software state, and when the
video device is opened for streaming, surely that's when the hardware
in the chain between the source and the capture device should be
powered up and programmed.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-30 22:28     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:28 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
> index 66d10d8..9e2d26d 100644
> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
> @@ -52,3 +52,9 @@
>  &sata {
>  	status = "okay";
>  };
> +
> +&ipu1_csi1_from_mipi_vc1 {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> +
> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> index 795b5a5..bca9fed 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
...
> +/* Incoming port from sensor */
> +&mipi_csi_from_mipi_sensor {
> +        remote-endpoint = <&ov5640_to_mipi_csi>;
> +        data-lanes = <0 1>;
> +        clock-lanes = <2>;
> +};
> +

Applying: ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
.git/rebase-apply/patch:33: new blank line at EOF.
+
.git/rebase-apply/patch:201: new blank line at EOF.
+
warning: 2 lines add whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-30 22:28     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:28 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
> index 66d10d8..9e2d26d 100644
> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
> @@ -52,3 +52,9 @@
>  &sata {
>  	status = "okay";
>  };
> +
> +&ipu1_csi1_from_mipi_vc1 {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> +
> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> index 795b5a5..bca9fed 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
...
> +/* Incoming port from sensor */
> +&mipi_csi_from_mipi_sensor {
> +        remote-endpoint = <&ov5640_to_mipi_csi>;
> +        data-lanes = <0 1>;
> +        clock-lanes = <2>;
> +};
> +

Applying: ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
.git/rebase-apply/patch:33: new blank line at EOF.
+
.git/rebase-apply/patch:201: new blank line at EOF.
+
warning: 2 lines add whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-30 22:28     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:28 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
> index 66d10d8..9e2d26d 100644
> --- a/arch/arm/boot/dts/imx6q-sabrelite.dts
> +++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
> @@ -52,3 +52,9 @@
>  &sata {
>  	status = "okay";
>  };
> +
> +&ipu1_csi1_from_mipi_vc1 {
> +	data-lanes = <0 1>;
> +	clock-lanes = <2>;
> +};
> +
> diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> index 795b5a5..bca9fed 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
...
> +/* Incoming port from sensor */
> +&mipi_csi_from_mipi_sensor {
> +        remote-endpoint = <&ov5640_to_mipi_csi>;
> +        data-lanes = <0 1>;
> +        clock-lanes = <2>;
> +};
> +

Applying: ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
.git/rebase-apply/patch:33: new blank line at EOF.
+
.git/rebase-apply/patch:201: new blank line at EOF.
+
warning: 2 lines add whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-07  2:11 ` [PATCH v3 16/24] media: Add i.MX media core driver Steve Longerbeam
  2017-01-13 15:20     ` Philipp Zabel
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  2017-02-02 22:44     ` Russell King - ARM Linux
  2017-02-02 22:50     ` Russell King - ARM Linux
  3 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> Add the core media driver for i.MX SOC.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: Add i.MX media core driver
.git/rebase-apply/patch:516: new blank line at EOF.
+
.git/rebase-apply/patch:528: new blank line at EOF.
+
.git/rebase-apply/patch:556: new blank line at EOF.
+
warning: 3 lines add whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> Add the core media driver for i.MX SOC.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: Add i.MX media core driver
.git/rebase-apply/patch:516: new blank line at EOF.
+
.git/rebase-apply/patch:528: new blank line at EOF.
+
.git/rebase-apply/patch:556: new blank line at EOF.
+
warning: 3 lines add whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> Add the core media driver for i.MX SOC.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: Add i.MX media core driver
.git/rebase-apply/patch:516: new blank line at EOF.
+
.git/rebase-apply/patch:528: new blank line at EOF.
+
.git/rebase-apply/patch:556: new blank line at EOF.
+
warning: 3 lines add whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

warning: 3 lines add whitespace errors.
Applying: media: imx: Add CSI subdev driver
.git/rebase-apply/patch:38: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>

warning: 3 lines add whitespace errors.
Applying: media: imx: Add CSI subdev driver
.git/rebase-apply/patch:38: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

warning: 3 lines add whitespace errors.
Applying: media: imx: Add CSI subdev driver
.git/rebase-apply/patch:38: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:37PM -0800, Steve Longerbeam wrote:
> This is a set of three media entity subdevice drivers for the i.MX
> Image Converter. The i.MX IC module contains three independent
> "tasks":
> 
> - Pre-processing Encode task: video frames are routed directly from
>   the CSI and can be scaled, color-space converted, and rotated.
>   Scaled output is limited to 1024x1024 resolution. Output frames
>   are routed to the camera interface entities (camif).
> 
> - Pre-processing Viewfinder task: this task can perform the same
>   conversions as the pre-process encode task, but in addition can
>   be used for hardware motion compensated deinterlacing. Frames can
>   come either directly from the CSI or from the SMFC entities (memory
>   buffers via IDMAC channels). Scaled output is limited to 1024x1024
>   resolution. Output frames can be routed to various sinks including
>   the post-processing task entities.
> 
> - Post-processing task: same conversions as pre-process encode. However
>   this entity sends frames to the i.MX IPU image converter which supports
>   image tiling, which allows scaled output up to 4096x4096 resolution.
>   Output frames can be routed to the camera interfaces.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: imx: Add IC subdev drivers
.git/rebase-apply/patch:3054: new blank line at EOF.
+
warning: 1 line adds whitespace errors.


-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:37PM -0800, Steve Longerbeam wrote:
> This is a set of three media entity subdevice drivers for the i.MX
> Image Converter. The i.MX IC module contains three independent
> "tasks":
> 
> - Pre-processing Encode task: video frames are routed directly from
>   the CSI and can be scaled, color-space converted, and rotated.
>   Scaled output is limited to 1024x1024 resolution. Output frames
>   are routed to the camera interface entities (camif).
> 
> - Pre-processing Viewfinder task: this task can perform the same
>   conversions as the pre-process encode task, but in addition can
>   be used for hardware motion compensated deinterlacing. Frames can
>   come either directly from the CSI or from the SMFC entities (memory
>   buffers via IDMAC channels). Scaled output is limited to 1024x1024
>   resolution. Output frames can be routed to various sinks including
>   the post-processing task entities.
> 
> - Post-processing task: same conversions as pre-process encode. However
>   this entity sends frames to the i.MX IPU image converter which supports
>   image tiling, which allows scaled output up to 4096x4096 resolution.
>   Output frames can be routed to the camera interfaces.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>

Applying: media: imx: Add IC subdev drivers
.git/rebase-apply/patch:3054: new blank line at EOF.
+
warning: 1 line adds whitespace errors.


-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 19/24] media: imx: Add IC subdev drivers
@ 2017-01-30 22:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:29 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:37PM -0800, Steve Longerbeam wrote:
> This is a set of three media entity subdevice drivers for the i.MX
> Image Converter. The i.MX IC module contains three independent
> "tasks":
> 
> - Pre-processing Encode task: video frames are routed directly from
>   the CSI and can be scaled, color-space converted, and rotated.
>   Scaled output is limited to 1024x1024 resolution. Output frames
>   are routed to the camera interface entities (camif).
> 
> - Pre-processing Viewfinder task: this task can perform the same
>   conversions as the pre-process encode task, but in addition can
>   be used for hardware motion compensated deinterlacing. Frames can
>   come either directly from the CSI or from the SMFC entities (memory
>   buffers via IDMAC channels). Scaled output is limited to 1024x1024
>   resolution. Output frames can be routed to various sinks including
>   the post-processing task entities.
> 
> - Post-processing task: same conversions as pre-process encode. However
>   this entity sends frames to the i.MX IPU image converter which supports
>   image tiling, which allows scaled output up to 4096x4096 resolution.
>   Output frames can be routed to the camera interfaces.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: imx: Add IC subdev drivers
.git/rebase-apply/patch:3054: new blank line at EOF.
+
warning: 1 line adds whitespace errors.


-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-30 22:30     ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
> for sensors with a MIPI CSI2 interface.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: imx: Add MIPI CSI-2 Receiver subdev driver
.git/rebase-apply/patch:522: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-30 22:30     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
> for sensors with a MIPI CSI2 interface.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: imx: Add MIPI CSI-2 Receiver subdev driver
.git/rebase-apply/patch:522: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-30 22:30     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:30 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
> for sensors with a MIPI CSI2 interface.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>

Applying: media: imx: Add MIPI CSI-2 Receiver subdev driver
.git/rebase-apply/patch:522: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-30 22:51     ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:51 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> +	ov5640: camera@40 {
> +		compatible = "ovti,ov5640";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5640>;
> +		clocks = <&mipi_xclk>;
> +		clock-names = "xclk";
> +		reg = <0x40>;
> +		xclk = <22000000>;
> +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> +
> +		port {
> +			#address-cells = <1>;
> +			#size-cells = <0>;
> +
> +			ov5640_to_mipi_csi: endpoint@1 {
> +				reg = <1>;
> +				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
> +				data-lanes = <0 1>;
> +				clock-lanes = <2>;

How do you envision a four-lane sensor being described?

	data-lanes = <0 1 3 4>;
	clock-lanes = <2>;

?

The binding document for video-interfaces.txt says:

- clock-lanes: an array of physical clock lane indexes. Position of an entry
  determines the logical lane number, while the value of an entry indicates
  physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes = <0>;",
  which places the clock lane on hardware lane 0. This property is valid for
  serial busses only (e.g. MIPI CSI-2). Note that for the MIPI CSI-2 bus this
  array contains only one entry.

So I think you need to have a good reason to make the clock lane non-zero.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-30 22:51     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:51 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> +	ov5640: camera@40 {
> +		compatible = "ovti,ov5640";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5640>;
> +		clocks = <&mipi_xclk>;
> +		clock-names = "xclk";
> +		reg = <0x40>;
> +		xclk = <22000000>;
> +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> +
> +		port {
> +			#address-cells = <1>;
> +			#size-cells = <0>;
> +
> +			ov5640_to_mipi_csi: endpoint@1 {
> +				reg = <1>;
> +				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
> +				data-lanes = <0 1>;
> +				clock-lanes = <2>;

How do you envision a four-lane sensor being described?

	data-lanes = <0 1 3 4>;
	clock-lanes = <2>;

?

The binding document for video-interfaces.txt says:

- clock-lanes: an array of physical clock lane indexes. Position of an entry
  determines the logical lane number, while the value of an entry indicates
  physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes = <0>;",
  which places the clock lane on hardware lane 0. This property is valid for
  serial busses only (e.g. MIPI CSI-2). Note that for the MIPI CSI-2 bus this
  array contains only one entry.

So I think you need to have a good reason to make the clock lane non-zero.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-01-30 22:51     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 22:51 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> +	ov5640: camera at 40 {
> +		compatible = "ovti,ov5640";
> +		pinctrl-names = "default";
> +		pinctrl-0 = <&pinctrl_ov5640>;
> +		clocks = <&mipi_xclk>;
> +		clock-names = "xclk";
> +		reg = <0x40>;
> +		xclk = <22000000>;
> +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> +
> +		port {
> +			#address-cells = <1>;
> +			#size-cells = <0>;
> +
> +			ov5640_to_mipi_csi: endpoint at 1 {
> +				reg = <1>;
> +				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
> +				data-lanes = <0 1>;
> +				clock-lanes = <2>;

How do you envision a four-lane sensor being described?

	data-lanes = <0 1 3 4>;
	clock-lanes = <2>;

?

The binding document for video-interfaces.txt says:

- clock-lanes: an array of physical clock lane indexes. Position of an entry
  determines the logical lane number, while the value of an entry indicates
  physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes = <0>;",
  which places the clock lane on hardware lane 0. This property is valid for
  serial busses only (e.g. MIPI CSI-2). Note that for the MIPI CSI-2 bus this
  array contains only one entry.

So I think you need to have a good reason to make the clock lane non-zero.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-30 23:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 23:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:40PM -0800, Steve Longerbeam wrote:
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> +       select IMX_MIPI_CSI2
> +       default y

Why is this defaulting to y?  New drivers should not default to enabled
unless they are replacing some already pre-existing functionality.
Ditto for the other camera driver.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-30 23:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 23:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:40PM -0800, Steve Longerbeam wrote:
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> +       select IMX_MIPI_CSI2
> +       default y

Why is this defaulting to y?  New drivers should not default to enabled
unless they are replacing some already pre-existing functionality.
Ditto for the other camera driver.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-30 23:29     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-30 23:29 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:40PM -0800, Steve Longerbeam wrote:
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> +       select IMX_MIPI_CSI2
> +       default y

Why is this defaulting to y?  New drivers should not default to enabled
unless they are replacing some already pre-existing functionality.
Ditto for the other camera driver.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  0:01     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:01 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
> +{
> +	if (enable) {
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
> +	} else {
> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
> +	}
> +}
> +
> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
> +{
> +	imxcsi2_enable(csi2, false);
> +
> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +
> +	imxcsi2_enable(csi2, true);
> +}
> +
> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
> +{
> +	u32 reg;
> +	int i;
> +
> +	/* wait for mipi sensor ready */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg != 0x200)
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* wait for mipi stable */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_ERR1);
> +		if (reg == 0x0)
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for controller timeout, err1 = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* finally let's wait for active clock on the clock lane */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg & (1 << 8))
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for active clock timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
> +		  imxcsi2_read(csi2, CSI2_VERSION));
> +
> +	return 0;
> +}
...
> +static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	if (on && !csi2->on) {
> +		v4l2_info(&csi2->sd, "power ON\n");
> +		clk_prepare_enable(csi2->cfg_clk);
> +		clk_prepare_enable(csi2->dphy_clk);
> +		imxcsi2_set_lanes(csi2);
> +		imxcsi2_reset(csi2);

The iMX6 manuals call for a very specific seven sequence of initialisation
for CSI2, which begins with:

1. reset the D-PHY.
2. place MIPI sensor in LP-11 state
3. perform D-PHY initialisation
4. configure CSI2 lanes and de-assert resets and shutdown signals

Since you reset the CSI2 at power up and then release it, how do you
guarantee that the published sequence is followed?

With Philipp's driver, this is easy, because there is a prepare_stream
callback which gives the sensor an opportunity to get everything
correctly configured according to the negotiated parameters, and place
the sensor in LP-11 state.

Some sensors do not power up in LP-11 state, but need to be programmed
fully before being asked to momentarily stream.  Only at that point is
the sensor guaranteed to be in the required LP-11 state.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  0:01     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:01 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
> +{
> +	if (enable) {
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
> +	} else {
> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
> +	}
> +}
> +
> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
> +{
> +	imxcsi2_enable(csi2, false);
> +
> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +
> +	imxcsi2_enable(csi2, true);
> +}
> +
> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
> +{
> +	u32 reg;
> +	int i;
> +
> +	/* wait for mipi sensor ready */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg != 0x200)
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* wait for mipi stable */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_ERR1);
> +		if (reg == 0x0)
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for controller timeout, err1 = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* finally let's wait for active clock on the clock lane */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg & (1 << 8))
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for active clock timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
> +		  imxcsi2_read(csi2, CSI2_VERSION));
> +
> +	return 0;
> +}
...
> +static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	if (on && !csi2->on) {
> +		v4l2_info(&csi2->sd, "power ON\n");
> +		clk_prepare_enable(csi2->cfg_clk);
> +		clk_prepare_enable(csi2->dphy_clk);
> +		imxcsi2_set_lanes(csi2);
> +		imxcsi2_reset(csi2);

The iMX6 manuals call for a very specific seven sequence of initialisation
for CSI2, which begins with:

1. reset the D-PHY.
2. place MIPI sensor in LP-11 state
3. perform D-PHY initialisation
4. configure CSI2 lanes and de-assert resets and shutdown signals

Since you reset the CSI2 at power up and then release it, how do you
guarantee that the published sequence is followed?

With Philipp's driver, this is easy, because there is a prepare_stream
callback which gives the sensor an opportunity to get everything
correctly configured according to the negotiated parameters, and place
the sensor in LP-11 state.

Some sensors do not power up in LP-11 state, but need to be programmed
fully before being asked to momentarily stream.  Only at that point is
the sensor guaranteed to be in the required LP-11 state.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  0:01     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:01 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
> +{
> +	if (enable) {
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
> +	} else {
> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
> +	}
> +}
> +
> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
> +{
> +	imxcsi2_enable(csi2, false);
> +
> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +
> +	imxcsi2_enable(csi2, true);
> +}
> +
> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
> +{
> +	u32 reg;
> +	int i;
> +
> +	/* wait for mipi sensor ready */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg != 0x200)
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* wait for mipi stable */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_ERR1);
> +		if (reg == 0x0)
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for controller timeout, err1 = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* finally let's wait for active clock on the clock lane */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg & (1 << 8))
> +			break;
> +		usleep_range(10000, 20000);
> +	}
> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for active clock timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
> +		  imxcsi2_read(csi2, CSI2_VERSION));
> +
> +	return 0;
> +}
...
> +static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	if (on && !csi2->on) {
> +		v4l2_info(&csi2->sd, "power ON\n");
> +		clk_prepare_enable(csi2->cfg_clk);
> +		clk_prepare_enable(csi2->dphy_clk);
> +		imxcsi2_set_lanes(csi2);
> +		imxcsi2_reset(csi2);

The iMX6 manuals call for a very specific seven sequence of initialisation
for CSI2, which begins with:

1. reset the D-PHY.
2. place MIPI sensor in LP-11 state
3. perform D-PHY initialisation
4. configure CSI2 lanes and de-assert resets and shutdown signals

Since you reset the CSI2 at power up and then release it, how do you
guarantee that the published sequence is followed?

With Philipp's driver, this is easy, because there is a prepare_stream
callback which gives the sensor an opportunity to get everything
correctly configured according to the negotiated parameters, and place
the sensor in LP-11 state.

Some sensors do not power up in LP-11 state, but need to be programmed
fully before being asked to momentarily stream.  Only at that point is
the sensor guaranteed to be in the required LP-11 state.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-31  0:31     ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:31 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
...
> +#define DEVICE_NAME "imx6-mipi-csi2"

Why is the device/driver named imx6-mipi-csi2, but the module named
imx-mipi-csi2 - could there be some consistency here please?

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  0:31     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:31 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
...
> +#define DEVICE_NAME "imx6-mipi-csi2"

Why is the device/driver named imx6-mipi-csi2, but the module named
imx-mipi-csi2 - could there be some consistency here please?

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  0:31     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:31 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
...
> +#define DEVICE_NAME "imx6-mipi-csi2"

Why is the device/driver named imx6-mipi-csi2, but the module named
imx-mipi-csi2 - could there be some consistency here please?

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-31  0:45   ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:45 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devel, devicetree, Steve Longerbeam,
	linux-kernel, linux-arm-kernel, linux-media

On Fri, Jan 06, 2017 at 06:11:18PM -0800, Steve Longerbeam wrote:
> Philipp Zabel (3):
>   ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
>     connections
>   add mux and video interface bridge entity functions
>   platform: add video-multiplexer subdevice driver
> 
> Steve Longerbeam (21):
>   [media] dt-bindings: Add bindings for i.MX media driver
>   ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
>   ARM: dts: imx6qdl: add media device
>   ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
>   ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
>   ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
>   ARM: dts: imx6-sabreauto: create i2cmux for i2c3
>   ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
>   ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
>   ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
>   UAPI: Add media UAPI Kbuild file
>   media: Add userspace header file for i.MX
>   media: Add i.MX media core driver
>   media: imx: Add CSI subdev driver
>   media: imx: Add SMFC subdev driver
>   media: imx: Add IC subdev drivers
>   media: imx: Add Camera Interface subdev driver
>   media: imx: Add MIPI CSI-2 Receiver subdev driver
>   media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
>   media: imx: Add Parallel OV5642 sensor subdev driver
>   ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

Hi,

Trying this driver with an imx219 camera (which works with Philipp's
driver) results in not much happening... no /dev/media* node for it,
no subdevs, no nothing.  No clues as to what's missing either.  Only
messages from imx-media are from registering the various subdevs.

[   37.444877] imx-media: Registered subdev imx6-mipi-csi2
[   37.444973] imx-media: Registered subdev imx219 0-0010
[   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
[   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
[   38.869425] imx-media: Registered subdev ipu1_ic_pp0
[   38.870086] imx-media: Registered subdev ipu1_ic_pp1
[   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
[   38.871743] imx-media: Registered subdev ipu1_smfc0
[   38.873043] imx-media: Registered subdev ipu1_smfc1
[   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
[   38.875027] imx-media: Registered subdev ipu2_smfc0
[   38.875320] imx-media: Registered subdev ipu2_ic_pp0
[   38.877148] imx-media: Registered subdev ipu2_smfc1
[   38.877436] imx-media: Registered subdev ipu2_ic_pp1
[   38.932089] imx-media: Registered subdev camif0
[   38.956538] imx-media: Registered subdev camif1
[   38.959148] imx-media: Registered subdev camif2
[   38.964353] imx-media: Registered subdev camif3
[  206.502077] imx-media: Registered subdev ipu1_csi0
[  206.503304] imx-media: Registered subdev ipu1_csi1
[  206.503814] imx-media: Registered subdev ipu2_csi0
[  206.504281] imx-media: Registered subdev ipu2_csi1

I also get:

[   37.200072] imx6-mipi-csi2: data lanes: 2
[   37.200077] imx6-mipi-csi2: flags: 0x00000200

and from what I can see, all modules from drivers/staging/media/imx/ are
loaded (had to load imx-csi by hand because of the brokenness in the
drivers/gpu/ipu code attaching an device_node pointer after registering
the platform device, which changes what userspace sees in the modalias
file.)

Any clues at what to look at?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  0:45   ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:45 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Fri, Jan 06, 2017 at 06:11:18PM -0800, Steve Longerbeam wrote:
> Philipp Zabel (3):
>   ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
>     connections
>   add mux and video interface bridge entity functions
>   platform: add video-multiplexer subdevice driver
> 
> Steve Longerbeam (21):
>   [media] dt-bindings: Add bindings for i.MX media driver
>   ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
>   ARM: dts: imx6qdl: add media device
>   ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
>   ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
>   ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
>   ARM: dts: imx6-sabreauto: create i2cmux for i2c3
>   ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
>   ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
>   ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
>   UAPI: Add media UAPI Kbuild file
>   media: Add userspace header file for i.MX
>   media: Add i.MX media core driver
>   media: imx: Add CSI subdev driver
>   media: imx: Add SMFC subdev driver
>   media: imx: Add IC subdev drivers
>   media: imx: Add Camera Interface subdev driver
>   media: imx: Add MIPI CSI-2 Receiver subdev driver
>   media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
>   media: imx: Add Parallel OV5642 sensor subdev driver
>   ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

Hi,

Trying this driver with an imx219 camera (which works with Philipp's
driver) results in not much happening... no /dev/media* node for it,
no subdevs, no nothing.  No clues as to what's missing either.  Only
messages from imx-media are from registering the various subdevs.

[   37.444877] imx-media: Registered subdev imx6-mipi-csi2
[   37.444973] imx-media: Registered subdev imx219 0-0010
[   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
[   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
[   38.869425] imx-media: Registered subdev ipu1_ic_pp0
[   38.870086] imx-media: Registered subdev ipu1_ic_pp1
[   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
[   38.871743] imx-media: Registered subdev ipu1_smfc0
[   38.873043] imx-media: Registered subdev ipu1_smfc1
[   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
[   38.875027] imx-media: Registered subdev ipu2_smfc0
[   38.875320] imx-media: Registered subdev ipu2_ic_pp0
[   38.877148] imx-media: Registered subdev ipu2_smfc1
[   38.877436] imx-media: Registered subdev ipu2_ic_pp1
[   38.932089] imx-media: Registered subdev camif0
[   38.956538] imx-media: Registered subdev camif1
[   38.959148] imx-media: Registered subdev camif2
[   38.964353] imx-media: Registered subdev camif3
[  206.502077] imx-media: Registered subdev ipu1_csi0
[  206.503304] imx-media: Registered subdev ipu1_csi1
[  206.503814] imx-media: Registered subdev ipu2_csi0
[  206.504281] imx-media: Registered subdev ipu2_csi1

I also get:

[   37.200072] imx6-mipi-csi2: data lanes: 2
[   37.200077] imx6-mipi-csi2: flags: 0x00000200

and from what I can see, all modules from drivers/staging/media/imx/ are
loaded (had to load imx-csi by hand because of the brokenness in the
drivers/gpu/ipu code attaching an device_node pointer after registering
the platform device, which changes what userspace sees in the modalias
file.)

Any clues at what to look at?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  0:45   ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  0:45 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:18PM -0800, Steve Longerbeam wrote:
> Philipp Zabel (3):
>   ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
>     connections
>   add mux and video interface bridge entity functions
>   platform: add video-multiplexer subdevice driver
> 
> Steve Longerbeam (21):
>   [media] dt-bindings: Add bindings for i.MX media driver
>   ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
>   ARM: dts: imx6qdl: add media device
>   ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
>   ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
>   ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
>   ARM: dts: imx6-sabreauto: create i2cmux for i2c3
>   ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
>   ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
>   ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
>   UAPI: Add media UAPI Kbuild file
>   media: Add userspace header file for i.MX
>   media: Add i.MX media core driver
>   media: imx: Add CSI subdev driver
>   media: imx: Add SMFC subdev driver
>   media: imx: Add IC subdev drivers
>   media: imx: Add Camera Interface subdev driver
>   media: imx: Add MIPI CSI-2 Receiver subdev driver
>   media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
>   media: imx: Add Parallel OV5642 sensor subdev driver
>   ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

Hi,

Trying this driver with an imx219 camera (which works with Philipp's
driver) results in not much happening... no /dev/media* node for it,
no subdevs, no nothing.  No clues as to what's missing either.  Only
messages from imx-media are from registering the various subdevs.

[   37.444877] imx-media: Registered subdev imx6-mipi-csi2
[   37.444973] imx-media: Registered subdev imx219 0-0010
[   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
[   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
[   38.869425] imx-media: Registered subdev ipu1_ic_pp0
[   38.870086] imx-media: Registered subdev ipu1_ic_pp1
[   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
[   38.871743] imx-media: Registered subdev ipu1_smfc0
[   38.873043] imx-media: Registered subdev ipu1_smfc1
[   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
[   38.875027] imx-media: Registered subdev ipu2_smfc0
[   38.875320] imx-media: Registered subdev ipu2_ic_pp0
[   38.877148] imx-media: Registered subdev ipu2_smfc1
[   38.877436] imx-media: Registered subdev ipu2_ic_pp1
[   38.932089] imx-media: Registered subdev camif0
[   38.956538] imx-media: Registered subdev camif1
[   38.959148] imx-media: Registered subdev camif2
[   38.964353] imx-media: Registered subdev camif3
[  206.502077] imx-media: Registered subdev ipu1_csi0
[  206.503304] imx-media: Registered subdev ipu1_csi1
[  206.503814] imx-media: Registered subdev ipu2_csi0
[  206.504281] imx-media: Registered subdev ipu2_csi1

I also get:

[   37.200072] imx6-mipi-csi2: data lanes: 2
[   37.200077] imx6-mipi-csi2: flags: 0x00000200

and from what I can see, all modules from drivers/staging/media/imx/ are
loaded (had to load imx-csi by hand because of the brokenness in the
drivers/gpu/ipu code attaching an device_node pointer after registering
the platform device, which changes what userspace sees in the modalias
file.)

Any clues at what to look at?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31  0:45   ` Russell King - ARM Linux
  (?)
@ 2017-01-31  1:06     ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  1:06 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 12:45:11AM +0000, Russell King - ARM Linux wrote:
> Trying this driver with an imx219 camera (which works with Philipp's
> driver) results in not much happening... no /dev/media* node for it,
> no subdevs, no nothing.  No clues as to what's missing either.  Only
> messages from imx-media are from registering the various subdevs.

Another issue:

imx_csi                 5491  4
imx_camif              11654  4
imx_ic                 23961  8
imx_smfc                6639  4
imx_media              23308  1 imx_csi
imx_mipi_csi2           5544  1
imx_media_common       12701  6 imx_csi,imx_smfc,imx_media,imx_mipi_csi2,imx_camif,imx_ic
imx219                 21205  2

So how does one remove any of these modules, say, while developing a
camera driver?  Having to reboot to test an update makes it painfully
slow for testing.

Philipp's driver can do this (once the unload bugs are fixed, which I
have patches for).

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  1:06     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  1:06 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 12:45:11AM +0000, Russell King - ARM Linux wrote:
> Trying this driver with an imx219 camera (which works with Philipp's
> driver) results in not much happening... no /dev/media* node for it,
> no subdevs, no nothing.  No clues as to what's missing either.  Only
> messages from imx-media are from registering the various subdevs.

Another issue:

imx_csi                 5491  4
imx_camif              11654  4
imx_ic                 23961  8
imx_smfc                6639  4
imx_media              23308  1 imx_csi
imx_mipi_csi2           5544  1
imx_media_common       12701  6 imx_csi,imx_smfc,imx_media,imx_mipi_csi2,imx_camif,imx_ic
imx219                 21205  2

So how does one remove any of these modules, say, while developing a
camera driver?  Having to reboot to test an update makes it painfully
slow for testing.

Philipp's driver can do this (once the unload bugs are fixed, which I
have patches for).

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  1:06     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31  1:06 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 12:45:11AM +0000, Russell King - ARM Linux wrote:
> Trying this driver with an imx219 camera (which works with Philipp's
> driver) results in not much happening... no /dev/media* node for it,
> no subdevs, no nothing.  No clues as to what's missing either.  Only
> messages from imx-media are from registering the various subdevs.

Another issue:

imx_csi                 5491  4
imx_camif              11654  4
imx_ic                 23961  8
imx_smfc                6639  4
imx_media              23308  1 imx_csi
imx_mipi_csi2           5544  1
imx_media_common       12701  6 imx_csi,imx_smfc,imx_media,imx_mipi_csi2,imx_camif,imx_ic
imx219                 21205  2

So how does one remove any of these modules, say, while developing a
camera driver?  Having to reboot to test an update makes it painfully
slow for testing.

Philipp's driver can do this (once the unload bugs are fixed, which I
have patches for).

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  1:22     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  1:22 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devel, devicetree, Steve Longerbeam,
	linux-kernel, linux-arm-kernel, linux-media



On 01/30/2017 04:45 PM, Russell King - ARM Linux wrote:
>
> Hi,
>
> Trying this driver with an imx219 camera (which works with Philipp's
> driver) results in not much happening... no /dev/media* node for it,
> no subdevs, no nothing.  No clues as to what's missing either.  Only
> messages from imx-media are from registering the various subdevs.
>
> [   37.444877] imx-media: Registered subdev imx6-mipi-csi2
> [   37.444973] imx-media: Registered subdev imx219 0-0010
> [   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
> [   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
> [   38.869425] imx-media: Registered subdev ipu1_ic_pp0
> [   38.870086] imx-media: Registered subdev ipu1_ic_pp1
> [   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
> [   38.871743] imx-media: Registered subdev ipu1_smfc0
> [   38.873043] imx-media: Registered subdev ipu1_smfc1
> [   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
> [   38.875027] imx-media: Registered subdev ipu2_smfc0
> [   38.875320] imx-media: Registered subdev ipu2_ic_pp0
> [   38.877148] imx-media: Registered subdev ipu2_smfc1
> [   38.877436] imx-media: Registered subdev ipu2_ic_pp1
> [   38.932089] imx-media: Registered subdev camif0
> [   38.956538] imx-media: Registered subdev camif1
> [   38.959148] imx-media: Registered subdev camif2
> [   38.964353] imx-media: Registered subdev camif3
> [  206.502077] imx-media: Registered subdev ipu1_csi0
> [  206.503304] imx-media: Registered subdev ipu1_csi1
> [  206.503814] imx-media: Registered subdev ipu2_csi0
> [  206.504281] imx-media: Registered subdev ipu2_csi1
>
> I also get:
>
> [   37.200072] imx6-mipi-csi2: data lanes: 2
> [   37.200077] imx6-mipi-csi2: flags: 0x00000200
>
> and from what I can see, all modules from drivers/staging/media/imx/ are
> loaded (had to load imx-csi by hand because of the brokenness in the
> drivers/gpu/ipu code attaching an device_node pointer after registering
> the platform device, which changes what userspace sees in the modalias
> file.)
>
> Any clues at what to look at?

Hi Russell,

I'm not familiar with IMX219, can you send me the source for the
imx219 subdev? I don't see it in 4.10-rc1.

I'm also having trouble finding a datasheet for it, but from what
I've read, it has a MIPI CSI-2 interface. It should work fine as long
as it presents a single source pad, registers asynchronously, and
sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

Since I see it was registered asynchronously from the above, it
must have been added to the device tree. But given that there
is no /dev/media? node, the media driver is probably waiting for
another subdev to register, I don't know what that would be.

Can you send me the full patch on top of the v3 driver and I'll
try to find what's missing.

Edit: I see a subdev that is missing: the video mux. Did you enable
CONFIG_VIDEO_MULTIPLEXER?

Finally, what platform does this IMX219 sensor module plug into?


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  1:22     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  1:22 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b,
	devicetree-u79uwXL29TY76Z2rM5mHXA, Steve Longerbeam,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, linux-media



On 01/30/2017 04:45 PM, Russell King - ARM Linux wrote:
>
> Hi,
>
> Trying this driver with an imx219 camera (which works with Philipp's
> driver) results in not much happening... no /dev/media* node for it,
> no subdevs, no nothing.  No clues as to what's missing either.  Only
> messages from imx-media are from registering the various subdevs.
>
> [   37.444877] imx-media: Registered subdev imx6-mipi-csi2
> [   37.444973] imx-media: Registered subdev imx219 0-0010
> [   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
> [   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
> [   38.869425] imx-media: Registered subdev ipu1_ic_pp0
> [   38.870086] imx-media: Registered subdev ipu1_ic_pp1
> [   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
> [   38.871743] imx-media: Registered subdev ipu1_smfc0
> [   38.873043] imx-media: Registered subdev ipu1_smfc1
> [   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
> [   38.875027] imx-media: Registered subdev ipu2_smfc0
> [   38.875320] imx-media: Registered subdev ipu2_ic_pp0
> [   38.877148] imx-media: Registered subdev ipu2_smfc1
> [   38.877436] imx-media: Registered subdev ipu2_ic_pp1
> [   38.932089] imx-media: Registered subdev camif0
> [   38.956538] imx-media: Registered subdev camif1
> [   38.959148] imx-media: Registered subdev camif2
> [   38.964353] imx-media: Registered subdev camif3
> [  206.502077] imx-media: Registered subdev ipu1_csi0
> [  206.503304] imx-media: Registered subdev ipu1_csi1
> [  206.503814] imx-media: Registered subdev ipu2_csi0
> [  206.504281] imx-media: Registered subdev ipu2_csi1
>
> I also get:
>
> [   37.200072] imx6-mipi-csi2: data lanes: 2
> [   37.200077] imx6-mipi-csi2: flags: 0x00000200
>
> and from what I can see, all modules from drivers/staging/media/imx/ are
> loaded (had to load imx-csi by hand because of the brokenness in the
> drivers/gpu/ipu code attaching an device_node pointer after registering
> the platform device, which changes what userspace sees in the modalias
> file.)
>
> Any clues at what to look at?

Hi Russell,

I'm not familiar with IMX219, can you send me the source for the
imx219 subdev? I don't see it in 4.10-rc1.

I'm also having trouble finding a datasheet for it, but from what
I've read, it has a MIPI CSI-2 interface. It should work fine as long
as it presents a single source pad, registers asynchronously, and
sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

Since I see it was registered asynchronously from the above, it
must have been added to the device tree. But given that there
is no /dev/media? node, the media driver is probably waiting for
another subdev to register, I don't know what that would be.

Can you send me the full patch on top of the v3 driver and I'll
try to find what's missing.

Edit: I see a subdev that is missing: the video mux. Did you enable
CONFIG_VIDEO_MULTIPLEXER?

Finally, what platform does this IMX219 sensor module plug into?


Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  1:22     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  1:22 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/30/2017 04:45 PM, Russell King - ARM Linux wrote:
>
> Hi,
>
> Trying this driver with an imx219 camera (which works with Philipp's
> driver) results in not much happening... no /dev/media* node for it,
> no subdevs, no nothing.  No clues as to what's missing either.  Only
> messages from imx-media are from registering the various subdevs.
>
> [   37.444877] imx-media: Registered subdev imx6-mipi-csi2
> [   37.444973] imx-media: Registered subdev imx219 0-0010
> [   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
> [   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
> [   38.869425] imx-media: Registered subdev ipu1_ic_pp0
> [   38.870086] imx-media: Registered subdev ipu1_ic_pp1
> [   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
> [   38.871743] imx-media: Registered subdev ipu1_smfc0
> [   38.873043] imx-media: Registered subdev ipu1_smfc1
> [   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
> [   38.875027] imx-media: Registered subdev ipu2_smfc0
> [   38.875320] imx-media: Registered subdev ipu2_ic_pp0
> [   38.877148] imx-media: Registered subdev ipu2_smfc1
> [   38.877436] imx-media: Registered subdev ipu2_ic_pp1
> [   38.932089] imx-media: Registered subdev camif0
> [   38.956538] imx-media: Registered subdev camif1
> [   38.959148] imx-media: Registered subdev camif2
> [   38.964353] imx-media: Registered subdev camif3
> [  206.502077] imx-media: Registered subdev ipu1_csi0
> [  206.503304] imx-media: Registered subdev ipu1_csi1
> [  206.503814] imx-media: Registered subdev ipu2_csi0
> [  206.504281] imx-media: Registered subdev ipu2_csi1
>
> I also get:
>
> [   37.200072] imx6-mipi-csi2: data lanes: 2
> [   37.200077] imx6-mipi-csi2: flags: 0x00000200
>
> and from what I can see, all modules from drivers/staging/media/imx/ are
> loaded (had to load imx-csi by hand because of the brokenness in the
> drivers/gpu/ipu code attaching an device_node pointer after registering
> the platform device, which changes what userspace sees in the modalias
> file.)
>
> Any clues at what to look at?

Hi Russell,

I'm not familiar with IMX219, can you send me the source for the
imx219 subdev? I don't see it in 4.10-rc1.

I'm also having trouble finding a datasheet for it, but from what
I've read, it has a MIPI CSI-2 interface. It should work fine as long
as it presents a single source pad, registers asynchronously, and
sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

Since I see it was registered asynchronously from the above, it
must have been added to the device tree. But given that there
is no /dev/media? node, the media driver is probably waiting for
another subdev to register, I don't know what that would be.

Can you send me the full patch on top of the v3 driver and I'll
try to find what's missing.

Edit: I see a subdev that is missing: the video mux. Did you enable
CONFIG_VIDEO_MULTIPLEXER?

Finally, what platform does this IMX219 sensor module plug into?


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31  1:06     ` Russell King - ARM Linux
  (?)
@ 2017-01-31  2:06       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  2:06 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/30/2017 05:06 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 12:45:11AM +0000, Russell King - ARM Linux wrote:
>> Trying this driver with an imx219 camera (which works with Philipp's
>> driver) results in not much happening... no /dev/media* node for it,
>> no subdevs, no nothing.  No clues as to what's missing either.  Only
>> messages from imx-media are from registering the various subdevs.
> Another issue:
>
> imx_csi                 5491  4
> imx_camif              11654  4
> imx_ic                 23961  8
> imx_smfc                6639  4
> imx_media              23308  1 imx_csi
> imx_mipi_csi2           5544  1
> imx_media_common       12701  6 imx_csi,imx_smfc,imx_media,imx_mipi_csi2,imx_camif,imx_ic
> imx219                 21205  2
>
> So how does one remove any of these modules, say, while developing a
> camera driver?  Having to reboot to test an update makes it painfully
> slow for testing.

Unload is not working yet, it's on the TODO list.

But FWIW, here's how it currently looks in version 4
(on the SabreSD):

imx_media_csi           9663  4
imx_media_ic           12688  6
imx_media_capture      10201  2 imx_media_ic,imx_media_csi
imx_media_vdic          6909  2
imx_mipi_csi2           6293  1
ov5640_mipi            25988  1
imx_media              15532  0
imx_media_common       16093  6 
imx_media_ic,imx_media,imx_media_csi,imx_mipi_cs
i2,imx_media_capture,imx_media_vdic


Steve


>
> Philipp's driver can do this (once the unload bugs are fixed, which I
> have patches for).
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  2:06       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  2:06 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/30/2017 05:06 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 12:45:11AM +0000, Russell King - ARM Linux wrote:
>> Trying this driver with an imx219 camera (which works with Philipp's
>> driver) results in not much happening... no /dev/media* node for it,
>> no subdevs, no nothing.  No clues as to what's missing either.  Only
>> messages from imx-media are from registering the various subdevs.
> Another issue:
>
> imx_csi                 5491  4
> imx_camif              11654  4
> imx_ic                 23961  8
> imx_smfc                6639  4
> imx_media              23308  1 imx_csi
> imx_mipi_csi2           5544  1
> imx_media_common       12701  6 imx_csi,imx_smfc,imx_media,imx_mipi_csi2,imx_camif,imx_ic
> imx219                 21205  2
>
> So how does one remove any of these modules, say, while developing a
> camera driver?  Having to reboot to test an update makes it painfully
> slow for testing.

Unload is not working yet, it's on the TODO list.

But FWIW, here's how it currently looks in version 4
(on the SabreSD):

imx_media_csi           9663  4
imx_media_ic           12688  6
imx_media_capture      10201  2 imx_media_ic,imx_media_csi
imx_media_vdic          6909  2
imx_mipi_csi2           6293  1
ov5640_mipi            25988  1
imx_media              15532  0
imx_media_common       16093  6 
imx_media_ic,imx_media,imx_media_csi,imx_mipi_cs
i2,imx_media_capture,imx_media_vdic


Steve


>
> Philipp's driver can do this (once the unload bugs are fixed, which I
> have patches for).
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  2:06       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  2:06 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/30/2017 05:06 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 12:45:11AM +0000, Russell King - ARM Linux wrote:
>> Trying this driver with an imx219 camera (which works with Philipp's
>> driver) results in not much happening... no /dev/media* node for it,
>> no subdevs, no nothing.  No clues as to what's missing either.  Only
>> messages from imx-media are from registering the various subdevs.
> Another issue:
>
> imx_csi                 5491  4
> imx_camif              11654  4
> imx_ic                 23961  8
> imx_smfc                6639  4
> imx_media              23308  1 imx_csi
> imx_mipi_csi2           5544  1
> imx_media_common       12701  6 imx_csi,imx_smfc,imx_media,imx_mipi_csi2,imx_camif,imx_ic
> imx219                 21205  2
>
> So how does one remove any of these modules, say, while developing a
> camera driver?  Having to reboot to test an update makes it painfully
> slow for testing.

Unload is not working yet, it's on the TODO list.

But FWIW, here's how it currently looks in version 4
(on the SabreSD):

imx_media_csi           9663  4
imx_media_ic           12688  6
imx_media_capture      10201  2 imx_media_ic,imx_media_csi
imx_media_vdic          6909  2
imx_mipi_csi2           6293  1
ov5640_mipi            25988  1
imx_media              15532  0
imx_media_common       16093  6 
imx_media_ic,imx_media,imx_media_csi,imx_mipi_cs
i2,imx_media_capture,imx_media_vdic


Steve


>
> Philipp's driver can do this (once the unload bugs are fixed, which I
> have patches for).
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-01-31  0:31     ` Russell King - ARM Linux
  (?)
@ 2017-01-31  2:11       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  2:11 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/30/2017 04:31 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
>> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
> ...
>> +#define DEVICE_NAME "imx6-mipi-csi2"
> Why is the device/driver named imx6-mipi-csi2, but the module named
> imx-mipi-csi2 - could there be some consistency here please?

right, that was missed after renaming the device node. Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  2:11       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  2:11 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/30/2017 04:31 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
>> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
> ...
>> +#define DEVICE_NAME "imx6-mipi-csi2"
> Why is the device/driver named imx6-mipi-csi2, but the module named
> imx-mipi-csi2 - could there be some consistency here please?

right, that was missed after renaming the device node. Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  2:11       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  2:11 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/30/2017 04:31 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
>> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
> ...
>> +#define DEVICE_NAME "imx6-mipi-csi2"
> Why is the device/driver named imx6-mipi-csi2, but the module named
> imx-mipi-csi2 - could there be some consistency here please?

right, that was missed after renaming the device node. Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  2017-01-30 23:29     ` Russell King - ARM Linux
  (?)
@ 2017-01-31  3:31       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  3:31 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/30/2017 03:29 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:40PM -0800, Steve Longerbeam wrote:
>> +config IMX_OV5640_MIPI
>> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
>> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
>> +       select IMX_MIPI_CSI2
>> +       default y
> Why is this defaulting to y?  New drivers should not default to enabled
> unless they are replacing some already pre-existing functionality.
> Ditto for the other camera driver.

The ov564x sensors are required for the SabreSD and SabreLite/Nitrogen
reference boards (if VIDEO_IMX_CAMERA is enabled that is). But they're not
required for other platforms, so you're right I shouldn't have defaulted 
these
to yes. Fixed.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-31  3:31       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  3:31 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/30/2017 03:29 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:40PM -0800, Steve Longerbeam wrote:
>> +config IMX_OV5640_MIPI
>> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
>> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
>> +       select IMX_MIPI_CSI2
>> +       default y
> Why is this defaulting to y?  New drivers should not default to enabled
> unless they are replacing some already pre-existing functionality.
> Ditto for the other camera driver.

The ov564x sensors are required for the SabreSD and SabreLite/Nitrogen
reference boards (if VIDEO_IMX_CAMERA is enabled that is). But they're not
required for other platforms, so you're right I shouldn't have defaulted 
these
to yes. Fixed.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-01-31  3:31       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31  3:31 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/30/2017 03:29 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:40PM -0800, Steve Longerbeam wrote:
>> +config IMX_OV5640_MIPI
>> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
>> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
>> +       select IMX_MIPI_CSI2
>> +       default y
> Why is this defaulting to y?  New drivers should not default to enabled
> unless they are replacing some already pre-existing functionality.
> Ditto for the other camera driver.

The ov564x sensors are required for the SabreSD and SabreLite/Nitrogen
reference boards (if VIDEO_IMX_CAMERA is enabled that is). But they're not
required for other platforms, so you're right I shouldn't have defaulted 
these
to yes. Fixed.


Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  9:49       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31  9:49 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, 2017-01-31 at 00:01 +0000, Russell King - ARM Linux wrote:
[...]
> The iMX6 manuals call for a very specific seven sequence of initialisation
> for CSI2, which begins with:
> 
> 1. reset the D-PHY.
> 2. place MIPI sensor in LP-11 state
> 3. perform D-PHY initialisation
> 4. configure CSI2 lanes and de-assert resets and shutdown signals
> 
> Since you reset the CSI2 at power up and then release it, how do you
> guarantee that the published sequence is followed?
> 
> With Philipp's driver, this is easy, because there is a prepare_stream
> callback which gives the sensor an opportunity to get everything
> correctly configured according to the negotiated parameters, and place
> the sensor in LP-11 state.
> 
> Some sensors do not power up in LP-11 state, but need to be programmed
> fully before being asked to momentarily stream.  Only at that point is
> the sensor guaranteed to be in the required LP-11 state.

Do you expect that 1. and 2. could depend on the negotiated parameters
in any way on some hardware? I had removed the prepare_stream callback
from my driver in v2 because for my use case at least the above sequence
could be realized by

1. in imx-mipi-csi2 s_power(1)
2. in MIPI sensor s_power(1)
3./4. in imx-mipi-csi2 s_stream(1)
4. in MIPI sensor s_stream(1)

as long as the sensor is correctly put back into LP-11 in s_stream(0).

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  9:49       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31  9:49 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

On Tue, 2017-01-31 at 00:01 +0000, Russell King - ARM Linux wrote:
[...]
> The iMX6 manuals call for a very specific seven sequence of initialisation
> for CSI2, which begins with:
> 
> 1. reset the D-PHY.
> 2. place MIPI sensor in LP-11 state
> 3. perform D-PHY initialisation
> 4. configure CSI2 lanes and de-assert resets and shutdown signals
> 
> Since you reset the CSI2 at power up and then release it, how do you
> guarantee that the published sequence is followed?
> 
> With Philipp's driver, this is easy, because there is a prepare_stream
> callback which gives the sensor an opportunity to get everything
> correctly configured according to the negotiated parameters, and place
> the sensor in LP-11 state.
> 
> Some sensors do not power up in LP-11 state, but need to be programmed
> fully before being asked to momentarily stream.  Only at that point is
> the sensor guaranteed to be in the required LP-11 state.

Do you expect that 1. and 2. could depend on the negotiated parameters
in any way on some hardware? I had removed the prepare_stream callback
from my driver in v2 because for my use case at least the above sequence
could be realized by

1. in imx-mipi-csi2 s_power(1)
2. in MIPI sensor s_power(1)
3./4. in imx-mipi-csi2 s_stream(1)
4. in MIPI sensor s_stream(1)

as long as the sensor is correctly put back into LP-11 in s_stream(0).

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-01-31  9:49       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31  9:49 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, 2017-01-31 at 00:01 +0000, Russell King - ARM Linux wrote:
[...]
> The iMX6 manuals call for a very specific seven sequence of initialisation
> for CSI2, which begins with:
> 
> 1. reset the D-PHY.
> 2. place MIPI sensor in LP-11 state
> 3. perform D-PHY initialisation
> 4. configure CSI2 lanes and de-assert resets and shutdown signals
> 
> Since you reset the CSI2 at power up and then release it, how do you
> guarantee that the published sequence is followed?
> 
> With Philipp's driver, this is easy, because there is a prepare_stream
> callback which gives the sensor an opportunity to get everything
> correctly configured according to the negotiated parameters, and place
> the sensor in LP-11 state.
> 
> Some sensors do not power up in LP-11 state, but need to be programmed
> fully before being asked to momentarily stream.  Only at that point is
> the sensor guaranteed to be in the required LP-11 state.

Do you expect that 1. and 2. could depend on the negotiated parameters
in any way on some hardware? I had removed the prepare_stream callback
from my driver in v2 because for my use case at least the above sequence
could be realized by

1. in imx-mipi-csi2 s_power(1)
2. in MIPI sensor s_power(1)
3./4. in imx-mipi-csi2 s_stream(1)
4. in MIPI sensor s_stream(1)

as long as the sensor is correctly put back into LP-11 in s_stream(0).

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  9:49       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31  9:49 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Russell King - ARM Linux, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devel, devicetree,
	Steve Longerbeam, linux-kernel, linux-arm-kernel, linux-media

On Mon, 2017-01-30 at 17:22 -0800, Steve Longerbeam wrote:
> 
> On 01/30/2017 04:45 PM, Russell King - ARM Linux wrote:
> >
> > Hi,
> >
> > Trying this driver with an imx219 camera (which works with Philipp's
> > driver) results in not much happening... no /dev/media* node for it,
> > no subdevs, no nothing.  No clues as to what's missing either.  Only
> > messages from imx-media are from registering the various subdevs.
> >
> > [   37.444877] imx-media: Registered subdev imx6-mipi-csi2
> > [   37.444973] imx-media: Registered subdev imx219 0-0010
> > [   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
> > [   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
> > [   38.869425] imx-media: Registered subdev ipu1_ic_pp0
> > [   38.870086] imx-media: Registered subdev ipu1_ic_pp1
> > [   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
> > [   38.871743] imx-media: Registered subdev ipu1_smfc0
> > [   38.873043] imx-media: Registered subdev ipu1_smfc1
> > [   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
> > [   38.875027] imx-media: Registered subdev ipu2_smfc0
> > [   38.875320] imx-media: Registered subdev ipu2_ic_pp0
> > [   38.877148] imx-media: Registered subdev ipu2_smfc1
> > [   38.877436] imx-media: Registered subdev ipu2_ic_pp1
> > [   38.932089] imx-media: Registered subdev camif0
> > [   38.956538] imx-media: Registered subdev camif1
> > [   38.959148] imx-media: Registered subdev camif2
> > [   38.964353] imx-media: Registered subdev camif3
> > [  206.502077] imx-media: Registered subdev ipu1_csi0
> > [  206.503304] imx-media: Registered subdev ipu1_csi1
> > [  206.503814] imx-media: Registered subdev ipu2_csi0
> > [  206.504281] imx-media: Registered subdev ipu2_csi1
> >
> > I also get:
> >
> > [   37.200072] imx6-mipi-csi2: data lanes: 2
> > [   37.200077] imx6-mipi-csi2: flags: 0x00000200
> >
> > and from what I can see, all modules from drivers/staging/media/imx/ are
> > loaded (had to load imx-csi by hand because of the brokenness in the
> > drivers/gpu/ipu code attaching an device_node pointer after registering
> > the platform device, which changes what userspace sees in the modalias
> > file.)
> >
> > Any clues at what to look at?
> 
> Hi Russell,
> 
> I'm not familiar with IMX219, can you send me the source for the
> imx219 subdev? I don't see it in 4.10-rc1.
> 
> I'm also having trouble finding a datasheet for it, but from what
> I've read, it has a MIPI CSI-2 interface. It should work fine as long
> as it presents a single source pad, registers asynchronously, and
> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

What about MEDIA_ENT_F_VID_IF_BRIDGE?

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  9:49       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31  9:49 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Russell King - ARM Linux, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b,
	devicetree-u79uwXL29TY76Z2rM5mHXA, Steve Longerbeam,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWPdpHbCvnp+Ag

On Mon, 2017-01-30 at 17:22 -0800, Steve Longerbeam wrote:
> 
> On 01/30/2017 04:45 PM, Russell King - ARM Linux wrote:
> >
> > Hi,
> >
> > Trying this driver with an imx219 camera (which works with Philipp's
> > driver) results in not much happening... no /dev/media* node for it,
> > no subdevs, no nothing.  No clues as to what's missing either.  Only
> > messages from imx-media are from registering the various subdevs.
> >
> > [   37.444877] imx-media: Registered subdev imx6-mipi-csi2
> > [   37.444973] imx-media: Registered subdev imx219 0-0010
> > [   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
> > [   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
> > [   38.869425] imx-media: Registered subdev ipu1_ic_pp0
> > [   38.870086] imx-media: Registered subdev ipu1_ic_pp1
> > [   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
> > [   38.871743] imx-media: Registered subdev ipu1_smfc0
> > [   38.873043] imx-media: Registered subdev ipu1_smfc1
> > [   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
> > [   38.875027] imx-media: Registered subdev ipu2_smfc0
> > [   38.875320] imx-media: Registered subdev ipu2_ic_pp0
> > [   38.877148] imx-media: Registered subdev ipu2_smfc1
> > [   38.877436] imx-media: Registered subdev ipu2_ic_pp1
> > [   38.932089] imx-media: Registered subdev camif0
> > [   38.956538] imx-media: Registered subdev camif1
> > [   38.959148] imx-media: Registered subdev camif2
> > [   38.964353] imx-media: Registered subdev camif3
> > [  206.502077] imx-media: Registered subdev ipu1_csi0
> > [  206.503304] imx-media: Registered subdev ipu1_csi1
> > [  206.503814] imx-media: Registered subdev ipu2_csi0
> > [  206.504281] imx-media: Registered subdev ipu2_csi1
> >
> > I also get:
> >
> > [   37.200072] imx6-mipi-csi2: data lanes: 2
> > [   37.200077] imx6-mipi-csi2: flags: 0x00000200
> >
> > and from what I can see, all modules from drivers/staging/media/imx/ are
> > loaded (had to load imx-csi by hand because of the brokenness in the
> > drivers/gpu/ipu code attaching an device_node pointer after registering
> > the platform device, which changes what userspace sees in the modalias
> > file.)
> >
> > Any clues at what to look at?
> 
> Hi Russell,
> 
> I'm not familiar with IMX219, can you send me the source for the
> imx219 subdev? I don't see it in 4.10-rc1.
> 
> I'm also having trouble finding a datasheet for it, but from what
> I've read, it has a MIPI CSI-2 interface. It should work fine as long
> as it presents a single source pad, registers asynchronously, and
> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

What about MEDIA_ENT_F_VID_IF_BRIDGE?

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31  9:49       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31  9:49 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, 2017-01-30 at 17:22 -0800, Steve Longerbeam wrote:
> 
> On 01/30/2017 04:45 PM, Russell King - ARM Linux wrote:
> >
> > Hi,
> >
> > Trying this driver with an imx219 camera (which works with Philipp's
> > driver) results in not much happening... no /dev/media* node for it,
> > no subdevs, no nothing.  No clues as to what's missing either.  Only
> > messages from imx-media are from registering the various subdevs.
> >
> > [   37.444877] imx-media: Registered subdev imx6-mipi-csi2
> > [   37.444973] imx-media: Registered subdev imx219 0-0010
> > [   38.868740] imx-media: Registered subdev ipu1_ic_prpenc
> > [   38.869265] imx-media: Registered subdev ipu1_ic_prpvf
> > [   38.869425] imx-media: Registered subdev ipu1_ic_pp0
> > [   38.870086] imx-media: Registered subdev ipu1_ic_pp1
> > [   38.871510] imx-media: Registered subdev ipu2_ic_prpenc
> > [   38.871743] imx-media: Registered subdev ipu1_smfc0
> > [   38.873043] imx-media: Registered subdev ipu1_smfc1
> > [   38.873225] imx-media: Registered subdev ipu2_ic_prpvf
> > [   38.875027] imx-media: Registered subdev ipu2_smfc0
> > [   38.875320] imx-media: Registered subdev ipu2_ic_pp0
> > [   38.877148] imx-media: Registered subdev ipu2_smfc1
> > [   38.877436] imx-media: Registered subdev ipu2_ic_pp1
> > [   38.932089] imx-media: Registered subdev camif0
> > [   38.956538] imx-media: Registered subdev camif1
> > [   38.959148] imx-media: Registered subdev camif2
> > [   38.964353] imx-media: Registered subdev camif3
> > [  206.502077] imx-media: Registered subdev ipu1_csi0
> > [  206.503304] imx-media: Registered subdev ipu1_csi1
> > [  206.503814] imx-media: Registered subdev ipu2_csi0
> > [  206.504281] imx-media: Registered subdev ipu2_csi1
> >
> > I also get:
> >
> > [   37.200072] imx6-mipi-csi2: data lanes: 2
> > [   37.200077] imx6-mipi-csi2: flags: 0x00000200
> >
> > and from what I can see, all modules from drivers/staging/media/imx/ are
> > loaded (had to load imx-csi by hand because of the brokenness in the
> > drivers/gpu/ipu code attaching an device_node pointer after registering
> > the platform device, which changes what userspace sees in the modalias
> > file.)
> >
> > Any clues at what to look at?
> 
> Hi Russell,
> 
> I'm not familiar with IMX219, can you send me the source for the
> imx219 subdev? I don't see it in 4.10-rc1.
> 
> I'm also having trouble finding a datasheet for it, but from what
> I've read, it has a MIPI CSI-2 interface. It should work fine as long
> as it presents a single source pad, registers asynchronously, and
> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

What about MEDIA_ENT_F_VID_IF_BRIDGE?

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-30 13:06                 ` Russell King - ARM Linux
  (?)
@ 2017-01-31 10:09                   ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 10:09 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam,
	Laurent Pinchart

On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > The central issue seems to be that I think media pad links / media bus
> > formats should describe physical links, such as parallel or serial
> > buses, and the formats of pixels flowing through them, whereas Steve
> > would like to extend them to describe software transports and in-memory
> > formats.
> 
> This probably isn't the right place to attach this comment in this
> thread, but... the issue of media bus formats matching physical buses
> is an argument that I think is already lost.

It is unfortunate that the parallel format definitions have been reused
for the MIPI logical formats, but I suppose we have to live with that.
Still, I think this is not a reason to open the floodgates and start
describing in-memory formats using MEDIA_BUS_FMT_*

Does at least the combination of logical format and number of lanes
uniquiely describe the physical format?
For the 4-lane LVDS bus formats there are JEIDA/VESA variants where just
the bit ordering is different (MEDIA_BUS_FMT_RGB888_1X7X4_SPWG,
MEDIA_BUS_FMT_RGB888_1X7X4_JEIDA).

> For example, take the 10-bit bayer formats:
> 
> #define MEDIA_BUS_FMT_SBGGR10_1X10              0x3007
> #define MEDIA_BUS_FMT_SGBRG10_1X10              0x300e
> #define MEDIA_BUS_FMT_SGRBG10_1X10              0x300a
> #define MEDIA_BUS_FMT_SRGGB10_1X10              0x300f
> 
> These are commonly used on CSI serial buses (see the smiapp driver for
> example).  From the description at the top of the file, it says the
> 1X10 means that one pixel is transferred as one 10-bit sample.
>
> However, the format on wire is somewhat different - four pixels are
> transmitted over five bytes:
> 
> 	P0	P1	P2	P3	P0	P1	P2	P3
> 	8-bit	8-bit	8-bit	8-bit	2-bit	2-bit	2-bit	2-bit
> 
> This gives two problems:
> 1) it doesn't fit in any sensible kind of "one pixel transferred as
>    N M-bit samples" description because the pixel/sample values
>    (depending how you look at them) are broken up.
> 
> 2) changing this will probably be a user visible change, as things
>    like smiapp are already in use.
> 
> So, I think what we actually have is the media bus formats describing
> the _logical_ bus format.  Yes, one pixel is transferred as one 10-bit
> sample in this case.

Yes. I suppose one way to look at it is that the MIPI CSI-2 specified
formats are representations of corresponding parallel bus formats.

> To help illustrate my point, consider the difference between
> MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> 
> So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> interesting:
> 
> 	first byte	2nd	3rd
> lane 1	P0 9:2		S0	P7 9:2
> lane 2	P1 9:2		P4 9:2	S1
> lane 3	P2 9:2		P5 9:2	P8 9:2
> lane 4	P3 9:2		P6 9:2	P9 9:2
> 
> S0 = P0/P1/P2/P3 least significant two bits
> S1 = P4/P5/P6/P7 least significant two bits
> 
> or 2 lane CSI:
> 	first byte	2nd	3rd	4th	5th
> lane 1	P0 9:2		P2	S0	P5	P7
> lane 2	P1 9:2		P3	P4	P6	S1
> 
> or 1 lane CSI:
> lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...

These do look like three different media bus formats to me.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 10:09                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 10:09 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > The central issue seems to be that I think media pad links / media bus
> > formats should describe physical links, such as parallel or serial
> > buses, and the formats of pixels flowing through them, whereas Steve
> > would like to extend them to describe software transports and in-memory
> > formats.
> 
> This probably isn't the right place to attach this comment in this
> thread, but... the issue of media bus formats matching physical buses
> is an argument that I think is already lost.

It is unfortunate that the parallel format definitions have been reused
for the MIPI logical formats, but I suppose we have to live with that.
Still, I think this is not a reason to open the floodgates and start
describing in-memory formats using MEDIA_BUS_FMT_*

Does at least the combination of logical format and number of lanes
uniquiely describe the physical format?
For the 4-lane LVDS bus formats there are JEIDA/VESA variants where just
the bit ordering is different (MEDIA_BUS_FMT_RGB888_1X7X4_SPWG,
MEDIA_BUS_FMT_RGB888_1X7X4_JEIDA).

> For example, take the 10-bit bayer formats:
> 
> #define MEDIA_BUS_FMT_SBGGR10_1X10              0x3007
> #define MEDIA_BUS_FMT_SGBRG10_1X10              0x300e
> #define MEDIA_BUS_FMT_SGRBG10_1X10              0x300a
> #define MEDIA_BUS_FMT_SRGGB10_1X10              0x300f
> 
> These are commonly used on CSI serial buses (see the smiapp driver for
> example).  From the description at the top of the file, it says the
> 1X10 means that one pixel is transferred as one 10-bit sample.
>
> However, the format on wire is somewhat different - four pixels are
> transmitted over five bytes:
> 
> 	P0	P1	P2	P3	P0	P1	P2	P3
> 	8-bit	8-bit	8-bit	8-bit	2-bit	2-bit	2-bit	2-bit
> 
> This gives two problems:
> 1) it doesn't fit in any sensible kind of "one pixel transferred as
>    N M-bit samples" description because the pixel/sample values
>    (depending how you look at them) are broken up.
> 
> 2) changing this will probably be a user visible change, as things
>    like smiapp are already in use.
> 
> So, I think what we actually have is the media bus formats describing
> the _logical_ bus format.  Yes, one pixel is transferred as one 10-bit
> sample in this case.

Yes. I suppose one way to look at it is that the MIPI CSI-2 specified
formats are representations of corresponding parallel bus formats.

> To help illustrate my point, consider the difference between
> MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> 
> So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> interesting:
> 
> 	first byte	2nd	3rd
> lane 1	P0 9:2		S0	P7 9:2
> lane 2	P1 9:2		P4 9:2	S1
> lane 3	P2 9:2		P5 9:2	P8 9:2
> lane 4	P3 9:2		P6 9:2	P9 9:2
> 
> S0 = P0/P1/P2/P3 least significant two bits
> S1 = P4/P5/P6/P7 least significant two bits
> 
> or 2 lane CSI:
> 	first byte	2nd	3rd	4th	5th
> lane 1	P0 9:2		P2	S0	P5	P7
> lane 2	P1 9:2		P3	P4	P6	S1
> 
> or 1 lane CSI:
> lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...

These do look like three different media bus formats to me.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 10:09                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 10:09 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > The central issue seems to be that I think media pad links / media bus
> > formats should describe physical links, such as parallel or serial
> > buses, and the formats of pixels flowing through them, whereas Steve
> > would like to extend them to describe software transports and in-memory
> > formats.
> 
> This probably isn't the right place to attach this comment in this
> thread, but... the issue of media bus formats matching physical buses
> is an argument that I think is already lost.

It is unfortunate that the parallel format definitions have been reused
for the MIPI logical formats, but I suppose we have to live with that.
Still, I think this is not a reason to open the floodgates and start
describing in-memory formats using MEDIA_BUS_FMT_*

Does at least the combination of logical format and number of lanes
uniquiely describe the physical format?
For the 4-lane LVDS bus formats there are JEIDA/VESA variants where just
the bit ordering is different (MEDIA_BUS_FMT_RGB888_1X7X4_SPWG,
MEDIA_BUS_FMT_RGB888_1X7X4_JEIDA).

> For example, take the 10-bit bayer formats:
> 
> #define MEDIA_BUS_FMT_SBGGR10_1X10              0x3007
> #define MEDIA_BUS_FMT_SGBRG10_1X10              0x300e
> #define MEDIA_BUS_FMT_SGRBG10_1X10              0x300a
> #define MEDIA_BUS_FMT_SRGGB10_1X10              0x300f
> 
> These are commonly used on CSI serial buses (see the smiapp driver for
> example).  From the description at the top of the file, it says the
> 1X10 means that one pixel is transferred as one 10-bit sample.
>
> However, the format on wire is somewhat different - four pixels are
> transmitted over five bytes:
> 
> 	P0	P1	P2	P3	P0	P1	P2	P3
> 	8-bit	8-bit	8-bit	8-bit	2-bit	2-bit	2-bit	2-bit
> 
> This gives two problems:
> 1) it doesn't fit in any sensible kind of "one pixel transferred as
>    N M-bit samples" description because the pixel/sample values
>    (depending how you look at them) are broken up.
> 
> 2) changing this will probably be a user visible change, as things
>    like smiapp are already in use.
> 
> So, I think what we actually have is the media bus formats describing
> the _logical_ bus format.  Yes, one pixel is transferred as one 10-bit
> sample in this case.

Yes. I suppose one way to look at it is that the MIPI CSI-2 specified
formats are representations of corresponding parallel bus formats.

> To help illustrate my point, consider the difference between
> MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> 
> So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> interesting:
> 
> 	first byte	2nd	3rd
> lane 1	P0 9:2		S0	P7 9:2
> lane 2	P1 9:2		P4 9:2	S1
> lane 3	P2 9:2		P5 9:2	P8 9:2
> lane 4	P3 9:2		P6 9:2	P9 9:2
> 
> S0 = P0/P1/P2/P3 least significant two bits
> S1 = P4/P5/P6/P7 least significant two bits
> 
> or 2 lane CSI:
> 	first byte	2nd	3rd	4th	5th
> lane 1	P0 9:2		P2	S0	P5	P7
> lane 2	P1 9:2		P3	P4	P6	S1
> 
> or 1 lane CSI:
> lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...

These do look like three different media bus formats to me.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31  1:22     ` Steve Longerbeam
  (?)
@ 2017-01-31 10:21       ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 10:21 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
> Edit: I see a subdev that is missing: the video mux. Did you enable
> CONFIG_VIDEO_MULTIPLEXER?

Yes, and that's where the problem is - the video-multiplexer is 
missing the module aliases to allow it to be automatically loaded.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 10:21       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 10:21 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
> Edit: I see a subdev that is missing: the video mux. Did you enable
> CONFIG_VIDEO_MULTIPLEXER?

Yes, and that's where the problem is - the video-multiplexer is 
missing the module aliases to allow it to be automatically loaded.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 10:21       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 10:21 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
> Edit: I see a subdev that is missing: the video mux. Did you enable
> CONFIG_VIDEO_MULTIPLEXER?

Yes, and that's where the problem is - the video-multiplexer is 
missing the module aliases to allow it to be automatically loaded.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-01-31 10:30     ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 10:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---

The lack of s_frame_interval/g_frame_interval in this driver means:

media-ctl -v -d /dev/media1 --set-v4l2 '"imx6-mipi-csi2":1[fmt:SGBRG8/512x512@1/30]'

Opening media device /dev/media1
Enumerating entities
Found 29 entities
Enumerating pads and links
Setting up format SGBRG8 512x512 on pad imx6-mipi-csi2/1
Format set: SGBRG8 512x512
Setting up frame interval 1/30 on entity imx6-mipi-csi2
Unable to set frame interval: Inappropriate ioctl for device (-25)Unable to setup formats: Inappropriate ioctl for device (25)

which causes the setup of the next element in the chain to also fail
(because the above media-ctl command doesn't set the sink on the
ipu1 csi0 mux.)

It seems to me that without the frame interval supported throughout the
chain, there's no way for an application to properly negotiate the
capture parameters via the "try" mechanism, since it has no idea whether
the frame rate it wants is supported throughout the subdev chain.  Eg,
the sensor may be able to do 100fps but there could be something in the
pipeline that restricts it due to bandwidth.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 10:30     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 10:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---

The lack of s_frame_interval/g_frame_interval in this driver means:

media-ctl -v -d /dev/media1 --set-v4l2 '"imx6-mipi-csi2":1[fmt:SGBRG8/512x512@1/30]'

Opening media device /dev/media1
Enumerating entities
Found 29 entities
Enumerating pads and links
Setting up format SGBRG8 512x512 on pad imx6-mipi-csi2/1
Format set: SGBRG8 512x512
Setting up frame interval 1/30 on entity imx6-mipi-csi2
Unable to set frame interval: Inappropriate ioctl for device (-25)Unable to setup formats: Inappropriate ioctl for device (25)

which causes the setup of the next element in the chain to also fail
(because the above media-ctl command doesn't set the sink on the
ipu1 csi0 mux.)

It seems to me that without the frame interval supported throughout the
chain, there's no way for an application to properly negotiate the
capture parameters via the "try" mechanism, since it has no idea whether
the frame rate it wants is supported throughout the subdev chain.  Eg,
the sensor may be able to do 100fps but there could be something in the
pipeline that restricts it due to bandwidth.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 10:30     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 10:30 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---

The lack of s_frame_interval/g_frame_interval in this driver means:

media-ctl -v -d /dev/media1 --set-v4l2 '"imx6-mipi-csi2":1[fmt:SGBRG8/512x512 at 1/30]'

Opening media device /dev/media1
Enumerating entities
Found 29 entities
Enumerating pads and links
Setting up format SGBRG8 512x512 on pad imx6-mipi-csi2/1
Format set: SGBRG8 512x512
Setting up frame interval 1/30 on entity imx6-mipi-csi2
Unable to set frame interval: Inappropriate ioctl for device (-25)Unable to setup formats: Inappropriate ioctl for device (25)

which causes the setup of the next element in the chain to also fail
(because the above media-ctl command doesn't set the sink on the
ipu1 csi0 mux.)

It seems to me that without the frame interval supported throughout the
chain, there's no way for an application to properly negotiate the
capture parameters via the "try" mechanism, since it has no idea whether
the frame rate it wants is supported throughout the subdev chain.  Eg,
the sensor may be able to do 100fps but there could be something in the
pipeline that restricts it due to bandwidth.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31  1:22     ` Steve Longerbeam
  (?)
@ 2017-01-31 11:00       ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 11:00 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
> I'm also having trouble finding a datasheet for it, but from what
> I've read, it has a MIPI CSI-2 interface. It should work fine as long
> as it presents a single source pad, registers asynchronously, and
> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

Yes, it is MIPI CSI-2, and yes it has a single source pad, registers
asynchronously, but that's about as far as it goes.

The structure is a camera sensor followed by some processing.  So just
like the smiapp code, I've ended up with multiple subdevs describing
each stage of the sensors pipeline.

Just like smiapp, the camera sensor block (which is the very far end
of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
front of that is the binner, which just like smiapp gets a separate
entity.  It's this entity which is connected to the mipi-csi2 subdev.

Unlike smiapp, which does not set an entity function, I set my binner
entity as MEDIA_ENT_F_PROC_VIDEO_SCALER on the basis that that is
what V4L2 documentation recommend:

    -  ..  row 27

       ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:

       -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``

       -  Video scaler. An entity capable of video scaling must have
          at least one sink pad and one source pad, and scale the
          video frame(s) received on its sink pad(s) to a different
          resolution output on its source pad(s). The range of
          supported scaling ratios is entity-specific and can differ
          between the horizontal and vertical directions (in particular
          scaling can be supported in one direction only). Binning and
          skipping are considered as scaling.

This causes attempts to configure the ipu1_csi0 interface to fail:

media-ctl -v -d /dev/media1 --set-v4l2 '"ipu1_csi0":1[fmt:SGBRG8/512x512@1/30]'
Opening media device /dev/media1
Enumerating entities
Found 29 entities
Enumerating pads and links
Setting up format SGBRG8 512x512 on pad ipu1_csi0/1
Unable to set format: No such device (-19)
Unable to setup formats: No such device (19)

and in the kernel log:

ipu1_csi0: no sensor attached

And yes, I already know that my next problem is going to be that the bayer
formats are not supported in your driver (just like Philipp's driver) but
adding them should not be difficult... but only once this issue is resolved.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 11:00       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 11:00 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
> I'm also having trouble finding a datasheet for it, but from what
> I've read, it has a MIPI CSI-2 interface. It should work fine as long
> as it presents a single source pad, registers asynchronously, and
> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

Yes, it is MIPI CSI-2, and yes it has a single source pad, registers
asynchronously, but that's about as far as it goes.

The structure is a camera sensor followed by some processing.  So just
like the smiapp code, I've ended up with multiple subdevs describing
each stage of the sensors pipeline.

Just like smiapp, the camera sensor block (which is the very far end
of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
front of that is the binner, which just like smiapp gets a separate
entity.  It's this entity which is connected to the mipi-csi2 subdev.

Unlike smiapp, which does not set an entity function, I set my binner
entity as MEDIA_ENT_F_PROC_VIDEO_SCALER on the basis that that is
what V4L2 documentation recommend:

    -  ..  row 27

       ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:

       -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``

       -  Video scaler. An entity capable of video scaling must have
          at least one sink pad and one source pad, and scale the
          video frame(s) received on its sink pad(s) to a different
          resolution output on its source pad(s). The range of
          supported scaling ratios is entity-specific and can differ
          between the horizontal and vertical directions (in particular
          scaling can be supported in one direction only). Binning and
          skipping are considered as scaling.

This causes attempts to configure the ipu1_csi0 interface to fail:

media-ctl -v -d /dev/media1 --set-v4l2 '"ipu1_csi0":1[fmt:SGBRG8/512x512@1/30]'
Opening media device /dev/media1
Enumerating entities
Found 29 entities
Enumerating pads and links
Setting up format SGBRG8 512x512 on pad ipu1_csi0/1
Unable to set format: No such device (-19)
Unable to setup formats: No such device (19)

and in the kernel log:

ipu1_csi0: no sensor attached

And yes, I already know that my next problem is going to be that the bayer
formats are not supported in your driver (just like Philipp's driver) but
adding them should not be difficult... but only once this issue is resolved.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 11:00       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 11:00 UTC (permalink / raw)
  To: linux-arm-kernel

On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
> I'm also having trouble finding a datasheet for it, but from what
> I've read, it has a MIPI CSI-2 interface. It should work fine as long
> as it presents a single source pad, registers asynchronously, and
> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.

Yes, it is MIPI CSI-2, and yes it has a single source pad, registers
asynchronously, but that's about as far as it goes.

The structure is a camera sensor followed by some processing.  So just
like the smiapp code, I've ended up with multiple subdevs describing
each stage of the sensors pipeline.

Just like smiapp, the camera sensor block (which is the very far end
of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
front of that is the binner, which just like smiapp gets a separate
entity.  It's this entity which is connected to the mipi-csi2 subdev.

Unlike smiapp, which does not set an entity function, I set my binner
entity as MEDIA_ENT_F_PROC_VIDEO_SCALER on the basis that that is
what V4L2 documentation recommend:

    -  ..  row 27

       ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:

       -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``

       -  Video scaler. An entity capable of video scaling must have
          at least one sink pad and one source pad, and scale the
          video frame(s) received on its sink pad(s) to a different
          resolution output on its source pad(s). The range of
          supported scaling ratios is entity-specific and can differ
          between the horizontal and vertical directions (in particular
          scaling can be supported in one direction only). Binning and
          skipping are considered as scaling.

This causes attempts to configure the ipu1_csi0 interface to fail:

media-ctl -v -d /dev/media1 --set-v4l2 '"ipu1_csi0":1[fmt:SGBRG8/512x512 at 1/30]'
Opening media device /dev/media1
Enumerating entities
Found 29 entities
Enumerating pads and links
Setting up format SGBRG8 512x512 on pad ipu1_csi0/1
Unable to set format: No such device (-19)
Unable to setup formats: No such device (19)

and in the kernel log:

ipu1_csi0: no sensor attached

And yes, I already know that my next problem is going to be that the bayer
formats are not supported in your driver (just like Philipp's driver) but
adding them should not be difficult... but only once this issue is resolved.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 11:20     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 11:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> +static int csi_link_validate(struct v4l2_subdev *sd,
> +			     struct media_link *link,
> +			     struct v4l2_subdev_format *source_fmt,
> +			     struct v4l2_subdev_format *sink_fmt)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	bool is_csi2;
> +	int ret;
> +
> +	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
> +	if (ret)
> +		return ret;
> +
> +	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(priv->sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		ret = PTR_ERR(priv->sensor);
> +		priv->sensor = NULL;
> +		return ret;
> +	}
> +
> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> +			       &priv->sensor_mbus_cfg);
> +	if (ret)
> +		return ret;
> +
> +	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
> +
> +	if (is_csi2) {
> +		int vc_num = 0;
> +		/*
> +		 * NOTE! It seems the virtual channels from the mipi csi-2
> +		 * receiver are used only for routing by the video mux's,
> +		 * or for hard-wired routing to the CSI's. Once the stream
> +		 * enters the CSI's however, they are treated internally
> +		 * in the IPU as virtual channel 0.
> +		 */
> +#if 0
> +		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
> +							  &priv->sd.entity);
> +		if (vc_num < 0)
> +			return vc_num;
> +#endif
> +		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
> +					  &priv->format_mbus[priv->input_pad]);

This seems (at least to me) to go against the spirit of the subdev
negotiation.  Here, you seem to bypass the negotiation performed
between the CSI and the attached device, wanting to grab the
format from the CSI2 sensor directly.  That bypasses negotiation
performed at the CSI2 subdev and video muxes.

The same goes for the frame rate monitoring code - imho, the frame
rate should be something that results from negotiation with the
neighbouring element, not by poking at some entity that is several
entities away.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 11:20     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 11:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> +static int csi_link_validate(struct v4l2_subdev *sd,
> +			     struct media_link *link,
> +			     struct v4l2_subdev_format *source_fmt,
> +			     struct v4l2_subdev_format *sink_fmt)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	bool is_csi2;
> +	int ret;
> +
> +	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
> +	if (ret)
> +		return ret;
> +
> +	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(priv->sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		ret = PTR_ERR(priv->sensor);
> +		priv->sensor = NULL;
> +		return ret;
> +	}
> +
> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> +			       &priv->sensor_mbus_cfg);
> +	if (ret)
> +		return ret;
> +
> +	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
> +
> +	if (is_csi2) {
> +		int vc_num = 0;
> +		/*
> +		 * NOTE! It seems the virtual channels from the mipi csi-2
> +		 * receiver are used only for routing by the video mux's,
> +		 * or for hard-wired routing to the CSI's. Once the stream
> +		 * enters the CSI's however, they are treated internally
> +		 * in the IPU as virtual channel 0.
> +		 */
> +#if 0
> +		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
> +							  &priv->sd.entity);
> +		if (vc_num < 0)
> +			return vc_num;
> +#endif
> +		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
> +					  &priv->format_mbus[priv->input_pad]);

This seems (at least to me) to go against the spirit of the subdev
negotiation.  Here, you seem to bypass the negotiation performed
between the CSI and the attached device, wanting to grab the
format from the CSI2 sensor directly.  That bypasses negotiation
performed at the CSI2 subdev and video muxes.

The same goes for the frame rate monitoring code - imho, the frame
rate should be something that results from negotiation with the
neighbouring element, not by poking at some entity that is several
entities away.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 11:20     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 11:20 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> +static int csi_link_validate(struct v4l2_subdev *sd,
> +			     struct media_link *link,
> +			     struct v4l2_subdev_format *source_fmt,
> +			     struct v4l2_subdev_format *sink_fmt)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	bool is_csi2;
> +	int ret;
> +
> +	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
> +	if (ret)
> +		return ret;
> +
> +	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
> +	if (IS_ERR(priv->sensor)) {
> +		v4l2_err(&priv->sd, "no sensor attached\n");
> +		ret = PTR_ERR(priv->sensor);
> +		priv->sensor = NULL;
> +		return ret;
> +	}
> +
> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> +			       &priv->sensor_mbus_cfg);
> +	if (ret)
> +		return ret;
> +
> +	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
> +
> +	if (is_csi2) {
> +		int vc_num = 0;
> +		/*
> +		 * NOTE! It seems the virtual channels from the mipi csi-2
> +		 * receiver are used only for routing by the video mux's,
> +		 * or for hard-wired routing to the CSI's. Once the stream
> +		 * enters the CSI's however, they are treated internally
> +		 * in the IPU as virtual channel 0.
> +		 */
> +#if 0
> +		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
> +							  &priv->sd.entity);
> +		if (vc_num < 0)
> +			return vc_num;
> +#endif
> +		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
> +					  &priv->format_mbus[priv->input_pad]);

This seems (at least to me) to go against the spirit of the subdev
negotiation.  Here, you seem to bypass the negotiation performed
between the CSI and the attached device, wanting to grab the
format from the CSI2 sensor directly.  That bypasses negotiation
performed at the CSI2 subdev and video muxes.

The same goes for the frame rate monitoring code - imho, the frame
rate should be something that results from negotiation with the
neighbouring element, not by poking at some entity that is several
entities away.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31 10:09                   ` Philipp Zabel
  (?)
@ 2017-01-31 13:14                     ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 13:14 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam,
	Laurent Pinchart

On Tue, Jan 31, 2017 at 11:09:24AM +0100, Philipp Zabel wrote:
> On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > To help illustrate my point, consider the difference between
> > MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> > MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> > 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> > separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> > 
> > So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> > bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> > interesting:
> > 
> > 	first byte	2nd	3rd
> > lane 1	P0 9:2		S0	P7 9:2
> > lane 2	P1 9:2		P4 9:2	S1
> > lane 3	P2 9:2		P5 9:2	P8 9:2
> > lane 4	P3 9:2		P6 9:2	P9 9:2
> > 
> > S0 = P0/P1/P2/P3 least significant two bits
> > S1 = P4/P5/P6/P7 least significant two bits
> > 
> > or 2 lane CSI:
> > 	first byte	2nd	3rd	4th	5th
> > lane 1	P0 9:2		P2	S0	P5	P7
> > lane 2	P1 9:2		P3	P4	P6	S1
> > 
> > or 1 lane CSI:
> > lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...
> 
> These do look like three different media bus formats to me.

This isn't limited to the serial side - the parallel bus side between
the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
the CS0/1 interfaces is much the same with 10-bit bayer.

Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
up on the least significant 8 bits of the 32-bit bus, lane 3 on the
top 8-bits.

Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
kind of the 2-lane case above...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 13:14                     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 13:14 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Hans Verkuil, Steve Longerbeam, Laurent Pinchart, robert.jarzmik,
	devel, markus.heiser, laurent.pinchart+renesas, geert,
	Steve Longerbeam, linux-media, devicetree, arnd, mchehab,
	bparrot, robh+dt, horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 11:09:24AM +0100, Philipp Zabel wrote:
> On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > To help illustrate my point, consider the difference between
> > MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> > MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> > 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> > separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> > 
> > So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> > bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> > interesting:
> > 
> > 	first byte	2nd	3rd
> > lane 1	P0 9:2		S0	P7 9:2
> > lane 2	P1 9:2		P4 9:2	S1
> > lane 3	P2 9:2		P5 9:2	P8 9:2
> > lane 4	P3 9:2		P6 9:2	P9 9:2
> > 
> > S0 = P0/P1/P2/P3 least significant two bits
> > S1 = P4/P5/P6/P7 least significant two bits
> > 
> > or 2 lane CSI:
> > 	first byte	2nd	3rd	4th	5th
> > lane 1	P0 9:2		P2	S0	P5	P7
> > lane 2	P1 9:2		P3	P4	P6	S1
> > 
> > or 1 lane CSI:
> > lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...
> 
> These do look like three different media bus formats to me.

This isn't limited to the serial side - the parallel bus side between
the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
the CS0/1 interfaces is much the same with 10-bit bayer.

Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
up on the least significant 8 bits of the 32-bit bus, lane 3 on the
top 8-bits.

Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
kind of the 2-lane case above...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 13:14                     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 13:14 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 11:09:24AM +0100, Philipp Zabel wrote:
> On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > To help illustrate my point, consider the difference between
> > MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> > MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> > 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> > separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> > 
> > So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> > bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> > interesting:
> > 
> > 	first byte	2nd	3rd
> > lane 1	P0 9:2		S0	P7 9:2
> > lane 2	P1 9:2		P4 9:2	S1
> > lane 3	P2 9:2		P5 9:2	P8 9:2
> > lane 4	P3 9:2		P6 9:2	P9 9:2
> > 
> > S0 = P0/P1/P2/P3 least significant two bits
> > S1 = P4/P5/P6/P7 least significant two bits
> > 
> > or 2 lane CSI:
> > 	first byte	2nd	3rd	4th	5th
> > lane 1	P0 9:2		P2	S0	P5	P7
> > lane 2	P1 9:2		P3	P4	P6	S1
> > 
> > or 1 lane CSI:
> > lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...
> 
> These do look like three different media bus formats to me.

This isn't limited to the serial side - the parallel bus side between
the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
the CS0/1 interfaces is much the same with 10-bit bayer.

Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
up on the least significant 8 bits of the 32-bit bus, lane 3 on the
top 8-bits.

Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
kind of the 2-lane case above...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 13:35                       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 13:35 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam,
	Laurent Pinchart

On Tue, 2017-01-31 at 13:14 +0000, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 11:09:24AM +0100, Philipp Zabel wrote:
> > On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > > To help illustrate my point, consider the difference between
> > > MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> > > MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> > > 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> > > separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> > > 
> > > So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> > > bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> > > interesting:
> > > 
> > > 	first byte	2nd	3rd
> > > lane 1	P0 9:2		S0	P7 9:2
> > > lane 2	P1 9:2		P4 9:2	S1
> > > lane 3	P2 9:2		P5 9:2	P8 9:2
> > > lane 4	P3 9:2		P6 9:2	P9 9:2
> > > 
> > > S0 = P0/P1/P2/P3 least significant two bits
> > > S1 = P4/P5/P6/P7 least significant two bits
> > > 
> > > or 2 lane CSI:
> > > 	first byte	2nd	3rd	4th	5th
> > > lane 1	P0 9:2		P2	S0	P5	P7
> > > lane 2	P1 9:2		P3	P4	P6	S1
> > > 
> > > or 1 lane CSI:
> > > lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...
> > 
> > These do look like three different media bus formats to me.
> 
> This isn't limited to the serial side - the parallel bus side between
> the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
> the CS0/1 interfaces is much the same with 10-bit bayer.
> 
> Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
> up on the least significant 8 bits of the 32-bit bus, lane 3 on the
> top 8-bits.
> 
> Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
> kind of the 2-lane case above...

You are right, on the parallel buses the format most definitely is not
MEDIA_BUS_FMT_SBGGR10_1X10. We don't have any representation of the
32-bit bus between CSI2 host and CSI2IPU gasket because we model the two
as a single entity, but the four 16-bit parallel buses between the
CSI2IPU gasket and the IPU1/2 CSI0/1 probably should be set to a custom
format describing this accurately.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 13:35                       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 13:35 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

On Tue, 2017-01-31 at 13:14 +0000, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 11:09:24AM +0100, Philipp Zabel wrote:
> > On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > > To help illustrate my point, consider the difference between
> > > MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> > > MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> > > 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> > > separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> > > 
> > > So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> > > bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> > > interesting:
> > > 
> > > 	first byte	2nd	3rd
> > > lane 1	P0 9:2		S0	P7 9:2
> > > lane 2	P1 9:2		P4 9:2	S1
> > > lane 3	P2 9:2		P5 9:2	P8 9:2
> > > lane 4	P3 9:2		P6 9:2	P9 9:2
> > > 
> > > S0 = P0/P1/P2/P3 least significant two bits
> > > S1 = P4/P5/P6/P7 least significant two bits
> > > 
> > > or 2 lane CSI:
> > > 	first byte	2nd	3rd	4th	5th
> > > lane 1	P0 9:2		P2	S0	P5	P7
> > > lane 2	P1 9:2		P3	P4	P6	S1
> > > 
> > > or 1 lane CSI:
> > > lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...
> > 
> > These do look like three different media bus formats to me.
> 
> This isn't limited to the serial side - the parallel bus side between
> the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
> the CS0/1 interfaces is much the same with 10-bit bayer.
> 
> Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
> up on the least significant 8 bits of the 32-bit bus, lane 3 on the
> top 8-bits.
> 
> Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
> kind of the 2-lane case above...

You are right, on the parallel buses the format most definitely is not
MEDIA_BUS_FMT_SBGGR10_1X10. We don't have any representation of the
32-bit bus between CSI2 host and CSI2IPU gasket because we model the two
as a single entity, but the four 16-bit parallel buses between the
CSI2IPU gasket and the IPU1/2 CSI0/1 probably should be set to a custom
format describing this accurately.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 13:35                       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 13:35 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, 2017-01-31 at 13:14 +0000, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 11:09:24AM +0100, Philipp Zabel wrote:
> > On Mon, 2017-01-30 at 13:06 +0000, Russell King - ARM Linux wrote:
> > > To help illustrate my point, consider the difference between
> > > MEDIA_BUS_FMT_RGB565_1X16 and MEDIA_BUS_FMT_RGB565_2X8_BE or
> > > MEDIA_BUS_FMT_RGB565_2X8_LE.  RGB565_1X16 means 1 pixel over an effective
> > > 16-bit wide bus (if it's not 16-bit, then it has to be broken up into
> > > separate "samples".)  RGB565_2X8 means 1 pixel as two 8-bit samples.
> > > 
> > > So, the 10-bit bayer is 1 pixel as 1.25 bytes.  Or is it, over a serial
> > > bus.  Using the RGB565 case, 10-bit bayer over a 4 lane CSI bus becomes
> > > interesting:
> > > 
> > > 	first byte	2nd	3rd
> > > lane 1	P0 9:2		S0	P7 9:2
> > > lane 2	P1 9:2		P4 9:2	S1
> > > lane 3	P2 9:2		P5 9:2	P8 9:2
> > > lane 4	P3 9:2		P6 9:2	P9 9:2
> > > 
> > > S0 = P0/P1/P2/P3 least significant two bits
> > > S1 = P4/P5/P6/P7 least significant two bits
> > > 
> > > or 2 lane CSI:
> > > 	first byte	2nd	3rd	4th	5th
> > > lane 1	P0 9:2		P2	S0	P5	P7
> > > lane 2	P1 9:2		P3	P4	P6	S1
> > > 
> > > or 1 lane CSI:
> > > lane 1	P0 P1 P2 P3 S0 P4 P5 P6 P7 S1 P8 P9 ...
> > 
> > These do look like three different media bus formats to me.
> 
> This isn't limited to the serial side - the parallel bus side between
> the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
> the CS0/1 interfaces is much the same with 10-bit bayer.
> 
> Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
> up on the least significant 8 bits of the 32-bit bus, lane 3 on the
> top 8-bits.
> 
> Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
> kind of the 2-lane case above...

You are right, on the parallel buses the format most definitely is not
MEDIA_BUS_FMT_SBGGR10_1X10. We don't have any representation of the
32-bit bus between CSI2 host and CSI2IPU gasket because we model the two
as a single entity, but the four 16-bit parallel buses between the
CSI2IPU gasket and the IPU1/2 CSI0/1 probably should be set to a custom
format describing this accurately.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 13:42       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 13:42 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> should complain about this.

... and more.

Driver Info:
        Driver name   : imx-media-camif
        Card type     : imx-media-camif
        Bus info      :
        Driver version: 4.10.0
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format

Compliance test for device /dev/video3 (not using libv4l2):

Required ioctls:
                fail: v4l2-compliance.cpp(244): string empty
                fail: v4l2-compliance.cpp(297): check_ustring(vcap.bus_info, sizeof(vcap.bus_info))
        test VIDIOC_QUERYCAP: FAIL

Allow for multiple opens:
        test second video open: OK
                fail: v4l2-compliance.cpp(244): string empty
                fail: v4l2-compliance.cpp(297): check_ustring(vcap.bus_info, sizeof(vcap.bus_info))
        test VIDIOC_QUERYCAP: FAIL
        test VIDIOC_G/S_PRIORITY: OK

Debug ioctls:
        test VIDIOC_DBG_G/S_REGISTER: OK
        test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
        test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
        test VIDIOC_ENUMAUDIO: OK (Not Supported)
                fail: v4l2-test-input-output.cpp(382): std == 0
                fail: v4l2-test-input-output.cpp(437): invalid attributes for input 0
        test VIDIOC_G/S/ENUMINPUT: FAIL
        test VIDIOC_G/S_AUDIO: OK (Not Supported)
        Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
        test VIDIOC_G/S_MODULATOR: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_ENUMAUDOUT: OK (Not Supported)
        test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
        test VIDIOC_G/S_AUDOUT: OK (Not Supported)
        Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
        test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
        test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
        test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
        test VIDIOC_G/S_EDID: OK (Not Supported)

        Control ioctls:
                test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
                test VIDIOC_QUERYCTRL: OK
                test VIDIOC_G/S_CTRL: OK
                test VIDIOC_G/S/TRY_EXT_CTRLS: OK
                fail: v4l2-test-controls.cpp(779): subscribe event for control 'Camera Controls' failed
                test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
                test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
                Standard Controls: 13 Private Controls: 0

        Format ioctls:
                test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
                test VIDIOC_G/S_PARM: OK (Not Supported)
                test VIDIOC_G_FBUF: OK (Not Supported)
                fail: v4l2-test-formats.cpp(414): unknown pixelformat 42474752 for buftype 1
                test VIDIOC_G_FMT: FAIL
                test VIDIOC_TRY_FMT: OK (Not Supported)
                test VIDIOC_S_FMT: OK (Not Supported)
                test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)

        Codec ioctls:
                test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
                test VIDIOC_G_ENC_INDEX: OK (Not Supported)
                test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

        Buffer ioctls:
                test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
                fail: v4l2-test-buffers.cpp(500): q.has_expbuf(node)
                test VIDIOC_EXPBUF: FAIL


Total: 39, Succeeded: 33, Failed: 6, Warnings: 0

Not all of these may be a result of Steve's code - this is running against
my gradually modified version to support bayer formats (which seems to be
the cause of the v4l2-test-formats.cpp failure... for some reason the
driver isn't enumerating all the formats.)

And that reason is the way that the formats are enumerated:

static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
                                  struct v4l2_fmtdesc *f)
{
        const struct imx_media_pixfmt *cc;
        u32 code;
        int ret;

        ret = imx_media_enum_format(&code, f->index, true, true);
        if (ret)
                return ret;
        cc = imx_media_find_format(0, code, true, true);
        if (!cc)
                return -EINVAL;

When imx_media_enum_format() hits this entry in the table:

        }, {
                .fourcc = V4L2_PIX_FMT_BGR24,
                .cs     = IPUV3_COLORSPACE_RGB,
                .bpp    = 24,
        }, {

becaues there's no .codes defined:

int imx_media_enum_format(u32 *code, u32 index, bool allow_rgb,
                          bool allow_planar)
{
...
        *code = fmt->codes[0];
        return 0;
}

So, we end up calling imx_media_find_format(0, 0, true, true), which
fails, returning NULL.  That causes camif_enum_fmt_vid_cap() to
return -EINVAL.

So everything past this entry is unable to be enumerated.

I think this is a really round-about way of enumerating the pixel
formats when there are soo many entries in the table which have no
media bus code - there's absolutely no way that any of those entries
can ever be enumerated in this fashion, so they might as well not be
in the table...

That's my present solution to this problem, to #if 0 out all the
entries without any .codes field.  I think the real answer is that
this needs a _separate_ function to enumerate the pixel formats for
camif_enum_fmt_vid_cap().  However, there may be other issues lurking
that I've not yet found (still trying to get this code to work...)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 13:42       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 13:42 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> should complain about this.

... and more.

Driver Info:
        Driver name   : imx-media-camif
        Card type     : imx-media-camif
        Bus info      :
        Driver version: 4.10.0
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format

Compliance test for device /dev/video3 (not using libv4l2):

Required ioctls:
                fail: v4l2-compliance.cpp(244): string empty
                fail: v4l2-compliance.cpp(297): check_ustring(vcap.bus_info, sizeof(vcap.bus_info))
        test VIDIOC_QUERYCAP: FAIL

Allow for multiple opens:
        test second video open: OK
                fail: v4l2-compliance.cpp(244): string empty
                fail: v4l2-compliance.cpp(297): check_ustring(vcap.bus_info, sizeof(vcap.bus_info))
        test VIDIOC_QUERYCAP: FAIL
        test VIDIOC_G/S_PRIORITY: OK

Debug ioctls:
        test VIDIOC_DBG_G/S_REGISTER: OK
        test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
        test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
        test VIDIOC_ENUMAUDIO: OK (Not Supported)
                fail: v4l2-test-input-output.cpp(382): std == 0
                fail: v4l2-test-input-output.cpp(437): invalid attributes for input 0
        test VIDIOC_G/S/ENUMINPUT: FAIL
        test VIDIOC_G/S_AUDIO: OK (Not Supported)
        Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
        test VIDIOC_G/S_MODULATOR: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_ENUMAUDOUT: OK (Not Supported)
        test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
        test VIDIOC_G/S_AUDOUT: OK (Not Supported)
        Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
        test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
        test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
        test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
        test VIDIOC_G/S_EDID: OK (Not Supported)

        Control ioctls:
                test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
                test VIDIOC_QUERYCTRL: OK
                test VIDIOC_G/S_CTRL: OK
                test VIDIOC_G/S/TRY_EXT_CTRLS: OK
                fail: v4l2-test-controls.cpp(779): subscribe event for control 'Camera Controls' failed
                test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
                test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
                Standard Controls: 13 Private Controls: 0

        Format ioctls:
                test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
                test VIDIOC_G/S_PARM: OK (Not Supported)
                test VIDIOC_G_FBUF: OK (Not Supported)
                fail: v4l2-test-formats.cpp(414): unknown pixelformat 42474752 for buftype 1
                test VIDIOC_G_FMT: FAIL
                test VIDIOC_TRY_FMT: OK (Not Supported)
                test VIDIOC_S_FMT: OK (Not Supported)
                test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)

        Codec ioctls:
                test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
                test VIDIOC_G_ENC_INDEX: OK (Not Supported)
                test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

        Buffer ioctls:
                test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
                fail: v4l2-test-buffers.cpp(500): q.has_expbuf(node)
                test VIDIOC_EXPBUF: FAIL


Total: 39, Succeeded: 33, Failed: 6, Warnings: 0

Not all of these may be a result of Steve's code - this is running against
my gradually modified version to support bayer formats (which seems to be
the cause of the v4l2-test-formats.cpp failure... for some reason the
driver isn't enumerating all the formats.)

And that reason is the way that the formats are enumerated:

static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
                                  struct v4l2_fmtdesc *f)
{
        const struct imx_media_pixfmt *cc;
        u32 code;
        int ret;

        ret = imx_media_enum_format(&code, f->index, true, true);
        if (ret)
                return ret;
        cc = imx_media_find_format(0, code, true, true);
        if (!cc)
                return -EINVAL;

When imx_media_enum_format() hits this entry in the table:

        }, {
                .fourcc = V4L2_PIX_FMT_BGR24,
                .cs     = IPUV3_COLORSPACE_RGB,
                .bpp    = 24,
        }, {

becaues there's no .codes defined:

int imx_media_enum_format(u32 *code, u32 index, bool allow_rgb,
                          bool allow_planar)
{
...
        *code = fmt->codes[0];
        return 0;
}

So, we end up calling imx_media_find_format(0, 0, true, true), which
fails, returning NULL.  That causes camif_enum_fmt_vid_cap() to
return -EINVAL.

So everything past this entry is unable to be enumerated.

I think this is a really round-about way of enumerating the pixel
formats when there are soo many entries in the table which have no
media bus code - there's absolutely no way that any of those entries
can ever be enumerated in this fashion, so they might as well not be
in the table...

That's my present solution to this problem, to #if 0 out all the
entries without any .codes field.  I think the real answer is that
this needs a _separate_ function to enumerate the pixel formats for
camif_enum_fmt_vid_cap().  However, there may be other issues lurking
that I've not yet found (still trying to get this code to work...)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 13:42       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 13:42 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> should complain about this.

... and more.

Driver Info:
        Driver name   : imx-media-camif
        Card type     : imx-media-camif
        Bus info      :
        Driver version: 4.10.0
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format

Compliance test for device /dev/video3 (not using libv4l2):

Required ioctls:
                fail: v4l2-compliance.cpp(244): string empty
                fail: v4l2-compliance.cpp(297): check_ustring(vcap.bus_info, sizeof(vcap.bus_info))
        test VIDIOC_QUERYCAP: FAIL

Allow for multiple opens:
        test second video open: OK
                fail: v4l2-compliance.cpp(244): string empty
                fail: v4l2-compliance.cpp(297): check_ustring(vcap.bus_info, sizeof(vcap.bus_info))
        test VIDIOC_QUERYCAP: FAIL
        test VIDIOC_G/S_PRIORITY: OK

Debug ioctls:
        test VIDIOC_DBG_G/S_REGISTER: OK
        test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
        test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
        test VIDIOC_ENUMAUDIO: OK (Not Supported)
                fail: v4l2-test-input-output.cpp(382): std == 0
                fail: v4l2-test-input-output.cpp(437): invalid attributes for input 0
        test VIDIOC_G/S/ENUMINPUT: FAIL
        test VIDIOC_G/S_AUDIO: OK (Not Supported)
        Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
        test VIDIOC_G/S_MODULATOR: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_ENUMAUDOUT: OK (Not Supported)
        test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
        test VIDIOC_G/S_AUDOUT: OK (Not Supported)
        Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
        test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
        test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
        test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
        test VIDIOC_G/S_EDID: OK (Not Supported)

        Control ioctls:
                test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
                test VIDIOC_QUERYCTRL: OK
                test VIDIOC_G/S_CTRL: OK
                test VIDIOC_G/S/TRY_EXT_CTRLS: OK
                fail: v4l2-test-controls.cpp(779): subscribe event for control 'Camera Controls' failed
                test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
                test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
                Standard Controls: 13 Private Controls: 0

        Format ioctls:
                test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
                test VIDIOC_G/S_PARM: OK (Not Supported)
                test VIDIOC_G_FBUF: OK (Not Supported)
                fail: v4l2-test-formats.cpp(414): unknown pixelformat 42474752 for buftype 1
                test VIDIOC_G_FMT: FAIL
                test VIDIOC_TRY_FMT: OK (Not Supported)
                test VIDIOC_S_FMT: OK (Not Supported)
                test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)

        Codec ioctls:
                test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
                test VIDIOC_G_ENC_INDEX: OK (Not Supported)
                test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

        Buffer ioctls:
                test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
                fail: v4l2-test-buffers.cpp(500): q.has_expbuf(node)
                test VIDIOC_EXPBUF: FAIL


Total: 39, Succeeded: 33, Failed: 6, Warnings: 0

Not all of these may be a result of Steve's code - this is running against
my gradually modified version to support bayer formats (which seems to be
the cause of the v4l2-test-formats.cpp failure... for some reason the
driver isn't enumerating all the formats.)

And that reason is the way that the formats are enumerated:

static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
                                  struct v4l2_fmtdesc *f)
{
        const struct imx_media_pixfmt *cc;
        u32 code;
        int ret;

        ret = imx_media_enum_format(&code, f->index, true, true);
        if (ret)
                return ret;
        cc = imx_media_find_format(0, code, true, true);
        if (!cc)
                return -EINVAL;

When imx_media_enum_format() hits this entry in the table:

        }, {
                .fourcc = V4L2_PIX_FMT_BGR24,
                .cs     = IPUV3_COLORSPACE_RGB,
                .bpp    = 24,
        }, {

becaues there's no .codes defined:

int imx_media_enum_format(u32 *code, u32 index, bool allow_rgb,
                          bool allow_planar)
{
...
        *code = fmt->codes[0];
        return 0;
}

So, we end up calling imx_media_find_format(0, 0, true, true), which
fails, returning NULL.  That causes camif_enum_fmt_vid_cap() to
return -EINVAL.

So everything past this entry is unable to be enumerated.

I think this is a really round-about way of enumerating the pixel
formats when there are soo many entries in the table which have no
media bus code - there's absolutely no way that any of those entries
can ever be enumerated in this fashion, so they might as well not be
in the table...

That's my present solution to this problem, to #if 0 out all the
entries without any .codes field.  I think the real answer is that
this needs a _separate_ function to enumerate the pixel formats for
camif_enum_fmt_vid_cap().  However, there may be other issues lurking
that I've not yet found (still trying to get this code to work...)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-07  2:11 ` Steve Longerbeam
  (?)
@ 2017-01-31 13:54   ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 13:54 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
observations:

# Link pipeline
media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"

# Provide an EDID to the HDMI source
v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
# At this point the HDMI source is enabled and sends a 1080p60 signal
# Configure detected DV timings
media-ctl --set-dv "'tc358743 1-000f':0"

# Set pad formats
media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"

v4l2-ctl -d /dev/video4 -V
# This still is configured to 640x480, which is inconsistent with
# the 'ipu1_csi0':2 pad format. The pad set_fmt above should
# have set this, too.

v4l2-ctl --list-formats -d /dev/video4
# This lists all the RGB formats, which it shouldn't. There is
# no CSC in this pipeline, so we should be limited to YUV formats
# only.

# Set capture format
v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY

v4l2-ctl -d /dev/video4 -V
# Now the capture format is correctly configured to 1920x1080.

v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
# This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
# frame interval. In the future this should list fractions achievable
# via frame skipping.

v4l2-compliance -d /dev/video4
# This fails two tests:
# fail: v4l2-test-input-output.cpp(383): std == 0
# fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
# test VIDIOC_G/S/ENUMINPUT: FAIL
# and
# fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
# test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL

# (Slowly) stream JPEG images to a display host:
gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink

I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
timeout" message when starting streaming.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 13:54   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 13:54 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

Hi Steve,

I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
observations:

# Link pipeline
media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"

# Provide an EDID to the HDMI source
v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
# At this point the HDMI source is enabled and sends a 1080p60 signal
# Configure detected DV timings
media-ctl --set-dv "'tc358743 1-000f':0"

# Set pad formats
media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"

v4l2-ctl -d /dev/video4 -V
# This still is configured to 640x480, which is inconsistent with
# the 'ipu1_csi0':2 pad format. The pad set_fmt above should
# have set this, too.

v4l2-ctl --list-formats -d /dev/video4
# This lists all the RGB formats, which it shouldn't. There is
# no CSC in this pipeline, so we should be limited to YUV formats
# only.

# Set capture format
v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY

v4l2-ctl -d /dev/video4 -V
# Now the capture format is correctly configured to 1920x1080.

v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
# This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
# frame interval. In the future this should list fractions achievable
# via frame skipping.

v4l2-compliance -d /dev/video4
# This fails two tests:
# fail: v4l2-test-input-output.cpp(383): std == 0
# fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
# test VIDIOC_G/S/ENUMINPUT: FAIL
# and
# fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
# test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL

# (Slowly) stream JPEG images to a display host:
gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink

I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
timeout" message when starting streaming.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 13:54   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 13:54 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
observations:

# Link pipeline
media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"

# Provide an EDID to the HDMI source
v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
# At this point the HDMI source is enabled and sends a 1080p60 signal
# Configure detected DV timings
media-ctl --set-dv "'tc358743 1-000f':0"

# Set pad formats
media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"

v4l2-ctl -d /dev/video4 -V
# This still is configured to 640x480, which is inconsistent with
# the 'ipu1_csi0':2 pad format. The pad set_fmt above should
# have set this, too.

v4l2-ctl --list-formats -d /dev/video4
# This lists all the RGB formats, which it shouldn't. There is
# no CSC in this pipeline, so we should be limited to YUV formats
# only.

# Set capture format
v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY

v4l2-ctl -d /dev/video4 -V
# Now the capture format is correctly configured to 1920x1080.

v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
# This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
# frame interval. In the future this should list fractions achievable
# via frame skipping.

v4l2-compliance -d /dev/video4
# This fails two tests:
# fail: v4l2-test-input-output.cpp(383): std == 0
# fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
# test VIDIOC_G/S/ENUMINPUT: FAIL
# and
# fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
# test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL

# (Slowly) stream JPEG images to a display host:
gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink

I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
timeout" message when starting streaming.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31 13:35                       ` Philipp Zabel
  (?)
@ 2017-01-31 14:04                         ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 14:04 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam,
	Laurent Pinchart

On Tue, Jan 31, 2017 at 02:35:00PM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 13:14 +0000, Russell King - ARM Linux wrote:
> > This isn't limited to the serial side - the parallel bus side between
> > the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
> > the CS0/1 interfaces is much the same with 10-bit bayer.
> > 
> > Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
> > up on the least significant 8 bits of the 32-bit bus, lane 3 on the
> > top 8-bits.
> > 
> > Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
> > kind of the 2-lane case above...
> 
> You are right, on the parallel buses the format most definitely is not
> MEDIA_BUS_FMT_SBGGR10_1X10. We don't have any representation of the
> 32-bit bus between CSI2 host and CSI2IPU gasket because we model the two
> as a single entity, but the four 16-bit parallel buses between the
> CSI2IPU gasket and the IPU1/2 CSI0/1 probably should be set to a custom
> format describing this accurately.

Yep.  I should also point out that there's a very odd transformation
going on somewhere, and I don't yet know where.

The sensor is definitely outputting GBRG format, but what seems to get
written into memory is RGGB format.  It's somewhere post CSI, because
when I was using the (broken) CSI compander with 10 bit bayer, the red
compander channel affected the red channel output from the camera, but
changed the green component written to memory... it's very much like
either the first line gets lost somewhere, or the odd/even lines are
transposed.  It could also be a gstreamer bug.  As I say, it's not
something I've looked into deeply enough yet... too many other issues
to chase down!

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 14:04                         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 14:04 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Hans Verkuil, Steve Longerbeam, Laurent Pinchart, robert.jarzmik,
	devel, markus.heiser, laurent.pinchart+renesas, geert,
	Steve Longerbeam, linux-media, devicetree, arnd, mchehab,
	bparrot, robh+dt, horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 02:35:00PM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 13:14 +0000, Russell King - ARM Linux wrote:
> > This isn't limited to the serial side - the parallel bus side between
> > the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
> > the CS0/1 interfaces is much the same with 10-bit bayer.
> > 
> > Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
> > up on the least significant 8 bits of the 32-bit bus, lane 3 on the
> > top 8-bits.
> > 
> > Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
> > kind of the 2-lane case above...
> 
> You are right, on the parallel buses the format most definitely is not
> MEDIA_BUS_FMT_SBGGR10_1X10. We don't have any representation of the
> 32-bit bus between CSI2 host and CSI2IPU gasket because we model the two
> as a single entity, but the four 16-bit parallel buses between the
> CSI2IPU gasket and the IPU1/2 CSI0/1 probably should be set to a custom
> format describing this accurately.

Yep.  I should also point out that there's a very odd transformation
going on somewhere, and I don't yet know where.

The sensor is definitely outputting GBRG format, but what seems to get
written into memory is RGGB format.  It's somewhere post CSI, because
when I was using the (broken) CSI compander with 10 bit bayer, the red
compander channel affected the red channel output from the camera, but
changed the green component written to memory... it's very much like
either the first line gets lost somewhere, or the odd/even lines are
transposed.  It could also be a gstreamer bug.  As I say, it's not
something I've looked into deeply enough yet... too many other issues
to chase down!

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 14:04                         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 14:04 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 02:35:00PM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 13:14 +0000, Russell King - ARM Linux wrote:
> > This isn't limited to the serial side - the parallel bus side between
> > the CSI2 interface and CSI2IPU wrapper, and the CSI2IPU wrapper and
> > the CS0/1 interfaces is much the same with 10-bit bayer.
> > 
> > Think of the CSI2 <-> CSI2IPU bit as the 4-lane case, lane 0 ending
> > up on the least significant 8 bits of the 32-bit bus, lane 3 on the
> > top 8-bits.
> > 
> > Post CSI2IPU, it talks about a 16-bit bus in the diagrams, so that's
> > kind of the 2-lane case above...
> 
> You are right, on the parallel buses the format most definitely is not
> MEDIA_BUS_FMT_SBGGR10_1X10. We don't have any representation of the
> 32-bit bus between CSI2 host and CSI2IPU gasket because we model the two
> as a single entity, but the four 16-bit parallel buses between the
> CSI2IPU gasket and the IPU1/2 CSI0/1 probably should be set to a custom
> format describing this accurately.

Yep.  I should also point out that there's a very odd transformation
going on somewhere, and I don't yet know where.

The sensor is definitely outputting GBRG format, but what seems to get
written into memory is RGGB format.  It's somewhere post CSI, because
when I was using the (broken) CSI compander with 10 bit bayer, the red
compander channel affected the red channel output from the camera, but
changed the green component written to memory... it's very much like
either the first line gets lost somewhere, or the odd/even lines are
transposed.  It could also be a gstreamer bug.  As I say, it's not
something I've looked into deeply enough yet... too many other issues
to chase down!

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31 13:54   ` Philipp Zabel
  (?)
@ 2017-01-31 14:25     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 14:25 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, 2017-01-31 at 14:54 +0100, Philipp Zabel wrote:
> Hi Steve,
> 
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
> 
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
> 
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
> 
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"

I noticed this seems to get ignored. The format is incorrectly set to
UYVY even though I request UYVY2X8 (see CSI2IPU chapter, Figure 19-10.
YUV422-8 data reception in the reference manual), but it seems to work
anyway.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 14:25     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 14:25 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, 2017-01-31 at 14:54 +0100, Philipp Zabel wrote:
> Hi Steve,
> 
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
> 
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
> 
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
> 
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"

I noticed this seems to get ignored. The format is incorrectly set to
UYVY even though I request UYVY2X8 (see CSI2IPU chapter, Figure 19-10.
YUV422-8 data reception in the reference manual), but it seems to work
anyway.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 14:25     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 14:25 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, 2017-01-31 at 14:54 +0100, Philipp Zabel wrote:
> Hi Steve,
> 
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
> 
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
> 
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
> 
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"

I noticed this seems to get ignored. The format is incorrectly set to
UYVY even though I request UYVY2X8 (see CSI2IPU chapter, Figure 19-10.
YUV422-8 data reception in the reference manual), but it seems to work
anyway.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31 14:25     ` Philipp Zabel
  (?)
@ 2017-01-31 15:03       ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 15:03 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, Jan 31, 2017 at 03:25:10PM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 14:54 +0100, Philipp Zabel wrote:
> > Hi Steve,
> > 
> > I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> > with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> > observations:
> > 
> > # Link pipeline
> > media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> > media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> > media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
> > 
> > # Provide an EDID to the HDMI source
> > v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> > # At this point the HDMI source is enabled and sends a 1080p60 signal
> > # Configure detected DV timings
> > media-ctl --set-dv "'tc358743 1-000f':0"
> > 
> > # Set pad formats
> > media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> > media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> 
> I noticed this seems to get ignored. The format is incorrectly set to
> UYVY even though I request UYVY2X8 (see CSI2IPU chapter, Figure 19-10.
> YUV422-8 data reception in the reference manual), but it seems to work
> anyway.

That's because the driver at imx-csi level bypasses the proper media pad
formats on its sink pads, and instead goes poking about at the "sensor"
to get the format.

This is one of the reasons it wants to know which entity is the "sensor".

The "sensor" stuff in there needs to be scrapped...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 15:03       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 15:03 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, Steve Longerbeam, linux-media,
	devicetree, arnd, mchehab, bparrot, robh+dt, horms+renesas,
	tiffany.lin, linux-arm-kernel, niklas.soderlund+renesas, gregkh,
	linux-kernel, jean-christophe.trotin, kernel, fabio.estevam,
	shawnguo, sudipm.mukherjee

On Tue, Jan 31, 2017 at 03:25:10PM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 14:54 +0100, Philipp Zabel wrote:
> > Hi Steve,
> > 
> > I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> > with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> > observations:
> > 
> > # Link pipeline
> > media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> > media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> > media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
> > 
> > # Provide an EDID to the HDMI source
> > v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> > # At this point the HDMI source is enabled and sends a 1080p60 signal
> > # Configure detected DV timings
> > media-ctl --set-dv "'tc358743 1-000f':0"
> > 
> > # Set pad formats
> > media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> > media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> 
> I noticed this seems to get ignored. The format is incorrectly set to
> UYVY even though I request UYVY2X8 (see CSI2IPU chapter, Figure 19-10.
> YUV422-8 data reception in the reference manual), but it seems to work
> anyway.

That's because the driver at imx-csi level bypasses the proper media pad
formats on its sink pads, and instead goes poking about at the "sensor"
to get the format.

This is one of the reasons it wants to know which entity is the "sensor".

The "sensor" stuff in there needs to be scrapped...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 15:03       ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 15:03 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 03:25:10PM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 14:54 +0100, Philipp Zabel wrote:
> > Hi Steve,
> > 
> > I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> > with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> > observations:
> > 
> > # Link pipeline
> > media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> > media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> > media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
> > 
> > # Provide an EDID to the HDMI source
> > v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> > # At this point the HDMI source is enabled and sends a 1080p60 signal
> > # Configure detected DV timings
> > media-ctl --set-dv "'tc358743 1-000f':0"
> > 
> > # Set pad formats
> > media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> > media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> 
> I noticed this seems to get ignored. The format is incorrectly set to
> UYVY even though I request UYVY2X8 (see CSI2IPU chapter, Figure 19-10.
> YUV422-8 data reception in the reference manual), but it seems to work
> anyway.

That's because the driver at imx-csi level bypasses the proper media pad
formats on its sink pads, and instead goes poking about at the "sensor"
to get the format.

This is one of the reasons it wants to know which entity is the "sensor".

The "sensor" stuff in there needs to be scrapped...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 15:51     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 15:51 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
[...]
> +static int csi_set_fmt(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       struct v4l2_subdev_format *sdformat)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	struct v4l2_rect crop;
> +	int ret;
> +
> +	if (sdformat->pad >= CSI_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.code = infmt->code;
> +		sdformat->format.field = infmt->field;
> +		crop.left = priv->crop.left;
> +		crop.top = priv->crop.top;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop);

This is the wrong way around, see also below.

Here the the output sdformat->format.width/height should be set to the
priv->crop.width/height (or priv->crop.width/height / 2, to enable
downscaling). The crop rectangle should not be changed by an output pad
set_fmt.

> +		if (ret)
> +			return ret;
> +		sdformat->format.width = crop.width;
> +		sdformat->format.height = crop.height;
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		/* Update the crop window if this is output pad  */
> +		if (sdformat->pad == priv->output_pad)
> +			priv->crop = crop;

The crop rectangle instead should be reset to the full input frame when
the input pad format is set.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 15:51     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 15:51 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
[...]
> +static int csi_set_fmt(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       struct v4l2_subdev_format *sdformat)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	struct v4l2_rect crop;
> +	int ret;
> +
> +	if (sdformat->pad >= CSI_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.code = infmt->code;
> +		sdformat->format.field = infmt->field;
> +		crop.left = priv->crop.left;
> +		crop.top = priv->crop.top;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop);

This is the wrong way around, see also below.

Here the the output sdformat->format.width/height should be set to the
priv->crop.width/height (or priv->crop.width/height / 2, to enable
downscaling). The crop rectangle should not be changed by an output pad
set_fmt.

> +		if (ret)
> +			return ret;
> +		sdformat->format.width = crop.width;
> +		sdformat->format.height = crop.height;
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		/* Update the crop window if this is output pad  */
> +		if (sdformat->pad == priv->output_pad)
> +			priv->crop = crop;

The crop rectangle instead should be reset to the full input frame when
the input pad format is set.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 15:51     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 15:51 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
[...]
> +static int csi_set_fmt(struct v4l2_subdev *sd,
> +		       struct v4l2_subdev_pad_config *cfg,
> +		       struct v4l2_subdev_format *sdformat)
> +{
> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> +	struct v4l2_rect crop;
> +	int ret;
> +
> +	if (sdformat->pad >= CSI_NUM_PADS)
> +		return -EINVAL;
> +
> +	if (priv->stream_on)
> +		return -EBUSY;
> +
> +	infmt = &priv->format_mbus[priv->input_pad];
> +	outfmt = &priv->format_mbus[priv->output_pad];
> +
> +	if (sdformat->pad == priv->output_pad) {
> +		sdformat->format.code = infmt->code;
> +		sdformat->format.field = infmt->field;
> +		crop.left = priv->crop.left;
> +		crop.top = priv->crop.top;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop);

This is the wrong way around, see also below.

Here the the output sdformat->format.width/height should be set to the
priv->crop.width/height (or priv->crop.width/height / 2, to enable
downscaling). The crop rectangle should not be changed by an output pad
set_fmt.

> +		if (ret)
> +			return ret;
> +		sdformat->format.width = crop.width;
> +		sdformat->format.height = crop.height;
> +	}
> +
> +	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		cfg->try_fmt = sdformat->format;
> +	} else {
> +		priv->format_mbus[sdformat->pad] = sdformat->format;
> +		/* Update the crop window if this is output pad  */
> +		if (sdformat->pad == priv->output_pad)
> +			priv->crop = crop;

The crop rectangle instead should be reset to the full input frame when
the input pad format is set.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
  2017-01-31 15:51     ` Philipp Zabel
  (?)
@ 2017-01-31 16:48       ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 16:48 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, 2017-01-31 at 16:51 +0100, Philipp Zabel wrote:
> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> [...]
> > +static int csi_set_fmt(struct v4l2_subdev *sd,
> > +		       struct v4l2_subdev_pad_config *cfg,
> > +		       struct v4l2_subdev_format *sdformat)
> > +{
> > +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> > +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> > +	struct v4l2_rect crop;
> > +	int ret;
> > +
> > +	if (sdformat->pad >= CSI_NUM_PADS)
> > +		return -EINVAL;
> > +
> > +	if (priv->stream_on)
> > +		return -EBUSY;
> > +
> > +	infmt = &priv->format_mbus[priv->input_pad];
> > +	outfmt = &priv->format_mbus[priv->output_pad];
> > +
> > +	if (sdformat->pad == priv->output_pad) {
> > +		sdformat->format.code = infmt->code;
> > +		sdformat->format.field = infmt->field;
> > +		crop.left = priv->crop.left;
> > +		crop.top = priv->crop.top;
> > +		crop.width = sdformat->format.width;
> > +		crop.height = sdformat->format.height;
> > +		ret = csi_try_crop(priv, &crop);
> 
> This is the wrong way around, see also below.
> 
> Here the the output sdformat->format.width/height should be set to the
> priv->crop.width/height (or priv->crop.width/height / 2, to enable
> downscaling). The crop rectangle should not be changed by an output pad
> set_fmt.
[...]
> The crop rectangle instead should be reset to the full input frame when
> the input pad format is set.

How about this:

----------8<----------
diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
index 5e80a0871b139..8220e4204a125 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
 	if_fmt.field = outfmt->field;
 
 	ipu_csi_set_window(priv->csi, &priv->crop);
+	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
+					priv->crop.height == 2 * outfmt->height);
 
 	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
 
@@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	switch (sdformat->pad) {
 	case CSI_SRC_PAD_DIRECT:
 	case CSI_SRC_PAD_IDMAC:
-		crop.left = priv->crop.left;
-		crop.top = priv->crop.top;
-		crop.width = sdformat->format.width;
-		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, sensor);
-		if (ret)
-			return ret;
-		sdformat->format.width = crop.width;
-		sdformat->format.height = crop.height;
+		if (sdformat->format.width < priv->crop.width * 3 / 4)
+			sdformat->format.width = priv->crop.width / 2;
+		else
+			sdformat->format.width = priv->crop.width;
+		if (sdformat->format.height < priv->crop.height * 3 / 4)
+			sdformat->format.height = priv->crop.height / 2;
+		else
+			sdformat->format.height = priv->crop.height;
 
 		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
 			cc = imx_media_find_format(0, sdformat->format.code,
@@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 		}
 		break;
 	case CSI_SINK_PAD:
+		crop.left = 0;
+		crop.top = 0;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop, sensor);
+		if (ret)
+			return ret;
+
 		cc = imx_media_find_format(0, sdformat->format.code,
 					   true, false);
 		if (!cc) {
@@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	} else {
 		priv->format_mbus[sdformat->pad] = sdformat->format;
 		priv->cc[sdformat->pad] = cc;
-		/* Update the crop window if this is an output pad  */
-		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
-		    sdformat->pad == CSI_SRC_PAD_IDMAC)
+		/* Reset the crop window if this is the input pad  */
+		if (sdformat->pad == CSI_SINK_PAD)
 			priv->crop = crop;
 	}
 
---------->8----------

regards
Philipp

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 16:48       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 16:48 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, 2017-01-31 at 16:51 +0100, Philipp Zabel wrote:
> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> [...]
> > +static int csi_set_fmt(struct v4l2_subdev *sd,
> > +		       struct v4l2_subdev_pad_config *cfg,
> > +		       struct v4l2_subdev_format *sdformat)
> > +{
> > +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> > +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> > +	struct v4l2_rect crop;
> > +	int ret;
> > +
> > +	if (sdformat->pad >= CSI_NUM_PADS)
> > +		return -EINVAL;
> > +
> > +	if (priv->stream_on)
> > +		return -EBUSY;
> > +
> > +	infmt = &priv->format_mbus[priv->input_pad];
> > +	outfmt = &priv->format_mbus[priv->output_pad];
> > +
> > +	if (sdformat->pad == priv->output_pad) {
> > +		sdformat->format.code = infmt->code;
> > +		sdformat->format.field = infmt->field;
> > +		crop.left = priv->crop.left;
> > +		crop.top = priv->crop.top;
> > +		crop.width = sdformat->format.width;
> > +		crop.height = sdformat->format.height;
> > +		ret = csi_try_crop(priv, &crop);
> 
> This is the wrong way around, see also below.
> 
> Here the the output sdformat->format.width/height should be set to the
> priv->crop.width/height (or priv->crop.width/height / 2, to enable
> downscaling). The crop rectangle should not be changed by an output pad
> set_fmt.
[...]
> The crop rectangle instead should be reset to the full input frame when
> the input pad format is set.

How about this:

----------8<----------
diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
index 5e80a0871b139..8220e4204a125 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
 	if_fmt.field = outfmt->field;
 
 	ipu_csi_set_window(priv->csi, &priv->crop);
+	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
+					priv->crop.height == 2 * outfmt->height);
 
 	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
 
@@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	switch (sdformat->pad) {
 	case CSI_SRC_PAD_DIRECT:
 	case CSI_SRC_PAD_IDMAC:
-		crop.left = priv->crop.left;
-		crop.top = priv->crop.top;
-		crop.width = sdformat->format.width;
-		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, sensor);
-		if (ret)
-			return ret;
-		sdformat->format.width = crop.width;
-		sdformat->format.height = crop.height;
+		if (sdformat->format.width < priv->crop.width * 3 / 4)
+			sdformat->format.width = priv->crop.width / 2;
+		else
+			sdformat->format.width = priv->crop.width;
+		if (sdformat->format.height < priv->crop.height * 3 / 4)
+			sdformat->format.height = priv->crop.height / 2;
+		else
+			sdformat->format.height = priv->crop.height;
 
 		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
 			cc = imx_media_find_format(0, sdformat->format.code,
@@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 		}
 		break;
 	case CSI_SINK_PAD:
+		crop.left = 0;
+		crop.top = 0;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop, sensor);
+		if (ret)
+			return ret;
+
 		cc = imx_media_find_format(0, sdformat->format.code,
 					   true, false);
 		if (!cc) {
@@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	} else {
 		priv->format_mbus[sdformat->pad] = sdformat->format;
 		priv->cc[sdformat->pad] = cc;
-		/* Update the crop window if this is an output pad  */
-		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
-		    sdformat->pad == CSI_SRC_PAD_IDMAC)
+		/* Reset the crop window if this is the input pad  */
+		if (sdformat->pad == CSI_SINK_PAD)
 			priv->crop = crop;
 	}
 
---------->8----------

regards
Philipp

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-01-31 16:48       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-01-31 16:48 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, 2017-01-31 at 16:51 +0100, Philipp Zabel wrote:
> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> [...]
> > +static int csi_set_fmt(struct v4l2_subdev *sd,
> > +		       struct v4l2_subdev_pad_config *cfg,
> > +		       struct v4l2_subdev_format *sdformat)
> > +{
> > +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> > +	struct v4l2_mbus_framefmt *infmt, *outfmt;
> > +	struct v4l2_rect crop;
> > +	int ret;
> > +
> > +	if (sdformat->pad >= CSI_NUM_PADS)
> > +		return -EINVAL;
> > +
> > +	if (priv->stream_on)
> > +		return -EBUSY;
> > +
> > +	infmt = &priv->format_mbus[priv->input_pad];
> > +	outfmt = &priv->format_mbus[priv->output_pad];
> > +
> > +	if (sdformat->pad == priv->output_pad) {
> > +		sdformat->format.code = infmt->code;
> > +		sdformat->format.field = infmt->field;
> > +		crop.left = priv->crop.left;
> > +		crop.top = priv->crop.top;
> > +		crop.width = sdformat->format.width;
> > +		crop.height = sdformat->format.height;
> > +		ret = csi_try_crop(priv, &crop);
> 
> This is the wrong way around, see also below.
> 
> Here the the output sdformat->format.width/height should be set to the
> priv->crop.width/height (or priv->crop.width/height / 2, to enable
> downscaling). The crop rectangle should not be changed by an output pad
> set_fmt.
[...]
> The crop rectangle instead should be reset to the full input frame when
> the input pad format is set.

How about this:

----------8<----------
diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
index 5e80a0871b139..8220e4204a125 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
 	if_fmt.field = outfmt->field;
 
 	ipu_csi_set_window(priv->csi, &priv->crop);
+	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
+					priv->crop.height == 2 * outfmt->height);
 
 	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
 
@@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	switch (sdformat->pad) {
 	case CSI_SRC_PAD_DIRECT:
 	case CSI_SRC_PAD_IDMAC:
-		crop.left = priv->crop.left;
-		crop.top = priv->crop.top;
-		crop.width = sdformat->format.width;
-		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, sensor);
-		if (ret)
-			return ret;
-		sdformat->format.width = crop.width;
-		sdformat->format.height = crop.height;
+		if (sdformat->format.width < priv->crop.width * 3 / 4)
+			sdformat->format.width = priv->crop.width / 2;
+		else
+			sdformat->format.width = priv->crop.width;
+		if (sdformat->format.height < priv->crop.height * 3 / 4)
+			sdformat->format.height = priv->crop.height / 2;
+		else
+			sdformat->format.height = priv->crop.height;
 
 		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
 			cc = imx_media_find_format(0, sdformat->format.code,
@@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 		}
 		break;
 	case CSI_SINK_PAD:
+		crop.left = 0;
+		crop.top = 0;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop, sensor);
+		if (ret)
+			return ret;
+
 		cc = imx_media_find_format(0, sdformat->format.code,
 					   true, false);
 		if (!cc) {
@@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	} else {
 		priv->format_mbus[sdformat->pad] = sdformat->format;
 		priv->cc[sdformat->pad] = cc;
-		/* Update the crop window if this is an output pad  */
-		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
-		    sdformat->pad == CSI_SRC_PAD_IDMAC)
+		/* Reset the crop window if this is the input pad  */
+		if (sdformat->pad == CSI_SINK_PAD)
 			priv->crop = crop;
 	}
 
---------->8----------

regards
Philipp

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-31 13:42       ` Russell King - ARM Linux
@ 2017-01-31 18:21         ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 18:21 UTC (permalink / raw)
  To: Russell King - ARM Linux, Hans Verkuil
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	nick, markus.heiser, p.zabel, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam



On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>> should complain about this.
> ... and more.

Right, in version 3 that you are working with, no v4l2-compliance fixes were
in yet. A lot of the compliance errors are fixed, please look in latest 
branch
imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.


<snip>
>
>
>
> Total: 39, Succeeded: 33, Failed: 6, Warnings: 0
>
> Not all of these may be a result of Steve's code - this is running against
> my gradually modified version to support bayer formats (which seems to be
> the cause of the v4l2-test-formats.cpp failure... for some reason the
> driver isn't enumerating all the formats.)
>
> And that reason is the way that the formats are enumerated:
>
> static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
>                                    struct v4l2_fmtdesc *f)
> {
>          const struct imx_media_pixfmt *cc;
>          u32 code;
>          int ret;
>
>          ret = imx_media_enum_format(&code, f->index, true, true);
>          if (ret)
>                  return ret;
>          cc = imx_media_find_format(0, code, true, true);
>          if (!cc)
>                  return -EINVAL;
>
> When imx_media_enum_format() hits this entry in the table:
>
>          }, {
>                  .fourcc = V4L2_PIX_FMT_BGR24,
>                  .cs     = IPUV3_COLORSPACE_RGB,
>                  .bpp    = 24,
>          }, {
>
> becaues there's no .codes defined:
>
> int imx_media_enum_format(u32 *code, u32 index, bool allow_rgb,
>                            bool allow_planar)
> {
> ...
>          *code = fmt->codes[0];
>          return 0;
> }
>
> So, we end up calling imx_media_find_format(0, 0, true, true), which
> fails, returning NULL.  That causes camif_enum_fmt_vid_cap() to
> return -EINVAL.
>
> So everything past this entry is unable to be enumerated.
>
> I think this is a really round-about way of enumerating the pixel
> formats when there are soo many entries in the table which have no
> media bus code - there's absolutely no way that any of those entries
> can ever be enumerated in this fashion, so they might as well not be
> in the table...
>
> That's my present solution to this problem, to #if 0 out all the
> entries without any .codes field.  I think the real answer is that
> this needs a _separate_ function to enumerate the pixel formats for
> camif_enum_fmt_vid_cap().  However, there may be other issues lurking
> that I've not yet found (still trying to get this code to work...)

I believe this has been fixed in imx-media-staging-md-wip as well,
see imx-media-capture.c:capture_enum_fmt_vid_cap()

Camif subdev is gone, replaced with a set of exported functions
that allow attaching a capture device (and v4l2 interface) to a
calling subdev's output pad. See imx-media-capture.c.

The subdev's capture device interface is the only subdev that
can request a planar format from imx_media_enum_format().
All the others now (the non-device node pads), request only RGB
or packed YUV, or the IPU internal formats for IPU internal connections,
and these are the first entries in the table. The planar formats all are at
the end, which can only be enumerated by the capture device interfaces.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 18:21         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 18:21 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>> should complain about this.
> ... and more.

Right, in version 3 that you are working with, no v4l2-compliance fixes were
in yet. A lot of the compliance errors are fixed, please look in latest 
branch
imx-media-staging-md-wip at git at github.com:slongerbeam/mediatree.git.


<snip>
>
>
>
> Total: 39, Succeeded: 33, Failed: 6, Warnings: 0
>
> Not all of these may be a result of Steve's code - this is running against
> my gradually modified version to support bayer formats (which seems to be
> the cause of the v4l2-test-formats.cpp failure... for some reason the
> driver isn't enumerating all the formats.)
>
> And that reason is the way that the formats are enumerated:
>
> static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
>                                    struct v4l2_fmtdesc *f)
> {
>          const struct imx_media_pixfmt *cc;
>          u32 code;
>          int ret;
>
>          ret = imx_media_enum_format(&code, f->index, true, true);
>          if (ret)
>                  return ret;
>          cc = imx_media_find_format(0, code, true, true);
>          if (!cc)
>                  return -EINVAL;
>
> When imx_media_enum_format() hits this entry in the table:
>
>          }, {
>                  .fourcc = V4L2_PIX_FMT_BGR24,
>                  .cs     = IPUV3_COLORSPACE_RGB,
>                  .bpp    = 24,
>          }, {
>
> becaues there's no .codes defined:
>
> int imx_media_enum_format(u32 *code, u32 index, bool allow_rgb,
>                            bool allow_planar)
> {
> ...
>          *code = fmt->codes[0];
>          return 0;
> }
>
> So, we end up calling imx_media_find_format(0, 0, true, true), which
> fails, returning NULL.  That causes camif_enum_fmt_vid_cap() to
> return -EINVAL.
>
> So everything past this entry is unable to be enumerated.
>
> I think this is a really round-about way of enumerating the pixel
> formats when there are soo many entries in the table which have no
> media bus code - there's absolutely no way that any of those entries
> can ever be enumerated in this fashion, so they might as well not be
> in the table...
>
> That's my present solution to this problem, to #if 0 out all the
> entries without any .codes field.  I think the real answer is that
> this needs a _separate_ function to enumerate the pixel formats for
> camif_enum_fmt_vid_cap().  However, there may be other issues lurking
> that I've not yet found (still trying to get this code to work...)

I believe this has been fixed in imx-media-staging-md-wip as well,
see imx-media-capture.c:capture_enum_fmt_vid_cap()

Camif subdev is gone, replaced with a set of exported functions
that allow attaching a capture device (and v4l2 interface) to a
calling subdev's output pad. See imx-media-capture.c.

The subdev's capture device interface is the only subdev that
can request a planar format from imx_media_enum_format().
All the others now (the non-device node pads), request only RGB
or packed YUV, or the IPU internal formats for IPU internal connections,
and these are the first entries in the table. The planar formats all are at
the end, which can only be enumerated by the capture device interfaces.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-31 18:21         ` Steve Longerbeam
  (?)
@ 2017-01-31 20:33           ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 20:33 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Hans Verkuil, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> >On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> >>Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> >>should complain about this.
> >... and more.
> 
> Right, in version 3 that you are working with, no v4l2-compliance fixes were
> in yet. A lot of the compliance errors are fixed, please look in latest
> branch
> imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.

Sorry, I'm not prepared to pull random trees from github as there's
no easy way to see what's in the branch.

I've always disliked github because its web interface makes it soo
difficult to navigate around git trees hosted there.  You can see
a commit, you can see a diff of the commit.  You can get a list of
branches.  But there seems to be no way to get a list of commits
similar to "git log" or even a one-line summary of each commit on
a branch.  If there is, it's completely non-obvious (which I think is
much of the problem with github, it's web interface is horrendous.)

Or you can clone/pull the tree without knowing what you're fetching
(eg, what the tree is based upon.)

Or you can waste time clicking repeatedly on the "parent" commit link
on each patch working your way back through the history...

Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
from the linux-media tree (I didn't try and go back any further.)
As I don't want to take a whole pile of other changes into my tree,
I'm certainly not going to pull from your github tree.  Sorry.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 20:33           ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 20:33 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Hans Verkuil, Steve Longerbeam, robert.jarzmik, devel,
	markus.heiser, laurent.pinchart+renesas, geert, linux-media,
	devicetree, kernel, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> >On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> >>Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> >>should complain about this.
> >... and more.
> 
> Right, in version 3 that you are working with, no v4l2-compliance fixes were
> in yet. A lot of the compliance errors are fixed, please look in latest
> branch
> imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.

Sorry, I'm not prepared to pull random trees from github as there's
no easy way to see what's in the branch.

I've always disliked github because its web interface makes it soo
difficult to navigate around git trees hosted there.  You can see
a commit, you can see a diff of the commit.  You can get a list of
branches.  But there seems to be no way to get a list of commits
similar to "git log" or even a one-line summary of each commit on
a branch.  If there is, it's completely non-obvious (which I think is
much of the problem with github, it's web interface is horrendous.)

Or you can clone/pull the tree without knowing what you're fetching
(eg, what the tree is based upon.)

Or you can waste time clicking repeatedly on the "parent" commit link
on each patch working your way back through the history...

Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
from the linux-media tree (I didn't try and go back any further.)
As I don't want to take a whole pile of other changes into my tree,
I'm certainly not going to pull from your github tree.  Sorry.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 20:33           ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 20:33 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> >On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> >>Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> >>should complain about this.
> >... and more.
> 
> Right, in version 3 that you are working with, no v4l2-compliance fixes were
> in yet. A lot of the compliance errors are fixed, please look in latest
> branch
> imx-media-staging-md-wip at git at github.com:slongerbeam/mediatree.git.

Sorry, I'm not prepared to pull random trees from github as there's
no easy way to see what's in the branch.

I've always disliked github because its web interface makes it soo
difficult to navigate around git trees hosted there.  You can see
a commit, you can see a diff of the commit.  You can get a list of
branches.  But there seems to be no way to get a list of commits
similar to "git log" or even a one-line summary of each commit on
a branch.  If there is, it's completely non-obvious (which I think is
much of the problem with github, it's web interface is horrendous.)

Or you can clone/pull the tree without knowing what you're fetching
(eg, what the tree is based upon.)

Or you can waste time clicking repeatedly on the "parent" commit link
on each patch working your way back through the history...

Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
from the linux-media tree (I didn't try and go back any further.)
As I don't want to take a whole pile of other changes into my tree,
I'm certainly not going to pull from your github tree.  Sorry.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 21:55             ` Ian Arkver
  0 siblings, 0 replies; 549+ messages in thread
From: Ian Arkver @ 2017-01-31 21:55 UTC (permalink / raw)
  To: Russell King - ARM Linux, Steve Longerbeam
  Cc: Hans Verkuil, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On 31/01/17 20:33, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>> should complain about this.
>>> ... and more.
>>
>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>> in yet. A lot of the compliance errors are fixed, please look in latest
>> branch
>> imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.
>
> Sorry, I'm not prepared to pull random trees from github as there's
> no easy way to see what's in the branch.
>
> I've always disliked github because its web interface makes it soo
> difficult to navigate around git trees hosted there.  You can see
> a commit, you can see a diff of the commit.  You can get a list of
> branches.  But there seems to be no way to get a list of commits
> similar to "git log" or even a one-line summary of each commit on
> a branch.  If there is, it's completely non-obvious (which I think is
> much of the problem with github, it's web interface is horrendous.)
>
> Or you can clone/pull the tree without knowing what you're fetching
> (eg, what the tree is based upon.)
>
> Or you can waste time clicking repeatedly on the "parent" commit link
> on each patch working your way back through the history...
>
> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
> from the linux-media tree (I didn't try and go back any further.)
> As I don't want to take a whole pile of other changes into my tree,
> I'm certainly not going to pull from your github tree.  Sorry.
>

https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip

It's under the "Compare" button from the main view. It would be nice 
though if the first commit's parent was some clearly tagged start point.

Regards,
Ian

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 21:55             ` Ian Arkver
  0 siblings, 0 replies; 549+ messages in thread
From: Ian Arkver @ 2017-01-31 21:55 UTC (permalink / raw)
  To: Russell King - ARM Linux, Steve Longerbeam
  Cc: Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

On 31/01/17 20:33, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>> should complain about this.
>>> ... and more.
>>
>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>> in yet. A lot of the compliance errors are fixed, please look in latest
>> branch
>> imx-media-staging-md-wip at git-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org:slongerbeam/mediatree.git.
>
> Sorry, I'm not prepared to pull random trees from github as there's
> no easy way to see what's in the branch.
>
> I've always disliked github because its web interface makes it soo
> difficult to navigate around git trees hosted there.  You can see
> a commit, you can see a diff of the commit.  You can get a list of
> branches.  But there seems to be no way to get a list of commits
> similar to "git log" or even a one-line summary of each commit on
> a branch.  If there is, it's completely non-obvious (which I think is
> much of the problem with github, it's web interface is horrendous.)
>
> Or you can clone/pull the tree without knowing what you're fetching
> (eg, what the tree is based upon.)
>
> Or you can waste time clicking repeatedly on the "parent" commit link
> on each patch working your way back through the history...
>
> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
> from the linux-media tree (I didn't try and go back any further.)
> As I don't want to take a whole pile of other changes into my tree,
> I'm certainly not going to pull from your github tree.  Sorry.
>

https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip

It's under the "Compare" button from the main view. It would be nice 
though if the first commit's parent was some clearly tagged start point.

Regards,
Ian
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 21:55             ` Ian Arkver
  0 siblings, 0 replies; 549+ messages in thread
From: Ian Arkver @ 2017-01-31 21:55 UTC (permalink / raw)
  To: linux-arm-kernel

On 31/01/17 20:33, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>> should complain about this.
>>> ... and more.
>>
>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>> in yet. A lot of the compliance errors are fixed, please look in latest
>> branch
>> imx-media-staging-md-wip at git at github.com:slongerbeam/mediatree.git.
>
> Sorry, I'm not prepared to pull random trees from github as there's
> no easy way to see what's in the branch.
>
> I've always disliked github because its web interface makes it soo
> difficult to navigate around git trees hosted there.  You can see
> a commit, you can see a diff of the commit.  You can get a list of
> branches.  But there seems to be no way to get a list of commits
> similar to "git log" or even a one-line summary of each commit on
> a branch.  If there is, it's completely non-obvious (which I think is
> much of the problem with github, it's web interface is horrendous.)
>
> Or you can clone/pull the tree without knowing what you're fetching
> (eg, what the tree is based upon.)
>
> Or you can waste time clicking repeatedly on the "parent" commit link
> on each patch working your way back through the history...
>
> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
> from the linux-media tree (I didn't try and go back any further.)
> As I don't want to take a whole pile of other changes into my tree,
> I'm certainly not going to pull from your github tree.  Sorry.
>

https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip

It's under the "Compare" button from the main view. It would be nice 
though if the first commit's parent was some clearly tagged start point.

Regards,
Ian

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-31 21:55             ` Ian Arkver
  (?)
@ 2017-01-31 22:04               ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 22:04 UTC (permalink / raw)
  To: Ian Arkver
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
> On 31/01/17 20:33, Russell King - ARM Linux wrote:
> >On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
> >>On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> >>>On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> >>>>Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> >>>>should complain about this.
> >>>... and more.
> >>
> >>Right, in version 3 that you are working with, no v4l2-compliance fixes were
> >>in yet. A lot of the compliance errors are fixed, please look in latest
> >>branch
> >>imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.
> >
> >Sorry, I'm not prepared to pull random trees from github as there's
> >no easy way to see what's in the branch.
> >
> >I've always disliked github because its web interface makes it soo
> >difficult to navigate around git trees hosted there.  You can see
> >a commit, you can see a diff of the commit.  You can get a list of
> >branches.  But there seems to be no way to get a list of commits
> >similar to "git log" or even a one-line summary of each commit on
> >a branch.  If there is, it's completely non-obvious (which I think is
> >much of the problem with github, it's web interface is horrendous.)
> >
> >Or you can clone/pull the tree without knowing what you're fetching
> >(eg, what the tree is based upon.)
> >
> >Or you can waste time clicking repeatedly on the "parent" commit link
> >on each patch working your way back through the history...
> >
> >Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
> >from the linux-media tree (I didn't try and go back any further.)
> >As I don't want to take a whole pile of other changes into my tree,
> >I'm certainly not going to pull from your github tree.  Sorry.
> >
> 
> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
> 
> It's under the "Compare" button from the main view. It would be nice though
> if the first commit's parent was some clearly tagged start point.

I don't want master though, I want v4.10-rc1, and if I ask for that
it tells me it knows nothing about v4.10-rc1, despite the fact that's
a tag in the mainline kernel repository which was merged into the
linux-media tree that this tree is based upon.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 22:04               ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 22:04 UTC (permalink / raw)
  To: Ian Arkver
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Hans Verkuil, Steve Longerbeam, robert.jarzmik, devel,
	markus.heiser, laurent.pinchart+renesas, geert, Steve Longerbeam,
	linux-media, devicetree, kernel, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
> On 31/01/17 20:33, Russell King - ARM Linux wrote:
> >On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
> >>On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> >>>On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> >>>>Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> >>>>should complain about this.
> >>>... and more.
> >>
> >>Right, in version 3 that you are working with, no v4l2-compliance fixes were
> >>in yet. A lot of the compliance errors are fixed, please look in latest
> >>branch
> >>imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.
> >
> >Sorry, I'm not prepared to pull random trees from github as there's
> >no easy way to see what's in the branch.
> >
> >I've always disliked github because its web interface makes it soo
> >difficult to navigate around git trees hosted there.  You can see
> >a commit, you can see a diff of the commit.  You can get a list of
> >branches.  But there seems to be no way to get a list of commits
> >similar to "git log" or even a one-line summary of each commit on
> >a branch.  If there is, it's completely non-obvious (which I think is
> >much of the problem with github, it's web interface is horrendous.)
> >
> >Or you can clone/pull the tree without knowing what you're fetching
> >(eg, what the tree is based upon.)
> >
> >Or you can waste time clicking repeatedly on the "parent" commit link
> >on each patch working your way back through the history...
> >
> >Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
> >from the linux-media tree (I didn't try and go back any further.)
> >As I don't want to take a whole pile of other changes into my tree,
> >I'm certainly not going to pull from your github tree.  Sorry.
> >
> 
> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
> 
> It's under the "Compare" button from the main view. It would be nice though
> if the first commit's parent was some clearly tagged start point.

I don't want master though, I want v4.10-rc1, and if I ask for that
it tells me it knows nothing about v4.10-rc1, despite the fact that's
a tag in the mainline kernel repository which was merged into the
linux-media tree that this tree is based upon.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 22:04               ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 22:04 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
> On 31/01/17 20:33, Russell King - ARM Linux wrote:
> >On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
> >>On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
> >>>On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
> >>>>Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
> >>>>should complain about this.
> >>>... and more.
> >>
> >>Right, in version 3 that you are working with, no v4l2-compliance fixes were
> >>in yet. A lot of the compliance errors are fixed, please look in latest
> >>branch
> >>imx-media-staging-md-wip at git at github.com:slongerbeam/mediatree.git.
> >
> >Sorry, I'm not prepared to pull random trees from github as there's
> >no easy way to see what's in the branch.
> >
> >I've always disliked github because its web interface makes it soo
> >difficult to navigate around git trees hosted there.  You can see
> >a commit, you can see a diff of the commit.  You can get a list of
> >branches.  But there seems to be no way to get a list of commits
> >similar to "git log" or even a one-line summary of each commit on
> >a branch.  If there is, it's completely non-obvious (which I think is
> >much of the problem with github, it's web interface is horrendous.)
> >
> >Or you can clone/pull the tree without knowing what you're fetching
> >(eg, what the tree is based upon.)
> >
> >Or you can waste time clicking repeatedly on the "parent" commit link
> >on each patch working your way back through the history...
> >
> >Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
> >from the linux-media tree (I didn't try and go back any further.)
> >As I don't want to take a whole pile of other changes into my tree,
> >I'm certainly not going to pull from your github tree.  Sorry.
> >
> 
> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
> 
> It's under the "Compare" button from the main view. It would be nice though
> if the first commit's parent was some clearly tagged start point.

I don't want master though, I want v4.10-rc1, and if I ask for that
it tells me it knows nothing about v4.10-rc1, despite the fact that's
a tag in the mainline kernel repository which was merged into the
linux-media tree that this tree is based upon.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-31 22:04               ` Russell King - ARM Linux
  (?)
@ 2017-01-31 22:33                 ` Ian Arkver
  -1 siblings, 0 replies; 549+ messages in thread
From: Ian Arkver @ 2017-01-31 22:33 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On 31/01/17 22:04, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
>> On 31/01/17 20:33, Russell King - ARM Linux wrote:
>>> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>>>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>>>> should complain about this.
>>>>> ... and more.
>>>>
>>>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>>>> in yet. A lot of the compliance errors are fixed, please look in latest
>>>> branch
>>>> imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.
>>>
>>> Sorry, I'm not prepared to pull random trees from github as there's
>>> no easy way to see what's in the branch.
>>>
>>> I've always disliked github because its web interface makes it soo
>>> difficult to navigate around git trees hosted there.  You can see
>>> a commit, you can see a diff of the commit.  You can get a list of
>>> branches.  But there seems to be no way to get a list of commits
>>> similar to "git log" or even a one-line summary of each commit on
>>> a branch.  If there is, it's completely non-obvious (which I think is
>>> much of the problem with github, it's web interface is horrendous.)
>>>
>>> Or you can clone/pull the tree without knowing what you're fetching
>>> (eg, what the tree is based upon.)
>>>
>>> Or you can waste time clicking repeatedly on the "parent" commit link
>>> on each patch working your way back through the history...
>>>
>>> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
>> >from the linux-media tree (I didn't try and go back any further.)
>>> As I don't want to take a whole pile of other changes into my tree,
>>> I'm certainly not going to pull from your github tree.  Sorry.
>>>
>>
>> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
>>
>> It's under the "Compare" button from the main view. It would be nice though
>> if the first commit's parent was some clearly tagged start point.
>
> I don't want master though, I want v4.10-rc1, and if I ask for that
> it tells me it knows nothing about v4.10-rc1, despite the fact that's
> a tag in the mainline kernel repository which was merged into the
> linux-media tree that this tree is based upon.
>

Yeah, that's what I meant about the first parent's commit not being a 
clearly tagged branch point. At least you get the series on one page. 
Maybe it's time for a rebase or a v4 series Steve?

Personally, I use a bare repo with multiple remotes and fetch branches 
from various trees. Then gitk --all --since(etc) is pretty good at 
giving the overview picture. You don't need to pull the commits over 
into any of your working branches if you don't want to.

Regards,
Ian

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 22:33                 ` Ian Arkver
  0 siblings, 0 replies; 549+ messages in thread
From: Ian Arkver @ 2017-01-31 22:33 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	Hans Verkuil, Steve Longerbeam, robert.jarzmik, devel,
	markus.heiser, laurent.pinchart+renesas, geert, Steve Longerbeam,
	linux-media, devicetree, kernel, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On 31/01/17 22:04, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
>> On 31/01/17 20:33, Russell King - ARM Linux wrote:
>>> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>>>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>>>> should complain about this.
>>>>> ... and more.
>>>>
>>>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>>>> in yet. A lot of the compliance errors are fixed, please look in latest
>>>> branch
>>>> imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.
>>>
>>> Sorry, I'm not prepared to pull random trees from github as there's
>>> no easy way to see what's in the branch.
>>>
>>> I've always disliked github because its web interface makes it soo
>>> difficult to navigate around git trees hosted there.  You can see
>>> a commit, you can see a diff of the commit.  You can get a list of
>>> branches.  But there seems to be no way to get a list of commits
>>> similar to "git log" or even a one-line summary of each commit on
>>> a branch.  If there is, it's completely non-obvious (which I think is
>>> much of the problem with github, it's web interface is horrendous.)
>>>
>>> Or you can clone/pull the tree without knowing what you're fetching
>>> (eg, what the tree is based upon.)
>>>
>>> Or you can waste time clicking repeatedly on the "parent" commit link
>>> on each patch working your way back through the history...
>>>
>>> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
>> >from the linux-media tree (I didn't try and go back any further.)
>>> As I don't want to take a whole pile of other changes into my tree,
>>> I'm certainly not going to pull from your github tree.  Sorry.
>>>
>>
>> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
>>
>> It's under the "Compare" button from the main view. It would be nice though
>> if the first commit's parent was some clearly tagged start point.
>
> I don't want master though, I want v4.10-rc1, and if I ask for that
> it tells me it knows nothing about v4.10-rc1, despite the fact that's
> a tag in the mainline kernel repository which was merged into the
> linux-media tree that this tree is based upon.
>

Yeah, that's what I meant about the first parent's commit not being a 
clearly tagged branch point. At least you get the series on one page. 
Maybe it's time for a rebase or a v4 series Steve?

Personally, I use a bare repo with multiple remotes and fetch branches 
from various trees. Then gitk --all --since(etc) is pretty good at 
giving the overview picture. You don't need to pull the commits over 
into any of your working branches if you don't want to.

Regards,
Ian

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 22:33                 ` Ian Arkver
  0 siblings, 0 replies; 549+ messages in thread
From: Ian Arkver @ 2017-01-31 22:33 UTC (permalink / raw)
  To: linux-arm-kernel

On 31/01/17 22:04, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
>> On 31/01/17 20:33, Russell King - ARM Linux wrote:
>>> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>>>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>>>> should complain about this.
>>>>> ... and more.
>>>>
>>>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>>>> in yet. A lot of the compliance errors are fixed, please look in latest
>>>> branch
>>>> imx-media-staging-md-wip at git at github.com:slongerbeam/mediatree.git.
>>>
>>> Sorry, I'm not prepared to pull random trees from github as there's
>>> no easy way to see what's in the branch.
>>>
>>> I've always disliked github because its web interface makes it soo
>>> difficult to navigate around git trees hosted there.  You can see
>>> a commit, you can see a diff of the commit.  You can get a list of
>>> branches.  But there seems to be no way to get a list of commits
>>> similar to "git log" or even a one-line summary of each commit on
>>> a branch.  If there is, it's completely non-obvious (which I think is
>>> much of the problem with github, it's web interface is horrendous.)
>>>
>>> Or you can clone/pull the tree without knowing what you're fetching
>>> (eg, what the tree is based upon.)
>>>
>>> Or you can waste time clicking repeatedly on the "parent" commit link
>>> on each patch working your way back through the history...
>>>
>>> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
>> >from the linux-media tree (I didn't try and go back any further.)
>>> As I don't want to take a whole pile of other changes into my tree,
>>> I'm certainly not going to pull from your github tree.  Sorry.
>>>
>>
>> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
>>
>> It's under the "Compare" button from the main view. It would be nice though
>> if the first commit's parent was some clearly tagged start point.
>
> I don't want master though, I want v4.10-rc1, and if I ask for that
> it tells me it knows nothing about v4.10-rc1, despite the fact that's
> a tag in the mainline kernel repository which was merged into the
> linux-media tree that this tree is based upon.
>

Yeah, that's what I meant about the first parent's commit not being a 
clearly tagged branch point. At least you get the series on one page. 
Maybe it's time for a rebase or a v4 series Steve?

Personally, I use a bare repo with multiple remotes and fetch branches 
from various trees. Then gitk --all --since(etc) is pretty good at 
giving the overview picture. You don't need to pull the commits over 
into any of your working branches if you don't want to.

Regards,
Ian

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 22:36                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 22:36 UTC (permalink / raw)
  To: Russell King - ARM Linux, Ian Arkver
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel



On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
>> On 31/01/17 20:33, Russell King - ARM Linux wrote:
>>> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>>>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>>>> should complain about this.
>>>>> ... and more.
>>>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>>>> in yet. A lot of the compliance errors are fixed, please look in latest
>>>> branch
>>>> imx-media-staging-md-wip at git@github.com:slongerbeam/mediatree.git.
>>> Sorry, I'm not prepared to pull random trees from github as there's
>>> no easy way to see what's in the branch.
>>>
>>> I've always disliked github because its web interface makes it soo
>>> difficult to navigate around git trees hosted there.  You can see
>>> a commit, you can see a diff of the commit.  You can get a list of
>>> branches.  But there seems to be no way to get a list of commits
>>> similar to "git log" or even a one-line summary of each commit on
>>> a branch.  If there is, it's completely non-obvious (which I think is
>>> much of the problem with github, it's web interface is horrendous.)
>>>
>>> Or you can clone/pull the tree without knowing what you're fetching
>>> (eg, what the tree is based upon.)
>>>
>>> Or you can waste time clicking repeatedly on the "parent" commit link
>>> on each patch working your way back through the history...
>>>
>>> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
>> >from the linux-media tree (I didn't try and go back any further.)
>>> As I don't want to take a whole pile of other changes into my tree,
>>> I'm certainly not going to pull from your github tree.  Sorry.
>>>
>> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
>>
>> It's under the "Compare" button from the main view. It would be nice though
>> if the first commit's parent was some clearly tagged start point.
> I don't want master though, I want v4.10-rc1, and if I ask for that
> it tells me it knows nothing about v4.10-rc1, despite the fact that's
> a tag in the mainline kernel repository which was merged into the
> linux-media tree that this tree is based upon.

Hi Russell, yes git@github.com:slongerbeam/mediatree.git is a fork
of the linux-media tree, and the imx-media-staging-md-wip branch
is up-to-date with master, currently at 4.10-rc1.

You don't need to use the web interface, just git clone the repo.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 22:36                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 22:36 UTC (permalink / raw)
  To: Russell King - ARM Linux, Ian Arkver
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA, linux-a



On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
>> On 31/01/17 20:33, Russell King - ARM Linux wrote:
>>> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>>>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>>>> should complain about this.
>>>>> ... and more.
>>>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>>>> in yet. A lot of the compliance errors are fixed, please look in latest
>>>> branch
>>>> imx-media-staging-md-wip at git-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org:slongerbeam/mediatree.git.
>>> Sorry, I'm not prepared to pull random trees from github as there's
>>> no easy way to see what's in the branch.
>>>
>>> I've always disliked github because its web interface makes it soo
>>> difficult to navigate around git trees hosted there.  You can see
>>> a commit, you can see a diff of the commit.  You can get a list of
>>> branches.  But there seems to be no way to get a list of commits
>>> similar to "git log" or even a one-line summary of each commit on
>>> a branch.  If there is, it's completely non-obvious (which I think is
>>> much of the problem with github, it's web interface is horrendous.)
>>>
>>> Or you can clone/pull the tree without knowing what you're fetching
>>> (eg, what the tree is based upon.)
>>>
>>> Or you can waste time clicking repeatedly on the "parent" commit link
>>> on each patch working your way back through the history...
>>>
>>> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
>> >from the linux-media tree (I didn't try and go back any further.)
>>> As I don't want to take a whole pile of other changes into my tree,
>>> I'm certainly not going to pull from your github tree.  Sorry.
>>>
>> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
>>
>> It's under the "Compare" button from the main view. It would be nice though
>> if the first commit's parent was some clearly tagged start point.
> I don't want master though, I want v4.10-rc1, and if I ask for that
> it tells me it knows nothing about v4.10-rc1, despite the fact that's
> a tag in the mainline kernel repository which was merged into the
> linux-media tree that this tree is based upon.

Hi Russell, yes git-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org:slongerbeam/mediatree.git is a fork
of the linux-media tree, and the imx-media-staging-md-wip branch
is up-to-date with master, currently at 4.10-rc1.

You don't need to use the web interface, just git clone the repo.

Steve


--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 22:36                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 22:36 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 09:55:29PM +0000, Ian Arkver wrote:
>> On 31/01/17 20:33, Russell King - ARM Linux wrote:
>>> On Tue, Jan 31, 2017 at 10:21:26AM -0800, Steve Longerbeam wrote:
>>>> On 01/31/2017 05:42 AM, Russell King - ARM Linux wrote:
>>>>> On Fri, Jan 20, 2017 at 03:38:28PM +0100, Hans Verkuil wrote:
>>>>>> Should be set to something like 'platform:imx-media-camif'. v4l2-compliance
>>>>>> should complain about this.
>>>>> ... and more.
>>>> Right, in version 3 that you are working with, no v4l2-compliance fixes were
>>>> in yet. A lot of the compliance errors are fixed, please look in latest
>>>> branch
>>>> imx-media-staging-md-wip at git at github.com:slongerbeam/mediatree.git.
>>> Sorry, I'm not prepared to pull random trees from github as there's
>>> no easy way to see what's in the branch.
>>>
>>> I've always disliked github because its web interface makes it soo
>>> difficult to navigate around git trees hosted there.  You can see
>>> a commit, you can see a diff of the commit.  You can get a list of
>>> branches.  But there seems to be no way to get a list of commits
>>> similar to "git log" or even a one-line summary of each commit on
>>> a branch.  If there is, it's completely non-obvious (which I think is
>>> much of the problem with github, it's web interface is horrendous.)
>>>
>>> Or you can clone/pull the tree without knowing what you're fetching
>>> (eg, what the tree is based upon.)
>>>
>>> Or you can waste time clicking repeatedly on the "parent" commit link
>>> on each patch working your way back through the history...
>>>
>>> Well, it looks like it's bsaed on 4.10-rc1 with who-knows-what work
>> >from the linux-media tree (I didn't try and go back any further.)
>>> As I don't want to take a whole pile of other changes into my tree,
>>> I'm certainly not going to pull from your github tree.  Sorry.
>>>
>> https://github.com/slongerbeam/mediatree/compare/master...imx-media-staging-md-wip
>>
>> It's under the "Compare" button from the main view. It would be nice though
>> if the first commit's parent was some clearly tagged start point.
> I don't want master though, I want v4.10-rc1, and if I ask for that
> it tells me it knows nothing about v4.10-rc1, despite the fact that's
> a tag in the mainline kernel repository which was merged into the
> linux-media tree that this tree is based upon.

Hi Russell, yes git at github.com:slongerbeam/mediatree.git is a fork
of the linux-media tree, and the imx-media-staging-md-wip branch
is up-to-date with master, currently at 4.10-rc1.

You don't need to use the web interface, just git clone the repo.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-31 22:36                 ` Steve Longerbeam
  (?)
@ 2017-01-31 23:30                   ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 23:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Ian Arkver, Steve Longerbeam, Hans Verkuil, robh+dt,
	mark.rutland, shawnguo, kernel, fabio.estevam, mchehab, nick,
	markus.heiser, p.zabel, laurent.pinchart+renesas, bparrot, geert,
	arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel

On Tue, Jan 31, 2017 at 02:36:53PM -0800, Steve Longerbeam wrote:
> On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
> >I don't want master though, I want v4.10-rc1, and if I ask for that
> >it tells me it knows nothing about v4.10-rc1, despite the fact that's
> >a tag in the mainline kernel repository which was merged into the
> >linux-media tree that this tree is based upon.
> 
> Hi Russell, yes git@github.com:slongerbeam/mediatree.git is a fork
> of the linux-media tree, and the imx-media-staging-md-wip branch
> is up-to-date with master, currently at 4.10-rc1.

"up to date" is different from "contains other stuff other than is in
4.10-rc1".

What I see in your tree is that your code is based off a merge commit
between something called "patchwork" (which I assume is a branch in
the media tree containing stuff commited from patch work) and v4.10-rc1.

Now, you don't get a commit when merging unless there's changes that
aren't in the commit you're merging - if "patchwork" was up to date
with v4.10-rc1, then git would have done a "fast forward" to v4.10-rc1.

Therefore, while it may be "up to date" with v4.10-rc1 in so far that
it's had v4.10-rc1 merged into it, that's not what I've been saying.
There are other changes below that merge commit which aren't in
v4.10-rc1.  It's those other changes that I'm talking about, and it's
those other changes I do not want without knowing what they are.

It may be that those other changes have since been merged into
v4.10-rc6 - but github's web interface can't show me that.  In fact,
github's web interface is pretty damned useless as far as this stuff
goes.

So, what I'll get if I clone or pull your imx-media-staging-md-wip
branch is, yes, a copy of all your changes, but _also_ all the
changes that are in the media tree that _aren't_ in mainline at the
point that v4.10-rc1 was merged.

> You don't need to use the web interface, just git clone the repo.

You're assuming I want to work off the top of your commits.  I don't.
I've got other dependencies.

Then there's yet another problem - lets say that I get a copy of your
patches that haven't been on the mailing list, and I then want to make
a comment about it.  I can't reply to a patch that hasn't been on the
mailing list.  So, the long established mechanism by which the Linux
community does patch review breaks down.

So no, sorry, I'm not fetching your tree, and I will persist with your
v3 patch set for the time being.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 23:30                   ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 23:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, Ian Arkver,
	songjun.wu, Hans Verkuil, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, Steve Longerbeam, linux-media,
	devicetree, kernel, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 02:36:53PM -0800, Steve Longerbeam wrote:
> On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
> >I don't want master though, I want v4.10-rc1, and if I ask for that
> >it tells me it knows nothing about v4.10-rc1, despite the fact that's
> >a tag in the mainline kernel repository which was merged into the
> >linux-media tree that this tree is based upon.
> 
> Hi Russell, yes git@github.com:slongerbeam/mediatree.git is a fork
> of the linux-media tree, and the imx-media-staging-md-wip branch
> is up-to-date with master, currently at 4.10-rc1.

"up to date" is different from "contains other stuff other than is in
4.10-rc1".

What I see in your tree is that your code is based off a merge commit
between something called "patchwork" (which I assume is a branch in
the media tree containing stuff commited from patch work) and v4.10-rc1.

Now, you don't get a commit when merging unless there's changes that
aren't in the commit you're merging - if "patchwork" was up to date
with v4.10-rc1, then git would have done a "fast forward" to v4.10-rc1.

Therefore, while it may be "up to date" with v4.10-rc1 in so far that
it's had v4.10-rc1 merged into it, that's not what I've been saying.
There are other changes below that merge commit which aren't in
v4.10-rc1.  It's those other changes that I'm talking about, and it's
those other changes I do not want without knowing what they are.

It may be that those other changes have since been merged into
v4.10-rc6 - but github's web interface can't show me that.  In fact,
github's web interface is pretty damned useless as far as this stuff
goes.

So, what I'll get if I clone or pull your imx-media-staging-md-wip
branch is, yes, a copy of all your changes, but _also_ all the
changes that are in the media tree that _aren't_ in mainline at the
point that v4.10-rc1 was merged.

> You don't need to use the web interface, just git clone the repo.

You're assuming I want to work off the top of your commits.  I don't.
I've got other dependencies.

Then there's yet another problem - lets say that I get a copy of your
patches that haven't been on the mailing list, and I then want to make
a comment about it.  I can't reply to a patch that hasn't been on the
mailing list.  So, the long established mechanism by which the Linux
community does patch review breaks down.

So no, sorry, I'm not fetching your tree, and I will persist with your
v3 patch set for the time being.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 23:30                   ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-01-31 23:30 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 02:36:53PM -0800, Steve Longerbeam wrote:
> On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
> >I don't want master though, I want v4.10-rc1, and if I ask for that
> >it tells me it knows nothing about v4.10-rc1, despite the fact that's
> >a tag in the mainline kernel repository which was merged into the
> >linux-media tree that this tree is based upon.
> 
> Hi Russell, yes git at github.com:slongerbeam/mediatree.git is a fork
> of the linux-media tree, and the imx-media-staging-md-wip branch
> is up-to-date with master, currently at 4.10-rc1.

"up to date" is different from "contains other stuff other than is in
4.10-rc1".

What I see in your tree is that your code is based off a merge commit
between something called "patchwork" (which I assume is a branch in
the media tree containing stuff commited from patch work) and v4.10-rc1.

Now, you don't get a commit when merging unless there's changes that
aren't in the commit you're merging - if "patchwork" was up to date
with v4.10-rc1, then git would have done a "fast forward" to v4.10-rc1.

Therefore, while it may be "up to date" with v4.10-rc1 in so far that
it's had v4.10-rc1 merged into it, that's not what I've been saying.
There are other changes below that merge commit which aren't in
v4.10-rc1.  It's those other changes that I'm talking about, and it's
those other changes I do not want without knowing what they are.

It may be that those other changes have since been merged into
v4.10-rc6 - but github's web interface can't show me that.  In fact,
github's web interface is pretty damned useless as far as this stuff
goes.

So, what I'll get if I clone or pull your imx-media-staging-md-wip
branch is, yes, a copy of all your changes, but _also_ all the
changes that are in the media tree that _aren't_ in mainline at the
point that v4.10-rc1 was merged.

> You don't need to use the web interface, just git clone the repo.

You're assuming I want to work off the top of your commits.  I don't.
I've got other dependencies.

Then there's yet another problem - lets say that I get a copy of your
patches that haven't been on the mailing list, and I then want to make
a comment about it.  I can't reply to a patch that hasn't been on the
mailing list.  So, the long established mechanism by which the Linux
community does patch review breaks down.

So no, sorry, I'm not fetching your tree, and I will persist with your
v3 patch set for the time being.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
  2017-01-31 23:30                   ` Russell King - ARM Linux
  (?)
@ 2017-01-31 23:41                     ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 23:41 UTC (permalink / raw)
  To: Russell King - ARM Linux, Steve Longerbeam
  Cc: Ian Arkver, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel



On 01/31/2017 03:30 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 02:36:53PM -0800, Steve Longerbeam wrote:
>> On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
>>> I don't want master though, I want v4.10-rc1, and if I ask for that
>>> it tells me it knows nothing about v4.10-rc1, despite the fact that's
>>> a tag in the mainline kernel repository which was merged into the
>>> linux-media tree that this tree is based upon.
>> Hi Russell, yes git@github.com:slongerbeam/mediatree.git is a fork
>> of the linux-media tree, and the imx-media-staging-md-wip branch
>> is up-to-date with master, currently at 4.10-rc1.
> "up to date" is different from "contains other stuff other than is in
> 4.10-rc1".
>
> What I see in your tree is that your code is based off a merge commit
> between something called "patchwork" (which I assume is a branch in
> the media tree containing stuff commited from patch work) and v4.10-rc1.
>
> Now, you don't get a commit when merging unless there's changes that
> aren't in the commit you're merging - if "patchwork" was up to date
> with v4.10-rc1, then git would have done a "fast forward" to v4.10-rc1.
>
> Therefore, while it may be "up to date" with v4.10-rc1 in so far that
> it's had v4.10-rc1 merged into it, that's not what I've been saying.
> There are other changes below that merge commit which aren't in
> v4.10-rc1.  It's those other changes that I'm talking about, and it's
> those other changes I do not want without knowing what they are.
>
> It may be that those other changes have since been merged into
> v4.10-rc6 - but github's web interface can't show me that.  In fact,
> github's web interface is pretty damned useless as far as this stuff
> goes.
>
> So, what I'll get if I clone or pull your imx-media-staging-md-wip
> branch is, yes, a copy of all your changes, but _also_ all the
> changes that are in the media tree that _aren't_ in mainline at the
> point that v4.10-rc1 was merged.
>
>> You don't need to use the web interface, just git clone the repo.
> You're assuming I want to work off the top of your commits.  I don't.
> I've got other dependencies.

Well, I was suggesting cloning it just to have a look at the new
code, but I understand you don't want to attempt to bring in your
SMIA/bayer format changes and run from this branch, due to the
other changes in the mediatree.

I suppose I should post the next version then.

Trouble is, I see issues in the current driver that prevents working
with your SMIA pipeline. But I guess that will have to be worked out
in another version.

Steve

>
> Then there's yet another problem - lets say that I get a copy of your
> patches that haven't been on the mailing list, and I then want to make
> a comment about it.  I can't reply to a patch that hasn't been on the
> mailing list.  So, the long established mechanism by which the Linux
> community does patch review breaks down.
>
> So no, sorry, I'm not fetching your tree, and I will persist with your
> v3 patch set for the time being.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 23:41                     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 23:41 UTC (permalink / raw)
  To: Russell King - ARM Linux, Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, Ian Arkver,
	songjun.wu, Hans Verkuil, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/31/2017 03:30 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 02:36:53PM -0800, Steve Longerbeam wrote:
>> On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
>>> I don't want master though, I want v4.10-rc1, and if I ask for that
>>> it tells me it knows nothing about v4.10-rc1, despite the fact that's
>>> a tag in the mainline kernel repository which was merged into the
>>> linux-media tree that this tree is based upon.
>> Hi Russell, yes git@github.com:slongerbeam/mediatree.git is a fork
>> of the linux-media tree, and the imx-media-staging-md-wip branch
>> is up-to-date with master, currently at 4.10-rc1.
> "up to date" is different from "contains other stuff other than is in
> 4.10-rc1".
>
> What I see in your tree is that your code is based off a merge commit
> between something called "patchwork" (which I assume is a branch in
> the media tree containing stuff commited from patch work) and v4.10-rc1.
>
> Now, you don't get a commit when merging unless there's changes that
> aren't in the commit you're merging - if "patchwork" was up to date
> with v4.10-rc1, then git would have done a "fast forward" to v4.10-rc1.
>
> Therefore, while it may be "up to date" with v4.10-rc1 in so far that
> it's had v4.10-rc1 merged into it, that's not what I've been saying.
> There are other changes below that merge commit which aren't in
> v4.10-rc1.  It's those other changes that I'm talking about, and it's
> those other changes I do not want without knowing what they are.
>
> It may be that those other changes have since been merged into
> v4.10-rc6 - but github's web interface can't show me that.  In fact,
> github's web interface is pretty damned useless as far as this stuff
> goes.
>
> So, what I'll get if I clone or pull your imx-media-staging-md-wip
> branch is, yes, a copy of all your changes, but _also_ all the
> changes that are in the media tree that _aren't_ in mainline at the
> point that v4.10-rc1 was merged.
>
>> You don't need to use the web interface, just git clone the repo.
> You're assuming I want to work off the top of your commits.  I don't.
> I've got other dependencies.

Well, I was suggesting cloning it just to have a look at the new
code, but I understand you don't want to attempt to bring in your
SMIA/bayer format changes and run from this branch, due to the
other changes in the mediatree.

I suppose I should post the next version then.

Trouble is, I see issues in the current driver that prevents working
with your SMIA pipeline. But I guess that will have to be worked out
in another version.

Steve

>
> Then there's yet another problem - lets say that I get a copy of your
> patches that haven't been on the mailing list, and I then want to make
> a comment about it.  I can't reply to a patch that hasn't been on the
> mailing list.  So, the long established mechanism by which the Linux
> community does patch review breaks down.
>
> So no, sorry, I'm not fetching your tree, and I will persist with your
> v3 patch set for the time being.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-01-31 23:41                     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 23:41 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 03:30 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 02:36:53PM -0800, Steve Longerbeam wrote:
>> On 01/31/2017 02:04 PM, Russell King - ARM Linux wrote:
>>> I don't want master though, I want v4.10-rc1, and if I ask for that
>>> it tells me it knows nothing about v4.10-rc1, despite the fact that's
>>> a tag in the mainline kernel repository which was merged into the
>>> linux-media tree that this tree is based upon.
>> Hi Russell, yes git at github.com:slongerbeam/mediatree.git is a fork
>> of the linux-media tree, and the imx-media-staging-md-wip branch
>> is up-to-date with master, currently at 4.10-rc1.
> "up to date" is different from "contains other stuff other than is in
> 4.10-rc1".
>
> What I see in your tree is that your code is based off a merge commit
> between something called "patchwork" (which I assume is a branch in
> the media tree containing stuff commited from patch work) and v4.10-rc1.
>
> Now, you don't get a commit when merging unless there's changes that
> aren't in the commit you're merging - if "patchwork" was up to date
> with v4.10-rc1, then git would have done a "fast forward" to v4.10-rc1.
>
> Therefore, while it may be "up to date" with v4.10-rc1 in so far that
> it's had v4.10-rc1 merged into it, that's not what I've been saying.
> There are other changes below that merge commit which aren't in
> v4.10-rc1.  It's those other changes that I'm talking about, and it's
> those other changes I do not want without knowing what they are.
>
> It may be that those other changes have since been merged into
> v4.10-rc6 - but github's web interface can't show me that.  In fact,
> github's web interface is pretty damned useless as far as this stuff
> goes.
>
> So, what I'll get if I clone or pull your imx-media-staging-md-wip
> branch is, yes, a copy of all your changes, but _also_ all the
> changes that are in the media tree that _aren't_ in mainline at the
> point that v4.10-rc1 was merged.
>
>> You don't need to use the web interface, just git clone the repo.
> You're assuming I want to work off the top of your commits.  I don't.
> I've got other dependencies.

Well, I was suggesting cloning it just to have a look at the new
code, but I understand you don't want to attempt to bring in your
SMIA/bayer format changes and run from this branch, due to the
other changes in the mediatree.

I suppose I should post the next version then.

Trouble is, I see issues in the current driver that prevents working
with your SMIA pipeline. But I guess that will have to be worked out
in another version.

Steve

>
> Then there's yet another problem - lets say that I get a copy of your
> patches that haven't been on the mailing list, and I then want to make
> a comment about it.  I can't reply to a patch that hasn't been on the
> mailing list.  So, the long established mechanism by which the Linux
> community does patch review breaks down.
>
> So no, sorry, I'm not fetching your tree, and I will persist with your
> v3 patch set for the time being.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 23:43         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 23:43 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
> On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
>> I'm also having trouble finding a datasheet for it, but from what
>> I've read, it has a MIPI CSI-2 interface. It should work fine as long
>> as it presents a single source pad, registers asynchronously, and
>> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.
> Yes, it is MIPI CSI-2, and yes it has a single source pad, registers
> asynchronously, but that's about as far as it goes.
>
> The structure is a camera sensor followed by some processing.  So just
> like the smiapp code, I've ended up with multiple subdevs describing
> each stage of the sensors pipeline.
>
> Just like smiapp, the camera sensor block (which is the very far end
> of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
> front of that is the binner, which just like smiapp gets a separate
> entity.  It's this entity which is connected to the mipi-csi2 subdev.

wow, ok got it.

So the sensor pipeline and binner, and the OF graph connecting
them, are described in the device tree I presume.

The OF graph AFAIK, has no information about which ports are sinks
and which are sources, so of_parse_subdev() tries to determine that
based on the compatible string of the device node. So ATM
of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
video-multiplexer, and camera sensors upstream from the CSI ports
in the OF graph.

I realize that's not a robust solution, and is the reason for the
"no sensor attached" below.

Is there any way to determine from the OF graph the data-direction
of a port (whether it is a sink or a source)? If so it will make
of_parse_subdev() much more robust.

Steve

>
> Unlike smiapp, which does not set an entity function, I set my binner
> entity as MEDIA_ENT_F_PROC_VIDEO_SCALER on the basis that that is
> what V4L2 documentation recommend:
>
>      -  ..  row 27
>
>         ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:
>
>         -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``
>
>         -  Video scaler. An entity capable of video scaling must have
>            at least one sink pad and one source pad, and scale the
>            video frame(s) received on its sink pad(s) to a different
>            resolution output on its source pad(s). The range of
>            supported scaling ratios is entity-specific and can differ
>            between the horizontal and vertical directions (in particular
>            scaling can be supported in one direction only). Binning and
>            skipping are considered as scaling.
>
> This causes attempts to configure the ipu1_csi0 interface to fail:
>
> media-ctl -v -d /dev/media1 --set-v4l2 '"ipu1_csi0":1[fmt:SGBRG8/512x512@1/30]'
> Opening media device /dev/media1
> Enumerating entities
> Found 29 entities
> Enumerating pads and links
> Setting up format SGBRG8 512x512 on pad ipu1_csi0/1
> Unable to set format: No such device (-19)
> Unable to setup formats: No such device (19)
>
> and in the kernel log:
>
> ipu1_csi0: no sensor attached
>
> And yes, I already know that my next problem is going to be that the bayer
> formats are not supported in your driver (just like Philipp's driver) but
> adding them should not be difficult... but only once this issue is resolved.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 23:43         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 23:43 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland-5wv7dgnIgG8, andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	nick-gcszYUEDH4VrovVCs/uTlw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, Steve Longerbeam,
	robert.jarzmik-GANU6spQydw,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	geert-Td1EMuHUCqxL1ZNQvxDV9g, linux-media-u79uwXL29TY76Z2rM5mHXA,
	devicetree-u79uwXL29TY76Z2rM5mHXA, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	arnd-r2nGTMty4D4, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	bparrot-l0cyMroinI0, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	jean-christophe.trotin-qxv4g6HH51o,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, sudipm.m



On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
> On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
>> I'm also having trouble finding a datasheet for it, but from what
>> I've read, it has a MIPI CSI-2 interface. It should work fine as long
>> as it presents a single source pad, registers asynchronously, and
>> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.
> Yes, it is MIPI CSI-2, and yes it has a single source pad, registers
> asynchronously, but that's about as far as it goes.
>
> The structure is a camera sensor followed by some processing.  So just
> like the smiapp code, I've ended up with multiple subdevs describing
> each stage of the sensors pipeline.
>
> Just like smiapp, the camera sensor block (which is the very far end
> of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
> front of that is the binner, which just like smiapp gets a separate
> entity.  It's this entity which is connected to the mipi-csi2 subdev.

wow, ok got it.

So the sensor pipeline and binner, and the OF graph connecting
them, are described in the device tree I presume.

The OF graph AFAIK, has no information about which ports are sinks
and which are sources, so of_parse_subdev() tries to determine that
based on the compatible string of the device node. So ATM
of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
video-multiplexer, and camera sensors upstream from the CSI ports
in the OF graph.

I realize that's not a robust solution, and is the reason for the
"no sensor attached" below.

Is there any way to determine from the OF graph the data-direction
of a port (whether it is a sink or a source)? If so it will make
of_parse_subdev() much more robust.

Steve

>
> Unlike smiapp, which does not set an entity function, I set my binner
> entity as MEDIA_ENT_F_PROC_VIDEO_SCALER on the basis that that is
> what V4L2 documentation recommend:
>
>      -  ..  row 27
>
>         ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:
>
>         -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``
>
>         -  Video scaler. An entity capable of video scaling must have
>            at least one sink pad and one source pad, and scale the
>            video frame(s) received on its sink pad(s) to a different
>            resolution output on its source pad(s). The range of
>            supported scaling ratios is entity-specific and can differ
>            between the horizontal and vertical directions (in particular
>            scaling can be supported in one direction only). Binning and
>            skipping are considered as scaling.
>
> This causes attempts to configure the ipu1_csi0 interface to fail:
>
> media-ctl -v -d /dev/media1 --set-v4l2 '"ipu1_csi0":1[fmt:SGBRG8/512x512@1/30]'
> Opening media device /dev/media1
> Enumerating entities
> Found 29 entities
> Enumerating pads and links
> Setting up format SGBRG8 512x512 on pad ipu1_csi0/1
> Unable to set format: No such device (-19)
> Unable to setup formats: No such device (19)
>
> and in the kernel log:
>
> ipu1_csi0: no sensor attached
>
> And yes, I already know that my next problem is going to be that the bayer
> formats are not supported in your driver (just like Philipp's driver) but
> adding them should not be difficult... but only once this issue is resolved.
>

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-01-31 23:43         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-01-31 23:43 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
> On Mon, Jan 30, 2017 at 05:22:01PM -0800, Steve Longerbeam wrote:
>> I'm also having trouble finding a datasheet for it, but from what
>> I've read, it has a MIPI CSI-2 interface. It should work fine as long
>> as it presents a single source pad, registers asynchronously, and
>> sets its entity function to MEDIA_ENT_F_CAM_SENSOR.
> Yes, it is MIPI CSI-2, and yes it has a single source pad, registers
> asynchronously, but that's about as far as it goes.
>
> The structure is a camera sensor followed by some processing.  So just
> like the smiapp code, I've ended up with multiple subdevs describing
> each stage of the sensors pipeline.
>
> Just like smiapp, the camera sensor block (which is the very far end
> of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
> front of that is the binner, which just like smiapp gets a separate
> entity.  It's this entity which is connected to the mipi-csi2 subdev.

wow, ok got it.

So the sensor pipeline and binner, and the OF graph connecting
them, are described in the device tree I presume.

The OF graph AFAIK, has no information about which ports are sinks
and which are sources, so of_parse_subdev() tries to determine that
based on the compatible string of the device node. So ATM
of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
video-multiplexer, and camera sensors upstream from the CSI ports
in the OF graph.

I realize that's not a robust solution, and is the reason for the
"no sensor attached" below.

Is there any way to determine from the OF graph the data-direction
of a port (whether it is a sink or a source)? If so it will make
of_parse_subdev() much more robust.

Steve

>
> Unlike smiapp, which does not set an entity function, I set my binner
> entity as MEDIA_ENT_F_PROC_VIDEO_SCALER on the basis that that is
> what V4L2 documentation recommend:
>
>      -  ..  row 27
>
>         ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:
>
>         -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``
>
>         -  Video scaler. An entity capable of video scaling must have
>            at least one sink pad and one source pad, and scale the
>            video frame(s) received on its sink pad(s) to a different
>            resolution output on its source pad(s). The range of
>            supported scaling ratios is entity-specific and can differ
>            between the horizontal and vertical directions (in particular
>            scaling can be supported in one direction only). Binning and
>            skipping are considered as scaling.
>
> This causes attempts to configure the ipu1_csi0 interface to fail:
>
> media-ctl -v -d /dev/media1 --set-v4l2 '"ipu1_csi0":1[fmt:SGBRG8/512x512 at 1/30]'
> Opening media device /dev/media1
> Enumerating entities
> Found 29 entities
> Enumerating pads and links
> Setting up format SGBRG8 512x512 on pad ipu1_csi0/1
> Unable to set format: No such device (-19)
> Unable to setup formats: No such device (19)
>
> and in the kernel log:
>
> ipu1_csi0: no sensor attached
>
> And yes, I already know that my next problem is going to be that the bayer
> formats are not supported in your driver (just like Philipp's driver) but
> adding them should not be difficult... but only once this issue is resolved.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-01-31 23:43         ` Steve Longerbeam
  (?)
@ 2017-02-01  0:23           ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  0:23 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 03:43:22PM -0800, Steve Longerbeam wrote:
> 
> 
> On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
> >Just like smiapp, the camera sensor block (which is the very far end
> >of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
> >front of that is the binner, which just like smiapp gets a separate
> >entity.  It's this entity which is connected to the mipi-csi2 subdev.
> 
> wow, ok got it.
> 
> So the sensor pipeline and binner, and the OF graph connecting
> them, are described in the device tree I presume.

No - because the binner and sensor are on the same die, it's even
one single device, there's no real separation of the two devices.

The reason there's no real separation is because the binning is done
as part of the process of reading the array - sometimes before the
analog voltage is converted to its digital pixel value representation.

The separation comes because of the requirements of the v4l2 media
subdevs, which requires scalers to have a sink pad and a source pad.
(Please see the v4l2 documentation, I think I've already quoted this:

       ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:

       -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``

       -  Video scaler. An entity capable of video scaling must have
          at least one sink pad and one source pad, and scale the
          video frame(s) received on its sink pad(s) to a different
          resolution output on its source pad(s). The range of
          supported scaling ratios is entity-specific and can differ
          between the horizontal and vertical directions (in particular
          scaling can be supported in one direction only). Binning and
          skipping are considered as scaling.

(Oh yes, I see it was the mail to which you were replying to...)

So, in order to configure the scaling (which can be none, /2 and /4)
we have to expose two subdevs - one which is the scaler, and has a
source pad connected to the upstream (in this case CSI2 receiver)
and a sink pad immutably connected to the camera sensor.

Since the split is entirely down to the V4L2 implementation requirements,
it's not something that should be reflected in DT - we don't put
implementation details in DT.

It's just the same (as I've already said) as the SMIAPP camera driver,
the reason I'm pointing you towards that is because this is an already
mainlined camera driver which nicely illustrates how my driver is
structured from the v4l2 subdev API point of view.

> The OF graph AFAIK, has no information about which ports are sinks
> and which are sources, so of_parse_subdev() tries to determine that
> based on the compatible string of the device node. So ATM
> of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
> video-multiplexer, and camera sensors upstream from the CSI ports
> in the OF graph.
> 
> I realize that's not a robust solution, and is the reason for the
> "no sensor attached" below.
> 
> Is there any way to determine from the OF graph the data-direction
> of a port (whether it is a sink or a source)? If so it will make
> of_parse_subdev() much more robust.

I'm not sure why you need to know the data direction.  I think the
issue here is how you're going about dealing with the subdevs.
Here's the subdev setup:

+---------camera--------+
| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
+-----------------------+

How the subdevs are supposed to work is that you start from the first
subdev in sequence (in this case the pixel array) and negotiate with
the driver through the TRY formats on its source pad, as well as
negotiating with the sink pad of the directly neighbouring subdev.

The neighbouring subdev propagates the configuration in a driver
specific manner from its source pad to the sink pad, giving a default
configuration at its source.

This process repeats throughout the chain all the way up to the bridge
device.

Now, where things go wrong is that you want to know what each type of
these subdevs are, and the reason you want that is so you can do this
(for example - I know similar stuff happens with the "sensor" stuff
further up the chain as well):

+---------camera--------+
| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
+-----------------------+                                |
              ^--I-want-your-bus-format-and-frame-rate---'

which goes against the negotiation mechanism of v4l2 subdevs.  This
is broken - it bypass the subdev negotiation that has been performed
on the intervening subdevs between the "sensor" and the csi0 subdevs,
so if there were a subdev in that chain that (eg) reduced the frame
rate by discarding the odd frames, you'd be working with the wrong
frame interval for your frame interval monitor at csi.

Note that the "MEDIA_ENT_F_PROC_VIDEO_SCALER" subdev type is documented
as not only supports scaling as in changing the size of the image, but
also in terms of skipping frames, which means a reduction in frame rate.

So, for your FIM, you need to know if there is any reduction in frame
rate through that pipeline, and looking for a "MEDIA_ENT_F_CAM_SENSOR"
subdev node isn't going to tell you that.  The frame rate really needs
to be carried through and I suspect you need to accept the frame rate
into each subdev so it can be passed along the chain by the application
configuring the pipeline.

The last bit from the above is the "I-want-your-bus-format" bit which
I haven't fully worked out how to eliminate - I understand the reason
you need that (so you can appropriately configure the CSI with the CSI2
data type code.)  The CSI2 data type code comes from the format
configured on the CSI sink pad, so the only information you're really
using there is "are we sinking data from a CSI2 interface."

You _could_ walk down the graph from the CSI looking for a subdev that
responds to g_mbus_config that reports CSI2, but I'm not sure that's
going to last - I've seen an email from Hans saying that he'd like
g_mbus_config to go away (to patch 13/24, for the vidsw_g_mbus_config()
function):

"I am not certain this op is needed at all. In the current kernel this
 op is onlyused by soc_camera, pxa_camera and omap3isp (somewhat dubious).
 Normally this information should come from the device tree and there
 should be no need for this op.

 My (tentative) long-term plan was to get rid of this op.

 If you don't need it, then I recommend it is removed."

So, if that goes away, the CSI subdev needs a completely different way
to get that information, and it shouldn't be coming from the camera
sensor subdev, but (imho) really from the CSI2 subdev.

This is probably something that needs to be discussed with media people
to work out how to replace the g_mbus_config call with something more
acceptable to resolve this issue.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  0:23           ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  0:23 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 03:43:22PM -0800, Steve Longerbeam wrote:
> 
> 
> On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
> >Just like smiapp, the camera sensor block (which is the very far end
> >of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
> >front of that is the binner, which just like smiapp gets a separate
> >entity.  It's this entity which is connected to the mipi-csi2 subdev.
> 
> wow, ok got it.
> 
> So the sensor pipeline and binner, and the OF graph connecting
> them, are described in the device tree I presume.

No - because the binner and sensor are on the same die, it's even
one single device, there's no real separation of the two devices.

The reason there's no real separation is because the binning is done
as part of the process of reading the array - sometimes before the
analog voltage is converted to its digital pixel value representation.

The separation comes because of the requirements of the v4l2 media
subdevs, which requires scalers to have a sink pad and a source pad.
(Please see the v4l2 documentation, I think I've already quoted this:

       ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:

       -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``

       -  Video scaler. An entity capable of video scaling must have
          at least one sink pad and one source pad, and scale the
          video frame(s) received on its sink pad(s) to a different
          resolution output on its source pad(s). The range of
          supported scaling ratios is entity-specific and can differ
          between the horizontal and vertical directions (in particular
          scaling can be supported in one direction only). Binning and
          skipping are considered as scaling.

(Oh yes, I see it was the mail to which you were replying to...)

So, in order to configure the scaling (which can be none, /2 and /4)
we have to expose two subdevs - one which is the scaler, and has a
source pad connected to the upstream (in this case CSI2 receiver)
and a sink pad immutably connected to the camera sensor.

Since the split is entirely down to the V4L2 implementation requirements,
it's not something that should be reflected in DT - we don't put
implementation details in DT.

It's just the same (as I've already said) as the SMIAPP camera driver,
the reason I'm pointing you towards that is because this is an already
mainlined camera driver which nicely illustrates how my driver is
structured from the v4l2 subdev API point of view.

> The OF graph AFAIK, has no information about which ports are sinks
> and which are sources, so of_parse_subdev() tries to determine that
> based on the compatible string of the device node. So ATM
> of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
> video-multiplexer, and camera sensors upstream from the CSI ports
> in the OF graph.
> 
> I realize that's not a robust solution, and is the reason for the
> "no sensor attached" below.
> 
> Is there any way to determine from the OF graph the data-direction
> of a port (whether it is a sink or a source)? If so it will make
> of_parse_subdev() much more robust.

I'm not sure why you need to know the data direction.  I think the
issue here is how you're going about dealing with the subdevs.
Here's the subdev setup:

+---------camera--------+
| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
+-----------------------+

How the subdevs are supposed to work is that you start from the first
subdev in sequence (in this case the pixel array) and negotiate with
the driver through the TRY formats on its source pad, as well as
negotiating with the sink pad of the directly neighbouring subdev.

The neighbouring subdev propagates the configuration in a driver
specific manner from its source pad to the sink pad, giving a default
configuration at its source.

This process repeats throughout the chain all the way up to the bridge
device.

Now, where things go wrong is that you want to know what each type of
these subdevs are, and the reason you want that is so you can do this
(for example - I know similar stuff happens with the "sensor" stuff
further up the chain as well):

+---------camera--------+
| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
+-----------------------+                                |
              ^--I-want-your-bus-format-and-frame-rate---'

which goes against the negotiation mechanism of v4l2 subdevs.  This
is broken - it bypass the subdev negotiation that has been performed
on the intervening subdevs between the "sensor" and the csi0 subdevs,
so if there were a subdev in that chain that (eg) reduced the frame
rate by discarding the odd frames, you'd be working with the wrong
frame interval for your frame interval monitor at csi.

Note that the "MEDIA_ENT_F_PROC_VIDEO_SCALER" subdev type is documented
as not only supports scaling as in changing the size of the image, but
also in terms of skipping frames, which means a reduction in frame rate.

So, for your FIM, you need to know if there is any reduction in frame
rate through that pipeline, and looking for a "MEDIA_ENT_F_CAM_SENSOR"
subdev node isn't going to tell you that.  The frame rate really needs
to be carried through and I suspect you need to accept the frame rate
into each subdev so it can be passed along the chain by the application
configuring the pipeline.

The last bit from the above is the "I-want-your-bus-format" bit which
I haven't fully worked out how to eliminate - I understand the reason
you need that (so you can appropriately configure the CSI with the CSI2
data type code.)  The CSI2 data type code comes from the format
configured on the CSI sink pad, so the only information you're really
using there is "are we sinking data from a CSI2 interface."

You _could_ walk down the graph from the CSI looking for a subdev that
responds to g_mbus_config that reports CSI2, but I'm not sure that's
going to last - I've seen an email from Hans saying that he'd like
g_mbus_config to go away (to patch 13/24, for the vidsw_g_mbus_config()
function):

"I am not certain this op is needed at all. In the current kernel this
 op is onlyused by soc_camera, pxa_camera and omap3isp (somewhat dubious).
 Normally this information should come from the device tree and there
 should be no need for this op.

 My (tentative) long-term plan was to get rid of this op.

 If you don't need it, then I recommend it is removed."

So, if that goes away, the CSI subdev needs a completely different way
to get that information, and it shouldn't be coming from the camera
sensor subdev, but (imho) really from the CSI2 subdev.

This is probably something that needs to be discussed with media people
to work out how to replace the g_mbus_config call with something more
acceptable to resolve this issue.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  0:23           ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  0:23 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 03:43:22PM -0800, Steve Longerbeam wrote:
> 
> 
> On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
> >Just like smiapp, the camera sensor block (which is the very far end
> >of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
> >front of that is the binner, which just like smiapp gets a separate
> >entity.  It's this entity which is connected to the mipi-csi2 subdev.
> 
> wow, ok got it.
> 
> So the sensor pipeline and binner, and the OF graph connecting
> them, are described in the device tree I presume.

No - because the binner and sensor are on the same die, it's even
one single device, there's no real separation of the two devices.

The reason there's no real separation is because the binning is done
as part of the process of reading the array - sometimes before the
analog voltage is converted to its digital pixel value representation.

The separation comes because of the requirements of the v4l2 media
subdevs, which requires scalers to have a sink pad and a source pad.
(Please see the v4l2 documentation, I think I've already quoted this:

       ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:

       -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``

       -  Video scaler. An entity capable of video scaling must have
          at least one sink pad and one source pad, and scale the
          video frame(s) received on its sink pad(s) to a different
          resolution output on its source pad(s). The range of
          supported scaling ratios is entity-specific and can differ
          between the horizontal and vertical directions (in particular
          scaling can be supported in one direction only). Binning and
          skipping are considered as scaling.

(Oh yes, I see it was the mail to which you were replying to...)

So, in order to configure the scaling (which can be none, /2 and /4)
we have to expose two subdevs - one which is the scaler, and has a
source pad connected to the upstream (in this case CSI2 receiver)
and a sink pad immutably connected to the camera sensor.

Since the split is entirely down to the V4L2 implementation requirements,
it's not something that should be reflected in DT - we don't put
implementation details in DT.

It's just the same (as I've already said) as the SMIAPP camera driver,
the reason I'm pointing you towards that is because this is an already
mainlined camera driver which nicely illustrates how my driver is
structured from the v4l2 subdev API point of view.

> The OF graph AFAIK, has no information about which ports are sinks
> and which are sources, so of_parse_subdev() tries to determine that
> based on the compatible string of the device node. So ATM
> of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
> video-multiplexer, and camera sensors upstream from the CSI ports
> in the OF graph.
> 
> I realize that's not a robust solution, and is the reason for the
> "no sensor attached" below.
> 
> Is there any way to determine from the OF graph the data-direction
> of a port (whether it is a sink or a source)? If so it will make
> of_parse_subdev() much more robust.

I'm not sure why you need to know the data direction.  I think the
issue here is how you're going about dealing with the subdevs.
Here's the subdev setup:

+---------camera--------+
| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
+-----------------------+

How the subdevs are supposed to work is that you start from the first
subdev in sequence (in this case the pixel array) and negotiate with
the driver through the TRY formats on its source pad, as well as
negotiating with the sink pad of the directly neighbouring subdev.

The neighbouring subdev propagates the configuration in a driver
specific manner from its source pad to the sink pad, giving a default
configuration at its source.

This process repeats throughout the chain all the way up to the bridge
device.

Now, where things go wrong is that you want to know what each type of
these subdevs are, and the reason you want that is so you can do this
(for example - I know similar stuff happens with the "sensor" stuff
further up the chain as well):

+---------camera--------+
| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
+-----------------------+                                |
              ^--I-want-your-bus-format-and-frame-rate---'

which goes against the negotiation mechanism of v4l2 subdevs.  This
is broken - it bypass the subdev negotiation that has been performed
on the intervening subdevs between the "sensor" and the csi0 subdevs,
so if there were a subdev in that chain that (eg) reduced the frame
rate by discarding the odd frames, you'd be working with the wrong
frame interval for your frame interval monitor at csi.

Note that the "MEDIA_ENT_F_PROC_VIDEO_SCALER" subdev type is documented
as not only supports scaling as in changing the size of the image, but
also in terms of skipping frames, which means a reduction in frame rate.

So, for your FIM, you need to know if there is any reduction in frame
rate through that pipeline, and looking for a "MEDIA_ENT_F_CAM_SENSOR"
subdev node isn't going to tell you that.  The frame rate really needs
to be carried through and I suspect you need to accept the frame rate
into each subdev so it can be passed along the chain by the application
configuring the pipeline.

The last bit from the above is the "I-want-your-bus-format" bit which
I haven't fully worked out how to eliminate - I understand the reason
you need that (so you can appropriately configure the CSI with the CSI2
data type code.)  The CSI2 data type code comes from the format
configured on the CSI sink pad, so the only information you're really
using there is "are we sinking data from a CSI2 interface."

You _could_ walk down the graph from the CSI looking for a subdev that
responds to g_mbus_config that reports CSI2, but I'm not sure that's
going to last - I've seen an email from Hans saying that he'd like
g_mbus_config to go away (to patch 13/24, for the vidsw_g_mbus_config()
function):

"I am not certain this op is needed at all. In the current kernel this
 op is onlyused by soc_camera, pxa_camera and omap3isp (somewhat dubious).
 Normally this information should come from the device tree and there
 should be no need for this op.

 My (tentative) long-term plan was to get rid of this op.

 If you don't need it, then I recommend it is removed."

So, if that goes away, the CSI subdev needs a completely different way
to get that information, and it shouldn't be coming from the camera
sensor subdev, but (imho) really from the CSI2 subdev.

This is probably something that needs to be discussed with media people
to work out how to replace the g_mbus_config call with something more
acceptable to resolve this issue.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
  2017-01-31 11:20     ` Russell King - ARM Linux
  (?)
@ 2017-02-01  0:31       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  0:31 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/31/2017 03:20 AM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
>> +static int csi_link_validate(struct v4l2_subdev *sd,
>> +			     struct media_link *link,
>> +			     struct v4l2_subdev_format *source_fmt,
>> +			     struct v4l2_subdev_format *sink_fmt)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	bool is_csi2;
>> +	int ret;
>> +
>> +	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
>> +	if (IS_ERR(priv->sensor)) {
>> +		v4l2_err(&priv->sd, "no sensor attached\n");
>> +		ret = PTR_ERR(priv->sensor);
>> +		priv->sensor = NULL;
>> +		return ret;
>> +	}
>> +
>> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
>> +			       &priv->sensor_mbus_cfg);
>> +	if (ret)
>> +		return ret;
>> +
>> +	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
>> +
>> +	if (is_csi2) {
>> +		int vc_num = 0;
>> +		/*
>> +		 * NOTE! It seems the virtual channels from the mipi csi-2
>> +		 * receiver are used only for routing by the video mux's,
>> +		 * or for hard-wired routing to the CSI's. Once the stream
>> +		 * enters the CSI's however, they are treated internally
>> +		 * in the IPU as virtual channel 0.
>> +		 */
>> +#if 0
>> +		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
>> +							  &priv->sd.entity);
>> +		if (vc_num < 0)
>> +			return vc_num;
>> +#endif
>> +		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
>> +					  &priv->format_mbus[priv->input_pad]);
> This seems (at least to me) to go against the spirit of the subdev
> negotiation.  Here, you seem to bypass the negotiation performed
> between the CSI and the attached device, wanting to grab the
> format from the CSI2 sensor directly.  That bypasses negotiation
> performed at the CSI2 subdev and video muxes.

This isn't so much grabbing a pad format, it is determining
which source pad at the imx6-mipi-csi2 receiver subdev is
reached from this CSI (which determines the virtual channel
the CSI is receiving).

If there was a way to determine the vc# via a status register
in the CSI, that would be perfect, but there isn't. In fact, the CSIs
seem to be ignoring the meta-data bus sent by the CSI2IPU gasket
which contains this info, or that bus is not being routed to the CSIs.
As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register.
Any other value programmed there results in no data from the CSI.

And even the presence of CSI_MIPI_DI register makes no sense to me,
the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data
bus, so I don't understand why it needs to be told.

Anyway I can just yank this disabled code, it's only there for documentation
purposes.


>
> The same goes for the frame rate monitoring code - imho, the frame
> rate should be something that results from negotiation with the
> neighbouring element, not by poking at some entity that is several
> entities away.

I will fix this once the g_frame_interval op is implemented in every
subdev. I believe you mentioned this already, but g_frame_interval
will need to be chained until it reaches the originating sensor.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-02-01  0:31       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  0:31 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/31/2017 03:20 AM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
>> +static int csi_link_validate(struct v4l2_subdev *sd,
>> +			     struct media_link *link,
>> +			     struct v4l2_subdev_format *source_fmt,
>> +			     struct v4l2_subdev_format *sink_fmt)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	bool is_csi2;
>> +	int ret;
>> +
>> +	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
>> +	if (IS_ERR(priv->sensor)) {
>> +		v4l2_err(&priv->sd, "no sensor attached\n");
>> +		ret = PTR_ERR(priv->sensor);
>> +		priv->sensor = NULL;
>> +		return ret;
>> +	}
>> +
>> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
>> +			       &priv->sensor_mbus_cfg);
>> +	if (ret)
>> +		return ret;
>> +
>> +	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
>> +
>> +	if (is_csi2) {
>> +		int vc_num = 0;
>> +		/*
>> +		 * NOTE! It seems the virtual channels from the mipi csi-2
>> +		 * receiver are used only for routing by the video mux's,
>> +		 * or for hard-wired routing to the CSI's. Once the stream
>> +		 * enters the CSI's however, they are treated internally
>> +		 * in the IPU as virtual channel 0.
>> +		 */
>> +#if 0
>> +		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
>> +							  &priv->sd.entity);
>> +		if (vc_num < 0)
>> +			return vc_num;
>> +#endif
>> +		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
>> +					  &priv->format_mbus[priv->input_pad]);
> This seems (at least to me) to go against the spirit of the subdev
> negotiation.  Here, you seem to bypass the negotiation performed
> between the CSI and the attached device, wanting to grab the
> format from the CSI2 sensor directly.  That bypasses negotiation
> performed at the CSI2 subdev and video muxes.

This isn't so much grabbing a pad format, it is determining
which source pad at the imx6-mipi-csi2 receiver subdev is
reached from this CSI (which determines the virtual channel
the CSI is receiving).

If there was a way to determine the vc# via a status register
in the CSI, that would be perfect, but there isn't. In fact, the CSIs
seem to be ignoring the meta-data bus sent by the CSI2IPU gasket
which contains this info, or that bus is not being routed to the CSIs.
As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register.
Any other value programmed there results in no data from the CSI.

And even the presence of CSI_MIPI_DI register makes no sense to me,
the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data
bus, so I don't understand why it needs to be told.

Anyway I can just yank this disabled code, it's only there for documentation
purposes.


>
> The same goes for the frame rate monitoring code - imho, the frame
> rate should be something that results from negotiation with the
> neighbouring element, not by poking at some entity that is several
> entities away.

I will fix this once the g_frame_interval op is implemented in every
subdev. I believe you mentioned this already, but g_frame_interval
will need to be chained until it reaches the originating sensor.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-02-01  0:31       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  0:31 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 03:20 AM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
>> +static int csi_link_validate(struct v4l2_subdev *sd,
>> +			     struct media_link *link,
>> +			     struct v4l2_subdev_format *source_fmt,
>> +			     struct v4l2_subdev_format *sink_fmt)
>> +{
>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> +	bool is_csi2;
>> +	int ret;
>> +
>> +	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
>> +	if (IS_ERR(priv->sensor)) {
>> +		v4l2_err(&priv->sd, "no sensor attached\n");
>> +		ret = PTR_ERR(priv->sensor);
>> +		priv->sensor = NULL;
>> +		return ret;
>> +	}
>> +
>> +	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
>> +			       &priv->sensor_mbus_cfg);
>> +	if (ret)
>> +		return ret;
>> +
>> +	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
>> +
>> +	if (is_csi2) {
>> +		int vc_num = 0;
>> +		/*
>> +		 * NOTE! It seems the virtual channels from the mipi csi-2
>> +		 * receiver are used only for routing by the video mux's,
>> +		 * or for hard-wired routing to the CSI's. Once the stream
>> +		 * enters the CSI's however, they are treated internally
>> +		 * in the IPU as virtual channel 0.
>> +		 */
>> +#if 0
>> +		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
>> +							  &priv->sd.entity);
>> +		if (vc_num < 0)
>> +			return vc_num;
>> +#endif
>> +		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
>> +					  &priv->format_mbus[priv->input_pad]);
> This seems (at least to me) to go against the spirit of the subdev
> negotiation.  Here, you seem to bypass the negotiation performed
> between the CSI and the attached device, wanting to grab the
> format from the CSI2 sensor directly.  That bypasses negotiation
> performed at the CSI2 subdev and video muxes.

This isn't so much grabbing a pad format, it is determining
which source pad at the imx6-mipi-csi2 receiver subdev is
reached from this CSI (which determines the virtual channel
the CSI is receiving).

If there was a way to determine the vc# via a status register
in the CSI, that would be perfect, but there isn't. In fact, the CSIs
seem to be ignoring the meta-data bus sent by the CSI2IPU gasket
which contains this info, or that bus is not being routed to the CSIs.
As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register.
Any other value programmed there results in no data from the CSI.

And even the presence of CSI_MIPI_DI register makes no sense to me,
the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data
bus, so I don't understand why it needs to be told.

Anyway I can just yank this disabled code, it's only there for documentation
purposes.


>
> The same goes for the frame rate monitoring code - imho, the frame
> rate should be something that results from negotiation with the
> neighbouring element, not by poking at some entity that is several
> entities away.

I will fix this once the g_frame_interval op is implemented in every
subdev. I believe you mentioned this already, but g_frame_interval
will need to be chained until it reaches the originating sensor.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-02-01  0:58         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  0:58 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, Jan 31, 2017 at 04:31:55PM -0800, Steve Longerbeam wrote:
> 
> 
> On 01/31/2017 03:20 AM, Russell King - ARM Linux wrote:
> >On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> >>+static int csi_link_validate(struct v4l2_subdev *sd,
> >>+			     struct media_link *link,
> >>+			     struct v4l2_subdev_format *source_fmt,
> >>+			     struct v4l2_subdev_format *sink_fmt)
> >>+{
> >>+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> >>+	bool is_csi2;
> >>+	int ret;
> >>+
> >>+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
> >>+	if (ret)
> >>+		return ret;
> >>+
> >>+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
> >>+	if (IS_ERR(priv->sensor)) {
> >>+		v4l2_err(&priv->sd, "no sensor attached\n");
> >>+		ret = PTR_ERR(priv->sensor);
> >>+		priv->sensor = NULL;
> >>+		return ret;
> >>+	}
> >>+
> >>+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> >>+			       &priv->sensor_mbus_cfg);
> >>+	if (ret)
> >>+		return ret;
> >>+
> >>+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
> >>+
> >>+	if (is_csi2) {
> >>+		int vc_num = 0;
> >>+		/*
> >>+		 * NOTE! It seems the virtual channels from the mipi csi-2
> >>+		 * receiver are used only for routing by the video mux's,
> >>+		 * or for hard-wired routing to the CSI's. Once the stream
> >>+		 * enters the CSI's however, they are treated internally
> >>+		 * in the IPU as virtual channel 0.
> >>+		 */
> >>+#if 0
> >>+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
> >>+							  &priv->sd.entity);
> >>+		if (vc_num < 0)
> >>+			return vc_num;
> >>+#endif
> >>+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
> >>+					  &priv->format_mbus[priv->input_pad]);
> >This seems (at least to me) to go against the spirit of the subdev
> >negotiation.  Here, you seem to bypass the negotiation performed
> >between the CSI and the attached device, wanting to grab the
> >format from the CSI2 sensor directly.  That bypasses negotiation
> >performed at the CSI2 subdev and video muxes.
> 
> This isn't so much grabbing a pad format, it is determining
> which source pad at the imx6-mipi-csi2 receiver subdev is
> reached from this CSI (which determines the virtual channel
> the CSI is receiving).
> 
> If there was a way to determine the vc# via a status register
> in the CSI, that would be perfect, but there isn't. In fact, the CSIs
> seem to be ignoring the meta-data bus sent by the CSI2IPU gasket
> which contains this info, or that bus is not being routed to the CSIs.
> As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register.
> Any other value programmed there results in no data from the CSI.
> 
> And even the presence of CSI_MIPI_DI register makes no sense to me,
> the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data
> bus, so I don't understand why it needs to be told.

The CSI_MIPI_DI register selects the data stream we want from the
CSI2 camera.

CSI2 cameras can produce many different data streams - for example,
a CSI2 camera can, for the same image, produce a YUV encoded frame
and a jpeg-encoded frame.  These are sent over the CSI2 serial bus
using two different data types.

The CSI2IPU converts the serial data streams into a parallel data
stream, feeding that into the CSI layer.  The CSI layer, in
conjunction with the CSI_MIPI_DI register, selects one of these
streams to pass on to the SMFC and other blocks.

>From what I've read, the CSI can also be programmed to pass other
streams on as well, but I've never tried that.

In my particular case, the IMX219 camera produces a complete frame
using the RAW8 or RAW10 MIPI data type, and also produces two lines
of register data per frame, encoded using another data type.  I
think it should be possible to program the CSI to pass this other
data on as "generic data" through a different FIFO and have it
written to memory, which makes it possible to know the camera
settings for each frame.  (This isn't something I'm interested in
though, I'm just using it as an example of why, possibly, there's
that register in the CSI block.)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-02-01  0:58         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  0:58 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Tue, Jan 31, 2017 at 04:31:55PM -0800, Steve Longerbeam wrote:
> 
> 
> On 01/31/2017 03:20 AM, Russell King - ARM Linux wrote:
> >On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> >>+static int csi_link_validate(struct v4l2_subdev *sd,
> >>+			     struct media_link *link,
> >>+			     struct v4l2_subdev_format *source_fmt,
> >>+			     struct v4l2_subdev_format *sink_fmt)
> >>+{
> >>+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> >>+	bool is_csi2;
> >>+	int ret;
> >>+
> >>+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
> >>+	if (ret)
> >>+		return ret;
> >>+
> >>+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
> >>+	if (IS_ERR(priv->sensor)) {
> >>+		v4l2_err(&priv->sd, "no sensor attached\n");
> >>+		ret = PTR_ERR(priv->sensor);
> >>+		priv->sensor = NULL;
> >>+		return ret;
> >>+	}
> >>+
> >>+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> >>+			       &priv->sensor_mbus_cfg);
> >>+	if (ret)
> >>+		return ret;
> >>+
> >>+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
> >>+
> >>+	if (is_csi2) {
> >>+		int vc_num = 0;
> >>+		/*
> >>+		 * NOTE! It seems the virtual channels from the mipi csi-2
> >>+		 * receiver are used only for routing by the video mux's,
> >>+		 * or for hard-wired routing to the CSI's. Once the stream
> >>+		 * enters the CSI's however, they are treated internally
> >>+		 * in the IPU as virtual channel 0.
> >>+		 */
> >>+#if 0
> >>+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
> >>+							  &priv->sd.entity);
> >>+		if (vc_num < 0)
> >>+			return vc_num;
> >>+#endif
> >>+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
> >>+					  &priv->format_mbus[priv->input_pad]);
> >This seems (at least to me) to go against the spirit of the subdev
> >negotiation.  Here, you seem to bypass the negotiation performed
> >between the CSI and the attached device, wanting to grab the
> >format from the CSI2 sensor directly.  That bypasses negotiation
> >performed at the CSI2 subdev and video muxes.
> 
> This isn't so much grabbing a pad format, it is determining
> which source pad at the imx6-mipi-csi2 receiver subdev is
> reached from this CSI (which determines the virtual channel
> the CSI is receiving).
> 
> If there was a way to determine the vc# via a status register
> in the CSI, that would be perfect, but there isn't. In fact, the CSIs
> seem to be ignoring the meta-data bus sent by the CSI2IPU gasket
> which contains this info, or that bus is not being routed to the CSIs.
> As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register.
> Any other value programmed there results in no data from the CSI.
> 
> And even the presence of CSI_MIPI_DI register makes no sense to me,
> the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data
> bus, so I don't understand why it needs to be told.

The CSI_MIPI_DI register selects the data stream we want from the
CSI2 camera.

CSI2 cameras can produce many different data streams - for example,
a CSI2 camera can, for the same image, produce a YUV encoded frame
and a jpeg-encoded frame.  These are sent over the CSI2 serial bus
using two different data types.

The CSI2IPU converts the serial data streams into a parallel data
stream, feeding that into the CSI layer.  The CSI layer, in
conjunction with the CSI_MIPI_DI register, selects one of these
streams to pass on to the SMFC and other blocks.

>From what I've read, the CSI can also be programmed to pass other
streams on as well, but I've never tried that.

In my particular case, the IMX219 camera produces a complete frame
using the RAW8 or RAW10 MIPI data type, and also produces two lines
of register data per frame, encoded using another data type.  I
think it should be possible to program the CSI to pass this other
data on as "generic data" through a different FIFO and have it
written to memory, which makes it possible to know the camera
settings for each frame.  (This isn't something I'm interested in
though, I'm just using it as an example of why, possibly, there's
that register in the CSI block.)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-02-01  0:58         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  0:58 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 04:31:55PM -0800, Steve Longerbeam wrote:
> 
> 
> On 01/31/2017 03:20 AM, Russell King - ARM Linux wrote:
> >On Fri, Jan 06, 2017 at 06:11:35PM -0800, Steve Longerbeam wrote:
> >>+static int csi_link_validate(struct v4l2_subdev *sd,
> >>+			     struct media_link *link,
> >>+			     struct v4l2_subdev_format *source_fmt,
> >>+			     struct v4l2_subdev_format *sink_fmt)
> >>+{
> >>+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
> >>+	bool is_csi2;
> >>+	int ret;
> >>+
> >>+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
> >>+	if (ret)
> >>+		return ret;
> >>+
> >>+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
> >>+	if (IS_ERR(priv->sensor)) {
> >>+		v4l2_err(&priv->sd, "no sensor attached\n");
> >>+		ret = PTR_ERR(priv->sensor);
> >>+		priv->sensor = NULL;
> >>+		return ret;
> >>+	}
> >>+
> >>+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
> >>+			       &priv->sensor_mbus_cfg);
> >>+	if (ret)
> >>+		return ret;
> >>+
> >>+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
> >>+
> >>+	if (is_csi2) {
> >>+		int vc_num = 0;
> >>+		/*
> >>+		 * NOTE! It seems the virtual channels from the mipi csi-2
> >>+		 * receiver are used only for routing by the video mux's,
> >>+		 * or for hard-wired routing to the CSI's. Once the stream
> >>+		 * enters the CSI's however, they are treated internally
> >>+		 * in the IPU as virtual channel 0.
> >>+		 */
> >>+#if 0
> >>+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
> >>+							  &priv->sd.entity);
> >>+		if (vc_num < 0)
> >>+			return vc_num;
> >>+#endif
> >>+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
> >>+					  &priv->format_mbus[priv->input_pad]);
> >This seems (at least to me) to go against the spirit of the subdev
> >negotiation.  Here, you seem to bypass the negotiation performed
> >between the CSI and the attached device, wanting to grab the
> >format from the CSI2 sensor directly.  That bypasses negotiation
> >performed at the CSI2 subdev and video muxes.
> 
> This isn't so much grabbing a pad format, it is determining
> which source pad at the imx6-mipi-csi2 receiver subdev is
> reached from this CSI (which determines the virtual channel
> the CSI is receiving).
> 
> If there was a way to determine the vc# via a status register
> in the CSI, that would be perfect, but there isn't. In fact, the CSIs
> seem to be ignoring the meta-data bus sent by the CSI2IPU gasket
> which contains this info, or that bus is not being routed to the CSIs.
> As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register.
> Any other value programmed there results in no data from the CSI.
> 
> And even the presence of CSI_MIPI_DI register makes no sense to me,
> the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data
> bus, so I don't understand why it needs to be told.

The CSI_MIPI_DI register selects the data stream we want from the
CSI2 camera.

CSI2 cameras can produce many different data streams - for example,
a CSI2 camera can, for the same image, produce a YUV encoded frame
and a jpeg-encoded frame.  These are sent over the CSI2 serial bus
using two different data types.

The CSI2IPU converts the serial data streams into a parallel data
stream, feeding that into the CSI layer.  The CSI layer, in
conjunction with the CSI_MIPI_DI register, selects one of these
streams to pass on to the SMFC and other blocks.

>From what I've read, the CSI can also be programmed to pass other
streams on as well, but I've never tried that.

In my particular case, the IMX219 camera produces a complete frame
using the RAW8 or RAW10 MIPI data type, and also produces two lines
of register data per frame, encoded using another data type.  I
think it should be possible to program the CSI to pass this other
data on as "generic data" through a different FIFO and have it
written to memory, which makes it possible to know the camera
settings for each frame.  (This isn't something I'm interested in
though, I'm just using it as an example of why, possibly, there's
that register in the CSI block.)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01  1:02         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:02 UTC (permalink / raw)
  To: Philipp Zabel, Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, laurent.pinchart+renesas, bparrot,
	geert, arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel,
	Steve Longerbeam



On 01/31/2017 01:49 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 00:01 +0000, Russell King - ARM Linux wrote:
> [...]
>> The iMX6 manuals call for a very specific seven sequence of initialisation
>> for CSI2, which begins with:
>>
>> 1. reset the D-PHY.
>> 2. place MIPI sensor in LP-11 state
>> 3. perform D-PHY initialisation
>> 4. configure CSI2 lanes and de-assert resets and shutdown signals
>>
>> Since you reset the CSI2 at power up and then release it, how do you
>> guarantee that the published sequence is followed?

Hi Russell,

In "40.3.1 Startup Sequence", it states that step 1 is to "De-assert
CSI2 presetn signal (global reset)". I can't find any description of
this signal in the manual, that statement is the only mention of it. I
don't know if this the D-PHY reset signal,  it sounds more like
CSI2_RESETN (CSI-2 host controller reset).

In any case, I re-reviewed the published sequence in the manual,
and it does look like there are a couple problems.

The pipeline power up sequence in imx-media driver is as follows:
s_power(ON) op is called first on the imx6-mipi-csi2, in which CSI2 and
D-PHY resets are asserted and then de-asserted via the CSI2_RESETN
and CSI2_DPHY_RSTZ registers, the D-PHY is initialized, and lanes set.

At this point the MIPI sensor may be powered down (in fact, in OV5640
case, the PWDN pin is asserted). So there could be a problem here,
I don't think the D-PHY is considered in the LP-11 stop state when the
D-PHY master is powered off :). A fix might simply be to reverse power
on, sensor first so that it can be placed in LP-11, then imx6-mipi-csi2.

The following steps are carried out by s_stream() calls. Sensor s_stream(ON)
is called first which starts a clock on the clock lane. Then imx6-mipi-csi2
s_stream(ON) in which the PHY_STATE register is polled to confirm the D-PHY
stop state, then looks for active clock on lock lane.

There could be a problem there too. Again should be fixed simply by
calling stream-on on the imx6-mipi-csi2 first, then sensor.

So I will try the following sequence:

1. sensor power on (put D-PHY in LP-11 stop state).
2. csi-2 power on (deassert CSI2 and D-PHY resets, D-PHY init, verify 
LP-11).
3. sensor stream on (starts clock on clock lane).
4. csi-2 stream on (confirm clock on clock lane).

That comes closest to meeting the sequence requirements.

But this also puts a requirement on MIPI sensors that s_power(ON)
should only place the D_PHY in LP-11, and _not_ start the clock lane.
But perhaps that is correct behavior anyway.

Steve


>> With Philipp's driver, this is easy, because there is a prepare_stream
>> callback which gives the sensor an opportunity to get everything
>> correctly configured according to the negotiated parameters, and place
>> the sensor in LP-11 state.
>>
>> Some sensors do not power up in LP-11 state, but need to be programmed
>> fully before being asked to momentarily stream.  Only at that point is
>> the sensor guaranteed to be in the required LP-11 state.
> Do you expect that 1. and 2. could depend on the negotiated parameters
> in any way on some hardware? I had removed the prepare_stream callback
> from my driver in v2 because for my use case at least the above sequence
> could be realized by
>
> 1. in imx-mipi-csi2 s_power(1)
> 2. in MIPI sensor s_power(1)
> 3./4. in imx-mipi-csi2 s_stream(1)
> 4. in MIPI sensor s_stream(1)
>
> as long as the sensor is correctly put back into LP-11 in s_stream(0).
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01  1:02         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:02 UTC (permalink / raw)
  To: Philipp Zabel, Russell King - ARM Linux
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/31/2017 01:49 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 00:01 +0000, Russell King - ARM Linux wrote:
> [...]
>> The iMX6 manuals call for a very specific seven sequence of initialisation
>> for CSI2, which begins with:
>>
>> 1. reset the D-PHY.
>> 2. place MIPI sensor in LP-11 state
>> 3. perform D-PHY initialisation
>> 4. configure CSI2 lanes and de-assert resets and shutdown signals
>>
>> Since you reset the CSI2 at power up and then release it, how do you
>> guarantee that the published sequence is followed?

Hi Russell,

In "40.3.1 Startup Sequence", it states that step 1 is to "De-assert
CSI2 presetn signal (global reset)". I can't find any description of
this signal in the manual, that statement is the only mention of it. I
don't know if this the D-PHY reset signal,  it sounds more like
CSI2_RESETN (CSI-2 host controller reset).

In any case, I re-reviewed the published sequence in the manual,
and it does look like there are a couple problems.

The pipeline power up sequence in imx-media driver is as follows:
s_power(ON) op is called first on the imx6-mipi-csi2, in which CSI2 and
D-PHY resets are asserted and then de-asserted via the CSI2_RESETN
and CSI2_DPHY_RSTZ registers, the D-PHY is initialized, and lanes set.

At this point the MIPI sensor may be powered down (in fact, in OV5640
case, the PWDN pin is asserted). So there could be a problem here,
I don't think the D-PHY is considered in the LP-11 stop state when the
D-PHY master is powered off :). A fix might simply be to reverse power
on, sensor first so that it can be placed in LP-11, then imx6-mipi-csi2.

The following steps are carried out by s_stream() calls. Sensor s_stream(ON)
is called first which starts a clock on the clock lane. Then imx6-mipi-csi2
s_stream(ON) in which the PHY_STATE register is polled to confirm the D-PHY
stop state, then looks for active clock on lock lane.

There could be a problem there too. Again should be fixed simply by
calling stream-on on the imx6-mipi-csi2 first, then sensor.

So I will try the following sequence:

1. sensor power on (put D-PHY in LP-11 stop state).
2. csi-2 power on (deassert CSI2 and D-PHY resets, D-PHY init, verify 
LP-11).
3. sensor stream on (starts clock on clock lane).
4. csi-2 stream on (confirm clock on clock lane).

That comes closest to meeting the sequence requirements.

But this also puts a requirement on MIPI sensors that s_power(ON)
should only place the D_PHY in LP-11, and _not_ start the clock lane.
But perhaps that is correct behavior anyway.

Steve


>> With Philipp's driver, this is easy, because there is a prepare_stream
>> callback which gives the sensor an opportunity to get everything
>> correctly configured according to the negotiated parameters, and place
>> the sensor in LP-11 state.
>>
>> Some sensors do not power up in LP-11 state, but need to be programmed
>> fully before being asked to momentarily stream.  Only at that point is
>> the sensor guaranteed to be in the required LP-11 state.
> Do you expect that 1. and 2. could depend on the negotiated parameters
> in any way on some hardware? I had removed the prepare_stream callback
> from my driver in v2 because for my use case at least the above sequence
> could be realized by
>
> 1. in imx-mipi-csi2 s_power(1)
> 2. in MIPI sensor s_power(1)
> 3./4. in imx-mipi-csi2 s_stream(1)
> 4. in MIPI sensor s_stream(1)
>
> as long as the sensor is correctly put back into LP-11 in s_stream(0).
>
> regards
> Philipp
>

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01  1:02         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:02 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 01:49 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 00:01 +0000, Russell King - ARM Linux wrote:
> [...]
>> The iMX6 manuals call for a very specific seven sequence of initialisation
>> for CSI2, which begins with:
>>
>> 1. reset the D-PHY.
>> 2. place MIPI sensor in LP-11 state
>> 3. perform D-PHY initialisation
>> 4. configure CSI2 lanes and de-assert resets and shutdown signals
>>
>> Since you reset the CSI2 at power up and then release it, how do you
>> guarantee that the published sequence is followed?

Hi Russell,

In "40.3.1 Startup Sequence", it states that step 1 is to "De-assert
CSI2 presetn signal (global reset)". I can't find any description of
this signal in the manual, that statement is the only mention of it. I
don't know if this the D-PHY reset signal,  it sounds more like
CSI2_RESETN (CSI-2 host controller reset).

In any case, I re-reviewed the published sequence in the manual,
and it does look like there are a couple problems.

The pipeline power up sequence in imx-media driver is as follows:
s_power(ON) op is called first on the imx6-mipi-csi2, in which CSI2 and
D-PHY resets are asserted and then de-asserted via the CSI2_RESETN
and CSI2_DPHY_RSTZ registers, the D-PHY is initialized, and lanes set.

At this point the MIPI sensor may be powered down (in fact, in OV5640
case, the PWDN pin is asserted). So there could be a problem here,
I don't think the D-PHY is considered in the LP-11 stop state when the
D-PHY master is powered off :). A fix might simply be to reverse power
on, sensor first so that it can be placed in LP-11, then imx6-mipi-csi2.

The following steps are carried out by s_stream() calls. Sensor s_stream(ON)
is called first which starts a clock on the clock lane. Then imx6-mipi-csi2
s_stream(ON) in which the PHY_STATE register is polled to confirm the D-PHY
stop state, then looks for active clock on lock lane.

There could be a problem there too. Again should be fixed simply by
calling stream-on on the imx6-mipi-csi2 first, then sensor.

So I will try the following sequence:

1. sensor power on (put D-PHY in LP-11 stop state).
2. csi-2 power on (deassert CSI2 and D-PHY resets, D-PHY init, verify 
LP-11).
3. sensor stream on (starts clock on clock lane).
4. csi-2 stream on (confirm clock on clock lane).

That comes closest to meeting the sequence requirements.

But this also puts a requirement on MIPI sensors that s_power(ON)
should only place the D_PHY in LP-11, and _not_ start the clock lane.
But perhaps that is correct behavior anyway.

Steve


>> With Philipp's driver, this is easy, because there is a prepare_stream
>> callback which gives the sensor an opportunity to get everything
>> correctly configured according to the negotiated parameters, and place
>> the sensor in LP-11 state.
>>
>> Some sensors do not power up in LP-11 state, but need to be programmed
>> fully before being asked to momentarily stream.  Only at that point is
>> the sensor guaranteed to be in the required LP-11 state.
> Do you expect that 1. and 2. could depend on the negotiated parameters
> in any way on some hardware? I had removed the prepare_stream callback
> from my driver in v2 because for my use case at least the above sequence
> could be realized by
>
> 1. in imx-mipi-csi2 s_power(1)
> 2. in MIPI sensor s_power(1)
> 3./4. in imx-mipi-csi2 s_stream(1)
> 4. in MIPI sensor s_stream(1)
>
> as long as the sensor is correctly put back into LP-11 in s_stream(0).
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-02-01  1:02         ` Steve Longerbeam
  (?)
@ 2017-02-01  1:13           ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  1:13 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Tue, Jan 31, 2017 at 05:02:40PM -0800, Steve Longerbeam wrote:
> But this also puts a requirement on MIPI sensors that s_power(ON)
> should only place the D_PHY in LP-11, and _not_ start the clock lane.
> But perhaps that is correct behavior anyway.

If the CSI2 DPHY is held in reset state, it shouldn't matter what the
sensor does.  In the case of IMX219, it needs a full setup of the
device, including enabling it to stream (so it starts the clock lane
etc) in order to get it into LP-11 state.  Merely disabling the XCLR
signal leaves the lanes grounded.

I do seem to remember reading in one of the MIPI specs that this is
rather expected behaviour, though I can't point at a paragraph this
late in the night.

So, the only way to satisfy these requirements is this order:

- assert PHY reset signals (so blocking any activity on the CSI lanes)
- initialise the sensor (including allowing it to start streaming and
  then stopping the stream - at that point, the lanes will be in LP-11.)
- deassert the resets as per the iMX6 documentation and follow the
  remaining procedure.

I'll look at your other points tomorrow.

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01  1:13           ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  1:13 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, Philipp Zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

Hi Steve,

On Tue, Jan 31, 2017 at 05:02:40PM -0800, Steve Longerbeam wrote:
> But this also puts a requirement on MIPI sensors that s_power(ON)
> should only place the D_PHY in LP-11, and _not_ start the clock lane.
> But perhaps that is correct behavior anyway.

If the CSI2 DPHY is held in reset state, it shouldn't matter what the
sensor does.  In the case of IMX219, it needs a full setup of the
device, including enabling it to stream (so it starts the clock lane
etc) in order to get it into LP-11 state.  Merely disabling the XCLR
signal leaves the lanes grounded.

I do seem to remember reading in one of the MIPI specs that this is
rather expected behaviour, though I can't point at a paragraph this
late in the night.

So, the only way to satisfy these requirements is this order:

- assert PHY reset signals (so blocking any activity on the CSI lanes)
- initialise the sensor (including allowing it to start streaming and
  then stopping the stream - at that point, the lanes will be in LP-11.)
- deassert the resets as per the iMX6 documentation and follow the
  remaining procedure.

I'll look at your other points tomorrow.

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01  1:13           ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  1:13 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Tue, Jan 31, 2017 at 05:02:40PM -0800, Steve Longerbeam wrote:
> But this also puts a requirement on MIPI sensors that s_power(ON)
> should only place the D_PHY in LP-11, and _not_ start the clock lane.
> But perhaps that is correct behavior anyway.

If the CSI2 DPHY is held in reset state, it shouldn't matter what the
sensor does.  In the case of IMX219, it needs a full setup of the
device, including enabling it to stream (so it starts the clock lane
etc) in order to get it into LP-11 state.  Merely disabling the XCLR
signal leaves the lanes grounded.

I do seem to remember reading in one of the MIPI specs that this is
rather expected behaviour, though I can't point at a paragraph this
late in the night.

So, the only way to satisfy these requirements is this order:

- assert PHY reset signals (so blocking any activity on the CSI lanes)
- initialise the sensor (including allowing it to start streaming and
  then stopping the stream - at that point, the lanes will be in LP-11.)
- deassert the resets as per the iMX6 documentation and follow the
  remaining procedure.

I'll look at your other points tomorrow.

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  1:26     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:26 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/31/2017 05:54 AM, Philipp Zabel wrote:
> Hi Steve,
>
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
>
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
>
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
>
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>
> v4l2-ctl -d /dev/video4 -V
> # This still is configured to 640x480, which is inconsistent with
> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> # have set this, too.

Because you've only configured the source pads,
and not the sink pads. The ipu_csi source format is
dependent on the sink format - output crop window is
limited by max input sensor frame, and since sink pad is
still at 640x480, output is reduced to that.

Maybe I'm missing something, is it expected behavior that
a source format should be automatically propagated to
the sink?

>
> v4l2-ctl --list-formats -d /dev/video4
> # This lists all the RGB formats, which it shouldn't. There is
> # no CSC in this pipeline, so we should be limited to YUV formats
> # only.

right, need to fix that. Probably by poking the attached
source subdev (csi or prpenc/vf) for its supported formats.


>
> # Set capture format
> v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY
>
> v4l2-ctl -d /dev/video4 -V
> # Now the capture format is correctly configured to 1920x1080.
>
> v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
> # This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
> # frame interval. In the future this should list fractions achievable
> # via frame skipping.

yes, need to implement g_frame_interval.

> v4l2-compliance -d /dev/video4
> # This fails two tests:
> # fail: v4l2-test-input-output.cpp(383): std == 0
> # fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
> # test VIDIOC_G/S/ENUMINPUT: FAIL
> # and
> # fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
> # test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>
> # (Slowly) stream JPEG images to a display host:
> gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink
>
> I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
> timeout" message when starting streaming.

It's hard to say what is going on there, it would be great if I could get my
hands on a Nitrogen6X with the tc35874 to help you debug.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  1:26     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:26 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 01/31/2017 05:54 AM, Philipp Zabel wrote:
> Hi Steve,
>
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
>
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
>
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
>
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>
> v4l2-ctl -d /dev/video4 -V
> # This still is configured to 640x480, which is inconsistent with
> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> # have set this, too.

Because you've only configured the source pads,
and not the sink pads. The ipu_csi source format is
dependent on the sink format - output crop window is
limited by max input sensor frame, and since sink pad is
still at 640x480, output is reduced to that.

Maybe I'm missing something, is it expected behavior that
a source format should be automatically propagated to
the sink?

>
> v4l2-ctl --list-formats -d /dev/video4
> # This lists all the RGB formats, which it shouldn't. There is
> # no CSC in this pipeline, so we should be limited to YUV formats
> # only.

right, need to fix that. Probably by poking the attached
source subdev (csi or prpenc/vf) for its supported formats.


>
> # Set capture format
> v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY
>
> v4l2-ctl -d /dev/video4 -V
> # Now the capture format is correctly configured to 1920x1080.
>
> v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
> # This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
> # frame interval. In the future this should list fractions achievable
> # via frame skipping.

yes, need to implement g_frame_interval.

> v4l2-compliance -d /dev/video4
> # This fails two tests:
> # fail: v4l2-test-input-output.cpp(383): std == 0
> # fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
> # test VIDIOC_G/S/ENUMINPUT: FAIL
> # and
> # fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
> # test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>
> # (Slowly) stream JPEG images to a display host:
> gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink
>
> I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
> timeout" message when starting streaming.

It's hard to say what is going on there, it would be great if I could get my
hands on a Nitrogen6X with the tc35874 to help you debug.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  1:26     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:26 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 05:54 AM, Philipp Zabel wrote:
> Hi Steve,
>
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
>
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
>
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
>
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>
> v4l2-ctl -d /dev/video4 -V
> # This still is configured to 640x480, which is inconsistent with
> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> # have set this, too.

Because you've only configured the source pads,
and not the sink pads. The ipu_csi source format is
dependent on the sink format - output crop window is
limited by max input sensor frame, and since sink pad is
still at 640x480, output is reduced to that.

Maybe I'm missing something, is it expected behavior that
a source format should be automatically propagated to
the sink?

>
> v4l2-ctl --list-formats -d /dev/video4
> # This lists all the RGB formats, which it shouldn't. There is
> # no CSC in this pipeline, so we should be limited to YUV formats
> # only.

right, need to fix that. Probably by poking the attached
source subdev (csi or prpenc/vf) for its supported formats.


>
> # Set capture format
> v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY
>
> v4l2-ctl -d /dev/video4 -V
> # Now the capture format is correctly configured to 1920x1080.
>
> v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
> # This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
> # frame interval. In the future this should list fractions achievable
> # via frame skipping.

yes, need to implement g_frame_interval.

> v4l2-compliance -d /dev/video4
> # This fails two tests:
> # fail: v4l2-test-input-output.cpp(383): std == 0
> # fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
> # test VIDIOC_G/S/ENUMINPUT: FAIL
> # and
> # fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
> # test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>
> # (Slowly) stream JPEG images to a display host:
> gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink
>
> I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
> timeout" message when starting streaming.

It's hard to say what is going on there, it would be great if I could get my
hands on a Nitrogen6X with the tc35874 to help you debug.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
  2017-01-31 16:48       ` Philipp Zabel
  (?)
@ 2017-02-01  1:40         ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:40 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 01/31/2017 08:48 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 16:51 +0100, Philipp Zabel wrote:
>> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
>> [...]
>>> +static int csi_set_fmt(struct v4l2_subdev *sd,
>>> +		       struct v4l2_subdev_pad_config *cfg,
>>> +		       struct v4l2_subdev_format *sdformat)
>>> +{
>>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>>> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
>>> +	struct v4l2_rect crop;
>>> +	int ret;
>>> +
>>> +	if (sdformat->pad >= CSI_NUM_PADS)
>>> +		return -EINVAL;
>>> +
>>> +	if (priv->stream_on)
>>> +		return -EBUSY;
>>> +
>>> +	infmt = &priv->format_mbus[priv->input_pad];
>>> +	outfmt = &priv->format_mbus[priv->output_pad];
>>> +
>>> +	if (sdformat->pad == priv->output_pad) {
>>> +		sdformat->format.code = infmt->code;
>>> +		sdformat->format.field = infmt->field;
>>> +		crop.left = priv->crop.left;
>>> +		crop.top = priv->crop.top;
>>> +		crop.width = sdformat->format.width;
>>> +		crop.height = sdformat->format.height;
>>> +		ret = csi_try_crop(priv, &crop);
>> This is the wrong way around, see also below.
>>
>> Here the the output sdformat->format.width/height should be set to the
>> priv->crop.width/height (or priv->crop.width/height / 2, to enable
>> downscaling). The crop rectangle should not be changed by an output pad
>> set_fmt.
> [...]
>> The crop rectangle instead should be reset to the full input frame when
>> the input pad format is set.
> How about this:

Thanks for the patch, I'll try it tomorrow.

Steve

>
> ----------8<----------
> diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
> index 5e80a0871b139..8220e4204a125 100644
> --- a/drivers/staging/media/imx/imx-media-csi.c
> +++ b/drivers/staging/media/imx/imx-media-csi.c
> @@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
>   	if_fmt.field = outfmt->field;
>   
>   	ipu_csi_set_window(priv->csi, &priv->crop);
> +	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
> +					priv->crop.height == 2 * outfmt->height);
>   
>   	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
>   
> @@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   	switch (sdformat->pad) {
>   	case CSI_SRC_PAD_DIRECT:
>   	case CSI_SRC_PAD_IDMAC:
> -		crop.left = priv->crop.left;
> -		crop.top = priv->crop.top;
> -		crop.width = sdformat->format.width;
> -		crop.height = sdformat->format.height;
> -		ret = csi_try_crop(priv, &crop, sensor);
> -		if (ret)
> -			return ret;
> -		sdformat->format.width = crop.width;
> -		sdformat->format.height = crop.height;
> +		if (sdformat->format.width < priv->crop.width * 3 / 4)
> +			sdformat->format.width = priv->crop.width / 2;
> +		else
> +			sdformat->format.width = priv->crop.width;
> +		if (sdformat->format.height < priv->crop.height * 3 / 4)
> +			sdformat->format.height = priv->crop.height / 2;
> +		else
> +			sdformat->format.height = priv->crop.height;
>   
>   		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
>   			cc = imx_media_find_format(0, sdformat->format.code,
> @@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   		}
>   		break;
>   	case CSI_SINK_PAD:
> +		crop.left = 0;
> +		crop.top = 0;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop, sensor);
> +		if (ret)
> +			return ret;
> +
>   		cc = imx_media_find_format(0, sdformat->format.code,
>   					   true, false);
>   		if (!cc) {
> @@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   	} else {
>   		priv->format_mbus[sdformat->pad] = sdformat->format;
>   		priv->cc[sdformat->pad] = cc;
> -		/* Update the crop window if this is an output pad  */
> -		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
> -		    sdformat->pad == CSI_SRC_PAD_IDMAC)
> +		/* Reset the crop window if this is the input pad  */
> +		if (sdformat->pad == CSI_SINK_PAD)
>   			priv->crop = crop;
>   	}
>   
> ---------->8----------
>
> regards
> Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-02-01  1:40         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:40 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 01/31/2017 08:48 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 16:51 +0100, Philipp Zabel wrote:
>> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
>> [...]
>>> +static int csi_set_fmt(struct v4l2_subdev *sd,
>>> +		       struct v4l2_subdev_pad_config *cfg,
>>> +		       struct v4l2_subdev_format *sdformat)
>>> +{
>>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>>> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
>>> +	struct v4l2_rect crop;
>>> +	int ret;
>>> +
>>> +	if (sdformat->pad >= CSI_NUM_PADS)
>>> +		return -EINVAL;
>>> +
>>> +	if (priv->stream_on)
>>> +		return -EBUSY;
>>> +
>>> +	infmt = &priv->format_mbus[priv->input_pad];
>>> +	outfmt = &priv->format_mbus[priv->output_pad];
>>> +
>>> +	if (sdformat->pad == priv->output_pad) {
>>> +		sdformat->format.code = infmt->code;
>>> +		sdformat->format.field = infmt->field;
>>> +		crop.left = priv->crop.left;
>>> +		crop.top = priv->crop.top;
>>> +		crop.width = sdformat->format.width;
>>> +		crop.height = sdformat->format.height;
>>> +		ret = csi_try_crop(priv, &crop);
>> This is the wrong way around, see also below.
>>
>> Here the the output sdformat->format.width/height should be set to the
>> priv->crop.width/height (or priv->crop.width/height / 2, to enable
>> downscaling). The crop rectangle should not be changed by an output pad
>> set_fmt.
> [...]
>> The crop rectangle instead should be reset to the full input frame when
>> the input pad format is set.
> How about this:

Thanks for the patch, I'll try it tomorrow.

Steve

>
> ----------8<----------
> diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
> index 5e80a0871b139..8220e4204a125 100644
> --- a/drivers/staging/media/imx/imx-media-csi.c
> +++ b/drivers/staging/media/imx/imx-media-csi.c
> @@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
>   	if_fmt.field = outfmt->field;
>   
>   	ipu_csi_set_window(priv->csi, &priv->crop);
> +	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
> +					priv->crop.height == 2 * outfmt->height);
>   
>   	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
>   
> @@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   	switch (sdformat->pad) {
>   	case CSI_SRC_PAD_DIRECT:
>   	case CSI_SRC_PAD_IDMAC:
> -		crop.left = priv->crop.left;
> -		crop.top = priv->crop.top;
> -		crop.width = sdformat->format.width;
> -		crop.height = sdformat->format.height;
> -		ret = csi_try_crop(priv, &crop, sensor);
> -		if (ret)
> -			return ret;
> -		sdformat->format.width = crop.width;
> -		sdformat->format.height = crop.height;
> +		if (sdformat->format.width < priv->crop.width * 3 / 4)
> +			sdformat->format.width = priv->crop.width / 2;
> +		else
> +			sdformat->format.width = priv->crop.width;
> +		if (sdformat->format.height < priv->crop.height * 3 / 4)
> +			sdformat->format.height = priv->crop.height / 2;
> +		else
> +			sdformat->format.height = priv->crop.height;
>   
>   		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
>   			cc = imx_media_find_format(0, sdformat->format.code,
> @@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   		}
>   		break;
>   	case CSI_SINK_PAD:
> +		crop.left = 0;
> +		crop.top = 0;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop, sensor);
> +		if (ret)
> +			return ret;
> +
>   		cc = imx_media_find_format(0, sdformat->format.code,
>   					   true, false);
>   		if (!cc) {
> @@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   	} else {
>   		priv->format_mbus[sdformat->pad] = sdformat->format;
>   		priv->cc[sdformat->pad] = cc;
> -		/* Update the crop window if this is an output pad  */
> -		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
> -		    sdformat->pad == CSI_SRC_PAD_IDMAC)
> +		/* Reset the crop window if this is the input pad  */
> +		if (sdformat->pad == CSI_SINK_PAD)
>   			priv->crop = crop;
>   	}
>   
> ---------->8----------
>
> regards
> Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 17/24] media: imx: Add CSI subdev driver
@ 2017-02-01  1:40         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:40 UTC (permalink / raw)
  To: linux-arm-kernel



On 01/31/2017 08:48 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 16:51 +0100, Philipp Zabel wrote:
>> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
>> [...]
>>> +static int csi_set_fmt(struct v4l2_subdev *sd,
>>> +		       struct v4l2_subdev_pad_config *cfg,
>>> +		       struct v4l2_subdev_format *sdformat)
>>> +{
>>> +	struct csi_priv *priv = v4l2_get_subdevdata(sd);
>>> +	struct v4l2_mbus_framefmt *infmt, *outfmt;
>>> +	struct v4l2_rect crop;
>>> +	int ret;
>>> +
>>> +	if (sdformat->pad >= CSI_NUM_PADS)
>>> +		return -EINVAL;
>>> +
>>> +	if (priv->stream_on)
>>> +		return -EBUSY;
>>> +
>>> +	infmt = &priv->format_mbus[priv->input_pad];
>>> +	outfmt = &priv->format_mbus[priv->output_pad];
>>> +
>>> +	if (sdformat->pad == priv->output_pad) {
>>> +		sdformat->format.code = infmt->code;
>>> +		sdformat->format.field = infmt->field;
>>> +		crop.left = priv->crop.left;
>>> +		crop.top = priv->crop.top;
>>> +		crop.width = sdformat->format.width;
>>> +		crop.height = sdformat->format.height;
>>> +		ret = csi_try_crop(priv, &crop);
>> This is the wrong way around, see also below.
>>
>> Here the the output sdformat->format.width/height should be set to the
>> priv->crop.width/height (or priv->crop.width/height / 2, to enable
>> downscaling). The crop rectangle should not be changed by an output pad
>> set_fmt.
> [...]
>> The crop rectangle instead should be reset to the full input frame when
>> the input pad format is set.
> How about this:

Thanks for the patch, I'll try it tomorrow.

Steve

>
> ----------8<----------
> diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c
> index 5e80a0871b139..8220e4204a125 100644
> --- a/drivers/staging/media/imx/imx-media-csi.c
> +++ b/drivers/staging/media/imx/imx-media-csi.c
> @@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
>   	if_fmt.field = outfmt->field;
>   
>   	ipu_csi_set_window(priv->csi, &priv->crop);
> +	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
> +					priv->crop.height == 2 * outfmt->height);
>   
>   	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
>   
> @@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   	switch (sdformat->pad) {
>   	case CSI_SRC_PAD_DIRECT:
>   	case CSI_SRC_PAD_IDMAC:
> -		crop.left = priv->crop.left;
> -		crop.top = priv->crop.top;
> -		crop.width = sdformat->format.width;
> -		crop.height = sdformat->format.height;
> -		ret = csi_try_crop(priv, &crop, sensor);
> -		if (ret)
> -			return ret;
> -		sdformat->format.width = crop.width;
> -		sdformat->format.height = crop.height;
> +		if (sdformat->format.width < priv->crop.width * 3 / 4)
> +			sdformat->format.width = priv->crop.width / 2;
> +		else
> +			sdformat->format.width = priv->crop.width;
> +		if (sdformat->format.height < priv->crop.height * 3 / 4)
> +			sdformat->format.height = priv->crop.height / 2;
> +		else
> +			sdformat->format.height = priv->crop.height;
>   
>   		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
>   			cc = imx_media_find_format(0, sdformat->format.code,
> @@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   		}
>   		break;
>   	case CSI_SINK_PAD:
> +		crop.left = 0;
> +		crop.top = 0;
> +		crop.width = sdformat->format.width;
> +		crop.height = sdformat->format.height;
> +		ret = csi_try_crop(priv, &crop, sensor);
> +		if (ret)
> +			return ret;
> +
>   		cc = imx_media_find_format(0, sdformat->format.code,
>   					   true, false);
>   		if (!cc) {
> @@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
>   	} else {
>   		priv->format_mbus[sdformat->pad] = sdformat->format;
>   		priv->cc[sdformat->pad] = cc;
> -		/* Update the crop window if this is an output pad  */
> -		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
> -		    sdformat->pad == CSI_SRC_PAD_IDMAC)
> +		/* Reset the crop window if this is the input pad  */
> +		if (sdformat->pad == CSI_SINK_PAD)
>   			priv->crop = crop;
>   	}
>   
> ---------->8----------
>
> regards
> Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-01  0:23           ` Russell King - ARM Linux
  (?)
@ 2017-02-01  1:54             ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:54 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee


On 01/31/2017 04:23 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 03:43:22PM -0800, Steve Longerbeam wrote:
>>
>> On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
>>> Just like smiapp, the camera sensor block (which is the very far end
>>> of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
>>> front of that is the binner, which just like smiapp gets a separate
>>> entity.  It's this entity which is connected to the mipi-csi2 subdev.
>> wow, ok got it.
>>
>> So the sensor pipeline and binner, and the OF graph connecting
>> them, are described in the device tree I presume.
> No - because the binner and sensor are on the same die, it's even
> one single device, there's no real separation of the two devices.
>
> The reason there's no real separation is because the binning is done
> as part of the process of reading the array - sometimes before the
> analog voltage is converted to its digital pixel value representation.
>
> The separation comes because of the requirements of the v4l2 media
> subdevs, which requires scalers to have a sink pad and a source pad.
> (Please see the v4l2 documentation, I think I've already quoted this:
>
>         ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:
>
>         -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``
>
>         -  Video scaler. An entity capable of video scaling must have
>            at least one sink pad and one source pad, and scale the
>            video frame(s) received on its sink pad(s) to a different
>            resolution output on its source pad(s). The range of
>            supported scaling ratios is entity-specific and can differ
>            between the horizontal and vertical directions (in particular
>            scaling can be supported in one direction only). Binning and
>            skipping are considered as scaling.
>
> (Oh yes, I see it was the mail to which you were replying to...)
>
> So, in order to configure the scaling (which can be none, /2 and /4)
> we have to expose two subdevs - one which is the scaler, and has a
> source pad connected to the upstream (in this case CSI2 receiver)
> and a sink pad immutably connected to the camera sensor.
>
> Since the split is entirely down to the V4L2 implementation requirements,
> it's not something that should be reflected in DT - we don't put
> implementation details in DT.
>
> It's just the same (as I've already said) as the SMIAPP camera driver,
> the reason I'm pointing you towards that is because this is an already
> mainlined camera driver which nicely illustrates how my driver is
> structured from the v4l2 subdev API point of view.
>
>> The OF graph AFAIK, has no information about which ports are sinks
>> and which are sources, so of_parse_subdev() tries to determine that
>> based on the compatible string of the device node. So ATM
>> of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
>> video-multiplexer, and camera sensors upstream from the CSI ports
>> in the OF graph.
>>
>> I realize that's not a robust solution, and is the reason for the
>> "no sensor attached" below.
>>
>> Is there any way to determine from the OF graph the data-direction
>> of a port (whether it is a sink or a source)? If so it will make
>> of_parse_subdev() much more robust.
> I'm not sure why you need to know the data direction.

First, thank you for the explanation, it clears up a lot.

But of_parse_subdev() is used to parse the OF graph starting
from the CSI ports, to discover all the nodes to add to subdev
async registration. It also forms the media link info to be used
later to create the media graph, after all discovered subdevs
have come online (registered themselves). This happens at
driver load time, it doesn't have anything to do with pad
negotiation.

>    I think the
> issue here is how you're going about dealing with the subdevs.
> Here's the subdev setup:
>
> +---------camera--------+
> | pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> +-----------------------+
>
> How the subdevs are supposed to work is that you start from the first
> subdev in sequence (in this case the pixel array) and negotiate with
> the driver through the TRY formats on its source pad, as well as
> negotiating with the sink pad of the directly neighbouring subdev.
>
> The neighbouring subdev propagates the configuration in a driver
> specific manner from its source pad to the sink pad, giving a default
> configuration at its source.

Let me try to re-phrase. You mean the subdev's set_fmt(), when
configured  its source pad(s), should call set_fmt() at the connected
sink subdev to automatically propagate the format to the sink's pad?

>
> This process repeats throughout the chain all the way up to the bridge
> device.
>
> Now, where things go wrong is that you want to know what each type of
> these subdevs are, and the reason you want that is so you can do this
> (for example - I know similar stuff happens with the "sensor" stuff
> further up the chain as well):
>
> +---------camera--------+
> | pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> +-----------------------+                                |
>                ^--I-want-your-bus-format-and-frame-rate---'
>
> which goes against the negotiation mechanism of v4l2 subdevs.  This
> is broken - it bypass the subdev negotiation that has been performed
> on the intervening subdevs between the "sensor" and the csi0 subdevs,
> so if there were a subdev in that chain that (eg) reduced the frame
> rate by discarding the odd frames, you'd be working with the wrong
> frame interval for your frame interval monitor at csi.
>
> Note that the "MEDIA_ENT_F_PROC_VIDEO_SCALER" subdev type is documented
> as not only supports scaling as in changing the size of the image, but
> also in terms of skipping frames, which means a reduction in frame rate.
>
> So, for your FIM, you need to know if there is any reduction in frame
> rate through that pipeline, and looking for a "MEDIA_ENT_F_CAM_SENSOR"
> subdev node isn't going to tell you that.  The frame rate really needs
> to be carried through and I suspect you need to accept the frame rate
> into each subdev so it can be passed along the chain by the application
> configuring the pipeline.
>
> The last bit from the above is the "I-want-your-bus-format" bit which
> I haven't fully worked out how to eliminate - I understand the reason
> you need that (so you can appropriately configure the CSI with the CSI2
> data type code.)  The CSI2 data type code comes from the format
> configured on the CSI sink pad, so the only information you're really
> using there is "are we sinking data from a CSI2 interface."
>
> You _could_ walk down the graph from the CSI looking for a subdev that
> responds to g_mbus_config that reports CSI2, but I'm not sure that's
> going to last - I've seen an email from Hans saying that he'd like
> g_mbus_config to go away (to patch 13/24, for the vidsw_g_mbus_config()
> function):
>
> "I am not certain this op is needed at all. In the current kernel this
>   op is onlyused by soc_camera, pxa_camera and omap3isp (somewhat dubious).
>   Normally this information should come from the device tree and there
>   should be no need for this op.
>
>   My (tentative) long-term plan was to get rid of this op.
>
>   If you don't need it, then I recommend it is removed."
>
> So, if that goes away, the CSI subdev needs a completely different way
> to get that information, and it shouldn't be coming from the camera
> sensor subdev, but (imho) really from the CSI2 subdev.
>
> This is probably something that needs to be discussed with media people
> to work out how to replace the g_mbus_config call with something more
> acceptable to resolve this issue.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  1:54             ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:54 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee


On 01/31/2017 04:23 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 03:43:22PM -0800, Steve Longerbeam wrote:
>>
>> On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
>>> Just like smiapp, the camera sensor block (which is the very far end
>>> of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
>>> front of that is the binner, which just like smiapp gets a separate
>>> entity.  It's this entity which is connected to the mipi-csi2 subdev.
>> wow, ok got it.
>>
>> So the sensor pipeline and binner, and the OF graph connecting
>> them, are described in the device tree I presume.
> No - because the binner and sensor are on the same die, it's even
> one single device, there's no real separation of the two devices.
>
> The reason there's no real separation is because the binning is done
> as part of the process of reading the array - sometimes before the
> analog voltage is converted to its digital pixel value representation.
>
> The separation comes because of the requirements of the v4l2 media
> subdevs, which requires scalers to have a sink pad and a source pad.
> (Please see the v4l2 documentation, I think I've already quoted this:
>
>         ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:
>
>         -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``
>
>         -  Video scaler. An entity capable of video scaling must have
>            at least one sink pad and one source pad, and scale the
>            video frame(s) received on its sink pad(s) to a different
>            resolution output on its source pad(s). The range of
>            supported scaling ratios is entity-specific and can differ
>            between the horizontal and vertical directions (in particular
>            scaling can be supported in one direction only). Binning and
>            skipping are considered as scaling.
>
> (Oh yes, I see it was the mail to which you were replying to...)
>
> So, in order to configure the scaling (which can be none, /2 and /4)
> we have to expose two subdevs - one which is the scaler, and has a
> source pad connected to the upstream (in this case CSI2 receiver)
> and a sink pad immutably connected to the camera sensor.
>
> Since the split is entirely down to the V4L2 implementation requirements,
> it's not something that should be reflected in DT - we don't put
> implementation details in DT.
>
> It's just the same (as I've already said) as the SMIAPP camera driver,
> the reason I'm pointing you towards that is because this is an already
> mainlined camera driver which nicely illustrates how my driver is
> structured from the v4l2 subdev API point of view.
>
>> The OF graph AFAIK, has no information about which ports are sinks
>> and which are sources, so of_parse_subdev() tries to determine that
>> based on the compatible string of the device node. So ATM
>> of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
>> video-multiplexer, and camera sensors upstream from the CSI ports
>> in the OF graph.
>>
>> I realize that's not a robust solution, and is the reason for the
>> "no sensor attached" below.
>>
>> Is there any way to determine from the OF graph the data-direction
>> of a port (whether it is a sink or a source)? If so it will make
>> of_parse_subdev() much more robust.
> I'm not sure why you need to know the data direction.

First, thank you for the explanation, it clears up a lot.

But of_parse_subdev() is used to parse the OF graph starting
from the CSI ports, to discover all the nodes to add to subdev
async registration. It also forms the media link info to be used
later to create the media graph, after all discovered subdevs
have come online (registered themselves). This happens at
driver load time, it doesn't have anything to do with pad
negotiation.

>    I think the
> issue here is how you're going about dealing with the subdevs.
> Here's the subdev setup:
>
> +---------camera--------+
> | pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> +-----------------------+
>
> How the subdevs are supposed to work is that you start from the first
> subdev in sequence (in this case the pixel array) and negotiate with
> the driver through the TRY formats on its source pad, as well as
> negotiating with the sink pad of the directly neighbouring subdev.
>
> The neighbouring subdev propagates the configuration in a driver
> specific manner from its source pad to the sink pad, giving a default
> configuration at its source.

Let me try to re-phrase. You mean the subdev's set_fmt(), when
configured  its source pad(s), should call set_fmt() at the connected
sink subdev to automatically propagate the format to the sink's pad?

>
> This process repeats throughout the chain all the way up to the bridge
> device.
>
> Now, where things go wrong is that you want to know what each type of
> these subdevs are, and the reason you want that is so you can do this
> (for example - I know similar stuff happens with the "sensor" stuff
> further up the chain as well):
>
> +---------camera--------+
> | pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> +-----------------------+                                |
>                ^--I-want-your-bus-format-and-frame-rate---'
>
> which goes against the negotiation mechanism of v4l2 subdevs.  This
> is broken - it bypass the subdev negotiation that has been performed
> on the intervening subdevs between the "sensor" and the csi0 subdevs,
> so if there were a subdev in that chain that (eg) reduced the frame
> rate by discarding the odd frames, you'd be working with the wrong
> frame interval for your frame interval monitor at csi.
>
> Note that the "MEDIA_ENT_F_PROC_VIDEO_SCALER" subdev type is documented
> as not only supports scaling as in changing the size of the image, but
> also in terms of skipping frames, which means a reduction in frame rate.
>
> So, for your FIM, you need to know if there is any reduction in frame
> rate through that pipeline, and looking for a "MEDIA_ENT_F_CAM_SENSOR"
> subdev node isn't going to tell you that.  The frame rate really needs
> to be carried through and I suspect you need to accept the frame rate
> into each subdev so it can be passed along the chain by the application
> configuring the pipeline.
>
> The last bit from the above is the "I-want-your-bus-format" bit which
> I haven't fully worked out how to eliminate - I understand the reason
> you need that (so you can appropriately configure the CSI with the CSI2
> data type code.)  The CSI2 data type code comes from the format
> configured on the CSI sink pad, so the only information you're really
> using there is "are we sinking data from a CSI2 interface."
>
> You _could_ walk down the graph from the CSI looking for a subdev that
> responds to g_mbus_config that reports CSI2, but I'm not sure that's
> going to last - I've seen an email from Hans saying that he'd like
> g_mbus_config to go away (to patch 13/24, for the vidsw_g_mbus_config()
> function):
>
> "I am not certain this op is needed at all. In the current kernel this
>   op is onlyused by soc_camera, pxa_camera and omap3isp (somewhat dubious).
>   Normally this information should come from the device tree and there
>   should be no need for this op.
>
>   My (tentative) long-term plan was to get rid of this op.
>
>   If you don't need it, then I recommend it is removed."
>
> So, if that goes away, the CSI subdev needs a completely different way
> to get that information, and it shouldn't be coming from the camera
> sensor subdev, but (imho) really from the CSI2 subdev.
>
> This is probably something that needs to be discussed with media people
> to work out how to replace the g_mbus_config call with something more
> acceptable to resolve this issue.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  1:54             ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01  1:54 UTC (permalink / raw)
  To: linux-arm-kernel


On 01/31/2017 04:23 PM, Russell King - ARM Linux wrote:
> On Tue, Jan 31, 2017 at 03:43:22PM -0800, Steve Longerbeam wrote:
>>
>> On 01/31/2017 03:00 AM, Russell King - ARM Linux wrote:
>>> Just like smiapp, the camera sensor block (which is the very far end
>>> of the pipeline) is marked with MEDIA_ENT_F_CAM_SENSOR.  However, in
>>> front of that is the binner, which just like smiapp gets a separate
>>> entity.  It's this entity which is connected to the mipi-csi2 subdev.
>> wow, ok got it.
>>
>> So the sensor pipeline and binner, and the OF graph connecting
>> them, are described in the device tree I presume.
> No - because the binner and sensor are on the same die, it's even
> one single device, there's no real separation of the two devices.
>
> The reason there's no real separation is because the binning is done
> as part of the process of reading the array - sometimes before the
> analog voltage is converted to its digital pixel value representation.
>
> The separation comes because of the requirements of the v4l2 media
> subdevs, which requires scalers to have a sink pad and a source pad.
> (Please see the v4l2 documentation, I think I've already quoted this:
>
>         ..  _MEDIA-ENT-F-PROC-VIDEO-SCALER:
>
>         -  ``MEDIA_ENT_F_PROC_VIDEO_SCALER``
>
>         -  Video scaler. An entity capable of video scaling must have
>            at least one sink pad and one source pad, and scale the
>            video frame(s) received on its sink pad(s) to a different
>            resolution output on its source pad(s). The range of
>            supported scaling ratios is entity-specific and can differ
>            between the horizontal and vertical directions (in particular
>            scaling can be supported in one direction only). Binning and
>            skipping are considered as scaling.
>
> (Oh yes, I see it was the mail to which you were replying to...)
>
> So, in order to configure the scaling (which can be none, /2 and /4)
> we have to expose two subdevs - one which is the scaler, and has a
> source pad connected to the upstream (in this case CSI2 receiver)
> and a sink pad immutably connected to the camera sensor.
>
> Since the split is entirely down to the V4L2 implementation requirements,
> it's not something that should be reflected in DT - we don't put
> implementation details in DT.
>
> It's just the same (as I've already said) as the SMIAPP camera driver,
> the reason I'm pointing you towards that is because this is an already
> mainlined camera driver which nicely illustrates how my driver is
> structured from the v4l2 subdev API point of view.
>
>> The OF graph AFAIK, has no information about which ports are sinks
>> and which are sources, so of_parse_subdev() tries to determine that
>> based on the compatible string of the device node. So ATM
>> of_parse_subdev() assumes there is nothing but the imx6-mipi-csi2,
>> video-multiplexer, and camera sensors upstream from the CSI ports
>> in the OF graph.
>>
>> I realize that's not a robust solution, and is the reason for the
>> "no sensor attached" below.
>>
>> Is there any way to determine from the OF graph the data-direction
>> of a port (whether it is a sink or a source)? If so it will make
>> of_parse_subdev() much more robust.
> I'm not sure why you need to know the data direction.

First, thank you for the explanation, it clears up a lot.

But of_parse_subdev() is used to parse the OF graph starting
from the CSI ports, to discover all the nodes to add to subdev
async registration. It also forms the media link info to be used
later to create the media graph, after all discovered subdevs
have come online (registered themselves). This happens at
driver load time, it doesn't have anything to do with pad
negotiation.

>    I think the
> issue here is how you're going about dealing with the subdevs.
> Here's the subdev setup:
>
> +---------camera--------+
> | pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> +-----------------------+
>
> How the subdevs are supposed to work is that you start from the first
> subdev in sequence (in this case the pixel array) and negotiate with
> the driver through the TRY formats on its source pad, as well as
> negotiating with the sink pad of the directly neighbouring subdev.
>
> The neighbouring subdev propagates the configuration in a driver
> specific manner from its source pad to the sink pad, giving a default
> configuration at its source.

Let me try to re-phrase. You mean the subdev's set_fmt(), when
configured  its source pad(s), should call set_fmt() at the connected
sink subdev to automatically propagate the format to the sink's pad?

>
> This process repeats throughout the chain all the way up to the bridge
> device.
>
> Now, where things go wrong is that you want to know what each type of
> these subdevs are, and the reason you want that is so you can do this
> (for example - I know similar stuff happens with the "sensor" stuff
> further up the chain as well):
>
> +---------camera--------+
> | pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> +-----------------------+                                |
>                ^--I-want-your-bus-format-and-frame-rate---'
>
> which goes against the negotiation mechanism of v4l2 subdevs.  This
> is broken - it bypass the subdev negotiation that has been performed
> on the intervening subdevs between the "sensor" and the csi0 subdevs,
> so if there were a subdev in that chain that (eg) reduced the frame
> rate by discarding the odd frames, you'd be working with the wrong
> frame interval for your frame interval monitor at csi.
>
> Note that the "MEDIA_ENT_F_PROC_VIDEO_SCALER" subdev type is documented
> as not only supports scaling as in changing the size of the image, but
> also in terms of skipping frames, which means a reduction in frame rate.
>
> So, for your FIM, you need to know if there is any reduction in frame
> rate through that pipeline, and looking for a "MEDIA_ENT_F_CAM_SENSOR"
> subdev node isn't going to tell you that.  The frame rate really needs
> to be carried through and I suspect you need to accept the frame rate
> into each subdev so it can be passed along the chain by the application
> configuring the pipeline.
>
> The last bit from the above is the "I-want-your-bus-format" bit which
> I haven't fully worked out how to eliminate - I understand the reason
> you need that (so you can appropriately configure the CSI with the CSI2
> data type code.)  The CSI2 data type code comes from the format
> configured on the CSI sink pad, so the only information you're really
> using there is "are we sinking data from a CSI2 interface."
>
> You _could_ walk down the graph from the CSI looking for a subdev that
> responds to g_mbus_config that reports CSI2, but I'm not sure that's
> going to last - I've seen an email from Hans saying that he'd like
> g_mbus_config to go away (to patch 13/24, for the vidsw_g_mbus_config()
> function):
>
> "I am not certain this op is needed at all. In the current kernel this
>   op is onlyused by soc_camera, pxa_camera and omap3isp (somewhat dubious).
>   Normally this information should come from the device tree and there
>   should be no need for this op.
>
>   My (tentative) long-term plan was to get rid of this op.
>
>   If you don't need it, then I recommend it is removed."
>
> So, if that goes away, the CSI subdev needs a completely different way
> to get that information, and it shouldn't be coming from the camera
> sensor subdev, but (imho) really from the CSI2 subdev.
>
> This is probably something that needs to be discussed with media people
> to work out how to replace the g_mbus_config call with something more
> acceptable to resolve this issue.
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-01  1:54             ` Steve Longerbeam
  (?)
@ 2017-02-01  9:20               ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  9:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 05:54:52PM -0800, Steve Longerbeam wrote:
> On 01/31/2017 04:23 PM, Russell King - ARM Linux wrote:
> First, thank you for the explanation, it clears up a lot.
> 
> But of_parse_subdev() is used to parse the OF graph starting
> from the CSI ports, to discover all the nodes to add to subdev
> async registration. It also forms the media link info to be used
> later to create the media graph, after all discovered subdevs
> have come online (registered themselves). This happens at
> driver load time, it doesn't have anything to do with pad
> negotiation.

Right, but I'm discussing why you need to know which is the sensor
subdev, and why you need to get parameters from the sensor subdev.

> >   I think the
> >issue here is how you're going about dealing with the subdevs.
> >Here's the subdev setup:
> >
> >+---------camera--------+
> >| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> >+-----------------------+
> >
> >How the subdevs are supposed to work is that you start from the first
> >subdev in sequence (in this case the pixel array) and negotiate with
> >the driver through the TRY formats on its source pad, as well as
> >negotiating with the sink pad of the directly neighbouring subdev.
> >
> >The neighbouring subdev propagates the configuration in a driver
> >specific manner from its source pad to the sink pad, giving a default
> >configuration at its source.
> 
> Let me try to re-phrase. You mean the subdev's set_fmt(), when
> configured  its source pad(s), should call set_fmt() at the connected
> sink subdev to automatically propagate the format to the sink's pad?

No.  For any individual subdev, if you configure it's _sink_ then it
should propagate the configuration to its _source_, potentially
modifying the configuration according to its function.  It should
never forward the configuration to the other side of any links.

The responsibility for setting up the neighbours source pad is the
userspace media application.

See Documentation/media/uapi/v4l/dev-subdev.rst and the section
named "Format Negotiation".

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  9:20               ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  9:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Tue, Jan 31, 2017 at 05:54:52PM -0800, Steve Longerbeam wrote:
> On 01/31/2017 04:23 PM, Russell King - ARM Linux wrote:
> First, thank you for the explanation, it clears up a lot.
> 
> But of_parse_subdev() is used to parse the OF graph starting
> from the CSI ports, to discover all the nodes to add to subdev
> async registration. It also forms the media link info to be used
> later to create the media graph, after all discovered subdevs
> have come online (registered themselves). This happens at
> driver load time, it doesn't have anything to do with pad
> negotiation.

Right, but I'm discussing why you need to know which is the sensor
subdev, and why you need to get parameters from the sensor subdev.

> >   I think the
> >issue here is how you're going about dealing with the subdevs.
> >Here's the subdev setup:
> >
> >+---------camera--------+
> >| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> >+-----------------------+
> >
> >How the subdevs are supposed to work is that you start from the first
> >subdev in sequence (in this case the pixel array) and negotiate with
> >the driver through the TRY formats on its source pad, as well as
> >negotiating with the sink pad of the directly neighbouring subdev.
> >
> >The neighbouring subdev propagates the configuration in a driver
> >specific manner from its source pad to the sink pad, giving a default
> >configuration at its source.
> 
> Let me try to re-phrase. You mean the subdev's set_fmt(), when
> configured  its source pad(s), should call set_fmt() at the connected
> sink subdev to automatically propagate the format to the sink's pad?

No.  For any individual subdev, if you configure it's _sink_ then it
should propagate the configuration to its _source_, potentially
modifying the configuration according to its function.  It should
never forward the configuration to the other side of any links.

The responsibility for setting up the neighbours source pad is the
userspace media application.

See Documentation/media/uapi/v4l/dev-subdev.rst and the section
named "Format Negotiation".

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  9:20               ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01  9:20 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jan 31, 2017 at 05:54:52PM -0800, Steve Longerbeam wrote:
> On 01/31/2017 04:23 PM, Russell King - ARM Linux wrote:
> First, thank you for the explanation, it clears up a lot.
> 
> But of_parse_subdev() is used to parse the OF graph starting
> from the CSI ports, to discover all the nodes to add to subdev
> async registration. It also forms the media link info to be used
> later to create the media graph, after all discovered subdevs
> have come online (registered themselves). This happens at
> driver load time, it doesn't have anything to do with pad
> negotiation.

Right, but I'm discussing why you need to know which is the sensor
subdev, and why you need to get parameters from the sensor subdev.

> >   I think the
> >issue here is how you're going about dealing with the subdevs.
> >Here's the subdev setup:
> >
> >+---------camera--------+
> >| pixel array -> binner |---> csi2 --> ipu1csi0 mux --> csi0 --> smfc --> idmac
> >+-----------------------+
> >
> >How the subdevs are supposed to work is that you start from the first
> >subdev in sequence (in this case the pixel array) and negotiate with
> >the driver through the TRY formats on its source pad, as well as
> >negotiating with the sink pad of the directly neighbouring subdev.
> >
> >The neighbouring subdev propagates the configuration in a driver
> >specific manner from its source pad to the sink pad, giving a default
> >configuration at its source.
> 
> Let me try to re-phrase. You mean the subdev's set_fmt(), when
> configured  its source pad(s), should call set_fmt() at the connected
> sink subdev to automatically propagate the format to the sink's pad?

No.  For any individual subdev, if you configure it's _sink_ then it
should propagate the configuration to its _source_, potentially
modifying the configuration according to its function.  It should
never forward the configuration to the other side of any links.

The responsibility for setting up the neighbours source pad is the
userspace media application.

See Documentation/media/uapi/v4l/dev-subdev.rst and the section
named "Format Negotiation".

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  9:30       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-01  9:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
[...]
> > # Set pad formats
> > media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> > media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
> >
> > v4l2-ctl -d /dev/video4 -V
> > # This still is configured to 640x480, which is inconsistent with
> > # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> > # have set this, too.
> 
> Because you've only configured the source pads,
> and not the sink pads. The ipu_csi source format is
> dependent on the sink format - output crop window is
> limited by max input sensor frame, and since sink pad is
> still at 640x480, output is reduced to that.

No, it is set (see below). What happens is that capture_g_fmt_vid_cap
just returns the capture devices' priv->vdev.fmt, even if it is
incompatible with the connected csi subdevice's output pad format.

priv->vdev.fmt was never changed from the default set in
imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
were not called yet.

> Maybe I'm missing something, is it expected behavior that
> a source format should be automatically propagated to
> the sink?

media-ctl propagates the output pad format to all remote subdevices'
input pads for all enabled links:

https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c#n693

> > v4l2-ctl --list-formats -d /dev/video4
> > # This lists all the RGB formats, which it shouldn't. There is
> > # no CSC in this pipeline, so we should be limited to YUV formats
> > # only.
> 
> right, need to fix that. Probably by poking the attached
> source subdev (csi or prpenc/vf) for its supported formats.

You are right, in bayer/raw mode only one specific format should be
listed, depending on the CSI output pad format.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  9:30       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-01  9:30 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
[...]
> > # Set pad formats
> > media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> > media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
> >
> > v4l2-ctl -d /dev/video4 -V
> > # This still is configured to 640x480, which is inconsistent with
> > # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> > # have set this, too.
> 
> Because you've only configured the source pads,
> and not the sink pads. The ipu_csi source format is
> dependent on the sink format - output crop window is
> limited by max input sensor frame, and since sink pad is
> still at 640x480, output is reduced to that.

No, it is set (see below). What happens is that capture_g_fmt_vid_cap
just returns the capture devices' priv->vdev.fmt, even if it is
incompatible with the connected csi subdevice's output pad format.

priv->vdev.fmt was never changed from the default set in
imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
were not called yet.

> Maybe I'm missing something, is it expected behavior that
> a source format should be automatically propagated to
> the sink?

media-ctl propagates the output pad format to all remote subdevices'
input pads for all enabled links:

https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c#n693

> > v4l2-ctl --list-formats -d /dev/video4
> > # This lists all the RGB formats, which it shouldn't. There is
> > # no CSC in this pipeline, so we should be limited to YUV formats
> > # only.
> 
> right, need to fix that. Probably by poking the attached
> source subdev (csi or prpenc/vf) for its supported formats.

You are right, in bayer/raw mode only one specific format should be
listed, depending on the CSI output pad format.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01  9:30       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-01  9:30 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
[...]
> > # Set pad formats
> > media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> > media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> > media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
> >
> > v4l2-ctl -d /dev/video4 -V
> > # This still is configured to 640x480, which is inconsistent with
> > # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> > # have set this, too.
> 
> Because you've only configured the source pads,
> and not the sink pads. The ipu_csi source format is
> dependent on the sink format - output crop window is
> limited by max input sensor frame, and since sink pad is
> still at 640x480, output is reduced to that.

No, it is set (see below). What happens is that capture_g_fmt_vid_cap
just returns the capture devices' priv->vdev.fmt, even if it is
incompatible with the connected csi subdevice's output pad format.

priv->vdev.fmt was never changed from the default set in
imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
were not called yet.

> Maybe I'm missing something, is it expected behavior that
> a source format should be automatically propagated to
> the sink?

media-ctl propagates the output pad format to all remote subdevices'
input pads for all enabled links:

https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c#n693

> > v4l2-ctl --list-formats -d /dev/video4
> > # This lists all the RGB formats, which it shouldn't. There is
> > # no CSC in this pipeline, so we should be limited to YUV formats
> > # only.
> 
> right, need to fix that. Probably by poking the attached
> source subdev (csi or prpenc/vf) for its supported formats.

You are right, in bayer/raw mode only one specific format should be
listed, depending on the CSI output pad format.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-01  9:30       ` Philipp Zabel
  (?)
@ 2017-02-01 10:11         ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 10:11 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, mark.rutland, andrew-ct.chen, minghsiu.tsai,
	nick, songjun.wu, hverkuil, Steve Longerbeam, robert.jarzmik,
	devel, markus.heiser, laurent.pinchart+renesas, geert,
	linux-media, devicetree, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Wed, Feb 01, 2017 at 10:30:57AM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> [...]
> > right, need to fix that. Probably by poking the attached
> > source subdev (csi or prpenc/vf) for its supported formats.
> 
> You are right, in bayer/raw mode only one specific format should be
> listed, depending on the CSI output pad format.

It depends what Steve means by "source subdev".

It should be the next sub-device below the bridge - if we have this
setup of subdev's:

---> CSI ---> SMFC ---> IDMAC

then the format configured at the SMFC's output pad is what matters,
not what was configured at CSI.

It's the responsibility of SMFC and CSI to make sure that their source
pads are configured with a compatible format for their corresponding
source pad, and it's also the sink subdev's responsibility to check
that the configuration across a link is valid (possibly via
v4l2_subdev_link_validate(), or a more intensive or relaxed test if
required.)

For example:

- when CSI's source pad is configured with a RGGB output format,
  userspace media-ctl will also set that on SMFC's sink pad.
- when SMFC's sink pad is configured, SMFC should configure it's
  source pad with an identical format (RGGB).
- when SMFC's source pad is configured, it should refuse to change
  the format, because SMFC can't modify pixel the format - it's
  just a buffer.

When starting to stream (Documentation/media/kapi/v4l2-subdev.rst) the
link validation function is called, and:

- the SMFC driver's link_validate function will be called to validate
  the CSI -> SMFC link.  This allows the SMFC to be sure that there's
  a compatible configuration - and, since the link does not allow
  format conversion, it should verify that the format on the CSI's
  source pad is the same as SMFC's sink pad.

Not only does this match what the hardware's doing, it also means that,
because there's no format conversion between the CSI's hardware output
and IDMAC, we don't need to care about trying to fetch the CSI's source
pad configuration from the IDMAC end - we can fetch that information
from our neighbour's SMFC's source pad _or_ our own sink pad if we have
one.

To see why this is an important, consider what the effect would be if
SMFC did have the capability to change the pixel format.  That means the
format presented to the IDMAC block would depend on the configuration of
SMFC, and the CSI's source pad format is no longer relevant to IDMAC.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01 10:11         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 10:11 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, Steve Longerbeam, linux-media, devicetree, arnd, mchehab,
	bparrot, robh+dt, horms+renesas, tiffany.lin,
	laurent.pinchart+renesas, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Wed, Feb 01, 2017 at 10:30:57AM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> [...]
> > right, need to fix that. Probably by poking the attached
> > source subdev (csi or prpenc/vf) for its supported formats.
> 
> You are right, in bayer/raw mode only one specific format should be
> listed, depending on the CSI output pad format.

It depends what Steve means by "source subdev".

It should be the next sub-device below the bridge - if we have this
setup of subdev's:

---> CSI ---> SMFC ---> IDMAC

then the format configured at the SMFC's output pad is what matters,
not what was configured at CSI.

It's the responsibility of SMFC and CSI to make sure that their source
pads are configured with a compatible format for their corresponding
source pad, and it's also the sink subdev's responsibility to check
that the configuration across a link is valid (possibly via
v4l2_subdev_link_validate(), or a more intensive or relaxed test if
required.)

For example:

- when CSI's source pad is configured with a RGGB output format,
  userspace media-ctl will also set that on SMFC's sink pad.
- when SMFC's sink pad is configured, SMFC should configure it's
  source pad with an identical format (RGGB).
- when SMFC's source pad is configured, it should refuse to change
  the format, because SMFC can't modify pixel the format - it's
  just a buffer.

When starting to stream (Documentation/media/kapi/v4l2-subdev.rst) the
link validation function is called, and:

- the SMFC driver's link_validate function will be called to validate
  the CSI -> SMFC link.  This allows the SMFC to be sure that there's
  a compatible configuration - and, since the link does not allow
  format conversion, it should verify that the format on the CSI's
  source pad is the same as SMFC's sink pad.

Not only does this match what the hardware's doing, it also means that,
because there's no format conversion between the CSI's hardware output
and IDMAC, we don't need to care about trying to fetch the CSI's source
pad configuration from the IDMAC end - we can fetch that information
from our neighbour's SMFC's source pad _or_ our own sink pad if we have
one.

To see why this is an important, consider what the effect would be if
SMFC did have the capability to change the pixel format.  That means the
format presented to the IDMAC block would depend on the configuration of
SMFC, and the CSI's source pad format is no longer relevant to IDMAC.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01 10:11         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 10:11 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Feb 01, 2017 at 10:30:57AM +0100, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> [...]
> > right, need to fix that. Probably by poking the attached
> > source subdev (csi or prpenc/vf) for its supported formats.
> 
> You are right, in bayer/raw mode only one specific format should be
> listed, depending on the CSI output pad format.

It depends what Steve means by "source subdev".

It should be the next sub-device below the bridge - if we have this
setup of subdev's:

---> CSI ---> SMFC ---> IDMAC

then the format configured at the SMFC's output pad is what matters,
not what was configured at CSI.

It's the responsibility of SMFC and CSI to make sure that their source
pads are configured with a compatible format for their corresponding
source pad, and it's also the sink subdev's responsibility to check
that the configuration across a link is valid (possibly via
v4l2_subdev_link_validate(), or a more intensive or relaxed test if
required.)

For example:

- when CSI's source pad is configured with a RGGB output format,
  userspace media-ctl will also set that on SMFC's sink pad.
- when SMFC's sink pad is configured, SMFC should configure it's
  source pad with an identical format (RGGB).
- when SMFC's source pad is configured, it should refuse to change
  the format, because SMFC can't modify pixel the format - it's
  just a buffer.

When starting to stream (Documentation/media/kapi/v4l2-subdev.rst) the
link validation function is called, and:

- the SMFC driver's link_validate function will be called to validate
  the CSI -> SMFC link.  This allows the SMFC to be sure that there's
  a compatible configuration - and, since the link does not allow
  format conversion, it should verify that the format on the CSI's
  source pad is the same as SMFC's sink pad.

Not only does this match what the hardware's doing, it also means that,
because there's no format conversion between the CSI's hardware output
and IDMAC, we don't need to care about trying to fetch the CSI's source
pad configuration from the IDMAC end - we can fetch that information
from our neighbour's SMFC's source pad _or_ our own sink pad if we have
one.

To see why this is an important, consider what the effect would be if
SMFC did have the capability to change the pixel format.  That means the
format presented to the IDMAC block would depend on the configuration of
SMFC, and the CSI's source pad format is no longer relevant to IDMAC.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01 10:42           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-01 10:42 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, mark.rutland, andrew-ct.chen, minghsiu.tsai,
	nick, songjun.wu, hverkuil, Steve Longerbeam, robert.jarzmik,
	devel, markus.heiser, laurent.pinchart+renesas, geert,
	linux-media, devicetree, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Wed, 2017-02-01 at 10:11 +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 01, 2017 at 10:30:57AM +0100, Philipp Zabel wrote:
> > On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> > [...]
> > > right, need to fix that. Probably by poking the attached
> > > source subdev (csi or prpenc/vf) for its supported formats.
> > 
> > You are right, in bayer/raw mode only one specific format should be
> > listed, depending on the CSI output pad format.
> 
> It depends what Steve means by "source subdev".
> 
> It should be the next sub-device below the bridge - if we have this
> setup of subdev's:
> 
> ---> CSI ---> SMFC ---> IDMAC
> 
> then the format configured at the SMFC's output pad is what matters,
> not what was configured at CSI.

Right, it's just that in the latest version there is no v4l2_subdev for
fifos and idmac. There is the capture interface entity that represents
one of the IDMAC write channels, but that doesn't have a pad and format
configuration.
The SMFC entity was removed because the fifo can be considered part of
the link between CSI and IDMAC. There is no manual configuration
necessary as the SMFC itself can't do anything to the data that flows
through it. There is no reason to present it to userspace as a no-op
entity.
So in the direct CSI -> SMFC -> IDMAC case, the CSI source pad now is
the nearest neighbor pad to the IDMAC capture video device.

> It's the responsibility of SMFC and CSI to make sure that their source
> pads are configured with a compatible format for their corresponding
> source pad, and it's also the sink subdev's responsibility to check
> that the configuration across a link is valid (possibly via
> v4l2_subdev_link_validate(), or a more intensive or relaxed test if
> required.)
> 
> For example:
> 
> - when CSI's source pad is configured with a RGGB output format,
>   userspace media-ctl will also set that on SMFC's sink pad.
> - when SMFC's sink pad is configured, SMFC should configure it's
>   source pad with an identical format (RGGB).
> - when SMFC's source pad is configured, it should refuse to change
>   the format, because SMFC can't modify pixel the format - it's
>   just a buffer.
>
> When starting to stream (Documentation/media/kapi/v4l2-subdev.rst) the
> link validation function is called, and:
> 
> - the SMFC driver's link_validate function will be called to validate
>   the CSI -> SMFC link.  This allows the SMFC to be sure that there's
>   a compatible configuration - and, since the link does not allow
>   format conversion, it should verify that the format on the CSI's
>   source pad is the same as SMFC's sink pad.
> 
> Not only does this match what the hardware's doing, it also means that,
> because there's no format conversion between the CSI's hardware output
> and IDMAC, we don't need to care about trying to fetch the CSI's source
> pad configuration from the IDMAC end - we can fetch that information
> from our neighbour's SMFC's source pad _or_ our own sink pad if we have
> one.

See above. All this is correct for the remaining entities, just the CSI
source pad now takes the role of the SMFC source pad as nearest neighbor
to the IDMAC capture video device.

> To see why this is an important, consider what the effect would be if
> SMFC did have the capability to change the pixel format.  That means the
> format presented to the IDMAC block would depend on the configuration of
> SMFC, and the CSI's source pad format is no longer relevant to IDMAC.

Yes, this is exactly the case for the CSI -> IC PRP -> IC PRPVF -> IDMAC
route, as the IC can do color space conversion. Here, (only) the IC
PRPVF source pad should determine the capture video device's format, and
the negotiation between CSI->IC PRP and between IC PRP->IC PRPVF should
happen as you say.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01 10:42           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-01 10:42 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, mark.rutland-5wv7dgnIgG8,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	nick-gcszYUEDH4VrovVCs/uTlw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, Steve Longerbeam,
	robert.jarzmik-GANU6spQydw,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	geert-Td1EMuHUCqxL1ZNQvxDV9g, linux-media-u79uwXL29TY76Z2rM5mHXA,
	devicetree-u79uwXL29TY76Z2rM5mHXA, arnd-r2nGTMty4D4,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, bparrot-l0cyMroinI0,
	robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	jean-christophe.trotin-qxv4g6HH51o,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A

On Wed, 2017-02-01 at 10:11 +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 01, 2017 at 10:30:57AM +0100, Philipp Zabel wrote:
> > On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> > [...]
> > > right, need to fix that. Probably by poking the attached
> > > source subdev (csi or prpenc/vf) for its supported formats.
> > 
> > You are right, in bayer/raw mode only one specific format should be
> > listed, depending on the CSI output pad format.
> 
> It depends what Steve means by "source subdev".
> 
> It should be the next sub-device below the bridge - if we have this
> setup of subdev's:
> 
> ---> CSI ---> SMFC ---> IDMAC
> 
> then the format configured at the SMFC's output pad is what matters,
> not what was configured at CSI.

Right, it's just that in the latest version there is no v4l2_subdev for
fifos and idmac. There is the capture interface entity that represents
one of the IDMAC write channels, but that doesn't have a pad and format
configuration.
The SMFC entity was removed because the fifo can be considered part of
the link between CSI and IDMAC. There is no manual configuration
necessary as the SMFC itself can't do anything to the data that flows
through it. There is no reason to present it to userspace as a no-op
entity.
So in the direct CSI -> SMFC -> IDMAC case, the CSI source pad now is
the nearest neighbor pad to the IDMAC capture video device.

> It's the responsibility of SMFC and CSI to make sure that their source
> pads are configured with a compatible format for their corresponding
> source pad, and it's also the sink subdev's responsibility to check
> that the configuration across a link is valid (possibly via
> v4l2_subdev_link_validate(), or a more intensive or relaxed test if
> required.)
> 
> For example:
> 
> - when CSI's source pad is configured with a RGGB output format,
>   userspace media-ctl will also set that on SMFC's sink pad.
> - when SMFC's sink pad is configured, SMFC should configure it's
>   source pad with an identical format (RGGB).
> - when SMFC's source pad is configured, it should refuse to change
>   the format, because SMFC can't modify pixel the format - it's
>   just a buffer.
>
> When starting to stream (Documentation/media/kapi/v4l2-subdev.rst) the
> link validation function is called, and:
> 
> - the SMFC driver's link_validate function will be called to validate
>   the CSI -> SMFC link.  This allows the SMFC to be sure that there's
>   a compatible configuration - and, since the link does not allow
>   format conversion, it should verify that the format on the CSI's
>   source pad is the same as SMFC's sink pad.
> 
> Not only does this match what the hardware's doing, it also means that,
> because there's no format conversion between the CSI's hardware output
> and IDMAC, we don't need to care about trying to fetch the CSI's source
> pad configuration from the IDMAC end - we can fetch that information
> from our neighbour's SMFC's source pad _or_ our own sink pad if we have
> one.

See above. All this is correct for the remaining entities, just the CSI
source pad now takes the role of the SMFC source pad as nearest neighbor
to the IDMAC capture video device.

> To see why this is an important, consider what the effect would be if
> SMFC did have the capability to change the pixel format.  That means the
> format presented to the IDMAC block would depend on the configuration of
> SMFC, and the CSI's source pad format is no longer relevant to IDMAC.

Yes, this is exactly the case for the CSI -> IC PRP -> IC PRPVF -> IDMAC
route, as the IC can do color space conversion. Here, (only) the IC
PRPVF source pad should determine the capture video device's format, and
the negotiation between CSI->IC PRP and between IC PRP->IC PRPVF should
happen as you say.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01 10:42           ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-01 10:42 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, 2017-02-01 at 10:11 +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 01, 2017 at 10:30:57AM +0100, Philipp Zabel wrote:
> > On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> > [...]
> > > right, need to fix that. Probably by poking the attached
> > > source subdev (csi or prpenc/vf) for its supported formats.
> > 
> > You are right, in bayer/raw mode only one specific format should be
> > listed, depending on the CSI output pad format.
> 
> It depends what Steve means by "source subdev".
> 
> It should be the next sub-device below the bridge - if we have this
> setup of subdev's:
> 
> ---> CSI ---> SMFC ---> IDMAC
> 
> then the format configured at the SMFC's output pad is what matters,
> not what was configured at CSI.

Right, it's just that in the latest version there is no v4l2_subdev for
fifos and idmac. There is the capture interface entity that represents
one of the IDMAC write channels, but that doesn't have a pad and format
configuration.
The SMFC entity was removed because the fifo can be considered part of
the link between CSI and IDMAC. There is no manual configuration
necessary as the SMFC itself can't do anything to the data that flows
through it. There is no reason to present it to userspace as a no-op
entity.
So in the direct CSI -> SMFC -> IDMAC case, the CSI source pad now is
the nearest neighbor pad to the IDMAC capture video device.

> It's the responsibility of SMFC and CSI to make sure that their source
> pads are configured with a compatible format for their corresponding
> source pad, and it's also the sink subdev's responsibility to check
> that the configuration across a link is valid (possibly via
> v4l2_subdev_link_validate(), or a more intensive or relaxed test if
> required.)
> 
> For example:
> 
> - when CSI's source pad is configured with a RGGB output format,
>   userspace media-ctl will also set that on SMFC's sink pad.
> - when SMFC's sink pad is configured, SMFC should configure it's
>   source pad with an identical format (RGGB).
> - when SMFC's source pad is configured, it should refuse to change
>   the format, because SMFC can't modify pixel the format - it's
>   just a buffer.
>
> When starting to stream (Documentation/media/kapi/v4l2-subdev.rst) the
> link validation function is called, and:
> 
> - the SMFC driver's link_validate function will be called to validate
>   the CSI -> SMFC link.  This allows the SMFC to be sure that there's
>   a compatible configuration - and, since the link does not allow
>   format conversion, it should verify that the format on the CSI's
>   source pad is the same as SMFC's sink pad.
> 
> Not only does this match what the hardware's doing, it also means that,
> because there's no format conversion between the CSI's hardware output
> and IDMAC, we don't need to care about trying to fetch the CSI's source
> pad configuration from the IDMAC end - we can fetch that information
> from our neighbour's SMFC's source pad _or_ our own sink pad if we have
> one.

See above. All this is correct for the remaining entities, just the CSI
source pad now takes the role of the SMFC source pad as nearest neighbor
to the IDMAC capture video device.

> To see why this is an important, consider what the effect would be if
> SMFC did have the capability to change the pixel format.  That means the
> format presented to the IDMAC block would depend on the configuration of
> SMFC, and the CSI's source pad format is no longer relevant to IDMAC.

Yes, this is exactly the case for the CSI -> IC PRP -> IC PRPVF -> IDMAC
route, as the IC can do color space conversion. Here, (only) the IC
PRPVF source pad should determine the capture video device's format, and
the negotiation between CSI->IC PRP and between IC PRP->IC PRPVF should
happen as you say.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-01 10:42           ` Philipp Zabel
  (?)
@ 2017-02-01 10:53             ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 10:53 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, mark.rutland, andrew-ct.chen, minghsiu.tsai,
	nick, songjun.wu, hverkuil, Steve Longerbeam, robert.jarzmik,
	devel, markus.heiser, laurent.pinchart+renesas, geert,
	linux-media, devicetree, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Wed, Feb 01, 2017 at 11:42:31AM +0100, Philipp Zabel wrote:
> On Wed, 2017-02-01 at 10:11 +0000, Russell King - ARM Linux wrote:
> Right, it's just that in the latest version there is no v4l2_subdev for
> fifos and idmac. There is the capture interface entity that represents
> one of the IDMAC write channels, but that doesn't have a pad and format
> configuration.
> The SMFC entity was removed because the fifo can be considered part of
> the link between CSI and IDMAC. There is no manual configuration
> necessary as the SMFC itself can't do anything to the data that flows
> through it. There is no reason to present it to userspace as a no-op
> entity.
> So in the direct CSI -> SMFC -> IDMAC case, the CSI source pad now is
> the nearest neighbor pad to the IDMAC capture video device.

Ok, that sounds fine then.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01 10:53             ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 10:53 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, Steve Longerbeam, linux-media, devicetree, arnd, mchehab,
	bparrot, robh+dt, horms+renesas, tiffany.lin,
	laurent.pinchart+renesas, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Wed, Feb 01, 2017 at 11:42:31AM +0100, Philipp Zabel wrote:
> On Wed, 2017-02-01 at 10:11 +0000, Russell King - ARM Linux wrote:
> Right, it's just that in the latest version there is no v4l2_subdev for
> fifos and idmac. There is the capture interface entity that represents
> one of the IDMAC write channels, but that doesn't have a pad and format
> configuration.
> The SMFC entity was removed because the fifo can be considered part of
> the link between CSI and IDMAC. There is no manual configuration
> necessary as the SMFC itself can't do anything to the data that flows
> through it. There is no reason to present it to userspace as a no-op
> entity.
> So in the direct CSI -> SMFC -> IDMAC case, the CSI source pad now is
> the nearest neighbor pad to the IDMAC capture video device.

Ok, that sounds fine then.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-01 10:53             ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 10:53 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Feb 01, 2017 at 11:42:31AM +0100, Philipp Zabel wrote:
> On Wed, 2017-02-01 at 10:11 +0000, Russell King - ARM Linux wrote:
> Right, it's just that in the latest version there is no v4l2_subdev for
> fifos and idmac. There is the capture interface entity that represents
> one of the IDMAC write channels, but that doesn't have a pad and format
> configuration.
> The SMFC entity was removed because the fifo can be considered part of
> the link between CSI and IDMAC. There is no manual configuration
> necessary as the SMFC itself can't do anything to the data that flows
> through it. There is no reason to present it to userspace as a no-op
> entity.
> So in the direct CSI -> SMFC -> IDMAC case, the CSI source pad now is
> the nearest neighbor pad to the IDMAC capture video device.

Ok, that sounds fine then.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 18/24] media: imx: Add SMFC subdev driver
@ 2017-02-01 18:39     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 18:39 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Fri, Jan 06, 2017 at 06:11:36PM -0800, Steve Longerbeam wrote:
> +/*
> + * Min/Max supported width and heights.
> + *
> + * We allow planar output from the SMFC, so we have to align
> + * output width by 16 pixels to meet IDMAC alignment requirements,
> + * which also means input width must have the same alignment.
> + */
> +#define MIN_W       176
> +#define MIN_H       144
> +#define MAX_W      8192
> +#define MAX_H      4096
> +#define W_ALIGN    4 /* multiple of 16 pixels */

Does this only apply to planar formats?

I notice Philipp's driver allows 8 pixel alignment.  If it's only for
planar formats, it ought to determine the alignment based on the
requested format rather than hard-coding it to the maximum alignment
of all the supported formats.

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 18/24] media: imx: Add SMFC subdev driver
@ 2017-02-01 18:39     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 18:39 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

Hi Steve,

On Fri, Jan 06, 2017 at 06:11:36PM -0800, Steve Longerbeam wrote:
> +/*
> + * Min/Max supported width and heights.
> + *
> + * We allow planar output from the SMFC, so we have to align
> + * output width by 16 pixels to meet IDMAC alignment requirements,
> + * which also means input width must have the same alignment.
> + */
> +#define MIN_W       176
> +#define MIN_H       144
> +#define MAX_W      8192
> +#define MAX_H      4096
> +#define W_ALIGN    4 /* multiple of 16 pixels */

Does this only apply to planar formats?

I notice Philipp's driver allows 8 pixel alignment.  If it's only for
planar formats, it ought to determine the alignment based on the
requested format rather than hard-coding it to the maximum alignment
of all the supported formats.

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 18/24] media: imx: Add SMFC subdev driver
@ 2017-02-01 18:39     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 18:39 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Fri, Jan 06, 2017 at 06:11:36PM -0800, Steve Longerbeam wrote:
> +/*
> + * Min/Max supported width and heights.
> + *
> + * We allow planar output from the SMFC, so we have to align
> + * output width by 16 pixels to meet IDMAC alignment requirements,
> + * which also means input width must have the same alignment.
> + */
> +#define MIN_W       176
> +#define MIN_H       144
> +#define MAX_W      8192
> +#define MAX_H      4096
> +#define W_ALIGN    4 /* multiple of 16 pixels */

Does this only apply to planar formats?

I notice Philipp's driver allows 8 pixel alignment.  If it's only for
planar formats, it ought to determine the alignment based on the
requested format rather than hard-coding it to the maximum alignment
of all the supported formats.

Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 18/24] media: imx: Add SMFC subdev driver
  2017-02-01 18:39     ` Russell King - ARM Linux
  (?)
@ 2017-02-01 18:52       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01 18:52 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 02/01/2017 10:39 AM, Russell King - ARM Linux wrote:
> Hi Steve,
>
> On Fri, Jan 06, 2017 at 06:11:36PM -0800, Steve Longerbeam wrote:
>> +/*
>> + * Min/Max supported width and heights.
>> + *
>> + * We allow planar output from the SMFC, so we have to align
>> + * output width by 16 pixels to meet IDMAC alignment requirements,
>> + * which also means input width must have the same alignment.
>> + */
>> +#define MIN_W       176
>> +#define MIN_H       144
>> +#define MAX_W      8192
>> +#define MAX_H      4096
>> +#define W_ALIGN    4 /* multiple of 16 pixels */
> Does this only apply to planar formats?
>
> I notice Philipp's driver allows 8 pixel alignment.  If it's only for
> planar formats, it ought to determine the alignment based on the
> requested format rather than hard-coding it to the maximum alignment
> of all the supported formats.

yeah, I got lazy/tired there. I will fix this.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 18/24] media: imx: Add SMFC subdev driver
@ 2017-02-01 18:52       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01 18:52 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/01/2017 10:39 AM, Russell King - ARM Linux wrote:
> Hi Steve,
>
> On Fri, Jan 06, 2017 at 06:11:36PM -0800, Steve Longerbeam wrote:
>> +/*
>> + * Min/Max supported width and heights.
>> + *
>> + * We allow planar output from the SMFC, so we have to align
>> + * output width by 16 pixels to meet IDMAC alignment requirements,
>> + * which also means input width must have the same alignment.
>> + */
>> +#define MIN_W       176
>> +#define MIN_H       144
>> +#define MAX_W      8192
>> +#define MAX_H      4096
>> +#define W_ALIGN    4 /* multiple of 16 pixels */
> Does this only apply to planar formats?
>
> I notice Philipp's driver allows 8 pixel alignment.  If it's only for
> planar formats, it ought to determine the alignment based on the
> requested format rather than hard-coding it to the maximum alignment
> of all the supported formats.

yeah, I got lazy/tired there. I will fix this.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 18/24] media: imx: Add SMFC subdev driver
@ 2017-02-01 18:52       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-01 18:52 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/01/2017 10:39 AM, Russell King - ARM Linux wrote:
> Hi Steve,
>
> On Fri, Jan 06, 2017 at 06:11:36PM -0800, Steve Longerbeam wrote:
>> +/*
>> + * Min/Max supported width and heights.
>> + *
>> + * We allow planar output from the SMFC, so we have to align
>> + * output width by 16 pixels to meet IDMAC alignment requirements,
>> + * which also means input width must have the same alignment.
>> + */
>> +#define MIN_W       176
>> +#define MIN_H       144
>> +#define MAX_W      8192
>> +#define MAX_H      4096
>> +#define W_ALIGN    4 /* multiple of 16 pixels */
> Does this only apply to planar formats?
>
> I notice Philipp's driver allows 8 pixel alignment.  If it's only for
> planar formats, it ought to determine the alignment based on the
> requested format rather than hard-coding it to the maximum alignment
> of all the supported formats.

yeah, I got lazy/tired there. I will fix this.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01 23:44     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 23:44 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_pad_config *cfg,
> +			   struct v4l2_subdev_format *sdformat)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	sdformat->format = csi2->format_mbus;
> +
> +	return 0;
> +}

Hi Steve,

This isn't correct, and I suspect the other get_fmt implementations are
the same - I've just checked imx-csi.c, and that also appears to have
the same issue.

When get_fmt() is called with sdformat->which == V4L2_SUBDEV_FORMAT_TRY,
you need to return the try format rather than the current format.  See
the second paragraph of Documentation/media/uapi/v4l/dev-subdev.rst's
"Format Negotiation" section, where it talks about using
V4L2_SUBDEV_FORMAT_TRY with both VIDIOC_SUBDEV_G_FMT and
VIDIOC_SUBDEV_S_FMT.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01 23:44     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 23:44 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_pad_config *cfg,
> +			   struct v4l2_subdev_format *sdformat)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	sdformat->format = csi2->format_mbus;
> +
> +	return 0;
> +}

Hi Steve,

This isn't correct, and I suspect the other get_fmt implementations are
the same - I've just checked imx-csi.c, and that also appears to have
the same issue.

When get_fmt() is called with sdformat->which == V4L2_SUBDEV_FORMAT_TRY,
you need to return the try format rather than the current format.  See
the second paragraph of Documentation/media/uapi/v4l/dev-subdev.rst's
"Format Negotiation" section, where it talks about using
V4L2_SUBDEV_FORMAT_TRY with both VIDIOC_SUBDEV_G_FMT and
VIDIOC_SUBDEV_S_FMT.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-01 23:44     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-01 23:44 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_pad_config *cfg,
> +			   struct v4l2_subdev_format *sdformat)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	sdformat->format = csi2->format_mbus;
> +
> +	return 0;
> +}

Hi Steve,

This isn't correct, and I suspect the other get_fmt implementations are
the same - I've just checked imx-csi.c, and that also appears to have
the same issue.

When get_fmt() is called with sdformat->which == V4L2_SUBDEV_FORMAT_TRY,
you need to return the try format rather than the current format.  See
the second paragraph of Documentation/media/uapi/v4l/dev-subdev.rst's
"Format Negotiation" section, where it talks about using
V4L2_SUBDEV_FORMAT_TRY with both VIDIOC_SUBDEV_G_FMT and
VIDIOC_SUBDEV_S_FMT.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-02  0:04       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02  0:04 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 02/01/2017 03:44 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
>> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_pad_config *cfg,
>> +			   struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +
>> +	sdformat->format = csi2->format_mbus;
>> +
>> +	return 0;
>> +}
> Hi Steve,
>
> This isn't correct, and I suspect the other get_fmt implementations are
> the same - I've just checked imx-csi.c, and that also appears to have
> the same issue.
>
> When get_fmt() is called with sdformat->which == V4L2_SUBDEV_FORMAT_TRY,
> you need to return the try format rather than the current format.  See
> the second paragraph of Documentation/media/uapi/v4l/dev-subdev.rst's
> "Format Negotiation" section, where it talks about using
> V4L2_SUBDEV_FORMAT_TRY with both VIDIOC_SUBDEV_G_FMT and
> VIDIOC_SUBDEV_S_FMT.

Yes that's wrong. I'll fix.

Btw I read over Documentation/media/uapi/v4l/dev-subdev.rst (can't
remember if I ever did!), and it clears up a lot. I do see I'm doing some
other things wrong as well:

-  Formats should be propagated from sink pads to source pads. Modifying
    a format on a source pad should not modify the format on any sink
    pad.

I don't believe I'm affecting the source pad formats during sink pad
negotiation, or vice-versa, yet. But I will, once the pixel width alignment
optimization is implemented based on whether the output format is planar.
And I'll keep this direction-of-propagation rule in mind when I do so.

-  Sub-devices that scale frames using variable scaling factors should
    reset the scale factors to default values when sink pads formats are
    modified. If the 1:1 scaling ratio is supported, this means that
    source pads formats should be reset to the sink pads formats.

I'm not resetting the scaling factors on sink pad format change in
the scaling subdevs (imx-ic-prpenc and imx-ic-prpvf). Will fix that.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-02  0:04       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02  0:04 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 02/01/2017 03:44 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
>> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_pad_config *cfg,
>> +			   struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +
>> +	sdformat->format = csi2->format_mbus;
>> +
>> +	return 0;
>> +}
> Hi Steve,
>
> This isn't correct, and I suspect the other get_fmt implementations are
> the same - I've just checked imx-csi.c, and that also appears to have
> the same issue.
>
> When get_fmt() is called with sdformat->which == V4L2_SUBDEV_FORMAT_TRY,
> you need to return the try format rather than the current format.  See
> the second paragraph of Documentation/media/uapi/v4l/dev-subdev.rst's
> "Format Negotiation" section, where it talks about using
> V4L2_SUBDEV_FORMAT_TRY with both VIDIOC_SUBDEV_G_FMT and
> VIDIOC_SUBDEV_S_FMT.

Yes that's wrong. I'll fix.

Btw I read over Documentation/media/uapi/v4l/dev-subdev.rst (can't
remember if I ever did!), and it clears up a lot. I do see I'm doing some
other things wrong as well:

-  Formats should be propagated from sink pads to source pads. Modifying
    a format on a source pad should not modify the format on any sink
    pad.

I don't believe I'm affecting the source pad formats during sink pad
negotiation, or vice-versa, yet. But I will, once the pixel width alignment
optimization is implemented based on whether the output format is planar.
And I'll keep this direction-of-propagation rule in mind when I do so.

-  Sub-devices that scale frames using variable scaling factors should
    reset the scale factors to default values when sink pads formats are
    modified. If the 1:1 scaling ratio is supported, this means that
    source pads formats should be reset to the sink pads formats.

I'm not resetting the scaling factors on sink pad format change in
the scaling subdevs (imx-ic-prpenc and imx-ic-prpvf). Will fix that.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-02  0:04       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02  0:04 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/01/2017 03:44 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:39PM -0800, Steve Longerbeam wrote:
>> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_pad_config *cfg,
>> +			   struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +
>> +	sdformat->format = csi2->format_mbus;
>> +
>> +	return 0;
>> +}
> Hi Steve,
>
> This isn't correct, and I suspect the other get_fmt implementations are
> the same - I've just checked imx-csi.c, and that also appears to have
> the same issue.
>
> When get_fmt() is called with sdformat->which == V4L2_SUBDEV_FORMAT_TRY,
> you need to return the try format rather than the current format.  See
> the second paragraph of Documentation/media/uapi/v4l/dev-subdev.rst's
> "Format Negotiation" section, where it talks about using
> V4L2_SUBDEV_FORMAT_TRY with both VIDIOC_SUBDEV_G_FMT and
> VIDIOC_SUBDEV_S_FMT.

Yes that's wrong. I'll fix.

Btw I read over Documentation/media/uapi/v4l/dev-subdev.rst (can't
remember if I ever did!), and it clears up a lot. I do see I'm doing some
other things wrong as well:

-  Formats should be propagated from sink pads to source pads. Modifying
    a format on a source pad should not modify the format on any sink
    pad.

I don't believe I'm affecting the source pad formats during sink pad
negotiation, or vice-versa, yet. But I will, once the pixel width alignment
optimization is implemented based on whether the output format is planar.
And I'll keep this direction-of-propagation rule in mind when I do so.

-  Sub-devices that scale frames using variable scaling factors should
    reset the scale factors to default values when sink pads formats are
    modified. If the 1:1 scaling ratio is supported, this means that
    source pads formats should be reset to the sink pads formats.

I'm not resetting the scaling factors on sink pad format change in
the scaling subdevs (imx-ic-prpenc and imx-ic-prpvf). Will fix that.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02  0:19         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02  0:19 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 02/01/2017 01:30 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> [...]
>>> # Set pad formats
>>> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
>>> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
>>> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
>>> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>>>
>>> v4l2-ctl -d /dev/video4 -V
>>> # This still is configured to 640x480, which is inconsistent with
>>> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
>>> # have set this, too.
>> Because you've only configured the source pads,
>> and not the sink pads. The ipu_csi source format is
>> dependent on the sink format - output crop window is
>> limited by max input sensor frame, and since sink pad is
>> still at 640x480, output is reduced to that.
> No, it is set (see below). What happens is that capture_g_fmt_vid_cap
> just returns the capture devices' priv->vdev.fmt, even if it is
> incompatible with the connected csi subdevice's output pad format.
>
> priv->vdev.fmt was never changed from the default set in
> imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
> were not called yet.

Ah, yep, this is a bug. Need to modify the capture device's
width/height at .set_fmt() in the subdev's device-node source
pad (csi and prpenc/vf).

>> Maybe I'm missing something, is it expected behavior that
>> a source format should be automatically propagated to
>> the sink?
> media-ctl propagates the output pad format to all remote subdevices'
> input pads for all enabled links:
>
> https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c#n693

Ah cool, I wasn't aware media-ctl did this, but it makes sense and
makes it easier on the user.

Steve

>
>>> v4l2-ctl --list-formats -d /dev/video4
>>> # This lists all the RGB formats, which it shouldn't. There is
>>> # no CSC in this pipeline, so we should be limited to YUV formats
>>> # only.
>> right, need to fix that. Probably by poking the attached
>> source subdev (csi or prpenc/vf) for its supported formats.
> You are right, in bayer/raw mode only one specific format should be
> listed, depending on the CSI output pad format.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02  0:19         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02  0:19 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam



On 02/01/2017 01:30 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> [...]
>>> # Set pad formats
>>> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
>>> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
>>> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
>>> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>>>
>>> v4l2-ctl -d /dev/video4 -V
>>> # This still is configured to 640x480, which is inconsistent with
>>> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
>>> # have set this, too.
>> Because you've only configured the source pads,
>> and not the sink pads. The ipu_csi source format is
>> dependent on the sink format - output crop window is
>> limited by max input sensor frame, and since sink pad is
>> still at 640x480, output is reduced to that.
> No, it is set (see below). What happens is that capture_g_fmt_vid_cap
> just returns the capture devices' priv->vdev.fmt, even if it is
> incompatible with the connected csi subdevice's output pad format.
>
> priv->vdev.fmt was never changed from the default set in
> imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
> were not called yet.

Ah, yep, this is a bug. Need to modify the capture device's
width/height at .set_fmt() in the subdev's device-node source
pad (csi and prpenc/vf).

>> Maybe I'm missing something, is it expected behavior that
>> a source format should be automatically propagated to
>> the sink?
> media-ctl propagates the output pad format to all remote subdevices'
> input pads for all enabled links:
>
> https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c#n693

Ah cool, I wasn't aware media-ctl did this, but it makes sense and
makes it easier on the user.

Steve

>
>>> v4l2-ctl --list-formats -d /dev/video4
>>> # This lists all the RGB formats, which it shouldn't. There is
>>> # no CSC in this pipeline, so we should be limited to YUV formats
>>> # only.
>> right, need to fix that. Probably by poking the attached
>> source subdev (csi or prpenc/vf) for its supported formats.
> You are right, in bayer/raw mode only one specific format should be
> listed, depending on the CSI output pad format.
>
> regards
> Philipp
>

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02  0:19         ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02  0:19 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/01/2017 01:30 AM, Philipp Zabel wrote:
> On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> [...]
>>> # Set pad formats
>>> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
>>> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
>>> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
>>> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>>>
>>> v4l2-ctl -d /dev/video4 -V
>>> # This still is configured to 640x480, which is inconsistent with
>>> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
>>> # have set this, too.
>> Because you've only configured the source pads,
>> and not the sink pads. The ipu_csi source format is
>> dependent on the sink format - output crop window is
>> limited by max input sensor frame, and since sink pad is
>> still at 640x480, output is reduced to that.
> No, it is set (see below). What happens is that capture_g_fmt_vid_cap
> just returns the capture devices' priv->vdev.fmt, even if it is
> incompatible with the connected csi subdevice's output pad format.
>
> priv->vdev.fmt was never changed from the default set in
> imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
> were not called yet.

Ah, yep, this is a bug. Need to modify the capture device's
width/height at .set_fmt() in the subdev's device-node source
pad (csi and prpenc/vf).

>> Maybe I'm missing something, is it expected behavior that
>> a source format should be automatically propagated to
>> the sink?
> media-ctl propagates the output pad format to all remote subdevices'
> input pads for all enabled links:
>
> https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c#n693

Ah cool, I wasn't aware media-ctl did this, but it makes sense and
makes it easier on the user.

Steve

>
>>> v4l2-ctl --list-formats -d /dev/video4
>>> # This lists all the RGB formats, which it shouldn't. There is
>>> # no CSC in this pipeline, so we should be limited to YUV formats
>>> # only.
>> right, need to fix that. Probably by poking the attached
>> source subdev (csi or prpenc/vf) for its supported formats.
> You are right, in bayer/raw mode only one specific format should be
> listed, depending on the CSI output pad format.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-02-02 10:36     ` Laurent Pinchart
  -1 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-02 10:36 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

Thank you for the patch. Many of the comments below apply to the ov5642 driver 
too, please take them into account when reworking patch 23/24.

On Friday 06 Jan 2017 18:11:40 Steve Longerbeam wrote:
> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
> branch, modified heavily to bring forward to latest interfaces and code
> cleanup.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig       |    8 +
>  drivers/staging/media/imx/Makefile      |    2 +
>  drivers/staging/media/imx/ov5640-mipi.c | 2348 ++++++++++++++++++++++++++++

You're missing DT bindings.

The driver should go to drivers/media/i2c/ as it should not be specific to the 
i.MX6, and you can just call it ov5640.c.

>  3 files changed, 2358 insertions(+)
>  create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig
> b/drivers/staging/media/imx/Kconfig index ce2d2c8..09f373d 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>  	---help---
>  	  A video4linux camera capture driver for i.MX5/6.
> 
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA

The sensor driver is generic, it shouldn't depend on IMX. It should however 
depend on at least I2C and OF by the look of it.

> +       select IMX_MIPI_CSI2
> +       default y
> +       ---help---
> +         MIPI CSI-2 OV5640 Camera support.
> +
>  endmenu
>  endif

[snip]

> diff --git a/drivers/staging/media/imx/ov5640-mipi.c
> b/drivers/staging/media/imx/ov5640-mipi.c new file mode 100644
> index 0000000..54647a7
> --- /dev/null
> +++ b/drivers/staging/media/imx/ov5640-mipi.c
> @@ -0,0 +1,2348 @@
> +/*
> + * Copyright (c) 2014 Mentor Graphics Inc.
> + * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights
> Reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/init.h>
> +#include <linux/slab.h>
> +#include <linux/ctype.h>
> +#include <linux/types.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/i2c.h>
> +#include <linux/of_device.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/regulator/consumer.h>
> +#include <linux/clk.h>
> +#include <linux/clk-provider.h>
> +#include <linux/clkdev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>

Pet peeve of mine, please sort the headers alphabetically. It makes it easier 
to locate duplicated.

> +
> +#define OV5640_VOLTAGE_ANALOG               2800000
> +#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
> +#define OV5640_VOLTAGE_DIGITAL_IO           1800000
> +
> +#define MIN_FPS 15
> +#define MAX_FPS 30
> +#define DEFAULT_FPS 30
> +
> +/* min/typical/max system clock (xclk) frequencies */
> +#define OV5640_XCLK_MIN  6000000
> +#define OV5640_XCLK_TYP 24000000
> +#define OV5640_XCLK_MAX 54000000
> +
> +/* min/typical/max pixel clock (mclk) frequencies */
> +#define OV5640_MCLK_MIN 48000000
> +#define OV5640_MCLK_TYP 48000000
> +#define OV5640_MCLK_MAX 96000000
> +
> +#define OV5640_CHIP_ID  0x300A
> +#define OV5640_SLAVE_ID 0x3100
> +#define OV5640_DEFAULT_SLAVE_ID 0x3c

You're mixing lower-case and upper-case hex constants. Let's pick one. Kernel 
code usually favours lower-case.

Please define macros for all the other numerical constants you use in the 
driver (register addresses and values). The large registers tables can be an 
exception if you don't have access to the information, but for registers 
written manually, hardcoding numerical values isn't good.

> +
> +#define OV5640_MAX_CONTROLS 64
> +
> +enum ov5640_mode {
> +	ov5640_mode_MIN = 0,
> +	ov5640_mode_QCIF_176_144 = 0,
> +	ov5640_mode_QVGA_320_240,
> +	ov5640_mode_VGA_640_480,
> +	ov5640_mode_NTSC_720_480,
> +	ov5640_mode_PAL_720_576,
> +	ov5640_mode_XGA_1024_768,
> +	ov5640_mode_720P_1280_720,
> +	ov5640_mode_1080P_1920_1080,
> +	ov5640_mode_QSXGA_2592_1944,
> +	ov5640_num_modes,
> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/

Please add spaces after /* and before */.

Enumerated values should be all upper-case.

> +};
> +
> +enum ov5640_frame_rate {
> +	ov5640_15_fps,
> +	ov5640_30_fps
> +};
> +
> +static int ov5640_framerates[] = {
> +	[ov5640_15_fps] = 15,
> +	[ov5640_30_fps] = 30,
> +};
> +#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
> +
> +/* image size under 1280 * 960 are SUBSAMPLING
> + * image size upper 1280 * 960 are SCALING
> + */

The kernel multi-line comment style is

/*
 * text
 * text
 */

> +enum ov5640_downsize_mode {
> +	SUBSAMPLING,
> +	SCALING,
> +};
> +
> +struct reg_value {
> +	u16 reg_addr;
> +	u8 val;
> +	u8 mask;
> +	u32 delay_ms;
> +};
> +
> +struct ov5640_mode_info {
> +	enum ov5640_mode mode;
> +	enum ov5640_downsize_mode dn_mode;
> +	u32 width;
> +	u32 height;
> +	struct reg_value *init_data_ptr;
> +	u32 init_data_size;
> +};
> +
> +struct ov5640_dev {
> +	struct i2c_client *i2c_client;
> +	struct device *dev;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad;
> +	struct v4l2_ctrl_handler ctrl_hdl;
> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
> +	struct v4l2_mbus_framefmt fmt;
> +	struct v4l2_captureparm streamcap;
> +	struct clk *xclk; /* system clock to OV5640 */
> +	int xclk_freq;    /* requested xclk freq from devicetree */
> +
> +	enum ov5640_mode current_mode;

Store a (const) pointer to the corresponding ov5640_mode_info instead, it will 
simplify the code and allow you to get rid of the ov5640_mode enum.

> +	enum ov5640_frame_rate current_fr;
> +
> +	bool on;
> +	bool awb_on;
> +	bool agc_on;
> +
> +	/* cached control settings */
> +	int ctrl_cache[OV5640_MAX_CONTROLS];
> +
> +	struct gpio_desc *reset_gpio;
> +	struct gpio_desc *pwdn_gpio;
> +	struct gpio_desc *gp_gpio;
> +
> +	int prev_sysclk, prev_hts;
> +	int ae_low, ae_high, ae_target;

Can't these be unsigned int ?

> +
> +	struct regulator *io_regulator;
> +	struct regulator *core_regulator;
> +	struct regulator *analog_regulator;
> +	struct regulator *gpo_regulator;
> +};
> +
> +static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct ov5640_dev, sd);
> +}
> +
> +static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
> +{
> +	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
> +}
> +
> +struct ov5640_control {
> +	struct v4l2_queryctrl ctrl;
> +	int (*set)(struct ov5640_dev *sensor, int value);
> +};
> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
> +static void ov5640_reset(struct ov5640_dev *sensor);
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_gain(struct ov5640_dev *sensor);

No forward declarations please. You should reorder functions as needed (and of 
course still group related functions together).

> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
> +
> +	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
> +	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
> +	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
> +	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
> +	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
> +	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
> +	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
> +	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
> +	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
> +	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
> +	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
> +	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
> +	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
> +	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
> +	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
> +	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
> +	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
> +	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
> +	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
> +	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
> +	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
> +	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
> +	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
> +	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
> +	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
> +	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
> +	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
> +	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
> +	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
> +	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
> +	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
> +	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
> +	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
> +	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
> +	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
> +	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
> +	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
> +	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
> +	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
> +	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
> +	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
> +	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
> +	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
> +	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
> +	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
> +	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
> +	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
> +	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
> +	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
> +	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
> +	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
> +	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
> +	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
> +	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
> +	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
> +	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
> +	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
> +	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
> +	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
> +	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
> +	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
> +	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
> +	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
> +	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
> +	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
> +	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
> +	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
> +};

You only use the delay feature of the registers tables twice, once after 
writing the first two registers (to select the clock source and perform a 
software reset) and once at the very end. Remove it, write the first two 
registers manually in the code with a manual delay afterwards, and add another 
manual delay after writing the whole table.

I'm actually wondering whether you couldn't remove the 300ms delay at the end, 
the 50/60Hz control register (0x3c00) doesn't look like it needs a delay after 
being written.

[snip]

> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */

Don't turn the stream on in the init sequences, it should only be turned on 
from the .s_stream() operation.

> +};
> +
> +static struct ov5640_mode_info
> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {

There's very few differences between the 15fps and 30fps tables. It would be 
better if you could merge them, and manually write the registers that differ.

> +	{
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_15fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_15fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_15fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_15fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_15fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_15fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_15fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_15fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
> +		 ov5640_setting_15fps_QSXGA_2592_1944,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
> +	}, {
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_30fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_30fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_30fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_30fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_30fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_30fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_30fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_30fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
> +	},
> +};
> +
> +static int ov5640_probe(struct i2c_client *adapter,
> +			const struct i2c_device_id *device_id);
> +static int ov5640_remove(struct i2c_client *client);

No forward declarations please.

> +static int ov5640_init_slave_id(struct ov5640_dev *sensor)
> +{
> +	struct i2c_msg msg;
> +	u8 buf[4];
> +	int ret;
> +
> +	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
> +		return 0;
> +
> +	buf[0] = OV5640_SLAVE_ID >> 8;
> +	buf[1] = OV5640_SLAVE_ID & 0xff;
> +	buf[2] = sensor->i2c_client->addr << 1;
> +	msg.addr = OV5640_DEFAULT_SLAVE_ID;
> +	msg.flags = 0;
> +	msg.len = 3;
> +	msg.buf = buf;
> +
> +	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
> +	if (ret < 0) {
> +		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
> +{
> +	u8 buf[3] = {0};
> +	int ret;
> +
> +	buf[0] = reg >> 8;
> +	buf[1] = reg & 0xff;
> +	buf[2] = val;
> +
> +	ret = i2c_master_send(sensor->i2c_client, buf, 3);
> +	if (ret < 0) {
> +		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
> +			__func__, reg, val);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
> +{
> +	u8 reg_buf[2] = {0};
> +	u8 read_val = 0;
> +
> +	reg_buf[0] = reg >> 8;
> +	reg_buf[1] = reg & 0xff;
> +
> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
> +			__func__, reg);
> +		return -EIO;
> +	}
> +
> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
> +			__func__, reg, read_val);
> +		return -EIO;
> +	}

Wouldn't i2c_transfer() be more efficient here ?

> +	*val = read_val;
> +	return 0;
> +}
> +
> +#define OV5640_READ_REG(s, r, v) {				\
> +		ret = ov5640_read_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}

No. No. No no no. Don't ever return from a macro. Hiding the return makes 
following the code flow much more difficult, it's just asking for trouble.

And don't use externally defined variables (ret in this case), that's also 
asking for trouble.

> +#define OV5640_WRITE_REG(s, r, v) {				\
> +		ret = ov5640_write_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
> +{
> +	u8 hi, lo;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &hi);
> +	OV5640_READ_REG(sensor, reg+1, &lo);
> +
> +	*val = ((u16)hi << 8) | (u16)lo;
> +	return 0;
> +}
> +#define OV5640_READ_REG16(s, r, v) {				\
> +		ret = ov5640_read_reg16((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, reg, val >> 8);
> +	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
> +	return 0;
> +}
> +#define OV5640_WRITE_REG16(s, r, v) {				\
> +		ret = ov5640_write_reg16((s), (r), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
> +			  u8 mask, u8 val)
> +{
> +	u8 readval;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &readval);
> +
> +	readval &= ~mask;
> +	val &= mask;
> +	val |= readval;
> +
> +	OV5640_WRITE_REG(sensor, reg, val);
> +	return 0;
> +}
> +#define OV5640_MOD_REG(s, r, m, v) {				\
> +		ret = ov5640_mod_reg((s), (r), (m), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}

If you need to modify registers a lot, switch to regmap for register access. 
It will provide you with a cache, removing the need to read registers back 
from the device.

> +/* download ov5640 settings to sensor through i2c */
> +static int ov5640_load_regs(struct ov5640_dev *sensor,
> +			    struct reg_value *regs,
> +			    int size)

size is never negative.

> +{
> +	register u32 delay_ms = 0;
> +	register u16 reg_addr = 0;
> +	register u8 mask = 0;
> +	register u8 val = 0;

register ? The compiler is nowadays likely smarter than us when it comes to 
register allocation.

There's also no need to initialize the variables to 0.

> +	int i, ret;

And i isn't either.

> +
> +	for (i = 0; i < size; ++i, ++regs) {
> +		delay_ms = regs->delay_ms;
> +		reg_addr = regs->reg_addr;
> +		val = regs->val;
> +		mask = regs->mask;
> +
> +		if (mask) {
> +			OV5640_MOD_REG(sensor, reg_addr, mask, val);
> +		} else {
> +			OV5640_WRITE_REG(sensor, reg_addr, val);
> +		}
> +		if (delay_ms)
> +			usleep_range(1000*delay_ms, 1000*delay_ms+100);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
> +	return 0;
> +}
> +
> +static int ov5640_get_sysclk(struct ov5640_dev *sensor)
> +{
> +	 /* calculate sysclk */
> +	int xvclk = sensor->xclk_freq / 10000;
> +	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
> +	int sclk_rdiv_map[] = {1, 2, 4, 8};
> +	int bit_div2x = 1, sclk_rdiv, sysclk;
> +	u8 temp1, temp2;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3034, &temp1);
> +	temp2 = temp1 & 0x0f;
> +	if (temp2 == 8 || temp2 == 10)
> +		bit_div2x = temp2 / 2;
> +
> +	OV5640_READ_REG(sensor, 0x3035, &temp1);
> +	sysdiv = temp1>>4;
> +	if (sysdiv == 0)
> +		sysdiv = 16;
> +
> +	OV5640_READ_REG(sensor, 0x3036, &temp1);
> +	multiplier = temp1;
> +
> +	OV5640_READ_REG(sensor, 0x3037, &temp1);
> +	prediv = temp1 & 0x0f;
> +	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
> +
> +	OV5640_READ_REG(sensor, 0x3108, &temp1);
> +	temp2 = temp1 & 0x03;
> +	sclk_rdiv = sclk_rdiv_map[temp2];
> +
> +	VCO = xvclk * multiplier / prediv;
> +
> +	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
> +
> +	return sysclk;
> +}
> +
> +static int ov5640_set_night_mode(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u8 mode;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3a00, &mode);
> +	mode &= 0xfb;
> +	OV5640_WRITE_REG(sensor, 0x3a00, mode);
> +	return 0;
> +}
> +
> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u16 HTS;

Function names and variables are lower case.

> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380c, &HTS);
> +	return HTS;
> +}
> +
> +static int ov5640_get_VTS(struct ov5640_dev *sensor)
> +{
> +	u16 VTS;
> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380e, &VTS);
> +	return VTS;
> +}
> +
> +static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
> +	return 0;
> +}
> +
> +static int ov5640_get_light_freq(struct ov5640_dev *sensor)
> +{
> +	/* get banding filter value */
> +	u8 temp, temp1;
> +	int light_freq = 0;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3c01, &temp);
> +
> +	if (temp & 0x80) {
> +		/* manual */
> +		OV5640_READ_REG(sensor, 0x3c00, &temp1);
> +		if (temp1 & 0x04) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +			light_freq = 60;
> +		}
> +	} else {
> +		/* auto */
> +		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
> +		if (temp1 & 0x01) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +		}
> +	}
> +
> +	return light_freq;
> +}
> +
> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
> +{
> +	int prev_vts;
> +	int band_step60, max_band60, band_step50, max_band50;

Aren't these values unsigned ?

> +	int ret;
> +
> +	/* read preview PCLK */
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_sysclk = ret;
> +	/* read preview HTS */
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_hts = ret;
> +
> +	/* read preview VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_vts = ret;
> +
> +	/* calculate banding filter */
> +	/* 60Hz */
> +	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
> +	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
> +
> +	max_band60 = (int)((prev_vts-4)/band_step60);
> +	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
> +
> +	/* 50Hz */
> +	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
> +	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
> +
> +	max_band50 = (int)((prev_vts-4)/band_step50);
> +	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
> +{
> +	/* stable in high */
> +	int fast_high, fast_low;

Aren't these values unsigned ?

> +	int ret;
> +
> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
> +
> +	fast_high = sensor->ae_high<<1;

Missing spaces around <<

> +	if (fast_high > 255)
> +		fast_high = 255;
> +
> +	fast_low = sensor->ae_low >> 1;
> +
> +	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
> +
> +	return 0;
> +}
> +
> +static int ov5640_binning_on(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3821, &temp);
> +	temp &= 0xfe;
> +
> +	return temp ? 1 : 0;
> +}
> +
> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
> +{
> +	u8 temp, channel = sensor->ep.base.id;

The endpoint id isn't meant to select a virtual channel. V4L2 has no virtual 
channel API at the moment, you can hardcode the VC to 0 for now.

> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x4814, &temp);
> +	temp &= ~(3 << 6);
> +	temp |= (channel << 6);
> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
> +
> +	return 0;
> +}
> +
> +static enum ov5640_mode
> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
> +			 int width, int height)

How about using v4l2_find_nearest_format() ?

> +{
> +	int i;
> +
> +	for (i = ov5640_num_modes - 1; i >= 0; i--) {
> +		if (ov5640_mode_info_data[0][i].width <= width &&
> +		    ov5640_mode_info_data[0][i].height <= height)
> +			break;
> +	}
> +
> +	if (i < 0)
> +		i = 0;
> +
> +	return (enum ov5640_mode)i;
> +}
> +
> +/*
> + * sensor changes between scaling and subsampling, go through
> + * exposure calculation
> + */
> +static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
> +					    enum ov5640_frame_rate frame_rate,
> +					    enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	u8 average;
> +	int prev_shutter, prev_gain16;
> +	int cap_shutter, cap_gain16;
> +	int cap_sysclk, cap_hts, cap_vts;
> +	int light_freq, cap_bandfilt, cap_maxband;
> +	long cap_gain16_shutter;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* auto focus */
> +	/* ov5640_auto_focus();//if no af function, just skip it */
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read preview shutter */
> +	ret = ov5640_get_exposure(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_shutter = ret;
> +	ret = ov5640_binning_on(sensor);
> +	if (ret < 0)
> +		return ret;
> +	if (ret && mode != ov5640_mode_720P_1280_720 &&
> +	    mode != ov5640_mode_1080P_1920_1080)
> +		prev_shutter *= 2;
> +
> +	/* read preview gain */
> +	ret = ov5640_get_gain(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_gain16 = ret;
> +
> +	/* get average */
> +	OV5640_READ_REG(sensor, 0x56a1, &average);
> +
> +	/* turn off night mode for capture */
> +	ret = ov5640_set_night_mode(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* turn off overlay */
> +	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
> +	   just skip it */
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read capture VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_vts = ret;
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_hts = ret;
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_sysclk = ret;
> +
> +	/* calculate capture banding filter */
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	light_freq = ret;
> +
> +	if (light_freq == 60) {
> +		/* 60Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
> +	} else {
> +		/* 50Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts;
> +	}
> +	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
> +
> +	/* calculate capture shutter/gain16 */
> +	if (average > sensor->ae_low && average < sensor->ae_high) {
> +		/* in stable range */
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts *
> +			sensor->ae_target / average;
> +	} else {
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts;
> +	}
> +
> +	/* gain to shutter */
> +	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
> +		/* shutter < 1/100 */
> +		cap_shutter = cap_gain16_shutter / 16;
> +		if (cap_shutter < 1)
> +			cap_shutter = 1;
> +
> +		cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		if (cap_gain16 < 16)
> +			cap_gain16 = 16;
> +	} else {
> +		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
> +			/* exposure reach max */
> +			cap_shutter = cap_bandfilt * cap_maxband;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		} else {
> +			/* 1/100 < (cap_shutter = n/100) =< max */
> +			cap_shutter =
> +				((int)(cap_gain16_shutter / 16 / 
cap_bandfilt))
> +				* cap_bandfilt;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		}
> +	}
> +
> +	/* write capture gain */
> +	ret = ov5640_set_gain(sensor, cap_gain16);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* write capture shutter */
> +	if (cap_shutter > (cap_vts - 4)) {
> +		cap_vts = cap_shutter + 4;
> +		ret = ov5640_set_VTS(sensor, cap_vts);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	ret = ov5640_set_exposure(sensor, cap_shutter);
> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_stream(sensor, true);
> +}
> +
> +/*
> + * if sensor changes inside scaling or subsampling
> + * change mode directly
> + */
> +static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
> +				     enum ov5640_frame_rate frame_rate,
> +				     enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, true);

This will turn streaming on when you set the format, which I don't think is 
correct. You should only turn streaming on in the .s_stream() operation. The 
same comment applies to the previous function.

> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_agc(sensor, true);
> +}
> +
> +static int ov5640_change_mode(struct ov5640_dev *sensor,
> +			      enum ov5640_frame_rate frame_rate,
> +			      enum ov5640_mode mode,
> +			      enum ov5640_mode orig_mode)
> +{
> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
> +	    mode != ov5640_mode_INIT) {
> +		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
> +		return -EINVAL;
> +	}
> +
> +	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
> +	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
> +	if (mode == ov5640_mode_INIT) {
> +		mode_data = ov5640_init_setting_30fps_VGA;
> +		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
> +
> +		sensor->fmt.width = 640;
> +		sensor->fmt.height = 480;

Don't reset the format here. The format must be preserved across subdev 
open/close.

> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +		if (ret < 0)
> +			return ret;
> +
> +		mode_data = ov5640_setting_30fps_VGA_640_480;
> +		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
> +			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
> +		/* change between subsampling and scaling
> +		 * go through exposure calucation */
> +		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
> +							  mode);
> +	} else {
> +		/* change inside subsampling or scaling
> +		 * download firmware directly */
> +		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
> +	}
> +
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_bandingfilter(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_virtual_channel(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* restore controls */
> +	ov5640_restore_ctrls(sensor);
> +
> +	if (ret >= 0 && mode != ov5640_mode_INIT) {
> +		sensor->current_mode = mode;
> +		sensor->current_fr = frame_rate;
> +	}
> +
> +	return 0;
> +}
> +
> +/* restore the last set video mode after chip power-on */
> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
> +{
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	/* first we need to set some initial register values */
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				    ov5640_mode_INIT, ov5640_mode_INIT);

I wouldn't use ov5640_change_mode() here. You only need to apply the init 
register values, just call ov5640_load_regs(). The rest of the 
ov5640_change_mode() isn't needed, as you're calling it below. This will also 
allow you to get rid of ov5640_mode_INIT, simplifying the logic.

> +	if (ret < 0)
> +		return ret;
> +
> +	/* now restore the last capture mode */
> +	return ov5640_change_mode(sensor,
> +				  sensor->current_fr,
> +				  sensor->current_mode,
> +				  ov5640_mode_VGA_640_480);

You can fit that in fewer lines while still staying within the 80 columns 
limit.

> +}
> +
> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
> +{
> +	int ret;
> +
> +	if (sensor->io_regulator) {
> +		ret = regulator_enable(sensor->io_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->core_regulator) {
> +		ret = regulator_enable(sensor->core_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->gpo_regulator) {
> +		ret = regulator_enable(sensor->gpo_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->analog_regulator) {
> +		ret = regulator_enable(sensor->analog_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
> +			return ret;
> +		}
> +	}

Maybe you should use the bulk regulator API ?

> +
> +	return 0;
> +}
> +
> +static void ov5640_regulators_off(struct ov5640_dev *sensor)
> +{
> +	if (sensor->analog_regulator)
> +		regulator_disable(sensor->analog_regulator);
> +	if (sensor->core_regulator)
> +		regulator_disable(sensor->core_regulator);
> +	if (sensor->io_regulator)
> +		regulator_disable(sensor->io_regulator);
> +	if (sensor->gpo_regulator)
> +		regulator_disable(sensor->gpo_regulator);
> +}
> +
> +/* --------------- Subdev Operations --------------- */
> +
> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	int ret;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (on && !sensor->on) {

The .s_power() calls need to be ref-counted, similarly to how the regulator 
enable/disable calls work. See the mt9p031 sensor driver for an example.

> +		if (sensor->xclk)
> +			clk_prepare_enable(sensor->xclk);
> +
> +		ret = ov5640_regulators_on(sensor);
> +		if (ret)
> +			return ret;
> +
> +		ov5640_reset(sensor);
> +		ov5640_power(sensor, true);
> +
> +		ret = ov5640_init_slave_id(sensor);

Why is this needed ?

> +		if (ret)
> +			return ret;
> +
> +		ret = ov5640_restore_mode(sensor);
> +		if (ret)
> +			return ret;
> +
> +		/*
> +		 * NOTE: Freescale adds a long delay (600 msec) after
> +		 * powering up and programming a mode on the ov5640-mipi
> +		 * camera (search for "msec_wait4stable" in FSL's
> +		 * ov5640_mipi.c), which equivalently would need to go
> +		 * right here. If we run into MIPI CSI-2 receiver dphy
> +		 * ready timeouts, it might be a clue to add that delay
> +		 * here.
> +		 */
> +	} else if (!on && sensor->on) {
> +		ov5640_power(sensor, false);
> +
> +		ov5640_regulators_off(sensor);
> +
> +		if (sensor->xclk)
> +			clk_disable_unprepare(sensor->xclk);
> +	}
> +
> +	sensor->on = on;
> +
> +	return 0;
> +}
> +
> +static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_captureparm *cparm = &a->parm.capture;
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* This is the only case currently handled. */
> +	memset(a, 0, sizeof(*a));
> +	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	cparm->capability = sensor->streamcap.capability;
> +	cparm->timeperframe = sensor->streamcap.timeperframe;
> +	cparm->capturemode = sensor->streamcap.capturemode;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
> +	enum ov5640_frame_rate frame_rate;
> +	u32 tgt_fps;	/* target frames per secound */
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* Check that the new frame rate is allowed. */
> +	if ((timeperframe->numerator == 0) ||
> +	    (timeperframe->denominator == 0)) {
> +		timeperframe->denominator = DEFAULT_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps > MAX_FPS) {
> +		timeperframe->denominator = MAX_FPS;
> +		timeperframe->numerator = 1;
> +	} else if (tgt_fps < MIN_FPS) {
> +		timeperframe->denominator = MIN_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	/* Actual frame rate we use */
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps == 15)
> +		frame_rate = ov5640_15_fps;
> +	else if (tgt_fps == 30)
> +		frame_rate = ov5640_30_fps;
> +	else {
> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
> +			 tgt_fps);

Don't print an error message that is userspace-triggerable, we have enough 
ways for applications to flood the kernel log already :-)

> +		return -EINVAL;
> +	}
> +
> +	ret = ov5640_change_mode(sensor, frame_rate,
> +				 sensor->current_mode,
> +				 sensor->current_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	sensor->streamcap.timeperframe = *timeperframe;
> +
> +	return 0;
> +}
> +
> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	format->format = sensor->fmt;

You need to handle the TRY format here. You can have a look at the mt9p031 
driver for an example on how to do so.

> +
> +	return 0;
> +}
> +
> +static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
> +				   struct v4l2_mbus_framefmt *fmt,
> +				   enum ov5640_mode *new_mode)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
> +
> +	fmt->width = ov5640_mode_info_data[0][mode].width;
> +	fmt->height = ov5640_mode_info_data[0][mode].height;
> +	fmt->code = sensor->fmt.code;
> +
> +	if (new_mode)
> +		*new_mode = mode;
> +	return 0;
> +}
> +
> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode new_mode;
> +	int ret;
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
> +		if (ret)
> +			return ret;

You can move this code above the test to avoid the duplicated call below.

> +		cfg->try_fmt = format->format;

Please use the v4l2_subdev_get_try_format() function to get the try format, 
don't dereference cfg directly.

> +		return 0;
> +	}
> +
> +	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
> +	if (ret)
> +		return ret;
> +
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				 new_mode, sensor->current_mode);
> +	if (ret >= 0)
> +		sensor->fmt = format->format;
> +
> +	return ret;
> +}
> +
> +
> +/*
> + * Sensor Controls.
> + */
> +
> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);

According to the kernel coding style, you need curly braces around the else 
branch if you use them around the if branch.

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
> +		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);

Ditto

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
> +		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);

Ditto

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	sensor->awb_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
> +	return 0;
> +}
> +
> +static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +#if 0

No compiled-out code please.

> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +#endif
> +
> +static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
> +{
> +	u16 max_exp = 0;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
> +	if (value < max_exp) {
> +		u32 exp = value << 4;
> +
> +		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
> +	}
> +
> +	return 0;
> +}
> +
> +/* read exposure, in number of line periods */
> +static int ov5640_get_exposure(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int exp, ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG(sensor, 0x3500, &temp);
> +	exp = ((int)temp & 0x0f) << 16;
> +	OV5640_READ_REG(sensor, 0x3501, &temp);
> +	exp |= ((int)temp << 8);
> +	OV5640_READ_REG(sensor, 0x3502, &temp);
> +	exp |= (int)temp;
> +
> +	return exp >> 4;
> +}
> +
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	/* this enables/disables both AEC and AGC */
> +	sensor->agc_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;

You can create a cluster with the autogain and manual gain controls 
(v4l2_ctrl_auto_cluster) to have the control framework disabling the manual 
control automatically when autogain is enabled.

> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
> +	return 0;
> +}
> +
> +static int ov5640_get_gain(struct ov5640_dev *sensor)
> +{
> +	u16 gain;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
> +
> +	return gain & 0x3ff;
> +}
> +
> +#if 0
> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
> +	return 0;
> +}
> +#endif

You can use a control to expose the test pattern function, it's quite useful.

> +static struct ov5640_control ov5640_ctrls[] = {
> +	{
> +		.set = ov5640_set_agc,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTOGAIN,
> +			.name = "Auto Gain/Exposure Control",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_exposure,
> +		.ctrl = {
> +			.id = V4L2_CID_EXPOSURE,
> +			.name = "Exposure",
> +			.minimum = 0,
> +			.maximum = 65535,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_gain,
> +		.ctrl = {
> +			.id = V4L2_CID_GAIN,
> +			.name = "Gain",
> +			.minimum = 0,
> +			.maximum = 1023,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_hue,
> +		.ctrl = {
> +			.id = V4L2_CID_HUE,
> +			.name = "Hue",
> +			.minimum = 0,
> +			.maximum = 359,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_contrast,
> +		.ctrl = {
> +			.id = V4L2_CID_CONTRAST,
> +			.name = "Contrast",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_saturation,
> +		.ctrl = {
> +			.id = V4L2_CID_SATURATION,
> +			.name = "Saturation",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 64,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_awb,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTO_WHITE_BALANCE,
> +			.name = "Auto White Balance",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_red_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_RED_BALANCE,
> +			.name = "Red Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_blue_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_BLUE_BALANCE,
> +			.name = "Blue Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	},
> +};
> +#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
> +
> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
> +{
> +	struct ov5640_control *ret = NULL;
> +	int i;

i is never negative, it should be unsigned int.

> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		if (id == ov5640_ctrls[i].ctrl.id) {
> +			ret = &ov5640_ctrls[i];
> +			break;
> +		}
> +	}
> +
> +	if (ret && index)
> +		*index = i;
> +	return ret;
> +}
> +
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;
> +	int i;

i is never negative, it should be unsigned int.

> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +		c->set(sensor, sensor->ctrl_cache[i]);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
> +	struct ov5640_control *c;
> +	int ret = 0;
> +	int i;
> +
> +	c = ov5640_get_ctrl(ctrl->id, &i);

You can inline the function call here as it's not used anywhere else.

> +	if (!c)
> +		return -EINVAL;
> +
> +	ret = c->set(sensor, ctrl->val);
> +	/* update cached value if no error */
> +	if (!ret)
> +		sensor->ctrl_cache[i] = ctrl->val;
> +
> +	return ret;
> +}
> +
> +static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
> +	.s_ctrl = ov5640_s_ctrl,
> +};
> +
> +static int ov5640_init_controls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;

I would name this ctrl or control, c can be a bit confusing. You can also 
declare it inside the loop.

> +	int i;

i can never be negative, you can make it an unsigned int.

> +
> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +
> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
> +				  c->ctrl.id, c->ctrl.minimum, c-
>ctrl.maximum,
> +				  c->ctrl.step, c->ctrl.default_value);
> +	}
> +
> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
> +	if (sensor->ctrl_hdl.error) {
> +		int err = sensor->ctrl_hdl.error;
> +
> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);

I'm not sure this brings much value.

> +		return err;
> +	}
> +	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);

You shouldn't call this function here for the reason explained below.

> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_frame_size_enum *fse)
> +{
> +	if (fse->pad != 0)
> +		return -EINVAL;
> +	if (fse->index >= ov5640_num_modes)
> +		return -EINVAL;
> +
> +	fse->min_width = fse->max_width =
> +		ov5640_mode_info_data[0][fse->index].width;
> +	fse->min_height = fse->max_height =
> +		ov5640_mode_info_data[0][fse->index].height;
> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_interval(
> +	struct v4l2_subdev *sd,
> +	struct v4l2_subdev_pad_config *cfg,
> +	struct v4l2_subdev_frame_interval_enum *fie)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	if (fie->pad != 0)
> +		return -EINVAL;
> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
> +		return -EINVAL;
> +
> +	if (fie->width == 0 || fie->height == 0)
> +		return -EINVAL;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);

You should find an exact mode here, not the nearest one. If with and height 
don't match, return -EINVAL. That will replace with above width == 0 and 
height == 0 test.

> +	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
> +		return -EINVAL;
> +
> +	fie->interval.numerator = 1;
> +	fie->interval.denominator = ov5640_framerates[fie->index];
> +
> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
> +		fie->width, fie->height, fie->index, fie-
>interval.denominator);

I'm not sure this is very useful, you can use ftrace if you want to trace 
function calls.

> +	return 0;
> +}
> +
> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
> +			    u32 output, u32 config)
> +{
> +	return (input != 0) ? -EINVAL : 0;
> +}

The g_input_status and s_routing subdev operations are not mandatory, you 
don't have to implement them as the sensor doesn't have multiple inputs.

[snip]

> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

You can use ftrace to trace function calls, there's no need to add debugging 
statements that duplicate the functionality.

> +	return ov5640_set_stream(sensor, enable);
> +}
> +
> +static struct v4l2_subdev_core_ops ov5640_core_ops = {
> +	.s_power = ov5640_s_power,
> +};
> +
> +static struct v4l2_subdev_video_ops ov5640_video_ops = {
> +	.s_parm = ov5640_s_parm,
> +	.g_parm = ov5640_g_parm,
> +	.g_input_status = ov5640_g_input_status,
> +	.s_routing = ov5640_s_routing,
> +	.g_mbus_config  = ov5640_g_mbus_config,
> +	.s_stream = ov5640_s_stream,
> +};
> +
> +static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
> +	.enum_mbus_code = ov5640_enum_mbus_code,
> +	.get_fmt = ov5640_get_fmt,
> +	.set_fmt = ov5640_set_fmt,
> +	.enum_frame_size = ov5640_enum_frame_size,
> +	.enum_frame_interval = ov5640_enum_frame_interval,
> +};
> +
> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
> +	.core = &ov5640_core_ops,
> +	.video = &ov5640_video_ops,
> +	.pad = &ov5640_pad_ops,
> +};

All these structures should be static const.

> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
> +{
> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
> +}
> +
> +static void ov5640_reset(struct ov5640_dev *sensor)
> +{
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +
> +	/* camera power cycle */
> +	ov5640_power(sensor, false);
> +	usleep_range(5000, 10000);
> +	ov5640_power(sensor, true);
> +	usleep_range(5000, 10000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 1);
> +	usleep_range(1000, 2000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +	usleep_range(5000, 10000);
> +}
> +
> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
> +{
> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
> +	if (!IS_ERR(sensor->io_regulator)) {
> +		regulator_set_voltage(sensor->io_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_IO,
> +				      OV5640_VOLTAGE_DIGITAL_IO);
> +	} else {
> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
> +			__func__);
> +		sensor->io_regulator = NULL;

How about making the power supplies mandatory in DT instead ? They are 
mandatory after all, if they're not controllable DT should just declare fixed 
supplies.

> +	}
> +
> +	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
> +	if (!IS_ERR(sensor->core_regulator)) {
> +		regulator_set_voltage(sensor->core_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_CORE,
> +				      OV5640_VOLTAGE_DIGITAL_CORE);
> +	} else {
> +		sensor->core_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
> +			__func__);
> +	}
> +
> +	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
> +	if (!IS_ERR(sensor->analog_regulator)) {
> +		regulator_set_voltage(sensor->analog_regulator,
> +				      OV5640_VOLTAGE_ANALOG,
> +				      OV5640_VOLTAGE_ANALOG);
> +	} else {
> +		sensor->analog_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
> +			__func__);
> +	}
> +}
> +
> +static int ov5640_probe(struct i2c_client *client,
> +			const struct i2c_device_id *id)
> +{
> +	struct device *dev = &client->dev;
> +	struct device_node *endpoint;
> +	struct ov5640_dev *sensor;
> +	int i, xclk, ret;

i and xclk are never negative, you can make them unsigned int.

> +
> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);

Please use sizeof(*variable) instead of sizeof(type).

devm_kzalloc() doesn't play nicely with dynamic removal of devices. We're in 
the process of fixing related race conditions in the media subsystem. In order 
not to make the problem worse, please use kzalloc() instead.

> +	if (!sensor)
> +		return -ENOMEM;
> +
> +	sensor->i2c_client = client;
> +	sensor->dev = dev;

Do you really need to store both i2c_client and dev, given that the latter is 
just &client->dev ?

> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
> +	sensor->fmt.width = 640;
> +	sensor->fmt.height = 480;
> +	sensor->fmt.field = V4L2_FIELD_NONE;
> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
> +					   V4L2_CAP_TIMEPERFRAME;

Please fix the indentation.

> +	sensor->streamcap.capturemode = 0;
> +	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
> +	sensor->streamcap.timeperframe.numerator = 1;
> +
> +	sensor->current_mode = ov5640_mode_VGA_640_480;
> +	sensor->current_fr = ov5640_30_fps;
> +
> +	sensor->ae_target = 52;
> +
> +	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
> +	if (!endpoint) {
> +		dev_err(dev, "endpoint node not found\n");
> +		return -EINVAL;
> +	}
> +
> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
> +		return -EINVAL;

You're leaking endpoint here. You should move the of_node_put() call right 
after the v4l2_of_parse_endpoint() call.

> +	}
> +	of_node_put(endpoint);
> +
> +	/* get system clock (xclk) frequency */
> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);

Instead of adding a custom DT property for this, use assigned-clock-rates. You 
won't need to set it manually in the driver, and can verify its frequency with 
clk_get_rate().

> +	if (!ret) {
> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {

Are your register tables above independent of the clock frequency ? You should 
ideally compute register values at runtime instead of hardcoding them, but 
given the lack of information from Omnivision I understand this isn't 
possible. You thus need to be stricter here and reject any value other than 
the nominal frequency.

> +			dev_err(dev, "invalid xclk frequency\n");
> +			return -EINVAL;
> +		}
> +		sensor->xclk_freq = xclk;
> +	}
> +
> +	/* get system clock (xclk) */
> +	sensor->xclk = devm_clk_get(dev, "xclk");
> +	if (!IS_ERR(sensor->xclk)) {
> +		if (!sensor->xclk_freq) {
> +			dev_err(dev, "xclk requires xclk frequency!\n");
> +			return -EINVAL;
> +		}
> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
> +	} else {
> +		/* assume system clock enabled by default */
> +		sensor->xclk = NULL;

Please don't. The clock should be mandatory.

> +	}
> +
> +	/* request power down pin */
> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);

Are the GPIOs mandatory or optional ? Can a system tie some of them to ground 
or a voltage rail, or do they need to always be manually controllable ?

> +	if (IS_ERR(sensor->pwdn_gpio)) {
> +		dev_err(dev, "request for power down gpio failed\n");
> +		return PTR_ERR(sensor->pwdn_gpio);
> +	}
> +
> +	/* request reset pin */
> +	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
> +	if (IS_ERR(sensor->reset_gpio)) {
> +		dev_err(dev, "request for reset gpio failed\n");
> +		return PTR_ERR(sensor->reset_gpio);
> +	}
> +
> +	/* initialize the cached controls to their defaults */
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		struct ov5640_control *c = &ov5640_ctrls[i];
> +
> +		sensor->ctrl_cache[i] = c->ctrl.default_value;
> +	}
> +	sensor->awb_on = sensor->agc_on = true;
> +
> +	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
> +
> +	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
> +	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> +	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
> +	if (ret)
> +		return ret;
> +
> +	ov5640_get_regulators(sensor);
> +
> +	ret = ov5640_s_power(&sensor->sd, 1);
> +	if (ret)
> +		goto entity_cleanup;
> +	ret = ov5640_init_controls(sensor);
> +	if (ret)
> +		goto power_off;
> +
> +	ret = ov5640_s_power(&sensor->sd, 0);
> +	if (ret)
> +		goto free_ctrls;

Writing the controls here is pointless, as powering the chip down will lose 
all the values. You shouldn't call v4l2_ctrl_handler_setup() in 
ov5640_init_controls(), and you can then remove the ov5640_s_power() calls 
here.

> +	ret = v4l2_async_register_subdev(&sensor->sd);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +power_off:
> +	ov5640_s_power(&sensor->sd, 0);
> +entity_cleanup:
> +	media_entity_cleanup(&sensor->sd.entity);
> +	ov5640_regulators_off(sensor);

Won't ov5640_s_power(0) already disable the regulators ?

> +	return ret;
> +}
> +
> +/*!
> + * ov5640 I2C detach function
> + *
> + * @param client            struct i2c_client *
> + * @return  Error code indicating success or failure
> + */

That's not the kerneldoc comment style. Given that this is the only documented 
function, and that the comment is completely useless, you can just drop it.

> +static int ov5640_remove(struct i2c_client *client)
> +{
> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	ov5640_regulators_off(sensor);
> +
> +	v4l2_async_unregister_subdev(&sensor->sd);
> +	media_entity_cleanup(&sensor->sd.entity);
> +	v4l2_device_unregister_subdev(sd);

This function is called by v4l2_async_unregister_subdev(), there's no need to 
duplicate it.

> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +	return 0;
> +}

[snip]

> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");

MODULE_AUTHOR isn't a synonym for copyright ownership. I don't think you 
should add Freescale as an author. If you know who wrote the original code you 
can list that developer explicitly.

> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
> +MODULE_LICENSE("GPL");
> +MODULE_VERSION("1.0");

Version numbers are never updated. I wouldn't bother adding one.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-02-02 10:36     ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-02 10:36 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve

Hi Steve,

Thank you for the patch. Many of the comments below apply to the ov5642 driver 
too, please take them into account when reworking patch 23/24.

On Friday 06 Jan 2017 18:11:40 Steve Longerbeam wrote:
> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
> branch, modified heavily to bring forward to latest interfaces and code
> cleanup.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig       |    8 +
>  drivers/staging/media/imx/Makefile      |    2 +
>  drivers/staging/media/imx/ov5640-mipi.c | 2348 ++++++++++++++++++++++++++++

You're missing DT bindings.

The driver should go to drivers/media/i2c/ as it should not be specific to the 
i.MX6, and you can just call it ov5640.c.

>  3 files changed, 2358 insertions(+)
>  create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig
> b/drivers/staging/media/imx/Kconfig index ce2d2c8..09f373d 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>  	---help---
>  	  A video4linux camera capture driver for i.MX5/6.
> 
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA

The sensor driver is generic, it shouldn't depend on IMX. It should however 
depend on at least I2C and OF by the look of it.

> +       select IMX_MIPI_CSI2
> +       default y
> +       ---help---
> +         MIPI CSI-2 OV5640 Camera support.
> +
>  endmenu
>  endif

[snip]

> diff --git a/drivers/staging/media/imx/ov5640-mipi.c
> b/drivers/staging/media/imx/ov5640-mipi.c new file mode 100644
> index 0000000..54647a7
> --- /dev/null
> +++ b/drivers/staging/media/imx/ov5640-mipi.c
> @@ -0,0 +1,2348 @@
> +/*
> + * Copyright (c) 2014 Mentor Graphics Inc.
> + * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights
> Reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/init.h>
> +#include <linux/slab.h>
> +#include <linux/ctype.h>
> +#include <linux/types.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/i2c.h>
> +#include <linux/of_device.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/regulator/consumer.h>
> +#include <linux/clk.h>
> +#include <linux/clk-provider.h>
> +#include <linux/clkdev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>

Pet peeve of mine, please sort the headers alphabetically. It makes it easier 
to locate duplicated.

> +
> +#define OV5640_VOLTAGE_ANALOG               2800000
> +#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
> +#define OV5640_VOLTAGE_DIGITAL_IO           1800000
> +
> +#define MIN_FPS 15
> +#define MAX_FPS 30
> +#define DEFAULT_FPS 30
> +
> +/* min/typical/max system clock (xclk) frequencies */
> +#define OV5640_XCLK_MIN  6000000
> +#define OV5640_XCLK_TYP 24000000
> +#define OV5640_XCLK_MAX 54000000
> +
> +/* min/typical/max pixel clock (mclk) frequencies */
> +#define OV5640_MCLK_MIN 48000000
> +#define OV5640_MCLK_TYP 48000000
> +#define OV5640_MCLK_MAX 96000000
> +
> +#define OV5640_CHIP_ID  0x300A
> +#define OV5640_SLAVE_ID 0x3100
> +#define OV5640_DEFAULT_SLAVE_ID 0x3c

You're mixing lower-case and upper-case hex constants. Let's pick one. Kernel 
code usually favours lower-case.

Please define macros for all the other numerical constants you use in the 
driver (register addresses and values). The large registers tables can be an 
exception if you don't have access to the information, but for registers 
written manually, hardcoding numerical values isn't good.

> +
> +#define OV5640_MAX_CONTROLS 64
> +
> +enum ov5640_mode {
> +	ov5640_mode_MIN = 0,
> +	ov5640_mode_QCIF_176_144 = 0,
> +	ov5640_mode_QVGA_320_240,
> +	ov5640_mode_VGA_640_480,
> +	ov5640_mode_NTSC_720_480,
> +	ov5640_mode_PAL_720_576,
> +	ov5640_mode_XGA_1024_768,
> +	ov5640_mode_720P_1280_720,
> +	ov5640_mode_1080P_1920_1080,
> +	ov5640_mode_QSXGA_2592_1944,
> +	ov5640_num_modes,
> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/

Please add spaces after /* and before */.

Enumerated values should be all upper-case.

> +};
> +
> +enum ov5640_frame_rate {
> +	ov5640_15_fps,
> +	ov5640_30_fps
> +};
> +
> +static int ov5640_framerates[] = {
> +	[ov5640_15_fps] = 15,
> +	[ov5640_30_fps] = 30,
> +};
> +#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
> +
> +/* image size under 1280 * 960 are SUBSAMPLING
> + * image size upper 1280 * 960 are SCALING
> + */

The kernel multi-line comment style is

/*
 * text
 * text
 */

> +enum ov5640_downsize_mode {
> +	SUBSAMPLING,
> +	SCALING,
> +};
> +
> +struct reg_value {
> +	u16 reg_addr;
> +	u8 val;
> +	u8 mask;
> +	u32 delay_ms;
> +};
> +
> +struct ov5640_mode_info {
> +	enum ov5640_mode mode;
> +	enum ov5640_downsize_mode dn_mode;
> +	u32 width;
> +	u32 height;
> +	struct reg_value *init_data_ptr;
> +	u32 init_data_size;
> +};
> +
> +struct ov5640_dev {
> +	struct i2c_client *i2c_client;
> +	struct device *dev;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad;
> +	struct v4l2_ctrl_handler ctrl_hdl;
> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
> +	struct v4l2_mbus_framefmt fmt;
> +	struct v4l2_captureparm streamcap;
> +	struct clk *xclk; /* system clock to OV5640 */
> +	int xclk_freq;    /* requested xclk freq from devicetree */
> +
> +	enum ov5640_mode current_mode;

Store a (const) pointer to the corresponding ov5640_mode_info instead, it will 
simplify the code and allow you to get rid of the ov5640_mode enum.

> +	enum ov5640_frame_rate current_fr;
> +
> +	bool on;
> +	bool awb_on;
> +	bool agc_on;
> +
> +	/* cached control settings */
> +	int ctrl_cache[OV5640_MAX_CONTROLS];
> +
> +	struct gpio_desc *reset_gpio;
> +	struct gpio_desc *pwdn_gpio;
> +	struct gpio_desc *gp_gpio;
> +
> +	int prev_sysclk, prev_hts;
> +	int ae_low, ae_high, ae_target;

Can't these be unsigned int ?

> +
> +	struct regulator *io_regulator;
> +	struct regulator *core_regulator;
> +	struct regulator *analog_regulator;
> +	struct regulator *gpo_regulator;
> +};
> +
> +static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct ov5640_dev, sd);
> +}
> +
> +static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
> +{
> +	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
> +}
> +
> +struct ov5640_control {
> +	struct v4l2_queryctrl ctrl;
> +	int (*set)(struct ov5640_dev *sensor, int value);
> +};
> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
> +static void ov5640_reset(struct ov5640_dev *sensor);
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_gain(struct ov5640_dev *sensor);

No forward declarations please. You should reorder functions as needed (and of 
course still group related functions together).

> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
> +
> +	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
> +	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
> +	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
> +	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
> +	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
> +	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
> +	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
> +	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
> +	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
> +	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
> +	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
> +	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
> +	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
> +	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
> +	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
> +	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
> +	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
> +	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
> +	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
> +	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
> +	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
> +	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
> +	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
> +	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
> +	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
> +	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
> +	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
> +	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
> +	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
> +	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
> +	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
> +	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
> +	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
> +	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
> +	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
> +	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
> +	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
> +	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
> +	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
> +	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
> +	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
> +	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
> +	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
> +	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
> +	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
> +	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
> +	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
> +	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
> +	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
> +	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
> +	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
> +	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
> +	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
> +	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
> +	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
> +	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
> +	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
> +	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
> +	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
> +	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
> +	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
> +	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
> +	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
> +	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
> +	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
> +	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
> +	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
> +};

You only use the delay feature of the registers tables twice, once after 
writing the first two registers (to select the clock source and perform a 
software reset) and once at the very end. Remove it, write the first two 
registers manually in the code with a manual delay afterwards, and add another 
manual delay after writing the whole table.

I'm actually wondering whether you couldn't remove the 300ms delay at the end, 
the 50/60Hz control register (0x3c00) doesn't look like it needs a delay after 
being written.

[snip]

> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */

Don't turn the stream on in the init sequences, it should only be turned on 
from the .s_stream() operation.

> +};
> +
> +static struct ov5640_mode_info
> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {

There's very few differences between the 15fps and 30fps tables. It would be 
better if you could merge them, and manually write the registers that differ.

> +	{
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_15fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_15fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_15fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_15fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_15fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_15fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_15fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_15fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
> +		 ov5640_setting_15fps_QSXGA_2592_1944,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
> +	}, {
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_30fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_30fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_30fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_30fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_30fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_30fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_30fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_30fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
> +	},
> +};
> +
> +static int ov5640_probe(struct i2c_client *adapter,
> +			const struct i2c_device_id *device_id);
> +static int ov5640_remove(struct i2c_client *client);

No forward declarations please.

> +static int ov5640_init_slave_id(struct ov5640_dev *sensor)
> +{
> +	struct i2c_msg msg;
> +	u8 buf[4];
> +	int ret;
> +
> +	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
> +		return 0;
> +
> +	buf[0] = OV5640_SLAVE_ID >> 8;
> +	buf[1] = OV5640_SLAVE_ID & 0xff;
> +	buf[2] = sensor->i2c_client->addr << 1;
> +	msg.addr = OV5640_DEFAULT_SLAVE_ID;
> +	msg.flags = 0;
> +	msg.len = 3;
> +	msg.buf = buf;
> +
> +	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
> +	if (ret < 0) {
> +		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
> +{
> +	u8 buf[3] = {0};
> +	int ret;
> +
> +	buf[0] = reg >> 8;
> +	buf[1] = reg & 0xff;
> +	buf[2] = val;
> +
> +	ret = i2c_master_send(sensor->i2c_client, buf, 3);
> +	if (ret < 0) {
> +		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
> +			__func__, reg, val);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
> +{
> +	u8 reg_buf[2] = {0};
> +	u8 read_val = 0;
> +
> +	reg_buf[0] = reg >> 8;
> +	reg_buf[1] = reg & 0xff;
> +
> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
> +			__func__, reg);
> +		return -EIO;
> +	}
> +
> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
> +			__func__, reg, read_val);
> +		return -EIO;
> +	}

Wouldn't i2c_transfer() be more efficient here ?

> +	*val = read_val;
> +	return 0;
> +}
> +
> +#define OV5640_READ_REG(s, r, v) {				\
> +		ret = ov5640_read_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}

No. No. No no no. Don't ever return from a macro. Hiding the return makes 
following the code flow much more difficult, it's just asking for trouble.

And don't use externally defined variables (ret in this case), that's also 
asking for trouble.

> +#define OV5640_WRITE_REG(s, r, v) {				\
> +		ret = ov5640_write_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
> +{
> +	u8 hi, lo;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &hi);
> +	OV5640_READ_REG(sensor, reg+1, &lo);
> +
> +	*val = ((u16)hi << 8) | (u16)lo;
> +	return 0;
> +}
> +#define OV5640_READ_REG16(s, r, v) {				\
> +		ret = ov5640_read_reg16((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, reg, val >> 8);
> +	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
> +	return 0;
> +}
> +#define OV5640_WRITE_REG16(s, r, v) {				\
> +		ret = ov5640_write_reg16((s), (r), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
> +			  u8 mask, u8 val)
> +{
> +	u8 readval;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &readval);
> +
> +	readval &= ~mask;
> +	val &= mask;
> +	val |= readval;
> +
> +	OV5640_WRITE_REG(sensor, reg, val);
> +	return 0;
> +}
> +#define OV5640_MOD_REG(s, r, m, v) {				\
> +		ret = ov5640_mod_reg((s), (r), (m), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}

If you need to modify registers a lot, switch to regmap for register access. 
It will provide you with a cache, removing the need to read registers back 
from the device.

> +/* download ov5640 settings to sensor through i2c */
> +static int ov5640_load_regs(struct ov5640_dev *sensor,
> +			    struct reg_value *regs,
> +			    int size)

size is never negative.

> +{
> +	register u32 delay_ms = 0;
> +	register u16 reg_addr = 0;
> +	register u8 mask = 0;
> +	register u8 val = 0;

register ? The compiler is nowadays likely smarter than us when it comes to 
register allocation.

There's also no need to initialize the variables to 0.

> +	int i, ret;

And i isn't either.

> +
> +	for (i = 0; i < size; ++i, ++regs) {
> +		delay_ms = regs->delay_ms;
> +		reg_addr = regs->reg_addr;
> +		val = regs->val;
> +		mask = regs->mask;
> +
> +		if (mask) {
> +			OV5640_MOD_REG(sensor, reg_addr, mask, val);
> +		} else {
> +			OV5640_WRITE_REG(sensor, reg_addr, val);
> +		}
> +		if (delay_ms)
> +			usleep_range(1000*delay_ms, 1000*delay_ms+100);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
> +	return 0;
> +}
> +
> +static int ov5640_get_sysclk(struct ov5640_dev *sensor)
> +{
> +	 /* calculate sysclk */
> +	int xvclk = sensor->xclk_freq / 10000;
> +	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
> +	int sclk_rdiv_map[] = {1, 2, 4, 8};
> +	int bit_div2x = 1, sclk_rdiv, sysclk;
> +	u8 temp1, temp2;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3034, &temp1);
> +	temp2 = temp1 & 0x0f;
> +	if (temp2 == 8 || temp2 == 10)
> +		bit_div2x = temp2 / 2;
> +
> +	OV5640_READ_REG(sensor, 0x3035, &temp1);
> +	sysdiv = temp1>>4;
> +	if (sysdiv == 0)
> +		sysdiv = 16;
> +
> +	OV5640_READ_REG(sensor, 0x3036, &temp1);
> +	multiplier = temp1;
> +
> +	OV5640_READ_REG(sensor, 0x3037, &temp1);
> +	prediv = temp1 & 0x0f;
> +	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
> +
> +	OV5640_READ_REG(sensor, 0x3108, &temp1);
> +	temp2 = temp1 & 0x03;
> +	sclk_rdiv = sclk_rdiv_map[temp2];
> +
> +	VCO = xvclk * multiplier / prediv;
> +
> +	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
> +
> +	return sysclk;
> +}
> +
> +static int ov5640_set_night_mode(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u8 mode;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3a00, &mode);
> +	mode &= 0xfb;
> +	OV5640_WRITE_REG(sensor, 0x3a00, mode);
> +	return 0;
> +}
> +
> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u16 HTS;

Function names and variables are lower case.

> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380c, &HTS);
> +	return HTS;
> +}
> +
> +static int ov5640_get_VTS(struct ov5640_dev *sensor)
> +{
> +	u16 VTS;
> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380e, &VTS);
> +	return VTS;
> +}
> +
> +static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
> +	return 0;
> +}
> +
> +static int ov5640_get_light_freq(struct ov5640_dev *sensor)
> +{
> +	/* get banding filter value */
> +	u8 temp, temp1;
> +	int light_freq = 0;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3c01, &temp);
> +
> +	if (temp & 0x80) {
> +		/* manual */
> +		OV5640_READ_REG(sensor, 0x3c00, &temp1);
> +		if (temp1 & 0x04) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +			light_freq = 60;
> +		}
> +	} else {
> +		/* auto */
> +		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
> +		if (temp1 & 0x01) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +		}
> +	}
> +
> +	return light_freq;
> +}
> +
> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
> +{
> +	int prev_vts;
> +	int band_step60, max_band60, band_step50, max_band50;

Aren't these values unsigned ?

> +	int ret;
> +
> +	/* read preview PCLK */
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_sysclk = ret;
> +	/* read preview HTS */
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_hts = ret;
> +
> +	/* read preview VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_vts = ret;
> +
> +	/* calculate banding filter */
> +	/* 60Hz */
> +	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
> +	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
> +
> +	max_band60 = (int)((prev_vts-4)/band_step60);
> +	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
> +
> +	/* 50Hz */
> +	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
> +	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
> +
> +	max_band50 = (int)((prev_vts-4)/band_step50);
> +	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
> +{
> +	/* stable in high */
> +	int fast_high, fast_low;

Aren't these values unsigned ?

> +	int ret;
> +
> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
> +
> +	fast_high = sensor->ae_high<<1;

Missing spaces around <<

> +	if (fast_high > 255)
> +		fast_high = 255;
> +
> +	fast_low = sensor->ae_low >> 1;
> +
> +	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
> +
> +	return 0;
> +}
> +
> +static int ov5640_binning_on(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3821, &temp);
> +	temp &= 0xfe;
> +
> +	return temp ? 1 : 0;
> +}
> +
> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
> +{
> +	u8 temp, channel = sensor->ep.base.id;

The endpoint id isn't meant to select a virtual channel. V4L2 has no virtual 
channel API at the moment, you can hardcode the VC to 0 for now.

> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x4814, &temp);
> +	temp &= ~(3 << 6);
> +	temp |= (channel << 6);
> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
> +
> +	return 0;
> +}
> +
> +static enum ov5640_mode
> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
> +			 int width, int height)

How about using v4l2_find_nearest_format() ?

> +{
> +	int i;
> +
> +	for (i = ov5640_num_modes - 1; i >= 0; i--) {
> +		if (ov5640_mode_info_data[0][i].width <= width &&
> +		    ov5640_mode_info_data[0][i].height <= height)
> +			break;
> +	}
> +
> +	if (i < 0)
> +		i = 0;
> +
> +	return (enum ov5640_mode)i;
> +}
> +
> +/*
> + * sensor changes between scaling and subsampling, go through
> + * exposure calculation
> + */
> +static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
> +					    enum ov5640_frame_rate frame_rate,
> +					    enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	u8 average;
> +	int prev_shutter, prev_gain16;
> +	int cap_shutter, cap_gain16;
> +	int cap_sysclk, cap_hts, cap_vts;
> +	int light_freq, cap_bandfilt, cap_maxband;
> +	long cap_gain16_shutter;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* auto focus */
> +	/* ov5640_auto_focus();//if no af function, just skip it */
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read preview shutter */
> +	ret = ov5640_get_exposure(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_shutter = ret;
> +	ret = ov5640_binning_on(sensor);
> +	if (ret < 0)
> +		return ret;
> +	if (ret && mode != ov5640_mode_720P_1280_720 &&
> +	    mode != ov5640_mode_1080P_1920_1080)
> +		prev_shutter *= 2;
> +
> +	/* read preview gain */
> +	ret = ov5640_get_gain(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_gain16 = ret;
> +
> +	/* get average */
> +	OV5640_READ_REG(sensor, 0x56a1, &average);
> +
> +	/* turn off night mode for capture */
> +	ret = ov5640_set_night_mode(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* turn off overlay */
> +	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
> +	   just skip it */
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read capture VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_vts = ret;
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_hts = ret;
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_sysclk = ret;
> +
> +	/* calculate capture banding filter */
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	light_freq = ret;
> +
> +	if (light_freq == 60) {
> +		/* 60Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
> +	} else {
> +		/* 50Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts;
> +	}
> +	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
> +
> +	/* calculate capture shutter/gain16 */
> +	if (average > sensor->ae_low && average < sensor->ae_high) {
> +		/* in stable range */
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts *
> +			sensor->ae_target / average;
> +	} else {
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts;
> +	}
> +
> +	/* gain to shutter */
> +	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
> +		/* shutter < 1/100 */
> +		cap_shutter = cap_gain16_shutter / 16;
> +		if (cap_shutter < 1)
> +			cap_shutter = 1;
> +
> +		cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		if (cap_gain16 < 16)
> +			cap_gain16 = 16;
> +	} else {
> +		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
> +			/* exposure reach max */
> +			cap_shutter = cap_bandfilt * cap_maxband;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		} else {
> +			/* 1/100 < (cap_shutter = n/100) =< max */
> +			cap_shutter =
> +				((int)(cap_gain16_shutter / 16 / 
cap_bandfilt))
> +				* cap_bandfilt;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		}
> +	}
> +
> +	/* write capture gain */
> +	ret = ov5640_set_gain(sensor, cap_gain16);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* write capture shutter */
> +	if (cap_shutter > (cap_vts - 4)) {
> +		cap_vts = cap_shutter + 4;
> +		ret = ov5640_set_VTS(sensor, cap_vts);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	ret = ov5640_set_exposure(sensor, cap_shutter);
> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_stream(sensor, true);
> +}
> +
> +/*
> + * if sensor changes inside scaling or subsampling
> + * change mode directly
> + */
> +static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
> +				     enum ov5640_frame_rate frame_rate,
> +				     enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, true);

This will turn streaming on when you set the format, which I don't think is 
correct. You should only turn streaming on in the .s_stream() operation. The 
same comment applies to the previous function.

> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_agc(sensor, true);
> +}
> +
> +static int ov5640_change_mode(struct ov5640_dev *sensor,
> +			      enum ov5640_frame_rate frame_rate,
> +			      enum ov5640_mode mode,
> +			      enum ov5640_mode orig_mode)
> +{
> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
> +	    mode != ov5640_mode_INIT) {
> +		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
> +		return -EINVAL;
> +	}
> +
> +	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
> +	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
> +	if (mode == ov5640_mode_INIT) {
> +		mode_data = ov5640_init_setting_30fps_VGA;
> +		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
> +
> +		sensor->fmt.width = 640;
> +		sensor->fmt.height = 480;

Don't reset the format here. The format must be preserved across subdev 
open/close.

> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +		if (ret < 0)
> +			return ret;
> +
> +		mode_data = ov5640_setting_30fps_VGA_640_480;
> +		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
> +			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
> +		/* change between subsampling and scaling
> +		 * go through exposure calucation */
> +		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
> +							  mode);
> +	} else {
> +		/* change inside subsampling or scaling
> +		 * download firmware directly */
> +		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
> +	}
> +
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_bandingfilter(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_virtual_channel(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* restore controls */
> +	ov5640_restore_ctrls(sensor);
> +
> +	if (ret >= 0 && mode != ov5640_mode_INIT) {
> +		sensor->current_mode = mode;
> +		sensor->current_fr = frame_rate;
> +	}
> +
> +	return 0;
> +}
> +
> +/* restore the last set video mode after chip power-on */
> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
> +{
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	/* first we need to set some initial register values */
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				    ov5640_mode_INIT, ov5640_mode_INIT);

I wouldn't use ov5640_change_mode() here. You only need to apply the init 
register values, just call ov5640_load_regs(). The rest of the 
ov5640_change_mode() isn't needed, as you're calling it below. This will also 
allow you to get rid of ov5640_mode_INIT, simplifying the logic.

> +	if (ret < 0)
> +		return ret;
> +
> +	/* now restore the last capture mode */
> +	return ov5640_change_mode(sensor,
> +				  sensor->current_fr,
> +				  sensor->current_mode,
> +				  ov5640_mode_VGA_640_480);

You can fit that in fewer lines while still staying within the 80 columns 
limit.

> +}
> +
> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
> +{
> +	int ret;
> +
> +	if (sensor->io_regulator) {
> +		ret = regulator_enable(sensor->io_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->core_regulator) {
> +		ret = regulator_enable(sensor->core_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->gpo_regulator) {
> +		ret = regulator_enable(sensor->gpo_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->analog_regulator) {
> +		ret = regulator_enable(sensor->analog_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
> +			return ret;
> +		}
> +	}

Maybe you should use the bulk regulator API ?

> +
> +	return 0;
> +}
> +
> +static void ov5640_regulators_off(struct ov5640_dev *sensor)
> +{
> +	if (sensor->analog_regulator)
> +		regulator_disable(sensor->analog_regulator);
> +	if (sensor->core_regulator)
> +		regulator_disable(sensor->core_regulator);
> +	if (sensor->io_regulator)
> +		regulator_disable(sensor->io_regulator);
> +	if (sensor->gpo_regulator)
> +		regulator_disable(sensor->gpo_regulator);
> +}
> +
> +/* --------------- Subdev Operations --------------- */
> +
> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	int ret;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (on && !sensor->on) {

The .s_power() calls need to be ref-counted, similarly to how the regulator 
enable/disable calls work. See the mt9p031 sensor driver for an example.

> +		if (sensor->xclk)
> +			clk_prepare_enable(sensor->xclk);
> +
> +		ret = ov5640_regulators_on(sensor);
> +		if (ret)
> +			return ret;
> +
> +		ov5640_reset(sensor);
> +		ov5640_power(sensor, true);
> +
> +		ret = ov5640_init_slave_id(sensor);

Why is this needed ?

> +		if (ret)
> +			return ret;
> +
> +		ret = ov5640_restore_mode(sensor);
> +		if (ret)
> +			return ret;
> +
> +		/*
> +		 * NOTE: Freescale adds a long delay (600 msec) after
> +		 * powering up and programming a mode on the ov5640-mipi
> +		 * camera (search for "msec_wait4stable" in FSL's
> +		 * ov5640_mipi.c), which equivalently would need to go
> +		 * right here. If we run into MIPI CSI-2 receiver dphy
> +		 * ready timeouts, it might be a clue to add that delay
> +		 * here.
> +		 */
> +	} else if (!on && sensor->on) {
> +		ov5640_power(sensor, false);
> +
> +		ov5640_regulators_off(sensor);
> +
> +		if (sensor->xclk)
> +			clk_disable_unprepare(sensor->xclk);
> +	}
> +
> +	sensor->on = on;
> +
> +	return 0;
> +}
> +
> +static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_captureparm *cparm = &a->parm.capture;
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* This is the only case currently handled. */
> +	memset(a, 0, sizeof(*a));
> +	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	cparm->capability = sensor->streamcap.capability;
> +	cparm->timeperframe = sensor->streamcap.timeperframe;
> +	cparm->capturemode = sensor->streamcap.capturemode;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
> +	enum ov5640_frame_rate frame_rate;
> +	u32 tgt_fps;	/* target frames per secound */
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* Check that the new frame rate is allowed. */
> +	if ((timeperframe->numerator == 0) ||
> +	    (timeperframe->denominator == 0)) {
> +		timeperframe->denominator = DEFAULT_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps > MAX_FPS) {
> +		timeperframe->denominator = MAX_FPS;
> +		timeperframe->numerator = 1;
> +	} else if (tgt_fps < MIN_FPS) {
> +		timeperframe->denominator = MIN_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	/* Actual frame rate we use */
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps == 15)
> +		frame_rate = ov5640_15_fps;
> +	else if (tgt_fps == 30)
> +		frame_rate = ov5640_30_fps;
> +	else {
> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
> +			 tgt_fps);

Don't print an error message that is userspace-triggerable, we have enough 
ways for applications to flood the kernel log already :-)

> +		return -EINVAL;
> +	}
> +
> +	ret = ov5640_change_mode(sensor, frame_rate,
> +				 sensor->current_mode,
> +				 sensor->current_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	sensor->streamcap.timeperframe = *timeperframe;
> +
> +	return 0;
> +}
> +
> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	format->format = sensor->fmt;

You need to handle the TRY format here. You can have a look at the mt9p031 
driver for an example on how to do so.

> +
> +	return 0;
> +}
> +
> +static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
> +				   struct v4l2_mbus_framefmt *fmt,
> +				   enum ov5640_mode *new_mode)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
> +
> +	fmt->width = ov5640_mode_info_data[0][mode].width;
> +	fmt->height = ov5640_mode_info_data[0][mode].height;
> +	fmt->code = sensor->fmt.code;
> +
> +	if (new_mode)
> +		*new_mode = mode;
> +	return 0;
> +}
> +
> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode new_mode;
> +	int ret;
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
> +		if (ret)
> +			return ret;

You can move this code above the test to avoid the duplicated call below.

> +		cfg->try_fmt = format->format;

Please use the v4l2_subdev_get_try_format() function to get the try format, 
don't dereference cfg directly.

> +		return 0;
> +	}
> +
> +	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
> +	if (ret)
> +		return ret;
> +
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				 new_mode, sensor->current_mode);
> +	if (ret >= 0)
> +		sensor->fmt = format->format;
> +
> +	return ret;
> +}
> +
> +
> +/*
> + * Sensor Controls.
> + */
> +
> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);

According to the kernel coding style, you need curly braces around the else 
branch if you use them around the if branch.

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
> +		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);

Ditto

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
> +		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);

Ditto

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	sensor->awb_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
> +	return 0;
> +}
> +
> +static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +#if 0

No compiled-out code please.

> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +#endif
> +
> +static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
> +{
> +	u16 max_exp = 0;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
> +	if (value < max_exp) {
> +		u32 exp = value << 4;
> +
> +		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
> +	}
> +
> +	return 0;
> +}
> +
> +/* read exposure, in number of line periods */
> +static int ov5640_get_exposure(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int exp, ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG(sensor, 0x3500, &temp);
> +	exp = ((int)temp & 0x0f) << 16;
> +	OV5640_READ_REG(sensor, 0x3501, &temp);
> +	exp |= ((int)temp << 8);
> +	OV5640_READ_REG(sensor, 0x3502, &temp);
> +	exp |= (int)temp;
> +
> +	return exp >> 4;
> +}
> +
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	/* this enables/disables both AEC and AGC */
> +	sensor->agc_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;

You can create a cluster with the autogain and manual gain controls 
(v4l2_ctrl_auto_cluster) to have the control framework disabling the manual 
control automatically when autogain is enabled.

> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
> +	return 0;
> +}
> +
> +static int ov5640_get_gain(struct ov5640_dev *sensor)
> +{
> +	u16 gain;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
> +
> +	return gain & 0x3ff;
> +}
> +
> +#if 0
> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
> +	return 0;
> +}
> +#endif

You can use a control to expose the test pattern function, it's quite useful.

> +static struct ov5640_control ov5640_ctrls[] = {
> +	{
> +		.set = ov5640_set_agc,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTOGAIN,
> +			.name = "Auto Gain/Exposure Control",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_exposure,
> +		.ctrl = {
> +			.id = V4L2_CID_EXPOSURE,
> +			.name = "Exposure",
> +			.minimum = 0,
> +			.maximum = 65535,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_gain,
> +		.ctrl = {
> +			.id = V4L2_CID_GAIN,
> +			.name = "Gain",
> +			.minimum = 0,
> +			.maximum = 1023,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_hue,
> +		.ctrl = {
> +			.id = V4L2_CID_HUE,
> +			.name = "Hue",
> +			.minimum = 0,
> +			.maximum = 359,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_contrast,
> +		.ctrl = {
> +			.id = V4L2_CID_CONTRAST,
> +			.name = "Contrast",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_saturation,
> +		.ctrl = {
> +			.id = V4L2_CID_SATURATION,
> +			.name = "Saturation",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 64,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_awb,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTO_WHITE_BALANCE,
> +			.name = "Auto White Balance",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_red_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_RED_BALANCE,
> +			.name = "Red Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_blue_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_BLUE_BALANCE,
> +			.name = "Blue Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	},
> +};
> +#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
> +
> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
> +{
> +	struct ov5640_control *ret = NULL;
> +	int i;

i is never negative, it should be unsigned int.

> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		if (id == ov5640_ctrls[i].ctrl.id) {
> +			ret = &ov5640_ctrls[i];
> +			break;
> +		}
> +	}
> +
> +	if (ret && index)
> +		*index = i;
> +	return ret;
> +}
> +
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;
> +	int i;

i is never negative, it should be unsigned int.

> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +		c->set(sensor, sensor->ctrl_cache[i]);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
> +	struct ov5640_control *c;
> +	int ret = 0;
> +	int i;
> +
> +	c = ov5640_get_ctrl(ctrl->id, &i);

You can inline the function call here as it's not used anywhere else.

> +	if (!c)
> +		return -EINVAL;
> +
> +	ret = c->set(sensor, ctrl->val);
> +	/* update cached value if no error */
> +	if (!ret)
> +		sensor->ctrl_cache[i] = ctrl->val;
> +
> +	return ret;
> +}
> +
> +static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
> +	.s_ctrl = ov5640_s_ctrl,
> +};
> +
> +static int ov5640_init_controls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;

I would name this ctrl or control, c can be a bit confusing. You can also 
declare it inside the loop.

> +	int i;

i can never be negative, you can make it an unsigned int.

> +
> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +
> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
> +				  c->ctrl.id, c->ctrl.minimum, c-
>ctrl.maximum,
> +				  c->ctrl.step, c->ctrl.default_value);
> +	}
> +
> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
> +	if (sensor->ctrl_hdl.error) {
> +		int err = sensor->ctrl_hdl.error;
> +
> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);

I'm not sure this brings much value.

> +		return err;
> +	}
> +	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);

You shouldn't call this function here for the reason explained below.

> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_frame_size_enum *fse)
> +{
> +	if (fse->pad != 0)
> +		return -EINVAL;
> +	if (fse->index >= ov5640_num_modes)
> +		return -EINVAL;
> +
> +	fse->min_width = fse->max_width =
> +		ov5640_mode_info_data[0][fse->index].width;
> +	fse->min_height = fse->max_height =
> +		ov5640_mode_info_data[0][fse->index].height;
> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_interval(
> +	struct v4l2_subdev *sd,
> +	struct v4l2_subdev_pad_config *cfg,
> +	struct v4l2_subdev_frame_interval_enum *fie)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	if (fie->pad != 0)
> +		return -EINVAL;
> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
> +		return -EINVAL;
> +
> +	if (fie->width == 0 || fie->height == 0)
> +		return -EINVAL;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);

You should find an exact mode here, not the nearest one. If with and height 
don't match, return -EINVAL. That will replace with above width == 0 and 
height == 0 test.

> +	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
> +		return -EINVAL;
> +
> +	fie->interval.numerator = 1;
> +	fie->interval.denominator = ov5640_framerates[fie->index];
> +
> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
> +		fie->width, fie->height, fie->index, fie-
>interval.denominator);

I'm not sure this is very useful, you can use ftrace if you want to trace 
function calls.

> +	return 0;
> +}
> +
> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
> +			    u32 output, u32 config)
> +{
> +	return (input != 0) ? -EINVAL : 0;
> +}

The g_input_status and s_routing subdev operations are not mandatory, you 
don't have to implement them as the sensor doesn't have multiple inputs.

[snip]

> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

You can use ftrace to trace function calls, there's no need to add debugging 
statements that duplicate the functionality.

> +	return ov5640_set_stream(sensor, enable);
> +}
> +
> +static struct v4l2_subdev_core_ops ov5640_core_ops = {
> +	.s_power = ov5640_s_power,
> +};
> +
> +static struct v4l2_subdev_video_ops ov5640_video_ops = {
> +	.s_parm = ov5640_s_parm,
> +	.g_parm = ov5640_g_parm,
> +	.g_input_status = ov5640_g_input_status,
> +	.s_routing = ov5640_s_routing,
> +	.g_mbus_config  = ov5640_g_mbus_config,
> +	.s_stream = ov5640_s_stream,
> +};
> +
> +static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
> +	.enum_mbus_code = ov5640_enum_mbus_code,
> +	.get_fmt = ov5640_get_fmt,
> +	.set_fmt = ov5640_set_fmt,
> +	.enum_frame_size = ov5640_enum_frame_size,
> +	.enum_frame_interval = ov5640_enum_frame_interval,
> +};
> +
> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
> +	.core = &ov5640_core_ops,
> +	.video = &ov5640_video_ops,
> +	.pad = &ov5640_pad_ops,
> +};

All these structures should be static const.

> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
> +{
> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
> +}
> +
> +static void ov5640_reset(struct ov5640_dev *sensor)
> +{
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +
> +	/* camera power cycle */
> +	ov5640_power(sensor, false);
> +	usleep_range(5000, 10000);
> +	ov5640_power(sensor, true);
> +	usleep_range(5000, 10000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 1);
> +	usleep_range(1000, 2000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +	usleep_range(5000, 10000);
> +}
> +
> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
> +{
> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
> +	if (!IS_ERR(sensor->io_regulator)) {
> +		regulator_set_voltage(sensor->io_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_IO,
> +				      OV5640_VOLTAGE_DIGITAL_IO);
> +	} else {
> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
> +			__func__);
> +		sensor->io_regulator = NULL;

How about making the power supplies mandatory in DT instead ? They are 
mandatory after all, if they're not controllable DT should just declare fixed 
supplies.

> +	}
> +
> +	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
> +	if (!IS_ERR(sensor->core_regulator)) {
> +		regulator_set_voltage(sensor->core_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_CORE,
> +				      OV5640_VOLTAGE_DIGITAL_CORE);
> +	} else {
> +		sensor->core_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
> +			__func__);
> +	}
> +
> +	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
> +	if (!IS_ERR(sensor->analog_regulator)) {
> +		regulator_set_voltage(sensor->analog_regulator,
> +				      OV5640_VOLTAGE_ANALOG,
> +				      OV5640_VOLTAGE_ANALOG);
> +	} else {
> +		sensor->analog_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
> +			__func__);
> +	}
> +}
> +
> +static int ov5640_probe(struct i2c_client *client,
> +			const struct i2c_device_id *id)
> +{
> +	struct device *dev = &client->dev;
> +	struct device_node *endpoint;
> +	struct ov5640_dev *sensor;
> +	int i, xclk, ret;

i and xclk are never negative, you can make them unsigned int.

> +
> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);

Please use sizeof(*variable) instead of sizeof(type).

devm_kzalloc() doesn't play nicely with dynamic removal of devices. We're in 
the process of fixing related race conditions in the media subsystem. In order 
not to make the problem worse, please use kzalloc() instead.

> +	if (!sensor)
> +		return -ENOMEM;
> +
> +	sensor->i2c_client = client;
> +	sensor->dev = dev;

Do you really need to store both i2c_client and dev, given that the latter is 
just &client->dev ?

> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
> +	sensor->fmt.width = 640;
> +	sensor->fmt.height = 480;
> +	sensor->fmt.field = V4L2_FIELD_NONE;
> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
> +					   V4L2_CAP_TIMEPERFRAME;

Please fix the indentation.

> +	sensor->streamcap.capturemode = 0;
> +	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
> +	sensor->streamcap.timeperframe.numerator = 1;
> +
> +	sensor->current_mode = ov5640_mode_VGA_640_480;
> +	sensor->current_fr = ov5640_30_fps;
> +
> +	sensor->ae_target = 52;
> +
> +	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
> +	if (!endpoint) {
> +		dev_err(dev, "endpoint node not found\n");
> +		return -EINVAL;
> +	}
> +
> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
> +		return -EINVAL;

You're leaking endpoint here. You should move the of_node_put() call right 
after the v4l2_of_parse_endpoint() call.

> +	}
> +	of_node_put(endpoint);
> +
> +	/* get system clock (xclk) frequency */
> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);

Instead of adding a custom DT property for this, use assigned-clock-rates. You 
won't need to set it manually in the driver, and can verify its frequency with 
clk_get_rate().

> +	if (!ret) {
> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {

Are your register tables above independent of the clock frequency ? You should 
ideally compute register values at runtime instead of hardcoding them, but 
given the lack of information from Omnivision I understand this isn't 
possible. You thus need to be stricter here and reject any value other than 
the nominal frequency.

> +			dev_err(dev, "invalid xclk frequency\n");
> +			return -EINVAL;
> +		}
> +		sensor->xclk_freq = xclk;
> +	}
> +
> +	/* get system clock (xclk) */
> +	sensor->xclk = devm_clk_get(dev, "xclk");
> +	if (!IS_ERR(sensor->xclk)) {
> +		if (!sensor->xclk_freq) {
> +			dev_err(dev, "xclk requires xclk frequency!\n");
> +			return -EINVAL;
> +		}
> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
> +	} else {
> +		/* assume system clock enabled by default */
> +		sensor->xclk = NULL;

Please don't. The clock should be mandatory.

> +	}
> +
> +	/* request power down pin */
> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);

Are the GPIOs mandatory or optional ? Can a system tie some of them to ground 
or a voltage rail, or do they need to always be manually controllable ?

> +	if (IS_ERR(sensor->pwdn_gpio)) {
> +		dev_err(dev, "request for power down gpio failed\n");
> +		return PTR_ERR(sensor->pwdn_gpio);
> +	}
> +
> +	/* request reset pin */
> +	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
> +	if (IS_ERR(sensor->reset_gpio)) {
> +		dev_err(dev, "request for reset gpio failed\n");
> +		return PTR_ERR(sensor->reset_gpio);
> +	}
> +
> +	/* initialize the cached controls to their defaults */
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		struct ov5640_control *c = &ov5640_ctrls[i];
> +
> +		sensor->ctrl_cache[i] = c->ctrl.default_value;
> +	}
> +	sensor->awb_on = sensor->agc_on = true;
> +
> +	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
> +
> +	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
> +	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> +	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
> +	if (ret)
> +		return ret;
> +
> +	ov5640_get_regulators(sensor);
> +
> +	ret = ov5640_s_power(&sensor->sd, 1);
> +	if (ret)
> +		goto entity_cleanup;
> +	ret = ov5640_init_controls(sensor);
> +	if (ret)
> +		goto power_off;
> +
> +	ret = ov5640_s_power(&sensor->sd, 0);
> +	if (ret)
> +		goto free_ctrls;

Writing the controls here is pointless, as powering the chip down will lose 
all the values. You shouldn't call v4l2_ctrl_handler_setup() in 
ov5640_init_controls(), and you can then remove the ov5640_s_power() calls 
here.

> +	ret = v4l2_async_register_subdev(&sensor->sd);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +power_off:
> +	ov5640_s_power(&sensor->sd, 0);
> +entity_cleanup:
> +	media_entity_cleanup(&sensor->sd.entity);
> +	ov5640_regulators_off(sensor);

Won't ov5640_s_power(0) already disable the regulators ?

> +	return ret;
> +}
> +
> +/*!
> + * ov5640 I2C detach function
> + *
> + * @param client            struct i2c_client *
> + * @return  Error code indicating success or failure
> + */

That's not the kerneldoc comment style. Given that this is the only documented 
function, and that the comment is completely useless, you can just drop it.

> +static int ov5640_remove(struct i2c_client *client)
> +{
> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	ov5640_regulators_off(sensor);
> +
> +	v4l2_async_unregister_subdev(&sensor->sd);
> +	media_entity_cleanup(&sensor->sd.entity);
> +	v4l2_device_unregister_subdev(sd);

This function is called by v4l2_async_unregister_subdev(), there's no need to 
duplicate it.

> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +	return 0;
> +}

[snip]

> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");

MODULE_AUTHOR isn't a synonym for copyright ownership. I don't think you 
should add Freescale as an author. If you know who wrote the original code you 
can list that developer explicitly.

> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
> +MODULE_LICENSE("GPL");
> +MODULE_VERSION("1.0");

Version numbers are never updated. I wouldn't bother adding one.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-02-02 10:36     ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-02 10:36 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

Thank you for the patch. Many of the comments below apply to the ov5642 driver 
too, please take them into account when reworking patch 23/24.

On Friday 06 Jan 2017 18:11:40 Steve Longerbeam wrote:
> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
> branch, modified heavily to bring forward to latest interfaces and code
> cleanup.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Kconfig       |    8 +
>  drivers/staging/media/imx/Makefile      |    2 +
>  drivers/staging/media/imx/ov5640-mipi.c | 2348 ++++++++++++++++++++++++++++

You're missing DT bindings.

The driver should go to drivers/media/i2c/ as it should not be specific to the 
i.MX6, and you can just call it ov5640.c.

>  3 files changed, 2358 insertions(+)
>  create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
> 
> diff --git a/drivers/staging/media/imx/Kconfig
> b/drivers/staging/media/imx/Kconfig index ce2d2c8..09f373d 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>  	---help---
>  	  A video4linux camera capture driver for i.MX5/6.
> 
> +config IMX_OV5640_MIPI
> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
> +       depends on GPIOLIB && VIDEO_IMX_CAMERA

The sensor driver is generic, it shouldn't depend on IMX. It should however 
depend on at least I2C and OF by the look of it.

> +       select IMX_MIPI_CSI2
> +       default y
> +       ---help---
> +         MIPI CSI-2 OV5640 Camera support.
> +
>  endmenu
>  endif

[snip]

> diff --git a/drivers/staging/media/imx/ov5640-mipi.c
> b/drivers/staging/media/imx/ov5640-mipi.c new file mode 100644
> index 0000000..54647a7
> --- /dev/null
> +++ b/drivers/staging/media/imx/ov5640-mipi.c
> @@ -0,0 +1,2348 @@
> +/*
> + * Copyright (c) 2014 Mentor Graphics Inc.
> + * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights
> Reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/init.h>
> +#include <linux/slab.h>
> +#include <linux/ctype.h>
> +#include <linux/types.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/i2c.h>
> +#include <linux/of_device.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/regulator/consumer.h>
> +#include <linux/clk.h>
> +#include <linux/clk-provider.h>
> +#include <linux/clkdev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-async.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>

Pet peeve of mine, please sort the headers alphabetically. It makes it easier 
to locate duplicated.

> +
> +#define OV5640_VOLTAGE_ANALOG               2800000
> +#define OV5640_VOLTAGE_DIGITAL_CORE         1500000
> +#define OV5640_VOLTAGE_DIGITAL_IO           1800000
> +
> +#define MIN_FPS 15
> +#define MAX_FPS 30
> +#define DEFAULT_FPS 30
> +
> +/* min/typical/max system clock (xclk) frequencies */
> +#define OV5640_XCLK_MIN  6000000
> +#define OV5640_XCLK_TYP 24000000
> +#define OV5640_XCLK_MAX 54000000
> +
> +/* min/typical/max pixel clock (mclk) frequencies */
> +#define OV5640_MCLK_MIN 48000000
> +#define OV5640_MCLK_TYP 48000000
> +#define OV5640_MCLK_MAX 96000000
> +
> +#define OV5640_CHIP_ID  0x300A
> +#define OV5640_SLAVE_ID 0x3100
> +#define OV5640_DEFAULT_SLAVE_ID 0x3c

You're mixing lower-case and upper-case hex constants. Let's pick one. Kernel 
code usually favours lower-case.

Please define macros for all the other numerical constants you use in the 
driver (register addresses and values). The large registers tables can be an 
exception if you don't have access to the information, but for registers 
written manually, hardcoding numerical values isn't good.

> +
> +#define OV5640_MAX_CONTROLS 64
> +
> +enum ov5640_mode {
> +	ov5640_mode_MIN = 0,
> +	ov5640_mode_QCIF_176_144 = 0,
> +	ov5640_mode_QVGA_320_240,
> +	ov5640_mode_VGA_640_480,
> +	ov5640_mode_NTSC_720_480,
> +	ov5640_mode_PAL_720_576,
> +	ov5640_mode_XGA_1024_768,
> +	ov5640_mode_720P_1280_720,
> +	ov5640_mode_1080P_1920_1080,
> +	ov5640_mode_QSXGA_2592_1944,
> +	ov5640_num_modes,
> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/

Please add spaces after /* and before */.

Enumerated values should be all upper-case.

> +};
> +
> +enum ov5640_frame_rate {
> +	ov5640_15_fps,
> +	ov5640_30_fps
> +};
> +
> +static int ov5640_framerates[] = {
> +	[ov5640_15_fps] = 15,
> +	[ov5640_30_fps] = 30,
> +};
> +#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
> +
> +/* image size under 1280 * 960 are SUBSAMPLING
> + * image size upper 1280 * 960 are SCALING
> + */

The kernel multi-line comment style is

/*
 * text
 * text
 */

> +enum ov5640_downsize_mode {
> +	SUBSAMPLING,
> +	SCALING,
> +};
> +
> +struct reg_value {
> +	u16 reg_addr;
> +	u8 val;
> +	u8 mask;
> +	u32 delay_ms;
> +};
> +
> +struct ov5640_mode_info {
> +	enum ov5640_mode mode;
> +	enum ov5640_downsize_mode dn_mode;
> +	u32 width;
> +	u32 height;
> +	struct reg_value *init_data_ptr;
> +	u32 init_data_size;
> +};
> +
> +struct ov5640_dev {
> +	struct i2c_client *i2c_client;
> +	struct device *dev;
> +	struct v4l2_subdev sd;
> +	struct media_pad pad;
> +	struct v4l2_ctrl_handler ctrl_hdl;
> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
> +	struct v4l2_mbus_framefmt fmt;
> +	struct v4l2_captureparm streamcap;
> +	struct clk *xclk; /* system clock to OV5640 */
> +	int xclk_freq;    /* requested xclk freq from devicetree */
> +
> +	enum ov5640_mode current_mode;

Store a (const) pointer to the corresponding ov5640_mode_info instead, it will 
simplify the code and allow you to get rid of the ov5640_mode enum.

> +	enum ov5640_frame_rate current_fr;
> +
> +	bool on;
> +	bool awb_on;
> +	bool agc_on;
> +
> +	/* cached control settings */
> +	int ctrl_cache[OV5640_MAX_CONTROLS];
> +
> +	struct gpio_desc *reset_gpio;
> +	struct gpio_desc *pwdn_gpio;
> +	struct gpio_desc *gp_gpio;
> +
> +	int prev_sysclk, prev_hts;
> +	int ae_low, ae_high, ae_target;

Can't these be unsigned int ?

> +
> +	struct regulator *io_regulator;
> +	struct regulator *core_regulator;
> +	struct regulator *analog_regulator;
> +	struct regulator *gpo_regulator;
> +};
> +
> +static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct ov5640_dev, sd);
> +}
> +
> +static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
> +{
> +	return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
> +}
> +
> +struct ov5640_control {
> +	struct v4l2_queryctrl ctrl;
> +	int (*set)(struct ov5640_dev *sensor, int value);
> +};
> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
> +static void ov5640_reset(struct ov5640_dev *sensor);
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
> +static int ov5640_get_gain(struct ov5640_dev *sensor);

No forward declarations please. You should reorder functions as needed (and of 
course still group related functions together).

> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
> +
> +	{0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
> +	{0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
> +	{0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
> +	{0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
> +	{0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
> +	{0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
> +	{0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
> +	{0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
> +	{0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
> +	{0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
> +	{0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
> +	{0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
> +	{0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
> +	{0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
> +	{0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
> +	{0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
> +	{0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
> +	{0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
> +	{0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
> +	{0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
> +	{0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
> +	{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
> +	{0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
> +	{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
> +	{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
> +	{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
> +	{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
> +	{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
> +	{0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
> +	{0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
> +	{0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
> +	{0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
> +	{0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
> +	{0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
> +	{0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
> +	{0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
> +	{0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
> +	{0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
> +	{0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
> +	{0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
> +	{0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
> +	{0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
> +	{0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
> +	{0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
> +	{0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
> +	{0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
> +	{0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
> +	{0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
> +	{0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
> +	{0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
> +	{0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
> +	{0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
> +	{0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
> +	{0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
> +	{0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
> +	{0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
> +	{0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
> +	{0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
> +	{0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
> +	{0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
> +	{0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
> +	{0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
> +	{0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
> +	{0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
> +	{0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
> +	{0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
> +	{0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
> +	{0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
> +	{0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
> +	{0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
> +	{0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
> +	{0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
> +	{0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
> +	{0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
> +};

You only use the delay feature of the registers tables twice, once after 
writing the first two registers (to select the clock source and perform a 
software reset) and once at the very end. Remove it, write the first two 
registers manually in the code with a manual delay afterwards, and add another 
manual delay after writing the whole table.

I'm actually wondering whether you couldn't remove the 300ms delay at the end, 
the 50/60Hz control register (0x3c00) doesn't look like it needs a delay after 
being written.

[snip]

> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
> +	{0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
> +	{0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
> +	{0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
> +	{0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
> +	{0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
> +	{0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
> +	{0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
> +	{0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
> +	{0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
> +	{0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
> +	{0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
> +	{0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
> +	{0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
> +	{0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
> +	{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
> +	{0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
> +	{0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
> +	{0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */

Don't turn the stream on in the init sequences, it should only be turned on 
from the .s_stream() operation.

> +};
> +
> +static struct ov5640_mode_info
> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {

There's very few differences between the 15fps and 30fps tables. It would be 
better if you could merge them, and manually write the registers that differ.

> +	{
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_15fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_15fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_15fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_15fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_15fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_15fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_15fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_15fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
> +		 ov5640_setting_15fps_QSXGA_2592_1944,
> +		 ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
> +	}, {
> +		{ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
> +		 ov5640_setting_30fps_QCIF_176_144,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
> +		{ov5640_mode_QVGA_320_240, SUBSAMPLING, 320,  240,
> +		 ov5640_setting_30fps_QVGA_320_240,
> +		 ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
> +		{ov5640_mode_VGA_640_480, SUBSAMPLING, 640,  480,
> +		 ov5640_setting_30fps_VGA_640_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
> +		{ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
> +		 ov5640_setting_30fps_NTSC_720_480,
> +		 ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
> +		{ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
> +		 ov5640_setting_30fps_PAL_720_576,
> +		 ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
> +		{ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
> +		 ov5640_setting_30fps_XGA_1024_768,
> +		 ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
> +		{ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
> +		 ov5640_setting_30fps_720P_1280_720,
> +		 ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
> +		{ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
> +		 ov5640_setting_30fps_1080P_1920_1080,
> +		 ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
> +		{ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
> +	},
> +};
> +
> +static int ov5640_probe(struct i2c_client *adapter,
> +			const struct i2c_device_id *device_id);
> +static int ov5640_remove(struct i2c_client *client);

No forward declarations please.

> +static int ov5640_init_slave_id(struct ov5640_dev *sensor)
> +{
> +	struct i2c_msg msg;
> +	u8 buf[4];
> +	int ret;
> +
> +	if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
> +		return 0;
> +
> +	buf[0] = OV5640_SLAVE_ID >> 8;
> +	buf[1] = OV5640_SLAVE_ID & 0xff;
> +	buf[2] = sensor->i2c_client->addr << 1;
> +	msg.addr = OV5640_DEFAULT_SLAVE_ID;
> +	msg.flags = 0;
> +	msg.len = 3;
> +	msg.buf = buf;
> +
> +	ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
> +	if (ret < 0) {
> +		dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
> +{
> +	u8 buf[3] = {0};
> +	int ret;
> +
> +	buf[0] = reg >> 8;
> +	buf[1] = reg & 0xff;
> +	buf[2] = val;
> +
> +	ret = i2c_master_send(sensor->i2c_client, buf, 3);
> +	if (ret < 0) {
> +		v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
> +			__func__, reg, val);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
> +{
> +	u8 reg_buf[2] = {0};
> +	u8 read_val = 0;
> +
> +	reg_buf[0] = reg >> 8;
> +	reg_buf[1] = reg & 0xff;
> +
> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
> +			__func__, reg);
> +		return -EIO;
> +	}
> +
> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
> +			__func__, reg, read_val);
> +		return -EIO;
> +	}

Wouldn't i2c_transfer() be more efficient here ?

> +	*val = read_val;
> +	return 0;
> +}
> +
> +#define OV5640_READ_REG(s, r, v) {				\
> +		ret = ov5640_read_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}

No. No. No no no. Don't ever return from a macro. Hiding the return makes 
following the code flow much more difficult, it's just asking for trouble.

And don't use externally defined variables (ret in this case), that's also 
asking for trouble.

> +#define OV5640_WRITE_REG(s, r, v) {				\
> +		ret = ov5640_write_reg((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
> +{
> +	u8 hi, lo;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &hi);
> +	OV5640_READ_REG(sensor, reg+1, &lo);
> +
> +	*val = ((u16)hi << 8) | (u16)lo;
> +	return 0;
> +}
> +#define OV5640_READ_REG16(s, r, v) {				\
> +		ret = ov5640_read_reg16((s), (r), (v));		\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, reg, val >> 8);
> +	OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
> +	return 0;
> +}
> +#define OV5640_WRITE_REG16(s, r, v) {				\
> +		ret = ov5640_write_reg16((s), (r), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}
> +
> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
> +			  u8 mask, u8 val)
> +{
> +	u8 readval;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, reg, &readval);
> +
> +	readval &= ~mask;
> +	val &= mask;
> +	val |= readval;
> +
> +	OV5640_WRITE_REG(sensor, reg, val);
> +	return 0;
> +}
> +#define OV5640_MOD_REG(s, r, m, v) {				\
> +		ret = ov5640_mod_reg((s), (r), (m), (v));	\
> +		if (ret)					\
> +			return ret;				\
> +	}

If you need to modify registers a lot, switch to regmap for register access. 
It will provide you with a cache, removing the need to read registers back 
from the device.

> +/* download ov5640 settings to sensor through i2c */
> +static int ov5640_load_regs(struct ov5640_dev *sensor,
> +			    struct reg_value *regs,
> +			    int size)

size is never negative.

> +{
> +	register u32 delay_ms = 0;
> +	register u16 reg_addr = 0;
> +	register u8 mask = 0;
> +	register u8 val = 0;

register ? The compiler is nowadays likely smarter than us when it comes to 
register allocation.

There's also no need to initialize the variables to 0.

> +	int i, ret;

And i isn't either.

> +
> +	for (i = 0; i < size; ++i, ++regs) {
> +		delay_ms = regs->delay_ms;
> +		reg_addr = regs->reg_addr;
> +		val = regs->val;
> +		mask = regs->mask;
> +
> +		if (mask) {
> +			OV5640_MOD_REG(sensor, reg_addr, mask, val);
> +		} else {
> +			OV5640_WRITE_REG(sensor, reg_addr, val);
> +		}
> +		if (delay_ms)
> +			usleep_range(1000*delay_ms, 1000*delay_ms+100);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);
> +	return 0;
> +}
> +
> +static int ov5640_get_sysclk(struct ov5640_dev *sensor)
> +{
> +	 /* calculate sysclk */
> +	int xvclk = sensor->xclk_freq / 10000;
> +	int multiplier, prediv, VCO, sysdiv, pll_rdiv;
> +	int sclk_rdiv_map[] = {1, 2, 4, 8};
> +	int bit_div2x = 1, sclk_rdiv, sysclk;
> +	u8 temp1, temp2;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3034, &temp1);
> +	temp2 = temp1 & 0x0f;
> +	if (temp2 == 8 || temp2 == 10)
> +		bit_div2x = temp2 / 2;
> +
> +	OV5640_READ_REG(sensor, 0x3035, &temp1);
> +	sysdiv = temp1>>4;
> +	if (sysdiv == 0)
> +		sysdiv = 16;
> +
> +	OV5640_READ_REG(sensor, 0x3036, &temp1);
> +	multiplier = temp1;
> +
> +	OV5640_READ_REG(sensor, 0x3037, &temp1);
> +	prediv = temp1 & 0x0f;
> +	pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
> +
> +	OV5640_READ_REG(sensor, 0x3108, &temp1);
> +	temp2 = temp1 & 0x03;
> +	sclk_rdiv = sclk_rdiv_map[temp2];
> +
> +	VCO = xvclk * multiplier / prediv;
> +
> +	sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
> +
> +	return sysclk;
> +}
> +
> +static int ov5640_set_night_mode(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u8 mode;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3a00, &mode);
> +	mode &= 0xfb;
> +	OV5640_WRITE_REG(sensor, 0x3a00, mode);
> +	return 0;
> +}
> +
> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
> +{
> +	 /* read HTS from register settings */
> +	u16 HTS;

Function names and variables are lower case.

> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380c, &HTS);
> +	return HTS;
> +}
> +
> +static int ov5640_get_VTS(struct ov5640_dev *sensor)
> +{
> +	u16 VTS;
> +	int ret;
> +
> +	OV5640_READ_REG16(sensor, 0x380e, &VTS);
> +	return VTS;
> +}
> +
> +static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
> +{
> +	int ret;
> +
> +	OV5640_WRITE_REG16(sensor, 0x380e, VTS);
> +	return 0;
> +}
> +
> +static int ov5640_get_light_freq(struct ov5640_dev *sensor)
> +{
> +	/* get banding filter value */
> +	u8 temp, temp1;
> +	int light_freq = 0;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3c01, &temp);
> +
> +	if (temp & 0x80) {
> +		/* manual */
> +		OV5640_READ_REG(sensor, 0x3c00, &temp1);
> +		if (temp1 & 0x04) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +			light_freq = 60;
> +		}
> +	} else {
> +		/* auto */
> +		OV5640_READ_REG(sensor, 0x3c0c, &temp1);
> +		if (temp1 & 0x01) {
> +			/* 50Hz */
> +			light_freq = 50;
> +		} else {
> +			/* 60Hz */
> +		}
> +	}
> +
> +	return light_freq;
> +}
> +
> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
> +{
> +	int prev_vts;
> +	int band_step60, max_band60, band_step50, max_band50;

Aren't these values unsigned ?

> +	int ret;
> +
> +	/* read preview PCLK */
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_sysclk = ret;
> +	/* read preview HTS */
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	sensor->prev_hts = ret;
> +
> +	/* read preview VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_vts = ret;
> +
> +	/* calculate banding filter */
> +	/* 60Hz */
> +	band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
> +	OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
> +
> +	max_band60 = (int)((prev_vts-4)/band_step60);
> +	OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
> +
> +	/* 50Hz */
> +	band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
> +	OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
> +
> +	max_band50 = (int)((prev_vts-4)/band_step50);
> +	OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
> +{
> +	/* stable in high */
> +	int fast_high, fast_low;

Aren't these values unsigned ?

> +	int ret;
> +
> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
> +
> +	fast_high = sensor->ae_high<<1;

Missing spaces around <<

> +	if (fast_high > 255)
> +		fast_high = 255;
> +
> +	fast_low = sensor->ae_low >> 1;
> +
> +	OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
> +	OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
> +	OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);
> +
> +	return 0;
> +}
> +
> +static int ov5640_binning_on(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x3821, &temp);
> +	temp &= 0xfe;
> +
> +	return temp ? 1 : 0;
> +}
> +
> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
> +{
> +	u8 temp, channel = sensor->ep.base.id;

The endpoint id isn't meant to select a virtual channel. V4L2 has no virtual 
channel API at the moment, you can hardcode the VC to 0 for now.

> +	int ret;
> +
> +	OV5640_READ_REG(sensor, 0x4814, &temp);
> +	temp &= ~(3 << 6);
> +	temp |= (channel << 6);
> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
> +
> +	return 0;
> +}
> +
> +static enum ov5640_mode
> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
> +			 int width, int height)

How about using v4l2_find_nearest_format() ?

> +{
> +	int i;
> +
> +	for (i = ov5640_num_modes - 1; i >= 0; i--) {
> +		if (ov5640_mode_info_data[0][i].width <= width &&
> +		    ov5640_mode_info_data[0][i].height <= height)
> +			break;
> +	}
> +
> +	if (i < 0)
> +		i = 0;
> +
> +	return (enum ov5640_mode)i;
> +}
> +
> +/*
> + * sensor changes between scaling and subsampling, go through
> + * exposure calculation
> + */
> +static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
> +					    enum ov5640_frame_rate frame_rate,
> +					    enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	u8 average;
> +	int prev_shutter, prev_gain16;
> +	int cap_shutter, cap_gain16;
> +	int cap_sysclk, cap_hts, cap_vts;
> +	int light_freq, cap_bandfilt, cap_maxband;
> +	long cap_gain16_shutter;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* auto focus */
> +	/* ov5640_auto_focus();//if no af function, just skip it */
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read preview shutter */
> +	ret = ov5640_get_exposure(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_shutter = ret;
> +	ret = ov5640_binning_on(sensor);
> +	if (ret < 0)
> +		return ret;
> +	if (ret && mode != ov5640_mode_720P_1280_720 &&
> +	    mode != ov5640_mode_1080P_1920_1080)
> +		prev_shutter *= 2;
> +
> +	/* read preview gain */
> +	ret = ov5640_get_gain(sensor);
> +	if (ret < 0)
> +		return ret;
> +	prev_gain16 = ret;
> +
> +	/* get average */
> +	OV5640_READ_REG(sensor, 0x56a1, &average);
> +
> +	/* turn off night mode for capture */
> +	ret = ov5640_set_night_mode(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* turn off overlay */
> +	/* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
> +	   just skip it */
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* read capture VTS */
> +	ret = ov5640_get_VTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_vts = ret;
> +	ret = ov5640_get_HTS(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_hts = ret;
> +	ret = ov5640_get_sysclk(sensor);
> +	if (ret < 0)
> +		return ret;
> +	cap_sysclk = ret;
> +
> +	/* calculate capture banding filter */
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	light_freq = ret;
> +
> +	if (light_freq == 60) {
> +		/* 60Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
> +	} else {
> +		/* 50Hz */
> +		cap_bandfilt = cap_sysclk * 100 / cap_hts;
> +	}
> +	cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
> +
> +	/* calculate capture shutter/gain16 */
> +	if (average > sensor->ae_low && average < sensor->ae_high) {
> +		/* in stable range */
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts *
> +			sensor->ae_target / average;
> +	} else {
> +		cap_gain16_shutter =
> +			prev_gain16 * prev_shutter *
> +			cap_sysclk / sensor->prev_sysclk *
> +			sensor->prev_hts / cap_hts;
> +	}
> +
> +	/* gain to shutter */
> +	if (cap_gain16_shutter < (cap_bandfilt * 16)) {
> +		/* shutter < 1/100 */
> +		cap_shutter = cap_gain16_shutter / 16;
> +		if (cap_shutter < 1)
> +			cap_shutter = 1;
> +
> +		cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		if (cap_gain16 < 16)
> +			cap_gain16 = 16;
> +	} else {
> +		if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
> +			/* exposure reach max */
> +			cap_shutter = cap_bandfilt * cap_maxband;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		} else {
> +			/* 1/100 < (cap_shutter = n/100) =< max */
> +			cap_shutter =
> +				((int)(cap_gain16_shutter / 16 / 
cap_bandfilt))
> +				* cap_bandfilt;
> +			cap_gain16 = cap_gain16_shutter / cap_shutter;
> +		}
> +	}
> +
> +	/* write capture gain */
> +	ret = ov5640_set_gain(sensor, cap_gain16);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* write capture shutter */
> +	if (cap_shutter > (cap_vts - 4)) {
> +		cap_vts = cap_shutter + 4;
> +		ret = ov5640_set_VTS(sensor, cap_vts);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	ret = ov5640_set_exposure(sensor, cap_shutter);
> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_stream(sensor, true);
> +}
> +
> +/*
> + * if sensor changes inside scaling or subsampling
> + * change mode directly
> + */
> +static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
> +				     enum ov5640_frame_rate frame_rate,
> +				     enum ov5640_mode mode)
> +{
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;
> +
> +	/* check if the input mode and frame rate is valid */
> +	mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
> +	mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
> +
> +	sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
> +	sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
> +
> +	if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
> +	    mode_data == NULL || mode_size == 0)
> +		return -EINVAL;
> +
> +	/* turn off AE/AG */
> +	ret = ov5640_set_agc(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, false);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Write capture setting */
> +	ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_stream(sensor, true);

This will turn streaming on when you set the format, which I don't think is 
correct. You should only turn streaming on in the .s_stream() operation. The 
same comment applies to the previous function.

> +	if (ret < 0)
> +		return ret;
> +
> +	return ov5640_set_agc(sensor, true);
> +}
> +
> +static int ov5640_change_mode(struct ov5640_dev *sensor,
> +			      enum ov5640_frame_rate frame_rate,
> +			      enum ov5640_mode mode,
> +			      enum ov5640_mode orig_mode)
> +{
> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
> +	struct reg_value *mode_data = NULL;
> +	int mode_size = 0;
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
> +	    mode != ov5640_mode_INIT) {
> +		v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");
> +		return -EINVAL;
> +	}
> +
> +	dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
> +	orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
> +	if (mode == ov5640_mode_INIT) {
> +		mode_data = ov5640_init_setting_30fps_VGA;
> +		mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
> +
> +		sensor->fmt.width = 640;
> +		sensor->fmt.height = 480;

Don't reset the format here. The format must be preserved across subdev 
open/close.

> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +		if (ret < 0)
> +			return ret;
> +
> +		mode_data = ov5640_setting_30fps_VGA_640_480;
> +		mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
> +		ret = ov5640_load_regs(sensor, mode_data, mode_size);
> +	} else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
> +			(dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
> +		/* change between subsampling and scaling
> +		 * go through exposure calucation */
> +		ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
> +							  mode);
> +	} else {
> +		/* change inside subsampling or scaling
> +		 * download firmware directly */
> +		ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
> +	}
> +
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = ov5640_set_AE_target(sensor, sensor->ae_target);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_get_light_freq(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_bandingfilter(sensor);
> +	if (ret < 0)
> +		return ret;
> +	ret = ov5640_set_virtual_channel(sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* restore controls */
> +	ov5640_restore_ctrls(sensor);
> +
> +	if (ret >= 0 && mode != ov5640_mode_INIT) {
> +		sensor->current_mode = mode;
> +		sensor->current_fr = frame_rate;
> +	}
> +
> +	return 0;
> +}
> +
> +/* restore the last set video mode after chip power-on */
> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
> +{
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	/* first we need to set some initial register values */
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				    ov5640_mode_INIT, ov5640_mode_INIT);

I wouldn't use ov5640_change_mode() here. You only need to apply the init 
register values, just call ov5640_load_regs(). The rest of the 
ov5640_change_mode() isn't needed, as you're calling it below. This will also 
allow you to get rid of ov5640_mode_INIT, simplifying the logic.

> +	if (ret < 0)
> +		return ret;
> +
> +	/* now restore the last capture mode */
> +	return ov5640_change_mode(sensor,
> +				  sensor->current_fr,
> +				  sensor->current_mode,
> +				  ov5640_mode_VGA_640_480);

You can fit that in fewer lines while still staying within the 80 columns 
limit.

> +}
> +
> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
> +{
> +	int ret;
> +
> +	if (sensor->io_regulator) {
> +		ret = regulator_enable(sensor->io_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->core_regulator) {
> +		ret = regulator_enable(sensor->core_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->gpo_regulator) {
> +		ret = regulator_enable(sensor->gpo_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
> +			return ret;
> +		}
> +	}
> +	if (sensor->analog_regulator) {
> +		ret = regulator_enable(sensor->analog_regulator);
> +		if (ret) {
> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
> +			return ret;
> +		}
> +	}

Maybe you should use the bulk regulator API ?

> +
> +	return 0;
> +}
> +
> +static void ov5640_regulators_off(struct ov5640_dev *sensor)
> +{
> +	if (sensor->analog_regulator)
> +		regulator_disable(sensor->analog_regulator);
> +	if (sensor->core_regulator)
> +		regulator_disable(sensor->core_regulator);
> +	if (sensor->io_regulator)
> +		regulator_disable(sensor->io_regulator);
> +	if (sensor->gpo_regulator)
> +		regulator_disable(sensor->gpo_regulator);
> +}
> +
> +/* --------------- Subdev Operations --------------- */
> +
> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	int ret;
> +
> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> +	if (on && !sensor->on) {

The .s_power() calls need to be ref-counted, similarly to how the regulator 
enable/disable calls work. See the mt9p031 sensor driver for an example.

> +		if (sensor->xclk)
> +			clk_prepare_enable(sensor->xclk);
> +
> +		ret = ov5640_regulators_on(sensor);
> +		if (ret)
> +			return ret;
> +
> +		ov5640_reset(sensor);
> +		ov5640_power(sensor, true);
> +
> +		ret = ov5640_init_slave_id(sensor);

Why is this needed ?

> +		if (ret)
> +			return ret;
> +
> +		ret = ov5640_restore_mode(sensor);
> +		if (ret)
> +			return ret;
> +
> +		/*
> +		 * NOTE: Freescale adds a long delay (600 msec) after
> +		 * powering up and programming a mode on the ov5640-mipi
> +		 * camera (search for "msec_wait4stable" in FSL's
> +		 * ov5640_mipi.c), which equivalently would need to go
> +		 * right here. If we run into MIPI CSI-2 receiver dphy
> +		 * ready timeouts, it might be a clue to add that delay
> +		 * here.
> +		 */
> +	} else if (!on && sensor->on) {
> +		ov5640_power(sensor, false);
> +
> +		ov5640_regulators_off(sensor);
> +
> +		if (sensor->xclk)
> +			clk_disable_unprepare(sensor->xclk);
> +	}
> +
> +	sensor->on = on;
> +
> +	return 0;
> +}
> +
> +static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_captureparm *cparm = &a->parm.capture;
> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* This is the only case currently handled. */
> +	memset(a, 0, sizeof(*a));
> +	a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	cparm->capability = sensor->streamcap.capability;
> +	cparm->timeperframe = sensor->streamcap.timeperframe;
> +	cparm->capturemode = sensor->streamcap.capturemode;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
> +	enum ov5640_frame_rate frame_rate;
> +	u32 tgt_fps;	/* target frames per secound */
> +	int ret = 0;

No need to initialize ret to 0.

> +
> +	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return -EINVAL;
> +
> +	/* Check that the new frame rate is allowed. */
> +	if ((timeperframe->numerator == 0) ||
> +	    (timeperframe->denominator == 0)) {
> +		timeperframe->denominator = DEFAULT_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps > MAX_FPS) {
> +		timeperframe->denominator = MAX_FPS;
> +		timeperframe->numerator = 1;
> +	} else if (tgt_fps < MIN_FPS) {
> +		timeperframe->denominator = MIN_FPS;
> +		timeperframe->numerator = 1;
> +	}
> +
> +	/* Actual frame rate we use */
> +	tgt_fps = timeperframe->denominator / timeperframe->numerator;
> +
> +	if (tgt_fps == 15)
> +		frame_rate = ov5640_15_fps;
> +	else if (tgt_fps == 30)
> +		frame_rate = ov5640_30_fps;
> +	else {
> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
> +			 tgt_fps);

Don't print an error message that is userspace-triggerable, we have enough 
ways for applications to flood the kernel log already :-)

> +		return -EINVAL;
> +	}
> +
> +	ret = ov5640_change_mode(sensor, frame_rate,
> +				 sensor->current_mode,
> +				 sensor->current_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	sensor->streamcap.timeperframe = *timeperframe;
> +
> +	return 0;
> +}
> +
> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	format->format = sensor->fmt;

You need to handle the TRY format here. You can have a look at the mt9p031 
driver for an example on how to do so.

> +
> +	return 0;
> +}
> +
> +static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
> +				   struct v4l2_mbus_framefmt *fmt,
> +				   enum ov5640_mode *new_mode)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
> +
> +	fmt->width = ov5640_mode_info_data[0][mode].width;
> +	fmt->height = ov5640_mode_info_data[0][mode].height;
> +	fmt->code = sensor->fmt.code;
> +
> +	if (new_mode)
> +		*new_mode = mode;
> +	return 0;
> +}
> +
> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_pad_config *cfg,
> +			  struct v4l2_subdev_format *format)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode new_mode;
> +	int ret;
> +
> +	if (format->pad != 0)
> +		return -EINVAL;
> +
> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
> +		if (ret)
> +			return ret;

You can move this code above the test to avoid the duplicated call below.

> +		cfg->try_fmt = format->format;

Please use the v4l2_subdev_get_try_format() function to get the try format, 
don't dereference cfg directly.

> +		return 0;
> +	}
> +
> +	ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);
> +	if (ret)
> +		return ret;
> +
> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
> +				 new_mode, sensor->current_mode);
> +	if (ret >= 0)
> +		sensor->fmt = format->format;
> +
> +	return ret;
> +}
> +
> +
> +/*
> + * Sensor Controls.
> + */
> +
> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);

According to the kernel coding style, you need curly braces around the else 
branch if you use them around the if branch.

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
> +		OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);

Ditto

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (value) {
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
> +		OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
> +	} else
> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);

Ditto

> +
> +	return 0;
> +}
> +
> +static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	sensor->awb_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);
> +	return 0;
> +}
> +
> +static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +#if 0

No compiled-out code please.

> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +#endif
> +
> +static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->awb_on)
> +		return -EINVAL;
> +
> +	OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
> +	OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);
> +	return 0;
> +}
> +
> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
> +{
> +	u16 max_exp = 0;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350c, &max_exp);
> +	if (value < max_exp) {
> +		u32 exp = value << 4;
> +
> +		OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
> +		OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);
> +	}
> +
> +	return 0;
> +}
> +
> +/* read exposure, in number of line periods */
> +static int ov5640_get_exposure(struct ov5640_dev *sensor)
> +{
> +	u8 temp;
> +	int exp, ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG(sensor, 0x3500, &temp);
> +	exp = ((int)temp & 0x0f) << 16;
> +	OV5640_READ_REG(sensor, 0x3501, &temp);
> +	exp |= ((int)temp << 8);
> +	OV5640_READ_REG(sensor, 0x3502, &temp);
> +	exp |= (int)temp;
> +
> +	return exp >> 4;
> +}
> +
> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	/* this enables/disables both AEC and AGC */
> +	sensor->agc_on = value ? true : false;
> +	OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);
> +
> +	return 0;
> +}
> +
> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;

You can create a cluster with the autogain and manual gain controls 
(v4l2_ctrl_auto_cluster) to have the control framework disabling the manual 
control automatically when autogain is enabled.

> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
> +	return 0;
> +}
> +
> +static int ov5640_get_gain(struct ov5640_dev *sensor)
> +{
> +	u16 gain;
> +	int ret;
> +
> +	if (sensor->agc_on)
> +		return -EINVAL;
> +
> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
> +
> +	return gain & 0x3ff;
> +}
> +
> +#if 0
> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
> +{
> +	int ret;
> +
> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
> +	return 0;
> +}
> +#endif

You can use a control to expose the test pattern function, it's quite useful.

> +static struct ov5640_control ov5640_ctrls[] = {
> +	{
> +		.set = ov5640_set_agc,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTOGAIN,
> +			.name = "Auto Gain/Exposure Control",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_exposure,
> +		.ctrl = {
> +			.id = V4L2_CID_EXPOSURE,
> +			.name = "Exposure",
> +			.minimum = 0,
> +			.maximum = 65535,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_gain,
> +		.ctrl = {
> +			.id = V4L2_CID_GAIN,
> +			.name = "Gain",
> +			.minimum = 0,
> +			.maximum = 1023,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_hue,
> +		.ctrl = {
> +			.id = V4L2_CID_HUE,
> +			.name = "Hue",
> +			.minimum = 0,
> +			.maximum = 359,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_contrast,
> +		.ctrl = {
> +			.id = V4L2_CID_CONTRAST,
> +			.name = "Contrast",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_saturation,
> +		.ctrl = {
> +			.id = V4L2_CID_SATURATION,
> +			.name = "Saturation",
> +			.minimum = 0,
> +			.maximum = 255,
> +			.step = 1,
> +			.default_value = 64,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_awb,
> +		.ctrl = {
> +			.id = V4L2_CID_AUTO_WHITE_BALANCE,
> +			.name = "Auto White Balance",
> +			.minimum = 0,
> +			.maximum = 1,
> +			.step = 1,
> +			.default_value = 1,
> +			.type = V4L2_CTRL_TYPE_BOOLEAN,
> +		},
> +	}, {
> +		.set = ov5640_set_red_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_RED_BALANCE,
> +			.name = "Red Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	}, {
> +		.set = ov5640_set_blue_balance,
> +		.ctrl = {
> +			.id = V4L2_CID_BLUE_BALANCE,
> +			.name = "Blue Balance",
> +			.minimum = 0,
> +			.maximum = 4095,
> +			.step = 1,
> +			.default_value = 0,
> +			.type = V4L2_CTRL_TYPE_INTEGER,
> +		},
> +	},
> +};
> +#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
> +
> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
> +{
> +	struct ov5640_control *ret = NULL;
> +	int i;

i is never negative, it should be unsigned int.

> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		if (id == ov5640_ctrls[i].ctrl.id) {
> +			ret = &ov5640_ctrls[i];
> +			break;
> +		}
> +	}
> +
> +	if (ret && index)
> +		*index = i;
> +	return ret;
> +}
> +
> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;
> +	int i;

i is never negative, it should be unsigned int.

> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +		c->set(sensor, sensor->ctrl_cache[i]);
> +	}
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
> +	struct ov5640_control *c;
> +	int ret = 0;
> +	int i;
> +
> +	c = ov5640_get_ctrl(ctrl->id, &i);

You can inline the function call here as it's not used anywhere else.

> +	if (!c)
> +		return -EINVAL;
> +
> +	ret = c->set(sensor, ctrl->val);
> +	/* update cached value if no error */
> +	if (!ret)
> +		sensor->ctrl_cache[i] = ctrl->val;
> +
> +	return ret;
> +}
> +
> +static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
> +	.s_ctrl = ov5640_s_ctrl,
> +};
> +
> +static int ov5640_init_controls(struct ov5640_dev *sensor)
> +{
> +	struct ov5640_control *c;

I would name this ctrl or control, c can be a bit confusing. You can also 
declare it inside the loop.

> +	int i;

i can never be negative, you can make it an unsigned int.

> +
> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
> +
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		c = &ov5640_ctrls[i];
> +
> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
> +				  c->ctrl.id, c->ctrl.minimum, c-
>ctrl.maximum,
> +				  c->ctrl.step, c->ctrl.default_value);
> +	}
> +
> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
> +	if (sensor->ctrl_hdl.error) {
> +		int err = sensor->ctrl_hdl.error;
> +
> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);

I'm not sure this brings much value.

> +		return err;
> +	}
> +	v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);

You shouldn't call this function here for the reason explained below.

> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_frame_size_enum *fse)
> +{
> +	if (fse->pad != 0)
> +		return -EINVAL;
> +	if (fse->index >= ov5640_num_modes)
> +		return -EINVAL;
> +
> +	fse->min_width = fse->max_width =
> +		ov5640_mode_info_data[0][fse->index].width;
> +	fse->min_height = fse->max_height =
> +		ov5640_mode_info_data[0][fse->index].height;
> +
> +	return 0;
> +}
> +
> +static int ov5640_enum_frame_interval(
> +	struct v4l2_subdev *sd,
> +	struct v4l2_subdev_pad_config *cfg,
> +	struct v4l2_subdev_frame_interval_enum *fie)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +	enum ov5640_mode mode;
> +
> +	if (fie->pad != 0)
> +		return -EINVAL;
> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
> +		return -EINVAL;
> +
> +	if (fie->width == 0 || fie->height == 0)
> +		return -EINVAL;
> +
> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);

You should find an exact mode here, not the nearest one. If with and height 
don't match, return -EINVAL. That will replace with above width == 0 and 
height == 0 test.

> +	if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
> +		return -EINVAL;
> +
> +	fie->interval.numerator = 1;
> +	fie->interval.denominator = ov5640_framerates[fie->index];
> +
> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
> +		fie->width, fie->height, fie->index, fie-
>interval.denominator);

I'm not sure this is very useful, you can use ftrace if you want to trace 
function calls.

> +	return 0;
> +}
> +
> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
> +
> +	return 0;
> +}
> +
> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
> +			    u32 output, u32 config)
> +{
> +	return (input != 0) ? -EINVAL : 0;
> +}

The g_input_status and s_routing subdev operations are not mandatory, you 
don't have to implement them as the sensor doesn't have multiple inputs.

[snip]

> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

You can use ftrace to trace function calls, there's no need to add debugging 
statements that duplicate the functionality.

> +	return ov5640_set_stream(sensor, enable);
> +}
> +
> +static struct v4l2_subdev_core_ops ov5640_core_ops = {
> +	.s_power = ov5640_s_power,
> +};
> +
> +static struct v4l2_subdev_video_ops ov5640_video_ops = {
> +	.s_parm = ov5640_s_parm,
> +	.g_parm = ov5640_g_parm,
> +	.g_input_status = ov5640_g_input_status,
> +	.s_routing = ov5640_s_routing,
> +	.g_mbus_config  = ov5640_g_mbus_config,
> +	.s_stream = ov5640_s_stream,
> +};
> +
> +static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
> +	.enum_mbus_code = ov5640_enum_mbus_code,
> +	.get_fmt = ov5640_get_fmt,
> +	.set_fmt = ov5640_set_fmt,
> +	.enum_frame_size = ov5640_enum_frame_size,
> +	.enum_frame_interval = ov5640_enum_frame_interval,
> +};
> +
> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
> +	.core = &ov5640_core_ops,
> +	.video = &ov5640_video_ops,
> +	.pad = &ov5640_pad_ops,
> +};

All these structures should be static const.

> +
> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
> +{
> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
> +}
> +
> +static void ov5640_reset(struct ov5640_dev *sensor)
> +{
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +
> +	/* camera power cycle */
> +	ov5640_power(sensor, false);
> +	usleep_range(5000, 10000);
> +	ov5640_power(sensor, true);
> +	usleep_range(5000, 10000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 1);
> +	usleep_range(1000, 2000);
> +
> +	gpiod_set_value(sensor->reset_gpio, 0);
> +	usleep_range(5000, 10000);
> +}
> +
> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
> +{
> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
> +	if (!IS_ERR(sensor->io_regulator)) {
> +		regulator_set_voltage(sensor->io_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_IO,
> +				      OV5640_VOLTAGE_DIGITAL_IO);
> +	} else {
> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
> +			__func__);
> +		sensor->io_regulator = NULL;

How about making the power supplies mandatory in DT instead ? They are 
mandatory after all, if they're not controllable DT should just declare fixed 
supplies.

> +	}
> +
> +	sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
> +	if (!IS_ERR(sensor->core_regulator)) {
> +		regulator_set_voltage(sensor->core_regulator,
> +				      OV5640_VOLTAGE_DIGITAL_CORE,
> +				      OV5640_VOLTAGE_DIGITAL_CORE);
> +	} else {
> +		sensor->core_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
> +			__func__);
> +	}
> +
> +	sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
> +	if (!IS_ERR(sensor->analog_regulator)) {
> +		regulator_set_voltage(sensor->analog_regulator,
> +				      OV5640_VOLTAGE_ANALOG,
> +				      OV5640_VOLTAGE_ANALOG);
> +	} else {
> +		sensor->analog_regulator = NULL;
> +		dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
> +			__func__);
> +	}
> +}
> +
> +static int ov5640_probe(struct i2c_client *client,
> +			const struct i2c_device_id *id)
> +{
> +	struct device *dev = &client->dev;
> +	struct device_node *endpoint;
> +	struct ov5640_dev *sensor;
> +	int i, xclk, ret;

i and xclk are never negative, you can make them unsigned int.

> +
> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);

Please use sizeof(*variable) instead of sizeof(type).

devm_kzalloc() doesn't play nicely with dynamic removal of devices. We're in 
the process of fixing related race conditions in the media subsystem. In order 
not to make the problem worse, please use kzalloc() instead.

> +	if (!sensor)
> +		return -ENOMEM;
> +
> +	sensor->i2c_client = client;
> +	sensor->dev = dev;

Do you really need to store both i2c_client and dev, given that the latter is 
just &client->dev ?

> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
> +	sensor->fmt.width = 640;
> +	sensor->fmt.height = 480;
> +	sensor->fmt.field = V4L2_FIELD_NONE;
> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
> +					   V4L2_CAP_TIMEPERFRAME;

Please fix the indentation.

> +	sensor->streamcap.capturemode = 0;
> +	sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
> +	sensor->streamcap.timeperframe.numerator = 1;
> +
> +	sensor->current_mode = ov5640_mode_VGA_640_480;
> +	sensor->current_fr = ov5640_30_fps;
> +
> +	sensor->ae_target = 52;
> +
> +	endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
> +	if (!endpoint) {
> +		dev_err(dev, "endpoint node not found\n");
> +		return -EINVAL;
> +	}
> +
> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
> +		return -EINVAL;

You're leaking endpoint here. You should move the of_node_put() call right 
after the v4l2_of_parse_endpoint() call.

> +	}
> +	of_node_put(endpoint);
> +
> +	/* get system clock (xclk) frequency */
> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);

Instead of adding a custom DT property for this, use assigned-clock-rates. You 
won't need to set it manually in the driver, and can verify its frequency with 
clk_get_rate().

> +	if (!ret) {
> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {

Are your register tables above independent of the clock frequency ? You should 
ideally compute register values at runtime instead of hardcoding them, but 
given the lack of information from Omnivision I understand this isn't 
possible. You thus need to be stricter here and reject any value other than 
the nominal frequency.

> +			dev_err(dev, "invalid xclk frequency\n");
> +			return -EINVAL;
> +		}
> +		sensor->xclk_freq = xclk;
> +	}
> +
> +	/* get system clock (xclk) */
> +	sensor->xclk = devm_clk_get(dev, "xclk");
> +	if (!IS_ERR(sensor->xclk)) {
> +		if (!sensor->xclk_freq) {
> +			dev_err(dev, "xclk requires xclk frequency!\n");
> +			return -EINVAL;
> +		}
> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
> +	} else {
> +		/* assume system clock enabled by default */
> +		sensor->xclk = NULL;

Please don't. The clock should be mandatory.

> +	}
> +
> +	/* request power down pin */
> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);

Are the GPIOs mandatory or optional ? Can a system tie some of them to ground 
or a voltage rail, or do they need to always be manually controllable ?

> +	if (IS_ERR(sensor->pwdn_gpio)) {
> +		dev_err(dev, "request for power down gpio failed\n");
> +		return PTR_ERR(sensor->pwdn_gpio);
> +	}
> +
> +	/* request reset pin */
> +	sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
> +	if (IS_ERR(sensor->reset_gpio)) {
> +		dev_err(dev, "request for reset gpio failed\n");
> +		return PTR_ERR(sensor->reset_gpio);
> +	}
> +
> +	/* initialize the cached controls to their defaults */
> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
> +		struct ov5640_control *c = &ov5640_ctrls[i];
> +
> +		sensor->ctrl_cache[i] = c->ctrl.default_value;
> +	}
> +	sensor->awb_on = sensor->agc_on = true;
> +
> +	v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
> +
> +	sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
> +	sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> +	ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);
> +	if (ret)
> +		return ret;
> +
> +	ov5640_get_regulators(sensor);
> +
> +	ret = ov5640_s_power(&sensor->sd, 1);
> +	if (ret)
> +		goto entity_cleanup;
> +	ret = ov5640_init_controls(sensor);
> +	if (ret)
> +		goto power_off;
> +
> +	ret = ov5640_s_power(&sensor->sd, 0);
> +	if (ret)
> +		goto free_ctrls;

Writing the controls here is pointless, as powering the chip down will lose 
all the values. You shouldn't call v4l2_ctrl_handler_setup() in 
ov5640_init_controls(), and you can then remove the ov5640_s_power() calls 
here.

> +	ret = v4l2_async_register_subdev(&sensor->sd);
> +	if (ret)
> +		goto free_ctrls;
> +
> +	return 0;
> +
> +free_ctrls:
> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +power_off:
> +	ov5640_s_power(&sensor->sd, 0);
> +entity_cleanup:
> +	media_entity_cleanup(&sensor->sd.entity);
> +	ov5640_regulators_off(sensor);

Won't ov5640_s_power(0) already disable the regulators ?

> +	return ret;
> +}
> +
> +/*!
> + * ov5640 I2C detach function
> + *
> + * @param client            struct i2c_client *
> + * @return  Error code indicating success or failure
> + */

That's not the kerneldoc comment style. Given that this is the only documented 
function, and that the comment is completely useless, you can just drop it.

> +static int ov5640_remove(struct i2c_client *client)
> +{
> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
> +
> +	ov5640_regulators_off(sensor);
> +
> +	v4l2_async_unregister_subdev(&sensor->sd);
> +	media_entity_cleanup(&sensor->sd.entity);
> +	v4l2_device_unregister_subdev(sd);

This function is called by v4l2_async_unregister_subdev(), there's no need to 
duplicate it.

> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
> +
> +	return 0;
> +}

[snip]

> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");

MODULE_AUTHOR isn't a synonym for copyright ownership. I don't think you 
should add Freescale as an author. If you know who wrote the original code you 
can list that developer explicitly.

> +MODULE_AUTHOR("Steve Longerbeam <steve_longerbeam@mentor.com>");
> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
> +MODULE_LICENSE("GPL");
> +MODULE_VERSION("1.0");

Version numbers are never updated. I wouldn't bother adding one.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-02 11:50     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-02 11:50 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
> for sensors with a MIPI CSI2 interface.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile        |   1 +
>  drivers/staging/media/imx/imx-mipi-csi2.c | 501 ++++++++++++++++++++++++++++++
>  2 files changed, 502 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index fe9e992..0decef7 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
> diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
> new file mode 100644
> index 0000000..daa6e1d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
> @@ -0,0 +1,501 @@
> +/*
> + * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
> + *
> + * Copyright (c) 2012-2014 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/irq.h>
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +/*
> + * there must be 5 pads: 1 input pad from sensor, and
> + * the 4 virtual channel output pads
> + */
> +#define CSI2_NUM_SINK_PADS  1
> +#define CSI2_NUM_SRC_PADS   4
> +#define CSI2_NUM_PADS       5
> +
> +struct imxcsi2_dev {
> +	struct device          *dev;
> +	struct imx_media_dev   *md;
> +	struct v4l2_subdev      sd;
> +	struct media_pad       pad[CSI2_NUM_PADS];
> +	struct v4l2_mbus_framefmt format_mbus;
> +	struct v4l2_subdev     *src_sd;
> +	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];

I see no reason to store pointers to the remote v4l2_subdevs.

> +	int                    input_pad;
> +	struct clk             *dphy_clk;
> +	struct clk             *cfg_clk;
> +	struct clk             *pix_clk; /* what is this? */
> +	void __iomem           *base;
> +	int                     intr1;
> +	int                     intr2;

The interrupts are not used, I'd remove them and the dead code in
_probe.

> +	struct v4l2_of_bus_mipi_csi2 bus;
> +	bool                    on;
> +	bool                    stream_on;
> +};
> +
> +#define DEVICE_NAME "imx6-mipi-csi2"
> +
> +/* Register offsets */
> +#define CSI2_VERSION            0x000
> +#define CSI2_N_LANES            0x004
> +#define CSI2_PHY_SHUTDOWNZ      0x008
> +#define CSI2_DPHY_RSTZ          0x00c
> +#define CSI2_RESETN             0x010
> +#define CSI2_PHY_STATE          0x014
> +#define CSI2_DATA_IDS_1         0x018
> +#define CSI2_DATA_IDS_2         0x01c
> +#define CSI2_ERR1               0x020
> +#define CSI2_ERR2               0x024
> +#define CSI2_MSK1               0x028
> +#define CSI2_MSK2               0x02c
> +#define CSI2_PHY_TST_CTRL0      0x030
> +#define CSI2_PHY_TST_CTRL1      0x034
> +#define CSI2_SFT_RESET          0xf00
> +
> +static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
> +{
> +	return container_of(sdev, struct imxcsi2_dev, sd);
> +}
> +
> +static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
> +{
> +	return readl(csi2->base + regoff);
> +}
> +
> +static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
> +				 unsigned int regoff)
> +{
> +	writel(val, csi2->base + regoff);
> +}

Do those two wrappers really make the code more readable?

> +static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
> +{
> +	int lanes = csi2->bus.num_data_lanes;
> +
> +	imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
> +}
> +
> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
> +{
> +	if (enable) {
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);

Given that these registers only contain a single bit, and bits 31:1 are
documented as reserved, 0, I think these should write 1 instead of
0xffffffff.

> +	} else {
> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
> +	}
> +}
> +
> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
> +{
> +	imxcsi2_enable(csi2, false);
> +
> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);

These magic constants should be replaced with proper defines for the
documented bitfields, if available.

#define PHY_TESTCLR		BIT(0)
#define PHY_TESTCLK		BIT(1)

#define PHY_TESTEN		BIT(16)

	/* Clear PHY test interface */
	imxcsi2_write(csi2, PHY_TESTCLR, CSI2_PHY_TST_CTRL0);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

	/* Raise test interface strobe signal */
	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);

	/* Configure address write on falling edge and lower strobe signal */
	u8 addr = 0x44;
	imxcsi2_write(csi2, PHY_TESTEN | addr, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

	/* Configure data write on rising edge and raise strobe signal */
	u8 data = 0x14;
	imxcsi2_write(csi2, data, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);

	/* Clear strobe signal */
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

The whole sequence should probably be encapsulated in a
dw_mipi_dphy_write function.

Actually, this exact function already exists as dw_mipi_dsi_phy_write in
drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
register 0x44 might contain a field called HSFREQRANGE_SEL.

> +	imxcsi2_enable(csi2, true);
> +}
> +
> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
> +{
> +	u32 reg;
> +	int i;
> +
> +	/* wait for mipi sensor ready */

More specifically, wait for the clock lane module to leave ULP state.

> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg != 0x200)

Magic constants are bad. This is PHY_RXULPSCLKNOT (clock lane module in
ultra low power state).

> +			break;
> +		usleep_range(10000, 20000);
> +	}

How about breaking this out into a wait function, or even better, using
readl_poll_timeout instead of open coding these loops multiple times?

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* wait for mipi stable */

Wait for error free transmission?

> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_ERR1);
> +		if (reg == 0x0)
> +			break;
> +		usleep_range(10000, 20000);
> +	}

readl_poll_timeout

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for controller timeout, err1 = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* finally let's wait for active clock on the clock lane */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg & (1 << 8))

Yes, and that is PHY_RXCLKACTIVEHS.

> +			break;
> +		usleep_range(10000, 20000);
> +	}

readl_poll_timeout

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for active clock timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
> +		  imxcsi2_read(csi2, CSI2_VERSION));

Noisy. (These are already removed for the next version.)

> +	return 0;
> +}
> +
> +/*
> + * V4L2 subdev operations
> + */
> +
> +static int imxcsi2_link_setup(struct media_entity *entity,
> +			      const struct media_pad *local,
> +			      const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (csi2->sink_sd[local->index])
> +				return -EBUSY;
> +			csi2->sink_sd[local->index] = remote_sd;
> +		} else {
> +			csi2->sink_sd[local->index] = NULL;
> +		}
> +	} else {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (csi2->src_sd)
> +				return -EBUSY;
> +			csi2->src_sd = remote_sd;
> +		} else {
> +			csi2->src_sd = NULL;
> +		}
> +	}

This whole code block is just to check if there is another link already
active on the given pad. This could be stored in a boolean or a bit, no
need to store pointers to remote subdevices.

> +	return 0;
> +}
> +
> +static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	if (on && !csi2->on) {
> +		v4l2_info(&csi2->sd, "power ON\n");
> +		clk_prepare_enable(csi2->cfg_clk);
> +		clk_prepare_enable(csi2->dphy_clk);
> +		imxcsi2_set_lanes(csi2);
> +		imxcsi2_reset(csi2);
> +	} else if (!on && csi2->on) {
> +		v4l2_info(&csi2->sd, "power OFF\n");
> +		imxcsi2_enable(csi2, false);
> +		clk_disable_unprepare(csi2->dphy_clk);
> +		clk_disable_unprepare(csi2->cfg_clk);
> +	}
> +
> +	csi2->on = on;
> +	return 0;
> +}
> +
> +static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +	int i, ret = 0;
> +
> +	if (!csi2->src_sd)
> +		return -EPIPE;
> +	for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
> +		if (csi2->sink_sd[i])
> +			break;
> +	}
> +	if (i >= CSI2_NUM_SRC_PADS)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	if (enable && !csi2->stream_on) {
> +		ret = clk_prepare_enable(csi2->pix_clk);
> +		if (ret)
> +			return ret;
> +
> +		ret = imxcsi2_dphy_wait(csi2);
> +		if (ret) {
> +			clk_disable_unprepare(csi2->pix_clk);
> +			return ret;
> +		}
> +	} else if (!enable && csi2->stream_on) {
> +		clk_disable_unprepare(csi2->pix_clk);
> +	}
> +
> +	csi2->stream_on = enable;
> +	return 0;
> +}
> +
> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_pad_config *cfg,
> +			   struct v4l2_subdev_format *sdformat)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	sdformat->format = csi2->format_mbus;

The output formats are different from the input formats, see the media
bus format discussion in the other thread. The input pad is the MIPI
CSI-2 bus, but the four output pads are connected to the muxes / CSIs
via a 16-bit parallel bus.

So if the input format is UYVY8_1X16, for example, the output should be
set to UYVY8_2X8.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-02 11:50     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-02 11:50 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
> for sensors with a MIPI CSI2 interface.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
> ---
>  drivers/staging/media/imx/Makefile        |   1 +
>  drivers/staging/media/imx/imx-mipi-csi2.c | 501 ++++++++++++++++++++++++++++++
>  2 files changed, 502 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index fe9e992..0decef7 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
> diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
> new file mode 100644
> index 0000000..daa6e1d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
> @@ -0,0 +1,501 @@
> +/*
> + * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
> + *
> + * Copyright (c) 2012-2014 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/irq.h>
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +/*
> + * there must be 5 pads: 1 input pad from sensor, and
> + * the 4 virtual channel output pads
> + */
> +#define CSI2_NUM_SINK_PADS  1
> +#define CSI2_NUM_SRC_PADS   4
> +#define CSI2_NUM_PADS       5
> +
> +struct imxcsi2_dev {
> +	struct device          *dev;
> +	struct imx_media_dev   *md;
> +	struct v4l2_subdev      sd;
> +	struct media_pad       pad[CSI2_NUM_PADS];
> +	struct v4l2_mbus_framefmt format_mbus;
> +	struct v4l2_subdev     *src_sd;
> +	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];

I see no reason to store pointers to the remote v4l2_subdevs.

> +	int                    input_pad;
> +	struct clk             *dphy_clk;
> +	struct clk             *cfg_clk;
> +	struct clk             *pix_clk; /* what is this? */
> +	void __iomem           *base;
> +	int                     intr1;
> +	int                     intr2;

The interrupts are not used, I'd remove them and the dead code in
_probe.

> +	struct v4l2_of_bus_mipi_csi2 bus;
> +	bool                    on;
> +	bool                    stream_on;
> +};
> +
> +#define DEVICE_NAME "imx6-mipi-csi2"
> +
> +/* Register offsets */
> +#define CSI2_VERSION            0x000
> +#define CSI2_N_LANES            0x004
> +#define CSI2_PHY_SHUTDOWNZ      0x008
> +#define CSI2_DPHY_RSTZ          0x00c
> +#define CSI2_RESETN             0x010
> +#define CSI2_PHY_STATE          0x014
> +#define CSI2_DATA_IDS_1         0x018
> +#define CSI2_DATA_IDS_2         0x01c
> +#define CSI2_ERR1               0x020
> +#define CSI2_ERR2               0x024
> +#define CSI2_MSK1               0x028
> +#define CSI2_MSK2               0x02c
> +#define CSI2_PHY_TST_CTRL0      0x030
> +#define CSI2_PHY_TST_CTRL1      0x034
> +#define CSI2_SFT_RESET          0xf00
> +
> +static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
> +{
> +	return container_of(sdev, struct imxcsi2_dev, sd);
> +}
> +
> +static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
> +{
> +	return readl(csi2->base + regoff);
> +}
> +
> +static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
> +				 unsigned int regoff)
> +{
> +	writel(val, csi2->base + regoff);
> +}

Do those two wrappers really make the code more readable?

> +static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
> +{
> +	int lanes = csi2->bus.num_data_lanes;
> +
> +	imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
> +}
> +
> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
> +{
> +	if (enable) {
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);

Given that these registers only contain a single bit, and bits 31:1 are
documented as reserved, 0, I think these should write 1 instead of
0xffffffff.

> +	} else {
> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
> +	}
> +}
> +
> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
> +{
> +	imxcsi2_enable(csi2, false);
> +
> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);

These magic constants should be replaced with proper defines for the
documented bitfields, if available.

#define PHY_TESTCLR		BIT(0)
#define PHY_TESTCLK		BIT(1)

#define PHY_TESTEN		BIT(16)

	/* Clear PHY test interface */
	imxcsi2_write(csi2, PHY_TESTCLR, CSI2_PHY_TST_CTRL0);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

	/* Raise test interface strobe signal */
	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);

	/* Configure address write on falling edge and lower strobe signal */
	u8 addr = 0x44;
	imxcsi2_write(csi2, PHY_TESTEN | addr, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

	/* Configure data write on rising edge and raise strobe signal */
	u8 data = 0x14;
	imxcsi2_write(csi2, data, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);

	/* Clear strobe signal */
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

The whole sequence should probably be encapsulated in a
dw_mipi_dphy_write function.

Actually, this exact function already exists as dw_mipi_dsi_phy_write in
drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
register 0x44 might contain a field called HSFREQRANGE_SEL.

> +	imxcsi2_enable(csi2, true);
> +}
> +
> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
> +{
> +	u32 reg;
> +	int i;
> +
> +	/* wait for mipi sensor ready */

More specifically, wait for the clock lane module to leave ULP state.

> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg != 0x200)

Magic constants are bad. This is PHY_RXULPSCLKNOT (clock lane module in
ultra low power state).

> +			break;
> +		usleep_range(10000, 20000);
> +	}

How about breaking this out into a wait function, or even better, using
readl_poll_timeout instead of open coding these loops multiple times?

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* wait for mipi stable */

Wait for error free transmission?

> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_ERR1);
> +		if (reg == 0x0)
> +			break;
> +		usleep_range(10000, 20000);
> +	}

readl_poll_timeout

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for controller timeout, err1 = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* finally let's wait for active clock on the clock lane */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg & (1 << 8))

Yes, and that is PHY_RXCLKACTIVEHS.

> +			break;
> +		usleep_range(10000, 20000);
> +	}

readl_poll_timeout

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for active clock timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
> +		  imxcsi2_read(csi2, CSI2_VERSION));

Noisy. (These are already removed for the next version.)

> +	return 0;
> +}
> +
> +/*
> + * V4L2 subdev operations
> + */
> +
> +static int imxcsi2_link_setup(struct media_entity *entity,
> +			      const struct media_pad *local,
> +			      const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (csi2->sink_sd[local->index])
> +				return -EBUSY;
> +			csi2->sink_sd[local->index] = remote_sd;
> +		} else {
> +			csi2->sink_sd[local->index] = NULL;
> +		}
> +	} else {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (csi2->src_sd)
> +				return -EBUSY;
> +			csi2->src_sd = remote_sd;
> +		} else {
> +			csi2->src_sd = NULL;
> +		}
> +	}

This whole code block is just to check if there is another link already
active on the given pad. This could be stored in a boolean or a bit, no
need to store pointers to remote subdevices.

> +	return 0;
> +}
> +
> +static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	if (on && !csi2->on) {
> +		v4l2_info(&csi2->sd, "power ON\n");
> +		clk_prepare_enable(csi2->cfg_clk);
> +		clk_prepare_enable(csi2->dphy_clk);
> +		imxcsi2_set_lanes(csi2);
> +		imxcsi2_reset(csi2);
> +	} else if (!on && csi2->on) {
> +		v4l2_info(&csi2->sd, "power OFF\n");
> +		imxcsi2_enable(csi2, false);
> +		clk_disable_unprepare(csi2->dphy_clk);
> +		clk_disable_unprepare(csi2->cfg_clk);
> +	}
> +
> +	csi2->on = on;
> +	return 0;
> +}
> +
> +static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +	int i, ret = 0;
> +
> +	if (!csi2->src_sd)
> +		return -EPIPE;
> +	for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
> +		if (csi2->sink_sd[i])
> +			break;
> +	}
> +	if (i >= CSI2_NUM_SRC_PADS)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	if (enable && !csi2->stream_on) {
> +		ret = clk_prepare_enable(csi2->pix_clk);
> +		if (ret)
> +			return ret;
> +
> +		ret = imxcsi2_dphy_wait(csi2);
> +		if (ret) {
> +			clk_disable_unprepare(csi2->pix_clk);
> +			return ret;
> +		}
> +	} else if (!enable && csi2->stream_on) {
> +		clk_disable_unprepare(csi2->pix_clk);
> +	}
> +
> +	csi2->stream_on = enable;
> +	return 0;
> +}
> +
> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_pad_config *cfg,
> +			   struct v4l2_subdev_format *sdformat)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	sdformat->format = csi2->format_mbus;

The output formats are different from the input formats, see the media
bus format discussion in the other thread. The input pad is the MIPI
CSI-2 bus, but the four output pads are connected to the muxes / CSIs
via a 16-bit parallel bus.

So if the input format is UYVY8_1X16, for example, the output should be
set to UYVY8_2X8.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-02 11:50     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-02 11:50 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
> for sensors with a MIPI CSI2 interface.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  drivers/staging/media/imx/Makefile        |   1 +
>  drivers/staging/media/imx/imx-mipi-csi2.c | 501 ++++++++++++++++++++++++++++++
>  2 files changed, 502 insertions(+)
>  create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
> 
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index fe9e992..0decef7 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
>  obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
> diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
> new file mode 100644
> index 0000000..daa6e1d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c
> @@ -0,0 +1,501 @@
> +/*
> + * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
> + *
> + * Copyright (c) 2012-2014 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/irq.h>
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +/*
> + * there must be 5 pads: 1 input pad from sensor, and
> + * the 4 virtual channel output pads
> + */
> +#define CSI2_NUM_SINK_PADS  1
> +#define CSI2_NUM_SRC_PADS   4
> +#define CSI2_NUM_PADS       5
> +
> +struct imxcsi2_dev {
> +	struct device          *dev;
> +	struct imx_media_dev   *md;
> +	struct v4l2_subdev      sd;
> +	struct media_pad       pad[CSI2_NUM_PADS];
> +	struct v4l2_mbus_framefmt format_mbus;
> +	struct v4l2_subdev     *src_sd;
> +	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];

I see no reason to store pointers to the remote v4l2_subdevs.

> +	int                    input_pad;
> +	struct clk             *dphy_clk;
> +	struct clk             *cfg_clk;
> +	struct clk             *pix_clk; /* what is this? */
> +	void __iomem           *base;
> +	int                     intr1;
> +	int                     intr2;

The interrupts are not used, I'd remove them and the dead code in
_probe.

> +	struct v4l2_of_bus_mipi_csi2 bus;
> +	bool                    on;
> +	bool                    stream_on;
> +};
> +
> +#define DEVICE_NAME "imx6-mipi-csi2"
> +
> +/* Register offsets */
> +#define CSI2_VERSION            0x000
> +#define CSI2_N_LANES            0x004
> +#define CSI2_PHY_SHUTDOWNZ      0x008
> +#define CSI2_DPHY_RSTZ          0x00c
> +#define CSI2_RESETN             0x010
> +#define CSI2_PHY_STATE          0x014
> +#define CSI2_DATA_IDS_1         0x018
> +#define CSI2_DATA_IDS_2         0x01c
> +#define CSI2_ERR1               0x020
> +#define CSI2_ERR2               0x024
> +#define CSI2_MSK1               0x028
> +#define CSI2_MSK2               0x02c
> +#define CSI2_PHY_TST_CTRL0      0x030
> +#define CSI2_PHY_TST_CTRL1      0x034
> +#define CSI2_SFT_RESET          0xf00
> +
> +static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
> +{
> +	return container_of(sdev, struct imxcsi2_dev, sd);
> +}
> +
> +static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
> +{
> +	return readl(csi2->base + regoff);
> +}
> +
> +static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
> +				 unsigned int regoff)
> +{
> +	writel(val, csi2->base + regoff);
> +}

Do those two wrappers really make the code more readable?

> +static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
> +{
> +	int lanes = csi2->bus.num_data_lanes;
> +
> +	imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
> +}
> +
> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
> +{
> +	if (enable) {
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);

Given that these registers only contain a single bit, and bits 31:1 are
documented as reserved, 0, I think these should write 1 instead of
0xffffffff.

> +	} else {
> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
> +	}
> +}
> +
> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
> +{
> +	imxcsi2_enable(csi2, false);
> +
> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);

These magic constants should be replaced with proper defines for the
documented bitfields, if available.

#define PHY_TESTCLR		BIT(0)
#define PHY_TESTCLK		BIT(1)

#define PHY_TESTEN		BIT(16)

	/* Clear PHY test interface */
	imxcsi2_write(csi2, PHY_TESTCLR, CSI2_PHY_TST_CTRL0);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

	/* Raise test interface strobe signal */
	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);

	/* Configure address write on falling edge and lower strobe signal */
	u8 addr = 0x44;
	imxcsi2_write(csi2, PHY_TESTEN | addr, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

	/* Configure data write on rising edge and raise strobe signal */
	u8 data = 0x14;
	imxcsi2_write(csi2, data, CSI2_PHY_TST_CTRL1);
	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);

	/* Clear strobe signal */
	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);

The whole sequence should probably be encapsulated in a
dw_mipi_dphy_write function.

Actually, this exact function already exists as dw_mipi_dsi_phy_write in
drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
register 0x44 might contain a field called HSFREQRANGE_SEL.

> +	imxcsi2_enable(csi2, true);
> +}
> +
> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
> +{
> +	u32 reg;
> +	int i;
> +
> +	/* wait for mipi sensor ready */

More specifically, wait for the clock lane module to leave ULP state.

> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg != 0x200)

Magic constants are bad. This is PHY_RXULPSCLKNOT (clock lane module in
ultra low power state).

> +			break;
> +		usleep_range(10000, 20000);
> +	}

How about breaking this out into a wait function, or even better, using
readl_poll_timeout instead of open coding these loops multiple times?

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* wait for mipi stable */

Wait for error free transmission?

> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_ERR1);
> +		if (reg == 0x0)
> +			break;
> +		usleep_range(10000, 20000);
> +	}

readl_poll_timeout

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for controller timeout, err1 = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	/* finally let's wait for active clock on the clock lane */
> +	for (i = 0; i < 50; i++) {
> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
> +		if (reg & (1 << 8))

Yes, and that is PHY_RXCLKACTIVEHS.

> +			break;
> +		usleep_range(10000, 20000);
> +	}

readl_poll_timeout

> +
> +	if (i >= 50) {
> +		v4l2_err(&csi2->sd,
> +			 "wait for active clock timeout, phy_state = 0x%08x\n",
> +			 reg);
> +		return -ETIME;
> +	}
> +
> +	v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
> +		  imxcsi2_read(csi2, CSI2_VERSION));

Noisy. (These are already removed for the next version.)

> +	return 0;
> +}
> +
> +/*
> + * V4L2 subdev operations
> + */
> +
> +static int imxcsi2_link_setup(struct media_entity *entity,
> +			      const struct media_pad *local,
> +			      const struct media_pad *remote, u32 flags)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +	struct v4l2_subdev *remote_sd;
> +
> +	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
> +		local->entity->name);
> +
> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +
> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (csi2->sink_sd[local->index])
> +				return -EBUSY;
> +			csi2->sink_sd[local->index] = remote_sd;
> +		} else {
> +			csi2->sink_sd[local->index] = NULL;
> +		}
> +	} else {
> +		if (flags & MEDIA_LNK_FL_ENABLED) {
> +			if (csi2->src_sd)
> +				return -EBUSY;
> +			csi2->src_sd = remote_sd;
> +		} else {
> +			csi2->src_sd = NULL;
> +		}
> +	}

This whole code block is just to check if there is another link already
active on the given pad. This could be stored in a boolean or a bit, no
need to store pointers to remote subdevices.

> +	return 0;
> +}
> +
> +static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	if (on && !csi2->on) {
> +		v4l2_info(&csi2->sd, "power ON\n");
> +		clk_prepare_enable(csi2->cfg_clk);
> +		clk_prepare_enable(csi2->dphy_clk);
> +		imxcsi2_set_lanes(csi2);
> +		imxcsi2_reset(csi2);
> +	} else if (!on && csi2->on) {
> +		v4l2_info(&csi2->sd, "power OFF\n");
> +		imxcsi2_enable(csi2, false);
> +		clk_disable_unprepare(csi2->dphy_clk);
> +		clk_disable_unprepare(csi2->cfg_clk);
> +	}
> +
> +	csi2->on = on;
> +	return 0;
> +}
> +
> +static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +	int i, ret = 0;
> +
> +	if (!csi2->src_sd)
> +		return -EPIPE;
> +	for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
> +		if (csi2->sink_sd[i])
> +			break;
> +	}
> +	if (i >= CSI2_NUM_SRC_PADS)
> +		return -EPIPE;
> +
> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +
> +	if (enable && !csi2->stream_on) {
> +		ret = clk_prepare_enable(csi2->pix_clk);
> +		if (ret)
> +			return ret;
> +
> +		ret = imxcsi2_dphy_wait(csi2);
> +		if (ret) {
> +			clk_disable_unprepare(csi2->pix_clk);
> +			return ret;
> +		}
> +	} else if (!enable && csi2->stream_on) {
> +		clk_disable_unprepare(csi2->pix_clk);
> +	}
> +
> +	csi2->stream_on = enable;
> +	return 0;
> +}
> +
> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_pad_config *cfg,
> +			   struct v4l2_subdev_format *sdformat)
> +{
> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> +
> +	sdformat->format = csi2->format_mbus;

The output formats are different from the input formats, see the media
bus format discussion in the other thread. The input pad is the MIPI
CSI-2 bus, but the four output pads are connected to the muxes / CSIs
via a 16-bit parallel bus.

So if the input format is UYVY8_1X16, for example, the output should be
set to UYVY8_2X8.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:22   ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 17:22 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devel, devicetree, Steve Longerbeam,
	linux-kernel, linux-arm-kernel, linux-media

I seem to be getting some sort of memory corruption with this driver.

I've had two instances now of uninitialised spinlocks in
imx_media_dma_buf_get_active() which show that the spinlock being
taken in this function is all-zeros.

That very quickly leads to an oops, where I've seen buf->ring is
NULL in imx_media_dma_buf_set_active().

Not quite sure what's going on, but the trigger (at least for me) is
to change my gstreamer pipeline from:

DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! bayer2rgbneon ! xvimagesink

to

DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! queue ! bayer2rgbneon ! xvimagesink

and it seems to take as little as two or three attempts to provoke the
kernel to totally die.

I've just tried a third time.  I can run the first gstreamer command
five times.  The I ran the second command and immediately got this:

INFO: trying to register non-static key.
the code is fine but needs lockdep annotation.
turning off the locking correctness validator.
CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0086af8>] (register_lock_class+0x1d4/0x554)
 r6:c1400408 r5:00000000 r4:00000000 r3:ee47a4c0
[<c0086924>] (register_lock_class) from [<c0089474>] (__lock_acquire+0x80/0x17b0)
 r10:d995f760 r9:c0a70384 r8:00000000 r7:c0a38680 r6:00000000 r5:ee47a4c0
 r4:c1400408
[<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
 r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:d995f760 r5:600f0193
 r4:00000000
[<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
 r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7a8c r5:600f0193
 r4:d995f750
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
 r6:e98b2c10 r5:d995f750 r4:d995f600
[<bf0d7a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf12c4b8>] (imx_smfc_eof_interrupt+0x60/0x124 [imx_smfc])
 r5:ee935dc4 r4:ee935c10
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
------------[ cut here ]------------
WARNING: CPU: 0 PID: 1008 at /home/rmk/git/linux-rmk/drivers/staging/media/imx/imx-smfc.c:159 imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc]
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
 r6:bf12d004 r5:00000000 r4:00000000 r3:ee47a4c0
[<c0033134>] (__warn) from [<c0033264>] (warn_slowpath_null+0x28/0x30)
 r10:ed501e64 r8:00000000 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
[<c003323c>] (warn_slowpath_null) from [<bf12c570>] (imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc])
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
---[ end trace 36356ae8b82a114e ]---
Unable to handle kernel NULL pointer dereference at virtual address 00000154
pgd = ed790000
[00000154] *pgd=00000000
Internal error: Oops: 5 [#1] SMP ARM
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 1008 Comm: Xorg Tainted: G        WC      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
task: ee47a4c0 task.stack: ed500000
PC is at do_raw_spin_lock+0x10/0x1d0
LR is at _raw_spin_lock_irqsave+0x54/0x60
pc : [<c008df34>]    lr : [<c07016fc>]    psr: 600f0193
sp : ed501d78  ip : ed501db0  fp : ed501dac
r10: ed501e64  r9 : c09e04ec  r8 : 00000000
r7 : 00000139  r6 : bf0d7bc8  r5 : 600f0193  r4 : 00000150
r3 : ee47a4c0  r2 : 00000000  r1 : ed501d58  r0 : 00000150
Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 3d79004a  DAC: 00000051
Process Xorg (pid: 1008, stack limit = 0xed500210)
Stack: (0xed501d78 to 0xed502000)
1d60:                                                       c011ad20 c09e04ec
1d80: c00867fc 00000150 600f0193 bf0d7bc8 00000139 00000000 c09e04ec ed501e64
1da0: ed501ddc ed501db0 c07016fc c008df30 00000001 00000000 bf0d7bc8 bf12c570
1dc0: ee935c10 d995f724 00000150 00000124 ed501dfc ed501de0 bf0d7bc8 c07016b4
1de0: ee935c10 ee935dc4 e98b2c10 00000139 ed501e1c ed501e00 bf12c4d0 bf0d7bb0
1e00: bf12c458 ebfb6d40 e98b2c00 e98b2c10 ed501e5c ed501e20 c009f5dc bf12c464
1e20: 00000001 c09e04ec 00000000 e98b2c00 c009f9f8 e98b2c00 e98b2c00 e98b2c10
1e40: 00000000 00000009 f4001100 ed501fb0 ed501e7c ed501e60 c009f984 c009f544
1e60: c0701d10 00000000 e98b2c00 e98b2c60 ed501e9c ed501e80 c009fa00 c009f96c
1e80: c09d0418 e98b2c00 e98b2c60 e98b2c10 ed501ebc ed501ea0 c00a3174 c009f9cc
1ea0: c00a30c4 00000000 ed501f08 ee4a3010 ed501ecc ed501ec0 c009ecf0 c00a30d0
1ec0: ed501efc ed501ed0 c0409328 c009ecdc c09d0448 00000001 0000003d ef1efc10
1ee0: c09e756c ee4a3010 00000026 ef008400 ed501f44 ed501f00 c0409458 c040928c
1f00: 00000001 00000000 00000001 00000002 00000003 0000000a 0000000b 0000000c
1f20: 0000000d 0000000e ed501f44 c09d52d0 00000000 00000000 ed501f54 ed501f48
1f40: c009ecf0 c0409408 ed501f7c ed501f58 c009ee24 c009ecdc ed501fb0 f4000100
1f60: f400010c c09e0af0 000003eb c0a38a78 ed501fac ed501f80 c00094c8 c009edd4
1f80: ee47a4c0 b6904220 600f0030 ffffffff 10c5387d 10c5387d 8114957c 7f79b000
1fa0: 00000000 ed501fb0 c0014dd8 c0009488 b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff 00000000 00000000
Backtrace:
[<c008df24>] (do_raw_spin_lock) from [<c07016fc>] (_raw_spin_lock_irqsave+0x54/0x60)
 r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7bc8 r5:600f0193
 r4:00000150
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7bc8>] (imx_media_dma_buf_set_active+0x24/0x68 [imx_media_common])
 r6:00000124 r5:00000150 r4:d995f724
[<bf0d7ba4>] (imx_media_dma_buf_set_active [imx_media_common]) from [<bf12c4d0>] (imx_smfc_eof_interrupt+0x78/0x124 [imx_smfc])
 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
Code: e1a0c00d e92ddff0 e24cb004 e24dd00c (e5902004)
---[ end trace 36356ae8b82a114f ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D WC      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: events dbs_work_handler
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60010193 r5:ffffffff r4:00000000 r3:ee6b8000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
 r6:c0a70028 r5:00000001 r4:00000004 r3:ee6b8000
[<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
 r10:ee6b5ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
 r4:f4000100
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6b5ba8 to 0xee6b5bf0)
5ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b5d2c
5bc0: c00177e8 00000000 00000001 ee6b5d2c 00000000 ee6b5c24 c09e0af4 ee6b5bf8
5be0: c0360f2c c00cd3a0 00010013 ffffffff
 r10:00000000 r9:ee6b4000 r8:00000001 r7:ee6b5bdc r6:ffffffff r5:00010013
 r4:c00cd3a0 r3:ee6b8000
[<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
 r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
[<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
 r10:00000002 r9:ffffffff r8:00000002 r7:ee6b5d2c r6:c00177e8 r5:00000000
 r4:ffffffff
[<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
[<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
 r7:ee6b5d24 r6:00000000 r5:00000000 r4:ffffffff
[<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
[<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
 r8:ee6b5d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
[<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
 r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
 r4:ef006900
[<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
[<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
 r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
 r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
[<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
 r10:ee592100 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
 r4:c141d574
[<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
[<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
 r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
 r4:ee843800
[<c0525210>] (__cpufreq_driver_target) from [<c0528e84>] (od_dbs_update+0xe4/0x168)
 r10:e9b77fc0 r9:c09e04ec r8:ee862480 r7:ee843800 r6:ee862000 r5:ee862480
 r4:ee862000
[<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee843800 r6:ee862004 r5:00000000 r4:ee862068
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b5f08 r6:ef7c9940 r5:ee862068 r4:ef101e00 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ef101e18 r5:ef7c9974
 r4:ef101e00
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ef101e00 r7:ef101f38 r6:ef0ffa40 r5:00000000
 r4:ef101f00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ef0ffa40 r3:ee6b4000
---[ end Kernel panic - not syncing: Fatal exception in interrupt

I thought, maybe, it's the IPU overwriting past the end of the buffer,
but I've added checks and that doesn't seem to have fired.  I also
wondered if it was some kind of use-after-free of the ring, so I made
imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
kfree()ing it... doesn't look like it's that either.  I'm going to
continue poking to see if I can figure out what's going on.

The oops at 0x00000154 is due to "ring" in imx_media_dma_buf_set_active()
being NULL.  "buf" in that instance (contained in r4) is 0xd995f724.

I'm just seeing if I can track that down by adding

	WARN_ON(buf->ring != priv->out_ring);

in imx-smfc.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:22   ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 17:22 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b,
	devicetree-u79uwXL29TY76Z2rM5mHXA, Steve Longerbeam,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, linux-media

I seem to be getting some sort of memory corruption with this driver.

I've had two instances now of uninitialised spinlocks in
imx_media_dma_buf_get_active() which show that the spinlock being
taken in this function is all-zeros.

That very quickly leads to an oops, where I've seen buf->ring is
NULL in imx_media_dma_buf_set_active().

Not quite sure what's going on, but the trigger (at least for me) is
to change my gstreamer pipeline from:

DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! bayer2rgbneon ! xvimagesink

to

DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! queue ! bayer2rgbneon ! xvimagesink

and it seems to take as little as two or three attempts to provoke the
kernel to totally die.

I've just tried a third time.  I can run the first gstreamer command
five times.  The I ran the second command and immediately got this:

INFO: trying to register non-static key.
the code is fine but needs lockdep annotation.
turning off the locking correctness validator.
CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0086af8>] (register_lock_class+0x1d4/0x554)
 r6:c1400408 r5:00000000 r4:00000000 r3:ee47a4c0
[<c0086924>] (register_lock_class) from [<c0089474>] (__lock_acquire+0x80/0x17b0)
 r10:d995f760 r9:c0a70384 r8:00000000 r7:c0a38680 r6:00000000 r5:ee47a4c0
 r4:c1400408
[<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
 r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:d995f760 r5:600f0193
 r4:00000000
[<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
 r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7a8c r5:600f0193
 r4:d995f750
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
 r6:e98b2c10 r5:d995f750 r4:d995f600
[<bf0d7a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf12c4b8>] (imx_smfc_eof_interrupt+0x60/0x124 [imx_smfc])
 r5:ee935dc4 r4:ee935c10
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
------------[ cut here ]------------
WARNING: CPU: 0 PID: 1008 at /home/rmk/git/linux-rmk/drivers/staging/media/imx/imx-smfc.c:159 imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc]
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
 r6:bf12d004 r5:00000000 r4:00000000 r3:ee47a4c0
[<c0033134>] (__warn) from [<c0033264>] (warn_slowpath_null+0x28/0x30)
 r10:ed501e64 r8:00000000 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
[<c003323c>] (warn_slowpath_null) from [<bf12c570>] (imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc])
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
---[ end trace 36356ae8b82a114e ]---
Unable to handle kernel NULL pointer dereference at virtual address 00000154
pgd = ed790000
[00000154] *pgd=00000000
Internal error: Oops: 5 [#1] SMP ARM
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 1008 Comm: Xorg Tainted: G        WC      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
task: ee47a4c0 task.stack: ed500000
PC is at do_raw_spin_lock+0x10/0x1d0
LR is at _raw_spin_lock_irqsave+0x54/0x60
pc : [<c008df34>]    lr : [<c07016fc>]    psr: 600f0193
sp : ed501d78  ip : ed501db0  fp : ed501dac
r10: ed501e64  r9 : c09e04ec  r8 : 00000000
r7 : 00000139  r6 : bf0d7bc8  r5 : 600f0193  r4 : 00000150
r3 : ee47a4c0  r2 : 00000000  r1 : ed501d58  r0 : 00000150
Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 3d79004a  DAC: 00000051
Process Xorg (pid: 1008, stack limit = 0xed500210)
Stack: (0xed501d78 to 0xed502000)
1d60:                                                       c011ad20 c09e04ec
1d80: c00867fc 00000150 600f0193 bf0d7bc8 00000139 00000000 c09e04ec ed501e64
1da0: ed501ddc ed501db0 c07016fc c008df30 00000001 00000000 bf0d7bc8 bf12c570
1dc0: ee935c10 d995f724 00000150 00000124 ed501dfc ed501de0 bf0d7bc8 c07016b4
1de0: ee935c10 ee935dc4 e98b2c10 00000139 ed501e1c ed501e00 bf12c4d0 bf0d7bb0
1e00: bf12c458 ebfb6d40 e98b2c00 e98b2c10 ed501e5c ed501e20 c009f5dc bf12c464
1e20: 00000001 c09e04ec 00000000 e98b2c00 c009f9f8 e98b2c00 e98b2c00 e98b2c10
1e40: 00000000 00000009 f4001100 ed501fb0 ed501e7c ed501e60 c009f984 c009f544
1e60: c0701d10 00000000 e98b2c00 e98b2c60 ed501e9c ed501e80 c009fa00 c009f96c
1e80: c09d0418 e98b2c00 e98b2c60 e98b2c10 ed501ebc ed501ea0 c00a3174 c009f9cc
1ea0: c00a30c4 00000000 ed501f08 ee4a3010 ed501ecc ed501ec0 c009ecf0 c00a30d0
1ec0: ed501efc ed501ed0 c0409328 c009ecdc c09d0448 00000001 0000003d ef1efc10
1ee0: c09e756c ee4a3010 00000026 ef008400 ed501f44 ed501f00 c0409458 c040928c
1f00: 00000001 00000000 00000001 00000002 00000003 0000000a 0000000b 0000000c
1f20: 0000000d 0000000e ed501f44 c09d52d0 00000000 00000000 ed501f54 ed501f48
1f40: c009ecf0 c0409408 ed501f7c ed501f58 c009ee24 c009ecdc ed501fb0 f4000100
1f60: f400010c c09e0af0 000003eb c0a38a78 ed501fac ed501f80 c00094c8 c009edd4
1f80: ee47a4c0 b6904220 600f0030 ffffffff 10c5387d 10c5387d 8114957c 7f79b000
1fa0: 00000000 ed501fb0 c0014dd8 c0009488 b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff 00000000 00000000
Backtrace:
[<c008df24>] (do_raw_spin_lock) from [<c07016fc>] (_raw_spin_lock_irqsave+0x54/0x60)
 r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7bc8 r5:600f0193
 r4:00000150
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7bc8>] (imx_media_dma_buf_set_active+0x24/0x68 [imx_media_common])
 r6:00000124 r5:00000150 r4:d995f724
[<bf0d7ba4>] (imx_media_dma_buf_set_active [imx_media_common]) from [<bf12c4d0>] (imx_smfc_eof_interrupt+0x78/0x124 [imx_smfc])
 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
Code: e1a0c00d e92ddff0 e24cb004 e24dd00c (e5902004)
---[ end trace 36356ae8b82a114f ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D WC      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: events dbs_work_handler
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60010193 r5:ffffffff r4:00000000 r3:ee6b8000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
 r6:c0a70028 r5:00000001 r4:00000004 r3:ee6b8000
[<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
 r10:ee6b5ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
 r4:f4000100
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6b5ba8 to 0xee6b5bf0)
5ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b5d2c
5bc0: c00177e8 00000000 00000001 ee6b5d2c 00000000 ee6b5c24 c09e0af4 ee6b5bf8
5be0: c0360f2c c00cd3a0 00010013 ffffffff
 r10:00000000 r9:ee6b4000 r8:00000001 r7:ee6b5bdc r6:ffffffff r5:00010013
 r4:c00cd3a0 r3:ee6b8000
[<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
 r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
[<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
 r10:00000002 r9:ffffffff r8:00000002 r7:ee6b5d2c r6:c00177e8 r5:00000000
 r4:ffffffff
[<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
[<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
 r7:ee6b5d24 r6:00000000 r5:00000000 r4:ffffffff
[<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
[<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
 r8:ee6b5d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
[<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
 r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
 r4:ef006900
[<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
[<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
 r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
 r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
[<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
 r10:ee592100 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
 r4:c141d574
[<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
[<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
 r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
 r4:ee843800
[<c0525210>] (__cpufreq_driver_target) from [<c0528e84>] (od_dbs_update+0xe4/0x168)
 r10:e9b77fc0 r9:c09e04ec r8:ee862480 r7:ee843800 r6:ee862000 r5:ee862480
 r4:ee862000
[<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee843800 r6:ee862004 r5:00000000 r4:ee862068
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b5f08 r6:ef7c9940 r5:ee862068 r4:ef101e00 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ef101e18 r5:ef7c9974
 r4:ef101e00
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ef101e00 r7:ef101f38 r6:ef0ffa40 r5:00000000
 r4:ef101f00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ef0ffa40 r3:ee6b4000
---[ end Kernel panic - not syncing: Fatal exception in interrupt

I thought, maybe, it's the IPU overwriting past the end of the buffer,
but I've added checks and that doesn't seem to have fired.  I also
wondered if it was some kind of use-after-free of the ring, so I made
imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
kfree()ing it... doesn't look like it's that either.  I'm going to
continue poking to see if I can figure out what's going on.

The oops at 0x00000154 is due to "ring" in imx_media_dma_buf_set_active()
being NULL.  "buf" in that instance (contained in r4) is 0xd995f724.

I'm just seeing if I can track that down by adding

	WARN_ON(buf->ring != priv->out_ring);

in imx-smfc.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:22   ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 17:22 UTC (permalink / raw)
  To: linux-arm-kernel

I seem to be getting some sort of memory corruption with this driver.

I've had two instances now of uninitialised spinlocks in
imx_media_dma_buf_get_active() which show that the spinlock being
taken in this function is all-zeros.

That very quickly leads to an oops, where I've seen buf->ring is
NULL in imx_media_dma_buf_set_active().

Not quite sure what's going on, but the trigger (at least for me) is
to change my gstreamer pipeline from:

DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! bayer2rgbneon ! xvimagesink

to

DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! queue ! bayer2rgbneon ! xvimagesink

and it seems to take as little as two or three attempts to provoke the
kernel to totally die.

I've just tried a third time.  I can run the first gstreamer command
five times.  The I ran the second command and immediately got this:

INFO: trying to register non-static key.
the code is fine but needs lockdep annotation.
turning off the locking correctness validator.
CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0086af8>] (register_lock_class+0x1d4/0x554)
 r6:c1400408 r5:00000000 r4:00000000 r3:ee47a4c0
[<c0086924>] (register_lock_class) from [<c0089474>] (__lock_acquire+0x80/0x17b0)
 r10:d995f760 r9:c0a70384 r8:00000000 r7:c0a38680 r6:00000000 r5:ee47a4c0
 r4:c1400408
[<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
 r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:d995f760 r5:600f0193
 r4:00000000
[<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
 r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7a8c r5:600f0193
 r4:d995f750
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
 r6:e98b2c10 r5:d995f750 r4:d995f600
[<bf0d7a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf12c4b8>] (imx_smfc_eof_interrupt+0x60/0x124 [imx_smfc])
 r5:ee935dc4 r4:ee935c10
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
------------[ cut here ]------------
WARNING: CPU: 0 PID: 1008 at /home/rmk/git/linux-rmk/drivers/staging/media/imx/imx-smfc.c:159 imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc]
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
 r6:bf12d004 r5:00000000 r4:00000000 r3:ee47a4c0
[<c0033134>] (__warn) from [<c0033264>] (warn_slowpath_null+0x28/0x30)
 r10:ed501e64 r8:00000000 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
[<c003323c>] (warn_slowpath_null) from [<bf12c570>] (imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc])
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
---[ end trace 36356ae8b82a114e ]---
Unable to handle kernel NULL pointer dereference at virtual address 00000154
pgd = ed790000
[00000154] *pgd=00000000
Internal error: Oops: 5 [#1] SMP ARM
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 1008 Comm: Xorg Tainted: G        WC      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
task: ee47a4c0 task.stack: ed500000
PC is at do_raw_spin_lock+0x10/0x1d0
LR is at _raw_spin_lock_irqsave+0x54/0x60
pc : [<c008df34>]    lr : [<c07016fc>]    psr: 600f0193
sp : ed501d78  ip : ed501db0  fp : ed501dac
r10: ed501e64  r9 : c09e04ec  r8 : 00000000
r7 : 00000139  r6 : bf0d7bc8  r5 : 600f0193  r4 : 00000150
r3 : ee47a4c0  r2 : 00000000  r1 : ed501d58  r0 : 00000150
Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 3d79004a  DAC: 00000051
Process Xorg (pid: 1008, stack limit = 0xed500210)
Stack: (0xed501d78 to 0xed502000)
1d60:                                                       c011ad20 c09e04ec
1d80: c00867fc 00000150 600f0193 bf0d7bc8 00000139 00000000 c09e04ec ed501e64
1da0: ed501ddc ed501db0 c07016fc c008df30 00000001 00000000 bf0d7bc8 bf12c570
1dc0: ee935c10 d995f724 00000150 00000124 ed501dfc ed501de0 bf0d7bc8 c07016b4
1de0: ee935c10 ee935dc4 e98b2c10 00000139 ed501e1c ed501e00 bf12c4d0 bf0d7bb0
1e00: bf12c458 ebfb6d40 e98b2c00 e98b2c10 ed501e5c ed501e20 c009f5dc bf12c464
1e20: 00000001 c09e04ec 00000000 e98b2c00 c009f9f8 e98b2c00 e98b2c00 e98b2c10
1e40: 00000000 00000009 f4001100 ed501fb0 ed501e7c ed501e60 c009f984 c009f544
1e60: c0701d10 00000000 e98b2c00 e98b2c60 ed501e9c ed501e80 c009fa00 c009f96c
1e80: c09d0418 e98b2c00 e98b2c60 e98b2c10 ed501ebc ed501ea0 c00a3174 c009f9cc
1ea0: c00a30c4 00000000 ed501f08 ee4a3010 ed501ecc ed501ec0 c009ecf0 c00a30d0
1ec0: ed501efc ed501ed0 c0409328 c009ecdc c09d0448 00000001 0000003d ef1efc10
1ee0: c09e756c ee4a3010 00000026 ef008400 ed501f44 ed501f00 c0409458 c040928c
1f00: 00000001 00000000 00000001 00000002 00000003 0000000a 0000000b 0000000c
1f20: 0000000d 0000000e ed501f44 c09d52d0 00000000 00000000 ed501f54 ed501f48
1f40: c009ecf0 c0409408 ed501f7c ed501f58 c009ee24 c009ecdc ed501fb0 f4000100
1f60: f400010c c09e0af0 000003eb c0a38a78 ed501fac ed501f80 c00094c8 c009edd4
1f80: ee47a4c0 b6904220 600f0030 ffffffff 10c5387d 10c5387d 8114957c 7f79b000
1fa0: 00000000 ed501fb0 c0014dd8 c0009488 b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff 00000000 00000000
Backtrace:
[<c008df24>] (do_raw_spin_lock) from [<c07016fc>] (_raw_spin_lock_irqsave+0x54/0x60)
 r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7bc8 r5:600f0193
 r4:00000150
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7bc8>] (imx_media_dma_buf_set_active+0x24/0x68 [imx_media_common])
 r6:00000124 r5:00000150 r4:d995f724
[<bf0d7ba4>] (imx_media_dma_buf_set_active [imx_media_common]) from [<bf12c4d0>] (imx_smfc_eof_interrupt+0x78/0x124 [imx_smfc])
 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
[<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
 r4:e98b2c00
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e98b2c60 r4:e98b2c00
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
[<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
Exception stack(0xed501fb0 to 0xed501ff8)
1fa0:                                     b698b4e0 00000000 0042c000 b698c708
1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
 r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
 r4:b6904220 r3:ee47a4c0
Code: e1a0c00d e92ddff0 e24cb004 e24dd00c (e5902004)
---[ end trace 36356ae8b82a114f ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D WC      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: events dbs_work_handler
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60010193 r5:ffffffff r4:00000000 r3:ee6b8000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
 r6:c0a70028 r5:00000001 r4:00000004 r3:ee6b8000
[<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
 r10:ee6b5ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
 r4:f4000100
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6b5ba8 to 0xee6b5bf0)
5ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b5d2c
5bc0: c00177e8 00000000 00000001 ee6b5d2c 00000000 ee6b5c24 c09e0af4 ee6b5bf8
5be0: c0360f2c c00cd3a0 00010013 ffffffff
 r10:00000000 r9:ee6b4000 r8:00000001 r7:ee6b5bdc r6:ffffffff r5:00010013
 r4:c00cd3a0 r3:ee6b8000
[<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
 r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
[<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
 r10:00000002 r9:ffffffff r8:00000002 r7:ee6b5d2c r6:c00177e8 r5:00000000
 r4:ffffffff
[<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
[<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
 r7:ee6b5d24 r6:00000000 r5:00000000 r4:ffffffff
[<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
[<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
 r8:ee6b5d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
[<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
 r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
 r4:ef006900
[<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
[<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
 r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
 r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
[<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
 r10:ee592100 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
 r4:c141d574
[<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
[<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
 r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
 r4:ee843800
[<c0525210>] (__cpufreq_driver_target) from [<c0528e84>] (od_dbs_update+0xe4/0x168)
 r10:e9b77fc0 r9:c09e04ec r8:ee862480 r7:ee843800 r6:ee862000 r5:ee862480
 r4:ee862000
[<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee843800 r6:ee862004 r5:00000000 r4:ee862068
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b5f08 r6:ef7c9940 r5:ee862068 r4:ef101e00 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ef101e18 r5:ef7c9974
 r4:ef101e00
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ef101e00 r7:ef101f38 r6:ef0ffa40 r5:00000000
 r4:ef101f00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ef0ffa40 r3:ee6b4000
---[ end Kernel panic - not syncing: Fatal exception in interrupt

I thought, maybe, it's the IPU overwriting past the end of the buffer,
but I've added checks and that doesn't seem to have fired.  I also
wondered if it was some kind of use-after-free of the ring, so I made
imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
kfree()ing it... doesn't look like it's that either.  I'm going to
continue poking to see if I can figure out what's going on.

The oops at 0x00000154 is due to "ring" in imx_media_dma_buf_set_active()
being NULL.  "buf" in that instance (contained in r4) is 0xd995f724.

I'm just seeing if I can track that down by adding

	WARN_ON(buf->ring != priv->out_ring);

in imx-smfc.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-02 17:22   ` Russell King - ARM Linux
  (?)
@ 2017-02-02 17:31     ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 17:31 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devel, devicetree, Steve Longerbeam,
	linux-kernel, linux-arm-kernel, linux-media

Hi Russell,

I don't recommend spending too much time debugging this
OOPS. The dma buffer ring has been removed completely
in version 4 (which I'm trying to get ready to post hopefully
by end of this week).

Steve


On 02/02/2017 09:22 AM, Russell King - ARM Linux wrote:
> I seem to be getting some sort of memory corruption with this driver.
>
> I've had two instances now of uninitialised spinlocks in
> imx_media_dma_buf_get_active() which show that the spinlock being
> taken in this function is all-zeros.
>
> That very quickly leads to an oops, where I've seen buf->ring is
> NULL in imx_media_dma_buf_set_active().
>
> Not quite sure what's going on, but the trigger (at least for me) is
> to change my gstreamer pipeline from:
>
> DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! bayer2rgbneon ! xvimagesink
>
> to
>
> DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! queue ! bayer2rgbneon ! xvimagesink
>
> and it seems to take as little as two or three attempts to provoke the
> kernel to totally die.
>
> I've just tried a third time.  I can run the first gstreamer command
> five times.  The I ran the second command and immediately got this:
>
> INFO: trying to register non-static key.
> the code is fine but needs lockdep annotation.
> turning off the locking correctness validator.
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0086af8>] (register_lock_class+0x1d4/0x554)
>   r6:c1400408 r5:00000000 r4:00000000 r3:ee47a4c0
> [<c0086924>] (register_lock_class) from [<c0089474>] (__lock_acquire+0x80/0x17b0)
>   r10:d995f760 r9:c0a70384 r8:00000000 r7:c0a38680 r6:00000000 r5:ee47a4c0
>   r4:c1400408
> [<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
>   r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:d995f760 r5:600f0193
>   r4:00000000
> [<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
>   r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7a8c r5:600f0193
>   r4:d995f750
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
>   r6:e98b2c10 r5:d995f750 r4:d995f600
> [<bf0d7a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf12c4b8>] (imx_smfc_eof_interrupt+0x60/0x124 [imx_smfc])
>   r5:ee935dc4 r4:ee935c10
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 1008 at /home/rmk/git/linux-rmk/drivers/staging/media/imx/imx-smfc.c:159 imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc]
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
>   r6:bf12d004 r5:00000000 r4:00000000 r3:ee47a4c0
> [<c0033134>] (__warn) from [<c0033264>] (warn_slowpath_null+0x28/0x30)
>   r10:ed501e64 r8:00000000 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
> [<c003323c>] (warn_slowpath_null) from [<bf12c570>] (imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc])
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> ---[ end trace 36356ae8b82a114e ]---
> Unable to handle kernel NULL pointer dereference at virtual address 00000154
> pgd = ed790000
> [00000154] *pgd=00000000
> Internal error: Oops: 5 [#1] SMP ARM
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G        WC      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> task: ee47a4c0 task.stack: ed500000
> PC is at do_raw_spin_lock+0x10/0x1d0
> LR is at _raw_spin_lock_irqsave+0x54/0x60
> pc : [<c008df34>]    lr : [<c07016fc>]    psr: 600f0193
> sp : ed501d78  ip : ed501db0  fp : ed501dac
> r10: ed501e64  r9 : c09e04ec  r8 : 00000000
> r7 : 00000139  r6 : bf0d7bc8  r5 : 600f0193  r4 : 00000150
> r3 : ee47a4c0  r2 : 00000000  r1 : ed501d58  r0 : 00000150
> Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> Control: 10c5387d  Table: 3d79004a  DAC: 00000051
> Process Xorg (pid: 1008, stack limit = 0xed500210)
> Stack: (0xed501d78 to 0xed502000)
> 1d60:                                                       c011ad20 c09e04ec
> 1d80: c00867fc 00000150 600f0193 bf0d7bc8 00000139 00000000 c09e04ec ed501e64
> 1da0: ed501ddc ed501db0 c07016fc c008df30 00000001 00000000 bf0d7bc8 bf12c570
> 1dc0: ee935c10 d995f724 00000150 00000124 ed501dfc ed501de0 bf0d7bc8 c07016b4
> 1de0: ee935c10 ee935dc4 e98b2c10 00000139 ed501e1c ed501e00 bf12c4d0 bf0d7bb0
> 1e00: bf12c458 ebfb6d40 e98b2c00 e98b2c10 ed501e5c ed501e20 c009f5dc bf12c464
> 1e20: 00000001 c09e04ec 00000000 e98b2c00 c009f9f8 e98b2c00 e98b2c00 e98b2c10
> 1e40: 00000000 00000009 f4001100 ed501fb0 ed501e7c ed501e60 c009f984 c009f544
> 1e60: c0701d10 00000000 e98b2c00 e98b2c60 ed501e9c ed501e80 c009fa00 c009f96c
> 1e80: c09d0418 e98b2c00 e98b2c60 e98b2c10 ed501ebc ed501ea0 c00a3174 c009f9cc
> 1ea0: c00a30c4 00000000 ed501f08 ee4a3010 ed501ecc ed501ec0 c009ecf0 c00a30d0
> 1ec0: ed501efc ed501ed0 c0409328 c009ecdc c09d0448 00000001 0000003d ef1efc10
> 1ee0: c09e756c ee4a3010 00000026 ef008400 ed501f44 ed501f00 c0409458 c040928c
> 1f00: 00000001 00000000 00000001 00000002 00000003 0000000a 0000000b 0000000c
> 1f20: 0000000d 0000000e ed501f44 c09d52d0 00000000 00000000 ed501f54 ed501f48
> 1f40: c009ecf0 c0409408 ed501f7c ed501f58 c009ee24 c009ecdc ed501fb0 f4000100
> 1f60: f400010c c09e0af0 000003eb c0a38a78 ed501fac ed501f80 c00094c8 c009edd4
> 1f80: ee47a4c0 b6904220 600f0030 ffffffff 10c5387d 10c5387d 8114957c 7f79b000
> 1fa0: 00000000 ed501fb0 c0014dd8 c0009488 b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff 00000000 00000000
> Backtrace:
> [<c008df24>] (do_raw_spin_lock) from [<c07016fc>] (_raw_spin_lock_irqsave+0x54/0x60)
>   r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7bc8 r5:600f0193
>   r4:00000150
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7bc8>] (imx_media_dma_buf_set_active+0x24/0x68 [imx_media_common])
>   r6:00000124 r5:00000150 r4:d995f724
> [<bf0d7ba4>] (imx_media_dma_buf_set_active [imx_media_common]) from [<bf12c4d0>] (imx_smfc_eof_interrupt+0x78/0x124 [imx_smfc])
>   r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> Code: e1a0c00d e92ddff0 e24cb004 e24dd00c (e5902004)
> ---[ end trace 36356ae8b82a114f ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> CPU1: stopping
> CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D WC      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: events dbs_work_handler
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:60010193 r5:ffffffff r4:00000000 r3:ee6b8000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
>   r6:c0a70028 r5:00000001 r4:00000004 r3:ee6b8000
> [<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
>   r10:ee6b5ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
>   r4:f4000100
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6b5ba8 to 0xee6b5bf0)
> 5ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b5d2c
> 5bc0: c00177e8 00000000 00000001 ee6b5d2c 00000000 ee6b5c24 c09e0af4 ee6b5bf8
> 5be0: c0360f2c c00cd3a0 00010013 ffffffff
>   r10:00000000 r9:ee6b4000 r8:00000001 r7:ee6b5bdc r6:ffffffff r5:00010013
>   r4:c00cd3a0 r3:ee6b8000
> [<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
>   r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
> [<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
>   r10:00000002 r9:ffffffff r8:00000002 r7:ee6b5d2c r6:c00177e8 r5:00000000
>   r4:ffffffff
> [<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
> [<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
>   r7:ee6b5d24 r6:00000000 r5:00000000 r4:ffffffff
> [<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
> [<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
>   r8:ee6b5d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
> [<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
>   r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
>   r4:ef006900
> [<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
> [<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
>   r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
>   r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
> [<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
>   r10:ee592100 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
>   r4:c141d574
> [<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
> [<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
>   r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
>   r4:ee843800
> [<c0525210>] (__cpufreq_driver_target) from [<c0528e84>] (od_dbs_update+0xe4/0x168)
>   r10:e9b77fc0 r9:c09e04ec r8:ee862480 r7:ee843800 r6:ee862000 r5:ee862480
>   r4:ee862000
> [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee843800 r6:ee862004 r5:00000000 r4:ee862068
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b5f08 r6:ef7c9940 r5:ee862068 r4:ef101e00 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ef101e18 r5:ef7c9974
>   r4:ef101e00
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ef101e00 r7:ef101f38 r6:ef0ffa40 r5:00000000
>   r4:ef101f00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ef0ffa40 r3:ee6b4000
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>
> I thought, maybe, it's the IPU overwriting past the end of the buffer,
> but I've added checks and that doesn't seem to have fired.  I also
> wondered if it was some kind of use-after-free of the ring, so I made
> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
> kfree()ing it... doesn't look like it's that either.  I'm going to
> continue poking to see if I can figure out what's going on.
>
> The oops at 0x00000154 is due to "ring" in imx_media_dma_buf_set_active()
> being NULL.  "buf" in that instance (contained in r4) is 0xd995f724.
>
> I'm just seeing if I can track that down by adding
>
> 	WARN_ON(buf->ring != priv->out_ring);
>
> in imx-smfc.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:31     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 17:31 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

Hi Russell,

I don't recommend spending too much time debugging this
OOPS. The dma buffer ring has been removed completely
in version 4 (which I'm trying to get ready to post hopefully
by end of this week).

Steve


On 02/02/2017 09:22 AM, Russell King - ARM Linux wrote:
> I seem to be getting some sort of memory corruption with this driver.
>
> I've had two instances now of uninitialised spinlocks in
> imx_media_dma_buf_get_active() which show that the spinlock being
> taken in this function is all-zeros.
>
> That very quickly leads to an oops, where I've seen buf->ring is
> NULL in imx_media_dma_buf_set_active().
>
> Not quite sure what's going on, but the trigger (at least for me) is
> to change my gstreamer pipeline from:
>
> DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! bayer2rgbneon ! xvimagesink
>
> to
>
> DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! queue ! bayer2rgbneon ! xvimagesink
>
> and it seems to take as little as two or three attempts to provoke the
> kernel to totally die.
>
> I've just tried a third time.  I can run the first gstreamer command
> five times.  The I ran the second command and immediately got this:
>
> INFO: trying to register non-static key.
> the code is fine but needs lockdep annotation.
> turning off the locking correctness validator.
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0086af8>] (register_lock_class+0x1d4/0x554)
>   r6:c1400408 r5:00000000 r4:00000000 r3:ee47a4c0
> [<c0086924>] (register_lock_class) from [<c0089474>] (__lock_acquire+0x80/0x17b0)
>   r10:d995f760 r9:c0a70384 r8:00000000 r7:c0a38680 r6:00000000 r5:ee47a4c0
>   r4:c1400408
> [<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
>   r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:d995f760 r5:600f0193
>   r4:00000000
> [<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
>   r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7a8c r5:600f0193
>   r4:d995f750
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
>   r6:e98b2c10 r5:d995f750 r4:d995f600
> [<bf0d7a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf12c4b8>] (imx_smfc_eof_interrupt+0x60/0x124 [imx_smfc])
>   r5:ee935dc4 r4:ee935c10
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 1008 at /home/rmk/git/linux-rmk/drivers/staging/media/imx/imx-smfc.c:159 imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc]
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
>   r6:bf12d004 r5:00000000 r4:00000000 r3:ee47a4c0
> [<c0033134>] (__warn) from [<c0033264>] (warn_slowpath_null+0x28/0x30)
>   r10:ed501e64 r8:00000000 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
> [<c003323c>] (warn_slowpath_null) from [<bf12c570>] (imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc])
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> ---[ end trace 36356ae8b82a114e ]---
> Unable to handle kernel NULL pointer dereference at virtual address 00000154
> pgd = ed790000
> [00000154] *pgd=00000000
> Internal error: Oops: 5 [#1] SMP ARM
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G        WC      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> task: ee47a4c0 task.stack: ed500000
> PC is at do_raw_spin_lock+0x10/0x1d0
> LR is at _raw_spin_lock_irqsave+0x54/0x60
> pc : [<c008df34>]    lr : [<c07016fc>]    psr: 600f0193
> sp : ed501d78  ip : ed501db0  fp : ed501dac
> r10: ed501e64  r9 : c09e04ec  r8 : 00000000
> r7 : 00000139  r6 : bf0d7bc8  r5 : 600f0193  r4 : 00000150
> r3 : ee47a4c0  r2 : 00000000  r1 : ed501d58  r0 : 00000150
> Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> Control: 10c5387d  Table: 3d79004a  DAC: 00000051
> Process Xorg (pid: 1008, stack limit = 0xed500210)
> Stack: (0xed501d78 to 0xed502000)
> 1d60:                                                       c011ad20 c09e04ec
> 1d80: c00867fc 00000150 600f0193 bf0d7bc8 00000139 00000000 c09e04ec ed501e64
> 1da0: ed501ddc ed501db0 c07016fc c008df30 00000001 00000000 bf0d7bc8 bf12c570
> 1dc0: ee935c10 d995f724 00000150 00000124 ed501dfc ed501de0 bf0d7bc8 c07016b4
> 1de0: ee935c10 ee935dc4 e98b2c10 00000139 ed501e1c ed501e00 bf12c4d0 bf0d7bb0
> 1e00: bf12c458 ebfb6d40 e98b2c00 e98b2c10 ed501e5c ed501e20 c009f5dc bf12c464
> 1e20: 00000001 c09e04ec 00000000 e98b2c00 c009f9f8 e98b2c00 e98b2c00 e98b2c10
> 1e40: 00000000 00000009 f4001100 ed501fb0 ed501e7c ed501e60 c009f984 c009f544
> 1e60: c0701d10 00000000 e98b2c00 e98b2c60 ed501e9c ed501e80 c009fa00 c009f96c
> 1e80: c09d0418 e98b2c00 e98b2c60 e98b2c10 ed501ebc ed501ea0 c00a3174 c009f9cc
> 1ea0: c00a30c4 00000000 ed501f08 ee4a3010 ed501ecc ed501ec0 c009ecf0 c00a30d0
> 1ec0: ed501efc ed501ed0 c0409328 c009ecdc c09d0448 00000001 0000003d ef1efc10
> 1ee0: c09e756c ee4a3010 00000026 ef008400 ed501f44 ed501f00 c0409458 c040928c
> 1f00: 00000001 00000000 00000001 00000002 00000003 0000000a 0000000b 0000000c
> 1f20: 0000000d 0000000e ed501f44 c09d52d0 00000000 00000000 ed501f54 ed501f48
> 1f40: c009ecf0 c0409408 ed501f7c ed501f58 c009ee24 c009ecdc ed501fb0 f4000100
> 1f60: f400010c c09e0af0 000003eb c0a38a78 ed501fac ed501f80 c00094c8 c009edd4
> 1f80: ee47a4c0 b6904220 600f0030 ffffffff 10c5387d 10c5387d 8114957c 7f79b000
> 1fa0: 00000000 ed501fb0 c0014dd8 c0009488 b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff 00000000 00000000
> Backtrace:
> [<c008df24>] (do_raw_spin_lock) from [<c07016fc>] (_raw_spin_lock_irqsave+0x54/0x60)
>   r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7bc8 r5:600f0193
>   r4:00000150
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7bc8>] (imx_media_dma_buf_set_active+0x24/0x68 [imx_media_common])
>   r6:00000124 r5:00000150 r4:d995f724
> [<bf0d7ba4>] (imx_media_dma_buf_set_active [imx_media_common]) from [<bf12c4d0>] (imx_smfc_eof_interrupt+0x78/0x124 [imx_smfc])
>   r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> Code: e1a0c00d e92ddff0 e24cb004 e24dd00c (e5902004)
> ---[ end trace 36356ae8b82a114f ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> CPU1: stopping
> CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D WC      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: events dbs_work_handler
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:60010193 r5:ffffffff r4:00000000 r3:ee6b8000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
>   r6:c0a70028 r5:00000001 r4:00000004 r3:ee6b8000
> [<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
>   r10:ee6b5ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
>   r4:f4000100
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6b5ba8 to 0xee6b5bf0)
> 5ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b5d2c
> 5bc0: c00177e8 00000000 00000001 ee6b5d2c 00000000 ee6b5c24 c09e0af4 ee6b5bf8
> 5be0: c0360f2c c00cd3a0 00010013 ffffffff
>   r10:00000000 r9:ee6b4000 r8:00000001 r7:ee6b5bdc r6:ffffffff r5:00010013
>   r4:c00cd3a0 r3:ee6b8000
> [<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
>   r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
> [<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
>   r10:00000002 r9:ffffffff r8:00000002 r7:ee6b5d2c r6:c00177e8 r5:00000000
>   r4:ffffffff
> [<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
> [<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
>   r7:ee6b5d24 r6:00000000 r5:00000000 r4:ffffffff
> [<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
> [<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
>   r8:ee6b5d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
> [<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
>   r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
>   r4:ef006900
> [<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
> [<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
>   r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
>   r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
> [<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
>   r10:ee592100 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
>   r4:c141d574
> [<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
> [<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
>   r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
>   r4:ee843800
> [<c0525210>] (__cpufreq_driver_target) from [<c0528e84>] (od_dbs_update+0xe4/0x168)
>   r10:e9b77fc0 r9:c09e04ec r8:ee862480 r7:ee843800 r6:ee862000 r5:ee862480
>   r4:ee862000
> [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee843800 r6:ee862004 r5:00000000 r4:ee862068
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b5f08 r6:ef7c9940 r5:ee862068 r4:ef101e00 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ef101e18 r5:ef7c9974
>   r4:ef101e00
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ef101e00 r7:ef101f38 r6:ef0ffa40 r5:00000000
>   r4:ef101f00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ef0ffa40 r3:ee6b4000
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>
> I thought, maybe, it's the IPU overwriting past the end of the buffer,
> but I've added checks and that doesn't seem to have fired.  I also
> wondered if it was some kind of use-after-free of the ring, so I made
> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
> kfree()ing it... doesn't look like it's that either.  I'm going to
> continue poking to see if I can figure out what's going on.
>
> The oops at 0x00000154 is due to "ring" in imx_media_dma_buf_set_active()
> being NULL.  "buf" in that instance (contained in r4) is 0xd995f724.
>
> I'm just seeing if I can track that down by adding
>
> 	WARN_ON(buf->ring != priv->out_ring);
>
> in imx-smfc.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:31     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 17:31 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Russell,

I don't recommend spending too much time debugging this
OOPS. The dma buffer ring has been removed completely
in version 4 (which I'm trying to get ready to post hopefully
by end of this week).

Steve


On 02/02/2017 09:22 AM, Russell King - ARM Linux wrote:
> I seem to be getting some sort of memory corruption with this driver.
>
> I've had two instances now of uninitialised spinlocks in
> imx_media_dma_buf_get_active() which show that the spinlock being
> taken in this function is all-zeros.
>
> That very quickly leads to an oops, where I've seen buf->ring is
> NULL in imx_media_dma_buf_set_active().
>
> Not quite sure what's going on, but the trigger (at least for me) is
> to change my gstreamer pipeline from:
>
> DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! bayer2rgbneon ! xvimagesink
>
> to
>
> DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/video3 ! queue ! bayer2rgbneon ! xvimagesink
>
> and it seems to take as little as two or three attempts to provoke the
> kernel to totally die.
>
> I've just tried a third time.  I can run the first gstreamer command
> five times.  The I ran the second command and immediately got this:
>
> INFO: trying to register non-static key.
> the code is fine but needs lockdep annotation.
> turning off the locking correctness validator.
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0086af8>] (register_lock_class+0x1d4/0x554)
>   r6:c1400408 r5:00000000 r4:00000000 r3:ee47a4c0
> [<c0086924>] (register_lock_class) from [<c0089474>] (__lock_acquire+0x80/0x17b0)
>   r10:d995f760 r9:c0a70384 r8:00000000 r7:c0a38680 r6:00000000 r5:ee47a4c0
>   r4:c1400408
> [<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
>   r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:d995f760 r5:600f0193
>   r4:00000000
> [<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
>   r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7a8c r5:600f0193
>   r4:d995f750
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
>   r6:e98b2c10 r5:d995f750 r4:d995f600
> [<bf0d7a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf12c4b8>] (imx_smfc_eof_interrupt+0x60/0x124 [imx_smfc])
>   r5:ee935dc4 r4:ee935c10
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 1008 at /home/rmk/git/linux-rmk/drivers/staging/media/imx/imx-smfc.c:159 imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc]
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:600f0193 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
>   r6:bf12d004 r5:00000000 r4:00000000 r3:ee47a4c0
> [<c0033134>] (__warn) from [<c0033264>] (warn_slowpath_null+0x28/0x30)
>   r10:ed501e64 r8:00000000 r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
> [<c003323c>] (warn_slowpath_null) from [<bf12c570>] (imx_smfc_eof_interrupt+0x118/0x124 [imx_smfc])
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> ---[ end trace 36356ae8b82a114e ]---
> Unable to handle kernel NULL pointer dereference at virtual address 00000154
> pgd = ed790000
> [00000154] *pgd=00000000
> Internal error: Oops: 5 [#1] SMP ARM
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 uvcvideo snd_soc_fsl_asoc_card snd_soc_imx_spdif imx_media(C) imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 caam video_multiplexer imx_sdma imx2_wdt rc_cec snd_soc_fsl_ssi coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core snd_soc_fsl_spdif imx_pcm_dma videobuf2_vmalloc dw_hdmi_ahb_audio dw_hdmi_cec videobuf2_memops imx_thermal etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 1008 Comm: Xorg Tainted: G        WC      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> task: ee47a4c0 task.stack: ed500000
> PC is at do_raw_spin_lock+0x10/0x1d0
> LR is at _raw_spin_lock_irqsave+0x54/0x60
> pc : [<c008df34>]    lr : [<c07016fc>]    psr: 600f0193
> sp : ed501d78  ip : ed501db0  fp : ed501dac
> r10: ed501e64  r9 : c09e04ec  r8 : 00000000
> r7 : 00000139  r6 : bf0d7bc8  r5 : 600f0193  r4 : 00000150
> r3 : ee47a4c0  r2 : 00000000  r1 : ed501d58  r0 : 00000150
> Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> Control: 10c5387d  Table: 3d79004a  DAC: 00000051
> Process Xorg (pid: 1008, stack limit = 0xed500210)
> Stack: (0xed501d78 to 0xed502000)
> 1d60:                                                       c011ad20 c09e04ec
> 1d80: c00867fc 00000150 600f0193 bf0d7bc8 00000139 00000000 c09e04ec ed501e64
> 1da0: ed501ddc ed501db0 c07016fc c008df30 00000001 00000000 bf0d7bc8 bf12c570
> 1dc0: ee935c10 d995f724 00000150 00000124 ed501dfc ed501de0 bf0d7bc8 c07016b4
> 1de0: ee935c10 ee935dc4 e98b2c10 00000139 ed501e1c ed501e00 bf12c4d0 bf0d7bb0
> 1e00: bf12c458 ebfb6d40 e98b2c00 e98b2c10 ed501e5c ed501e20 c009f5dc bf12c464
> 1e20: 00000001 c09e04ec 00000000 e98b2c00 c009f9f8 e98b2c00 e98b2c00 e98b2c10
> 1e40: 00000000 00000009 f4001100 ed501fb0 ed501e7c ed501e60 c009f984 c009f544
> 1e60: c0701d10 00000000 e98b2c00 e98b2c60 ed501e9c ed501e80 c009fa00 c009f96c
> 1e80: c09d0418 e98b2c00 e98b2c60 e98b2c10 ed501ebc ed501ea0 c00a3174 c009f9cc
> 1ea0: c00a30c4 00000000 ed501f08 ee4a3010 ed501ecc ed501ec0 c009ecf0 c00a30d0
> 1ec0: ed501efc ed501ed0 c0409328 c009ecdc c09d0448 00000001 0000003d ef1efc10
> 1ee0: c09e756c ee4a3010 00000026 ef008400 ed501f44 ed501f00 c0409458 c040928c
> 1f00: 00000001 00000000 00000001 00000002 00000003 0000000a 0000000b 0000000c
> 1f20: 0000000d 0000000e ed501f44 c09d52d0 00000000 00000000 ed501f54 ed501f48
> 1f40: c009ecf0 c0409408 ed501f7c ed501f58 c009ee24 c009ecdc ed501fb0 f4000100
> 1f60: f400010c c09e0af0 000003eb c0a38a78 ed501fac ed501f80 c00094c8 c009edd4
> 1f80: ee47a4c0 b6904220 600f0030 ffffffff 10c5387d 10c5387d 8114957c 7f79b000
> 1fa0: 00000000 ed501fb0 c0014dd8 c0009488 b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff 00000000 00000000
> Backtrace:
> [<c008df24>] (do_raw_spin_lock) from [<c07016fc>] (_raw_spin_lock_irqsave+0x54/0x60)
>   r10:ed501e64 r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d7bc8 r5:600f0193
>   r4:00000150
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d7bc8>] (imx_media_dma_buf_set_active+0x24/0x68 [imx_media_common])
>   r6:00000124 r5:00000150 r4:d995f724
> [<bf0d7ba4>] (imx_media_dma_buf_set_active [imx_media_common]) from [<bf12c4d0>] (imx_smfc_eof_interrupt+0x78/0x124 [imx_smfc])
>   r7:00000139 r6:e98b2c10 r5:ee935dc4 r4:ee935c10
> [<bf12c458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e98b2c10 r5:e98b2c00 r4:ebfb6d40 r3:bf12c458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ed501fb0 r9:f4001100 r8:00000009 r7:00000000 r6:e98b2c10 r5:e98b2c00
>   r4:e98b2c00
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e98b2c60 r4:e98b2c00
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e98b2c10 r5:e98b2c60 r4:e98b2c00 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ed501f08 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ed501fb0
> [<c000947c>] (gic_handle_irq) from [<c0014dd8>] (__irq_usr+0x58/0x80)
> Exception stack(0xed501fb0 to 0xed501ff8)
> 1fa0:                                     b698b4e0 00000000 0042c000 b698c708
> 1fc0: 00000010 81231b10 81231b18 80e89670 b698b4e0 8114957c 7f79b000 81149438
> 1fe0: 7f79b248 bee08b98 7f708609 b6904220 600f0030 ffffffff
>   r10:7f79b000 r9:8114957c r8:10c5387d r7:10c5387d r6:ffffffff r5:600f0030
>   r4:b6904220 r3:ee47a4c0
> Code: e1a0c00d e92ddff0 e24cb004 e24dd00c (e5902004)
> ---[ end trace 36356ae8b82a114f ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> CPU1: stopping
> CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D WC      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: events dbs_work_handler
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:60010193 r5:ffffffff r4:00000000 r3:ee6b8000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
>   r6:c0a70028 r5:00000001 r4:00000004 r3:ee6b8000
> [<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
>   r10:ee6b5ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
>   r4:f4000100
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6b5ba8 to 0xee6b5bf0)
> 5ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b5d2c
> 5bc0: c00177e8 00000000 00000001 ee6b5d2c 00000000 ee6b5c24 c09e0af4 ee6b5bf8
> 5be0: c0360f2c c00cd3a0 00010013 ffffffff
>   r10:00000000 r9:ee6b4000 r8:00000001 r7:ee6b5bdc r6:ffffffff r5:00010013
>   r4:c00cd3a0 r3:ee6b8000
> [<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
>   r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
> [<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
>   r10:00000002 r9:ffffffff r8:00000002 r7:ee6b5d2c r6:c00177e8 r5:00000000
>   r4:ffffffff
> [<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
> [<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
>   r7:ee6b5d24 r6:00000000 r5:00000000 r4:ffffffff
> [<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
> [<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
>   r8:ee6b5d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
> [<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
>   r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
>   r4:ef006900
> [<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
> [<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
>   r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
>   r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
> [<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
>   r10:ee592100 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
>   r4:c141d574
> [<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
> [<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
>   r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
>   r4:ee843800
> [<c0525210>] (__cpufreq_driver_target) from [<c0528e84>] (od_dbs_update+0xe4/0x168)
>   r10:e9b77fc0 r9:c09e04ec r8:ee862480 r7:ee843800 r6:ee862000 r5:ee862480
>   r4:ee862000
> [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee843800 r6:ee862004 r5:00000000 r4:ee862068
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b5f08 r6:ef7c9940 r5:ee862068 r4:ef101e00 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ef101e18 r5:ef7c9974
>   r4:ef101e00
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ef101e00 r7:ef101f38 r6:ef0ffa40 r5:00000000
>   r4:ef101f00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ef0ffa40 r3:ee6b4000
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>
> I thought, maybe, it's the IPU overwriting past the end of the buffer,
> but I've added checks and that doesn't seem to have fired.  I also
> wondered if it was some kind of use-after-free of the ring, so I made
> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
> kfree()ing it... doesn't look like it's that either.  I'm going to
> continue poking to see if I can figure out what's going on.
>
> The oops at 0x00000154 is due to "ring" in imx_media_dma_buf_set_active()
> being NULL.  "buf" in that instance (contained in r4) is 0xd995f724.
>
> I'm just seeing if I can track that down by adding
>
> 	WARN_ON(buf->ring != priv->out_ring);
>
> in imx-smfc.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:56     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 17:56 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Thu, Feb 02, 2017 at 05:22:46PM +0000, Russell King - ARM Linux wrote:
> I thought, maybe, it's the IPU overwriting past the end of the buffer,
> but I've added checks and that doesn't seem to have fired.  I also
> wondered if it was some kind of use-after-free of the ring, so I made
> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
> kfree()ing it... doesn't look like it's that either.  I'm going to
> continue poking to see if I can figure out what's going on.

I take that back... here's a use-after-free of that buffer, on the
very first run:

Alignment trap: not handling instruction e1921f9f at [<c00894c4>]
Unhandled fault: alignment exception (0x001) at 0x5a5a5b5e
pgd = c0004000
[5a5a5b5e] *pgd=00000000
Internal error: : 1 [#1] SMP ARM
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_spdif snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card imx_media(C) uvcvideo imx_mipi_csi2(C) imx_media_common(C) imx219 snd_soc_sgtl5000 snd_soc_imx_audmux caam video_multiplexer imx_sdma imx2_wdt coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core rc_cec snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_vmalloc videobuf2_memops imx_pcm_dma imx_thermal dw_hdmi_ahb_audio dw_hdmi_cec etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 99 Comm: kworker/0:3 Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: lru-add-drain wq_barrier_func
task: ee4e24c0 task.stack: ee6da000
PC is at __lock_acquire+0xd4/0x17b0
LR is at lock_acquire+0xd8/0x250
pc : [<c00894c8>]    lr : [<c008b108>]    psr: 20070193
sp : ee6dbb60  ip : 00000001  fp : ee6dbbe4
r10: e9efad60  r9 : c0a70384  r8 : 00000000
r7 : c0a38680  r6 : 00000000  r5 : ee4e24c0  r4 : c1400408
r3 : 00000000  r2 : 5a5a5b5e  r1 : 00000000  r0 : 5a5a5a5a
Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 3d7ec04a  DAC: 00000051
Process kworker/0:3 (pid: 99, stack limit = 0xee6da210)
Stack: (0xee6dbb60 to 0xee6dc000)
bb60: c0a38680 00000002 c0b9d8c4 ee4e29a8 ee6dbc04 ee6dbb80 c0089708 c0088d44
bb80: ee6dbb9c 0000050f c00867fc c0086728 ee6dbbf4 ee6dbba0 87eba239 c035aa2f
bba0: 00000001 ee4e29a8 c00c4f84 00000001 00000026 0560e36b ffffffff 00000000
bbc0: 60070193 e9efad60 00000000 00000000 c0a70384 00000000 ee6dbc3c ee6dbbe8
bbe0: c008b108 c0089400 00000001 00000080 00000000 bf0d2a8c 00000000 00000000
bc00: c008b108 c0089400 00000001 c09e04ec 00000000 e9efad50 60070193 bf0d2a8c
bc20: 00000139 00000000 c09e04ec ee6dbcec ee6dbc6c ee6dbc40 c07016f4 c008b03c
bc40: 00000001 00000000 bf0d2a8c ee6dbcec ee6dbc84 e9efac00 e9efad50 ee9785c4
bc60: ee6dbc84 ee6dbc70 bf0d2a8c c07016b4 ee978410 e9efb400 ee6dbca4 ee6dbc88
bc80: bf1224b8 bf0d2a7c bf122458 ee88d4c0 e9efb400 e9efb410 ee6dbce4 ee6dbca8
bca0: c009f5dc bf122464 00000001 c09e04ec 00000000 e9efb400 c009f9f8 e9efb400
bcc0: e9efb400 e9efb410 00000000 00000009 f4001100 ee6dbe38 ee6dbd04 ee6dbce8
bce0: c009f984 c009f544 c0701d10 00000000 e9efb400 e9efb460 ee6dbd24 ee6dbd08
bd00: c009fa00 c009f96c c09d0418 e9efb400 e9efb460 e9efb410 ee6dbd44 ee6dbd28
bd20: c00a3174 c009f9cc c00a30c4 00000000 ee6dbd90 ee4a3010 ee6dbd54 ee6dbd48
bd40: c009ecf0 c00a30d0 ee6dbd84 ee6dbd58 c0409328 c009ecdc c09d0448 00000001
bd60: 00000026 ef1efc10 c09e756c ee4a3010 00000026 ef008400 ee6dbdcc ee6dbd88
bd80: c0409458 c040928c 00000001 00000000 00000001 00000002 00000003 0000000a
bda0: 0000000b 0000000c 0000000d 0000000e ee6dbdcc c09d52d0 00000000 00000000
bdc0: ee6dbddc ee6dbdd0 c009ecf0 c0409408 ee6dbe04 ee6dbde0 c009ee24 c009ecdc
bde0: ee6dbe38 f4000100 f400010c c09e0af0 000003eb c0a38a78 ee6dbe34 ee6dbe08
be00: c00094c8 c009edd4 ee4e24c0 c0701d50 20070013 ffffffff ee6dbe6c ef7be600
be20: ee6da000 c09f5dc6 ee6dbe9c ee6dbe38 c00149f0 c0009488 00000001 ee4e2988
be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
be80: 00000051 00000000 ddb9799c ddb97998 ee6dbebc ee6dbea0 c0083824 c0701d20
bea0: c004e9c4 ee6e6d80 ddb97978 ef7ba940 ee6dbecc ee6dbec0 c004e9d8 c00837e8
bec0: ee6dbf2c ee6dbed0 c0050958 c004e9d0 00000001 00000000 c0050898 00000000
bee0: c0701d8c ee4e24c0 0000000f 00000000 c0a73e7c c0bc8834 00000000 c08947f8
bf00: 00000008 ee6e6d80 ee6e6d98 ee6e6d98 00000008 ef7ba940 ef7ba940 c09dd900
bf20: ee6dbf44 ee6dbf30 c0050e78 c0050774 ee6e6d80 ef7ba974 ee6dbf7c ee6dbf48
bf40: c0051094 c0050e54 00000000 ee6e8ac0 ee509eb8 ee509e80 00000000 ee6e8ac0
bf60: ee509eb8 ee6e6d80 ef0c9e58 c0050e88 ee6dbfac ee6dbf80 c0057b90 c0050e94
bf80: ee6da000 ee6e8ac0 c0057a88 00000000 00000000 00000000 00000000 00000000
bfa0: 00000000 ee6dbfb0 c000fdf0 c0057a94 00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 3fffd861 3fffdc61
Backtrace:
[<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
 r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:e9efad60 r5:60070193
 r4:00000000
[<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
 r10:ee6dbcec r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d2a8c r5:60070193
 r4:e9efad50
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d2a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
 r6:ee9785c4 r5:e9efad50 r4:e9efac00
[<bf0d2a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf1224b8>] (imx_smfc_eof_interrupt+0x60/0x168 [imx_smfc])
 r5:e9efb400 r4:ee978410
[<bf122458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e9efb410 r5:e9efb400 r4:ee88d4c0 r3:bf122458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ee6dbe38 r9:f4001100 r8:00000009 r7:00000000 r6:e9efb410 r5:e9efb400
 r4:e9efb400
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e9efb460 r4:e9efb400
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e9efb410 r5:e9efb460 r4:e9efb400 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ee6dbd90 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ee6dbe38
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6dbe38 to 0xee6dbe80)
be20:                                                       00000001 ee4e2988
be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
 r10:c09f5dc6 r9:ee6da000 r8:ef7be600 r7:ee6dbe6c r6:ffffffff r5:20070013
 r4:c0701d50 r3:ee4e24c0
[<c0701d14>] (_raw_spin_unlock_irqrestore) from [<c0083824>] (complete+0x48/0x4c)
 r5:ddb97998 r4:ddb9799c
[<c00837dc>] (complete) from [<c004e9d8>] (wq_barrier_func+0x14/0x18)
 r6:ef7ba940 r5:ddb97978 r4:ee6e6d80 r3:c004e9c4
[<c004e9c4>] (wq_barrier_func) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
[<c0050768>] (process_one_work) from [<c0050e78>] (process_scheduled_works+0x30/0x40)
 r10:c09dd900 r9:ef7ba940 r8:ef7ba940 r7:00000008 r6:ee6e6d98 r5:ee6e6d98
 r4:ee6e6d80
[<c0050e48>] (process_scheduled_works) from [<c0051094>] (worker_thread+0x20c/0x4c8)
 r5:ef7ba974 r4:ee6e6d80
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0c9e58 r8:ee6e6d80 r7:ee509eb8 r6:ee6e8ac0 r5:00000000
 r4:ee509e80
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ee6e8ac0 r3:ee6da000
Code: 0affffe9 e2802f41 f592f000 e1921f9f (e2811001)
---[ end trace 2e91a0629044cda4 ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D  C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: events dbs_work_handler
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60000193 r5:ffffffff r4:00000000 r3:ee4e6e40
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
 r6:c0a70028 r5:00000001 r4:00000004 r3:ee4e6e40
[<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
 r10:ee6b7ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
 r4:f4000100
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6b7ba8 to 0xee6b7bf0)
7ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b7d2c
7bc0: c00177e8 00000000 00000001 ee6b7d2c 00000000 ee6b7c24 c09e0af4 ee6b7bf8
7be0: c0360f2c c00cd3a0 00000013 ffffffff
 r10:00000000 r9:ee6b6000 r8:00000001 r7:ee6b7bdc r6:ffffffff r5:00000013
 r4:c00cd3a0 r3:ee4e6e40
[<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
 r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
[<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
 r10:00000002 r9:ffffffff r8:00000002 r7:ee6b7d2c r6:c00177e8 r5:00000000
 r4:ffffffff
[<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
[<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
 r7:ee6b7d24 r6:00000000 r5:00000000 r4:ffffffff
[<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
[<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
 r8:ee6b7d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
[<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
 r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
 r4:ef006900
[<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
[<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
 r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
 r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
[<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
 r10:ee719c00 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
 r4:c141d574
[<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
[<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
 r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
 r4:ee726000
[<c0525210>] (__cpufreq_driver_target) from [<c0528e40>] (od_dbs_update+0xa0/0x168)
 r10:eb522380 r9:c09e04ec r8:ebd79d80 r7:ee726000 r6:ebd79e40 r5:ebd79d80
 r4:eb522380
[<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
 r4:ee509900
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
 r4:ee509a00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24[  165.007974] [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
 r4:ee509900
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
 r4:ee509a00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ef0ff1c0 r3:ee6b6000
---[ end Kernel panic - not syncing: Fatal exception in interrupt

This happens because (a little more debugging - notably a dump_stack()
in imx_media_free_dma_buf_ring()):

CPU: 0 PID: 2322 Comm: v4l2src0:src Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:a0010013 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<bf0dc0b4>] (imx_media_free_dma_buf_ring+0x18/0x74 [imx_media_common])
 r6:bf059540 r5:ffffffea r4:ee40ce00 r3:00000004
[<bf0dc09c>] (imx_media_free_dma_buf_ring [imx_media_common]) from [<bf1590dc>] (camif_buf_prepare+0x9c/0x130 [imx_camif])
 r5:ffffffea r4:d008f010
[<bf159040>] (camif_buf_prepare [imx_camif]) from [<bf054b1c>] (__buf_prepare+0x130/0x1dc [videobuf2_core])
 r6:bf059540 r5:d004f800 r4:00000000
[<bf0549ec>] (__buf_prepare [videobuf2_core]) from [<bf054c20>] (vb2_core_qbuf+0x58/0x324 [videobuf2_core])
 r6:e74b1e20 r5:d004f800 r4:d008f6b0
[<bf054bc8>] (vb2_core_qbuf [videobuf2_core]) from [<bf068028>] (vb2_qbuf+0x58/0x80 [videobuf2_v4l2])
 r10:c0a57704 r9:e74b1e20 r8:e7417140 r7:bf15977c r6:e974c980 r5:e74b1e20
 r4:d008f6b0 r3:00000000
[<bf067fd0>] (vb2_qbuf [videobuf2_v4l2]) from [<bf068098>] (vb2_ioctl_qbuf+0x48/0x4c [videobuf2_v4l2])
 r5:e74b1e20 r4:e7417140
[<bf068050>] (vb2_ioctl_qbuf [videobuf2_v4l2]) from [<c04f746c>] (v4l_qbuf+0x44/0x48)
 r5:e74b1e20 r4:e7417140
[<c04f7428>] (v4l_qbuf) from [<c04f4660>] (__video_do_ioctl+0x270/0x304)
 r7:00000000 r6:e974c980 r5:d008f018 r4:c044560f
[<c04f43f0>] (__video_do_ioctl) from [<c04f6b48>] (video_usercopy+0x12c/0x85c)
 r10:00000000 r9:c04f43f0 r8:00000000 r7:e74b1e20 r6:b5408888 r5:00000000
 r4:c044560f
[<c04f6a1c>] (video_usercopy) from [<c04f7290>] (video_ioctl2+0x18/0x1c)
 r10:d024e568 r9:e74b0000 r8:d008f64c r7:c044560f r6:b5408888 r5:e7417140
 r4:d008f018
[<c04f7278>] (video_ioctl2) from [<c04f1524>] (v4l2_ioctl+0xa4/0xc4)
[<c04f1480>] (v4l2_ioctl) from [<c0188b6c>] (do_vfs_ioctl+0x98/0x9a0)
 r8:b5408888 r7:0000000d r6:0000000d r5:e7417140 r4:c01894b0 r3:c04f1480
[<c0188ad4>] (do_vfs_ioctl) from [<c01894b0>] (SyS_ioctl+0x3c/0x60)
 r10:00000000 r9:e74b0000 r8:b5408888 r7:0000000d r6:c044560f r5:e7417140
 r4:e7417141
[<c0189474>] (SyS_ioctl) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
 r8:c000ff04 r7:00000036 r6:00066800 r5:b68fa000 r4:b540887c r3:00000000

This is totally broken, and here's why.  Immediately before the above
are these lines:

[  114.120099] ipu1_smfc0: stream ON
[  114.234338] imx6-mipi-csi2: stream ON
[  114.258187] imx6-mipi-csi2: ready, dphy version 0x3130302a
[  114.263767] imx6-mipi-csi2: stream ON
[  114.267495] ipu1_csi0: stream ON

At the "ipu1_smfc0" stream on message, smfc calls imx_smfc_start().
imx_smfc_start() asks for the ring:

        /* ask the sink for the buffer ring */
        ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
                               IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
                               &priv->out_ring);

camif provides the ring:

static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
        case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
                if (!priv->in_ring)
                        return -EINVAL;
                ring = (struct imx_media_dma_buf_ring **)arg;
                *ring = priv->in_ring;
                break;

So, smfc contains a copy of the pointer to camif's priv->in_ring.

Things continue, and we get to camif_buf_prepare():

static int camif_buf_prepare(struct vb2_buffer *vb)
{
...
        if (!priv->in_ring) {
                priv->in_ring = imx_media_alloc_dma_buf_ring(
                        priv->md, &priv->src_sd->entity, &priv->sd.entity,
                        sizeimage, vq->num_buffers, false);
                if (IS_ERR(priv->in_ring)) {
                        v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
                        ret = PTR_ERR(priv->in_ring);
                        priv->in_ring = NULL;
                        return ret;
                }
        }

Well, if we haven't setup priv->in_ring by now... is anything going to
work?

Then we do this:

        ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
        if (ret)
                goto free_ring;

        return 0;

free_ring:
        imx_media_free_dma_buf_ring(priv->in_ring);
        priv->in_ring = NULL;
        return ret;

and for whatever reason we end up falling out through free_ring.  This
is VERY bad news, because it means that the ring which SMFC took a copy
of is now freed beneath its feet.

It doesn't matter if you later reallocate it, it could very well end up
with a different pointer from kmalloc().

SMFC continues along unknowing that its priv->out_ring is now invalid, and
it tries to use it as if it is still valid, leading to two things:

1. potentially stamping over memory that has been given to someone else
   (possibly inodes, resulting in filesystem corruption should that
    memory get written back to disk)
2. dereferencing pointers to other random memory leading to who-knows-what.

and that is _very_ bad.

This is way too serious a bug to justify any further testing.

So here endeth my interest in this driver until a new set of patches
appears for review. :p

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:56     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 17:56 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland-5wv7dgnIgG8, andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	nick-gcszYUEDH4VrovVCs/uTlw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, Steve Longerbeam,
	robert.jarzmik-GANU6spQydw,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	geert-Td1EMuHUCqxL1ZNQvxDV9g, linux-media-u79uwXL29TY76Z2rM5mHXA,
	devicetree-u79uwXL29TY76Z2rM5mHXA, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	arnd-r2nGTMty4D4, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	bparrot-l0cyMroinI0, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	jean-christophe.trotin-qxv4g6HH51o,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, sudipm.m

On Thu, Feb 02, 2017 at 05:22:46PM +0000, Russell King - ARM Linux wrote:
> I thought, maybe, it's the IPU overwriting past the end of the buffer,
> but I've added checks and that doesn't seem to have fired.  I also
> wondered if it was some kind of use-after-free of the ring, so I made
> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
> kfree()ing it... doesn't look like it's that either.  I'm going to
> continue poking to see if I can figure out what's going on.

I take that back... here's a use-after-free of that buffer, on the
very first run:

Alignment trap: not handling instruction e1921f9f at [<c00894c4>]
Unhandled fault: alignment exception (0x001) at 0x5a5a5b5e
pgd = c0004000
[5a5a5b5e] *pgd=00000000
Internal error: : 1 [#1] SMP ARM
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_spdif snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card imx_media(C) uvcvideo imx_mipi_csi2(C) imx_media_common(C) imx219 snd_soc_sgtl5000 snd_soc_imx_audmux caam video_multiplexer imx_sdma imx2_wdt coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core rc_cec snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_vmalloc videobuf2_memops imx_pcm_dma imx_thermal dw_hdmi_ahb_audio dw_hdmi_cec etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 99 Comm: kworker/0:3 Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: lru-add-drain wq_barrier_func
task: ee4e24c0 task.stack: ee6da000
PC is at __lock_acquire+0xd4/0x17b0
LR is at lock_acquire+0xd8/0x250
pc : [<c00894c8>]    lr : [<c008b108>]    psr: 20070193
sp : ee6dbb60  ip : 00000001  fp : ee6dbbe4
r10: e9efad60  r9 : c0a70384  r8 : 00000000
r7 : c0a38680  r6 : 00000000  r5 : ee4e24c0  r4 : c1400408
r3 : 00000000  r2 : 5a5a5b5e  r1 : 00000000  r0 : 5a5a5a5a
Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 3d7ec04a  DAC: 00000051
Process kworker/0:3 (pid: 99, stack limit = 0xee6da210)
Stack: (0xee6dbb60 to 0xee6dc000)
bb60: c0a38680 00000002 c0b9d8c4 ee4e29a8 ee6dbc04 ee6dbb80 c0089708 c0088d44
bb80: ee6dbb9c 0000050f c00867fc c0086728 ee6dbbf4 ee6dbba0 87eba239 c035aa2f
bba0: 00000001 ee4e29a8 c00c4f84 00000001 00000026 0560e36b ffffffff 00000000
bbc0: 60070193 e9efad60 00000000 00000000 c0a70384 00000000 ee6dbc3c ee6dbbe8
bbe0: c008b108 c0089400 00000001 00000080 00000000 bf0d2a8c 00000000 00000000
bc00: c008b108 c0089400 00000001 c09e04ec 00000000 e9efad50 60070193 bf0d2a8c
bc20: 00000139 00000000 c09e04ec ee6dbcec ee6dbc6c ee6dbc40 c07016f4 c008b03c
bc40: 00000001 00000000 bf0d2a8c ee6dbcec ee6dbc84 e9efac00 e9efad50 ee9785c4
bc60: ee6dbc84 ee6dbc70 bf0d2a8c c07016b4 ee978410 e9efb400 ee6dbca4 ee6dbc88
bc80: bf1224b8 bf0d2a7c bf122458 ee88d4c0 e9efb400 e9efb410 ee6dbce4 ee6dbca8
bca0: c009f5dc bf122464 00000001 c09e04ec 00000000 e9efb400 c009f9f8 e9efb400
bcc0: e9efb400 e9efb410 00000000 00000009 f4001100 ee6dbe38 ee6dbd04 ee6dbce8
bce0: c009f984 c009f544 c0701d10 00000000 e9efb400 e9efb460 ee6dbd24 ee6dbd08
bd00: c009fa00 c009f96c c09d0418 e9efb400 e9efb460 e9efb410 ee6dbd44 ee6dbd28
bd20: c00a3174 c009f9cc c00a30c4 00000000 ee6dbd90 ee4a3010 ee6dbd54 ee6dbd48
bd40: c009ecf0 c00a30d0 ee6dbd84 ee6dbd58 c0409328 c009ecdc c09d0448 00000001
bd60: 00000026 ef1efc10 c09e756c ee4a3010 00000026 ef008400 ee6dbdcc ee6dbd88
bd80: c0409458 c040928c 00000001 00000000 00000001 00000002 00000003 0000000a
bda0: 0000000b 0000000c 0000000d 0000000e ee6dbdcc c09d52d0 00000000 00000000
bdc0: ee6dbddc ee6dbdd0 c009ecf0 c0409408 ee6dbe04 ee6dbde0 c009ee24 c009ecdc
bde0: ee6dbe38 f4000100 f400010c c09e0af0 000003eb c0a38a78 ee6dbe34 ee6dbe08
be00: c00094c8 c009edd4 ee4e24c0 c0701d50 20070013 ffffffff ee6dbe6c ef7be600
be20: ee6da000 c09f5dc6 ee6dbe9c ee6dbe38 c00149f0 c0009488 00000001 ee4e2988
be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
be80: 00000051 00000000 ddb9799c ddb97998 ee6dbebc ee6dbea0 c0083824 c0701d20
bea0: c004e9c4 ee6e6d80 ddb97978 ef7ba940 ee6dbecc ee6dbec0 c004e9d8 c00837e8
bec0: ee6dbf2c ee6dbed0 c0050958 c004e9d0 00000001 00000000 c0050898 00000000
bee0: c0701d8c ee4e24c0 0000000f 00000000 c0a73e7c c0bc8834 00000000 c08947f8
bf00: 00000008 ee6e6d80 ee6e6d98 ee6e6d98 00000008 ef7ba940 ef7ba940 c09dd900
bf20: ee6dbf44 ee6dbf30 c0050e78 c0050774 ee6e6d80 ef7ba974 ee6dbf7c ee6dbf48
bf40: c0051094 c0050e54 00000000 ee6e8ac0 ee509eb8 ee509e80 00000000 ee6e8ac0
bf60: ee509eb8 ee6e6d80 ef0c9e58 c0050e88 ee6dbfac ee6dbf80 c0057b90 c0050e94
bf80: ee6da000 ee6e8ac0 c0057a88 00000000 00000000 00000000 00000000 00000000
bfa0: 00000000 ee6dbfb0 c000fdf0 c0057a94 00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 3fffd861 3fffdc61
Backtrace:
[<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
 r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:e9efad60 r5:60070193
 r4:00000000
[<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
 r10:ee6dbcec r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d2a8c r5:60070193
 r4:e9efad50
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d2a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
 r6:ee9785c4 r5:e9efad50 r4:e9efac00
[<bf0d2a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf1224b8>] (imx_smfc_eof_interrupt+0x60/0x168 [imx_smfc])
 r5:e9efb400 r4:ee978410
[<bf122458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e9efb410 r5:e9efb400 r4:ee88d4c0 r3:bf122458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ee6dbe38 r9:f4001100 r8:00000009 r7:00000000 r6:e9efb410 r5:e9efb400
 r4:e9efb400
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e9efb460 r4:e9efb400
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e9efb410 r5:e9efb460 r4:e9efb400 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ee6dbd90 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ee6dbe38
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6dbe38 to 0xee6dbe80)
be20:                                                       00000001 ee4e2988
be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
 r10:c09f5dc6 r9:ee6da000 r8:ef7be600 r7:ee6dbe6c r6:ffffffff r5:20070013
 r4:c0701d50 r3:ee4e24c0
[<c0701d14>] (_raw_spin_unlock_irqrestore) from [<c0083824>] (complete+0x48/0x4c)
 r5:ddb97998 r4:ddb9799c
[<c00837dc>] (complete) from [<c004e9d8>] (wq_barrier_func+0x14/0x18)
 r6:ef7ba940 r5:ddb97978 r4:ee6e6d80 r3:c004e9c4
[<c004e9c4>] (wq_barrier_func) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
[<c0050768>] (process_one_work) from [<c0050e78>] (process_scheduled_works+0x30/0x40)
 r10:c09dd900 r9:ef7ba940 r8:ef7ba940 r7:00000008 r6:ee6e6d98 r5:ee6e6d98
 r4:ee6e6d80
[<c0050e48>] (process_scheduled_works) from [<c0051094>] (worker_thread+0x20c/0x4c8)
 r5:ef7ba974 r4:ee6e6d80
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0c9e58 r8:ee6e6d80 r7:ee509eb8 r6:ee6e8ac0 r5:00000000
 r4:ee509e80
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ee6e8ac0 r3:ee6da000
Code: 0affffe9 e2802f41 f592f000 e1921f9f (e2811001)
---[ end trace 2e91a0629044cda4 ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D  C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: events dbs_work_handler
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60000193 r5:ffffffff r4:00000000 r3:ee4e6e40
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
 r6:c0a70028 r5:00000001 r4:00000004 r3:ee4e6e40
[<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
 r10:ee6b7ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
 r4:f4000100
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6b7ba8 to 0xee6b7bf0)
7ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b7d2c
7bc0: c00177e8 00000000 00000001 ee6b7d2c 00000000 ee6b7c24 c09e0af4 ee6b7bf8
7be0: c0360f2c c00cd3a0 00000013 ffffffff
 r10:00000000 r9:ee6b6000 r8:00000001 r7:ee6b7bdc r6:ffffffff r5:00000013
 r4:c00cd3a0 r3:ee4e6e40
[<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
 r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
[<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
 r10:00000002 r9:ffffffff r8:00000002 r7:ee6b7d2c r6:c00177e8 r5:00000000
 r4:ffffffff
[<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
[<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
 r7:ee6b7d24 r6:00000000 r5:00000000 r4:ffffffff
[<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
[<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
 r8:ee6b7d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
[<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
 r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
 r4:ef006900
[<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
[<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
 r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
 r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
[<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
 r10:ee719c00 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
 r4:c141d574
[<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
[<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
 r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
 r4:ee726000
[<c0525210>] (__cpufreq_driver_target) from [<c0528e40>] (od_dbs_update+0xa0/0x168)
 r10:eb522380 r9:c09e04ec r8:ebd79d80 r7:ee726000 r6:ebd79e40 r5:ebd79d80
 r4:eb522380
[<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
 r4:ee509900
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
 r4:ee509a00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24[  165.007974] [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
 r4:ee509900
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
 r4:ee509a00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ef0ff1c0 r3:ee6b6000
---[ end Kernel panic - not syncing: Fatal exception in interrupt

This happens because (a little more debugging - notably a dump_stack()
in imx_media_free_dma_buf_ring()):

CPU: 0 PID: 2322 Comm: v4l2src0:src Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:a0010013 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<bf0dc0b4>] (imx_media_free_dma_buf_ring+0x18/0x74 [imx_media_common])
 r6:bf059540 r5:ffffffea r4:ee40ce00 r3:00000004
[<bf0dc09c>] (imx_media_free_dma_buf_ring [imx_media_common]) from [<bf1590dc>] (camif_buf_prepare+0x9c/0x130 [imx_camif])
 r5:ffffffea r4:d008f010
[<bf159040>] (camif_buf_prepare [imx_camif]) from [<bf054b1c>] (__buf_prepare+0x130/0x1dc [videobuf2_core])
 r6:bf059540 r5:d004f800 r4:00000000
[<bf0549ec>] (__buf_prepare [videobuf2_core]) from [<bf054c20>] (vb2_core_qbuf+0x58/0x324 [videobuf2_core])
 r6:e74b1e20 r5:d004f800 r4:d008f6b0
[<bf054bc8>] (vb2_core_qbuf [videobuf2_core]) from [<bf068028>] (vb2_qbuf+0x58/0x80 [videobuf2_v4l2])
 r10:c0a57704 r9:e74b1e20 r8:e7417140 r7:bf15977c r6:e974c980 r5:e74b1e20
 r4:d008f6b0 r3:00000000
[<bf067fd0>] (vb2_qbuf [videobuf2_v4l2]) from [<bf068098>] (vb2_ioctl_qbuf+0x48/0x4c [videobuf2_v4l2])
 r5:e74b1e20 r4:e7417140
[<bf068050>] (vb2_ioctl_qbuf [videobuf2_v4l2]) from [<c04f746c>] (v4l_qbuf+0x44/0x48)
 r5:e74b1e20 r4:e7417140
[<c04f7428>] (v4l_qbuf) from [<c04f4660>] (__video_do_ioctl+0x270/0x304)
 r7:00000000 r6:e974c980 r5:d008f018 r4:c044560f
[<c04f43f0>] (__video_do_ioctl) from [<c04f6b48>] (video_usercopy+0x12c/0x85c)
 r10:00000000 r9:c04f43f0 r8:00000000 r7:e74b1e20 r6:b5408888 r5:00000000
 r4:c044560f
[<c04f6a1c>] (video_usercopy) from [<c04f7290>] (video_ioctl2+0x18/0x1c)
 r10:d024e568 r9:e74b0000 r8:d008f64c r7:c044560f r6:b5408888 r5:e7417140
 r4:d008f018
[<c04f7278>] (video_ioctl2) from [<c04f1524>] (v4l2_ioctl+0xa4/0xc4)
[<c04f1480>] (v4l2_ioctl) from [<c0188b6c>] (do_vfs_ioctl+0x98/0x9a0)
 r8:b5408888 r7:0000000d r6:0000000d r5:e7417140 r4:c01894b0 r3:c04f1480
[<c0188ad4>] (do_vfs_ioctl) from [<c01894b0>] (SyS_ioctl+0x3c/0x60)
 r10:00000000 r9:e74b0000 r8:b5408888 r7:0000000d r6:c044560f r5:e7417140
 r4:e7417141
[<c0189474>] (SyS_ioctl) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
 r8:c000ff04 r7:00000036 r6:00066800 r5:b68fa000 r4:b540887c r3:00000000

This is totally broken, and here's why.  Immediately before the above
are these lines:

[  114.120099] ipu1_smfc0: stream ON
[  114.234338] imx6-mipi-csi2: stream ON
[  114.258187] imx6-mipi-csi2: ready, dphy version 0x3130302a
[  114.263767] imx6-mipi-csi2: stream ON
[  114.267495] ipu1_csi0: stream ON

At the "ipu1_smfc0" stream on message, smfc calls imx_smfc_start().
imx_smfc_start() asks for the ring:

        /* ask the sink for the buffer ring */
        ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
                               IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
                               &priv->out_ring);

camif provides the ring:

static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
        case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
                if (!priv->in_ring)
                        return -EINVAL;
                ring = (struct imx_media_dma_buf_ring **)arg;
                *ring = priv->in_ring;
                break;

So, smfc contains a copy of the pointer to camif's priv->in_ring.

Things continue, and we get to camif_buf_prepare():

static int camif_buf_prepare(struct vb2_buffer *vb)
{
...
        if (!priv->in_ring) {
                priv->in_ring = imx_media_alloc_dma_buf_ring(
                        priv->md, &priv->src_sd->entity, &priv->sd.entity,
                        sizeimage, vq->num_buffers, false);
                if (IS_ERR(priv->in_ring)) {
                        v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
                        ret = PTR_ERR(priv->in_ring);
                        priv->in_ring = NULL;
                        return ret;
                }
        }

Well, if we haven't setup priv->in_ring by now... is anything going to
work?

Then we do this:

        ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
        if (ret)
                goto free_ring;

        return 0;

free_ring:
        imx_media_free_dma_buf_ring(priv->in_ring);
        priv->in_ring = NULL;
        return ret;

and for whatever reason we end up falling out through free_ring.  This
is VERY bad news, because it means that the ring which SMFC took a copy
of is now freed beneath its feet.

It doesn't matter if you later reallocate it, it could very well end up
with a different pointer from kmalloc().

SMFC continues along unknowing that its priv->out_ring is now invalid, and
it tries to use it as if it is still valid, leading to two things:

1. potentially stamping over memory that has been given to someone else
   (possibly inodes, resulting in filesystem corruption should that
    memory get written back to disk)
2. dereferencing pointers to other random memory leading to who-knows-what.

and that is _very_ bad.

This is way too serious a bug to justify any further testing.

So here endeth my interest in this driver until a new set of patches
appears for review. :p

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 17:56     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 17:56 UTC (permalink / raw)
  To: linux-arm-kernel

On Thu, Feb 02, 2017 at 05:22:46PM +0000, Russell King - ARM Linux wrote:
> I thought, maybe, it's the IPU overwriting past the end of the buffer,
> but I've added checks and that doesn't seem to have fired.  I also
> wondered if it was some kind of use-after-free of the ring, so I made
> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
> kfree()ing it... doesn't look like it's that either.  I'm going to
> continue poking to see if I can figure out what's going on.

I take that back... here's a use-after-free of that buffer, on the
very first run:

Alignment trap: not handling instruction e1921f9f at [<c00894c4>]
Unhandled fault: alignment exception (0x001) at 0x5a5a5b5e
pgd = c0004000
[5a5a5b5e] *pgd=00000000
Internal error: : 1 [#1] SMP ARM
Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_spdif snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card imx_media(C) uvcvideo imx_mipi_csi2(C) imx_media_common(C) imx219 snd_soc_sgtl5000 snd_soc_imx_audmux caam video_multiplexer imx_sdma imx2_wdt coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core rc_cec snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_vmalloc videobuf2_memops imx_pcm_dma imx_thermal dw_hdmi_ahb_audio dw_hdmi_cec etnaviv fuse rc_pinnacle_pctv_hd
CPU: 0 PID: 99 Comm: kworker/0:3 Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: lru-add-drain wq_barrier_func
task: ee4e24c0 task.stack: ee6da000
PC is at __lock_acquire+0xd4/0x17b0
LR is at lock_acquire+0xd8/0x250
pc : [<c00894c8>]    lr : [<c008b108>]    psr: 20070193
sp : ee6dbb60  ip : 00000001  fp : ee6dbbe4
r10: e9efad60  r9 : c0a70384  r8 : 00000000
r7 : c0a38680  r6 : 00000000  r5 : ee4e24c0  r4 : c1400408
r3 : 00000000  r2 : 5a5a5b5e  r1 : 00000000  r0 : 5a5a5a5a
Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 3d7ec04a  DAC: 00000051
Process kworker/0:3 (pid: 99, stack limit = 0xee6da210)
Stack: (0xee6dbb60 to 0xee6dc000)
bb60: c0a38680 00000002 c0b9d8c4 ee4e29a8 ee6dbc04 ee6dbb80 c0089708 c0088d44
bb80: ee6dbb9c 0000050f c00867fc c0086728 ee6dbbf4 ee6dbba0 87eba239 c035aa2f
bba0: 00000001 ee4e29a8 c00c4f84 00000001 00000026 0560e36b ffffffff 00000000
bbc0: 60070193 e9efad60 00000000 00000000 c0a70384 00000000 ee6dbc3c ee6dbbe8
bbe0: c008b108 c0089400 00000001 00000080 00000000 bf0d2a8c 00000000 00000000
bc00: c008b108 c0089400 00000001 c09e04ec 00000000 e9efad50 60070193 bf0d2a8c
bc20: 00000139 00000000 c09e04ec ee6dbcec ee6dbc6c ee6dbc40 c07016f4 c008b03c
bc40: 00000001 00000000 bf0d2a8c ee6dbcec ee6dbc84 e9efac00 e9efad50 ee9785c4
bc60: ee6dbc84 ee6dbc70 bf0d2a8c c07016b4 ee978410 e9efb400 ee6dbca4 ee6dbc88
bc80: bf1224b8 bf0d2a7c bf122458 ee88d4c0 e9efb400 e9efb410 ee6dbce4 ee6dbca8
bca0: c009f5dc bf122464 00000001 c09e04ec 00000000 e9efb400 c009f9f8 e9efb400
bcc0: e9efb400 e9efb410 00000000 00000009 f4001100 ee6dbe38 ee6dbd04 ee6dbce8
bce0: c009f984 c009f544 c0701d10 00000000 e9efb400 e9efb460 ee6dbd24 ee6dbd08
bd00: c009fa00 c009f96c c09d0418 e9efb400 e9efb460 e9efb410 ee6dbd44 ee6dbd28
bd20: c00a3174 c009f9cc c00a30c4 00000000 ee6dbd90 ee4a3010 ee6dbd54 ee6dbd48
bd40: c009ecf0 c00a30d0 ee6dbd84 ee6dbd58 c0409328 c009ecdc c09d0448 00000001
bd60: 00000026 ef1efc10 c09e756c ee4a3010 00000026 ef008400 ee6dbdcc ee6dbd88
bd80: c0409458 c040928c 00000001 00000000 00000001 00000002 00000003 0000000a
bda0: 0000000b 0000000c 0000000d 0000000e ee6dbdcc c09d52d0 00000000 00000000
bdc0: ee6dbddc ee6dbdd0 c009ecf0 c0409408 ee6dbe04 ee6dbde0 c009ee24 c009ecdc
bde0: ee6dbe38 f4000100 f400010c c09e0af0 000003eb c0a38a78 ee6dbe34 ee6dbe08
be00: c00094c8 c009edd4 ee4e24c0 c0701d50 20070013 ffffffff ee6dbe6c ef7be600
be20: ee6da000 c09f5dc6 ee6dbe9c ee6dbe38 c00149f0 c0009488 00000001 ee4e2988
be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
be80: 00000051 00000000 ddb9799c ddb97998 ee6dbebc ee6dbea0 c0083824 c0701d20
bea0: c004e9c4 ee6e6d80 ddb97978 ef7ba940 ee6dbecc ee6dbec0 c004e9d8 c00837e8
bec0: ee6dbf2c ee6dbed0 c0050958 c004e9d0 00000001 00000000 c0050898 00000000
bee0: c0701d8c ee4e24c0 0000000f 00000000 c0a73e7c c0bc8834 00000000 c08947f8
bf00: 00000008 ee6e6d80 ee6e6d98 ee6e6d98 00000008 ef7ba940 ef7ba940 c09dd900
bf20: ee6dbf44 ee6dbf30 c0050e78 c0050774 ee6e6d80 ef7ba974 ee6dbf7c ee6dbf48
bf40: c0051094 c0050e54 00000000 ee6e8ac0 ee509eb8 ee509e80 00000000 ee6e8ac0
bf60: ee509eb8 ee6e6d80 ef0c9e58 c0050e88 ee6dbfac ee6dbf80 c0057b90 c0050e94
bf80: ee6da000 ee6e8ac0 c0057a88 00000000 00000000 00000000 00000000 00000000
bfa0: 00000000 ee6dbfb0 c000fdf0 c0057a94 00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 3fffd861 3fffdc61
Backtrace:
[<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
 r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:e9efad60 r5:60070193
 r4:00000000
[<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
 r10:ee6dbcec r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d2a8c r5:60070193
 r4:e9efad50
[<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d2a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
 r6:ee9785c4 r5:e9efad50 r4:e9efac00
[<bf0d2a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf1224b8>] (imx_smfc_eof_interrupt+0x60/0x168 [imx_smfc])
 r5:e9efb400 r4:ee978410
[<bf122458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
 r6:e9efb410 r5:e9efb400 r4:ee88d4c0 r3:bf122458
[<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
 r10:ee6dbe38 r9:f4001100 r8:00000009 r7:00000000 r6:e9efb410 r5:e9efb400
 r4:e9efb400
[<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
 r5:e9efb460 r4:e9efb400
[<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
 r6:e9efb410 r5:e9efb460 r4:e9efb400 r3:c09d0418
[<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:ee4a3010 r5:ee6dbd90 r4:00000000 r3:c00a30c4
[<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
[<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
 r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
[<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
 r6:00000000 r5:00000000 r4:c09d52d0
[<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
[<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ee6dbe38
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6dbe38 to 0xee6dbe80)
be20:                                                       00000001 ee4e2988
be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
 r10:c09f5dc6 r9:ee6da000 r8:ef7be600 r7:ee6dbe6c r6:ffffffff r5:20070013
 r4:c0701d50 r3:ee4e24c0
[<c0701d14>] (_raw_spin_unlock_irqrestore) from [<c0083824>] (complete+0x48/0x4c)
 r5:ddb97998 r4:ddb9799c
[<c00837dc>] (complete) from [<c004e9d8>] (wq_barrier_func+0x14/0x18)
 r6:ef7ba940 r5:ddb97978 r4:ee6e6d80 r3:c004e9c4
[<c004e9c4>] (wq_barrier_func) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
[<c0050768>] (process_one_work) from [<c0050e78>] (process_scheduled_works+0x30/0x40)
 r10:c09dd900 r9:ef7ba940 r8:ef7ba940 r7:00000008 r6:ee6e6d98 r5:ee6e6d98
 r4:ee6e6d80
[<c0050e48>] (process_scheduled_works) from [<c0051094>] (worker_thread+0x20c/0x4c8)
 r5:ef7ba974 r4:ee6e6d80
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0c9e58 r8:ee6e6d80 r7:ee509eb8 r6:ee6e8ac0 r5:00000000
 r4:ee509e80
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ee6e8ac0 r3:ee6da000
Code: 0affffe9 e2802f41 f592f000 e1921f9f (e2811001)
---[ end trace 2e91a0629044cda4 ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D  C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Workqueue: events dbs_work_handler
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60000193 r5:ffffffff r4:00000000 r3:ee4e6e40
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
 r6:c0a70028 r5:00000001 r4:00000004 r3:ee4e6e40
[<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
 r10:ee6b7ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
 r4:f4000100
[<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
Exception stack(0xee6b7ba8 to 0xee6b7bf0)
7ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b7d2c
7bc0: c00177e8 00000000 00000001 ee6b7d2c 00000000 ee6b7c24 c09e0af4 ee6b7bf8
7be0: c0360f2c c00cd3a0 00000013 ffffffff
 r10:00000000 r9:ee6b6000 r8:00000001 r7:ee6b7bdc r6:ffffffff r5:00000013
 r4:c00cd3a0 r3:ee4e6e40
[<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
 r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
[<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
 r10:00000002 r9:ffffffff r8:00000002 r7:ee6b7d2c r6:c00177e8 r5:00000000
 r4:ffffffff
[<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
[<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
 r7:ee6b7d24 r6:00000000 r5:00000000 r4:ffffffff
[<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
[<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
 r8:ee6b7d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
[<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
 r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
 r4:ef006900
[<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
[<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
 r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
 r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
[<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
 r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
[<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
 r10:ee719c00 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
 r4:c141d574
[<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
[<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
 r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
 r4:ee726000
[<c0525210>] (__cpufreq_driver_target) from [<c0528e40>] (od_dbs_update+0xa0/0x168)
 r10:eb522380 r9:c09e04ec r8:ebd79d80 r7:ee726000 r6:ebd79e40 r5:ebd79d80
 r4:eb522380
[<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
 r4:ee509900
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
 r4:ee509a00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24[  165.007974] [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
 r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
[<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
 r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
[<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
 r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
 r4:ee509900
[<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
 r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
 r4:ee509a00
[<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
 r4:ef0ff1c0 r3:ee6b6000
---[ end Kernel panic - not syncing: Fatal exception in interrupt

This happens because (a little more debugging - notably a dump_stack()
in imx_media_free_dma_buf_ring()):

CPU: 0 PID: 2322 Comm: v4l2src0:src Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:a0010013 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<bf0dc0b4>] (imx_media_free_dma_buf_ring+0x18/0x74 [imx_media_common])
 r6:bf059540 r5:ffffffea r4:ee40ce00 r3:00000004
[<bf0dc09c>] (imx_media_free_dma_buf_ring [imx_media_common]) from [<bf1590dc>] (camif_buf_prepare+0x9c/0x130 [imx_camif])
 r5:ffffffea r4:d008f010
[<bf159040>] (camif_buf_prepare [imx_camif]) from [<bf054b1c>] (__buf_prepare+0x130/0x1dc [videobuf2_core])
 r6:bf059540 r5:d004f800 r4:00000000
[<bf0549ec>] (__buf_prepare [videobuf2_core]) from [<bf054c20>] (vb2_core_qbuf+0x58/0x324 [videobuf2_core])
 r6:e74b1e20 r5:d004f800 r4:d008f6b0
[<bf054bc8>] (vb2_core_qbuf [videobuf2_core]) from [<bf068028>] (vb2_qbuf+0x58/0x80 [videobuf2_v4l2])
 r10:c0a57704 r9:e74b1e20 r8:e7417140 r7:bf15977c r6:e974c980 r5:e74b1e20
 r4:d008f6b0 r3:00000000
[<bf067fd0>] (vb2_qbuf [videobuf2_v4l2]) from [<bf068098>] (vb2_ioctl_qbuf+0x48/0x4c [videobuf2_v4l2])
 r5:e74b1e20 r4:e7417140
[<bf068050>] (vb2_ioctl_qbuf [videobuf2_v4l2]) from [<c04f746c>] (v4l_qbuf+0x44/0x48)
 r5:e74b1e20 r4:e7417140
[<c04f7428>] (v4l_qbuf) from [<c04f4660>] (__video_do_ioctl+0x270/0x304)
 r7:00000000 r6:e974c980 r5:d008f018 r4:c044560f
[<c04f43f0>] (__video_do_ioctl) from [<c04f6b48>] (video_usercopy+0x12c/0x85c)
 r10:00000000 r9:c04f43f0 r8:00000000 r7:e74b1e20 r6:b5408888 r5:00000000
 r4:c044560f
[<c04f6a1c>] (video_usercopy) from [<c04f7290>] (video_ioctl2+0x18/0x1c)
 r10:d024e568 r9:e74b0000 r8:d008f64c r7:c044560f r6:b5408888 r5:e7417140
 r4:d008f018
[<c04f7278>] (video_ioctl2) from [<c04f1524>] (v4l2_ioctl+0xa4/0xc4)
[<c04f1480>] (v4l2_ioctl) from [<c0188b6c>] (do_vfs_ioctl+0x98/0x9a0)
 r8:b5408888 r7:0000000d r6:0000000d r5:e7417140 r4:c01894b0 r3:c04f1480
[<c0188ad4>] (do_vfs_ioctl) from [<c01894b0>] (SyS_ioctl+0x3c/0x60)
 r10:00000000 r9:e74b0000 r8:b5408888 r7:0000000d r6:c044560f r5:e7417140
 r4:e7417141
[<c0189474>] (SyS_ioctl) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
 r8:c000ff04 r7:00000036 r6:00066800 r5:b68fa000 r4:b540887c r3:00000000

This is totally broken, and here's why.  Immediately before the above
are these lines:

[  114.120099] ipu1_smfc0: stream ON
[  114.234338] imx6-mipi-csi2: stream ON
[  114.258187] imx6-mipi-csi2: ready, dphy version 0x3130302a
[  114.263767] imx6-mipi-csi2: stream ON
[  114.267495] ipu1_csi0: stream ON

At the "ipu1_smfc0" stream on message, smfc calls imx_smfc_start().
imx_smfc_start() asks for the ring:

        /* ask the sink for the buffer ring */
        ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
                               IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
                               &priv->out_ring);

camif provides the ring:

static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
        case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
                if (!priv->in_ring)
                        return -EINVAL;
                ring = (struct imx_media_dma_buf_ring **)arg;
                *ring = priv->in_ring;
                break;

So, smfc contains a copy of the pointer to camif's priv->in_ring.

Things continue, and we get to camif_buf_prepare():

static int camif_buf_prepare(struct vb2_buffer *vb)
{
...
        if (!priv->in_ring) {
                priv->in_ring = imx_media_alloc_dma_buf_ring(
                        priv->md, &priv->src_sd->entity, &priv->sd.entity,
                        sizeimage, vq->num_buffers, false);
                if (IS_ERR(priv->in_ring)) {
                        v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
                        ret = PTR_ERR(priv->in_ring);
                        priv->in_ring = NULL;
                        return ret;
                }
        }

Well, if we haven't setup priv->in_ring by now... is anything going to
work?

Then we do this:

        ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
        if (ret)
                goto free_ring;

        return 0;

free_ring:
        imx_media_free_dma_buf_ring(priv->in_ring);
        priv->in_ring = NULL;
        return ret;

and for whatever reason we end up falling out through free_ring.  This
is VERY bad news, because it means that the ring which SMFC took a copy
of is now freed beneath its feet.

It doesn't matter if you later reallocate it, it could very well end up
with a different pointer from kmalloc().

SMFC continues along unknowing that its priv->out_ring is now invalid, and
it tries to use it as if it is still valid, leading to two things:

1. potentially stamping over memory that has been given to someone else
   (possibly inodes, resulting in filesystem corruption should that
    memory get written back to disk)
2. dereferencing pointers to other random memory leading to who-knows-what.

and that is _very_ bad.

This is way too serious a bug to justify any further testing.

So here endeth my interest in this driver until a new set of patches
appears for review. :p

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-02 17:56     ` Russell King - ARM Linux
  (?)
@ 2017-02-02 18:26       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 18:26 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 05:22:46PM +0000, Russell King - ARM Linux wrote:
>> I thought, maybe, it's the IPU overwriting past the end of the buffer,
>> but I've added checks and that doesn't seem to have fired.  I also
>> wondered if it was some kind of use-after-free of the ring, so I made
>> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
>> kfree()ing it... doesn't look like it's that either.  I'm going to
>> continue poking to see if I can figure out what's going on.
> I take that back... here's a use-after-free of that buffer, on the
> very first run:
>
> Alignment trap: not handling instruction e1921f9f at [<c00894c4>]
> Unhandled fault: alignment exception (0x001) at 0x5a5a5b5e
> pgd = c0004000
> [5a5a5b5e] *pgd=00000000
> Internal error: : 1 [#1] SMP ARM
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_spdif snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card imx_media(C) uvcvideo imx_mipi_csi2(C) imx_media_common(C) imx219 snd_soc_sgtl5000 snd_soc_imx_audmux caam video_multiplexer imx_sdma imx2_wdt coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core rc_cec snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_vmalloc videobuf2_memops imx_pcm_dma imx_thermal dw_hdmi_ahb_audio dw_hdmi_cec etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 99 Comm: kworker/0:3 Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: lru-add-drain wq_barrier_func
> task: ee4e24c0 task.stack: ee6da000
> PC is at __lock_acquire+0xd4/0x17b0
> LR is at lock_acquire+0xd8/0x250
> pc : [<c00894c8>]    lr : [<c008b108>]    psr: 20070193
> sp : ee6dbb60  ip : 00000001  fp : ee6dbbe4
> r10: e9efad60  r9 : c0a70384  r8 : 00000000
> r7 : c0a38680  r6 : 00000000  r5 : ee4e24c0  r4 : c1400408
> r3 : 00000000  r2 : 5a5a5b5e  r1 : 00000000  r0 : 5a5a5a5a
> Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> Control: 10c5387d  Table: 3d7ec04a  DAC: 00000051
> Process kworker/0:3 (pid: 99, stack limit = 0xee6da210)
> Stack: (0xee6dbb60 to 0xee6dc000)
> bb60: c0a38680 00000002 c0b9d8c4 ee4e29a8 ee6dbc04 ee6dbb80 c0089708 c0088d44
> bb80: ee6dbb9c 0000050f c00867fc c0086728 ee6dbbf4 ee6dbba0 87eba239 c035aa2f
> bba0: 00000001 ee4e29a8 c00c4f84 00000001 00000026 0560e36b ffffffff 00000000
> bbc0: 60070193 e9efad60 00000000 00000000 c0a70384 00000000 ee6dbc3c ee6dbbe8
> bbe0: c008b108 c0089400 00000001 00000080 00000000 bf0d2a8c 00000000 00000000
> bc00: c008b108 c0089400 00000001 c09e04ec 00000000 e9efad50 60070193 bf0d2a8c
> bc20: 00000139 00000000 c09e04ec ee6dbcec ee6dbc6c ee6dbc40 c07016f4 c008b03c
> bc40: 00000001 00000000 bf0d2a8c ee6dbcec ee6dbc84 e9efac00 e9efad50 ee9785c4
> bc60: ee6dbc84 ee6dbc70 bf0d2a8c c07016b4 ee978410 e9efb400 ee6dbca4 ee6dbc88
> bc80: bf1224b8 bf0d2a7c bf122458 ee88d4c0 e9efb400 e9efb410 ee6dbce4 ee6dbca8
> bca0: c009f5dc bf122464 00000001 c09e04ec 00000000 e9efb400 c009f9f8 e9efb400
> bcc0: e9efb400 e9efb410 00000000 00000009 f4001100 ee6dbe38 ee6dbd04 ee6dbce8
> bce0: c009f984 c009f544 c0701d10 00000000 e9efb400 e9efb460 ee6dbd24 ee6dbd08
> bd00: c009fa00 c009f96c c09d0418 e9efb400 e9efb460 e9efb410 ee6dbd44 ee6dbd28
> bd20: c00a3174 c009f9cc c00a30c4 00000000 ee6dbd90 ee4a3010 ee6dbd54 ee6dbd48
> bd40: c009ecf0 c00a30d0 ee6dbd84 ee6dbd58 c0409328 c009ecdc c09d0448 00000001
> bd60: 00000026 ef1efc10 c09e756c ee4a3010 00000026 ef008400 ee6dbdcc ee6dbd88
> bd80: c0409458 c040928c 00000001 00000000 00000001 00000002 00000003 0000000a
> bda0: 0000000b 0000000c 0000000d 0000000e ee6dbdcc c09d52d0 00000000 00000000
> bdc0: ee6dbddc ee6dbdd0 c009ecf0 c0409408 ee6dbe04 ee6dbde0 c009ee24 c009ecdc
> bde0: ee6dbe38 f4000100 f400010c c09e0af0 000003eb c0a38a78 ee6dbe34 ee6dbe08
> be00: c00094c8 c009edd4 ee4e24c0 c0701d50 20070013 ffffffff ee6dbe6c ef7be600
> be20: ee6da000 c09f5dc6 ee6dbe9c ee6dbe38 c00149f0 c0009488 00000001 ee4e2988
> be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
> be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
> be80: 00000051 00000000 ddb9799c ddb97998 ee6dbebc ee6dbea0 c0083824 c0701d20
> bea0: c004e9c4 ee6e6d80 ddb97978 ef7ba940 ee6dbecc ee6dbec0 c004e9d8 c00837e8
> bec0: ee6dbf2c ee6dbed0 c0050958 c004e9d0 00000001 00000000 c0050898 00000000
> bee0: c0701d8c ee4e24c0 0000000f 00000000 c0a73e7c c0bc8834 00000000 c08947f8
> bf00: 00000008 ee6e6d80 ee6e6d98 ee6e6d98 00000008 ef7ba940 ef7ba940 c09dd900
> bf20: ee6dbf44 ee6dbf30 c0050e78 c0050774 ee6e6d80 ef7ba974 ee6dbf7c ee6dbf48
> bf40: c0051094 c0050e54 00000000 ee6e8ac0 ee509eb8 ee509e80 00000000 ee6e8ac0
> bf60: ee509eb8 ee6e6d80 ef0c9e58 c0050e88 ee6dbfac ee6dbf80 c0057b90 c0050e94
> bf80: ee6da000 ee6e8ac0 c0057a88 00000000 00000000 00000000 00000000 00000000
> bfa0: 00000000 ee6dbfb0 c000fdf0 c0057a94 00000000 00000000 00000000 00000000
> bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 3fffd861 3fffdc61
> Backtrace:
> [<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
>   r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:e9efad60 r5:60070193
>   r4:00000000
> [<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
>   r10:ee6dbcec r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d2a8c r5:60070193
>   r4:e9efad50
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d2a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
>   r6:ee9785c4 r5:e9efad50 r4:e9efac00
> [<bf0d2a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf1224b8>] (imx_smfc_eof_interrupt+0x60/0x168 [imx_smfc])
>   r5:e9efb400 r4:ee978410
> [<bf122458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e9efb410 r5:e9efb400 r4:ee88d4c0 r3:bf122458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ee6dbe38 r9:f4001100 r8:00000009 r7:00000000 r6:e9efb410 r5:e9efb400
>   r4:e9efb400
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e9efb460 r4:e9efb400
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e9efb410 r5:e9efb460 r4:e9efb400 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ee6dbd90 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ee6dbe38
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6dbe38 to 0xee6dbe80)
> be20:                                                       00000001 ee4e2988
> be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
> be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
>   r10:c09f5dc6 r9:ee6da000 r8:ef7be600 r7:ee6dbe6c r6:ffffffff r5:20070013
>   r4:c0701d50 r3:ee4e24c0
> [<c0701d14>] (_raw_spin_unlock_irqrestore) from [<c0083824>] (complete+0x48/0x4c)
>   r5:ddb97998 r4:ddb9799c
> [<c00837dc>] (complete) from [<c004e9d8>] (wq_barrier_func+0x14/0x18)
>   r6:ef7ba940 r5:ddb97978 r4:ee6e6d80 r3:c004e9c4
> [<c004e9c4>] (wq_barrier_func) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
> [<c0050768>] (process_one_work) from [<c0050e78>] (process_scheduled_works+0x30/0x40)
>   r10:c09dd900 r9:ef7ba940 r8:ef7ba940 r7:00000008 r6:ee6e6d98 r5:ee6e6d98
>   r4:ee6e6d80
> [<c0050e48>] (process_scheduled_works) from [<c0051094>] (worker_thread+0x20c/0x4c8)
>   r5:ef7ba974 r4:ee6e6d80
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0c9e58 r8:ee6e6d80 r7:ee509eb8 r6:ee6e8ac0 r5:00000000
>   r4:ee509e80
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ee6e8ac0 r3:ee6da000
> Code: 0affffe9 e2802f41 f592f000 e1921f9f (e2811001)
> ---[ end trace 2e91a0629044cda4 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> CPU1: stopping
> CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D  C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: events dbs_work_handler
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:60000193 r5:ffffffff r4:00000000 r3:ee4e6e40
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
>   r6:c0a70028 r5:00000001 r4:00000004 r3:ee4e6e40
> [<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
>   r10:ee6b7ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
>   r4:f4000100
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6b7ba8 to 0xee6b7bf0)
> 7ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b7d2c
> 7bc0: c00177e8 00000000 00000001 ee6b7d2c 00000000 ee6b7c24 c09e0af4 ee6b7bf8
> 7be0: c0360f2c c00cd3a0 00000013 ffffffff
>   r10:00000000 r9:ee6b6000 r8:00000001 r7:ee6b7bdc r6:ffffffff r5:00000013
>   r4:c00cd3a0 r3:ee4e6e40
> [<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
>   r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
> [<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
>   r10:00000002 r9:ffffffff r8:00000002 r7:ee6b7d2c r6:c00177e8 r5:00000000
>   r4:ffffffff
> [<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
> [<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
>   r7:ee6b7d24 r6:00000000 r5:00000000 r4:ffffffff
> [<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
> [<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
>   r8:ee6b7d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
> [<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
>   r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
>   r4:ef006900
> [<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
> [<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
>   r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
>   r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
> [<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
>   r10:ee719c00 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
>   r4:c141d574
> [<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
> [<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
>   r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
>   r4:ee726000
> [<c0525210>] (__cpufreq_driver_target) from [<c0528e40>] (od_dbs_update+0xa0/0x168)
>   r10:eb522380 r9:c09e04ec r8:ebd79d80 r7:ee726000 r6:ebd79e40 r5:ebd79d80
>   r4:eb522380
> [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
>   r4:ee509900
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
>   r4:ee509a00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24[  165.007974] [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
>   r4:ee509900
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
>   r4:ee509a00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ef0ff1c0 r3:ee6b6000
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>
> This happens because (a little more debugging - notably a dump_stack()
> in imx_media_free_dma_buf_ring()):
>
> CPU: 0 PID: 2322 Comm: v4l2src0:src Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:a0010013 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<bf0dc0b4>] (imx_media_free_dma_buf_ring+0x18/0x74 [imx_media_common])
>   r6:bf059540 r5:ffffffea r4:ee40ce00 r3:00000004
> [<bf0dc09c>] (imx_media_free_dma_buf_ring [imx_media_common]) from [<bf1590dc>] (camif_buf_prepare+0x9c/0x130 [imx_camif])
>   r5:ffffffea r4:d008f010
> [<bf159040>] (camif_buf_prepare [imx_camif]) from [<bf054b1c>] (__buf_prepare+0x130/0x1dc [videobuf2_core])
>   r6:bf059540 r5:d004f800 r4:00000000
> [<bf0549ec>] (__buf_prepare [videobuf2_core]) from [<bf054c20>] (vb2_core_qbuf+0x58/0x324 [videobuf2_core])
>   r6:e74b1e20 r5:d004f800 r4:d008f6b0
> [<bf054bc8>] (vb2_core_qbuf [videobuf2_core]) from [<bf068028>] (vb2_qbuf+0x58/0x80 [videobuf2_v4l2])
>   r10:c0a57704 r9:e74b1e20 r8:e7417140 r7:bf15977c r6:e974c980 r5:e74b1e20
>   r4:d008f6b0 r3:00000000
> [<bf067fd0>] (vb2_qbuf [videobuf2_v4l2]) from [<bf068098>] (vb2_ioctl_qbuf+0x48/0x4c [videobuf2_v4l2])
>   r5:e74b1e20 r4:e7417140
> [<bf068050>] (vb2_ioctl_qbuf [videobuf2_v4l2]) from [<c04f746c>] (v4l_qbuf+0x44/0x48)
>   r5:e74b1e20 r4:e7417140
> [<c04f7428>] (v4l_qbuf) from [<c04f4660>] (__video_do_ioctl+0x270/0x304)
>   r7:00000000 r6:e974c980 r5:d008f018 r4:c044560f
> [<c04f43f0>] (__video_do_ioctl) from [<c04f6b48>] (video_usercopy+0x12c/0x85c)
>   r10:00000000 r9:c04f43f0 r8:00000000 r7:e74b1e20 r6:b5408888 r5:00000000
>   r4:c044560f
> [<c04f6a1c>] (video_usercopy) from [<c04f7290>] (video_ioctl2+0x18/0x1c)
>   r10:d024e568 r9:e74b0000 r8:d008f64c r7:c044560f r6:b5408888 r5:e7417140
>   r4:d008f018
> [<c04f7278>] (video_ioctl2) from [<c04f1524>] (v4l2_ioctl+0xa4/0xc4)
> [<c04f1480>] (v4l2_ioctl) from [<c0188b6c>] (do_vfs_ioctl+0x98/0x9a0)
>   r8:b5408888 r7:0000000d r6:0000000d r5:e7417140 r4:c01894b0 r3:c04f1480
> [<c0188ad4>] (do_vfs_ioctl) from [<c01894b0>] (SyS_ioctl+0x3c/0x60)
>   r10:00000000 r9:e74b0000 r8:b5408888 r7:0000000d r6:c044560f r5:e7417140
>   r4:e7417141
> [<c0189474>] (SyS_ioctl) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
>   r8:c000ff04 r7:00000036 r6:00066800 r5:b68fa000 r4:b540887c r3:00000000
>
> This is totally broken, and here's why.  Immediately before the above
> are these lines:
>
> [  114.120099] ipu1_smfc0: stream ON
> [  114.234338] imx6-mipi-csi2: stream ON
> [  114.258187] imx6-mipi-csi2: ready, dphy version 0x3130302a
> [  114.263767] imx6-mipi-csi2: stream ON
> [  114.267495] ipu1_csi0: stream ON
>
> At the "ipu1_smfc0" stream on message, smfc calls imx_smfc_start().
> imx_smfc_start() asks for the ring:
>
>          /* ask the sink for the buffer ring */
>          ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
>                                 IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
>                                 &priv->out_ring);
>
> camif provides the ring:
>
> static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
>          case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
>                  if (!priv->in_ring)
>                          return -EINVAL;
>                  ring = (struct imx_media_dma_buf_ring **)arg;
>                  *ring = priv->in_ring;
>                  break;
>
> So, smfc contains a copy of the pointer to camif's priv->in_ring.
>
> Things continue, and we get to camif_buf_prepare():
>
> static int camif_buf_prepare(struct vb2_buffer *vb)
> {
> ...
>          if (!priv->in_ring) {
>                  priv->in_ring = imx_media_alloc_dma_buf_ring(
>                          priv->md, &priv->src_sd->entity, &priv->sd.entity,
>                          sizeimage, vq->num_buffers, false);
>                  if (IS_ERR(priv->in_ring)) {
>                          v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
>                          ret = PTR_ERR(priv->in_ring);
>                          priv->in_ring = NULL;
>                          return ret;
>                  }
>          }
>
> Well, if we haven't setup priv->in_ring by now... is anything going to
> work?
>
> Then we do this:
>
>          ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
>          if (ret)
>                  goto free_ring;
>
>          return 0;
>
> free_ring:
>          imx_media_free_dma_buf_ring(priv->in_ring);
>          priv->in_ring = NULL;
>          return ret;
>
> and for whatever reason we end up falling out through free_ring.  This
> is VERY bad news, because it means that the ring which SMFC took a copy
> of is now freed beneath its feet.

Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
returned error, the ring should not have been freed, it should have only
returned the error. And further bad stuff happens from that point on.

But all of this is gone in version 4.

Steve

>
> It doesn't matter if you later reallocate it, it could very well end up
> with a different pointer from kmalloc().
>
> SMFC continues along unknowing that its priv->out_ring is now invalid, and
> it tries to use it as if it is still valid, leading to two things:
>
> 1. potentially stamping over memory that has been given to someone else
>     (possibly inodes, resulting in filesystem corruption should that
>      memory get written back to disk)
> 2. dereferencing pointers to other random memory leading to who-knows-what.
>
> and that is _very_ bad.
>
> This is way too serious a bug to justify any further testing.
>
> So here endeth my interest in this driver until a new set of patches
> appears for review. :p
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 18:26       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 18:26 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 05:22:46PM +0000, Russell King - ARM Linux wrote:
>> I thought, maybe, it's the IPU overwriting past the end of the buffer,
>> but I've added checks and that doesn't seem to have fired.  I also
>> wondered if it was some kind of use-after-free of the ring, so I made
>> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
>> kfree()ing it... doesn't look like it's that either.  I'm going to
>> continue poking to see if I can figure out what's going on.
> I take that back... here's a use-after-free of that buffer, on the
> very first run:
>
> Alignment trap: not handling instruction e1921f9f at [<c00894c4>]
> Unhandled fault: alignment exception (0x001) at 0x5a5a5b5e
> pgd = c0004000
> [5a5a5b5e] *pgd=00000000
> Internal error: : 1 [#1] SMP ARM
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_spdif snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card imx_media(C) uvcvideo imx_mipi_csi2(C) imx_media_common(C) imx219 snd_soc_sgtl5000 snd_soc_imx_audmux caam video_multiplexer imx_sdma imx2_wdt coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core rc_cec snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_vmalloc videobuf2_memops imx_pcm_dma imx_thermal dw_hdmi_ahb_audio dw_hdmi_cec etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 99 Comm: kworker/0:3 Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: lru-add-drain wq_barrier_func
> task: ee4e24c0 task.stack: ee6da000
> PC is at __lock_acquire+0xd4/0x17b0
> LR is at lock_acquire+0xd8/0x250
> pc : [<c00894c8>]    lr : [<c008b108>]    psr: 20070193
> sp : ee6dbb60  ip : 00000001  fp : ee6dbbe4
> r10: e9efad60  r9 : c0a70384  r8 : 00000000
> r7 : c0a38680  r6 : 00000000  r5 : ee4e24c0  r4 : c1400408
> r3 : 00000000  r2 : 5a5a5b5e  r1 : 00000000  r0 : 5a5a5a5a
> Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> Control: 10c5387d  Table: 3d7ec04a  DAC: 00000051
> Process kworker/0:3 (pid: 99, stack limit = 0xee6da210)
> Stack: (0xee6dbb60 to 0xee6dc000)
> bb60: c0a38680 00000002 c0b9d8c4 ee4e29a8 ee6dbc04 ee6dbb80 c0089708 c0088d44
> bb80: ee6dbb9c 0000050f c00867fc c0086728 ee6dbbf4 ee6dbba0 87eba239 c035aa2f
> bba0: 00000001 ee4e29a8 c00c4f84 00000001 00000026 0560e36b ffffffff 00000000
> bbc0: 60070193 e9efad60 00000000 00000000 c0a70384 00000000 ee6dbc3c ee6dbbe8
> bbe0: c008b108 c0089400 00000001 00000080 00000000 bf0d2a8c 00000000 00000000
> bc00: c008b108 c0089400 00000001 c09e04ec 00000000 e9efad50 60070193 bf0d2a8c
> bc20: 00000139 00000000 c09e04ec ee6dbcec ee6dbc6c ee6dbc40 c07016f4 c008b03c
> bc40: 00000001 00000000 bf0d2a8c ee6dbcec ee6dbc84 e9efac00 e9efad50 ee9785c4
> bc60: ee6dbc84 ee6dbc70 bf0d2a8c c07016b4 ee978410 e9efb400 ee6dbca4 ee6dbc88
> bc80: bf1224b8 bf0d2a7c bf122458 ee88d4c0 e9efb400 e9efb410 ee6dbce4 ee6dbca8
> bca0: c009f5dc bf122464 00000001 c09e04ec 00000000 e9efb400 c009f9f8 e9efb400
> bcc0: e9efb400 e9efb410 00000000 00000009 f4001100 ee6dbe38 ee6dbd04 ee6dbce8
> bce0: c009f984 c009f544 c0701d10 00000000 e9efb400 e9efb460 ee6dbd24 ee6dbd08
> bd00: c009fa00 c009f96c c09d0418 e9efb400 e9efb460 e9efb410 ee6dbd44 ee6dbd28
> bd20: c00a3174 c009f9cc c00a30c4 00000000 ee6dbd90 ee4a3010 ee6dbd54 ee6dbd48
> bd40: c009ecf0 c00a30d0 ee6dbd84 ee6dbd58 c0409328 c009ecdc c09d0448 00000001
> bd60: 00000026 ef1efc10 c09e756c ee4a3010 00000026 ef008400 ee6dbdcc ee6dbd88
> bd80: c0409458 c040928c 00000001 00000000 00000001 00000002 00000003 0000000a
> bda0: 0000000b 0000000c 0000000d 0000000e ee6dbdcc c09d52d0 00000000 00000000
> bdc0: ee6dbddc ee6dbdd0 c009ecf0 c0409408 ee6dbe04 ee6dbde0 c009ee24 c009ecdc
> bde0: ee6dbe38 f4000100 f400010c c09e0af0 000003eb c0a38a78 ee6dbe34 ee6dbe08
> be00: c00094c8 c009edd4 ee4e24c0 c0701d50 20070013 ffffffff ee6dbe6c ef7be600
> be20: ee6da000 c09f5dc6 ee6dbe9c ee6dbe38 c00149f0 c0009488 00000001 ee4e2988
> be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
> be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
> be80: 00000051 00000000 ddb9799c ddb97998 ee6dbebc ee6dbea0 c0083824 c0701d20
> bea0: c004e9c4 ee6e6d80 ddb97978 ef7ba940 ee6dbecc ee6dbec0 c004e9d8 c00837e8
> bec0: ee6dbf2c ee6dbed0 c0050958 c004e9d0 00000001 00000000 c0050898 00000000
> bee0: c0701d8c ee4e24c0 0000000f 00000000 c0a73e7c c0bc8834 00000000 c08947f8
> bf00: 00000008 ee6e6d80 ee6e6d98 ee6e6d98 00000008 ef7ba940 ef7ba940 c09dd900
> bf20: ee6dbf44 ee6dbf30 c0050e78 c0050774 ee6e6d80 ef7ba974 ee6dbf7c ee6dbf48
> bf40: c0051094 c0050e54 00000000 ee6e8ac0 ee509eb8 ee509e80 00000000 ee6e8ac0
> bf60: ee509eb8 ee6e6d80 ef0c9e58 c0050e88 ee6dbfac ee6dbf80 c0057b90 c0050e94
> bf80: ee6da000 ee6e8ac0 c0057a88 00000000 00000000 00000000 00000000 00000000
> bfa0: 00000000 ee6dbfb0 c000fdf0 c0057a94 00000000 00000000 00000000 00000000
> bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 3fffd861 3fffdc61
> Backtrace:
> [<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
>   r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:e9efad60 r5:60070193
>   r4:00000000
> [<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
>   r10:ee6dbcec r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d2a8c r5:60070193
>   r4:e9efad50
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d2a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
>   r6:ee9785c4 r5:e9efad50 r4:e9efac00
> [<bf0d2a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf1224b8>] (imx_smfc_eof_interrupt+0x60/0x168 [imx_smfc])
>   r5:e9efb400 r4:ee978410
> [<bf122458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e9efb410 r5:e9efb400 r4:ee88d4c0 r3:bf122458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ee6dbe38 r9:f4001100 r8:00000009 r7:00000000 r6:e9efb410 r5:e9efb400
>   r4:e9efb400
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e9efb460 r4:e9efb400
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e9efb410 r5:e9efb460 r4:e9efb400 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ee6dbd90 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ee6dbe38
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6dbe38 to 0xee6dbe80)
> be20:                                                       00000001 ee4e2988
> be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
> be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
>   r10:c09f5dc6 r9:ee6da000 r8:ef7be600 r7:ee6dbe6c r6:ffffffff r5:20070013
>   r4:c0701d50 r3:ee4e24c0
> [<c0701d14>] (_raw_spin_unlock_irqrestore) from [<c0083824>] (complete+0x48/0x4c)
>   r5:ddb97998 r4:ddb9799c
> [<c00837dc>] (complete) from [<c004e9d8>] (wq_barrier_func+0x14/0x18)
>   r6:ef7ba940 r5:ddb97978 r4:ee6e6d80 r3:c004e9c4
> [<c004e9c4>] (wq_barrier_func) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
> [<c0050768>] (process_one_work) from [<c0050e78>] (process_scheduled_works+0x30/0x40)
>   r10:c09dd900 r9:ef7ba940 r8:ef7ba940 r7:00000008 r6:ee6e6d98 r5:ee6e6d98
>   r4:ee6e6d80
> [<c0050e48>] (process_scheduled_works) from [<c0051094>] (worker_thread+0x20c/0x4c8)
>   r5:ef7ba974 r4:ee6e6d80
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0c9e58 r8:ee6e6d80 r7:ee509eb8 r6:ee6e8ac0 r5:00000000
>   r4:ee509e80
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ee6e8ac0 r3:ee6da000
> Code: 0affffe9 e2802f41 f592f000 e1921f9f (e2811001)
> ---[ end trace 2e91a0629044cda4 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> CPU1: stopping
> CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D  C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: events dbs_work_handler
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:60000193 r5:ffffffff r4:00000000 r3:ee4e6e40
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
>   r6:c0a70028 r5:00000001 r4:00000004 r3:ee4e6e40
> [<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
>   r10:ee6b7ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
>   r4:f4000100
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6b7ba8 to 0xee6b7bf0)
> 7ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b7d2c
> 7bc0: c00177e8 00000000 00000001 ee6b7d2c 00000000 ee6b7c24 c09e0af4 ee6b7bf8
> 7be0: c0360f2c c00cd3a0 00000013 ffffffff
>   r10:00000000 r9:ee6b6000 r8:00000001 r7:ee6b7bdc r6:ffffffff r5:00000013
>   r4:c00cd3a0 r3:ee4e6e40
> [<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
>   r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
> [<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
>   r10:00000002 r9:ffffffff r8:00000002 r7:ee6b7d2c r6:c00177e8 r5:00000000
>   r4:ffffffff
> [<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
> [<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
>   r7:ee6b7d24 r6:00000000 r5:00000000 r4:ffffffff
> [<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
> [<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
>   r8:ee6b7d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
> [<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
>   r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
>   r4:ef006900
> [<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
> [<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
>   r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
>   r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
> [<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
>   r10:ee719c00 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
>   r4:c141d574
> [<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
> [<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
>   r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
>   r4:ee726000
> [<c0525210>] (__cpufreq_driver_target) from [<c0528e40>] (od_dbs_update+0xa0/0x168)
>   r10:eb522380 r9:c09e04ec r8:ebd79d80 r7:ee726000 r6:ebd79e40 r5:ebd79d80
>   r4:eb522380
> [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
>   r4:ee509900
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
>   r4:ee509a00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24[  165.007974] [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
>   r4:ee509900
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
>   r4:ee509a00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ef0ff1c0 r3:ee6b6000
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>
> This happens because (a little more debugging - notably a dump_stack()
> in imx_media_free_dma_buf_ring()):
>
> CPU: 0 PID: 2322 Comm: v4l2src0:src Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:a0010013 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<bf0dc0b4>] (imx_media_free_dma_buf_ring+0x18/0x74 [imx_media_common])
>   r6:bf059540 r5:ffffffea r4:ee40ce00 r3:00000004
> [<bf0dc09c>] (imx_media_free_dma_buf_ring [imx_media_common]) from [<bf1590dc>] (camif_buf_prepare+0x9c/0x130 [imx_camif])
>   r5:ffffffea r4:d008f010
> [<bf159040>] (camif_buf_prepare [imx_camif]) from [<bf054b1c>] (__buf_prepare+0x130/0x1dc [videobuf2_core])
>   r6:bf059540 r5:d004f800 r4:00000000
> [<bf0549ec>] (__buf_prepare [videobuf2_core]) from [<bf054c20>] (vb2_core_qbuf+0x58/0x324 [videobuf2_core])
>   r6:e74b1e20 r5:d004f800 r4:d008f6b0
> [<bf054bc8>] (vb2_core_qbuf [videobuf2_core]) from [<bf068028>] (vb2_qbuf+0x58/0x80 [videobuf2_v4l2])
>   r10:c0a57704 r9:e74b1e20 r8:e7417140 r7:bf15977c r6:e974c980 r5:e74b1e20
>   r4:d008f6b0 r3:00000000
> [<bf067fd0>] (vb2_qbuf [videobuf2_v4l2]) from [<bf068098>] (vb2_ioctl_qbuf+0x48/0x4c [videobuf2_v4l2])
>   r5:e74b1e20 r4:e7417140
> [<bf068050>] (vb2_ioctl_qbuf [videobuf2_v4l2]) from [<c04f746c>] (v4l_qbuf+0x44/0x48)
>   r5:e74b1e20 r4:e7417140
> [<c04f7428>] (v4l_qbuf) from [<c04f4660>] (__video_do_ioctl+0x270/0x304)
>   r7:00000000 r6:e974c980 r5:d008f018 r4:c044560f
> [<c04f43f0>] (__video_do_ioctl) from [<c04f6b48>] (video_usercopy+0x12c/0x85c)
>   r10:00000000 r9:c04f43f0 r8:00000000 r7:e74b1e20 r6:b5408888 r5:00000000
>   r4:c044560f
> [<c04f6a1c>] (video_usercopy) from [<c04f7290>] (video_ioctl2+0x18/0x1c)
>   r10:d024e568 r9:e74b0000 r8:d008f64c r7:c044560f r6:b5408888 r5:e7417140
>   r4:d008f018
> [<c04f7278>] (video_ioctl2) from [<c04f1524>] (v4l2_ioctl+0xa4/0xc4)
> [<c04f1480>] (v4l2_ioctl) from [<c0188b6c>] (do_vfs_ioctl+0x98/0x9a0)
>   r8:b5408888 r7:0000000d r6:0000000d r5:e7417140 r4:c01894b0 r3:c04f1480
> [<c0188ad4>] (do_vfs_ioctl) from [<c01894b0>] (SyS_ioctl+0x3c/0x60)
>   r10:00000000 r9:e74b0000 r8:b5408888 r7:0000000d r6:c044560f r5:e7417140
>   r4:e7417141
> [<c0189474>] (SyS_ioctl) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
>   r8:c000ff04 r7:00000036 r6:00066800 r5:b68fa000 r4:b540887c r3:00000000
>
> This is totally broken, and here's why.  Immediately before the above
> are these lines:
>
> [  114.120099] ipu1_smfc0: stream ON
> [  114.234338] imx6-mipi-csi2: stream ON
> [  114.258187] imx6-mipi-csi2: ready, dphy version 0x3130302a
> [  114.263767] imx6-mipi-csi2: stream ON
> [  114.267495] ipu1_csi0: stream ON
>
> At the "ipu1_smfc0" stream on message, smfc calls imx_smfc_start().
> imx_smfc_start() asks for the ring:
>
>          /* ask the sink for the buffer ring */
>          ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
>                                 IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
>                                 &priv->out_ring);
>
> camif provides the ring:
>
> static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
>          case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
>                  if (!priv->in_ring)
>                          return -EINVAL;
>                  ring = (struct imx_media_dma_buf_ring **)arg;
>                  *ring = priv->in_ring;
>                  break;
>
> So, smfc contains a copy of the pointer to camif's priv->in_ring.
>
> Things continue, and we get to camif_buf_prepare():
>
> static int camif_buf_prepare(struct vb2_buffer *vb)
> {
> ...
>          if (!priv->in_ring) {
>                  priv->in_ring = imx_media_alloc_dma_buf_ring(
>                          priv->md, &priv->src_sd->entity, &priv->sd.entity,
>                          sizeimage, vq->num_buffers, false);
>                  if (IS_ERR(priv->in_ring)) {
>                          v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
>                          ret = PTR_ERR(priv->in_ring);
>                          priv->in_ring = NULL;
>                          return ret;
>                  }
>          }
>
> Well, if we haven't setup priv->in_ring by now... is anything going to
> work?
>
> Then we do this:
>
>          ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
>          if (ret)
>                  goto free_ring;
>
>          return 0;
>
> free_ring:
>          imx_media_free_dma_buf_ring(priv->in_ring);
>          priv->in_ring = NULL;
>          return ret;
>
> and for whatever reason we end up falling out through free_ring.  This
> is VERY bad news, because it means that the ring which SMFC took a copy
> of is now freed beneath its feet.

Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
returned error, the ring should not have been freed, it should have only
returned the error. And further bad stuff happens from that point on.

But all of this is gone in version 4.

Steve

>
> It doesn't matter if you later reallocate it, it could very well end up
> with a different pointer from kmalloc().
>
> SMFC continues along unknowing that its priv->out_ring is now invalid, and
> it tries to use it as if it is still valid, leading to two things:
>
> 1. potentially stamping over memory that has been given to someone else
>     (possibly inodes, resulting in filesystem corruption should that
>      memory get written back to disk)
> 2. dereferencing pointers to other random memory leading to who-knows-what.
>
> and that is _very_ bad.
>
> This is way too serious a bug to justify any further testing.
>
> So here endeth my interest in this driver until a new set of patches
> appears for review. :p
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 18:26       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 18:26 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 05:22:46PM +0000, Russell King - ARM Linux wrote:
>> I thought, maybe, it's the IPU overwriting past the end of the buffer,
>> but I've added checks and that doesn't seem to have fired.  I also
>> wondered if it was some kind of use-after-free of the ring, so I made
>> imx_media_free_dma_buf_ring() memset the ring to 0x5a5a5a5a before
>> kfree()ing it... doesn't look like it's that either.  I'm going to
>> continue poking to see if I can figure out what's going on.
> I take that back... here's a use-after-free of that buffer, on the
> very first run:
>
> Alignment trap: not handling instruction e1921f9f at [<c00894c4>]
> Unhandled fault: alignment exception (0x001) at 0x5a5a5b5e
> pgd = c0004000
> [5a5a5b5e] *pgd=00000000
> Internal error: : 1 [#1] SMP ARM
> Modules linked in: imx_csi(C) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_spdif snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card imx_media(C) uvcvideo imx_mipi_csi2(C) imx_media_common(C) imx219 snd_soc_sgtl5000 snd_soc_imx_audmux caam video_multiplexer imx_sdma imx2_wdt coda v4l2_mem2mem videobuf2_v4l2 videobuf2_dma_contig videobuf2_core rc_cec snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_vmalloc videobuf2_memops imx_pcm_dma imx_thermal dw_hdmi_ahb_audio dw_hdmi_cec etnaviv fuse rc_pinnacle_pctv_hd
> CPU: 0 PID: 99 Comm: kworker/0:3 Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: lru-add-drain wq_barrier_func
> task: ee4e24c0 task.stack: ee6da000
> PC is at __lock_acquire+0xd4/0x17b0
> LR is at lock_acquire+0xd8/0x250
> pc : [<c00894c8>]    lr : [<c008b108>]    psr: 20070193
> sp : ee6dbb60  ip : 00000001  fp : ee6dbbe4
> r10: e9efad60  r9 : c0a70384  r8 : 00000000
> r7 : c0a38680  r6 : 00000000  r5 : ee4e24c0  r4 : c1400408
> r3 : 00000000  r2 : 5a5a5b5e  r1 : 00000000  r0 : 5a5a5a5a
> Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> Control: 10c5387d  Table: 3d7ec04a  DAC: 00000051
> Process kworker/0:3 (pid: 99, stack limit = 0xee6da210)
> Stack: (0xee6dbb60 to 0xee6dc000)
> bb60: c0a38680 00000002 c0b9d8c4 ee4e29a8 ee6dbc04 ee6dbb80 c0089708 c0088d44
> bb80: ee6dbb9c 0000050f c00867fc c0086728 ee6dbbf4 ee6dbba0 87eba239 c035aa2f
> bba0: 00000001 ee4e29a8 c00c4f84 00000001 00000026 0560e36b ffffffff 00000000
> bbc0: 60070193 e9efad60 00000000 00000000 c0a70384 00000000 ee6dbc3c ee6dbbe8
> bbe0: c008b108 c0089400 00000001 00000080 00000000 bf0d2a8c 00000000 00000000
> bc00: c008b108 c0089400 00000001 c09e04ec 00000000 e9efad50 60070193 bf0d2a8c
> bc20: 00000139 00000000 c09e04ec ee6dbcec ee6dbc6c ee6dbc40 c07016f4 c008b03c
> bc40: 00000001 00000000 bf0d2a8c ee6dbcec ee6dbc84 e9efac00 e9efad50 ee9785c4
> bc60: ee6dbc84 ee6dbc70 bf0d2a8c c07016b4 ee978410 e9efb400 ee6dbca4 ee6dbc88
> bc80: bf1224b8 bf0d2a7c bf122458 ee88d4c0 e9efb400 e9efb410 ee6dbce4 ee6dbca8
> bca0: c009f5dc bf122464 00000001 c09e04ec 00000000 e9efb400 c009f9f8 e9efb400
> bcc0: e9efb400 e9efb410 00000000 00000009 f4001100 ee6dbe38 ee6dbd04 ee6dbce8
> bce0: c009f984 c009f544 c0701d10 00000000 e9efb400 e9efb460 ee6dbd24 ee6dbd08
> bd00: c009fa00 c009f96c c09d0418 e9efb400 e9efb460 e9efb410 ee6dbd44 ee6dbd28
> bd20: c00a3174 c009f9cc c00a30c4 00000000 ee6dbd90 ee4a3010 ee6dbd54 ee6dbd48
> bd40: c009ecf0 c00a30d0 ee6dbd84 ee6dbd58 c0409328 c009ecdc c09d0448 00000001
> bd60: 00000026 ef1efc10 c09e756c ee4a3010 00000026 ef008400 ee6dbdcc ee6dbd88
> bd80: c0409458 c040928c 00000001 00000000 00000001 00000002 00000003 0000000a
> bda0: 0000000b 0000000c 0000000d 0000000e ee6dbdcc c09d52d0 00000000 00000000
> bdc0: ee6dbddc ee6dbdd0 c009ecf0 c0409408 ee6dbe04 ee6dbde0 c009ee24 c009ecdc
> bde0: ee6dbe38 f4000100 f400010c c09e0af0 000003eb c0a38a78 ee6dbe34 ee6dbe08
> be00: c00094c8 c009edd4 ee4e24c0 c0701d50 20070013 ffffffff ee6dbe6c ef7be600
> be20: ee6da000 c09f5dc6 ee6dbe9c ee6dbe38 c00149f0 c0009488 00000001 ee4e2988
> be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
> be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
> be80: 00000051 00000000 ddb9799c ddb97998 ee6dbebc ee6dbea0 c0083824 c0701d20
> bea0: c004e9c4 ee6e6d80 ddb97978 ef7ba940 ee6dbecc ee6dbec0 c004e9d8 c00837e8
> bec0: ee6dbf2c ee6dbed0 c0050958 c004e9d0 00000001 00000000 c0050898 00000000
> bee0: c0701d8c ee4e24c0 0000000f 00000000 c0a73e7c c0bc8834 00000000 c08947f8
> bf00: 00000008 ee6e6d80 ee6e6d98 ee6e6d98 00000008 ef7ba940 ef7ba940 c09dd900
> bf20: ee6dbf44 ee6dbf30 c0050e78 c0050774 ee6e6d80 ef7ba974 ee6dbf7c ee6dbf48
> bf40: c0051094 c0050e54 00000000 ee6e8ac0 ee509eb8 ee509e80 00000000 ee6e8ac0
> bf60: ee509eb8 ee6e6d80 ef0c9e58 c0050e88 ee6dbfac ee6dbf80 c0057b90 c0050e94
> bf80: ee6da000 ee6e8ac0 c0057a88 00000000 00000000 00000000 00000000 00000000
> bfa0: 00000000 ee6dbfb0 c000fdf0 c0057a94 00000000 00000000 00000000 00000000
> bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 3fffd861 3fffdc61
> Backtrace:
> [<c00893f4>] (__lock_acquire) from [<c008b108>] (lock_acquire+0xd8/0x250)
>   r10:00000000 r9:c0a70384 r8:00000000 r7:00000000 r6:e9efad60 r5:60070193
>   r4:00000000
> [<c008b030>] (lock_acquire) from [<c07016f4>] (_raw_spin_lock_irqsave+0x4c/0x60)
>   r10:ee6dbcec r9:c09e04ec r8:00000000 r7:00000139 r6:bf0d2a8c r5:60070193
>   r4:e9efad50
> [<c07016a8>] (_raw_spin_lock_irqsave) from [<bf0d2a8c>] (imx_media_dma_buf_get_active+0x1c/0x94 [imx_media_common])
>   r6:ee9785c4 r5:e9efad50 r4:e9efac00
> [<bf0d2a70>] (imx_media_dma_buf_get_active [imx_media_common]) from [<bf1224b8>] (imx_smfc_eof_interrupt+0x60/0x168 [imx_smfc])
>   r5:e9efb400 r4:ee978410
> [<bf122458>] (imx_smfc_eof_interrupt [imx_smfc]) from [<c009f5dc>] (__handle_irq_event_percpu+0xa4/0x428)
>   r6:e9efb410 r5:e9efb400 r4:ee88d4c0 r3:bf122458
> [<c009f538>] (__handle_irq_event_percpu) from [<c009f984>] (handle_irq_event_percpu+0x24/0x60)
>   r10:ee6dbe38 r9:f4001100 r8:00000009 r7:00000000 r6:e9efb410 r5:e9efb400
>   r4:e9efb400
> [<c009f960>] (handle_irq_event_percpu) from [<c009fa00>] (handle_irq_event+0x40/0x64)
>   r5:e9efb460 r4:e9efb400
> [<c009f9c0>] (handle_irq_event) from [<c00a3174>] (handle_level_irq+0xb0/0x138)
>   r6:e9efb410 r5:e9efb460 r4:e9efb400 r3:c09d0418
> [<c00a30c4>] (handle_level_irq) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:ee4a3010 r5:ee6dbd90 r4:00000000 r3:c00a30c4
> [<c009ecd0>] (generic_handle_irq) from [<c0409328>] (ipu_irq_handle+0xa8/0xd8)
> [<c0409280>] (ipu_irq_handle) from [<c0409458>] (ipu_irq_handler+0x5c/0xb4)
>   r8:ef008400 r7:00000026 r6:ee4a3010 r5:c09e756c r4:ef1efc10
> [<c04093fc>] (ipu_irq_handler) from [<c009ecf0>] (generic_handle_irq+0x20/0x30)
>   r6:00000000 r5:00000000 r4:c09d52d0
> [<c009ecd0>] (generic_handle_irq) from [<c009ee24>] (__handle_domain_irq+0x5c/0xb8)
> [<c009edc8>] (__handle_domain_irq) from [<c00094c8>] (gic_handle_irq+0x4c/0x9c)
>   r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c r4:f4000100 r3:ee6dbe38
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6dbe38 to 0xee6dbe80)
> be20:                                                       00000001 ee4e2988
> be40: 00000000 60070093 20070013 ddb9799c 20070013 ee6dbef0 ef7be600 c09e04ec
> be60: c09f5dc6 ee6dbe9c 00000288 ee6dbe88 c008b60c c0701d50 20070013 ffffffff
>   r10:c09f5dc6 r9:ee6da000 r8:ef7be600 r7:ee6dbe6c r6:ffffffff r5:20070013
>   r4:c0701d50 r3:ee4e24c0
> [<c0701d14>] (_raw_spin_unlock_irqrestore) from [<c0083824>] (complete+0x48/0x4c)
>   r5:ddb97998 r4:ddb9799c
> [<c00837dc>] (complete) from [<c004e9d8>] (wq_barrier_func+0x14/0x18)
>   r6:ef7ba940 r5:ddb97978 r4:ee6e6d80 r3:c004e9c4
> [<c004e9c4>] (wq_barrier_func) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
> [<c0050768>] (process_one_work) from [<c0050e78>] (process_scheduled_works+0x30/0x40)
>   r10:c09dd900 r9:ef7ba940 r8:ef7ba940 r7:00000008 r6:ee6e6d98 r5:ee6e6d98
>   r4:ee6e6d80
> [<c0050e48>] (process_scheduled_works) from [<c0051094>] (worker_thread+0x20c/0x4c8)
>   r5:ef7ba974 r4:ee6e6d80
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0c9e58 r8:ee6e6d80 r7:ee509eb8 r6:ee6e8ac0 r5:00000000
>   r4:ee509e80
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ee6e8ac0 r3:ee6da000
> Code: 0affffe9 e2802f41 f592f000 e1921f9f (e2811001)
> ---[ end trace 2e91a0629044cda4 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> CPU1: stopping
> CPU: 1 PID: 91 Comm: kworker/1:1 Tainted: G      D  C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Workqueue: events dbs_work_handler
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:60000193 r5:ffffffff r4:00000000 r3:ee4e6e40
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<c0016a68>] (handle_IPI+0x1b4/0x364)
>   r6:c0a70028 r5:00000001 r4:00000004 r3:ee4e6e40
> [<c00168b4>] (handle_IPI) from [<c000950c>] (gic_handle_irq+0x90/0x9c)
>   r10:ee6b7ba8 r9:f4001100 r8:c0a38a78 r7:000003eb r6:c09e0af0 r5:f400010c
>   r4:f4000100
> [<c000947c>] (gic_handle_irq) from [<c00149f0>] (__irq_svc+0x70/0x98)
> Exception stack(0xee6b7ba8 to 0xee6b7bf0)
> 7ba0:                   00000000 00000004 00000003 00000003 00000001 ee6b7d2c
> 7bc0: c00177e8 00000000 00000001 ee6b7d2c 00000000 ee6b7c24 c09e0af4 ee6b7bf8
> 7be0: c0360f2c c00cd3a0 00000013 ffffffff
>   r10:00000000 r9:ee6b6000 r8:00000001 r7:ee6b7bdc r6:ffffffff r5:00000013
>   r4:c00cd3a0 r3:ee4e6e40
> [<c00cd2a4>] (smp_call_function_single) from [<c00cd668>] (smp_call_function_many+0x270/0x2bc)
>   r7:c09e04ec r6:c09e04ec r5:00000001 r4:c09e05c8
> [<c00cd3f8>] (smp_call_function_many) from [<c00cd818>] (smp_call_function+0x30/0x38)
>   r10:00000002 r9:ffffffff r8:00000002 r7:ee6b7d2c r6:c00177e8 r5:00000000
>   r4:ffffffff
> [<c00cd7e8>] (smp_call_function) from [<c00cd860>] (on_each_cpu+0x18/0x58)
> [<c00cd848>] (on_each_cpu) from [<c0017890>] (twd_rate_change+0x2c/0x38)
>   r7:ee6b7d24 r6:00000000 r5:00000000 r4:ffffffff
> [<c0017864>] (twd_rate_change) from [<c00593a4>] (notifier_call_chain+0x4c/0x8c)
> [<c0059358>] (notifier_call_chain) from [<c00596b8>] (__srcu_notifier_call_chain+0x78/0xac)
>   r8:ee6b7d24 r7:00000000 r6:ef0069e4 r5:ef006948 r4:ef006904 r3:ffffffff
> [<c0059640>] (__srcu_notifier_call_chain) from [<c005970c>] (srcu_notifier_call_chain+0x20/0x28)
>   r10:ef024e00 r9:c09e04ec r8:c0a720bc r7:00000002 r6:ef02a080 r5:c0a39b40
>   r4:ef006900
> [<c00596ec>] (srcu_notifier_call_chain) from [<c0385788>] (__clk_notify+0x74/0x7c)
> [<c0385714>] (__clk_notify) from [<c0385860>] (__clk_recalc_rates+0xd0/0xe0)
>   r7:00000001 r6:179a7b00 r5:00000002 r4:ef02a080
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef02a080 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c0385818>] (__clk_recalc_rates+0x88/0xe0)
>   r6:2f34f600 r5:00000002 r4:ef033380 r3:179a7b00
> [<c0385790>] (__clk_recalc_rates) from [<c03892bc>] (clk_core_set_parent+0x1a8/0x410)
>   r6:ef02a400 r5:00000000 r4:ef02a480 r3:c0a38680
> [<c0389114>] (clk_core_set_parent) from [<c0389748>] (clk_set_parent+0x24/0x28)
>   r10:ee719c00 r9:001312d0 r8:3b5dc100 r7:000c15c0 r6:000f32a0 r5:00000002
>   r4:c141d574
> [<c0389724>] (clk_set_parent) from [<c052ab7c>] (imx6q_set_target+0x258/0x52c)
> [<c052a924>] (imx6q_set_target) from [<c0525360>] (__cpufreq_driver_target+0x150/0x528)
>   r10:000f32a0 r9:00000000 r8:00000001 r7:c141d424 r6:00000002 r5:00000000
>   r4:ee726000
> [<c0525210>] (__cpufreq_driver_target) from [<c0528e40>] (od_dbs_update+0xa0/0x168)
>   r10:eb522380 r9:c09e04ec r8:ebd79d80 r7:ee726000 r6:ebd79e40 r5:ebd79d80
>   r4:eb522380
> [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
>   r4:ee509900
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
>   r4:ee509a00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24[  165.007974] [<c0528da0>] (od_dbs_update) from [<c0529c24>] (dbs_work_handler+0x38/0x60)
>   r10:00000001 r8:c0a5b424 r7:ee726000 r6:ebd79e44 r5:00000000 r4:ebd79ea8
> [<c0529bec>] (dbs_work_handler) from [<c0050958>] (process_one_work+0x1f0/0x6e0)
>   r8:ef7ccc00 r7:ee6b7f08 r6:ef7c9940 r5:ebd79ea8 r4:ee509900 r3:c0529bec
> [<c0050768>] (process_one_work) from [<c0050eb8>] (worker_thread+0x30/0x4c8)
>   r10:c09dd900 r9:ef7c9940 r8:ef7c9940 r7:00000008 r6:ee509918 r5:ef7c9974
>   r4:ee509900
> [<c0050e88>] (worker_thread) from [<c0057b90>] (kthread+0x108/0x140)
>   r10:c0050e88 r9:ef0f3e58 r8:ee509900 r7:ee509a38 r6:ef0ff1c0 r5:00000000
>   r4:ee509a00
> [<c0057a88>] (kthread) from [<c000fdf0>] (ret_from_fork+0x14/0x24)
>   r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0057a88
>   r4:ef0ff1c0 r3:ee6b6000
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>
> This happens because (a little more debugging - notably a dump_stack()
> in imx_media_free_dma_buf_ring()):
>
> CPU: 0 PID: 2322 Comm: v4l2src0:src Tainted: G         C      4.10.0-rc6+ #2103
> Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> Backtrace:
> [<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
>   r6:a0010013 r5:ffffffff r4:00000000 r3:00000000
> [<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
> [<c0333444>] (dump_stack) from [<bf0dc0b4>] (imx_media_free_dma_buf_ring+0x18/0x74 [imx_media_common])
>   r6:bf059540 r5:ffffffea r4:ee40ce00 r3:00000004
> [<bf0dc09c>] (imx_media_free_dma_buf_ring [imx_media_common]) from [<bf1590dc>] (camif_buf_prepare+0x9c/0x130 [imx_camif])
>   r5:ffffffea r4:d008f010
> [<bf159040>] (camif_buf_prepare [imx_camif]) from [<bf054b1c>] (__buf_prepare+0x130/0x1dc [videobuf2_core])
>   r6:bf059540 r5:d004f800 r4:00000000
> [<bf0549ec>] (__buf_prepare [videobuf2_core]) from [<bf054c20>] (vb2_core_qbuf+0x58/0x324 [videobuf2_core])
>   r6:e74b1e20 r5:d004f800 r4:d008f6b0
> [<bf054bc8>] (vb2_core_qbuf [videobuf2_core]) from [<bf068028>] (vb2_qbuf+0x58/0x80 [videobuf2_v4l2])
>   r10:c0a57704 r9:e74b1e20 r8:e7417140 r7:bf15977c r6:e974c980 r5:e74b1e20
>   r4:d008f6b0 r3:00000000
> [<bf067fd0>] (vb2_qbuf [videobuf2_v4l2]) from [<bf068098>] (vb2_ioctl_qbuf+0x48/0x4c [videobuf2_v4l2])
>   r5:e74b1e20 r4:e7417140
> [<bf068050>] (vb2_ioctl_qbuf [videobuf2_v4l2]) from [<c04f746c>] (v4l_qbuf+0x44/0x48)
>   r5:e74b1e20 r4:e7417140
> [<c04f7428>] (v4l_qbuf) from [<c04f4660>] (__video_do_ioctl+0x270/0x304)
>   r7:00000000 r6:e974c980 r5:d008f018 r4:c044560f
> [<c04f43f0>] (__video_do_ioctl) from [<c04f6b48>] (video_usercopy+0x12c/0x85c)
>   r10:00000000 r9:c04f43f0 r8:00000000 r7:e74b1e20 r6:b5408888 r5:00000000
>   r4:c044560f
> [<c04f6a1c>] (video_usercopy) from [<c04f7290>] (video_ioctl2+0x18/0x1c)
>   r10:d024e568 r9:e74b0000 r8:d008f64c r7:c044560f r6:b5408888 r5:e7417140
>   r4:d008f018
> [<c04f7278>] (video_ioctl2) from [<c04f1524>] (v4l2_ioctl+0xa4/0xc4)
> [<c04f1480>] (v4l2_ioctl) from [<c0188b6c>] (do_vfs_ioctl+0x98/0x9a0)
>   r8:b5408888 r7:0000000d r6:0000000d r5:e7417140 r4:c01894b0 r3:c04f1480
> [<c0188ad4>] (do_vfs_ioctl) from [<c01894b0>] (SyS_ioctl+0x3c/0x60)
>   r10:00000000 r9:e74b0000 r8:b5408888 r7:0000000d r6:c044560f r5:e7417140
>   r4:e7417141
> [<c0189474>] (SyS_ioctl) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
>   r8:c000ff04 r7:00000036 r6:00066800 r5:b68fa000 r4:b540887c r3:00000000
>
> This is totally broken, and here's why.  Immediately before the above
> are these lines:
>
> [  114.120099] ipu1_smfc0: stream ON
> [  114.234338] imx6-mipi-csi2: stream ON
> [  114.258187] imx6-mipi-csi2: ready, dphy version 0x3130302a
> [  114.263767] imx6-mipi-csi2: stream ON
> [  114.267495] ipu1_csi0: stream ON
>
> At the "ipu1_smfc0" stream on message, smfc calls imx_smfc_start().
> imx_smfc_start() asks for the ring:
>
>          /* ask the sink for the buffer ring */
>          ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
>                                 IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
>                                 &priv->out_ring);
>
> camif provides the ring:
>
> static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
>          case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
>                  if (!priv->in_ring)
>                          return -EINVAL;
>                  ring = (struct imx_media_dma_buf_ring **)arg;
>                  *ring = priv->in_ring;
>                  break;
>
> So, smfc contains a copy of the pointer to camif's priv->in_ring.
>
> Things continue, and we get to camif_buf_prepare():
>
> static int camif_buf_prepare(struct vb2_buffer *vb)
> {
> ...
>          if (!priv->in_ring) {
>                  priv->in_ring = imx_media_alloc_dma_buf_ring(
>                          priv->md, &priv->src_sd->entity, &priv->sd.entity,
>                          sizeimage, vq->num_buffers, false);
>                  if (IS_ERR(priv->in_ring)) {
>                          v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
>                          ret = PTR_ERR(priv->in_ring);
>                          priv->in_ring = NULL;
>                          return ret;
>                  }
>          }
>
> Well, if we haven't setup priv->in_ring by now... is anything going to
> work?
>
> Then we do this:
>
>          ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
>          if (ret)
>                  goto free_ring;
>
>          return 0;
>
> free_ring:
>          imx_media_free_dma_buf_ring(priv->in_ring);
>          priv->in_ring = NULL;
>          return ret;
>
> and for whatever reason we end up falling out through free_ring.  This
> is VERY bad news, because it means that the ring which SMFC took a copy
> of is now freed beneath its feet.

Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
returned error, the ring should not have been freed, it should have only
returned the error. And further bad stuff happens from that point on.

But all of this is gone in version 4.

Steve

>
> It doesn't matter if you later reallocate it, it could very well end up
> with a different pointer from kmalloc().
>
> SMFC continues along unknowing that its priv->out_ring is now invalid, and
> it tries to use it as if it is still valid, leading to two things:
>
> 1. potentially stamping over memory that has been given to someone else
>     (possibly inodes, resulting in filesystem corruption should that
>      memory get written back to disk)
> 2. dereferencing pointers to other random memory leading to who-knows-what.
>
> and that is _very_ bad.
>
> This is way too serious a bug to justify any further testing.
>
> So here endeth my interest in this driver until a new set of patches
> appears for review. :p
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-02 18:26       ` Steve Longerbeam
  (?)
@ 2017-02-02 18:58         ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 18:58 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Thu, Feb 02, 2017 at 10:26:55AM -0800, Steve Longerbeam wrote:
> On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
> >and for whatever reason we end up falling out through free_ring.  This
> >is VERY bad news, because it means that the ring which SMFC took a copy
> >of is now freed beneath its feet.
> 
> Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
> returned error, the ring should not have been freed, it should have only
> returned the error. And further bad stuff happens from that point on.
> 
> But all of this is gone in version 4.

I think there's an error in how you think the queue_setup() works.

camif_queue_setup() always returns the number of buffers between
IMX_MEDIA_MIN_RING_BUFS and IMX_MEDIA_MAX_RING_BUFS.  However, it seems
that, looking through the videobuf2-core.c code, that the value is
passed to __vb2_queue_alloc() to allocate the specified number of
_additional_ buffers over and on-top of the existing q->num_buffers:

static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
                             unsigned int num_buffers, unsigned int num_planes,
                             const unsigned plane_sizes[VB2_MAX_PLANES])
{
        for (buffer = 0; buffer < num_buffers; ++buffer) {
...
                vb->index = q->num_buffers + buffer;

and

int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
                unsigned int *count)
{
        unsigned int num_buffers, allocated_buffers, num_planes = 0;
...
        num_buffers = min_t(unsigned int, *count, VB2_MAX_FRAME);
        num_buffers = max_t(unsigned int, num_buffers, q->min_buffers_needed);
...
        /*
         * Ask the driver how many buffers and planes per buffer it requires.
         * Driver also sets the size and allocator context for each plane.
         */
        ret = call_qop(q, queue_setup, q, &num_buffers, &num_planes,
                       plane_sizes, q->alloc_devs);
        if (ret)
                return ret;

        /* Finally, allocate buffers and video memory */
        allocated_buffers =
                __vb2_queue_alloc(q, memory, num_buffers, num_planes, plane_sizes);

or:

int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
                unsigned int *count, unsigned requested_planes,
                const unsigned requested_sizes[])
{
        unsigned int num_planes = 0, num_buffers, allocated_buffers;
...
        num_buffers = min(*count, VB2_MAX_FRAME - q->num_buffers);
        if (requested_planes && requested_sizes) {
                num_planes = requested_planes;
...
        /*
         * Ask the driver, whether the requested number of buffers, planes per
         * buffer and their sizes are acceptable
         */
        ret = call_qop(q, queue_setup, q, &num_buffers,
                       &num_planes, plane_sizes, q->alloc_devs);
        if (ret)
                return ret;

        /* Finally, allocate buffers and video memory */
        allocated_buffers = __vb2_queue_alloc(q, memory, num_buffers,
                                num_planes, plane_sizes);


It seems to me that if you don't take account of the existing queue
size, your camif_queue_setup() has the side effect that each time
either of these are called.  Hence, the vb2 queue increases by the
same amount each time, which is probably what you don't want.

The documentation on queue_setup() leaves much to be desired:

 * @queue_setup:        called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
 *                      handlers before memory allocation. It can be called
 *                      twice: if the original number of requested buffers
 *                      could not be allocated, then it will be called a
 *                      second time with the actually allocated number of
 *                      buffers to verify if that is OK.
 *                      The driver should return the required number of buffers
 *                      in \*num_buffers, the required number of planes per
 *                      buffer in \*num_planes, the size of each plane should be
 *                      set in the sizes\[\] array and optional per-plane
 *                      allocator specific device in the alloc_devs\[\] array.
 *                      When called from VIDIOC_REQBUFS,() \*num_planes == 0,
 *                      the driver has to use the currently configured format to
 *                      determine the plane sizes and \*num_buffers is the total
 *                      number of buffers that are being allocated. When called
 *                      from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
 *                      describes the requested number of planes and sizes\[\]
 *                      contains the requested plane sizes. If either
 *                      \*num_planes or the requested sizes are invalid callback
 *                      must return %-EINVAL. In this case \*num_buffers are
 *                      being allocated additionally to q->num_buffers.

That's really really ambiguous, because the "In this case" part doesn't
really tell you which case it's talking about - but it seems to me looking
at the code that it's referring to the VIDIOC_CREATE_BUFS case.

As you support both .vidioc_create_bufs and .vidioc_reqbufs, it seems
to me that you're not handling the VIDIOC_CREATE_BUFS case correctly.

Can you please make sure that your next version resolves that?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 18:58         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 18:58 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Thu, Feb 02, 2017 at 10:26:55AM -0800, Steve Longerbeam wrote:
> On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
> >and for whatever reason we end up falling out through free_ring.  This
> >is VERY bad news, because it means that the ring which SMFC took a copy
> >of is now freed beneath its feet.
> 
> Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
> returned error, the ring should not have been freed, it should have only
> returned the error. And further bad stuff happens from that point on.
> 
> But all of this is gone in version 4.

I think there's an error in how you think the queue_setup() works.

camif_queue_setup() always returns the number of buffers between
IMX_MEDIA_MIN_RING_BUFS and IMX_MEDIA_MAX_RING_BUFS.  However, it seems
that, looking through the videobuf2-core.c code, that the value is
passed to __vb2_queue_alloc() to allocate the specified number of
_additional_ buffers over and on-top of the existing q->num_buffers:

static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
                             unsigned int num_buffers, unsigned int num_planes,
                             const unsigned plane_sizes[VB2_MAX_PLANES])
{
        for (buffer = 0; buffer < num_buffers; ++buffer) {
...
                vb->index = q->num_buffers + buffer;

and

int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
                unsigned int *count)
{
        unsigned int num_buffers, allocated_buffers, num_planes = 0;
...
        num_buffers = min_t(unsigned int, *count, VB2_MAX_FRAME);
        num_buffers = max_t(unsigned int, num_buffers, q->min_buffers_needed);
...
        /*
         * Ask the driver how many buffers and planes per buffer it requires.
         * Driver also sets the size and allocator context for each plane.
         */
        ret = call_qop(q, queue_setup, q, &num_buffers, &num_planes,
                       plane_sizes, q->alloc_devs);
        if (ret)
                return ret;

        /* Finally, allocate buffers and video memory */
        allocated_buffers =
                __vb2_queue_alloc(q, memory, num_buffers, num_planes, plane_sizes);

or:

int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
                unsigned int *count, unsigned requested_planes,
                const unsigned requested_sizes[])
{
        unsigned int num_planes = 0, num_buffers, allocated_buffers;
...
        num_buffers = min(*count, VB2_MAX_FRAME - q->num_buffers);
        if (requested_planes && requested_sizes) {
                num_planes = requested_planes;
...
        /*
         * Ask the driver, whether the requested number of buffers, planes per
         * buffer and their sizes are acceptable
         */
        ret = call_qop(q, queue_setup, q, &num_buffers,
                       &num_planes, plane_sizes, q->alloc_devs);
        if (ret)
                return ret;

        /* Finally, allocate buffers and video memory */
        allocated_buffers = __vb2_queue_alloc(q, memory, num_buffers,
                                num_planes, plane_sizes);


It seems to me that if you don't take account of the existing queue
size, your camif_queue_setup() has the side effect that each time
either of these are called.  Hence, the vb2 queue increases by the
same amount each time, which is probably what you don't want.

The documentation on queue_setup() leaves much to be desired:

 * @queue_setup:        called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
 *                      handlers before memory allocation. It can be called
 *                      twice: if the original number of requested buffers
 *                      could not be allocated, then it will be called a
 *                      second time with the actually allocated number of
 *                      buffers to verify if that is OK.
 *                      The driver should return the required number of buffers
 *                      in \*num_buffers, the required number of planes per
 *                      buffer in \*num_planes, the size of each plane should be
 *                      set in the sizes\[\] array and optional per-plane
 *                      allocator specific device in the alloc_devs\[\] array.
 *                      When called from VIDIOC_REQBUFS,() \*num_planes == 0,
 *                      the driver has to use the currently configured format to
 *                      determine the plane sizes and \*num_buffers is the total
 *                      number of buffers that are being allocated. When called
 *                      from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
 *                      describes the requested number of planes and sizes\[\]
 *                      contains the requested plane sizes. If either
 *                      \*num_planes or the requested sizes are invalid callback
 *                      must return %-EINVAL. In this case \*num_buffers are
 *                      being allocated additionally to q->num_buffers.

That's really really ambiguous, because the "In this case" part doesn't
really tell you which case it's talking about - but it seems to me looking
at the code that it's referring to the VIDIOC_CREATE_BUFS case.

As you support both .vidioc_create_bufs and .vidioc_reqbufs, it seems
to me that you're not handling the VIDIOC_CREATE_BUFS case correctly.

Can you please make sure that your next version resolves that?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 18:58         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 18:58 UTC (permalink / raw)
  To: linux-arm-kernel

On Thu, Feb 02, 2017 at 10:26:55AM -0800, Steve Longerbeam wrote:
> On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
> >and for whatever reason we end up falling out through free_ring.  This
> >is VERY bad news, because it means that the ring which SMFC took a copy
> >of is now freed beneath its feet.
> 
> Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
> returned error, the ring should not have been freed, it should have only
> returned the error. And further bad stuff happens from that point on.
> 
> But all of this is gone in version 4.

I think there's an error in how you think the queue_setup() works.

camif_queue_setup() always returns the number of buffers between
IMX_MEDIA_MIN_RING_BUFS and IMX_MEDIA_MAX_RING_BUFS.  However, it seems
that, looking through the videobuf2-core.c code, that the value is
passed to __vb2_queue_alloc() to allocate the specified number of
_additional_ buffers over and on-top of the existing q->num_buffers:

static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
                             unsigned int num_buffers, unsigned int num_planes,
                             const unsigned plane_sizes[VB2_MAX_PLANES])
{
        for (buffer = 0; buffer < num_buffers; ++buffer) {
...
                vb->index = q->num_buffers + buffer;

and

int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
                unsigned int *count)
{
        unsigned int num_buffers, allocated_buffers, num_planes = 0;
...
        num_buffers = min_t(unsigned int, *count, VB2_MAX_FRAME);
        num_buffers = max_t(unsigned int, num_buffers, q->min_buffers_needed);
...
        /*
         * Ask the driver how many buffers and planes per buffer it requires.
         * Driver also sets the size and allocator context for each plane.
         */
        ret = call_qop(q, queue_setup, q, &num_buffers, &num_planes,
                       plane_sizes, q->alloc_devs);
        if (ret)
                return ret;

        /* Finally, allocate buffers and video memory */
        allocated_buffers =
                __vb2_queue_alloc(q, memory, num_buffers, num_planes, plane_sizes);

or:

int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
                unsigned int *count, unsigned requested_planes,
                const unsigned requested_sizes[])
{
        unsigned int num_planes = 0, num_buffers, allocated_buffers;
...
        num_buffers = min(*count, VB2_MAX_FRAME - q->num_buffers);
        if (requested_planes && requested_sizes) {
                num_planes = requested_planes;
...
        /*
         * Ask the driver, whether the requested number of buffers, planes per
         * buffer and their sizes are acceptable
         */
        ret = call_qop(q, queue_setup, q, &num_buffers,
                       &num_planes, plane_sizes, q->alloc_devs);
        if (ret)
                return ret;

        /* Finally, allocate buffers and video memory */
        allocated_buffers = __vb2_queue_alloc(q, memory, num_buffers,
                                num_planes, plane_sizes);


It seems to me that if you don't take account of the existing queue
size, your camif_queue_setup() has the side effect that each time
either of these are called.  Hence, the vb2 queue increases by the
same amount each time, which is probably what you don't want.

The documentation on queue_setup() leaves much to be desired:

 * @queue_setup:        called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
 *                      handlers before memory allocation. It can be called
 *                      twice: if the original number of requested buffers
 *                      could not be allocated, then it will be called a
 *                      second time with the actually allocated number of
 *                      buffers to verify if that is OK.
 *                      The driver should return the required number of buffers
 *                      in \*num_buffers, the required number of planes per
 *                      buffer in \*num_planes, the size of each plane should be
 *                      set in the sizes\[\] array and optional per-plane
 *                      allocator specific device in the alloc_devs\[\] array.
 *                      When called from VIDIOC_REQBUFS,() \*num_planes == 0,
 *                      the driver has to use the currently configured format to
 *                      determine the plane sizes and \*num_buffers is the total
 *                      number of buffers that are being allocated. When called
 *                      from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
 *                      describes the requested number of planes and sizes\[\]
 *                      contains the requested plane sizes. If either
 *                      \*num_planes or the requested sizes are invalid callback
 *                      must return %-EINVAL. In this case \*num_buffers are
 *                      being allocated additionally to q->num_buffers.

That's really really ambiguous, because the "In this case" part doesn't
really tell you which case it's talking about - but it seems to me looking
at the code that it's referring to the VIDIOC_CREATE_BUFS case.

As you support both .vidioc_create_bufs and .vidioc_reqbufs, it seems
to me that you're not handling the VIDIOC_CREATE_BUFS case correctly.

Can you please make sure that your next version resolves that?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-02 18:58         ` Russell King - ARM Linux
  (?)
@ 2017-02-02 19:12           ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 19:12 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/02/2017 10:58 AM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 10:26:55AM -0800, Steve Longerbeam wrote:
>> On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
>>> and for whatever reason we end up falling out through free_ring.  This
>>> is VERY bad news, because it means that the ring which SMFC took a copy
>>> of is now freed beneath its feet.
>> Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
>> returned error, the ring should not have been freed, it should have only
>> returned the error. And further bad stuff happens from that point on.
>>
>> But all of this is gone in version 4.
> I think there's an error in how you think the queue_setup() works.
>
> camif_queue_setup() always returns the number of buffers between
> IMX_MEDIA_MIN_RING_BUFS and IMX_MEDIA_MAX_RING_BUFS.  However, it seems
> that, looking through the videobuf2-core.c code, that the value is
> passed to __vb2_queue_alloc() to allocate the specified number of
> _additional_ buffers over and on-top of the existing q->num_buffers:
>
> static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>                               unsigned int num_buffers, unsigned int num_planes,
>                               const unsigned plane_sizes[VB2_MAX_PLANES])
> {
>          for (buffer = 0; buffer < num_buffers; ++buffer) {
> ...
>                  vb->index = q->num_buffers + buffer;
>
> and
>
> int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
>                  unsigned int *count)
> {
>          unsigned int num_buffers, allocated_buffers, num_planes = 0;
> ...
>          num_buffers = min_t(unsigned int, *count, VB2_MAX_FRAME);
>          num_buffers = max_t(unsigned int, num_buffers, q->min_buffers_needed);
> ...
>          /*
>           * Ask the driver how many buffers and planes per buffer it requires.
>           * Driver also sets the size and allocator context for each plane.
>           */
>          ret = call_qop(q, queue_setup, q, &num_buffers, &num_planes,
>                         plane_sizes, q->alloc_devs);
>          if (ret)
>                  return ret;
>
>          /* Finally, allocate buffers and video memory */
>          allocated_buffers =
>                  __vb2_queue_alloc(q, memory, num_buffers, num_planes, plane_sizes);
>
> or:
>
> int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
>                  unsigned int *count, unsigned requested_planes,
>                  const unsigned requested_sizes[])
> {
>          unsigned int num_planes = 0, num_buffers, allocated_buffers;
> ...
>          num_buffers = min(*count, VB2_MAX_FRAME - q->num_buffers);
>          if (requested_planes && requested_sizes) {
>                  num_planes = requested_planes;
> ...
>          /*
>           * Ask the driver, whether the requested number of buffers, planes per
>           * buffer and their sizes are acceptable
>           */
>          ret = call_qop(q, queue_setup, q, &num_buffers,
>                         &num_planes, plane_sizes, q->alloc_devs);
>          if (ret)
>                  return ret;
>
>          /* Finally, allocate buffers and video memory */
>          allocated_buffers = __vb2_queue_alloc(q, memory, num_buffers,
>                                  num_planes, plane_sizes);
>
>
> It seems to me that if you don't take account of the existing queue
> size, your camif_queue_setup() has the side effect that each time
> either of these are called.  Hence, the vb2 queue increases by the
> same amount each time, which is probably what you don't want.
>
> The documentation on queue_setup() leaves much to be desired:
>
>   * @queue_setup:        called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
>   *                      handlers before memory allocation. It can be called
>   *                      twice: if the original number of requested buffers
>   *                      could not be allocated, then it will be called a
>   *                      second time with the actually allocated number of
>   *                      buffers to verify if that is OK.
>   *                      The driver should return the required number of buffers
>   *                      in \*num_buffers, the required number of planes per
>   *                      buffer in \*num_planes, the size of each plane should be
>   *                      set in the sizes\[\] array and optional per-plane
>   *                      allocator specific device in the alloc_devs\[\] array.
>   *                      When called from VIDIOC_REQBUFS,() \*num_planes == 0,
>   *                      the driver has to use the currently configured format to
>   *                      determine the plane sizes and \*num_buffers is the total
>   *                      number of buffers that are being allocated. When called
>   *                      from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
>   *                      describes the requested number of planes and sizes\[\]
>   *                      contains the requested plane sizes. If either
>   *                      \*num_planes or the requested sizes are invalid callback
>   *                      must return %-EINVAL. In this case \*num_buffers are
>   *                      being allocated additionally to q->num_buffers.
>
> That's really really ambiguous, because the "In this case" part doesn't
> really tell you which case it's talking about - but it seems to me looking
> at the code that it's referring to the VIDIOC_CREATE_BUFS case.

Yes, I caught this when adding fixes from v4l2-compliance testing, which
is not part of the version 3 driver. I agree it is a confusing API. When
called from VIDIOC_CREATE_BUFS (indicated by *num_planes != 0),
*num_buffers is supposed to be requested buffers _in addition_ to
already requested q->num_buffers, which is important info and
should be emphasized a little more than the "oh by the way" fashion
in the prototype description, IMHO.

>
> As you support both .vidioc_create_bufs and .vidioc_reqbufs, it seems
> to me that you're not handling the VIDIOC_CREATE_BUFS case correctly.
>
> Can you please make sure that your next version resolves that?

Here is the current .queue_setup() op in imx-media-capture.c:

static int capture_queue_setup(struct vb2_queue *vq,
                                unsigned int *nbuffers,
                                unsigned int *nplanes,
                                unsigned int sizes[],
                                struct device *alloc_devs[])
{
         struct capture_priv *priv = vb2_get_drv_priv(vq);
         struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
         unsigned int count = *nbuffers;

         if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
                 return -EINVAL;

         if (*nplanes) {
                 if (*nplanes != 1 || sizes[0] < pix->sizeimage)
                         return -EINVAL;
                 count += vq->num_buffers;
         }

         while (pix->sizeimage * count > VID_MEM_LIMIT)
                 count--;

         if (*nplanes)
                 *nbuffers = (count < vq->num_buffers) ? 0 :
                         count - vq->num_buffers;
         else
                 *nbuffers = count;

         *nplanes = 1;
         sizes[0] = pix->sizeimage;

         return 0;
}

>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 19:12           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 19:12 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/02/2017 10:58 AM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 10:26:55AM -0800, Steve Longerbeam wrote:
>> On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
>>> and for whatever reason we end up falling out through free_ring.  This
>>> is VERY bad news, because it means that the ring which SMFC took a copy
>>> of is now freed beneath its feet.
>> Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
>> returned error, the ring should not have been freed, it should have only
>> returned the error. And further bad stuff happens from that point on.
>>
>> But all of this is gone in version 4.
> I think there's an error in how you think the queue_setup() works.
>
> camif_queue_setup() always returns the number of buffers between
> IMX_MEDIA_MIN_RING_BUFS and IMX_MEDIA_MAX_RING_BUFS.  However, it seems
> that, looking through the videobuf2-core.c code, that the value is
> passed to __vb2_queue_alloc() to allocate the specified number of
> _additional_ buffers over and on-top of the existing q->num_buffers:
>
> static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>                               unsigned int num_buffers, unsigned int num_planes,
>                               const unsigned plane_sizes[VB2_MAX_PLANES])
> {
>          for (buffer = 0; buffer < num_buffers; ++buffer) {
> ...
>                  vb->index = q->num_buffers + buffer;
>
> and
>
> int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
>                  unsigned int *count)
> {
>          unsigned int num_buffers, allocated_buffers, num_planes = 0;
> ...
>          num_buffers = min_t(unsigned int, *count, VB2_MAX_FRAME);
>          num_buffers = max_t(unsigned int, num_buffers, q->min_buffers_needed);
> ...
>          /*
>           * Ask the driver how many buffers and planes per buffer it requires.
>           * Driver also sets the size and allocator context for each plane.
>           */
>          ret = call_qop(q, queue_setup, q, &num_buffers, &num_planes,
>                         plane_sizes, q->alloc_devs);
>          if (ret)
>                  return ret;
>
>          /* Finally, allocate buffers and video memory */
>          allocated_buffers =
>                  __vb2_queue_alloc(q, memory, num_buffers, num_planes, plane_sizes);
>
> or:
>
> int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
>                  unsigned int *count, unsigned requested_planes,
>                  const unsigned requested_sizes[])
> {
>          unsigned int num_planes = 0, num_buffers, allocated_buffers;
> ...
>          num_buffers = min(*count, VB2_MAX_FRAME - q->num_buffers);
>          if (requested_planes && requested_sizes) {
>                  num_planes = requested_planes;
> ...
>          /*
>           * Ask the driver, whether the requested number of buffers, planes per
>           * buffer and their sizes are acceptable
>           */
>          ret = call_qop(q, queue_setup, q, &num_buffers,
>                         &num_planes, plane_sizes, q->alloc_devs);
>          if (ret)
>                  return ret;
>
>          /* Finally, allocate buffers and video memory */
>          allocated_buffers = __vb2_queue_alloc(q, memory, num_buffers,
>                                  num_planes, plane_sizes);
>
>
> It seems to me that if you don't take account of the existing queue
> size, your camif_queue_setup() has the side effect that each time
> either of these are called.  Hence, the vb2 queue increases by the
> same amount each time, which is probably what you don't want.
>
> The documentation on queue_setup() leaves much to be desired:
>
>   * @queue_setup:        called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
>   *                      handlers before memory allocation. It can be called
>   *                      twice: if the original number of requested buffers
>   *                      could not be allocated, then it will be called a
>   *                      second time with the actually allocated number of
>   *                      buffers to verify if that is OK.
>   *                      The driver should return the required number of buffers
>   *                      in \*num_buffers, the required number of planes per
>   *                      buffer in \*num_planes, the size of each plane should be
>   *                      set in the sizes\[\] array and optional per-plane
>   *                      allocator specific device in the alloc_devs\[\] array.
>   *                      When called from VIDIOC_REQBUFS,() \*num_planes == 0,
>   *                      the driver has to use the currently configured format to
>   *                      determine the plane sizes and \*num_buffers is the total
>   *                      number of buffers that are being allocated. When called
>   *                      from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
>   *                      describes the requested number of planes and sizes\[\]
>   *                      contains the requested plane sizes. If either
>   *                      \*num_planes or the requested sizes are invalid callback
>   *                      must return %-EINVAL. In this case \*num_buffers are
>   *                      being allocated additionally to q->num_buffers.
>
> That's really really ambiguous, because the "In this case" part doesn't
> really tell you which case it's talking about - but it seems to me looking
> at the code that it's referring to the VIDIOC_CREATE_BUFS case.

Yes, I caught this when adding fixes from v4l2-compliance testing, which
is not part of the version 3 driver. I agree it is a confusing API. When
called from VIDIOC_CREATE_BUFS (indicated by *num_planes != 0),
*num_buffers is supposed to be requested buffers _in addition_ to
already requested q->num_buffers, which is important info and
should be emphasized a little more than the "oh by the way" fashion
in the prototype description, IMHO.

>
> As you support both .vidioc_create_bufs and .vidioc_reqbufs, it seems
> to me that you're not handling the VIDIOC_CREATE_BUFS case correctly.
>
> Can you please make sure that your next version resolves that?

Here is the current .queue_setup() op in imx-media-capture.c:

static int capture_queue_setup(struct vb2_queue *vq,
                                unsigned int *nbuffers,
                                unsigned int *nplanes,
                                unsigned int sizes[],
                                struct device *alloc_devs[])
{
         struct capture_priv *priv = vb2_get_drv_priv(vq);
         struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
         unsigned int count = *nbuffers;

         if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
                 return -EINVAL;

         if (*nplanes) {
                 if (*nplanes != 1 || sizes[0] < pix->sizeimage)
                         return -EINVAL;
                 count += vq->num_buffers;
         }

         while (pix->sizeimage * count > VID_MEM_LIMIT)
                 count--;

         if (*nplanes)
                 *nbuffers = (count < vq->num_buffers) ? 0 :
                         count - vq->num_buffers;
         else
                 *nbuffers = count;

         *nplanes = 1;
         sizes[0] = pix->sizeimage;

         return 0;
}

>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 19:12           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-02 19:12 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/02/2017 10:58 AM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 10:26:55AM -0800, Steve Longerbeam wrote:
>> On 02/02/2017 09:56 AM, Russell King - ARM Linux wrote:
>>> and for whatever reason we end up falling out through free_ring.  This
>>> is VERY bad news, because it means that the ring which SMFC took a copy
>>> of is now freed beneath its feet.
>> Yes, that is bad. That was a bug, if imx_media_dma_buf_queue_from_vb()
>> returned error, the ring should not have been freed, it should have only
>> returned the error. And further bad stuff happens from that point on.
>>
>> But all of this is gone in version 4.
> I think there's an error in how you think the queue_setup() works.
>
> camif_queue_setup() always returns the number of buffers between
> IMX_MEDIA_MIN_RING_BUFS and IMX_MEDIA_MAX_RING_BUFS.  However, it seems
> that, looking through the videobuf2-core.c code, that the value is
> passed to __vb2_queue_alloc() to allocate the specified number of
> _additional_ buffers over and on-top of the existing q->num_buffers:
>
> static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>                               unsigned int num_buffers, unsigned int num_planes,
>                               const unsigned plane_sizes[VB2_MAX_PLANES])
> {
>          for (buffer = 0; buffer < num_buffers; ++buffer) {
> ...
>                  vb->index = q->num_buffers + buffer;
>
> and
>
> int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
>                  unsigned int *count)
> {
>          unsigned int num_buffers, allocated_buffers, num_planes = 0;
> ...
>          num_buffers = min_t(unsigned int, *count, VB2_MAX_FRAME);
>          num_buffers = max_t(unsigned int, num_buffers, q->min_buffers_needed);
> ...
>          /*
>           * Ask the driver how many buffers and planes per buffer it requires.
>           * Driver also sets the size and allocator context for each plane.
>           */
>          ret = call_qop(q, queue_setup, q, &num_buffers, &num_planes,
>                         plane_sizes, q->alloc_devs);
>          if (ret)
>                  return ret;
>
>          /* Finally, allocate buffers and video memory */
>          allocated_buffers =
>                  __vb2_queue_alloc(q, memory, num_buffers, num_planes, plane_sizes);
>
> or:
>
> int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
>                  unsigned int *count, unsigned requested_planes,
>                  const unsigned requested_sizes[])
> {
>          unsigned int num_planes = 0, num_buffers, allocated_buffers;
> ...
>          num_buffers = min(*count, VB2_MAX_FRAME - q->num_buffers);
>          if (requested_planes && requested_sizes) {
>                  num_planes = requested_planes;
> ...
>          /*
>           * Ask the driver, whether the requested number of buffers, planes per
>           * buffer and their sizes are acceptable
>           */
>          ret = call_qop(q, queue_setup, q, &num_buffers,
>                         &num_planes, plane_sizes, q->alloc_devs);
>          if (ret)
>                  return ret;
>
>          /* Finally, allocate buffers and video memory */
>          allocated_buffers = __vb2_queue_alloc(q, memory, num_buffers,
>                                  num_planes, plane_sizes);
>
>
> It seems to me that if you don't take account of the existing queue
> size, your camif_queue_setup() has the side effect that each time
> either of these are called.  Hence, the vb2 queue increases by the
> same amount each time, which is probably what you don't want.
>
> The documentation on queue_setup() leaves much to be desired:
>
>   * @queue_setup:        called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
>   *                      handlers before memory allocation. It can be called
>   *                      twice: if the original number of requested buffers
>   *                      could not be allocated, then it will be called a
>   *                      second time with the actually allocated number of
>   *                      buffers to verify if that is OK.
>   *                      The driver should return the required number of buffers
>   *                      in \*num_buffers, the required number of planes per
>   *                      buffer in \*num_planes, the size of each plane should be
>   *                      set in the sizes\[\] array and optional per-plane
>   *                      allocator specific device in the alloc_devs\[\] array.
>   *                      When called from VIDIOC_REQBUFS,() \*num_planes == 0,
>   *                      the driver has to use the currently configured format to
>   *                      determine the plane sizes and \*num_buffers is the total
>   *                      number of buffers that are being allocated. When called
>   *                      from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
>   *                      describes the requested number of planes and sizes\[\]
>   *                      contains the requested plane sizes. If either
>   *                      \*num_planes or the requested sizes are invalid callback
>   *                      must return %-EINVAL. In this case \*num_buffers are
>   *                      being allocated additionally to q->num_buffers.
>
> That's really really ambiguous, because the "In this case" part doesn't
> really tell you which case it's talking about - but it seems to me looking
> at the code that it's referring to the VIDIOC_CREATE_BUFS case.

Yes, I caught this when adding fixes from v4l2-compliance testing, which
is not part of the version 3 driver. I agree it is a confusing API. When
called from VIDIOC_CREATE_BUFS (indicated by *num_planes != 0),
*num_buffers is supposed to be requested buffers _in addition_ to
already requested q->num_buffers, which is important info and
should be emphasized a little more than the "oh by the way" fashion
in the prototype description, IMHO.

>
> As you support both .vidioc_create_bufs and .vidioc_reqbufs, it seems
> to me that you're not handling the VIDIOC_CREATE_BUFS case correctly.
>
> Can you please make sure that your next version resolves that?

Here is the current .queue_setup() op in imx-media-capture.c:

static int capture_queue_setup(struct vb2_queue *vq,
                                unsigned int *nbuffers,
                                unsigned int *nplanes,
                                unsigned int sizes[],
                                struct device *alloc_devs[])
{
         struct capture_priv *priv = vb2_get_drv_priv(vq);
         struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
         unsigned int count = *nbuffers;

         if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
                 return -EINVAL;

         if (*nplanes) {
                 if (*nplanes != 1 || sizes[0] < pix->sizeimage)
                         return -EINVAL;
                 count += vq->num_buffers;
         }

         while (pix->sizeimage * count > VID_MEM_LIMIT)
                 count--;

         if (*nplanes)
                 *nbuffers = (count < vq->num_buffers) ? 0 :
                         count - vq->num_buffers;
         else
                 *nbuffers = count;

         *nplanes = 1;
         sizes[0] = pix->sizeimage;

         return 0;
}

>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-02 19:12           ` Steve Longerbeam
  (?)
@ 2017-02-02 22:29             ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Thu, Feb 02, 2017 at 11:12:41AM -0800, Steve Longerbeam wrote:
> Here is the current .queue_setup() op in imx-media-capture.c:
> 
> static int capture_queue_setup(struct vb2_queue *vq,
>                                unsigned int *nbuffers,
>                                unsigned int *nplanes,
>                                unsigned int sizes[],
>                                struct device *alloc_devs[])
> {
>         struct capture_priv *priv = vb2_get_drv_priv(vq);
>         struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
>         unsigned int count = *nbuffers;
> 
>         if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>                 return -EINVAL;
> 
>         if (*nplanes) {
>                 if (*nplanes != 1 || sizes[0] < pix->sizeimage)
>                         return -EINVAL;
>                 count += vq->num_buffers;
>         }
> 
>         while (pix->sizeimage * count > VID_MEM_LIMIT)
>                 count--;

That's a weird way of writing:

	unsigned int max_num = VID_MEM_LIMIT / pix->sizeimage;
	count = max(count, max_num);

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 22:29             ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:29 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser, Steve Longerbeam,
	geert, linux-media, devicetree, p.zabel, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, laurent.pinchart+renesas,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Thu, Feb 02, 2017 at 11:12:41AM -0800, Steve Longerbeam wrote:
> Here is the current .queue_setup() op in imx-media-capture.c:
> 
> static int capture_queue_setup(struct vb2_queue *vq,
>                                unsigned int *nbuffers,
>                                unsigned int *nplanes,
>                                unsigned int sizes[],
>                                struct device *alloc_devs[])
> {
>         struct capture_priv *priv = vb2_get_drv_priv(vq);
>         struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
>         unsigned int count = *nbuffers;
> 
>         if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>                 return -EINVAL;
> 
>         if (*nplanes) {
>                 if (*nplanes != 1 || sizes[0] < pix->sizeimage)
>                         return -EINVAL;
>                 count += vq->num_buffers;
>         }
> 
>         while (pix->sizeimage * count > VID_MEM_LIMIT)
>                 count--;

That's a weird way of writing:

	unsigned int max_num = VID_MEM_LIMIT / pix->sizeimage;
	count = max(count, max_num);

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-02 22:29             ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:29 UTC (permalink / raw)
  To: linux-arm-kernel

On Thu, Feb 02, 2017 at 11:12:41AM -0800, Steve Longerbeam wrote:
> Here is the current .queue_setup() op in imx-media-capture.c:
> 
> static int capture_queue_setup(struct vb2_queue *vq,
>                                unsigned int *nbuffers,
>                                unsigned int *nplanes,
>                                unsigned int sizes[],
>                                struct device *alloc_devs[])
> {
>         struct capture_priv *priv = vb2_get_drv_priv(vq);
>         struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
>         unsigned int count = *nbuffers;
> 
>         if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>                 return -EINVAL;
> 
>         if (*nplanes) {
>                 if (*nplanes != 1 || sizes[0] < pix->sizeimage)
>                         return -EINVAL;
>                 count += vq->num_buffers;
>         }
> 
>         while (pix->sizeimage * count > VID_MEM_LIMIT)
>                 count--;

That's a weird way of writing:

	unsigned int max_num = VID_MEM_LIMIT / pix->sizeimage;
	count = max(count, max_num);

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-02-02 22:35     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:35 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:38PM -0800, Steve Longerbeam wrote:
> +struct camif_priv {
> +	struct device         *dev;
> +	struct video_device    vfd;

You can't do this.

> +static struct video_device camif_videodev = {
> +	.fops		= &camif_fops,
> +	.ioctl_ops	= &camif_ioctl_ops,
> +	.minor		= -1,
> +	.release	= video_device_release,
> +	.vfl_dir	= VFL_DIR_RX,
> +	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
> +};

> +static int camif_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct camif_priv *priv;
> +	struct video_device *vfd;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;

You kmalloc this structure, so this structure has the lifetime of
the driver being bound to the platform device.

> +	vfd = &priv->vfd;
> +	*vfd = camif_videodev;

However, "*vfd" contains a struct device, and you _correctly_ set the
release function for "*vfd" to video_device_release via camif_videodev.

However, if you try to rmmod imx-media, then you end up with a kernel
warning that you're freeing memory containing a held lock, and later
chaos ensues because kmalloc has been corrupted.

The root cause of this is embedding the device structure within the
video_device into the driver's private data.  *Any* structure what so
ever that contains a kref is reference counted, and that includes
struct device, and therefore also includes struct video_device.  What
that means is that its lifetime is _not_ under _your_ control, and
you may not free it except through its release function (which is
video_device_release().)  However, that also tries to kfree (with an
offset of 4) your private data, which results in the warning and the
corrupted kmalloc free lists.

The solution is simple, make "vfd" a pointer in your private data
structure and kmalloc() it separately, letting video_device_release()
kfree() that data when it needs to.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-02-02 22:35     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:35 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:38PM -0800, Steve Longerbeam wrote:
> +struct camif_priv {
> +	struct device         *dev;
> +	struct video_device    vfd;

You can't do this.

> +static struct video_device camif_videodev = {
> +	.fops		= &camif_fops,
> +	.ioctl_ops	= &camif_ioctl_ops,
> +	.minor		= -1,
> +	.release	= video_device_release,
> +	.vfl_dir	= VFL_DIR_RX,
> +	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
> +};

> +static int camif_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct camif_priv *priv;
> +	struct video_device *vfd;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;

You kmalloc this structure, so this structure has the lifetime of
the driver being bound to the platform device.

> +	vfd = &priv->vfd;
> +	*vfd = camif_videodev;

However, "*vfd" contains a struct device, and you _correctly_ set the
release function for "*vfd" to video_device_release via camif_videodev.

However, if you try to rmmod imx-media, then you end up with a kernel
warning that you're freeing memory containing a held lock, and later
chaos ensues because kmalloc has been corrupted.

The root cause of this is embedding the device structure within the
video_device into the driver's private data.  *Any* structure what so
ever that contains a kref is reference counted, and that includes
struct device, and therefore also includes struct video_device.  What
that means is that its lifetime is _not_ under _your_ control, and
you may not free it except through its release function (which is
video_device_release().)  However, that also tries to kfree (with an
offset of 4) your private data, which results in the warning and the
corrupted kmalloc free lists.

The solution is simple, make "vfd" a pointer in your private data
structure and kmalloc() it separately, letting video_device_release()
kfree() that data when it needs to.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-02-02 22:35     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:35 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:38PM -0800, Steve Longerbeam wrote:
> +struct camif_priv {
> +	struct device         *dev;
> +	struct video_device    vfd;

You can't do this.

> +static struct video_device camif_videodev = {
> +	.fops		= &camif_fops,
> +	.ioctl_ops	= &camif_ioctl_ops,
> +	.minor		= -1,
> +	.release	= video_device_release,
> +	.vfl_dir	= VFL_DIR_RX,
> +	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
> +};

> +static int camif_probe(struct platform_device *pdev)
> +{
> +	struct imx_media_internal_sd_platformdata *pdata;
> +	struct camif_priv *priv;
> +	struct video_device *vfd;
> +	int ret;
> +
> +	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;

You kmalloc this structure, so this structure has the lifetime of
the driver being bound to the platform device.

> +	vfd = &priv->vfd;
> +	*vfd = camif_videodev;

However, "*vfd" contains a struct device, and you _correctly_ set the
release function for "*vfd" to video_device_release via camif_videodev.

However, if you try to rmmod imx-media, then you end up with a kernel
warning that you're freeing memory containing a held lock, and later
chaos ensues because kmalloc has been corrupted.

The root cause of this is embedding the device structure within the
video_device into the driver's private data.  *Any* structure what so
ever that contains a kref is reference counted, and that includes
struct device, and therefore also includes struct video_device.  What
that means is that its lifetime is _not_ under _your_ control, and
you may not free it except through its release function (which is
video_device_release().)  However, that also tries to kfree (with an
offset of 4) your private data, which results in the warning and the
corrupted kmalloc free lists.

The solution is simple, make "vfd" a pointer in your private data
structure and kmalloc() it separately, letting video_device_release()
kfree() that data when it needs to.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-02-02 22:44     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:44 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> +struct imx_media_dev {
> +	struct media_device md;
> +	struct v4l2_device  v4l2_dev;

This is similarly buggy.

struct v4l2_device {
        struct device *dev;
#if defined(CONFIG_MEDIA_CONTROLLER)
        struct media_device *mdev;
#endif
        struct list_head subdevs;
        spinlock_t lock;
        char name[V4L2_DEVICE_NAME_SIZE];
        void (*notify)(struct v4l2_subdev *sd,
                        unsigned int notification, void *arg);
        struct v4l2_ctrl_handler *ctrl_handler;
        struct v4l2_prio_state prio;
        struct kref ref;
        void (*release)(struct v4l2_device *v4l2_dev);
};

Notice the kref and release function.  This is the only way the
memory backing "struct v4l2_device" may be released.  If you wish to
embed this structure into another structure, then the lifetime of
that other structure is determined by this one.  IOW, when this
release function is called, only then may you kfree() the memory
backing struct imx_media_dev.

> +	struct device *dev;

And... do you need all these struct device pointers?

        imxmd->dev = dev;
        imxmd->md.dev = dev;

As media_device already contains a pointer, can't you re-use that?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-02-02 22:44     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:44 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> +struct imx_media_dev {
> +	struct media_device md;
> +	struct v4l2_device  v4l2_dev;

This is similarly buggy.

struct v4l2_device {
        struct device *dev;
#if defined(CONFIG_MEDIA_CONTROLLER)
        struct media_device *mdev;
#endif
        struct list_head subdevs;
        spinlock_t lock;
        char name[V4L2_DEVICE_NAME_SIZE];
        void (*notify)(struct v4l2_subdev *sd,
                        unsigned int notification, void *arg);
        struct v4l2_ctrl_handler *ctrl_handler;
        struct v4l2_prio_state prio;
        struct kref ref;
        void (*release)(struct v4l2_device *v4l2_dev);
};

Notice the kref and release function.  This is the only way the
memory backing "struct v4l2_device" may be released.  If you wish to
embed this structure into another structure, then the lifetime of
that other structure is determined by this one.  IOW, when this
release function is called, only then may you kfree() the memory
backing struct imx_media_dev.

> +	struct device *dev;

And... do you need all these struct device pointers?

        imxmd->dev = dev;
        imxmd->md.dev = dev;

As media_device already contains a pointer, can't you re-use that?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-02-02 22:44     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:44 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> +struct imx_media_dev {
> +	struct media_device md;
> +	struct v4l2_device  v4l2_dev;

This is similarly buggy.

struct v4l2_device {
        struct device *dev;
#if defined(CONFIG_MEDIA_CONTROLLER)
        struct media_device *mdev;
#endif
        struct list_head subdevs;
        spinlock_t lock;
        char name[V4L2_DEVICE_NAME_SIZE];
        void (*notify)(struct v4l2_subdev *sd,
                        unsigned int notification, void *arg);
        struct v4l2_ctrl_handler *ctrl_handler;
        struct v4l2_prio_state prio;
        struct kref ref;
        void (*release)(struct v4l2_device *v4l2_dev);
};

Notice the kref and release function.  This is the only way the
memory backing "struct v4l2_device" may be released.  If you wish to
embed this structure into another structure, then the lifetime of
that other structure is determined by this one.  IOW, when this
release function is called, only then may you kfree() the memory
backing struct imx_media_dev.

> +	struct device *dev;

And... do you need all these struct device pointers?

        imxmd->dev = dev;
        imxmd->md.dev = dev;

As media_device already contains a pointer, can't you re-use that?

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-01-07  2:11 ` [PATCH v3 16/24] media: Add i.MX media core driver Steve Longerbeam
  2017-01-13 15:20     ` Philipp Zabel
@ 2017-02-02 22:50     ` Russell King - ARM Linux
  2017-02-02 22:44     ` Russell King - ARM Linux
  2017-02-02 22:50     ` Russell King - ARM Linux
  3 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:50 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> +/* register an internal subdev as a platform device */
> +static struct imx_media_subdev *
> +add_internal_subdev(struct imx_media_dev *imxmd,
> +		    const struct internal_subdev *isd,
> +		    int ipu_id)
> +{
> +	struct imx_media_internal_sd_platformdata pdata;
> +	struct platform_device_info pdevinfo = {0};
> +	struct imx_media_subdev *imxsd;
> +	struct platform_device *pdev;
> +
> +	switch (isd->id->grp_id) {
> +	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
> +		pdata.grp_id = isd->id->grp_id +
> +			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
> +		break;
> +	default:
> +		pdata.grp_id = isd->id->grp_id;
> +		break;
> +	}
> +
> +	/* the id of IPU this subdev will control */
> +	pdata.ipu_id = ipu_id;
> +
> +	/* create subdev name */
> +	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
> +				    pdata.grp_id, ipu_id);
> +
> +	pdevinfo.name = isd->id->name;
> +	pdevinfo.id = ipu_id * num_isd + isd->id->index;
> +	pdevinfo.parent = imxmd->dev;
> +	pdevinfo.data = &pdata;
> +	pdevinfo.size_data = sizeof(pdata);
> +	pdevinfo.dma_mask = DMA_BIT_MASK(32);
> +
> +	pdev = platform_device_register_full(&pdevinfo);
> +	if (IS_ERR(pdev))
> +		return ERR_CAST(pdev);
> +
> +	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
> +	if (IS_ERR(imxsd))
> +		return imxsd;
> +
> +	imxsd->num_sink_pads = isd->num_sink_pads;
> +	imxsd->num_src_pads = isd->num_src_pads;
> +
> +	return imxsd;
> +}

You seem to create platform devices here, but I see nowhere that you
ever remove them - so if you get to the lucky point of being able to
rmmod imx-media and then try to re-insert it, you end up with a load
of kernel warnings, one for each device created this way, and
platform_device_register_full() fails:

WARNING: CPU: 0 PID: 2143 at /home/rmk/git/linux-rmk/fs/sysfs/dir.c:31 sysfs_warn_dup+0x64/0x80
sysfs: cannot create duplicate filename '/devices/soc0/soc/soc:media@0/imx-ipuv3-smfc.2'
Modules linked in: imx_media(C+) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card uvcvideo snd_soc_imx_spdif imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 video_multiplexer imx_sdma caam imx2_wdt rc_cec coda v4l2_mem2mem videobuf2_v4l2 snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_dma_contig imx_pcm_dma videobuf2_core videobuf2_vmalloc videobuf2_memops imx_thermal dw_hdmi_cec dw_hdmi_ahb_audio etnaviv fuse rc_pinnacle_pctv_hd [last unloaded: imx_media]
CPU: 0 PID: 2143 Comm: modprobe Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60000013 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
 r6:c08ad998 r5:00000000 r4:cf4e9a78 r3:ec984980
[<c0033134>] (__warn) from [<c00332f4>] (warn_slowpath_fmt+0x40/0x48)
 r10:e5800010 r8:ef1fa818 r7:ef1fa810 r6:ef202300 r5:e9809000 r4:ee868000
[<c00332b8>] (warn_slowpath_fmt) from [<c01f5918>] (sysfs_warn_dup+0x64/0x80)
 r3:ee868000 r2:c08ad9c0
[<c01f58b4>] (sysfs_warn_dup) from [<c01f5a10>] (sysfs_create_dir_ns+0x90/0x98)
 r6:ffffffef r5:ef202300 r4:eb5e3418
[<c01f5980>] (sysfs_create_dir_ns) from [<c03361a4>] (kobject_add_internal+0xa4/0x2d8)
 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336100>] (kobject_add_internal) from [<c033657c>] (kobject_add+0x50/0x98)
 r8:bf1f9018 r7:ef1fa810 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336530>] (kobject_add) from [<c0415fd8>] (device_add+0xc8/0x538)
 r3:ef1fa834 r2:00000000
 r6:eb5e3418 r5:00000000 r4:eb5e3410
[<c0415f10>] (device_add) from [<c041ae24>] (platform_device_add+0xb0/0x214)
 r10:e5800010 r9:00000000 r8:bf1f9018 r7:e5806fd4 r6:eb5e3410 r5:eb5e3400
 r4:00000000
[<c041ad74>] (platform_device_add) from [<c041b7f4>] (platform_device_register_full+0xe8/0x110)
 r7:e5806fd4 r6:00000002 r5:eb5e3400 r4:cf4e9c18
[<c041b70c>] (platform_device_register_full) from [<bf1f7f54>] (add_ipu_internal_subdevs+0x128/0x2c8 [imx_media])
 r5:bf1f9000 r4:00000918
[<bf1f7e2c>] (add_ipu_internal_subdevs [imx_media]) from [<bf1f8120>] (imx_media_add_internal_subdevs+0x2c/0x70 [imx_media])
 r10:00000048 r9:f31ceef8 r8:ef7f1d94 r7:e5800230 r6:e5800200 r5:e5800010
 r4:cf4e9c90
[<bf1f80f4>] (imx_media_add_internal_subdevs [imx_media]) from [<bf1f71ec>] (imx_media_probe+0xc4/0x1c0 [imx_media])
 r5:00000000 r4:e5800010
[<bf1f7128>] (imx_media_probe [imx_media]) from [<c041ac80>] (platform_drv_probe+0x58/0xb8)
 r8:00000000 r7:bf1fbd48 r6:fffffdfb r5:ef1fa810 r4:fffffffe
[<c041ac28>] (platform_drv_probe) from [<c0418c90>] (driver_probe_device+0x204/0x2c8)
 r7:bf1fbd48 r6:00000000 r5:c1419de8 r4:ef1fa810
[<c0418a8c>] (driver_probe_device) from [<c0418e10>] (__driver_attach+0xbc/0xc0)
 r10:00000124 r8:00000001 r7:00000000 r6:ef1fa844 r5:bf1fbd48 r4:ef1fa810
[<c0418d54>] (__driver_attach) from [<c04170b0>] (bus_for_each_dev+0x5c/0x90)
 r6:c0418d54 r5:bf1fbd48 r4:00000000 r3:00000000
[<c0417054>] (bus_for_each_dev) from [<c04184f4>] (driver_attach+0x24/0x28)
 r6:c0a45e90 r5:ed5a8980 r4:bf1fbd48
[<c04184d0>] (driver_attach) from [<c04181f4>] (bus_add_driver+0xf4/0x200)
[<c0418100>] (bus_add_driver) from [<c0419c90>] (driver_register+0x80/0xfc)
 r7:00000000 r6:bf1fe000 r5:c0a70528 r4:bf1fbd48
[<c0419c10>] (driver_register) from [<c041ab54>] (__platform_driver_register+0x48/0x4c)
 r5:c0a70528 r4:bf1fbdc0
[<c041ab0c>] (__platform_driver_register) from [<bf1fe018>] (imx_media_pdrv_init+0x18/0x24 [imx_media])
[<bf1fe000>] (imx_media_pdrv_init [imx_media]) from [<c00098ac>] (do_one_initcall+0x44/0x170)
[<c0009868>] (do_one_initcall) from [<c011b090>] (do_init_module+0x68/0x1d8)
 r8:00000001 r7:bf1fbdc0 r6:e9809540 r5:c0a70528 r4:bf1fbdc0
[<c011b028>] (do_init_module) from [<c00d2d44>] (load_module+0x195c/0x2080)
 r7:bf1fbdc0 r6:c09e04ec r5:c0a70528 r4:c09f5f27
[<c00d13e8>] (load_module) from [<c00d3640>] (SyS_finit_module+0x94/0xa0)
 r10:00000000 r9:cf4e8000 r8:7f6b2398 r7:00000003 r6:00000000 r5:00000000
 r4:7fffffff
[<c00d35ac>] (SyS_finit_module) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
 r8:c000ff04 r7:0000017b r6:80c95148 r5:80c95b10 r4:80c95370
---[ end trace 05abce0bbb26bc34 ]---

imx-media: add_internal_subdevs failed with -17
imx-media: probe of soc:media@0 failed with error -17

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-02-02 22:50     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:50 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> +/* register an internal subdev as a platform device */
> +static struct imx_media_subdev *
> +add_internal_subdev(struct imx_media_dev *imxmd,
> +		    const struct internal_subdev *isd,
> +		    int ipu_id)
> +{
> +	struct imx_media_internal_sd_platformdata pdata;
> +	struct platform_device_info pdevinfo = {0};
> +	struct imx_media_subdev *imxsd;
> +	struct platform_device *pdev;
> +
> +	switch (isd->id->grp_id) {
> +	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
> +		pdata.grp_id = isd->id->grp_id +
> +			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
> +		break;
> +	default:
> +		pdata.grp_id = isd->id->grp_id;
> +		break;
> +	}
> +
> +	/* the id of IPU this subdev will control */
> +	pdata.ipu_id = ipu_id;
> +
> +	/* create subdev name */
> +	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
> +				    pdata.grp_id, ipu_id);
> +
> +	pdevinfo.name = isd->id->name;
> +	pdevinfo.id = ipu_id * num_isd + isd->id->index;
> +	pdevinfo.parent = imxmd->dev;
> +	pdevinfo.data = &pdata;
> +	pdevinfo.size_data = sizeof(pdata);
> +	pdevinfo.dma_mask = DMA_BIT_MASK(32);
> +
> +	pdev = platform_device_register_full(&pdevinfo);
> +	if (IS_ERR(pdev))
> +		return ERR_CAST(pdev);
> +
> +	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
> +	if (IS_ERR(imxsd))
> +		return imxsd;
> +
> +	imxsd->num_sink_pads = isd->num_sink_pads;
> +	imxsd->num_src_pads = isd->num_src_pads;
> +
> +	return imxsd;
> +}

You seem to create platform devices here, but I see nowhere that you
ever remove them - so if you get to the lucky point of being able to
rmmod imx-media and then try to re-insert it, you end up with a load
of kernel warnings, one for each device created this way, and
platform_device_register_full() fails:

WARNING: CPU: 0 PID: 2143 at /home/rmk/git/linux-rmk/fs/sysfs/dir.c:31 sysfs_warn_dup+0x64/0x80
sysfs: cannot create duplicate filename '/devices/soc0/soc/soc:media@0/imx-ipuv3-smfc.2'
Modules linked in: imx_media(C+) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card uvcvideo snd_soc_imx_spdif imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 video_multiplexer imx_sdma caam imx2_wdt rc_cec coda v4l2_mem2mem videobuf2_v4l2 snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_dma_contig imx_pcm_dma videobuf2_core videobuf2_vmalloc videobuf2_memops imx_thermal dw_hdmi_cec dw_hdmi_ahb_audio etnaviv fuse rc_pinnacle_pctv_hd [last unloaded: imx_media]
CPU: 0 PID: 2143 Comm: modprobe Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60000013 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
 r6:c08ad998 r5:00000000 r4:cf4e9a78 r3:ec984980
[<c0033134>] (__warn) from [<c00332f4>] (warn_slowpath_fmt+0x40/0x48)
 r10:e5800010 r8:ef1fa818 r7:ef1fa810 r6:ef202300 r5:e9809000 r4:ee868000
[<c00332b8>] (warn_slowpath_fmt) from [<c01f5918>] (sysfs_warn_dup+0x64/0x80)
 r3:ee868000 r2:c08ad9c0
[<c01f58b4>] (sysfs_warn_dup) from [<c01f5a10>] (sysfs_create_dir_ns+0x90/0x98)
 r6:ffffffef r5:ef202300 r4:eb5e3418
[<c01f5980>] (sysfs_create_dir_ns) from [<c03361a4>] (kobject_add_internal+0xa4/0x2d8)
 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336100>] (kobject_add_internal) from [<c033657c>] (kobject_add+0x50/0x98)
 r8:bf1f9018 r7:ef1fa810 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336530>] (kobject_add) from [<c0415fd8>] (device_add+0xc8/0x538)
 r3:ef1fa834 r2:00000000
 r6:eb5e3418 r5:00000000 r4:eb5e3410
[<c0415f10>] (device_add) from [<c041ae24>] (platform_device_add+0xb0/0x214)
 r10:e5800010 r9:00000000 r8:bf1f9018 r7:e5806fd4 r6:eb5e3410 r5:eb5e3400
 r4:00000000
[<c041ad74>] (platform_device_add) from [<c041b7f4>] (platform_device_register_full+0xe8/0x110)
 r7:e5806fd4 r6:00000002 r5:eb5e3400 r4:cf4e9c18
[<c041b70c>] (platform_device_register_full) from [<bf1f7f54>] (add_ipu_internal_subdevs+0x128/0x2c8 [imx_media])
 r5:bf1f9000 r4:00000918
[<bf1f7e2c>] (add_ipu_internal_subdevs [imx_media]) from [<bf1f8120>] (imx_media_add_internal_subdevs+0x2c/0x70 [imx_media])
 r10:00000048 r9:f31ceef8 r8:ef7f1d94 r7:e5800230 r6:e5800200 r5:e5800010
 r4:cf4e9c90
[<bf1f80f4>] (imx_media_add_internal_subdevs [imx_media]) from [<bf1f71ec>] (imx_media_probe+0xc4/0x1c0 [imx_media])
 r5:00000000 r4:e5800010
[<bf1f7128>] (imx_media_probe [imx_media]) from [<c041ac80>] (platform_drv_probe+0x58/0xb8)
 r8:00000000 r7:bf1fbd48 r6:fffffdfb r5:ef1fa810 r4:fffffffe
[<c041ac28>] (platform_drv_probe) from [<c0418c90>] (driver_probe_device+0x204/0x2c8)
 r7:bf1fbd48 r6:00000000 r5:c1419de8 r4:ef1fa810
[<c0418a8c>] (driver_probe_device) from [<c0418e10>] (__driver_attach+0xbc/0xc0)
 r10:00000124 r8:00000001 r7:00000000 r6:ef1fa844 r5:bf1fbd48 r4:ef1fa810
[<c0418d54>] (__driver_attach) from [<c04170b0>] (bus_for_each_dev+0x5c/0x90)
 r6:c0418d54 r5:bf1fbd48 r4:00000000 r3:00000000
[<c0417054>] (bus_for_each_dev) from [<c04184f4>] (driver_attach+0x24/0x28)
 r6:c0a45e90 r5:ed5a8980 r4:bf1fbd48
[<c04184d0>] (driver_attach) from [<c04181f4>] (bus_add_driver+0xf4/0x200)
[<c0418100>] (bus_add_driver) from [<c0419c90>] (driver_register+0x80/0xfc)
 r7:00000000 r6:bf1fe000 r5:c0a70528 r4:bf1fbd48
[<c0419c10>] (driver_register) from [<c041ab54>] (__platform_driver_register+0x48/0x4c)
 r5:c0a70528 r4:bf1fbdc0
[<c041ab0c>] (__platform_driver_register) from [<bf1fe018>] (imx_media_pdrv_init+0x18/0x24 [imx_media])
[<bf1fe000>] (imx_media_pdrv_init [imx_media]) from [<c00098ac>] (do_one_initcall+0x44/0x170)
[<c0009868>] (do_one_initcall) from [<c011b090>] (do_init_module+0x68/0x1d8)
 r8:00000001 r7:bf1fbdc0 r6:e9809540 r5:c0a70528 r4:bf1fbdc0
[<c011b028>] (do_init_module) from [<c00d2d44>] (load_module+0x195c/0x2080)
 r7:bf1fbdc0 r6:c09e04ec r5:c0a70528 r4:c09f5f27
[<c00d13e8>] (load_module) from [<c00d3640>] (SyS_finit_module+0x94/0xa0)
 r10:00000000 r9:cf4e8000 r8:7f6b2398 r7:00000003 r6:00000000 r5:00000000
 r4:7fffffff
[<c00d35ac>] (SyS_finit_module) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
 r8:c000ff04 r7:0000017b r6:80c95148 r5:80c95b10 r4:80c95370
---[ end trace 05abce0bbb26bc34 ]---

imx-media: add_internal_subdevs failed with -17
imx-media: probe of soc:media@0 failed with error -17

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-02-02 22:50     ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-02 22:50 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
> +/* register an internal subdev as a platform device */
> +static struct imx_media_subdev *
> +add_internal_subdev(struct imx_media_dev *imxmd,
> +		    const struct internal_subdev *isd,
> +		    int ipu_id)
> +{
> +	struct imx_media_internal_sd_platformdata pdata;
> +	struct platform_device_info pdevinfo = {0};
> +	struct imx_media_subdev *imxsd;
> +	struct platform_device *pdev;
> +
> +	switch (isd->id->grp_id) {
> +	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
> +		pdata.grp_id = isd->id->grp_id +
> +			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
> +		break;
> +	default:
> +		pdata.grp_id = isd->id->grp_id;
> +		break;
> +	}
> +
> +	/* the id of IPU this subdev will control */
> +	pdata.ipu_id = ipu_id;
> +
> +	/* create subdev name */
> +	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
> +				    pdata.grp_id, ipu_id);
> +
> +	pdevinfo.name = isd->id->name;
> +	pdevinfo.id = ipu_id * num_isd + isd->id->index;
> +	pdevinfo.parent = imxmd->dev;
> +	pdevinfo.data = &pdata;
> +	pdevinfo.size_data = sizeof(pdata);
> +	pdevinfo.dma_mask = DMA_BIT_MASK(32);
> +
> +	pdev = platform_device_register_full(&pdevinfo);
> +	if (IS_ERR(pdev))
> +		return ERR_CAST(pdev);
> +
> +	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
> +	if (IS_ERR(imxsd))
> +		return imxsd;
> +
> +	imxsd->num_sink_pads = isd->num_sink_pads;
> +	imxsd->num_src_pads = isd->num_src_pads;
> +
> +	return imxsd;
> +}

You seem to create platform devices here, but I see nowhere that you
ever remove them - so if you get to the lucky point of being able to
rmmod imx-media and then try to re-insert it, you end up with a load
of kernel warnings, one for each device created this way, and
platform_device_register_full() fails:

WARNING: CPU: 0 PID: 2143 at /home/rmk/git/linux-rmk/fs/sysfs/dir.c:31 sysfs_warn_dup+0x64/0x80
sysfs: cannot create duplicate filename '/devices/soc0/soc/soc:media at 0/imx-ipuv3-smfc.2'
Modules linked in: imx_media(C+) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card uvcvideo snd_soc_imx_spdif imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 video_multiplexer imx_sdma caam imx2_wdt rc_cec coda v4l2_mem2mem videobuf2_v4l2 snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_dma_contig imx_pcm_dma videobuf2_core videobuf2_vmalloc videobuf2_memops imx_thermal dw_hdmi_cec dw_hdmi_ahb_audio etnaviv fuse rc_pinnacle_pctv_hd [last unloaded: imx_media]
CPU: 0 PID: 2143 Comm: modprobe Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60000013 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
 r6:c08ad998 r5:00000000 r4:cf4e9a78 r3:ec984980
[<c0033134>] (__warn) from [<c00332f4>] (warn_slowpath_fmt+0x40/0x48)
 r10:e5800010 r8:ef1fa818 r7:ef1fa810 r6:ef202300 r5:e9809000 r4:ee868000
[<c00332b8>] (warn_slowpath_fmt) from [<c01f5918>] (sysfs_warn_dup+0x64/0x80)
 r3:ee868000 r2:c08ad9c0
[<c01f58b4>] (sysfs_warn_dup) from [<c01f5a10>] (sysfs_create_dir_ns+0x90/0x98)
 r6:ffffffef r5:ef202300 r4:eb5e3418
[<c01f5980>] (sysfs_create_dir_ns) from [<c03361a4>] (kobject_add_internal+0xa4/0x2d8)
 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336100>] (kobject_add_internal) from [<c033657c>] (kobject_add+0x50/0x98)
 r8:bf1f9018 r7:ef1fa810 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336530>] (kobject_add) from [<c0415fd8>] (device_add+0xc8/0x538)
 r3:ef1fa834 r2:00000000
 r6:eb5e3418 r5:00000000 r4:eb5e3410
[<c0415f10>] (device_add) from [<c041ae24>] (platform_device_add+0xb0/0x214)
 r10:e5800010 r9:00000000 r8:bf1f9018 r7:e5806fd4 r6:eb5e3410 r5:eb5e3400
 r4:00000000
[<c041ad74>] (platform_device_add) from [<c041b7f4>] (platform_device_register_full+0xe8/0x110)
 r7:e5806fd4 r6:00000002 r5:eb5e3400 r4:cf4e9c18
[<c041b70c>] (platform_device_register_full) from [<bf1f7f54>] (add_ipu_internal_subdevs+0x128/0x2c8 [imx_media])
 r5:bf1f9000 r4:00000918
[<bf1f7e2c>] (add_ipu_internal_subdevs [imx_media]) from [<bf1f8120>] (imx_media_add_internal_subdevs+0x2c/0x70 [imx_media])
 r10:00000048 r9:f31ceef8 r8:ef7f1d94 r7:e5800230 r6:e5800200 r5:e5800010
 r4:cf4e9c90
[<bf1f80f4>] (imx_media_add_internal_subdevs [imx_media]) from [<bf1f71ec>] (imx_media_probe+0xc4/0x1c0 [imx_media])
 r5:00000000 r4:e5800010
[<bf1f7128>] (imx_media_probe [imx_media]) from [<c041ac80>] (platform_drv_probe+0x58/0xb8)
 r8:00000000 r7:bf1fbd48 r6:fffffdfb r5:ef1fa810 r4:fffffffe
[<c041ac28>] (platform_drv_probe) from [<c0418c90>] (driver_probe_device+0x204/0x2c8)
 r7:bf1fbd48 r6:00000000 r5:c1419de8 r4:ef1fa810
[<c0418a8c>] (driver_probe_device) from [<c0418e10>] (__driver_attach+0xbc/0xc0)
 r10:00000124 r8:00000001 r7:00000000 r6:ef1fa844 r5:bf1fbd48 r4:ef1fa810
[<c0418d54>] (__driver_attach) from [<c04170b0>] (bus_for_each_dev+0x5c/0x90)
 r6:c0418d54 r5:bf1fbd48 r4:00000000 r3:00000000
[<c0417054>] (bus_for_each_dev) from [<c04184f4>] (driver_attach+0x24/0x28)
 r6:c0a45e90 r5:ed5a8980 r4:bf1fbd48
[<c04184d0>] (driver_attach) from [<c04181f4>] (bus_add_driver+0xf4/0x200)
[<c0418100>] (bus_add_driver) from [<c0419c90>] (driver_register+0x80/0xfc)
 r7:00000000 r6:bf1fe000 r5:c0a70528 r4:bf1fbd48
[<c0419c10>] (driver_register) from [<c041ab54>] (__platform_driver_register+0x48/0x4c)
 r5:c0a70528 r4:bf1fbdc0
[<c041ab0c>] (__platform_driver_register) from [<bf1fe018>] (imx_media_pdrv_init+0x18/0x24 [imx_media])
[<bf1fe000>] (imx_media_pdrv_init [imx_media]) from [<c00098ac>] (do_one_initcall+0x44/0x170)
[<c0009868>] (do_one_initcall) from [<c011b090>] (do_init_module+0x68/0x1d8)
 r8:00000001 r7:bf1fbdc0 r6:e9809540 r5:c0a70528 r4:bf1fbdc0
[<c011b028>] (do_init_module) from [<c00d2d44>] (load_module+0x195c/0x2080)
 r7:bf1fbdc0 r6:c09e04ec r5:c0a70528 r4:c09f5f27
[<c00d13e8>] (load_module) from [<c00d3640>] (SyS_finit_module+0x94/0xa0)
 r10:00000000 r9:cf4e8000 r8:7f6b2398 r7:00000003 r6:00000000 r5:00000000
 r4:7fffffff
[<c00d35ac>] (SyS_finit_module) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
 r8:c000ff04 r7:0000017b r6:80c95148 r5:80c95b10 r4:80c95370
---[ end trace 05abce0bbb26bc34 ]---

imx-media: add_internal_subdevs failed with -17
imx-media: probe of soc:media at 0 failed with error -17

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* vb2 queue_setup documentation clarification (was "Re: [PATCH v3 00/24] i.MX Media Driver")
  2017-02-02 19:12           ` Steve Longerbeam
                             ` (2 preceding siblings ...)
  (?)
@ 2017-02-03 14:21           ` Laurent Pinchart
  2017-02-03 14:31             ` Hans Verkuil
  -1 siblings, 1 reply; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-03 14:21 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Russell King - ARM Linux, hverkuil, Steve Longerbeam,
	linux-media, p.zabel, Sakari Ailus

Hi Steve,

(stripping the CC list a bit and adding Sakari Ailus)

On Thursday 02 Feb 2017 11:12:41 Steve Longerbeam wrote:
> On 02/02/2017 10:58 AM, Russell King - ARM Linux wrote:

[snip]

> > It seems to me that if you don't take account of the existing queue
> > size, your camif_queue_setup() has the side effect that each time
> > either of these are called.  Hence, the vb2 queue increases by the
> > same amount each time, which is probably what you don't want.
> > 
> > The documentation on queue_setup() leaves much to be desired:
> >   * @queue_setup: called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS() 
> >   *          handlers before memory allocation. It can be  called
> >   *          twice: if the original number of requested  buffers
> >   *          could not be allocated, then it will be called a
> >   *          second time with the actually allocated number of
> >   *          buffers to verify if that is OK.
> >   *          The driver should return the required number of buffers
> >   *          in \*num_buffers, the required number of planes per
> >   *          buffer in \*num_planes, the size of each plane should be
> >   *          set in the sizes\[\] array and optional per-plane
> >   *          allocator specific device in the alloc_devs\[\] array.
> >   *          When called from VIDIOC_REQBUFS,() \*num_planes == 0,
> >   *          the driver has to use the currently configured format to
> >   *          determine the plane sizes and \*num_buffers is the total
> >   *          number of buffers that are being allocated. When called
> >   *          from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
> >   *          describes the requested number of planes and sizes\[\]
> >   *          contains the requested plane sizes. If either
> >   *          \*num_planes or the requested sizes are invalid callback
> >   *          must return %-EINVAL. In this case \*num_buffers are
> >   *          being allocated additionally to q->num_buffers.
> >
> > That's really really ambiguous, because the "In this case" part doesn't
> > really tell you which case it's talking about - but it seems to me looking
> > at the code that it's referring to the VIDIOC_CREATE_BUFS case.
> 
> Yes, I caught this when adding fixes from v4l2-compliance testing, which
> is not part of the version 3 driver. I agree it is a confusing API. When
> called from VIDIOC_CREATE_BUFS (indicated by *num_planes != 0),
> *num_buffers is supposed to be requested buffers _in addition_ to
> already requested q->num_buffers, which is important info and
> should be emphasized a little more than the "oh by the way" fashion
> in the prototype description, IMHO.

Hans, Sakari, any opinion ?

[snip]

-- 
Regards,

Laurent Pinchart


^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: vb2 queue_setup documentation clarification (was "Re: [PATCH v3 00/24] i.MX Media Driver")
  2017-02-03 14:21           ` vb2 queue_setup documentation clarification (was "Re: [PATCH v3 00/24] i.MX Media Driver") Laurent Pinchart
@ 2017-02-03 14:31             ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-02-03 14:31 UTC (permalink / raw)
  To: Laurent Pinchart, Steve Longerbeam
  Cc: Russell King - ARM Linux, Steve Longerbeam, linux-media, p.zabel,
	Sakari Ailus

On 03/02/17 15:21, Laurent Pinchart wrote:
> Hi Steve,
>
> (stripping the CC list a bit and adding Sakari Ailus)
>
> On Thursday 02 Feb 2017 11:12:41 Steve Longerbeam wrote:
>> On 02/02/2017 10:58 AM, Russell King - ARM Linux wrote:
>
> [snip]
>
>>> It seems to me that if you don't take account of the existing queue
>>> size, your camif_queue_setup() has the side effect that each time
>>> either of these are called.  Hence, the vb2 queue increases by the
>>> same amount each time, which is probably what you don't want.
>>>
>>> The documentation on queue_setup() leaves much to be desired:
>>>   * @queue_setup: called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
>>>   *          handlers before memory allocation. It can be  called
>>>   *          twice: if the original number of requested  buffers
>>>   *          could not be allocated, then it will be called a
>>>   *          second time with the actually allocated number of
>>>   *          buffers to verify if that is OK.
>>>   *          The driver should return the required number of buffers
>>>   *          in \*num_buffers, the required number of planes per
>>>   *          buffer in \*num_planes, the size of each plane should be
>>>   *          set in the sizes\[\] array and optional per-plane
>>>   *          allocator specific device in the alloc_devs\[\] array.
>>>   *          When called from VIDIOC_REQBUFS,() \*num_planes == 0,
>>>   *          the driver has to use the currently configured format to
>>>   *          determine the plane sizes and \*num_buffers is the total
>>>   *          number of buffers that are being allocated. When called
>>>   *          from VIDIOC_CREATE_BUFS,() \*num_planes != 0 and it
>>>   *          describes the requested number of planes and sizes\[\]
>>>   *          contains the requested plane sizes. If either
>>>   *          \*num_planes or the requested sizes are invalid callback
>>>   *          must return %-EINVAL. In this case \*num_buffers are
>>>   *          being allocated additionally to q->num_buffers.
>>>
>>> That's really really ambiguous, because the "In this case" part doesn't
>>> really tell you which case it's talking about - but it seems to me looking
>>> at the code that it's referring to the VIDIOC_CREATE_BUFS case.
>>
>> Yes, I caught this when adding fixes from v4l2-compliance testing, which
>> is not part of the version 3 driver. I agree it is a confusing API. When
>> called from VIDIOC_CREATE_BUFS (indicated by *num_planes != 0),
>> *num_buffers is supposed to be requested buffers _in addition_ to
>> already requested q->num_buffers, which is important info and
>> should be emphasized a little more than the "oh by the way" fashion
>> in the prototype description, IMHO.
>
> Hans, Sakari, any opinion ?

This certainly could be improved.

The total number of buffers is q->num_buffers + *num_buffer where q->num_buffers
is 0 when called from VIDIOC_REQBUFS and can be > 0 when called from VIDIOC_CREATEBUFS.

So if you want to ensure that a certain minimum number of buffers is allocated,
then you would code that as:

         if (q->num_buffers + *num_buffers < 3)
                 *num_buffers = 3 - q->num_buffers;

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-02  0:19         ` Steve Longerbeam
  (?)
@ 2017-02-03 14:41           ` Laurent Pinchart
  -1 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-03 14:41 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hello,

On Wednesday 01 Feb 2017 16:19:27 Steve Longerbeam wrote:
> On 02/01/2017 01:30 AM, Philipp Zabel wrote:
> > On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> > [...]
> > 
> >>> # Set pad formats
> >>> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> >>> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> >>> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> >>> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
> >>> 
> >>> v4l2-ctl -d /dev/video4 -V
> >>> # This still is configured to 640x480, which is inconsistent with
> >>> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> >>> # have set this, too.
> >> 
> >> Because you've only configured the source pads,
> >> and not the sink pads. The ipu_csi source format is
> >> dependent on the sink format - output crop window is
> >> limited by max input sensor frame, and since sink pad is
> >> still at 640x480, output is reduced to that.
> > 
> > No, it is set (see below). What happens is that capture_g_fmt_vid_cap
> > just returns the capture devices' priv->vdev.fmt, even if it is
> > incompatible with the connected csi subdevice's output pad format.
> > 
> > priv->vdev.fmt was never changed from the default set in
> > imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
> > were not called yet.
> 
> Ah, yep, this is a bug. Need to modify the capture device's
> width/height at .set_fmt() in the subdev's device-node source
> pad (csi and prpenc/vf).
> 
> >> Maybe I'm missing something, is it expected behavior that
> >> a source format should be automatically propagated to
> >> the sink?
> > 
> > media-ctl propagates the output pad format to all remote subdevices'
> > input pads for all enabled links:
> > 
> > https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c
> > #n693
>
> Ah cool, I wasn't aware media-ctl did this, but it makes sense and
> makes it easier on the user.

To be precise, userspace is responsible for propagating formats *between* 
subdevs (source to sink, over a link) and drivers for propagating formats *in* 
subdevs (sink to source, inside the subdev).

> >>> v4l2-ctl --list-formats -d /dev/video4
> >>> # This lists all the RGB formats, which it shouldn't. There is
> >>> # no CSC in this pipeline, so we should be limited to YUV formats
> >>> # only.
> >> 
> >> right, need to fix that. Probably by poking the attached
> >> source subdev (csi or prpenc/vf) for its supported formats.
> > 
> > You are right, in bayer/raw mode only one specific format should be
> > listed, depending on the CSI output pad format.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-03 14:41           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-03 14:41 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

Hello,

On Wednesday 01 Feb 2017 16:19:27 Steve Longerbeam wrote:
> On 02/01/2017 01:30 AM, Philipp Zabel wrote:
> > On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> > [...]
> > 
> >>> # Set pad formats
> >>> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> >>> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> >>> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> >>> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
> >>> 
> >>> v4l2-ctl -d /dev/video4 -V
> >>> # This still is configured to 640x480, which is inconsistent with
> >>> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> >>> # have set this, too.
> >> 
> >> Because you've only configured the source pads,
> >> and not the sink pads. The ipu_csi source format is
> >> dependent on the sink format - output crop window is
> >> limited by max input sensor frame, and since sink pad is
> >> still at 640x480, output is reduced to that.
> > 
> > No, it is set (see below). What happens is that capture_g_fmt_vid_cap
> > just returns the capture devices' priv->vdev.fmt, even if it is
> > incompatible with the connected csi subdevice's output pad format.
> > 
> > priv->vdev.fmt was never changed from the default set in
> > imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
> > were not called yet.
> 
> Ah, yep, this is a bug. Need to modify the capture device's
> width/height at .set_fmt() in the subdev's device-node source
> pad (csi and prpenc/vf).
> 
> >> Maybe I'm missing something, is it expected behavior that
> >> a source format should be automatically propagated to
> >> the sink?
> > 
> > media-ctl propagates the output pad format to all remote subdevices'
> > input pads for all enabled links:
> > 
> > https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c
> > #n693
>
> Ah cool, I wasn't aware media-ctl did this, but it makes sense and
> makes it easier on the user.

To be precise, userspace is responsible for propagating formats *between* 
subdevs (source to sink, over a link) and drivers for propagating formats *in* 
subdevs (sink to source, inside the subdev).

> >>> v4l2-ctl --list-formats -d /dev/video4
> >>> # This lists all the RGB formats, which it shouldn't. There is
> >>> # no CSC in this pipeline, so we should be limited to YUV formats
> >>> # only.
> >> 
> >> right, need to fix that. Probably by poking the attached
> >> source subdev (csi or prpenc/vf) for its supported formats.
> > 
> > You are right, in bayer/raw mode only one specific format should be
> > listed, depending on the CSI output pad format.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-03 14:41           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-03 14:41 UTC (permalink / raw)
  To: linux-arm-kernel

Hello,

On Wednesday 01 Feb 2017 16:19:27 Steve Longerbeam wrote:
> On 02/01/2017 01:30 AM, Philipp Zabel wrote:
> > On Tue, 2017-01-31 at 17:26 -0800, Steve Longerbeam wrote:
> > [...]
> > 
> >>> # Set pad formats
> >>> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> >>> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> >>> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> >>> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
> >>> 
> >>> v4l2-ctl -d /dev/video4 -V
> >>> # This still is configured to 640x480, which is inconsistent with
> >>> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> >>> # have set this, too.
> >> 
> >> Because you've only configured the source pads,
> >> and not the sink pads. The ipu_csi source format is
> >> dependent on the sink format - output crop window is
> >> limited by max input sensor frame, and since sink pad is
> >> still at 640x480, output is reduced to that.
> > 
> > No, it is set (see below). What happens is that capture_g_fmt_vid_cap
> > just returns the capture devices' priv->vdev.fmt, even if it is
> > incompatible with the connected csi subdevice's output pad format.
> > 
> > priv->vdev.fmt was never changed from the default set in
> > imx_media_capture_device_register, because capture_s/try_fmt_vid_cap
> > were not called yet.
> 
> Ah, yep, this is a bug. Need to modify the capture device's
> width/height at .set_fmt() in the subdev's device-node source
> pad (csi and prpenc/vf).
> 
> >> Maybe I'm missing something, is it expected behavior that
> >> a source format should be automatically propagated to
> >> the sink?
> > 
> > media-ctl propagates the output pad format to all remote subdevices'
> > input pads for all enabled links:
> > 
> > https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c
> > #n693
>
> Ah cool, I wasn't aware media-ctl did this, but it makes sense and
> makes it easier on the user.

To be precise, userspace is responsible for propagating formats *between* 
subdevs (source to sink, over a link) and drivers for propagating formats *in* 
subdevs (sink to source, inside the subdev).

> >>> v4l2-ctl --list-formats -d /dev/video4
> >>> # This lists all the RGB formats, which it shouldn't. There is
> >>> # no CSC in this pipeline, so we should be limited to YUV formats
> >>> # only.
> >> 
> >> right, need to fix that. Probably by poking the attached
> >> source subdev (csi or prpenc/vf) for its supported formats.
> > 
> > You are right, in bayer/raw mode only one specific format should be
> > listed, depending on the CSI output pad format.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
  2017-02-03 14:41           ` Laurent Pinchart
  (?)
@ 2017-02-03 17:56             ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-03 17:56 UTC (permalink / raw)
  To: Laurent Pinchart, Steve Longerbeam
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel



On 02/03/2017 06:41 AM, Laurent Pinchart wrote:
> Hello,
>
> On Wednesday 01 Feb 2017 16:19:27 Steve Longerbeam wrote:
>> On 02/01/2017 01:30 AM, Philipp Zabel wrote:
>>
>>> media-ctl propagates the output pad format to all remote subdevices'
>>> input pads for all enabled links:
>>>
>>> https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c
>>> #n693
>> Ah cool, I wasn't aware media-ctl did this, but it makes sense and
>> makes it easier on the user.
> To be precise, userspace is responsible for propagating formats *between*
> subdevs (source to sink, over a link) and drivers for propagating formats *in*
> subdevs (sink to source, inside the subdev).

Hi Laurent, yes thanks for that clarification.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-03 17:56             ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-03 17:56 UTC (permalink / raw)
  To: Laurent Pinchart, Steve Longerbeam
  Cc: Philipp Zabel, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b



On 02/03/2017 06:41 AM, Laurent Pinchart wrote:
> Hello,
>
> On Wednesday 01 Feb 2017 16:19:27 Steve Longerbeam wrote:
>> On 02/01/2017 01:30 AM, Philipp Zabel wrote:
>>
>>> media-ctl propagates the output pad format to all remote subdevices'
>>> input pads for all enabled links:
>>>
>>> https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c
>>> #n693
>> Ah cool, I wasn't aware media-ctl did this, but it makes sense and
>> makes it easier on the user.
> To be precise, userspace is responsible for propagating formats *between*
> subdevs (source to sink, over a link) and drivers for propagating formats *in*
> subdevs (sink to source, inside the subdev).

Hi Laurent, yes thanks for that clarification.

Steve
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-03 17:56             ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-03 17:56 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/03/2017 06:41 AM, Laurent Pinchart wrote:
> Hello,
>
> On Wednesday 01 Feb 2017 16:19:27 Steve Longerbeam wrote:
>> On 02/01/2017 01:30 AM, Philipp Zabel wrote:
>>
>>> media-ctl propagates the output pad format to all remote subdevices'
>>> input pads for all enabled links:
>>>
>>> https://git.linuxtv.org/v4l-utils.git/tree/utils/media-ctl/libv4l2subdev.c
>>> #n693
>> Ah cool, I wasn't aware media-ctl did this, but it makes sense and
>> makes it easier on the user.
> To be precise, userspace is responsible for propagating formats *between*
> subdevs (source to sink, over a link) and drivers for propagating formats *in*
> subdevs (sink to source, inside the subdev).

Hi Laurent, yes thanks for that clarification.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-03 18:49               ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-03 18:49 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/02/2017 02:29 PM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 11:12:41AM -0800, Steve Longerbeam wrote:
>> Here is the current .queue_setup() op in imx-media-capture.c:
>>
>> static int capture_queue_setup(struct vb2_queue *vq,
>>                                 unsigned int *nbuffers,
>>                                 unsigned int *nplanes,
>>                                 unsigned int sizes[],
>>                                 struct device *alloc_devs[])
>> {
>>          struct capture_priv *priv = vb2_get_drv_priv(vq);
>>          struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
>>          unsigned int count = *nbuffers;
>>
>>          if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>                  return -EINVAL;
>>
>>          if (*nplanes) {
>>                  if (*nplanes != 1 || sizes[0] < pix->sizeimage)
>>                          return -EINVAL;
>>                  count += vq->num_buffers;
>>          }
>>
>>          while (pix->sizeimage * count > VID_MEM_LIMIT)
>>                  count--;
> That's a weird way of writing:
>
> 	unsigned int max_num = VID_MEM_LIMIT / pix->sizeimage;
> 	count = max(count, max_num);

I think you mean min() there, but yes thanks, fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-03 18:49               ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-03 18:49 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland-5wv7dgnIgG8, andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	nick-gcszYUEDH4VrovVCs/uTlw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, Steve Longerbeam,
	robert.jarzmik-GANU6spQydw,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	geert-Td1EMuHUCqxL1ZNQvxDV9g, linux-media-u79uwXL29TY76Z2rM5mHXA,
	devicetree-u79uwXL29TY76Z2rM5mHXA, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	arnd-r2nGTMty4D4, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	bparrot-l0cyMroinI0, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	jean-christophe.trotin-qxv4g6HH51o,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, sudipm.m



On 02/02/2017 02:29 PM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 11:12:41AM -0800, Steve Longerbeam wrote:
>> Here is the current .queue_setup() op in imx-media-capture.c:
>>
>> static int capture_queue_setup(struct vb2_queue *vq,
>>                                 unsigned int *nbuffers,
>>                                 unsigned int *nplanes,
>>                                 unsigned int sizes[],
>>                                 struct device *alloc_devs[])
>> {
>>          struct capture_priv *priv = vb2_get_drv_priv(vq);
>>          struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
>>          unsigned int count = *nbuffers;
>>
>>          if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>                  return -EINVAL;
>>
>>          if (*nplanes) {
>>                  if (*nplanes != 1 || sizes[0] < pix->sizeimage)
>>                          return -EINVAL;
>>                  count += vq->num_buffers;
>>          }
>>
>>          while (pix->sizeimage * count > VID_MEM_LIMIT)
>>                  count--;
> That's a weird way of writing:
>
> 	unsigned int max_num = VID_MEM_LIMIT / pix->sizeimage;
> 	count = max(count, max_num);

I think you mean min() there, but yes thanks, fixed.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-03 18:49               ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-03 18:49 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/02/2017 02:29 PM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2017 at 11:12:41AM -0800, Steve Longerbeam wrote:
>> Here is the current .queue_setup() op in imx-media-capture.c:
>>
>> static int capture_queue_setup(struct vb2_queue *vq,
>>                                 unsigned int *nbuffers,
>>                                 unsigned int *nplanes,
>>                                 unsigned int sizes[],
>>                                 struct device *alloc_devs[])
>> {
>>          struct capture_priv *priv = vb2_get_drv_priv(vq);
>>          struct v4l2_pix_format *pix = &priv->vdev.fmt.fmt.pix;
>>          unsigned int count = *nbuffers;
>>
>>          if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>                  return -EINVAL;
>>
>>          if (*nplanes) {
>>                  if (*nplanes != 1 || sizes[0] < pix->sizeimage)
>>                          return -EINVAL;
>>                  count += vq->num_buffers;
>>          }
>>
>>          while (pix->sizeimage * count > VID_MEM_LIMIT)
>>                  count--;
> That's a weird way of writing:
>
> 	unsigned int max_num = VID_MEM_LIMIT / pix->sizeimage;
> 	count = max(count, max_num);

I think you mean min() there, but yes thanks, fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:17           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:17 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Monday 16 Jan 2017 13:55:23 Philipp Zabel wrote:
> On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
> > On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> > > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> > >> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2
> > >> sensor.
> > >> Both hang off the same i2c2 bus, so they require different (and non-
> > >> default) i2c slave addresses.
> > >> 
> > >> The OV5642 connects to the parallel-bus mux input port on
> > >> ipu1_csi0_mux.
> > >> 
> > >> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> > >> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> > >> 
> > >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> > >> ---
> > >> 
> > >>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
> > >>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
> > >>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 ++++++++++++++++++++++
> > >>   3 files changed, 129 insertions(+)
> > >> 
> > >> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> > >> b/arch/arm/boot/dts/imx6dl-sabrelite.dts index 0f06ca5..fec2524 100644
> > >> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> > >> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> 
> [...]
> 
> > >> @@ -299,6 +326,52 @@
> > >> 
> > >>   	pinctrl-names = "default";
> > >>   	pinctrl-0 = <&pinctrl_i2c2>;
> > >>   	status = "okay";
> > >> 
> > >> +
> > >> +	ov5640: camera@40 {
> > >> +		compatible = "ovti,ov5640";
> > >> +		pinctrl-names = "default";
> > >> +		pinctrl-0 = <&pinctrl_ov5640>;
> > >> +		clocks = <&mipi_xclk>;
> > >> +		clock-names = "xclk";
> > >> +		reg = <0x40>;
> > >> +		xclk = <22000000>;
> > > 
> > > This is superfluous, you can use clk_get_rate on mipi_xclk.
> > 
> > This property is actually there to tell the driver what to set the
> > rate to, with clk_set_rate(). So you are saying it would be better
> > to set the rate in the device tree and the driver should only
> > retrieve the rate?
> 
> Yes. Given that this is a reference clock input that is constant on a
> given board and never changes during runtime, I think this is the
> correct way. The clock will be fixed rate on most boards, I assume.

I think it's a bit worse than that. The ov5640 and ov5642 drivers should 
retrieve the clock rate and compute register values accordingly (PLL 
configuration parameters for instance, but most probably other values as 
well). Unfortunately, as usual with Omnivision, the lack of public and even 
non-public information results in drivers hardcoding large lists of 
register/value pairs that have been computed for a specific clock frequency. 
The drivers will thus not operate correctly if the clock is running at a 
different rate. Until that can be fixed, the best option is probably to assign 
the rate in the device tree as Philipp proposed, and to use clk_get_rate() in 
the driver to reject any rate other than 22MHz.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:17           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:17 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

On Monday 16 Jan 2017 13:55:23 Philipp Zabel wrote:
> On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
> > On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> > > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> > >> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2
> > >> sensor.
> > >> Both hang off the same i2c2 bus, so they require different (and non-
> > >> default) i2c slave addresses.
> > >> 
> > >> The OV5642 connects to the parallel-bus mux input port on
> > >> ipu1_csi0_mux.
> > >> 
> > >> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> > >> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> > >> 
> > >> Signed-off-by: Steve Longerbeam <steve_longerbeam-nmGgyN9QBj3QT0dZR+AlfA@public.gmane.org>
> > >> ---
> > >> 
> > >>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
> > >>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
> > >>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 ++++++++++++++++++++++
> > >>   3 files changed, 129 insertions(+)
> > >> 
> > >> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> > >> b/arch/arm/boot/dts/imx6dl-sabrelite.dts index 0f06ca5..fec2524 100644
> > >> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> > >> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> 
> [...]
> 
> > >> @@ -299,6 +326,52 @@
> > >> 
> > >>   	pinctrl-names = "default";
> > >>   	pinctrl-0 = <&pinctrl_i2c2>;
> > >>   	status = "okay";
> > >> 
> > >> +
> > >> +	ov5640: camera@40 {
> > >> +		compatible = "ovti,ov5640";
> > >> +		pinctrl-names = "default";
> > >> +		pinctrl-0 = <&pinctrl_ov5640>;
> > >> +		clocks = <&mipi_xclk>;
> > >> +		clock-names = "xclk";
> > >> +		reg = <0x40>;
> > >> +		xclk = <22000000>;
> > > 
> > > This is superfluous, you can use clk_get_rate on mipi_xclk.
> > 
> > This property is actually there to tell the driver what to set the
> > rate to, with clk_set_rate(). So you are saying it would be better
> > to set the rate in the device tree and the driver should only
> > retrieve the rate?
> 
> Yes. Given that this is a reference clock input that is constant on a
> given board and never changes during runtime, I think this is the
> correct way. The clock will be fixed rate on most boards, I assume.

I think it's a bit worse than that. The ov5640 and ov5642 drivers should 
retrieve the clock rate and compute register values accordingly (PLL 
configuration parameters for instance, but most probably other values as 
well). Unfortunately, as usual with Omnivision, the lack of public and even 
non-public information results in drivers hardcoding large lists of 
register/value pairs that have been computed for a specific clock frequency. 
The drivers will thus not operate correctly if the clock is running at a 
different rate. Until that can be fixed, the best option is probably to assign 
the rate in the device tree as Philipp proposed, and to use clk_get_rate() in 
the driver to reject any rate other than 22MHz.

-- 
Regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:17           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:17 UTC (permalink / raw)
  To: linux-arm-kernel

On Monday 16 Jan 2017 13:55:23 Philipp Zabel wrote:
> On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
> > On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> > > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> > >> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2
> > >> sensor.
> > >> Both hang off the same i2c2 bus, so they require different (and non-
> > >> default) i2c slave addresses.
> > >> 
> > >> The OV5642 connects to the parallel-bus mux input port on
> > >> ipu1_csi0_mux.
> > >> 
> > >> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> > >> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> > >> 
> > >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> > >> ---
> > >> 
> > >>   arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
> > >>   arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
> > >>   arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 ++++++++++++++++++++++
> > >>   3 files changed, 129 insertions(+)
> > >> 
> > >> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> > >> b/arch/arm/boot/dts/imx6dl-sabrelite.dts index 0f06ca5..fec2524 100644
> > >> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> > >> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> 
> [...]
> 
> > >> @@ -299,6 +326,52 @@
> > >> 
> > >>   	pinctrl-names = "default";
> > >>   	pinctrl-0 = <&pinctrl_i2c2>;
> > >>   	status = "okay";
> > >> 
> > >> +
> > >> +	ov5640: camera at 40 {
> > >> +		compatible = "ovti,ov5640";
> > >> +		pinctrl-names = "default";
> > >> +		pinctrl-0 = <&pinctrl_ov5640>;
> > >> +		clocks = <&mipi_xclk>;
> > >> +		clock-names = "xclk";
> > >> +		reg = <0x40>;
> > >> +		xclk = <22000000>;
> > > 
> > > This is superfluous, you can use clk_get_rate on mipi_xclk.
> > 
> > This property is actually there to tell the driver what to set the
> > rate to, with clk_set_rate(). So you are saying it would be better
> > to set the rate in the device tree and the driver should only
> > retrieve the rate?
> 
> Yes. Given that this is a reference clock input that is constant on a
> given board and never changes during runtime, I think this is the
> correct way. The clock will be fixed rate on most boards, I assume.

I think it's a bit worse than that. The ov5640 and ov5642 drivers should 
retrieve the clock rate and compute register values accordingly (PLL 
configuration parameters for instance, but most probably other values as 
well). Unfortunately, as usual with Omnivision, the lack of public and even 
non-public information results in drivers hardcoding large lists of 
register/value pairs that have been computed for a specific clock frequency. 
The drivers will thus not operate correctly if the clock is running at a 
different rate. Until that can be fixed, the best option is probably to assign 
the rate in the device tree as Philipp proposed, and to use clk_get_rate() in 
the driver to reject any rate other than 22MHz.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:24       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:24 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Russell,

On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> > +	ov5640: camera@40 {
> > +		compatible = "ovti,ov5640";
> > +		pinctrl-names = "default";
> > +		pinctrl-0 = <&pinctrl_ov5640>;
> > +		clocks = <&mipi_xclk>;
> > +		clock-names = "xclk";
> > +		reg = <0x40>;
> > +		xclk = <22000000>;
> > +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> > +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> > +
> > +		port {
> > +			#address-cells = <1>;
> > +			#size-cells = <0>;
> > +
> > +			ov5640_to_mipi_csi: endpoint@1 {
> > +				reg = <1>;
> > +				remote-endpoint = 
<&mipi_csi_from_mipi_sensor>;
> > +				data-lanes = <0 1>;
> > +				clock-lanes = <2>;
> 
> How do you envision a four-lane sensor being described?
> 
> 	data-lanes = <0 1 3 4>;
> 	clock-lanes = <2>;
> 
> ?
> 
> The binding document for video-interfaces.txt says:
> 
> - clock-lanes: an array of physical clock lane indexes. Position of an entry
> determines the logical lane number, while the value of an entry indicates
> physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes =
> <0>;", which places the clock lane on hardware lane 0. This property is
> valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI
> CSI-2 bus this array contains only one entry.
> 
> So I think you need to have a good reason to make the clock lane non-zero.

The purpose of the data-lanes and clock-lanes properties is to describe lane 
assignment for hardware that supports lane routing. As far as I know the 
OV5640 doesn't support lane routing and has dedicated pins for the clock and 
data lanes. The data-lanes and clock-lanes properties should probably not be 
specified at all.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:24       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:24 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA, devel-gWbeCf7V1WCQmaza687I9g

Hi Russell,

On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> > +	ov5640: camera@40 {
> > +		compatible = "ovti,ov5640";
> > +		pinctrl-names = "default";
> > +		pinctrl-0 = <&pinctrl_ov5640>;
> > +		clocks = <&mipi_xclk>;
> > +		clock-names = "xclk";
> > +		reg = <0x40>;
> > +		xclk = <22000000>;
> > +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> > +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> > +
> > +		port {
> > +			#address-cells = <1>;
> > +			#size-cells = <0>;
> > +
> > +			ov5640_to_mipi_csi: endpoint@1 {
> > +				reg = <1>;
> > +				remote-endpoint = 
<&mipi_csi_from_mipi_sensor>;
> > +				data-lanes = <0 1>;
> > +				clock-lanes = <2>;
> 
> How do you envision a four-lane sensor being described?
> 
> 	data-lanes = <0 1 3 4>;
> 	clock-lanes = <2>;
> 
> ?
> 
> The binding document for video-interfaces.txt says:
> 
> - clock-lanes: an array of physical clock lane indexes. Position of an entry
> determines the logical lane number, while the value of an entry indicates
> physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes =
> <0>;", which places the clock lane on hardware lane 0. This property is
> valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI
> CSI-2 bus this array contains only one entry.
> 
> So I think you need to have a good reason to make the clock lane non-zero.

The purpose of the data-lanes and clock-lanes properties is to describe lane 
assignment for hardware that supports lane routing. As far as I know the 
OV5640 doesn't support lane routing and has dedicated pins for the clock and 
data lanes. The data-lanes and clock-lanes properties should probably not be 
specified at all.

-- 
Regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:24       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:24 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Russell,

On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:24PM -0800, Steve Longerbeam wrote:
> > +	ov5640: camera at 40 {
> > +		compatible = "ovti,ov5640";
> > +		pinctrl-names = "default";
> > +		pinctrl-0 = <&pinctrl_ov5640>;
> > +		clocks = <&mipi_xclk>;
> > +		clock-names = "xclk";
> > +		reg = <0x40>;
> > +		xclk = <22000000>;
> > +		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> > +		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> > +
> > +		port {
> > +			#address-cells = <1>;
> > +			#size-cells = <0>;
> > +
> > +			ov5640_to_mipi_csi: endpoint at 1 {
> > +				reg = <1>;
> > +				remote-endpoint = 
<&mipi_csi_from_mipi_sensor>;
> > +				data-lanes = <0 1>;
> > +				clock-lanes = <2>;
> 
> How do you envision a four-lane sensor being described?
> 
> 	data-lanes = <0 1 3 4>;
> 	clock-lanes = <2>;
> 
> ?
> 
> The binding document for video-interfaces.txt says:
> 
> - clock-lanes: an array of physical clock lane indexes. Position of an entry
> determines the logical lane number, while the value of an entry indicates
> physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes =
> <0>;", which places the clock lane on hardware lane 0. This property is
> valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI
> CSI-2 bus this array contains only one entry.
> 
> So I think you need to have a good reason to make the clock lane non-zero.

The purpose of the data-lanes and clock-lanes properties is to describe lane 
assignment for hardware that supports lane routing. As far as I know the 
OV5640 doesn't support lane routing and has dedicated pins for the clock and 
data lanes. The data-lanes and clock-lanes properties should probably not be 
specified at all.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 12/24] add mux and video interface bridge entity functions
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-02-05 15:36     ` Laurent Pinchart
  -1 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:36 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

Hi Steve,

Thank you for the patch

On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
>  include/uapi/linux/media.h                        |  6 ++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/Documentation/media/uapi/mediactl/media-types.rst
> b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29
> 100644
> --- a/Documentation/media/uapi/mediactl/media-types.rst
> +++ b/Documentation/media/uapi/mediactl/media-types.rst
> @@ -298,6 +298,28 @@ Types and flags used to represent the media graph
> elements received on its sink pad and outputs the statistics data on
>  	  its source pad.
> 
> +    -  ..  row 29
> +
> +       ..  _MEDIA-ENT-F-MUX:
> +
> +       -  ``MEDIA_ENT_F_MUX``
> +
> +       - Video multiplexer. An entity capable of multiplexing must have at
> +         least two sink pads and one source pad, and must pass the video
> +         frame(s) received from the active sink pad to the source pad.
> Video
> +         frame(s) from the inactive sink pads are discarded.

Apart from the comment made by Hans regarding the macro name, this looks good 
to me.

> +    -  ..  row 30
> +
> +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> +
> +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> +
> +       - Video interface bridge. A video interface bridge entity must have
> at
> +         least one sink pad and one source pad. It receives video frame(s)
> on
> +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> +         converts them and outputs them on its source pad in another bus
> format
> +         (eDP, MIPI CSI-2, parallel, ...).


The first sentence mentions *at least* one sink pad and one source pad, and 
the second one refers to "its sink pad" and "its source pad". This is a bit 
confusing. An easy option would be to require a single sink and a single 
source pad, but that would exclude bridges that combine a multiplexer.

I also wonder whether "bridge" is the appropriate word here. Transceiver might 
be a better choice, to insist on the fact that one of the two pads corresponds 
to a physical interface that has special electrical properties (such as HDMI, 
eDP, or CSI-2 that all require PHYs).

>  ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
> 
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 4890787..08a8bfa 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -105,6 +105,12 @@ struct media_device_info {
>  #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
> 
>  /*
> + * Switch and bridge entitites
> + */
> +#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 
0x5001)
> +#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
> +
> +/*
>   * Connectors
>   */
>  /* It is a responsibility of the entity drivers to add connectors and links
> */

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 12/24] add mux and video interface bridge entity functions
@ 2017-02-05 15:36     ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:36 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	kernel, arnd, mchehab, bparrot, robh+dt, horms+renesas,
	tiffany.lin, linux-arm-kernel, niklas.soderlund+renesas, gregkh,
	linux-kernel, jean-christophe.trotin, p.zabel, fabio.estevam,
	shawnguo, sudipm.mukherjee

Hi Steve,

Thank you for the patch

On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
>  include/uapi/linux/media.h                        |  6 ++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/Documentation/media/uapi/mediactl/media-types.rst
> b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29
> 100644
> --- a/Documentation/media/uapi/mediactl/media-types.rst
> +++ b/Documentation/media/uapi/mediactl/media-types.rst
> @@ -298,6 +298,28 @@ Types and flags used to represent the media graph
> elements received on its sink pad and outputs the statistics data on
>  	  its source pad.
> 
> +    -  ..  row 29
> +
> +       ..  _MEDIA-ENT-F-MUX:
> +
> +       -  ``MEDIA_ENT_F_MUX``
> +
> +       - Video multiplexer. An entity capable of multiplexing must have at
> +         least two sink pads and one source pad, and must pass the video
> +         frame(s) received from the active sink pad to the source pad.
> Video
> +         frame(s) from the inactive sink pads are discarded.

Apart from the comment made by Hans regarding the macro name, this looks good 
to me.

> +    -  ..  row 30
> +
> +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> +
> +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> +
> +       - Video interface bridge. A video interface bridge entity must have
> at
> +         least one sink pad and one source pad. It receives video frame(s)
> on
> +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> +         converts them and outputs them on its source pad in another bus
> format
> +         (eDP, MIPI CSI-2, parallel, ...).


The first sentence mentions *at least* one sink pad and one source pad, and 
the second one refers to "its sink pad" and "its source pad". This is a bit 
confusing. An easy option would be to require a single sink and a single 
source pad, but that would exclude bridges that combine a multiplexer.

I also wonder whether "bridge" is the appropriate word here. Transceiver might 
be a better choice, to insist on the fact that one of the two pads corresponds 
to a physical interface that has special electrical properties (such as HDMI, 
eDP, or CSI-2 that all require PHYs).

>  ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
> 
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 4890787..08a8bfa 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -105,6 +105,12 @@ struct media_device_info {
>  #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
> 
>  /*
> + * Switch and bridge entitites
> + */
> +#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 
0x5001)
> +#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
> +
> +/*
>   * Connectors
>   */
>  /* It is a responsibility of the entity drivers to add connectors and links
> */

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 12/24] add mux and video interface bridge entity functions
@ 2017-02-05 15:36     ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:36 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

Thank you for the patch

On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
>  include/uapi/linux/media.h                        |  6 ++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/Documentation/media/uapi/mediactl/media-types.rst
> b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29
> 100644
> --- a/Documentation/media/uapi/mediactl/media-types.rst
> +++ b/Documentation/media/uapi/mediactl/media-types.rst
> @@ -298,6 +298,28 @@ Types and flags used to represent the media graph
> elements received on its sink pad and outputs the statistics data on
>  	  its source pad.
> 
> +    -  ..  row 29
> +
> +       ..  _MEDIA-ENT-F-MUX:
> +
> +       -  ``MEDIA_ENT_F_MUX``
> +
> +       - Video multiplexer. An entity capable of multiplexing must have at
> +         least two sink pads and one source pad, and must pass the video
> +         frame(s) received from the active sink pad to the source pad.
> Video
> +         frame(s) from the inactive sink pads are discarded.

Apart from the comment made by Hans regarding the macro name, this looks good 
to me.

> +    -  ..  row 30
> +
> +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> +
> +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> +
> +       - Video interface bridge. A video interface bridge entity must have
> at
> +         least one sink pad and one source pad. It receives video frame(s)
> on
> +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> +         converts them and outputs them on its source pad in another bus
> format
> +         (eDP, MIPI CSI-2, parallel, ...).


The first sentence mentions *at least* one sink pad and one source pad, and 
the second one refers to "its sink pad" and "its source pad". This is a bit 
confusing. An easy option would be to require a single sink and a single 
source pad, but that would exclude bridges that combine a multiplexer.

I also wonder whether "bridge" is the appropriate word here. Transceiver might 
be a better choice, to insist on the fact that one of the two pads corresponds 
to a physical interface that has special electrical properties (such as HDMI, 
eDP, or CSI-2 that all require PHYs).

>  ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
> 
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 4890787..08a8bfa 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -105,6 +105,12 @@ struct media_device_info {
>  #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
> 
>  /*
> + * Switch and bridge entitites
> + */
> +#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 
0x5001)
> +#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
> +
> +/*
>   * Connectors
>   */
>  /* It is a responsibility of the entity drivers to add connectors and links
> */

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
  2017-02-05 15:24       ` Laurent Pinchart
  (?)
@ 2017-02-05 15:37         ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-05 15:37 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Sun, Feb 05, 2017 at 05:24:30PM +0200, Laurent Pinchart wrote:
> Hi Russell,
> 
> On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:
> > > +			ov5640_to_mipi_csi: endpoint@1 {
> > > +				reg = <1>;
> > > +				remote-endpoint = 
> <&mipi_csi_from_mipi_sensor>;
> > > +				data-lanes = <0 1>;
> > > +				clock-lanes = <2>;
> > 
> > How do you envision a four-lane sensor being described?
> > 
> > 	data-lanes = <0 1 3 4>;
> > 	clock-lanes = <2>;
> > 
> > ?
> > 
> > The binding document for video-interfaces.txt says:
> > 
> > - clock-lanes: an array of physical clock lane indexes. Position of an entry
> > determines the logical lane number, while the value of an entry indicates
> > physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes =
> > <0>;", which places the clock lane on hardware lane 0. This property is
> > valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI
> > CSI-2 bus this array contains only one entry.
> > 
> > So I think you need to have a good reason to make the clock lane non-zero.
> 
> The purpose of the data-lanes and clock-lanes properties is to describe lane 
> assignment for hardware that supports lane routing. As far as I know the 
> OV5640 doesn't support lane routing and has dedicated pins for the clock and 
> data lanes. The data-lanes and clock-lanes properties should probably not be 
> specified at all.

You need at least data-lanes so you know how many data lanes are wired
between the camera and the mipi csi2 receiver.  Just because a camera
has (eg) four lanes does not mean you need to wire all four, and in
some cases less than four will be wired.

If data-lanes does not describe that, then all existing users of the
binding are abusing it:

$ grep data_lanes drivers/media/i2c -r
drivers/media/i2c/s5k5baf.c:                state->nlanes = ep.bus.mipi_csi2.num_data_lanes;
drivers/media/i2c/s5c73m3/s5c73m3-core.c:   if (ep.bus.mipi_csi2.num_data_lanes != S5C73M3_MIPI_DATA_LANES)
drivers/media/i2c/tc358743.c:           endpoint->bus.mipi_csi2.num_data_lanes == 0 ||
drivers/media/i2c/smiapp/smiapp-core.c:     hwcfg->lanes = bus_cfg->bus.mipi_csi2.num_data_lanes;

So all those drivers are using it for the _number_ of CSI2 lanes, and
are not touching the mapping in any way (not even checking that it is
an identity mapping.)  You could specify any mapping to these drivers,
as long as num_data_lanes came out right.

And... there's no point having a property in a binding if no one is
using it... and even more silly not to have a property that everyone
needs...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:37         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-05 15:37 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA, devel-gWbeCf7V1WCQmaza687I9g

On Sun, Feb 05, 2017 at 05:24:30PM +0200, Laurent Pinchart wrote:
> Hi Russell,
> 
> On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:
> > > +			ov5640_to_mipi_csi: endpoint@1 {
> > > +				reg = <1>;
> > > +				remote-endpoint = 
> <&mipi_csi_from_mipi_sensor>;
> > > +				data-lanes = <0 1>;
> > > +				clock-lanes = <2>;
> > 
> > How do you envision a four-lane sensor being described?
> > 
> > 	data-lanes = <0 1 3 4>;
> > 	clock-lanes = <2>;
> > 
> > ?
> > 
> > The binding document for video-interfaces.txt says:
> > 
> > - clock-lanes: an array of physical clock lane indexes. Position of an entry
> > determines the logical lane number, while the value of an entry indicates
> > physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes =
> > <0>;", which places the clock lane on hardware lane 0. This property is
> > valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI
> > CSI-2 bus this array contains only one entry.
> > 
> > So I think you need to have a good reason to make the clock lane non-zero.
> 
> The purpose of the data-lanes and clock-lanes properties is to describe lane 
> assignment for hardware that supports lane routing. As far as I know the 
> OV5640 doesn't support lane routing and has dedicated pins for the clock and 
> data lanes. The data-lanes and clock-lanes properties should probably not be 
> specified at all.

You need at least data-lanes so you know how many data lanes are wired
between the camera and the mipi csi2 receiver.  Just because a camera
has (eg) four lanes does not mean you need to wire all four, and in
some cases less than four will be wired.

If data-lanes does not describe that, then all existing users of the
binding are abusing it:

$ grep data_lanes drivers/media/i2c -r
drivers/media/i2c/s5k5baf.c:                state->nlanes = ep.bus.mipi_csi2.num_data_lanes;
drivers/media/i2c/s5c73m3/s5c73m3-core.c:   if (ep.bus.mipi_csi2.num_data_lanes != S5C73M3_MIPI_DATA_LANES)
drivers/media/i2c/tc358743.c:           endpoint->bus.mipi_csi2.num_data_lanes == 0 ||
drivers/media/i2c/smiapp/smiapp-core.c:     hwcfg->lanes = bus_cfg->bus.mipi_csi2.num_data_lanes;

So all those drivers are using it for the _number_ of CSI2 lanes, and
are not touching the mapping in any way (not even checking that it is
an identity mapping.)  You could specify any mapping to these drivers,
as long as num_data_lanes came out right.

And... there's no point having a property in a binding if no one is
using it... and even more silly not to have a property that everyone
needs...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
@ 2017-02-05 15:37         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-05 15:37 UTC (permalink / raw)
  To: linux-arm-kernel

On Sun, Feb 05, 2017 at 05:24:30PM +0200, Laurent Pinchart wrote:
> Hi Russell,
> 
> On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:
> > > +			ov5640_to_mipi_csi: endpoint at 1 {
> > > +				reg = <1>;
> > > +				remote-endpoint = 
> <&mipi_csi_from_mipi_sensor>;
> > > +				data-lanes = <0 1>;
> > > +				clock-lanes = <2>;
> > 
> > How do you envision a four-lane sensor being described?
> > 
> > 	data-lanes = <0 1 3 4>;
> > 	clock-lanes = <2>;
> > 
> > ?
> > 
> > The binding document for video-interfaces.txt says:
> > 
> > - clock-lanes: an array of physical clock lane indexes. Position of an entry
> > determines the logical lane number, while the value of an entry indicates
> > physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes =
> > <0>;", which places the clock lane on hardware lane 0. This property is
> > valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI
> > CSI-2 bus this array contains only one entry.
> > 
> > So I think you need to have a good reason to make the clock lane non-zero.
> 
> The purpose of the data-lanes and clock-lanes properties is to describe lane 
> assignment for hardware that supports lane routing. As far as I know the 
> OV5640 doesn't support lane routing and has dedicated pins for the clock and 
> data lanes. The data-lanes and clock-lanes properties should probably not be 
> specified at all.

You need at least data-lanes so you know how many data lanes are wired
between the camera and the mipi csi2 receiver.  Just because a camera
has (eg) four lanes does not mean you need to wire all four, and in
some cases less than four will be wired.

If data-lanes does not describe that, then all existing users of the
binding are abusing it:

$ grep data_lanes drivers/media/i2c -r
drivers/media/i2c/s5k5baf.c:                state->nlanes = ep.bus.mipi_csi2.num_data_lanes;
drivers/media/i2c/s5c73m3/s5c73m3-core.c:   if (ep.bus.mipi_csi2.num_data_lanes != S5C73M3_MIPI_DATA_LANES)
drivers/media/i2c/tc358743.c:           endpoint->bus.mipi_csi2.num_data_lanes == 0 ||
drivers/media/i2c/smiapp/smiapp-core.c:     hwcfg->lanes = bus_cfg->bus.mipi_csi2.num_data_lanes;

So all those drivers are using it for the _number_ of CSI2 lanes, and
are not touching the mapping in any way (not even checking that it is
an identity mapping.)  You could specify any mapping to these drivers,
as long as num_data_lanes came out right.

And... there's no point having a property in a binding if no one is
using it... and even more silly not to have a property that everyone
needs...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-05 15:48           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:48 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Philipp Zabel, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam

Hi Steve,

On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>> +
> >>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config
> >>> *cfg)
> >>> +{
> >>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> >>> +	struct media_pad *pad;
> >>> +	int ret;
> >>> +
> >>> +	if (vidsw->active == -1) {
> >>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
> >>> +		return -EINVAL;
> >>> +	}
> >>> +
> >>> +	/*
> >>> +	 * Retrieve media bus configuration from the entity connected to the
> >>> +	 * active input
> >>> +	 */
> >>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> >>> +	if (pad) {
> >>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> >>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> >>> +		if (ret == -ENOIOCTLCMD)
> >>> +			pad = NULL;
> >>> +		else if (ret < 0) {
> >>> +			dev_err(sd->dev, "failed to get source 
configuration\n");
> >>> +			return ret;
> >>> +		}
> >>> +	}
> >>> +	if (!pad) {
> >>> +		/* Mirror the input side on the output side */
> >>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> >>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> >>> +		    cfg->type == V4L2_MBUS_BT656)
> >>> +			cfg->flags = vidsw->endpoint[vidsw-
>active].bus.parallel.flags;
> >>> +	}
> >>> +
> >>> +	return 0;
> >>> +}
> >> 
> >> I am not certain this op is needed at all. In the current kernel this op
> >> is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious).
> >> Normally this information should come from the device tree and there
> >> should be no need for this op.
> >> 
> >> My (tentative) long-term plan was to get rid of this op.
> >> 
> >> If you don't need it, then I recommend it is removed.
> 
> Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
> sensor, and it was doing that to determine the sensor's bus type. This info
> was already available from parsing a v4l2_of_endpoint from the sensor node.
> So it was simple to remove the g_mbus_config calls, and instead rely on the
> parsed sensor v4l2_of_endpoint.

That's not a good point. The imx-media driver must not parse the sensor DT 
node as it is not aware of what bindings the sensor is compatible with. 
Information must instead be queried from the sensor subdev at runtime, through 
the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's 
certainly an option. It's only information provided by the sensor driver that 
you have no choice but query using a subdev operation.

> > We currently use this to make the CSI capture interface understand
> > whether its source from the MIPI CSI-2 or from the parallel bus. That is
> > probably something that should be fixed, but I'm not quite sure how.
> > 
> > The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
> > CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
> > MIPI specific metadata (virtual channel, data type).
> > 
> > Then the CSI2IPU gasket turns this input bus into four separate parallel
> > 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
> > carries the MIPI metadata. The incoming data is split into the four
> > outputs according to the MIPI virtual channel.
> > 
> > Two of these 16-bit + 8-bit parallel buses are routed through a
> > multiplexer before finally arriving at the CSI on the other side.
> > 
> > We need to configure the CSI to either use or ignore the data from the
> > 8-bit mct_di bus depending on whether the source of the mux is
> > configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
> > input.
> 
> Philipp, from my experience, the CSI_MIPI_DI register (configured
> by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0,
> otherwise no data is received from the MIPI CSI-2 sensor, regardless
> of the virtual channel the sensor is transmitting over.
> 
> So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU
> gasket are ignored by the CSI's, at least the virtual channel number is
> ignored.
> 
> For example, if the sensor is transmitting on vc 1, the gasket routes
> the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1
> is written with vc 1, no data is received.
> 
> Steve
> 
> > Currently we let g_mbus_config pretend that even the internal 32-bit +
> > metadata and 16-bit + 8-bit metadata parallel buses are of type
> > V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
> > CSI-2 receiver, if connected.
> > 
> > Without g_mbus_config we'd need to get that information from somewhere
> > else. One possibility would be to extend MEDIA_BUS formats to describe
> > these "parallelized MIPI data" buses separately.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-05 15:48           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:48 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Philipp Zabel, Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA, devel

Hi Steve,

On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>> +
> >>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config
> >>> *cfg)
> >>> +{
> >>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> >>> +	struct media_pad *pad;
> >>> +	int ret;
> >>> +
> >>> +	if (vidsw->active == -1) {
> >>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
> >>> +		return -EINVAL;
> >>> +	}
> >>> +
> >>> +	/*
> >>> +	 * Retrieve media bus configuration from the entity connected to the
> >>> +	 * active input
> >>> +	 */
> >>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> >>> +	if (pad) {
> >>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> >>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> >>> +		if (ret == -ENOIOCTLCMD)
> >>> +			pad = NULL;
> >>> +		else if (ret < 0) {
> >>> +			dev_err(sd->dev, "failed to get source 
configuration\n");
> >>> +			return ret;
> >>> +		}
> >>> +	}
> >>> +	if (!pad) {
> >>> +		/* Mirror the input side on the output side */
> >>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> >>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> >>> +		    cfg->type == V4L2_MBUS_BT656)
> >>> +			cfg->flags = vidsw->endpoint[vidsw-
>active].bus.parallel.flags;
> >>> +	}
> >>> +
> >>> +	return 0;
> >>> +}
> >> 
> >> I am not certain this op is needed at all. In the current kernel this op
> >> is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious).
> >> Normally this information should come from the device tree and there
> >> should be no need for this op.
> >> 
> >> My (tentative) long-term plan was to get rid of this op.
> >> 
> >> If you don't need it, then I recommend it is removed.
> 
> Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
> sensor, and it was doing that to determine the sensor's bus type. This info
> was already available from parsing a v4l2_of_endpoint from the sensor node.
> So it was simple to remove the g_mbus_config calls, and instead rely on the
> parsed sensor v4l2_of_endpoint.

That's not a good point. The imx-media driver must not parse the sensor DT 
node as it is not aware of what bindings the sensor is compatible with. 
Information must instead be queried from the sensor subdev at runtime, through 
the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's 
certainly an option. It's only information provided by the sensor driver that 
you have no choice but query using a subdev operation.

> > We currently use this to make the CSI capture interface understand
> > whether its source from the MIPI CSI-2 or from the parallel bus. That is
> > probably something that should be fixed, but I'm not quite sure how.
> > 
> > The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
> > CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
> > MIPI specific metadata (virtual channel, data type).
> > 
> > Then the CSI2IPU gasket turns this input bus into four separate parallel
> > 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
> > carries the MIPI metadata. The incoming data is split into the four
> > outputs according to the MIPI virtual channel.
> > 
> > Two of these 16-bit + 8-bit parallel buses are routed through a
> > multiplexer before finally arriving at the CSI on the other side.
> > 
> > We need to configure the CSI to either use or ignore the data from the
> > 8-bit mct_di bus depending on whether the source of the mux is
> > configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
> > input.
> 
> Philipp, from my experience, the CSI_MIPI_DI register (configured
> by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0,
> otherwise no data is received from the MIPI CSI-2 sensor, regardless
> of the virtual channel the sensor is transmitting over.
> 
> So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU
> gasket are ignored by the CSI's, at least the virtual channel number is
> ignored.
> 
> For example, if the sensor is transmitting on vc 1, the gasket routes
> the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1
> is written with vc 1, no data is received.
> 
> Steve
> 
> > Currently we let g_mbus_config pretend that even the internal 32-bit +
> > metadata and 16-bit + 8-bit metadata parallel buses are of type
> > V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
> > CSI-2 receiver, if connected.
> > 
> > Without g_mbus_config we'd need to get that information from somewhere
> > else. One possibility would be to extend MEDIA_BUS formats to describe
> > these "parallelized MIPI data" buses separately.

-- 
Regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-05 15:48           ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:48 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>> +
> >>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config
> >>> *cfg)
> >>> +{
> >>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> >>> +	struct media_pad *pad;
> >>> +	int ret;
> >>> +
> >>> +	if (vidsw->active == -1) {
> >>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
> >>> +		return -EINVAL;
> >>> +	}
> >>> +
> >>> +	/*
> >>> +	 * Retrieve media bus configuration from the entity connected to the
> >>> +	 * active input
> >>> +	 */
> >>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> >>> +	if (pad) {
> >>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> >>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> >>> +		if (ret == -ENOIOCTLCMD)
> >>> +			pad = NULL;
> >>> +		else if (ret < 0) {
> >>> +			dev_err(sd->dev, "failed to get source 
configuration\n");
> >>> +			return ret;
> >>> +		}
> >>> +	}
> >>> +	if (!pad) {
> >>> +		/* Mirror the input side on the output side */
> >>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> >>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> >>> +		    cfg->type == V4L2_MBUS_BT656)
> >>> +			cfg->flags = vidsw->endpoint[vidsw-
>active].bus.parallel.flags;
> >>> +	}
> >>> +
> >>> +	return 0;
> >>> +}
> >> 
> >> I am not certain this op is needed at all. In the current kernel this op
> >> is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious).
> >> Normally this information should come from the device tree and there
> >> should be no need for this op.
> >> 
> >> My (tentative) long-term plan was to get rid of this op.
> >> 
> >> If you don't need it, then I recommend it is removed.
> 
> Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
> sensor, and it was doing that to determine the sensor's bus type. This info
> was already available from parsing a v4l2_of_endpoint from the sensor node.
> So it was simple to remove the g_mbus_config calls, and instead rely on the
> parsed sensor v4l2_of_endpoint.

That's not a good point. The imx-media driver must not parse the sensor DT 
node as it is not aware of what bindings the sensor is compatible with. 
Information must instead be queried from the sensor subdev at runtime, through 
the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's 
certainly an option. It's only information provided by the sensor driver that 
you have no choice but query using a subdev operation.

> > We currently use this to make the CSI capture interface understand
> > whether its source from the MIPI CSI-2 or from the parallel bus. That is
> > probably something that should be fixed, but I'm not quite sure how.
> > 
> > The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI
> > CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the
> > MIPI specific metadata (virtual channel, data type).
> > 
> > Then the CSI2IPU gasket turns this input bus into four separate parallel
> > 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that
> > carries the MIPI metadata. The incoming data is split into the four
> > outputs according to the MIPI virtual channel.
> > 
> > Two of these 16-bit + 8-bit parallel buses are routed through a
> > multiplexer before finally arriving at the CSI on the other side.
> > 
> > We need to configure the CSI to either use or ignore the data from the
> > 8-bit mct_di bus depending on whether the source of the mux is
> > configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel
> > input.
> 
> Philipp, from my experience, the CSI_MIPI_DI register (configured
> by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0,
> otherwise no data is received from the MIPI CSI-2 sensor, regardless
> of the virtual channel the sensor is transmitting over.
> 
> So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU
> gasket are ignored by the CSI's, at least the virtual channel number is
> ignored.
> 
> For example, if the sensor is transmitting on vc 1, the gasket routes
> the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1
> is written with vc 1, no data is received.
> 
> Steve
> 
> > Currently we let g_mbus_config pretend that even the internal 32-bit +
> > metadata and 16-bit + 8-bit metadata parallel buses are of type
> > V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the
> > CSI-2 receiver, if connected.
> > 
> > Without g_mbus_config we'd need to get that information from somewhere
> > else. One possibility would be to extend MEDIA_BUS formats to describe
> > these "parallelized MIPI data" buses separately.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 15/24] media: Add userspace header file for i.MX
  2017-01-13 23:13       ` Steve Longerbeam
  (?)
@ 2017-02-05 15:50         ` Laurent Pinchart
  -1 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:50 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Friday 13 Jan 2017 15:13:33 Steve Longerbeam wrote:
> On 01/13/2017 04:05 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> This adds a header file for use by userspace programs wanting to interact
> >> with the i.MX media driver. It defines custom v4l2 controls and events
> >> generated by the i.MX v4l2 subdevices.
> >> 
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >> 
> >>   include/uapi/media/Kbuild |  1 +
> >>   include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
> >>   2 files changed, 31 insertions(+)
> >>   create mode 100644 include/uapi/media/imx.h
> >> 
> >> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
> >> index aafaa5a..fa78958 100644
> >> --- a/include/uapi/media/Kbuild
> >> +++ b/include/uapi/media/Kbuild
> >> @@ -1 +1,2 @@
> >> 
> >>   # UAPI Header export list
> >> 
> >> +header-y += imx.h
> >> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
> >> new file mode 100644
> >> index 0000000..2421d9c
> >> --- /dev/null
> >> +++ b/include/uapi/media/imx.h
> >> @@ -0,0 +1,30 @@
> >> +/*
> >> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License as published by
> >> the + * Free Software Foundation; either version 2 of the
> >> + * License, or (at your option) any later version
> >> + */
> >> +
> >> +#ifndef __UAPI_MEDIA_IMX_H__
> >> +#define __UAPI_MEDIA_IMX_H__
> >> +
> >> +/*
> >> + * events from the subdevs
> >> + */
> >> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
> >> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
> >> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
> >> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
> > 
> > Aren't these generic enough to warrant common events? I would think
> > there have to be other capture IP cores that can signal aborted frames
> > or frame timeouts.
> 
> Yes, agreed. A frame capture timeout, or frame interval error, are
> both generic concepts. At some point it would be great to make the
> Frame Interval Monitor generally available under v4l2-core. As for the
> EOF timeout event, I'll look into moving that into a generic V4L2 event.

I'd prefer generic events if possible, but regardless of whether that's 
possible, the events must be documented in the V4L2 specification.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-02-05 15:50         ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:50 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	kernel, arnd, mchehab, bparrot, robh+dt, horms+renesas,
	tiffany.lin, linux-arm-kernel, niklas.soderlund+renesas, gregkh,
	linux-kernel, jean-christophe.trotin, Philipp Zabel,
	fabio.estevam, shawnguo, sudipm.mukherjee

Hi Steve,

On Friday 13 Jan 2017 15:13:33 Steve Longerbeam wrote:
> On 01/13/2017 04:05 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> This adds a header file for use by userspace programs wanting to interact
> >> with the i.MX media driver. It defines custom v4l2 controls and events
> >> generated by the i.MX v4l2 subdevices.
> >> 
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >> 
> >>   include/uapi/media/Kbuild |  1 +
> >>   include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
> >>   2 files changed, 31 insertions(+)
> >>   create mode 100644 include/uapi/media/imx.h
> >> 
> >> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
> >> index aafaa5a..fa78958 100644
> >> --- a/include/uapi/media/Kbuild
> >> +++ b/include/uapi/media/Kbuild
> >> @@ -1 +1,2 @@
> >> 
> >>   # UAPI Header export list
> >> 
> >> +header-y += imx.h
> >> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
> >> new file mode 100644
> >> index 0000000..2421d9c
> >> --- /dev/null
> >> +++ b/include/uapi/media/imx.h
> >> @@ -0,0 +1,30 @@
> >> +/*
> >> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License as published by
> >> the + * Free Software Foundation; either version 2 of the
> >> + * License, or (at your option) any later version
> >> + */
> >> +
> >> +#ifndef __UAPI_MEDIA_IMX_H__
> >> +#define __UAPI_MEDIA_IMX_H__
> >> +
> >> +/*
> >> + * events from the subdevs
> >> + */
> >> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
> >> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
> >> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
> >> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
> > 
> > Aren't these generic enough to warrant common events? I would think
> > there have to be other capture IP cores that can signal aborted frames
> > or frame timeouts.
> 
> Yes, agreed. A frame capture timeout, or frame interval error, are
> both generic concepts. At some point it would be great to make the
> Frame Interval Monitor generally available under v4l2-core. As for the
> EOF timeout event, I'll look into moving that into a generic V4L2 event.

I'd prefer generic events if possible, but regardless of whether that's 
possible, the events must be documented in the V4L2 specification.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 15/24] media: Add userspace header file for i.MX
@ 2017-02-05 15:50         ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-05 15:50 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Friday 13 Jan 2017 15:13:33 Steve Longerbeam wrote:
> On 01/13/2017 04:05 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> This adds a header file for use by userspace programs wanting to interact
> >> with the i.MX media driver. It defines custom v4l2 controls and events
> >> generated by the i.MX v4l2 subdevices.
> >> 
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >> 
> >>   include/uapi/media/Kbuild |  1 +
> >>   include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
> >>   2 files changed, 31 insertions(+)
> >>   create mode 100644 include/uapi/media/imx.h
> >> 
> >> diff --git a/include/uapi/media/Kbuild b/include/uapi/media/Kbuild
> >> index aafaa5a..fa78958 100644
> >> --- a/include/uapi/media/Kbuild
> >> +++ b/include/uapi/media/Kbuild
> >> @@ -1 +1,2 @@
> >> 
> >>   # UAPI Header export list
> >> 
> >> +header-y += imx.h
> >> diff --git a/include/uapi/media/imx.h b/include/uapi/media/imx.h
> >> new file mode 100644
> >> index 0000000..2421d9c
> >> --- /dev/null
> >> +++ b/include/uapi/media/imx.h
> >> @@ -0,0 +1,30 @@
> >> +/*
> >> + * Copyright (c) 2014-2015 Mentor Graphics Inc.
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License as published by
> >> the + * Free Software Foundation; either version 2 of the
> >> + * License, or (at your option) any later version
> >> + */
> >> +
> >> +#ifndef __UAPI_MEDIA_IMX_H__
> >> +#define __UAPI_MEDIA_IMX_H__
> >> +
> >> +/*
> >> + * events from the subdevs
> >> + */
> >> +#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
> >> +#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
> >> +#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
> >> +#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
> > 
> > Aren't these generic enough to warrant common events? I would think
> > there have to be other capture IP cores that can signal aborted frames
> > or frame timeouts.
> 
> Yes, agreed. A frame capture timeout, or frame interval error, are
> both generic concepts. At some point it would be great to make the
> Frame Interval Monitor generally available under v4l2-core. As for the
> EOF timeout event, I'll look into moving that into a generic V4L2 event.

I'd prefer generic events if possible, but regardless of whether that's 
possible, the events must be documented in the V4L2 specification.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-05 15:48           ` Laurent Pinchart
  (?)
@ 2017-02-06  9:50             ` Hans Verkuil
  -1 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-02-06  9:50 UTC (permalink / raw)
  To: Laurent Pinchart, Steve Longerbeam
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam

On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> Hi Steve,
> 
> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>>>>> +
>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config
>>>>> *cfg)
>>>>> +{
>>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>>>> +	struct media_pad *pad;
>>>>> +	int ret;
>>>>> +
>>>>> +	if (vidsw->active == -1) {
>>>>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
>>>>> +		return -EINVAL;
>>>>> +	}
>>>>> +
>>>>> +	/*
>>>>> +	 * Retrieve media bus configuration from the entity connected to the
>>>>> +	 * active input
>>>>> +	 */
>>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>>>> +	if (pad) {
>>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>>>> +		if (ret == -ENOIOCTLCMD)
>>>>> +			pad = NULL;
>>>>> +		else if (ret < 0) {
>>>>> +			dev_err(sd->dev, "failed to get source 
> configuration\n");
>>>>> +			return ret;
>>>>> +		}
>>>>> +	}
>>>>> +	if (!pad) {
>>>>> +		/* Mirror the input side on the output side */
>>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>>>> +		    cfg->type == V4L2_MBUS_BT656)
>>>>> +			cfg->flags = vidsw->endpoint[vidsw-
>> active].bus.parallel.flags;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>
>>>> I am not certain this op is needed at all. In the current kernel this op
>>>> is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious).
>>>> Normally this information should come from the device tree and there
>>>> should be no need for this op.
>>>>
>>>> My (tentative) long-term plan was to get rid of this op.
>>>>
>>>> If you don't need it, then I recommend it is removed.
>>
>> Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
>> sensor, and it was doing that to determine the sensor's bus type. This info
>> was already available from parsing a v4l2_of_endpoint from the sensor node.
>> So it was simple to remove the g_mbus_config calls, and instead rely on the
>> parsed sensor v4l2_of_endpoint.
> 
> That's not a good point. The imx-media driver must not parse the sensor DT 
> node as it is not aware of what bindings the sensor is compatible with. 
> Information must instead be queried from the sensor subdev at runtime, through 
> the g_mbus_config() operation.
> 
> Of course, if you can get the information from the imx-media DT node, that's 
> certainly an option. It's only information provided by the sensor driver that 
> you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

The reason I am suspicious about this op is that it came from soc-camera and
predates the DT. The contents of v4l2_mbus_config seems very much like a HW
description to me, i.e. something that belongs in the DT.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-06  9:50             ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-02-06  9:50 UTC (permalink / raw)
  To: Laurent Pinchart, Steve Longerbeam
  Cc: Philipp Zabel, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> Hi Steve,
> 
> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>>>>> +
>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config
>>>>> *cfg)
>>>>> +{
>>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>>>> +	struct media_pad *pad;
>>>>> +	int ret;
>>>>> +
>>>>> +	if (vidsw->active == -1) {
>>>>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
>>>>> +		return -EINVAL;
>>>>> +	}
>>>>> +
>>>>> +	/*
>>>>> +	 * Retrieve media bus configuration from the entity connected to the
>>>>> +	 * active input
>>>>> +	 */
>>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>>>> +	if (pad) {
>>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>>>> +		if (ret == -ENOIOCTLCMD)
>>>>> +			pad = NULL;
>>>>> +		else if (ret < 0) {
>>>>> +			dev_err(sd->dev, "failed to get source 
> configuration\n");
>>>>> +			return ret;
>>>>> +		}
>>>>> +	}
>>>>> +	if (!pad) {
>>>>> +		/* Mirror the input side on the output side */
>>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>>>> +		    cfg->type == V4L2_MBUS_BT656)
>>>>> +			cfg->flags = vidsw->endpoint[vidsw-
>> active].bus.parallel.flags;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>
>>>> I am not certain this op is needed at all. In the current kernel this op
>>>> is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious).
>>>> Normally this information should come from the device tree and there
>>>> should be no need for this op.
>>>>
>>>> My (tentative) long-term plan was to get rid of this op.
>>>>
>>>> If you don't need it, then I recommend it is removed.
>>
>> Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
>> sensor, and it was doing that to determine the sensor's bus type. This info
>> was already available from parsing a v4l2_of_endpoint from the sensor node.
>> So it was simple to remove the g_mbus_config calls, and instead rely on the
>> parsed sensor v4l2_of_endpoint.
> 
> That's not a good point. The imx-media driver must not parse the sensor DT 
> node as it is not aware of what bindings the sensor is compatible with. 
> Information must instead be queried from the sensor subdev at runtime, through 
> the g_mbus_config() operation.
> 
> Of course, if you can get the information from the imx-media DT node, that's 
> certainly an option. It's only information provided by the sensor driver that 
> you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

The reason I am suspicious about this op is that it came from soc-camera and
predates the DT. The contents of v4l2_mbus_config seems very much like a HW
description to me, i.e. something that belongs in the DT.

Regards,

	Hans
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-06  9:50             ` Hans Verkuil
  0 siblings, 0 replies; 549+ messages in thread
From: Hans Verkuil @ 2017-02-06  9:50 UTC (permalink / raw)
  To: linux-arm-kernel

On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> Hi Steve,
> 
> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>>>>> +
>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config
>>>>> *cfg)
>>>>> +{
>>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>>>> +	struct media_pad *pad;
>>>>> +	int ret;
>>>>> +
>>>>> +	if (vidsw->active == -1) {
>>>>> +		dev_err(sd->dev, "no configuration for inactive mux\n");
>>>>> +		return -EINVAL;
>>>>> +	}
>>>>> +
>>>>> +	/*
>>>>> +	 * Retrieve media bus configuration from the entity connected to the
>>>>> +	 * active input
>>>>> +	 */
>>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>>>> +	if (pad) {
>>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>>>> +		if (ret == -ENOIOCTLCMD)
>>>>> +			pad = NULL;
>>>>> +		else if (ret < 0) {
>>>>> +			dev_err(sd->dev, "failed to get source 
> configuration\n");
>>>>> +			return ret;
>>>>> +		}
>>>>> +	}
>>>>> +	if (!pad) {
>>>>> +		/* Mirror the input side on the output side */
>>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>>>> +		    cfg->type == V4L2_MBUS_BT656)
>>>>> +			cfg->flags = vidsw->endpoint[vidsw-
>> active].bus.parallel.flags;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>
>>>> I am not certain this op is needed at all. In the current kernel this op
>>>> is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious).
>>>> Normally this information should come from the device tree and there
>>>> should be no need for this op.
>>>>
>>>> My (tentative) long-term plan was to get rid of this op.
>>>>
>>>> If you don't need it, then I recommend it is removed.
>>
>> Hi Hans, the imx-media driver was only calling g_mbus_config to the camera
>> sensor, and it was doing that to determine the sensor's bus type. This info
>> was already available from parsing a v4l2_of_endpoint from the sensor node.
>> So it was simple to remove the g_mbus_config calls, and instead rely on the
>> parsed sensor v4l2_of_endpoint.
> 
> That's not a good point. The imx-media driver must not parse the sensor DT 
> node as it is not aware of what bindings the sensor is compatible with. 
> Information must instead be queried from the sensor subdev at runtime, through 
> the g_mbus_config() operation.
> 
> Of course, if you can get the information from the imx-media DT node, that's 
> certainly an option. It's only information provided by the sensor driver that 
> you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

The reason I am suspicious about this op is that it came from soc-camera and
predates the DT. The contents of v4l2_mbus_config seems very much like a HW
description to me, i.e. something that belongs in the DT.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 12/24] add mux and video interface bridge entity functions
  2017-02-05 15:36     ` Laurent Pinchart
  (?)
@ 2017-02-06 10:27       ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-06 10:27 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

On Sun, 2017-02-05 at 17:36 +0200, Laurent Pinchart wrote:
> Hi Steve,
> 
> Thank you for the patch
> 
> On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > ---
> >  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
> >  include/uapi/linux/media.h                        |  6 ++++++
> >  2 files changed, 28 insertions(+)
> > 
> > diff --git a/Documentation/media/uapi/mediactl/media-types.rst
> > b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29
> > 100644
> > --- a/Documentation/media/uapi/mediactl/media-types.rst
> > +++ b/Documentation/media/uapi/mediactl/media-types.rst
> > @@ -298,6 +298,28 @@ Types and flags used to represent the media graph
> > elements received on its sink pad and outputs the statistics data on
> >  	  its source pad.
> > 
> > +    -  ..  row 29
> > +
> > +       ..  _MEDIA-ENT-F-MUX:
> > +
> > +       -  ``MEDIA_ENT_F_MUX``
> > +
> > +       - Video multiplexer. An entity capable of multiplexing must have at
> > +         least two sink pads and one source pad, and must pass the video
> > +         frame(s) received from the active sink pad to the source pad.
> > Video
> > +         frame(s) from the inactive sink pads are discarded.
> 
> Apart from the comment made by Hans regarding the macro name, this looks good 
> to me.
> 
> > +    -  ..  row 30
> > +
> > +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> > +
> > +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> > +
> > +       - Video interface bridge. A video interface bridge entity must have
> > at
> > +         least one sink pad and one source pad. It receives video frame(s)
> > on
> > +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> > +         converts them and outputs them on its source pad in another bus
> > format
> > +         (eDP, MIPI CSI-2, parallel, ...).
> 
> 
> The first sentence mentions *at least* one sink pad and one source pad, and 
> the second one refers to "its sink pad" and "its source pad". This is a bit 
> confusing. An easy option would be to require a single sink and a single 
> source pad, but that would exclude bridges that combine a multiplexer.

Would it be enough to just switch to plural?

"It receives video frame(s) on its sink pads in one bus format and
converts them and outputs them on its source pads in another bus
format"?

I fear if this is made too specific to single-input single-output
devices, a whole lot of multi-input devices will have to be split up
unnecessarily. Although in cases of freely configurable multiplexers, it
might also make sense to describe them as multiple entities. Especially
if they can run in parallel with different configurations.

> I also wonder whether "bridge" is the appropriate word here. Transceiver might 
> be a better choice, to insist on the fact that one of the two pads corresponds 
> to a physical interface that has special electrical properties (such as HDMI, 
> eDP, or CSI-2 that all require PHYs).

What media entity function would you suggest to use for the IPU CSI,
which basically is
 - a video interface bridge, because it converts media bus formats from
   an external (up to) 20-bit parallel bus to an internal 128-bit bus,
   with the option to either expand/compand/pack >8-bit per component
   pixel formats (so parts of a write pixel formatter)
 - also a video scaler, because it can crop and reduce width and/or
   height to 1/2 the original size

I wouldn't call it a transceiver, as it is completely contained inside
the SoC.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 12/24] add mux and video interface bridge entity functions
@ 2017-02-06 10:27       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-06 10:27 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Steve Longerbeam, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b

On Sun, 2017-02-05 at 17:36 +0200, Laurent Pinchart wrote:
> Hi Steve,
> 
> Thank you for the patch
> 
> On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> > 
> > Signed-off-by: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> > ---
> >  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
> >  include/uapi/linux/media.h                        |  6 ++++++
> >  2 files changed, 28 insertions(+)
> > 
> > diff --git a/Documentation/media/uapi/mediactl/media-types.rst
> > b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29
> > 100644
> > --- a/Documentation/media/uapi/mediactl/media-types.rst
> > +++ b/Documentation/media/uapi/mediactl/media-types.rst
> > @@ -298,6 +298,28 @@ Types and flags used to represent the media graph
> > elements received on its sink pad and outputs the statistics data on
> >  	  its source pad.
> > 
> > +    -  ..  row 29
> > +
> > +       ..  _MEDIA-ENT-F-MUX:
> > +
> > +       -  ``MEDIA_ENT_F_MUX``
> > +
> > +       - Video multiplexer. An entity capable of multiplexing must have at
> > +         least two sink pads and one source pad, and must pass the video
> > +         frame(s) received from the active sink pad to the source pad.
> > Video
> > +         frame(s) from the inactive sink pads are discarded.
> 
> Apart from the comment made by Hans regarding the macro name, this looks good 
> to me.
> 
> > +    -  ..  row 30
> > +
> > +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> > +
> > +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> > +
> > +       - Video interface bridge. A video interface bridge entity must have
> > at
> > +         least one sink pad and one source pad. It receives video frame(s)
> > on
> > +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> > +         converts them and outputs them on its source pad in another bus
> > format
> > +         (eDP, MIPI CSI-2, parallel, ...).
> 
> 
> The first sentence mentions *at least* one sink pad and one source pad, and 
> the second one refers to "its sink pad" and "its source pad". This is a bit 
> confusing. An easy option would be to require a single sink and a single 
> source pad, but that would exclude bridges that combine a multiplexer.

Would it be enough to just switch to plural?

"It receives video frame(s) on its sink pads in one bus format and
converts them and outputs them on its source pads in another bus
format"?

I fear if this is made too specific to single-input single-output
devices, a whole lot of multi-input devices will have to be split up
unnecessarily. Although in cases of freely configurable multiplexers, it
might also make sense to describe them as multiple entities. Especially
if they can run in parallel with different configurations.

> I also wonder whether "bridge" is the appropriate word here. Transceiver might 
> be a better choice, to insist on the fact that one of the two pads corresponds 
> to a physical interface that has special electrical properties (such as HDMI, 
> eDP, or CSI-2 that all require PHYs).

What media entity function would you suggest to use for the IPU CSI,
which basically is
 - a video interface bridge, because it converts media bus formats from
   an external (up to) 20-bit parallel bus to an internal 128-bit bus,
   with the option to either expand/compand/pack >8-bit per component
   pixel formats (so parts of a write pixel formatter)
 - also a video scaler, because it can crop and reduce width and/or
   height to 1/2 the original size

I wouldn't call it a transceiver, as it is completely contained inside
the SoC.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 12/24] add mux and video interface bridge entity functions
@ 2017-02-06 10:27       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-06 10:27 UTC (permalink / raw)
  To: linux-arm-kernel

On Sun, 2017-02-05 at 17:36 +0200, Laurent Pinchart wrote:
> Hi Steve,
> 
> Thank you for the patch
> 
> On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > ---
> >  Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
> >  include/uapi/linux/media.h                        |  6 ++++++
> >  2 files changed, 28 insertions(+)
> > 
> > diff --git a/Documentation/media/uapi/mediactl/media-types.rst
> > b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29
> > 100644
> > --- a/Documentation/media/uapi/mediactl/media-types.rst
> > +++ b/Documentation/media/uapi/mediactl/media-types.rst
> > @@ -298,6 +298,28 @@ Types and flags used to represent the media graph
> > elements received on its sink pad and outputs the statistics data on
> >  	  its source pad.
> > 
> > +    -  ..  row 29
> > +
> > +       ..  _MEDIA-ENT-F-MUX:
> > +
> > +       -  ``MEDIA_ENT_F_MUX``
> > +
> > +       - Video multiplexer. An entity capable of multiplexing must have at
> > +         least two sink pads and one source pad, and must pass the video
> > +         frame(s) received from the active sink pad to the source pad.
> > Video
> > +         frame(s) from the inactive sink pads are discarded.
> 
> Apart from the comment made by Hans regarding the macro name, this looks good 
> to me.
> 
> > +    -  ..  row 30
> > +
> > +       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
> > +
> > +       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
> > +
> > +       - Video interface bridge. A video interface bridge entity must have
> > at
> > +         least one sink pad and one source pad. It receives video frame(s)
> > on
> > +         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> > +         converts them and outputs them on its source pad in another bus
> > format
> > +         (eDP, MIPI CSI-2, parallel, ...).
> 
> 
> The first sentence mentions *at least* one sink pad and one source pad, and 
> the second one refers to "its sink pad" and "its source pad". This is a bit 
> confusing. An easy option would be to require a single sink and a single 
> source pad, but that would exclude bridges that combine a multiplexer.

Would it be enough to just switch to plural?

"It receives video frame(s) on its sink pads in one bus format and
converts them and outputs them on its source pads in another bus
format"?

I fear if this is made too specific to single-input single-output
devices, a whole lot of multi-input devices will have to be split up
unnecessarily. Although in cases of freely configurable multiplexers, it
might also make sense to describe them as multiple entities. Especially
if they can run in parallel with different configurations.

> I also wonder whether "bridge" is the appropriate word here. Transceiver might 
> be a better choice, to insist on the fact that one of the two pads corresponds 
> to a physical interface that has special electrical properties (such as HDMI, 
> eDP, or CSI-2 that all require PHYs).

What media entity function would you suggest to use for the IPU CSI,
which basically is
 - a video interface bridge, because it converts media bus formats from
   an external (up to) 20-bit parallel bus to an internal 128-bit bus,
   with the option to either expand/compand/pack >8-bit per component
   pixel formats (so parts of a write pixel formatter)
 - also a video scaler, because it can crop and reduce width and/or
   height to 1/2 the original size

I wouldn't call it a transceiver, as it is completely contained inside
the SoC.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-06 22:33               ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-06 22:33 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, Philipp Zabel, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus

Hi Hans,

(CC'ing Sakari)

On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>> +
> >>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>> v4l2_mbus_config
> >>>>> *cfg)
> >>>>> +{
> >>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> >>>>> +	struct media_pad *pad;
> >>>>> +	int ret;
> >>>>> +
> >>>>> +	if (vidsw->active == -1) {
> >>>>> +		dev_err(sd->dev, "no configuration for inactive 
mux\n");
> >>>>> +		return -EINVAL;
> >>>>> +	}
> >>>>> +
> >>>>> +	/*
> >>>>> +	 * Retrieve media bus configuration from the entity connected 
to the
> >>>>> +	 * active input
> >>>>> +	 */
> >>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> >>>>> +	if (pad) {
> >>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> >>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> >>>>> +		if (ret == -ENOIOCTLCMD)
> >>>>> +			pad = NULL;
> >>>>> +		else if (ret < 0) {
> >>>>> +			dev_err(sd->dev, "failed to get source
> >>>>> configuration\n");
> >>>>> +			return ret;
> >>>>> +		}
> >>>>> +	}
> >>>>> +	if (!pad) {
> >>>>> +		/* Mirror the input side on the output side */
> >>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> >>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> >>>>> +		    cfg->type == V4L2_MBUS_BT656)
> >>>>> +			cfg->flags = vidsw->endpoint[vidsw-
> >>>>> active].bus.parallel.flags;
> >>>>> +	}
> >>>>> +
> >>>>> +	return 0;
> >>>>> +}
> >>>> 
> >>>> I am not certain this op is needed at all. In the current kernel this
> >>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> >>>> dubious). Normally this information should come from the device tree
> >>>> and there should be no need for this op.
> >>>> 
> >>>> My (tentative) long-term plan was to get rid of this op.
> >>>> 
> >>>> If you don't need it, then I recommend it is removed.
> >> 
> >> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >> camera sensor, and it was doing that to determine the sensor's bus type.
> >> This info was already available from parsing a v4l2_of_endpoint from the
> >> sensor node. So it was simple to remove the g_mbus_config calls, and
> >> instead rely on the parsed sensor v4l2_of_endpoint.
> > 
> > That's not a good point.

(mea culpa, s/point/idea/)

> > The imx-media driver must not parse the sensor DT node as it is not aware
> > of what bindings the sensor is compatible with. Information must instead
> > be queried from the sensor subdev at runtime, through the g_mbus_config()
> > operation.
> > 
> > Of course, if you can get the information from the imx-media DT node,
> > that's certainly an option. It's only information provided by the sensor
> > driver that you have no choice but query using a subdev operation.
> 
> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using
> this?

It all depends on what type of information needs to be retrieved, and whether 
it can change at runtime or is fixed. Adding properties to the imx-media DT 
node is certainly fine as long as those properties describe the i.MX side.

In the omap3isp case, we use the operation to query whether parallel data 
contains embedded sync (BT.656) or uses separate h/v sync signals.

> The reason I am suspicious about this op is that it came from soc-camera and
> predates the DT. The contents of v4l2_mbus_config seems very much like a HW
> description to me, i.e. something that belongs in the DT.

Part of it is possibly outdated, but for buses that support multiple modes of 
operation (such as the parallel bus case described above) we need to make that 
information discoverable at runtime. Maybe this should be considered as 
related to Sakari's efforts to support VC/DT for CSI-2, and supported through 
the API he is working on.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-06 22:33               ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-06 22:33 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Steve Longerbeam, Philipp Zabel, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA

Hi Hans,

(CC'ing Sakari)

On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>> +
> >>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>> v4l2_mbus_config
> >>>>> *cfg)
> >>>>> +{
> >>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> >>>>> +	struct media_pad *pad;
> >>>>> +	int ret;
> >>>>> +
> >>>>> +	if (vidsw->active == -1) {
> >>>>> +		dev_err(sd->dev, "no configuration for inactive 
mux\n");
> >>>>> +		return -EINVAL;
> >>>>> +	}
> >>>>> +
> >>>>> +	/*
> >>>>> +	 * Retrieve media bus configuration from the entity connected 
to the
> >>>>> +	 * active input
> >>>>> +	 */
> >>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> >>>>> +	if (pad) {
> >>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> >>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> >>>>> +		if (ret == -ENOIOCTLCMD)
> >>>>> +			pad = NULL;
> >>>>> +		else if (ret < 0) {
> >>>>> +			dev_err(sd->dev, "failed to get source
> >>>>> configuration\n");
> >>>>> +			return ret;
> >>>>> +		}
> >>>>> +	}
> >>>>> +	if (!pad) {
> >>>>> +		/* Mirror the input side on the output side */
> >>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> >>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> >>>>> +		    cfg->type == V4L2_MBUS_BT656)
> >>>>> +			cfg->flags = vidsw->endpoint[vidsw-
> >>>>> active].bus.parallel.flags;
> >>>>> +	}
> >>>>> +
> >>>>> +	return 0;
> >>>>> +}
> >>>> 
> >>>> I am not certain this op is needed at all. In the current kernel this
> >>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> >>>> dubious). Normally this information should come from the device tree
> >>>> and there should be no need for this op.
> >>>> 
> >>>> My (tentative) long-term plan was to get rid of this op.
> >>>> 
> >>>> If you don't need it, then I recommend it is removed.
> >> 
> >> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >> camera sensor, and it was doing that to determine the sensor's bus type.
> >> This info was already available from parsing a v4l2_of_endpoint from the
> >> sensor node. So it was simple to remove the g_mbus_config calls, and
> >> instead rely on the parsed sensor v4l2_of_endpoint.
> > 
> > That's not a good point.

(mea culpa, s/point/idea/)

> > The imx-media driver must not parse the sensor DT node as it is not aware
> > of what bindings the sensor is compatible with. Information must instead
> > be queried from the sensor subdev at runtime, through the g_mbus_config()
> > operation.
> > 
> > Of course, if you can get the information from the imx-media DT node,
> > that's certainly an option. It's only information provided by the sensor
> > driver that you have no choice but query using a subdev operation.
> 
> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using
> this?

It all depends on what type of information needs to be retrieved, and whether 
it can change at runtime or is fixed. Adding properties to the imx-media DT 
node is certainly fine as long as those properties describe the i.MX side.

In the omap3isp case, we use the operation to query whether parallel data 
contains embedded sync (BT.656) or uses separate h/v sync signals.

> The reason I am suspicious about this op is that it came from soc-camera and
> predates the DT. The contents of v4l2_mbus_config seems very much like a HW
> description to me, i.e. something that belongs in the DT.

Part of it is possibly outdated, but for buses that support multiple modes of 
operation (such as the parallel bus case described above) we need to make that 
information discoverable at runtime. Maybe this should be considered as 
related to Sakari's efforts to support VC/DT for CSI-2, and supported through 
the API he is working on.

-- 
Regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-06 22:33               ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-06 22:33 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Hans,

(CC'ing Sakari)

On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>> +
> >>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>> v4l2_mbus_config
> >>>>> *cfg)
> >>>>> +{
> >>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
> >>>>> +	struct media_pad *pad;
> >>>>> +	int ret;
> >>>>> +
> >>>>> +	if (vidsw->active == -1) {
> >>>>> +		dev_err(sd->dev, "no configuration for inactive 
mux\n");
> >>>>> +		return -EINVAL;
> >>>>> +	}
> >>>>> +
> >>>>> +	/*
> >>>>> +	 * Retrieve media bus configuration from the entity connected 
to the
> >>>>> +	 * active input
> >>>>> +	 */
> >>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
> >>>>> +	if (pad) {
> >>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> >>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
> >>>>> +		if (ret == -ENOIOCTLCMD)
> >>>>> +			pad = NULL;
> >>>>> +		else if (ret < 0) {
> >>>>> +			dev_err(sd->dev, "failed to get source
> >>>>> configuration\n");
> >>>>> +			return ret;
> >>>>> +		}
> >>>>> +	}
> >>>>> +	if (!pad) {
> >>>>> +		/* Mirror the input side on the output side */
> >>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
> >>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
> >>>>> +		    cfg->type == V4L2_MBUS_BT656)
> >>>>> +			cfg->flags = vidsw->endpoint[vidsw-
> >>>>> active].bus.parallel.flags;
> >>>>> +	}
> >>>>> +
> >>>>> +	return 0;
> >>>>> +}
> >>>> 
> >>>> I am not certain this op is needed at all. In the current kernel this
> >>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> >>>> dubious). Normally this information should come from the device tree
> >>>> and there should be no need for this op.
> >>>> 
> >>>> My (tentative) long-term plan was to get rid of this op.
> >>>> 
> >>>> If you don't need it, then I recommend it is removed.
> >> 
> >> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >> camera sensor, and it was doing that to determine the sensor's bus type.
> >> This info was already available from parsing a v4l2_of_endpoint from the
> >> sensor node. So it was simple to remove the g_mbus_config calls, and
> >> instead rely on the parsed sensor v4l2_of_endpoint.
> > 
> > That's not a good point.

(mea culpa, s/point/idea/)

> > The imx-media driver must not parse the sensor DT node as it is not aware
> > of what bindings the sensor is compatible with. Information must instead
> > be queried from the sensor subdev at runtime, through the g_mbus_config()
> > operation.
> > 
> > Of course, if you can get the information from the imx-media DT node,
> > that's certainly an option. It's only information provided by the sensor
> > driver that you have no choice but query using a subdev operation.
> 
> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using
> this?

It all depends on what type of information needs to be retrieved, and whether 
it can change at runtime or is fixed. Adding properties to the imx-media DT 
node is certainly fine as long as those properties describe the i.MX side.

In the omap3isp case, we use the operation to query whether parallel data 
contains embedded sync (BT.656) or uses separate h/v sync signals.

> The reason I am suspicious about this op is that it came from soc-camera and
> predates the DT. The contents of v4l2_mbus_config seems very much like a HW
> description to me, i.e. something that belongs in the DT.

Part of it is possibly outdated, but for buses that support multiple modes of 
operation (such as the parallel bus case described above) we need to make that 
information discoverable at runtime. Maybe this should be considered as 
related to Sakari's efforts to support VC/DT for CSI-2, and supported through 
the API he is working on.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-06 22:33               ` Laurent Pinchart
  (?)
@ 2017-02-06 23:10                 ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-06 23:10 UTC (permalink / raw)
  To: Laurent Pinchart, Hans Verkuil
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus



On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> Hi Hans,
>
> (CC'ing Sakari)
>
> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>>>>>>> +
>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
>>>>>>> v4l2_mbus_config
>>>>>>> *cfg)
>>>>>>> +{
>>>>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>>>>>> +	struct media_pad *pad;
>>>>>>> +	int ret;
>>>>>>> +
>>>>>>> +	if (vidsw->active == -1) {
>>>>>>> +		dev_err(sd->dev, "no configuration for inactive
> mux\n");
>>>>>>> +		return -EINVAL;
>>>>>>> +	}
>>>>>>> +
>>>>>>> +	/*
>>>>>>> +	 * Retrieve media bus configuration from the entity connected
> to the
>>>>>>> +	 * active input
>>>>>>> +	 */
>>>>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>>>>>> +	if (pad) {
>>>>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>>>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>>>>>> +		if (ret == -ENOIOCTLCMD)
>>>>>>> +			pad = NULL;
>>>>>>> +		else if (ret < 0) {
>>>>>>> +			dev_err(sd->dev, "failed to get source
>>>>>>> configuration\n");
>>>>>>> +			return ret;
>>>>>>> +		}
>>>>>>> +	}
>>>>>>> +	if (!pad) {
>>>>>>> +		/* Mirror the input side on the output side */
>>>>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>>>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>>>>>> +		    cfg->type == V4L2_MBUS_BT656)
>>>>>>> +			cfg->flags = vidsw->endpoint[vidsw-
>>>>>>> active].bus.parallel.flags;
>>>>>>> +	}
>>>>>>> +
>>>>>>> +	return 0;
>>>>>>> +}
>>>>>> I am not certain this op is needed at all. In the current kernel this
>>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
>>>>>> dubious). Normally this information should come from the device tree
>>>>>> and there should be no need for this op.
>>>>>>
>>>>>> My (tentative) long-term plan was to get rid of this op.
>>>>>>
>>>>>> If you don't need it, then I recommend it is removed.
>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
>>>> camera sensor, and it was doing that to determine the sensor's bus type.
>>>> This info was already available from parsing a v4l2_of_endpoint from the
>>>> sensor node. So it was simple to remove the g_mbus_config calls, and
>>>> instead rely on the parsed sensor v4l2_of_endpoint.
>>> That's not a good point.
> (mea culpa, s/point/idea/)
>
>>> The imx-media driver must not parse the sensor DT node as it is not aware
>>> of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found
by parsing the OF graph, so it is known to be a camera sensor node at
that point.


>>>   Information must instead
>>> be queried from the sensor subdev at runtime, through the g_mbus_config()
>>> operation.
>>>
>>> Of course, if you can get the information from the imx-media DT node,
>>> that's certainly an option. It's only information provided by the sensor
>>> driver that you have no choice but query using a subdev operation.
>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using
>> this?
> It all depends on what type of information needs to be retrieved, and whether
> it can change at runtime or is fixed. Adding properties to the imx-media DT
> node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily
available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

The media bus type is not something that can be added to the
imx-media node since it contains no endpoint nodes.


> In the omap3isp case, we use the operation to query whether parallel data
> contains embedded sync (BT.656) or uses separate h/v sync signals.
>
>> The reason I am suspicious about this op is that it came from soc-camera and
>> predates the DT. The contents of v4l2_mbus_config seems very much like a HW
>> description to me, i.e. something that belongs in the DT.
> Part of it is possibly outdated, but for buses that support multiple modes of
> operation (such as the parallel bus case described above) we need to make that
> information discoverable at runtime. Maybe this should be considered as
> related to Sakari's efforts to support VC/DT for CSI-2, and supported through
> the API he is working on.

That sounds interesting, can you point me to some info on this effort?
I've been thinking the DT should contain virtual channel info for CSI-2
buses.

Steve

>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-06 23:10                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-06 23:10 UTC (permalink / raw)
  To: Laurent Pinchart, Hans Verkuil
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, sakari.ailus, nick,
	songjun.wu, Steve Longerbeam, robert.jarzmik, devel,
	markus.heiser, laurent.pinchart+renesas, linux, geert,
	Sascha Hauer, linux-media, devicetree, kernel, arnd, mchehab,
	bparrot, robh+dt, horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, Philipp Zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> Hi Hans,
>
> (CC'ing Sakari)
>
> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>>>>>>> +
>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
>>>>>>> v4l2_mbus_config
>>>>>>> *cfg)
>>>>>>> +{
>>>>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>>>>>> +	struct media_pad *pad;
>>>>>>> +	int ret;
>>>>>>> +
>>>>>>> +	if (vidsw->active == -1) {
>>>>>>> +		dev_err(sd->dev, "no configuration for inactive
> mux\n");
>>>>>>> +		return -EINVAL;
>>>>>>> +	}
>>>>>>> +
>>>>>>> +	/*
>>>>>>> +	 * Retrieve media bus configuration from the entity connected
> to the
>>>>>>> +	 * active input
>>>>>>> +	 */
>>>>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>>>>>> +	if (pad) {
>>>>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>>>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>>>>>> +		if (ret == -ENOIOCTLCMD)
>>>>>>> +			pad = NULL;
>>>>>>> +		else if (ret < 0) {
>>>>>>> +			dev_err(sd->dev, "failed to get source
>>>>>>> configuration\n");
>>>>>>> +			return ret;
>>>>>>> +		}
>>>>>>> +	}
>>>>>>> +	if (!pad) {
>>>>>>> +		/* Mirror the input side on the output side */
>>>>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>>>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>>>>>> +		    cfg->type == V4L2_MBUS_BT656)
>>>>>>> +			cfg->flags = vidsw->endpoint[vidsw-
>>>>>>> active].bus.parallel.flags;
>>>>>>> +	}
>>>>>>> +
>>>>>>> +	return 0;
>>>>>>> +}
>>>>>> I am not certain this op is needed at all. In the current kernel this
>>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
>>>>>> dubious). Normally this information should come from the device tree
>>>>>> and there should be no need for this op.
>>>>>>
>>>>>> My (tentative) long-term plan was to get rid of this op.
>>>>>>
>>>>>> If you don't need it, then I recommend it is removed.
>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
>>>> camera sensor, and it was doing that to determine the sensor's bus type.
>>>> This info was already available from parsing a v4l2_of_endpoint from the
>>>> sensor node. So it was simple to remove the g_mbus_config calls, and
>>>> instead rely on the parsed sensor v4l2_of_endpoint.
>>> That's not a good point.
> (mea culpa, s/point/idea/)
>
>>> The imx-media driver must not parse the sensor DT node as it is not aware
>>> of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found
by parsing the OF graph, so it is known to be a camera sensor node at
that point.


>>>   Information must instead
>>> be queried from the sensor subdev at runtime, through the g_mbus_config()
>>> operation.
>>>
>>> Of course, if you can get the information from the imx-media DT node,
>>> that's certainly an option. It's only information provided by the sensor
>>> driver that you have no choice but query using a subdev operation.
>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using
>> this?
> It all depends on what type of information needs to be retrieved, and whether
> it can change at runtime or is fixed. Adding properties to the imx-media DT
> node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily
available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

The media bus type is not something that can be added to the
imx-media node since it contains no endpoint nodes.


> In the omap3isp case, we use the operation to query whether parallel data
> contains embedded sync (BT.656) or uses separate h/v sync signals.
>
>> The reason I am suspicious about this op is that it came from soc-camera and
>> predates the DT. The contents of v4l2_mbus_config seems very much like a HW
>> description to me, i.e. something that belongs in the DT.
> Part of it is possibly outdated, but for buses that support multiple modes of
> operation (such as the parallel bus case described above) we need to make that
> information discoverable at runtime. Maybe this should be considered as
> related to Sakari's efforts to support VC/DT for CSI-2, and supported through
> the API he is working on.

That sounds interesting, can you point me to some info on this effort?
I've been thinking the DT should contain virtual channel info for CSI-2
buses.

Steve

>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-06 23:10                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-06 23:10 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> Hi Hans,
>
> (CC'ing Sakari)
>
> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
>>>>>>> +
>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
>>>>>>> v4l2_mbus_config
>>>>>>> *cfg)
>>>>>>> +{
>>>>>>> +	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
>>>>>>> +	struct media_pad *pad;
>>>>>>> +	int ret;
>>>>>>> +
>>>>>>> +	if (vidsw->active == -1) {
>>>>>>> +		dev_err(sd->dev, "no configuration for inactive
> mux\n");
>>>>>>> +		return -EINVAL;
>>>>>>> +	}
>>>>>>> +
>>>>>>> +	/*
>>>>>>> +	 * Retrieve media bus configuration from the entity connected
> to the
>>>>>>> +	 * active input
>>>>>>> +	 */
>>>>>>> +	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
>>>>>>> +	if (pad) {
>>>>>>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>>>>>>> +		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
>>>>>>> +		if (ret == -ENOIOCTLCMD)
>>>>>>> +			pad = NULL;
>>>>>>> +		else if (ret < 0) {
>>>>>>> +			dev_err(sd->dev, "failed to get source
>>>>>>> configuration\n");
>>>>>>> +			return ret;
>>>>>>> +		}
>>>>>>> +	}
>>>>>>> +	if (!pad) {
>>>>>>> +		/* Mirror the input side on the output side */
>>>>>>> +		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
>>>>>>> +		if (cfg->type == V4L2_MBUS_PARALLEL ||
>>>>>>> +		    cfg->type == V4L2_MBUS_BT656)
>>>>>>> +			cfg->flags = vidsw->endpoint[vidsw-
>>>>>>> active].bus.parallel.flags;
>>>>>>> +	}
>>>>>>> +
>>>>>>> +	return 0;
>>>>>>> +}
>>>>>> I am not certain this op is needed at all. In the current kernel this
>>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
>>>>>> dubious). Normally this information should come from the device tree
>>>>>> and there should be no need for this op.
>>>>>>
>>>>>> My (tentative) long-term plan was to get rid of this op.
>>>>>>
>>>>>> If you don't need it, then I recommend it is removed.
>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
>>>> camera sensor, and it was doing that to determine the sensor's bus type.
>>>> This info was already available from parsing a v4l2_of_endpoint from the
>>>> sensor node. So it was simple to remove the g_mbus_config calls, and
>>>> instead rely on the parsed sensor v4l2_of_endpoint.
>>> That's not a good point.
> (mea culpa, s/point/idea/)
>
>>> The imx-media driver must not parse the sensor DT node as it is not aware
>>> of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found
by parsing the OF graph, so it is known to be a camera sensor node at
that point.


>>>   Information must instead
>>> be queried from the sensor subdev at runtime, through the g_mbus_config()
>>> operation.
>>>
>>> Of course, if you can get the information from the imx-media DT node,
>>> that's certainly an option. It's only information provided by the sensor
>>> driver that you have no choice but query using a subdev operation.
>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using
>> this?
> It all depends on what type of information needs to be retrieved, and whether
> it can change at runtime or is fixed. Adding properties to the imx-media DT
> node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily
available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

The media bus type is not something that can be added to the
imx-media node since it contains no endpoint nodes.


> In the omap3isp case, we use the operation to query whether parallel data
> contains embedded sync (BT.656) or uses separate h/v sync signals.
>
>> The reason I am suspicious about this op is that it came from soc-camera and
>> predates the DT. The contents of v4l2_mbus_config seems very much like a HW
>> description to me, i.e. something that belongs in the DT.
> Part of it is possibly outdated, but for buses that support multiple modes of
> operation (such as the parallel bus case described above) we need to make that
> information discoverable at runtime. Maybe this should be considered as
> related to Sakari's efforts to support VC/DT for CSI-2, and supported through
> the API he is working on.

That sounds interesting, can you point me to some info on this effort?
I've been thinking the DT should contain virtual channel info for CSI-2
buses.

Steve

>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-02-07  1:52       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-07  1:52 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam


On 02/02/2017 02:35 PM, Russell King - ARM Linux wrote:
> <snip>
> However, "*vfd" contains a struct device, and you _correctly_ set the
> release function for "*vfd" to video_device_release via camif_videodev.
>
> However, if you try to rmmod imx-media, then you end up with a kernel
> warning that you're freeing memory containing a held lock, and later
> chaos ensues because kmalloc has been corrupted.
>
> The root cause of this is embedding the device structure within the
> video_device into the driver's private data.  *Any* structure what so
> ever that contains a kref is reference counted, and that includes
> struct device, and therefore also includes struct video_device.  What
> that means is that its lifetime is _not_ under _your_ control, and
> you may not free it except through its release function (which is
> video_device_release().)  However, that also tries to kfree (with an
> offset of 4) your private data, which results in the warning and the
> corrupted kmalloc free lists.
>
> The solution is simple, make "vfd" a pointer in your private data
> structure and kmalloc() it separately, letting video_device_release()
> kfree() that data when it needs to.

Thanks Russell for tracking this down. I remember doing this
when I was reviewing the code for opportunities to "optimize" :-/,
and carelessly caused this bug by not reviewing how video_device
is freed.

Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-02-07  1:52       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-07  1:52 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	hverkuil-qWit8jRvyhVmR6Xm/wNWPw, nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam


On 02/02/2017 02:35 PM, Russell King - ARM Linux wrote:
> <snip>
> However, "*vfd" contains a struct device, and you _correctly_ set the
> release function for "*vfd" to video_device_release via camif_videodev.
>
> However, if you try to rmmod imx-media, then you end up with a kernel
> warning that you're freeing memory containing a held lock, and later
> chaos ensues because kmalloc has been corrupted.
>
> The root cause of this is embedding the device structure within the
> video_device into the driver's private data.  *Any* structure what so
> ever that contains a kref is reference counted, and that includes
> struct device, and therefore also includes struct video_device.  What
> that means is that its lifetime is _not_ under _your_ control, and
> you may not free it except through its release function (which is
> video_device_release().)  However, that also tries to kfree (with an
> offset of 4) your private data, which results in the warning and the
> corrupted kmalloc free lists.
>
> The solution is simple, make "vfd" a pointer in your private data
> structure and kmalloc() it separately, letting video_device_release()
> kfree() that data when it needs to.

Thanks Russell for tracking this down. I remember doing this
when I was reviewing the code for opportunities to "optimize" :-/,
and carelessly caused this bug by not reviewing how video_device
is freed.

Fixed.

Steve

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver
@ 2017-02-07  1:52       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-07  1:52 UTC (permalink / raw)
  To: linux-arm-kernel


On 02/02/2017 02:35 PM, Russell King - ARM Linux wrote:
> <snip>
> However, "*vfd" contains a struct device, and you _correctly_ set the
> release function for "*vfd" to video_device_release via camif_videodev.
>
> However, if you try to rmmod imx-media, then you end up with a kernel
> warning that you're freeing memory containing a held lock, and later
> chaos ensues because kmalloc has been corrupted.
>
> The root cause of this is embedding the device structure within the
> video_device into the driver's private data.  *Any* structure what so
> ever that contains a kref is reference counted, and that includes
> struct device, and therefore also includes struct video_device.  What
> that means is that its lifetime is _not_ under _your_ control, and
> you may not free it except through its release function (which is
> video_device_release().)  However, that also tries to kfree (with an
> offset of 4) your private data, which results in the warning and the
> corrupted kmalloc free lists.
>
> The solution is simple, make "vfd" a pointer in your private data
> structure and kmalloc() it separately, letting video_device_release()
> kfree() that data when it needs to.

Thanks Russell for tracking this down. I remember doing this
when I was reviewing the code for opportunities to "optimize" :-/,
and carelessly caused this bug by not reviewing how video_device
is freed.

Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
  2017-02-02 22:50     ` Russell King - ARM Linux
  (?)
@ 2017-02-07  1:54       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-07  1:54 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, mchehab,
	hverkuil, nick, markus.heiser, p.zabel, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 02/02/2017 02:50 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
>> +/* register an internal subdev as a platform device */
>> +static struct imx_media_subdev *
>> +add_internal_subdev(struct imx_media_dev *imxmd,
>> +		    const struct internal_subdev *isd,
>> +		    int ipu_id)
>> +{
>> +	struct imx_media_internal_sd_platformdata pdata;
>> +	struct platform_device_info pdevinfo = {0};
>> +	struct imx_media_subdev *imxsd;
>> +	struct platform_device *pdev;
>> +
>> +	switch (isd->id->grp_id) {
>> +	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
>> +		pdata.grp_id = isd->id->grp_id +
>> +			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
>> +		break;
>> +	default:
>> +		pdata.grp_id = isd->id->grp_id;
>> +		break;
>> +	}
>> +
>> +	/* the id of IPU this subdev will control */
>> +	pdata.ipu_id = ipu_id;
>> +
>> +	/* create subdev name */
>> +	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
>> +				    pdata.grp_id, ipu_id);
>> +
>> +	pdevinfo.name = isd->id->name;
>> +	pdevinfo.id = ipu_id * num_isd + isd->id->index;
>> +	pdevinfo.parent = imxmd->dev;
>> +	pdevinfo.data = &pdata;
>> +	pdevinfo.size_data = sizeof(pdata);
>> +	pdevinfo.dma_mask = DMA_BIT_MASK(32);
>> +
>> +	pdev = platform_device_register_full(&pdevinfo);
>> +	if (IS_ERR(pdev))
>> +		return ERR_CAST(pdev);
>> +
>> +	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
>> +	if (IS_ERR(imxsd))
>> +		return imxsd;
>> +
>> +	imxsd->num_sink_pads = isd->num_sink_pads;
>> +	imxsd->num_src_pads = isd->num_src_pads;
>> +
>> +	return imxsd;
>> +}
> You seem to create platform devices here, but I see nowhere that you
> ever remove them - so if you get to the lucky point of being able to
> rmmod imx-media and then try to re-insert it, you end up with a load
> of kernel warnings, one for each device created this way, and
> platform_device_register_full() fails:

Right, I never free the platform devices for the internal subdevs.
Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-02-07  1:54       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-07  1:54 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/02/2017 02:50 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
>> +/* register an internal subdev as a platform device */
>> +static struct imx_media_subdev *
>> +add_internal_subdev(struct imx_media_dev *imxmd,
>> +		    const struct internal_subdev *isd,
>> +		    int ipu_id)
>> +{
>> +	struct imx_media_internal_sd_platformdata pdata;
>> +	struct platform_device_info pdevinfo = {0};
>> +	struct imx_media_subdev *imxsd;
>> +	struct platform_device *pdev;
>> +
>> +	switch (isd->id->grp_id) {
>> +	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
>> +		pdata.grp_id = isd->id->grp_id +
>> +			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
>> +		break;
>> +	default:
>> +		pdata.grp_id = isd->id->grp_id;
>> +		break;
>> +	}
>> +
>> +	/* the id of IPU this subdev will control */
>> +	pdata.ipu_id = ipu_id;
>> +
>> +	/* create subdev name */
>> +	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
>> +				    pdata.grp_id, ipu_id);
>> +
>> +	pdevinfo.name = isd->id->name;
>> +	pdevinfo.id = ipu_id * num_isd + isd->id->index;
>> +	pdevinfo.parent = imxmd->dev;
>> +	pdevinfo.data = &pdata;
>> +	pdevinfo.size_data = sizeof(pdata);
>> +	pdevinfo.dma_mask = DMA_BIT_MASK(32);
>> +
>> +	pdev = platform_device_register_full(&pdevinfo);
>> +	if (IS_ERR(pdev))
>> +		return ERR_CAST(pdev);
>> +
>> +	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
>> +	if (IS_ERR(imxsd))
>> +		return imxsd;
>> +
>> +	imxsd->num_sink_pads = isd->num_sink_pads;
>> +	imxsd->num_src_pads = isd->num_src_pads;
>> +
>> +	return imxsd;
>> +}
> You seem to create platform devices here, but I see nowhere that you
> ever remove them - so if you get to the lucky point of being able to
> rmmod imx-media and then try to re-insert it, you end up with a load
> of kernel warnings, one for each device created this way, and
> platform_device_register_full() fails:

Right, I never free the platform devices for the internal subdevs.
Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 16/24] media: Add i.MX media core driver
@ 2017-02-07  1:54       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-07  1:54 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/02/2017 02:50 PM, Russell King - ARM Linux wrote:
> On Fri, Jan 06, 2017 at 06:11:34PM -0800, Steve Longerbeam wrote:
>> +/* register an internal subdev as a platform device */
>> +static struct imx_media_subdev *
>> +add_internal_subdev(struct imx_media_dev *imxmd,
>> +		    const struct internal_subdev *isd,
>> +		    int ipu_id)
>> +{
>> +	struct imx_media_internal_sd_platformdata pdata;
>> +	struct platform_device_info pdevinfo = {0};
>> +	struct imx_media_subdev *imxsd;
>> +	struct platform_device *pdev;
>> +
>> +	switch (isd->id->grp_id) {
>> +	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
>> +		pdata.grp_id = isd->id->grp_id +
>> +			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
>> +		break;
>> +	default:
>> +		pdata.grp_id = isd->id->grp_id;
>> +		break;
>> +	}
>> +
>> +	/* the id of IPU this subdev will control */
>> +	pdata.ipu_id = ipu_id;
>> +
>> +	/* create subdev name */
>> +	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
>> +				    pdata.grp_id, ipu_id);
>> +
>> +	pdevinfo.name = isd->id->name;
>> +	pdevinfo.id = ipu_id * num_isd + isd->id->index;
>> +	pdevinfo.parent = imxmd->dev;
>> +	pdevinfo.data = &pdata;
>> +	pdevinfo.size_data = sizeof(pdata);
>> +	pdevinfo.dma_mask = DMA_BIT_MASK(32);
>> +
>> +	pdev = platform_device_register_full(&pdevinfo);
>> +	if (IS_ERR(pdev))
>> +		return ERR_CAST(pdev);
>> +
>> +	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
>> +	if (IS_ERR(imxsd))
>> +		return imxsd;
>> +
>> +	imxsd->num_sink_pads = isd->num_sink_pads;
>> +	imxsd->num_src_pads = isd->num_src_pads;
>> +
>> +	return imxsd;
>> +}
> You seem to create platform devices here, but I see nowhere that you
> ever remove them - so if you get to the lucky point of being able to
> rmmod imx-media and then try to re-insert it, you end up with a load
> of kernel warnings, one for each device created this way, and
> platform_device_register_full() fails:

Right, I never free the platform devices for the internal subdevs.
Fixed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 10:26                   ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 10:26 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Hans Verkuil, Philipp Zabel, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus

Hi Steve,

On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>> +
> >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>> v4l2_mbus_config *cfg)

[snip]

> >>>>>> I am not certain this op is needed at all. In the current kernel this
> >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> >>>>>> dubious). Normally this information should come from the device tree
> >>>>>> and there should be no need for this op.
> >>>>>> 
> >>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>> 
> >>>>>> If you don't need it, then I recommend it is removed.
> >>>> 
> >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>> type. This info was already available from parsing a v4l2_of_endpoint
> >>>> from the sensor node. So it was simple to remove the g_mbus_config
> >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> >>> 
> >>> That's not a good point.
> > 
> > (mea culpa, s/point/idea/)
> > 
> >>> The imx-media driver must not parse the sensor DT node as it is not
> >>> aware of what bindings the sensor is compatible with.
> 
> Hi Laurent,
> 
> I don't really understand this argument. The sensor node has been found
> by parsing the OF graph, so it is known to be a camera sensor node at
> that point.

All you know in the i.MX6 driver is that the remote node is a video source. 
You can rely on the fact that it implements the OF graph bindings to locate 
other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible 
string. Unless you match on sensor compat strings in the i.MX6 driver (which 
you shouldn't do, to keep the driver generic) you can't know for certain how 
to parse the sensor node DT properties. For all you know, the video source 
could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
can't even rely on the fact that it's a sensor.

> >>> Information must instead be queried from the sensor subdev at runtime,
> >>> through the g_mbus_config() operation.
> >>> 
> >>> Of course, if you can get the information from the imx-media DT node,
> >>> that's certainly an option. It's only information provided by the sensor
> >>> driver that you have no choice but query using a subdev operation.
> >> 
> >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >> using this?
> > 
> > It all depends on what type of information needs to be retrieved, and
> > whether it can change at runtime or is fixed. Adding properties to the
> > imx-media DT node is certainly fine as long as those properties describe
> > the i.MX side.
>
> In this case the info needed is the media bus type. That info is most easily
> available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> node.

I haven't had time to check the code in details yet, so I can't really comment 
on what you need and how it should be implemented exactly.

> The media bus type is not something that can be added to the
> imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

> > In the omap3isp case, we use the operation to query whether parallel data
> > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > 
> >> The reason I am suspicious about this op is that it came from soc-camera
> >> and predates the DT. The contents of v4l2_mbus_config seems very much
> >> like a HW description to me, i.e. something that belongs in the DT.
> > 
> > Part of it is possibly outdated, but for buses that support multiple modes
> > of operation (such as the parallel bus case described above) we need to
> > make that information discoverable at runtime. Maybe this should be
> > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > supported through the API he is working on.
> 
> That sounds interesting, can you point me to some info on this effort?

Sure.

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

> I've been thinking the DT should contain virtual channel info for CSI-2
> buses.

I don't think it should. CSI-2 virtual channels and data types should be 
handled as a software concept, and thus supported through driver code without 
involving DT.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 10:26                   ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 10:26 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Hans Verkuil, Philipp Zabel, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA, devel

Hi Steve,

On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>> +
> >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>> v4l2_mbus_config *cfg)

[snip]

> >>>>>> I am not certain this op is needed at all. In the current kernel this
> >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> >>>>>> dubious). Normally this information should come from the device tree
> >>>>>> and there should be no need for this op.
> >>>>>> 
> >>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>> 
> >>>>>> If you don't need it, then I recommend it is removed.
> >>>> 
> >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>> type. This info was already available from parsing a v4l2_of_endpoint
> >>>> from the sensor node. So it was simple to remove the g_mbus_config
> >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> >>> 
> >>> That's not a good point.
> > 
> > (mea culpa, s/point/idea/)
> > 
> >>> The imx-media driver must not parse the sensor DT node as it is not
> >>> aware of what bindings the sensor is compatible with.
> 
> Hi Laurent,
> 
> I don't really understand this argument. The sensor node has been found
> by parsing the OF graph, so it is known to be a camera sensor node at
> that point.

All you know in the i.MX6 driver is that the remote node is a video source. 
You can rely on the fact that it implements the OF graph bindings to locate 
other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible 
string. Unless you match on sensor compat strings in the i.MX6 driver (which 
you shouldn't do, to keep the driver generic) you can't know for certain how 
to parse the sensor node DT properties. For all you know, the video source 
could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
can't even rely on the fact that it's a sensor.

> >>> Information must instead be queried from the sensor subdev at runtime,
> >>> through the g_mbus_config() operation.
> >>> 
> >>> Of course, if you can get the information from the imx-media DT node,
> >>> that's certainly an option. It's only information provided by the sensor
> >>> driver that you have no choice but query using a subdev operation.
> >> 
> >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >> using this?
> > 
> > It all depends on what type of information needs to be retrieved, and
> > whether it can change at runtime or is fixed. Adding properties to the
> > imx-media DT node is certainly fine as long as those properties describe
> > the i.MX side.
>
> In this case the info needed is the media bus type. That info is most easily
> available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> node.

I haven't had time to check the code in details yet, so I can't really comment 
on what you need and how it should be implemented exactly.

> The media bus type is not something that can be added to the
> imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

> > In the omap3isp case, we use the operation to query whether parallel data
> > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > 
> >> The reason I am suspicious about this op is that it came from soc-camera
> >> and predates the DT. The contents of v4l2_mbus_config seems very much
> >> like a HW description to me, i.e. something that belongs in the DT.
> > 
> > Part of it is possibly outdated, but for buses that support multiple modes
> > of operation (such as the parallel bus case described above) we need to
> > make that information discoverable at runtime. Maybe this should be
> > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > supported through the API he is working on.
> 
> That sounds interesting, can you point me to some info on this effort?

Sure.

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

> I've been thinking the DT should contain virtual channel info for CSI-2
> buses.

I don't think it should. CSI-2 virtual channels and data types should be 
handled as a software concept, and thus supported through driver code without 
involving DT.

-- 
Regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 10:26                   ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 10:26 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>> +
> >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>> v4l2_mbus_config *cfg)

[snip]

> >>>>>> I am not certain this op is needed at all. In the current kernel this
> >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> >>>>>> dubious). Normally this information should come from the device tree
> >>>>>> and there should be no need for this op.
> >>>>>> 
> >>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>> 
> >>>>>> If you don't need it, then I recommend it is removed.
> >>>> 
> >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>> type. This info was already available from parsing a v4l2_of_endpoint
> >>>> from the sensor node. So it was simple to remove the g_mbus_config
> >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> >>> 
> >>> That's not a good point.
> > 
> > (mea culpa, s/point/idea/)
> > 
> >>> The imx-media driver must not parse the sensor DT node as it is not
> >>> aware of what bindings the sensor is compatible with.
> 
> Hi Laurent,
> 
> I don't really understand this argument. The sensor node has been found
> by parsing the OF graph, so it is known to be a camera sensor node at
> that point.

All you know in the i.MX6 driver is that the remote node is a video source. 
You can rely on the fact that it implements the OF graph bindings to locate 
other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible 
string. Unless you match on sensor compat strings in the i.MX6 driver (which 
you shouldn't do, to keep the driver generic) you can't know for certain how 
to parse the sensor node DT properties. For all you know, the video source 
could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
can't even rely on the fact that it's a sensor.

> >>> Information must instead be queried from the sensor subdev at runtime,
> >>> through the g_mbus_config() operation.
> >>> 
> >>> Of course, if you can get the information from the imx-media DT node,
> >>> that's certainly an option. It's only information provided by the sensor
> >>> driver that you have no choice but query using a subdev operation.
> >> 
> >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >> using this?
> > 
> > It all depends on what type of information needs to be retrieved, and
> > whether it can change at runtime or is fixed. Adding properties to the
> > imx-media DT node is certainly fine as long as those properties describe
> > the i.MX side.
>
> In this case the info needed is the media bus type. That info is most easily
> available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> node.

I haven't had time to check the code in details yet, so I can't really comment 
on what you need and how it should be implemented exactly.

> The media bus type is not something that can be added to the
> imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

> > In the omap3isp case, we use the operation to query whether parallel data
> > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > 
> >> The reason I am suspicious about this op is that it came from soc-camera
> >> and predates the DT. The contents of v4l2_mbus_config seems very much
> >> like a HW description to me, i.e. something that belongs in the DT.
> > 
> > Part of it is possibly outdated, but for buses that support multiple modes
> > of operation (such as the parallel bus case described above) we need to
> > make that information discoverable at runtime. Maybe this should be
> > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > supported through the API he is working on.
> 
> That sounds interesting, can you point me to some info on this effort?

Sure.

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

> I've been thinking the DT should contain virtual channel info for CSI-2
> buses.

I don't think it should. CSI-2 virtual channels and data types should be 
handled as a software concept, and thus supported through driver code without 
involving DT.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-07 10:26                   ` Laurent Pinchart
  (?)
@ 2017-02-07 10:41                     ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-07 10:41 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus

On Tue, 2017-02-07 at 12:26 +0200, Laurent Pinchart wrote:
> Hi Steve,
> 
> On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > >>>>>>> +
> > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > >>>>>>> v4l2_mbus_config *cfg)
> 
> [snip]
> 
> > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > >>>>>> dubious). Normally this information should come from the device tree
> > >>>>>> and there should be no need for this op.
> > >>>>>> 
> > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > >>>>>> 
> > >>>>>> If you don't need it, then I recommend it is removed.
> > >>>> 
> > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > >>> 
> > >>> That's not a good point.
> > > 
> > > (mea culpa, s/point/idea/)
> > > 
> > >>> The imx-media driver must not parse the sensor DT node as it is not
> > >>> aware of what bindings the sensor is compatible with.
> > 
> > Hi Laurent,
> > 
> > I don't really understand this argument. The sensor node has been found
> > by parsing the OF graph, so it is known to be a camera sensor node at
> > that point.
> 
> All you know in the i.MX6 driver is that the remote node is a video source. 
> You can rely on the fact that it implements the OF graph bindings to locate 
> other ports in that DT node, but that's more or less it.
> 
> DT properties are defined by DT bindings and thus qualified by a compatible 
> string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> you shouldn't do, to keep the driver generic) you can't know for certain how 
> to parse the sensor node DT properties. For all you know, the video source 
> could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> can't even rely on the fact that it's a sensor.
> 
> > >>> Information must instead be queried from the sensor subdev at runtime,
> > >>> through the g_mbus_config() operation.
> > >>> 
> > >>> Of course, if you can get the information from the imx-media DT node,
> > >>> that's certainly an option. It's only information provided by the sensor
> > >>> driver that you have no choice but query using a subdev operation.
> > >> 
> > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > >> using this?
> > > 
> > > It all depends on what type of information needs to be retrieved, and
> > > whether it can change at runtime or is fixed. Adding properties to the
> > > imx-media DT node is certainly fine as long as those properties describe
> > > the i.MX side.
> >
> > In this case the info needed is the media bus type. That info is most easily
> > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > node.
> 
> I haven't had time to check the code in details yet, so I can't really comment 
> on what you need and how it should be implemented exactly.
> 
> > The media bus type is not something that can be added to the
> > imx-media node since it contains no endpoint nodes.
> 
> Agreed. You have endpoints in the CSI nodes though.
> 
> > > In the omap3isp case, we use the operation to query whether parallel data
> > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > 
> > >> The reason I am suspicious about this op is that it came from soc-camera
> > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > >> like a HW description to me, i.e. something that belongs in the DT.
> > > 
> > > Part of it is possibly outdated, but for buses that support multiple modes
> > > of operation (such as the parallel bus case described above) we need to
> > > make that information discoverable at runtime. Maybe this should be
> > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > supported through the API he is working on.
> > 
> > That sounds interesting, can you point me to some info on this effort?
> 
> Sure.
> 
> http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> 
> > I've been thinking the DT should contain virtual channel info for CSI-2
> > buses.
> 
> I don't think it should. CSI-2 virtual channels and data types should be 
> handled as a software concept, and thus supported through driver code without 
> involving DT.

I agree. The CSI2IPU gasket is a bit special in that it distributes its
input data to four different parallel buses depending on the input's VC,
but upstream of the MIPI CSI-2 receiver, any virtual channel information
is purely a matter of the data sent over the CSI-2 link, and not board
specific hardware description.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 10:41                     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-07 10:41 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA, devel

On Tue, 2017-02-07 at 12:26 +0200, Laurent Pinchart wrote:
> Hi Steve,
> 
> On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > >>>>>>> +
> > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > >>>>>>> v4l2_mbus_config *cfg)
> 
> [snip]
> 
> > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > >>>>>> dubious). Normally this information should come from the device tree
> > >>>>>> and there should be no need for this op.
> > >>>>>> 
> > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > >>>>>> 
> > >>>>>> If you don't need it, then I recommend it is removed.
> > >>>> 
> > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > >>> 
> > >>> That's not a good point.
> > > 
> > > (mea culpa, s/point/idea/)
> > > 
> > >>> The imx-media driver must not parse the sensor DT node as it is not
> > >>> aware of what bindings the sensor is compatible with.
> > 
> > Hi Laurent,
> > 
> > I don't really understand this argument. The sensor node has been found
> > by parsing the OF graph, so it is known to be a camera sensor node at
> > that point.
> 
> All you know in the i.MX6 driver is that the remote node is a video source. 
> You can rely on the fact that it implements the OF graph bindings to locate 
> other ports in that DT node, but that's more or less it.
> 
> DT properties are defined by DT bindings and thus qualified by a compatible 
> string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> you shouldn't do, to keep the driver generic) you can't know for certain how 
> to parse the sensor node DT properties. For all you know, the video source 
> could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> can't even rely on the fact that it's a sensor.
> 
> > >>> Information must instead be queried from the sensor subdev at runtime,
> > >>> through the g_mbus_config() operation.
> > >>> 
> > >>> Of course, if you can get the information from the imx-media DT node,
> > >>> that's certainly an option. It's only information provided by the sensor
> > >>> driver that you have no choice but query using a subdev operation.
> > >> 
> > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > >> using this?
> > > 
> > > It all depends on what type of information needs to be retrieved, and
> > > whether it can change at runtime or is fixed. Adding properties to the
> > > imx-media DT node is certainly fine as long as those properties describe
> > > the i.MX side.
> >
> > In this case the info needed is the media bus type. That info is most easily
> > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > node.
> 
> I haven't had time to check the code in details yet, so I can't really comment 
> on what you need and how it should be implemented exactly.
> 
> > The media bus type is not something that can be added to the
> > imx-media node since it contains no endpoint nodes.
> 
> Agreed. You have endpoints in the CSI nodes though.
> 
> > > In the omap3isp case, we use the operation to query whether parallel data
> > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > 
> > >> The reason I am suspicious about this op is that it came from soc-camera
> > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > >> like a HW description to me, i.e. something that belongs in the DT.
> > > 
> > > Part of it is possibly outdated, but for buses that support multiple modes
> > > of operation (such as the parallel bus case described above) we need to
> > > make that information discoverable at runtime. Maybe this should be
> > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > supported through the API he is working on.
> > 
> > That sounds interesting, can you point me to some info on this effort?
> 
> Sure.
> 
> http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> 
> > I've been thinking the DT should contain virtual channel info for CSI-2
> > buses.
> 
> I don't think it should. CSI-2 virtual channels and data types should be 
> handled as a software concept, and thus supported through driver code without 
> involving DT.

I agree. The CSI2IPU gasket is a bit special in that it distributes its
input data to four different parallel buses depending on the input's VC,
but upstream of the MIPI CSI-2 receiver, any virtual channel information
is purely a matter of the data sent over the CSI-2 link, and not board
specific hardware description.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 10:41                     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-07 10:41 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, 2017-02-07 at 12:26 +0200, Laurent Pinchart wrote:
> Hi Steve,
> 
> On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > >>>>>>> +
> > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > >>>>>>> v4l2_mbus_config *cfg)
> 
> [snip]
> 
> > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > >>>>>> dubious). Normally this information should come from the device tree
> > >>>>>> and there should be no need for this op.
> > >>>>>> 
> > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > >>>>>> 
> > >>>>>> If you don't need it, then I recommend it is removed.
> > >>>> 
> > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > >>> 
> > >>> That's not a good point.
> > > 
> > > (mea culpa, s/point/idea/)
> > > 
> > >>> The imx-media driver must not parse the sensor DT node as it is not
> > >>> aware of what bindings the sensor is compatible with.
> > 
> > Hi Laurent,
> > 
> > I don't really understand this argument. The sensor node has been found
> > by parsing the OF graph, so it is known to be a camera sensor node at
> > that point.
> 
> All you know in the i.MX6 driver is that the remote node is a video source. 
> You can rely on the fact that it implements the OF graph bindings to locate 
> other ports in that DT node, but that's more or less it.
> 
> DT properties are defined by DT bindings and thus qualified by a compatible 
> string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> you shouldn't do, to keep the driver generic) you can't know for certain how 
> to parse the sensor node DT properties. For all you know, the video source 
> could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> can't even rely on the fact that it's a sensor.
> 
> > >>> Information must instead be queried from the sensor subdev at runtime,
> > >>> through the g_mbus_config() operation.
> > >>> 
> > >>> Of course, if you can get the information from the imx-media DT node,
> > >>> that's certainly an option. It's only information provided by the sensor
> > >>> driver that you have no choice but query using a subdev operation.
> > >> 
> > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > >> using this?
> > > 
> > > It all depends on what type of information needs to be retrieved, and
> > > whether it can change at runtime or is fixed. Adding properties to the
> > > imx-media DT node is certainly fine as long as those properties describe
> > > the i.MX side.
> >
> > In this case the info needed is the media bus type. That info is most easily
> > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > node.
> 
> I haven't had time to check the code in details yet, so I can't really comment 
> on what you need and how it should be implemented exactly.
> 
> > The media bus type is not something that can be added to the
> > imx-media node since it contains no endpoint nodes.
> 
> Agreed. You have endpoints in the CSI nodes though.
> 
> > > In the omap3isp case, we use the operation to query whether parallel data
> > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > 
> > >> The reason I am suspicious about this op is that it came from soc-camera
> > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > >> like a HW description to me, i.e. something that belongs in the DT.
> > > 
> > > Part of it is possibly outdated, but for buses that support multiple modes
> > > of operation (such as the parallel bus case described above) we need to
> > > make that information discoverable at runtime. Maybe this should be
> > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > supported through the API he is working on.
> > 
> > That sounds interesting, can you point me to some info on this effort?
> 
> Sure.
> 
> http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> 
> > I've been thinking the DT should contain virtual channel info for CSI-2
> > buses.
> 
> I don't think it should. CSI-2 virtual channels and data types should be 
> handled as a software concept, and thus supported through driver code without 
> involving DT.

I agree. The CSI2IPU gasket is a bit special in that it distributes its
input data to four different parallel buses depending on the input's VC,
but upstream of the MIPI CSI-2 receiver, any virtual channel information
is purely a matter of the data sent over the CSI-2 link, and not board
specific hardware description.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-07 10:26                   ` Laurent Pinchart
  (?)
@ 2017-02-07 13:36                     ` Benoit Parrot
  -1 siblings, 0 replies; 549+ messages in thread
From: Benoit Parrot @ 2017-02-07 13:36 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Steve Longerbeam, Hans Verkuil, Philipp Zabel, robh+dt,
	mark.rutland, shawnguo, kernel, fabio.estevam, linux, mchehab,
	nick, markus.heiser, laurent.pinchart+renesas, geert, arnd,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus

Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> Hi Steve,
> 
> On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > >>>>>>> +
> > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > >>>>>>> v4l2_mbus_config *cfg)
> 
> [snip]
> 
> > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > >>>>>> dubious). Normally this information should come from the device tree
> > >>>>>> and there should be no need for this op.
> > >>>>>> 
> > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > >>>>>> 
> > >>>>>> If you don't need it, then I recommend it is removed.
> > >>>> 
> > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > >>> 
> > >>> That's not a good point.
> > > 
> > > (mea culpa, s/point/idea/)
> > > 
> > >>> The imx-media driver must not parse the sensor DT node as it is not
> > >>> aware of what bindings the sensor is compatible with.
> > 
> > Hi Laurent,
> > 
> > I don't really understand this argument. The sensor node has been found
> > by parsing the OF graph, so it is known to be a camera sensor node at
> > that point.
> 
> All you know in the i.MX6 driver is that the remote node is a video source. 
> You can rely on the fact that it implements the OF graph bindings to locate 
> other ports in that DT node, but that's more or less it.
> 
> DT properties are defined by DT bindings and thus qualified by a compatible 
> string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> you shouldn't do, to keep the driver generic) you can't know for certain how 
> to parse the sensor node DT properties. For all you know, the video source 
> could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> can't even rely on the fact that it's a sensor.
> 
> > >>> Information must instead be queried from the sensor subdev at runtime,
> > >>> through the g_mbus_config() operation.
> > >>> 
> > >>> Of course, if you can get the information from the imx-media DT node,
> > >>> that's certainly an option. It's only information provided by the sensor
> > >>> driver that you have no choice but query using a subdev operation.
> > >> 
> > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > >> using this?
> > > 
> > > It all depends on what type of information needs to be retrieved, and
> > > whether it can change at runtime or is fixed. Adding properties to the
> > > imx-media DT node is certainly fine as long as those properties describe
> > > the i.MX side.
> >
> > In this case the info needed is the media bus type. That info is most easily
> > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > node.
> 
> I haven't had time to check the code in details yet, so I can't really comment 
> on what you need and how it should be implemented exactly.
> 
> > The media bus type is not something that can be added to the
> > imx-media node since it contains no endpoint nodes.
> 
> Agreed. You have endpoints in the CSI nodes though.
> 
> > > In the omap3isp case, we use the operation to query whether parallel data
> > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > 
> > >> The reason I am suspicious about this op is that it came from soc-camera
> > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > >> like a HW description to me, i.e. something that belongs in the DT.
> > > 
> > > Part of it is possibly outdated, but for buses that support multiple modes
> > > of operation (such as the parallel bus case described above) we need to
> > > make that information discoverable at runtime. Maybe this should be
> > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > supported through the API he is working on.
> > 
> > That sounds interesting, can you point me to some info on this effort?
> 
> Sure.
> 
> http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> 
> > I've been thinking the DT should contain virtual channel info for CSI-2
> > buses.
> 
> I don't think it should. CSI-2 virtual channels and data types should be 
> handled as a software concept, and thus supported through driver code without 
> involving DT.

Laurent,

So when you have a CSI2 port aggregator for instance where traffic from up
to 4 CSI2 sources where each source is now assigned its own VC by the
aggregator and interleaved into a single CSI2 Receiver. I was hoping that
in this case the VC would be DT discoverable as a specicic source identifier.
So the CSI-RX side could associate a specific source and create its own
video device. I am guessing that no such thing exist today?

Benoit

> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 13:36                     ` Benoit Parrot
  0 siblings, 0 replies; 549+ messages in thread
From: Benoit Parrot @ 2017-02-07 13:36 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, sakari.ailus, nick,
	songjun.wu, Hans Verkuil, Steve Longerbeam, robert.jarzmik,
	devel, markus.heiser, laurent.pinchart+renesas, linux, geert,
	Steve Longerbeam, Sascha Hauer, linux-media, devicetree,
	Philipp Zabel, arnd, mchehab, robh+dt, horms+renesas,
	tiffany.lin, linux-arm-kernel, niklas.soderlund+renesas, gregkh,
	linux-kernel, jean-christophe.trotin, kernel

Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> Hi Steve,
> 
> On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > >>>>>>> +
> > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > >>>>>>> v4l2_mbus_config *cfg)
> 
> [snip]
> 
> > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > >>>>>> dubious). Normally this information should come from the device tree
> > >>>>>> and there should be no need for this op.
> > >>>>>> 
> > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > >>>>>> 
> > >>>>>> If you don't need it, then I recommend it is removed.
> > >>>> 
> > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > >>> 
> > >>> That's not a good point.
> > > 
> > > (mea culpa, s/point/idea/)
> > > 
> > >>> The imx-media driver must not parse the sensor DT node as it is not
> > >>> aware of what bindings the sensor is compatible with.
> > 
> > Hi Laurent,
> > 
> > I don't really understand this argument. The sensor node has been found
> > by parsing the OF graph, so it is known to be a camera sensor node at
> > that point.
> 
> All you know in the i.MX6 driver is that the remote node is a video source. 
> You can rely on the fact that it implements the OF graph bindings to locate 
> other ports in that DT node, but that's more or less it.
> 
> DT properties are defined by DT bindings and thus qualified by a compatible 
> string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> you shouldn't do, to keep the driver generic) you can't know for certain how 
> to parse the sensor node DT properties. For all you know, the video source 
> could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> can't even rely on the fact that it's a sensor.
> 
> > >>> Information must instead be queried from the sensor subdev at runtime,
> > >>> through the g_mbus_config() operation.
> > >>> 
> > >>> Of course, if you can get the information from the imx-media DT node,
> > >>> that's certainly an option. It's only information provided by the sensor
> > >>> driver that you have no choice but query using a subdev operation.
> > >> 
> > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > >> using this?
> > > 
> > > It all depends on what type of information needs to be retrieved, and
> > > whether it can change at runtime or is fixed. Adding properties to the
> > > imx-media DT node is certainly fine as long as those properties describe
> > > the i.MX side.
> >
> > In this case the info needed is the media bus type. That info is most easily
> > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > node.
> 
> I haven't had time to check the code in details yet, so I can't really comment 
> on what you need and how it should be implemented exactly.
> 
> > The media bus type is not something that can be added to the
> > imx-media node since it contains no endpoint nodes.
> 
> Agreed. You have endpoints in the CSI nodes though.
> 
> > > In the omap3isp case, we use the operation to query whether parallel data
> > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > 
> > >> The reason I am suspicious about this op is that it came from soc-camera
> > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > >> like a HW description to me, i.e. something that belongs in the DT.
> > > 
> > > Part of it is possibly outdated, but for buses that support multiple modes
> > > of operation (such as the parallel bus case described above) we need to
> > > make that information discoverable at runtime. Maybe this should be
> > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > supported through the API he is working on.
> > 
> > That sounds interesting, can you point me to some info on this effort?
> 
> Sure.
> 
> http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> 
> > I've been thinking the DT should contain virtual channel info for CSI-2
> > buses.
> 
> I don't think it should. CSI-2 virtual channels and data types should be 
> handled as a software concept, and thus supported through driver code without 
> involving DT.

Laurent,

So when you have a CSI2 port aggregator for instance where traffic from up
to 4 CSI2 sources where each source is now assigned its own VC by the
aggregator and interleaved into a single CSI2 Receiver. I was hoping that
in this case the VC would be DT discoverable as a specicic source identifier.
So the CSI-RX side could associate a specific source and create its own
video device. I am guessing that no such thing exist today?

Benoit

> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 13:36                     ` Benoit Parrot
  0 siblings, 0 replies; 549+ messages in thread
From: Benoit Parrot @ 2017-02-07 13:36 UTC (permalink / raw)
  To: linux-arm-kernel

Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> Hi Steve,
> 
> On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > >>>>>>> +
> > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > >>>>>>> v4l2_mbus_config *cfg)
> 
> [snip]
> 
> > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > >>>>>> dubious). Normally this information should come from the device tree
> > >>>>>> and there should be no need for this op.
> > >>>>>> 
> > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > >>>>>> 
> > >>>>>> If you don't need it, then I recommend it is removed.
> > >>>> 
> > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > >>> 
> > >>> That's not a good point.
> > > 
> > > (mea culpa, s/point/idea/)
> > > 
> > >>> The imx-media driver must not parse the sensor DT node as it is not
> > >>> aware of what bindings the sensor is compatible with.
> > 
> > Hi Laurent,
> > 
> > I don't really understand this argument. The sensor node has been found
> > by parsing the OF graph, so it is known to be a camera sensor node at
> > that point.
> 
> All you know in the i.MX6 driver is that the remote node is a video source. 
> You can rely on the fact that it implements the OF graph bindings to locate 
> other ports in that DT node, but that's more or less it.
> 
> DT properties are defined by DT bindings and thus qualified by a compatible 
> string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> you shouldn't do, to keep the driver generic) you can't know for certain how 
> to parse the sensor node DT properties. For all you know, the video source 
> could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> can't even rely on the fact that it's a sensor.
> 
> > >>> Information must instead be queried from the sensor subdev at runtime,
> > >>> through the g_mbus_config() operation.
> > >>> 
> > >>> Of course, if you can get the information from the imx-media DT node,
> > >>> that's certainly an option. It's only information provided by the sensor
> > >>> driver that you have no choice but query using a subdev operation.
> > >> 
> > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > >> using this?
> > > 
> > > It all depends on what type of information needs to be retrieved, and
> > > whether it can change at runtime or is fixed. Adding properties to the
> > > imx-media DT node is certainly fine as long as those properties describe
> > > the i.MX side.
> >
> > In this case the info needed is the media bus type. That info is most easily
> > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > node.
> 
> I haven't had time to check the code in details yet, so I can't really comment 
> on what you need and how it should be implemented exactly.
> 
> > The media bus type is not something that can be added to the
> > imx-media node since it contains no endpoint nodes.
> 
> Agreed. You have endpoints in the CSI nodes though.
> 
> > > In the omap3isp case, we use the operation to query whether parallel data
> > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > 
> > >> The reason I am suspicious about this op is that it came from soc-camera
> > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > >> like a HW description to me, i.e. something that belongs in the DT.
> > > 
> > > Part of it is possibly outdated, but for buses that support multiple modes
> > > of operation (such as the parallel bus case described above) we need to
> > > make that information discoverable at runtime. Maybe this should be
> > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > supported through the API he is working on.
> > 
> > That sounds interesting, can you point me to some info on this effort?
> 
> Sure.
> 
> http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> 
> > I've been thinking the DT should contain virtual channel info for CSI-2
> > buses.
> 
> I don't think it should. CSI-2 virtual channels and data types should be 
> handled as a software concept, and thus supported through driver code without 
> involving DT.

Laurent,

So when you have a CSI2 port aggregator for instance where traffic from up
to 4 CSI2 sources where each source is now assigned its own VC by the
aggregator and interleaved into a single CSI2 Receiver. I was hoping that
in this case the VC would be DT discoverable as a specicic source identifier.
So the CSI-RX side could associate a specific source and create its own
video device. I am guessing that no such thing exist today?

Benoit

> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 14:11                       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 14:11 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, linux, mchehab, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus

Hi Philipp,

On Tuesday 07 Feb 2017 11:41:30 Philipp Zabel wrote:
> On Tue, 2017-02-07 at 12:26 +0200, Laurent Pinchart wrote:
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> >> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> >>> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >>>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>>>> +
> >>>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> >>>>>>>> I am not certain this op is needed at all. In the current kernel
> >>>>>>>> this op is only used by soc_camera, pxa_camera and omap3isp
> >>>>>>>> (somewhat dubious). Normally this information should come from the
> >>>>>>>> device tree and there should be no need for this op.
> >>>>>>>> 
> >>>>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>>>> 
> >>>>>>>> If you don't need it, then I recommend it is removed.
> >>>>>> 
> >>>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>>>> type. This info was already available from parsing a
> >>>>>> v4l2_of_endpoint from the sensor node. So it was simple to remove the
> >>>>>> g_mbus_config calls, and instead rely on the parsed sensor
> >>>>>> v4l2_of_endpoint.
> >>>>> 
> >>>>> That's not a good point.
> >>> 
> >>> (mea culpa, s/point/idea/)
> >>> 
> >>>>> The imx-media driver must not parse the sensor DT node as it is not
> >>>>> aware of what bindings the sensor is compatible with.
> >> 
> >> Hi Laurent,
> >> 
> >> I don't really understand this argument. The sensor node has been found
> >> by parsing the OF graph, so it is known to be a camera sensor node at
> >> that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video
> > source. You can rely on the fact that it implements the OF graph bindings
> > to locate other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a
> > compatible string. Unless you match on sensor compat strings in the i.MX6
> > driver (which you shouldn't do, to keep the driver generic) you can't know
> > for certain how to parse the sensor node DT properties. For all you know,
> > the video source could be a bridge such as an HDMI to CSI-2 converter for
> > instance, so you can't even rely on the fact that it's a sensor.
> > 
> >>>>> Information must instead be queried from the sensor subdev at
> >>>>> runtime, through the g_mbus_config() operation.
> >>>>> 
> >>>>> Of course, if you can get the information from the imx-media DT
> >>>>> node, that's certainly an option. It's only information provided by
> >>>>> the sensor driver that you have no choice but query using a subdev
> >>>>> operation.
> >>>> 
> >>>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >>>> using this?
> >>> 
> >>> It all depends on what type of information needs to be retrieved, and
> >>> whether it can change at runtime or is fixed. Adding properties to the
> >>> imx-media DT node is certainly fine as long as those properties
> >>> describe the i.MX side.
> >> 
> >> In this case the info needed is the media bus type. That info is most
> >> easily available by calling v4l2_of_parse_endpoint() on the sensor's
> >> endpoint node.
> > 
> > I haven't had time to check the code in details yet, so I can't really
> > comment on what you need and how it should be implemented exactly.
> > 
> >> The media bus type is not something that can be added to the
> >> imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> >>> In the omap3isp case, we use the operation to query whether parallel
> >>> data contains embedded sync (BT.656) or uses separate h/v sync signals.
> >>> 
> >>>> The reason I am suspicious about this op is that it came from
> >>>> soc-camera and predates the DT. The contents of v4l2_mbus_config seems
> >>>> very much like a HW description to me, i.e. something that belongs in
> >>>> the DT.
> >>> 
> >>> Part of it is possibly outdated, but for buses that support multiple
> >>> modes of operation (such as the parallel bus case described above) we
> >>> need to make that information discoverable at runtime. Maybe this should
> >>> be considered as related to Sakari's efforts to support VC/DT for CSI-2,
> >>> and supported through the API he is working on.
> >>
> >> That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> >> I've been thinking the DT should contain virtual channel info for CSI-2
> >> buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be
> > handled as a software concept, and thus supported through driver code
> > without involving DT.
> 
> I agree. The CSI2IPU gasket is a bit special in that it distributes its
> input data to four different parallel buses depending on the input's VC,
> but upstream of the MIPI CSI-2 receiver, any virtual channel information
> is purely a matter of the data sent over the CSI-2 link, and not board
> specific hardware description.

If the CSI2IPU gasket has four physical output buses then it can be modelled 
as an entity with four source pads. Only when VC/DT are multiplexed on the 
same physical bus do I think they should be handled without involving the 
device tree and the media controller graph.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 14:11                       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 14:11 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Steve Longerbeam, Hans Verkuil, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	linux-I+IVW8TIWO2tmTQ+vhA3Yw, mchehab-DgEjT+Ai2ygdnm+yROfE0A,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-m1Uo1GnMJf0b1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA, devel

Hi Philipp,

On Tuesday 07 Feb 2017 11:41:30 Philipp Zabel wrote:
> On Tue, 2017-02-07 at 12:26 +0200, Laurent Pinchart wrote:
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> >> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> >>> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >>>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>>>> +
> >>>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> >>>>>>>> I am not certain this op is needed at all. In the current kernel
> >>>>>>>> this op is only used by soc_camera, pxa_camera and omap3isp
> >>>>>>>> (somewhat dubious). Normally this information should come from the
> >>>>>>>> device tree and there should be no need for this op.
> >>>>>>>> 
> >>>>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>>>> 
> >>>>>>>> If you don't need it, then I recommend it is removed.
> >>>>>> 
> >>>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>>>> type. This info was already available from parsing a
> >>>>>> v4l2_of_endpoint from the sensor node. So it was simple to remove the
> >>>>>> g_mbus_config calls, and instead rely on the parsed sensor
> >>>>>> v4l2_of_endpoint.
> >>>>> 
> >>>>> That's not a good point.
> >>> 
> >>> (mea culpa, s/point/idea/)
> >>> 
> >>>>> The imx-media driver must not parse the sensor DT node as it is not
> >>>>> aware of what bindings the sensor is compatible with.
> >> 
> >> Hi Laurent,
> >> 
> >> I don't really understand this argument. The sensor node has been found
> >> by parsing the OF graph, so it is known to be a camera sensor node at
> >> that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video
> > source. You can rely on the fact that it implements the OF graph bindings
> > to locate other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a
> > compatible string. Unless you match on sensor compat strings in the i.MX6
> > driver (which you shouldn't do, to keep the driver generic) you can't know
> > for certain how to parse the sensor node DT properties. For all you know,
> > the video source could be a bridge such as an HDMI to CSI-2 converter for
> > instance, so you can't even rely on the fact that it's a sensor.
> > 
> >>>>> Information must instead be queried from the sensor subdev at
> >>>>> runtime, through the g_mbus_config() operation.
> >>>>> 
> >>>>> Of course, if you can get the information from the imx-media DT
> >>>>> node, that's certainly an option. It's only information provided by
> >>>>> the sensor driver that you have no choice but query using a subdev
> >>>>> operation.
> >>>> 
> >>>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >>>> using this?
> >>> 
> >>> It all depends on what type of information needs to be retrieved, and
> >>> whether it can change at runtime or is fixed. Adding properties to the
> >>> imx-media DT node is certainly fine as long as those properties
> >>> describe the i.MX side.
> >> 
> >> In this case the info needed is the media bus type. That info is most
> >> easily available by calling v4l2_of_parse_endpoint() on the sensor's
> >> endpoint node.
> > 
> > I haven't had time to check the code in details yet, so I can't really
> > comment on what you need and how it should be implemented exactly.
> > 
> >> The media bus type is not something that can be added to the
> >> imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> >>> In the omap3isp case, we use the operation to query whether parallel
> >>> data contains embedded sync (BT.656) or uses separate h/v sync signals.
> >>> 
> >>>> The reason I am suspicious about this op is that it came from
> >>>> soc-camera and predates the DT. The contents of v4l2_mbus_config seems
> >>>> very much like a HW description to me, i.e. something that belongs in
> >>>> the DT.
> >>> 
> >>> Part of it is possibly outdated, but for buses that support multiple
> >>> modes of operation (such as the parallel bus case described above) we
> >>> need to make that information discoverable at runtime. Maybe this should
> >>> be considered as related to Sakari's efforts to support VC/DT for CSI-2,
> >>> and supported through the API he is working on.
> >>
> >> That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> >> I've been thinking the DT should contain virtual channel info for CSI-2
> >> buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be
> > handled as a software concept, and thus supported through driver code
> > without involving DT.
> 
> I agree. The CSI2IPU gasket is a bit special in that it distributes its
> input data to four different parallel buses depending on the input's VC,
> but upstream of the MIPI CSI-2 receiver, any virtual channel information
> is purely a matter of the data sent over the CSI-2 link, and not board
> specific hardware description.

If the CSI2IPU gasket has four physical output buses then it can be modelled 
as an entity with four source pads. Only when VC/DT are multiplexed on the 
same physical bus do I think they should be handled without involving the 
device tree and the media controller graph.

-- 
Regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 14:11                       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 14:11 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Philipp,

On Tuesday 07 Feb 2017 11:41:30 Philipp Zabel wrote:
> On Tue, 2017-02-07 at 12:26 +0200, Laurent Pinchart wrote:
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> >> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> >>> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >>>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>>>> +
> >>>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> >>>>>>>> I am not certain this op is needed at all. In the current kernel
> >>>>>>>> this op is only used by soc_camera, pxa_camera and omap3isp
> >>>>>>>> (somewhat dubious). Normally this information should come from the
> >>>>>>>> device tree and there should be no need for this op.
> >>>>>>>> 
> >>>>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>>>> 
> >>>>>>>> If you don't need it, then I recommend it is removed.
> >>>>>> 
> >>>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>>>> type. This info was already available from parsing a
> >>>>>> v4l2_of_endpoint from the sensor node. So it was simple to remove the
> >>>>>> g_mbus_config calls, and instead rely on the parsed sensor
> >>>>>> v4l2_of_endpoint.
> >>>>> 
> >>>>> That's not a good point.
> >>> 
> >>> (mea culpa, s/point/idea/)
> >>> 
> >>>>> The imx-media driver must not parse the sensor DT node as it is not
> >>>>> aware of what bindings the sensor is compatible with.
> >> 
> >> Hi Laurent,
> >> 
> >> I don't really understand this argument. The sensor node has been found
> >> by parsing the OF graph, so it is known to be a camera sensor node at
> >> that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video
> > source. You can rely on the fact that it implements the OF graph bindings
> > to locate other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a
> > compatible string. Unless you match on sensor compat strings in the i.MX6
> > driver (which you shouldn't do, to keep the driver generic) you can't know
> > for certain how to parse the sensor node DT properties. For all you know,
> > the video source could be a bridge such as an HDMI to CSI-2 converter for
> > instance, so you can't even rely on the fact that it's a sensor.
> > 
> >>>>> Information must instead be queried from the sensor subdev at
> >>>>> runtime, through the g_mbus_config() operation.
> >>>>> 
> >>>>> Of course, if you can get the information from the imx-media DT
> >>>>> node, that's certainly an option. It's only information provided by
> >>>>> the sensor driver that you have no choice but query using a subdev
> >>>>> operation.
> >>>> 
> >>>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >>>> using this?
> >>> 
> >>> It all depends on what type of information needs to be retrieved, and
> >>> whether it can change at runtime or is fixed. Adding properties to the
> >>> imx-media DT node is certainly fine as long as those properties
> >>> describe the i.MX side.
> >> 
> >> In this case the info needed is the media bus type. That info is most
> >> easily available by calling v4l2_of_parse_endpoint() on the sensor's
> >> endpoint node.
> > 
> > I haven't had time to check the code in details yet, so I can't really
> > comment on what you need and how it should be implemented exactly.
> > 
> >> The media bus type is not something that can be added to the
> >> imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> >>> In the omap3isp case, we use the operation to query whether parallel
> >>> data contains embedded sync (BT.656) or uses separate h/v sync signals.
> >>> 
> >>>> The reason I am suspicious about this op is that it came from
> >>>> soc-camera and predates the DT. The contents of v4l2_mbus_config seems
> >>>> very much like a HW description to me, i.e. something that belongs in
> >>>> the DT.
> >>> 
> >>> Part of it is possibly outdated, but for buses that support multiple
> >>> modes of operation (such as the parallel bus case described above) we
> >>> need to make that information discoverable at runtime. Maybe this should
> >>> be considered as related to Sakari's efforts to support VC/DT for CSI-2,
> >>> and supported through the API he is working on.
> >>
> >> That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> >> I've been thinking the DT should contain virtual channel info for CSI-2
> >> buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be
> > handled as a software concept, and thus supported through driver code
> > without involving DT.
> 
> I agree. The CSI2IPU gasket is a bit special in that it distributes its
> input data to four different parallel buses depending on the input's VC,
> but upstream of the MIPI CSI-2 receiver, any virtual channel information
> is purely a matter of the data sent over the CSI-2 link, and not board
> specific hardware description.

If the CSI2IPU gasket has four physical output buses then it can be modelled 
as an entity with four source pads. Only when VC/DT are multiplexed on the 
same physical bus do I think they should be handled without involving the 
device tree and the media controller graph.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH] media: imx: csi: fix crop rectangle reset in sink set_fmt
@ 2017-02-07 16:18     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-07 16:18 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

The csi_try_crop call in set_fmt should compare the cropping rectangle
to the currently set input format, not to the previous input format.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
This is a patch against the current imx-media-staging-md-wip branch.
S_FMT wouldn't update the cropping rectangle during the first time it is
called, since the csi_try_crop call would compare to the input frame
format returned by __csi_get_fmt, which still contains the old value.
---
 drivers/staging/media/imx/imx-media-csi.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c
b/drivers/staging/media/imx/imx-media-csi.c
index 637d0f137938a..7c590bf94fad5 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -820,15 +820,13 @@ __csi_get_fmt(struct csi_priv *priv, struct
v4l2_subdev_pad_config *cfg,
 static int csi_try_crop(struct csi_priv *priv,
 			struct v4l2_rect *crop,
 			struct v4l2_subdev_pad_config *cfg,
-			enum v4l2_subdev_format_whence which,
+			struct v4l2_mbus_framefmt *infmt,
 			struct imx_media_subdev *sensor)
 {
 	struct v4l2_of_endpoint *sensor_ep;
-	struct v4l2_mbus_framefmt *infmt;
 	v4l2_std_id std;
 	int ret;
 
-	infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, which);
 	sensor_ep = &sensor->sensor_ep;
 
 	crop->width = min_t(__u32, infmt->width, crop->width);
@@ -1023,8 +1021,7 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 		crop.top = 0;
 		crop.width = sdformat->format.width;
 		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, cfg,
-				   sdformat->which, sensor);
+		ret = csi_try_crop(priv, &crop, cfg, &sdformat->format, sensor);
 		if (ret)
 			return ret;
 
@@ -1104,6 +1101,7 @@ static int csi_set_selection(struct v4l2_subdev
*sd,
 			     struct v4l2_subdev_selection *sel)
 {
 	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt;
 	struct imx_media_subdev *sensor;
 	int ret;
 
@@ -1133,7 +1131,8 @@ static int csi_set_selection(struct v4l2_subdev
*sd,
 		return 0;
 	}
 
-	ret = csi_try_crop(priv, &sel->r, cfg, sel->which, sensor);
+	infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, sel->which);
+	ret = csi_try_crop(priv, &sel->r, cfg, infmt, sensor);
 	if (ret)
 		return ret;
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH] media: imx: csi: fix crop rectangle reset in sink set_fmt
@ 2017-02-07 16:18     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-07 16:18 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

The csi_try_crop call in set_fmt should compare the cropping rectangle
to the currently set input format, not to the previous input format.

Signed-off-by: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
---
This is a patch against the current imx-media-staging-md-wip branch.
S_FMT wouldn't update the cropping rectangle during the first time it is
called, since the csi_try_crop call would compare to the input frame
format returned by __csi_get_fmt, which still contains the old value.
---
 drivers/staging/media/imx/imx-media-csi.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c
b/drivers/staging/media/imx/imx-media-csi.c
index 637d0f137938a..7c590bf94fad5 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -820,15 +820,13 @@ __csi_get_fmt(struct csi_priv *priv, struct
v4l2_subdev_pad_config *cfg,
 static int csi_try_crop(struct csi_priv *priv,
 			struct v4l2_rect *crop,
 			struct v4l2_subdev_pad_config *cfg,
-			enum v4l2_subdev_format_whence which,
+			struct v4l2_mbus_framefmt *infmt,
 			struct imx_media_subdev *sensor)
 {
 	struct v4l2_of_endpoint *sensor_ep;
-	struct v4l2_mbus_framefmt *infmt;
 	v4l2_std_id std;
 	int ret;
 
-	infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, which);
 	sensor_ep = &sensor->sensor_ep;
 
 	crop->width = min_t(__u32, infmt->width, crop->width);
@@ -1023,8 +1021,7 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 		crop.top = 0;
 		crop.width = sdformat->format.width;
 		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, cfg,
-				   sdformat->which, sensor);
+		ret = csi_try_crop(priv, &crop, cfg, &sdformat->format, sensor);
 		if (ret)
 			return ret;
 
@@ -1104,6 +1101,7 @@ static int csi_set_selection(struct v4l2_subdev
*sd,
 			     struct v4l2_subdev_selection *sel)
 {
 	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt;
 	struct imx_media_subdev *sensor;
 	int ret;
 
@@ -1133,7 +1131,8 @@ static int csi_set_selection(struct v4l2_subdev
*sd,
 		return 0;
 	}
 
-	ret = csi_try_crop(priv, &sel->r, cfg, sel->which, sensor);
+	infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, sel->which);
+	ret = csi_try_crop(priv, &sel->r, cfg, infmt, sensor);
 	if (ret)
 		return ret;
 
-- 
2.11.0


--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH] media: imx: csi: fix crop rectangle reset in sink set_fmt
@ 2017-02-07 16:18     ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-07 16:18 UTC (permalink / raw)
  To: linux-arm-kernel

The csi_try_crop call in set_fmt should compare the cropping rectangle
to the currently set input format, not to the previous input format.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
This is a patch against the current imx-media-staging-md-wip branch.
S_FMT wouldn't update the cropping rectangle during the first time it is
called, since the csi_try_crop call would compare to the input frame
format returned by __csi_get_fmt, which still contains the old value.
---
 drivers/staging/media/imx/imx-media-csi.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c
b/drivers/staging/media/imx/imx-media-csi.c
index 637d0f137938a..7c590bf94fad5 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -820,15 +820,13 @@ __csi_get_fmt(struct csi_priv *priv, struct
v4l2_subdev_pad_config *cfg,
 static int csi_try_crop(struct csi_priv *priv,
 			struct v4l2_rect *crop,
 			struct v4l2_subdev_pad_config *cfg,
-			enum v4l2_subdev_format_whence which,
+			struct v4l2_mbus_framefmt *infmt,
 			struct imx_media_subdev *sensor)
 {
 	struct v4l2_of_endpoint *sensor_ep;
-	struct v4l2_mbus_framefmt *infmt;
 	v4l2_std_id std;
 	int ret;
 
-	infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, which);
 	sensor_ep = &sensor->sensor_ep;
 
 	crop->width = min_t(__u32, infmt->width, crop->width);
@@ -1023,8 +1021,7 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 		crop.top = 0;
 		crop.width = sdformat->format.width;
 		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, cfg,
-				   sdformat->which, sensor);
+		ret = csi_try_crop(priv, &crop, cfg, &sdformat->format, sensor);
 		if (ret)
 			return ret;
 
@@ -1104,6 +1101,7 @@ static int csi_set_selection(struct v4l2_subdev
*sd,
 			     struct v4l2_subdev_selection *sel)
 {
 	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt;
 	struct imx_media_subdev *sensor;
 	int ret;
 
@@ -1133,7 +1131,8 @@ static int csi_set_selection(struct v4l2_subdev
*sd,
 		return 0;
 	}
 
-	ret = csi_try_crop(priv, &sel->r, cfg, sel->which, sensor);
+	infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, sel->which);
+	ret = csi_try_crop(priv, &sel->r, cfg, infmt, sensor);
 	if (ret)
 		return ret;
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-01-07  2:11   ` Steve Longerbeam
  (?)
@ 2017-02-07 20:46     ` Sakari Ailus
  -1 siblings, 0 replies; 549+ messages in thread
From: Sakari Ailus @ 2017-02-07 20:46 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam

Hi Steve,

On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"
> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.
> +
> +Example:
> +
> +syscon {
> +	compatible = "syscon", "simple-mfd";
> +
> +	mux {

Could you use standardised properties for this, i.e. ones defined in
Documentation/devicetree/bindings/media/video-interfaces.txt ?

This is very similar to another patch "[PATCH] devicetree: Add video bus
switch" posted by Pavel Machek recently. The problem with that is also
similar than with this one: how to pass the CSI-2 bus configuration to the
receiver.

There's some discussion here:

<URL:http://www.spinics.net/lists/linux-media/msg109493.html>

As Laurent already suggested, I think we should have a common solution for
the problem that, besides conveying the bus parameters to the receiver, also
encompasses CSI-2 virtual channels and data types.

That would mean finishing the series of patches in the branch I believe
Laurent already quoted here.

> +		compatible = "video-multiplexer";
> +		/* Single bit (1 << 19) in syscon register 0x04: */
> +		reg = <0x04>;
> +		bit-mask = <1>;
> +		bit-shift = <19>;
> +		#address-cells = <1>;
> +		#size-cells = <0>;
> +
> +		port@0 {
> +			reg = <0>;
> +
> +			mux_in0: endpoint {
> +				remote-endpoint = <&video_source0_out>;
> +			};
> +		};
> +
> +		port@1 {
> +			reg = <1>;
> +
> +			mux_in1: endpoint {
> +				remote-endpoint = <&video_source1_out>;
> +			};
> +		};
> +
> +		port@2 {
> +			reg = <2>;
> +
> +			mux_out: endpoint {
> +				remote-endpoint = <&capture_interface_in>;
> +			};
> +		};
> +	};
> +};

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 20:46     ` Sakari Ailus
  0 siblings, 0 replies; 549+ messages in thread
From: Sakari Ailus @ 2017-02-07 20:46 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, Sascha Hauer,
	linux-media, devicetree, kernel, arnd, mchehab, bparrot, robh+dt,
	horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, p.zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

Hi Steve,

On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"
> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.
> +
> +Example:
> +
> +syscon {
> +	compatible = "syscon", "simple-mfd";
> +
> +	mux {

Could you use standardised properties for this, i.e. ones defined in
Documentation/devicetree/bindings/media/video-interfaces.txt ?

This is very similar to another patch "[PATCH] devicetree: Add video bus
switch" posted by Pavel Machek recently. The problem with that is also
similar than with this one: how to pass the CSI-2 bus configuration to the
receiver.

There's some discussion here:

<URL:http://www.spinics.net/lists/linux-media/msg109493.html>

As Laurent already suggested, I think we should have a common solution for
the problem that, besides conveying the bus parameters to the receiver, also
encompasses CSI-2 virtual channels and data types.

That would mean finishing the series of patches in the branch I believe
Laurent already quoted here.

> +		compatible = "video-multiplexer";
> +		/* Single bit (1 << 19) in syscon register 0x04: */
> +		reg = <0x04>;
> +		bit-mask = <1>;
> +		bit-shift = <19>;
> +		#address-cells = <1>;
> +		#size-cells = <0>;
> +
> +		port@0 {
> +			reg = <0>;
> +
> +			mux_in0: endpoint {
> +				remote-endpoint = <&video_source0_out>;
> +			};
> +		};
> +
> +		port@1 {
> +			reg = <1>;
> +
> +			mux_in1: endpoint {
> +				remote-endpoint = <&video_source1_out>;
> +			};
> +		};
> +
> +		port@2 {
> +			reg = <2>;
> +
> +			mux_out: endpoint {
> +				remote-endpoint = <&capture_interface_in>;
> +			};
> +		};
> +	};
> +};

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 20:46     ` Sakari Ailus
  0 siblings, 0 replies; 549+ messages in thread
From: Sakari Ailus @ 2017-02-07 20:46 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> This driver can handle SoC internal and external video bus multiplexers,
> controlled either by register bit fields or by a GPIO. The subdevice
> passes through frame interval and mbus configuration of the active input
> to the output side.
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> --
> 
> - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
>   should be unregister.
> 
> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
>   to vidsw_remove().
> 
> - there was a line left over from a previous iteration that negated
>   the new way of determining the pad count just before it which
>   has been removed (num_pads = of_get_child_count(np)).
> 
> - Philipp Zabel has developed a set of patches that allow adding
>   to the subdev async notifier waiting list using a chaining method
>   from the async registered callbacks (v4l2_of_subdev_registered()
>   and the prep patches for that). For now, I've removed the use of
>   v4l2_of_subdev_registered() for the vidmux driver's registered
>   callback. This doesn't affect the functionality of this driver,
>   but allows for it to be merged now, before adding the chaining
>   support.
> 
> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> ---
>  .../bindings/media/video-multiplexer.txt           |  59 +++
>  drivers/media/platform/Kconfig                     |   8 +
>  drivers/media/platform/Makefile                    |   2 +
>  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
>  4 files changed, 541 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
>  create mode 100644 drivers/media/platform/video-multiplexer.c
> 
> diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> new file mode 100644
> index 0000000..9d133d9
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> @@ -0,0 +1,59 @@
> +Video Multiplexer
> +=================
> +
> +Video multiplexers allow to select between multiple input ports. Video received
> +on the active input port is passed through to the output port. Muxes described
> +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> +
> +Required properties:
> +- compatible : should be "video-multiplexer"
> +- reg: should be register base of the register containing the control bitfield
> +- bit-mask: bitmask of the control bitfield in the control register
> +- bit-shift: bit offset of the control bitfield in the control register
> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> +  may be given to switch between two inputs
> +- #address-cells: should be <1>
> +- #size-cells: should be <0>
> +- port@*: at least three port nodes containing endpoints connecting to the
> +  source and sink devices according to of_graph bindings. The last port is
> +  the output port, all others are inputs.
> +
> +Example:
> +
> +syscon {
> +	compatible = "syscon", "simple-mfd";
> +
> +	mux {

Could you use standardised properties for this, i.e. ones defined in
Documentation/devicetree/bindings/media/video-interfaces.txt ?

This is very similar to another patch "[PATCH] devicetree: Add video bus
switch" posted by Pavel Machek recently. The problem with that is also
similar than with this one: how to pass the CSI-2 bus configuration to the
receiver.

There's some discussion here:

<URL:http://www.spinics.net/lists/linux-media/msg109493.html>

As Laurent already suggested, I think we should have a common solution for
the problem that, besides conveying the bus parameters to the receiver, also
encompasses CSI-2 virtual channels and data types.

That would mean finishing the series of patches in the branch I believe
Laurent already quoted here.

> +		compatible = "video-multiplexer";
> +		/* Single bit (1 << 19) in syscon register 0x04: */
> +		reg = <0x04>;
> +		bit-mask = <1>;
> +		bit-shift = <19>;
> +		#address-cells = <1>;
> +		#size-cells = <0>;
> +
> +		port at 0 {
> +			reg = <0>;
> +
> +			mux_in0: endpoint {
> +				remote-endpoint = <&video_source0_out>;
> +			};
> +		};
> +
> +		port at 1 {
> +			reg = <1>;
> +
> +			mux_in1: endpoint {
> +				remote-endpoint = <&video_source1_out>;
> +			};
> +		};
> +
> +		port at 2 {
> +			reg = <2>;
> +
> +			mux_out: endpoint {
> +				remote-endpoint = <&capture_interface_in>;
> +			};
> +		};
> +	};
> +};

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus at iki.fi	XMPP: sailus at retiisi.org.uk

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-07 13:36                     ` Benoit Parrot
  (?)
@ 2017-02-07 20:50                       ` Sakari Ailus
  -1 siblings, 0 replies; 549+ messages in thread
From: Sakari Ailus @ 2017-02-07 20:50 UTC (permalink / raw)
  To: Benoit Parrot
  Cc: Laurent Pinchart, Steve Longerbeam, Hans Verkuil, Philipp Zabel,
	robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, nick, markus.heiser, laurent.pinchart+renesas, geert,
	arnd, sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus

Hi Benoit,

On Tue, Feb 07, 2017 at 07:36:48AM -0600, Benoit Parrot wrote:
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> > Hi Steve,
> > 
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > > >>>>>>> +
> > > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > > >>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> > > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > > >>>>>> dubious). Normally this information should come from the device tree
> > > >>>>>> and there should be no need for this op.
> > > >>>>>> 
> > > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > > >>>>>> 
> > > >>>>>> If you don't need it, then I recommend it is removed.
> > > >>>> 
> > > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > > >>> 
> > > >>> That's not a good point.
> > > > 
> > > > (mea culpa, s/point/idea/)
> > > > 
> > > >>> The imx-media driver must not parse the sensor DT node as it is not
> > > >>> aware of what bindings the sensor is compatible with.
> > > 
> > > Hi Laurent,
> > > 
> > > I don't really understand this argument. The sensor node has been found
> > > by parsing the OF graph, so it is known to be a camera sensor node at
> > > that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video source. 
> > You can rely on the fact that it implements the OF graph bindings to locate 
> > other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a compatible 
> > string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> > you shouldn't do, to keep the driver generic) you can't know for certain how 
> > to parse the sensor node DT properties. For all you know, the video source 
> > could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> > can't even rely on the fact that it's a sensor.
> > 
> > > >>> Information must instead be queried from the sensor subdev at runtime,
> > > >>> through the g_mbus_config() operation.
> > > >>> 
> > > >>> Of course, if you can get the information from the imx-media DT node,
> > > >>> that's certainly an option. It's only information provided by the sensor
> > > >>> driver that you have no choice but query using a subdev operation.
> > > >> 
> > > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > > >> using this?
> > > > 
> > > > It all depends on what type of information needs to be retrieved, and
> > > > whether it can change at runtime or is fixed. Adding properties to the
> > > > imx-media DT node is certainly fine as long as those properties describe
> > > > the i.MX side.
> > >
> > > In this case the info needed is the media bus type. That info is most easily
> > > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > > node.
> > 
> > I haven't had time to check the code in details yet, so I can't really comment 
> > on what you need and how it should be implemented exactly.
> > 
> > > The media bus type is not something that can be added to the
> > > imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> > > > In the omap3isp case, we use the operation to query whether parallel data
> > > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > > 
> > > >> The reason I am suspicious about this op is that it came from soc-camera
> > > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > > >> like a HW description to me, i.e. something that belongs in the DT.
> > > > 
> > > > Part of it is possibly outdated, but for buses that support multiple modes
> > > > of operation (such as the parallel bus case described above) we need to
> > > > make that information discoverable at runtime. Maybe this should be
> > > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > > supported through the API he is working on.
> > > 
> > > That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> > > I've been thinking the DT should contain virtual channel info for CSI-2
> > > buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be 
> > handled as a software concept, and thus supported through driver code without 
> > involving DT.
> 
> Laurent,
> 
> So when you have a CSI2 port aggregator for instance where traffic from up
> to 4 CSI2 sources where each source is now assigned its own VC by the
> aggregator and interleaved into a single CSI2 Receiver. I was hoping that
> in this case the VC would be DT discoverable as a specicic source identifier.
> So the CSI-RX side could associate a specific source and create its own
> video device. I am guessing that no such thing exist today?

This should be configurable in software: the sensors connected to the
aggregator may also output multiple streams. The number of streams may also
depend on the user specified configuration over the MC / V4L2 sub-device
interfaces. Thus, this needs to be configurable in software.

We do need additional patches on top of the current mediatree.git master
branch though, some of which are not yet written...

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 20:50                       ` Sakari Ailus
  0 siblings, 0 replies; 549+ messages in thread
From: Sakari Ailus @ 2017-02-07 20:50 UTC (permalink / raw)
  To: Benoit Parrot
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, sakari.ailus, nick,
	songjun.wu, Hans Verkuil, Steve Longerbeam, Laurent Pinchart,
	robert.jarzmik, devel, markus.heiser, laurent.pinchart+renesas,
	linux, geert, Steve Longerbeam, Sascha Hauer, linux-media,
	devicetree, Philipp Zabel, arnd, mchehab, robh+dt, horms+renesas,
	tiffany.lin, linux-arm-kernel, niklas.soderlund+renesas, gregkh,
	linux-kernel

Hi Benoit,

On Tue, Feb 07, 2017 at 07:36:48AM -0600, Benoit Parrot wrote:
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> > Hi Steve,
> > 
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > > >>>>>>> +
> > > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > > >>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> > > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > > >>>>>> dubious). Normally this information should come from the device tree
> > > >>>>>> and there should be no need for this op.
> > > >>>>>> 
> > > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > > >>>>>> 
> > > >>>>>> If you don't need it, then I recommend it is removed.
> > > >>>> 
> > > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > > >>> 
> > > >>> That's not a good point.
> > > > 
> > > > (mea culpa, s/point/idea/)
> > > > 
> > > >>> The imx-media driver must not parse the sensor DT node as it is not
> > > >>> aware of what bindings the sensor is compatible with.
> > > 
> > > Hi Laurent,
> > > 
> > > I don't really understand this argument. The sensor node has been found
> > > by parsing the OF graph, so it is known to be a camera sensor node at
> > > that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video source. 
> > You can rely on the fact that it implements the OF graph bindings to locate 
> > other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a compatible 
> > string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> > you shouldn't do, to keep the driver generic) you can't know for certain how 
> > to parse the sensor node DT properties. For all you know, the video source 
> > could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> > can't even rely on the fact that it's a sensor.
> > 
> > > >>> Information must instead be queried from the sensor subdev at runtime,
> > > >>> through the g_mbus_config() operation.
> > > >>> 
> > > >>> Of course, if you can get the information from the imx-media DT node,
> > > >>> that's certainly an option. It's only information provided by the sensor
> > > >>> driver that you have no choice but query using a subdev operation.
> > > >> 
> > > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > > >> using this?
> > > > 
> > > > It all depends on what type of information needs to be retrieved, and
> > > > whether it can change at runtime or is fixed. Adding properties to the
> > > > imx-media DT node is certainly fine as long as those properties describe
> > > > the i.MX side.
> > >
> > > In this case the info needed is the media bus type. That info is most easily
> > > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > > node.
> > 
> > I haven't had time to check the code in details yet, so I can't really comment 
> > on what you need and how it should be implemented exactly.
> > 
> > > The media bus type is not something that can be added to the
> > > imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> > > > In the omap3isp case, we use the operation to query whether parallel data
> > > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > > 
> > > >> The reason I am suspicious about this op is that it came from soc-camera
> > > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > > >> like a HW description to me, i.e. something that belongs in the DT.
> > > > 
> > > > Part of it is possibly outdated, but for buses that support multiple modes
> > > > of operation (such as the parallel bus case described above) we need to
> > > > make that information discoverable at runtime. Maybe this should be
> > > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > > supported through the API he is working on.
> > > 
> > > That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> > > I've been thinking the DT should contain virtual channel info for CSI-2
> > > buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be 
> > handled as a software concept, and thus supported through driver code without 
> > involving DT.
> 
> Laurent,
> 
> So when you have a CSI2 port aggregator for instance where traffic from up
> to 4 CSI2 sources where each source is now assigned its own VC by the
> aggregator and interleaved into a single CSI2 Receiver. I was hoping that
> in this case the VC would be DT discoverable as a specicic source identifier.
> So the CSI-RX side could associate a specific source and create its own
> video device. I am guessing that no such thing exist today?

This should be configurable in software: the sensors connected to the
aggregator may also output multiple streams. The number of streams may also
depend on the user specified configuration over the MC / V4L2 sub-device
interfaces. Thus, this needs to be configurable in software.

We do need additional patches on top of the current mediatree.git master
branch though, some of which are not yet written...

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 20:50                       ` Sakari Ailus
  0 siblings, 0 replies; 549+ messages in thread
From: Sakari Ailus @ 2017-02-07 20:50 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Benoit,

On Tue, Feb 07, 2017 at 07:36:48AM -0600, Benoit Parrot wrote:
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> > Hi Steve,
> > 
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> > > On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> > > > On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> > > >> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> > > >>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> > > >>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> > > >>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> > > >>>>>>> +
> > > >>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> > > >>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> > > >>>>>> I am not certain this op is needed at all. In the current kernel this
> > > >>>>>> op is only used by soc_camera, pxa_camera and omap3isp (somewhat
> > > >>>>>> dubious). Normally this information should come from the device tree
> > > >>>>>> and there should be no need for this op.
> > > >>>>>> 
> > > >>>>>> My (tentative) long-term plan was to get rid of this op.
> > > >>>>>> 
> > > >>>>>> If you don't need it, then I recommend it is removed.
> > > >>>> 
> > > >>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> > > >>>> camera sensor, and it was doing that to determine the sensor's bus
> > > >>>> type. This info was already available from parsing a v4l2_of_endpoint
> > > >>>> from the sensor node. So it was simple to remove the g_mbus_config
> > > >>>> calls, and instead rely on the parsed sensor v4l2_of_endpoint.
> > > >>> 
> > > >>> That's not a good point.
> > > > 
> > > > (mea culpa, s/point/idea/)
> > > > 
> > > >>> The imx-media driver must not parse the sensor DT node as it is not
> > > >>> aware of what bindings the sensor is compatible with.
> > > 
> > > Hi Laurent,
> > > 
> > > I don't really understand this argument. The sensor node has been found
> > > by parsing the OF graph, so it is known to be a camera sensor node at
> > > that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video source. 
> > You can rely on the fact that it implements the OF graph bindings to locate 
> > other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a compatible 
> > string. Unless you match on sensor compat strings in the i.MX6 driver (which 
> > you shouldn't do, to keep the driver generic) you can't know for certain how 
> > to parse the sensor node DT properties. For all you know, the video source 
> > could be a bridge such as an HDMI to CSI-2 converter for instance, so you 
> > can't even rely on the fact that it's a sensor.
> > 
> > > >>> Information must instead be queried from the sensor subdev at runtime,
> > > >>> through the g_mbus_config() operation.
> > > >>> 
> > > >>> Of course, if you can get the information from the imx-media DT node,
> > > >>> that's certainly an option. It's only information provided by the sensor
> > > >>> driver that you have no choice but query using a subdev operation.
> > > >> 
> > > >> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> > > >> using this?
> > > > 
> > > > It all depends on what type of information needs to be retrieved, and
> > > > whether it can change at runtime or is fixed. Adding properties to the
> > > > imx-media DT node is certainly fine as long as those properties describe
> > > > the i.MX side.
> > >
> > > In this case the info needed is the media bus type. That info is most easily
> > > available by calling v4l2_of_parse_endpoint() on the sensor's endpoint
> > > node.
> > 
> > I haven't had time to check the code in details yet, so I can't really comment 
> > on what you need and how it should be implemented exactly.
> > 
> > > The media bus type is not something that can be added to the
> > > imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> > > > In the omap3isp case, we use the operation to query whether parallel data
> > > > contains embedded sync (BT.656) or uses separate h/v sync signals.
> > > > 
> > > >> The reason I am suspicious about this op is that it came from soc-camera
> > > >> and predates the DT. The contents of v4l2_mbus_config seems very much
> > > >> like a HW description to me, i.e. something that belongs in the DT.
> > > > 
> > > > Part of it is possibly outdated, but for buses that support multiple modes
> > > > of operation (such as the parallel bus case described above) we need to
> > > > make that information discoverable at runtime. Maybe this should be
> > > > considered as related to Sakari's efforts to support VC/DT for CSI-2, and
> > > > supported through the API he is working on.
> > > 
> > > That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> > > I've been thinking the DT should contain virtual channel info for CSI-2
> > > buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be 
> > handled as a software concept, and thus supported through driver code without 
> > involving DT.
> 
> Laurent,
> 
> So when you have a CSI2 port aggregator for instance where traffic from up
> to 4 CSI2 sources where each source is now assigned its own VC by the
> aggregator and interleaved into a single CSI2 Receiver. I was hoping that
> in this case the VC would be DT discoverable as a specicic source identifier.
> So the CSI-RX side could associate a specific source and create its own
> video device. I am guessing that no such thing exist today?

This should be configurable in software: the sensors connected to the
aggregator may also output multiple streams. The number of streams may also
depend on the user specified configuration over the MC / V4L2 sub-device
interfaces. Thus, this needs to be configurable in software.

We do need additional patches on top of the current mediatree.git master
branch though, some of which are not yet written...

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus at iki.fi	XMPP: sailus at retiisi.org.uk

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-07 13:36                     ` Benoit Parrot
  (?)
@ 2017-02-07 23:04                       ` Laurent Pinchart
  -1 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 23:04 UTC (permalink / raw)
  To: Benoit Parrot
  Cc: Steve Longerbeam, Hans Verkuil, Philipp Zabel, robh+dt,
	mark.rutland, shawnguo, kernel, fabio.estevam, linux, mchehab,
	nick, markus.heiser, laurent.pinchart+renesas, geert, arnd,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, sakari.ailus

Hi Benoit,

On Tuesday 07 Feb 2017 07:36:48 Benoit Parrot wrote:
> Laurent Pinchart wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> >> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> >>> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >>>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>>>> +
> >>>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> >>>>>>>> I am not certain this op is needed at all. In the current kernel
> >>>>>>>> this op is only used by soc_camera, pxa_camera and omap3isp
> >>>>>>>> (somewhat dubious). Normally this information should come from the
> >>>>>>>> device tree and there should be no need for this op.
> >>>>>>>> 
> >>>>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>>>> 
> >>>>>>>> If you don't need it, then I recommend it is removed.
> >>>>>> 
> >>>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>>>> type. This info was already available from parsing a
> >>>>>> v4l2_of_endpoint from the sensor node. So it was simple to remove the
> >>>>>> g_mbus_config calls, and instead rely on the parsed sensor
> >>>>>> v4l2_of_endpoint.
> >>>>> 
> >>>>> That's not a good point.
> >>> 
> >>> (mea culpa, s/point/idea/)
> >>> 
> >>>>> The imx-media driver must not parse the sensor DT node as it is not
> >>>>> aware of what bindings the sensor is compatible with.
> >> 
> >> Hi Laurent,
> >> 
> >> I don't really understand this argument. The sensor node has been found
> >> by parsing the OF graph, so it is known to be a camera sensor node at
> >> that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video
> > source. You can rely on the fact that it implements the OF graph bindings
> > to locate other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a
> > compatible string. Unless you match on sensor compat strings in the i.MX6
> > driver (which you shouldn't do, to keep the driver generic) you can't know
> > for certain how to parse the sensor node DT properties. For all you know,
> > the video source could be a bridge such as an HDMI to CSI-2 converter for
> > instance, so you can't even rely on the fact that it's a sensor.
> > 
> >>>>> Information must instead be queried from the sensor subdev at
> >>>>> runtime, through the g_mbus_config() operation.
> >>>>> 
> >>>>> Of course, if you can get the information from the imx-media DT
> >>>>> node, that's certainly an option. It's only information provided by
> >>>>> the sensor driver that you have no choice but query using a subdev
> >>>>> operation.
> >>>> 
> >>>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >>>> using this?
> >>> 
> >>> It all depends on what type of information needs to be retrieved, and
> >>> whether it can change at runtime or is fixed. Adding properties to the
> >>> imx-media DT node is certainly fine as long as those properties
> >>> describe the i.MX side.
> >> 
> >> In this case the info needed is the media bus type. That info is most
> >> easily available by calling v4l2_of_parse_endpoint() on the sensor's
> >> endpoint node.
> > 
> > I haven't had time to check the code in details yet, so I can't really
> > comment on what you need and how it should be implemented exactly.
> > 
> >> The media bus type is not something that can be added to the
> >> imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> >>> In the omap3isp case, we use the operation to query whether parallel
> >>> data contains embedded sync (BT.656) or uses separate h/v sync signals.
> >>> 
> >>>> The reason I am suspicious about this op is that it came from
> >>>> soc-camera and predates the DT. The contents of v4l2_mbus_config seems
> >>>> very much like a HW description to me, i.e. something that belongs in
> >>>> the DT.
> >>> 
> >>> Part of it is possibly outdated, but for buses that support multiple
> >>> modes of operation (such as the parallel bus case described above) we
> >>> need to make that information discoverable at runtime. Maybe this should
> >>> be considered as related to Sakari's efforts to support VC/DT for CSI-2,
> >>> and supported through the API he is working on.
> >> 
> >> That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> >> I've been thinking the DT should contain virtual channel info for CSI-2
> >> buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be
> > handled as a software concept, and thus supported through driver code
> > without involving DT.
> 
> Laurent,
> 
> So when you have a CSI2 port aggregator for instance where traffic from up
> to 4 CSI2 sources where each source is now assigned its own VC by the
> aggregator and interleaved into a single CSI2 Receiver. I was hoping that
> in this case the VC would be DT discoverable as a specicic source
> identifier. So the CSI-RX side could associate a specific source and create
> its own video device.

In this specific example, I believe the aggregator should be modelled with 4 
input ports and one output port in DT, and with 4 sink pads and one source pad 
in MC. Information about the VCs multiplexed over the aggregator's source link 
should not be part of the device tree, but should be discoverable at runtime 
through V4L2 subdev operations. This would include information about how the 4 
input streams are routed to VCs inside the aggregator.

> I am guessing that no such thing exist today?

There's very little (to not say nothing) in terms of VC support in V4L2 today.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 23:04                       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 23:04 UTC (permalink / raw)
  To: Benoit Parrot
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, sakari.ailus, nick,
	songjun.wu, Hans Verkuil, Steve Longerbeam, robert.jarzmik,
	devel, markus.heiser, laurent.pinchart+renesas, linux, geert,
	Steve Longerbeam, Sascha Hauer, linux-media, devicetree,
	Philipp Zabel, arnd, mchehab, robh+dt, horms+renesas,
	tiffany.lin, linux-arm-kernel, niklas.soderlund+renesas, gregkh,
	linux-kernel, jean-christophe.trotin, kernel

Hi Benoit,

On Tuesday 07 Feb 2017 07:36:48 Benoit Parrot wrote:
> Laurent Pinchart wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> >> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> >>> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >>>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>>>> +
> >>>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> >>>>>>>> I am not certain this op is needed at all. In the current kernel
> >>>>>>>> this op is only used by soc_camera, pxa_camera and omap3isp
> >>>>>>>> (somewhat dubious). Normally this information should come from the
> >>>>>>>> device tree and there should be no need for this op.
> >>>>>>>> 
> >>>>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>>>> 
> >>>>>>>> If you don't need it, then I recommend it is removed.
> >>>>>> 
> >>>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>>>> type. This info was already available from parsing a
> >>>>>> v4l2_of_endpoint from the sensor node. So it was simple to remove the
> >>>>>> g_mbus_config calls, and instead rely on the parsed sensor
> >>>>>> v4l2_of_endpoint.
> >>>>> 
> >>>>> That's not a good point.
> >>> 
> >>> (mea culpa, s/point/idea/)
> >>> 
> >>>>> The imx-media driver must not parse the sensor DT node as it is not
> >>>>> aware of what bindings the sensor is compatible with.
> >> 
> >> Hi Laurent,
> >> 
> >> I don't really understand this argument. The sensor node has been found
> >> by parsing the OF graph, so it is known to be a camera sensor node at
> >> that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video
> > source. You can rely on the fact that it implements the OF graph bindings
> > to locate other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a
> > compatible string. Unless you match on sensor compat strings in the i.MX6
> > driver (which you shouldn't do, to keep the driver generic) you can't know
> > for certain how to parse the sensor node DT properties. For all you know,
> > the video source could be a bridge such as an HDMI to CSI-2 converter for
> > instance, so you can't even rely on the fact that it's a sensor.
> > 
> >>>>> Information must instead be queried from the sensor subdev at
> >>>>> runtime, through the g_mbus_config() operation.
> >>>>> 
> >>>>> Of course, if you can get the information from the imx-media DT
> >>>>> node, that's certainly an option. It's only information provided by
> >>>>> the sensor driver that you have no choice but query using a subdev
> >>>>> operation.
> >>>> 
> >>>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >>>> using this?
> >>> 
> >>> It all depends on what type of information needs to be retrieved, and
> >>> whether it can change at runtime or is fixed. Adding properties to the
> >>> imx-media DT node is certainly fine as long as those properties
> >>> describe the i.MX side.
> >> 
> >> In this case the info needed is the media bus type. That info is most
> >> easily available by calling v4l2_of_parse_endpoint() on the sensor's
> >> endpoint node.
> > 
> > I haven't had time to check the code in details yet, so I can't really
> > comment on what you need and how it should be implemented exactly.
> > 
> >> The media bus type is not something that can be added to the
> >> imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> >>> In the omap3isp case, we use the operation to query whether parallel
> >>> data contains embedded sync (BT.656) or uses separate h/v sync signals.
> >>> 
> >>>> The reason I am suspicious about this op is that it came from
> >>>> soc-camera and predates the DT. The contents of v4l2_mbus_config seems
> >>>> very much like a HW description to me, i.e. something that belongs in
> >>>> the DT.
> >>> 
> >>> Part of it is possibly outdated, but for buses that support multiple
> >>> modes of operation (such as the parallel bus case described above) we
> >>> need to make that information discoverable at runtime. Maybe this should
> >>> be considered as related to Sakari's efforts to support VC/DT for CSI-2,
> >>> and supported through the API he is working on.
> >> 
> >> That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> >> I've been thinking the DT should contain virtual channel info for CSI-2
> >> buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be
> > handled as a software concept, and thus supported through driver code
> > without involving DT.
> 
> Laurent,
> 
> So when you have a CSI2 port aggregator for instance where traffic from up
> to 4 CSI2 sources where each source is now assigned its own VC by the
> aggregator and interleaved into a single CSI2 Receiver. I was hoping that
> in this case the VC would be DT discoverable as a specicic source
> identifier. So the CSI-RX side could associate a specific source and create
> its own video device.

In this specific example, I believe the aggregator should be modelled with 4 
input ports and one output port in DT, and with 4 sink pads and one source pad 
in MC. Information about the VCs multiplexed over the aggregator's source link 
should not be part of the device tree, but should be discoverable at runtime 
through V4L2 subdev operations. This would include information about how the 4 
input streams are routed to VCs inside the aggregator.

> I am guessing that no such thing exist today?

There's very little (to not say nothing) in terms of VC support in V4L2 today.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-07 23:04                       ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-07 23:04 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Benoit,

On Tuesday 07 Feb 2017 07:36:48 Benoit Parrot wrote:
> Laurent Pinchart wrote on Tue [2017-Feb-07 12:26:32 +0200]:
> > On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
> >> On 02/06/2017 02:33 PM, Laurent Pinchart wrote:
> >>> On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
> >>>> On 02/05/2017 04:48 PM, Laurent Pinchart wrote:
> >>>>> On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
> >>>>>> On 01/24/2017 04:02 AM, Philipp Zabel wrote:
> >>>>>>> On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:
> >>>>>>>>> +
> >>>>>>>>> +int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct
> >>>>>>>>> v4l2_mbus_config *cfg)
> > 
> > [snip]
> > 
> >>>>>>>> I am not certain this op is needed at all. In the current kernel
> >>>>>>>> this op is only used by soc_camera, pxa_camera and omap3isp
> >>>>>>>> (somewhat dubious). Normally this information should come from the
> >>>>>>>> device tree and there should be no need for this op.
> >>>>>>>> 
> >>>>>>>> My (tentative) long-term plan was to get rid of this op.
> >>>>>>>> 
> >>>>>>>> If you don't need it, then I recommend it is removed.
> >>>>>> 
> >>>>>> Hi Hans, the imx-media driver was only calling g_mbus_config to the
> >>>>>> camera sensor, and it was doing that to determine the sensor's bus
> >>>>>> type. This info was already available from parsing a
> >>>>>> v4l2_of_endpoint from the sensor node. So it was simple to remove the
> >>>>>> g_mbus_config calls, and instead rely on the parsed sensor
> >>>>>> v4l2_of_endpoint.
> >>>>> 
> >>>>> That's not a good point.
> >>> 
> >>> (mea culpa, s/point/idea/)
> >>> 
> >>>>> The imx-media driver must not parse the sensor DT node as it is not
> >>>>> aware of what bindings the sensor is compatible with.
> >> 
> >> Hi Laurent,
> >> 
> >> I don't really understand this argument. The sensor node has been found
> >> by parsing the OF graph, so it is known to be a camera sensor node at
> >> that point.
> > 
> > All you know in the i.MX6 driver is that the remote node is a video
> > source. You can rely on the fact that it implements the OF graph bindings
> > to locate other ports in that DT node, but that's more or less it.
> > 
> > DT properties are defined by DT bindings and thus qualified by a
> > compatible string. Unless you match on sensor compat strings in the i.MX6
> > driver (which you shouldn't do, to keep the driver generic) you can't know
> > for certain how to parse the sensor node DT properties. For all you know,
> > the video source could be a bridge such as an HDMI to CSI-2 converter for
> > instance, so you can't even rely on the fact that it's a sensor.
> > 
> >>>>> Information must instead be queried from the sensor subdev at
> >>>>> runtime, through the g_mbus_config() operation.
> >>>>> 
> >>>>> Of course, if you can get the information from the imx-media DT
> >>>>> node, that's certainly an option. It's only information provided by
> >>>>> the sensor driver that you have no choice but query using a subdev
> >>>>> operation.
> >>>> 
> >>>> Shouldn't this come from the imx-media DT node? BTW, why is omap3isp
> >>>> using this?
> >>> 
> >>> It all depends on what type of information needs to be retrieved, and
> >>> whether it can change at runtime or is fixed. Adding properties to the
> >>> imx-media DT node is certainly fine as long as those properties
> >>> describe the i.MX side.
> >> 
> >> In this case the info needed is the media bus type. That info is most
> >> easily available by calling v4l2_of_parse_endpoint() on the sensor's
> >> endpoint node.
> > 
> > I haven't had time to check the code in details yet, so I can't really
> > comment on what you need and how it should be implemented exactly.
> > 
> >> The media bus type is not something that can be added to the
> >> imx-media node since it contains no endpoint nodes.
> > 
> > Agreed. You have endpoints in the CSI nodes though.
> > 
> >>> In the omap3isp case, we use the operation to query whether parallel
> >>> data contains embedded sync (BT.656) or uses separate h/v sync signals.
> >>> 
> >>>> The reason I am suspicious about this op is that it came from
> >>>> soc-camera and predates the DT. The contents of v4l2_mbus_config seems
> >>>> very much like a HW description to me, i.e. something that belongs in
> >>>> the DT.
> >>> 
> >>> Part of it is possibly outdated, but for buses that support multiple
> >>> modes of operation (such as the parallel bus case described above) we
> >>> need to make that information discoverable at runtime. Maybe this should
> >>> be considered as related to Sakari's efforts to support VC/DT for CSI-2,
> >>> and supported through the API he is working on.
> >> 
> >> That sounds interesting, can you point me to some info on this effort?
> > 
> > Sure.
> > 
> > http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc
> > 
> >> I've been thinking the DT should contain virtual channel info for CSI-2
> >> buses.
> > 
> > I don't think it should. CSI-2 virtual channels and data types should be
> > handled as a software concept, and thus supported through driver code
> > without involving DT.
> 
> Laurent,
> 
> So when you have a CSI2 port aggregator for instance where traffic from up
> to 4 CSI2 sources where each source is now assigned its own VC by the
> aggregator and interleaved into a single CSI2 Receiver. I was hoping that
> in this case the VC would be DT discoverable as a specicic source
> identifier. So the CSI-RX side could associate a specific source and create
> its own video device.

In this specific example, I believe the aggregator should be modelled with 4 
input ports and one output port in DT, and with 4 sink pads and one source pad 
in MC. Information about the VCs multiplexed over the aggregator's source link 
should not be part of the device tree, but should be discoverable at runtime 
through V4L2 subdev operations. This would include information about how the 4 
input streams are routed to VCs inside the aggregator.

> I am guessing that no such thing exist today?

There's very little (to not say nothing) in terms of VC support in V4L2 today.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-07 20:46     ` Sakari Ailus
  (?)
@ 2017-02-08  9:47       ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-08  9:47 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, Sebastian Reichel

Hi Sakari,

On Tue, 2017-02-07 at 22:46 +0200, Sakari Ailus wrote:
> Hi Steve,
> 
> On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > This driver can handle SoC internal and external video bus multiplexers,
> > controlled either by register bit fields or by a GPIO. The subdevice
> > passes through frame interval and mbus configuration of the active input
> > to the output side.
> > 
> > Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > --
> > 
> > - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
> >   should be unregister.
> > 
> > - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >   to vidsw_remove().
> > 
> > - there was a line left over from a previous iteration that negated
> >   the new way of determining the pad count just before it which
> >   has been removed (num_pads = of_get_child_count(np)).
> > 
> > - Philipp Zabel has developed a set of patches that allow adding
> >   to the subdev async notifier waiting list using a chaining method
> >   from the async registered callbacks (v4l2_of_subdev_registered()
> >   and the prep patches for that). For now, I've removed the use of
> >   v4l2_of_subdev_registered() for the vidmux driver's registered
> >   callback. This doesn't affect the functionality of this driver,
> >   but allows for it to be merged now, before adding the chaining
> >   support.
> > 
> > Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> > ---
> >  .../bindings/media/video-multiplexer.txt           |  59 +++
> >  drivers/media/platform/Kconfig                     |   8 +
> >  drivers/media/platform/Makefile                    |   2 +
> >  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
> >  4 files changed, 541 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
> >  create mode 100644 drivers/media/platform/video-multiplexer.c
> > 
> > diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > new file mode 100644
> > index 0000000..9d133d9
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > @@ -0,0 +1,59 @@
> > +Video Multiplexer
> > +=================
> > +
> > +Video multiplexers allow to select between multiple input ports. Video received
> > +on the active input port is passed through to the output port. Muxes described
> > +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> > +
> > +Required properties:
> > +- compatible : should be "video-multiplexer"
> > +- reg: should be register base of the register containing the control bitfield
> > +- bit-mask: bitmask of the control bitfield in the control register
> > +- bit-shift: bit offset of the control bitfield in the control register
> > +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> > +  may be given to switch between two inputs
> > +- #address-cells: should be <1>
> > +- #size-cells: should be <0>
> > +- port@*: at least three port nodes containing endpoints connecting to the
> > +  source and sink devices according to of_graph bindings. The last port is
> > +  the output port, all others are inputs.
> > +
> > +Example:
> > +
> > +syscon {
> > +	compatible = "syscon", "simple-mfd";
> > +
> > +	mux {
> 
> Could you use standardised properties for this, i.e. ones defined in
> Documentation/devicetree/bindings/media/video-interfaces.txt ?

What do you mean? Should we add the optional bus-width property to
describe the bit width of the parallel bus even though the driver
doesn't care about any of the properties?

> This is very similar to another patch "[PATCH] devicetree: Add video bus
> switch" posted by Pavel Machek recently. The problem with that is also
> similar than with this one: how to pass the CSI-2 bus configuration to the
> receiver.
>
> There's some discussion here:
> 
> <URL:http://www.spinics.net/lists/linux-media/msg109493.html>

[Added Sebastian do Cc:]

Yes, this is essentially the same driver, except that this driver also
handles MMIO-bitfield controlled muxes, and that the actual physical bus
(which the drivers currently don't care about) is MIPI CSI-2 in the
other case, and parallel this one.
They should probably be combined, or maybe split into two separate
drivers (MMIO controlled mux, GPIO controlled switch), possibly using
the same v4l2_subdev boilerplate.

> As Laurent already suggested, I think we should have a common solution for
> the problem that, besides conveying the bus parameters to the receiver, also
> encompasses CSI-2 virtual channels and data types.

Yes, that seems to be necessary, as certainly we can't configure the mux
output bus parameters in DT to a fixed setting.

> That would mean finishing the series of patches in the branch I believe
> Laurent already quoted here.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-08  9:47       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-08  9:47 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: Steve Longerbeam, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, linux, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel

Hi Sakari,

On Tue, 2017-02-07 at 22:46 +0200, Sakari Ailus wrote:
> Hi Steve,
> 
> On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > This driver can handle SoC internal and external video bus multiplexers,
> > controlled either by register bit fields or by a GPIO. The subdevice
> > passes through frame interval and mbus configuration of the active input
> > to the output side.
> > 
> > Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > --
> > 
> > - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
> >   should be unregister.
> > 
> > - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >   to vidsw_remove().
> > 
> > - there was a line left over from a previous iteration that negated
> >   the new way of determining the pad count just before it which
> >   has been removed (num_pads = of_get_child_count(np)).
> > 
> > - Philipp Zabel has developed a set of patches that allow adding
> >   to the subdev async notifier waiting list using a chaining method
> >   from the async registered callbacks (v4l2_of_subdev_registered()
> >   and the prep patches for that). For now, I've removed the use of
> >   v4l2_of_subdev_registered() for the vidmux driver's registered
> >   callback. This doesn't affect the functionality of this driver,
> >   but allows for it to be merged now, before adding the chaining
> >   support.
> > 
> > Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> > ---
> >  .../bindings/media/video-multiplexer.txt           |  59 +++
> >  drivers/media/platform/Kconfig                     |   8 +
> >  drivers/media/platform/Makefile                    |   2 +
> >  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
> >  4 files changed, 541 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
> >  create mode 100644 drivers/media/platform/video-multiplexer.c
> > 
> > diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > new file mode 100644
> > index 0000000..9d133d9
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > @@ -0,0 +1,59 @@
> > +Video Multiplexer
> > +=================
> > +
> > +Video multiplexers allow to select between multiple input ports. Video received
> > +on the active input port is passed through to the output port. Muxes described
> > +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> > +
> > +Required properties:
> > +- compatible : should be "video-multiplexer"
> > +- reg: should be register base of the register containing the control bitfield
> > +- bit-mask: bitmask of the control bitfield in the control register
> > +- bit-shift: bit offset of the control bitfield in the control register
> > +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> > +  may be given to switch between two inputs
> > +- #address-cells: should be <1>
> > +- #size-cells: should be <0>
> > +- port@*: at least three port nodes containing endpoints connecting to the
> > +  source and sink devices according to of_graph bindings. The last port is
> > +  the output port, all others are inputs.
> > +
> > +Example:
> > +
> > +syscon {
> > +	compatible = "syscon", "simple-mfd";
> > +
> > +	mux {
> 
> Could you use standardised properties for this, i.e. ones defined in
> Documentation/devicetree/bindings/media/video-interfaces.txt ?

What do you mean? Should we add the optional bus-width property to
describe the bit width of the parallel bus even though the driver
doesn't care about any of the properties?

> This is very similar to another patch "[PATCH] devicetree: Add video bus
> switch" posted by Pavel Machek recently. The problem with that is also
> similar than with this one: how to pass the CSI-2 bus configuration to the
> receiver.
>
> There's some discussion here:
> 
> <URL:http://www.spinics.net/lists/linux-media/msg109493.html>

[Added Sebastian do Cc:]

Yes, this is essentially the same driver, except that this driver also
handles MMIO-bitfield controlled muxes, and that the actual physical bus
(which the drivers currently don't care about) is MIPI CSI-2 in the
other case, and parallel this one.
They should probably be combined, or maybe split into two separate
drivers (MMIO controlled mux, GPIO controlled switch), possibly using
the same v4l2_subdev boilerplate.

> As Laurent already suggested, I think we should have a common solution for
> the problem that, besides conveying the bus parameters to the receiver, also
> encompasses CSI-2 virtual channels and data types.

Yes, that seems to be necessary, as certainly we can't configure the mux
output bus parameters in DT to a fixed setting.

> That would mean finishing the series of patches in the branch I believe
> Laurent already quoted here.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-08  9:47       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-08  9:47 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Sakari,

On Tue, 2017-02-07 at 22:46 +0200, Sakari Ailus wrote:
> Hi Steve,
> 
> On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> > From: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > This driver can handle SoC internal and external video bus multiplexers,
> > controlled either by register bit fields or by a GPIO. The subdevice
> > passes through frame interval and mbus configuration of the active input
> > to the output side.
> > 
> > Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > 
> > --
> > 
> > - fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
> >   should be unregister.
> > 
> > - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >   to vidsw_remove().
> > 
> > - there was a line left over from a previous iteration that negated
> >   the new way of determining the pad count just before it which
> >   has been removed (num_pads = of_get_child_count(np)).
> > 
> > - Philipp Zabel has developed a set of patches that allow adding
> >   to the subdev async notifier waiting list using a chaining method
> >   from the async registered callbacks (v4l2_of_subdev_registered()
> >   and the prep patches for that). For now, I've removed the use of
> >   v4l2_of_subdev_registered() for the vidmux driver's registered
> >   callback. This doesn't affect the functionality of this driver,
> >   but allows for it to be merged now, before adding the chaining
> >   support.
> > 
> > Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> > ---
> >  .../bindings/media/video-multiplexer.txt           |  59 +++
> >  drivers/media/platform/Kconfig                     |   8 +
> >  drivers/media/platform/Makefile                    |   2 +
> >  drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
> >  4 files changed, 541 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
> >  create mode 100644 drivers/media/platform/video-multiplexer.c
> > 
> > diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > new file mode 100644
> > index 0000000..9d133d9
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> > @@ -0,0 +1,59 @@
> > +Video Multiplexer
> > +=================
> > +
> > +Video multiplexers allow to select between multiple input ports. Video received
> > +on the active input port is passed through to the output port. Muxes described
> > +by this binding may be controlled by a syscon register bitfield or by a GPIO.
> > +
> > +Required properties:
> > +- compatible : should be "video-multiplexer"
> > +- reg: should be register base of the register containing the control bitfield
> > +- bit-mask: bitmask of the control bitfield in the control register
> > +- bit-shift: bit offset of the control bitfield in the control register
> > +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
> > +  may be given to switch between two inputs
> > +- #address-cells: should be <1>
> > +- #size-cells: should be <0>
> > +- port@*: at least three port nodes containing endpoints connecting to the
> > +  source and sink devices according to of_graph bindings. The last port is
> > +  the output port, all others are inputs.
> > +
> > +Example:
> > +
> > +syscon {
> > +	compatible = "syscon", "simple-mfd";
> > +
> > +	mux {
> 
> Could you use standardised properties for this, i.e. ones defined in
> Documentation/devicetree/bindings/media/video-interfaces.txt ?

What do you mean? Should we add the optional bus-width property to
describe the bit width of the parallel bus even though the driver
doesn't care about any of the properties?

> This is very similar to another patch "[PATCH] devicetree: Add video bus
> switch" posted by Pavel Machek recently. The problem with that is also
> similar than with this one: how to pass the CSI-2 bus configuration to the
> receiver.
>
> There's some discussion here:
> 
> <URL:http://www.spinics.net/lists/linux-media/msg109493.html>

[Added Sebastian do Cc:]

Yes, this is essentially the same driver, except that this driver also
handles MMIO-bitfield controlled muxes, and that the actual physical bus
(which the drivers currently don't care about) is MIPI CSI-2 in the
other case, and parallel this one.
They should probably be combined, or maybe split into two separate
drivers (MMIO controlled mux, GPIO controlled switch), possibly using
the same v4l2_subdev boilerplate.

> As Laurent already suggested, I think we should have a common solution for
> the problem that, besides conveying the bus parameters to the receiver, also
> encompasses CSI-2 virtual channels and data types.

Yes, that seems to be necessary, as certainly we can't configure the mux
output bus parameters in DT to a fixed setting.

> That would mean finishing the series of patches in the branch I believe
> Laurent already quoted here.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
  2017-02-08  9:47       ` Philipp Zabel
  (?)
@ 2017-02-08 10:05         ` Laurent Pinchart
  -1 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-08 10:05 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Sakari Ailus, Steve Longerbeam, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, linux, mchehab, hverkuil, nick,
	markus.heiser, laurent.pinchart+renesas, bparrot, geert, arnd,
	sudipm.mukherjee, minghsiu.tsai, tiffany.lin,
	jean-christophe.trotin, horms+renesas, niklas.soderlund+renesas,
	robert.jarzmik, songjun.wu, andrew-ct.chen, gregkh, devicetree,
	linux-kernel, linux-arm-kernel, linux-media, devel, Sascha Hauer,
	Steve Longerbeam, Sebastian Reichel

Hi Philipp,

On Wednesday 08 Feb 2017 10:47:15 Philipp Zabel wrote:
> On Tue, 2017-02-07 at 22:46 +0200, Sakari Ailus wrote:
> > On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> >> From: Philipp Zabel <p.zabel@pengutronix.de>
> >> 
> >> This driver can handle SoC internal and external video bus multiplexers,
> >> controlled either by register bit fields or by a GPIO. The subdevice
> >> passes through frame interval and mbus configuration of the active input
> >> to the output side.
> >> 
> >> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> >> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> >> 
> >> --
> >> 
> >> - fixed a cut&paste error in vidsw_remove():
> >> v4l2_async_register_subdev()
> >>   should be unregister.
> >> 
> >> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >>   to vidsw_remove().
> >> 
> >> - there was a line left over from a previous iteration that negated
> >>   the new way of determining the pad count just before it which
> >>   has been removed (num_pads = of_get_child_count(np)).
> >> 
> >> - Philipp Zabel has developed a set of patches that allow adding
> >>   to the subdev async notifier waiting list using a chaining method
> >>   from the async registered callbacks (v4l2_of_subdev_registered()
> >>   and the prep patches for that). For now, I've removed the use of
> >>   v4l2_of_subdev_registered() for the vidmux driver's registered
> >>   callback. This doesn't affect the functionality of this driver,
> >>   but allows for it to be merged now, before adding the chaining
> >>   support.
> >> 
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >> 
> >>  .../bindings/media/video-multiplexer.txt           |  59 +++
> >>  drivers/media/platform/Kconfig                     |   8 +
> >>  drivers/media/platform/Makefile                    |   2 +
> >>  drivers/media/platform/video-multiplexer.c         | 472 ++++++++++++++
> >>  4 files changed, 541 insertions(+)
> >>  create mode 100644
> >>  Documentation/devicetree/bindings/media/video-multiplexer.txt create
> >>  mode 100644 drivers/media/platform/video-multiplexer.c
> >> 
> >> diff --git
> >> a/Documentation/devicetree/bindings/media/video-multiplexer.txt
> >> b/Documentation/devicetree/bindings/media/video-multiplexer.txt new
> >> file mode 100644
> >> index 0000000..9d133d9
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> >> @@ -0,0 +1,59 @@
> >> +Video Multiplexer
> >> +=================
> >> +
> >> +Video multiplexers allow to select between multiple input ports. Video
> >> received
> >> +on the active input port is passed through to the output port. Muxes
> >> described
> >> +by this binding may be controlled by a syscon register bitfield or by a
> >> GPIO.
> >> +
> >> +Required properties:
> >> +- compatible : should be "video-multiplexer"
> >> +- reg: should be register base of the register containing the control
> >> bitfield
> >> +- bit-mask: bitmask of the control bitfield in the control register
> >> +- bit-shift: bit offset of the control bitfield in the control register
> >> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO
> >> phandle
> >> +  may be given to switch between two inputs
> >> +- #address-cells: should be <1>
> >> +- #size-cells: should be <0>
> >> +- port@*: at least three port nodes containing endpoints connecting to
> >> the
> >> +  source and sink devices according to of_graph bindings. The last port
> >> is
> >> +  the output port, all others are inputs.
> >> +
> >> +Example:
> >> +
> >> +syscon {
> >> +	compatible = "syscon", "simple-mfd";
> >> +
> >> +	mux {
> > 
> > Could you use standardised properties for this, i.e. ones defined in
> > Documentation/devicetree/bindings/media/video-interfaces.txt ?
> 
> What do you mean? Should we add the optional bus-width property to
> describe the bit width of the parallel bus even though the driver
> doesn't care about any of the properties?
> 
> > This is very similar to another patch "[PATCH] devicetree: Add video bus
> > switch" posted by Pavel Machek recently. The problem with that is also
> > similar than with this one: how to pass the CSI-2 bus configuration to the
> > receiver.
> > 
> > There's some discussion here:
> > 
> > <URL:http://www.spinics.net/lists/linux-media/msg109493.html>
> 
> [Added Sebastian do Cc:]
> 
> Yes, this is essentially the same driver, except that this driver also
> handles MMIO-bitfield controlled muxes, and that the actual physical bus
> (which the drivers currently don't care about) is MIPI CSI-2 in the
> other case, and parallel this one.
> They should probably be combined, or maybe split into two separate
> drivers (MMIO controlled mux, GPIO controlled switch), possibly using
> the same v4l2_subdev boilerplate.

I'd split the bindings in two with two separate compat strings, but we can 
handle both in a single driver as most of the code would be identical 
otherwise.

> > As Laurent already suggested, I think we should have a common solution for
> > the problem that, besides conveying the bus parameters to the receiver,
> > also encompasses CSI-2 virtual channels and data types.
> 
> Yes, that seems to be necessary, as certainly we can't configure the mux
> output bus parameters in DT to a fixed setting.
> 
> > That would mean finishing the series of patches in the branch I believe
> > Laurent already quoted here.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-08 10:05         ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-08 10:05 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, Steve Longerbeam,
	Sascha Hauer, linux-media, devicetree, arnd, mchehab, bparrot,
	robh+dt, horms+renesas, tiffany.lin, linux-arm-kernel,
	niklas.soderlund+renesas, gregkh, linux-kernel, Sakari Ailus,
	jean-christophe.trotin, kernel, Sebastian Reichel

Hi Philipp,

On Wednesday 08 Feb 2017 10:47:15 Philipp Zabel wrote:
> On Tue, 2017-02-07 at 22:46 +0200, Sakari Ailus wrote:
> > On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> >> From: Philipp Zabel <p.zabel@pengutronix.de>
> >> 
> >> This driver can handle SoC internal and external video bus multiplexers,
> >> controlled either by register bit fields or by a GPIO. The subdevice
> >> passes through frame interval and mbus configuration of the active input
> >> to the output side.
> >> 
> >> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> >> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> >> 
> >> --
> >> 
> >> - fixed a cut&paste error in vidsw_remove():
> >> v4l2_async_register_subdev()
> >>   should be unregister.
> >> 
> >> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >>   to vidsw_remove().
> >> 
> >> - there was a line left over from a previous iteration that negated
> >>   the new way of determining the pad count just before it which
> >>   has been removed (num_pads = of_get_child_count(np)).
> >> 
> >> - Philipp Zabel has developed a set of patches that allow adding
> >>   to the subdev async notifier waiting list using a chaining method
> >>   from the async registered callbacks (v4l2_of_subdev_registered()
> >>   and the prep patches for that). For now, I've removed the use of
> >>   v4l2_of_subdev_registered() for the vidmux driver's registered
> >>   callback. This doesn't affect the functionality of this driver,
> >>   but allows for it to be merged now, before adding the chaining
> >>   support.
> >> 
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >> 
> >>  .../bindings/media/video-multiplexer.txt           |  59 +++
> >>  drivers/media/platform/Kconfig                     |   8 +
> >>  drivers/media/platform/Makefile                    |   2 +
> >>  drivers/media/platform/video-multiplexer.c         | 472 ++++++++++++++
> >>  4 files changed, 541 insertions(+)
> >>  create mode 100644
> >>  Documentation/devicetree/bindings/media/video-multiplexer.txt create
> >>  mode 100644 drivers/media/platform/video-multiplexer.c
> >> 
> >> diff --git
> >> a/Documentation/devicetree/bindings/media/video-multiplexer.txt
> >> b/Documentation/devicetree/bindings/media/video-multiplexer.txt new
> >> file mode 100644
> >> index 0000000..9d133d9
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> >> @@ -0,0 +1,59 @@
> >> +Video Multiplexer
> >> +=================
> >> +
> >> +Video multiplexers allow to select between multiple input ports. Video
> >> received
> >> +on the active input port is passed through to the output port. Muxes
> >> described
> >> +by this binding may be controlled by a syscon register bitfield or by a
> >> GPIO.
> >> +
> >> +Required properties:
> >> +- compatible : should be "video-multiplexer"
> >> +- reg: should be register base of the register containing the control
> >> bitfield
> >> +- bit-mask: bitmask of the control bitfield in the control register
> >> +- bit-shift: bit offset of the control bitfield in the control register
> >> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO
> >> phandle
> >> +  may be given to switch between two inputs
> >> +- #address-cells: should be <1>
> >> +- #size-cells: should be <0>
> >> +- port@*: at least three port nodes containing endpoints connecting to
> >> the
> >> +  source and sink devices according to of_graph bindings. The last port
> >> is
> >> +  the output port, all others are inputs.
> >> +
> >> +Example:
> >> +
> >> +syscon {
> >> +	compatible = "syscon", "simple-mfd";
> >> +
> >> +	mux {
> > 
> > Could you use standardised properties for this, i.e. ones defined in
> > Documentation/devicetree/bindings/media/video-interfaces.txt ?
> 
> What do you mean? Should we add the optional bus-width property to
> describe the bit width of the parallel bus even though the driver
> doesn't care about any of the properties?
> 
> > This is very similar to another patch "[PATCH] devicetree: Add video bus
> > switch" posted by Pavel Machek recently. The problem with that is also
> > similar than with this one: how to pass the CSI-2 bus configuration to the
> > receiver.
> > 
> > There's some discussion here:
> > 
> > <URL:http://www.spinics.net/lists/linux-media/msg109493.html>
> 
> [Added Sebastian do Cc:]
> 
> Yes, this is essentially the same driver, except that this driver also
> handles MMIO-bitfield controlled muxes, and that the actual physical bus
> (which the drivers currently don't care about) is MIPI CSI-2 in the
> other case, and parallel this one.
> They should probably be combined, or maybe split into two separate
> drivers (MMIO controlled mux, GPIO controlled switch), possibly using
> the same v4l2_subdev boilerplate.

I'd split the bindings in two with two separate compat strings, but we can 
handle both in a single driver as most of the code would be identical 
otherwise.

> > As Laurent already suggested, I think we should have a common solution for
> > the problem that, besides conveying the bus parameters to the receiver,
> > also encompasses CSI-2 virtual channels and data types.
> 
> Yes, that seems to be necessary, as certainly we can't configure the mux
> output bus parameters in DT to a fixed setting.
> 
> > That would mean finishing the series of patches in the branch I believe
> > Laurent already quoted here.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 13/24] platform: add video-multiplexer subdevice driver
@ 2017-02-08 10:05         ` Laurent Pinchart
  0 siblings, 0 replies; 549+ messages in thread
From: Laurent Pinchart @ 2017-02-08 10:05 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Philipp,

On Wednesday 08 Feb 2017 10:47:15 Philipp Zabel wrote:
> On Tue, 2017-02-07 at 22:46 +0200, Sakari Ailus wrote:
> > On Fri, Jan 06, 2017 at 06:11:31PM -0800, Steve Longerbeam wrote:
> >> From: Philipp Zabel <p.zabel@pengutronix.de>
> >> 
> >> This driver can handle SoC internal and external video bus multiplexers,
> >> controlled either by register bit fields or by a GPIO. The subdevice
> >> passes through frame interval and mbus configuration of the active input
> >> to the output side.
> >> 
> >> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> >> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> >> 
> >> --
> >> 
> >> - fixed a cut&paste error in vidsw_remove():
> >> v4l2_async_register_subdev()
> >>   should be unregister.
> >> 
> >> - added media_entity_cleanup() and v4l2_device_unregister_subdev()
> >>   to vidsw_remove().
> >> 
> >> - there was a line left over from a previous iteration that negated
> >>   the new way of determining the pad count just before it which
> >>   has been removed (num_pads = of_get_child_count(np)).
> >> 
> >> - Philipp Zabel has developed a set of patches that allow adding
> >>   to the subdev async notifier waiting list using a chaining method
> >>   from the async registered callbacks (v4l2_of_subdev_registered()
> >>   and the prep patches for that). For now, I've removed the use of
> >>   v4l2_of_subdev_registered() for the vidmux driver's registered
> >>   callback. This doesn't affect the functionality of this driver,
> >>   but allows for it to be merged now, before adding the chaining
> >>   support.
> >> 
> >> Signed-off-by: Steve Longerbeam <steve_longerbeam@mentor.com>
> >> ---
> >> 
> >>  .../bindings/media/video-multiplexer.txt           |  59 +++
> >>  drivers/media/platform/Kconfig                     |   8 +
> >>  drivers/media/platform/Makefile                    |   2 +
> >>  drivers/media/platform/video-multiplexer.c         | 472 ++++++++++++++
> >>  4 files changed, 541 insertions(+)
> >>  create mode 100644
> >>  Documentation/devicetree/bindings/media/video-multiplexer.txt create
> >>  mode 100644 drivers/media/platform/video-multiplexer.c
> >> 
> >> diff --git
> >> a/Documentation/devicetree/bindings/media/video-multiplexer.txt
> >> b/Documentation/devicetree/bindings/media/video-multiplexer.txt new
> >> file mode 100644
> >> index 0000000..9d133d9
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
> >> @@ -0,0 +1,59 @@
> >> +Video Multiplexer
> >> +=================
> >> +
> >> +Video multiplexers allow to select between multiple input ports. Video
> >> received
> >> +on the active input port is passed through to the output port. Muxes
> >> described
> >> +by this binding may be controlled by a syscon register bitfield or by a
> >> GPIO.
> >> +
> >> +Required properties:
> >> +- compatible : should be "video-multiplexer"
> >> +- reg: should be register base of the register containing the control
> >> bitfield
> >> +- bit-mask: bitmask of the control bitfield in the control register
> >> +- bit-shift: bit offset of the control bitfield in the control register
> >> +- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO
> >> phandle
> >> +  may be given to switch between two inputs
> >> +- #address-cells: should be <1>
> >> +- #size-cells: should be <0>
> >> +- port@*: at least three port nodes containing endpoints connecting to
> >> the
> >> +  source and sink devices according to of_graph bindings. The last port
> >> is
> >> +  the output port, all others are inputs.
> >> +
> >> +Example:
> >> +
> >> +syscon {
> >> +	compatible = "syscon", "simple-mfd";
> >> +
> >> +	mux {
> > 
> > Could you use standardised properties for this, i.e. ones defined in
> > Documentation/devicetree/bindings/media/video-interfaces.txt ?
> 
> What do you mean? Should we add the optional bus-width property to
> describe the bit width of the parallel bus even though the driver
> doesn't care about any of the properties?
> 
> > This is very similar to another patch "[PATCH] devicetree: Add video bus
> > switch" posted by Pavel Machek recently. The problem with that is also
> > similar than with this one: how to pass the CSI-2 bus configuration to the
> > receiver.
> > 
> > There's some discussion here:
> > 
> > <URL:http://www.spinics.net/lists/linux-media/msg109493.html>
> 
> [Added Sebastian do Cc:]
> 
> Yes, this is essentially the same driver, except that this driver also
> handles MMIO-bitfield controlled muxes, and that the actual physical bus
> (which the drivers currently don't care about) is MIPI CSI-2 in the
> other case, and parallel this one.
> They should probably be combined, or maybe split into two separate
> drivers (MMIO controlled mux, GPIO controlled switch), possibly using
> the same v4l2_subdev boilerplate.

I'd split the bindings in two with two separate compat strings, but we can 
handle both in a single driver as most of the code would be identical 
otherwise.

> > As Laurent already suggested, I think we should have a common solution for
> > the problem that, besides conveying the bus parameters to the receiver,
> > also encompasses CSI-2 virtual channels and data types.
> 
> Yes, that seems to be necessary, as certainly we can't configure the mux
> output bus parameters in DT to a fixed setting.
> 
> > That would mean finishing the series of patches in the branch I believe
> > Laurent already quoted here.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-02-02 11:50     ` Philipp Zabel
  (?)
@ 2017-02-08 23:23       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-08 23:23 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 02/02/2017 03:50 AM, Philipp Zabel wrote:
>
>> +	struct v4l2_subdev     *src_sd;
>> +	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];
> I see no reason to store pointers to the remote v4l2_subdevs.
>
>> +	int                    input_pad;
>> +	struct clk             *dphy_clk;
>> +	struct clk             *cfg_clk;
>> +	struct clk             *pix_clk; /* what is this? */
>> +	void __iomem           *base;
>> +	int                     intr1;
>> +	int                     intr2;
> The interrupts are not used, I'd remove them and the dead code in
> _probe.

done.

>> +
>> +static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
>> +{
>> +	return readl(csi2->base + regoff);
>> +}
>> +
>> +static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
>> +				 unsigned int regoff)
>> +{
>> +	writel(val, csi2->base + regoff);
>> +}
> Do those two wrappers really make the code more readable?

It doesn't really matter to me either way, I removed these
macros.

>> +
>> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
>> +{
>> +	if (enable) {
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
> Given that these registers only contain a single bit, and bits 31:1 are
> documented as reserved, 0, I think these should write 1 instead of
> 0xffffffff.

Yes, these lines are lifted from the FSL BSP's mipi csi-2 driver. I
did notice this but left it in place because I was worried about
possible undocumented bits. I tried writing 1 to these registers
and the behavior is the same as before (still works).

>> +	} else {
>> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
>> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
>> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
>> +	}
>> +}
>> +
>> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
>> +{
>> +	imxcsi2_enable(csi2, false);
>> +
>> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> These magic constants should be replaced with proper defines for the
> documented bitfields, if available.
>
> #define PHY_TESTCLR		BIT(0)
> #define PHY_TESTCLK		BIT(1)
>
> #define PHY_TESTEN		BIT(16)
>
> 	/* Clear PHY test interface */
> 	imxcsi2_write(csi2, PHY_TESTCLR, CSI2_PHY_TST_CTRL0);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> 	/* Raise test interface strobe signal */
> 	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);
>
> 	/* Configure address write on falling edge and lower strobe signal */
> 	u8 addr = 0x44;
> 	imxcsi2_write(csi2, PHY_TESTEN | addr, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> 	/* Configure data write on rising edge and raise strobe signal */
> 	u8 data = 0x14;
> 	imxcsi2_write(csi2, data, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);
>
> 	/* Clear strobe signal */
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> The whole sequence should probably be encapsulated in a
> dw_mipi_dphy_write function.
>
> Actually, this exact function already exists as dw_mipi_dsi_phy_write in
> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> register 0x44 might contain a field called HSFREQRANGE_SEL.

Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
It's clear from that driver that there probably needs to be a fuller
treatment of the D-PHY programming here, but I don't know where
to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
in imxcsi2_reset() was also pulled from FSL, and that's all I really have
to go on for the D-PHY programming. I assume the D-PHY is also a
Synopsys core, like the host controller, but the i.MX6 manual doesn't
cover it.

In any case I've created dw_mipi_csi2_phy_write(), modeled after
dw_mipi_dsi_phy_write(). The "0x14" value is a value derived for
a target max bandwidth per lane of 300 Mbps, at least that is what
drivers/gpu/drm/rockchip/dw-mipi-dsi.c suggests. I've added a FIXME
note that effect, that this value should be derived based on the D-PHY
PLL clock rate and the desired max lane bandwidth.


>> +	imxcsi2_enable(csi2, true);
>> +}
>> +
>> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
>> +{
>> +	u32 reg;
>> +	int i;
>> +
>> +	/* wait for mipi sensor ready */
> More specifically, wait for the clock lane module to leave ULP state.

I've split this function in two:

imxcsi2_dphy_wait_lp_11()

which waits for !PHY_RXULPSCLKNOT, and stable/error-free
CSI-2 bus (CSI2_ERR1 == 0),

and

  imxcsi2_dphy_wait_clock_lane()

which waits for PHY_RXCLKACTIVEHS.

The former is called during s_power(1), the latter in s_stream(1).

>> +	for (i = 0; i < 50; i++) {
>> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
>> +		if (reg != 0x200)
> Magic constants are bad. This is PHY_RXULPSCLKNOT (clock lane module in
> ultra low power state).

Fixed.


>> +			break;
>> +		usleep_range(10000, 20000);
>> +	}
> How about breaking this out into a wait function, or even better, using
> readl_poll_timeout instead of open coding these loops multiple times?

Cool, thanks for pointing out that macro. I've switched to 
readl_poll_timeout()
everywhere.

>
>> +
>> +	if (i >= 50) {
>> +		v4l2_err(&csi2->sd,
>> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
>> +			 reg);
>> +		return -ETIME;
>> +	}
>> +
>> +	/* wait for mipi stable */
> Wait for error free transmission?

Changed to /* wait until no errors on bus */.

>
>> +
>> +	if (i >= 50) {
>> +		v4l2_err(&csi2->sd,
>> +			 "wait for controller timeout, err1 = 0x%08x\n",
>> +			 reg);
>> +		return -ETIME;
>> +	}
>> +
>> +	/* finally let's wait for active clock on the clock lane */
>> +	for (i = 0; i < 50; i++) {
>> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
>> +		if (reg & (1 << 8))
> Yes, and that is PHY_RXCLKACTIVEHS.

done.


>> +	return 0;
>> +}
>> +
>> +/*
>> + * V4L2 subdev operations
>> + */
>> +
>> +static int imxcsi2_link_setup(struct media_entity *entity,
>> +			      const struct media_pad *local,
>> +			      const struct media_pad *remote, u32 flags)
>> +{
>> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +	struct v4l2_subdev *remote_sd;
>> +
>> +	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
>> +		local->entity->name);
>> +
>> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
>> +
>> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (csi2->sink_sd[local->index])
>> +				return -EBUSY;
>> +			csi2->sink_sd[local->index] = remote_sd;
>> +		} else {
>> +			csi2->sink_sd[local->index] = NULL;
>> +		}
>> +	} else {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (csi2->src_sd)
>> +				return -EBUSY;
>> +			csi2->src_sd = remote_sd;
>> +		} else {
>> +			csi2->src_sd = NULL;
>> +		}
>> +	}
> This whole code block is just to check if there is another link already
> active on the given pad. This could be stored in a boolean or a bit, no
> need to store pointers to remote subdevices.

I converted these to true booleans.

>
>> +
>> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_pad_config *cfg,
>> +			   struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +
>> +	sdformat->format = csi2->format_mbus;
> The output formats are different from the input formats, see the media
> bus format discussion in the other thread. The input pad is the MIPI
> CSI-2 bus, but the four output pads are connected to the muxes / CSIs
> via a 16-bit parallel bus.
>
> So if the input format is UYVY8_1X16, for example, the output should be
> set to UYVY8_2X8.

Since the output buses from the CSI2IPU gasket are 16-bit
parallel buses, shouldn't an input format UYVY8_1X16 also be
the same at the output?

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-08 23:23       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-08 23:23 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, linux, geert, linux-media, devicetree,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, kernel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/02/2017 03:50 AM, Philipp Zabel wrote:
>
>> +	struct v4l2_subdev     *src_sd;
>> +	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];
> I see no reason to store pointers to the remote v4l2_subdevs.
>
>> +	int                    input_pad;
>> +	struct clk             *dphy_clk;
>> +	struct clk             *cfg_clk;
>> +	struct clk             *pix_clk; /* what is this? */
>> +	void __iomem           *base;
>> +	int                     intr1;
>> +	int                     intr2;
> The interrupts are not used, I'd remove them and the dead code in
> _probe.

done.

>> +
>> +static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
>> +{
>> +	return readl(csi2->base + regoff);
>> +}
>> +
>> +static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
>> +				 unsigned int regoff)
>> +{
>> +	writel(val, csi2->base + regoff);
>> +}
> Do those two wrappers really make the code more readable?

It doesn't really matter to me either way, I removed these
macros.

>> +
>> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
>> +{
>> +	if (enable) {
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
> Given that these registers only contain a single bit, and bits 31:1 are
> documented as reserved, 0, I think these should write 1 instead of
> 0xffffffff.

Yes, these lines are lifted from the FSL BSP's mipi csi-2 driver. I
did notice this but left it in place because I was worried about
possible undocumented bits. I tried writing 1 to these registers
and the behavior is the same as before (still works).

>> +	} else {
>> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
>> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
>> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
>> +	}
>> +}
>> +
>> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
>> +{
>> +	imxcsi2_enable(csi2, false);
>> +
>> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> These magic constants should be replaced with proper defines for the
> documented bitfields, if available.
>
> #define PHY_TESTCLR		BIT(0)
> #define PHY_TESTCLK		BIT(1)
>
> #define PHY_TESTEN		BIT(16)
>
> 	/* Clear PHY test interface */
> 	imxcsi2_write(csi2, PHY_TESTCLR, CSI2_PHY_TST_CTRL0);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> 	/* Raise test interface strobe signal */
> 	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);
>
> 	/* Configure address write on falling edge and lower strobe signal */
> 	u8 addr = 0x44;
> 	imxcsi2_write(csi2, PHY_TESTEN | addr, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> 	/* Configure data write on rising edge and raise strobe signal */
> 	u8 data = 0x14;
> 	imxcsi2_write(csi2, data, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);
>
> 	/* Clear strobe signal */
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> The whole sequence should probably be encapsulated in a
> dw_mipi_dphy_write function.
>
> Actually, this exact function already exists as dw_mipi_dsi_phy_write in
> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> register 0x44 might contain a field called HSFREQRANGE_SEL.

Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
It's clear from that driver that there probably needs to be a fuller
treatment of the D-PHY programming here, but I don't know where
to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
in imxcsi2_reset() was also pulled from FSL, and that's all I really have
to go on for the D-PHY programming. I assume the D-PHY is also a
Synopsys core, like the host controller, but the i.MX6 manual doesn't
cover it.

In any case I've created dw_mipi_csi2_phy_write(), modeled after
dw_mipi_dsi_phy_write(). The "0x14" value is a value derived for
a target max bandwidth per lane of 300 Mbps, at least that is what
drivers/gpu/drm/rockchip/dw-mipi-dsi.c suggests. I've added a FIXME
note that effect, that this value should be derived based on the D-PHY
PLL clock rate and the desired max lane bandwidth.


>> +	imxcsi2_enable(csi2, true);
>> +}
>> +
>> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
>> +{
>> +	u32 reg;
>> +	int i;
>> +
>> +	/* wait for mipi sensor ready */
> More specifically, wait for the clock lane module to leave ULP state.

I've split this function in two:

imxcsi2_dphy_wait_lp_11()

which waits for !PHY_RXULPSCLKNOT, and stable/error-free
CSI-2 bus (CSI2_ERR1 == 0),

and

  imxcsi2_dphy_wait_clock_lane()

which waits for PHY_RXCLKACTIVEHS.

The former is called during s_power(1), the latter in s_stream(1).

>> +	for (i = 0; i < 50; i++) {
>> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
>> +		if (reg != 0x200)
> Magic constants are bad. This is PHY_RXULPSCLKNOT (clock lane module in
> ultra low power state).

Fixed.


>> +			break;
>> +		usleep_range(10000, 20000);
>> +	}
> How about breaking this out into a wait function, or even better, using
> readl_poll_timeout instead of open coding these loops multiple times?

Cool, thanks for pointing out that macro. I've switched to 
readl_poll_timeout()
everywhere.

>
>> +
>> +	if (i >= 50) {
>> +		v4l2_err(&csi2->sd,
>> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
>> +			 reg);
>> +		return -ETIME;
>> +	}
>> +
>> +	/* wait for mipi stable */
> Wait for error free transmission?

Changed to /* wait until no errors on bus */.

>
>> +
>> +	if (i >= 50) {
>> +		v4l2_err(&csi2->sd,
>> +			 "wait for controller timeout, err1 = 0x%08x\n",
>> +			 reg);
>> +		return -ETIME;
>> +	}
>> +
>> +	/* finally let's wait for active clock on the clock lane */
>> +	for (i = 0; i < 50; i++) {
>> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
>> +		if (reg & (1 << 8))
> Yes, and that is PHY_RXCLKACTIVEHS.

done.


>> +	return 0;
>> +}
>> +
>> +/*
>> + * V4L2 subdev operations
>> + */
>> +
>> +static int imxcsi2_link_setup(struct media_entity *entity,
>> +			      const struct media_pad *local,
>> +			      const struct media_pad *remote, u32 flags)
>> +{
>> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +	struct v4l2_subdev *remote_sd;
>> +
>> +	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
>> +		local->entity->name);
>> +
>> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
>> +
>> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (csi2->sink_sd[local->index])
>> +				return -EBUSY;
>> +			csi2->sink_sd[local->index] = remote_sd;
>> +		} else {
>> +			csi2->sink_sd[local->index] = NULL;
>> +		}
>> +	} else {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (csi2->src_sd)
>> +				return -EBUSY;
>> +			csi2->src_sd = remote_sd;
>> +		} else {
>> +			csi2->src_sd = NULL;
>> +		}
>> +	}
> This whole code block is just to check if there is another link already
> active on the given pad. This could be stored in a boolean or a bit, no
> need to store pointers to remote subdevices.

I converted these to true booleans.

>
>> +
>> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_pad_config *cfg,
>> +			   struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +
>> +	sdformat->format = csi2->format_mbus;
> The output formats are different from the input formats, see the media
> bus format discussion in the other thread. The input pad is the MIPI
> CSI-2 bus, but the four output pads are connected to the muxes / CSIs
> via a 16-bit parallel bus.
>
> So if the input format is UYVY8_1X16, for example, the output should be
> set to UYVY8_2X8.

Since the output buses from the CSI2IPU gasket are 16-bit
parallel buses, shouldn't an input format UYVY8_1X16 also be
the same at the output?

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-08 23:23       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-08 23:23 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/02/2017 03:50 AM, Philipp Zabel wrote:
>
>> +	struct v4l2_subdev     *src_sd;
>> +	struct v4l2_subdev     *sink_sd[CSI2_NUM_SRC_PADS];
> I see no reason to store pointers to the remote v4l2_subdevs.
>
>> +	int                    input_pad;
>> +	struct clk             *dphy_clk;
>> +	struct clk             *cfg_clk;
>> +	struct clk             *pix_clk; /* what is this? */
>> +	void __iomem           *base;
>> +	int                     intr1;
>> +	int                     intr2;
> The interrupts are not used, I'd remove them and the dead code in
> _probe.

done.

>> +
>> +static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
>> +{
>> +	return readl(csi2->base + regoff);
>> +}
>> +
>> +static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
>> +				 unsigned int regoff)
>> +{
>> +	writel(val, csi2->base + regoff);
>> +}
> Do those two wrappers really make the code more readable?

It doesn't really matter to me either way, I removed these
macros.

>> +
>> +static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
>> +{
>> +	if (enable) {
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
>> +		imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
> Given that these registers only contain a single bit, and bits 31:1 are
> documented as reserved, 0, I think these should write 1 instead of
> 0xffffffff.

Yes, these lines are lifted from the FSL BSP's mipi csi-2 driver. I
did notice this but left it in place because I was worried about
possible undocumented bits. I tried writing 1 to these registers
and the behavior is the same as before (still works).

>> +	} else {
>> +		imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
>> +		imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
>> +		imxcsi2_write(csi2, 0x0, CSI2_RESETN);
>> +	}
>> +}
>> +
>> +static void imxcsi2_reset(struct imxcsi2_dev *csi2)
>> +{
>> +	imxcsi2_enable(csi2, false);
>> +
>> +	imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
>> +	imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
>> +	imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
> These magic constants should be replaced with proper defines for the
> documented bitfields, if available.
>
> #define PHY_TESTCLR		BIT(0)
> #define PHY_TESTCLK		BIT(1)
>
> #define PHY_TESTEN		BIT(16)
>
> 	/* Clear PHY test interface */
> 	imxcsi2_write(csi2, PHY_TESTCLR, CSI2_PHY_TST_CTRL0);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> 	/* Raise test interface strobe signal */
> 	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);
>
> 	/* Configure address write on falling edge and lower strobe signal */
> 	u8 addr = 0x44;
> 	imxcsi2_write(csi2, PHY_TESTEN | addr, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> 	/* Configure data write on rising edge and raise strobe signal */
> 	u8 data = 0x14;
> 	imxcsi2_write(csi2, data, CSI2_PHY_TST_CTRL1);
> 	imxcsi2_write(csi2, PHY_TESTCLK, CSI2_PHY_TST_CTRL0);
>
> 	/* Clear strobe signal */
> 	imxcsi2_write(csi2, 0, CSI2_PHY_TST_CTRL0);
>
> The whole sequence should probably be encapsulated in a
> dw_mipi_dphy_write function.
>
> Actually, this exact function already exists as dw_mipi_dsi_phy_write in
> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> register 0x44 might contain a field called HSFREQRANGE_SEL.

Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
It's clear from that driver that there probably needs to be a fuller
treatment of the D-PHY programming here, but I don't know where
to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
in imxcsi2_reset() was also pulled from FSL, and that's all I really have
to go on for the D-PHY programming. I assume the D-PHY is also a
Synopsys core, like the host controller, but the i.MX6 manual doesn't
cover it.

In any case I've created dw_mipi_csi2_phy_write(), modeled after
dw_mipi_dsi_phy_write(). The "0x14" value is a value derived for
a target max bandwidth per lane of 300 Mbps, at least that is what
drivers/gpu/drm/rockchip/dw-mipi-dsi.c suggests. I've added a FIXME
note that effect, that this value should be derived based on the D-PHY
PLL clock rate and the desired max lane bandwidth.


>> +	imxcsi2_enable(csi2, true);
>> +}
>> +
>> +static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
>> +{
>> +	u32 reg;
>> +	int i;
>> +
>> +	/* wait for mipi sensor ready */
> More specifically, wait for the clock lane module to leave ULP state.

I've split this function in two:

imxcsi2_dphy_wait_lp_11()

which waits for !PHY_RXULPSCLKNOT, and stable/error-free
CSI-2 bus (CSI2_ERR1 == 0),

and

  imxcsi2_dphy_wait_clock_lane()

which waits for PHY_RXCLKACTIVEHS.

The former is called during s_power(1), the latter in s_stream(1).

>> +	for (i = 0; i < 50; i++) {
>> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
>> +		if (reg != 0x200)
> Magic constants are bad. This is PHY_RXULPSCLKNOT (clock lane module in
> ultra low power state).

Fixed.


>> +			break;
>> +		usleep_range(10000, 20000);
>> +	}
> How about breaking this out into a wait function, or even better, using
> readl_poll_timeout instead of open coding these loops multiple times?

Cool, thanks for pointing out that macro. I've switched to 
readl_poll_timeout()
everywhere.

>
>> +
>> +	if (i >= 50) {
>> +		v4l2_err(&csi2->sd,
>> +			 "wait for clock lane timeout, phy_state = 0x%08x\n",
>> +			 reg);
>> +		return -ETIME;
>> +	}
>> +
>> +	/* wait for mipi stable */
> Wait for error free transmission?

Changed to /* wait until no errors on bus */.

>
>> +
>> +	if (i >= 50) {
>> +		v4l2_err(&csi2->sd,
>> +			 "wait for controller timeout, err1 = 0x%08x\n",
>> +			 reg);
>> +		return -ETIME;
>> +	}
>> +
>> +	/* finally let's wait for active clock on the clock lane */
>> +	for (i = 0; i < 50; i++) {
>> +		reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
>> +		if (reg & (1 << 8))
> Yes, and that is PHY_RXCLKACTIVEHS.

done.


>> +	return 0;
>> +}
>> +
>> +/*
>> + * V4L2 subdev operations
>> + */
>> +
>> +static int imxcsi2_link_setup(struct media_entity *entity,
>> +			      const struct media_pad *local,
>> +			      const struct media_pad *remote, u32 flags)
>> +{
>> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +	struct v4l2_subdev *remote_sd;
>> +
>> +	dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
>> +		local->entity->name);
>> +
>> +	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
>> +
>> +	if (local->flags & MEDIA_PAD_FL_SOURCE) {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (csi2->sink_sd[local->index])
>> +				return -EBUSY;
>> +			csi2->sink_sd[local->index] = remote_sd;
>> +		} else {
>> +			csi2->sink_sd[local->index] = NULL;
>> +		}
>> +	} else {
>> +		if (flags & MEDIA_LNK_FL_ENABLED) {
>> +			if (csi2->src_sd)
>> +				return -EBUSY;
>> +			csi2->src_sd = remote_sd;
>> +		} else {
>> +			csi2->src_sd = NULL;
>> +		}
>> +	}
> This whole code block is just to check if there is another link already
> active on the given pad. This could be stored in a boolean or a bit, no
> need to store pointers to remote subdevices.

I converted these to true booleans.

>
>> +
>> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_pad_config *cfg,
>> +			   struct v4l2_subdev_format *sdformat)
>> +{
>> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> +
>> +	sdformat->format = csi2->format_mbus;
> The output formats are different from the input formats, see the media
> bus format discussion in the other thread. The input pad is the MIPI
> CSI-2 bus, but the four output pads are connected to the muxes / CSIs
> via a 16-bit parallel bus.
>
> So if the input format is UYVY8_1X16, for example, the output should be
> set to UYVY8_2X8.

Since the output buses from the CSI2IPU gasket are 16-bit
parallel buses, shouldn't an input format UYVY8_1X16 also be
the same at the output?

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-02-08 23:23       ` Steve Longerbeam
  (?)
@ 2017-02-08 23:42         ` Russell King - ARM Linux
  -1 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-08 23:42 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
> >Actually, this exact function already exists as dw_mipi_dsi_phy_write in
> >drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> >register 0x44 might contain a field called HSFREQRANGE_SEL.
> 
> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
> It's clear from that driver that there probably needs to be a fuller
> treatment of the D-PHY programming here, but I don't know where
> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
> in imxcsi2_reset() was also pulled from FSL, and that's all I really have
> to go on for the D-PHY programming. I assume the D-PHY is also a
> Synopsys core, like the host controller, but the i.MX6 manual doesn't
> cover it.

Why exactly?  What problems are you seeing that would necessitate a
more detailed programming of the D-PHY?  From my testing, I can wind
a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
per lane with your existing code without any issues.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-08 23:42         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-08 23:42 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, Philipp Zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee

On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
> >Actually, this exact function already exists as dw_mipi_dsi_phy_write in
> >drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> >register 0x44 might contain a field called HSFREQRANGE_SEL.
> 
> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
> It's clear from that driver that there probably needs to be a fuller
> treatment of the D-PHY programming here, but I don't know where
> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
> in imxcsi2_reset() was also pulled from FSL, and that's all I really have
> to go on for the D-PHY programming. I assume the D-PHY is also a
> Synopsys core, like the host controller, but the i.MX6 manual doesn't
> cover it.

Why exactly?  What problems are you seeing that would necessitate a
more detailed programming of the D-PHY?  From my testing, I can wind
a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
per lane with your existing code without any issues.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-08 23:42         ` Russell King - ARM Linux
  0 siblings, 0 replies; 549+ messages in thread
From: Russell King - ARM Linux @ 2017-02-08 23:42 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
> >Actually, this exact function already exists as dw_mipi_dsi_phy_write in
> >drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> >register 0x44 might contain a field called HSFREQRANGE_SEL.
> 
> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
> It's clear from that driver that there probably needs to be a fuller
> treatment of the D-PHY programming here, but I don't know where
> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
> in imxcsi2_reset() was also pulled from FSL, and that's all I really have
> to go on for the D-PHY programming. I assume the D-PHY is also a
> Synopsys core, like the host controller, but the i.MX6 manual doesn't
> cover it.

Why exactly?  What problems are you seeing that would necessitate a
more detailed programming of the D-PHY?  From my testing, I can wind
a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
per lane with your existing code without any issues.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-02-08 23:23       ` Steve Longerbeam
  (?)
@ 2017-02-09  9:43         ` Philipp Zabel
  -1 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-09  9:43 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Wed, 2017-02-08 at 15:23 -0800, Steve Longerbeam wrote:
[...]
> >> +
> >> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> >> +			   struct v4l2_subdev_pad_config *cfg,
> >> +			   struct v4l2_subdev_format *sdformat)
> >> +{
> >> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> >> +
> >> +	sdformat->format = csi2->format_mbus;
> > The output formats are different from the input formats, see the media
> > bus format discussion in the other thread. The input pad is the MIPI
> > CSI-2 bus, but the four output pads are connected to the muxes / CSIs
> > via a 16-bit parallel bus.
> >
> > So if the input format is UYVY8_1X16, for example, the output should be
> > set to UYVY8_2X8.
> 
> Since the output buses from the CSI2IPU gasket are 16-bit
> parallel buses, shouldn't an input format UYVY8_1X16 also be
> the same at the output?

I looked at the reference manual again, and I think I have read this
incorrectly, probably confused by the coloring in Figure 19-10 "YUV422-8
data reception" in the CSI2IPU chapter. It looks like indeed the 16-bit
bus carries UYVY8_1X16, whereas the internal 32-bit bus between MIPI
CSI-2 decoder and the CSI2IPU carries two samples (complete UYVY) per
pixel clock.
So UYVY is straight forward. It's the YUV420, RGB, and RAW formats that
would be interesting to describe correctly.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-09  9:43         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-09  9:43 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

On Wed, 2017-02-08 at 15:23 -0800, Steve Longerbeam wrote:
[...]
> >> +
> >> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> >> +			   struct v4l2_subdev_pad_config *cfg,
> >> +			   struct v4l2_subdev_format *sdformat)
> >> +{
> >> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> >> +
> >> +	sdformat->format = csi2->format_mbus;
> > The output formats are different from the input formats, see the media
> > bus format discussion in the other thread. The input pad is the MIPI
> > CSI-2 bus, but the four output pads are connected to the muxes / CSIs
> > via a 16-bit parallel bus.
> >
> > So if the input format is UYVY8_1X16, for example, the output should be
> > set to UYVY8_2X8.
> 
> Since the output buses from the CSI2IPU gasket are 16-bit
> parallel buses, shouldn't an input format UYVY8_1X16 also be
> the same at the output?

I looked at the reference manual again, and I think I have read this
incorrectly, probably confused by the coloring in Figure 19-10 "YUV422-8
data reception" in the CSI2IPU chapter. It looks like indeed the 16-bit
bus carries UYVY8_1X16, whereas the internal 32-bit bus between MIPI
CSI-2 decoder and the CSI2IPU carries two samples (complete UYVY) per
pixel clock.
So UYVY is straight forward. It's the YUV420, RGB, and RAW formats that
would be interesting to describe correctly.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-09  9:43         ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-09  9:43 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, 2017-02-08 at 15:23 -0800, Steve Longerbeam wrote:
[...]
> >> +
> >> +static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
> >> +			   struct v4l2_subdev_pad_config *cfg,
> >> +			   struct v4l2_subdev_format *sdformat)
> >> +{
> >> +	struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> >> +
> >> +	sdformat->format = csi2->format_mbus;
> > The output formats are different from the input formats, see the media
> > bus format discussion in the other thread. The input pad is the MIPI
> > CSI-2 bus, but the four output pads are connected to the muxes / CSIs
> > via a 16-bit parallel bus.
> >
> > So if the input format is UYVY8_1X16, for example, the output should be
> > set to UYVY8_2X8.
> 
> Since the output buses from the CSI2IPU gasket are 16-bit
> parallel buses, shouldn't an input format UYVY8_1X16 also be
> the same at the output?

I looked at the reference manual again, and I think I have read this
incorrectly, probably confused by the coloring in Figure 19-10 "YUV422-8
data reception" in the CSI2IPU chapter. It looks like indeed the 16-bit
bus carries UYVY8_1X16, whereas the internal 32-bit bus between MIPI
CSI-2 decoder and the CSI2IPU carries two samples (complete UYVY) per
pixel clock.
So UYVY is straight forward. It's the YUV420, RGB, and RAW formats that
would be interesting to describe correctly.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-02-08 23:42         ` Russell King - ARM Linux
  (?)
@ 2017-02-09 23:49           ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-09 23:49 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>> Actually, this exact function already exists as dw_mipi_dsi_phy_write in
>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>> It's clear from that driver that there probably needs to be a fuller
>> treatment of the D-PHY programming here, but I don't know where
>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>> in imxcsi2_reset() was also pulled from FSL, and that's all I really have
>> to go on for the D-PHY programming. I assume the D-PHY is also a
>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>> cover it.
> Why exactly?  What problems are you seeing that would necessitate a
> more detailed programming of the D-PHY?  From my testing, I can wind
> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
> per lane with your existing code without any issues.

That's good to hear.

Just from my experience with struggles to get the CSI2 receiver
up and running with an active clock lane, and my suspicions that
some of that could be due to my lack of understanding of the D-PHY
programming.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-09 23:49           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-09 23:49 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, Philipp Zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>> Actually, this exact function already exists as dw_mipi_dsi_phy_write in
>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>> It's clear from that driver that there probably needs to be a fuller
>> treatment of the D-PHY programming here, but I don't know where
>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>> in imxcsi2_reset() was also pulled from FSL, and that's all I really have
>> to go on for the D-PHY programming. I assume the D-PHY is also a
>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>> cover it.
> Why exactly?  What problems are you seeing that would necessitate a
> more detailed programming of the D-PHY?  From my testing, I can wind
> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
> per lane with your existing code without any issues.

That's good to hear.

Just from my experience with struggles to get the CSI2 receiver
up and running with an active clock lane, and my suspicions that
some of that could be due to my lack of understanding of the D-PHY
programming.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-09 23:49           ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-09 23:49 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>> Actually, this exact function already exists as dw_mipi_dsi_phy_write in
>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>> It's clear from that driver that there probably needs to be a fuller
>> treatment of the D-PHY programming here, but I don't know where
>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>> in imxcsi2_reset() was also pulled from FSL, and that's all I really have
>> to go on for the D-PHY programming. I assume the D-PHY is also a
>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>> cover it.
> Why exactly?  What problems are you seeing that would necessitate a
> more detailed programming of the D-PHY?  From my testing, I can wind
> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
> per lane with your existing code without any issues.

That's good to hear.

Just from my experience with struggles to get the CSI2 receiver
up and running with an active clock lane, and my suspicions that
some of that could be due to my lack of understanding of the D-PHY
programming.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-02-09 23:49           ` Steve Longerbeam
  (?)
@ 2017-02-09 23:51             ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-09 23:51 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: Philipp Zabel, robh+dt, mark.rutland, shawnguo, kernel,
	fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam



On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
>
>
> On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
>> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>>> Actually, this exact function already exists as 
>>>> dw_mipi_dsi_phy_write in
>>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>>> It's clear from that driver that there probably needs to be a fuller
>>> treatment of the D-PHY programming here, but I don't know where
>>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>>> in imxcsi2_reset() was also pulled from FSL, and that's all I really 
>>> have
>>> to go on for the D-PHY programming. I assume the D-PHY is also a
>>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>>> cover it.
>> Why exactly?  What problems are you seeing that would necessitate a
>> more detailed programming of the D-PHY?  From my testing, I can wind
>> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
>> per lane with your existing code without any issues.
>
> That's good to hear.
>
> Just from my experience with struggles to get the CSI2 receiver
> up and running with an active clock lane, and my suspicions that
> some of that could be due to my lack of understanding of the D-PHY
> programming.

But I should add that after a re-org of the sequence, it looks more stable
now. Tested on both the SabreSD and SabreLite with the OV5640.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-09 23:51             ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-09 23:51 UTC (permalink / raw)
  To: Russell King - ARM Linux
  Cc: mark.rutland, andrew-ct.chen, minghsiu.tsai, nick, songjun.wu,
	hverkuil, Steve Longerbeam, robert.jarzmik, devel, markus.heiser,
	laurent.pinchart+renesas, geert, linux-media, devicetree, kernel,
	arnd, mchehab, bparrot, robh+dt, horms+renesas, tiffany.lin,
	linux-arm-kernel, niklas.soderlund+renesas, gregkh, linux-kernel,
	jean-christophe.trotin, Philipp Zabel, fabio.estevam, shawnguo,
	sudipm.mukherjee



On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
>
>
> On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
>> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>>> Actually, this exact function already exists as 
>>>> dw_mipi_dsi_phy_write in
>>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>>> It's clear from that driver that there probably needs to be a fuller
>>> treatment of the D-PHY programming here, but I don't know where
>>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>>> in imxcsi2_reset() was also pulled from FSL, and that's all I really 
>>> have
>>> to go on for the D-PHY programming. I assume the D-PHY is also a
>>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>>> cover it.
>> Why exactly?  What problems are you seeing that would necessitate a
>> more detailed programming of the D-PHY?  From my testing, I can wind
>> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
>> per lane with your existing code without any issues.
>
> That's good to hear.
>
> Just from my experience with struggles to get the CSI2 receiver
> up and running with an active clock lane, and my suspicions that
> some of that could be due to my lack of understanding of the D-PHY
> programming.

But I should add that after a re-org of the sequence, it looks more stable
now. Tested on both the SabreSD and SabreLite with the OV5640.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-09 23:51             ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-09 23:51 UTC (permalink / raw)
  To: linux-arm-kernel



On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
>
>
> On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
>> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>>> Actually, this exact function already exists as 
>>>> dw_mipi_dsi_phy_write in
>>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>>> It's clear from that driver that there probably needs to be a fuller
>>> treatment of the D-PHY programming here, but I don't know where
>>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>>> in imxcsi2_reset() was also pulled from FSL, and that's all I really 
>>> have
>>> to go on for the D-PHY programming. I assume the D-PHY is also a
>>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>>> cover it.
>> Why exactly?  What problems are you seeing that would necessitate a
>> more detailed programming of the D-PHY?  From my testing, I can wind
>> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
>> per lane with your existing code without any issues.
>
> That's good to hear.
>
> Just from my experience with struggles to get the CSI2 receiver
> up and running with an active clock lane, and my suspicions that
> some of that could be due to my lack of understanding of the D-PHY
> programming.

But I should add that after a re-org of the sequence, it looks more stable
now. Tested on both the SabreSD and SabreLite with the OV5640.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  2017-02-02 10:36     ` Laurent Pinchart
  (?)
  (?)
@ 2017-02-12 22:46     ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-12 22:46 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

[-- Attachment #1: Type: text/plain, Size: 29611 bytes --]



On 02/02/2017 02:36 AM, Laurent Pinchart wrote:
> Hi Steve,
>
> Thank you for the patch. Many of the comments below apply to the ov5642 driver
> too, please take them into account when reworking patch 23/24.
>
> On Friday 06 Jan 2017 18:11:40 Steve Longerbeam wrote:
>> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
>> branch, modified heavily to bring forward to latest interfaces and code
>> cleanup.
>>
>> Signed-off-by: Steve Longerbeam<steve_longerbeam@mentor.com>
>> ---
>>   drivers/staging/media/imx/Kconfig       |    8 +
>>   drivers/staging/media/imx/Makefile      |    2 +
>>   drivers/staging/media/imx/ov5640-mipi.c | 2348 ++++++++++++++++++++++++++++
> You're missing DT bindings.

Done, created Documentation/devicetree/bindings/media/i2c/ov5640.txt.

> The driver should go to drivers/media/i2c/ as it should not be specific to the
> i.MX6, and you can just call it ov5640.c.

Done.

>> diff --git a/drivers/staging/media/imx/Kconfig
>> b/drivers/staging/media/imx/Kconfig index ce2d2c8..09f373d 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>>   	---help---
>>   	  A video4linux camera capture driver for i.MX5/6.
>>
>> +config IMX_OV5640_MIPI
>> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
>> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> The sensor driver is generic, it shouldn't depend on IMX. It should however
> depend on at least I2C and OF by the look of it.

Done.

>
>> +
>> +#include <linux/module.h>
>> +#include <linux/init.h>
>> +#include <linux/slab.h>
>> +#include <linux/ctype.h>
>> +#include <linux/types.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/i2c.h>
>> +#include <linux/of_device.h>
>> +#include <linux/gpio/consumer.h>
>> +#include <linux/regulator/consumer.h>
>> +#include <linux/clk.h>
>> +#include <linux/clk-provider.h>
>> +#include <linux/clkdev.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/v4l2-async.h>
>> +#include <media/v4l2-of.h>
>> +#include <media/v4l2-ctrls.h>
> Pet peeve of mine, please sort the headers alphabetically. It makes it easier
> to locate duplicated.

Fixed.

>> +
>> +#define OV5640_CHIP_ID  0x300A
>> +#define OV5640_SLAVE_ID 0x3100
>> +#define OV5640_DEFAULT_SLAVE_ID 0x3c
> You're mixing lower-case and upper-case hex constants. Let's pick one. Kernel
> code usually favours lower-case.

Fixed.

> Please define macros for all the other numerical constants you use in the
> driver (register addresses and values). The large registers tables can be an
> exception if you don't have access to the information, but for registers
> written manually, hardcoding numerical values isn't good.

Done.

>> +
>> +#define OV5640_MAX_CONTROLS 64
>> +
>> +enum ov5640_mode {
>> +	ov5640_mode_MIN = 0,
>> +	ov5640_mode_QCIF_176_144 = 0,
>> +	ov5640_mode_QVGA_320_240,
>> +	ov5640_mode_VGA_640_480,
>> +	ov5640_mode_NTSC_720_480,
>> +	ov5640_mode_PAL_720_576,
>> +	ov5640_mode_XGA_1024_768,
>> +	ov5640_mode_720P_1280_720,
>> +	ov5640_mode_1080P_1920_1080,
>> +	ov5640_mode_QSXGA_2592_1944,
>> +	ov5640_num_modes,
>> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/
> Please add spaces after /* and before */.
>
> Enumerated values should be all upper-case.

Fixed (and ov5640_mode_INIT is removed).
>> +
>> +/* image size under 1280 * 960 are SUBSAMPLING
>> + * image size upper 1280 * 960 are SCALING
>> + */
> The kernel multi-line comment style is
>
> /*
>   * text
>   * text
>   */

Fixed.

>> +
>> +struct ov5640_dev {
>> +	struct i2c_client *i2c_client;
>> +	struct device *dev;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad;
>> +	struct v4l2_ctrl_handler ctrl_hdl;
>> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
>> +	struct v4l2_mbus_framefmt fmt;
>> +	struct v4l2_captureparm streamcap;
>> +	struct clk *xclk; /* system clock to OV5640 */
>> +	int xclk_freq;    /* requested xclk freq from devicetree */
>> +
>> +	enum ov5640_mode current_mode;
> Store a (const) pointer to the corresponding ov5640_mode_info instead, it will
> simplify the code and allow you to get rid of the ov5640_mode enum.

Done.

>> +
>> +	int prev_sysclk, prev_hts;
>> +	int ae_low, ae_high, ae_target;
> Can't these be unsigned int ?

yep, an old left-over Freescale-ism, fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
>> +static void ov5640_reset(struct ov5640_dev *sensor);
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
>> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
>> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_gain(struct ov5640_dev *sensor);
> No forward declarations please. You should reorder functions as needed (and of
> course still group related functions together).

Fixed.

>> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
>> +
>> <snip>
>> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
>> +};
> You only use the delay feature of the registers tables twice, once after
> writing the first two registers (to select the clock source and perform a
> software reset) and once at the very end.

There is a delay in other places as well. There is a 1 msec delay after
setting a PLL multiplier register, and another after programming the
15 fps 2592x1944 mode. I'd prefer to keep the delay support in place
for now. Later, removing these delays can be experimented with.

>   Remove it, write the first two
> registers manually in the code with a manual delay afterwards, and add another
> manual delay after writing the whole table.
>
> I'm actually wondering whether you couldn't remove the 300ms delay at the end,
> the 50/60Hz control register (0x3c00) doesn't look like it needs a delay after
> being written.
>
> [snip]
>
>> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
>> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
>> <snip>
>> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
> Don't turn the stream on in the init sequences, it should only be turned on
> from the .s_stream() operation.

Fixed. More old FSL-isms.

>> +};
>> +
>> +static struct ov5640_mode_info
>> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
> There's very few differences between the 15fps and 30fps tables. It would be
> better if you could merge them, and manually write the registers that differ.

Sorting that out will be a lot of work, I don't have the time, it will have
to wait for future fixes. The register tables are likely bloated with 
power-on
reset values that need to be pruned anyway (there is a FIXME added to that
effect).

>> +
>> +static int ov5640_probe(struct i2c_client *adapter,
>> +			const struct i2c_device_id *device_id);
>> +static int ov5640_remove(struct i2c_client *client);
> No forward declarations please.

Fixed, more FSL-isms.

>> +
>> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
>> +{
>> +	u8 reg_buf[2] = {0};
>> +	u8 read_val = 0;
>> +
>> +	reg_buf[0] = reg >> 8;
>> +	reg_buf[1] = reg & 0xff;
>> +
>> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
>> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
>> +			__func__, reg);
>> +		return -EIO;
>> +	}
>> +
>> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
>> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
>> +			__func__, reg, read_val);
>> +		return -EIO;
>> +	}
> Wouldn't i2c_transfer() be more efficient here ?

Yep, done, for every register access function.

>> +	*val = read_val;
>> +	return 0;
>> +}
>> +
>> +#define OV5640_READ_REG(s, r, v) {				\
>> +		ret = ov5640_read_reg((s), (r), (v));		\
>> +		if (ret)					\
>> +			return ret;				\
>> +	}
> No. No. No no no. Don't ever return from a macro. Hiding the return makes
> following the code flow much more difficult, it's just asking for trouble.
>
> And don't use externally defined variables (ret in this case), that's also
> asking for trouble.

Fixed.

>> +
>> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
>> +			  u8 mask, u8 val)
>> +{
>> +	u8 readval;
>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, reg, &readval);
>> +
>> +	readval &= ~mask;
>> +	val &= mask;
>> +	val |= readval;
>> +
>> +	OV5640_WRITE_REG(sensor, reg, val);
>> +	return 0;
>> +}
>>
>> If you need to modify registers a lot, switch to regmap for register access.
>> It will provide you with a cache, removing the need to read registers back
>> from the device.

ov5640_mod_reg() is only called in a few places. If this needs more use
later, will convert to regmap.

>> +/* download ov5640 settings to sensor through i2c */
>> +static int ov5640_load_regs(struct ov5640_dev *sensor,
>> +			    struct reg_value *regs,
>> +			    int size)
> size is never negative.

Fixed, actually new prototype is much simpler:

static int ov5640_load_regs(struct ov5640_dev *sensor,
                 const struct ov5640_mode_info *mode);

>> +{
>> +	register u32 delay_ms = 0;
>> +	register u16 reg_addr = 0;
>> +	register u8 mask = 0;
>> +	register u8 val = 0;
> register ? The compiler is nowadays likely smarter than us when it comes to
> register allocation.

Yeah, nutty. I didn't write that, more old FSL-isms.
>> +
>> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
>> +{
>> +	 /* read HTS from register settings */
>> +	u16 HTS;
> Function names and variables are lower case.

Fixed.

>> +
>> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
>> +{
>> +	int prev_vts;
>> +	int band_step60, max_band60, band_step50, max_band50;
> Aren't these values unsigned ?

Yep, fixed.

>> +
>> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
>> +{
>> +	/* stable in high */
>> +	int fast_high, fast_low;
> Aren't these values unsigned ?

Fixed.

>> +	int ret;
>> +
>> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
>> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
>> +
>> +	fast_high = sensor->ae_high<<1;
> Missing spaces around <<

Fixed.

>> +
>> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
>> +{
>> +	u8 temp, channel = sensor->ep.base.id;
> The endpoint id isn't meant to select a virtual channel. V4L2 has no virtual
> channel API at the moment, you can hardcode the VC to 0 for now.

Yes, imx-media bridge driver _really_ needs a subdev/V4L2 API to
set the virtual channel that a CSI-2 sensor will transmit on, because
it fully supports receiving on any of the 4 virtual channels (via the
4 source pads from the imx6-mipi-csi2 / CSI2IPU gasket entity).

For now I've hardcoded the ov5640 to transmit over channel 1. This is
good for the imx-media graph because it allows simultaneous capture
from a parallel sensor via ipu1_csi0, while the ov5640 captures via
ipu1_csi1.

Channel 1 may not be good for other bridges though. Maybe this would
be a good candidate for a module parameter.

>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, 0x4814, &temp);
>> +	temp &= ~(3 << 6);
>> +	temp |= (channel << 6);
>> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
>> +
>> +	return 0;
>> +}
>> +
>> +static enum ov5640_mode
>> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
>> +			 int width, int height)
> How about using v4l2_find_nearest_format() ?

It doesn't really fit ATM, see below.

>
>> +
>> +	ret = ov5640_set_stream(sensor, true);
> This will turn streaming on when you set the format, which I don't think is
> correct. You should only turn streaming on in the .s_stream() operation. The
> same comment applies to the previous function.

Fixed, FSL-ism. Tested and still works fine.


>> +
>> +static int ov5640_change_mode(struct ov5640_dev *sensor,
>> +			      enum ov5640_frame_rate frame_rate,
>> +			      enum ov5640_mode mode,
>> +			      enum ov5640_mode orig_mode)
>> +{
>> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
>> +	struct reg_value *mode_data = NULL;
>> +	int mode_size = 0;
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +		sensor->fmt.width = 640;
>> +		sensor->fmt.height = 480;
> Don't reset the format here. The format must be preserved across subdev
> open/close.

Fixed.

>> +
>> +/* restore the last set video mode after chip power-on */
>> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
>> +{
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +
>> +	/* first we need to set some initial register values */
>> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
>> +				    ov5640_mode_INIT, ov5640_mode_INIT);
> I wouldn't use ov5640_change_mode() here. You only need to apply the init
> register values, just call ov5640_load_regs(). The rest of the
> ov5640_change_mode() isn't needed, as you're calling it below. This will also
> allow you to get rid of ov5640_mode_INIT, simplifying the logic.

Fixed. I created an init register table, ov5640_mode_init_data, which is
loaded by ov5640_restore_mode().


>
>> +
>> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->io_regulator) {
>> +		ret = regulator_enable(sensor->io_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->core_regulator) {
>> +		ret = regulator_enable(sensor->core_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->gpo_regulator) {
>> +		ret = regulator_enable(sensor->gpo_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->analog_regulator) {
>> +		ret = regulator_enable(sensor->analog_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
> Maybe you should use the bulk regulator API ?

Done.

>> +
>> +/* --------------- Subdev Operations --------------- */
>> +
>> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	int ret;
>> +
>> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> +	if (on && !sensor->on) {
> The .s_power() calls need to be ref-counted, similarly to how the regulator
> enable/disable calls work. See the mt9p031 sensor driver for an example.

Done.

>> +		if (sensor->xclk)
>> +			clk_prepare_enable(sensor->xclk);
>> +
>> +		ret = ov5640_regulators_on(sensor);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ov5640_reset(sensor);
>> +		ov5640_power(sensor, true);
>> +
>> +		ret = ov5640_init_slave_id(sensor);
> Why is this needed ?

Because on SabreLite, the ov5642 is at the same default
i2c slave address, so they both need to answer to a non-default
address.

>> +
>> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
>> +	enum ov5640_frame_rate frame_rate;
>> +	u32 tgt_fps;	/* target frames per secound */
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +	else {
>> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
>> +			 tgt_fps);
> Don't print an error message that is userspace-triggerable, we have enough
> ways for applications to flood the kernel log already :-)

Removed.

>> +
>> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	format->format = sensor->fmt;
> You need to handle the TRY format here. You can have a look at the mt9p031
> driver for an example on how to do so.

Right, that was caught and fixed earlier.

>> +
>> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode new_mode;
>> +	int ret;
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
>> +		if (ret)
>> +			return ret;
> You can move this code above the test to avoid the duplicated call below.

Fixed.

>> +		cfg->try_fmt = format->format;
> Please use the v4l2_subdev_get_try_format() function to get the try format,
> don't dereference cfg directly.

Done.

>> +
>> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (value) {
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
>> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
>> +	} else
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
> According to the kernel coding style, you need curly braces around the else
> branch if you use them around the if branch.

Fixed everywhere.

>
>
>> +
>> +#if 0
> No compiled-out code please.

Removed ov5640_set_green_balance().

>> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->awb_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
>> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
>> +	return 0;
>> +}
>> +#endif
>> +
>> <snip>
>> +
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
> You can create a cluster with the autogain and manual gain controls
> (v4l2_ctrl_auto_cluster) to have the control framework disabling the manual
> control automatically when autogain is enabled.

Done. Also made white-balance and exposure auto clusters as well.
And in the process, got rid of the control table as Hans had suggested
earlier.


>> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
>> +	return 0;
>> +}
>> +
>> +static int ov5640_get_gain(struct ov5640_dev *sensor)
>> +{
>> +	u16 gain;
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
>> +
>> +	return gain & 0x3ff;
>> +}
>> +
>> +#if 0
>> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
>> +	return 0;
>> +}
>> +#endif
> You can use a control to expose the test pattern function, it's quite useful.

Done, as a menu control for color bars.

>> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
>> +{
>> +	struct ov5640_control *ret = NULL;
>> +	int i;
> i is never negative, it should be unsigned int.

Fixed.


>> +
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
> i is never negative, it should be unsigned int.

restore_ctrls is gone from earlier fixes.

>> +
>> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
>> +{
>> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
>> +	struct ov5640_control *c;
>> +	int ret = 0;
>> +	int i;
>> +
>> +	c = ov5640_get_ctrl(ctrl->id, &i);
> You can inline the function call here as it's not used anywhere else.

Done.

>> +
>> +static int ov5640_init_controls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
> I would name this ctrl or control, c can be a bit confusing. You can also
> declare it inside the loop.

Done.

>> +	int i;
> i can never be negative, you can make it an unsigned int.

Done.

>> +
>> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +
>> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
>> +				  c->ctrl.id, c->ctrl.minimum, c-
>> ctrl.maximum,
>> +				  c->ctrl.step, c->ctrl.default_value);
>> +	}
>> +
>> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
>> +	if (sensor->ctrl_hdl.error) {
>> +		int err = sensor->ctrl_hdl.error;
>> +
>> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +
>> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
> I'm not sure this brings much value.

Removed.

>
>> +
>> +static int ov5640_enum_frame_interval(
>> +	struct v4l2_subdev *sd,
>> +	struct v4l2_subdev_pad_config *cfg,
>> +	struct v4l2_subdev_frame_interval_enum *fie)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode mode;
>> +
>> +	if (fie->pad != 0)
>> +		return -EINVAL;
>> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
>> +		return -EINVAL;
>> +
>> +	if (fie->width == 0 || fie->height == 0)
>> +		return -EINVAL;
>> +
>> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
> You should find an exact mode here, not the nearest one. If with and height
> don't match, return -EINVAL. That will replace with above width == 0 and
> height == 0 test.

Fixed, I modified ov5640_find_mode() (renamed from 
ov5640_find_nearest_mode())
to take a "nearest" boolean, which is passed as false here.

>> +
>> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
>> +		fie->width, fie->height, fie->index, fie-
>> interval.denominator);
> I'm not sure this is very useful, you can use ftrace if you want to trace
> function calls.

Removed.
>> +	return 0;
>> +}
>> +
>> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
>> +
>> +	return 0;
>> +}
>> +
>> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
>> +			    u32 output, u32 config)
>> +{
>> +	return (input != 0) ? -EINVAL : 0;
>> +}
> The g_input_status and s_routing subdev operations are not mandatory, you
> don't have to implement them as the sensor doesn't have multiple inputs.

Removed.

>> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> You can use ftrace to trace function calls, there's no need to add debugging
> statements that duplicate the functionality.

Removed.

>> +
>> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
>> +	.core = &ov5640_core_ops,
>> +	.video = &ov5640_video_ops,
>> +	.pad = &ov5640_pad_ops,
>> +};
> All these structures should be static const.

Fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
>> +{
>> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
>> +}
>> +
>> +static void ov5640_reset(struct ov5640_dev *sensor)
>> +{
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +
>> +	/* camera power cycle */
>> +	ov5640_power(sensor, false);
>> +	usleep_range(5000, 10000);
>> +	ov5640_power(sensor, true);
>> +	usleep_range(5000, 10000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 1);
>> +	usleep_range(1000, 2000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +	usleep_range(5000, 10000);
>> +}
>> +
>> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
>> +{
>> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
>> +	if (!IS_ERR(sensor->io_regulator)) {
>> +		regulator_set_voltage(sensor->io_regulator,
>> +				      OV5640_VOLTAGE_DIGITAL_IO,
>> +				      OV5640_VOLTAGE_DIGITAL_IO);
>> +	} else {
>> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
>> +			__func__);
>> +		sensor->io_regulator = NULL;
> How about making the power supplies mandatory in DT instead ? They are
> mandatory after all, if they're not controllable DT should just declare fixed
> supplies.

I guess that makes sense. They are now mandatory.

>> +
>> +static int ov5640_probe(struct i2c_client *client,
>> +			const struct i2c_device_id *id)
>> +{
>> +	struct device *dev = &client->dev;
>> +	struct device_node *endpoint;
>> +	struct ov5640_dev *sensor;
>> +	int i, xclk, ret;
> i and xclk are never negative, you can make them unsigned int.

This was removed from earlier cleanups.

>> +
>> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
> Please use sizeof(*variable) instead of sizeof(type).

Done.

> devm_kzalloc() doesn't play nicely with dynamic removal of devices. We're in
> the process of fixing related race conditions in the media subsystem. In order
> not to make the problem worse, please use kzalloc() instead.
>
>> +	if (!sensor)
>> +		return -ENOMEM;
>> +
>> +	sensor->i2c_client = client;
>> +	sensor->dev = dev;
> Do you really need to store both i2c_client and dev, given that the latter is
> just &client->dev ?

It was only for convenience, but sensor->dev was now only used in a
couple places, so sensor->dev removed.

>
>> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
>> +	sensor->fmt.width = 640;
>> +	sensor->fmt.height = 480;
>> +	sensor->fmt.field = V4L2_FIELD_NONE;
>> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
>> +					   V4L2_CAP_TIMEPERFRAME;
> Please fix the indentation.

Fixed.

>> +
>> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
>> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
>> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
>> +		return -EINVAL;
> You're leaking endpoint here. You should move the of_node_put() call right
> after the v4l2_of_parse_endpoint() call.

oops, Fixed.

>> +
>> +	/* get system clock (xclk) frequency */
>> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
> Instead of adding a custom DT property for this, use assigned-clock-rates. You
> won't need to set it manually in the driver, and can verify its frequency with
> clk_get_rate().

This was done in earlier fixups.

>> +	if (!ret) {
>> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
> Are your register tables above independent of the clock frequency ? You should
> ideally compute register values at runtime instead of hardcoding them, but
> given the lack of information from Omnivision I understand this isn't
> possible. You thus need to be stricter here and reject any value other than
> the nominal frequency.

I went back and looked at the original driver from Freescale that these
register tables are based on, and that driver does allow a range between
6 and 24 MHz. I've updated the min/max for xclk to that range. I agree
though that this will need some experimentation to verify these xclk ranges
actually work.


>
>> +
>> +	/* get system clock (xclk) */
>> +	sensor->xclk = devm_clk_get(dev, "xclk");
>> +	if (!IS_ERR(sensor->xclk)) {
>> +		if (!sensor->xclk_freq) {
>> +			dev_err(dev, "xclk requires xclk frequency!\n");
>> +			return -EINVAL;
>> +		}
>> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
>> +	} else {
>> +		/* assume system clock enabled by default */
>> +		sensor->xclk = NULL;
> Please don't. The clock should be mandatory.

This was done in earlier fixups.


>> +	}
>> +
>> +	/* request power down pin */
>> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
> Are the GPIOs mandatory or optional ? Can a system tie some of them to ground
> or a voltage rail, or do they need to always be manually controllable ?

Yes, they could be tied to ground or a rail, and are not required. I
made both reset and pwdn gpios optional.

>
>> +
>> +	ret = ov5640_s_power(&sensor->sd, 1);
>> +	if (ret)
>> +		goto entity_cleanup;
>> +	ret = ov5640_init_controls(sensor);
>> +	if (ret)
>> +		goto power_off;
>> +
>> +	ret = ov5640_s_power(&sensor->sd, 0);
>> +	if (ret)
>> +		goto free_ctrls;
> Writing the controls here is pointless, as powering the chip down will lose
> all the values. You shouldn't call v4l2_ctrl_handler_setup() in
> ov5640_init_controls(), and you can then remove the ov5640_s_power() calls
> here.

Good idea, done.

>> +
>> +free_ctrls:
>> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +power_off:
>> +	ov5640_s_power(&sensor->sd, 0);
>> +entity_cleanup:
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	ov5640_regulators_off(sensor);
> Won't ov5640_s_power(0) already disable the regulators ?

Not applicable anymore, as there's no need to enable power in probe.

>> +	return ret;
>> +}
>> +
>> +/*!
>> + * ov5640 I2C detach function
>> + *
>> + * @param client            struct i2c_client *
>> + * @return  Error code indicating success or failure
>> + */
> That's not the kerneldoc comment style. Given that this is the only documented
> function, and that the comment is completely useless, you can just drop it.

yuck, old FSL-isms, removed.

>> +static int ov5640_remove(struct i2c_client *client)
>> +{
>> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	ov5640_regulators_off(sensor);
>> +
>> +	v4l2_async_unregister_subdev(&sensor->sd);
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	v4l2_device_unregister_subdev(sd);
> This function is called by v4l2_async_unregister_subdev(), there's no need to
> duplicate it.

Removed.

>> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
> MODULE_AUTHOR isn't a synonym for copyright ownership. I don't think you
> should add Freescale as an author. If you know who wrote the original code you
> can list that developer explicitly.

I have no clue who the original author was, I just removed
MODULE_AUTHOR(s).

>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
>> +MODULE_LICENSE("GPL");
>> +MODULE_VERSION("1.0");
> Version numbers are never updated. I wouldn't bother adding one.

Removed.

Steve


[-- Attachment #2: Type: text/html, Size: 44161 bytes --]

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
  2017-02-02 10:36     ` Laurent Pinchart
  (?)
@ 2017-02-12 22:53       ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-12 22:53 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

(resending text only)


On 02/02/2017 02:36 AM, Laurent Pinchart wrote:
> Hi Steve,
>
> Thank you for the patch. Many of the comments below apply to the ov5642 driver
> too, please take them into account when reworking patch 23/24.
>
> On Friday 06 Jan 2017 18:11:40 Steve Longerbeam wrote:
>> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
>> branch, modified heavily to bring forward to latest interfaces and code
>> cleanup.
>>
>> Signed-off-by: Steve Longerbeam<steve_longerbeam@mentor.com>
>> ---
>>   drivers/staging/media/imx/Kconfig       |    8 +
>>   drivers/staging/media/imx/Makefile      |    2 +
>>   drivers/staging/media/imx/ov5640-mipi.c | 2348 ++++++++++++++++++++++++++++
> You're missing DT bindings.

Done, created Documentation/devicetree/bindings/media/i2c/ov5640.txt.

> The driver should go to drivers/media/i2c/ as it should not be specific to the
> i.MX6, and you can just call it ov5640.c.

Done.

>> diff --git a/drivers/staging/media/imx/Kconfig
>> b/drivers/staging/media/imx/Kconfig index ce2d2c8..09f373d 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>>   	---help---
>>   	  A video4linux camera capture driver for i.MX5/6.
>>
>> +config IMX_OV5640_MIPI
>> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
>> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> The sensor driver is generic, it shouldn't depend on IMX. It should however
> depend on at least I2C and OF by the look of it.

Done.

>
>> +
>> +#include <linux/module.h>
>> +#include <linux/init.h>
>> +#include <linux/slab.h>
>> +#include <linux/ctype.h>
>> +#include <linux/types.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/i2c.h>
>> +#include <linux/of_device.h>
>> +#include <linux/gpio/consumer.h>
>> +#include <linux/regulator/consumer.h>
>> +#include <linux/clk.h>
>> +#include <linux/clk-provider.h>
>> +#include <linux/clkdev.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/v4l2-async.h>
>> +#include <media/v4l2-of.h>
>> +#include <media/v4l2-ctrls.h>
> Pet peeve of mine, please sort the headers alphabetically. It makes it easier
> to locate duplicated.

Fixed.

>> +
>> +#define OV5640_CHIP_ID  0x300A
>> +#define OV5640_SLAVE_ID 0x3100
>> +#define OV5640_DEFAULT_SLAVE_ID 0x3c
> You're mixing lower-case and upper-case hex constants. Let's pick one. Kernel
> code usually favours lower-case.

Fixed.

> Please define macros for all the other numerical constants you use in the
> driver (register addresses and values). The large registers tables can be an
> exception if you don't have access to the information, but for registers
> written manually, hardcoding numerical values isn't good.

Done.

>> +
>> +#define OV5640_MAX_CONTROLS 64
>> +
>> +enum ov5640_mode {
>> +	ov5640_mode_MIN = 0,
>> +	ov5640_mode_QCIF_176_144 = 0,
>> +	ov5640_mode_QVGA_320_240,
>> +	ov5640_mode_VGA_640_480,
>> +	ov5640_mode_NTSC_720_480,
>> +	ov5640_mode_PAL_720_576,
>> +	ov5640_mode_XGA_1024_768,
>> +	ov5640_mode_720P_1280_720,
>> +	ov5640_mode_1080P_1920_1080,
>> +	ov5640_mode_QSXGA_2592_1944,
>> +	ov5640_num_modes,
>> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/
> Please add spaces after /* and before */.
>
> Enumerated values should be all upper-case.

Fixed (and ov5640_mode_INIT is removed).
>> +
>> +/* image size under 1280 * 960 are SUBSAMPLING
>> + * image size upper 1280 * 960 are SCALING
>> + */
> The kernel multi-line comment style is
>
> /*
>   * text
>   * text
>   */

Fixed.

>> +
>> +struct ov5640_dev {
>> +	struct i2c_client *i2c_client;
>> +	struct device *dev;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad;
>> +	struct v4l2_ctrl_handler ctrl_hdl;
>> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
>> +	struct v4l2_mbus_framefmt fmt;
>> +	struct v4l2_captureparm streamcap;
>> +	struct clk *xclk; /* system clock to OV5640 */
>> +	int xclk_freq;    /* requested xclk freq from devicetree */
>> +
>> +	enum ov5640_mode current_mode;
> Store a (const) pointer to the corresponding ov5640_mode_info instead, it will
> simplify the code and allow you to get rid of the ov5640_mode enum.

Done.

>> +
>> +	int prev_sysclk, prev_hts;
>> +	int ae_low, ae_high, ae_target;
> Can't these be unsigned int ?

yep, an old left-over Freescale-ism, fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
>> +static void ov5640_reset(struct ov5640_dev *sensor);
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
>> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
>> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_gain(struct ov5640_dev *sensor);
> No forward declarations please. You should reorder functions as needed (and of
> course still group related functions together).

Fixed.

>> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
>> +
>> <snip>
>> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
>> +};
> You only use the delay feature of the registers tables twice, once after
> writing the first two registers (to select the clock source and perform a
> software reset) and once at the very end.

There is a delay in other places as well. There is a 1 msec delay after
setting a PLL multiplier register, and another after programming the
15 fps 2592x1944 mode. I'd prefer to keep the delay support in place
for now. Later, removing these delays can be experimented with.

>   Remove it, write the first two
> registers manually in the code with a manual delay afterwards, and add another
> manual delay after writing the whole table.
>
> I'm actually wondering whether you couldn't remove the 300ms delay at the end,
> the 50/60Hz control register (0x3c00) doesn't look like it needs a delay after
> being written.
>
> [snip]
>
>> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
>> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
>> <snip>
>> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
> Don't turn the stream on in the init sequences, it should only be turned on
> from the .s_stream() operation.

Fixed. More old FSL-isms.

>> +};
>> +
>> +static struct ov5640_mode_info
>> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
> There's very few differences between the 15fps and 30fps tables. It would be
> better if you could merge them, and manually write the registers that differ.

Sorting that out will be a lot of work, I don't have the time, it will have
to wait for future fixes. The register tables are likely bloated with 
power-on
reset values that need to be pruned anyway (there is a FIXME added to that
effect).

>> +
>> +static int ov5640_probe(struct i2c_client *adapter,
>> +			const struct i2c_device_id *device_id);
>> +static int ov5640_remove(struct i2c_client *client);
> No forward declarations please.

Fixed, more FSL-isms.

>> +
>> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
>> +{
>> +	u8 reg_buf[2] = {0};
>> +	u8 read_val = 0;
>> +
>> +	reg_buf[0] = reg >> 8;
>> +	reg_buf[1] = reg & 0xff;
>> +
>> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
>> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
>> +			__func__, reg);
>> +		return -EIO;
>> +	}
>> +
>> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
>> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
>> +			__func__, reg, read_val);
>> +		return -EIO;
>> +	}
> Wouldn't i2c_transfer() be more efficient here ?

Yep, done, for every register access function.

>> +	*val = read_val;
>> +	return 0;
>> +}
>> +
>> +#define OV5640_READ_REG(s, r, v) {				\
>> +		ret = ov5640_read_reg((s), (r), (v));		\
>> +		if (ret)					\
>> +			return ret;				\
>> +	}
> No. No. No no no. Don't ever return from a macro. Hiding the return makes
> following the code flow much more difficult, it's just asking for trouble.
>
> And don't use externally defined variables (ret in this case), that's also
> asking for trouble.

Fixed.

>> +
>> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
>> +			  u8 mask, u8 val)
>> +{
>> +	u8 readval;
>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, reg, &readval);
>> +
>> +	readval &= ~mask;
>> +	val &= mask;
>> +	val |= readval;
>> +
>> +	OV5640_WRITE_REG(sensor, reg, val);
>> +	return 0;
>> +}
>>
>> If you need to modify registers a lot, switch to regmap for register access.
>> It will provide you with a cache, removing the need to read registers back
>> from the device.

ov5640_mod_reg() is only called in a few places. If this needs more use
later, will convert to regmap.

>> +/* download ov5640 settings to sensor through i2c */
>> +static int ov5640_load_regs(struct ov5640_dev *sensor,
>> +			    struct reg_value *regs,
>> +			    int size)
> size is never negative.

Fixed, actually new prototype is much simpler:

static int ov5640_load_regs(struct ov5640_dev *sensor,
                 const struct ov5640_mode_info *mode);

>> +{
>> +	register u32 delay_ms = 0;
>> +	register u16 reg_addr = 0;
>> +	register u8 mask = 0;
>> +	register u8 val = 0;
> register ? The compiler is nowadays likely smarter than us when it comes to
> register allocation.

Yeah, nutty. I didn't write that, more old FSL-isms.
>> +
>> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
>> +{
>> +	 /* read HTS from register settings */
>> +	u16 HTS;
> Function names and variables are lower case.

Fixed.

>> +
>> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
>> +{
>> +	int prev_vts;
>> +	int band_step60, max_band60, band_step50, max_band50;
> Aren't these values unsigned ?

Yep, fixed.

>> +
>> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
>> +{
>> +	/* stable in high */
>> +	int fast_high, fast_low;
> Aren't these values unsigned ?

Fixed.

>> +	int ret;
>> +
>> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
>> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
>> +
>> +	fast_high = sensor->ae_high<<1;
> Missing spaces around <<

Fixed.

>> +
>> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
>> +{
>> +	u8 temp, channel = sensor->ep.base.id;
> The endpoint id isn't meant to select a virtual channel. V4L2 has no virtual
> channel API at the moment, you can hardcode the VC to 0 for now.

Yes, imx-media bridge driver _really_ needs a subdev/V4L2 API to
set the virtual channel that a CSI-2 sensor will transmit on, because
it fully supports receiving on any of the 4 virtual channels (via the
4 source pads from the imx6-mipi-csi2 / CSI2IPU gasket entity).

For now I've hardcoded the ov5640 to transmit over channel 1. This is
good for the imx-media graph because it allows simultaneous capture
from a parallel sensor via ipu1_csi0, while the ov5640 captures via
ipu1_csi1.

Channel 1 may not be good for other bridges though. Maybe this would
be a good candidate for a module parameter.

>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, 0x4814, &temp);
>> +	temp &= ~(3 << 6);
>> +	temp |= (channel << 6);
>> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
>> +
>> +	return 0;
>> +}
>> +
>> +static enum ov5640_mode
>> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
>> +			 int width, int height)
> How about using v4l2_find_nearest_format() ?

It doesn't really fit ATM, see below.

>
>> +
>> +	ret = ov5640_set_stream(sensor, true);
> This will turn streaming on when you set the format, which I don't think is
> correct. You should only turn streaming on in the .s_stream() operation. The
> same comment applies to the previous function.

Fixed, FSL-ism. Tested and still works fine.


>> +
>> +static int ov5640_change_mode(struct ov5640_dev *sensor,
>> +			      enum ov5640_frame_rate frame_rate,
>> +			      enum ov5640_mode mode,
>> +			      enum ov5640_mode orig_mode)
>> +{
>> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
>> +	struct reg_value *mode_data = NULL;
>> +	int mode_size = 0;
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +		sensor->fmt.width = 640;
>> +		sensor->fmt.height = 480;
> Don't reset the format here. The format must be preserved across subdev
> open/close.

Fixed.

>> +
>> +/* restore the last set video mode after chip power-on */
>> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
>> +{
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +
>> +	/* first we need to set some initial register values */
>> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
>> +				    ov5640_mode_INIT, ov5640_mode_INIT);
> I wouldn't use ov5640_change_mode() here. You only need to apply the init
> register values, just call ov5640_load_regs(). The rest of the
> ov5640_change_mode() isn't needed, as you're calling it below. This will also
> allow you to get rid of ov5640_mode_INIT, simplifying the logic.

Fixed. I created an init register table, ov5640_mode_init_data, which is
loaded by ov5640_restore_mode().


>
>> +
>> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->io_regulator) {
>> +		ret = regulator_enable(sensor->io_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->core_regulator) {
>> +		ret = regulator_enable(sensor->core_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->gpo_regulator) {
>> +		ret = regulator_enable(sensor->gpo_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->analog_regulator) {
>> +		ret = regulator_enable(sensor->analog_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
> Maybe you should use the bulk regulator API ?

Done.

>> +
>> +/* --------------- Subdev Operations --------------- */
>> +
>> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	int ret;
>> +
>> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> +	if (on && !sensor->on) {
> The .s_power() calls need to be ref-counted, similarly to how the regulator
> enable/disable calls work. See the mt9p031 sensor driver for an example.

Done.

>> +		if (sensor->xclk)
>> +			clk_prepare_enable(sensor->xclk);
>> +
>> +		ret = ov5640_regulators_on(sensor);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ov5640_reset(sensor);
>> +		ov5640_power(sensor, true);
>> +
>> +		ret = ov5640_init_slave_id(sensor);
> Why is this needed ?

Because on SabreLite, the ov5642 is at the same default
i2c slave address, so they both need to answer to a non-default
address.

>> +
>> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
>> +	enum ov5640_frame_rate frame_rate;
>> +	u32 tgt_fps;	/* target frames per secound */
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +	else {
>> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
>> +			 tgt_fps);
> Don't print an error message that is userspace-triggerable, we have enough
> ways for applications to flood the kernel log already :-)

Removed.

>> +
>> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	format->format = sensor->fmt;
> You need to handle the TRY format here. You can have a look at the mt9p031
> driver for an example on how to do so.

Right, that was caught and fixed earlier.

>> +
>> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode new_mode;
>> +	int ret;
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
>> +		if (ret)
>> +			return ret;
> You can move this code above the test to avoid the duplicated call below.

Fixed.

>> +		cfg->try_fmt = format->format;
> Please use the v4l2_subdev_get_try_format() function to get the try format,
> don't dereference cfg directly.

Done.

>> +
>> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (value) {
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
>> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
>> +	} else
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
> According to the kernel coding style, you need curly braces around the else
> branch if you use them around the if branch.

Fixed everywhere.

>
>
>> +
>> +#if 0
> No compiled-out code please.

Removed ov5640_set_green_balance().

>> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->awb_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
>> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
>> +	return 0;
>> +}
>> +#endif
>> +
>> <snip>
>> +
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
> You can create a cluster with the autogain and manual gain controls
> (v4l2_ctrl_auto_cluster) to have the control framework disabling the manual
> control automatically when autogain is enabled.

Done. Also made white-balance and exposure auto clusters as well.
And in the process, got rid of the control table as Hans had suggested
earlier.


>> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
>> +	return 0;
>> +}
>> +
>> +static int ov5640_get_gain(struct ov5640_dev *sensor)
>> +{
>> +	u16 gain;
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
>> +
>> +	return gain & 0x3ff;
>> +}
>> +
>> +#if 0
>> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
>> +	return 0;
>> +}
>> +#endif
> You can use a control to expose the test pattern function, it's quite useful.

Done, as a menu control for color bars.

>> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
>> +{
>> +	struct ov5640_control *ret = NULL;
>> +	int i;
> i is never negative, it should be unsigned int.

Fixed.


>> +
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
> i is never negative, it should be unsigned int.

restore_ctrls is gone from earlier fixes.

>> +
>> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
>> +{
>> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
>> +	struct ov5640_control *c;
>> +	int ret = 0;
>> +	int i;
>> +
>> +	c = ov5640_get_ctrl(ctrl->id, &i);
> You can inline the function call here as it's not used anywhere else.

Done.

>> +
>> +static int ov5640_init_controls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
> I would name this ctrl or control, c can be a bit confusing. You can also
> declare it inside the loop.

Done.

>> +	int i;
> i can never be negative, you can make it an unsigned int.

Done.

>> +
>> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +
>> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
>> +				  c->ctrl.id, c->ctrl.minimum, c-
>> ctrl.maximum,
>> +				  c->ctrl.step, c->ctrl.default_value);
>> +	}
>> +
>> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
>> +	if (sensor->ctrl_hdl.error) {
>> +		int err = sensor->ctrl_hdl.error;
>> +
>> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +
>> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
> I'm not sure this brings much value.

Removed.

>
>> +
>> +static int ov5640_enum_frame_interval(
>> +	struct v4l2_subdev *sd,
>> +	struct v4l2_subdev_pad_config *cfg,
>> +	struct v4l2_subdev_frame_interval_enum *fie)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode mode;
>> +
>> +	if (fie->pad != 0)
>> +		return -EINVAL;
>> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
>> +		return -EINVAL;
>> +
>> +	if (fie->width == 0 || fie->height == 0)
>> +		return -EINVAL;
>> +
>> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
> You should find an exact mode here, not the nearest one. If with and height
> don't match, return -EINVAL. That will replace with above width == 0 and
> height == 0 test.

Fixed, I modified ov5640_find_mode() (renamed from 
ov5640_find_nearest_mode())
to take a "nearest" boolean, which is passed as false here.

>> +
>> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
>> +		fie->width, fie->height, fie->index, fie-
>> interval.denominator);
> I'm not sure this is very useful, you can use ftrace if you want to trace
> function calls.

Removed.
>> +	return 0;
>> +}
>> +
>> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
>> +
>> +	return 0;
>> +}
>> +
>> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
>> +			    u32 output, u32 config)
>> +{
>> +	return (input != 0) ? -EINVAL : 0;
>> +}
> The g_input_status and s_routing subdev operations are not mandatory, you
> don't have to implement them as the sensor doesn't have multiple inputs.

Removed.

>> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> You can use ftrace to trace function calls, there's no need to add debugging
> statements that duplicate the functionality.

Removed.

>> +
>> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
>> +	.core = &ov5640_core_ops,
>> +	.video = &ov5640_video_ops,
>> +	.pad = &ov5640_pad_ops,
>> +};
> All these structures should be static const.

Fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
>> +{
>> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
>> +}
>> +
>> +static void ov5640_reset(struct ov5640_dev *sensor)
>> +{
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +
>> +	/* camera power cycle */
>> +	ov5640_power(sensor, false);
>> +	usleep_range(5000, 10000);
>> +	ov5640_power(sensor, true);
>> +	usleep_range(5000, 10000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 1);
>> +	usleep_range(1000, 2000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +	usleep_range(5000, 10000);
>> +}
>> +
>> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
>> +{
>> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
>> +	if (!IS_ERR(sensor->io_regulator)) {
>> +		regulator_set_voltage(sensor->io_regulator,
>> +				      OV5640_VOLTAGE_DIGITAL_IO,
>> +				      OV5640_VOLTAGE_DIGITAL_IO);
>> +	} else {
>> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
>> +			__func__);
>> +		sensor->io_regulator = NULL;
> How about making the power supplies mandatory in DT instead ? They are
> mandatory after all, if they're not controllable DT should just declare fixed
> supplies.

I guess that makes sense. They are now mandatory.

>> +
>> +static int ov5640_probe(struct i2c_client *client,
>> +			const struct i2c_device_id *id)
>> +{
>> +	struct device *dev = &client->dev;
>> +	struct device_node *endpoint;
>> +	struct ov5640_dev *sensor;
>> +	int i, xclk, ret;
> i and xclk are never negative, you can make them unsigned int.

This was removed from earlier cleanups.

>> +
>> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
> Please use sizeof(*variable) instead of sizeof(type).

Done.

> devm_kzalloc() doesn't play nicely with dynamic removal of devices. We're in
> the process of fixing related race conditions in the media subsystem. In order
> not to make the problem worse, please use kzalloc() instead.
>
>> +	if (!sensor)
>> +		return -ENOMEM;
>> +
>> +	sensor->i2c_client = client;
>> +	sensor->dev = dev;
> Do you really need to store both i2c_client and dev, given that the latter is
> just &client->dev ?

It was only for convenience, but sensor->dev was now only used in a
couple places, so sensor->dev removed.

>> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
>> +	sensor->fmt.width = 640;
>> +	sensor->fmt.height = 480;
>> +	sensor->fmt.field = V4L2_FIELD_NONE;
>> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
>> +					   V4L2_CAP_TIMEPERFRAME;
> Please fix the indentation.

Fixed.

>> +
>> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
>> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
>> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
>> +		return -EINVAL;
> You're leaking endpoint here. You should move the of_node_put() call right
> after the v4l2_of_parse_endpoint() call.

oops, Fixed.

>> +
>> +	/* get system clock (xclk) frequency */
>> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
> Instead of adding a custom DT property for this, use assigned-clock-rates. You
> won't need to set it manually in the driver, and can verify its frequency with
> clk_get_rate().

This was done in earlier fixups.

>> +	if (!ret) {
>> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
> Are your register tables above independent of the clock frequency ? You should
> ideally compute register values at runtime instead of hardcoding them, but
> given the lack of information from Omnivision I understand this isn't
> possible. You thus need to be stricter here and reject any value other than
> the nominal frequency.

I went back and looked at the original driver from Freescale that these
register tables are based on, and that driver does allow a range between
6 and 24 MHz. I've updated the min/max for xclk to that range. I agree
though that this will need some experimentation to verify these xclk ranges
actually work.


>> +
>> +	/* get system clock (xclk) */
>> +	sensor->xclk = devm_clk_get(dev, "xclk");
>> +	if (!IS_ERR(sensor->xclk)) {
>> +		if (!sensor->xclk_freq) {
>> +			dev_err(dev, "xclk requires xclk frequency!\n");
>> +			return -EINVAL;
>> +		}
>> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
>> +	} else {
>> +		/* assume system clock enabled by default */
>> +		sensor->xclk = NULL;
> Please don't. The clock should be mandatory.

This was done in earlier fixups.


>> +	}
>> +
>> +	/* request power down pin */
>> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
> Are the GPIOs mandatory or optional ? Can a system tie some of them to ground
> or a voltage rail, or do they need to always be manually controllable ?

Yes, they could be tied to ground or a rail, and are not required. I
made both reset and pwdn gpios optional.

>> +
>> +	ret = ov5640_s_power(&sensor->sd, 1);
>> +	if (ret)
>> +		goto entity_cleanup;
>> +	ret = ov5640_init_controls(sensor);
>> +	if (ret)
>> +		goto power_off;
>> +
>> +	ret = ov5640_s_power(&sensor->sd, 0);
>> +	if (ret)
>> +		goto free_ctrls;
> Writing the controls here is pointless, as powering the chip down will lose
> all the values. You shouldn't call v4l2_ctrl_handler_setup() in
> ov5640_init_controls(), and you can then remove the ov5640_s_power() calls
> here.

Good idea, done.

>> +
>> +free_ctrls:
>> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +power_off:
>> +	ov5640_s_power(&sensor->sd, 0);
>> +entity_cleanup:
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	ov5640_regulators_off(sensor);
> Won't ov5640_s_power(0) already disable the regulators ?

Not applicable anymore, as there's no need to enable power in probe.

>> +	return ret;
>> +}
>> +
>> +/*!
>> + * ov5640 I2C detach function
>> + *
>> + * @param client            struct i2c_client *
>> + * @return  Error code indicating success or failure
>> + */
> That's not the kerneldoc comment style. Given that this is the only documented
> function, and that the comment is completely useless, you can just drop it.

yuck, old FSL-isms, removed.

>> +static int ov5640_remove(struct i2c_client *client)
>> +{
>> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	ov5640_regulators_off(sensor);
>> +
>> +	v4l2_async_unregister_subdev(&sensor->sd);
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	v4l2_device_unregister_subdev(sd);
> This function is called by v4l2_async_unregister_subdev(), there's no need to
> duplicate it.

Removed.

>> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
> MODULE_AUTHOR isn't a synonym for copyright ownership. I don't think you
> should add Freescale as an author. If you know who wrote the original code you
> can list that developer explicitly.

I have no clue who the original author was, I just removed
MODULE_AUTHOR(s).

>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
>> +MODULE_LICENSE("GPL");
>> +MODULE_VERSION("1.0");
> Version numbers are never updated. I wouldn't bother adding one.

Removed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-02-12 22:53       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-12 22:53 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, p.zabel,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve

(resending text only)


On 02/02/2017 02:36 AM, Laurent Pinchart wrote:
> Hi Steve,
>
> Thank you for the patch. Many of the comments below apply to the ov5642 driver
> too, please take them into account when reworking patch 23/24.
>
> On Friday 06 Jan 2017 18:11:40 Steve Longerbeam wrote:
>> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
>> branch, modified heavily to bring forward to latest interfaces and code
>> cleanup.
>>
>> Signed-off-by: Steve Longerbeam<steve_longerbeam@mentor.com>
>> ---
>>   drivers/staging/media/imx/Kconfig       |    8 +
>>   drivers/staging/media/imx/Makefile      |    2 +
>>   drivers/staging/media/imx/ov5640-mipi.c | 2348 ++++++++++++++++++++++++++++
> You're missing DT bindings.

Done, created Documentation/devicetree/bindings/media/i2c/ov5640.txt.

> The driver should go to drivers/media/i2c/ as it should not be specific to the
> i.MX6, and you can just call it ov5640.c.

Done.

>> diff --git a/drivers/staging/media/imx/Kconfig
>> b/drivers/staging/media/imx/Kconfig index ce2d2c8..09f373d 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>>   	---help---
>>   	  A video4linux camera capture driver for i.MX5/6.
>>
>> +config IMX_OV5640_MIPI
>> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
>> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> The sensor driver is generic, it shouldn't depend on IMX. It should however
> depend on at least I2C and OF by the look of it.

Done.

>
>> +
>> +#include <linux/module.h>
>> +#include <linux/init.h>
>> +#include <linux/slab.h>
>> +#include <linux/ctype.h>
>> +#include <linux/types.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/i2c.h>
>> +#include <linux/of_device.h>
>> +#include <linux/gpio/consumer.h>
>> +#include <linux/regulator/consumer.h>
>> +#include <linux/clk.h>
>> +#include <linux/clk-provider.h>
>> +#include <linux/clkdev.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/v4l2-async.h>
>> +#include <media/v4l2-of.h>
>> +#include <media/v4l2-ctrls.h>
> Pet peeve of mine, please sort the headers alphabetically. It makes it easier
> to locate duplicated.

Fixed.

>> +
>> +#define OV5640_CHIP_ID  0x300A
>> +#define OV5640_SLAVE_ID 0x3100
>> +#define OV5640_DEFAULT_SLAVE_ID 0x3c
> You're mixing lower-case and upper-case hex constants. Let's pick one. Kernel
> code usually favours lower-case.

Fixed.

> Please define macros for all the other numerical constants you use in the
> driver (register addresses and values). The large registers tables can be an
> exception if you don't have access to the information, but for registers
> written manually, hardcoding numerical values isn't good.

Done.

>> +
>> +#define OV5640_MAX_CONTROLS 64
>> +
>> +enum ov5640_mode {
>> +	ov5640_mode_MIN = 0,
>> +	ov5640_mode_QCIF_176_144 = 0,
>> +	ov5640_mode_QVGA_320_240,
>> +	ov5640_mode_VGA_640_480,
>> +	ov5640_mode_NTSC_720_480,
>> +	ov5640_mode_PAL_720_576,
>> +	ov5640_mode_XGA_1024_768,
>> +	ov5640_mode_720P_1280_720,
>> +	ov5640_mode_1080P_1920_1080,
>> +	ov5640_mode_QSXGA_2592_1944,
>> +	ov5640_num_modes,
>> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/
> Please add spaces after /* and before */.
>
> Enumerated values should be all upper-case.

Fixed (and ov5640_mode_INIT is removed).
>> +
>> +/* image size under 1280 * 960 are SUBSAMPLING
>> + * image size upper 1280 * 960 are SCALING
>> + */
> The kernel multi-line comment style is
>
> /*
>   * text
>   * text
>   */

Fixed.

>> +
>> +struct ov5640_dev {
>> +	struct i2c_client *i2c_client;
>> +	struct device *dev;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad;
>> +	struct v4l2_ctrl_handler ctrl_hdl;
>> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
>> +	struct v4l2_mbus_framefmt fmt;
>> +	struct v4l2_captureparm streamcap;
>> +	struct clk *xclk; /* system clock to OV5640 */
>> +	int xclk_freq;    /* requested xclk freq from devicetree */
>> +
>> +	enum ov5640_mode current_mode;
> Store a (const) pointer to the corresponding ov5640_mode_info instead, it will
> simplify the code and allow you to get rid of the ov5640_mode enum.

Done.

>> +
>> +	int prev_sysclk, prev_hts;
>> +	int ae_low, ae_high, ae_target;
> Can't these be unsigned int ?

yep, an old left-over Freescale-ism, fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
>> +static void ov5640_reset(struct ov5640_dev *sensor);
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
>> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
>> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_gain(struct ov5640_dev *sensor);
> No forward declarations please. You should reorder functions as needed (and of
> course still group related functions together).

Fixed.

>> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
>> +
>> <snip>
>> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
>> +};
> You only use the delay feature of the registers tables twice, once after
> writing the first two registers (to select the clock source and perform a
> software reset) and once at the very end.

There is a delay in other places as well. There is a 1 msec delay after
setting a PLL multiplier register, and another after programming the
15 fps 2592x1944 mode. I'd prefer to keep the delay support in place
for now. Later, removing these delays can be experimented with.

>   Remove it, write the first two
> registers manually in the code with a manual delay afterwards, and add another
> manual delay after writing the whole table.
>
> I'm actually wondering whether you couldn't remove the 300ms delay at the end,
> the 50/60Hz control register (0x3c00) doesn't look like it needs a delay after
> being written.
>
> [snip]
>
>> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
>> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
>> <snip>
>> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
> Don't turn the stream on in the init sequences, it should only be turned on
> from the .s_stream() operation.

Fixed. More old FSL-isms.

>> +};
>> +
>> +static struct ov5640_mode_info
>> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
> There's very few differences between the 15fps and 30fps tables. It would be
> better if you could merge them, and manually write the registers that differ.

Sorting that out will be a lot of work, I don't have the time, it will have
to wait for future fixes. The register tables are likely bloated with 
power-on
reset values that need to be pruned anyway (there is a FIXME added to that
effect).

>> +
>> +static int ov5640_probe(struct i2c_client *adapter,
>> +			const struct i2c_device_id *device_id);
>> +static int ov5640_remove(struct i2c_client *client);
> No forward declarations please.

Fixed, more FSL-isms.

>> +
>> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
>> +{
>> +	u8 reg_buf[2] = {0};
>> +	u8 read_val = 0;
>> +
>> +	reg_buf[0] = reg >> 8;
>> +	reg_buf[1] = reg & 0xff;
>> +
>> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
>> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
>> +			__func__, reg);
>> +		return -EIO;
>> +	}
>> +
>> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
>> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
>> +			__func__, reg, read_val);
>> +		return -EIO;
>> +	}
> Wouldn't i2c_transfer() be more efficient here ?

Yep, done, for every register access function.

>> +	*val = read_val;
>> +	return 0;
>> +}
>> +
>> +#define OV5640_READ_REG(s, r, v) {				\
>> +		ret = ov5640_read_reg((s), (r), (v));		\
>> +		if (ret)					\
>> +			return ret;				\
>> +	}
> No. No. No no no. Don't ever return from a macro. Hiding the return makes
> following the code flow much more difficult, it's just asking for trouble.
>
> And don't use externally defined variables (ret in this case), that's also
> asking for trouble.

Fixed.

>> +
>> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
>> +			  u8 mask, u8 val)
>> +{
>> +	u8 readval;
>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, reg, &readval);
>> +
>> +	readval &= ~mask;
>> +	val &= mask;
>> +	val |= readval;
>> +
>> +	OV5640_WRITE_REG(sensor, reg, val);
>> +	return 0;
>> +}
>>
>> If you need to modify registers a lot, switch to regmap for register access.
>> It will provide you with a cache, removing the need to read registers back
>> from the device.

ov5640_mod_reg() is only called in a few places. If this needs more use
later, will convert to regmap.

>> +/* download ov5640 settings to sensor through i2c */
>> +static int ov5640_load_regs(struct ov5640_dev *sensor,
>> +			    struct reg_value *regs,
>> +			    int size)
> size is never negative.

Fixed, actually new prototype is much simpler:

static int ov5640_load_regs(struct ov5640_dev *sensor,
                 const struct ov5640_mode_info *mode);

>> +{
>> +	register u32 delay_ms = 0;
>> +	register u16 reg_addr = 0;
>> +	register u8 mask = 0;
>> +	register u8 val = 0;
> register ? The compiler is nowadays likely smarter than us when it comes to
> register allocation.

Yeah, nutty. I didn't write that, more old FSL-isms.
>> +
>> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
>> +{
>> +	 /* read HTS from register settings */
>> +	u16 HTS;
> Function names and variables are lower case.

Fixed.

>> +
>> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
>> +{
>> +	int prev_vts;
>> +	int band_step60, max_band60, band_step50, max_band50;
> Aren't these values unsigned ?

Yep, fixed.

>> +
>> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
>> +{
>> +	/* stable in high */
>> +	int fast_high, fast_low;
> Aren't these values unsigned ?

Fixed.

>> +	int ret;
>> +
>> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
>> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
>> +
>> +	fast_high = sensor->ae_high<<1;
> Missing spaces around <<

Fixed.

>> +
>> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
>> +{
>> +	u8 temp, channel = sensor->ep.base.id;
> The endpoint id isn't meant to select a virtual channel. V4L2 has no virtual
> channel API at the moment, you can hardcode the VC to 0 for now.

Yes, imx-media bridge driver _really_ needs a subdev/V4L2 API to
set the virtual channel that a CSI-2 sensor will transmit on, because
it fully supports receiving on any of the 4 virtual channels (via the
4 source pads from the imx6-mipi-csi2 / CSI2IPU gasket entity).

For now I've hardcoded the ov5640 to transmit over channel 1. This is
good for the imx-media graph because it allows simultaneous capture
from a parallel sensor via ipu1_csi0, while the ov5640 captures via
ipu1_csi1.

Channel 1 may not be good for other bridges though. Maybe this would
be a good candidate for a module parameter.

>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, 0x4814, &temp);
>> +	temp &= ~(3 << 6);
>> +	temp |= (channel << 6);
>> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
>> +
>> +	return 0;
>> +}
>> +
>> +static enum ov5640_mode
>> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
>> +			 int width, int height)
> How about using v4l2_find_nearest_format() ?

It doesn't really fit ATM, see below.

>
>> +
>> +	ret = ov5640_set_stream(sensor, true);
> This will turn streaming on when you set the format, which I don't think is
> correct. You should only turn streaming on in the .s_stream() operation. The
> same comment applies to the previous function.

Fixed, FSL-ism. Tested and still works fine.


>> +
>> +static int ov5640_change_mode(struct ov5640_dev *sensor,
>> +			      enum ov5640_frame_rate frame_rate,
>> +			      enum ov5640_mode mode,
>> +			      enum ov5640_mode orig_mode)
>> +{
>> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
>> +	struct reg_value *mode_data = NULL;
>> +	int mode_size = 0;
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +		sensor->fmt.width = 640;
>> +		sensor->fmt.height = 480;
> Don't reset the format here. The format must be preserved across subdev
> open/close.

Fixed.

>> +
>> +/* restore the last set video mode after chip power-on */
>> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
>> +{
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +
>> +	/* first we need to set some initial register values */
>> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
>> +				    ov5640_mode_INIT, ov5640_mode_INIT);
> I wouldn't use ov5640_change_mode() here. You only need to apply the init
> register values, just call ov5640_load_regs(). The rest of the
> ov5640_change_mode() isn't needed, as you're calling it below. This will also
> allow you to get rid of ov5640_mode_INIT, simplifying the logic.

Fixed. I created an init register table, ov5640_mode_init_data, which is
loaded by ov5640_restore_mode().


>
>> +
>> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->io_regulator) {
>> +		ret = regulator_enable(sensor->io_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->core_regulator) {
>> +		ret = regulator_enable(sensor->core_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->gpo_regulator) {
>> +		ret = regulator_enable(sensor->gpo_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->analog_regulator) {
>> +		ret = regulator_enable(sensor->analog_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
> Maybe you should use the bulk regulator API ?

Done.

>> +
>> +/* --------------- Subdev Operations --------------- */
>> +
>> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	int ret;
>> +
>> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> +	if (on && !sensor->on) {
> The .s_power() calls need to be ref-counted, similarly to how the regulator
> enable/disable calls work. See the mt9p031 sensor driver for an example.

Done.

>> +		if (sensor->xclk)
>> +			clk_prepare_enable(sensor->xclk);
>> +
>> +		ret = ov5640_regulators_on(sensor);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ov5640_reset(sensor);
>> +		ov5640_power(sensor, true);
>> +
>> +		ret = ov5640_init_slave_id(sensor);
> Why is this needed ?

Because on SabreLite, the ov5642 is at the same default
i2c slave address, so they both need to answer to a non-default
address.

>> +
>> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
>> +	enum ov5640_frame_rate frame_rate;
>> +	u32 tgt_fps;	/* target frames per secound */
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +	else {
>> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
>> +			 tgt_fps);
> Don't print an error message that is userspace-triggerable, we have enough
> ways for applications to flood the kernel log already :-)

Removed.

>> +
>> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	format->format = sensor->fmt;
> You need to handle the TRY format here. You can have a look at the mt9p031
> driver for an example on how to do so.

Right, that was caught and fixed earlier.

>> +
>> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode new_mode;
>> +	int ret;
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
>> +		if (ret)
>> +			return ret;
> You can move this code above the test to avoid the duplicated call below.

Fixed.

>> +		cfg->try_fmt = format->format;
> Please use the v4l2_subdev_get_try_format() function to get the try format,
> don't dereference cfg directly.

Done.

>> +
>> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (value) {
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
>> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
>> +	} else
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
> According to the kernel coding style, you need curly braces around the else
> branch if you use them around the if branch.

Fixed everywhere.

>
>
>> +
>> +#if 0
> No compiled-out code please.

Removed ov5640_set_green_balance().

>> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->awb_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
>> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
>> +	return 0;
>> +}
>> +#endif
>> +
>> <snip>
>> +
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
> You can create a cluster with the autogain and manual gain controls
> (v4l2_ctrl_auto_cluster) to have the control framework disabling the manual
> control automatically when autogain is enabled.

Done. Also made white-balance and exposure auto clusters as well.
And in the process, got rid of the control table as Hans had suggested
earlier.


>> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
>> +	return 0;
>> +}
>> +
>> +static int ov5640_get_gain(struct ov5640_dev *sensor)
>> +{
>> +	u16 gain;
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
>> +
>> +	return gain & 0x3ff;
>> +}
>> +
>> +#if 0
>> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
>> +	return 0;
>> +}
>> +#endif
> You can use a control to expose the test pattern function, it's quite useful.

Done, as a menu control for color bars.

>> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
>> +{
>> +	struct ov5640_control *ret = NULL;
>> +	int i;
> i is never negative, it should be unsigned int.

Fixed.


>> +
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
> i is never negative, it should be unsigned int.

restore_ctrls is gone from earlier fixes.

>> +
>> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
>> +{
>> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
>> +	struct ov5640_control *c;
>> +	int ret = 0;
>> +	int i;
>> +
>> +	c = ov5640_get_ctrl(ctrl->id, &i);
> You can inline the function call here as it's not used anywhere else.

Done.

>> +
>> +static int ov5640_init_controls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
> I would name this ctrl or control, c can be a bit confusing. You can also
> declare it inside the loop.

Done.

>> +	int i;
> i can never be negative, you can make it an unsigned int.

Done.

>> +
>> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +
>> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
>> +				  c->ctrl.id, c->ctrl.minimum, c-
>> ctrl.maximum,
>> +				  c->ctrl.step, c->ctrl.default_value);
>> +	}
>> +
>> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
>> +	if (sensor->ctrl_hdl.error) {
>> +		int err = sensor->ctrl_hdl.error;
>> +
>> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +
>> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
> I'm not sure this brings much value.

Removed.

>
>> +
>> +static int ov5640_enum_frame_interval(
>> +	struct v4l2_subdev *sd,
>> +	struct v4l2_subdev_pad_config *cfg,
>> +	struct v4l2_subdev_frame_interval_enum *fie)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode mode;
>> +
>> +	if (fie->pad != 0)
>> +		return -EINVAL;
>> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
>> +		return -EINVAL;
>> +
>> +	if (fie->width == 0 || fie->height == 0)
>> +		return -EINVAL;
>> +
>> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
> You should find an exact mode here, not the nearest one. If with and height
> don't match, return -EINVAL. That will replace with above width == 0 and
> height == 0 test.

Fixed, I modified ov5640_find_mode() (renamed from 
ov5640_find_nearest_mode())
to take a "nearest" boolean, which is passed as false here.

>> +
>> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
>> +		fie->width, fie->height, fie->index, fie-
>> interval.denominator);
> I'm not sure this is very useful, you can use ftrace if you want to trace
> function calls.

Removed.
>> +	return 0;
>> +}
>> +
>> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
>> +
>> +	return 0;
>> +}
>> +
>> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
>> +			    u32 output, u32 config)
>> +{
>> +	return (input != 0) ? -EINVAL : 0;
>> +}
> The g_input_status and s_routing subdev operations are not mandatory, you
> don't have to implement them as the sensor doesn't have multiple inputs.

Removed.

>> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> You can use ftrace to trace function calls, there's no need to add debugging
> statements that duplicate the functionality.

Removed.

>> +
>> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
>> +	.core = &ov5640_core_ops,
>> +	.video = &ov5640_video_ops,
>> +	.pad = &ov5640_pad_ops,
>> +};
> All these structures should be static const.

Fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
>> +{
>> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
>> +}
>> +
>> +static void ov5640_reset(struct ov5640_dev *sensor)
>> +{
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +
>> +	/* camera power cycle */
>> +	ov5640_power(sensor, false);
>> +	usleep_range(5000, 10000);
>> +	ov5640_power(sensor, true);
>> +	usleep_range(5000, 10000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 1);
>> +	usleep_range(1000, 2000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +	usleep_range(5000, 10000);
>> +}
>> +
>> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
>> +{
>> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
>> +	if (!IS_ERR(sensor->io_regulator)) {
>> +		regulator_set_voltage(sensor->io_regulator,
>> +				      OV5640_VOLTAGE_DIGITAL_IO,
>> +				      OV5640_VOLTAGE_DIGITAL_IO);
>> +	} else {
>> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
>> +			__func__);
>> +		sensor->io_regulator = NULL;
> How about making the power supplies mandatory in DT instead ? They are
> mandatory after all, if they're not controllable DT should just declare fixed
> supplies.

I guess that makes sense. They are now mandatory.

>> +
>> +static int ov5640_probe(struct i2c_client *client,
>> +			const struct i2c_device_id *id)
>> +{
>> +	struct device *dev = &client->dev;
>> +	struct device_node *endpoint;
>> +	struct ov5640_dev *sensor;
>> +	int i, xclk, ret;
> i and xclk are never negative, you can make them unsigned int.

This was removed from earlier cleanups.

>> +
>> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
> Please use sizeof(*variable) instead of sizeof(type).

Done.

> devm_kzalloc() doesn't play nicely with dynamic removal of devices. We're in
> the process of fixing related race conditions in the media subsystem. In order
> not to make the problem worse, please use kzalloc() instead.
>
>> +	if (!sensor)
>> +		return -ENOMEM;
>> +
>> +	sensor->i2c_client = client;
>> +	sensor->dev = dev;
> Do you really need to store both i2c_client and dev, given that the latter is
> just &client->dev ?

It was only for convenience, but sensor->dev was now only used in a
couple places, so sensor->dev removed.

>> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
>> +	sensor->fmt.width = 640;
>> +	sensor->fmt.height = 480;
>> +	sensor->fmt.field = V4L2_FIELD_NONE;
>> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
>> +					   V4L2_CAP_TIMEPERFRAME;
> Please fix the indentation.

Fixed.

>> +
>> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
>> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
>> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
>> +		return -EINVAL;
> You're leaking endpoint here. You should move the of_node_put() call right
> after the v4l2_of_parse_endpoint() call.

oops, Fixed.

>> +
>> +	/* get system clock (xclk) frequency */
>> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
> Instead of adding a custom DT property for this, use assigned-clock-rates. You
> won't need to set it manually in the driver, and can verify its frequency with
> clk_get_rate().

This was done in earlier fixups.

>> +	if (!ret) {
>> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
> Are your register tables above independent of the clock frequency ? You should
> ideally compute register values at runtime instead of hardcoding them, but
> given the lack of information from Omnivision I understand this isn't
> possible. You thus need to be stricter here and reject any value other than
> the nominal frequency.

I went back and looked at the original driver from Freescale that these
register tables are based on, and that driver does allow a range between
6 and 24 MHz. I've updated the min/max for xclk to that range. I agree
though that this will need some experimentation to verify these xclk ranges
actually work.


>> +
>> +	/* get system clock (xclk) */
>> +	sensor->xclk = devm_clk_get(dev, "xclk");
>> +	if (!IS_ERR(sensor->xclk)) {
>> +		if (!sensor->xclk_freq) {
>> +			dev_err(dev, "xclk requires xclk frequency!\n");
>> +			return -EINVAL;
>> +		}
>> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
>> +	} else {
>> +		/* assume system clock enabled by default */
>> +		sensor->xclk = NULL;
> Please don't. The clock should be mandatory.

This was done in earlier fixups.


>> +	}
>> +
>> +	/* request power down pin */
>> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
> Are the GPIOs mandatory or optional ? Can a system tie some of them to ground
> or a voltage rail, or do they need to always be manually controllable ?

Yes, they could be tied to ground or a rail, and are not required. I
made both reset and pwdn gpios optional.

>> +
>> +	ret = ov5640_s_power(&sensor->sd, 1);
>> +	if (ret)
>> +		goto entity_cleanup;
>> +	ret = ov5640_init_controls(sensor);
>> +	if (ret)
>> +		goto power_off;
>> +
>> +	ret = ov5640_s_power(&sensor->sd, 0);
>> +	if (ret)
>> +		goto free_ctrls;
> Writing the controls here is pointless, as powering the chip down will lose
> all the values. You shouldn't call v4l2_ctrl_handler_setup() in
> ov5640_init_controls(), and you can then remove the ov5640_s_power() calls
> here.

Good idea, done.

>> +
>> +free_ctrls:
>> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +power_off:
>> +	ov5640_s_power(&sensor->sd, 0);
>> +entity_cleanup:
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	ov5640_regulators_off(sensor);
> Won't ov5640_s_power(0) already disable the regulators ?

Not applicable anymore, as there's no need to enable power in probe.

>> +	return ret;
>> +}
>> +
>> +/*!
>> + * ov5640 I2C detach function
>> + *
>> + * @param client            struct i2c_client *
>> + * @return  Error code indicating success or failure
>> + */
> That's not the kerneldoc comment style. Given that this is the only documented
> function, and that the comment is completely useless, you can just drop it.

yuck, old FSL-isms, removed.

>> +static int ov5640_remove(struct i2c_client *client)
>> +{
>> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	ov5640_regulators_off(sensor);
>> +
>> +	v4l2_async_unregister_subdev(&sensor->sd);
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	v4l2_device_unregister_subdev(sd);
> This function is called by v4l2_async_unregister_subdev(), there's no need to
> duplicate it.

Removed.

>> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
> MODULE_AUTHOR isn't a synonym for copyright ownership. I don't think you
> should add Freescale as an author. If you know who wrote the original code you
> can list that developer explicitly.

I have no clue who the original author was, I just removed
MODULE_AUTHOR(s).

>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
>> +MODULE_LICENSE("GPL");
>> +MODULE_VERSION("1.0");
> Version numbers are never updated. I wouldn't bother adding one.

Removed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
@ 2017-02-12 22:53       ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-12 22:53 UTC (permalink / raw)
  To: linux-arm-kernel

(resending text only)


On 02/02/2017 02:36 AM, Laurent Pinchart wrote:
> Hi Steve,
>
> Thank you for the patch. Many of the comments below apply to the ov5642 driver
> too, please take them into account when reworking patch 23/24.
>
> On Friday 06 Jan 2017 18:11:40 Steve Longerbeam wrote:
>> This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
>> branch, modified heavily to bring forward to latest interfaces and code
>> cleanup.
>>
>> Signed-off-by: Steve Longerbeam<steve_longerbeam@mentor.com>
>> ---
>>   drivers/staging/media/imx/Kconfig       |    8 +
>>   drivers/staging/media/imx/Makefile      |    2 +
>>   drivers/staging/media/imx/ov5640-mipi.c | 2348 ++++++++++++++++++++++++++++
> You're missing DT bindings.

Done, created Documentation/devicetree/bindings/media/i2c/ov5640.txt.

> The driver should go to drivers/media/i2c/ as it should not be specific to the
> i.MX6, and you can just call it ov5640.c.

Done.

>> diff --git a/drivers/staging/media/imx/Kconfig
>> b/drivers/staging/media/imx/Kconfig index ce2d2c8..09f373d 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
>>   	---help---
>>   	  A video4linux camera capture driver for i.MX5/6.
>>
>> +config IMX_OV5640_MIPI
>> +       tristate "OmniVision OV5640 MIPI CSI-2 camera support"
>> +       depends on GPIOLIB && VIDEO_IMX_CAMERA
> The sensor driver is generic, it shouldn't depend on IMX. It should however
> depend on at least I2C and OF by the look of it.

Done.

>
>> +
>> +#include <linux/module.h>
>> +#include <linux/init.h>
>> +#include <linux/slab.h>
>> +#include <linux/ctype.h>
>> +#include <linux/types.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/i2c.h>
>> +#include <linux/of_device.h>
>> +#include <linux/gpio/consumer.h>
>> +#include <linux/regulator/consumer.h>
>> +#include <linux/clk.h>
>> +#include <linux/clk-provider.h>
>> +#include <linux/clkdev.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/v4l2-async.h>
>> +#include <media/v4l2-of.h>
>> +#include <media/v4l2-ctrls.h>
> Pet peeve of mine, please sort the headers alphabetically. It makes it easier
> to locate duplicated.

Fixed.

>> +
>> +#define OV5640_CHIP_ID  0x300A
>> +#define OV5640_SLAVE_ID 0x3100
>> +#define OV5640_DEFAULT_SLAVE_ID 0x3c
> You're mixing lower-case and upper-case hex constants. Let's pick one. Kernel
> code usually favours lower-case.

Fixed.

> Please define macros for all the other numerical constants you use in the
> driver (register addresses and values). The large registers tables can be an
> exception if you don't have access to the information, but for registers
> written manually, hardcoding numerical values isn't good.

Done.

>> +
>> +#define OV5640_MAX_CONTROLS 64
>> +
>> +enum ov5640_mode {
>> +	ov5640_mode_MIN = 0,
>> +	ov5640_mode_QCIF_176_144 = 0,
>> +	ov5640_mode_QVGA_320_240,
>> +	ov5640_mode_VGA_640_480,
>> +	ov5640_mode_NTSC_720_480,
>> +	ov5640_mode_PAL_720_576,
>> +	ov5640_mode_XGA_1024_768,
>> +	ov5640_mode_720P_1280_720,
>> +	ov5640_mode_1080P_1920_1080,
>> +	ov5640_mode_QSXGA_2592_1944,
>> +	ov5640_num_modes,
>> +	ov5640_mode_INIT = 0xff, /*only for sensor init*/
> Please add spaces after /* and before */.
>
> Enumerated values should be all upper-case.

Fixed (and ov5640_mode_INIT is removed).
>> +
>> +/* image size under 1280 * 960 are SUBSAMPLING
>> + * image size upper 1280 * 960 are SCALING
>> + */
> The kernel multi-line comment style is
>
> /*
>   * text
>   * text
>   */

Fixed.

>> +
>> +struct ov5640_dev {
>> +	struct i2c_client *i2c_client;
>> +	struct device *dev;
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pad;
>> +	struct v4l2_ctrl_handler ctrl_hdl;
>> +	struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
>> +	struct v4l2_mbus_framefmt fmt;
>> +	struct v4l2_captureparm streamcap;
>> +	struct clk *xclk; /* system clock to OV5640 */
>> +	int xclk_freq;    /* requested xclk freq from devicetree */
>> +
>> +	enum ov5640_mode current_mode;
> Store a (const) pointer to the corresponding ov5640_mode_info instead, it will
> simplify the code and allow you to get rid of the ov5640_mode enum.

Done.

>> +
>> +	int prev_sysclk, prev_hts;
>> +	int ae_low, ae_high, ae_target;
> Can't these be unsigned int ?

yep, an old left-over Freescale-ism, fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable);
>> +static void ov5640_reset(struct ov5640_dev *sensor);
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
>> +static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
>> +static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_exposure(struct ov5640_dev *sensor);
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
>> +static int ov5640_get_gain(struct ov5640_dev *sensor);
> No forward declarations please. You should reorder functions as needed (and of
> course still group related functions together).

Fixed.

>> +static struct reg_value ov5640_init_setting_30fps_VGA[] = {
>> +
>> <snip>
>> +	{0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
>> +};
> You only use the delay feature of the registers tables twice, once after
> writing the first two registers (to select the clock source and perform a
> software reset) and once at the very end.

There is a delay in other places as well. There is a 1 msec delay after
setting a PLL multiplier register, and another after programming the
15 fps 2592x1944 mode. I'd prefer to keep the delay support in place
for now. Later, removing these delays can be experimented with.

>   Remove it, write the first two
> registers manually in the code with a manual delay afterwards, and add another
> manual delay after writing the whole table.
>
> I'm actually wondering whether you couldn't remove the 300ms delay at the end,
> the 50/60Hz control register (0x3c00) doesn't look like it needs a delay after
> being written.
>
> [snip]
>
>> +static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
>> +	{0x4202, 0x0f, 0, 0},	/* stream off the sensor */
>> <snip>
>> +	{0x4202, 0x00, 0, 0},	/* stream on the sensor */
> Don't turn the stream on in the init sequences, it should only be turned on
> from the .s_stream() operation.

Fixed. More old FSL-isms.

>> +};
>> +
>> +static struct ov5640_mode_info
>> +ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
> There's very few differences between the 15fps and 30fps tables. It would be
> better if you could merge them, and manually write the registers that differ.

Sorting that out will be a lot of work, I don't have the time, it will have
to wait for future fixes. The register tables are likely bloated with 
power-on
reset values that need to be pruned anyway (there is a FIXME added to that
effect).

>> +
>> +static int ov5640_probe(struct i2c_client *adapter,
>> +			const struct i2c_device_id *device_id);
>> +static int ov5640_remove(struct i2c_client *client);
> No forward declarations please.

Fixed, more FSL-isms.

>> +
>> +static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
>> +{
>> +	u8 reg_buf[2] = {0};
>> +	u8 read_val = 0;
>> +
>> +	reg_buf[0] = reg >> 8;
>> +	reg_buf[1] = reg & 0xff;
>> +
>> +	if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
>> +		v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
>> +			__func__, reg);
>> +		return -EIO;
>> +	}
>> +
>> +	if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
>> +		v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
>> +			__func__, reg, read_val);
>> +		return -EIO;
>> +	}
> Wouldn't i2c_transfer() be more efficient here ?

Yep, done, for every register access function.

>> +	*val = read_val;
>> +	return 0;
>> +}
>> +
>> +#define OV5640_READ_REG(s, r, v) {				\
>> +		ret = ov5640_read_reg((s), (r), (v));		\
>> +		if (ret)					\
>> +			return ret;				\
>> +	}
> No. No. No no no. Don't ever return from a macro. Hiding the return makes
> following the code flow much more difficult, it's just asking for trouble.
>
> And don't use externally defined variables (ret in this case), that's also
> asking for trouble.

Fixed.

>> +
>> +static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
>> +			  u8 mask, u8 val)
>> +{
>> +	u8 readval;
>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, reg, &readval);
>> +
>> +	readval &= ~mask;
>> +	val &= mask;
>> +	val |= readval;
>> +
>> +	OV5640_WRITE_REG(sensor, reg, val);
>> +	return 0;
>> +}
>>
>> If you need to modify registers a lot, switch to regmap for register access.
>> It will provide you with a cache, removing the need to read registers back
>> from the device.

ov5640_mod_reg() is only called in a few places. If this needs more use
later, will convert to regmap.

>> +/* download ov5640 settings to sensor through i2c */
>> +static int ov5640_load_regs(struct ov5640_dev *sensor,
>> +			    struct reg_value *regs,
>> +			    int size)
> size is never negative.

Fixed, actually new prototype is much simpler:

static int ov5640_load_regs(struct ov5640_dev *sensor,
                 const struct ov5640_mode_info *mode);

>> +{
>> +	register u32 delay_ms = 0;
>> +	register u16 reg_addr = 0;
>> +	register u8 mask = 0;
>> +	register u8 val = 0;
> register ? The compiler is nowadays likely smarter than us when it comes to
> register allocation.

Yeah, nutty. I didn't write that, more old FSL-isms.
>> +
>> +static int ov5640_get_HTS(struct ov5640_dev *sensor)
>> +{
>> +	 /* read HTS from register settings */
>> +	u16 HTS;
> Function names and variables are lower case.

Fixed.

>> +
>> +static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
>> +{
>> +	int prev_vts;
>> +	int band_step60, max_band60, band_step50, max_band50;
> Aren't these values unsigned ?

Yep, fixed.

>> +
>> +static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
>> +{
>> +	/* stable in high */
>> +	int fast_high, fast_low;
> Aren't these values unsigned ?

Fixed.

>> +	int ret;
>> +
>> +	sensor->ae_low = target * 23 / 25;	/* 0.92 */
>> +	sensor->ae_high = target * 27 / 25;	/* 1.08 */
>> +
>> +	fast_high = sensor->ae_high<<1;
> Missing spaces around <<

Fixed.

>> +
>> +static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
>> +{
>> +	u8 temp, channel = sensor->ep.base.id;
> The endpoint id isn't meant to select a virtual channel. V4L2 has no virtual
> channel API at the moment, you can hardcode the VC to 0 for now.

Yes, imx-media bridge driver _really_ needs a subdev/V4L2 API to
set the virtual channel that a CSI-2 sensor will transmit on, because
it fully supports receiving on any of the 4 virtual channels (via the
4 source pads from the imx6-mipi-csi2 / CSI2IPU gasket entity).

For now I've hardcoded the ov5640 to transmit over channel 1. This is
good for the imx-media graph because it allows simultaneous capture
from a parallel sensor via ipu1_csi0, while the ov5640 captures via
ipu1_csi1.

Channel 1 may not be good for other bridges though. Maybe this would
be a good candidate for a module parameter.

>> +	int ret;
>> +
>> +	OV5640_READ_REG(sensor, 0x4814, &temp);
>> +	temp &= ~(3 << 6);
>> +	temp |= (channel << 6);
>> +	OV5640_WRITE_REG(sensor, 0x4814, temp);
>> +
>> +	return 0;
>> +}
>> +
>> +static enum ov5640_mode
>> +ov5640_find_nearest_mode(struct ov5640_dev *sensor,
>> +			 int width, int height)
> How about using v4l2_find_nearest_format() ?

It doesn't really fit ATM, see below.

>
>> +
>> +	ret = ov5640_set_stream(sensor, true);
> This will turn streaming on when you set the format, which I don't think is
> correct. You should only turn streaming on in the .s_stream() operation. The
> same comment applies to the previous function.

Fixed, FSL-ism. Tested and still works fine.


>> +
>> +static int ov5640_change_mode(struct ov5640_dev *sensor,
>> +			      enum ov5640_frame_rate frame_rate,
>> +			      enum ov5640_mode mode,
>> +			      enum ov5640_mode orig_mode)
>> +{
>> +	enum ov5640_downsize_mode dn_mode, orig_dn_mode;
>> +	struct reg_value *mode_data = NULL;
>> +	int mode_size = 0;
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +		sensor->fmt.width = 640;
>> +		sensor->fmt.height = 480;
> Don't reset the format here. The format must be preserved across subdev
> open/close.

Fixed.

>> +
>> +/* restore the last set video mode after chip power-on */
>> +static int ov5640_restore_mode(struct ov5640_dev *sensor)
>> +{
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +
>> +	/* first we need to set some initial register values */
>> +	ret = ov5640_change_mode(sensor, sensor->current_fr,
>> +				    ov5640_mode_INIT, ov5640_mode_INIT);
> I wouldn't use ov5640_change_mode() here. You only need to apply the init
> register values, just call ov5640_load_regs(). The rest of the
> ov5640_change_mode() isn't needed, as you're calling it below. This will also
> allow you to get rid of ov5640_mode_INIT, simplifying the logic.

Fixed. I created an init register table, ov5640_mode_init_data, which is
loaded by ov5640_restore_mode().


>
>> +
>> +static int ov5640_regulators_on(struct ov5640_dev *sensor)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->io_regulator) {
>> +		ret = regulator_enable(sensor->io_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "io reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->core_regulator) {
>> +		ret = regulator_enable(sensor->core_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "core reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->gpo_regulator) {
>> +		ret = regulator_enable(sensor->gpo_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "gpo reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
>> +	if (sensor->analog_regulator) {
>> +		ret = regulator_enable(sensor->analog_regulator);
>> +		if (ret) {
>> +			v4l2_err(&sensor->sd, "analog reg enable failed\n");
>> +			return ret;
>> +		}
>> +	}
> Maybe you should use the bulk regulator API ?

Done.

>> +
>> +/* --------------- Subdev Operations --------------- */
>> +
>> +static int ov5640_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	int ret;
>> +
>> +	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> +	if (on && !sensor->on) {
> The .s_power() calls need to be ref-counted, similarly to how the regulator
> enable/disable calls work. See the mt9p031 sensor driver for an example.

Done.

>> +		if (sensor->xclk)
>> +			clk_prepare_enable(sensor->xclk);
>> +
>> +		ret = ov5640_regulators_on(sensor);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ov5640_reset(sensor);
>> +		ov5640_power(sensor, true);
>> +
>> +		ret = ov5640_init_slave_id(sensor);
> Why is this needed ?

Because on SabreLite, the ov5642 is at the same default
i2c slave address, so they both need to answer to a non-default
address.

>> +
>> +static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
>> +	enum ov5640_frame_rate frame_rate;
>> +	u32 tgt_fps;	/* target frames per secound */
>> +	int ret = 0;
> No need to initialize ret to 0.

Fixed.

>> +	else {
>> +		v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
>> +			 tgt_fps);
> Don't print an error message that is userspace-triggerable, we have enough
> ways for applications to flood the kernel log already :-)

Removed.

>> +
>> +static int ov5640_get_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	format->format = sensor->fmt;
> You need to handle the TRY format here. You can have a look at the mt9p031
> driver for an example on how to do so.

Right, that was caught and fixed earlier.

>> +
>> +static int ov5640_set_fmt(struct v4l2_subdev *sd,
>> +			  struct v4l2_subdev_pad_config *cfg,
>> +			  struct v4l2_subdev_format *format)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode new_mode;
>> +	int ret;
>> +
>> +	if (format->pad != 0)
>> +		return -EINVAL;
>> +
>> +	if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		ret = ov5640_try_fmt_internal(sd, &format->format, NULL);
>> +		if (ret)
>> +			return ret;
> You can move this code above the test to avoid the duplicated call below.

Fixed.

>> +		cfg->try_fmt = format->format;
> Please use the v4l2_subdev_get_try_format() function to get the try format,
> don't dereference cfg directly.

Done.

>> +
>> +static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (value) {
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
>> +		OV5640_WRITE_REG16(sensor, 0x5581, value);
>> +	} else
>> +		OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);
> According to the kernel coding style, you need curly braces around the else
> branch if you use them around the if branch.

Fixed everywhere.

>
>
>> +
>> +#if 0
> No compiled-out code please.

Removed ov5640_set_green_balance().

>> +static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->awb_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
>> +	OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
>> +	return 0;
>> +}
>> +#endif
>> +
>> <snip>
>> +
>> +static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
> You can create a cluster with the autogain and manual gain controls
> (v4l2_ctrl_auto_cluster) to have the control framework disabling the manual
> control automatically when autogain is enabled.

Done. Also made white-balance and exposure auto clusters as well.
And in the process, got rid of the control table as Hans had suggested
earlier.


>> +	OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);
>> +	return 0;
>> +}
>> +
>> +static int ov5640_get_gain(struct ov5640_dev *sensor)
>> +{
>> +	u16 gain;
>> +	int ret;
>> +
>> +	if (sensor->agc_on)
>> +		return -EINVAL;
>> +
>> +	OV5640_READ_REG16(sensor, 0x350a, &gain);
>> +
>> +	return gain & 0x3ff;
>> +}
>> +
>> +#if 0
>> +static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
>> +{
>> +	int ret;
>> +
>> +	OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
>> +	return 0;
>> +}
>> +#endif
> You can use a control to expose the test pattern function, it's quite useful.

Done, as a menu control for color bars.

>> +static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
>> +{
>> +	struct ov5640_control *ret = NULL;
>> +	int i;
> i is never negative, it should be unsigned int.

Fixed.


>> +
>> +static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
>> +	int i;
> i is never negative, it should be unsigned int.

restore_ctrls is gone from earlier fixes.

>> +
>> +static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
>> +{
>> +	struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
>> +	struct ov5640_control *c;
>> +	int ret = 0;
>> +	int i;
>> +
>> +	c = ov5640_get_ctrl(ctrl->id, &i);
> You can inline the function call here as it's not used anywhere else.

Done.

>> +
>> +static int ov5640_init_controls(struct ov5640_dev *sensor)
>> +{
>> +	struct ov5640_control *c;
> I would name this ctrl or control, c can be a bit confusing. You can also
> declare it inside the loop.

Done.

>> +	int i;
> i can never be negative, you can make it an unsigned int.

Done.

>> +
>> +	v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
>> +
>> +	for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
>> +		c = &ov5640_ctrls[i];
>> +
>> +		v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
>> +				  c->ctrl.id, c->ctrl.minimum, c-
>> ctrl.maximum,
>> +				  c->ctrl.step, c->ctrl.default_value);
>> +	}
>> +
>> +	sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
>> +	if (sensor->ctrl_hdl.error) {
>> +		int err = sensor->ctrl_hdl.error;
>> +
>> +		v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +
>> +		v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
> I'm not sure this brings much value.

Removed.

>
>> +
>> +static int ov5640_enum_frame_interval(
>> +	struct v4l2_subdev *sd,
>> +	struct v4l2_subdev_pad_config *cfg,
>> +	struct v4l2_subdev_frame_interval_enum *fie)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +	enum ov5640_mode mode;
>> +
>> +	if (fie->pad != 0)
>> +		return -EINVAL;
>> +	if (fie->index < 0 || fie->index >= ov5640_num_framerates)
>> +		return -EINVAL;
>> +
>> +	if (fie->width == 0 || fie->height == 0)
>> +		return -EINVAL;
>> +
>> +	mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
> You should find an exact mode here, not the nearest one. If with and height
> don't match, return -EINVAL. That will replace with above width == 0 and
> height == 0 test.

Fixed, I modified ov5640_find_mode() (renamed from 
ov5640_find_nearest_mode())
to take a "nearest" boolean, which is passed as false here.

>> +
>> +	dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
>> +		fie->width, fie->height, fie->index, fie-
>> interval.denominator);
> I'm not sure this is very useful, you can use ftrace if you want to trace
> function calls.

Removed.
>> +	return 0;
>> +}
>> +
>> +static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	*status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;
>> +
>> +	return 0;
>> +}
>> +
>> +static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
>> +			    u32 output, u32 config)
>> +{
>> +	return (input != 0) ? -EINVAL : 0;
>> +}
> The g_input_status and s_routing subdev operations are not mandatory, you
> don't have to implement them as the sensor doesn't have multiple inputs.

Removed.

>> +static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> You can use ftrace to trace function calls, there's no need to add debugging
> statements that duplicate the functionality.

Removed.

>> +
>> +static struct v4l2_subdev_ops ov5640_subdev_ops = {
>> +	.core = &ov5640_core_ops,
>> +	.video = &ov5640_video_ops,
>> +	.pad = &ov5640_pad_ops,
>> +};
> All these structures should be static const.

Fixed.

>> +
>> +static void ov5640_power(struct ov5640_dev *sensor, bool enable)
>> +{
>> +	gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
>> +}
>> +
>> +static void ov5640_reset(struct ov5640_dev *sensor)
>> +{
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +
>> +	/* camera power cycle */
>> +	ov5640_power(sensor, false);
>> +	usleep_range(5000, 10000);
>> +	ov5640_power(sensor, true);
>> +	usleep_range(5000, 10000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 1);
>> +	usleep_range(1000, 2000);
>> +
>> +	gpiod_set_value(sensor->reset_gpio, 0);
>> +	usleep_range(5000, 10000);
>> +}
>> +
>> +static void ov5640_get_regulators(struct ov5640_dev *sensor)
>> +{
>> +	sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
>> +	if (!IS_ERR(sensor->io_regulator)) {
>> +		regulator_set_voltage(sensor->io_regulator,
>> +				      OV5640_VOLTAGE_DIGITAL_IO,
>> +				      OV5640_VOLTAGE_DIGITAL_IO);
>> +	} else {
>> +		dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
>> +			__func__);
>> +		sensor->io_regulator = NULL;
> How about making the power supplies mandatory in DT instead ? They are
> mandatory after all, if they're not controllable DT should just declare fixed
> supplies.

I guess that makes sense. They are now mandatory.

>> +
>> +static int ov5640_probe(struct i2c_client *client,
>> +			const struct i2c_device_id *id)
>> +{
>> +	struct device *dev = &client->dev;
>> +	struct device_node *endpoint;
>> +	struct ov5640_dev *sensor;
>> +	int i, xclk, ret;
> i and xclk are never negative, you can make them unsigned int.

This was removed from earlier cleanups.

>> +
>> +	sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
> Please use sizeof(*variable) instead of sizeof(type).

Done.

> devm_kzalloc() doesn't play nicely with dynamic removal of devices. We're in
> the process of fixing related race conditions in the media subsystem. In order
> not to make the problem worse, please use kzalloc() instead.
>
>> +	if (!sensor)
>> +		return -ENOMEM;
>> +
>> +	sensor->i2c_client = client;
>> +	sensor->dev = dev;
> Do you really need to store both i2c_client and dev, given that the latter is
> just &client->dev ?

It was only for convenience, but sensor->dev was now only used in a
couple places, so sensor->dev removed.

>> +	sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
>> +	sensor->fmt.width = 640;
>> +	sensor->fmt.height = 480;
>> +	sensor->fmt.field = V4L2_FIELD_NONE;
>> +	sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
>> +					   V4L2_CAP_TIMEPERFRAME;
> Please fix the indentation.

Fixed.

>> +
>> +	v4l2_of_parse_endpoint(endpoint, &sensor->ep);
>> +	if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
>> +		dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
>> +		return -EINVAL;
> You're leaking endpoint here. You should move the of_node_put() call right
> after the v4l2_of_parse_endpoint() call.

oops, Fixed.

>> +
>> +	/* get system clock (xclk) frequency */
>> +	ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
> Instead of adding a custom DT property for this, use assigned-clock-rates. You
> won't need to set it manually in the driver, and can verify its frequency with
> clk_get_rate().

This was done in earlier fixups.

>> +	if (!ret) {
>> +		if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
> Are your register tables above independent of the clock frequency ? You should
> ideally compute register values at runtime instead of hardcoding them, but
> given the lack of information from Omnivision I understand this isn't
> possible. You thus need to be stricter here and reject any value other than
> the nominal frequency.

I went back and looked at the original driver from Freescale that these
register tables are based on, and that driver does allow a range between
6 and 24 MHz. I've updated the min/max for xclk to that range. I agree
though that this will need some experimentation to verify these xclk ranges
actually work.


>> +
>> +	/* get system clock (xclk) */
>> +	sensor->xclk = devm_clk_get(dev, "xclk");
>> +	if (!IS_ERR(sensor->xclk)) {
>> +		if (!sensor->xclk_freq) {
>> +			dev_err(dev, "xclk requires xclk frequency!\n");
>> +			return -EINVAL;
>> +		}
>> +		clk_set_rate(sensor->xclk, sensor->xclk_freq);
>> +	} else {
>> +		/* assume system clock enabled by default */
>> +		sensor->xclk = NULL;
> Please don't. The clock should be mandatory.

This was done in earlier fixups.


>> +	}
>> +
>> +	/* request power down pin */
>> +	sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
> Are the GPIOs mandatory or optional ? Can a system tie some of them to ground
> or a voltage rail, or do they need to always be manually controllable ?

Yes, they could be tied to ground or a rail, and are not required. I
made both reset and pwdn gpios optional.

>> +
>> +	ret = ov5640_s_power(&sensor->sd, 1);
>> +	if (ret)
>> +		goto entity_cleanup;
>> +	ret = ov5640_init_controls(sensor);
>> +	if (ret)
>> +		goto power_off;
>> +
>> +	ret = ov5640_s_power(&sensor->sd, 0);
>> +	if (ret)
>> +		goto free_ctrls;
> Writing the controls here is pointless, as powering the chip down will lose
> all the values. You shouldn't call v4l2_ctrl_handler_setup() in
> ov5640_init_controls(), and you can then remove the ov5640_s_power() calls
> here.

Good idea, done.

>> +
>> +free_ctrls:
>> +	v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
>> +power_off:
>> +	ov5640_s_power(&sensor->sd, 0);
>> +entity_cleanup:
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	ov5640_regulators_off(sensor);
> Won't ov5640_s_power(0) already disable the regulators ?

Not applicable anymore, as there's no need to enable power in probe.

>> +	return ret;
>> +}
>> +
>> +/*!
>> + * ov5640 I2C detach function
>> + *
>> + * @param client            struct i2c_client *
>> + * @return  Error code indicating success or failure
>> + */
> That's not the kerneldoc comment style. Given that this is the only documented
> function, and that the comment is completely useless, you can just drop it.

yuck, old FSL-isms, removed.

>> +static int ov5640_remove(struct i2c_client *client)
>> +{
>> +	struct v4l2_subdev *sd = i2c_get_clientdata(client);
>> +	struct ov5640_dev *sensor = to_ov5640_dev(sd);
>> +
>> +	ov5640_regulators_off(sensor);
>> +
>> +	v4l2_async_unregister_subdev(&sensor->sd);
>> +	media_entity_cleanup(&sensor->sd.entity);
>> +	v4l2_device_unregister_subdev(sd);
> This function is called by v4l2_async_unregister_subdev(), there's no need to
> duplicate it.

Removed.

>> +MODULE_AUTHOR("Freescale Semiconductor, Inc.");
> MODULE_AUTHOR isn't a synonym for copyright ownership. I don't think you
> should add Freescale as an author. If you know who wrote the original code you
> can list that developer explicitly.

I have no clue who the original author was, I just removed
MODULE_AUTHOR(s).

>> +MODULE_AUTHOR("Steve Longerbeam<steve_longerbeam@mentor.com>");
>> +MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
>> +MODULE_LICENSE("GPL");
>> +MODULE_VERSION("1.0");
> Version numbers are never updated. I wouldn't bother adding one.

Removed.

Steve

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-13  9:20               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-13  9:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Russell King - ARM Linux, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Thu, 2017-02-09 at 15:51 -0800, Steve Longerbeam wrote:
> 
> On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
> >
> >
> > On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
> >> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
> >>>> Actually, this exact function already exists as 
> >>>> dw_mipi_dsi_phy_write in
> >>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> >>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
> >>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
> >>> It's clear from that driver that there probably needs to be a fuller
> >>> treatment of the D-PHY programming here, but I don't know where
> >>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
> >>> in imxcsi2_reset() was also pulled from FSL, and that's all I really 
> >>> have
> >>> to go on for the D-PHY programming. I assume the D-PHY is also a
> >>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
> >>> cover it.
> >> Why exactly?  What problems are you seeing that would necessitate a
> >> more detailed programming of the D-PHY?  From my testing, I can wind
> >> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
> >> per lane with your existing code without any issues.
> >
> > That's good to hear.
> >
> > Just from my experience with struggles to get the CSI2 receiver
> > up and running with an active clock lane, and my suspicions that
> > some of that could be due to my lack of understanding of the D-PHY
> > programming.
> 
> But I should add that after a re-org of the sequence, it looks more stable
> now. Tested on both the SabreSD and SabreLite with the OV5640.

It seems the OV5640 driver never puts its the CSI-2 lanes into stop
state while not streaming. With the recent s_stream reordering,
streaming from TC358743 does not work anymore, since imx6-mipi-csi2
s_stream is called before tc358743 s_stream, while all lanes are still
in stop state. Then it waits for the clock lane to become active and
fails. I have applied the following patch to revert the reordering
locally to get it to work again.

The initialization order, as Russell pointed out, should be:

1. reset the D-PHY.
2. place MIPI sensor in LP-11 state
3. perform D-PHY initialisation
4. configure CSI2 lanes and de-assert resets and shutdown signals

So csi2_s_stream should wait for stop state on all lanes (the result of
2.) before dphy_init (3.), not wait for active clock afterwards. That
should happen only after sensor_s_stream was called. So unless we are
allowed to reorder steps 1. and 2., we might need the prepare_stream
callback after all.

More specifically, the chapter 40.3.1 "Startup Sequence" in the i.MX6DQ
reference manual states:

1. Deassert presetn signal (global reset)
   - This is probably the APB global reset, so not something we have to
     care about.
2. Configure MIPI Camera Sensor to put all Tx lanes in PL-11 state
3. D-PHY initialization (write 0x14 to address 0x44)
4. CSI2 Controller programming
   - Set N_LANES
   - Deassert PHY_SHUTDOWNZ
   - Deassert PHY_RSTZ
   - Deassert CSI2_RESETN
5. Read the PHY status register (PHY_STATE) to confirm that all data and
   clock lanes of the D-PHY are in Stop State
6. Configure the MIPI Camera Sensor to start transmitting a clock on the
   D-PHY clock lane
7. CSI2 Controller programming - Read the PHY status register
   (PHY_STATE) to confirm that the D-PHY is receiving a clock on the
   D-PHY clock lane

2. can be done in sensor s_power (which tc358743 currently does) only if
the sensor can still be configured in step 6.
Also, 3./4./5. are csi2 code, 6. is sensor code, and 7. is csi2 code
again. For reliable startup we need csi2 receiver code to be called on
both sides of the sensor enabling its clock lane.

----------8<----------
>From f12758caa3e1d05980cd2ac07587d125449fc860 Mon Sep 17 00:00:00 2001
From: Philipp Zabel <p.zabel@pengutronix.de>
Date: Mon, 13 Feb 2017 09:28:27 +0100
Subject: [PATCH] media: imx: revert streamon sequence change

Without this patch, starting streaming from a TC358743 MIPI CSI-2 source
fails with the following error messages:

    imx6-mipi-csi2: clock lane timeout, phy_state = 0x000006f0
    ipu1_csi0: pipeline_set_stream failed with -110

The phy state above has the stopstateclk, rxulpsclknot, and
stopstatedata[3:0] bits set: at this point all lanes are in stop state,
no lane is in ultra low power state or active.
This is no suprise, since tc358743 s_stream(sd, 1) has not been called
due to the recently changed ordering. The imx6-mipi-csi2 s_stream does
wait for the clock lane to become active, so csi2_s_stream must happen
after tc358743_s_stream.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/staging/media/imx/imx-media-utils.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 81eabcf76a66f..495ccefa3cd2a 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -782,8 +782,8 @@ static const u32 stream_on_seq[] = {
 	IMX_MEDIA_GRP_ID_IC_PRPENC,
 	IMX_MEDIA_GRP_ID_IC_PRP,
 	IMX_MEDIA_GRP_ID_VDIC,
-	IMX_MEDIA_GRP_ID_CSI2,
 	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
 	IMX_MEDIA_GRP_ID_VIDMUX,
 	IMX_MEDIA_GRP_ID_CSI,
 };
@@ -795,8 +795,8 @@ static const u32 stream_off_seq[] = {
 	IMX_MEDIA_GRP_ID_VDIC,
 	IMX_MEDIA_GRP_ID_CSI,
 	IMX_MEDIA_GRP_ID_VIDMUX,
-	IMX_MEDIA_GRP_ID_SENSOR,
 	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
 };
 
 #define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
-- 
2.11.0
---------->8----------

regards
Philipp

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-13  9:20               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-13  9:20 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Russell King - ARM Linux, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve

Hi Steve,

On Thu, 2017-02-09 at 15:51 -0800, Steve Longerbeam wrote:
> 
> On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
> >
> >
> > On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
> >> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
> >>>> Actually, this exact function already exists as 
> >>>> dw_mipi_dsi_phy_write in
> >>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> >>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
> >>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
> >>> It's clear from that driver that there probably needs to be a fuller
> >>> treatment of the D-PHY programming here, but I don't know where
> >>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
> >>> in imxcsi2_reset() was also pulled from FSL, and that's all I really 
> >>> have
> >>> to go on for the D-PHY programming. I assume the D-PHY is also a
> >>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
> >>> cover it.
> >> Why exactly?  What problems are you seeing that would necessitate a
> >> more detailed programming of the D-PHY?  From my testing, I can wind
> >> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
> >> per lane with your existing code without any issues.
> >
> > That's good to hear.
> >
> > Just from my experience with struggles to get the CSI2 receiver
> > up and running with an active clock lane, and my suspicions that
> > some of that could be due to my lack of understanding of the D-PHY
> > programming.
> 
> But I should add that after a re-org of the sequence, it looks more stable
> now. Tested on both the SabreSD and SabreLite with the OV5640.

It seems the OV5640 driver never puts its the CSI-2 lanes into stop
state while not streaming. With the recent s_stream reordering,
streaming from TC358743 does not work anymore, since imx6-mipi-csi2
s_stream is called before tc358743 s_stream, while all lanes are still
in stop state. Then it waits for the clock lane to become active and
fails. I have applied the following patch to revert the reordering
locally to get it to work again.

The initialization order, as Russell pointed out, should be:

1. reset the D-PHY.
2. place MIPI sensor in LP-11 state
3. perform D-PHY initialisation
4. configure CSI2 lanes and de-assert resets and shutdown signals

So csi2_s_stream should wait for stop state on all lanes (the result of
2.) before dphy_init (3.), not wait for active clock afterwards. That
should happen only after sensor_s_stream was called. So unless we are
allowed to reorder steps 1. and 2., we might need the prepare_stream
callback after all.

More specifically, the chapter 40.3.1 "Startup Sequence" in the i.MX6DQ
reference manual states:

1. Deassert presetn signal (global reset)
   - This is probably the APB global reset, so not something we have to
     care about.
2. Configure MIPI Camera Sensor to put all Tx lanes in PL-11 state
3. D-PHY initialization (write 0x14 to address 0x44)
4. CSI2 Controller programming
   - Set N_LANES
   - Deassert PHY_SHUTDOWNZ
   - Deassert PHY_RSTZ
   - Deassert CSI2_RESETN
5. Read the PHY status register (PHY_STATE) to confirm that all data and
   clock lanes of the D-PHY are in Stop State
6. Configure the MIPI Camera Sensor to start transmitting a clock on the
   D-PHY clock lane
7. CSI2 Controller programming - Read the PHY status register
   (PHY_STATE) to confirm that the D-PHY is receiving a clock on the
   D-PHY clock lane

2. can be done in sensor s_power (which tc358743 currently does) only if
the sensor can still be configured in step 6.
Also, 3./4./5. are csi2 code, 6. is sensor code, and 7. is csi2 code
again. For reliable startup we need csi2 receiver code to be called on
both sides of the sensor enabling its clock lane.

----------8<----------
>From f12758caa3e1d05980cd2ac07587d125449fc860 Mon Sep 17 00:00:00 2001
From: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
Date: Mon, 13 Feb 2017 09:28:27 +0100
Subject: [PATCH] media: imx: revert streamon sequence change

Without this patch, starting streaming from a TC358743 MIPI CSI-2 source
fails with the following error messages:

    imx6-mipi-csi2: clock lane timeout, phy_state = 0x000006f0
    ipu1_csi0: pipeline_set_stream failed with -110

The phy state above has the stopstateclk, rxulpsclknot, and
stopstatedata[3:0] bits set: at this point all lanes are in stop state,
no lane is in ultra low power state or active.
This is no suprise, since tc358743 s_stream(sd, 1) has not been called
due to the recently changed ordering. The imx6-mipi-csi2 s_stream does
wait for the clock lane to become active, so csi2_s_stream must happen
after tc358743_s_stream.

Signed-off-by: Philipp Zabel <p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
---
 drivers/staging/media/imx/imx-media-utils.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 81eabcf76a66f..495ccefa3cd2a 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -782,8 +782,8 @@ static const u32 stream_on_seq[] = {
 	IMX_MEDIA_GRP_ID_IC_PRPENC,
 	IMX_MEDIA_GRP_ID_IC_PRP,
 	IMX_MEDIA_GRP_ID_VDIC,
-	IMX_MEDIA_GRP_ID_CSI2,
 	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
 	IMX_MEDIA_GRP_ID_VIDMUX,
 	IMX_MEDIA_GRP_ID_CSI,
 };
@@ -795,8 +795,8 @@ static const u32 stream_off_seq[] = {
 	IMX_MEDIA_GRP_ID_VDIC,
 	IMX_MEDIA_GRP_ID_CSI,
 	IMX_MEDIA_GRP_ID_VIDMUX,
-	IMX_MEDIA_GRP_ID_SENSOR,
 	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
 };
 
 #define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
-- 
2.11.0
---------->8----------

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-13  9:20               ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-13  9:20 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Thu, 2017-02-09 at 15:51 -0800, Steve Longerbeam wrote:
> 
> On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
> >
> >
> > On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
> >> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
> >>>> Actually, this exact function already exists as 
> >>>> dw_mipi_dsi_phy_write in
> >>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
> >>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
> >>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
> >>> It's clear from that driver that there probably needs to be a fuller
> >>> treatment of the D-PHY programming here, but I don't know where
> >>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
> >>> in imxcsi2_reset() was also pulled from FSL, and that's all I really 
> >>> have
> >>> to go on for the D-PHY programming. I assume the D-PHY is also a
> >>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
> >>> cover it.
> >> Why exactly?  What problems are you seeing that would necessitate a
> >> more detailed programming of the D-PHY?  From my testing, I can wind
> >> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
> >> per lane with your existing code without any issues.
> >
> > That's good to hear.
> >
> > Just from my experience with struggles to get the CSI2 receiver
> > up and running with an active clock lane, and my suspicions that
> > some of that could be due to my lack of understanding of the D-PHY
> > programming.
> 
> But I should add that after a re-org of the sequence, it looks more stable
> now. Tested on both the SabreSD and SabreLite with the OV5640.

It seems the OV5640 driver never puts its the CSI-2 lanes into stop
state while not streaming. With the recent s_stream reordering,
streaming from TC358743 does not work anymore, since imx6-mipi-csi2
s_stream is called before tc358743 s_stream, while all lanes are still
in stop state. Then it waits for the clock lane to become active and
fails. I have applied the following patch to revert the reordering
locally to get it to work again.

The initialization order, as Russell pointed out, should be:

1. reset the D-PHY.
2. place MIPI sensor in LP-11 state
3. perform D-PHY initialisation
4. configure CSI2 lanes and de-assert resets and shutdown signals

So csi2_s_stream should wait for stop state on all lanes (the result of
2.) before dphy_init (3.), not wait for active clock afterwards. That
should happen only after sensor_s_stream was called. So unless we are
allowed to reorder steps 1. and 2., we might need the prepare_stream
callback after all.

More specifically, the chapter 40.3.1 "Startup Sequence" in the i.MX6DQ
reference manual states:

1. Deassert presetn signal (global reset)
   - This is probably the APB global reset, so not something we have to
     care about.
2. Configure MIPI Camera Sensor to put all Tx lanes in PL-11 state
3. D-PHY initialization (write 0x14 to address 0x44)
4. CSI2 Controller programming
   - Set N_LANES
   - Deassert PHY_SHUTDOWNZ
   - Deassert PHY_RSTZ
   - Deassert CSI2_RESETN
5. Read the PHY status register (PHY_STATE) to confirm that all data and
   clock lanes of the D-PHY are in Stop State
6. Configure the MIPI Camera Sensor to start transmitting a clock on the
   D-PHY clock lane
7. CSI2 Controller programming - Read the PHY status register
   (PHY_STATE) to confirm that the D-PHY is receiving a clock on the
   D-PHY clock lane

2. can be done in sensor s_power (which tc358743 currently does) only if
the sensor can still be configured in step 6.
Also, 3./4./5. are csi2 code, 6. is sensor code, and 7. is csi2 code
again. For reliable startup we need csi2 receiver code to be called on
both sides of the sensor enabling its clock lane.

----------8<----------
>From f12758caa3e1d05980cd2ac07587d125449fc860 Mon Sep 17 00:00:00 2001
From: Philipp Zabel <p.zabel@pengutronix.de>
Date: Mon, 13 Feb 2017 09:28:27 +0100
Subject: [PATCH] media: imx: revert streamon sequence change

Without this patch, starting streaming from a TC358743 MIPI CSI-2 source
fails with the following error messages:

    imx6-mipi-csi2: clock lane timeout, phy_state = 0x000006f0
    ipu1_csi0: pipeline_set_stream failed with -110

The phy state above has the stopstateclk, rxulpsclknot, and
stopstatedata[3:0] bits set: at this point all lanes are in stop state,
no lane is in ultra low power state or active.
This is no suprise, since tc358743 s_stream(sd, 1) has not been called
due to the recently changed ordering. The imx6-mipi-csi2 s_stream does
wait for the clock lane to become active, so csi2_s_stream must happen
after tc358743_s_stream.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/staging/media/imx/imx-media-utils.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 81eabcf76a66f..495ccefa3cd2a 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -782,8 +782,8 @@ static const u32 stream_on_seq[] = {
 	IMX_MEDIA_GRP_ID_IC_PRPENC,
 	IMX_MEDIA_GRP_ID_IC_PRP,
 	IMX_MEDIA_GRP_ID_VDIC,
-	IMX_MEDIA_GRP_ID_CSI2,
 	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
 	IMX_MEDIA_GRP_ID_VIDMUX,
 	IMX_MEDIA_GRP_ID_CSI,
 };
@@ -795,8 +795,8 @@ static const u32 stream_off_seq[] = {
 	IMX_MEDIA_GRP_ID_VDIC,
 	IMX_MEDIA_GRP_ID_CSI,
 	IMX_MEDIA_GRP_ID_VIDMUX,
-	IMX_MEDIA_GRP_ID_SENSOR,
 	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
 };
 
 #define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
-- 
2.11.0
---------->8----------

regards
Philipp

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
  2017-02-13  9:20               ` Philipp Zabel
  (?)
  (?)
@ 2017-02-13 23:19               ` Steve Longerbeam
  -1 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-13 23:19 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Russell King - ARM Linux, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

[-- Attachment #1: Type: text/plain, Size: 7436 bytes --]

Hi Philipp,


On 02/13/2017 01:20 AM, Philipp Zabel wrote:
> Hi Steve,
>
> On Thu, 2017-02-09 at 15:51 -0800, Steve Longerbeam wrote:
>> On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
>>> On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
>>>> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>>>>> Actually, this exact function already exists as
>>>>>> dw_mipi_dsi_phy_write in
>>>>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>>>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>>>>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>>>>> It's clear from that driver that there probably needs to be a fuller
>>>>> treatment of the D-PHY programming here, but I don't know where
>>>>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>>>>> in imxcsi2_reset() was also pulled from FSL, and that's all I really
>>>>> have
>>>>> to go on for the D-PHY programming. I assume the D-PHY is also a
>>>>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>>>>> cover it.
>>>> Why exactly?  What problems are you seeing that would necessitate a
>>>> more detailed programming of the D-PHY?  From my testing, I can wind
>>>> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
>>>> per lane with your existing code without any issues.
>>> That's good to hear.
>>>
>>> Just from my experience with struggles to get the CSI2 receiver
>>> up and running with an active clock lane, and my suspicions that
>>> some of that could be due to my lack of understanding of the D-PHY
>>> programming.
>> But I should add that after a re-org of the sequence, it looks more stable
>> now. Tested on both the SabreSD and SabreLite with the OV5640.
> It seems the OV5640 driver never puts its the CSI-2 lanes into stop
> state while not streaming.

Yes I found that as well.

But good news, I finally managed to coax the OV5640's clock lane
into LP-11 state! It was accomplished by setting bit 5 in OV5640 register
0x4800. The datasheet describes this bit as "Gate clock lane when no
packet to transmit". But I may have also got this to work with the 
additional
write 1 to bits 4-6 in register 0x3019 ("MIPI CLK/data lane state in sleep
mode" - setting 1 to these bits selects LP-11 for sleep mode).

So I am now finally able to call csi2_dphy_wait_stopstate() in
csi2_s_stream(ON).

So for the TC35874, you shouldn't see a timeout in csi2_s_stream(ON)
any longer.

I have updated both ov5640.c and imx6-mipi-csi2.c in the wip branch.
Can you try again? I have not applied your patch below, because I
don't think that is needed anymore.

And speaking of the TC35874, I received this module for the SabreLite
from Boundary Devices (thanks BD!). So I can finally help you with
debug/test. Are there any patches you need to send to me (against
wip branch) for this support, or can I use what exists now in media
tree? Also any scripts or help with running.


>   With the recent s_stream reordering,
> streaming from TC358743 does not work anymore, since imx6-mipi-csi2
> s_stream is called before tc358743 s_stream, while all lanes are still
> in stop state. Then it waits for the clock lane to become active and
> fails. I have applied the following patch to revert the reordering
> locally to get it to work again.
>
> The initialization order, as Russell pointed out, should be:
>
> 1. reset the D-PHY.
> 2. place MIPI sensor in LP-11 state
> 3. perform D-PHY initialisation
> 4. configure CSI2 lanes and de-assert resets and shutdown signals
>
> So csi2_s_stream should wait for stop state on all lanes (the result of
> 2.) before dphy_init (3.), not wait for active clock afterwards. That
> should happen only after sensor_s_stream was called. So unless we are
> allowed to reorder steps 1. and 2., we might need the prepare_stream
> callback after all.

Agreed!

See my new FIXME comment in imx6-mipi-csi2.c.

I agree we might need a new subdev op .prepare_stream(). This
op would be implemented by imx6-mipi-csi2.c, and would carry
out steps 3, 4, 5 (instead of currently by csi2_s_stream()). Then
step 6 would finally become available as csi2_s_stream().

And then we must re-order stream on to start sensor first, then
csi2, as in your patch below.

Steve

>
> More specifically, the chapter 40.3.1 "Startup Sequence" in the i.MX6DQ
> reference manual states:
>
> 1. Deassert presetn signal (global reset)
>     - This is probably the APB global reset, so not something we have to
>       care about.
> 2. Configure MIPI Camera Sensor to put all Tx lanes in PL-11 state
> 3. D-PHY initialization (write 0x14 to address 0x44)
> 4. CSI2 Controller programming
>     - Set N_LANES
>     - Deassert PHY_SHUTDOWNZ
>     - Deassert PHY_RSTZ
>     - Deassert CSI2_RESETN
> 5. Read the PHY status register (PHY_STATE) to confirm that all data and
>     clock lanes of the D-PHY are in Stop State
> 6. Configure the MIPI Camera Sensor to start transmitting a clock on the
>     D-PHY clock lane
> 7. CSI2 Controller programming - Read the PHY status register
>     (PHY_STATE) to confirm that the D-PHY is receiving a clock on the
>     D-PHY clock lane
>
> 2. can be done in sensor s_power (which tc358743 currently does) only if
> the sensor can still be configured in step 6.
> Also, 3./4./5. are csi2 code, 6. is sensor code, and 7. is csi2 code
> again. For reliable startup we need csi2 receiver code to be called on
> both sides of the sensor enabling its clock lane.
> ----------8<----------
>  From f12758caa3e1d05980cd2ac07587d125449fc860 Mon Sep 17 00:00:00 2001
> From: Philipp Zabel<p.zabel@pengutronix.de>
> Date: Mon, 13 Feb 2017 09:28:27 +0100
> Subject: [PATCH] media: imx: revert streamon sequence change
>
> Without this patch, starting streaming from a TC358743 MIPI CSI-2 source
> fails with the following error messages:
>
>      imx6-mipi-csi2: clock lane timeout, phy_state = 0x000006f0
>      ipu1_csi0: pipeline_set_stream failed with -110
>
> The phy state above has the stopstateclk, rxulpsclknot, and
> stopstatedata[3:0] bits set: at this point all lanes are in stop state,
> no lane is in ultra low power state or active.
> This is no suprise, since tc358743 s_stream(sd, 1) has not been called
> due to the recently changed ordering. The imx6-mipi-csi2 s_stream does
> wait for the clock lane to become active, so csi2_s_stream must happen
> after tc358743_s_stream.
>
> Signed-off-by: Philipp Zabel<p.zabel@pengutronix.de>
> ---
>   drivers/staging/media/imx/imx-media-utils.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 81eabcf76a66f..495ccefa3cd2a 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -782,8 +782,8 @@ static const u32 stream_on_seq[] = {
>   	IMX_MEDIA_GRP_ID_IC_PRPENC,
>   	IMX_MEDIA_GRP_ID_IC_PRP,
>   	IMX_MEDIA_GRP_ID_VDIC,
> -	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
>   	IMX_MEDIA_GRP_ID_CSI,
>   };
> @@ -795,8 +795,8 @@ static const u32 stream_off_seq[] = {
>   	IMX_MEDIA_GRP_ID_VDIC,
>   	IMX_MEDIA_GRP_ID_CSI,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
> -	IMX_MEDIA_GRP_ID_SENSOR,
>   	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
>   };
>   
>   #define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)


[-- Attachment #2: Type: text/html, Size: 8934 bytes --]

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-13 23:20                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-13 23:20 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Russell King - ARM Linux, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

(#!##* Thunderbird! resending text only)

Hi Philipp,


On 02/13/2017 01:20 AM, Philipp Zabel wrote:
> Hi Steve,
>
> On Thu, 2017-02-09 at 15:51 -0800, Steve Longerbeam wrote:
>> On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
>>> On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
>>>> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>>>>> Actually, this exact function already exists as
>>>>>> dw_mipi_dsi_phy_write in
>>>>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>>>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>>>>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>>>>> It's clear from that driver that there probably needs to be a fuller
>>>>> treatment of the D-PHY programming here, but I don't know where
>>>>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>>>>> in imxcsi2_reset() was also pulled from FSL, and that's all I really
>>>>> have
>>>>> to go on for the D-PHY programming. I assume the D-PHY is also a
>>>>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>>>>> cover it.
>>>> Why exactly?  What problems are you seeing that would necessitate a
>>>> more detailed programming of the D-PHY?  From my testing, I can wind
>>>> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
>>>> per lane with your existing code without any issues.
>>> That's good to hear.
>>>
>>> Just from my experience with struggles to get the CSI2 receiver
>>> up and running with an active clock lane, and my suspicions that
>>> some of that could be due to my lack of understanding of the D-PHY
>>> programming.
>> But I should add that after a re-org of the sequence, it looks more stable
>> now. Tested on both the SabreSD and SabreLite with the OV5640.
> It seems the OV5640 driver never puts its the CSI-2 lanes into stop
> state while not streaming.

Yes I found that as well.

But good news, I finally managed to coax the OV5640's clock lane
into LP-11 state! It was accomplished by setting bit 5 in OV5640 register
0x4800. The datasheet describes this bit as "Gate clock lane when no
packet to transmit". But I may have also got this to work with the 
additional
write 1 to bits 4-6 in register 0x3019 ("MIPI CLK/data lane state in sleep
mode" - setting 1 to these bits selects LP-11 for sleep mode).

So I am now finally able to call csi2_dphy_wait_stopstate() in
csi2_s_stream(ON).

So for the TC35874, you shouldn't see a timeout in csi2_s_stream(ON)
any longer.

I have updated both ov5640.c and imx6-mipi-csi2.c in the wip branch.
Can you try again? I have not applied your patch below, because I
don't think that is needed anymore.

And speaking of the TC35874, I received this module for the SabreLite
from Boundary Devices (thanks BD!). So I can finally help you with
debug/test. Are there any patches you need to send to me (against
wip branch) for this support, or can I use what exists now in media
tree? Also any scripts or help with running.


>   With the recent s_stream reordering,
> streaming from TC358743 does not work anymore, since imx6-mipi-csi2
> s_stream is called before tc358743 s_stream, while all lanes are still
> in stop state. Then it waits for the clock lane to become active and
> fails. I have applied the following patch to revert the reordering
> locally to get it to work again.
>
> The initialization order, as Russell pointed out, should be:
>
> 1. reset the D-PHY.
> 2. place MIPI sensor in LP-11 state
> 3. perform D-PHY initialisation
> 4. configure CSI2 lanes and de-assert resets and shutdown signals
>
> So csi2_s_stream should wait for stop state on all lanes (the result of
> 2.) before dphy_init (3.), not wait for active clock afterwards. That
> should happen only after sensor_s_stream was called. So unless we are
> allowed to reorder steps 1. and 2., we might need the prepare_stream
> callback after all.

Agreed!

See my new FIXME comment in imx6-mipi-csi2.c.

I agree we might need a new subdev op .prepare_stream(). This
op would be implemented by imx6-mipi-csi2.c, and would carry
out steps 3, 4, 5 (instead of currently by csi2_s_stream()). Then
step 6 would finally become available as csi2_s_stream().

And then we must re-order stream on to start sensor first, then
csi2, as in your patch below.

Steve

> More specifically, the chapter 40.3.1 "Startup Sequence" in the i.MX6DQ
> reference manual states:
>
> 1. Deassert presetn signal (global reset)
>     - This is probably the APB global reset, so not something we have to
>       care about.
> 2. Configure MIPI Camera Sensor to put all Tx lanes in PL-11 state
> 3. D-PHY initialization (write 0x14 to address 0x44)
> 4. CSI2 Controller programming
>     - Set N_LANES
>     - Deassert PHY_SHUTDOWNZ
>     - Deassert PHY_RSTZ
>     - Deassert CSI2_RESETN
> 5. Read the PHY status register (PHY_STATE) to confirm that all data and
>     clock lanes of the D-PHY are in Stop State
> 6. Configure the MIPI Camera Sensor to start transmitting a clock on the
>     D-PHY clock lane
> 7. CSI2 Controller programming - Read the PHY status register
>     (PHY_STATE) to confirm that the D-PHY is receiving a clock on the
>     D-PHY clock lane
>
> 2. can be done in sensor s_power (which tc358743 currently does) only if
> the sensor can still be configured in step 6.
> Also, 3./4./5. are csi2 code, 6. is sensor code, and 7. is csi2 code
> again. For reliable startup we need csi2 receiver code to be called on
> both sides of the sensor enabling its clock lane.
> ----------8<----------
>  From f12758caa3e1d05980cd2ac07587d125449fc860 Mon Sep 17 00:00:00 2001
> From: Philipp Zabel<p.zabel@pengutronix.de>
> Date: Mon, 13 Feb 2017 09:28:27 +0100
> Subject: [PATCH] media: imx: revert streamon sequence change
>
> Without this patch, starting streaming from a TC358743 MIPI CSI-2 source
> fails with the following error messages:
>
>      imx6-mipi-csi2: clock lane timeout, phy_state = 0x000006f0
>      ipu1_csi0: pipeline_set_stream failed with -110
>
> The phy state above has the stopstateclk, rxulpsclknot, and
> stopstatedata[3:0] bits set: at this point all lanes are in stop state,
> no lane is in ultra low power state or active.
> This is no suprise, since tc358743 s_stream(sd, 1) has not been called
> due to the recently changed ordering. The imx6-mipi-csi2 s_stream does
> wait for the clock lane to become active, so csi2_s_stream must happen
> after tc358743_s_stream.
>
> Signed-off-by: Philipp Zabel<p.zabel@pengutronix.de>
> ---
>   drivers/staging/media/imx/imx-media-utils.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 81eabcf76a66f..495ccefa3cd2a 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -782,8 +782,8 @@ static const u32 stream_on_seq[] = {
>   	IMX_MEDIA_GRP_ID_IC_PRPENC,
>   	IMX_MEDIA_GRP_ID_IC_PRP,
>   	IMX_MEDIA_GRP_ID_VDIC,
> -	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
>   	IMX_MEDIA_GRP_ID_CSI,
>   };
> @@ -795,8 +795,8 @@ static const u32 stream_off_seq[] = {
>   	IMX_MEDIA_GRP_ID_VDIC,
>   	IMX_MEDIA_GRP_ID_CSI,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
> -	IMX_MEDIA_GRP_ID_SENSOR,
>   	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
>   };
>   
>   #define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-13 23:20                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-13 23:20 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: Russell King - ARM Linux, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve

(#!##* Thunderbird! resending text only)

Hi Philipp,


On 02/13/2017 01:20 AM, Philipp Zabel wrote:
> Hi Steve,
>
> On Thu, 2017-02-09 at 15:51 -0800, Steve Longerbeam wrote:
>> On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
>>> On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
>>>> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>>>>> Actually, this exact function already exists as
>>>>>> dw_mipi_dsi_phy_write in
>>>>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>>>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>>>>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>>>>> It's clear from that driver that there probably needs to be a fuller
>>>>> treatment of the D-PHY programming here, but I don't know where
>>>>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>>>>> in imxcsi2_reset() was also pulled from FSL, and that's all I really
>>>>> have
>>>>> to go on for the D-PHY programming. I assume the D-PHY is also a
>>>>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>>>>> cover it.
>>>> Why exactly?  What problems are you seeing that would necessitate a
>>>> more detailed programming of the D-PHY?  From my testing, I can wind
>>>> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
>>>> per lane with your existing code without any issues.
>>> That's good to hear.
>>>
>>> Just from my experience with struggles to get the CSI2 receiver
>>> up and running with an active clock lane, and my suspicions that
>>> some of that could be due to my lack of understanding of the D-PHY
>>> programming.
>> But I should add that after a re-org of the sequence, it looks more stable
>> now. Tested on both the SabreSD and SabreLite with the OV5640.
> It seems the OV5640 driver never puts its the CSI-2 lanes into stop
> state while not streaming.

Yes I found that as well.

But good news, I finally managed to coax the OV5640's clock lane
into LP-11 state! It was accomplished by setting bit 5 in OV5640 register
0x4800. The datasheet describes this bit as "Gate clock lane when no
packet to transmit". But I may have also got this to work with the 
additional
write 1 to bits 4-6 in register 0x3019 ("MIPI CLK/data lane state in sleep
mode" - setting 1 to these bits selects LP-11 for sleep mode).

So I am now finally able to call csi2_dphy_wait_stopstate() in
csi2_s_stream(ON).

So for the TC35874, you shouldn't see a timeout in csi2_s_stream(ON)
any longer.

I have updated both ov5640.c and imx6-mipi-csi2.c in the wip branch.
Can you try again? I have not applied your patch below, because I
don't think that is needed anymore.

And speaking of the TC35874, I received this module for the SabreLite
from Boundary Devices (thanks BD!). So I can finally help you with
debug/test. Are there any patches you need to send to me (against
wip branch) for this support, or can I use what exists now in media
tree? Also any scripts or help with running.


>   With the recent s_stream reordering,
> streaming from TC358743 does not work anymore, since imx6-mipi-csi2
> s_stream is called before tc358743 s_stream, while all lanes are still
> in stop state. Then it waits for the clock lane to become active and
> fails. I have applied the following patch to revert the reordering
> locally to get it to work again.
>
> The initialization order, as Russell pointed out, should be:
>
> 1. reset the D-PHY.
> 2. place MIPI sensor in LP-11 state
> 3. perform D-PHY initialisation
> 4. configure CSI2 lanes and de-assert resets and shutdown signals
>
> So csi2_s_stream should wait for stop state on all lanes (the result of
> 2.) before dphy_init (3.), not wait for active clock afterwards. That
> should happen only after sensor_s_stream was called. So unless we are
> allowed to reorder steps 1. and 2., we might need the prepare_stream
> callback after all.

Agreed!

See my new FIXME comment in imx6-mipi-csi2.c.

I agree we might need a new subdev op .prepare_stream(). This
op would be implemented by imx6-mipi-csi2.c, and would carry
out steps 3, 4, 5 (instead of currently by csi2_s_stream()). Then
step 6 would finally become available as csi2_s_stream().

And then we must re-order stream on to start sensor first, then
csi2, as in your patch below.

Steve

> More specifically, the chapter 40.3.1 "Startup Sequence" in the i.MX6DQ
> reference manual states:
>
> 1. Deassert presetn signal (global reset)
>     - This is probably the APB global reset, so not something we have to
>       care about.
> 2. Configure MIPI Camera Sensor to put all Tx lanes in PL-11 state
> 3. D-PHY initialization (write 0x14 to address 0x44)
> 4. CSI2 Controller programming
>     - Set N_LANES
>     - Deassert PHY_SHUTDOWNZ
>     - Deassert PHY_RSTZ
>     - Deassert CSI2_RESETN
> 5. Read the PHY status register (PHY_STATE) to confirm that all data and
>     clock lanes of the D-PHY are in Stop State
> 6. Configure the MIPI Camera Sensor to start transmitting a clock on the
>     D-PHY clock lane
> 7. CSI2 Controller programming - Read the PHY status register
>     (PHY_STATE) to confirm that the D-PHY is receiving a clock on the
>     D-PHY clock lane
>
> 2. can be done in sensor s_power (which tc358743 currently does) only if
> the sensor can still be configured in step 6.
> Also, 3./4./5. are csi2 code, 6. is sensor code, and 7. is csi2 code
> again. For reliable startup we need csi2 receiver code to be called on
> both sides of the sensor enabling its clock lane.
> ----------8<----------
>  From f12758caa3e1d05980cd2ac07587d125449fc860 Mon Sep 17 00:00:00 2001
> From: Philipp Zabel<p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> Date: Mon, 13 Feb 2017 09:28:27 +0100
> Subject: [PATCH] media: imx: revert streamon sequence change
>
> Without this patch, starting streaming from a TC358743 MIPI CSI-2 source
> fails with the following error messages:
>
>      imx6-mipi-csi2: clock lane timeout, phy_state = 0x000006f0
>      ipu1_csi0: pipeline_set_stream failed with -110
>
> The phy state above has the stopstateclk, rxulpsclknot, and
> stopstatedata[3:0] bits set: at this point all lanes are in stop state,
> no lane is in ultra low power state or active.
> This is no suprise, since tc358743 s_stream(sd, 1) has not been called
> due to the recently changed ordering. The imx6-mipi-csi2 s_stream does
> wait for the clock lane to become active, so csi2_s_stream must happen
> after tc358743_s_stream.
>
> Signed-off-by: Philipp Zabel<p.zabel-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
> ---
>   drivers/staging/media/imx/imx-media-utils.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 81eabcf76a66f..495ccefa3cd2a 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -782,8 +782,8 @@ static const u32 stream_on_seq[] = {
>   	IMX_MEDIA_GRP_ID_IC_PRPENC,
>   	IMX_MEDIA_GRP_ID_IC_PRP,
>   	IMX_MEDIA_GRP_ID_VDIC,
> -	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
>   	IMX_MEDIA_GRP_ID_CSI,
>   };
> @@ -795,8 +795,8 @@ static const u32 stream_off_seq[] = {
>   	IMX_MEDIA_GRP_ID_VDIC,
>   	IMX_MEDIA_GRP_ID_CSI,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
> -	IMX_MEDIA_GRP_ID_SENSOR,
>   	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
>   };
>   
>   #define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-13 23:20                 ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-13 23:20 UTC (permalink / raw)
  To: linux-arm-kernel

(#!##* Thunderbird! resending text only)

Hi Philipp,


On 02/13/2017 01:20 AM, Philipp Zabel wrote:
> Hi Steve,
>
> On Thu, 2017-02-09 at 15:51 -0800, Steve Longerbeam wrote:
>> On 02/09/2017 03:49 PM, Steve Longerbeam wrote:
>>> On 02/08/2017 03:42 PM, Russell King - ARM Linux wrote:
>>>> On Wed, Feb 08, 2017 at 03:23:53PM -0800, Steve Longerbeam wrote:
>>>>>> Actually, this exact function already exists as
>>>>>> dw_mipi_dsi_phy_write in
>>>>>> drivers/gpu/drm/rockchip/dw-mipi-dsi.c, and it looks like the D-PHY
>>>>>> register 0x44 might contain a field called HSFREQRANGE_SEL.
>>>>> Thanks for pointing out drivers/gpu/drm/rockchip/dw-mipi-dsi.c.
>>>>> It's clear from that driver that there probably needs to be a fuller
>>>>> treatment of the D-PHY programming here, but I don't know where
>>>>> to find the MIPI CSI-2 D-PHY documentation for the i.MX6. The code
>>>>> in imxcsi2_reset() was also pulled from FSL, and that's all I really
>>>>> have
>>>>> to go on for the D-PHY programming. I assume the D-PHY is also a
>>>>> Synopsys core, like the host controller, but the i.MX6 manual doesn't
>>>>> cover it.
>>>> Why exactly?  What problems are you seeing that would necessitate a
>>>> more detailed programming of the D-PHY?  From my testing, I can wind
>>>> a 2-lane MIPI bus on iMX6D from 912Mbps per lane down to (eg) 308Mbps
>>>> per lane with your existing code without any issues.
>>> That's good to hear.
>>>
>>> Just from my experience with struggles to get the CSI2 receiver
>>> up and running with an active clock lane, and my suspicions that
>>> some of that could be due to my lack of understanding of the D-PHY
>>> programming.
>> But I should add that after a re-org of the sequence, it looks more stable
>> now. Tested on both the SabreSD and SabreLite with the OV5640.
> It seems the OV5640 driver never puts its the CSI-2 lanes into stop
> state while not streaming.

Yes I found that as well.

But good news, I finally managed to coax the OV5640's clock lane
into LP-11 state! It was accomplished by setting bit 5 in OV5640 register
0x4800. The datasheet describes this bit as "Gate clock lane when no
packet to transmit". But I may have also got this to work with the 
additional
write 1 to bits 4-6 in register 0x3019 ("MIPI CLK/data lane state in sleep
mode" - setting 1 to these bits selects LP-11 for sleep mode).

So I am now finally able to call csi2_dphy_wait_stopstate() in
csi2_s_stream(ON).

So for the TC35874, you shouldn't see a timeout in csi2_s_stream(ON)
any longer.

I have updated both ov5640.c and imx6-mipi-csi2.c in the wip branch.
Can you try again? I have not applied your patch below, because I
don't think that is needed anymore.

And speaking of the TC35874, I received this module for the SabreLite
from Boundary Devices (thanks BD!). So I can finally help you with
debug/test. Are there any patches you need to send to me (against
wip branch) for this support, or can I use what exists now in media
tree? Also any scripts or help with running.


>   With the recent s_stream reordering,
> streaming from TC358743 does not work anymore, since imx6-mipi-csi2
> s_stream is called before tc358743 s_stream, while all lanes are still
> in stop state. Then it waits for the clock lane to become active and
> fails. I have applied the following patch to revert the reordering
> locally to get it to work again.
>
> The initialization order, as Russell pointed out, should be:
>
> 1. reset the D-PHY.
> 2. place MIPI sensor in LP-11 state
> 3. perform D-PHY initialisation
> 4. configure CSI2 lanes and de-assert resets and shutdown signals
>
> So csi2_s_stream should wait for stop state on all lanes (the result of
> 2.) before dphy_init (3.), not wait for active clock afterwards. That
> should happen only after sensor_s_stream was called. So unless we are
> allowed to reorder steps 1. and 2., we might need the prepare_stream
> callback after all.

Agreed!

See my new FIXME comment in imx6-mipi-csi2.c.

I agree we might need a new subdev op .prepare_stream(). This
op would be implemented by imx6-mipi-csi2.c, and would carry
out steps 3, 4, 5 (instead of currently by csi2_s_stream()). Then
step 6 would finally become available as csi2_s_stream().

And then we must re-order stream on to start sensor first, then
csi2, as in your patch below.

Steve

> More specifically, the chapter 40.3.1 "Startup Sequence" in the i.MX6DQ
> reference manual states:
>
> 1. Deassert presetn signal (global reset)
>     - This is probably the APB global reset, so not something we have to
>       care about.
> 2. Configure MIPI Camera Sensor to put all Tx lanes in PL-11 state
> 3. D-PHY initialization (write 0x14 to address 0x44)
> 4. CSI2 Controller programming
>     - Set N_LANES
>     - Deassert PHY_SHUTDOWNZ
>     - Deassert PHY_RSTZ
>     - Deassert CSI2_RESETN
> 5. Read the PHY status register (PHY_STATE) to confirm that all data and
>     clock lanes of the D-PHY are in Stop State
> 6. Configure the MIPI Camera Sensor to start transmitting a clock on the
>     D-PHY clock lane
> 7. CSI2 Controller programming - Read the PHY status register
>     (PHY_STATE) to confirm that the D-PHY is receiving a clock on the
>     D-PHY clock lane
>
> 2. can be done in sensor s_power (which tc358743 currently does) only if
> the sensor can still be configured in step 6.
> Also, 3./4./5. are csi2 code, 6. is sensor code, and 7. is csi2 code
> again. For reliable startup we need csi2 receiver code to be called on
> both sides of the sensor enabling its clock lane.
> ----------8<----------
>  From f12758caa3e1d05980cd2ac07587d125449fc860 Mon Sep 17 00:00:00 2001
> From: Philipp Zabel<p.zabel@pengutronix.de>
> Date: Mon, 13 Feb 2017 09:28:27 +0100
> Subject: [PATCH] media: imx: revert streamon sequence change
>
> Without this patch, starting streaming from a TC358743 MIPI CSI-2 source
> fails with the following error messages:
>
>      imx6-mipi-csi2: clock lane timeout, phy_state = 0x000006f0
>      ipu1_csi0: pipeline_set_stream failed with -110
>
> The phy state above has the stopstateclk, rxulpsclknot, and
> stopstatedata[3:0] bits set: at this point all lanes are in stop state,
> no lane is in ultra low power state or active.
> This is no suprise, since tc358743 s_stream(sd, 1) has not been called
> due to the recently changed ordering. The imx6-mipi-csi2 s_stream does
> wait for the clock lane to become active, so csi2_s_stream must happen
> after tc358743_s_stream.
>
> Signed-off-by: Philipp Zabel<p.zabel@pengutronix.de>
> ---
>   drivers/staging/media/imx/imx-media-utils.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 81eabcf76a66f..495ccefa3cd2a 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -782,8 +782,8 @@ static const u32 stream_on_seq[] = {
>   	IMX_MEDIA_GRP_ID_IC_PRPENC,
>   	IMX_MEDIA_GRP_ID_IC_PRP,
>   	IMX_MEDIA_GRP_ID_VDIC,
> -	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_SENSOR,
> +	IMX_MEDIA_GRP_ID_CSI2,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
>   	IMX_MEDIA_GRP_ID_CSI,
>   };
> @@ -795,8 +795,8 @@ static const u32 stream_off_seq[] = {
>   	IMX_MEDIA_GRP_ID_VDIC,
>   	IMX_MEDIA_GRP_ID_CSI,
>   	IMX_MEDIA_GRP_ID_VIDMUX,
> -	IMX_MEDIA_GRP_ID_SENSOR,
>   	IMX_MEDIA_GRP_ID_CSI2,
> +	IMX_MEDIA_GRP_ID_SENSOR,
>   };
>   
>   #define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-14 16:59                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-14 16:59 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Russell King - ARM Linux, robh+dt, mark.rutland, shawnguo,
	kernel, fabio.estevam, mchehab, hverkuil, nick, markus.heiser,
	laurent.pinchart+renesas, bparrot, geert, arnd, sudipm.mukherjee,
	minghsiu.tsai, tiffany.lin, jean-christophe.trotin,
	horms+renesas, niklas.soderlund+renesas, robert.jarzmik,
	songjun.wu, andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Mon, 2017-02-13 at 15:20 -0800, Steve Longerbeam wrote:
[...]
> > It seems the OV5640 driver never puts its the CSI-2 lanes into stop
> > state while not streaming.
> 
> Yes I found that as well.
> 
> But good news, I finally managed to coax the OV5640's clock lane
> into LP-11 state! It was accomplished by setting bit 5 in OV5640 register
> 0x4800. The datasheet describes this bit as "Gate clock lane when no
> packet to transmit". But I may have also got this to work with the 
> additional
> write 1 to bits 4-6 in register 0x3019 ("MIPI CLK/data lane state in sleep
> mode" - setting 1 to these bits selects LP-11 for sleep mode).
> 
> So I am now finally able to call csi2_dphy_wait_stopstate() in
> csi2_s_stream(ON).

That's good news.

> So for the TC35874, you shouldn't see a timeout in csi2_s_stream(ON)
> any longer.
> 
> I have updated both ov5640.c and imx6-mipi-csi2.c in the wip branch.
> Can you try again? I have not applied your patch below, because I
> don't think that is needed anymore.

Thanks, I'll test tomorrow.

> And speaking of the TC35874, I received this module for the SabreLite
> from Boundary Devices (thanks BD!). So I can finally help you with
> debug/test. Are there any patches you need to send to me (against
> wip branch) for this support, or can I use what exists now in media
> tree? Also any scripts or help with running.

That's even better news. I have pushed my my wip branch, which contains
some colorspace work and experiments to pass through query/g_/s_std
subdev calls so bypassing the pipeline can be avoided. Also, there's the
Nitrogen6X device tree that I've been using to test:

    https://git.pengutronix.de/git/pza/linux imx-media-staging-md-wip

> >   With the recent s_stream reordering,
> > streaming from TC358743 does not work anymore, since imx6-mipi-csi2
> > s_stream is called before tc358743 s_stream, while all lanes are still
> > in stop state. Then it waits for the clock lane to become active and
> > fails. I have applied the following patch to revert the reordering
> > locally to get it to work again.
> >
> > The initialization order, as Russell pointed out, should be:
> >
> > 1. reset the D-PHY.
> > 2. place MIPI sensor in LP-11 state
> > 3. perform D-PHY initialisation
> > 4. configure CSI2 lanes and de-assert resets and shutdown signals
> >
> > So csi2_s_stream should wait for stop state on all lanes (the result of
> > 2.) before dphy_init (3.), not wait for active clock afterwards. That
> > should happen only after sensor_s_stream was called. So unless we are
> > allowed to reorder steps 1. and 2., we might need the prepare_stream
> > callback after all.
> 
> Agreed!
> 
> See my new FIXME comment in imx6-mipi-csi2.c.

Looks good. I wonder if enabling the phy clock isn't part of step 3.
though.

> I agree we might need a new subdev op .prepare_stream(). This
> op would be implemented by imx6-mipi-csi2.c, and would carry
> out steps 3, 4, 5 (instead of currently by csi2_s_stream()). Then
> step 6 would finally become available as csi2_s_stream().
> 
> And then we must re-order stream on to start sensor first, then
> csi2, as in your patch below.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-14 16:59                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-14 16:59 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: Russell King - ARM Linux, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
	mark.rutland-5wv7dgnIgG8, shawnguo-DgEjT+Ai2ygdnm+yROfE0A,
	kernel-bIcnvbaLZ9MEGnE8C9+IrQ, fabio.estevam-3arQi8VN3Tc,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve

Hi Steve,

On Mon, 2017-02-13 at 15:20 -0800, Steve Longerbeam wrote:
[...]
> > It seems the OV5640 driver never puts its the CSI-2 lanes into stop
> > state while not streaming.
> 
> Yes I found that as well.
> 
> But good news, I finally managed to coax the OV5640's clock lane
> into LP-11 state! It was accomplished by setting bit 5 in OV5640 register
> 0x4800. The datasheet describes this bit as "Gate clock lane when no
> packet to transmit". But I may have also got this to work with the 
> additional
> write 1 to bits 4-6 in register 0x3019 ("MIPI CLK/data lane state in sleep
> mode" - setting 1 to these bits selects LP-11 for sleep mode).
> 
> So I am now finally able to call csi2_dphy_wait_stopstate() in
> csi2_s_stream(ON).

That's good news.

> So for the TC35874, you shouldn't see a timeout in csi2_s_stream(ON)
> any longer.
> 
> I have updated both ov5640.c and imx6-mipi-csi2.c in the wip branch.
> Can you try again? I have not applied your patch below, because I
> don't think that is needed anymore.

Thanks, I'll test tomorrow.

> And speaking of the TC35874, I received this module for the SabreLite
> from Boundary Devices (thanks BD!). So I can finally help you with
> debug/test. Are there any patches you need to send to me (against
> wip branch) for this support, or can I use what exists now in media
> tree? Also any scripts or help with running.

That's even better news. I have pushed my my wip branch, which contains
some colorspace work and experiments to pass through query/g_/s_std
subdev calls so bypassing the pipeline can be avoided. Also, there's the
Nitrogen6X device tree that I've been using to test:

    https://git.pengutronix.de/git/pza/linux imx-media-staging-md-wip

> >   With the recent s_stream reordering,
> > streaming from TC358743 does not work anymore, since imx6-mipi-csi2
> > s_stream is called before tc358743 s_stream, while all lanes are still
> > in stop state. Then it waits for the clock lane to become active and
> > fails. I have applied the following patch to revert the reordering
> > locally to get it to work again.
> >
> > The initialization order, as Russell pointed out, should be:
> >
> > 1. reset the D-PHY.
> > 2. place MIPI sensor in LP-11 state
> > 3. perform D-PHY initialisation
> > 4. configure CSI2 lanes and de-assert resets and shutdown signals
> >
> > So csi2_s_stream should wait for stop state on all lanes (the result of
> > 2.) before dphy_init (3.), not wait for active clock afterwards. That
> > should happen only after sensor_s_stream was called. So unless we are
> > allowed to reorder steps 1. and 2., we might need the prepare_stream
> > callback after all.
> 
> Agreed!
> 
> See my new FIXME comment in imx6-mipi-csi2.c.

Looks good. I wonder if enabling the phy clock isn't part of step 3.
though.

> I agree we might need a new subdev op .prepare_stream(). This
> op would be implemented by imx6-mipi-csi2.c, and would carry
> out steps 3, 4, 5 (instead of currently by csi2_s_stream()). Then
> step 6 would finally become available as csi2_s_stream().
> 
> And then we must re-order stream on to start sensor first, then
> csi2, as in your patch below.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver subdev driver
@ 2017-02-14 16:59                   ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-14 16:59 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Mon, 2017-02-13 at 15:20 -0800, Steve Longerbeam wrote:
[...]
> > It seems the OV5640 driver never puts its the CSI-2 lanes into stop
> > state while not streaming.
> 
> Yes I found that as well.
> 
> But good news, I finally managed to coax the OV5640's clock lane
> into LP-11 state! It was accomplished by setting bit 5 in OV5640 register
> 0x4800. The datasheet describes this bit as "Gate clock lane when no
> packet to transmit". But I may have also got this to work with the 
> additional
> write 1 to bits 4-6 in register 0x3019 ("MIPI CLK/data lane state in sleep
> mode" - setting 1 to these bits selects LP-11 for sleep mode).
> 
> So I am now finally able to call csi2_dphy_wait_stopstate() in
> csi2_s_stream(ON).

That's good news.

> So for the TC35874, you shouldn't see a timeout in csi2_s_stream(ON)
> any longer.
> 
> I have updated both ov5640.c and imx6-mipi-csi2.c in the wip branch.
> Can you try again? I have not applied your patch below, because I
> don't think that is needed anymore.

Thanks, I'll test tomorrow.

> And speaking of the TC35874, I received this module for the SabreLite
> from Boundary Devices (thanks BD!). So I can finally help you with
> debug/test. Are there any patches you need to send to me (against
> wip branch) for this support, or can I use what exists now in media
> tree? Also any scripts or help with running.

That's even better news. I have pushed my my wip branch, which contains
some colorspace work and experiments to pass through query/g_/s_std
subdev calls so bypassing the pipeline can be avoided. Also, there's the
Nitrogen6X device tree that I've been using to test:

    https://git.pengutronix.de/git/pza/linux imx-media-staging-md-wip

> >   With the recent s_stream reordering,
> > streaming from TC358743 does not work anymore, since imx6-mipi-csi2
> > s_stream is called before tc358743 s_stream, while all lanes are still
> > in stop state. Then it waits for the clock lane to become active and
> > fails. I have applied the following patch to revert the reordering
> > locally to get it to work again.
> >
> > The initialization order, as Russell pointed out, should be:
> >
> > 1. reset the D-PHY.
> > 2. place MIPI sensor in LP-11 state
> > 3. perform D-PHY initialisation
> > 4. configure CSI2 lanes and de-assert resets and shutdown signals
> >
> > So csi2_s_stream should wait for stop state on all lanes (the result of
> > 2.) before dphy_init (3.), not wait for active clock afterwards. That
> > should happen only after sensor_s_stream was called. So unless we are
> > allowed to reorder steps 1. and 2., we might need the prepare_stream
> > callback after all.
> 
> Agreed!
> 
> See my new FIXME comment in imx6-mipi-csi2.c.

Looks good. I wonder if enabling the phy clock isn't part of step 3.
though.

> I agree we might need a new subdev op .prepare_stream(). This
> op would be implemented by imx6-mipi-csi2.c, and would carry
> out steps 3, 4, 5 (instead of currently by csi2_s_stream()). Then
> step 6 would finally become available as csi2_s_stream().
> 
> And then we must re-order stream on to start sensor first, then
> csi2, as in your patch below.

regards
Philipp

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-15  2:27     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-15  2:27 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Philipp,

I've created a test branch off my imx-media-staging-md-wip called tc358743,
which cherry-picks a couple of your commits from your 
imx-media-staging-md-wip
branch:

[media] tc358743: set entity function to video interface bridge
[media] tc358743: put lanes in STOP state before starting streaming

And one more commit that enables the tc358743 in the DT for sabrelite:

ARM: dts: imx6-sabrelite: switch to tc358743

which is based off your work in imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi.

With that the tc358743 is loading fine, and is present in the media graph:

root@mx6q:~# dmesg | grep -i tc358
[   11.056799] imx-media: Registered subdev tc358743 1-000f
[   11.122133] imx-media: imx_media_create_link: tc358743 1-000f:0 -> 
imx6-mipi-csi2:0
[   11.490274] tc358743 1-000f: tc358743 found @ 0x1e (21a4000.i2c)


But I'm not able to get to testing streaming yet, see below.


On 01/31/2017 05:54 AM, Philipp Zabel wrote:
> Hi Steve,
>
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
>
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"

This works fine, I can create these links.

>
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex

I can probably generate this Intel hex file myself from sysfs
edid outputs, but for convenience do you mind sending me this
file? I have a 1080p HDMI source I can plug into the tc358743.

The other problem here is that my version of v4l2-ctl, built from
master branch of git@github.com:gjasny/v4l-utils.git, does not
support taking a subdev node:

root@mx6q:~# v4l2-ctl -d /dev/v4l-subdev15 --get-edid=format=hex
VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
/dev/v4l-subdev15: not a v4l2 node

Is this something you added yourself, or where can I find this version
of v4l2-ctrl?

Thanks,
Steve

> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
>
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>
> v4l2-ctl -d /dev/video4 -V
> # This still is configured to 640x480, which is inconsistent with
> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> # have set this, too.
>
> v4l2-ctl --list-formats -d /dev/video4
> # This lists all the RGB formats, which it shouldn't. There is
> # no CSC in this pipeline, so we should be limited to YUV formats
> # only.
>
> # Set capture format
> v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY
>
> v4l2-ctl -d /dev/video4 -V
> # Now the capture format is correctly configured to 1920x1080.
>
> v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
> # This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
> # frame interval. In the future this should list fractions achievable
> # via frame skipping.
>
> v4l2-compliance -d /dev/video4
> # This fails two tests:
> # fail: v4l2-test-input-output.cpp(383): std == 0
> # fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
> # test VIDIOC_G/S/ENUMINPUT: FAIL
> # and
> # fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
> # test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>
> # (Slowly) stream JPEG images to a display host:
> gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink
>
> I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
> timeout" message when starting streaming.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-15  2:27     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-15  2:27 UTC (permalink / raw)
  To: Philipp Zabel
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

Hi Philipp,

I've created a test branch off my imx-media-staging-md-wip called tc358743,
which cherry-picks a couple of your commits from your 
imx-media-staging-md-wip
branch:

[media] tc358743: set entity function to video interface bridge
[media] tc358743: put lanes in STOP state before starting streaming

And one more commit that enables the tc358743 in the DT for sabrelite:

ARM: dts: imx6-sabrelite: switch to tc358743

which is based off your work in imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi.

With that the tc358743 is loading fine, and is present in the media graph:

root@mx6q:~# dmesg | grep -i tc358
[   11.056799] imx-media: Registered subdev tc358743 1-000f
[   11.122133] imx-media: imx_media_create_link: tc358743 1-000f:0 -> 
imx6-mipi-csi2:0
[   11.490274] tc358743 1-000f: tc358743 found @ 0x1e (21a4000.i2c)


But I'm not able to get to testing streaming yet, see below.


On 01/31/2017 05:54 AM, Philipp Zabel wrote:
> Hi Steve,
>
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
>
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"

This works fine, I can create these links.

>
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex

I can probably generate this Intel hex file myself from sysfs
edid outputs, but for convenience do you mind sending me this
file? I have a 1080p HDMI source I can plug into the tc358743.

The other problem here is that my version of v4l2-ctl, built from
master branch of git-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org:gjasny/v4l-utils.git, does not
support taking a subdev node:

root@mx6q:~# v4l2-ctl -d /dev/v4l-subdev15 --get-edid=format=hex
VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
/dev/v4l-subdev15: not a v4l2 node

Is this something you added yourself, or where can I find this version
of v4l2-ctrl?

Thanks,
Steve

> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
>
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>
> v4l2-ctl -d /dev/video4 -V
> # This still is configured to 640x480, which is inconsistent with
> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> # have set this, too.
>
> v4l2-ctl --list-formats -d /dev/video4
> # This lists all the RGB formats, which it shouldn't. There is
> # no CSC in this pipeline, so we should be limited to YUV formats
> # only.
>
> # Set capture format
> v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY
>
> v4l2-ctl -d /dev/video4 -V
> # Now the capture format is correctly configured to 1920x1080.
>
> v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
> # This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
> # frame interval. In the future this should list fractions achievable
> # via frame skipping.
>
> v4l2-compliance -d /dev/video4
> # This fails two tests:
> # fail: v4l2-test-input-output.cpp(383): std == 0
> # fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
> # test VIDIOC_G/S/ENUMINPUT: FAIL
> # and
> # fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
> # test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>
> # (Slowly) stream JPEG images to a display host:
> gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink
>
> I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
> timeout" message when starting streaming.
>
> regards
> Philipp
>

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-15  2:27     ` Steve Longerbeam
  0 siblings, 0 replies; 549+ messages in thread
From: Steve Longerbeam @ 2017-02-15  2:27 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Philipp,

I've created a test branch off my imx-media-staging-md-wip called tc358743,
which cherry-picks a couple of your commits from your 
imx-media-staging-md-wip
branch:

[media] tc358743: set entity function to video interface bridge
[media] tc358743: put lanes in STOP state before starting streaming

And one more commit that enables the tc358743 in the DT for sabrelite:

ARM: dts: imx6-sabrelite: switch to tc358743

which is based off your work in imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi.

With that the tc358743 is loading fine, and is present in the media graph:

root at mx6q:~# dmesg | grep -i tc358
[   11.056799] imx-media: Registered subdev tc358743 1-000f
[   11.122133] imx-media: imx_media_create_link: tc358743 1-000f:0 -> 
imx6-mipi-csi2:0
[   11.490274] tc358743 1-000f: tc358743 found @ 0x1e (21a4000.i2c)


But I'm not able to get to testing streaming yet, see below.


On 01/31/2017 05:54 AM, Philipp Zabel wrote:
> Hi Steve,
>
> I have just tested the imx-media-staging-md-wip branch on a Nitrogen6X
> with a tc358743 (BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board). Some
> observations:
>
> # Link pipeline
> media-ctl -l "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]"
> media-ctl -l "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]"
> media-ctl -l "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"

This works fine, I can create these links.

>
> # Provide an EDID to the HDMI source
> v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex

I can probably generate this Intel hex file myself from sysfs
edid outputs, but for convenience do you mind sending me this
file? I have a 1080p HDMI source I can plug into the tc358743.

The other problem here is that my version of v4l2-ctl, built from
master branch of git at github.com:gjasny/v4l-utils.git, does not
support taking a subdev node:

root at mx6q:~# v4l2-ctl -d /dev/v4l-subdev15 --get-edid=format=hex
VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
/dev/v4l-subdev15: not a v4l2 node

Is this something you added yourself, or where can I find this version
of v4l2-ctrl?

Thanks,
Steve

> # At this point the HDMI source is enabled and sends a 1080p60 signal
> # Configure detected DV timings
> media-ctl --set-dv "'tc358743 1-000f':0"
>
> # Set pad formats
> media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY/1920x1080]"
> media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/1920x1080]"
> media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080]"
>
> v4l2-ctl -d /dev/video4 -V
> # This still is configured to 640x480, which is inconsistent with
> # the 'ipu1_csi0':2 pad format. The pad set_fmt above should
> # have set this, too.
>
> v4l2-ctl --list-formats -d /dev/video4
> # This lists all the RGB formats, which it shouldn't. There is
> # no CSC in this pipeline, so we should be limited to YUV formats
> # only.
>
> # Set capture format
> v4l2-ctl -d /dev/video4 -v width=1920,height=1080,pixelformat=UYVY
>
> v4l2-ctl -d /dev/video4 -V
> # Now the capture format is correctly configured to 1920x1080.
>
> v4l2-ctl -d 4 --list-frameintervals=width=1920,height=1080,pixelformat=UYVY
> # This lists nothing. We should at least provide the 'ipu1_csi0':2 pad
> # frame interval. In the future this should list fractions achievable
> # via frame skipping.
>
> v4l2-compliance -d /dev/video4
> # This fails two tests:
> # fail: v4l2-test-input-output.cpp(383): std == 0
> # fail: v4l2-test-input-output.cpp(449): invalid attributes for input 0
> # test VIDIOC_G/S/ENUMINPUT: FAIL
> # and
> # fail: v4l2-test-controls.cpp(782): subscribe event for control 'User Controls' failed
> # test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>
> # (Slowly) stream JPEG images to a display host:
> gst-launch-1.0 -v v4l2src device=/dev/video4 ! jpegenc ! rtpjpegpay ! udpsink
>
> I've done this a few times, and sometimes I only get a "ipu1_csi0: EOF
> timeout" message when starting streaming.
>
> regards
> Philipp
>

^ permalink raw reply	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-15  9:33       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-15  9:33 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt, mark.rutland, shawnguo, kernel, fabio.estevam, linux,
	mchehab, hverkuil, nick, markus.heiser, laurent.pinchart+renesas,
	bparrot, geert, arnd, sudipm.mukherjee, minghsiu.tsai,
	tiffany.lin, jean-christophe.trotin, horms+renesas,
	niklas.soderlund+renesas, robert.jarzmik, songjun.wu,
	andrew-ct.chen, gregkh, devicetree, linux-kernel,
	linux-arm-kernel, linux-media, devel, Steve Longerbeam

Hi Steve,

On Tue, 2017-02-14 at 18:27 -0800, Steve Longerbeam wrote:
[...]
> >
> > # Provide an EDID to the HDMI source
> > v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> 
> I can probably generate this Intel hex file myself from sysfs
> edid outputs, but for convenience do you mind sending me this
> file? I have a 1080p HDMI source I can plug into the tc358743.

I copied the EDID off of some random 1080p HDMI monitor,
probably using something like:

    xxd -g1 /sys/class/drm/card0-HDMI-A-1/edid | cut -c 9-56

----------8<----------
 00 ff ff ff ff ff ff 00 09 d1 89 78 45 54 00 00
 2a 14 01 03 80 35 1e 78 2e b8 45 a1 59 55 9f 28
 0d 50 54 a5 6b 80 81 c0 81 00 81 80 a9 c0 b3 00
 d1 c0 01 01 01 01 02 3a 80 18 71 38 2d 40 58 2c
 45 00 13 2a 21 00 00 1e 00 00 00 ff 00 4e 41 41
 30 36 32 39 36 53 4c 30 0a 20 00 00 00 fd 00 32
 4c 1e 53 11 00 0a 20 20 20 20 20 20 00 00 00 fc
 00 42 65 6e 51 20 47 4c 32 34 34 30 48 0a 01 18
 02 03 22 f1 4f 90 05 04 03 02 01 11 12 13 14 06
 07 15 16 1f 23 09 07 07 65 03 0c 00 10 00 83 01
 00 00 02 3a 80 18 71 38 2d 40 58 2c 45 00 13 2a
 21 00 00 1f 01 1d 80 18 71 1c 16 20 58 2c 25 00
 13 2a 21 00 00 9f 01 1d 00 72 51 d0 1e 20 6e 28
 55 00 13 2a 21 00 00 1e 8c 0a d0 8a 20 e0 2d 10
 10 3e 96 00 13 2a 21 00 00 18 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 eb
---------->8----------

> The other problem here is that my version of v4l2-ctl, built from
> master branch of git@github.com:gjasny/v4l-utils.git, does not
> support taking a subdev node:
> 
> root@mx6q:~# v4l2-ctl -d /dev/v4l-subdev15 --get-edid=format=hex
> VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
> /dev/v4l-subdev15: not a v4l2 node
> 
> Is this something you added yourself, or where can I find this version
> of v4l2-ctrl?

Ah right, I still have no proper fix for that. v4l-ctl bails out if it
can't VIDIOC_QUERYCAP, which is an ioctl not supported on subdevices.
I have just patched it out locally:

----------8<----------
diff --git a/utils/v4l2-ctl/v4l2-ctl.cpp b/utils/v4l2-ctl/v4l2-ctl.cpp
index 886a91d093ae..fa15a49375ae 100644
--- a/utils/v4l2-ctl/v4l2-ctl.cpp
+++ b/utils/v4l2-ctl/v4l2-ctl.cpp
@@ -1214,10 +1214,7 @@ int main(int argc, char **argv)
 	}
 
 	verbose = options[OptVerbose];
-	if (doioctl(fd, VIDIOC_QUERYCAP, &vcap)) {
-		fprintf(stderr, "%s: not a v4l2 node\n", device);
-		exit(1);
-	}
+	doioctl(fd, VIDIOC_QUERYCAP, &vcap);
 	capabilities = vcap.capabilities;
 	if (capabilities & V4L2_CAP_DEVICE_CAPS)
 		capabilities = vcap.device_caps;
---------->8----------

Note that setting the EDID is not necessary if you can force the mode on
your HDMI source.

regards
Philipp

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* Re: [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-15  9:33       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-15  9:33 UTC (permalink / raw)
  To: Steve Longerbeam
  Cc: robh+dt-DgEjT+Ai2ygdnm+yROfE0A, mark.rutland-5wv7dgnIgG8,
	shawnguo-DgEjT+Ai2ygdnm+yROfE0A, kernel-bIcnvbaLZ9MEGnE8C9+IrQ,
	fabio.estevam-3arQi8VN3Tc, linux-I+IVW8TIWO2tmTQ+vhA3Yw,
	mchehab-DgEjT+Ai2ygdnm+yROfE0A, hverkuil-qWit8jRvyhVmR6Xm/wNWPw,
	nick-gcszYUEDH4VrovVCs/uTlw,
	markus.heiser-O6JHGLzbNUwb1SvskN2V4Q,
	laurent.pinchart+renesas-ryLnwIuWjnjg/C1BVhZhaw,
	bparrot-l0cyMroinI0, geert-Td1EMuHUCqxL1ZNQvxDV9g,
	arnd-r2nGTMty4D4, sudipm.mukherjee-Re5JQEeQqe8AvxtiuMwx3w,
	minghsiu.tsai-NuS5LvNUpcJWk0Htik3J/w,
	tiffany.lin-NuS5LvNUpcJWk0Htik3J/w,
	jean-christophe.trotin-qxv4g6HH51o,
	horms+renesas-/R6kz+dDXgpPR4JQBCEnsQ,
	niklas.soderlund+renesas-1zkq55x86MTxsAP9Fp7wbw,
	robert.jarzmik-GANU6spQydw, songjun.wu-UWL1GkI3JZL3oGB3hsPCZA,
	andrew-ct.chen-NuS5LvNUpcJWk0Htik3J/w,
	gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
	devicetree-u79uwXL29TY76Z2rM5mHXA,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
	linux-media-u79uwXL29TY76Z2rM5mHXA,
	devel-gWbeCf7V1WCQmaza687I9mD2FQJk+8+b, Steve Longerbeam

Hi Steve,

On Tue, 2017-02-14 at 18:27 -0800, Steve Longerbeam wrote:
[...]
> >
> > # Provide an EDID to the HDMI source
> > v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> 
> I can probably generate this Intel hex file myself from sysfs
> edid outputs, but for convenience do you mind sending me this
> file? I have a 1080p HDMI source I can plug into the tc358743.

I copied the EDID off of some random 1080p HDMI monitor,
probably using something like:

    xxd -g1 /sys/class/drm/card0-HDMI-A-1/edid | cut -c 9-56

----------8<----------
 00 ff ff ff ff ff ff 00 09 d1 89 78 45 54 00 00
 2a 14 01 03 80 35 1e 78 2e b8 45 a1 59 55 9f 28
 0d 50 54 a5 6b 80 81 c0 81 00 81 80 a9 c0 b3 00
 d1 c0 01 01 01 01 02 3a 80 18 71 38 2d 40 58 2c
 45 00 13 2a 21 00 00 1e 00 00 00 ff 00 4e 41 41
 30 36 32 39 36 53 4c 30 0a 20 00 00 00 fd 00 32
 4c 1e 53 11 00 0a 20 20 20 20 20 20 00 00 00 fc
 00 42 65 6e 51 20 47 4c 32 34 34 30 48 0a 01 18
 02 03 22 f1 4f 90 05 04 03 02 01 11 12 13 14 06
 07 15 16 1f 23 09 07 07 65 03 0c 00 10 00 83 01
 00 00 02 3a 80 18 71 38 2d 40 58 2c 45 00 13 2a
 21 00 00 1f 01 1d 80 18 71 1c 16 20 58 2c 25 00
 13 2a 21 00 00 9f 01 1d 00 72 51 d0 1e 20 6e 28
 55 00 13 2a 21 00 00 1e 8c 0a d0 8a 20 e0 2d 10
 10 3e 96 00 13 2a 21 00 00 18 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 eb
---------->8----------

> The other problem here is that my version of v4l2-ctl, built from
> master branch of git-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org:gjasny/v4l-utils.git, does not
> support taking a subdev node:
> 
> root@mx6q:~# v4l2-ctl -d /dev/v4l-subdev15 --get-edid=format=hex
> VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
> /dev/v4l-subdev15: not a v4l2 node
> 
> Is this something you added yourself, or where can I find this version
> of v4l2-ctrl?

Ah right, I still have no proper fix for that. v4l-ctl bails out if it
can't VIDIOC_QUERYCAP, which is an ioctl not supported on subdevices.
I have just patched it out locally:

----------8<----------
diff --git a/utils/v4l2-ctl/v4l2-ctl.cpp b/utils/v4l2-ctl/v4l2-ctl.cpp
index 886a91d093ae..fa15a49375ae 100644
--- a/utils/v4l2-ctl/v4l2-ctl.cpp
+++ b/utils/v4l2-ctl/v4l2-ctl.cpp
@@ -1214,10 +1214,7 @@ int main(int argc, char **argv)
 	}
 
 	verbose = options[OptVerbose];
-	if (doioctl(fd, VIDIOC_QUERYCAP, &vcap)) {
-		fprintf(stderr, "%s: not a v4l2 node\n", device);
-		exit(1);
-	}
+	doioctl(fd, VIDIOC_QUERYCAP, &vcap);
 	capabilities = vcap.capabilities;
 	if (capabilities & V4L2_CAP_DEVICE_CAPS)
 		capabilities = vcap.device_caps;
---------->8----------

Note that setting the EDID is not necessary if you can force the mode on
your HDMI source.

regards
Philipp

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 549+ messages in thread
* [PATCH v3 00/24] i.MX Media Driver
@ 2017-02-15  9:33       ` Philipp Zabel
  0 siblings, 0 replies; 549+ messages in thread
From: Philipp Zabel @ 2017-02-15  9:33 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Steve,

On Tue, 2017-02-14 at 18:27 -0800, Steve Longerbeam wrote:
[...]
> >
> > # Provide an EDID to the HDMI source
> > v4l2-ctl -d /dev/v4l-subdev2 --set-edid=file=edid-1080p.hex
> 
> I can probably generate this Intel hex file myself from sysfs
> edid outputs, but for convenience do you mind sending me this
> file? I have a 1080p HDMI source I can plug into the tc358743.

I copied the EDID off of some random 1080p HDMI monitor,
probably using something like:

    xxd -g1 /sys/class/drm/card0-HDMI-A-1/edid | cut -c 9-56

----------8<----------
 00 ff ff ff ff ff ff 00 09 d1 89 78 45 54 00 00
 2a 14 01 03 80 35 1e 78 2e b8 45 a1 59 55 9f 28
 0d 50 54 a5 6b 80 81 c0 81 00 81 80 a9 c0 b3 00
 d1 c0 01 01 01 01 02 3a 80 18 71 38 2d 40 58 2c
 45 00 13 2a 21 00 00 1e 00 00 00 ff 00 4e 41 41
 30 36 32 39 36 53 4c 30 0a 20 00 00 00 fd 00 32
 4c 1e 53 11 00 0a 20 20 20 20 20 20 00 00 00 fc
 00 42 65 6e 51 20 47 4c 32 34 34 30 48 0a 01 18
 02 03 22 f1 4f 90 05 04 03 02 01 11 12 13 14 06
 07 15 16 1f 23 09 07 07 65 03 0c 00 10 00 83 01
 00 00 02 3a 80 18 71 38 2d 40 58 2c 45 00 13 2a
 21 00 00 1f 01 1d 80 18 71 1c 16 20 58 2c 25 00
 13 2a 21 00 00 9f 01 1d 00 72 51 d0 1e 20 6e 28
 55 00 13 2a 21 00 00 1e 8c 0a d0 8a 20 e0 2d 10
 10 3e 96 00 13 2a 21 00 00 18 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 eb
---------->8----------

> The other problem here is that my version of v4l2-ctl, built from
> master branch of git at github.com:gjasny/v4l-utils.git, does not
> support taking a subdev node:
> 
> root at mx6q:~# v4l2-ctl -d /dev/v4l-subdev15 --get-edid=format=hex
> VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
> /dev/v4l-subdev15: not a v4l2 node
> 
> Is this something you added yourself, or where can I find this version
> of v4l2-ctrl?

Ah right, I still have no proper fix for that. v4l-ctl bails out if it
can't VIDIOC_QUERYCAP, which is an ioctl not supported on subdevices.
I have just patched it out locally:

----------8<----------
diff --git a/utils/v4l2-ctl/v4l2-ctl.cpp b/utils/v4l2-ctl/v4l2-ctl.cpp
index 886a91d093ae..fa15a49375ae 100644
--- a/utils/v4l2-ctl/v4l2-ctl.cpp
+++ b/utils/v4l2-ctl/v4l2-ctl.cpp
@@ -1214,10 +1214,7 @@ int main(int argc, char **argv)
 	}
 
 	verbose = options[OptVerbose];
-	if (doioctl(fd, VIDIOC_QUERYCAP, &vcap)) {
-		fprintf(stderr, "%s: not a v4l2 node\n", device);
-		exit(1);
-	}
+	doioctl(fd, VIDIOC_QUERYCAP, &vcap);
 	capabilities = vcap.capabilities;
 	if (capabilities & V4L2_CAP_DEVICE_CAPS)
 		capabilities = vcap.device_caps;
---------->8----------

Note that setting the EDID is not necessary if you can force the mode on
your HDMI source.

regards
Philipp

^ permalink raw reply related	[flat|nested] 549+ messages in thread
end of thread, other threads:[~2017-02-15  9:34 UTC | newest]

Thread overview: 549+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-07  2:11 [PATCH v3 00/24] i.MX Media Driver Steve Longerbeam
2017-01-07  2:11 ` Steve Longerbeam
2017-01-07  2:11 ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-13 11:55   ` Philipp Zabel
2017-01-13 11:55     ` Philipp Zabel
2017-01-13 11:55     ` Philipp Zabel
2017-01-13 19:03     ` Steve Longerbeam
2017-01-13 19:03       ` Steve Longerbeam
2017-01-13 19:03       ` Steve Longerbeam
2017-01-16 12:09       ` Philipp Zabel
2017-01-16 12:09         ` Philipp Zabel
2017-01-16 12:09         ` Philipp Zabel
2017-01-16 17:13         ` Steve Longerbeam
2017-01-16 17:13           ` Steve Longerbeam
2017-01-16 17:13           ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-13 11:57   ` Philipp Zabel
2017-01-13 11:57     ` Philipp Zabel
2017-01-13 11:57     ` Philipp Zabel
2017-01-13 22:40     ` Steve Longerbeam
2017-01-13 22:40       ` Steve Longerbeam
2017-01-13 22:40       ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 03/24] ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their connections Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 04/24] ARM: dts: imx6qdl: add media device Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 05/24] ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-13 11:59   ` Philipp Zabel
2017-01-13 11:59     ` Philipp Zabel
2017-01-13 11:59     ` Philipp Zabel
2017-01-07  2:11 ` [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-13 12:03   ` Philipp Zabel
2017-01-13 12:03     ` Philipp Zabel
2017-01-13 12:03     ` Philipp Zabel
2017-01-13 23:04     ` Steve Longerbeam
2017-01-13 23:04       ` Steve Longerbeam
2017-01-13 23:04       ` Steve Longerbeam
2017-01-16 12:55       ` Philipp Zabel
2017-01-16 12:55         ` Philipp Zabel
2017-01-16 12:55         ` Philipp Zabel
2017-01-16 17:15         ` Steve Longerbeam
2017-01-16 17:15           ` Steve Longerbeam
2017-01-16 17:15           ` Steve Longerbeam
2017-02-05 15:17         ` Laurent Pinchart
2017-02-05 15:17           ` Laurent Pinchart
2017-02-05 15:17           ` Laurent Pinchart
2017-01-30 22:28   ` Russell King - ARM Linux
2017-01-30 22:28     ` Russell King - ARM Linux
2017-01-30 22:28     ` Russell King - ARM Linux
2017-01-30 22:51   ` Russell King - ARM Linux
2017-01-30 22:51     ` Russell King - ARM Linux
2017-01-30 22:51     ` Russell King - ARM Linux
2017-02-05 15:24     ` Laurent Pinchart
2017-02-05 15:24       ` Laurent Pinchart
2017-02-05 15:24       ` Laurent Pinchart
2017-02-05 15:37       ` Russell King - ARM Linux
2017-02-05 15:37         ` Russell King - ARM Linux
2017-02-05 15:37         ` Russell King - ARM Linux
2017-01-07  2:11 ` [PATCH v3 07/24] ARM: dts: imx6-sabresd: " Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 08/24] ARM: dts: imx6-sabreauto: create i2cmux for i2c3 Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 09/24] ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-12 19:37   ` Tim Harvey
2017-01-12 19:37     ` Tim Harvey
2017-01-12 19:37     ` Tim Harvey
2017-01-12 23:40     ` Steve Longerbeam
2017-01-12 23:40       ` Steve Longerbeam
2017-01-12 23:40       ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 11/24] ARM: dts: imx6-sabreauto: add the ADV7180 video decoder Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 12/24] add mux and video interface bridge entity functions Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-20 13:56   ` Hans Verkuil
2017-01-20 13:56     ` Hans Verkuil
2017-02-05 15:36   ` Laurent Pinchart
2017-02-05 15:36     ` Laurent Pinchart
2017-02-05 15:36     ` Laurent Pinchart
2017-02-06 10:27     ` Philipp Zabel
2017-02-06 10:27       ` Philipp Zabel
2017-02-06 10:27       ` Philipp Zabel
2017-01-07  2:11 ` [PATCH v3 13/24] platform: add video-multiplexer subdevice driver Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-10  5:35   ` Rob Herring
2017-01-10  5:35     ` Rob Herring
2017-01-10  5:35     ` Rob Herring
2017-01-20 14:03   ` Hans Verkuil
2017-01-20 14:03     ` Hans Verkuil
2017-01-20 14:03     ` Hans Verkuil
2017-01-24 12:02     ` Philipp Zabel
2017-01-24 12:02       ` Philipp Zabel
2017-01-24 12:02       ` Philipp Zabel
2017-01-25  2:07       ` Steve Longerbeam
2017-01-25  2:07         ` Steve Longerbeam
2017-01-25  2:07         ` Steve Longerbeam
2017-02-05 15:48         ` Laurent Pinchart
2017-02-05 15:48           ` Laurent Pinchart
2017-02-05 15:48           ` Laurent Pinchart
2017-02-06  9:50           ` Hans Verkuil
2017-02-06  9:50             ` Hans Verkuil
2017-02-06  9:50             ` Hans Verkuil
2017-02-06 22:33             ` Laurent Pinchart
2017-02-06 22:33               ` Laurent Pinchart
2017-02-06 22:33               ` Laurent Pinchart
2017-02-06 23:10               ` Steve Longerbeam
2017-02-06 23:10                 ` Steve Longerbeam
2017-02-06 23:10                 ` Steve Longerbeam
2017-02-07 10:26                 ` Laurent Pinchart
2017-02-07 10:26                   ` Laurent Pinchart
2017-02-07 10:26                   ` Laurent Pinchart
2017-02-07 10:41                   ` Philipp Zabel
2017-02-07 10:41                     ` Philipp Zabel
2017-02-07 10:41                     ` Philipp Zabel
2017-02-07 14:11                     ` Laurent Pinchart
2017-02-07 14:11                       ` Laurent Pinchart
2017-02-07 14:11                       ` Laurent Pinchart
2017-02-07 13:36                   ` Benoit Parrot
2017-02-07 13:36                     ` Benoit Parrot
2017-02-07 13:36                     ` Benoit Parrot
2017-02-07 20:50                     ` Sakari Ailus
2017-02-07 20:50                       ` Sakari Ailus
2017-02-07 20:50                       ` Sakari Ailus
2017-02-07 23:04                     ` Laurent Pinchart
2017-02-07 23:04                       ` Laurent Pinchart
2017-02-07 23:04                       ` Laurent Pinchart
2017-01-24 12:44   ` Javier Martinez Canillas
2017-01-24 12:44     ` Javier Martinez Canillas
2017-01-24 12:44     ` Javier Martinez Canillas
2017-01-24 12:44     ` Javier Martinez Canillas
2017-01-26  1:22     ` Steve Longerbeam
2017-01-26  1:22       ` Steve Longerbeam
2017-01-26  1:22       ` Steve Longerbeam
2017-01-26  1:22       ` Steve Longerbeam
2017-02-07 20:46   ` Sakari Ailus
2017-02-07 20:46     ` Sakari Ailus
2017-02-07 20:46     ` Sakari Ailus
2017-02-08  9:47     ` Philipp Zabel
2017-02-08  9:47       ` Philipp Zabel
2017-02-08  9:47       ` Philipp Zabel
2017-02-08 10:05       ` Laurent Pinchart
2017-02-08 10:05         ` Laurent Pinchart
2017-02-08 10:05         ` Laurent Pinchart
2017-01-07  2:11 ` [PATCH v3 14/24] UAPI: Add media UAPI Kbuild file Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 15/24] media: Add userspace header file for i.MX Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-13 12:05   ` Philipp Zabel
2017-01-13 12:05     ` Philipp Zabel
2017-01-13 12:05     ` Philipp Zabel
2017-01-13 23:13     ` Steve Longerbeam
2017-01-13 23:13       ` Steve Longerbeam
2017-01-13 23:13       ` Steve Longerbeam
2017-02-05 15:50       ` Laurent Pinchart
2017-02-05 15:50         ` Laurent Pinchart
2017-02-05 15:50         ` Laurent Pinchart
2017-01-07  2:11 ` [PATCH v3 16/24] media: Add i.MX media core driver Steve Longerbeam
2017-01-13 15:20   ` Philipp Zabel
2017-01-13 15:20     ` Philipp Zabel
2017-01-13 15:20     ` Philipp Zabel
2017-01-14 22:42     ` Steve Longerbeam
2017-01-19  1:44       ` Steve Longerbeam
2017-01-19  1:44         ` Steve Longerbeam
2017-01-19  1:44         ` Steve Longerbeam
2017-01-24 11:39         ` Philipp Zabel
2017-01-24 11:39           ` Philipp Zabel
2017-01-24 11:39           ` Philipp Zabel
2017-01-14 22:46     ` Steve Longerbeam
2017-01-14 22:46       ` Steve Longerbeam
2017-01-14 22:46       ` Steve Longerbeam
2017-01-16 13:47       ` Philipp Zabel
2017-01-16 13:47         ` Philipp Zabel
2017-01-16 13:47         ` Philipp Zabel
2017-01-23  2:31         ` Steve Longerbeam
2017-01-23  2:31           ` Steve Longerbeam
2017-01-23  2:31           ` Steve Longerbeam
2017-01-23 11:13           ` Philipp Zabel
2017-01-23 11:13             ` Philipp Zabel
2017-01-23 11:13             ` Philipp Zabel
2017-01-24  1:38             ` Steve Longerbeam
2017-01-24  1:38               ` Steve Longerbeam
2017-01-24  1:38               ` Steve Longerbeam
2017-01-24  1:45               ` Steve Longerbeam
2017-01-24  1:45                 ` Steve Longerbeam
2017-01-24  1:45                 ` Steve Longerbeam
2017-01-24 11:37                 ` Philipp Zabel
2017-01-24 11:37                   ` Philipp Zabel
2017-01-24 11:37                   ` Philipp Zabel
2017-01-30 15:28             ` Russell King - ARM Linux
2017-01-30 15:28               ` Russell King - ARM Linux
2017-01-30 15:28               ` Russell King - ARM Linux
2017-01-30 22:29   ` Russell King - ARM Linux
2017-01-30 22:29     ` Russell King - ARM Linux
2017-01-30 22:29     ` Russell King - ARM Linux
2017-02-02 22:44   ` Russell King - ARM Linux
2017-02-02 22:44     ` Russell King - ARM Linux
2017-02-02 22:44     ` Russell King - ARM Linux
2017-02-02 22:50   ` Russell King - ARM Linux
2017-02-02 22:50     ` Russell King - ARM Linux
2017-02-02 22:50     ` Russell King - ARM Linux
2017-02-07  1:54     ` Steve Longerbeam
2017-02-07  1:54       ` Steve Longerbeam
2017-02-07  1:54       ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 17/24] media: imx: Add CSI subdev driver Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-16 15:03   ` Philipp Zabel
2017-01-16 15:03     ` Philipp Zabel
2017-01-16 15:03     ` Philipp Zabel
2017-01-16 21:15     ` Steve Longerbeam
2017-01-16 21:15       ` Steve Longerbeam
2017-01-16 21:15       ` Steve Longerbeam
2017-01-30 22:29   ` Russell King - ARM Linux
2017-01-30 22:29     ` Russell King - ARM Linux
2017-01-30 22:29     ` Russell King - ARM Linux
2017-01-31 10:30   ` Russell King - ARM Linux
2017-01-31 10:30     ` Russell King - ARM Linux
2017-01-31 10:30     ` Russell King - ARM Linux
2017-01-31 11:20   ` Russell King - ARM Linux
2017-01-31 11:20     ` Russell King - ARM Linux
2017-01-31 11:20     ` Russell King - ARM Linux
2017-02-01  0:31     ` Steve Longerbeam
2017-02-01  0:31       ` Steve Longerbeam
2017-02-01  0:31       ` Steve Longerbeam
2017-02-01  0:58       ` Russell King - ARM Linux
2017-02-01  0:58         ` Russell King - ARM Linux
2017-02-01  0:58         ` Russell King - ARM Linux
2017-01-31 15:51   ` Philipp Zabel
2017-01-31 15:51     ` Philipp Zabel
2017-01-31 15:51     ` Philipp Zabel
2017-01-31 16:48     ` Philipp Zabel
2017-01-31 16:48       ` Philipp Zabel
2017-01-31 16:48       ` Philipp Zabel
2017-02-01  1:40       ` Steve Longerbeam
2017-02-01  1:40         ` Steve Longerbeam
2017-02-01  1:40         ` Steve Longerbeam
2017-02-07 16:18   ` [PATCH] media: imx: csi: fix crop rectangle reset in sink set_fmt Philipp Zabel
2017-02-07 16:18     ` Philipp Zabel
2017-02-07 16:18     ` Philipp Zabel
2017-01-07  2:11 ` [PATCH v3 18/24] media: imx: Add SMFC subdev driver Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-02-01 18:39   ` Russell King - ARM Linux
2017-02-01 18:39     ` Russell King - ARM Linux
2017-02-01 18:39     ` Russell King - ARM Linux
2017-02-01 18:52     ` Steve Longerbeam
2017-02-01 18:52       ` Steve Longerbeam
2017-02-01 18:52       ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 19/24] media: imx: Add IC subdev drivers Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-20 14:29   ` Hans Verkuil
2017-01-20 14:29     ` Hans Verkuil
2017-01-20 14:29     ` Hans Verkuil
2017-01-25  2:39     ` Steve Longerbeam
2017-01-25  2:39       ` Steve Longerbeam
2017-01-25  2:39       ` Steve Longerbeam
2017-01-30 22:29   ` Russell King - ARM Linux
2017-01-30 22:29     ` Russell King - ARM Linux
2017-01-30 22:29     ` Russell King - ARM Linux
2017-01-07  2:11 ` [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-20 14:38   ` Hans Verkuil
2017-01-20 14:38     ` Hans Verkuil
2017-01-20 14:38     ` Hans Verkuil
2017-01-24  2:15     ` Steve Longerbeam
2017-01-24  2:15       ` Steve Longerbeam
2017-01-31 13:42     ` Russell King - ARM Linux
2017-01-31 13:42       ` Russell King - ARM Linux
2017-01-31 13:42       ` Russell King - ARM Linux
2017-01-31 18:21       ` Steve Longerbeam
2017-01-31 18:21         ` Steve Longerbeam
2017-01-31 20:33         ` Russell King - ARM Linux
2017-01-31 20:33           ` Russell King - ARM Linux
2017-01-31 20:33           ` Russell King - ARM Linux
2017-01-31 21:55           ` Ian Arkver
2017-01-31 21:55             ` Ian Arkver
2017-01-31 21:55             ` Ian Arkver
2017-01-31 22:04             ` Russell King - ARM Linux
2017-01-31 22:04               ` Russell King - ARM Linux
2017-01-31 22:04               ` Russell King - ARM Linux
2017-01-31 22:33               ` Ian Arkver
2017-01-31 22:33                 ` Ian Arkver
2017-01-31 22:33                 ` Ian Arkver
2017-01-31 22:36               ` Steve Longerbeam
2017-01-31 22:36                 ` Steve Longerbeam
2017-01-31 22:36                 ` Steve Longerbeam
2017-01-31 23:30                 ` Russell King - ARM Linux
2017-01-31 23:30                   ` Russell King - ARM Linux
2017-01-31 23:30                   ` Russell King - ARM Linux
2017-01-31 23:41                   ` Steve Longerbeam
2017-01-31 23:41                     ` Steve Longerbeam
2017-01-31 23:41                     ` Steve Longerbeam
2017-02-02 22:35   ` Russell King - ARM Linux
2017-02-02 22:35     ` Russell King - ARM Linux
2017-02-02 22:35     ` Russell King - ARM Linux
2017-02-07  1:52     ` Steve Longerbeam
2017-02-07  1:52       ` Steve Longerbeam
2017-02-07  1:52       ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 21/24] media: imx: Add MIPI CSI-2 Receiver " Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-30 22:30   ` Russell King - ARM Linux
2017-01-30 22:30     ` Russell King - ARM Linux
2017-01-30 22:30     ` Russell King - ARM Linux
2017-01-31  0:01   ` Russell King - ARM Linux
2017-01-31  0:01     ` Russell King - ARM Linux
2017-01-31  0:01     ` Russell King - ARM Linux
2017-01-31  9:49     ` Philipp Zabel
2017-01-31  9:49       ` Philipp Zabel
2017-01-31  9:49       ` Philipp Zabel
2017-02-01  1:02       ` Steve Longerbeam
2017-02-01  1:02         ` Steve Longerbeam
2017-02-01  1:02         ` Steve Longerbeam
2017-02-01  1:13         ` Russell King - ARM Linux
2017-02-01  1:13           ` Russell King - ARM Linux
2017-02-01  1:13           ` Russell King - ARM Linux
2017-01-31  0:31   ` Russell King - ARM Linux
2017-01-31  0:31     ` Russell King - ARM Linux
2017-01-31  0:31     ` Russell King - ARM Linux
2017-01-31  2:11     ` Steve Longerbeam
2017-01-31  2:11       ` Steve Longerbeam
2017-01-31  2:11       ` Steve Longerbeam
2017-02-01 23:44   ` Russell King - ARM Linux
2017-02-01 23:44     ` Russell King - ARM Linux
2017-02-01 23:44     ` Russell King - ARM Linux
2017-02-02  0:04     ` Steve Longerbeam
2017-02-02  0:04       ` Steve Longerbeam
2017-02-02  0:04       ` Steve Longerbeam
2017-02-02 11:50   ` Philipp Zabel
2017-02-02 11:50     ` Philipp Zabel
2017-02-02 11:50     ` Philipp Zabel
2017-02-08 23:23     ` Steve Longerbeam
2017-02-08 23:23       ` Steve Longerbeam
2017-02-08 23:23       ` Steve Longerbeam
2017-02-08 23:42       ` Russell King - ARM Linux
2017-02-08 23:42         ` Russell King - ARM Linux
2017-02-08 23:42         ` Russell King - ARM Linux
2017-02-09 23:49         ` Steve Longerbeam
2017-02-09 23:49           ` Steve Longerbeam
2017-02-09 23:49           ` Steve Longerbeam
2017-02-09 23:51           ` Steve Longerbeam
2017-02-09 23:51             ` Steve Longerbeam
2017-02-09 23:51             ` Steve Longerbeam
2017-02-13  9:20             ` Philipp Zabel
2017-02-13  9:20               ` Philipp Zabel
2017-02-13  9:20               ` Philipp Zabel
2017-02-13 23:19               ` Steve Longerbeam
2017-02-13 23:20               ` Steve Longerbeam
2017-02-13 23:20                 ` Steve Longerbeam
2017-02-13 23:20                 ` Steve Longerbeam
2017-02-14 16:59                 ` Philipp Zabel
2017-02-14 16:59                   ` Philipp Zabel
2017-02-14 16:59                   ` Philipp Zabel
2017-02-09  9:43       ` Philipp Zabel
2017-02-09  9:43         ` Philipp Zabel
2017-02-09  9:43         ` Philipp Zabel
2017-01-07  2:11 ` [PATCH v3 22/24] media: imx: Add MIPI CSI-2 OV5640 sensor " Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-20 14:48   ` Hans Verkuil
2017-01-20 14:48     ` Hans Verkuil
2017-01-25 19:10     ` Steve Longerbeam
2017-01-25 19:10       ` Steve Longerbeam
2017-01-30 23:29   ` Russell King - ARM Linux
2017-01-30 23:29     ` Russell King - ARM Linux
2017-01-30 23:29     ` Russell King - ARM Linux
2017-01-31  3:31     ` Steve Longerbeam
2017-01-31  3:31       ` Steve Longerbeam
2017-01-31  3:31       ` Steve Longerbeam
2017-02-02 10:36   ` Laurent Pinchart
2017-02-02 10:36     ` Laurent Pinchart
2017-02-02 10:36     ` Laurent Pinchart
2017-02-12 22:46     ` Steve Longerbeam
2017-02-12 22:53     ` Steve Longerbeam
2017-02-12 22:53       ` Steve Longerbeam
2017-02-12 22:53       ` Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 23/24] media: imx: Add Parallel OV5642 " Steve Longerbeam
2017-01-07  2:11 ` [PATCH v3 24/24] ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-07  2:11   ` Steve Longerbeam
2017-01-11 23:14 ` [PATCH v3 00/24] i.MX Media Driver Tim Harvey
2017-01-11 23:14   ` Tim Harvey
2017-01-11 23:14   ` Tim Harvey
2017-01-12  3:22   ` Steve Longerbeam
2017-01-12  3:22     ` Steve Longerbeam
2017-01-12  3:22     ` Steve Longerbeam
2017-01-12 21:13     ` Tim Harvey
2017-01-12 22:32       ` Steve Longerbeam
2017-01-13 21:04         ` Tim Harvey
2017-01-20 13:52 ` Hans Verkuil
2017-01-20 13:52   ` Hans Verkuil
2017-01-20 13:52   ` Hans Verkuil
2017-01-20 16:31   ` Philipp Zabel
2017-01-20 16:31     ` Philipp Zabel
2017-01-20 16:31     ` Philipp Zabel
2017-01-20 18:40     ` Steve Longerbeam
2017-01-20 18:40       ` Steve Longerbeam
2017-01-20 20:39       ` Hans Verkuil
2017-01-20 20:39         ` Hans Verkuil
2017-01-20 20:39         ` Hans Verkuil
2017-01-23 11:00         ` Philipp Zabel
2017-01-23 11:00           ` Philipp Zabel
2017-01-23 11:00           ` Philipp Zabel
2017-01-23 11:08           ` Hans Verkuil
2017-01-23 11:08             ` Hans Verkuil
2017-01-23 11:08             ` Hans Verkuil
2017-01-24 11:28             ` Philipp Zabel
2017-01-24 11:28               ` Philipp Zabel
2017-01-24 11:28               ` Philipp Zabel
2017-01-23 23:08           ` Steve Longerbeam
2017-01-23 23:08             ` Steve Longerbeam
2017-01-24 11:27             ` Philipp Zabel
2017-01-24 11:27               ` Philipp Zabel
2017-01-24 11:27               ` Philipp Zabel
2017-01-28 19:27               ` Steve Longerbeam
2017-01-28 19:27                 ` Steve Longerbeam
2017-01-28 19:27                 ` Steve Longerbeam
2017-01-30 13:06               ` Russell King - ARM Linux
2017-01-30 13:06                 ` Russell King - ARM Linux
2017-01-30 13:06                 ` Russell King - ARM Linux
2017-01-31 10:09                 ` Philipp Zabel
2017-01-31 10:09                   ` Philipp Zabel
2017-01-31 10:09                   ` Philipp Zabel
2017-01-31 13:14                   ` Russell King - ARM Linux
2017-01-31 13:14                     ` Russell King - ARM Linux
2017-01-31 13:14                     ` Russell King - ARM Linux
2017-01-31 13:35                     ` Philipp Zabel
2017-01-31 13:35                       ` Philipp Zabel
2017-01-31 13:35                       ` Philipp Zabel
2017-01-31 14:04                       ` Russell King - ARM Linux
2017-01-31 14:04                         ` Russell King - ARM Linux
2017-01-31 14:04                         ` Russell King - ARM Linux
2017-01-31  0:45 ` Russell King - ARM Linux
2017-01-31  0:45   ` Russell King - ARM Linux
2017-01-31  0:45   ` Russell King - ARM Linux
2017-01-31  1:06   ` Russell King - ARM Linux
2017-01-31  1:06     ` Russell King - ARM Linux
2017-01-31  1:06     ` Russell King - ARM Linux
2017-01-31  2:06     ` Steve Longerbeam
2017-01-31  2:06       ` Steve Longerbeam
2017-01-31  2:06       ` Steve Longerbeam
2017-01-31  1:22   ` Steve Longerbeam
2017-01-31  1:22     ` Steve Longerbeam
2017-01-31  1:22     ` Steve Longerbeam
2017-01-31  9:49     ` Philipp Zabel
2017-01-31  9:49       ` Philipp Zabel
2017-01-31  9:49       ` Philipp Zabel
2017-01-31 10:21     ` Russell King - ARM Linux
2017-01-31 10:21       ` Russell King - ARM Linux
2017-01-31 10:21       ` Russell King - ARM Linux
2017-01-31 11:00     ` Russell King - ARM Linux
2017-01-31 11:00       ` Russell King - ARM Linux
2017-01-31 11:00       ` Russell King - ARM Linux
2017-01-31 23:43       ` Steve Longerbeam
2017-01-31 23:43         ` Steve Longerbeam
2017-01-31 23:43         ` Steve Longerbeam
2017-02-01  0:23         ` Russell King - ARM Linux
2017-02-01  0:23           ` Russell King - ARM Linux
2017-02-01  0:23           ` Russell King - ARM Linux
2017-02-01  1:54           ` Steve Longerbeam
2017-02-01  1:54             ` Steve Longerbeam
2017-02-01  1:54             ` Steve Longerbeam
2017-02-01  9:20             ` Russell King - ARM Linux
2017-02-01  9:20               ` Russell King - ARM Linux
2017-02-01  9:20               ` Russell King - ARM Linux
2017-01-31 13:54 ` Philipp Zabel
2017-01-31 13:54   ` Philipp Zabel
2017-01-31 13:54   ` Philipp Zabel
2017-01-31 14:25   ` Philipp Zabel
2017-01-31 14:25     ` Philipp Zabel
2017-01-31 14:25     ` Philipp Zabel
2017-01-31 15:03     ` Russell King - ARM Linux
2017-01-31 15:03       ` Russell King - ARM Linux
2017-01-31 15:03       ` Russell King - ARM Linux
2017-02-01  1:26   ` Steve Longerbeam
2017-02-01  1:26     ` Steve Longerbeam
2017-02-01  1:26     ` Steve Longerbeam
2017-02-01  9:30     ` Philipp Zabel
2017-02-01  9:30       ` Philipp Zabel
2017-02-01  9:30       ` Philipp Zabel
2017-02-01 10:11       ` Russell King - ARM Linux
2017-02-01 10:11         ` Russell King - ARM Linux
2017-02-01 10:11         ` Russell King - ARM Linux
2017-02-01 10:42         ` Philipp Zabel
2017-02-01 10:42           ` Philipp Zabel
2017-02-01 10:42           ` Philipp Zabel
2017-02-01 10:53           ` Russell King - ARM Linux
2017-02-01 10:53             ` Russell King - ARM Linux
2017-02-01 10:53             ` Russell King - ARM Linux
2017-02-02  0:19       ` Steve Longerbeam
2017-02-02  0:19         ` Steve Longerbeam
2017-02-02  0:19         ` Steve Longerbeam
2017-02-03 14:41         ` Laurent Pinchart
2017-02-03 14:41           ` Laurent Pinchart
2017-02-03 14:41           ` Laurent Pinchart
2017-02-03 17:56           ` Steve Longerbeam
2017-02-03 17:56             ` Steve Longerbeam
2017-02-03 17:56             ` Steve Longerbeam
2017-02-15  2:27   ` Steve Longerbeam
2017-02-15  2:27     ` Steve Longerbeam
2017-02-15  2:27     ` Steve Longerbeam
2017-02-15  9:33     ` Philipp Zabel
2017-02-15  9:33       ` Philipp Zabel
2017-02-15  9:33       ` Philipp Zabel
2017-02-02 17:22 ` Russell King - ARM Linux
2017-02-02 17:22   ` Russell King - ARM Linux
2017-02-02 17:22   ` Russell King - ARM Linux
2017-02-02 17:31   ` Steve Longerbeam
2017-02-02 17:31     ` Steve Longerbeam
2017-02-02 17:31     ` Steve Longerbeam
2017-02-02 17:56   ` Russell King - ARM Linux
2017-02-02 17:56     ` Russell King - ARM Linux
2017-02-02 17:56     ` Russell King - ARM Linux
2017-02-02 18:26     ` Steve Longerbeam
2017-02-02 18:26       ` Steve Longerbeam
2017-02-02 18:26       ` Steve Longerbeam
2017-02-02 18:58       ` Russell King - ARM Linux
2017-02-02 18:58         ` Russell King - ARM Linux
2017-02-02 18:58         ` Russell King - ARM Linux
2017-02-02 19:12         ` Steve Longerbeam
2017-02-02 19:12           ` Steve Longerbeam
2017-02-02 19:12           ` Steve Longerbeam
2017-02-02 22:29           ` Russell King - ARM Linux
2017-02-02 22:29             ` Russell King - ARM Linux
2017-02-02 22:29             ` Russell King - ARM Linux
2017-02-03 18:49             ` Steve Longerbeam
2017-02-03 18:49               ` Steve Longerbeam
2017-02-03 18:49               ` Steve Longerbeam
2017-02-03 14:21           ` vb2 queue_setup documentation clarification (was "Re: [PATCH v3 00/24] i.MX Media Driver") Laurent Pinchart
2017-02-03 14:31             ` Hans Verkuil

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.