[RFC PATCH V0 0/7] media: platform: Add support for Face Detection (FD) on mt8183 SoC

[RFC PATCH V0 0/7] media: platform: Add support for Face Detection (FD) on mt8183 SoC

[Jerry-ch Chen]

Hello,

This is the first version of the RFC patch series adding Face Detection
(FD) driver on Mediatek mt8183 SoC, which will be used in camera features
on CrOS application. It belongs to the first Mediatek's camera driver
series based on V4L2 and media controller framework. I posted the main part
of the FD driver as RFC to discuss first and would like some review
comments on the overall structure of the driver.

Face Detection (FD) unit provide hardware accelerated face detection
feature. It can detect different sizes of faces in a given image.
Furthermore, it has the capability to detect the faces of Rotation-in-Plane
from -180 to +180 degrees and Rotation-off-Plane from -90 to +90 degrees.

The driver is implemented with V4L2 and media controller framework. We have
the following entities describing the FD path.

1. Meta input (output video device): connects to FD sub device. It accepts
   the input parameter buffer from userspace. The metadata interface used
   currently is only a temporary solution to kick off driver development
   and is not ready for reviewed yet.

2. RAW (output video device): connects to FD sub device. It accepts input
   image buffer from userspace.

3. FD (sub device): connects to Meta output. When processing an image,
   FD hardware only returns the statistics of detected faces so it needs
   only one capture video devices to return the streaming data to the user.

4. Meta output (capture video device): Return the result of detected faces
   as metadata output.

   The overall file structure of the FD driver is as following:

* mtk_fd-dev-ctx-core.c: Implements common software flow of FD driver.
* mtk_fd-v4l2.c: Static FD contexts configuration.
* mtk_fd.c: Controls the hardware flow.
* mtk_fd-dev.c: Implements context-independent flow.
* mtk_fd-ctrl.c: Handles the HW ctrl request from userspace.
* mtk_fd-smem-drv.c: Provides the shared memory management required
operation. We reserved a memory region for the co-processor and FD to
exchange the hardware configuration data.
* mtk_fd-v4l2-util.c: Implements V4L2 and vb2 ops.

Jerry-ch Chen (7):
  dt-bindings: mt8183: Add binding for FD shared memory
  dts: arm64: mt8183: Add FD shared memory node
  dt-bindings: mt8183: Added FD-SMEM dt-bindings
  dt-bindings: mt8183: Added FD dt-bindings
  dts: arm64: mt8183: Add FD nodes
  media: platform: Add Mediatek FD driver KConfig
  platform: mtk-isp: Add Mediatek FD driver

 .../devicetree/bindings/media/mediatek,fd_smem.txt |   28 +
 .../bindings/media/mediatek,mt8183-fd.txt          |   30 +
 .../mediatek,reserve-memory-fd_smem.txt            |   44 +
 arch/arm64/boot/dts/mediatek/mt8183.dtsi           |   28 +
 drivers/media/platform/Kconfig                     |    2 +
 drivers/media/platform/mtk-isp/Kconfig             |   10 +
 drivers/media/platform/mtk-isp/Makefile            |   16 +
 drivers/media/platform/mtk-isp/fd/Makefile         |   38 +
 drivers/media/platform/mtk-isp/fd/mtk_fd-core.h    |  157 +++
 drivers/media/platform/mtk-isp/fd/mtk_fd-ctx.h     |  299 ++++++
 .../platform/mtk-isp/fd/mtk_fd-dev-ctx-core.c      |  917 +++++++++++++++++
 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.c     |  355 +++++++
 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.h     |  198 ++++
 .../media/platform/mtk-isp/fd/mtk_fd-smem-drv.c    |  452 +++++++++
 drivers/media/platform/mtk-isp/fd/mtk_fd-smem.h    |   25 +
 .../media/platform/mtk-isp/fd/mtk_fd-v4l2-util.c   | 1046 ++++++++++++++++++++
 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.c    |  115 +++
 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.h    |   36 +
 drivers/media/platform/mtk-isp/fd/mtk_fd.c         |  730 ++++++++++++++
 drivers/media/platform/mtk-isp/fd/mtk_fd.h         |  127 +++
 20 files changed, 4653 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,fd_smem.txt
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
 create mode 100644 Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-fd_smem.txt
 create mode 100644 drivers/media/platform/mtk-isp/Kconfig
 create mode 100644 drivers/media/platform/mtk-isp/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-core.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-ctx.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev-ctx-core.c
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.c
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-smem-drv.c
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-smem.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2-util.c
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.c
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.c
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h

-- 
1.9.1

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH V0 1/7] dt-bindings: mt8183: Add binding for FD shared memory

[Jerry-ch Chen]

This patch adds the binding for describing the shared memory
used to exchange meta data between the co-processor and Face
Detection (FD) unit of the camera system on Mediatek SoCs.

Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
---
 .../mediatek,reserve-memory-fd_smem.txt            | 44 ++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-fd_smem.txt

diff --git a/Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-fd_smem.txt b/Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-fd_smem.txt
new file mode 100644
index 0000000..52ae507
--- /dev/null
+++ b/Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-fd_smem.txt
@@ -0,0 +1,44 @@
+Mediatek FD Shared Memory binding
+
+This binding describes the shared memory, which serves the purpose of
+describing the shared memory region used to exchange data between Face
+Detection hardware (FD) and co-processor in Mediatek SoCs.
+
+The co-processor doesn't have the iommu so we need to use the physical
+address to access the shared buffer in the firmware.
+
+The Face Detection hardware (FD) can access memory through mt8183 IOMMU so
+it can use dma address to access the memory region.
+(See iommu/mediatek,iommu.txt for the detailed description of Mediatek IOMMU)
+
+
+Required properties:
+
+- compatible: must be "mediatek,reserve-memory-fd_smem"
+
+- reg: required for static allocation (see reserved-memory.txt for
+  the detailed usage)
+
+- alloc-range: required for dynamic allocation. The range must
+  between 0x00000400 and 0x100000000 due to the co-processer's
+  addressing limitation
+
+- size: required for dynamic allocation. The unit is bytes.
+  for Face Detection Unit, you need 1 MB at least.
+
+
+Example:
+
+The following example shows the FD shared memory setup for MT8183.
+
+	reserved-memory {
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges;
+		reserve-memory-fd_smem {
+			compatible = "mediatek,reserve-memory-fd_smem";
+			size = <0 0x00100000>;
+			alignment = <0 0x1000>;
+			alloc-ranges = <0 0x40000000 0 0x100000000>;
+		};
+	};
\ No newline at end of file
-- 
1.9.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH V0 2/7] dts: arm64: mt8183: Add FD shared memory node

[Jerry-ch Chen]

This patch adds a shared memory region used on mt8183 for exchanging
meta data between co-processor and Face Detection (FD) unit.

Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
---
 arch/arm64/boot/dts/mediatek/mt8183.dtsi | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/arm64/boot/dts/mediatek/mt8183.dtsi b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
index c3a516e..b3d8dfd 100644
--- a/arch/arm64/boot/dts/mediatek/mt8183.dtsi
+++ b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
@@ -134,6 +134,14 @@
 		clock-output-names = "clk26m";
 	};
 
+	reserve-memory-fd_smem {
+		compatible = "mediatek,reserve-memory-fd_smem";
+		no-map;
+		size = <0 0x00100000>;  /*1 MB share mem size */
+		alignment = <0 0x1000>;
+		alloc-ranges = <0 0x40000000 0 0x10000000>;
+	};
+
 	timer {
 		compatible = "arm,armv8-timer";
 		interrupt-parent = <&gic>;
-- 
1.9.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH V0 3/7] [media] dt-bindings: mt8183: Added FD-SMEM dt-bindings

[Jerry-ch Chen]

This patch adds the DT binding documentation for the shared memory
between Face Detection unit of the camera system and the co-processor
in Mediatek SoCs.

Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
---
 .../devicetree/bindings/media/mediatek,fd_smem.txt | 28 ++++++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,fd_smem.txt

diff --git a/Documentation/devicetree/bindings/media/mediatek,fd_smem.txt b/Documentation/devicetree/bindings/media/mediatek,fd_smem.txt
new file mode 100644
index 0000000..6f7c836
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/mediatek,fd_smem.txt
@@ -0,0 +1,28 @@
+Mediatek FD Shared Memory Device
+
+Mediatek FD Shared Memory Device is used to manage shared memory
+among CPU, FD and coprocessor. It is associated with a reserved
+memory region (Please see Documentation/devicetree/bindings/
+reserved-memory/mediatek,reserve-memory-fd_smem.txt) and
+provide the context to allocate memory with dma addresses.
+
+Required properties:
+- compatible: Shall be "mediatek,fd_smem"
+
+- iommus: Shall point to the respective IOMMU block with master port
+  as argument. Please set the ports which may be accessed
+  through the common path. You can see
+  Documentation/devicetree/bindings/iommu/mediatek,iommu.txt
+  for the detail.
+
+- mediatek,larb: must contain the local arbiters in the current SoCs.
+  Please set the larb of imgsys for FD if you are using FD function.
+  You can see Documentation/devicetree/bindings/memory-controllers/
+  mediatek,smi-larb.txt for the detail.
+
+Example:
+	fd_smem: fd_smem {
+		compatible = "mediatek,fd_smem";
+		mediatek,larb = <&larb5>;
+		iommus = <&iommu M4U_PORT_CAM_IMGI>;
+	};
-- 
1.9.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH V0 4/7] [media] dt-bindings: mt8183: Added FD dt-bindings

[Jerry-ch Chen]

This patch adds DT binding documentation for the Face Detection (FD)
unit of the camera system on Mediatek's SoCs.

Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
---
 .../bindings/media/mediatek,mt8183-fd.txt          | 30 ++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt

diff --git a/Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt b/Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
new file mode 100644
index 0000000..d8dbb89
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
@@ -0,0 +1,30 @@
+* Mediatek Face Detection Unit (FD)
+
+Face Detection (FD) unit is a typical memory-to-memory HW device.
+It provides hardware accelerated face detection function, and it
+is able to detect different poses of faces. FD will writre result
+of detected face into memory as output.
+
+Required properties:
+- compatible: "mediatek,fd"
+- reg: Must contain an entry for each entry in reg-names.
+- reg-names: Must include the following entries:
+- interrupts: interrupt number to the cpu.
+- clocks : clock name from clock manager
+- clock-names: must be main. It is the main clock of FD
+
+Example:
+	fd:fd@1502b000 {
+		compatible = "mediatek,fd";
+		mediatek,larb = <&larb5>;
+		mediatek,vpu = <&vpu>;
+		iommus = <&iommu M4U_PORT_CAM_FDVT_RP>,
+			 <&iommu M4U_PORT_CAM_FDVT_WR>,
+			 <&iommu M4U_PORT_CAM_FDVT_RB>;
+		reg = <0 0x1502b000 0 0x1000>;
+		interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>;
+		clocks = <&imgsys CLK_IMG_FDVT>;
+		clock-names = "FD_CLK_IMG_FDVT";
+		smem_device = <&fd_smem>;
+	};
+
-- 
1.9.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH V0 5/7] dts: arm64: mt8183: Add FD nodes

[Jerry-ch Chen]

This patch adds nodes for Face Detection (FD) unit. FD is embedded
in Mediatek SoCs and works with the co-processor to perform face
detection on the input data and image and output detected face result.

Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
---
 arch/arm64/boot/dts/mediatek/mt8183.dtsi | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/arm64/boot/dts/mediatek/mt8183.dtsi b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
index b3d8dfd..45c7e2f 100644
--- a/arch/arm64/boot/dts/mediatek/mt8183.dtsi
+++ b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
@@ -440,6 +440,26 @@
 			#clock-cells = <1>;
 		};
 
+		fd_smem: fd_smem {
+			compatible = "mediatek,fd_smem";
+			mediatek,larb = <&larb5>;
+			iommus = <&iommu M4U_PORT_CAM_IMGI>;
+		};
+
+		fd:fd@1502b000 {
+			compatible = "mediatek,fd";
+			mediatek,larb = <&larb5>;
+			mediatek,vpu = <&vpu>;
+			iommus = <&iommu M4U_PORT_CAM_FDVT_RP>,
+				 <&iommu M4U_PORT_CAM_FDVT_WR>,
+				 <&iommu M4U_PORT_CAM_FDVT_RB>;
+			reg = <0 0x1502b000 0 0x1000>;
+			interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>;
+			clocks = <&imgsys CLK_IMG_FDVT>;
+			clock-names = "FD_CLK_IMG_FD";
+			smem_device = <&fd_smem>;
+		};
+
 		vdecsys: syscon@16000000 {
 			compatible = "mediatek,mt8183-vdecsys", "syscon";
 			reg = <0 0x16000000 0 0x1000>;
-- 
1.9.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH V0 6/7] media: platform: Add Mediatek FD driver KConfig

[Jerry-ch Chen]

This patch adds KConfig for Mediatek Face Detection driver (FD).
FD is embedded in Mediatek SoCs. It can provide hardware
accelerated face detection function.

Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
---
 drivers/media/platform/Kconfig         |  2 ++
 drivers/media/platform/mtk-isp/Kconfig | 10 ++++++++++
 2 files changed, 12 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/Kconfig

diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index a505e9f..ef08d48 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -32,6 +32,8 @@ source "drivers/media/platform/davinci/Kconfig"
 
 source "drivers/media/platform/omap/Kconfig"
 
+source "drivers/media/platform/mtk-isp/Kconfig"
+
 config VIDEO_ASPEED
 	tristate "Aspeed AST2400 and AST2500 Video Engine driver"
 	depends on VIDEO_V4L2
diff --git a/drivers/media/platform/mtk-isp/Kconfig b/drivers/media/platform/mtk-isp/Kconfig
new file mode 100644
index 0000000..096d2cb
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/Kconfig
@@ -0,0 +1,10 @@
+config VIDEO_MEDIATEK_FD_SUPPORT
+	bool "Mediatek face detection processing function"
+
+	default n
+	help
+		Support the basic hardware accelerated face detectioin feature.
+
+		FD driver provide face detection function, it can detect
+		faces of Rotation-in-Plane from -180 degrees to +180 degrees
+		and Rotation-off-Plane from -90 degrees to +90 degrees.
-- 
1.9.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH V0 0/7] media: platform: Add support for Face Detection (FD) on mt8183 SoC

[Hans Verkuil]

Hi Jerry-ch Chen,

On 2/20/19 8:48 AM, Jerry-ch Chen wrote:
> Hello,
> 
> This is the first version of the RFC patch series adding Face Detection
> (FD) driver on Mediatek mt8183 SoC, which will be used in camera features
> on CrOS application. It belongs to the first Mediatek's camera driver
> series based on V4L2 and media controller framework. I posted the main part
> of the FD driver as RFC to discuss first and would like some review
> comments on the overall structure of the driver.
> 
> Face Detection (FD) unit provide hardware accelerated face detection
> feature. It can detect different sizes of faces in a given image.
> Furthermore, it has the capability to detect the faces of Rotation-in-Plane
> from -180 to +180 degrees and Rotation-off-Plane from -90 to +90 degrees.
> 
> The driver is implemented with V4L2 and media controller framework. We have
> the following entities describing the FD path.

Just a high-level comment before you post the next version of this series:

Please compile the latest version of v4l2-compliance (part of
git://linuxtv.org/v4l-utils.git) and run it against your driver:

v4l2-compliance -m /dev/mediaX

Whenever you post a new version of this series, please do a 'git pull' of
the v4l-utils repo, recompile and retest with v4l2-compliance and post the
test results in the cover letter.

Obviously, there should be no FAILs and probably no warnings.

I suspect that streaming (e.g. adding the -s10 option to v4l2-compliance)
probably won't work since v4l2-compliance doesn't know about the meta data
formats.

Regards,

	Hans

> 
> 1. Meta input (output video device): connects to FD sub device. It accepts
>    the input parameter buffer from userspace. The metadata interface used
>    currently is only a temporary solution to kick off driver development
>    and is not ready for reviewed yet.
> 
> 2. RAW (output video device): connects to FD sub device. It accepts input
>    image buffer from userspace.
> 
> 3. FD (sub device): connects to Meta output. When processing an image,
>    FD hardware only returns the statistics of detected faces so it needs
>    only one capture video devices to return the streaming data to the user.
> 
> 4. Meta output (capture video device): Return the result of detected faces
>    as metadata output.
> 
>    The overall file structure of the FD driver is as following:
> 
> * mtk_fd-dev-ctx-core.c: Implements common software flow of FD driver.
> * mtk_fd-v4l2.c: Static FD contexts configuration.
> * mtk_fd.c: Controls the hardware flow.
> * mtk_fd-dev.c: Implements context-independent flow.
> * mtk_fd-ctrl.c: Handles the HW ctrl request from userspace.
> * mtk_fd-smem-drv.c: Provides the shared memory management required
> operation. We reserved a memory region for the co-processor and FD to
> exchange the hardware configuration data.
> * mtk_fd-v4l2-util.c: Implements V4L2 and vb2 ops.
> 
> Jerry-ch Chen (7):
>   dt-bindings: mt8183: Add binding for FD shared memory
>   dts: arm64: mt8183: Add FD shared memory node
>   dt-bindings: mt8183: Added FD-SMEM dt-bindings
>   dt-bindings: mt8183: Added FD dt-bindings
>   dts: arm64: mt8183: Add FD nodes
>   media: platform: Add Mediatek FD driver KConfig
>   platform: mtk-isp: Add Mediatek FD driver
> 
>  .../devicetree/bindings/media/mediatek,fd_smem.txt |   28 +
>  .../bindings/media/mediatek,mt8183-fd.txt          |   30 +
>  .../mediatek,reserve-memory-fd_smem.txt            |   44 +
>  arch/arm64/boot/dts/mediatek/mt8183.dtsi           |   28 +
>  drivers/media/platform/Kconfig                     |    2 +
>  drivers/media/platform/mtk-isp/Kconfig             |   10 +
>  drivers/media/platform/mtk-isp/Makefile            |   16 +
>  drivers/media/platform/mtk-isp/fd/Makefile         |   38 +
>  drivers/media/platform/mtk-isp/fd/mtk_fd-core.h    |  157 +++
>  drivers/media/platform/mtk-isp/fd/mtk_fd-ctx.h     |  299 ++++++
>  .../platform/mtk-isp/fd/mtk_fd-dev-ctx-core.c      |  917 +++++++++++++++++
>  drivers/media/platform/mtk-isp/fd/mtk_fd-dev.c     |  355 +++++++
>  drivers/media/platform/mtk-isp/fd/mtk_fd-dev.h     |  198 ++++
>  .../media/platform/mtk-isp/fd/mtk_fd-smem-drv.c    |  452 +++++++++
>  drivers/media/platform/mtk-isp/fd/mtk_fd-smem.h    |   25 +
>  .../media/platform/mtk-isp/fd/mtk_fd-v4l2-util.c   | 1046 ++++++++++++++++++++
>  drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.c    |  115 +++
>  drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.h    |   36 +
>  drivers/media/platform/mtk-isp/fd/mtk_fd.c         |  730 ++++++++++++++
>  drivers/media/platform/mtk-isp/fd/mtk_fd.h         |  127 +++
>  20 files changed, 4653 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/mediatek,fd_smem.txt
>  create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
>  create mode 100644 Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-fd_smem.txt
>  create mode 100644 drivers/media/platform/mtk-isp/Kconfig
>  create mode 100644 drivers/media/platform/mtk-isp/Makefile
>  create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-core.h
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-ctx.h
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev-ctx-core.c
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.c
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.h
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-smem-drv.c
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-smem.h
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2-util.c
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.c
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.h
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.c
>  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH V0 0/7] media: platform: Add support for Face Detection (FD) on mt8183 SoC

[Jerry-ch Chen]

Hi Hans, Tomasz,

On Thu, 2019-03-14 at 16:40 +0800, Hans Verkuil wrote:
> Hi Jerry-ch Chen,
> 
> On 2/20/19 8:48 AM, Jerry-ch Chen wrote:
> > Hello,
> > 
> > This is the first version of the RFC patch series adding Face Detection
> > (FD) driver on Mediatek mt8183 SoC, which will be used in camera features
> > on CrOS application. It belongs to the first Mediatek's camera driver
> > series based on V4L2 and media controller framework. I posted the main part
> > of the FD driver as RFC to discuss first and would like some review
> > comments on the overall structure of the driver.
> > 
> > Face Detection (FD) unit provide hardware accelerated face detection
> > feature. It can detect different sizes of faces in a given image.
> > Furthermore, it has the capability to detect the faces of Rotation-in-Plane
> > from -180 to +180 degrees and Rotation-off-Plane from -90 to +90 degrees.
> > 
> > The driver is implemented with V4L2 and media controller framework. We have
> > the following entities describing the FD path.
> 
> Just a high-level comment before you post the next version of this series:
> 
> Please compile the latest version of v4l2-compliance (part of
> git://linuxtv.org/v4l-utils.git) and run it against your driver:
> 
> v4l2-compliance -m /dev/mediaX
> 
> Whenever you post a new version of this series, please do a 'git pull' of
> the v4l-utils repo, recompile and retest with v4l2-compliance and post the
> test results in the cover letter.
> 
> Obviously, there should be no FAILs and probably no warnings.
> 
> I suspect that streaming (e.g. adding the -s10 option to v4l2-compliance)
> probably won't work since v4l2-compliance doesn't know about the meta data
> formats.
> 
> Regards,
> 
> 	Hans
> 

Thanks for comments,
I am reworking FD driver based on general comments of P1 and DIP driver.
After that, I will upload the RFC V1 patch with the results of
v4l2-compliance in the cover-letter.

Best Regards,

	Jerry

> > 
> > 1. Meta input (output video device): connects to FD sub device. It accepts
> >    the input parameter buffer from userspace. The metadata interface used
> >    currently is only a temporary solution to kick off driver development
> >    and is not ready for reviewed yet.
> > 
> > 2. RAW (output video device): connects to FD sub device. It accepts input
> >    image buffer from userspace.
> > 
> > 3. FD (sub device): connects to Meta output. When processing an image,
> >    FD hardware only returns the statistics of detected faces so it needs
> >    only one capture video devices to return the streaming data to the user.
> > 
> > 4. Meta output (capture video device): Return the result of detected faces
> >    as metadata output.
> > 
> >    The overall file structure of the FD driver is as following:
> > 
> > * mtk_fd-dev-ctx-core.c: Implements common software flow of FD driver.
> > * mtk_fd-v4l2.c: Static FD contexts configuration.
> > * mtk_fd.c: Controls the hardware flow.
> > * mtk_fd-dev.c: Implements context-independent flow.
> > * mtk_fd-ctrl.c: Handles the HW ctrl request from userspace.
> > * mtk_fd-smem-drv.c: Provides the shared memory management required
> > operation. We reserved a memory region for the co-processor and FD to
> > exchange the hardware configuration data.
> > * mtk_fd-v4l2-util.c: Implements V4L2 and vb2 ops.
> > 
> > Jerry-ch Chen (7):
> >   dt-bindings: mt8183: Add binding for FD shared memory
> >   dts: arm64: mt8183: Add FD shared memory node
> >   dt-bindings: mt8183: Added FD-SMEM dt-bindings
> >   dt-bindings: mt8183: Added FD dt-bindings
> >   dts: arm64: mt8183: Add FD nodes
> >   media: platform: Add Mediatek FD driver KConfig
> >   platform: mtk-isp: Add Mediatek FD driver
> > 
> >  .../devicetree/bindings/media/mediatek,fd_smem.txt |   28 +
> >  .../bindings/media/mediatek,mt8183-fd.txt          |   30 +
> >  .../mediatek,reserve-memory-fd_smem.txt            |   44 +
> >  arch/arm64/boot/dts/mediatek/mt8183.dtsi           |   28 +
> >  drivers/media/platform/Kconfig                     |    2 +
> >  drivers/media/platform/mtk-isp/Kconfig             |   10 +
> >  drivers/media/platform/mtk-isp/Makefile            |   16 +
> >  drivers/media/platform/mtk-isp/fd/Makefile         |   38 +
> >  drivers/media/platform/mtk-isp/fd/mtk_fd-core.h    |  157 +++
> >  drivers/media/platform/mtk-isp/fd/mtk_fd-ctx.h     |  299 ++++++
> >  .../platform/mtk-isp/fd/mtk_fd-dev-ctx-core.c      |  917 +++++++++++++++++
> >  drivers/media/platform/mtk-isp/fd/mtk_fd-dev.c     |  355 +++++++
> >  drivers/media/platform/mtk-isp/fd/mtk_fd-dev.h     |  198 ++++
> >  .../media/platform/mtk-isp/fd/mtk_fd-smem-drv.c    |  452 +++++++++
> >  drivers/media/platform/mtk-isp/fd/mtk_fd-smem.h    |   25 +
> >  .../media/platform/mtk-isp/fd/mtk_fd-v4l2-util.c   | 1046 ++++++++++++++++++++
> >  drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.c    |  115 +++
> >  drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.h    |   36 +
> >  drivers/media/platform/mtk-isp/fd/mtk_fd.c         |  730 ++++++++++++++
> >  drivers/media/platform/mtk-isp/fd/mtk_fd.h         |  127 +++
> >  20 files changed, 4653 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/mediatek,fd_smem.txt
> >  create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
> >  create mode 100644 Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-fd_smem.txt
> >  create mode 100644 drivers/media/platform/mtk-isp/Kconfig
> >  create mode 100644 drivers/media/platform/mtk-isp/Makefile
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-core.h
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-ctx.h
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev-ctx-core.c
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.c
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-dev.h
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-smem-drv.c
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-smem.h
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2-util.c
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.c
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd-v4l2.h
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.c
> >  create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h
> > 
> 



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH V0 5/7] dts: arm64: mt8183: Add FD nodes

[Rob Herring]

On Wed, Feb 20, 2019 at 03:48:11PM +0800, Jerry-ch Chen wrote:
> This patch adds nodes for Face Detection (FD) unit. FD is embedded
> in Mediatek SoCs and works with the co-processor to perform face
> detection on the input data and image and output detected face result.
> 
> Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
> ---
>  arch/arm64/boot/dts/mediatek/mt8183.dtsi | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/arch/arm64/boot/dts/mediatek/mt8183.dtsi b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
> index b3d8dfd..45c7e2f 100644
> --- a/arch/arm64/boot/dts/mediatek/mt8183.dtsi
> +++ b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
> @@ -440,6 +440,26 @@
>  			#clock-cells = <1>;
>  		};
>  
> +		fd_smem: fd_smem {
> +			compatible = "mediatek,fd_smem";
> +			mediatek,larb = <&larb5>;
> +			iommus = <&iommu M4U_PORT_CAM_IMGI>;

This doesn't look like an actual h/w device...

> +		};
> +
> +		fd:fd@1502b000 {

space after the :  ^
> +			compatible = "mediatek,fd";

Should be SoC specific.

> +			mediatek,larb = <&larb5>;
> +			mediatek,vpu = <&vpu>;
> +			iommus = <&iommu M4U_PORT_CAM_FDVT_RP>,
> +				 <&iommu M4U_PORT_CAM_FDVT_WR>,
> +				 <&iommu M4U_PORT_CAM_FDVT_RB>;
> +			reg = <0 0x1502b000 0 0x1000>;
> +			interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>;
> +			clocks = <&imgsys CLK_IMG_FDVT>;
> +			clock-names = "FD_CLK_IMG_FD";
> +			smem_device = <&fd_smem>;
> +		};
> +
>  		vdecsys: syscon@16000000 {
>  			compatible = "mediatek,mt8183-vdecsys", "syscon";
>  			reg = <0 0x16000000 0 0x1000>;
> -- 
> 1.9.1
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH V0 5/7] dts: arm64: mt8183: Add FD nodes

[Jerry-ch Chen]

On Tue, 2019-03-26 at 05:57 +0800, Rob Herring wrote:
> On Wed, Feb 20, 2019 at 03:48:11PM +0800, Jerry-ch Chen wrote:
> > This patch adds nodes for Face Detection (FD) unit. FD is embedded
> > in Mediatek SoCs and works with the co-processor to perform face
> > detection on the input data and image and output detected face result.
> > 
> > Signed-off-by: Jerry-ch Chen <jerry-ch.chen@mediatek.com>
> > ---
> >  arch/arm64/boot/dts/mediatek/mt8183.dtsi | 20 ++++++++++++++++++++
> >  1 file changed, 20 insertions(+)
> > 
> > diff --git a/arch/arm64/boot/dts/mediatek/mt8183.dtsi b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
> > index b3d8dfd..45c7e2f 100644
> > --- a/arch/arm64/boot/dts/mediatek/mt8183.dtsi
> > +++ b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
> > @@ -440,6 +440,26 @@
> >  			#clock-cells = <1>;
> >  		};
> >  
> > +		fd_smem: fd_smem {
> > +			compatible = "mediatek,fd_smem";
> > +			mediatek,larb = <&larb5>;
> > +			iommus = <&iommu M4U_PORT_CAM_IMGI>;
> 
> This doesn't look like an actual h/w device...
> 
Thank you for your comments.

fd_smem is not an actual h/w device and it will not be used in the
future. We are going to remove the fd smem driver and fd_smem node.

> > +		};
> > +
> > +		fd:fd@1502b000 {
> 
> space after the :  ^
will be fixed.

> > +			compatible = "mediatek,fd";
> 
> Should be SoC specific.
> 
will be fixed as "mediatek, mt8183-fd".

Thanks and Best Regards
Jerry
> > +			mediatek,larb = <&larb5>;
> > +			mediatek,vpu = <&vpu>;
> > +			iommus = <&iommu M4U_PORT_CAM_FDVT_RP>,
> > +				 <&iommu M4U_PORT_CAM_FDVT_WR>,
> > +				 <&iommu M4U_PORT_CAM_FDVT_RB>;
> > +			reg = <0 0x1502b000 0 0x1000>;
> > +			interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>;
> > +			clocks = <&imgsys CLK_IMG_FDVT>;
> > +			clock-names = "FD_CLK_IMG_FD";
> > +			smem_device = <&fd_smem>;
> > +		};
> > +
> >  		vdecsys: syscon@16000000 {
> >  			compatible = "mediatek,mt8183-vdecsys", "syscon";
> >  			reg = <0 0x16000000 0 0x1000>;
> > -- 
> > 1.9.1
> > 



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel