linux-media.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver
@ 2019-06-11  3:53 Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 1/9] dt-bindings: mt8183: Added camera ISP Pass 1 Jungo Lin
                   ` (8 more replies)
  0 siblings, 9 replies; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

Hello,

This RFC patch series adding the driver for Pass 1 (P1) unit in
Mediatek's camera ISP system on mt8183 SoC, which
will be used in camera features of CrOS. It's the first time Mediatek
develops ISP kernel drivers based on V4L2 and media controller
framework. I posted the main part of the ISP Pass 1 driver as RFC to
discuss first and would like some review comments on the overall
architecture of the driver.

Pass 1 unit processes image signal from sensor devices and accepts the
tuning parameters to adjust the image quality. It performs optical
black correction, defect pixel correction, W/IR imbalance correction
and lens shading correction for RAW processing.

The driver is implemented with V4L2 and media controller framework so
we have the following entities to describe the ISP pass 1 path.

(The current metadata interface used in meta input and partial meta
nodes is only a temporary solution to kick off the driver development
and is not ready to be reviewed yet.)

1. meta input (output video device): connect to ISP P1 sub device.
It accepts the tuning buffer from user.

2. ISP P1 (sub device): connect to partial meta 0/1/2/3,
main stream and packed out video devices. When processing an image,
Pass 1 hardware supports multiple output images with different sizes
and formats so it needs two capture video devices ("main stream" and
"packed out") to return the image data to the user.

3. main stream (capture video device): return the processed image data
which is used in capture scenario.

4. packed out (capture video device): return the processed image data
which is used in preview scenario.

5. partial meta 0 (capture video device): return the AE/AWB statistics.

6. partial meta 1 (capture video device): return the AF statistics.

7. partial meta 2 (capture video device): return the local contrast
   enhanced statistics.

8. partial meta 3 (capture video device): return the local motion
   vector statistics.

The overall patches of the series is:

* Patch 1 & 2 are dt-bindings & dts information related to ISP P1 driver.
* Patch 3 is Kconfig configuration for ISP P1 driver.
* Patch 4 extends the original V4L2 image & meta formats.
* Patch 5 add Mediatek specific v4l2 control ID for ISP P1 driver.
* Patch 6 provides V4L2 utility functions & default video devices
  configuration & initialization for ISP P1 driver.
* Patch 7 is the heart of ISP P1 driver. It handles the ISP
  HW configuration, provides interrupt handling and initializes
  the V4L2 device nodes and other functions.
* Patch 8 adds communication with the co-processor on the SoC through
  the SCP driver[2].
* Patch 9 provides ISP P1 shard memory management between ISP P1 &
  co-processor. It is controlled by child device of ISP P1 driver.

Here is ISP P1 media topology:
It is included the main/sub sensor & sen-inf sub-devices which are
implemented in below patch[1][4][5]:


/usr/bin/media-ctl -p -d /dev/media1

Media controller API version 4.19.43

Media device information
------------------------
driver          mtk-cam
model           MTK-ISP-P1-V4L2
serial
bus info        platform:1a000000.camisp
hw revision     0x0
driver version  4.19.43

Device topology
- entity 1: MTK-ISP-P1-V4L2 (12 pads, 8 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev5
    pad0: Sink
        <- "MK-ISP-P1-V4L2 meta input":0 []
    pad1: Source
        -> "MTK-ISP-P1-V4L2 main stream":0 [ENABLED]
    pad2: Source
        -> "MTK-ISP-P1-V4L2 packed out":0 [ENABLED]
    pad3: Source
        -> "MTK-ISP-P1-V4L2 partial meta 0":0 []
    pad4: Source
        -> "MTK-ISP-P1-V4L2 partial meta 1":0 []
    pad5: Source
        -> "MTK-ISP-P1-V4L2 partial meta 2":0 []
    pad6: Source
        -> "MTK-ISP-P1-V4L2 partial meta 3":0 []
    pad7: Source
    pad8: Source
    pad9: Source
    pad10: Source
    pad11: Sink
        <- "1a040000.seninf.mipi-csi":4 []

- entity 14: MTK-ISP-P1-V4L2 meta input (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video12
    pad0: Sink
        -> "MTK-ISP-P1-V4L2":0 []

- entity 20: MTK-ISP-P1-V4L2 main stream (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video13
    pad0: Source
        <- "MTK-ISP-P1-V4L2":1 [ENABLED]

- entity 26: MTK-ISP-P1-V4L2 packed out (1 pad, 1 link)
             type Node subtype V4L flags 0
            device node name /dev/video14
    pad0: Source
        <- "MTK-ISP-P1-V4L2":2 [ENABLED]

- entity 32: MTK-ISP-P1-V4L2 partial meta 0 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video15
    pad0: Source
        <- "MTK-ISP-P1-V4L2":3 []

- entity 38: MTK-ISP-P1-V4L2 partial meta 1 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video16
    pad0: Source
        <- "MTK-ISP-P1-V4L2":4 []

- entity 44: MTK-ISP-P1-V4L2 partial meta 2 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video17
    pad0: Source
        <- "MTK-ISP-P1-V4L2":5 []

- entity 50: MTK-ISP-P1-V4L2 partial meta 3 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video18
    pad0: Source
        <- "MTK-ISP-P1-V4L2":6 []

- entity 56: 1a040000.seninf.mipi-csi (12 pads, 3 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev6
    pad0: Sink
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
        <- "ov5695 2-0036":0 []
    pad1: Sink
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
        <- "ov2685 4-003c":0 []
    pad2: Sink
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad3: Sink
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad4: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
        -> "MTK-ISP-P1-V4L2":11 []
    pad5: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad6: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad7: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad8: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad9: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad10: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
    pad11: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]

- entity 69: ov5695 2-0036 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev7
    pad0: Source
        [fmt:SBGGR10_1X10/2592x1944 field:none colorspace:srgb]
        -> "1a040000.seninf.mipi-csi":0 []

- entity 73: ov2685 4-003c (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev8
    pad0: Source
        [fmt:SBGGR10_1X10/1600x1200 field:none colorspace:srgb]
        -> "1a040000.seninf.mipi-csi":1 []

===========
= history =
===========

version 3:
 - Remove ISP Pass 1 reserved memory device node and change to use SCP's
   reserved memory region. (Rob Herring)
 - Fix comments of ISP Pass 1 device node & dt-bindings document (Rob Herring)
 - Revise ISP Pass1 Kconfig
 - Add rst documents for Mediatek image formats (Hans Verkuil)
 - Fix kernel warning messages when running v4l2_compliance test
 - Move AFO buffer enqueue & de-queue from request API to non-request
 - mtk_cam-ctrl.h/mtk_cam-ctrl.c
   Revise Mediatek ISP Pass1 specific V4L2 control naming & file licence declaration (Hans Verkuil)
   Split GET_BIN_INFO control into two controls to get width & height in-dependently (Hans Verkuil)
 - mtk_cam-v4l2-util.h/mtk_cam-v4l2-util.c
   Merging mtk_cam-dev.c and mtk_cam-v4l2-util.c. (Drew Davenport)
   Remove the pix_mode argument in related functions and unreachable code. (Drew Davenport)
   Fix Drew's comments which are addressed in v2 patch
   Fix some Tomasz comments which are addressed in DIP's v1 patch[3]
 - mtk_cam-regs.h / mtk_cam.h / mtk_cam.c
   Fix Drew's comments which are addressed in v2 patch
   Fix some Tomasz comments which are addressed in DIP's v1 patch[3]
   Refactoring mtk_isp_config & mtk_isp_req_enqueue functions
 - mtk_cam-scp.h / mtk_cam-scp.c
   Move function declarations from mtk_cam.h to mtk_cam-scp.h (Drew Davenport)
   Fix some Tomasz comments which are addressed in DIP's v1 patch[3]
   Fix ISP de-initialize timing KE issue
 - mtk_cam-smem.h / mtk_cam-smem-dev.c
   Get the reserved shared memory via SCP driver (Tomasz Figa)

Todo:
 - Add rst documents for Mediatek meta formats
 - New meta buffer structure design & re-factoring

version 2:
 - Add 3A enhancement feature which includes:
   Separates 3A pipeline out of frame basis to improve
   AE/AWB (exposure and white balance) performance.
   Add 2 SCP sub-commands for 3A meta buffers.
 - Add new child device to manage P1 shared memory between P1 HW unit
   and co-processor.
 - Remove mediatek,cam_smem.txt & cam_smem dts node in mt8183.dtsi.
 - Revised document wording for dt-bindings documents & dts information.
 - Remove mtk_cam-ctx.h & mtk_cam-dev-ctx-core.c and move these
   source codes to mtk_cam-dev.h & mtk_cam-dev.c.
 - mtk_cam-dev.h / mtk_cam-dev.c
   Revised mtk_cam_video_device & mtk_cam_dev to remove unused structure fields
   or add comments.
   Revised buffer size for LMVO & LCSO.
   Fix pixel format utility function.
   Add vb2_dma_contig_set_max_seg_size to configure DMA max segment size.
 - mtk_cam-v4l2-util.c
   Refactoring V4L2 async mechanism with seninf driver only
   Refactoring CIO (Connection IO) implementation with active sensor
   Revised stream on function for 3A enhancement feature
   Add new V4L2 en-queue/de-queue utility functions for 3A enhancement feature
 - mtk_cam-regs.h / mtk_cam.h / mtk_cam.c
   Add meta buffer index register definitions
   Add meta DMA configuration function.
   Separate with frame-base and non-frame-base en-queue/de-queue functions
   Add isp_setup_scp_rproc function to get RPC handle
   Add mtk_cam_reserved_memory_init for shared memory management
 - mtk_cam-scp.h / mtk_cam-scp.c
   Add new meta strictures for 3A enhancement feature
   Add new IPI command utility function for 3A enhancement feature
   Enhance isp_composer_dma_sg_init function flow
   Shorten overall IPI command structure size
   Remove scp_state state checking
   Improve code readability
 - mtk_cam-smem.h / mtk_cam-smem-dev.c
   Add mtk_cam_alloc_smem_dev to allocate one new child device of ISP driver.
   Handling P1 driver 's reserved memory & allocate DMA buffers based on this
   memory region.

TODOs:
 - 3A enhancement feature bug fixing

version 1:
 - Revised driver sources based on Tomasz's comments including
   part1/2/3/4 in RFC V0 patch.
 - Remove DMA cache mechanism.
   Support two new video devices (LCSO/LMVO) for advance camera
   features.
 - Fixed v4l2-compliance test failure items.
 - Add private controls for Mediatek camera middle-ware.
 - Replace VPU driver's APIs with new SCP driver interface for
   co-processor communication.
 - Refactoring mtk_cam_scp.c to use ring-buffers mechanism for IPI
   commands RX handling.
 - Fix internal bugs.

TODOs:
 - Remove mtk_cam_smem_drv.c & mtk_cam_smem.h and implement DMA pool
   for shared memory management.
 - Revised file names.
 - Support non frame-sync AFO/AAO DMA buffers

version 0:
- Initial submission

==================
 Dependent patch
==================

Camera ISP P1 driver depends on seninf driver, SCP driver.
The patches are listed as following:

[1]. media: support Mediatek sensor interface driver
https://patchwork.kernel.org/cover/10979135/

[2]. Add support for mt8183 SCP
https://patchwork.kernel.org/cover/10972143/

[3]. [RFC,V1,6/6] platform: mtk-isp: Add Mediatek DIP driver
https://patchwork.kernel.org/patch/10905223/

[4]. WIP: media: ov5695: support ov5695 sensor in mt8183
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1614887

[5]. WIP: media: ov2685: support ov2685 sensor in mt8183
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1614885

==================
 Compliance test
==================

The v4l2-compliance is built with the below lastest patch.
https://git.linuxtv.org/v4l-utils.git/commit/utils/v4l2-compliance?id=2aff3ef768c42cfdbb31d143ee2286a6b46e9db0

Note 1.
This testing depends on the above sensors[4][5] patches.

Note 2.
Before passing streaming on testing, need to enable media links between
entity 1: MTK-ISP-P1-V4L2, entity 20: MTK-ISP-P1-V4L2 main stream
and entity 26: MTK-ISP-P1-V4L2 packed out.

/usr/bin/media-ctl -i -d /dev/media1
Enter a link to modify or enter to stop
1:1->20:0[1]
1:1->20:0[1]
Enter a link to modify or enter to stop
1:2->26:0[1]
1:2->26:0[1]
Enter a link to modify or enter to stop

v4l2-compliance test output:
/usr/bin/v4l2-compliance -m /dev/media1

v4l2-compliance SHA: not available, 32 bits

Compliance test for mtk-cam device /dev/media1:

Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43

Required ioctls:
    test MEDIA_IOC_DEVICE_INFO: OK

Allow for multiple opens:
    test second /dev/media1 open: OK
    test MEDIA_IOC_DEVICE_INFO: OK
    test for unlimited opens: OK

Media Controller ioctls:
    test MEDIA_IOC_G_TOPOLOGY: OK
    Entities: 11 Interfaces: 11 Pads: 33 Links: 21
    test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
    test MEDIA_IOC_SETUP_LINK: OK

Total for mtk-cam device /dev/media1: 7, Succeeded: 7, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for MTK-ISP-P1-V4L2 device /dev/video12:

Driver Info:
    Driver name      : MTK-ISP-P1-V4L2
    Card type        : MTK-ISP-P1-V4L2
    Bus info         : platform:1a000000.camisp
    Driver version   : 4.19.43
    Capabilities     : 0x8c200000
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x0c200000
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000010
    Type             : V4L Video
Entity Info:
    ID               : 0x0000000e (14)
    Name             : MTK-ISP-P1-V4L2 meta input
    Function         : V4L2 I/O
    Pad 0x0100000f   : 0: Sink
      Link 0x02000012: to remote pad 0x1000002 of entity 'MTK-ISP-P1-V4L2': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK
    test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
    test second /dev/video12 open: OK
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
    test VIDIOC_QUERYCTRL: OK (Not Supported)
    test VIDIOC_G/S_CTRL: OK (Not Supported)
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 0 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK

Total for MTK-ISP-P1-V4L2 device /dev/video12: 45, Succeeded: 45, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for MTK-ISP-P1-V4L2 device /dev/video13:

Driver Info:
    Driver name      : MTK-ISP-P1-V4L2
    Card type        : MTK-ISP-P1-V4L2
    Bus info         : platform:1a000000.camisp
    Driver version   : 4.19.43
    Capabilities     : 0x84201000
        Video Capture Multiplanar
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04201000
        Video Capture Multiplanar
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000016
    Type             : V4L Video
Entity Info:
    ID               : 0x00000014 (20)
    Name             : MTK-ISP-P1-V4L2 main stream
    Function         : V4L2 I/O
    Pad 0x01000015   : 0: Source
      Link 0x02000018: from remote pad 0x1000003 of entity 'MTK-ISP-P1-V4L2': Data, Enabled

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK
    test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
    test second /dev/video13 open: OK
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
    test VIDIOC_QUERYCTRL: OK
    test VIDIOC_G/S_CTRL: OK
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 1 Private Controls: 3

Format ioctls (Input 0):
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK

Codec ioctls (Input 0):
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK

Total for MTK-ISP-P1-V4L2 device /dev/video13: 45, Succeeded: 45, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for MTK-ISP-P1-V4L2 device /dev/video14:

Driver Info:
    Driver name      : MTK-ISP-P1-V4L2
    Card type        : MTK-ISP-P1-V4L2
    Bus info         : platform:1a000000.camisp
    Driver version   : 4.19.43
    Capabilities     : 0x84201000
        Video Capture Multiplanar
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04201000
        Video Capture Multiplanar
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x0300001c
    Type             : V4L Video
Entity Info:
    ID               : 0x0000001a (26)
    Name             : MTK-ISP-P1-V4L2 packed out
    Function         : V4L2 I/O
    Pad 0x0100001b   : 0: Source
      Link 0x0200001e: from remote pad 0x1000004 of entity 'MTK-ISP-P1-V4L2': Data, Enabled

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK
    test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
    test second /dev/video14 open: OK
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
    test VIDIOC_QUERYCTRL: OK
    test VIDIOC_G/S_CTRL: OK
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 1 Private Controls: 3

Format ioctls (Input 0):
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK

Codec ioctls (Input 0):
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK

Total for MTK-ISP-P1-V4L2 device /dev/video14: 45, Succeeded: 45, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for MTK-ISP-P1-V4L2 device /dev/video15:

Driver Info:
    Driver name      : MTK-ISP-P1-V4L2
    Card type        : MTK-ISP-P1-V4L2
    Bus info         : platform:1a000000.camisp
    Driver version   : 4.19.43
    Capabilities     : 0x84a00000
        Metadata Capture
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04a00000
        Metadata Capture
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000022
    Type             : V4L Video
Entity Info:
    ID               : 0x00000020 (32)
    Name             : MTK-ISP-P1-V4L2 partial meta 0
    Function         : V4L2 I/O
    Pad 0x01000021   : 0: Source
      Link 0x02000024: from remote pad 0x1000005 of entity 'MTK-ISP-P1-V4L2': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK
    test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
    test second /dev/video15 open: OK
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
    test VIDIOC_QUERYCTRL: OK (Not Supported)
    test VIDIOC_G/S_CTRL: OK (Not Supported)
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 0 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK

Total for MTK-ISP-P1-V4L2 device /dev/video15: 45, Succeeded: 45, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for MTK-ISP-P1-V4L2 device /dev/video16:

Driver Info:
    Driver name      : MTK-ISP-P1-V4L2
    Card type        : MTK-ISP-P1-V4L2
    Bus info         : platform:1a000000.camisp
    Driver version   : 4.19.43
    Capabilities     : 0x84a00000
        Metadata Capture
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04a00000
        Metadata Capture
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000028
    Type             : V4L Video
Entity Info:
    ID               : 0x00000026 (38)
    Name             : MTK-ISP-P1-V4L2 partial meta 1
    Function         : V4L2 I/O
    Pad 0x01000027   : 0: Source
      Link 0x0200002a: from remote pad 0x1000006 of entity 'MTK-ISP-P1-V4L2': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK
    test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
    test second /dev/video16 open: OK
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
    test VIDIOC_QUERYCTRL: OK (Not Supported)
    test VIDIOC_G/S_CTRL: OK (Not Supported)
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 0 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK

Total for MTK-ISP-P1-V4L2 device /dev/video16: 45, Succeeded: 45, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for MTK-ISP-P1-V4L2 device /dev/video17:

Driver Info:
    Driver name      : MTK-ISP-P1-V4L2
    Card type        : MTK-ISP-P1-V4L2
    Bus info         : platform:1a000000.camisp
    Driver version   : 4.19.43
    Capabilities     : 0x84a00000
        Metadata Capture
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04a00000
        Metadata Capture
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x0300002e
    Type             : V4L Video
Entity Info:
    ID               : 0x0000002c (44)
    Name             : MTK-ISP-P1-V4L2 partial meta 2
    Function         : V4L2 I/O
    Pad 0x0100002d   : 0: Source
      Link 0x02000030: from remote pad 0x1000007 of entity 'MTK-ISP-P1-V4L2': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK
    test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
    test second /dev/video17 open: OK
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
    test VIDIOC_QUERYCTRL: OK (Not Supported)
    test VIDIOC_G/S_CTRL: OK (Not Supported)
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 0 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK

Total for MTK-ISP-P1-V4L2 device /dev/video17: 45, Succeeded: 45, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for MTK-ISP-P1-V4L2 device /dev/video18:

Driver Info:
    Driver name      : MTK-ISP-P1-V4L2
    Card type        : MTK-ISP-P1-V4L2
    Bus info         : platform:1a000000.camisp
    Driver version   : 4.19.43
    Capabilities     : 0x84a00000
        Metadata Capture
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04a00000
        Metadata Capture
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000034
    Type             : V4L Video
Entity Info:
    ID               : 0x00000032 (50)
    Name             : MTK-ISP-P1-V4L2 partial meta 3
    Function         : V4L2 I/O
    Pad 0x01000033   : 0: Source
      Link 0x02000036: from remote pad 0x1000008 of entity 'MTK-ISP-P1-V4L2': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK
    test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
    test second /dev/video18 open: OK
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
    test VIDIOC_QUERYCTRL: OK (Not Supported)
    test VIDIOC_G/S_CTRL: OK (Not Supported)
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 0 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK

Total for MTK-ISP-P1-V4L2 device /dev/video18: 45, Succeeded: 45, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-cam device /dev/v4l-subdev5:

Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x0300004f
    Type             : V4L Sub-Device
Entity Info:
    ID               : 0x00000001 (1)
    Name             : MTK-ISP-P1-V4L2
    Function         : Video Statistics
    Pad 0x01000002   : 0: Sink
      Link 0x02000012: from remote pad 0x100000f of entity 'MTK-ISP-P1-V4L2 meta input': Data
    Pad 0x01000003   : 1: Source
      Link 0x02000018: to remote pad 0x1000015 of entity 'MTK-ISP-P1-V4L2 main stream': Data, Enabled
    Pad 0x01000004   : 2: Source
      Link 0x0200001e: to remote pad 0x100001b of entity 'MTK-ISP-P1-V4L2 packed out': Data, Enabled
    Pad 0x01000005   : 3: Source
      Link 0x02000024: to remote pad 0x1000021 of entity 'MTK-ISP-P1-V4L2 partial meta 0': Data
    Pad 0x01000006   : 4: Source
      Link 0x0200002a: to remote pad 0x1000027 of entity 'MTK-ISP-P1-V4L2 partial meta 1': Data
    Pad 0x01000007   : 5: Source
      Link 0x02000030: to remote pad 0x100002d of entity 'MTK-ISP-P1-V4L2 partial meta 2': Data
    Pad 0x01000008   : 6: Source
      Link 0x02000036: to remote pad 0x1000033 of entity 'MTK-ISP-P1-V4L2 partial meta 3': Data
    Pad 0x01000009   : 7: Source
    Pad 0x0100000a   : 8: Source
    Pad 0x0100000b   : 9: Source
    Pad 0x0100000c   : 10: Source
    Pad 0x0100000d   : 11: Sink
      Link 0x0200004d: from remote pad 0x100003d of entity '1a040000.seninf.mipi-csi': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK

Allow for multiple opens:
    test second /dev/v4l-subdev5 open: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Sub-Device ioctls (Sink Pad 0):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 1):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 2):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 3):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 4):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 5):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 6):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 7):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 8):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 9):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 10):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Sink Pad 11):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
    test VIDIOC_QUERYCTRL: OK (Not Supported)
    test VIDIOC_G/S_CTRL: OK (Not Supported)
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 0 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK (Not Supported)
    test VIDIOC_TRY_FMT: OK (Not Supported)
    test VIDIOC_S_FMT: OK (Not Supported)
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
    test VIDIOC_EXPBUF: OK (Not Supported)
    test Requests: OK (Not Supported)

Total for mtk-cam device /dev/v4l-subdev5: 125, Succeeded: 125, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-cam device /dev/v4l-subdev6:

Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000051
    Type             : V4L Sub-Device
Entity Info:
    ID               : 0x00000038 (56)
    Name             : 1a040000.seninf.mipi-csi
    Function         : Video Interface Bridge
    Pad 0x01000039   : 0: Sink
      Link 0x02000047: from remote pad 0x1000046 of entity 'ov5695 2-0036': Data
    Pad 0x0100003a   : 1: Sink
      Link 0x0200004b: from remote pad 0x100004a of entity 'ov2685 4-003c': Data
    Pad 0x0100003b   : 2: Sink
    Pad 0x0100003c   : 3: Sink
    Pad 0x0100003d   : 4: Source
      Link 0x0200004d: to remote pad 0x100000d of entity 'MTK-ISP-P1-V4L2': Data
    Pad 0x0100003e   : 5: Source
    Pad 0x0100003f   : 6: Source
    Pad 0x01000040   : 7: Source
    Pad 0x01000041   : 8: Source
    Pad 0x01000042   : 9: Source
    Pad 0x01000043   : 10: Source
    Pad 0x01000044   : 11: Source

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK

Allow for multiple opens:
    test second /dev/v4l-subdev6 open: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Sub-Device ioctls (Sink Pad 0):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Sink Pad 1):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Sink Pad 2):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Sink Pad 3):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 4):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 5):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 6):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 7):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 8):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 9):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 10):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Sub-Device ioctls (Source Pad 11):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
    test VIDIOC_QUERYCTRL: OK
    test VIDIOC_G/S_CTRL: OK
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 2 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK (Not Supported)
    test VIDIOC_TRY_FMT: OK (Not Supported)
    test VIDIOC_S_FMT: OK (Not Supported)
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
    test VIDIOC_EXPBUF: OK (Not Supported)
    test Requests: OK (Not Supported)

Total for mtk-cam device /dev/v4l-subdev6: 125, Succeeded: 125, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-cam device /dev/v4l-subdev7:

Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000053
    Type             : V4L Sub-Device
Entity Info:
    ID               : 0x00000045 (69)
    Name             : ov5695 2-0036
    Function         : Camera Sensor
    Pad 0x01000046   : 0: Source
      Link 0x02000047: to remote pad 0x1000039 of entity '1a040000.seninf.mipi-csi': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK

Allow for multiple opens:
    test second /dev/v4l-subdev7 open: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Sub-Device ioctls (Source Pad 0):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
    test VIDIOC_QUERYCTRL: OK
    test VIDIOC_G/S_CTRL: OK
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 11 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK (Not Supported)
    test VIDIOC_TRY_FMT: OK (Not Supported)
    test VIDIOC_S_FMT: OK (Not Supported)
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
    test VIDIOC_EXPBUF: OK (Not Supported)
    test Requests: OK (Not Supported)

Total for mtk-cam device /dev/v4l-subdev7: 48, Succeeded: 48, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-cam device /dev/v4l-subdev8:

Media Driver Info:
    Driver name      : mtk-cam
    Model            : MTK-ISP-P1-V4L2
    Serial           : 
    Bus info         : platform:1a000000.camisp
    Media version    : 4.19.43
    Hardware revision: 0x00000000 (0)
    Driver version   : 4.19.43
Interface Info:
    ID               : 0x03000055
    Type             : V4L Sub-Device
Entity Info:
    ID               : 0x00000049 (73)
    Name             : ov2685 4-003c
    Function         : Camera Sensor
    Pad 0x0100004a   : 0: Source
      Link 0x0200004b: to remote pad 0x100003a of entity '1a040000.seninf.mipi-csi': Data

Required ioctls:
    test MC information (see 'Media Driver Info' above): OK

Allow for multiple opens:
    test second /dev/v4l-subdev8 open: OK
    test for unlimited opens: OK

Debug ioctls:
    test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

Sub-Device ioctls (Source Pad 0):
    test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
    test Try VIDIOC_SUBDEV_G/S_FMT: OK
    test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
    test Active VIDIOC_SUBDEV_G/S_FMT: OK
    test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK (Not Supported)
    test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Control ioctls:
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
    test VIDIOC_QUERYCTRL: OK
    test VIDIOC_G/S_CTRL: OK
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 10 Private Controls: 0

Format ioctls:
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK (Not Supported)
    test VIDIOC_TRY_FMT: OK (Not Supported)
    test VIDIOC_S_FMT: OK (Not Supported)
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK (Not Supported)

Codec ioctls:
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
    test VIDIOC_EXPBUF: OK (Not Supported)
    test Requests: OK (Not Supported)

Total for mtk-cam device /dev/v4l-subdev8: 48, Succeeded: 48, Failed: 0, Warnings: 0

Grand Total for mtk-cam device /dev/media1: 668, Succeeded: 668, Failed: 0, Warnings: 0

--------------------------------------------------------------------------------

Jungo Lin (9):
  dt-bindings: mt8183: Added camera ISP Pass 1
  dts: arm64: mt8183: Add ISP Pass 1 nodes
  media: platform: Add Mediatek ISP Pass 1 driver Kconfig
  media: platform: Add Mediatek ISP P1 image & meta formats
  media: platform: Add Mediatek ISP P1 V4L2 control
  media: platform: Add Mediatek ISP P1 V4L2 functions
  media: platform: Add Mediatek ISP P1 device driver
  media: platform: Add Mediatek ISP P1 SCP communication
  media: platform: Add Mediatek ISP P1 shared memory device

 .../bindings/media/mediatek,camisp.txt        |   57 +
 Documentation/media/uapi/v4l/pixfmt-mtb8.rst  |   49 +
 Documentation/media/uapi/v4l/pixfmt-mtba.rst  |   62 +
 Documentation/media/uapi/v4l/pixfmt-mtbc.rst  |   58 +
 Documentation/media/uapi/v4l/pixfmt-mtbe.rst  |   70 +
 Documentation/media/uapi/v4l/pixfmt-mtf8.rst  |   75 +
 Documentation/media/uapi/v4l/pixfmt-mtfa.rst  |   87 +
 Documentation/media/uapi/v4l/pixfmt-mtfc.rst  |  107 ++
 Documentation/media/uapi/v4l/pixfmt-mtfe.rst  |  107 ++
 arch/arm64/boot/dts/mediatek/mt8183.dtsi      |   24 +
 drivers/media/platform/Kconfig                |    2 +
 drivers/media/platform/mtk-isp/Kconfig        |   17 +
 drivers/media/platform/mtk-isp/Makefile       |    3 +
 .../media/platform/mtk-isp/isp_50/Makefile    |    5 +
 .../platform/mtk-isp/isp_50/cam/Makefile      |    9 +
 .../mtk-isp/isp_50/cam/mtk_cam-ctrl.c         |  138 ++
 .../mtk-isp/isp_50/cam/mtk_cam-ctrl.h         |   38 +
 .../mtk-isp/isp_50/cam/mtk_cam-regs.h         |  126 ++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.c |  371 ++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.h |  207 ++
 .../mtk-isp/isp_50/cam/mtk_cam-smem.c         |  304 +++
 .../mtk-isp/isp_50/cam/mtk_cam-smem.h         |   18 +
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c    | 1674 +++++++++++++++++
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h    |  173 ++
 .../platform/mtk-isp/isp_50/cam/mtk_cam.c     | 1087 +++++++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam.h     |  251 +++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   13 +
 include/uapi/linux/v4l2-controls.h            |    4 +
 include/uapi/linux/videodev2.h                |   17 +
 29 files changed, 5153 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,camisp.txt
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtb8.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtba.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtbc.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtbe.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtf8.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtfa.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtfc.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtfe.rst
 create mode 100644 drivers/media/platform/mtk-isp/Kconfig
 create mode 100644 drivers/media/platform/mtk-isp/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h

-- 
2.18.0


^ permalink raw reply	[flat|nested] 45+ messages in thread

* [RFC,v3 1/9] dt-bindings: mt8183: Added camera ISP Pass 1
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 2/9] dts: arm64: mt8183: Add ISP Pass 1 nodes Jungo Lin
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

This patch adds DT binding document for the Pass 1 (P1) unit in
Mediatek's camera ISP system. The Pass 1 unit grabs the sensor data
out from the sensor interface, applies ISP image effects from tuning
data and outputs the image data or statistics data to DRAM.

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
 .../bindings/media/mediatek,camisp.txt        | 57 +++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,camisp.txt

diff --git a/Documentation/devicetree/bindings/media/mediatek,camisp.txt b/Documentation/devicetree/bindings/media/mediatek,camisp.txt
new file mode 100644
index 000000000000..50a8b4d9ac8e
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/mediatek,camisp.txt
@@ -0,0 +1,57 @@
+* Mediatek Image Signal Processor Pass 1 (ISP P1)
+
+The Pass 1 unit of Mediatek's camera ISP system grabs the sensor data out
+from the sensor interface, applies ISP effects from tuning data and outputs
+the image data and statistics data to DRAM. Furthermore, Pass 1 unit has
+the ability to output two different resolutions frames at the same time to
+increase the performance of the camera application.
+
+Required properties:
+- compatible: Must be "mediatek,mt8183-camisp" for MT8183.
+- reg: Physical base address of the camera function block registers and
+  length of memory mapped region. Must contain an entry for each entry
+  in reg-names.
+- reg-names: Must include the following entries:
+  "cam_sys": Camsys base function block
+  "cam_uni": Camera UNI function block
+  "cam_a": Single camera ISP P1 hardware module A
+  "cam_b": Single camera ISP P1 hardware module B
+- interrupts: Interrupt number to the CPU.
+- iommus: Shall point to the respective IOMMU block with master port
+  as argument, see Documentation/devicetree/bindings/iommu/mediatek,iommu.txt
+  for details.
+- clocks: A list of phandle and clock specifier pairs as listed
+  in clock-names property, see
+  Documentation/devicetree/bindings/clock/clock-bindings.txt for details.
+- clock-names: Must be "camsys_cam_cgpdn" and "camsys_camtg_cgpdn".
+- mediatek,larb: Must contain the local arbiters in the current SoCs, see
+  Documentation/devicetree/bindings/memory-controllers/mediatek,smi-larb.txt
+  for details.
+- mediatek,scp : The node of system control processor (SCP), see
+  Documentation/devicetree/bindings/remoteproc/mtk,scp.txt for details.
+
+Example:
+SoC specific DT entry:
+
+		camisp: camisp@1a000000 {
+			compatible = "mediatek,mt8183-camisp", "syscon";
+			reg = <0 0x1a000000 0 0x1000>,
+			      <0 0x1a003000 0 0x1000>,
+			      <0 0x1a004000 0 0x2000>,
+			      <0 0x1a006000 0 0x2000>;
+			reg-names = "cam_sys",
+				    "cam_uni",
+				    "cam_a",
+				    "cam_b";
+			interrupts = <GIC_SPI 253 IRQ_TYPE_LEVEL_LOW>,
+				     <GIC_SPI 254 IRQ_TYPE_LEVEL_LOW>,
+				     <GIC_SPI 255 IRQ_TYPE_LEVEL_LOW>;
+			iommus = <&iommu M4U_PORT_CAM_IMGO>;
+			clocks = <&camsys CLK_CAM_CAM>,
+				 <&camsys CLK_CAM_CAMTG>;
+			clock-names = "camsys_cam_cgpdn",
+				      "camsys_camtg_cgpdn";
+			mediatek,larb = <&larb3>,
+					<&larb6>;
+			mediatek,scp = <&scp>;
+		};
\ No newline at end of file
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 2/9] dts: arm64: mt8183: Add ISP Pass 1 nodes
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 1/9] dt-bindings: mt8183: Added camera ISP Pass 1 Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 3/9] media: platform: Add Mediatek ISP Pass 1 driver Kconfig Jungo Lin
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

Add nodes for Pass 1 unit of Mediatek's camera ISP system.
Pass 1 unit embedded in Mediatek SoCs, works with the
co-processor to process image signal from the image sensor
and output RAW image data.

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
 arch/arm64/boot/dts/mediatek/mt8183.dtsi | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/arch/arm64/boot/dts/mediatek/mt8183.dtsi b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
index 75c4881bbe5e..8a725357c594 100644
--- a/arch/arm64/boot/dts/mediatek/mt8183.dtsi
+++ b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
@@ -369,5 +369,29 @@
 			reg = <0 0x1a000000 0 0x1000>;
 			#clock-cells = <1>;
 		};
+
+		camisp: camisp@1a000000 {
+			compatible = "mediatek,mt8183-camisp", "syscon";
+			reg = <0 0x1a000000 0 0x1000>,
+			      <0 0x1a003000 0 0x1000>,
+			      <0 0x1a004000 0 0x2000>,
+			      <0 0x1a006000 0 0x2000>;
+			reg-names = "cam_sys",
+				    "cam_uni",
+				    "cam_a",
+				    "cam_b";
+			interrupts = <GIC_SPI 253 IRQ_TYPE_LEVEL_LOW>,
+				     <GIC_SPI 254 IRQ_TYPE_LEVEL_LOW>,
+				     <GIC_SPI 255 IRQ_TYPE_LEVEL_LOW>;
+			iommus = <&iommu M4U_PORT_CAM_IMGO>;
+			clocks = <&camsys CLK_CAM_CAM>,
+				 <&camsys CLK_CAM_CAMTG>;
+			clock-names = "camsys_cam_cgpdn",
+				      "camsys_camtg_cgpdn";
+			mediatek,larb = <&larb3>,
+					<&larb6>;
+			mediatek,scp = <&scp>;
+		};
+
 	};
 };
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 3/9] media: platform: Add Mediatek ISP Pass 1 driver Kconfig
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 1/9] dt-bindings: mt8183: Added camera ISP Pass 1 Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 2/9] dts: arm64: mt8183: Add ISP Pass 1 nodes Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 4/9] media: platform: Add Mediatek ISP P1 image & meta formats Jungo Lin
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

This patch adds Kconfig for Pass 1 (P1) unit driver of Mediatek's
camera ISP system. ISP P1 unit is embedded in Mediatek SoCs. It
provides RAW processing which includes optical black correction,
defect pixel correction, W/IR imbalance correction and lens
shading correction.

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
 drivers/media/platform/Kconfig         |  2 ++
 drivers/media/platform/mtk-isp/Kconfig | 17 +++++++++++++++++
 2 files changed, 19 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/Kconfig

diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index 011c1c2fcf19..8e2b65d757e5 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -32,6 +32,8 @@ source "drivers/media/platform/davinci/Kconfig"
 
 source "drivers/media/platform/omap/Kconfig"
 
+source "drivers/media/platform/mtk-isp/Kconfig"
+
 config VIDEO_ASPEED
 	tristate "Aspeed AST2400 and AST2500 Video Engine driver"
 	depends on VIDEO_V4L2
diff --git a/drivers/media/platform/mtk-isp/Kconfig b/drivers/media/platform/mtk-isp/Kconfig
new file mode 100644
index 000000000000..983b79c261fa
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/Kconfig
@@ -0,0 +1,17 @@
+config VIDEO_MEDIATEK_ISP_PASS1
+	bool "Mediatek Pass 1 image processing function"
+	depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
+	depends on MEDIA_CONTROLLER && VIDEO_V4L2_SUBDEV_API
+	select V4L2_FWNODE
+	select VIDEOBUF2_DMA_CONTIG
+	default n
+	help
+		Pass 1 driver controls 3A (auto-focus, exposure,
+		and white balance) with tuning feature and outputs
+		the captured image buffers in Mediatek's camera system.
+
+		Choose y if you want to use Mediatek SoCs to create image
+		captured application such as video recording and still image
+		capturing.
+
+
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 4/9] media: platform: Add Mediatek ISP P1 image & meta formats
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
                   ` (2 preceding siblings ...)
  2019-06-11  3:53 ` [RFC,v3 3/9] media: platform: Add Mediatek ISP Pass 1 driver Kconfig Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 5/9] media: platform: Add Mediatek ISP P1 V4L2 control Jungo Lin
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

Add packed/full-g bayer formats with 8/10/12/14 bit
for image output. Add Pass 1 (P1) specific meta formats for
parameter processing and 3A/other statistics.

(The current metadata interface used in meta input and partial
meta nodes is only a temporary solution to kick off the driver
development and is not ready to be reviewed yet.)

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
 Documentation/media/uapi/v4l/pixfmt-mtb8.rst |  49 +++++++++
 Documentation/media/uapi/v4l/pixfmt-mtba.rst |  62 +++++++++++
 Documentation/media/uapi/v4l/pixfmt-mtbc.rst |  58 ++++++++++
 Documentation/media/uapi/v4l/pixfmt-mtbe.rst |  70 ++++++++++++
 Documentation/media/uapi/v4l/pixfmt-mtf8.rst |  75 +++++++++++++
 Documentation/media/uapi/v4l/pixfmt-mtfa.rst |  87 +++++++++++++++
 Documentation/media/uapi/v4l/pixfmt-mtfc.rst | 107 +++++++++++++++++++
 Documentation/media/uapi/v4l/pixfmt-mtfe.rst | 107 +++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c         |  13 +++
 include/uapi/linux/videodev2.h               |  17 +++
 10 files changed, 645 insertions(+)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtb8.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtba.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtbc.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtbe.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtf8.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtfa.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtfc.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-mtfe.rst

diff --git a/Documentation/media/uapi/v4l/pixfmt-mtb8.rst b/Documentation/media/uapi/v4l/pixfmt-mtb8.rst
new file mode 100644
index 000000000000..2337ccd66277
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtb8.rst
@@ -0,0 +1,49 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_B8:
+
+*******************************
+V4L2_PIX_FMT_MTISP_B8 ('MTB8')
+*******************************
+
+8-bit Packed Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format, meaning all the data for a pixel lie
+next to each other in memory, with a depth of 8 bits per pixel.
+Each sample is stored in a byte.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Byte Order.**
+Each cell is one byte.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00`
+      - G\ :sub:`01`
+      - B\ :sub:`02`
+      - G\ :sub:`03`
+    * - start + 4:
+      - G\ :sub:`10`
+      - R\ :sub:`11`
+      - G\ :sub:`12`
+      - R\ :sub:`13`
+    * - start + 8:
+      - B\ :sub:`20`
+      - G\ :sub:`21`
+      - B\ :sub:`22`
+      - G\ :sub:`23`
+    * - start + 12:
+      - G\ :sub:`30`
+      - R\ :sub:`31`
+      - G\ :sub:`32`
+      - R\ :sub:`33`
\ No newline at end of file
diff --git a/Documentation/media/uapi/v4l/pixfmt-mtba.rst b/Documentation/media/uapi/v4l/pixfmt-mtba.rst
new file mode 100644
index 000000000000..ade51d5472b0
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtba.rst
@@ -0,0 +1,62 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_B10:
+
+*******************************
+V4L2_PIX_FMT_MTISP_B10 ('MTBA')
+*******************************
+
+10-bit Packed Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format, meaning all the data for a pixel lie
+next to each other with no padding in memory, with a depth of 10 bits per pixel.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Byte Order.**
+Each cell is one byte.
+
+pixels cross the byte boundary and have a ratio of 5 bytes for each 4 pixels.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00low bits 7--0`
+      - G\ :sub:`01low bits 5--0` (bits 7--2) B\ :sub:`00high bits 9--8`\ (bits 1--0)
+    * - start + 2:
+      - B\ :sub:`02low bits 3--0`\ (bits 7--4) G\ :sub:`01high bits 9--6`\ (bits 3--0)
+      - G\ :sub:`03low bits 1--0`\ (bits 7--6) B\ :sub:`02high bits 9--4`\ (bits 5--0)
+    * - start + 4:
+      - G\ :sub:`03high bits 9--2`
+    * - start + 6:
+      - G\ :sub:`10low bits 7--0`
+      - R\ :sub:`11low bits 5--0`\ (bits 7--2) G\ :sub:`10high bits 9--8`\ (bits 1--0)
+    * - start + 8:
+      - G\ :sub:`12low bits 3--0`\ (bits 7--4) R\ :sub:`11high bits 9--6`\ (bits 3--0)
+      - R\ :sub:`13low bits 1--0`\ (bits 7--6) G\ :sub:`12high bits 9--4`\ (bits 5--0)
+    * - start + 10:
+      - R\ :sub:`13high bits 9--2`
+    * - start + 12:
+      - B\ :sub:`20low bits 7--0`
+      - G\ :sub:`21low bits 5--0`\ (bits 7--2) B\ :sub:`20high bits 9--8`\ (bits 1--0)
+    * - start + 14:
+      - B\ :sub:`22low bits 3--0`\ (bits 7--4) G\ :sub:`21high bits 9--6`\ (bits 3--0)
+      - G\ :sub:`23low bits 1--0`\ (bits 7--6) B\ :sub:`22high bits 9--4`\ (bits 5--0)
+    * - start + 16:
+      - G\ :sub:`23high bits 9--2`
+    * - start + 18:
+      - G\ :sub:`30low bits 7--0`
+      - R\ :sub:`31low bits 5--0`\ (bits 7--2) G\ :sub:`30high bits 9--8`\ (bits 1--0)
+    * - start + 20:
+      - G\ :sub:`32low bits 3--0`\ (bits 7--4) R\ :sub:`31high bits 9--6`\ (bits 3--0)
+      - R\ :sub:`33low bits 1--0`\ (bits 7--6) G\ :sub:`32high bits 9--4`\ (bits 5--0)
+    * - start + 22:
+      - R\ :sub:`33high bits 9--2` (bits 7--0)
\ No newline at end of file
diff --git a/Documentation/media/uapi/v4l/pixfmt-mtbc.rst b/Documentation/media/uapi/v4l/pixfmt-mtbc.rst
new file mode 100644
index 000000000000..b122600fddb5
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtbc.rst
@@ -0,0 +1,58 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_B12:
+
+*******************************
+V4L2_PIX_FMT_MTISP_B12 ('MTBC')
+*******************************
+
+12-bit Packed Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format, meaning all the data for a pixel lie
+next to each other with no padding in memory, with a depth of 12 bits per pixel.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Byte Order.**
+Each cell is one byte.
+
+pixels cross the byte boundary and have a ratio of 6 bytes for each 4 pixels.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00lowbits 7--0`
+      - G\ :sub:`01lowbits 3--0`\ (bits 7--4) B\ :sub:`00highbits 11--8`\ (bits 3--0)
+      - G\ :sub:`01highbits 7--0`
+      - B\ :sub:`02lowbits 7--0`
+      - G\ :sub:`03lowbits 3--0`\ (bits 7--4) B\ :sub:`02highbits 11--8`\ (bits 3--0)
+      - G\ :sub:`03highbits 7--0`
+    * - start + 6:
+      - G\ :sub:`10lowbits 7--0`
+      - R\ :sub:`11lowbits 3--0`\ (bits 7--4) G\ :sub:`10highbits 11--8`\ (bits 3--0)
+      - R\ :sub:`11highbits 7--0`
+      - G\ :sub:`12lowbits 7--0`
+      - R\ :sub:`13lowbits 3--0`\ (bits 7--4) G\ :sub:`12highbits 11--8`\ (bits 3--0)
+      - R\ :sub:`13highbits 7--0`
+    * - start + 12:
+      - B\ :sub:`20lowbits 7--0`
+      - G\ :sub:`21lowbits 3--0`\ (bits 7--4) B\ :sub:`20highbits 11--8`\ (bits 3--0)
+      - G\ :sub:`21highbits 7--0`
+      - B\ :sub:`22lowbits 7--0`
+      - G\ :sub:`23lowbits 3--0`\ (bits 7--4) B\ :sub:`22highbits 11--8`\ (bits 3--0)
+      - G\ :sub:`23highbits 7--0`
+    * - start + 18:
+      - G\ :sub:`30lowbits 7--0`
+      - R\ :sub:`31lowbits 3--0`\ (bits 7--4) G\ :sub:`30highbits 11--8`\ (bits 3--0)
+      - R\ :sub:`31highbits 7--0`
+      - G\ :sub:`32lowbits 7--0`
+      - R\ :sub:`33lowbits 3--0`\ (bits 7--4) G\ :sub:`32highbits 11--8`\ (bits 3--0)
+      - R\ :sub:`33highbits 7--0`
diff --git a/Documentation/media/uapi/v4l/pixfmt-mtbe.rst b/Documentation/media/uapi/v4l/pixfmt-mtbe.rst
new file mode 100644
index 000000000000..4b9bc9a62504
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtbe.rst
@@ -0,0 +1,70 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_B14:
+
+*******************************
+V4L2_PIX_FMT_MTISP_B14 ('MTBE')
+*******************************
+
+14-bit Packed Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format, meaning all the data for a pixel lie
+next to each other with no padding in memory, with a depth of 14 bits per pixel.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Byte Order.**
+Each cell is one byte.
+
+pixels cross the byte boundary and have a ratio of 7 bytes for each 4 pixels.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00low bits 7--0`
+      - G\ :sub:`01low bits 1--0`\ (bits 7--6) B\ :sub:`00high bits 13--8`\ (bits 5--0)
+      - G\ :sub:`01low bits 9--2`\
+      - B\ :sub:`02low bits 3--0`\ (bits 7--4) G\ :sub:`01high bits 13--10`\ (bits 3--0)
+    * - start + 4:
+      - B\ :sub:`02low bits 11--4`\
+      - G\ :sub:`03low bits 5--0`\ (bits 7--2) B\ :sub:`02high bits 13--12`\ (bits 1--0)
+      - G\ :sub:`03high bits 13--6`\
+      -
+    * - start + 8:
+      - G\ :sub:`10low bits 7--0`
+      - R\ :sub:`11low bits 1--0`\ (bits 7--6) G\ :sub:`10high bits 13--8`\ (bits 5--0)
+      - R\ :sub:`11low bits 9--2`\
+      - G\ :sub:`12low bits 3--0`\ (bits 7--4) R\ :sub:`11high bits 13--10`\ (bits 3--0)
+    * - start + 12:
+      - G\ :sub:`12low bits 11--4`\
+      - R\ :sub:`13low bits 5--0`\ (bits 7--2) G\ :sub:`12high bits 13--12`\ (bits 1--0)
+      - R\ :sub:`13high bits 13--6`\
+      -
+    * - start + 16:
+      - B\ :sub:`20low bits 7--0`
+      - G\ :sub:`21low bits 1--0`\ (bits 7--6) B\ :sub:`20high bits 13--8`\ (bits 5--0)
+      - G\ :sub:`21low bits 9--2`\
+      - B\ :sub:`22low bits 3--0`\ (bits 7--4) G\ :sub:`21high bits 13--10`\ (bits 3--0)
+    * - start + 20:
+      - B\ :sub:`22low bits 11--4`\
+      - G\ :sub:`23low bits 5--0`\ (bits 7--2) B\ :sub:`22high bits 13--12`\ (bits 1--0)
+      - G\ :sub:`23high bits 13--6`\
+      -
+    * - start + 24:
+      - G\ :sub:`30low bits 7--0`
+      - R\ :sub:`31low bits 1--0`\ (bits 7--6) G\ :sub:`30high bits 13--8`\ (bits 5--0)
+      - R\ :sub:`31low bits 9--2`\
+      - G\ :sub:`32low bits 3--0`\ (bits 7--4) R\ :sub:`31high bits 13--10`\ (bits 3--0)
+    * - start + 28:
+      - G\ :sub:`32low bits 11--4`\
+      - R\ :sub:`33low bits 5--0`\ (bits 7--2) G\ :sub:`32high bits 13--12`\ (bits 1--0)
+      - R\ :sub:`33high bits 13--6`\
+      -
\ No newline at end of file
diff --git a/Documentation/media/uapi/v4l/pixfmt-mtf8.rst b/Documentation/media/uapi/v4l/pixfmt-mtf8.rst
new file mode 100644
index 000000000000..51c9ddc4e20d
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtf8.rst
@@ -0,0 +1,75 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_F8:
+
+*******************************
+V4L2_PIX_FMT_MTISP_F8 ('MTF8')
+*******************************
+
+8-bit Packed Full-G Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format with a depth of 8 bits per pixel.
+Full-G means 1 more pixel for green channel every 2 pixels.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Bit-packed representation.**
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - B\ :sub:`00`
+      - FG\ :sub:`01`
+      - G\ :sub:`02`
+      - B\ :sub:`03`
+      - FG\ :sub:`04`
+      - G\ :sub:`05`
+    * - G\ :sub:`10`
+      - R\ :sub:`11`
+      - FG\ :sub:`12`
+      - G\ :sub:`13`
+      - R\ :sub:`14`
+      - FG\ :sub:`15`
+
+**Byte Order.**
+Each cell is one byte.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00`
+      - FG\ :sub:`01`
+      - G\ :sub:`02`
+      - B\ :sub:`03`
+      - FG\ :sub:`04`
+      - G\ :sub:`05`
+    * - start + 6:
+      - G\ :sub:`10`
+      - R\ :sub:`11`
+      - FG\ :sub:`12`
+      - G\ :sub:`13`
+      - R\ :sub:`14`
+      - FG\ :sub:`15`
+    * - start + 12:
+      - B\ :sub:`20`
+      - FG\ :sub:`21`
+      - G\ :sub:`22`
+      - B\ :sub:`23`
+      - FG\ :sub:`24`
+      - G\ :sub:`25`
+    * - start + 18:
+      - G\ :sub:`30`
+      - R\ :sub:`31`
+      - FG\ :sub:`32`
+      - G\ :sub:`33`
+      - R\ :sub:`34`
+      - FG\ :sub:`35`
\ No newline at end of file
diff --git a/Documentation/media/uapi/v4l/pixfmt-mtfa.rst b/Documentation/media/uapi/v4l/pixfmt-mtfa.rst
new file mode 100644
index 000000000000..68421c44f5e7
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtfa.rst
@@ -0,0 +1,87 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_F10:
+
+*******************************
+V4L2_PIX_FMT_MTISP_F10 ('MTFA')
+*******************************
+
+10-bit Packed Full-G Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format with a depth of 10 bits per pixel.
+Full-G means 1 more pixel for green channel every 2 pixels.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Bit-packed representation.**
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - B\ :sub:`00`
+      - FG\ :sub:`01`
+      - G\ :sub:`02`
+      - B\ :sub:`03`
+      - FG\ :sub:`04`
+      - G\ :sub:`05`
+    * - G\ :sub:`10`
+      - R\ :sub:`11`
+      - FG\ :sub:`12`
+      - G\ :sub:`13`
+      - R\ :sub:`14`
+      - FG\ :sub:`15`
+
+**Byte Order.**
+Each cell is one byte.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00low bits 7--0`
+      - FG\ :sub:`01low bits 5--0`\ (bits 7--2) B\ :sub:`00high bits 9--8`\ (bits 1--0)
+      - G\ :sub:`02low bits 3--0`\ (bits 7--4) FG\ :sub:`01high bits 9--6`\ (bits 3--0)
+      - B\ :sub:`03low bits 1--0`\ (bits 7--6) G\ :sub:`02high bits 9--4`\ (bits 5--0)
+    * - start + 4:
+      - B\ :sub:`03high bits 9--2`
+      - FG\ :sub:`04low bits 7--0`
+      - G\ :sub:`05low bits 5--0`\ (bits 7--2) FG\ :sub:`04high bits 9--8`\ (bits 1--0)
+      - G\ :sub:`05high bits 3--0`
+    * - start + 8:
+      - G\ :sub:`10low bits 7--0`
+      - R\ :sub:`11low bits 5--0`\ (bits 7--2) G\ :sub:`10high bits 9--8`\ (bits 1--0)
+      - FG\ :sub:`12low bits 3--0`\ (bits 7--4) R\ :sub:`11high bits 9--6`\ (bits 3--0)
+      - G\ :sub:`13low bits 1--0`\ (bits 7--6) FG\ :sub:`12high bits 9--4`\ (bits 5--0)
+    * - start + 12:
+      - G\ :sub:`13high bits 9--2`
+      - R\ :sub:`14low bits 7--0`
+      - FG\ :sub:`15low bits 5--0`\ (bits 7--2) R\ :sub:`14high bits 9--8`\ (bits 1--0)
+      - FG\ :sub:`15high bits 3--0`
+    * - start + 16:
+      - B\ :sub:`20low bits 7--0`
+      - FG\ :sub:`21low bits 5--0`\ (bits 7--2) B\ :sub:`20high bits 9--8`\ (bits 1--0)
+      - G\ :sub:`22low bits 3--0`\ (bits 7--4) FG\ :sub:`21high bits 9--6`\ (bits 3--0)
+      - B\ :sub:`23low bits 1--0`\ (bits 7--6) G\ :sub:`22high bits 9--4`\ (bits 5--0)
+    * - start + 20:
+      - B\ :sub:`23high bits 9--2`
+      - FG\ :sub:`24low bits 7--0`
+      - G\ :sub:`25low bits 5--0`\ (bits 7--2) FG\ :sub:`24high bits 9--8`\ (bits 1--0)
+      - G\ :sub:`25high bits 3--0`
+    * - start + 24:
+      - G\ :sub:`30low bits 7--0`
+      - R\ :sub:`31low bits 5--0`\ (bits 7--2) G\ :sub:`30high bits 9--8`\ (bits 1--0)
+      - FG\ :sub:`32low bits 3--0`\ (bits 7--4) R\ :sub:`31high bits 9--6`\ (bits 3--0)
+      - G\ :sub:`33low bits 1--0`\ (bits 7--6) FG\ :sub:`32high bits 9--4`\ (bits 5--0)
+    * - start + 28:
+      - G\ :sub:`33high bits 9--2`
+      - R\ :sub:`34low bits 7--0`
+      - FG\ :sub:`35low bits 5--0`\ (bits 7--2) R\ :sub:`34high bits 9--8`\ (bits 1--0)
+      - FG\ :sub:`35high bits 3--0`
\ No newline at end of file
diff --git a/Documentation/media/uapi/v4l/pixfmt-mtfc.rst b/Documentation/media/uapi/v4l/pixfmt-mtfc.rst
new file mode 100644
index 000000000000..a3535f5435fa
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtfc.rst
@@ -0,0 +1,107 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_F12:
+
+*******************************
+V4L2_PIX_FMT_MTISP_F12 ('MTFC')
+*******************************
+
+12-bit Packed Full-G Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format with a depth of 12 bits per pixel.
+Full-G means 1 more pixel for green channel every 2 pixels.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Bit-packed representation.**
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - B\ :sub:`00`
+      - FG\ :sub:`01`
+      - G\ :sub:`02`
+      - B\ :sub:`03`
+      - FG\ :sub:`04`
+      - G\ :sub:`05`
+    * - G\ :sub:`10`
+      - R\ :sub:`11`
+      - FG\ :sub:`12`
+      - G\ :sub:`13`
+      - R\ :sub:`14`
+      - FG\ :sub:`15`
+
+**Byte Order.**
+Each cell is one byte.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00low bits 7--0`
+      - FG\ :sub:`01low bits 3--0`\ (bits 7--4) B\ :sub:`00high bits 11--8`\ (bits 3--0)
+    * - start + 2:
+      - FG\ :sub:`01high bits 7--0`
+      - G\ :sub:`02low bits 7--0`
+    * - start + 4:
+      - B\ :sub:`03low bits 3--0`\ (bits 7--4) G\ :sub:`02high bits 11--8`\ (bits 3--0)
+      - B\ :sub:`03high bits 7--0`
+    * - start + 6:
+      - FG\ :sub:`04low bits 7--0`
+      - G\ :sub:`05low bits 3--0`\ (bits 7--4) FG\ :sub:`04high bits 11--8`\ (bits 3--0)
+    * - start + 8:
+      - G\ :sub:`05high bits 7--0`
+      -
+    * - start + 10:
+      - G\ :sub:`10low bits 7--0`
+      - R\ :sub:`11low bits 3--0`\ (bits 7--4) G\ :sub:`10high bits 11--8`\ (bits 3--0)
+    * - start + 12:
+      - R\ :sub:`11high bits 7--0`
+      - FG\ :sub:`12low bits 7--0`
+    * - start + 14:
+      - G\ :sub:`13low bits 3--0`\ (bits 7--4) FG\ :sub:`12high bits 11--8`\ (bits 3--0)
+      - G\ :sub:`13high bits 7--0`
+    * - start + 16:
+      - R\ :sub:`14low bits 7--0`
+      - FG\ :sub:`15low bits 3--0`\ (bits 7--4) R\ :sub:`14high bits 11--8`\ (bits 3--0)
+    * - start + 18:
+      - FG\ :sub:`15high bits 7--0`
+      -
+    * - start + 20:
+      - B\ :sub:`20low bits 7--0`
+      - FG\ :sub:`21low bits 3--0`\ (bits 7--4) B\ :sub:`20high bits 11--8`\ (bits 3--0)
+    * - start + 22:
+      - FG\ :sub:`21high bits 7--0`
+      - G\ :sub:`22low bits 7--0`
+    * - start + 24:
+      - B\ :sub:`23low bits 3--0`\ (bits 7--4) G\ :sub:`22high bits 11--8`\ (bits 3--0)
+      - B\ :sub:`23high bits 7--0`
+    * - start + 26:
+      - FG\ :sub:`24low bits 7--0`
+      - G\ :sub:`25low bits 3--0`\ (bits 7--4) FG\ :sub:`24high bits 11--8`\ (bits 3--0)
+    * - start + 28:
+      - G\ :sub:`25high bits 7--0`
+      -
+    * - start + 30:
+      - G\ :sub:`30low bits 7--0`
+      - R\ :sub:`31low bits 3--0`\ (bits 7--4) G\ :sub:`30high bits 11--8`\ (bits 3--0)
+    * - start + 32:
+      - R\ :sub:`31high bits 7--0`
+      - FG\ :sub:`32low bits 7--0`
+    * - start + 34:
+      - G\ :sub:`33low bits 3--0`\ (bits 7--4) FG\ :sub:`32high bits 11--8`\ (bits 3--0)
+      - G\ :sub:`33high bits 7--0`
+    * - start + 36:
+      - R\ :sub:`34low bits 7--0`
+      - FG\ :sub:`35low bits 3--0`\ (bits 7--4) R\ :sub:`34high bits 11--8`\ (bits 3--0)
+    * - start + 38:
+      - FG\ :sub:`35high bits 7--0`
+      -
\ No newline at end of file
diff --git a/Documentation/media/uapi/v4l/pixfmt-mtfe.rst b/Documentation/media/uapi/v4l/pixfmt-mtfe.rst
new file mode 100644
index 000000000000..324a258e897f
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-mtfe.rst
@@ -0,0 +1,107 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _V4L2-PIX-FMT-MTISP_F14:
+
+*******************************
+V4L2_PIX_FMT_MTISP_F14 ('MTFE')
+*******************************
+
+14-bit Packed Full-G Bayer formats.
+
+
+Description
+===========
+
+The four pixel formats are used by Mediatek ISP.
+This is a packed format with a depth of 14 bits per pixel.
+Full-G means 1 more pixel for green channel every 2 pixels.
+The least significant byte is stored at lower memory addresses (little-endian).
+The RGB byte order follows raw sRGB / Bayer format from sensor.
+Below is an example of conventional RGB byte order BGGR.
+
+**Bit-packed representation.**
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - B\ :sub:`00`
+      - FG\ :sub:`01`
+      - G\ :sub:`02`
+      - B\ :sub:`03`
+      - FG\ :sub:`04`
+      - G\ :sub:`05`
+    * - G\ :sub:`10`
+      - R\ :sub:`11`
+      - FG\ :sub:`12`
+      - G\ :sub:`13`
+      - R\ :sub:`14`
+      - FG\ :sub:`15`
+
+**Byte Order.**
+Each cell is one byte.
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+
+    * - start + 0:
+      - B\ :sub:`00low bits 7--0`
+      - FG\ :sub:`01low bits 1--0`\ (bits 7--6) B\ :sub:`00high bits 13--8`\ (bits 5--0)
+      - FG\ :sub:`01low bits 9--2`
+      - G\ :sub:`02low bits 3--0`\ (bits 7--4) FG\ :sub:`01high bits 13--10`\ (bits 3--0)
+    * - start + 4:
+      - G\ :sub:`02low bits 11--4`
+      - B\ :sub:`03low bits 5--0`\ (bits 7--2) G\ :sub:`02high bits 13--12`\ (bits 1--0)
+      - B\ :sub:`03high bits 13--6`
+      - FG\ :sub:`04low bits 7--0`
+    * - start + 8:
+      - G\ :sub:`05low bits 1--0`\ (bits 7--6) FG\ :sub:`04high bits 13--8`\ (bits 5--0)
+      - G\ :sub:`05high bits 9--2`
+      - G\ :sub:`05high bits 13--10`
+      -
+    * - start + 12:
+      - G\ :sub:`10low bits 7--0`
+      - R\ :sub:`11low bits 1--0`\ (bits 7--6) G\ :sub:`10high bits 13--8`\ (bits 5--0)
+      - R\ :sub:`11low bits 9--2`
+      - FG\ :sub:`12low bits 3--0`\ (bits 7--4) R\ :sub:`11high bits 13--10`\ (bits 3--0)
+    * - start + 16:
+      - FG\ :sub:`12low bits 11--4`
+      - G\ :sub:`13low bits 5--0`\ (bits 7--2) FG\ :sub:`12high bits 13--12`\ (bits 1--0)
+      - G\ :sub:`13high bits 13--6`
+      - R\ :sub:`14low bits 7--0`
+    * - start + 20:
+      - FG\ :sub:`15low bits 1--0`\ (bits 7--6) R\ :sub:`14high bits 13--8`\ (bits 5--0)
+      - FG\ :sub:`15high bits 9--2`
+      - FG\ :sub:`15high bits 13--10`
+      -
+    * - start + 24:
+      - B\ :sub:`20low bits 7--0`
+      - FG\ :sub:`21low bits 1--0`\ (bits 7--6) B\ :sub:`20high bits 13--8`\ (bits 5--0)
+      - FG\ :sub:`21low bits 9--2`
+      - G\ :sub:`22low bits 3--0`\ (bits 7--4) FG\ :sub:`21high bits 13--10`\ (bits 3--0)
+    * - start + 28:
+      - G\ :sub:`22low bits 11--4`
+      - B\ :sub:`23low bits 5--0`\ (bits 7--2) G\ :sub:`22high bits 13--12`\ (bits 1--0)
+      - B\ :sub:`23high bits 13--6`
+      - FG\ :sub:`24low bits 7--0`
+    * - start + 32:
+      - G\ :sub:`25low bits 1--0`\ (bits 7--6) FG\ :sub:`24high bits 13--8`\ (bits 5--0)
+      - G\ :sub:`25high bits 9--2`
+      - G\ :sub:`25high bits 13--10`
+      -
+    * - start + 36:
+      - G\ :sub:`30low bits 7--0`
+      - R\ :sub:`31low bits 1--0`\ (bits 7--6) G\ :sub:`30high bits 13--8`\ (bits 5--0)
+      - R\ :sub:`31low bits 9--2`
+      - FG\ :sub:`32low bits 3--0`\ (bits 7--4) R\ :sub:`31high bits 13--10`\ (bits 3--0)
+    * - start + 40:
+      - FG\ :sub:`32low bits 11--4`
+      - G\ :sub:`33low bits 5--0`\ (bits 7--2) FG\ :sub:`32high bits 13--12`\ (bits 1--0)
+      - G\ :sub:`33high bits 13--6`
+      - R\ :sub:`34low bits 7--0`
+    * - start + 44:
+      - FG\ :sub:`35low bits 1--0`\ (bits 7--6) R\ :sub:`34high bits 13--8`\ (bits 5--0)
+      - FG\ :sub:`35high bits 9--2`
+      - FG\ :sub:`35high bits 13--10`
+      -
\ No newline at end of file
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index ac87c3e37280..2f536fedd9c4 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1297,6 +1297,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_KONICA420:	descr = "GSPCA KONICA420"; break;
 	case V4L2_PIX_FMT_HSV24:	descr = "24-bit HSV 8-8-8"; break;
 	case V4L2_PIX_FMT_HSV32:	descr = "32-bit XHSV 8-8-8-8"; break;
+	case V4L2_PIX_FMT_MTISP_B8:	descr = "8-bit Packed Bayer format"; break;
+	case V4L2_PIX_FMT_MTISP_F8:	descr = "8-bit Packed Full-G Bayer format"; break;
+	case V4L2_PIX_FMT_MTISP_B10:	descr = "10-bit Packed Bayer format"; break;
+	case V4L2_PIX_FMT_MTISP_F10:	descr = "10-bit Packed Full-G Bayer format"; break;
+	case V4L2_PIX_FMT_MTISP_B12:	descr = "12-bit Packed Bayer format"; break;
+	case V4L2_PIX_FMT_MTISP_F12:	descr = "12-bit Packed Full-G Bayer format"; break;
+	case V4L2_PIX_FMT_MTISP_B14:	descr = "14-bit Packed Bayer format"; break;
+	case V4L2_PIX_FMT_MTISP_F14:	descr = "14-bit Packed Full-G Bayer format"; break;
 	case V4L2_SDR_FMT_CU8:		descr = "Complex U8"; break;
 	case V4L2_SDR_FMT_CU16LE:	descr = "Complex U16LE"; break;
 	case V4L2_SDR_FMT_CS8:		descr = "Complex S8"; break;
@@ -1312,6 +1320,11 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_META_FMT_VSP1_HGO:	descr = "R-Car VSP1 1-D Histogram"; break;
 	case V4L2_META_FMT_VSP1_HGT:	descr = "R-Car VSP1 2-D Histogram"; break;
 	case V4L2_META_FMT_UVC:		descr = "UVC payload header metadata"; break;
+	case V4L2_META_FMT_MTISP_3A:	descr = "AE/AWB Histogram"; break;
+	case V4L2_META_FMT_MTISP_AF:	descr = "AF Histogram"; break;
+	case V4L2_META_FMT_MTISP_LCS:	descr = "Local contrast enhanced statistics"; break;
+	case V4L2_META_FMT_MTISP_LMV:	descr = "Local motion vector Histogram"; break;
+	case V4L2_META_FMT_MTISP_PARAMS: descr = "MTK ISP tuning metadata"; break;
 
 	default:
 		/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 1050a75fb7ef..ef51911fcfe4 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -728,6 +728,16 @@ struct v4l2_pix_format {
 #define V4L2_PIX_FMT_IPU3_SGRBG10	v4l2_fourcc('i', 'p', '3', 'G') /* IPU3 packed 10-bit GRBG bayer */
 #define V4L2_PIX_FMT_IPU3_SRGGB10	v4l2_fourcc('i', 'p', '3', 'r') /* IPU3 packed 10-bit RGGB bayer */
 
+/* Vendor specific - Mediatek ISP bayer formats */
+#define V4L2_PIX_FMT_MTISP_B8	v4l2_fourcc('M', 'T', 'B', '8') /* Packed bayer format,  8-bit */
+#define V4L2_PIX_FMT_MTISP_B10	v4l2_fourcc('M', 'T', 'B', 'A') /* Packed bayer format, 10-bit */
+#define V4L2_PIX_FMT_MTISP_B12	v4l2_fourcc('M', 'T', 'B', 'C') /* Packed bayer format, 12-bit */
+#define V4L2_PIX_FMT_MTISP_B14	v4l2_fourcc('M', 'T', 'B', 'E') /* Packed bayer format, 14-bit */
+#define V4L2_PIX_FMT_MTISP_F8	v4l2_fourcc('M', 'T', 'F', '8') /* Full-G bayer format,  8-bit */
+#define V4L2_PIX_FMT_MTISP_F10	v4l2_fourcc('M', 'T', 'F', 'A') /* Full-G bayer format, 10-bit */
+#define V4L2_PIX_FMT_MTISP_F12	v4l2_fourcc('M', 'T', 'F', 'C') /* Full-G bayer format, 12-bit */
+#define V4L2_PIX_FMT_MTISP_F14	v4l2_fourcc('M', 'T', 'F', 'E') /* Full-G bayer format, 14-bit */
+
 /* SDR formats - used only for Software Defined Radio devices */
 #define V4L2_SDR_FMT_CU8          v4l2_fourcc('C', 'U', '0', '8') /* IQ u8 */
 #define V4L2_SDR_FMT_CU16LE       v4l2_fourcc('C', 'U', '1', '6') /* IQ u16le */
@@ -750,6 +760,13 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_UVC         v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
 #define V4L2_META_FMT_D4XX        v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
 
+/* Vendor specific - Mediatek ISP parameters for firmware */
+#define V4L2_META_FMT_MTISP_PARAMS	v4l2_fourcc('M', 'T', 'f', 'p') /* ISP tuning parameters */
+#define V4L2_META_FMT_MTISP_3A		v4l2_fourcc('M', 'T', 'f', 'a') /* AE/AWB histogram */
+#define V4L2_META_FMT_MTISP_AF		v4l2_fourcc('M', 'T', 'f', 'f') /* AF histogram */
+#define V4L2_META_FMT_MTISP_LCS	v4l2_fourcc('M', 'T', 'f', 'c') /* Local contrast enhanced statistics */
+#define V4L2_META_FMT_MTISP_LMV	v4l2_fourcc('M', 'T', 'f', 'm') /* Local motion vector histogram */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 5/9] media: platform: Add Mediatek ISP P1 V4L2 control
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
                   ` (3 preceding siblings ...)
  2019-06-11  3:53 ` [RFC,v3 4/9] media: platform: Add Mediatek ISP P1 image & meta formats Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-07-01  5:50   ` Tomasz Figa
  2019-06-11  3:53 ` [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions Jungo Lin
                   ` (3 subsequent siblings)
  8 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

Reserved Mediatek ISP P1 V4L2 control number with 16.
Moreover, add two V4L2 controls for ISP P1 user space
usage.

1. V4L2_CID_MTK_GET_BIN_INFO
- Provide the image output width & height in case
camera binning mode is enabled.

2. V4L2_CID_MTK_RAW_PATH
- Export the path control of the main stream to user space.
One is pure raw and the other is processing raw.
The default value is 0 which outputs the pure raw bayer image
from sesnor, without image processing in ISP HW.

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
 drivers/media/platform/mtk-isp/Makefile       |   3 +
 .../media/platform/mtk-isp/isp_50/Makefile    |   5 +
 .../platform/mtk-isp/isp_50/cam/Makefile      |   5 +
 .../mtk-isp/isp_50/cam/mtk_cam-ctrl.c         | 138 ++++++++++++++++++
 .../mtk-isp/isp_50/cam/mtk_cam-ctrl.h         |  38 +++++
 include/uapi/linux/v4l2-controls.h            |   4 +
 6 files changed, 193 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.h

diff --git a/drivers/media/platform/mtk-isp/Makefile b/drivers/media/platform/mtk-isp/Makefile
new file mode 100644
index 000000000000..c17fb3fc3340
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-y += isp_50/
diff --git a/drivers/media/platform/mtk-isp/isp_50/Makefile b/drivers/media/platform/mtk-isp/isp_50/Makefile
new file mode 100644
index 000000000000..8498fe70e418
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ifeq ($(CONFIG_VIDEO_MEDIATEK_ISP_PASS1),y)
+obj-y += cam/
+endif
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
new file mode 100644
index 000000000000..53fb69d3add6
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+mtk-cam-isp-objs += mtk_cam-ctrl.o
+
+obj-$(CONFIG_VIDEO_MEDIATEK_ISP_PASS1) += mtk-cam-isp.o
\ No newline at end of file
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.c
new file mode 100644
index 000000000000..31d801c82495
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.c
@@ -0,0 +1,138 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+
+#include <linux/device.h>
+#include <linux/platform_device.h>
+
+#include "mtk_cam-ctrl.h"
+#include "mtk_cam.h"
+
+static int handle_ctrl_get_bin_info(struct v4l2_ctrl *ctrl, int is_width)
+{
+	struct mtk_cam_dev *cam_dev = ctrl->priv;
+	struct v4l2_format *fmt;
+
+	fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_MAIN_STREAM_OUT].vdev_fmt;
+
+	dev_dbg(&cam_dev->pdev->dev, "Get bin info w*h:%d*%d is_width:%d",
+		fmt->fmt.pix_mp.width, fmt->fmt.pix_mp.height, is_width);
+
+	if (is_width)
+		ctrl->val = fmt->fmt.pix_mp.width;
+	else
+		ctrl->val = fmt->fmt.pix_mp.height;
+
+	return 0;
+}
+
+static int handle_ctrl_get_process_raw(struct v4l2_ctrl *ctrl)
+{
+	struct mtk_cam_dev *cam_dev = ctrl->priv;
+	struct isp_p1_device *p1_dev = get_p1_device(&cam_dev->pdev->dev);
+
+	ctrl->val = (p1_dev->isp_ctx.isp_raw_path == ISP_PROCESS_RAW_PATH);
+
+	dev_dbg(&cam_dev->pdev->dev, "Get process raw:%d", ctrl->val);
+
+	return 0;
+}
+
+static int handle_ctrl_set_process_raw(struct v4l2_ctrl *ctrl)
+{
+	struct mtk_cam_dev *cam_dev = ctrl->priv;
+	struct isp_p1_device *p1_dev = get_p1_device(&cam_dev->pdev->dev);
+
+	p1_dev->isp_ctx.isp_raw_path = (ctrl->val) ?
+		ISP_PROCESS_RAW_PATH : ISP_PURE_RAW_PATH;
+	dev_dbg(&cam_dev->pdev->dev, "Set process raw:%d", ctrl->val);
+	return 0;
+}
+
+static int mtk_cam_dev_g_ctrl(struct v4l2_ctrl *ctrl)
+{
+	switch (ctrl->id) {
+	case V4L2_CID_MTK_PROCESSING_RAW:
+		handle_ctrl_get_process_raw(ctrl);
+		break;
+	case V4L2_CID_MTK_GET_BIN_WIDTH:
+		handle_ctrl_get_bin_info(ctrl, 1);
+		break;
+	case V4L2_CID_MTK_GET_BIN_HEIGTH:
+		handle_ctrl_get_bin_info(ctrl, 0);
+		break;
+	default:
+		return -EINVAL;
+	}
+	return 0;
+}
+
+static int mtk_cam_dev_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	switch (ctrl->id) {
+	case V4L2_CID_MTK_PROCESSING_RAW:
+		return handle_ctrl_set_process_raw(ctrl);
+	default:
+		return -EINVAL;
+	}
+}
+
+static const struct v4l2_ctrl_ops mtk_cam_dev_ctrl_ops = {
+	.g_volatile_ctrl = mtk_cam_dev_g_ctrl,
+	.s_ctrl = mtk_cam_dev_s_ctrl,
+};
+
+struct v4l2_ctrl_config mtk_cam_controls[] = {
+	{
+	.ops = &mtk_cam_dev_ctrl_ops,
+	.id = V4L2_CID_MTK_PROCESSING_RAW,
+	.name = "MTK CAM PROCESSING RAW",
+	.type = V4L2_CTRL_TYPE_BOOLEAN,
+	.min = 0,
+	.max = 1,
+	.step = 1,
+	.def = 1,
+	},
+	{
+	.ops = &mtk_cam_dev_ctrl_ops,
+	.id = V4L2_CID_MTK_GET_BIN_WIDTH,
+	.name = "MTK CAM GET BIN WIDTH",
+	.type = V4L2_CTRL_TYPE_INTEGER,
+	.min = IMG_MIN_WIDTH,
+	.max = IMG_MAX_WIDTH,
+	.step = 1,
+	.def = IMG_MAX_WIDTH,
+	.flags = V4L2_CTRL_FLAG_READ_ONLY | V4L2_CTRL_FLAG_VOLATILE,
+	},
+	{
+	.ops = &mtk_cam_dev_ctrl_ops,
+	.id = V4L2_CID_MTK_GET_BIN_HEIGTH,
+	.name = "MTK CAM GET BIN HEIGHT",
+	.type = V4L2_CTRL_TYPE_INTEGER,
+	.min = IMG_MIN_HEIGHT,
+	.max = IMG_MAX_HEIGHT,
+	.step = 1,
+	.def = IMG_MAX_HEIGHT,
+	.flags = V4L2_CTRL_FLAG_READ_ONLY | V4L2_CTRL_FLAG_VOLATILE,
+	},
+};
+
+int mtk_cam_ctrl_init(struct mtk_cam_dev *cam_dev,
+		      struct v4l2_ctrl_handler *hdl)
+{
+	unsigned int i;
+
+	/* Initialized HW controls, allow V4L2_CID_MTK_CAM_MAX ctrls */
+	v4l2_ctrl_handler_init(hdl, V4L2_CID_MTK_CAM_MAX);
+	if (hdl->error) {
+		v4l2_ctrl_handler_free(hdl);
+		return hdl->error;
+	}
+
+	for (i = 0; i < ARRAY_SIZE(mtk_cam_controls); i++)
+		v4l2_ctrl_new_custom(hdl, &mtk_cam_controls[i], cam_dev);
+
+	dev_dbg(&cam_dev->pdev->dev, "%s done", __func__);
+
+	return 0;
+}
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.h
new file mode 100644
index 000000000000..0f9349ae0b07
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctrl.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ */
+
+#ifndef __MTK_CAM_CTRL_H__
+#define __MTK_CAM_CTRL_H__
+
+#include <media/v4l2-ctrls.h>
+
+#include "mtk_cam-v4l2-util.h"
+
+/* The base for the MTK Camera ISP P1 driver controls.
+ * We reserve 16 controls for this driver.
+ */
+#define V4L2_CID_MTK_CAM_BASE			V4L2_CID_USER_MTK_CAM_BASE
+
+/* Control MTK ISP P1 main stream to process raw image data or not.
+ * The default value is 0 which outputs the pure raw bayer data from sensor,
+ * without image processing in ISP HW.
+ */
+#define V4L2_CID_MTK_PROCESSING_RAW		(V4L2_CID_MTK_CAM_BASE + 1)
+
+/* MTK ISP P1 HW supports frontal binning function.
+ * If this function is enabled, the 3A algo. may get the new image resolution
+ * which is binned by ISP P1. If this function is disabled or no supported,
+ * the image resolution will be equal to configured image format.
+ * For this control, it is read only.
+ */
+#define V4L2_CID_MTK_GET_BIN_WIDTH		(V4L2_CID_MTK_CAM_BASE + 2)
+#define V4L2_CID_MTK_GET_BIN_HEIGTH		(V4L2_CID_MTK_CAM_BASE + 3)
+
+#define V4L2_CID_MTK_CAM_MAX			16
+
+int mtk_cam_ctrl_init(struct mtk_cam_dev *cam_dev,
+		      struct v4l2_ctrl_handler *hdl);
+
+#endif /* __MTK_CAM_CTRL_H__ */
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index 37807f23231e..2db99716f40d 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -192,6 +192,10 @@ enum v4l2_colorfx {
  * We reserve 16 controls for this driver. */
 #define V4L2_CID_USER_IMX_BASE			(V4L2_CID_USER_BASE + 0x10b0)
 
+/* The base for the mediatek ISP Pass 1 driver controls */
+/* We reserve 16 controls for this driver. */
+#define V4L2_CID_USER_MTK_CAM_BASE		(V4L2_CID_USER_BASE + 0x10c0)
+
 /* MPEG-class control IDs */
 /* The MPEG controls are applicable to all codec controls
  * and the 'MPEG' part of the define is historical */
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
                   ` (4 preceding siblings ...)
  2019-06-11  3:53 ` [RFC,v3 5/9] media: platform: Add Mediatek ISP P1 V4L2 control Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-07-10  9:54   ` Tomasz Figa
  2019-06-11  3:53 ` [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver Jungo Lin
                   ` (2 subsequent siblings)
  8 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

Implement standard V4L2 video driver that utilizes V4L2
and media framework APIs. In this driver, supports one media
device, one sub-device and seven video devices during
initialization. Moreover, it also connects with sensor and
seninf drivers with V4L2 async APIs.

(The current metadata interface used in meta input and partial
meta nodes is only a temporary solution to kick off the driver
development and is not ready to be reviewed yet.)

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
This patch depends on "media: support Mediatek sensor interface driver"[1].

ISP P1 sub-device communicates with seninf sub-device with CIO.

[1]. media: support Mediatek sensor interface driver
https://patchwork.kernel.org/cover/10979135/
---
 .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c    | 1674 +++++++++++++++++
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h    |  173 ++
 3 files changed, 1848 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h

diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
index 53fb69d3add6..7558593e63f0 100644
--- a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 
 mtk-cam-isp-objs += mtk_cam-ctrl.o
+mtk-cam-isp-objs += mtk_cam-v4l2-util.o
 
 obj-$(CONFIG_VIDEO_MEDIATEK_ISP_PASS1) += mtk-cam-isp.o
\ No newline at end of file
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
new file mode 100644
index 000000000000..117398ed29d2
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
@@ -0,0 +1,1674 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018 Mediatek Corporation.
+ * Copyright (c) 2017 Intel Corporation.
+ *
+ * MTK_CAM-v4l2-util is highly based on Intel IPU3 ImgU driver.
+ *
+ */
+
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/videodev2.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/of_platform.h>
+#include <media/media-entity.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-common.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-fwnode.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-mc.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "mtk_cam.h"
+#include "mtk_cam-ctrl.h"
+#include "mtk_cam-smem.h"
+#include "mtk_cam-v4l2-util.h"
+
+#define MTK_CAM_CIO_PAD_SRC			4
+#define MTK_CAM_CIO_PAD_SINK			11
+
+static inline struct mtk_cam_video_device *
+file_to_mtk_cam_node(struct file *__file)
+{
+	return container_of(video_devdata(__file),
+		struct mtk_cam_video_device, vdev);
+}
+
+static inline struct mtk_cam_dev *
+mtk_cam_subdev_to_dev(struct v4l2_subdev *__sd)
+{
+	return container_of(__sd,
+		struct mtk_cam_dev, subdev);
+}
+
+static inline struct mtk_cam_dev *
+mtk_cam_mdev_to_dev(struct media_device *__mdev)
+{
+	return container_of(__mdev,
+		struct mtk_cam_dev, media_dev);
+}
+
+static inline struct mtk_cam_video_device *
+mtk_cam_vbq_to_vdev(struct vb2_queue *__vq)
+{
+	return container_of(__vq,
+		struct mtk_cam_video_device, vbq);
+}
+
+static inline struct mtk_cam_dev_request *
+mtk_cam_req_to_dev_req(struct media_request *__req)
+{
+	return container_of(__req,
+		struct mtk_cam_dev_request, req);
+}
+
+static inline struct mtk_cam_dev_buffer *
+mtk_cam_vb2_buf_to_dev_buf(struct vb2_buffer *__vb)
+{
+	return container_of(__vb,
+		struct mtk_cam_dev_buffer, vbb.vb2_buf);
+}
+
+static void mtk_cam_req_try_isp_queue(struct mtk_cam_dev *cam_dev,
+				      struct media_request *new_req)
+{
+	struct mtk_cam_dev_request *req, *req_safe, *cam_dev_req;
+	struct device *dev = &cam_dev->pdev->dev;
+
+	dev_dbg(dev, "%s new req:%d", __func__, !new_req);
+
+	if (!cam_dev->streaming) {
+		cam_dev_req = mtk_cam_req_to_dev_req(new_req);
+		spin_lock(&cam_dev->req_lock);
+		list_add_tail(&cam_dev_req->list, &cam_dev->req_list);
+		spin_unlock(&cam_dev->req_lock);
+		dev_dbg(dev, "%s: stream off, no ISP enqueue\n", __func__);
+		return;
+	}
+
+	/* Normal enqueue flow */
+	if (new_req) {
+		mtk_isp_req_enqueue(dev, new_req);
+		return;
+	}
+
+	/* Flush all media requests wehen first stream on */
+	list_for_each_entry_safe(req, req_safe, &cam_dev->req_list, list) {
+		list_del(&req->list);
+		mtk_isp_req_enqueue(dev, &req->req);
+	}
+}
+
+static void mtk_cam_req_queue(struct media_request *req)
+{
+	struct mtk_cam_dev *cam_dev = mtk_cam_mdev_to_dev(req->mdev);
+
+	vb2_request_queue(req);
+	mtk_cam_req_try_isp_queue(cam_dev, req);
+}
+
+static struct media_request *mtk_cam_req_alloc(struct media_device *mdev)
+{
+	struct mtk_cam_dev_request *cam_dev_req;
+
+	cam_dev_req = kzalloc(sizeof(*cam_dev_req), GFP_KERNEL);
+
+	return &cam_dev_req->req;
+}
+
+static void mtk_cam_req_free(struct media_request *req)
+{
+	struct mtk_cam_dev_request *cam_dev_req = mtk_cam_req_to_dev_req(req);
+
+	kfree(cam_dev_req);
+}
+
+static __u32 img_get_pixel_byte_by_fmt(__u32 pix_fmt)
+{
+	switch (pix_fmt) {
+	case V4L2_PIX_FMT_MTISP_B8:
+	case V4L2_PIX_FMT_MTISP_F8:
+		return 8;
+	case V4L2_PIX_FMT_MTISP_B10:
+	case V4L2_PIX_FMT_MTISP_F10:
+		return 10;
+	case V4L2_PIX_FMT_MTISP_B12:
+	case V4L2_PIX_FMT_MTISP_F12:
+		return 12;
+	case V4L2_PIX_FMT_MTISP_B14:
+	case V4L2_PIX_FMT_MTISP_F14:
+		return 14;
+	default:
+		return 0;
+	}
+}
+
+static __u32 img_cal_main_stream_stride(struct device *dev, __u32 width,
+					__u32 pix_fmt)
+{
+	__u32 stride;
+	__u32 pixel_byte = img_get_pixel_byte_by_fmt(pix_fmt);
+
+	width = ALIGN(width, 4);
+	stride = ALIGN(DIV_ROUND_UP(width * pixel_byte, 8), 2);
+
+	dev_dbg(dev, "main width:%d, stride:%d\n", width, stride);
+
+	return stride;
+}
+
+static __u32 img_cal_packed_out_stride(struct device *dev, __u32 width,
+				       __u32 pix_fmt)
+{
+	__u32 stride;
+	__u32 pixel_byte = img_get_pixel_byte_by_fmt(pix_fmt);
+
+	width = ALIGN(width, 4);
+	stride = DIV_ROUND_UP(width * 3, 2);
+	stride = DIV_ROUND_UP(stride * pixel_byte, 8);
+
+	if (pix_fmt == V4L2_PIX_FMT_MTISP_F10)
+		stride = ALIGN(stride, 4);
+
+	dev_dbg(dev, "packed width:%d, stride:%d\n", width, stride);
+
+	return stride;
+}
+
+static __u32 img_cal_stride(struct device *dev,
+			    int node_id,
+			    __u32 width,
+			    __u32 pix_fmt)
+{
+	__u32 bpl;
+
+	/* Currently, only support one_pixel_mode */
+	if (node_id == MTK_CAM_P1_MAIN_STREAM_OUT)
+		bpl = img_cal_main_stream_stride(dev, width, pix_fmt);
+	else if (node_id == MTK_CAM_P1_PACKED_BIN_OUT)
+		bpl = img_cal_packed_out_stride(dev, width, pix_fmt);
+
+	/* For DIP HW constrained, it needs 4 byte alignment */
+	bpl = ALIGN(bpl, 4);
+
+	return bpl;
+}
+
+static const struct v4l2_format *
+mtk_cam_dev_find_fmt(struct mtk_cam_dev_node_desc *desc, u32 format)
+{
+	unsigned int i;
+	const struct v4l2_format *dev_fmt;
+
+	for (i = 0; i < desc->num_fmts; i++) {
+		dev_fmt = &desc->fmts[i];
+		if (dev_fmt->fmt.pix_mp.pixelformat == format)
+			return dev_fmt;
+	}
+
+	return NULL;
+}
+
+/* Calcuate mplane pix format */
+static void
+mtk_cam_dev_cal_mplane_fmt(struct device *dev,
+			   struct v4l2_pix_format_mplane *dest_fmt,
+			   unsigned int node_id)
+{
+	unsigned int i;
+	__u32 bpl, sizeimage, imagsize;
+
+	imagsize = 0;
+	for (i = 0 ; i < dest_fmt->num_planes; ++i) {
+		bpl = img_cal_stride(dev,
+				     node_id,
+				     dest_fmt->width,
+				     dest_fmt->pixelformat);
+		sizeimage = bpl * dest_fmt->height;
+		imagsize += sizeimage;
+		dest_fmt->plane_fmt[i].bytesperline = bpl;
+		dest_fmt->plane_fmt[i].sizeimage = sizeimage;
+		memset(dest_fmt->plane_fmt[i].reserved,
+		       0, sizeof(dest_fmt->plane_fmt[i].reserved));
+		dev_dbg(dev, "plane:%d,bpl:%d,sizeimage:%u\n",
+			i,  bpl, dest_fmt->plane_fmt[i].sizeimage);
+	}
+
+	if (dest_fmt->num_planes == 1)
+		dest_fmt->plane_fmt[0].sizeimage = imagsize;
+}
+
+static void
+mtk_cam_dev_set_img_fmt(struct device *dev,
+			struct v4l2_pix_format_mplane *dest_fmt,
+			const struct v4l2_pix_format_mplane *src_fmt,
+			unsigned int node_id)
+{
+	dest_fmt->width = src_fmt->width;
+	dest_fmt->height = src_fmt->height;
+	dest_fmt->pixelformat = src_fmt->pixelformat;
+	dest_fmt->field = src_fmt->field;
+	dest_fmt->colorspace = src_fmt->colorspace;
+	dest_fmt->num_planes = src_fmt->num_planes;
+	/* Use default */
+	dest_fmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+	dest_fmt->quantization = V4L2_QUANTIZATION_DEFAULT;
+	dest_fmt->xfer_func =
+		V4L2_MAP_XFER_FUNC_DEFAULT(dest_fmt->colorspace);
+	memset(dest_fmt->reserved, 0, sizeof(dest_fmt->reserved));
+
+	dev_dbg(dev, "%s: Dest Fmt:%c%c%c%c, w*h:%d*%d\n",
+		__func__,
+		(dest_fmt->pixelformat & 0xFF),
+		(dest_fmt->pixelformat >> 8) & 0xFF,
+		(dest_fmt->pixelformat >> 16) & 0xFF,
+		(dest_fmt->pixelformat >> 24) & 0xFF,
+		dest_fmt->width,
+		dest_fmt->height);
+
+	mtk_cam_dev_cal_mplane_fmt(dev, dest_fmt, node_id);
+}
+
+/* Get the default format setting */
+static void
+mtk_cam_dev_load_default_fmt(struct device *dev,
+			     struct mtk_cam_dev_node_desc *queue_desc,
+			     struct v4l2_format *dest)
+{
+	const struct v4l2_format *default_fmt =
+		&queue_desc->fmts[queue_desc->default_fmt_idx];
+
+	dest->type = queue_desc->buf_type;
+
+	/* Configure default format based on node type */
+	if (queue_desc->image) {
+		mtk_cam_dev_set_img_fmt(dev,
+					&dest->fmt.pix_mp,
+					&default_fmt->fmt.pix_mp,
+					queue_desc->id);
+	} else {
+		dest->fmt.meta.dataformat = default_fmt->fmt.meta.dataformat;
+		dest->fmt.meta.buffersize = default_fmt->fmt.meta.buffersize;
+	}
+}
+
+static int mtk_cam_isp_open(struct file *file)
+{
+	struct mtk_cam_dev *cam_dev = video_drvdata(file);
+	struct device *dev = &cam_dev->pdev->dev;
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+	int ret;
+
+	mutex_lock(&cam_dev->lock);
+	ret = v4l2_fh_open(file);
+	if (ret)
+		goto unlock;
+
+	ret = v4l2_pipeline_pm_use(&node->vdev.entity, 1);
+	if (ret)
+		dev_err(dev, "%s fail:%d", __func__, ret);
+
+unlock:
+	mutex_unlock(&cam_dev->lock);
+
+	return ret;
+}
+
+static int mtk_cam_isp_release(struct file *file)
+{
+	struct mtk_cam_dev *cam_dev = video_drvdata(file);
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+
+	mutex_lock(&cam_dev->lock);
+	v4l2_pipeline_pm_use(&node->vdev.entity, 0);
+	vb2_fop_release(file);
+	mutex_unlock(&cam_dev->lock);
+
+	return 0;
+}
+
+static struct v4l2_subdev *
+mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
+{
+	struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
+	struct media_entity *entity;
+	struct device *dev = &cam_dev->pdev->dev;
+	struct v4l2_subdev *sensor;
+
+	media_device_for_each_entity(entity, mdev) {
+		dev_dbg(dev, "media entity: %s:0x%x\n",
+			entity->name, entity->function);
+		if (entity->function == MEDIA_ENT_F_CAM_SENSOR &&
+		    entity->stream_count) {
+			sensor = media_entity_to_v4l2_subdev(entity);
+			dev_dbg(dev, "Sensor found: %s\n", entity->name);
+			break;
+		}
+	}
+
+	if (!sensor)
+		dev_err(dev, "Sensor is not connected\n");
+
+	return sensor;
+}
+
+static int mtk_cam_cio_stream_on(struct mtk_cam_dev *cam_dev)
+{
+	struct device *dev = &cam_dev->pdev->dev;
+	int ret;
+
+	/* Align vb2_core_streamon design */
+	if (cam_dev->streaming) {
+		dev_warn(dev, "already streaming\n", dev);
+		return 0;
+	}
+
+	if (!cam_dev->seninf) {
+		dev_err(dev, "no seninf connected:%d\n", ret);
+		return -EPERM;
+	}
+
+	/* Get active sensor from graph topology */
+	cam_dev->sensor = mtk_cam_cio_get_active_sensor(cam_dev);
+	if (!cam_dev->sensor)
+		return -EPERM;
+
+	ret = mtk_isp_config(dev);
+	if (ret)
+		return -EPERM;
+
+	/* Seninf must stream on first */
+	ret = v4l2_subdev_call(cam_dev->seninf, video, s_stream, 1);
+	if (ret) {
+		dev_err(dev, "%s stream on failed:%d\n",
+			cam_dev->seninf->entity.name, ret);
+		return -EPERM;
+	}
+
+	ret = v4l2_subdev_call(cam_dev->sensor, video, s_stream, 1);
+	if (ret) {
+		dev_err(dev, "%s stream on failed:%d\n",
+			cam_dev->sensor->entity.name, ret);
+		goto fail_sensor_on;
+	}
+
+	cam_dev->streaming = true;
+	mtk_cam_req_try_isp_queue(cam_dev, NULL);
+	isp_composer_stream(dev, 1);
+	dev_dbg(dev, "streamed on Pass 1\n");
+
+	return 0;
+
+fail_sensor_on:
+	v4l2_subdev_call(cam_dev->seninf, video, s_stream, 0);
+
+	return -EPERM;
+}
+
+static int mtk_cam_cio_stream_off(struct mtk_cam_dev *cam_dev)
+{
+	struct device *dev = &cam_dev->pdev->dev;
+	int ret;
+
+	if (!cam_dev->streaming) {
+		dev_warn(dev, "already stream off");
+		return 0;
+	}
+
+	ret = v4l2_subdev_call(cam_dev->sensor, video, s_stream, 0);
+	if (ret) {
+		dev_err(dev, "%s stream off failed:%d\n",
+			cam_dev->sensor->entity.name, ret);
+		return -EPERM;
+	}
+
+	ret = v4l2_subdev_call(cam_dev->seninf, video, s_stream, 0);
+	if (ret) {
+		dev_err(dev, "%s stream off failed:%d\n",
+			cam_dev->seninf->entity.name, ret);
+		return -EPERM;
+	}
+
+	isp_composer_stream(dev, 0);
+	cam_dev->streaming = false;
+	dev_dbg(dev, "streamed off Pass 1\n");
+
+	return 0;
+}
+
+static int mtk_cam_sd_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct mtk_cam_dev *cam_dev = mtk_cam_subdev_to_dev(sd);
+
+	if (enable)
+		return mtk_cam_cio_stream_on(cam_dev);
+	else
+		return mtk_cam_cio_stream_off(cam_dev);
+}
+
+static int mtk_cam_sd_subscribe_event(struct v4l2_subdev *subdev,
+				      struct v4l2_fh *fh,
+				      struct v4l2_event_subscription *sub)
+{
+	switch (sub->type) {
+	case V4L2_EVENT_FRAME_SYNC:
+		return v4l2_event_subscribe(fh, sub, 0, NULL);
+	default:
+		return -EINVAL;
+	}
+}
+
+static int mtk_cam_sd_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct mtk_cam_dev *cam_dev = mtk_cam_subdev_to_dev(sd);
+
+	dev_dbg(&cam_dev->pdev->dev, "%s:%d", __func__, on);
+
+	return on ? mtk_isp_power_init(cam_dev) :
+		    mtk_isp_power_release(&cam_dev->pdev->dev);
+}
+
+static int mtk_cam_media_link_setup(struct media_entity *entity,
+				    const struct media_pad *local,
+				    const struct media_pad *remote, u32 flags)
+{
+	struct mtk_cam_dev *cam_dev =
+		container_of(entity, struct mtk_cam_dev, subdev.entity);
+	u32 pad = local->index;
+
+	dev_dbg(&cam_dev->pdev->dev, "%s: %d -> %d flags:0x%x\n",
+		__func__, pad, remote->index, flags);
+
+	if (pad < MTK_CAM_P1_TOTAL_NODES)
+		cam_dev->vdev_nodes[pad].enabled =
+			!!(flags & MEDIA_LNK_FL_ENABLED);
+
+	return 0;
+}
+
+static void mtk_cam_vb2_buf_queue(struct vb2_buffer *vb)
+{
+	struct mtk_cam_dev *mtk_cam_dev = vb2_get_drv_priv(vb->vb2_queue);
+	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
+	struct device *dev = &mtk_cam_dev->pdev->dev;
+	struct mtk_cam_dev_buffer *buf;
+
+	buf = mtk_cam_vb2_buf_to_dev_buf(vb);
+
+	dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
+		__func__,
+		node->id,
+		buf->vbb.request_fd,
+		buf->vbb.vb2_buf.index);
+
+	/* For request buffers en-queue, handled in mtk_cam_req_try_queue */
+	if (vb->vb2_queue->uses_requests)
+		return;
+
+	/* Added the buffer into the tracking list */
+	spin_lock(&node->slock);
+	list_add_tail(&buf->list, &node->pending_list);
+	spin_unlock(&node->slock);
+
+	mtk_isp_enqueue(dev, node->desc.dma_port, buf);
+}
+
+static int mtk_cam_vb2_buf_init(struct vb2_buffer *vb)
+{
+	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vb->vb2_queue);
+	struct device *smem_dev = cam_dev->smem_dev;
+	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
+	struct mtk_cam_dev_buffer *buf;
+
+	buf = mtk_cam_vb2_buf_to_dev_buf(vb);
+	buf->node_id = node->id;
+	buf->daddr = vb2_dma_contig_plane_dma_addr(&buf->vbb.vb2_buf, 0);
+	buf->scp_addr = 0;
+
+	/* scp address is only valid for meta input buffer */
+	if (node->desc.smem_alloc)
+		buf->scp_addr = mtk_cam_smem_iova_to_scp_addr(smem_dev,
+							      buf->daddr);
+
+	return 0;
+}
+
+static int mtk_cam_vb2_buf_prepare(struct vb2_buffer *vb)
+{
+	struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb);
+	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
+	const struct v4l2_format *fmt = &node->vdev_fmt;
+	unsigned int size;
+
+	if (vb->vb2_queue->type == V4L2_BUF_TYPE_META_OUTPUT ||
+	    vb->vb2_queue->type == V4L2_BUF_TYPE_META_CAPTURE)
+		size = fmt->fmt.meta.buffersize;
+	else
+		size = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
+
+	if (vb2_plane_size(vb, 0) < size)
+		return -EINVAL;
+
+	v4l2_buf->field = V4L2_FIELD_NONE;
+	vb2_set_plane_payload(vb, 0, size);
+
+	return 0;
+}
+
+static int mtk_cam_vb2_queue_setup(struct vb2_queue *vq,
+				   unsigned int *num_buffers,
+				   unsigned int *num_planes,
+				   unsigned int sizes[],
+				   struct device *alloc_devs[])
+{
+	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
+	unsigned int max_buffer_count = node->desc.max_buf_count;
+	const struct v4l2_format *fmt = &node->vdev_fmt;
+	unsigned int size;
+
+	/* Check the limitation of buffer size */
+	if (max_buffer_count)
+		*num_buffers = clamp_val(*num_buffers, 1, max_buffer_count);
+
+	if (vq->type == V4L2_BUF_TYPE_META_OUTPUT ||
+	    vq->type == V4L2_BUF_TYPE_META_CAPTURE)
+		size = fmt->fmt.meta.buffersize;
+	else
+		size = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
+
+	/* Add for q.create_bufs with fmt.g_sizeimage(p) / 2 test */
+	if (*num_planes) {
+		if (sizes[0] < size)
+			return -EINVAL;
+	} else {
+		*num_planes = 1;
+		sizes[0] = size;
+	}
+
+	return 0;
+}
+
+static void mtk_cam_vb2_return_all_buffers(struct mtk_cam_dev *cam_dev,
+					   struct mtk_cam_video_device *node,
+					   enum vb2_buffer_state state)
+{
+	struct mtk_cam_dev_buffer *b, *b0;
+	struct mtk_cam_dev_request *req, *req0;
+	struct media_request_object *obj, *obj0;
+	struct vb2_buffer *vb;
+
+	dev_dbg(&cam_dev->pdev->dev, "%s: node:%s", __func__, node->vdev.name);
+
+	/* Return all buffers */
+	spin_lock(&node->slock);
+	list_for_each_entry_safe(b, b0, &node->pending_list, list) {
+		vb = &b->vbb.vb2_buf;
+		if (vb->state == VB2_BUF_STATE_ACTIVE)
+			vb2_buffer_done(vb, state);
+		list_del(&b->list);
+	}
+	spin_unlock(&node->slock);
+
+	spin_lock(&cam_dev->req_lock);
+	list_for_each_entry_safe(req, req0, &cam_dev->req_list, list) {
+		list_for_each_entry_safe(obj, obj0, &req->req.objects, list) {
+			vb = container_of(obj, struct vb2_buffer, req_obj);
+			if (vb->state == VB2_BUF_STATE_ACTIVE)
+				vb2_buffer_done(vb, state);
+		}
+		list_del(&req->list);
+	}
+	spin_unlock(&cam_dev->req_lock);
+
+	if (node->vbq.uses_requests)
+		mtk_isp_req_flush_buffers(&cam_dev->pdev->dev);
+}
+
+static int mtk_cam_vb2_start_streaming(struct vb2_queue *vq,
+				       unsigned int count)
+{
+	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
+	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
+	struct device *dev = &cam_dev->pdev->dev;
+	unsigned int node_count = cam_dev->subdev.entity.use_count;
+	int ret;
+
+	if (!node->enabled) {
+		dev_err(dev, "Node:%d is not enable\n", node->id);
+		ret = -ENOLINK;
+		goto fail_no_link;
+	}
+
+	dev_dbg(dev, "%s: count info:%d:%d", __func__,
+		atomic_read(&cam_dev->streamed_node_count), node_count);
+
+	if (atomic_inc_return(&cam_dev->streamed_node_count) < node_count)
+		return 0;
+
+	/* Start streaming of the whole pipeline now */
+	ret = media_pipeline_start(&node->vdev.entity, &cam_dev->pipeline);
+	if (ret) {
+		dev_err(dev, "%s: Node:%d failed\n", __func__, node->id);
+		goto fail_start_pipeline;
+	}
+
+	/* Stream on sub-devices node */
+	ret = v4l2_subdev_call(&cam_dev->subdev, video, s_stream, 1);
+	if (ret) {
+		dev_err(dev, "Node:%d s_stream on failed:%d\n", node->id, ret);
+		goto fail_stream_on;
+	}
+
+	return 0;
+
+fail_stream_on:
+	media_pipeline_stop(&node->vdev.entity);
+fail_start_pipeline:
+	atomic_dec(&cam_dev->streamed_node_count);
+fail_no_link:
+	mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_QUEUED);
+
+	return ret;
+}
+
+static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
+{
+	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
+	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
+	struct device *dev = &cam_dev->pdev->dev;
+
+	if (!node->enabled)
+		return;
+
+	mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_ERROR);
+
+	dev_dbg(dev, "%s: count info:%d", __func__,
+		cam_dev->subdev.entity.stream_count);
+
+	/* Check the first node to stream-off */
+	if (!cam_dev->subdev.entity.stream_count)
+		return;
+
+	media_pipeline_stop(&node->vdev.entity);
+
+	if (v4l2_subdev_call(&cam_dev->subdev, video, s_stream, 0))
+		dev_err(dev, "failed to stop streaming\n");
+}
+
+static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
+{
+	struct mtk_cam_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
+
+	v4l2_ctrl_request_complete(vb->req_obj.req,
+				   dev->v4l2_dev.ctrl_handler);
+}
+
+static int mtk_cam_vidioc_querycap(struct file *file, void *fh,
+				   struct v4l2_capability *cap)
+{
+	struct mtk_cam_dev *cam_dev = video_drvdata(file);
+
+	strscpy(cap->driver, MTK_CAM_DEV_P1_NAME, sizeof(cap->driver));
+	strscpy(cap->card, MTK_CAM_DEV_P1_NAME, sizeof(cap->card));
+	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
+		 dev_name(cam_dev->media_dev.dev));
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_enum_fmt(struct file *file, void *fh,
+				   struct v4l2_fmtdesc *f)
+{
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+
+	if (f->index >= node->desc.num_fmts)
+		return -EINVAL;
+
+	f->pixelformat = node->desc.fmts[f->index].fmt.pix_mp.pixelformat;
+	f->flags = 0;
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_g_fmt(struct file *file, void *fh,
+				struct v4l2_format *f)
+{
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+
+	if (!node->desc.num_fmts)
+		return -EINVAL;
+
+	f->fmt = node->vdev_fmt.fmt;
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_try_fmt(struct file *file, void *fh,
+				  struct v4l2_format *in_fmt)
+{
+	struct mtk_cam_dev *cam_dev = video_drvdata(file);
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+	const struct v4l2_format *dev_fmt;
+	__u32  width, height;
+
+	dev_dbg(&cam_dev->pdev->dev, "%s: fmt:%c%c%c%c, w*h:%u*%u\n",
+		__func__,
+		(in_fmt->fmt.pix_mp.pixelformat & 0xFF),
+		(in_fmt->fmt.pix_mp.pixelformat >> 8) & 0xFF,
+		(in_fmt->fmt.pix_mp.pixelformat >> 16) & 0xFF,
+		(in_fmt->fmt.pix_mp.pixelformat >> 24) & 0xFF,
+		in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height);
+
+	width = in_fmt->fmt.pix_mp.width;
+	height = in_fmt->fmt.pix_mp.height;
+
+	dev_fmt = mtk_cam_dev_find_fmt(&node->desc,
+				       in_fmt->fmt.pix_mp.pixelformat);
+	if (dev_fmt) {
+		mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
+					&in_fmt->fmt.pix_mp,
+					&dev_fmt->fmt.pix_mp,
+					node->id);
+	} else {
+		mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
+					     &node->desc, in_fmt);
+	}
+	in_fmt->fmt.pix_mp.width = clamp_t(u32,
+					   width,
+					   CAM_MIN_WIDTH,
+					   in_fmt->fmt.pix_mp.width);
+	in_fmt->fmt.pix_mp.height = clamp_t(u32,
+					    height,
+					    CAM_MIN_HEIGHT,
+					    in_fmt->fmt.pix_mp.height);
+	mtk_cam_dev_cal_mplane_fmt(&cam_dev->pdev->dev,
+				   &in_fmt->fmt.pix_mp, node->id);
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_s_fmt(struct file *file, void *fh,
+				struct v4l2_format *f)
+{
+	struct mtk_cam_dev *cam_dev = video_drvdata(file);
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+
+	if (cam_dev->streaming)
+		return -EBUSY;
+
+	/* Get the valid format */
+	mtk_cam_vidioc_try_fmt(file, fh, f);
+
+	/* Configure to video device */
+	mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
+				&node->vdev_fmt.fmt.pix_mp,
+				&f->fmt.pix_mp,
+				node->id);
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_enum_input(struct file *file, void *fh,
+				     struct v4l2_input *input)
+{
+	if (input->index)
+		return -EINVAL;
+
+	strscpy(input->name, "camera", sizeof(input->name));
+	input->type = V4L2_INPUT_TYPE_CAMERA;
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_g_input(struct file *file, void *fh,
+				  unsigned int *input)
+{
+	*input = 0;
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_s_input(struct file *file,
+				  void *fh, unsigned int input)
+{
+	return input == 0 ? 0 : -EINVAL;
+}
+
+static int mtk_cam_vidioc_enum_framesizes(struct file *filp, void *priv,
+					  struct v4l2_frmsizeenum *sizes)
+{
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(filp);
+	const struct v4l2_format *dev_fmt;
+
+	dev_fmt = mtk_cam_dev_find_fmt(&node->desc, sizes->pixel_format);
+	if (!dev_fmt || sizes->index)
+		return -EINVAL;
+
+	sizes->type = node->desc.frmsizes->type;
+	memcpy(&sizes->stepwise, &node->desc.frmsizes->stepwise,
+	       sizeof(sizes->stepwise));
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_meta_enum_fmt(struct file *file, void *fh,
+					struct v4l2_fmtdesc *f)
+{
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+
+	if (f->index)
+		return -EINVAL;
+
+	strscpy(f->description, node->desc.description,
+		sizeof(node->desc.description));
+	f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
+	f->flags = 0;
+
+	return 0;
+}
+
+static int mtk_cam_vidioc_g_meta_fmt(struct file *file, void *fh,
+				     struct v4l2_format *f)
+{
+	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
+
+	f->fmt.meta.dataformat = node->vdev_fmt.fmt.meta.dataformat;
+	f->fmt.meta.buffersize = node->vdev_fmt.fmt.meta.buffersize;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_core_ops mtk_cam_subdev_core_ops = {
+	.subscribe_event = mtk_cam_sd_subscribe_event,
+	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
+	.s_power = mtk_cam_sd_s_power,
+};
+
+static const struct v4l2_subdev_video_ops mtk_cam_subdev_video_ops = {
+	.s_stream =  mtk_cam_sd_s_stream,
+};
+
+static const struct v4l2_subdev_ops mtk_cam_subdev_ops = {
+	.core = &mtk_cam_subdev_core_ops,
+	.video = &mtk_cam_subdev_video_ops,
+};
+
+static const struct media_entity_operations mtk_cam_media_ops = {
+	.link_setup = mtk_cam_media_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static const struct vb2_ops mtk_cam_vb2_ops = {
+	.queue_setup = mtk_cam_vb2_queue_setup,
+	.wait_prepare = vb2_ops_wait_prepare,
+	.wait_finish = vb2_ops_wait_finish,
+	.buf_init = mtk_cam_vb2_buf_init,
+	.buf_prepare = mtk_cam_vb2_buf_prepare,
+	.start_streaming = mtk_cam_vb2_start_streaming,
+	.stop_streaming = mtk_cam_vb2_stop_streaming,
+	.buf_queue = mtk_cam_vb2_buf_queue,
+	.buf_request_complete = mtk_cam_vb2_buf_request_complete,
+};
+
+static const struct v4l2_file_operations mtk_cam_v4l2_fops = {
+	.unlocked_ioctl = video_ioctl2,
+	.open = mtk_cam_isp_open,
+	.release = mtk_cam_isp_release,
+	.poll = vb2_fop_poll,
+	.mmap = vb2_fop_mmap,
+#ifdef CONFIG_COMPAT
+	.compat_ioctl32 = v4l2_compat_ioctl32,
+#endif
+};
+
+static const struct media_device_ops mtk_cam_media_req_ops = {
+	.link_notify = v4l2_pipeline_link_notify,
+	.req_alloc = mtk_cam_req_alloc,
+	.req_free = mtk_cam_req_free,
+	.req_validate = vb2_request_validate,
+	.req_queue = mtk_cam_req_queue,
+};
+
+static int mtk_cam_media_register(struct device *dev,
+				  struct media_device *media_dev)
+{
+	media_dev->dev = dev;
+	strscpy(media_dev->model, MTK_CAM_DEV_P1_NAME,
+		sizeof(media_dev->model));
+	snprintf(media_dev->bus_info, sizeof(media_dev->bus_info),
+		 "platform:%s", dev_name(dev));
+	media_dev->hw_revision = 0;
+	media_device_init(media_dev);
+	media_dev->ops = &mtk_cam_media_req_ops;
+
+	return media_device_register(media_dev);
+}
+
+static int mtk_cam_video_register_device(struct mtk_cam_dev *cam_dev, u32 i)
+{
+	struct device *dev = &cam_dev->pdev->dev;
+	struct mtk_cam_video_device *node = &cam_dev->vdev_nodes[i];
+	struct video_device *vdev = &node->vdev;
+	struct vb2_queue *vbq = &node->vbq;
+	u32 output = !cam_dev->vdev_nodes[i].desc.capture;
+	u32 link_flags = cam_dev->vdev_nodes[i].desc.link_flags;
+	int ret;
+
+	cam_dev->subdev_pads[i].flags = output ?
+		MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
+
+	/* Initialize media entities */
+	ret = media_entity_pads_init(&vdev->entity, 1, &node->vdev_pad);
+	if (ret) {
+		dev_err(dev, "failed initialize media pad:%d\n", ret);
+		return ret;
+	}
+	node->enabled = false;
+	node->id = i;
+	node->vdev_pad.flags = cam_dev->subdev_pads[i].flags;
+	mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
+				     &node->desc,
+				     &node->vdev_fmt);
+
+	/* Initialize vbq */
+	vbq->type = node->vdev_fmt.type;
+	if (vbq->type == V4L2_BUF_TYPE_META_OUTPUT)
+		vbq->io_modes = VB2_MMAP;
+	else
+		vbq->io_modes = VB2_MMAP | VB2_DMABUF;
+
+	if (node->desc.smem_alloc) {
+		vbq->bidirectional = 1;
+		vbq->dev = cam_dev->smem_dev;
+	} else {
+		vbq->dev = &cam_dev->pdev->dev;
+	}
+
+	if (vbq->type == V4L2_BUF_TYPE_META_CAPTURE)
+		vdev->entity.function =
+			MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
+	vbq->ops = &mtk_cam_vb2_ops;
+	vbq->mem_ops = &vb2_dma_contig_memops;
+	vbq->buf_struct_size = sizeof(struct mtk_cam_dev_buffer);
+	vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	vbq->min_buffers_needed = 0;	/* Can streamon w/o buffers */
+	/* Put the process hub sub device in the vb2 private data */
+	vbq->drv_priv = cam_dev;
+	vbq->lock = &node->lock;
+	vbq->supports_requests = true;
+
+	ret = vb2_queue_init(vbq);
+	if (ret) {
+		dev_err(dev, "failed to init. vb2 queue:%d\n", ret);
+		goto fail_vb2_queue;
+	}
+
+	/* Initialize vdev */
+	snprintf(vdev->name, sizeof(vdev->name), "%s %s",
+		 MTK_CAM_DEV_P1_NAME, node->desc.name);
+	/* set cap/type/ioctl_ops of the video device */
+	vdev->device_caps = node->desc.cap | V4L2_CAP_STREAMING;
+	vdev->ioctl_ops = node->desc.ioctl_ops;
+	vdev->fops = &mtk_cam_v4l2_fops;
+	vdev->release = video_device_release_empty;
+	vdev->lock = &node->lock;
+	vdev->v4l2_dev = &cam_dev->v4l2_dev;
+	vdev->queue = &node->vbq;
+	vdev->vfl_dir = output ? VFL_DIR_TX : VFL_DIR_RX;
+	vdev->entity.ops = NULL;
+	/* Enable private control for image video devices */
+	if (node->desc.image) {
+		mtk_cam_ctrl_init(cam_dev, &node->ctrl_handler);
+		vdev->ctrl_handler = &node->ctrl_handler;
+	}
+	video_set_drvdata(vdev, cam_dev);
+	dev_dbg(dev, "register vdev:%d:%s\n", i, vdev->name);
+
+	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
+	if (ret) {
+		dev_err(dev, "failed to register vde:%d\n", ret);
+		goto fail_vdev;
+	}
+
+	/* Create link between video node and the subdev pad */
+	if (output) {
+		ret = media_create_pad_link(&vdev->entity, 0,
+					    &cam_dev->subdev.entity,
+					    i, link_flags);
+	} else {
+		ret = media_create_pad_link(&cam_dev->subdev.entity,
+					    i, &vdev->entity, 0,
+					    link_flags);
+	}
+	if (ret)
+		goto fail_link;
+
+	/* Initialize miscellaneous variables */
+	mutex_init(&node->lock);
+	spin_lock_init(&node->slock);
+	INIT_LIST_HEAD(&node->pending_list);
+
+	return 0;
+
+fail_link:
+	video_unregister_device(vdev);
+fail_vdev:
+	vb2_queue_release(vbq);
+fail_vb2_queue:
+	media_entity_cleanup(&vdev->entity);
+
+	return ret;
+}
+
+static int mtk_cam_mem2mem2_v4l2_register(struct mtk_cam_dev *cam_dev)
+{
+	struct device *dev = &cam_dev->pdev->dev;
+	/* Total pad numbers is video devices + one seninf pad */
+	unsigned int num_subdev_pads = MTK_CAM_CIO_PAD_SINK + 1;
+	unsigned int i;
+	int ret;
+
+	ret = mtk_cam_media_register(dev,
+				     &cam_dev->media_dev);
+	if (ret) {
+		dev_err(dev, "failed to register media device:%d\n", ret);
+		return ret;
+	}
+	dev_info(dev, "Register media device: %s, 0x%pK",
+		 MTK_CAM_DEV_P1_NAME, cam_dev->media_dev);
+
+	/* Set up v4l2 device */
+	cam_dev->v4l2_dev.mdev = &cam_dev->media_dev;
+	ret = v4l2_device_register(dev, &cam_dev->v4l2_dev);
+	if (ret) {
+		dev_err(dev, "failed to register V4L2 device:%d\n", ret);
+		goto fail_v4l2_dev;
+	}
+	dev_info(dev, "Register v4l2 device: 0x%pK", cam_dev->v4l2_dev);
+
+	/* Initialize subdev media entity */
+	cam_dev->subdev_pads = devm_kcalloc(dev, num_subdev_pads,
+					    sizeof(*cam_dev->subdev_pads),
+					    GFP_KERNEL);
+	if (!cam_dev->subdev_pads) {
+		ret = -ENOMEM;
+		goto fail_subdev_pads;
+	}
+
+	ret = media_entity_pads_init(&cam_dev->subdev.entity,
+				     num_subdev_pads,
+				     cam_dev->subdev_pads);
+	if (ret) {
+		dev_err(dev, "failed initialize media pads:%d:\n", ret);
+		goto fail_subdev_pads;
+	}
+
+	/* Initialize all pads with MEDIA_PAD_FL_SOURCE */
+	for (i = 0; i < num_subdev_pads; i++)
+		cam_dev->subdev_pads[i].flags = MEDIA_PAD_FL_SOURCE;
+
+	/* Customize the last one pad as CIO sink pad. */
+	cam_dev->subdev_pads[MTK_CAM_CIO_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
+
+	/* Initialize subdev */
+	v4l2_subdev_init(&cam_dev->subdev, &mtk_cam_subdev_ops);
+	cam_dev->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
+	cam_dev->subdev.entity.ops = &mtk_cam_media_ops;
+	cam_dev->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
+				V4L2_SUBDEV_FL_HAS_EVENTS;
+	snprintf(cam_dev->subdev.name, sizeof(cam_dev->subdev.name),
+		 "%s", MTK_CAM_DEV_P1_NAME);
+	v4l2_set_subdevdata(&cam_dev->subdev, cam_dev);
+
+	ret = v4l2_device_register_subdev(&cam_dev->v4l2_dev, &cam_dev->subdev);
+	if (ret) {
+		dev_err(dev, "failed initialize subdev:%d\n", ret);
+		goto fail_subdev;
+	}
+	dev_info(dev, "register subdev: %s\n", cam_dev->subdev.name);
+
+	/* Create video nodes and links */
+	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++) {
+		ret = mtk_cam_video_register_device(cam_dev, i);
+		if (ret)
+			goto fail_video_register;
+	}
+
+	vb2_dma_contig_set_max_seg_size(dev, DMA_BIT_MASK(32));
+
+	return 0;
+
+fail_video_register:
+	i--;
+	for (; i >= 0; i--) {
+		video_unregister_device(&cam_dev->vdev_nodes[i].vdev);
+		media_entity_cleanup(&cam_dev->vdev_nodes[i].vdev.entity);
+		mutex_destroy(&cam_dev->vdev_nodes[i].lock);
+	}
+fail_subdev:
+	media_entity_cleanup(&cam_dev->subdev.entity);
+fail_subdev_pads:
+	v4l2_device_unregister(&cam_dev->v4l2_dev);
+fail_v4l2_dev:
+	dev_err(dev, "fail_v4l2_dev mdev: 0x%pK:%d", &cam_dev->media_dev, ret);
+	media_device_unregister(&cam_dev->media_dev);
+	media_device_cleanup(&cam_dev->media_dev);
+
+	return ret;
+}
+
+static int mtk_cam_v4l2_unregister(struct mtk_cam_dev *cam_dev)
+{
+	unsigned int i;
+	struct mtk_cam_video_device *dev;
+
+	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++) {
+		dev = &cam_dev->vdev_nodes[i];
+		video_unregister_device(&dev->vdev);
+		media_entity_cleanup(&dev->vdev.entity);
+		mutex_destroy(&dev->lock);
+		if (dev->desc.image)
+			v4l2_ctrl_handler_free(&dev->ctrl_handler);
+	}
+
+	vb2_dma_contig_clear_max_seg_size(&cam_dev->pdev->dev);
+
+	v4l2_device_unregister_subdev(&cam_dev->subdev);
+	media_entity_cleanup(&cam_dev->subdev.entity);
+	kfree(cam_dev->subdev_pads);
+
+	v4l2_device_unregister(&cam_dev->v4l2_dev);
+	media_device_unregister(&cam_dev->media_dev);
+	media_device_cleanup(&cam_dev->media_dev);
+
+	return 0;
+}
+
+static int mtk_cam_dev_complete(struct v4l2_async_notifier *notifier)
+{
+	struct mtk_cam_dev *cam_dev =
+		container_of(notifier, struct mtk_cam_dev, notifier);
+	struct device *dev = &cam_dev->pdev->dev;
+	int ret;
+
+	ret = media_create_pad_link(&cam_dev->seninf->entity,
+				    MTK_CAM_CIO_PAD_SRC,
+				    &cam_dev->subdev.entity,
+				    MTK_CAM_CIO_PAD_SINK,
+				    0);
+	if (ret) {
+		dev_err(dev, "fail to create pad link %s %s err:%d\n",
+			cam_dev->seninf->entity.name,
+			cam_dev->subdev.entity.name,
+			ret);
+		return ret;
+	}
+
+	dev_info(dev, "Complete the v4l2 registration\n");
+
+	ret = v4l2_device_register_subdev_nodes(&cam_dev->v4l2_dev);
+	if (ret) {
+		dev_err(dev, "failed initialize subdev nodes:%d\n", ret);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int mtk_cam_dev_notifier_bound(struct v4l2_async_notifier *notifier,
+				      struct v4l2_subdev *sd,
+				      struct v4l2_async_subdev *asd)
+{
+	struct mtk_cam_dev *cam_dev =
+		container_of(notifier, struct mtk_cam_dev, notifier);
+
+	cam_dev->seninf = sd;
+	dev_info(&cam_dev->pdev->dev, "%s is bounded\n", sd->entity.name);
+	return 0;
+}
+
+static void mtk_cam_dev_notifier_unbind(struct v4l2_async_notifier *notifier,
+					struct v4l2_subdev *sd,
+					struct v4l2_async_subdev *asd)
+{
+	struct mtk_cam_dev *cam_dev =
+		container_of(notifier, struct mtk_cam_dev, notifier);
+
+	cam_dev->seninf = NULL;
+	dev_dbg(&cam_dev->pdev->dev, "%s is unbounded\n", sd->entity.name);
+}
+
+static int mtk_cam_dev_notifier_complete(struct v4l2_async_notifier *notifier)
+{
+	return mtk_cam_dev_complete(notifier);
+}
+
+static const struct v4l2_async_notifier_operations mtk_cam_async_ops = {
+	.bound = mtk_cam_dev_notifier_bound,
+	.unbind = mtk_cam_dev_notifier_unbind,
+	.complete = mtk_cam_dev_notifier_complete,
+};
+
+static int mtk_cam_v4l2_async_register(struct mtk_cam_dev *cam_dev)
+{
+	struct device *dev = &cam_dev->pdev->dev;
+	int ret;
+
+	ret = v4l2_async_notifier_parse_fwnode_endpoints(dev,
+		&cam_dev->notifier, sizeof(struct v4l2_async_subdev),
+		NULL);
+	if (ret)
+		return ret;
+
+	if (!cam_dev->notifier.num_subdevs)
+		return -ENODEV;
+
+	cam_dev->notifier.ops = &mtk_cam_async_ops;
+	dev_info(&cam_dev->pdev->dev, "mtk_cam v4l2_async_notifier_register\n");
+	ret = v4l2_async_notifier_register(&cam_dev->v4l2_dev,
+					   &cam_dev->notifier);
+	if (ret) {
+		dev_err(&cam_dev->pdev->dev,
+			"failed to register async notifier : %d\n", ret);
+		v4l2_async_notifier_cleanup(&cam_dev->notifier);
+	}
+
+	return ret;
+}
+
+static void mtk_cam_v4l2_async_unregister(struct mtk_cam_dev *cam_dev)
+{
+	v4l2_async_notifier_unregister(&cam_dev->notifier);
+	v4l2_async_notifier_cleanup(&cam_dev->notifier);
+}
+
+static const struct v4l2_ioctl_ops mtk_cam_v4l2_vcap_ioctl_ops = {
+	.vidioc_querycap = mtk_cam_vidioc_querycap,
+	.vidioc_enum_framesizes = mtk_cam_vidioc_enum_framesizes,
+	.vidioc_enum_fmt_vid_cap_mplane = mtk_cam_vidioc_enum_fmt,
+	.vidioc_g_fmt_vid_cap_mplane = mtk_cam_vidioc_g_fmt,
+	.vidioc_s_fmt_vid_cap_mplane = mtk_cam_vidioc_s_fmt,
+	.vidioc_try_fmt_vid_cap_mplane = mtk_cam_vidioc_try_fmt,
+	.vidioc_enum_input = mtk_cam_vidioc_enum_input,
+	.vidioc_g_input = mtk_cam_vidioc_g_input,
+	.vidioc_s_input = mtk_cam_vidioc_s_input,
+	/* buffer queue management */
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
+	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_cap_ioctl_ops = {
+	.vidioc_querycap = mtk_cam_vidioc_querycap,
+	.vidioc_enum_fmt_meta_cap = mtk_cam_vidioc_meta_enum_fmt,
+	.vidioc_g_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
+	.vidioc_s_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
+	.vidioc_try_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+};
+
+static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_out_ioctl_ops = {
+	.vidioc_querycap = mtk_cam_vidioc_querycap,
+	.vidioc_enum_fmt_meta_out = mtk_cam_vidioc_meta_enum_fmt,
+	.vidioc_g_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
+	.vidioc_s_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
+	.vidioc_try_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+};
+
+static const struct v4l2_format meta_fmts[] = {
+	{
+		.fmt.meta = {
+			.dataformat = V4L2_META_FMT_MTISP_PARAMS,
+			.buffersize = 128 * PAGE_SIZE,
+		},
+	},
+	{
+		.fmt.meta = {
+			.dataformat = V4L2_META_FMT_MTISP_3A,
+			.buffersize = 300 * PAGE_SIZE,
+		},
+	},
+	{
+		.fmt.meta = {
+			.dataformat = V4L2_META_FMT_MTISP_AF,
+			.buffersize = 160 * PAGE_SIZE,
+		},
+	},
+	{
+		.fmt.meta = {
+			.dataformat = V4L2_META_FMT_MTISP_LCS,
+			.buffersize = 72 * PAGE_SIZE,
+		},
+	},
+	{
+		.fmt.meta = {
+			.dataformat = V4L2_META_FMT_MTISP_LMV,
+			.buffersize = 256,
+		},
+	},
+};
+
+static const struct v4l2_format stream_out_fmts[] = {
+	{
+		.fmt.pix_mp = {
+			.width = IMG_MAX_WIDTH,
+			.height = IMG_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_B8,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_SRGB,
+			.num_planes = 1,
+		},
+	},
+	{
+		.fmt.pix_mp = {
+			.width = IMG_MAX_WIDTH,
+			.height = IMG_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_B10,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_SRGB,
+			.num_planes = 1,
+		},
+	},
+	{
+		.fmt.pix_mp = {
+			.width = IMG_MAX_WIDTH,
+			.height = IMG_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_B12,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_SRGB,
+			.num_planes = 1,
+		},
+	},
+	{
+		.fmt.pix_mp = {
+			.width = IMG_MAX_WIDTH,
+			.height = IMG_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_B14,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_SRGB,
+			.num_planes = 1,
+		},
+	},
+};
+
+static const struct v4l2_format bin_out_fmts[] = {
+	{
+		.fmt.pix_mp = {
+			.width = RRZ_MAX_WIDTH,
+			.height = RRZ_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_F8,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_RAW,
+			.num_planes = 1,
+		},
+	},
+	{
+		.fmt.pix_mp = {
+			.width = RRZ_MAX_WIDTH,
+			.height = RRZ_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_F10,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_RAW,
+			.num_planes = 1,
+		},
+	},
+	{
+		.fmt.pix_mp = {
+			.width = RRZ_MAX_WIDTH,
+			.height = RRZ_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_F12,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_RAW,
+			.num_planes = 1,
+		},
+	},
+	{
+		.fmt.pix_mp = {
+			.width = RRZ_MAX_WIDTH,
+			.height = RRZ_MAX_HEIGHT,
+			.pixelformat = V4L2_PIX_FMT_MTISP_F14,
+			.field = V4L2_FIELD_NONE,
+			.colorspace = V4L2_COLORSPACE_RAW,
+			.num_planes = 1,
+		},
+	},
+};
+
+static const struct v4l2_frmsizeenum img_frm_size_nums[] = {
+	{
+		.index = 0,
+		.type = V4L2_FRMSIZE_TYPE_CONTINUOUS,
+		.stepwise = {
+			.max_width = IMG_MAX_WIDTH,
+			.min_width = IMG_MIN_WIDTH,
+			.max_height = IMG_MAX_HEIGHT,
+			.min_height = IMG_MIN_HEIGHT,
+			.step_height = 1,
+			.step_width = 1,
+		},
+	},
+	{
+		.index = 0,
+		.type = V4L2_FRMSIZE_TYPE_CONTINUOUS,
+		.stepwise = {
+			.max_width = RRZ_MAX_WIDTH,
+			.min_width = RRZ_MIN_WIDTH,
+			.max_height = RRZ_MAX_HEIGHT,
+			.min_height = RRZ_MIN_HEIGHT,
+			.step_height = 1,
+			.step_width = 1,
+		},
+	},
+};
+
+static const struct
+mtk_cam_dev_node_desc output_queues[MTK_CAM_P1_TOTAL_OUTPUT] = {
+	{
+		.id = MTK_CAM_P1_META_IN_0,
+		.name = "meta input",
+		.description = "ISP tuning parameters",
+		.cap = V4L2_CAP_META_OUTPUT,
+		.buf_type = V4L2_BUF_TYPE_META_OUTPUT,
+		.link_flags = 0,
+		.capture = false,
+		.image = false,
+		.smem_alloc = true,
+		.fmts = meta_fmts,
+		.num_fmts = ARRAY_SIZE(meta_fmts),
+		.default_fmt_idx = 0,
+		.max_buf_count = 10,
+		.ioctl_ops = &mtk_cam_v4l2_meta_out_ioctl_ops,
+	},
+};
+
+static const struct
+mtk_cam_dev_node_desc capture_queues[MTK_CAM_P1_TOTAL_CAPTURE] = {
+	{
+		.id = MTK_CAM_P1_MAIN_STREAM_OUT,
+		.name = "main stream",
+		.cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE,
+		.buf_type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
+		.link_flags = 0,
+		.capture = true,
+		.image = true,
+		.smem_alloc = false,
+		.dma_port = R_IMGO,
+		.fmts = stream_out_fmts,
+		.num_fmts = ARRAY_SIZE(stream_out_fmts),
+		.default_fmt_idx = 1,
+		.ioctl_ops = &mtk_cam_v4l2_vcap_ioctl_ops,
+		.frmsizes = &img_frm_size_nums[0],
+	},
+	{
+		.id = MTK_CAM_P1_PACKED_BIN_OUT,
+		.name = "packed out",
+		.cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE,
+		.buf_type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
+		.link_flags = 0,
+		.capture = true,
+		.image = true,
+		.smem_alloc = false,
+		.dma_port = R_RRZO,
+		.fmts = bin_out_fmts,
+		.num_fmts = ARRAY_SIZE(bin_out_fmts),
+		.default_fmt_idx = 1,
+		.ioctl_ops = &mtk_cam_v4l2_vcap_ioctl_ops,
+		.frmsizes = &img_frm_size_nums[1],
+	},
+	{
+		.id = MTK_CAM_P1_META_OUT_0,
+		.name = "partial meta 0",
+		.description = "AE/AWB histogram",
+		.cap = V4L2_CAP_META_CAPTURE,
+		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
+		.link_flags = 0,
+		.capture = true,
+		.image = false,
+		.smem_alloc = false,
+		.dma_port = R_AAO | R_FLKO | R_PSO,
+		.fmts = meta_fmts,
+		.num_fmts = ARRAY_SIZE(meta_fmts),
+		.default_fmt_idx = 1,
+		.max_buf_count = 5,
+		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
+	},
+	{
+		.id = MTK_CAM_P1_META_OUT_1,
+		.name = "partial meta 1",
+		.description = "AF histogram",
+		.cap = V4L2_CAP_META_CAPTURE,
+		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
+		.link_flags = 0,
+		.capture = true,
+		.image = false,
+		.smem_alloc = false,
+		.dma_port = R_AFO,
+		.fmts = meta_fmts,
+		.num_fmts = ARRAY_SIZE(meta_fmts),
+		.default_fmt_idx = 2,
+		.max_buf_count = 5,
+		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
+	},
+	{
+		.id = MTK_CAM_P1_META_OUT_2,
+		.name = "partial meta 2",
+		.description = "Local contrast enhanced statistics",
+		.cap = V4L2_CAP_META_CAPTURE,
+		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
+		.link_flags = 0,
+		.capture = true,
+		.image = false,
+		.smem_alloc = false,
+		.dma_port = R_LCSO,
+		.fmts = meta_fmts,
+		.num_fmts = ARRAY_SIZE(meta_fmts),
+		.default_fmt_idx = 3,
+		.max_buf_count = 10,
+		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
+	},
+	{
+		.id = MTK_CAM_P1_META_OUT_3,
+		.name = "partial meta 3",
+		.description = "Local motion vector histogram",
+		.cap = V4L2_CAP_META_CAPTURE,
+		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
+		.link_flags = 0,
+		.capture = true,
+		.image = false,
+		.smem_alloc = false,
+		.dma_port = R_LMVO,
+		.fmts = meta_fmts,
+		.num_fmts = ARRAY_SIZE(meta_fmts),
+		.default_fmt_idx = 4,
+		.max_buf_count = 10,
+		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
+	},
+};
+
+/* The helper to configure the device context */
+static void mtk_cam_dev_queue_setup(struct mtk_cam_dev *cam_dev)
+{
+	unsigned int i, node_idx;
+
+	node_idx = 0;
+
+	/* Setup the output queue */
+	for (i = 0; i < MTK_CAM_P1_TOTAL_OUTPUT; i++)
+		cam_dev->vdev_nodes[node_idx++].desc = output_queues[i];
+
+	/* Setup the capture queue */
+	for (i = 0; i < MTK_CAM_P1_TOTAL_CAPTURE; i++)
+		cam_dev->vdev_nodes[node_idx++].desc = capture_queues[i];
+}
+
+int mtk_cam_dev_init(struct platform_device *pdev,
+		     struct mtk_cam_dev *cam_dev)
+{
+	int ret;
+
+	cam_dev->pdev = pdev;
+	mtk_cam_dev_queue_setup(cam_dev);
+	/* v4l2 sub-device registration */
+
+	dev_dbg(&cam_dev->pdev->dev, "mem2mem2.name: %s\n",
+		MTK_CAM_DEV_P1_NAME);
+	ret = mtk_cam_mem2mem2_v4l2_register(cam_dev);
+	if (ret)
+		return ret;
+
+	ret = mtk_cam_v4l2_async_register(cam_dev);
+	if (ret) {
+		mtk_cam_v4l2_unregister(cam_dev);
+		return ret;
+	}
+
+	spin_lock_init(&cam_dev->req_lock);
+	INIT_LIST_HEAD(&cam_dev->req_list);
+	mutex_init(&cam_dev->lock);
+
+	return 0;
+}
+
+int mtk_cam_dev_release(struct platform_device *pdev,
+			struct mtk_cam_dev *cam_dev)
+{
+	mtk_cam_v4l2_async_unregister(cam_dev);
+	mtk_cam_v4l2_unregister(cam_dev);
+
+	mutex_destroy(&cam_dev->lock);
+
+	return 0;
+}
+
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
new file mode 100644
index 000000000000..825cdf20643a
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
@@ -0,0 +1,173 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ */
+
+#ifndef __MTK_CAM_DEV_V4L2_H__
+#define __MTK_CAM_DEV_V4L2_H__
+
+#include <linux/device.h>
+#include <linux/types.h>
+#include <linux/platform_device.h>
+#include <linux/spinlock.h>
+#include <linux/videodev2.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-core.h>
+#include <media/videobuf2-v4l2.h>
+
+#define MTK_CAM_DEV_P1_NAME			"MTK-ISP-P1-V4L2"
+
+#define MTK_CAM_P1_META_IN_0			0
+#define MTK_CAM_P1_TOTAL_OUTPUT		1
+
+#define MTK_CAM_P1_MAIN_STREAM_OUT		1
+#define MTK_CAM_P1_PACKED_BIN_OUT		2
+#define MTK_CAM_P1_META_OUT_0			3
+#define MTK_CAM_P1_META_OUT_1			4
+#define MTK_CAM_P1_META_OUT_2			5
+#define MTK_CAM_P1_META_OUT_3			6
+#define MTK_CAM_P1_TOTAL_CAPTURE		6
+
+#define MTK_CAM_P1_TOTAL_NODES			7
+
+struct mtk_cam_dev_request {
+	struct media_request	req;
+	struct list_head	list;
+};
+
+struct mtk_cam_dev_buffer {
+	struct vb2_v4l2_buffer	vbb;
+	struct list_head	list;
+	/* Intenal part */
+	dma_addr_t		daddr;
+	dma_addr_t		scp_addr;
+	unsigned int		node_id;
+};
+
+/*
+ * struct mtk_cam_dev_node_desc - node attributes
+ *
+ * @id:		 id of the context queue
+ * @name:	 media entity name
+ * @description: descritpion of node
+ * @cap:	 mapped to V4L2 capabilities
+ * @buf_type:	 mapped to V4L2 buffer type
+ * @dma_port:	 the dma port associated to the buffer
+ * @link_flags:	 default media link flags
+ * @smem_alloc:	 using the cam_smem_drv as alloc ctx or not
+ * @capture:	 true for capture queue (device to user)
+ *		 false for output queue (from user to device)
+ * @image:	 true for image node, false for meta node
+ * @num_fmts:	 the number of supported formats
+ * @default_fmt_idx: default format of this node
+ * @max_buf_count: maximum V4L2 buffer count
+ * @ioctl_ops:  mapped to v4l2_ioctl_ops
+ * @fmts:	supported format
+ * @frmsizes:	supported frame size number
+ *
+ */
+struct mtk_cam_dev_node_desc {
+	u8 id;
+	char *name;
+	char *description;
+	u32 cap;
+	u32 buf_type;
+	u32 dma_port;
+	u32 link_flags;
+	u8 smem_alloc:1;
+	u8 capture:1;
+	u8 image:1;
+	u8 num_fmts;
+	u8 default_fmt_idx;
+	u8 max_buf_count;
+	const struct v4l2_ioctl_ops *ioctl_ops;
+	const struct v4l2_format *fmts;
+	const struct v4l2_frmsizeenum *frmsizes;
+};
+
+/*
+ * struct mtk_cam_video_device - Mediatek video device structure.
+ *
+ * @id:		Id for mtk_cam_dev_node_desc or mem2mem2_nodes array
+ * @enabled:	Indicate the device is enabled or not
+ * @vdev_fmt:	The V4L2 format of video device
+ * @vdev_apd:	The media pad graph object of video device
+ * @vbq:	A videobuf queue of video device
+ * @desc:	The node attributes of video device
+ * @ctrl_handler:	The control handler of video device
+ * @pending_list:	List for pending buffers before enqueuing into driver
+ * @lock:	Serializes vb2 queue and video device operations.
+ * @slock:	Protect for pending_list.
+ *
+ */
+struct mtk_cam_video_device {
+	unsigned int id;
+	unsigned int enabled;
+	struct v4l2_format vdev_fmt;
+	struct mtk_cam_dev_node_desc desc;
+	struct video_device vdev;
+	struct media_pad vdev_pad;
+	struct vb2_queue vbq;
+	struct v4l2_ctrl_handler ctrl_handler;
+	struct list_head pending_list;
+	/* Used for vbq & vdev */
+	struct mutex lock;
+	/* protect for pending_list */
+	spinlock_t slock;
+};
+
+/*
+ * struct mtk_cam_dev - Mediatek camera device structure.
+ *
+ * @pdev:	Pointer to platform device
+ * @smem_pdev:	Pointer to shared memory platform device
+ * @pipeline:	Media pipeline information
+ * @media_dev:	Media device
+ * @subdev:	The V4L2 sub-device
+ * @v4l2_dev:	The V4L2 device driver
+ * @notifier:	The v4l2_device notifier data
+ * @subdev_pads: Pointer to the number of media pads of this sub-device
+ * @ctrl_handler: The control handler
+ * @vdev_nodes: The array list of mtk_cam_video_device nodes
+ * @seninf:	Pointer to the seninf sub-device
+ * @sensor:	Pointer to the active sensor V4L2 sub-device when streaming on
+ * @lock:       The mutex protecting video device open/release operations
+ * @streaming:	Indicate the overall streaming status is on or off
+ * @streamed_node_count: The number of V4L2 video device nodes are streaming on
+ * @req_list:	Lins to keep media requests before streaming on
+ * @req_lock:	Protect the req_list data
+ *
+ * Below is the graph topology for Camera IO connection.
+ * sensor 1 (main) --> sensor IF --> P1 sub-device
+ * sensor 2 (sub)  -->
+ *
+ */
+struct mtk_cam_dev {
+	struct platform_device *pdev;
+	struct device *smem_dev;
+	struct media_pipeline pipeline;
+	struct media_device media_dev;
+	struct v4l2_subdev subdev;
+	struct v4l2_device v4l2_dev;
+	struct v4l2_async_notifier notifier;
+	struct media_pad *subdev_pads;
+	struct v4l2_ctrl_handler ctrl_handler;
+	struct mtk_cam_video_device vdev_nodes[MTK_CAM_P1_TOTAL_NODES];
+	struct v4l2_subdev *seninf;
+	struct v4l2_subdev *sensor;
+	/* protect video device open/release operations */
+	struct mutex lock;
+	unsigned int streaming:1;
+	atomic_t streamed_node_count;
+	struct list_head req_list;
+	/* protect for req_list */
+	spinlock_t req_lock;
+};
+
+int mtk_cam_dev_init(struct platform_device *pdev,
+		     struct mtk_cam_dev *cam_dev);
+int mtk_cam_dev_release(struct platform_device *pdev,
+			struct mtk_cam_dev *cam_dev);
+#endif /* __MTK_CAM_DEV_V4L2_H__ */
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
                   ` (5 preceding siblings ...)
  2019-06-11  3:53 ` [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-07-10  9:56   ` Tomasz Figa
  2019-06-11  3:53 ` [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication Jungo Lin
  2019-06-11  3:53 ` [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device Jungo Lin
  8 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

This patch adds the Mediatek ISP P1 HW control device driver.
It handles the ISP HW configuration, provides interrupt handling and
initializes the V4L2 device nodes and other functions.

(The current metadata interface used in meta input and partial
meta nodes is only a temporary solution to kick off the driver
development and is not ready to be reviewed yet.)

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
 .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
 .../mtk-isp/isp_50/cam/mtk_cam-regs.h         |  126 ++
 .../platform/mtk-isp/isp_50/cam/mtk_cam.c     | 1087 +++++++++++++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam.h     |  243 ++++
 4 files changed, 1457 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h

diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
index 7558593e63f0..30df10983f6a 100644
--- a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
@@ -2,5 +2,6 @@
 
 mtk-cam-isp-objs += mtk_cam-ctrl.o
 mtk-cam-isp-objs += mtk_cam-v4l2-util.o
+mtk-cam-isp-objs += mtk_cam.o
 
 obj-$(CONFIG_VIDEO_MEDIATEK_ISP_PASS1) += mtk-cam-isp.o
\ No newline at end of file
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
new file mode 100644
index 000000000000..9e59a6bfc6b7
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
@@ -0,0 +1,126 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ */
+
+#ifndef _CAM_REGS_H
+#define _CAM_REGS_H
+
+/* TG Bit Mask */
+#define VFDATA_EN_BIT			BIT(0)
+#define CMOS_EN_BIT			BIT(0)
+
+/* normal signal bit */
+#define VS_INT_ST			BIT(0)
+#define HW_PASS1_DON_ST		BIT(11)
+#define SOF_INT_ST			BIT(12)
+#define SW_PASS1_DON_ST		BIT(30)
+
+/* err status bit */
+#define TG_ERR_ST			BIT(4)
+#define TG_GBERR_ST			BIT(5)
+#define CQ_CODE_ERR_ST			BIT(6)
+#define CQ_APB_ERR_ST			BIT(7)
+#define CQ_VS_ERR_ST			BIT(8)
+#define AMX_ERR_ST			BIT(15)
+#define RMX_ERR_ST			BIT(16)
+#define BMX_ERR_ST			BIT(17)
+#define RRZO_ERR_ST			BIT(18)
+#define AFO_ERR_ST			BIT(19)
+#define IMGO_ERR_ST			BIT(20)
+#define AAO_ERR_ST			BIT(21)
+#define PSO_ERR_ST			BIT(22)
+#define LCSO_ERR_ST			BIT(23)
+#define BNR_ERR_ST			BIT(24)
+#define LSCI_ERR_ST			BIT(25)
+#define DMA_ERR_ST			BIT(29)
+
+/* CAM DMA done status */
+#define FLKO_DONE_ST			BIT(4)
+#define AFO_DONE_ST			BIT(5)
+#define AAO_DONE_ST			BIT(7)
+#define PSO_DONE_ST			BIT(14)
+
+/* IRQ signal mask */
+#define INT_ST_MASK_CAM		( \
+					VS_INT_ST |\
+					SOF_INT_ST |\
+					HW_PASS1_DON_ST |\
+					SW_PASS1_DON_ST)
+
+/* IRQ Error Mask */
+#define INT_ST_MASK_CAM_ERR		( \
+					TG_ERR_ST |\
+					TG_GBERR_ST |\
+					CQ_CODE_ERR_ST |\
+					CQ_APB_ERR_ST |\
+					CQ_VS_ERR_ST |\
+					BNR_ERR_ST |\
+					RMX_ERR_ST |\
+					BMX_ERR_ST |\
+					BNR_ERR_ST |\
+					LSCI_ERR_ST |\
+					DMA_ERR_ST)
+
+/* IRQ Signal Log Mask */
+#define INT_ST_LOG_MASK_CAM		( \
+					SOF_INT_ST |\
+					SW_PASS1_DON_ST |\
+					HW_PASS1_DON_ST |\
+					VS_INT_ST |\
+					TG_ERR_ST |\
+					TG_GBERR_ST |\
+					RRZO_ERR_ST |\
+					AFO_ERR_ST |\
+					IMGO_ERR_ST |\
+					AAO_ERR_ST |\
+					DMA_ERR_ST)
+
+/* DMA Event Notification Mask */
+#define DMA_ST_MASK_CAM		( \
+					AAO_DONE_ST |\
+					AFO_DONE_ST)
+
+/* Status check */
+#define REG_CTL_EN			0x0004
+#define REG_CTL_DMA_EN			0x0008
+#define REG_CTL_FMT_SEL		0x0010
+#define REG_CTL_EN2			0x0018
+#define REG_CTL_RAW_INT_EN		0x0020
+#define REG_CTL_RAW_INT_STAT		0x0024
+#define REG_CTL_RAW_INT2_STAT		0x0034
+
+#define REG_TG_SEN_MODE		0x0230
+#define REG_TG_VF_CON			0x0234
+
+#define REG_IMGO_BASE_ADDR		0x1020
+#define REG_RRZO_BASE_ADDR		0x1050
+
+/* Error status log */
+#define REG_IMGO_ERR_STAT		0x1360
+#define REG_RRZO_ERR_STAT		0x1364
+#define REG_AAO_ERR_STAT		0x1368
+#define REG_AFO_ERR_STAT		0x136c
+#define REG_LCSO_ERR_STAT		0x1370
+#define REG_UFEO_ERR_STAT		0x1374
+#define REG_PDO_ERR_STAT		0x1378
+#define REG_BPCI_ERR_STAT		0x137c
+#define REG_LSCI_ERR_STAT		0x1384
+#define REG_PDI_ERR_STAT		0x138c
+#define REG_LMVO_ERR_STAT		0x1390
+#define REG_FLKO_ERR_STAT		0x1394
+#define REG_PSO_ERR_STAT		0x13a0
+
+/* ISP command */
+#define REG_CQ_THR0_BASEADDR		0x0198
+#define REG_HW_FRAME_NUM		0x13b8
+
+/* META */
+#define REG_META0_VB2_INDEX		0x14dc
+#define REG_META1_VB2_INDEX		0x151c
+
+/* FBC */
+#define REG_AAO_FBC_STATUS		0x013c
+#define REG_AFO_FBC_STATUS		0x0134
+
+#endif	/* _CAM_REGS_H */
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
new file mode 100644
index 000000000000..c5a3babed69d
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
@@ -0,0 +1,1087 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+
+#include <linux/atomic.h>
+#include <linux/cdev.h>
+#include <linux/compat.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/ktime.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/of_irq.h>
+#include <linux/of_address.h>
+#include <linux/platform_device.h>
+#include <linux/platform_data/mtk_scp.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/sched/clock.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/videodev2.h>
+#include <linux/vmalloc.h>
+#include <media/v4l2-event.h>
+
+#include "mtk_cam.h"
+#include "mtk_cam-regs.h"
+#include "mtk_cam-smem.h"
+
+static const struct of_device_id mtk_isp_of_ids[] = {
+	{.compatible = "mediatek,mt8183-camisp",},
+	{}
+};
+MODULE_DEVICE_TABLE(of, mtk_isp_of_ids);
+
+/* List of clocks required by isp cam */
+static const char * const mtk_isp_clks[] = {
+	"camsys_cam_cgpdn", "camsys_camtg_cgpdn"
+};
+
+static void isp_dump_dma_status(struct isp_device *isp_dev)
+{
+	dev_err(isp_dev->dev,
+		"IMGO:0x%x, RRZO:0x%x, AAO=0x%x, AFO=0x%x, LMVO=0x%x\n",
+		readl(isp_dev->regs + REG_IMGO_ERR_STAT),
+		readl(isp_dev->regs + REG_RRZO_ERR_STAT),
+		readl(isp_dev->regs + REG_AAO_ERR_STAT),
+		readl(isp_dev->regs + REG_AFO_ERR_STAT),
+		readl(isp_dev->regs + REG_LMVO_ERR_STAT));
+	dev_err(isp_dev->dev,
+		"LCSO=0x%x, PSO=0x%x, FLKO=0x%x, BPCI:0x%x, LSCI=0x%x\n",
+		readl(isp_dev->regs + REG_LCSO_ERR_STAT),
+		readl(isp_dev->regs + REG_PSO_ERR_STAT),
+		readl(isp_dev->regs + REG_FLKO_ERR_STAT),
+		readl(isp_dev->regs + REG_BPCI_ERR_STAT),
+		readl(isp_dev->regs + REG_LSCI_ERR_STAT));
+}
+
+static void mtk_cam_dev_event_frame_sync(struct mtk_cam_dev *cam_dev,
+					 __u32 frame_seq_no)
+{
+	struct v4l2_event event;
+
+	memset(&event, 0, sizeof(event));
+	event.type = V4L2_EVENT_FRAME_SYNC;
+	event.u.frame_sync.frame_sequence = frame_seq_no;
+	v4l2_event_queue(cam_dev->subdev.devnode, &event);
+}
+
+static void mtk_cam_dev_job_finish(struct mtk_isp_p1_ctx *isp_ctx,
+				   unsigned int request_fd,
+				   unsigned int frame_seq_no,
+				   struct list_head *list_buf,
+				   enum vb2_buffer_state state)
+{
+	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
+	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
+	struct mtk_cam_dev_buffer *buf, *b0;
+	u64    timestamp;
+
+	if (!cam_dev->streaming)
+		return;
+
+	dev_dbg(&p1_dev->pdev->dev, "%s request fd:%d frame_seq:%d state:%d\n",
+		__func__, request_fd, frame_seq_no, state);
+
+	/*
+	 * Set the buffer's VB2 status so that the user can dequeue
+	 * the buffer.
+	 */
+	timestamp = ktime_get_ns();
+	list_for_each_entry_safe(buf, b0, list_buf, list) {
+		list_del(&buf->list);
+		buf->vbb.vb2_buf.timestamp = timestamp;
+		buf->vbb.sequence = frame_seq_no;
+		if (buf->vbb.vb2_buf.state == VB2_BUF_STATE_ACTIVE)
+			vb2_buffer_done(&buf->vbb.vb2_buf, state);
+	}
+}
+
+static void isp_deque_frame(struct isp_p1_device *p1_dev,
+			    unsigned int node_id, int vb2_index,
+			    int frame_seq_no)
+{
+	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
+	struct device *dev = &p1_dev->pdev->dev;
+	struct mtk_cam_video_device *node = &cam_dev->vdev_nodes[node_id];
+	struct mtk_cam_dev_buffer *b, *b0;
+	struct vb2_buffer *vb;
+
+	if (!cam_dev->vdev_nodes[node_id].enabled || !cam_dev->streaming)
+		return;
+
+	spin_lock(&node->slock);
+	b = list_first_entry(&node->pending_list,
+			     struct mtk_cam_dev_buffer,
+			     list);
+	list_for_each_entry_safe(b, b0, &node->pending_list, list) {
+		vb = &b->vbb.vb2_buf;
+		if (!vb->vb2_queue->uses_requests &&
+		    vb->index == vb2_index &&
+		    vb->state == VB2_BUF_STATE_ACTIVE) {
+			dev_dbg(dev, "%s:%d:%d", __func__, node_id, vb2_index);
+			vb->timestamp = ktime_get_ns();
+			b->vbb.sequence = frame_seq_no;
+			vb2_buffer_done(vb, VB2_BUF_STATE_DONE);
+			list_del(&b->list);
+			break;
+		}
+	}
+	spin_unlock(&node->slock);
+}
+
+static void isp_deque_request_frame(struct isp_p1_device *p1_dev,
+				    int frame_seq_no)
+{
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct device *dev = &p1_dev->pdev->dev;
+	struct mtk_isp_queue_job *framejob, *tmp;
+	struct isp_queue *p1_enqueue_list = &isp_ctx->p1_enqueue_list;
+
+	/* Match dequeue work and enqueue frame */
+	spin_lock(&p1_enqueue_list->lock);
+	list_for_each_entry_safe(framejob, tmp, &p1_enqueue_list->queue,
+				 list_entry) {
+		dev_dbg(dev,
+			"%s frame_seq_no:%d, target frame_seq_no:%d\n",
+			__func__,
+			framejob->frame_seq_no, frame_seq_no);
+		/* Match by the en-queued request number */
+		if (framejob->frame_seq_no == frame_seq_no) {
+			/* Pass to user space */
+			mtk_cam_dev_job_finish(isp_ctx,
+					       framejob->request_fd,
+					       framejob->frame_seq_no,
+					       &framejob->list_buf,
+					       VB2_BUF_STATE_DONE);
+			atomic_dec(&p1_enqueue_list->queue_cnt);
+			dev_dbg(dev,
+				"frame_seq_no:%d is done, queue_cnt:%d\n",
+				framejob->frame_seq_no,
+				atomic_read(&p1_enqueue_list->queue_cnt));
+
+			/* Remove only when frame ready */
+			list_del(&framejob->list_entry);
+			kfree(framejob);
+			break;
+		} else if (framejob->frame_seq_no < frame_seq_no) {
+			/* Pass to user space for frame drop */
+			mtk_cam_dev_job_finish(isp_ctx,
+					       framejob->request_fd,
+					       framejob->frame_seq_no,
+					       &framejob->list_buf,
+					       VB2_BUF_STATE_ERROR);
+			atomic_dec(&p1_enqueue_list->queue_cnt);
+			dev_warn(dev,
+				 "frame_seq_no:%d drop, queue_cnt:%d\n",
+				 framejob->frame_seq_no,
+				 atomic_read(&p1_enqueue_list->queue_cnt));
+
+			/* Remove only drop frame */
+			list_del(&framejob->list_entry);
+			kfree(framejob);
+		} else {
+			break;
+		}
+	}
+	spin_unlock(&p1_enqueue_list->lock);
+}
+
+static int isp_deque_work(void *data)
+{
+	struct isp_p1_device *p1_dev = (struct isp_p1_device *)data;
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct mtk_cam_dev_stat_event_data event_data;
+	atomic_t *irq_data_end = &isp_ctx->irq_data_end;
+	atomic_t *irq_data_start = &isp_ctx->irq_data_start;
+	unsigned long flags;
+	int ret, i;
+
+	while (1) {
+		ret = wait_event_interruptible(isp_ctx->isp_deque_thread.wq,
+					       (atomic_read(irq_data_end) !=
+					       atomic_read(irq_data_start)) ||
+					       kthread_should_stop());
+
+		if (kthread_should_stop())
+			break;
+
+		spin_lock_irqsave(&isp_ctx->irq_dequeue_lock, flags);
+		i = atomic_read(&isp_ctx->irq_data_start);
+		memcpy(&event_data, &isp_ctx->irq_event_datas[i],
+		       sizeof(event_data));
+		atomic_set(&isp_ctx->irq_data_start, ++i & 0x3);
+		spin_unlock_irqrestore(&isp_ctx->irq_dequeue_lock, flags);
+
+		if (event_data.irq_status_mask & HW_PASS1_DON_ST &&
+		    event_data.dma_status_mask & AAO_DONE_ST) {
+			isp_deque_frame(p1_dev,
+					MTK_CAM_P1_META_OUT_0,
+					event_data.meta0_vb2_index,
+					event_data.frame_seq_no);
+		}
+		if (event_data.dma_status_mask & AFO_DONE_ST) {
+			isp_deque_frame(p1_dev,
+					MTK_CAM_P1_META_OUT_1,
+					event_data.meta1_vb2_index,
+					event_data.frame_seq_no);
+		}
+		if (event_data.irq_status_mask & SW_PASS1_DON_ST) {
+			isp_deque_frame(p1_dev,
+					MTK_CAM_P1_META_OUT_0,
+					event_data.meta0_vb2_index,
+					event_data.frame_seq_no);
+			isp_deque_frame(p1_dev,
+					MTK_CAM_P1_META_OUT_1,
+					event_data.meta1_vb2_index,
+					event_data.frame_seq_no);
+			isp_deque_request_frame(p1_dev,
+						event_data.frame_seq_no);
+		}
+	}
+
+	return 0;
+}
+
+static int irq_handle_sof(struct isp_device *isp_dev,
+			  dma_addr_t base_addr,
+			  unsigned int frame_num)
+{
+	unsigned int addr_offset;
+	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
+	int cq_num = atomic_read(&p1_dev->isp_ctx.composed_frame_id);
+
+	isp_dev->sof_count += 1;
+
+	if (cq_num <= frame_num) {
+		dev_dbg(isp_dev->dev,
+			"SOF_INT_ST, wait next, cq_num:%d, frame_num:%d",
+			cq_num, frame_num);
+		atomic_set(&p1_dev->isp_ctx.composing_frame, 0);
+		return cq_num;
+	}
+	atomic_set(&p1_dev->isp_ctx.composing_frame, cq_num - frame_num);
+
+	addr_offset = CQ_ADDRESS_OFFSET * (frame_num % CQ_BUFFER_COUNT);
+	writel(base_addr + addr_offset, isp_dev->regs + REG_CQ_THR0_BASEADDR);
+	dev_dbg(isp_dev->dev,
+		"SOF_INT_ST, update next, cq_num:%d, frame_num:%d cq_addr:0x%x",
+		cq_num, frame_num, addr_offset);
+
+	return cq_num;
+}
+
+static void irq_handle_notify_event(struct isp_device *isp_dev,
+				    unsigned int irq_status,
+				    unsigned int dma_status,
+				    bool sof_only)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct device *dev = isp_dev->dev;
+	unsigned long flags;
+	int i;
+
+	if (irq_status & VS_INT_ST) {
+		/* Notify specific HW events to user space */
+		mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev,
+					     isp_dev->current_frame);
+		dev_dbg(dev,
+			"frame sync is sent:%d:%d\n",
+			isp_dev->sof_count,
+			isp_dev->current_frame);
+		if (sof_only)
+			return;
+	}
+
+	if (irq_status & SW_PASS1_DON_ST) {
+		/* Notify TX thread to send if TX frame is blocked */
+		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
+	}
+
+	spin_lock_irqsave(&isp_ctx->irq_dequeue_lock, flags);
+	i = atomic_read(&isp_ctx->irq_data_end);
+	isp_ctx->irq_event_datas[i].frame_seq_no = isp_dev->current_frame;
+	isp_ctx->irq_event_datas[i].meta0_vb2_index = isp_dev->meta0_vb2_index;
+	isp_ctx->irq_event_datas[i].meta1_vb2_index = isp_dev->meta1_vb2_index;
+	isp_ctx->irq_event_datas[i].irq_status_mask =
+		(irq_status & INT_ST_MASK_CAM);
+	isp_ctx->irq_event_datas[i].dma_status_mask =
+		(dma_status & DMA_ST_MASK_CAM);
+	atomic_set(&isp_ctx->irq_data_end, ++i & 0x3);
+	spin_unlock_irqrestore(&isp_ctx->irq_dequeue_lock, flags);
+
+	wake_up_interruptible(&isp_ctx->isp_deque_thread.wq);
+
+	dev_dbg(dev,
+		"%s IRQ:0x%x DMA:0x%x seq:%d idx0:%d idx1:%d\n",
+		__func__,
+		(irq_status & INT_ST_MASK_CAM),
+		(dma_status & DMA_ST_MASK_CAM),
+		isp_dev->current_frame,
+		isp_dev->meta0_vb2_index,
+		isp_dev->meta1_vb2_index);
+}
+
+irqreturn_t isp_irq_cam(int irq, void *data)
+{
+	struct isp_device *isp_dev = (struct isp_device *)data;
+	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct device *dev = isp_dev->dev;
+	unsigned int cam_idx, cq_num, hw_frame_num;
+	unsigned int meta0_vb2_index, meta1_vb2_index;
+	unsigned int irq_status, err_status, dma_status;
+	unsigned int aao_fbc, afo_fbc;
+	unsigned long flags;
+
+	/* Check the streaming is off or not */
+	if (!p1_dev->cam_dev.streaming)
+		return IRQ_HANDLED;
+
+	cam_idx = isp_dev->isp_hw_module - ISP_CAM_A_IDX;
+	cq_num = 0;
+
+	spin_lock_irqsave(&isp_dev->spinlock_irq, flags);
+	irq_status = readl(isp_dev->regs + REG_CTL_RAW_INT_STAT);
+	dma_status = readl(isp_dev->regs + REG_CTL_RAW_INT2_STAT);
+	hw_frame_num = readl(isp_dev->regs + REG_HW_FRAME_NUM);
+	meta0_vb2_index = readl(isp_dev->regs + REG_META0_VB2_INDEX);
+	meta1_vb2_index = readl(isp_dev->regs + REG_META1_VB2_INDEX);
+	aao_fbc = readl(isp_dev->regs + REG_AAO_FBC_STATUS);
+	afo_fbc = readl(isp_dev->regs + REG_AFO_FBC_STATUS);
+	spin_unlock_irqrestore(&isp_dev->spinlock_irq, flags);
+
+	/* Ignore unnecessary IRQ */
+	if (!irq_status && (!(dma_status & DMA_ST_MASK_CAM)))
+		return IRQ_HANDLED;
+
+	err_status = irq_status & INT_ST_MASK_CAM_ERR;
+
+	/* Sof, done order check */
+	if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST)) {
+		dev_dbg(dev, "sof_done block cnt:%d\n", isp_dev->sof_count);
+
+		/* Notify IRQ event and enqueue frame */
+		irq_handle_notify_event(isp_dev, irq_status, dma_status, 0);
+		isp_dev->current_frame = hw_frame_num;
+		isp_dev->meta0_vb2_index = meta0_vb2_index;
+		isp_dev->meta1_vb2_index = meta1_vb2_index;
+	} else {
+		if (irq_status & SOF_INT_ST) {
+			isp_dev->current_frame = hw_frame_num;
+			isp_dev->meta0_vb2_index = meta0_vb2_index;
+			isp_dev->meta1_vb2_index = meta1_vb2_index;
+		}
+		irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
+	}
+
+	if (irq_status & SOF_INT_ST)
+		cq_num = irq_handle_sof(isp_dev, isp_ctx->scp_mem_iova,
+					hw_frame_num);
+
+	/* Check ISP error status */
+	if (err_status) {
+		dev_err(dev,
+			"raw_int_err:0x%x/0x%x\n",
+			irq_status, err_status);
+		/* Show DMA errors in detail */
+		if (err_status & DMA_ERR_ST)
+			isp_dump_dma_status(isp_dev);
+	}
+
+	if (irq_status & INT_ST_LOG_MASK_CAM)
+		dev_dbg(dev, IRQ_STAT_STR,
+			'A' + cam_idx,
+			isp_dev->sof_count,
+			irq_status,
+			dma_status,
+			hw_frame_num,
+			cq_num,
+			aao_fbc,
+			afo_fbc);
+
+	return IRQ_HANDLED;
+}
+
+static int isp_setup_scp_rproc(struct isp_p1_device *p1_dev)
+{
+	phandle rproc_phandle;
+	struct device *dev = &p1_dev->pdev->dev;
+	int ret;
+
+	p1_dev->scp_pdev = scp_get_pdev(p1_dev->pdev);
+	if (!p1_dev->scp_pdev) {
+		dev_err(dev, "Failed to get scp device\n");
+		return -ENODEV;
+	}
+
+	ret = of_property_read_u32(dev->of_node, "mediatek,scp",
+				   &rproc_phandle);
+	if (ret) {
+		dev_err(dev, "fail to get rproc_phandle:%d\n", ret);
+		return -EINVAL;
+	}
+
+	p1_dev->rproc_handle = rproc_get_by_phandle(rproc_phandle);
+	dev_dbg(dev, "p1 rproc_phandle: 0x%pK\n\n", p1_dev->rproc_handle);
+	if (!p1_dev->rproc_handle) {
+		dev_err(dev, "fail to get rproc_handle\n");
+		return -EINVAL;
+	}
+
+	ret = rproc_boot(p1_dev->rproc_handle);
+	if (ret) {
+		/*
+		 * Return 0 if downloading firmware successfully,
+		 * otherwise it is failed
+		 */
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int isp_init_context(struct isp_p1_device *p1_dev)
+{
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct device *dev = &p1_dev->pdev->dev;
+	unsigned int i;
+
+	dev_dbg(dev, "init irq work thread\n");
+	if (!isp_ctx->isp_deque_thread.thread) {
+		init_waitqueue_head(&isp_ctx->isp_deque_thread.wq);
+		isp_ctx->isp_deque_thread.thread =
+			kthread_run(isp_deque_work, (void *)p1_dev,
+				    "isp_deque_work");
+		if (IS_ERR(isp_ctx->isp_deque_thread.thread)) {
+			dev_err(dev, "unable to alloc kthread\n");
+			isp_ctx->isp_deque_thread.thread = NULL;
+			return -ENOMEM;
+		}
+	}
+	spin_lock_init(&isp_ctx->irq_dequeue_lock);
+	mutex_init(&isp_ctx->lock);
+
+	INIT_LIST_HEAD(&isp_ctx->p1_enqueue_list.queue);
+	atomic_set(&isp_ctx->p1_enqueue_list.queue_cnt, 0);
+
+	for (i = 0; i < ISP_DEV_NODE_NUM; i++)
+		spin_lock_init(&p1_dev->isp_devs[i].spinlock_irq);
+
+	spin_lock_init(&isp_ctx->p1_enqueue_list.lock);
+	spin_lock_init(&isp_ctx->composer_txlist.lock);
+
+	atomic_set(&isp_ctx->irq_data_end, 0);
+	atomic_set(&isp_ctx->irq_data_start, 0);
+
+	return 0;
+}
+
+static int isp_uninit_context(struct device *dev)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct mtk_isp_queue_job *framejob, *tmp_framejob;
+
+	spin_lock_irq(&isp_ctx->p1_enqueue_list.lock);
+	list_for_each_entry_safe(framejob, tmp_framejob,
+				 &isp_ctx->p1_enqueue_list.queue, list_entry) {
+		list_del(&framejob->list_entry);
+		kfree(framejob);
+	}
+	spin_unlock_irq(&isp_ctx->p1_enqueue_list.lock);
+
+	if (isp_ctx->isp_deque_thread.thread) {
+		kthread_stop(isp_ctx->isp_deque_thread.thread);
+		wake_up_interruptible(&isp_ctx->isp_deque_thread.wq);
+		isp_ctx->isp_deque_thread.thread = NULL;
+	}
+
+	mutex_destroy(&isp_ctx->lock);
+
+	return 0;
+}
+
+static unsigned int get_enabled_dma_ports(struct mtk_cam_dev *cam_dev)
+{
+	unsigned int enabled_dma_ports, i;
+
+	/* Get the enabled meta DMA ports */
+	enabled_dma_ports = 0;
+
+	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++)
+		if (cam_dev->vdev_nodes[i].enabled)
+			enabled_dma_ports |=
+				cam_dev->vdev_nodes[i].desc.dma_port;
+
+	dev_dbg(&cam_dev->pdev->dev, "%s :0x%x", __func__, enabled_dma_ports);
+
+	return enabled_dma_ports;
+}
+
+/* Utility functions */
+static unsigned int get_sensor_pixel_id(unsigned int fmt)
+{
+	switch (fmt) {
+	case MEDIA_BUS_FMT_SBGGR8_1X8:
+	case MEDIA_BUS_FMT_SBGGR10_1X10:
+	case MEDIA_BUS_FMT_SBGGR12_1X12:
+	case MEDIA_BUS_FMT_SBGGR14_1X14:
+		return RAW_PXL_ID_B;
+	case MEDIA_BUS_FMT_SGBRG8_1X8:
+	case MEDIA_BUS_FMT_SGBRG10_1X10:
+	case MEDIA_BUS_FMT_SGBRG12_1X12:
+	case MEDIA_BUS_FMT_SGBRG14_1X14:
+		return RAW_PXL_ID_GB;
+	case MEDIA_BUS_FMT_SGRBG8_1X8:
+	case MEDIA_BUS_FMT_SGRBG10_1X10:
+	case MEDIA_BUS_FMT_SGRBG12_1X12:
+	case MEDIA_BUS_FMT_SGRBG14_1X14:
+		return RAW_PXL_ID_GR;
+	case MEDIA_BUS_FMT_SRGGB8_1X8:
+	case MEDIA_BUS_FMT_SRGGB10_1X10:
+	case MEDIA_BUS_FMT_SRGGB12_1X12:
+	case MEDIA_BUS_FMT_SRGGB14_1X14:
+		return RAW_PXL_ID_R;
+	default:
+		return RAW_PXL_ID_B;
+	}
+}
+
+static unsigned int get_sensor_fmt(unsigned int fmt)
+{
+	switch (fmt) {
+	case MEDIA_BUS_FMT_SBGGR8_1X8:
+	case MEDIA_BUS_FMT_SGBRG8_1X8:
+	case MEDIA_BUS_FMT_SGRBG8_1X8:
+	case MEDIA_BUS_FMT_SRGGB8_1X8:
+		return IMG_FMT_BAYER8;
+	case MEDIA_BUS_FMT_SBGGR10_1X10:
+	case MEDIA_BUS_FMT_SGBRG10_1X10:
+	case MEDIA_BUS_FMT_SGRBG10_1X10:
+	case MEDIA_BUS_FMT_SRGGB10_1X10:
+		return IMG_FMT_BAYER10;
+	case MEDIA_BUS_FMT_SBGGR12_1X12:
+	case MEDIA_BUS_FMT_SGBRG12_1X12:
+	case MEDIA_BUS_FMT_SGRBG12_1X12:
+	case MEDIA_BUS_FMT_SRGGB12_1X12:
+		return IMG_FMT_BAYER12;
+	case MEDIA_BUS_FMT_SBGGR14_1X14:
+	case MEDIA_BUS_FMT_SGBRG14_1X14:
+	case MEDIA_BUS_FMT_SGRBG14_1X14:
+	case MEDIA_BUS_FMT_SRGGB14_1X14:
+		return IMG_FMT_BAYER14;
+	default:
+		return IMG_FMT_UNKNOWN;
+	}
+}
+
+static unsigned int get_img_fmt(unsigned int fourcc)
+{
+	switch (fourcc) {
+	case V4L2_PIX_FMT_MTISP_B8:
+		return IMG_FMT_BAYER8;
+	case V4L2_PIX_FMT_MTISP_F8:
+		return IMG_FMT_FG_BAYER8;
+	case V4L2_PIX_FMT_MTISP_B10:
+		return IMG_FMT_BAYER10;
+	case V4L2_PIX_FMT_MTISP_F10:
+		return IMG_FMT_FG_BAYER10;
+	case V4L2_PIX_FMT_MTISP_B12:
+		return IMG_FMT_BAYER12;
+	case V4L2_PIX_FMT_MTISP_F12:
+		return IMG_FMT_FG_BAYER12;
+	case V4L2_PIX_FMT_MTISP_B14:
+		return IMG_FMT_BAYER14;
+	case V4L2_PIX_FMT_MTISP_F14:
+		return IMG_FMT_FG_BAYER14;
+	default:
+		return IMG_FMT_UNKNOWN;
+	}
+}
+
+static unsigned int get_pixel_byte(unsigned int fourcc)
+{
+	switch (fourcc) {
+	case V4L2_PIX_FMT_MTISP_B8:
+	case V4L2_PIX_FMT_MTISP_F8:
+		return 8;
+	case V4L2_PIX_FMT_MTISP_B10:
+	case V4L2_PIX_FMT_MTISP_F10:
+		return 10;
+	case V4L2_PIX_FMT_MTISP_B12:
+	case V4L2_PIX_FMT_MTISP_F12:
+		return 12;
+	case V4L2_PIX_FMT_MTISP_B14:
+	case V4L2_PIX_FMT_MTISP_F14:
+		return 14;
+	default:
+		return 10;
+	}
+}
+
+static void config_img_fmt(struct device *dev, struct p1_img_output *out_fmt,
+			   const struct v4l2_format *in_fmt,
+			   const struct v4l2_subdev_format *sd_format)
+{
+	out_fmt->img_fmt = get_img_fmt(in_fmt->fmt.pix_mp.pixelformat);
+	out_fmt->pixel_byte = get_pixel_byte(in_fmt->fmt.pix_mp.pixelformat);
+	out_fmt->size.w = in_fmt->fmt.pix_mp.width;
+	out_fmt->size.h = in_fmt->fmt.pix_mp.height;
+
+	out_fmt->size.stride = in_fmt->fmt.pix_mp.plane_fmt[0].bytesperline;
+	out_fmt->size.xsize = in_fmt->fmt.pix_mp.plane_fmt[0].bytesperline;
+
+	out_fmt->crop.left = 0x0;
+	out_fmt->crop.top = 0x0;
+
+	out_fmt->crop.width = sd_format->format.width;
+	out_fmt->crop.height = sd_format->format.height;
+
+	WARN_ONCE(in_fmt->fmt.pix_mp.width > out_fmt->crop.width ||
+		  in_fmt->fmt.pix_mp.height > out_fmt->crop.height,
+		  "img out:%d:%d in:%d:%d",
+		  in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height,
+		  out_fmt->crop.width, out_fmt->crop.height);
+
+	dev_dbg(dev, "pixel_byte:%d img_fmt:0x%x\n",
+		out_fmt->pixel_byte,
+		out_fmt->img_fmt);
+	dev_dbg(dev,
+		"param:size=%0dx%0d, stride:%d, xsize:%d, crop=%0dx%0d\n",
+		out_fmt->size.w, out_fmt->size.h,
+		out_fmt->size.stride, out_fmt->size.xsize,
+		out_fmt->crop.width, out_fmt->crop.height);
+}
+
+/* ISP P1 interface functions */
+int mtk_isp_power_init(struct mtk_cam_dev *cam_dev)
+{
+	struct device *dev = &cam_dev->pdev->dev;
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	int ret;
+
+	ret = isp_setup_scp_rproc(p1_dev);
+	if (ret)
+		return ret;
+
+	ret = isp_init_context(p1_dev);
+	if (ret)
+		return ret;
+
+	ret = isp_composer_init(dev);
+	if (ret)
+		goto composer_err;
+
+	pm_runtime_get_sync(dev);
+
+	/* ISP HW INIT */
+	isp_ctx->isp_hw_module = ISP_CAM_B_IDX;
+	/* Use pure RAW as default HW path */
+	isp_ctx->isp_raw_path = ISP_PURE_RAW_PATH;
+	atomic_set(&p1_dev->cam_dev.streamed_node_count, 0);
+
+	isp_composer_hw_init(dev);
+	/* Check enabled DMAs which is configured by media setup */
+	isp_composer_meta_config(dev, get_enabled_dma_ports(cam_dev));
+
+	dev_dbg(dev, "%s done\n", __func__);
+
+	return 0;
+
+composer_err:
+	isp_uninit_context(dev);
+
+	return ret;
+}
+
+int mtk_isp_power_release(struct device *dev)
+{
+	isp_composer_hw_deinit(dev);
+	isp_uninit_context(dev);
+
+	dev_dbg(dev, "%s done\n", __func__);
+
+	return 0;
+}
+
+int mtk_isp_config(struct device *dev)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct p1_config_param config_param;
+	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
+	struct v4l2_subdev_format sd_fmt;
+	unsigned int enabled_dma_ports;
+	struct v4l2_format *img_fmt;
+	int ret;
+
+	p1_dev->isp_devs[isp_ctx->isp_hw_module].current_frame = 0;
+	p1_dev->isp_devs[isp_ctx->isp_hw_module].sof_count = 0;
+
+	isp_ctx->frame_seq_no = 1;
+	atomic_set(&isp_ctx->composed_frame_id, 0);
+
+	/* Get the enabled DMA ports */
+	enabled_dma_ports = get_enabled_dma_ports(cam_dev);
+	dev_dbg(dev, "%s enable_dma_ports:0x%x", __func__, enabled_dma_ports);
+
+	/* Sensor config */
+	sd_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+	ret = v4l2_subdev_call(cam_dev->sensor, pad, get_fmt, NULL, &sd_fmt);
+
+	if (ret) {
+		dev_dbg(dev, "sensor g_fmt on failed:%d\n", ret);
+		return -EPERM;
+	}
+
+	dev_dbg(dev,
+		"get_fmt ret=%d, w=%d, h=%d, code=0x%x, field=%d, color=%d\n",
+		ret, sd_fmt.format.width, sd_fmt.format.height,
+		sd_fmt.format.code, sd_fmt.format.field,
+		sd_fmt.format.colorspace);
+
+	config_param.cfg_in_param.continuous = 0x1;
+	config_param.cfg_in_param.subsample = 0x0;
+	/* Fix to one pixel mode in default */
+	config_param.cfg_in_param.pixel_mode = 0x1;
+	/* Support normal pattern in default */
+	config_param.cfg_in_param.data_pattern = 0x0;
+
+	config_param.cfg_in_param.crop.left = 0x0;
+	config_param.cfg_in_param.crop.top = 0x0;
+
+	config_param.cfg_in_param.raw_pixel_id =
+		get_sensor_pixel_id(sd_fmt.format.code);
+	config_param.cfg_in_param.img_fmt = get_sensor_fmt(sd_fmt.format.code);
+	config_param.cfg_in_param.crop.width = sd_fmt.format.width;
+	config_param.cfg_in_param.crop.height = sd_fmt.format.height;
+
+	config_param.cfg_main_param.bypass = 1;
+	img_fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_MAIN_STREAM_OUT].vdev_fmt;
+	if ((enabled_dma_ports & R_IMGO) == R_IMGO) {
+		config_param.cfg_main_param.bypass = 0;
+		config_param.cfg_main_param.pure_raw = isp_ctx->isp_raw_path;
+		config_param.cfg_main_param.pure_raw_pack = 1;
+		config_img_fmt(dev, &config_param.cfg_main_param.output,
+			       img_fmt, &sd_fmt);
+	}
+
+	config_param.cfg_resize_param.bypass = 1;
+	img_fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_PACKED_BIN_OUT].vdev_fmt;
+	if ((enabled_dma_ports & R_RRZO) == R_RRZO) {
+		config_param.cfg_resize_param.bypass = 0;
+		config_img_fmt(dev, &config_param.cfg_resize_param.output,
+			       img_fmt, &sd_fmt);
+	}
+
+	/* Configure meta DMAs info. */
+	config_param.cfg_meta_param.enabled_meta_dmas = enabled_dma_ports;
+
+	isp_composer_hw_config(dev, &config_param);
+
+	dev_dbg(dev, "%s done\n", __func__);
+
+	return 0;
+}
+
+void mtk_isp_enqueue(struct device *dev, unsigned int dma_port,
+		     struct mtk_cam_dev_buffer *buffer)
+{
+	struct mtk_isp_scp_p1_cmd frameparams;
+
+	memset(&frameparams, 0, sizeof(frameparams));
+	frameparams.cmd_id = ISP_CMD_ENQUEUE_META;
+	frameparams.meta_frame.enabled_dma = dma_port;
+	frameparams.meta_frame.vb_index = buffer->vbb.vb2_buf.index;
+	frameparams.meta_frame.meta_addr.iova = buffer->daddr;
+	frameparams.meta_frame.meta_addr.scp_addr = buffer->scp_addr;
+
+	isp_composer_enqueue(dev, &frameparams, SCP_ISP_CMD);
+}
+
+void mtk_isp_req_flush_buffers(struct device *dev)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_queue_job *job, *j0;
+	struct mtk_cam_dev_buffer *buf, *b0;
+	struct isp_queue *p1_list = &p1_dev->isp_ctx.p1_enqueue_list;
+
+	if (!atomic_read(&p1_list->queue_cnt))
+		return;
+
+	spin_lock(&p1_list->lock);
+	list_for_each_entry_safe(job, j0, &p1_list->queue, list_entry) {
+		list_for_each_entry_safe(buf, b0, &job->list_buf, list) {
+			list_del(&buf->list);
+			if (buf->vbb.vb2_buf.state == VB2_BUF_STATE_ACTIVE)
+				vb2_buffer_done(&buf->vbb.vb2_buf,
+						VB2_BUF_STATE_ERROR);
+		}
+		list_del(&job->list_entry);
+		atomic_dec(&p1_list->queue_cnt);
+		kfree(job);
+	}
+	spin_unlock(&p1_list->lock);
+}
+
+void mtk_isp_req_enqueue(struct device *dev, struct media_request *req)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	struct p1_frame_param frameparams;
+	struct mtk_isp_queue_job *framejob;
+	struct media_request_object *obj, *obj_safe;
+	struct vb2_buffer *vb;
+	struct mtk_cam_dev_buffer *buf;
+
+	framejob = kzalloc(sizeof(*framejob), GFP_ATOMIC);
+	memset(framejob, 0, sizeof(*framejob));
+	memset(&frameparams, 0, sizeof(frameparams));
+	INIT_LIST_HEAD(&framejob->list_buf);
+
+	frameparams.frame_seq_no = isp_ctx->frame_seq_no++;
+	frameparams.sof_idx =
+		p1_dev->isp_devs[isp_ctx->isp_hw_module].sof_count;
+	framejob->frame_seq_no = frameparams.frame_seq_no;
+
+	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
+		vb = container_of(obj, struct vb2_buffer, req_obj);
+		buf = container_of(vb, struct mtk_cam_dev_buffer, vbb.vb2_buf);
+		framejob->request_fd = buf->vbb.request_fd;
+		frameparams.dma_buffers[buf->node_id].iova = buf->daddr;
+		frameparams.dma_buffers[buf->node_id].scp_addr = buf->scp_addr;
+		list_add_tail(&buf->list, &framejob->list_buf);
+	}
+
+	spin_lock(&isp_ctx->p1_enqueue_list.lock);
+	list_add_tail(&framejob->list_entry, &isp_ctx->p1_enqueue_list.queue);
+	atomic_inc(&isp_ctx->p1_enqueue_list.queue_cnt);
+	spin_unlock(&isp_ctx->p1_enqueue_list.lock);
+
+	isp_composer_enqueue(dev, &frameparams, SCP_ISP_FRAME);
+	dev_dbg(dev, "request fd:%d frame_seq_no:%d is queued cnt:%d\n",
+		framejob->request_fd,
+		frameparams.frame_seq_no,
+		atomic_read(&isp_ctx->p1_enqueue_list.queue_cnt));
+}
+
+static int enable_sys_clock(struct isp_p1_device *p1_dev)
+{
+	struct device *dev = &p1_dev->pdev->dev;
+	int ret;
+
+	dev_info(dev, "- %s\n", __func__);
+
+	ret = clk_bulk_prepare_enable(p1_dev->isp_ctx.num_clks,
+				      p1_dev->isp_ctx.clk_list);
+	if (ret)
+		goto clk_err;
+
+	return 0;
+
+clk_err:
+	dev_err(dev, "cannot pre-en isp_cam clock:%d\n", ret);
+	clk_bulk_disable_unprepare(p1_dev->isp_ctx.num_clks,
+				   p1_dev->isp_ctx.clk_list);
+	return ret;
+}
+
+static void disable_sys_clock(struct isp_p1_device *p1_dev)
+{
+	dev_info(&p1_dev->pdev->dev, "- %s\n", __func__);
+	clk_bulk_disable_unprepare(p1_dev->isp_ctx.num_clks,
+				   p1_dev->isp_ctx.clk_list);
+}
+
+static int mtk_isp_suspend(struct device *dev)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	int module = p1_dev->isp_ctx.isp_hw_module;
+	struct isp_device *isp_dev = &p1_dev->isp_devs[module];
+	unsigned int reg_val;
+
+	dev_dbg(dev, "- %s\n", __func__);
+
+	isp_dev = &p1_dev->isp_devs[module];
+	reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
+	if (reg_val & VFDATA_EN_BIT) {
+		dev_dbg(dev, "Cam:%d suspend, disable VF\n", module);
+		/* Disable view finder */
+		writel((reg_val & (~VFDATA_EN_BIT)),
+		       isp_dev->regs + REG_TG_VF_CON);
+		/*
+		 * After VF enable, the TG frame count will be reset to 0;
+		 */
+		reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
+		writel((reg_val & (~CMOS_EN_BIT)),
+		       isp_dev->regs +  + REG_TG_SEN_MODE);
+	}
+
+	disable_sys_clock(p1_dev);
+
+	return 0;
+}
+
+static int mtk_isp_resume(struct device *dev)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	int module = p1_dev->isp_ctx.isp_hw_module;
+	struct isp_device *isp_dev = &p1_dev->isp_devs[module];
+	unsigned int reg_val;
+
+	dev_dbg(dev, "- %s\n", __func__);
+
+	enable_sys_clock(p1_dev);
+
+	/* V4L2 stream-on phase & restore HW stream-on status */
+	if (p1_dev->cam_dev.streaming) {
+		dev_dbg(dev, "Cam:%d resume,enable VF\n", module);
+		/* Enable CMOS */
+		reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
+		writel((reg_val | CMOS_EN_BIT),
+		       isp_dev->regs + REG_TG_SEN_MODE);
+		/* Enable VF */
+		reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
+		writel((reg_val | VFDATA_EN_BIT),
+		       isp_dev->regs + REG_TG_VF_CON);
+	}
+
+	return 0;
+}
+
+static int mtk_isp_probe(struct platform_device *pdev)
+{
+	struct isp_p1_device *p1_dev;
+	struct mtk_isp_p1_ctx *isp_ctx;
+	struct isp_device *isp_dev;
+	struct device *dev = &pdev->dev;
+	struct resource *res;
+	int irq;
+	int ret;
+	unsigned int i;
+
+	p1_dev = devm_kzalloc(dev, sizeof(*p1_dev), GFP_KERNEL);
+	if (!p1_dev)
+		return -ENOMEM;
+
+	dev_set_drvdata(dev, p1_dev);
+	isp_ctx = &p1_dev->isp_ctx;
+	p1_dev->pdev = pdev;
+
+	for (i = ISP_CAMSYS_CONFIG_IDX; i < ISP_DEV_NODE_NUM; i++) {
+		isp_dev = &p1_dev->isp_devs[i];
+		isp_dev->isp_hw_module = i;
+		isp_dev->dev = dev;
+		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
+		isp_dev->regs = devm_ioremap_resource(dev, res);
+
+		dev_dbg(dev, "cam%u, map_addr=0x%lx\n",
+			i, (unsigned long)isp_dev->regs);
+
+		if (!isp_dev->regs)
+			return PTR_ERR(isp_dev->regs);
+
+		/* Support IRQ from ISP_CAM_A_IDX */
+		if (i >= ISP_CAM_A_IDX) {
+			/* Reg & interrupts index is shifted with 1  */
+			irq = platform_get_irq(pdev, i - 1);
+			if (irq) {
+				ret = devm_request_irq(dev, irq,
+						       isp_irq_cam,
+						       IRQF_SHARED,
+						       dev_driver_string(dev),
+						       (void *)isp_dev);
+				if (ret) {
+					dev_err(dev,
+						"req_irq fail, dev:%s irq=%d\n",
+						dev->of_node->name,
+						irq);
+					return ret;
+				}
+				dev_dbg(dev, "Registered irq=%d, ISR:%s\n",
+					irq, dev_driver_string(dev));
+			}
+		}
+		spin_lock_init(&isp_dev->spinlock_irq);
+	}
+
+	p1_dev->isp_ctx.num_clks = ARRAY_SIZE(mtk_isp_clks);
+	p1_dev->isp_ctx.clk_list =
+		devm_kcalloc(dev,
+			     p1_dev->isp_ctx.num_clks,
+			     sizeof(*p1_dev->isp_ctx.clk_list),
+			     GFP_KERNEL);
+	if (!p1_dev->isp_ctx.clk_list)
+		return -ENOMEM;
+
+	for (i = 0; i < p1_dev->isp_ctx.num_clks; ++i)
+		p1_dev->isp_ctx.clk_list->id = mtk_isp_clks[i];
+
+	ret = devm_clk_bulk_get(dev,
+				p1_dev->isp_ctx.num_clks,
+				p1_dev->isp_ctx.clk_list);
+	if (ret) {
+		dev_err(dev, "cannot get isp cam clock:%d\n", ret);
+		return ret;
+	}
+
+	/* Initialize reserved DMA memory */
+	ret = mtk_cam_reserved_memory_init(p1_dev);
+	if (ret) {
+		dev_err(dev, "failed to configure DMA memory:%d\n", ret);
+		goto err_init;
+	}
+
+	/* Initialize the v4l2 common part */
+	ret = mtk_cam_dev_init(pdev, &p1_dev->cam_dev);
+	if (ret)
+		goto err_init;
+
+	spin_lock_init(&isp_ctx->p1_enqueue_list.lock);
+	pm_runtime_enable(dev);
+
+	return 0;
+
+err_init:
+	if (p1_dev->cam_dev.smem_dev)
+		device_unregister(p1_dev->cam_dev.smem_dev);
+
+	return ret;
+}
+
+static int mtk_isp_remove(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct isp_p1_device *p1_dev = dev_get_drvdata(dev);
+
+	pm_runtime_disable(dev);
+	mtk_cam_dev_release(pdev, &p1_dev->cam_dev);
+
+	return 0;
+}
+
+static const struct dev_pm_ops mtk_isp_pm_ops = {
+	SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
+	SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)
+};
+
+static struct platform_driver mtk_isp_driver = {
+	.probe   = mtk_isp_probe,
+	.remove  = mtk_isp_remove,
+	.driver  = {
+		.name  = "mtk-cam",
+		.of_match_table = of_match_ptr(mtk_isp_of_ids),
+		.pm     = &mtk_isp_pm_ops,
+	}
+};
+
+module_platform_driver(mtk_isp_driver);
+
+MODULE_DESCRIPTION("Camera ISP driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
new file mode 100644
index 000000000000..6af3f569664c
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
@@ -0,0 +1,243 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ */
+
+#ifndef __CAMERA_ISP_H
+#define __CAMERA_ISP_H
+
+#include <linux/cdev.h>
+#include <linux/clk.h>
+#include <linux/interrupt.h>
+#include <linux/ioctl.h>
+#include <linux/irqreturn.h>
+#include <linux/miscdevice.h>
+#include <linux/pm_qos.h>
+#include <linux/scatterlist.h>
+
+#include "mtk_cam-scp.h"
+#include "mtk_cam-v4l2-util.h"
+
+#define CAM_A_MAX_WIDTH		3328
+#define CAM_A_MAX_HEIGHT		2496
+#define CAM_B_MAX_WIDTH		5376
+#define CAM_B_MAX_HEIGHT		4032
+
+#define CAM_MIN_WIDTH			80
+#define CAM_MIN_HEIGHT			60
+
+#define IMG_MAX_WIDTH			CAM_B_MAX_WIDTH
+#define IMG_MAX_HEIGHT			CAM_B_MAX_HEIGHT
+#define IMG_MIN_WIDTH			CAM_MIN_WIDTH
+#define IMG_MIN_HEIGHT			CAM_MIN_HEIGHT
+
+#define RRZ_MAX_WIDTH			CAM_B_MAX_WIDTH
+#define RRZ_MAX_HEIGHT			CAM_B_MAX_HEIGHT
+#define RRZ_MIN_WIDTH			CAM_MIN_WIDTH
+#define RRZ_MIN_HEIGHT			CAM_MIN_HEIGHT
+
+#define R_IMGO				BIT(0)
+#define R_RRZO				BIT(1)
+#define R_AAO				BIT(3)
+#define R_AFO				BIT(4)
+#define R_LCSO				BIT(5)
+#define R_PDO				BIT(6)
+#define R_LMVO				BIT(7)
+#define R_FLKO				BIT(8)
+#define R_RSSO				BIT(9)
+#define R_PSO				BIT(10)
+
+#define CQ_BUFFER_COUNT		3
+#define IRQ_DATA_BUF_SIZE		4
+#define CQ_ADDRESS_OFFSET		0x640
+
+#define ISP_COMPOSING_MAX_NUM		4
+#define ISP_FRAME_COMPOSING_MAX_NUM	3
+
+#define IRQ_STAT_STR	"cam%c, SOF_%d irq(0x%x), " \
+			"dma(0x%x), frame_num(%d)/cq_num(%d), " \
+			"fbc1(0x%x), fbc2(0x%x)\n"
+
+/*
+ * In order with the sequence of device nodes defined in dtsi rule,
+ * one hardware module should be mapping to one node.
+ */
+enum isp_dev_node_enum {
+	ISP_CAMSYS_CONFIG_IDX = 0,
+	ISP_CAM_UNI_IDX,
+	ISP_CAM_A_IDX,
+	ISP_CAM_B_IDX,
+	ISP_DEV_NODE_NUM
+};
+
+/* Image RAW path for ISP P1 module. */
+enum isp_raw_path_enum {
+	ISP_PROCESS_RAW_PATH = 0,
+	ISP_PURE_RAW_PATH
+};
+
+/* State for struct mtk_isp_p1_ctx: composer_state */
+enum  {
+	SCP_ON = 0,
+	SCP_OFF
+};
+
+enum {
+	IMG_FMT_UNKNOWN		= 0x0000,
+	IMG_FMT_RAW_START	= 0x2200,
+	IMG_FMT_BAYER8		= IMG_FMT_RAW_START,
+	IMG_FMT_BAYER10,
+	IMG_FMT_BAYER12,
+	IMG_FMT_BAYER14,
+	IMG_FMT_FG_BAYER8,
+	IMG_FMT_FG_BAYER10,
+	IMG_FMT_FG_BAYER12,
+	IMG_FMT_FG_BAYER14,
+};
+
+enum {
+	RAW_PXL_ID_B = 0,
+	RAW_PXL_ID_GB,
+	RAW_PXL_ID_GR,
+	RAW_PXL_ID_R
+};
+
+struct isp_queue {
+	struct list_head queue;
+	atomic_t queue_cnt;
+	spinlock_t lock; /* queue attributes protection */
+};
+
+struct isp_thread {
+	struct task_struct *thread;
+	wait_queue_head_t wq;
+};
+
+struct mtk_isp_queue_work {
+	union {
+		struct mtk_isp_scp_p1_cmd cmd;
+		struct p1_frame_param frameparams;
+	};
+	struct list_head list_entry;
+	enum mtk_isp_scp_type type;
+};
+
+struct mtk_cam_dev_stat_event_data {
+	__u32 frame_seq_no;
+	__u32 meta0_vb2_index;
+	__u32 meta1_vb2_index;
+	__u32 irq_status_mask;
+	__u32 dma_status_mask;
+};
+
+struct mtk_isp_queue_job {
+	struct list_head list_entry;
+	struct list_head list_buf;
+	unsigned int request_fd;
+	unsigned int frame_seq_no;
+};
+
+/*
+ * struct isp_device - the ISP device information
+ *
+ * @dev: Pointer to struct device
+ * @regs: Camera ISP base register address
+ * @spinlock_irq: Used to protect register read/write data
+ * @current_frame: Current frame sequence number, set when SOF
+ * @meta0_vb2_index: Meta0 vb2 buffer index, set when SOF
+ * @meta1_vb2_index: Meta1 vb2 buffer index, set when SOF
+ * @sof_count: The accumulated SOF counter
+ * @isp_hw_module: Identity camera A or B
+ *
+ */
+struct isp_device {
+	struct device *dev;
+	void __iomem *regs;
+	spinlock_t spinlock_irq; /* ISP reg setting integrity */
+	unsigned int current_frame;
+	unsigned int meta0_vb2_index;
+	unsigned int meta1_vb2_index;
+	u8 sof_count;
+	u8 isp_hw_module;
+};
+
+/*
+ * struct mtk_isp_p1_ctx - the ISP device information
+ *
+ * @composer_txlist: Queue for SCP TX data including SCP_ISP_CMD & SCP_ISP_FRAME
+ * @composer_tx_thread: TX Thread for SCP data tranmission
+ * @cmd_queued: The number of SCP_ISP_CMD commands will be sent
+ * @ipi_occupied: The total number of SCP TX data has beent sent
+ * @scp_state: The state of SCP control
+ * @composing_frame: The total number of SCP_ISP_FRAME has beent sent
+ * @composed_frame_id: The ack. frame sequence by SCP
+ * @composer_deinit_thread: The de-initialized thread
+ * @p1_enqueue_list: Queue for ISP frame buffers
+ * @isp_deque_thread: Thread for handling ISP frame buffers dequeue
+ * @irq_event_datas: Ring buffer for struct mtk_cam_dev_stat_event_data data
+ * @irq_data_start: Start index of irq_event_datas ring buffer
+ * @irq_data_end: End index of irq_event_datas ring buffer
+ * @irq_dequeue_lock: Lock to protect irq_event_datas ring buffer
+ * @scp_mem_pa: DMA address for SCP device
+ * @scp_mem_iova: DMA address for ISP HW DMA devices
+ * @frame_seq_no: Sequence number for ISP frame buffer
+ * @isp_hw_module: Active camera HW module
+ * @num_clks: The number of driver's clock
+ * @clk_list: The list of clock data
+ * @lock: Lock to protect context operations
+ *
+ */
+struct mtk_isp_p1_ctx {
+	struct isp_queue composer_txlist;
+	struct isp_thread composer_tx_thread;
+	atomic_t cmd_queued;
+	atomic_t ipi_occupied;
+	atomic_t scp_state;
+	atomic_t composing_frame;
+	atomic_t composed_frame_id;
+	struct isp_thread composer_deinit_thread;
+	struct isp_queue p1_enqueue_list;
+	struct isp_thread isp_deque_thread;
+	struct mtk_cam_dev_stat_event_data irq_event_datas[IRQ_DATA_BUF_SIZE];
+	atomic_t irq_data_start;
+	atomic_t irq_data_end;
+	spinlock_t irq_dequeue_lock; /* ISP frame dequeuq protection */
+	dma_addr_t scp_mem_pa;
+	dma_addr_t scp_mem_iova;
+	int frame_seq_no;
+	unsigned int isp_hw_module;
+	unsigned int isp_raw_path;
+	unsigned int num_clks;
+	struct clk_bulk_data *clk_list;
+	struct mutex lock; /* Protect context operations */
+};
+
+struct isp_p1_device {
+	struct platform_device *pdev;
+	struct platform_device *scp_pdev;
+	struct rproc *rproc_handle;
+	struct mtk_isp_p1_ctx isp_ctx;
+	struct mtk_cam_dev cam_dev;
+	struct isp_device isp_devs[ISP_DEV_NODE_NUM];
+};
+
+static inline struct isp_p1_device *
+p1_ctx_to_dev(const struct mtk_isp_p1_ctx *__p1_ctx)
+{
+	return container_of(__p1_ctx, struct isp_p1_device, isp_ctx);
+}
+
+static inline struct isp_p1_device *get_p1_device(struct device *dev)
+{
+	return ((struct isp_p1_device *)dev_get_drvdata(dev));
+}
+
+int mtk_isp_power_init(struct mtk_cam_dev *cam_dev);
+int mtk_isp_power_release(struct device *dev);
+int mtk_isp_config(struct device *dev);
+void mtk_isp_req_enqueue(struct device *dev, struct media_request *req);
+void mtk_isp_enqueue(struct device *dev, unsigned int dma_port,
+		     struct mtk_cam_dev_buffer *buffer);
+void mtk_isp_req_flush_buffers(struct device *dev);
+
+#endif /*__CAMERA_ISP_H*/
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
                   ` (6 preceding siblings ...)
  2019-06-11  3:53 ` [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-07-10  9:58   ` Tomasz Figa
  2019-06-11  3:53 ` [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device Jungo Lin
  8 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

This patch adds communication with the co-processor on the SoC
through the SCP driver. It supports bi-directional commands
to exchange data and perform command flow control function.

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
This patch depends on "Add support for mt8183 SCP"[1].

[1] https://patchwork.kernel.org/cover/10972143/
---
 .../platform/mtk-isp/isp_50/cam/Makefile      |   1 +
 .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.c | 371 ++++++++++++++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.h | 207 ++++++++++
 3 files changed, 579 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h

diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
index 30df10983f6a..95f0b1c8fa1c 100644
--- a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
@@ -3,5 +3,6 @@
 mtk-cam-isp-objs += mtk_cam-ctrl.o
 mtk-cam-isp-objs += mtk_cam-v4l2-util.o
 mtk-cam-isp-objs += mtk_cam.o
+mtk-cam-isp-objs += mtk_cam-scp.o
 
 obj-$(CONFIG_VIDEO_MEDIATEK_ISP_PASS1) += mtk-cam-isp.o
\ No newline at end of file
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
new file mode 100644
index 000000000000..04519d0b942f
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
@@ -0,0 +1,371 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+
+#include <linux/atomic.h>
+#include <linux/kthread.h>
+#include <linux/platform_data/mtk_scp.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/sched.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/vmalloc.h>
+
+#include "mtk_cam.h"
+
+static void isp_composer_deinit(struct mtk_isp_p1_ctx *isp_ctx)
+{
+	struct mtk_isp_queue_work *ipi_job, *tmp_ipi_job;
+	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
+
+	atomic_set(&isp_ctx->cmd_queued, 0);
+	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
+	atomic_set(&isp_ctx->composing_frame, 0);
+	atomic_set(&isp_ctx->ipi_occupied, 0);
+
+	spin_lock(&isp_ctx->composer_txlist.lock);
+	list_for_each_entry_safe(ipi_job, tmp_ipi_job,
+				 &isp_ctx->composer_txlist.queue,
+				 list_entry) {
+		list_del(&ipi_job->list_entry);
+		kfree(ipi_job);
+	}
+	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
+	spin_unlock(&isp_ctx->composer_txlist.lock);
+
+	mutex_lock(&isp_ctx->lock);
+	if (isp_ctx->composer_tx_thread.thread) {
+		kthread_stop(isp_ctx->composer_tx_thread.thread);
+		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
+		isp_ctx->composer_tx_thread.thread = NULL;
+	}
+
+	if (isp_ctx->composer_deinit_thread.thread) {
+		wake_up(&isp_ctx->composer_deinit_thread.wq);
+		isp_ctx->composer_deinit_thread.thread = NULL;
+	}
+	mutex_unlock(&isp_ctx->lock);
+
+	pm_runtime_put_sync(&p1_dev->pdev->dev);
+}
+
+/*
+ * Two kinds of flow control in isp_composer_tx_work.
+ *
+ * Case 1: IPI commands flow control. The maximum number of command queues is 3.
+ * There are two types of IPI commands (SCP_ISP_CMD/SCP_ISP_FRAME) in P1 driver.
+ * It is controlled by ipi_occupied.
+ * The priority of SCP_ISP_CMD is higher than SCP_ISP_FRAME.
+ *
+ * Case 2: Frame buffers flow control. The maximum number of frame buffers is 3.
+ * It is controlled by composing_frame.
+ * Frame buffer is sent by SCP_ISP_FRAME command.
+ */
+static int isp_composer_tx_work(void *data)
+{
+	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)data;
+	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
+	struct device *dev = &p1_dev->pdev->dev;
+	struct mtk_isp_queue_work *isp_composer_work, *tmp_ipi_job;
+	struct isp_queue *composer_txlist = &isp_ctx->composer_txlist;
+	int ret;
+
+	while (1) {
+		ret = wait_event_interruptible
+			(isp_ctx->composer_tx_thread.wq,
+			 (atomic_read(&composer_txlist->queue_cnt) > 0 &&
+			 atomic_read(&isp_ctx->ipi_occupied)
+				< ISP_COMPOSING_MAX_NUM &&
+			 atomic_read(&isp_ctx->composing_frame)
+				< ISP_FRAME_COMPOSING_MAX_NUM) ||
+			 (atomic_read(&isp_ctx->cmd_queued) > 0 &&
+			 atomic_read(&isp_ctx->ipi_occupied)
+				< ISP_COMPOSING_MAX_NUM) ||
+			 kthread_should_stop());
+
+		if (kthread_should_stop())
+			break;
+
+		spin_lock(&composer_txlist->lock);
+		if (atomic_read(&isp_ctx->cmd_queued) > 0) {
+			list_for_each_entry_safe(isp_composer_work, tmp_ipi_job,
+						 &composer_txlist->queue,
+						 list_entry) {
+				if (isp_composer_work->type == SCP_ISP_CMD) {
+					dev_dbg(dev, "Found a cmd\n");
+					break;
+				}
+			}
+		} else {
+			if (atomic_read(&isp_ctx->composing_frame) >=
+				ISP_FRAME_COMPOSING_MAX_NUM) {
+				spin_unlock(&composer_txlist->lock);
+				continue;
+			}
+			isp_composer_work =
+			    list_first_entry_or_null
+				(&composer_txlist->queue,
+				 struct mtk_isp_queue_work,
+				 list_entry);
+		}
+
+		list_del(&isp_composer_work->list_entry);
+		atomic_dec(&composer_txlist->queue_cnt);
+		spin_unlock(&composer_txlist->lock);
+
+		if (isp_composer_work->type == SCP_ISP_CMD) {
+			scp_ipi_send
+				(p1_dev->scp_pdev,
+				 SCP_IPI_ISP_CMD,
+				 &isp_composer_work->cmd,
+				 sizeof(isp_composer_work->cmd),
+				 0);
+			atomic_dec(&isp_ctx->cmd_queued);
+			atomic_inc(&isp_ctx->ipi_occupied);
+			dev_dbg(dev,
+				"%s cmd id %d sent, %d ipi buf occupied",
+				__func__,
+				isp_composer_work->cmd.cmd_id,
+				atomic_read(&isp_ctx->ipi_occupied));
+		} else if (isp_composer_work->type == SCP_ISP_FRAME) {
+			scp_ipi_send
+				(p1_dev->scp_pdev,
+				 SCP_IPI_ISP_FRAME,
+				 &isp_composer_work->frameparams,
+				 sizeof(isp_composer_work->frameparams),
+				 0);
+			atomic_inc(&isp_ctx->ipi_occupied);
+			atomic_inc(&isp_ctx->composing_frame);
+			dev_dbg(dev,
+				"%s frame %d sent, %d ipi, %d CQ bufs occupied",
+				__func__,
+				isp_composer_work->frameparams.frame_seq_no,
+				atomic_read(&isp_ctx->ipi_occupied),
+				atomic_read(&isp_ctx->composing_frame));
+		} else {
+			dev_err(dev,
+				"ignore IPI type: %d!\n",
+				isp_composer_work->type);
+		}
+		kfree(isp_composer_work);
+	}
+	return ret;
+}
+
+static int isp_composer_deinit_work(void *data)
+{
+	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)data;
+	struct isp_p1_device *p1_dev = p1_ctx_to_dev(data);
+	struct device *dev = &p1_dev->pdev->dev;
+
+	wait_event_interruptible(isp_ctx->composer_deinit_thread.wq,
+				 atomic_read(&isp_ctx->scp_state) == SCP_OFF ||
+				 kthread_should_stop());
+
+	dev_dbg(dev, "%s run deinit", __func__);
+	isp_composer_deinit(isp_ctx);
+
+	return 0;
+}
+
+static void isp_composer_handler(void *data, unsigned int len, void *priv)
+{
+	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)priv;
+	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
+	struct device *dev = &p1_dev->pdev->dev;
+	struct mtk_isp_scp_p1_cmd *ipi_msg;
+
+	ipi_msg = (struct mtk_isp_scp_p1_cmd *)data;
+
+	if (ipi_msg->cmd_id != ISP_CMD_ACK)
+		return;
+
+	if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
+		dev_dbg(dev, "ack frame_num:%d",
+			ipi_msg->ack_info.frame_seq_no);
+		atomic_set(&isp_ctx->composed_frame_id,
+			   ipi_msg->ack_info.frame_seq_no);
+	} else if (ipi_msg->ack_info.cmd_id == ISP_CMD_DEINIT) {
+		dev_dbg(dev, "ISP_CMD_DEINIT is acked");
+		atomic_set(&isp_ctx->scp_state, SCP_OFF);
+		wake_up_interruptible(&isp_ctx->composer_deinit_thread.wq);
+	}
+
+	atomic_dec_return(&isp_ctx->ipi_occupied);
+	wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
+}
+
+int isp_composer_init(struct device *dev)
+{
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	int ret;
+
+	ret = scp_ipi_register(p1_dev->scp_pdev,
+			       SCP_IPI_ISP_CMD,
+			       isp_composer_handler,
+			       isp_ctx);
+	if (ret)
+		return ret;
+
+	atomic_set(&isp_ctx->cmd_queued, 0);
+	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
+	atomic_set(&isp_ctx->composing_frame, 0);
+	atomic_set(&isp_ctx->ipi_occupied, 0);
+	atomic_set(&isp_ctx->scp_state, SCP_ON);
+
+	mutex_lock(&isp_ctx->lock);
+	if (!isp_ctx->composer_tx_thread.thread) {
+		init_waitqueue_head(&isp_ctx->composer_tx_thread.wq);
+		INIT_LIST_HEAD(&isp_ctx->composer_txlist.queue);
+		spin_lock_init(&isp_ctx->composer_txlist.lock);
+		isp_ctx->composer_tx_thread.thread =
+			kthread_run(isp_composer_tx_work, isp_ctx,
+				    "isp_composer_tx");
+		if (IS_ERR(isp_ctx->composer_tx_thread.thread)) {
+			dev_err(dev, "unable to start kthread\n");
+			isp_ctx->composer_tx_thread.thread = NULL;
+			goto nomem;
+		}
+	} else {
+		dev_warn(dev, "old tx thread is existed\n");
+	}
+
+	if (!isp_ctx->composer_deinit_thread.thread) {
+		init_waitqueue_head(&isp_ctx->composer_deinit_thread.wq);
+		isp_ctx->composer_deinit_thread.thread =
+			kthread_run(isp_composer_deinit_work, isp_ctx,
+				    "isp_composer_deinit_work");
+		if (IS_ERR(isp_ctx->composer_deinit_thread.thread)) {
+			dev_err(dev, "unable to start kthread\n");
+			isp_ctx->composer_deinit_thread.thread = NULL;
+			goto nomem;
+		}
+	} else {
+		dev_warn(dev, "old rx thread is existed\n");
+	}
+	mutex_unlock(&isp_ctx->lock);
+
+	return 0;
+
+nomem:
+	mutex_unlock(&isp_ctx->lock);
+
+	return -ENOMEM;
+}
+
+void isp_composer_enqueue(struct device *dev,
+			  void *data,
+			  enum mtk_isp_scp_type type)
+{
+	struct mtk_isp_queue_work *isp_composer_work;
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+
+	isp_composer_work = kzalloc(sizeof(*isp_composer_work), GFP_KERNEL);
+	isp_composer_work->type = type;
+
+	switch (type) {
+	case SCP_ISP_CMD:
+		memcpy(&isp_composer_work->cmd, data,
+		       sizeof(isp_composer_work->cmd));
+		dev_dbg(dev, "Enq ipi cmd id:%d\n",
+			isp_composer_work->cmd.cmd_id);
+
+		spin_lock(&isp_ctx->composer_txlist.lock);
+		list_add_tail(&isp_composer_work->list_entry,
+			      &isp_ctx->composer_txlist.queue);
+		atomic_inc(&isp_ctx->composer_txlist.queue_cnt);
+		spin_unlock(&isp_ctx->composer_txlist.lock);
+
+		atomic_inc(&isp_ctx->cmd_queued);
+		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
+		break;
+	case SCP_ISP_FRAME:
+		memcpy(&isp_composer_work->frameparams, data,
+		       sizeof(isp_composer_work->frameparams));
+		dev_dbg(dev, "Enq ipi frame_num:%d\n",
+			isp_composer_work->frameparams.frame_seq_no);
+
+		spin_lock(&isp_ctx->composer_txlist.lock);
+		list_add_tail(&isp_composer_work->list_entry,
+			      &isp_ctx->composer_txlist.queue);
+		atomic_inc(&isp_ctx->composer_txlist.queue_cnt);
+		spin_unlock(&isp_ctx->composer_txlist.lock);
+
+		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
+		break;
+	default:
+		break;
+	}
+}
+
+void isp_composer_hw_init(struct device *dev)
+{
+	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+
+	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
+	composer_tx_cmd.cmd_id = ISP_CMD_INIT;
+	composer_tx_cmd.frameparam.hw_module = isp_ctx->isp_hw_module;
+	composer_tx_cmd.frameparam.cq_addr.iova = isp_ctx->scp_mem_iova;
+	composer_tx_cmd.frameparam.cq_addr.scp_addr = isp_ctx->scp_mem_pa;
+	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
+}
+
+void isp_composer_meta_config(struct device *dev,
+			      unsigned int dma)
+{
+	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
+
+	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
+	composer_tx_cmd.cmd_id = ISP_CMD_CONFIG_META;
+	composer_tx_cmd.cfg_meta_out_param.enabled_meta_dmas = dma;
+	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
+}
+
+void isp_composer_hw_config(struct device *dev,
+			    struct p1_config_param *config_param)
+{
+	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
+
+	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
+	composer_tx_cmd.cmd_id = ISP_CMD_CONFIG;
+	memcpy(&composer_tx_cmd.config_param, config_param,
+	       sizeof(*config_param));
+	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
+}
+
+void isp_composer_stream(struct device *dev, int on)
+{
+	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
+
+	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
+	composer_tx_cmd.cmd_id = ISP_CMD_STREAM;
+	composer_tx_cmd.is_stream_on = on;
+	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
+}
+
+void isp_composer_hw_deinit(struct device *dev)
+{
+	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
+	struct isp_p1_device *p1_dev = get_p1_device(dev);
+	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
+	int ret;
+
+	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
+	composer_tx_cmd.cmd_id = ISP_CMD_DEINIT;
+	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
+
+	/* Wait for ISP_CMD_DEINIT command is handled done */
+	ret = wait_event_timeout(isp_ctx->composer_deinit_thread.wq,
+				 atomic_read(&isp_ctx->scp_state) == SCP_OFF,
+				 msecs_to_jiffies(2000));
+	if (ret)
+		return;
+
+	dev_warn(dev, "Timeout & local de-init\n");
+	isp_composer_deinit(isp_ctx);
+}
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
new file mode 100644
index 000000000000..fbd8593e9c2d
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
@@ -0,0 +1,207 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ */
+
+#ifndef _MTK_ISP_SCP_H
+#define _MTK_ISP_SCP_H
+
+#include <linux/types.h>
+
+#include "mtk_cam-v4l2-util.h"
+
+/*
+ * struct img_size - image size information.
+ *
+ * @w: image width, the unit is pixel
+ * @h: image height, the unit is pixel
+ * @xsize: bytes per line based on width.
+ * @stride: bytes per line when changing line.
+ *          Normally, calculate new STRIDE based on
+ *          xsize + HW constrain(page or align).
+ *
+ */
+struct img_size {
+	__u32 w;
+	__u32 h;
+	__u32 xsize;
+	__u32 stride;
+} __packed;
+
+/*
+ * struct img_buffer - buffer address information.
+ *
+ * @iova: DMA address for external devices.
+ * @scp_addr: SCP address for external co-process unit.
+ *
+ */
+struct img_buffer {
+	__u32 iova;
+	__u32 scp_addr;
+} __packed;
+
+struct p1_img_crop {
+	__u32 left;
+	__u32 top;
+	__u32 width;
+	__u32 height;
+} __packed;
+
+struct p1_img_output {
+	struct img_buffer buffer;
+	struct img_size size;
+	struct p1_img_crop crop;
+	__u8 pixel_byte;
+	__u32 img_fmt;
+} __packed;
+
+/*
+ * struct cfg_in_param - image input parameters structure.
+ *                       Normally, it comes from sensor information.
+ *
+ * @continuous: indicate the sensor mode.
+ *              1: continuous
+ *              0: single
+ * @subsample: indicate to enables SOF subsample or not.
+ * @pixel_mode: describe 1/2/4 pixels per clock cycle.
+ * @data_pattern: describe input data pattern.
+ * @raw_pixel_id: bayer sequence.
+ * @tg_fps: the fps rate of TG (time generator).
+ * @img_fmt: the image format of input source.
+ * @p1_img_crop: the crop configuration of input source.
+ *
+ */
+struct cfg_in_param {
+	__u8 continuous;
+	__u8 subsample;
+	__u8 pixel_mode;
+	__u8 data_pattern;
+	__u8 raw_pixel_id;
+	__u16 tg_fps;
+	__u32 img_fmt;
+	struct p1_img_crop crop;
+} __packed;
+
+/*
+ * struct cfg_main_out_param - the image output parameters of main stream.
+ *
+ * @bypass: indicate this device is enabled or disabled or not .
+ * @pure_raw: indicate the image path control.
+ *            1: pure raw
+ *            0: processing raw
+ * @pure_raw_pack: indicate the image is packed or not.
+ *                 1: packed mode
+ *                 0: unpacked mode
+ * @p1_img_output: the output image information.
+ *
+ */
+struct cfg_main_out_param {
+	/* Bypass main out parameters */
+	__u8 bypass;
+	/* Control HW image raw path */
+	__u8 pure_raw;
+	/* Control HW image pack function */
+	__u8 pure_raw_pack;
+	struct p1_img_output output;
+} __packed;
+
+/*
+ * struct cfg_resize_out_param - the image output parameters of
+ *                               packed out stream.
+ *
+ * @bypass: indicate this device is enabled or disabled or not .
+ * @p1_img_output: the output image information.
+ *
+ */
+struct cfg_resize_out_param {
+	/* Bypass resize parameters */
+	__u8 bypass;
+	struct p1_img_output output;
+} __packed;
+
+/*
+ * struct cfg_meta_out_param - output meta information.
+ *
+ * @enabled_meta_dmas: indicate which meta DMAs are enabled.
+ *
+ */
+struct cfg_meta_out_param {
+	__u32 enabled_meta_dmas;
+} __packed;
+
+struct p1_config_param {
+	/* Sensor/TG info */
+	struct cfg_in_param cfg_in_param;
+	/* IMGO DMA */
+	struct cfg_main_out_param cfg_main_param;
+	/* RRZO DMA */
+	struct cfg_resize_out_param cfg_resize_param;
+	/* 3A DMAs and other. */
+	struct cfg_meta_out_param cfg_meta_param;
+} __packed;
+
+struct p1_frame_param {
+	/* frame sequence number */
+	__u32 frame_seq_no;
+	/* SOF index */
+	__u32 sof_idx;
+	/* The memory address of tuning buffer from user space */
+	struct img_buffer dma_buffers[MTK_CAM_P1_TOTAL_NODES];
+} __packed;
+
+struct P1_meta_frame {
+	__u32 enabled_dma;
+	__u32 vb_index;
+	struct img_buffer meta_addr;
+} __packed;
+
+struct isp_init_info {
+	__u8 hw_module;
+	struct img_buffer cq_addr;
+} __packed;
+
+struct isp_ack_info {
+	__u8 cmd_id;
+	__u32 frame_seq_no;
+} __packed;
+
+enum mtk_isp_scp_cmds {
+	ISP_CMD_INIT,
+	ISP_CMD_CONFIG,
+	ISP_CMD_STREAM,
+	ISP_CMD_DEINIT,
+	ISP_CMD_ACK,
+	ISP_CMD_FRAME_ACK,
+	ISP_CMD_CONFIG_META,
+	ISP_CMD_ENQUEUE_META,
+	ISP_CMD_RESERVED,
+};
+
+struct mtk_isp_scp_p1_cmd {
+	__u8 cmd_id;
+	union {
+		struct isp_init_info frameparam;
+		struct p1_config_param config_param;
+		struct cfg_meta_out_param cfg_meta_out_param;
+		struct P1_meta_frame meta_frame;
+		__u8 is_stream_on;
+		struct isp_ack_info ack_info;
+	};
+} __packed;
+
+enum mtk_isp_scp_type {
+	SCP_ISP_CMD = 0,
+	SCP_ISP_FRAME,
+};
+
+int isp_composer_init(struct device *dev);
+void isp_composer_enqueue(struct device *dev, void *data,
+			  enum mtk_isp_scp_type type);
+void isp_composer_hw_init(struct device *dev);
+void isp_composer_hw_config(struct device *dev,
+			    struct p1_config_param *config_param);
+void isp_composer_hw_deinit(struct device *dev);
+void isp_composer_meta_config(struct device *dev, unsigned int dma);
+void isp_composer_stream(struct device *dev, int on);
+
+#endif /* _MTK_ISP_SCP_H */
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
                   ` (7 preceding siblings ...)
  2019-06-11  3:53 ` [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication Jungo Lin
@ 2019-06-11  3:53 ` Jungo Lin
  2019-07-01  7:25   ` Tomasz Figa
  8 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-06-11  3:53 UTC (permalink / raw)
  To: tfiga, hverkuil, laurent.pinchart, matthias.bgg, mchehab
  Cc: linux-media, linux-mediatek, linux-arm-kernel, devicetree,
	srv_heupstream, ddavenport, robh, sean.cheng, sj.huang,
	frederic.chen, ryan.yu, rynn.wu, jungo.lin, frankie.chiu

The purpose of this child device is to provide shared
memory management for exchanging tuning data between co-processor
and the Pass 1 unit of the camera ISP system, including cache
buffer handling.

Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
---
This patch depends on "Add support for mt8183 SCP"[1].

[1] https://patchwork.kernel.org/cover/10972143/
---
 .../platform/mtk-isp/isp_50/cam/Makefile      |   1 +
 .../mtk-isp/isp_50/cam/mtk_cam-smem.c         | 304 ++++++++++++++++++
 .../mtk-isp/isp_50/cam/mtk_cam-smem.h         |  18 ++
 3 files changed, 323 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h

diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
index 95f0b1c8fa1c..d545ca6f09c5 100644
--- a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
@@ -4,5 +4,6 @@ mtk-cam-isp-objs += mtk_cam-ctrl.o
 mtk-cam-isp-objs += mtk_cam-v4l2-util.o
 mtk-cam-isp-objs += mtk_cam.o
 mtk-cam-isp-objs += mtk_cam-scp.o
+mtk-cam-isp-objs += mtk_cam-smem.o
 
 obj-$(CONFIG_VIDEO_MEDIATEK_ISP_PASS1) += mtk-cam-isp.o
\ No newline at end of file
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
new file mode 100644
index 000000000000..a9845668ce10
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
@@ -0,0 +1,304 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+
+#include <asm/cacheflush.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include <linux/iommu.h>
+#include <linux/of.h>
+#include <linux/of_fdt.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/platform_data/mtk_scp.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "mtk_cam-smem.h"
+
+static struct dma_map_ops smem_dma_ops;
+
+struct mtk_cam_smem_dev {
+	struct device *dev;
+	struct sg_table sgt;
+	struct page **smem_pages;
+	dma_addr_t smem_base;
+	dma_addr_t smem_dma_base;
+	int smem_size;
+};
+
+struct dma_coherent_mem {
+	void		*virt_base;
+	dma_addr_t	device_base;
+	unsigned long	pfn_base;
+	int		size;
+	int		flags;
+	unsigned long	*bitmap;
+	spinlock_t	spinlock; /* dma_coherent_mem attributes protection */
+	bool		use_dev_dma_pfn_offset;
+};
+
+dma_addr_t mtk_cam_smem_iova_to_scp_addr(struct device *dev,
+					 dma_addr_t iova)
+{
+	struct iommu_domain *domain;
+	dma_addr_t addr, limit;
+	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
+
+	domain = iommu_get_domain_for_dev(dev);
+	if (!domain) {
+		dev_warn(dev, "No iommu group domain\n");
+		return 0;
+	}
+
+	addr = iommu_iova_to_phys(domain, iova);
+	limit = smem_dev->smem_base + smem_dev->smem_size;
+	if (addr < smem_dev->smem_base || addr >= limit) {
+		dev_err(dev,
+			"Unexpected scp_addr:%pad must >= %pad and < %pad)\n",
+			&addr, &smem_dev->smem_base, &limit);
+		return 0;
+	}
+	return addr;
+}
+
+static int mtk_cam_smem_get_sgtable(struct device *dev,
+				    struct sg_table *sgt,
+				    void *cpu_addr, dma_addr_t dma_addr,
+				    size_t size, unsigned long attrs)
+{
+	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
+	size_t pages_count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	dma_addr_t scp_addr = mtk_cam_smem_iova_to_scp_addr(dev, dma_addr);
+	u32 pages_start = (scp_addr - smem_dev->smem_base) >> PAGE_SHIFT;
+
+	dev_dbg(dev,
+		"%s:page:%u va:%pK scp addr:%pad, aligned size:%zu pages:%zu\n",
+		__func__, pages_start, cpu_addr, &scp_addr, size, pages_count);
+
+	return sg_alloc_table_from_pages(sgt,
+		smem_dev->smem_pages + pages_start,
+		pages_count, 0, size, GFP_KERNEL);
+}
+
+static void *mtk_cam_smem_get_cpu_addr(struct mtk_cam_smem_dev *smem_dev,
+				       dma_addr_t addr)
+{
+	struct device *dev = smem_dev->dev;
+	struct dma_coherent_mem *dma_mem = dev->dma_mem;
+
+	if (addr < smem_dev->smem_base ||
+	    addr > smem_dev->smem_base + smem_dev->smem_size) {
+		dev_err(dev, "Invalid scp_addr %pad from sg\n", &addr);
+		return NULL;
+	}
+	return dma_mem->virt_base + (addr - smem_dev->smem_base);
+}
+
+static void mtk_cam_smem_sync_sg_for_cpu(struct device *dev,
+					 struct scatterlist *sgl, int nelems,
+					 enum dma_data_direction dir)
+{
+	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
+	dma_addr_t scp_addr = sg_phys(sgl);
+	void *cpu_addr = mtk_cam_smem_get_cpu_addr(smem_dev, scp_addr);
+
+	dev_dbg(dev,
+		"__dma_unmap_area:scp_addr:%pad,vaddr:%pK,size:%d,dir:%d\n",
+		&scp_addr, cpu_addr, sgl->length, dir);
+	__dma_unmap_area(cpu_addr, sgl->length, dir);
+}
+
+static void mtk_cam_smem_sync_sg_for_device(struct device *dev,
+					    struct scatterlist *sgl,
+					    int nelems,
+					    enum dma_data_direction dir)
+{
+	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
+	dma_addr_t scp_addr = sg_phys(sgl);
+	void *cpu_addr = mtk_cam_smem_get_cpu_addr(smem_dev, scp_addr);
+
+	dev_dbg(dev,
+		"__dma_map_area:scp_addr:%pad,vaddr:%pK,size:%d,dir:%d\n",
+		&scp_addr, cpu_addr, sgl->length, dir);
+	__dma_map_area(cpu_addr, sgl->length, dir);
+}
+
+static void mtk_cam_smem_setup_dma_ops(struct device *dev,
+				       struct dma_map_ops *smem_ops)
+{
+	memcpy((void *)smem_ops, dev->dma_ops, sizeof(*smem_ops));
+	smem_ops->get_sgtable = mtk_cam_smem_get_sgtable;
+	smem_ops->sync_sg_for_device = mtk_cam_smem_sync_sg_for_device;
+	smem_ops->sync_sg_for_cpu = mtk_cam_smem_sync_sg_for_cpu;
+	set_dma_ops(dev, smem_ops);
+}
+
+static int mtk_cam_reserved_drm_sg_init(struct mtk_cam_smem_dev *smem_dev)
+{
+	u32 size_align, n_pages;
+	struct device *dev = smem_dev->dev;
+	struct sg_table *sgt = &smem_dev->sgt;
+	struct page **pages;
+	dma_addr_t dma_addr;
+	unsigned int i;
+	int ret;
+
+	smem_dev->smem_base = scp_get_reserve_mem_phys(SCP_ISP_MEM2_ID);
+	smem_dev->smem_size = scp_get_reserve_mem_size(SCP_ISP_MEM2_ID);
+	if (!smem_dev->smem_base || !smem_dev->smem_size)
+		return -EPROBE_DEFER;
+
+	dev_info(dev, "%s dev:0x%pK base:%pad size:%u MiB\n",
+		 __func__,
+		 smem_dev->dev,
+		 &smem_dev->smem_base,
+		 (smem_dev->smem_size / SZ_1M));
+
+	size_align = PAGE_ALIGN(smem_dev->smem_size);
+	n_pages = size_align >> PAGE_SHIFT;
+
+	pages = kmalloc_array(n_pages, sizeof(struct page *), GFP_KERNEL);
+	if (!pages)
+		return -ENOMEM;
+
+	for (i = 0; i < n_pages; i++)
+		pages[i] = phys_to_page(smem_dev->smem_base + i * PAGE_SIZE);
+
+	ret = sg_alloc_table_from_pages(sgt, pages, n_pages, 0,
+					size_align, GFP_KERNEL);
+	if (ret) {
+		dev_err(dev, "failed to alloca sg table:%d\n", ret);
+		goto fail_table_alloc;
+	}
+	sgt->nents = dma_map_sg_attrs(dev, sgt->sgl, sgt->orig_nents,
+				      DMA_BIDIRECTIONAL,
+				      DMA_ATTR_SKIP_CPU_SYNC);
+	if (!sgt->nents) {
+		dev_err(dev, "failed to dma sg map\n");
+		goto fail_map;
+	}
+
+	dma_addr = sg_dma_address(sgt->sgl);
+	ret = dma_declare_coherent_memory(dev, smem_dev->smem_base,
+					  dma_addr, size_align,
+					  DMA_MEMORY_EXCLUSIVE);
+	if (ret) {
+		dev_err(dev, "Unable to declare smem  memory:%d\n", ret);
+		goto fail_map;
+	}
+
+	dev_info(dev, "Coherent mem pa:%pad/%pad, size:%d\n",
+		 &smem_dev->smem_base, &dma_addr, size_align);
+
+	smem_dev->smem_size = size_align;
+	smem_dev->smem_pages = pages;
+	smem_dev->smem_dma_base = dma_addr;
+
+	return 0;
+
+fail_map:
+	sg_free_table(sgt);
+fail_table_alloc:
+	while (n_pages--)
+		__free_page(pages[n_pages]);
+	kfree(pages);
+
+	return -ENOMEM;
+}
+
+/* DMA memory related helper functions */
+static void mtk_cam_memdev_release(struct device *dev)
+{
+	vb2_dma_contig_clear_max_seg_size(dev);
+}
+
+static struct device *mtk_cam_alloc_smem_dev(struct device *dev,
+					     const char *name)
+{
+	struct device *child;
+	int ret;
+
+	child = devm_kzalloc(dev, sizeof(*child), GFP_KERNEL);
+	if (!child)
+		return NULL;
+
+	child->parent = dev;
+	child->iommu_group = dev->iommu_group;
+	child->release = mtk_cam_memdev_release;
+	dev_set_name(child, name);
+	set_dma_ops(child, get_dma_ops(dev));
+	child->dma_mask = dev->dma_mask;
+	ret = dma_set_coherent_mask(child, DMA_BIT_MASK(32));
+	if (ret)
+		return NULL;
+
+	vb2_dma_contig_set_max_seg_size(child, DMA_BIT_MASK(32));
+
+	if (device_register(child)) {
+		device_del(child);
+		return NULL;
+	}
+
+	return child;
+}
+
+static int mtk_cam_composer_dma_init(struct mtk_isp_p1_ctx *isp_ctx)
+{
+	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
+	struct device *dev = &p1_dev->pdev->dev;
+	u32 size;
+	dma_addr_t addr;
+
+	isp_ctx->scp_mem_pa = scp_get_reserve_mem_phys(SCP_ISP_MEM_ID);
+	size = PAGE_ALIGN(scp_get_reserve_mem_size(SCP_ISP_MEM_ID));
+	if (!isp_ctx->scp_mem_pa || !size)
+		return -EPROBE_DEFER;
+
+	dev_info(dev, "scp addr:%pad size:0x%x\n", &isp_ctx->scp_mem_pa, size);
+
+	/* get iova address */
+	addr = dma_map_page_attrs(dev, phys_to_page(isp_ctx->scp_mem_pa), 0,
+				  size, DMA_BIDIRECTIONAL,
+				  DMA_ATTR_SKIP_CPU_SYNC);
+	if (dma_mapping_error(dev, addr)) {
+		isp_ctx->scp_mem_pa = 0;
+		dev_err(dev, "Failed to map scp iova\n");
+		return -ENOMEM;
+	}
+
+	isp_ctx->scp_mem_iova = addr;
+
+	return 0;
+}
+
+int mtk_cam_reserved_memory_init(struct isp_p1_device *p1_dev)
+{
+	struct device *dev = &p1_dev->pdev->dev;
+	struct mtk_cam_smem_dev *smem_dev;
+	int ret;
+
+	ret = mtk_cam_composer_dma_init(&p1_dev->isp_ctx);
+	if (ret)
+		return ret;
+
+	/* Allocate context */
+	smem_dev = devm_kzalloc(dev, sizeof(*smem_dev), GFP_KERNEL);
+	if (!smem_dev)
+		return -ENOMEM;
+
+	smem_dev->dev = mtk_cam_alloc_smem_dev(dev, "cam-smem");
+	if (!smem_dev->dev) {
+		dev_err(dev, "failed to alloc smem device\n");
+		return -ENODEV;
+	}
+	dev_set_drvdata(smem_dev->dev, smem_dev);
+	p1_dev->cam_dev.smem_dev = smem_dev->dev;
+
+	ret = mtk_cam_reserved_drm_sg_init(smem_dev);
+	if (ret)
+		return ret;
+
+	mtk_cam_smem_setup_dma_ops(smem_dev->dev, &smem_dma_ops);
+
+	return 0;
+}
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h
new file mode 100644
index 000000000000..981d47178e99
--- /dev/null
+++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ */
+
+#ifndef __MTK_CAM_ISP_SMEM_H
+#define __MTK_CAM_ISP_SMEM_H
+
+#include <linux/dma-mapping.h>
+
+#include "mtk_cam.h"
+
+int mtk_cam_reserved_memory_init(struct isp_p1_device *p1_dev);
+dma_addr_t mtk_cam_smem_iova_to_scp_addr(struct device *smem_dev,
+					 dma_addr_t iova);
+
+#endif
+
-- 
2.18.0


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 5/9] media: platform: Add Mediatek ISP P1 V4L2 control
  2019-06-11  3:53 ` [RFC,v3 5/9] media: platform: Add Mediatek ISP P1 V4L2 control Jungo Lin
@ 2019-07-01  5:50   ` Tomasz Figa
  2019-07-02 11:34     ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-01  5:50 UTC (permalink / raw)
  To: Jungo Lin
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi Jungo,

On Tue, Jun 11, 2019 at 11:53:40AM +0800, Jungo Lin wrote:
> Reserved Mediatek ISP P1 V4L2 control number with 16.
> Moreover, add two V4L2 controls for ISP P1 user space
> usage.
> 
> 1. V4L2_CID_MTK_GET_BIN_INFO
> - Provide the image output width & height in case
> camera binning mode is enabled.

Could you explain with a bit more details what these binned width and height
would mean? How would they relate to the video CAPTURE node width and height?
Isn't this something that should be rather exposed as an appropriate
selection rectangle, instead of custom controls?

> 
> 2. V4L2_CID_MTK_RAW_PATH
> - Export the path control of the main stream to user space.
> One is pure raw and the other is processing raw.
> The default value is 0 which outputs the pure raw bayer image
> from sesnor, without image processing in ISP HW.

Is it just effectively a full processing bypass? The driver seems to only
update the related configuration when the streaming starts. Can it be
controlled per-frame?

Generally this sounds more like something that should be modelled using the
media topology, similar to the example below.

/----------------\   /-------------------\   /--------------\
|                |---|                   |   |              |
| Capture Subdev |   | Processing Subdev |-o-| CAPTURE node |
|                |-\ |                   | | |              |
\----------------/ | \-------------------/ | \--------------/
                   |                       |
                   \-----------------------/

Then the userspace can select whether it wants the image from the capture
interface directly or procesed by the ISP by configuring the media links
appropriately.

The current limitation of this model is that it can't be easily configured
per-frame, as media configurations are not included in the requests yet.

[snip]

> +static int handle_ctrl_get_bin_info(struct v4l2_ctrl *ctrl, int is_width)
> +{
> +	struct mtk_cam_dev *cam_dev = ctrl->priv;
> +	struct v4l2_format *fmt;
> +
> +	fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_MAIN_STREAM_OUT].vdev_fmt;
> +
> +	dev_dbg(&cam_dev->pdev->dev, "Get bin info w*h:%d*%d is_width:%d",
> +		fmt->fmt.pix_mp.width, fmt->fmt.pix_mp.height, is_width);
> +
> +	if (is_width)
> +		ctrl->val = fmt->fmt.pix_mp.width;
> +	else
> +		ctrl->val = fmt->fmt.pix_mp.height;

This seems to contradict to what the comment in the header says, because it just
always returns the video node format and doesn't seem to care about whether
binning is enabled or not.

> +
> +	return 0;
> +}
> +
> +static int handle_ctrl_get_process_raw(struct v4l2_ctrl *ctrl)
> +{
> +	struct mtk_cam_dev *cam_dev = ctrl->priv;
> +	struct isp_p1_device *p1_dev = get_p1_device(&cam_dev->pdev->dev);
> +
> +	ctrl->val = (p1_dev->isp_ctx.isp_raw_path == ISP_PROCESS_RAW_PATH);
> +
> +	dev_dbg(&cam_dev->pdev->dev, "Get process raw:%d", ctrl->val);
> +
> +	return 0;
> +}
> +
> +static int handle_ctrl_set_process_raw(struct v4l2_ctrl *ctrl)
> +{
> +	struct mtk_cam_dev *cam_dev = ctrl->priv;
> +	struct isp_p1_device *p1_dev = get_p1_device(&cam_dev->pdev->dev);
> +
> +	p1_dev->isp_ctx.isp_raw_path = (ctrl->val) ?
> +		ISP_PROCESS_RAW_PATH : ISP_PURE_RAW_PATH;
> +	dev_dbg(&cam_dev->pdev->dev, "Set process raw:%d", ctrl->val);
> +	return 0;
> +}
> +
> +static int mtk_cam_dev_g_ctrl(struct v4l2_ctrl *ctrl)

This is g_volatile_ctrl not, g_ctrl.

> +{
> +	switch (ctrl->id) {
> +	case V4L2_CID_MTK_PROCESSING_RAW:
> +		handle_ctrl_get_process_raw(ctrl);
> +		break;

No need to provide getters for non-volatile controls. The
framework manages them.

> +	case V4L2_CID_MTK_GET_BIN_WIDTH:
> +		handle_ctrl_get_bin_info(ctrl, 1);
> +		break;
> +	case V4L2_CID_MTK_GET_BIN_HEIGTH:
> +		handle_ctrl_get_bin_info(ctrl, 0);

It's trivial to get the value, so there isn't much benefit in having a
function to do so, especially if one needs something like a is_width
argument that further complicates the code.

> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +	return 0;
> +}
> +
> +static int mtk_cam_dev_s_ctrl(struct v4l2_ctrl *ctrl)
> +{
> +	switch (ctrl->id) {
> +	case V4L2_CID_MTK_PROCESSING_RAW:
> +		return handle_ctrl_set_process_raw(ctrl);

Same as above. The operation is too trivial to deserve a function.

> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
> +static const struct v4l2_ctrl_ops mtk_cam_dev_ctrl_ops = {
> +	.g_volatile_ctrl = mtk_cam_dev_g_ctrl,
> +	.s_ctrl = mtk_cam_dev_s_ctrl,
> +};
> +
> +struct v4l2_ctrl_config mtk_cam_controls[] = {
> +	{
> +	.ops = &mtk_cam_dev_ctrl_ops,
> +	.id = V4L2_CID_MTK_PROCESSING_RAW,
> +	.name = "MTK CAM PROCESSING RAW",
> +	.type = V4L2_CTRL_TYPE_BOOLEAN,
> +	.min = 0,
> +	.max = 1,
> +	.step = 1,
> +	.def = 1,
> +	},
> +	{
> +	.ops = &mtk_cam_dev_ctrl_ops,
> +	.id = V4L2_CID_MTK_GET_BIN_WIDTH,
> +	.name = "MTK CAM GET BIN WIDTH",
> +	.type = V4L2_CTRL_TYPE_INTEGER,
> +	.min = IMG_MIN_WIDTH,
> +	.max = IMG_MAX_WIDTH,
> +	.step = 1,
> +	.def = IMG_MAX_WIDTH,
> +	.flags = V4L2_CTRL_FLAG_READ_ONLY | V4L2_CTRL_FLAG_VOLATILE,
> +	},
> +	{
> +	.ops = &mtk_cam_dev_ctrl_ops,
> +	.id = V4L2_CID_MTK_GET_BIN_HEIGTH,
> +	.name = "MTK CAM GET BIN HEIGHT",
> +	.type = V4L2_CTRL_TYPE_INTEGER,
> +	.min = IMG_MIN_HEIGHT,
> +	.max = IMG_MAX_HEIGHT,
> +	.step = 1,
> +	.def = IMG_MAX_HEIGHT,
> +	.flags = V4L2_CTRL_FLAG_READ_ONLY | V4L2_CTRL_FLAG_VOLATILE,
> +	},
> +};
> +
> +int mtk_cam_ctrl_init(struct mtk_cam_dev *cam_dev,
> +		      struct v4l2_ctrl_handler *hdl)
> +{
> +	unsigned int i;
> +
> +	/* Initialized HW controls, allow V4L2_CID_MTK_CAM_MAX ctrls */
> +	v4l2_ctrl_handler_init(hdl, V4L2_CID_MTK_CAM_MAX);
> +	if (hdl->error) {

This should be checked at the end, after all the controls are added.

Best regards,
Tomasz


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-06-11  3:53 ` [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device Jungo Lin
@ 2019-07-01  7:25   ` Tomasz Figa
  2019-07-05  3:33     ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-01  7:25 UTC (permalink / raw)
  To: Jungo Lin
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi Jungo,

On Tue, Jun 11, 2019 at 11:53:44AM +0800, Jungo Lin wrote:
> The purpose of this child device is to provide shared
> memory management for exchanging tuning data between co-processor
> and the Pass 1 unit of the camera ISP system, including cache
> buffer handling.
> 

Looks like we haven't really progressed on getting this replaced with
something that doesn't require so much custom code. Let me propose something
better then.

We already have a reserved memory mode in DT. If it has a compatible string
of "shared-dma-pool", it would be registered in the coherent DMA framework
[1]. That would make it available for consumer devices to look-up.

Now if we add a "memory-region" property to the SCP device node and point it
to our reserved memory node, the SCP driver could look it up and hook to the
DMA mapping API using of_reserved_mem_device_init_by_idx[2].

That basically makes any dma_alloc_*(), dma_map_*(), etc. calls on the SCP
struct device use the coherent DMA ops, which operate on the assigned memory
pool. With that, the P1 driver could just directly use those calls to
manage the memory, without any custom code.

There is an example how this setup works in the s5p-mfc driver[3], but it
needs to be noted that it creates child nodes, because it can have more than
1 DMA port, which may need its own memory pool. In our case, we wouldn't
need child nodes and could just use the SCP device directly.

[1] https://elixir.bootlin.com/linux/v5.2-rc7/source/kernel/dma/coherent.c#L345
[2] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/of/of_reserved_mem.c#L312
[3] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/media/platform/s5p-mfc/s5p_mfc.c#L1075

Let me also post some specific comments below, in case we end up still
needing any of the code.

> Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> ---
> This patch depends on "Add support for mt8183 SCP"[1].
> 
> [1] https://patchwork.kernel.org/cover/10972143/
> ---
>  .../platform/mtk-isp/isp_50/cam/Makefile      |   1 +
>  .../mtk-isp/isp_50/cam/mtk_cam-smem.c         | 304 ++++++++++++++++++
>  .../mtk-isp/isp_50/cam/mtk_cam-smem.h         |  18 ++
>  3 files changed, 323 insertions(+)
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h
> 
> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
> index 95f0b1c8fa1c..d545ca6f09c5 100644
> --- a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
> @@ -4,5 +4,6 @@ mtk-cam-isp-objs += mtk_cam-ctrl.o
>  mtk-cam-isp-objs += mtk_cam-v4l2-util.o
>  mtk-cam-isp-objs += mtk_cam.o
>  mtk-cam-isp-objs += mtk_cam-scp.o
> +mtk-cam-isp-objs += mtk_cam-smem.o
>  
>  obj-$(CONFIG_VIDEO_MEDIATEK_ISP_PASS1) += mtk-cam-isp.o
> \ No newline at end of file
> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
> new file mode 100644
> index 000000000000..a9845668ce10
> --- /dev/null
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
> @@ -0,0 +1,304 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (c) 2018 MediaTek Inc.
> +
> +#include <asm/cacheflush.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/of.h>
> +#include <linux/of_fdt.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/platform_data/mtk_scp.h>
> +#include <media/videobuf2-dma-contig.h>
> +
> +#include "mtk_cam-smem.h"
> +
> +static struct dma_map_ops smem_dma_ops;
> +
> +struct mtk_cam_smem_dev {
> +	struct device *dev;
> +	struct sg_table sgt;
> +	struct page **smem_pages;
> +	dma_addr_t smem_base;
> +	dma_addr_t smem_dma_base;
> +	int smem_size;
> +};
> +
> +struct dma_coherent_mem {
> +	void		*virt_base;
> +	dma_addr_t	device_base;
> +	unsigned long	pfn_base;
> +	int		size;
> +	int		flags;
> +	unsigned long	*bitmap;
> +	spinlock_t	spinlock; /* dma_coherent_mem attributes protection */
> +	bool		use_dev_dma_pfn_offset;
> +};
> +
> +dma_addr_t mtk_cam_smem_iova_to_scp_addr(struct device *dev,
> +					 dma_addr_t iova)
> +{
> +	struct iommu_domain *domain;
> +	dma_addr_t addr, limit;
> +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> +
> +	domain = iommu_get_domain_for_dev(dev);
> +	if (!domain) {
> +		dev_warn(dev, "No iommu group domain\n");
> +		return 0;
> +	}
> +
> +	addr = iommu_iova_to_phys(domain, iova);
> +	limit = smem_dev->smem_base + smem_dev->smem_size;
> +	if (addr < smem_dev->smem_base || addr >= limit) {
> +		dev_err(dev,
> +			"Unexpected scp_addr:%pad must >= %pad and < %pad)\n",
> +			&addr, &smem_dev->smem_base, &limit);
> +		return 0;
> +	}
> +	return addr;
> +}

This isn't correct. One could pass an IOVA that wasn't allocated for the SCP
and then the address wouldn't be valid, because it would point outside of
the address range allowed for SCP to access and also it would only point to
the first page backing the IOVA.

The correct approach would be to always carry SCP DMA address and IOVA
together in some kind of struct describing such buffers.

> +
> +static int mtk_cam_smem_get_sgtable(struct device *dev,
> +				    struct sg_table *sgt,
> +				    void *cpu_addr, dma_addr_t dma_addr,
> +				    size_t size, unsigned long attrs)
> +{
> +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> +	size_t pages_count = PAGE_ALIGN(size) >> PAGE_SHIFT;
> +	dma_addr_t scp_addr = mtk_cam_smem_iova_to_scp_addr(dev, dma_addr);
> +	u32 pages_start = (scp_addr - smem_dev->smem_base) >> PAGE_SHIFT;
> +
> +	dev_dbg(dev,
> +		"%s:page:%u va:%pK scp addr:%pad, aligned size:%zu pages:%zu\n",
> +		__func__, pages_start, cpu_addr, &scp_addr, size, pages_count);
> +
> +	return sg_alloc_table_from_pages(sgt,
> +		smem_dev->smem_pages + pages_start,
> +		pages_count, 0, size, GFP_KERNEL);
> +}

This should be just dma_get_sgtable_attrs(), in the approach I suggested at
the top.

> +
> +static void *mtk_cam_smem_get_cpu_addr(struct mtk_cam_smem_dev *smem_dev,
> +				       dma_addr_t addr)
> +{
> +	struct device *dev = smem_dev->dev;
> +	struct dma_coherent_mem *dma_mem = dev->dma_mem;
> +
> +	if (addr < smem_dev->smem_base ||
> +	    addr > smem_dev->smem_base + smem_dev->smem_size) {

This is off by one, should be >=.

Also, this wouldn't really guarantee the CPU access the caller is going to
do is valid, because it doesn't consider the access operation size.

Generally I'd suggest designing the code so that it doesn't have to convert
offset addresses between different address spaces.

> +		dev_err(dev, "Invalid scp_addr %pad from sg\n", &addr);
> +		return NULL;
> +	}
> +	return dma_mem->virt_base + (addr - smem_dev->smem_base);
> +}
> +
> +static void mtk_cam_smem_sync_sg_for_cpu(struct device *dev,
> +					 struct scatterlist *sgl, int nelems,
> +					 enum dma_data_direction dir)
> +{
> +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> +	dma_addr_t scp_addr = sg_phys(sgl);
> +	void *cpu_addr = mtk_cam_smem_get_cpu_addr(smem_dev, scp_addr);
> +
> +	dev_dbg(dev,
> +		"__dma_unmap_area:scp_addr:%pad,vaddr:%pK,size:%d,dir:%d\n",
> +		&scp_addr, cpu_addr, sgl->length, dir);
> +	__dma_unmap_area(cpu_addr, sgl->length, dir);

It's not allowed to use this function anywhere outside of the DMA API
internals. See the comment [4].

[4] https://elixir.bootlin.com/linux/v5.2-rc7/source/arch/arm64/include/asm/cacheflush.h#L112

> +}
> +
> +static void mtk_cam_smem_sync_sg_for_device(struct device *dev,
> +					    struct scatterlist *sgl,
> +					    int nelems,
> +					    enum dma_data_direction dir)
> +{
> +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> +	dma_addr_t scp_addr = sg_phys(sgl);
> +	void *cpu_addr = mtk_cam_smem_get_cpu_addr(smem_dev, scp_addr);
> +
> +	dev_dbg(dev,
> +		"__dma_map_area:scp_addr:%pad,vaddr:%pK,size:%d,dir:%d\n",
> +		&scp_addr, cpu_addr, sgl->length, dir);
> +	__dma_map_area(cpu_addr, sgl->length, dir);

Ditto.

> +}
> +
> +static void mtk_cam_smem_setup_dma_ops(struct device *dev,
> +				       struct dma_map_ops *smem_ops)
> +{
> +	memcpy((void *)smem_ops, dev->dma_ops, sizeof(*smem_ops));
> +	smem_ops->get_sgtable = mtk_cam_smem_get_sgtable;
> +	smem_ops->sync_sg_for_device = mtk_cam_smem_sync_sg_for_device;
> +	smem_ops->sync_sg_for_cpu = mtk_cam_smem_sync_sg_for_cpu;
> +	set_dma_ops(dev, smem_ops);
> +}
> +
> +static int mtk_cam_reserved_drm_sg_init(struct mtk_cam_smem_dev *smem_dev)
> +{
> +	u32 size_align, n_pages;
> +	struct device *dev = smem_dev->dev;
> +	struct sg_table *sgt = &smem_dev->sgt;
> +	struct page **pages;
> +	dma_addr_t dma_addr;
> +	unsigned int i;
> +	int ret;
> +
> +	smem_dev->smem_base = scp_get_reserve_mem_phys(SCP_ISP_MEM2_ID);
> +	smem_dev->smem_size = scp_get_reserve_mem_size(SCP_ISP_MEM2_ID);
> +	if (!smem_dev->smem_base || !smem_dev->smem_size)
> +		return -EPROBE_DEFER;
> +
> +	dev_info(dev, "%s dev:0x%pK base:%pad size:%u MiB\n",
> +		 __func__,
> +		 smem_dev->dev,
> +		 &smem_dev->smem_base,
> +		 (smem_dev->smem_size / SZ_1M));
> +
> +	size_align = PAGE_ALIGN(smem_dev->smem_size);
> +	n_pages = size_align >> PAGE_SHIFT;
> +
> +	pages = kmalloc_array(n_pages, sizeof(struct page *), GFP_KERNEL);
> +	if (!pages)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < n_pages; i++)
> +		pages[i] = phys_to_page(smem_dev->smem_base + i * PAGE_SIZE);
> +
> +	ret = sg_alloc_table_from_pages(sgt, pages, n_pages, 0,
> +					size_align, GFP_KERNEL);
> +	if (ret) {
> +		dev_err(dev, "failed to alloca sg table:%d\n", ret);
> +		goto fail_table_alloc;
> +	}
> +	sgt->nents = dma_map_sg_attrs(dev, sgt->sgl, sgt->orig_nents,
> +				      DMA_BIDIRECTIONAL,
> +				      DMA_ATTR_SKIP_CPU_SYNC);
> +	if (!sgt->nents) {
> +		dev_err(dev, "failed to dma sg map\n");
> +		goto fail_map;
> +	}
> +
> +	dma_addr = sg_dma_address(sgt->sgl);
> +	ret = dma_declare_coherent_memory(dev, smem_dev->smem_base,
> +					  dma_addr, size_align,
> +					  DMA_MEMORY_EXCLUSIVE);
> +	if (ret) {
> +		dev_err(dev, "Unable to declare smem  memory:%d\n", ret);
> +		goto fail_map;
> +	}
> +
> +	dev_info(dev, "Coherent mem pa:%pad/%pad, size:%d\n",
> +		 &smem_dev->smem_base, &dma_addr, size_align);
> +
> +	smem_dev->smem_size = size_align;
> +	smem_dev->smem_pages = pages;
> +	smem_dev->smem_dma_base = dma_addr;
> +
> +	return 0;
> +
> +fail_map:
> +	sg_free_table(sgt);
> +fail_table_alloc:
> +	while (n_pages--)
> +		__free_page(pages[n_pages]);
> +	kfree(pages);
> +
> +	return -ENOMEM;
> +}
> +
> +/* DMA memory related helper functions */
> +static void mtk_cam_memdev_release(struct device *dev)
> +{
> +	vb2_dma_contig_clear_max_seg_size(dev);
> +}
> +
> +static struct device *mtk_cam_alloc_smem_dev(struct device *dev,
> +					     const char *name)
> +{
> +	struct device *child;
> +	int ret;
> +
> +	child = devm_kzalloc(dev, sizeof(*child), GFP_KERNEL);
> +	if (!child)
> +		return NULL;
> +
> +	child->parent = dev;
> +	child->iommu_group = dev->iommu_group;

This isn't something that can be set explicitly. It's an internal field of
the IOMMU subsystem.

> +	child->release = mtk_cam_memdev_release;
> +	dev_set_name(child, name);
> +	set_dma_ops(child, get_dma_ops(dev));
> +	child->dma_mask = dev->dma_mask;
> +	ret = dma_set_coherent_mask(child, DMA_BIT_MASK(32));
> +	if (ret)
> +		return NULL;
> +
> +	vb2_dma_contig_set_max_seg_size(child, DMA_BIT_MASK(32));
> +
> +	if (device_register(child)) {
> +		device_del(child);
> +		return NULL;
> +	}
> +
> +	return child;
> +}

We shouldn't need child devices, just one SCP device, as I mentioned above.

> +
> +static int mtk_cam_composer_dma_init(struct mtk_isp_p1_ctx *isp_ctx)
> +{
> +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> +	struct device *dev = &p1_dev->pdev->dev;
> +	u32 size;
> +	dma_addr_t addr;
> +
> +	isp_ctx->scp_mem_pa = scp_get_reserve_mem_phys(SCP_ISP_MEM_ID);
> +	size = PAGE_ALIGN(scp_get_reserve_mem_size(SCP_ISP_MEM_ID));
> +	if (!isp_ctx->scp_mem_pa || !size)
> +		return -EPROBE_DEFER;
> +
> +	dev_info(dev, "scp addr:%pad size:0x%x\n", &isp_ctx->scp_mem_pa, size);

This isn't something that deserves the "info" log level. Should be "dbg"
or removed.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 5/9] media: platform: Add Mediatek ISP P1 V4L2 control
  2019-07-01  5:50   ` Tomasz Figa
@ 2019-07-02 11:34     ` Jungo Lin
  0 siblings, 0 replies; 45+ messages in thread
From: Jungo Lin @ 2019-07-02 11:34 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi Tomasz,

On Mon, 2019-07-01 at 14:50 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Tue, Jun 11, 2019 at 11:53:40AM +0800, Jungo Lin wrote:
> > Reserved Mediatek ISP P1 V4L2 control number with 16.
> > Moreover, add two V4L2 controls for ISP P1 user space
> > usage.
> > 
> > 1. V4L2_CID_MTK_GET_BIN_INFO
> > - Provide the image output width & height in case
> > camera binning mode is enabled.
> 
> Could you explain with a bit more details what these binned width and height
> would mean? How would they relate to the video CAPTURE node width and height?
> Isn't this something that should be rather exposed as an appropriate
> selection rectangle, instead of custom controls?
> 

Frontal binning is Mediatek ISP P1 HW function and it is designed for
low power saving. The input sensor resolution will be divided by 2
including width & height in the source side of ISP HW and lower the
overall ISP throughput. So the ISP clock could be running in the low
speed for power saving. However, the image quality will be bad in this
mode. If the frontal binning is enabled by ISP driver, user space 3A
algorithm needs to know the new resolution of TG. Btw, current ISP
driver doesn't support this function. We just export this V4L2 contorl
for future support. But, after internal discussing, we will remove this
v4l2 control in next patch.

> > 
> > 2. V4L2_CID_MTK_RAW_PATH
> > - Export the path control of the main stream to user space.
> > One is pure raw and the other is processing raw.
> > The default value is 0 which outputs the pure raw bayer image
> > from sesnor, without image processing in ISP HW.
> 
> Is it just effectively a full processing bypass? The driver seems to only
> update the related configuration when the streaming starts. Can it be
> controlled per-frame?
> 
> Generally this sounds more like something that should be modelled using the
> media topology, similar to the example below.
> 
> /----------------\   /-------------------\   /--------------\
> |                |---|                   |   |              |
> | Capture Subdev |   | Processing Subdev |-o-| CAPTURE node |
> |                |-\ |                   | | |              |
> \----------------/ | \-------------------/ | \--------------/
>                    |                       |
>                    \-----------------------/
> 
> Then the userspace can select whether it wants the image from the capture
> interface directly or procesed by the ISP by configuring the media links
> appropriately.
> 
> The current limitation of this model is that it can't be easily configured
> per-frame, as media configurations are not included in the requests yet.
> 
> [snip]
> 

For this V4L2 control, it is designed for per stream, not per frame and
just implemented for future usage. As the same conclusion with the above
one, we will remove this, either in current version. If we have strong
requirement on this, we could adopt your suggestion to use media
topology to per stream control.


> > +static int handle_ctrl_get_bin_info(struct v4l2_ctrl *ctrl, int is_width)
> > +{
> > +	struct mtk_cam_dev *cam_dev = ctrl->priv;
> > +	struct v4l2_format *fmt;
> > +
> > +	fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_MAIN_STREAM_OUT].vdev_fmt;
> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "Get bin info w*h:%d*%d is_width:%d",
> > +		fmt->fmt.pix_mp.width, fmt->fmt.pix_mp.height, is_width);
> > +
> > +	if (is_width)
> > +		ctrl->val = fmt->fmt.pix_mp.width;
> > +	else
> > +		ctrl->val = fmt->fmt.pix_mp.height;
> 
> This seems to contradict to what the comment in the header says, because it just
> always returns the video node format and doesn't seem to care about whether
> binning is enabled or not.
> 

This is because binning mode is not supported in current driver's
implementation and no request on this function. We just create this for
future usage. In order to avoid confusion, we will remove this.

> > +
> > +	return 0;
> > +}
> > +
> > +static int handle_ctrl_get_process_raw(struct v4l2_ctrl *ctrl)
> > +{
> > +	struct mtk_cam_dev *cam_dev = ctrl->priv;
> > +	struct isp_p1_device *p1_dev = get_p1_device(&cam_dev->pdev->dev);
> > +
> > +	ctrl->val = (p1_dev->isp_ctx.isp_raw_path == ISP_PROCESS_RAW_PATH);
> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "Get process raw:%d", ctrl->val);
> > +
> > +	return 0;
> > +}
> > +
> > +static int handle_ctrl_set_process_raw(struct v4l2_ctrl *ctrl)
> > +{
> > +	struct mtk_cam_dev *cam_dev = ctrl->priv;
> > +	struct isp_p1_device *p1_dev = get_p1_device(&cam_dev->pdev->dev);
> > +
> > +	p1_dev->isp_ctx.isp_raw_path = (ctrl->val) ?
> > +		ISP_PROCESS_RAW_PATH : ISP_PURE_RAW_PATH;
> > +	dev_dbg(&cam_dev->pdev->dev, "Set process raw:%d", ctrl->val);
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_dev_g_ctrl(struct v4l2_ctrl *ctrl)
> 
> This is g_volatile_ctrl not, g_ctrl.
> 

Ok, thanks for your correction.

> > +{
> > +	switch (ctrl->id) {
> > +	case V4L2_CID_MTK_PROCESSING_RAW:
> > +		handle_ctrl_get_process_raw(ctrl);
> > +		break;
> 
> No need to provide getters for non-volatile controls. The
> framework manages them.
> 

Ok, thanks for your correction.

> > +	case V4L2_CID_MTK_GET_BIN_WIDTH:
> > +		handle_ctrl_get_bin_info(ctrl, 1);
> > +		break;
> > +	case V4L2_CID_MTK_GET_BIN_HEIGTH:
> > +		handle_ctrl_get_bin_info(ctrl, 0);
> 
> It's trivial to get the value, so there isn't much benefit in having a
> function to do so, especially if one needs something like a is_width
> argument that further complicates the code.
> 

Ok, we will pay attention in this of implementation next time.

> > +		break;
> > +	default:
> > +		return -EINVAL;
> > +	}
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_dev_s_ctrl(struct v4l2_ctrl *ctrl)
> > +{
> > +	switch (ctrl->id) {
> > +	case V4L2_CID_MTK_PROCESSING_RAW:
> > +		return handle_ctrl_set_process_raw(ctrl);
> 
> Same as above. The operation is too trivial to deserve a function.
> 

Ok, got it.

> > +	default:
> > +		return -EINVAL;
> > +	}
> > +}
> > +
> > +static const struct v4l2_ctrl_ops mtk_cam_dev_ctrl_ops = {
> > +	.g_volatile_ctrl = mtk_cam_dev_g_ctrl,
> > +	.s_ctrl = mtk_cam_dev_s_ctrl,
> > +};
> > +
> > +struct v4l2_ctrl_config mtk_cam_controls[] = {
> > +	{
> > +	.ops = &mtk_cam_dev_ctrl_ops,
> > +	.id = V4L2_CID_MTK_PROCESSING_RAW,
> > +	.name = "MTK CAM PROCESSING RAW",
> > +	.type = V4L2_CTRL_TYPE_BOOLEAN,
> > +	.min = 0,
> > +	.max = 1,
> > +	.step = 1,
> > +	.def = 1,
> > +	},
> > +	{
> > +	.ops = &mtk_cam_dev_ctrl_ops,
> > +	.id = V4L2_CID_MTK_GET_BIN_WIDTH,
> > +	.name = "MTK CAM GET BIN WIDTH",
> > +	.type = V4L2_CTRL_TYPE_INTEGER,
> > +	.min = IMG_MIN_WIDTH,
> > +	.max = IMG_MAX_WIDTH,
> > +	.step = 1,
> > +	.def = IMG_MAX_WIDTH,
> > +	.flags = V4L2_CTRL_FLAG_READ_ONLY | V4L2_CTRL_FLAG_VOLATILE,
> > +	},
> > +	{
> > +	.ops = &mtk_cam_dev_ctrl_ops,
> > +	.id = V4L2_CID_MTK_GET_BIN_HEIGTH,
> > +	.name = "MTK CAM GET BIN HEIGHT",
> > +	.type = V4L2_CTRL_TYPE_INTEGER,
> > +	.min = IMG_MIN_HEIGHT,
> > +	.max = IMG_MAX_HEIGHT,
> > +	.step = 1,
> > +	.def = IMG_MAX_HEIGHT,
> > +	.flags = V4L2_CTRL_FLAG_READ_ONLY | V4L2_CTRL_FLAG_VOLATILE,
> > +	},
> > +};
> > +
> > +int mtk_cam_ctrl_init(struct mtk_cam_dev *cam_dev,
> > +		      struct v4l2_ctrl_handler *hdl)
> > +{
> > +	unsigned int i;
> > +
> > +	/* Initialized HW controls, allow V4L2_CID_MTK_CAM_MAX ctrls */
> > +	v4l2_ctrl_handler_init(hdl, V4L2_CID_MTK_CAM_MAX);
> > +	if (hdl->error) {
> 
> This should be checked at the end, after all the controls are added.
> 

Ok, got it.

> Best regards,
> Tomasz
> 

Based on above comments, we will remove mtk_cam_ctrl.h & mtk_cam_ctrl.c
in next patch to avoid over design.

Thanks for your comments.

Jungo



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-01  7:25   ` Tomasz Figa
@ 2019-07-05  3:33     ` Jungo Lin
  2019-07-05  4:22       ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-05  3:33 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, sean.cheng, frederic.chen, rynn.wu, srv_heupstream,
	robh, ryan.yu, frankie.chiu, hverkuil, ddavenport, sj.huang,
	linux-mediatek, laurent.pinchart, matthias.bgg, mchehab,
	linux-arm-kernel, linux-media

Hi Tomasz,

On Mon, 2019-07-01 at 16:25 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Tue, Jun 11, 2019 at 11:53:44AM +0800, Jungo Lin wrote:
> > The purpose of this child device is to provide shared
> > memory management for exchanging tuning data between co-processor
> > and the Pass 1 unit of the camera ISP system, including cache
> > buffer handling.
> > 
> 
> Looks like we haven't really progressed on getting this replaced with
> something that doesn't require so much custom code. Let me propose something
> better then.
> 
> We already have a reserved memory mode in DT. If it has a compatible string
> of "shared-dma-pool", it would be registered in the coherent DMA framework
> [1]. That would make it available for consumer devices to look-up.
> 
> Now if we add a "memory-region" property to the SCP device node and point it
> to our reserved memory node, the SCP driver could look it up and hook to the
> DMA mapping API using of_reserved_mem_device_init_by_idx[2].
> 
> That basically makes any dma_alloc_*(), dma_map_*(), etc. calls on the SCP
> struct device use the coherent DMA ops, which operate on the assigned memory
> pool. With that, the P1 driver could just directly use those calls to
> manage the memory, without any custom code.
> 
> There is an example how this setup works in the s5p-mfc driver[3], but it
> needs to be noted that it creates child nodes, because it can have more than
> 1 DMA port, which may need its own memory pool. In our case, we wouldn't
> need child nodes and could just use the SCP device directly.
> 
> [1] https://elixir.bootlin.com/linux/v5.2-rc7/source/kernel/dma/coherent.c#L345
> [2] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/of/of_reserved_mem.c#L312
> [3] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/media/platform/s5p-mfc/s5p_mfc.c#L1075
> 
> Let me also post some specific comments below, in case we end up still
> needing any of the code.
> 

Thanks your suggestions.

After applying your suggestion in SCP device driver, we could remove
mtk_cam-smem.h/c. Currently, we use dma_alloc_coherent with SCP device
to get SCP address. We could touch the buffer with this SCP address in
SCP processor. 

After that, we use dma_map_page_attrs with P1 device which supports
IOMMU domain to get IOVA address. For this address, we will assign
it to our ISP HW device to proceed.

Below is the snippet for ISP P1 compose buffer initialization.

	ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
				 MAX_COMPOSER_SIZE, &addr, GFP_KERNEL);
	if (!ptr) {
		dev_err(dev, "failed to allocate compose memory\n");
		return -ENOMEM;
	}
	isp_ctx->scp_mem_pa = addr;
	dev_dbg(dev, "scp addr:%pad\n", &addr);

	/* get iova address */
	addr = dma_map_page_attrs(dev, phys_to_page(addr), 0,
				  MAX_COMPOSER_SIZE, DMA_BIDIRECTIONAL,
				  DMA_ATTR_SKIP_CPU_SYNC);
	if (dma_mapping_error(dev, addr)) {
		isp_ctx->scp_mem_pa = 0;
		dev_err(dev, "Failed to map scp iova\n");
		return -ENOMEM;
	}
	isp_ctx->scp_mem_iova = addr;

Moreover, we have another meta input buffer usage.
For this kind of buffer, it will be allocated by V4L2 framework
with dma_alloc_coherent with SCP device. In order to get IOVA,
we will add dma_map_page_attrs in vb2_ops' buf_init function.
In buf_cleanup function, we will call dma_unmap_page_attrs function.

Based on these current implementation, do you think it is correct?
If we got any wrong, please let us know. 

Btw, we also DMA_ATTR_NO_KERNEL_MAPPING DMA attribte to
avoid dma_sync_sg_for_device. Othewise, it will hit the KE.
Maybe we could not get the correct sg_table.
Do you think it is a bug and need to fix?

For this new implementation, it will apply ISP P1 & P2 drivers[1].

[1] https://patchwork.kernel.org/cover/10905221/

> > Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> > ---
> > This patch depends on "Add support for mt8183 SCP"[1].
> > 
> > [1] https://patchwork.kernel.org/cover/10972143/
> > ---
> >  .../platform/mtk-isp/isp_50/cam/Makefile      |   1 +
> >  .../mtk-isp/isp_50/cam/mtk_cam-smem.c         | 304 ++++++++++++++++++
> >  .../mtk-isp/isp_50/cam/mtk_cam-smem.h         |  18 ++
> >  3 files changed, 323 insertions(+)
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h
> > 
> > diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
> > index 95f0b1c8fa1c..d545ca6f09c5 100644
> > --- a/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/Makefile
> > @@ -4,5 +4,6 @@ mtk-cam-isp-objs += mtk_cam-ctrl.o
> >  mtk-cam-isp-objs += mtk_cam-v4l2-util.o
> >  mtk-cam-isp-objs += mtk_cam.o
> >  mtk-cam-isp-objs += mtk_cam-scp.o
> > +mtk-cam-isp-objs += mtk_cam-smem.o
> >  
> >  obj-$(CONFIG_VIDEO_MEDIATEK_ISP_PASS1) += mtk-cam-isp.o
> > \ No newline at end of file
> > diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
> > new file mode 100644
> > index 000000000000..a9845668ce10
> > --- /dev/null
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.c
> > @@ -0,0 +1,304 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +//
> > +// Copyright (c) 2018 MediaTek Inc.
> > +
> > +#include <asm/cacheflush.h>
> > +#include <linux/device.h>
> > +#include <linux/io.h>
> > +#include <linux/iommu.h>
> > +#include <linux/of.h>
> > +#include <linux/of_fdt.h>
> > +#include <linux/of_reserved_mem.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/platform_data/mtk_scp.h>
> > +#include <media/videobuf2-dma-contig.h>
> > +
> > +#include "mtk_cam-smem.h"
> > +
> > +static struct dma_map_ops smem_dma_ops;
> > +
> > +struct mtk_cam_smem_dev {
> > +	struct device *dev;
> > +	struct sg_table sgt;
> > +	struct page **smem_pages;
> > +	dma_addr_t smem_base;
> > +	dma_addr_t smem_dma_base;
> > +	int smem_size;
> > +};
> > +
> > +struct dma_coherent_mem {
> > +	void		*virt_base;
> > +	dma_addr_t	device_base;
> > +	unsigned long	pfn_base;
> > +	int		size;
> > +	int		flags;
> > +	unsigned long	*bitmap;
> > +	spinlock_t	spinlock; /* dma_coherent_mem attributes protection */
> > +	bool		use_dev_dma_pfn_offset;
> > +};
> > +
> > +dma_addr_t mtk_cam_smem_iova_to_scp_addr(struct device *dev,
> > +					 dma_addr_t iova)
> > +{
> > +	struct iommu_domain *domain;
> > +	dma_addr_t addr, limit;
> > +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> > +
> > +	domain = iommu_get_domain_for_dev(dev);
> > +	if (!domain) {
> > +		dev_warn(dev, "No iommu group domain\n");
> > +		return 0;
> > +	}
> > +
> > +	addr = iommu_iova_to_phys(domain, iova);
> > +	limit = smem_dev->smem_base + smem_dev->smem_size;
> > +	if (addr < smem_dev->smem_base || addr >= limit) {
> > +		dev_err(dev,
> > +			"Unexpected scp_addr:%pad must >= %pad and < %pad)\n",
> > +			&addr, &smem_dev->smem_base, &limit);
> > +		return 0;
> > +	}
> > +	return addr;
> > +}
> 
> This isn't correct. One could pass an IOVA that wasn't allocated for the SCP
> and then the address wouldn't be valid, because it would point outside of
> the address range allowed for SCP to access and also it would only point to
> the first page backing the IOVA.
> 
> The correct approach would be to always carry SCP DMA address and IOVA
> together in some kind of struct describing such buffers.
> 

We will remove this function in next patch & handle this in buf_init
function.

> > +
> > +static int mtk_cam_smem_get_sgtable(struct device *dev,
> > +				    struct sg_table *sgt,
> > +				    void *cpu_addr, dma_addr_t dma_addr,
> > +				    size_t size, unsigned long attrs)
> > +{
> > +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> > +	size_t pages_count = PAGE_ALIGN(size) >> PAGE_SHIFT;
> > +	dma_addr_t scp_addr = mtk_cam_smem_iova_to_scp_addr(dev, dma_addr);
> > +	u32 pages_start = (scp_addr - smem_dev->smem_base) >> PAGE_SHIFT;
> > +
> > +	dev_dbg(dev,
> > +		"%s:page:%u va:%pK scp addr:%pad, aligned size:%zu pages:%zu\n",
> > +		__func__, pages_start, cpu_addr, &scp_addr, size, pages_count);
> > +
> > +	return sg_alloc_table_from_pages(sgt,
> > +		smem_dev->smem_pages + pages_start,
> > +		pages_count, 0, size, GFP_KERNEL);
> > +}
> 
> This should be just dma_get_sgtable_attrs(), in the approach I suggested at
> the top.
> 

Yes, we will remove this in next patch.

> > +
> > +static void *mtk_cam_smem_get_cpu_addr(struct mtk_cam_smem_dev *smem_dev,
> > +				       dma_addr_t addr)
> > +{
> > +	struct device *dev = smem_dev->dev;
> > +	struct dma_coherent_mem *dma_mem = dev->dma_mem;
> > +
> > +	if (addr < smem_dev->smem_base ||
> > +	    addr > smem_dev->smem_base + smem_dev->smem_size) {
> 
> This is off by one, should be >=.
> 
> Also, this wouldn't really guarantee the CPU access the caller is going to
> do is valid, because it doesn't consider the access operation size.
> 
> Generally I'd suggest designing the code so that it doesn't have to convert
> offset addresses between different address spaces.
> 

Yes, we will remove this in next patch.

> > +		dev_err(dev, "Invalid scp_addr %pad from sg\n", &addr);
> > +		return NULL;
> > +	}
> > +	return dma_mem->virt_base + (addr - smem_dev->smem_base);
> > +}
> > +
> > +static void mtk_cam_smem_sync_sg_for_cpu(struct device *dev,
> > +					 struct scatterlist *sgl, int nelems,
> > +					 enum dma_data_direction dir)
> > +{
> > +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> > +	dma_addr_t scp_addr = sg_phys(sgl);
> > +	void *cpu_addr = mtk_cam_smem_get_cpu_addr(smem_dev, scp_addr);
> > +
> > +	dev_dbg(dev,
> > +		"__dma_unmap_area:scp_addr:%pad,vaddr:%pK,size:%d,dir:%d\n",
> > +		&scp_addr, cpu_addr, sgl->length, dir);
> > +	__dma_unmap_area(cpu_addr, sgl->length, dir);
> 
> It's not allowed to use this function anywhere outside of the DMA API
> internals. See the comment [4].
> 
> [4] https://elixir.bootlin.com/linux/v5.2-rc7/source/arch/arm64/include/asm/cacheflush.h#L112
> 

Ok, got it and remove this next patch.

> > +}
> > +
> > +static void mtk_cam_smem_sync_sg_for_device(struct device *dev,
> > +					    struct scatterlist *sgl,
> > +					    int nelems,
> > +					    enum dma_data_direction dir)
> > +{
> > +	struct mtk_cam_smem_dev *smem_dev = dev_get_drvdata(dev);
> > +	dma_addr_t scp_addr = sg_phys(sgl);
> > +	void *cpu_addr = mtk_cam_smem_get_cpu_addr(smem_dev, scp_addr);
> > +
> > +	dev_dbg(dev,
> > +		"__dma_map_area:scp_addr:%pad,vaddr:%pK,size:%d,dir:%d\n",
> > +		&scp_addr, cpu_addr, sgl->length, dir);
> > +	__dma_map_area(cpu_addr, sgl->length, dir);
> 
> Ditto.
> 

Ok, got it and remove this next patch.

> > +}
> > +
> > +static void mtk_cam_smem_setup_dma_ops(struct device *dev,
> > +				       struct dma_map_ops *smem_ops)
> > +{
> > +	memcpy((void *)smem_ops, dev->dma_ops, sizeof(*smem_ops));
> > +	smem_ops->get_sgtable = mtk_cam_smem_get_sgtable;
> > +	smem_ops->sync_sg_for_device = mtk_cam_smem_sync_sg_for_device;
> > +	smem_ops->sync_sg_for_cpu = mtk_cam_smem_sync_sg_for_cpu;
> > +	set_dma_ops(dev, smem_ops);
> > +}
> > +
> > +static int mtk_cam_reserved_drm_sg_init(struct mtk_cam_smem_dev *smem_dev)
> > +{
> > +	u32 size_align, n_pages;
> > +	struct device *dev = smem_dev->dev;
> > +	struct sg_table *sgt = &smem_dev->sgt;
> > +	struct page **pages;
> > +	dma_addr_t dma_addr;
> > +	unsigned int i;
> > +	int ret;
> > +
> > +	smem_dev->smem_base = scp_get_reserve_mem_phys(SCP_ISP_MEM2_ID);
> > +	smem_dev->smem_size = scp_get_reserve_mem_size(SCP_ISP_MEM2_ID);
> > +	if (!smem_dev->smem_base || !smem_dev->smem_size)
> > +		return -EPROBE_DEFER;
> > +
> > +	dev_info(dev, "%s dev:0x%pK base:%pad size:%u MiB\n",
> > +		 __func__,
> > +		 smem_dev->dev,
> > +		 &smem_dev->smem_base,
> > +		 (smem_dev->smem_size / SZ_1M));
> > +
> > +	size_align = PAGE_ALIGN(smem_dev->smem_size);
> > +	n_pages = size_align >> PAGE_SHIFT;
> > +
> > +	pages = kmalloc_array(n_pages, sizeof(struct page *), GFP_KERNEL);
> > +	if (!pages)
> > +		return -ENOMEM;
> > +
> > +	for (i = 0; i < n_pages; i++)
> > +		pages[i] = phys_to_page(smem_dev->smem_base + i * PAGE_SIZE);
> > +
> > +	ret = sg_alloc_table_from_pages(sgt, pages, n_pages, 0,
> > +					size_align, GFP_KERNEL);
> > +	if (ret) {
> > +		dev_err(dev, "failed to alloca sg table:%d\n", ret);
> > +		goto fail_table_alloc;
> > +	}
> > +	sgt->nents = dma_map_sg_attrs(dev, sgt->sgl, sgt->orig_nents,
> > +				      DMA_BIDIRECTIONAL,
> > +				      DMA_ATTR_SKIP_CPU_SYNC);
> > +	if (!sgt->nents) {
> > +		dev_err(dev, "failed to dma sg map\n");
> > +		goto fail_map;
> > +	}
> > +
> > +	dma_addr = sg_dma_address(sgt->sgl);
> > +	ret = dma_declare_coherent_memory(dev, smem_dev->smem_base,
> > +					  dma_addr, size_align,
> > +					  DMA_MEMORY_EXCLUSIVE);
> > +	if (ret) {
> > +		dev_err(dev, "Unable to declare smem  memory:%d\n", ret);
> > +		goto fail_map;
> > +	}
> > +
> > +	dev_info(dev, "Coherent mem pa:%pad/%pad, size:%d\n",
> > +		 &smem_dev->smem_base, &dma_addr, size_align);
> > +
> > +	smem_dev->smem_size = size_align;
> > +	smem_dev->smem_pages = pages;
> > +	smem_dev->smem_dma_base = dma_addr;
> > +
> > +	return 0;
> > +
> > +fail_map:
> > +	sg_free_table(sgt);
> > +fail_table_alloc:
> > +	while (n_pages--)
> > +		__free_page(pages[n_pages]);
> > +	kfree(pages);
> > +
> > +	return -ENOMEM;
> > +}
> > +
> > +/* DMA memory related helper functions */
> > +static void mtk_cam_memdev_release(struct device *dev)
> > +{
> > +	vb2_dma_contig_clear_max_seg_size(dev);
> > +}
> > +
> > +static struct device *mtk_cam_alloc_smem_dev(struct device *dev,
> > +					     const char *name)
> > +{
> > +	struct device *child;
> > +	int ret;
> > +
> > +	child = devm_kzalloc(dev, sizeof(*child), GFP_KERNEL);
> > +	if (!child)
> > +		return NULL;
> > +
> > +	child->parent = dev;
> > +	child->iommu_group = dev->iommu_group;
> 
> This isn't something that can be set explicitly. It's an internal field of
> the IOMMU subsystem.
> 
> > +	child->release = mtk_cam_memdev_release;
> > +	dev_set_name(child, name);
> > +	set_dma_ops(child, get_dma_ops(dev));
> > +	child->dma_mask = dev->dma_mask;
> > +	ret = dma_set_coherent_mask(child, DMA_BIT_MASK(32));
> > +	if (ret)
> > +		return NULL;
> > +
> > +	vb2_dma_contig_set_max_seg_size(child, DMA_BIT_MASK(32));
> > +
> > +	if (device_register(child)) {
> > +		device_del(child);
> > +		return NULL;
> > +	}
> > +
> > +	return child;
> > +}
> 
> We shouldn't need child devices, just one SCP device, as I mentioned above.
> 

Ok, got your point. Just keep one single SCP device for single reserved
memory range.

> > +
> > +static int mtk_cam_composer_dma_init(struct mtk_isp_p1_ctx *isp_ctx)
> > +{
> > +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	u32 size;
> > +	dma_addr_t addr;
> > +
> > +	isp_ctx->scp_mem_pa = scp_get_reserve_mem_phys(SCP_ISP_MEM_ID);
> > +	size = PAGE_ALIGN(scp_get_reserve_mem_size(SCP_ISP_MEM_ID));
> > +	if (!isp_ctx->scp_mem_pa || !size)
> > +		return -EPROBE_DEFER;
> > +
> > +	dev_info(dev, "scp addr:%pad size:0x%x\n", &isp_ctx->scp_mem_pa, size);
> 
> This isn't something that deserves the "info" log level. Should be "dbg"
> or removed.
> 

Ok, we will change the log level from info to debug.

> Best regards,
> Tomasz
> 
> _______________________________________________
> Linux-mediatek mailing list
> Linux-mediatek@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-mediatek

Thanks for your valued comments.

Best regards,


Jungo



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-05  3:33     ` Jungo Lin
@ 2019-07-05  4:22       ` Tomasz Figa
  2019-07-05  5:44         ` Jungo Lin
  2019-07-05  7:59         ` Jungo Lin
  0 siblings, 2 replies; 45+ messages in thread
From: Tomasz Figa @ 2019-07-05  4:22 UTC (permalink / raw)
  To: Jungo Lin
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi Jungo,

On Fri, Jul 5, 2019 at 12:33 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi Tomasz,
>
> On Mon, 2019-07-01 at 16:25 +0900, Tomasz Figa wrote:
> > Hi Jungo,
> >
> > On Tue, Jun 11, 2019 at 11:53:44AM +0800, Jungo Lin wrote:
> > > The purpose of this child device is to provide shared
> > > memory management for exchanging tuning data between co-processor
> > > and the Pass 1 unit of the camera ISP system, including cache
> > > buffer handling.
> > >
> >
> > Looks like we haven't really progressed on getting this replaced with
> > something that doesn't require so much custom code. Let me propose something
> > better then.
> >
> > We already have a reserved memory mode in DT. If it has a compatible string
> > of "shared-dma-pool", it would be registered in the coherent DMA framework
> > [1]. That would make it available for consumer devices to look-up.
> >
> > Now if we add a "memory-region" property to the SCP device node and point it
> > to our reserved memory node, the SCP driver could look it up and hook to the
> > DMA mapping API using of_reserved_mem_device_init_by_idx[2].
> >
> > That basically makes any dma_alloc_*(), dma_map_*(), etc. calls on the SCP
> > struct device use the coherent DMA ops, which operate on the assigned memory
> > pool. With that, the P1 driver could just directly use those calls to
> > manage the memory, without any custom code.
> >
> > There is an example how this setup works in the s5p-mfc driver[3], but it
> > needs to be noted that it creates child nodes, because it can have more than
> > 1 DMA port, which may need its own memory pool. In our case, we wouldn't
> > need child nodes and could just use the SCP device directly.
> >
> > [1] https://elixir.bootlin.com/linux/v5.2-rc7/source/kernel/dma/coherent.c#L345
> > [2] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/of/of_reserved_mem.c#L312
> > [3] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/media/platform/s5p-mfc/s5p_mfc.c#L1075
> >
> > Let me also post some specific comments below, in case we end up still
> > needing any of the code.
> >
>
> Thanks your suggestions.
>
> After applying your suggestion in SCP device driver, we could remove
> mtk_cam-smem.h/c. Currently, we use dma_alloc_coherent with SCP device
> to get SCP address. We could touch the buffer with this SCP address in
> SCP processor.
>
> After that, we use dma_map_page_attrs with P1 device which supports
> IOMMU domain to get IOVA address. For this address, we will assign
> it to our ISP HW device to proceed.
>
> Below is the snippet for ISP P1 compose buffer initialization.
>
>         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
>                                  MAX_COMPOSER_SIZE, &addr, GFP_KERNEL);
>         if (!ptr) {
>                 dev_err(dev, "failed to allocate compose memory\n");
>                 return -ENOMEM;
>         }
>         isp_ctx->scp_mem_pa = addr;

addr contains a DMA address, not a physical address. Could we call it
scp_mem_dma instead?

>         dev_dbg(dev, "scp addr:%pad\n", &addr);
>
>         /* get iova address */
>         addr = dma_map_page_attrs(dev, phys_to_page(addr), 0,

addr is a DMA address, so phys_to_page() can't be called on it. The
simplest thing here would be to use dma_map_single() with ptr as the
CPU address expected.

>                                   MAX_COMPOSER_SIZE, DMA_BIDIRECTIONAL,
>                                   DMA_ATTR_SKIP_CPU_SYNC);
>         if (dma_mapping_error(dev, addr)) {
>                 isp_ctx->scp_mem_pa = 0;

We also need to free the allocated memory.

>                 dev_err(dev, "Failed to map scp iova\n");
>                 return -ENOMEM;
>         }
>         isp_ctx->scp_mem_iova = addr;
>
> Moreover, we have another meta input buffer usage.
> For this kind of buffer, it will be allocated by V4L2 framework
> with dma_alloc_coherent with SCP device. In order to get IOVA,
> we will add dma_map_page_attrs in vb2_ops' buf_init function.
> In buf_cleanup function, we will call dma_unmap_page_attrs function.

As per above, we don't have access to the struct page we want to map.
We probably want to get the CPU VA using vb2_plane_vaddr() and call
dma_map_single() instead.

>
> Based on these current implementation, do you think it is correct?
> If we got any wrong, please let us know.
>
> Btw, we also DMA_ATTR_NO_KERNEL_MAPPING DMA attribte to
> avoid dma_sync_sg_for_device. Othewise, it will hit the KE.
> Maybe we could not get the correct sg_table.
> Do you think it is a bug and need to fix?

I think DMA_ATTR_NO_KERNEL_MAPPING is good to have for all the buffers
that don't need to be accessed from the kernel anyway, to avoid
unnecessary kernel mapping operations. However, for coherent memory
pool, it doesn't change anything, because the memory always has a
kernel mapping. We also need the kernel virtual address for
dma_map_single(). Also the flag doesn't eliminate the need to do the
sync, e.g. if the userspace accesses the buffer.

Could you give me more information about the failure you're seeing?
Where is the dma_sync_sg_for_device() called from? Where do you get
the sgtable from?

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-05  4:22       ` Tomasz Figa
@ 2019-07-05  5:44         ` Jungo Lin
  2019-07-05  7:59         ` Jungo Lin
  1 sibling, 0 replies; 45+ messages in thread
From: Jungo Lin @ 2019-07-05  5:44 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi, Tomasz:
On Fri, 2019-07-05 at 13:22 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Fri, Jul 5, 2019 at 12:33 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> >
> > Hi Tomasz,
> >
> > On Mon, 2019-07-01 at 16:25 +0900, Tomasz Figa wrote:
> > > Hi Jungo,
> > >
> > > On Tue, Jun 11, 2019 at 11:53:44AM +0800, Jungo Lin wrote:
> > > > The purpose of this child device is to provide shared
> > > > memory management for exchanging tuning data between co-processor
> > > > and the Pass 1 unit of the camera ISP system, including cache
> > > > buffer handling.
> > > >
> > >
> > > Looks like we haven't really progressed on getting this replaced with
> > > something that doesn't require so much custom code. Let me propose something
> > > better then.
> > >
> > > We already have a reserved memory mode in DT. If it has a compatible string
> > > of "shared-dma-pool", it would be registered in the coherent DMA framework
> > > [1]. That would make it available for consumer devices to look-up.
> > >
> > > Now if we add a "memory-region" property to the SCP device node and point it
> > > to our reserved memory node, the SCP driver could look it up and hook to the
> > > DMA mapping API using of_reserved_mem_device_init_by_idx[2].
> > >
> > > That basically makes any dma_alloc_*(), dma_map_*(), etc. calls on the SCP
> > > struct device use the coherent DMA ops, which operate on the assigned memory
> > > pool. With that, the P1 driver could just directly use those calls to
> > > manage the memory, without any custom code.
> > >
> > > There is an example how this setup works in the s5p-mfc driver[3], but it
> > > needs to be noted that it creates child nodes, because it can have more than
> > > 1 DMA port, which may need its own memory pool. In our case, we wouldn't
> > > need child nodes and could just use the SCP device directly.
> > >
> > > [1] https://elixir.bootlin.com/linux/v5.2-rc7/source/kernel/dma/coherent.c#L345
> > > [2] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/of/of_reserved_mem.c#L312
> > > [3] https://elixir.bootlin.com/linux/v5.2-rc7/source/drivers/media/platform/s5p-mfc/s5p_mfc.c#L1075
> > >
> > > Let me also post some specific comments below, in case we end up still
> > > needing any of the code.
> > >
> >
> > Thanks your suggestions.
> >
> > After applying your suggestion in SCP device driver, we could remove
> > mtk_cam-smem.h/c. Currently, we use dma_alloc_coherent with SCP device
> > to get SCP address. We could touch the buffer with this SCP address in
> > SCP processor.
> >
> > After that, we use dma_map_page_attrs with P1 device which supports
> > IOMMU domain to get IOVA address. For this address, we will assign
> > it to our ISP HW device to proceed.
> >
> > Below is the snippet for ISP P1 compose buffer initialization.
> >
> >         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
> >                                  MAX_COMPOSER_SIZE, &addr, GFP_KERNEL);
> >         if (!ptr) {
> >                 dev_err(dev, "failed to allocate compose memory\n");
> >                 return -ENOMEM;
> >         }
> >         isp_ctx->scp_mem_pa = addr;
> 
> addr contains a DMA address, not a physical address. Could we call it
> scp_mem_dma instead?
> 

Ok, we will rename this.

> >         dev_dbg(dev, "scp addr:%pad\n", &addr);
> >
> >         /* get iova address */
> >         addr = dma_map_page_attrs(dev, phys_to_page(addr), 0,
> 
> addr is a DMA address, so phys_to_page() can't be called on it. The
> simplest thing here would be to use dma_map_single() with ptr as the
> CPU address expected.
> 

Got it. We will revise to use dma_map_single() with ptr.

> >                                   MAX_COMPOSER_SIZE, DMA_BIDIRECTIONAL,
> >                                   DMA_ATTR_SKIP_CPU_SYNC);
> >         if (dma_mapping_error(dev, addr)) {
> >                 isp_ctx->scp_mem_pa = 0;
> 
> We also need to free the allocated memory.
> 

Ok, we will add the dma_unmap_single to free the allocated memory.

> >                 dev_err(dev, "Failed to map scp iova\n");
> >                 return -ENOMEM;
> >         }
> >         isp_ctx->scp_mem_iova = addr;
> >
> > Moreover, we have another meta input buffer usage.
> > For this kind of buffer, it will be allocated by V4L2 framework
> > with dma_alloc_coherent with SCP device. In order to get IOVA,
> > we will add dma_map_page_attrs in vb2_ops' buf_init function.
> > In buf_cleanup function, we will call dma_unmap_page_attrs function.
> 
> As per above, we don't have access to the struct page we want to map.
> We probably want to get the CPU VA using vb2_plane_vaddr() and call
> dma_map_single() instead.
> 

Got it. We will revise this to use dma_map_single() with CPU VA which is
got from vb2_plane_vaddr() function.

> >
> > Based on these current implementation, do you think it is correct?
> > If we got any wrong, please let us know.
> >
> > Btw, we also DMA_ATTR_NO_KERNEL_MAPPING DMA attribte to
> > avoid dma_sync_sg_for_device. Othewise, it will hit the KE.
> > Maybe we could not get the correct sg_table.
> > Do you think it is a bug and need to fix?
> 
> I think DMA_ATTR_NO_KERNEL_MAPPING is good to have for all the buffers
> that don't need to be accessed from the kernel anyway, to avoid
> unnecessary kernel mapping operations. However, for coherent memory
> pool, it doesn't change anything, because the memory always has a
> kernel mapping. We also need the kernel virtual address for
> dma_map_single(). Also the flag doesn't eliminate the need to do the
> sync, e.g. if the userspace accesses the buffer.
> 
> Could you give me more information about the failure you're seeing?
> Where is the dma_sync_sg_for_device() called from? Where do you get
> the sgtable from?
> 
> Best regards,
> Tomasz

Sorry. I forgot provide one information related to this issue.
Here is the call stack of panic KE if we enable DMA_ATTR_NON_CONSISTENT
DMA flag. Maybe we should not enable this flag for coherent memory pool.

[Function]
vb2_dc_alloc

[Code]
	if (!(buf->attrs & DMA_ATTR_NO_KERNEL_MAPPING) &&
	    (buf->attrs & DMA_ATTR_NON_CONSISTENT))
		buf->dma_sgt = vb2_dc_get_base_sgt(buf);

[KE]
[   59.234326] pstate: 80000005 (Nzcv daif -PAN -UAO)
[   59.234935] pc : __clean_dcache_area_poc+0x20/0x38
[   59.235537] lr : __swiotlb_sync_sg_for_device+0x74/0x9c
[   59.249430] Call trace:
[   59.249742]  __clean_dcache_area_poc+0x20/0x38
[   59.250303]  vb2_dc_prepare+0x5c/0x6c
[   59.250763]  __buf_prepare+0x790/0x8a4
[   59.251234]  vb2_req_prepare+0x38/0x68
[   59.251707]  vb2_request_validate+0x40/0x9c
[   59.252235]  media_request_ioctl+0x124/0x2a4
[   59.252774]  __arm64_compat_sys_ioctl+0xf4/0x25c
[   59.253356]  el0_svc_common+0xa4/0x154
[   59.253828]  el0_svc_compat_handler+0x2c/0x38
[   59.254377]  el0_svc_compat+0x8/0x18
[   59.254827] Code: 9ac32042 8b010001 d1000443 8a230000 (d50b7a20)
[   59.255592] ---[ end trace eb37ebade032c2fc ]---
[   59.256173] Kernel panic - not syncing: Fatal exception

Thanks,

Jungo



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-05  4:22       ` Tomasz Figa
  2019-07-05  5:44         ` Jungo Lin
@ 2019-07-05  7:59         ` Jungo Lin
  2019-07-23  7:20           ` Tomasz Figa
  1 sibling, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-05  7:59 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi Tomasz:

On Fri, 2019-07-05 at 13:22 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Fri, Jul 5, 2019 at 12:33 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> >
> > Hi Tomasz,

[snip]

> > After applying your suggestion in SCP device driver, we could remove
> > mtk_cam-smem.h/c. Currently, we use dma_alloc_coherent with SCP device
> > to get SCP address. We could touch the buffer with this SCP address in
> > SCP processor.
> >
> > After that, we use dma_map_page_attrs with P1 device which supports
> > IOMMU domain to get IOVA address. For this address, we will assign
> > it to our ISP HW device to proceed.
> >
> > Below is the snippet for ISP P1 compose buffer initialization.
> >
> >         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
> >                                  MAX_COMPOSER_SIZE, &addr, GFP_KERNEL);
> >         if (!ptr) {
> >                 dev_err(dev, "failed to allocate compose memory\n");
> >                 return -ENOMEM;
> >         }
> >         isp_ctx->scp_mem_pa = addr;
> 
> addr contains a DMA address, not a physical address. Could we call it
> scp_mem_dma instead?
> 
> >         dev_dbg(dev, "scp addr:%pad\n", &addr);
> >
> >         /* get iova address */
> >         addr = dma_map_page_attrs(dev, phys_to_page(addr), 0,
> 
> addr is a DMA address, so phys_to_page() can't be called on it. The
> simplest thing here would be to use dma_map_single() with ptr as the
> CPU address expected.
> 

We have changed to use ma_map_single() with ptr, but encounter IOMMU
error. From the debug log of iommu_dma_map_page[3], we got
0x0000000054800000 instead of expected address: 0x0000000050800000[2].
There is a address offset(0x4000000). If we change to use
dma_map_page_attrs with phys_to_page(addr), the address is correct as we
expected[2]. Do you have any suggestion on this issue? Do we miss
something?

[1]
[    1.344786] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
device_base:0x0000000050000000 dma:0x0000000050800000
virt_base:ffffff8014000000 va:ffffff8014800000

[    1.346890] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
va:ffffff8014800000

[    1.347864] iommu_dma_map_page:0x0000000054800000 offset:0
[    1.348562] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000

[2]
[    1.346738] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
device_base:0x0000000050000000 dma:0x0000000050800000
virt_base:ffffff8014000000 va:ffffff8014800000
[    1.348841] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
va:ffffff8014800000
[    1.349816] iommu_dma_map_page:0x0000000050800000 offset:0
[    1.350514] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000


[3]
dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
		unsigned long offset, size_t size, int prot)
{
	phys_addr_t phys = page_to_phys(page);
	pr_err("iommu_dma_map_page:%pa offset:%lu\n", &phys, offset);

	return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot,
			iommu_get_dma_domain(dev));
}

[snip]

Best regards,

Jungo


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-06-11  3:53 ` [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions Jungo Lin
@ 2019-07-10  9:54   ` Tomasz Figa
  2019-07-18  4:39     ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-10  9:54 UTC (permalink / raw)
  To: Jungo Lin
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi Jungo,

On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
> Implement standard V4L2 video driver that utilizes V4L2
> and media framework APIs. In this driver, supports one media
> device, one sub-device and seven video devices during
> initialization. Moreover, it also connects with sensor and
> seninf drivers with V4L2 async APIs.
> 
> (The current metadata interface used in meta input and partial
> meta nodes is only a temporary solution to kick off the driver
> development and is not ready to be reviewed yet.)
> 
> Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> ---
> This patch depends on "media: support Mediatek sensor interface driver"[1].
> 
> ISP P1 sub-device communicates with seninf sub-device with CIO.
> 
> [1]. media: support Mediatek sensor interface driver
> https://patchwork.kernel.org/cover/10979135/
> ---
>  .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
>  .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c    | 1674 +++++++++++++++++
>  .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h    |  173 ++
>  3 files changed, 1848 insertions(+)
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
> 

Thanks for the patch. Please see my comments inline.

[snip]

> +static void mtk_cam_req_try_isp_queue(struct mtk_cam_dev *cam_dev,
> +				      struct media_request *new_req)
> +{
> +	struct mtk_cam_dev_request *req, *req_safe, *cam_dev_req;
> +	struct device *dev = &cam_dev->pdev->dev;
> +
> +	dev_dbg(dev, "%s new req:%d", __func__, !new_req);
> +
> +	if (!cam_dev->streaming) {
> +		cam_dev_req = mtk_cam_req_to_dev_req(new_req);
> +		spin_lock(&cam_dev->req_lock);
> +		list_add_tail(&cam_dev_req->list, &cam_dev->req_list);
> +		spin_unlock(&cam_dev->req_lock);
> +		dev_dbg(dev, "%s: stream off, no ISP enqueue\n", __func__);
> +		return;
> +	}
> +
> +	/* Normal enqueue flow */
> +	if (new_req) {
> +		mtk_isp_req_enqueue(dev, new_req);
> +		return;
> +	}
> +
> +	/* Flush all media requests wehen first stream on */
> +	list_for_each_entry_safe(req, req_safe, &cam_dev->req_list, list) {
> +		list_del(&req->list);
> +		mtk_isp_req_enqueue(dev, &req->req);
> +	}
> +}

This will have to be redone, as per the other suggestions, but generally one
would have a function that tries to queue as much as possible from a list to
the hardware and another function that adds a request to the list and calls
the first function.

> +
> +static void mtk_cam_req_queue(struct media_request *req)
> +{
> +	struct mtk_cam_dev *cam_dev = mtk_cam_mdev_to_dev(req->mdev);
> +
> +	vb2_request_queue(req);
> +	mtk_cam_req_try_isp_queue(cam_dev, req);

Looks like this driver is suffering from versy similar problems in request
handling as the DIP driver used to.

I'd prefer to save my time and avoid repeating the same comments, so please
check my comments for the DIP driver and apply them to this one too:

https://patchwork.kernel.org/patch/10905223/

> +}
> +
> +static struct media_request *mtk_cam_req_alloc(struct media_device *mdev)
> +{
> +	struct mtk_cam_dev_request *cam_dev_req;
> +
> +	cam_dev_req = kzalloc(sizeof(*cam_dev_req), GFP_KERNEL);
> +
> +	return &cam_dev_req->req;
> +}
> +
> +static void mtk_cam_req_free(struct media_request *req)
> +{
> +	struct mtk_cam_dev_request *cam_dev_req = mtk_cam_req_to_dev_req(req);
> +
> +	kfree(cam_dev_req);
> +}
> +
> +static __u32 img_get_pixel_byte_by_fmt(__u32 pix_fmt)

Doesn't this function return bits not bytes?

> +{
> +	switch (pix_fmt) {
> +	case V4L2_PIX_FMT_MTISP_B8:
> +	case V4L2_PIX_FMT_MTISP_F8:
> +		return 8;
> +	case V4L2_PIX_FMT_MTISP_B10:
> +	case V4L2_PIX_FMT_MTISP_F10:
> +		return 10;
> +	case V4L2_PIX_FMT_MTISP_B12:
> +	case V4L2_PIX_FMT_MTISP_F12:
> +		return 12;
> +	case V4L2_PIX_FMT_MTISP_B14:
> +	case V4L2_PIX_FMT_MTISP_F14:
> +		return 14;
> +	default:
> +		return 0;
> +	}
> +}
> +
> +static __u32 img_cal_main_stream_stride(struct device *dev, __u32 width,
> +					__u32 pix_fmt)
> +{
> +	__u32 stride;
> +	__u32 pixel_byte = img_get_pixel_byte_by_fmt(pix_fmt);

The __ prefixed types should be used only inside UAPI. Please change the
driver to use the normal ones.

> +
> +	width = ALIGN(width, 4);

If there is some alignment requirement for width, it should be handled by
TRY_/S_FMT and here we should already assume everything properly aligned.

> +	stride = ALIGN(DIV_ROUND_UP(width * pixel_byte, 8), 2);
> +
> +	dev_dbg(dev, "main width:%d, stride:%d\n", width, stride);
> +
> +	return stride;
> +}
> +
> +static __u32 img_cal_packed_out_stride(struct device *dev, __u32 width,
> +				       __u32 pix_fmt)
> +{
> +	__u32 stride;
> +	__u32 pixel_byte = img_get_pixel_byte_by_fmt(pix_fmt);
> +
> +	width = ALIGN(width, 4);

Ditto.

> +	stride = DIV_ROUND_UP(width * 3, 2);

Could we introduce a local variable for this intermediate value, so that its
name could explain what the value is?

> +	stride = DIV_ROUND_UP(stride * pixel_byte, 8);
> +
> +	if (pix_fmt == V4L2_PIX_FMT_MTISP_F10)
> +		stride = ALIGN(stride, 4);

Is it expected that only the F10 format needs this alignment?

> +
> +	dev_dbg(dev, "packed width:%d, stride:%d\n", width, stride);
> +
> +	return stride;
> +}
> +
> +static __u32 img_cal_stride(struct device *dev,
> +			    int node_id,
> +			    __u32 width,
> +			    __u32 pix_fmt)
> +{
> +	__u32 bpl;
> +
> +	/* Currently, only support one_pixel_mode */
> +	if (node_id == MTK_CAM_P1_MAIN_STREAM_OUT)
> +		bpl = img_cal_main_stream_stride(dev, width, pix_fmt);
> +	else if (node_id == MTK_CAM_P1_PACKED_BIN_OUT)
> +		bpl = img_cal_packed_out_stride(dev, width, pix_fmt);
> +
> +	/* For DIP HW constrained, it needs 4 byte alignment */
> +	bpl = ALIGN(bpl, 4);
> +
> +	return bpl;
> +}
> +
> +static const struct v4l2_format *
> +mtk_cam_dev_find_fmt(struct mtk_cam_dev_node_desc *desc, u32 format)
> +{
> +	unsigned int i;
> +	const struct v4l2_format *dev_fmt;
> +
> +	for (i = 0; i < desc->num_fmts; i++) {
> +		dev_fmt = &desc->fmts[i];
> +		if (dev_fmt->fmt.pix_mp.pixelformat == format)
> +			return dev_fmt;
> +	}
> +
> +	return NULL;
> +}
> +
> +/* Calcuate mplane pix format */
> +static void
> +mtk_cam_dev_cal_mplane_fmt(struct device *dev,
> +			   struct v4l2_pix_format_mplane *dest_fmt,
> +			   unsigned int node_id)
> +{
> +	unsigned int i;
> +	__u32 bpl, sizeimage, imagsize;

Perhaps s/sizeimage/plane_size/ and s/imagsize/total_size/?

> +
> +	imagsize = 0;
> +	for (i = 0 ; i < dest_fmt->num_planes; ++i) {
> +		bpl = img_cal_stride(dev,
> +				     node_id,
> +				     dest_fmt->width,
> +				     dest_fmt->pixelformat);
> +		sizeimage = bpl * dest_fmt->height;
> +		imagsize += sizeimage;
> +		dest_fmt->plane_fmt[i].bytesperline = bpl;
> +		dest_fmt->plane_fmt[i].sizeimage = sizeimage;
> +		memset(dest_fmt->plane_fmt[i].reserved,
> +		       0, sizeof(dest_fmt->plane_fmt[i].reserved));

This memset is not needed. The core clears the reserved fields
automatically:

https://elixir.bootlin.com/linux/v5.2/source/drivers/media/v4l2-core/v4l2-ioctl.c#L1559

(We may want to backport the patch that added that to our 4.19 branch.)

> +		dev_dbg(dev, "plane:%d,bpl:%d,sizeimage:%u\n",
> +			i,  bpl, dest_fmt->plane_fmt[i].sizeimage);
> +	}
> +
> +	if (dest_fmt->num_planes == 1)
> +		dest_fmt->plane_fmt[0].sizeimage = imagsize;

Hmm, we only seem to support 1 plane raw formats in this driver. Does the
hardware support any formats with more than 1 plane? If not, all the code
should be simplified to just assume 1 plane.

> +}
> +
> +static void
> +mtk_cam_dev_set_img_fmt(struct device *dev,
> +			struct v4l2_pix_format_mplane *dest_fmt,
> +			const struct v4l2_pix_format_mplane *src_fmt,
> +			unsigned int node_id)
> +{
> +	dest_fmt->width = src_fmt->width;
> +	dest_fmt->height = src_fmt->height;
> +	dest_fmt->pixelformat = src_fmt->pixelformat;
> +	dest_fmt->field = src_fmt->field;
> +	dest_fmt->colorspace = src_fmt->colorspace;
> +	dest_fmt->num_planes = src_fmt->num_planes;
> +	/* Use default */
> +	dest_fmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +	dest_fmt->quantization = V4L2_QUANTIZATION_DEFAULT;
> +	dest_fmt->xfer_func =
> +		V4L2_MAP_XFER_FUNC_DEFAULT(dest_fmt->colorspace);
> +	memset(dest_fmt->reserved, 0, sizeof(dest_fmt->reserved));

Given that src_fmt should already be validated and have any fields adjusted
to match the driver requirements, wouldn't all the lines above be equivalent
to *dest_fmt = *src_fmt?

We probably want to move setting all the constant fields to
mtk_cam_vidioc_try_fmt().

> +
> +	dev_dbg(dev, "%s: Dest Fmt:%c%c%c%c, w*h:%d*%d\n",
> +		__func__,
> +		(dest_fmt->pixelformat & 0xFF),
> +		(dest_fmt->pixelformat >> 8) & 0xFF,
> +		(dest_fmt->pixelformat >> 16) & 0xFF,
> +		(dest_fmt->pixelformat >> 24) & 0xFF,
> +		dest_fmt->width,
> +		dest_fmt->height);
> +
> +	mtk_cam_dev_cal_mplane_fmt(dev, dest_fmt, node_id);

This should have been called already before this function was called,
because src_fmt should be already expected to contain valid settings. In
fact, this is already called in mtk_cam_vidioc_try_fmt().

> +}
> +
> +/* Get the default format setting */
> +static void
> +mtk_cam_dev_load_default_fmt(struct device *dev,

Please don't pass struct device pointer around, but instead just the main
driver data struct, which should be much more convenient for accessing
various driver data. Please fix the other functions as well.

> +			     struct mtk_cam_dev_node_desc *queue_desc,
> +			     struct v4l2_format *dest)
> +{
> +	const struct v4l2_format *default_fmt =
> +		&queue_desc->fmts[queue_desc->default_fmt_idx];
> +
> +	dest->type = queue_desc->buf_type;
> +
> +	/* Configure default format based on node type */
> +	if (queue_desc->image) {
> +		mtk_cam_dev_set_img_fmt(dev,
> +					&dest->fmt.pix_mp,
> +					&default_fmt->fmt.pix_mp,
> +					queue_desc->id);

We should probably just call mtk_cam_vidioc_s_fmt() here, with a dummy
v4l2_format struct and have any incorrect fields replaced by
mtk_cam_vidioc_try_fmt(), since it's the same logic, as if setting invalid
v4l2_format at runtime.

> +	} else {
> +		dest->fmt.meta.dataformat = default_fmt->fmt.meta.dataformat;
> +		dest->fmt.meta.buffersize = default_fmt->fmt.meta.buffersize;
> +	}
> +}
> +
> +static int mtk_cam_isp_open(struct file *file)
> +{
> +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> +	struct device *dev = &cam_dev->pdev->dev;
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +	int ret;
> +
> +	mutex_lock(&cam_dev->lock);
> +	ret = v4l2_fh_open(file);
> +	if (ret)
> +		goto unlock;
> +
> +	ret = v4l2_pipeline_pm_use(&node->vdev.entity, 1);

Please don't power on open. Normally applications keep the device nodes open
all the time, so they would keep everything powered on.

Normally this should be done as late as possible, ideally when starting the
streaming.

> +	if (ret)
> +		dev_err(dev, "%s fail:%d", __func__, ret);
> +
> +unlock:
> +	mutex_unlock(&cam_dev->lock);
> +
> +	return ret;
> +}
> +
> +static int mtk_cam_isp_release(struct file *file)
> +{
> +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +
> +	mutex_lock(&cam_dev->lock);
> +	v4l2_pipeline_pm_use(&node->vdev.entity, 0);
> +	vb2_fop_release(file);
> +	mutex_unlock(&cam_dev->lock);
> +
> +	return 0;
> +}

If we remove power handling from open and release, we should be able to just
use v4l2_fh_open() and vb2_fop_release() directly in the
v4l2_file_operations struct.

> +
> +static struct v4l2_subdev *
> +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> +{
> +	struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> +	struct media_entity *entity;
> +	struct device *dev = &cam_dev->pdev->dev;
> +	struct v4l2_subdev *sensor;

This variable would be unitialized if there is no streaming sensor. Was
there no compiler warning generated for this?

> +
> +	media_device_for_each_entity(entity, mdev) {
> +		dev_dbg(dev, "media entity: %s:0x%x\n",
> +			entity->name, entity->function);
> +		if (entity->function == MEDIA_ENT_F_CAM_SENSOR &&
> +		    entity->stream_count) {
> +			sensor = media_entity_to_v4l2_subdev(entity);
> +			dev_dbg(dev, "Sensor found: %s\n", entity->name);
> +			break;
> +		}
> +	}
> +
> +	if (!sensor)
> +		dev_err(dev, "Sensor is not connected\n");
> +
> +	return sensor;
> +}
> +
> +static int mtk_cam_cio_stream_on(struct mtk_cam_dev *cam_dev)
> +{
> +	struct device *dev = &cam_dev->pdev->dev;
> +	int ret;
> +
> +	/* Align vb2_core_streamon design */
> +	if (cam_dev->streaming) {
> +		dev_warn(dev, "already streaming\n", dev);
> +		return 0;
> +	}

Could we check this in the caller?

> +
> +	if (!cam_dev->seninf) {
> +		dev_err(dev, "no seninf connected:%d\n", ret);
> +		return -EPERM;

I don't think -EPERM is a good error code here. It's about a missing seninf
device, so perhaps -ENODEV?

> +	}
> +
> +	/* Get active sensor from graph topology */
> +	cam_dev->sensor = mtk_cam_cio_get_active_sensor(cam_dev);
> +	if (!cam_dev->sensor)
> +		return -EPERM;

> -ENODEV

> +
> +	ret = mtk_isp_config(dev);
> +	if (ret)
> +		return -EPERM;

Maybe just return ret?

> +
> +	/* Seninf must stream on first */
> +	ret = v4l2_subdev_call(cam_dev->seninf, video, s_stream, 1);
> +	if (ret) {
> +		dev_err(dev, "%s stream on failed:%d\n",
> +			cam_dev->seninf->entity.name, ret);
> +		return -EPERM;

return ret?

> +	}
> +
> +	ret = v4l2_subdev_call(cam_dev->sensor, video, s_stream, 1);
> +	if (ret) {
> +		dev_err(dev, "%s stream on failed:%d\n",
> +			cam_dev->sensor->entity.name, ret);
> +		goto fail_sensor_on;
> +	}
> +
> +	cam_dev->streaming = true;
> +	mtk_cam_req_try_isp_queue(cam_dev, NULL);
> +	isp_composer_stream(dev, 1);
> +	dev_dbg(dev, "streamed on Pass 1\n");
> +
> +	return 0;
> +
> +fail_sensor_on:
> +	v4l2_subdev_call(cam_dev->seninf, video, s_stream, 0);
> +
> +	return -EPERM;

return ret?

> +}
> +
> +static int mtk_cam_cio_stream_off(struct mtk_cam_dev *cam_dev)
> +{
> +	struct device *dev = &cam_dev->pdev->dev;
> +	int ret;
> +
> +	if (!cam_dev->streaming) {
> +		dev_warn(dev, "already stream off");
> +		return 0;
> +	}

Could we check this in the caller?

> +
> +	ret = v4l2_subdev_call(cam_dev->sensor, video, s_stream, 0);
> +	if (ret) {
> +		dev_err(dev, "%s stream off failed:%d\n",
> +			cam_dev->sensor->entity.name, ret);
> +		return -EPERM;
> +	}
> +
> +	ret = v4l2_subdev_call(cam_dev->seninf, video, s_stream, 0);
> +	if (ret) {
> +		dev_err(dev, "%s stream off failed:%d\n",
> +			cam_dev->seninf->entity.name, ret);
> +		return -EPERM;
> +	}
> +
> +	isp_composer_stream(dev, 0);

Shouldn't we synchronously wait for the streaming to stop here? Otherwise we
can't guarantee that the hardware releases all the memory that we're going
to free once this function returns.

> +	cam_dev->streaming = false;
> +	dev_dbg(dev, "streamed off Pass 1\n");
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_sd_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> +	struct mtk_cam_dev *cam_dev = mtk_cam_subdev_to_dev(sd);
> +
> +	if (enable)
> +		return mtk_cam_cio_stream_on(cam_dev);
> +	else
> +		return mtk_cam_cio_stream_off(cam_dev);
> +}
> +
> +static int mtk_cam_sd_subscribe_event(struct v4l2_subdev *subdev,
> +				      struct v4l2_fh *fh,
> +				      struct v4l2_event_subscription *sub)
> +{
> +	switch (sub->type) {
> +	case V4L2_EVENT_FRAME_SYNC:
> +		return v4l2_event_subscribe(fh, sub, 0, NULL);
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
> +static int mtk_cam_sd_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct mtk_cam_dev *cam_dev = mtk_cam_subdev_to_dev(sd);
> +
> +	dev_dbg(&cam_dev->pdev->dev, "%s:%d", __func__, on);
> +
> +	return on ? mtk_isp_power_init(cam_dev) :
> +		    mtk_isp_power_release(&cam_dev->pdev->dev);

s_power is a historical thing and we shouldn't be implementing it. Instead,
we should use runtime PM and call pm_runtime_get_sync(), pm_runtime_put()
whenever we start and stop streaming respectively.

> +}
> +
> +static int mtk_cam_media_link_setup(struct media_entity *entity,
> +				    const struct media_pad *local,
> +				    const struct media_pad *remote, u32 flags)
> +{
> +	struct mtk_cam_dev *cam_dev =
> +		container_of(entity, struct mtk_cam_dev, subdev.entity);
> +	u32 pad = local->index;
> +
> +	dev_dbg(&cam_dev->pdev->dev, "%s: %d -> %d flags:0x%x\n",
> +		__func__, pad, remote->index, flags);
> +
> +	if (pad < MTK_CAM_P1_TOTAL_NODES)

I assume this check is needed, because the pads with higher indexes are not
video nodes? If so, a comment would be helpful here.

> +		cam_dev->vdev_nodes[pad].enabled =
> +			!!(flags & MEDIA_LNK_FL_ENABLED);
> +
> +	return 0;
> +}
> +
> +static void mtk_cam_vb2_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct mtk_cam_dev *mtk_cam_dev = vb2_get_drv_priv(vb->vb2_queue);
> +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
> +	struct device *dev = &mtk_cam_dev->pdev->dev;
> +	struct mtk_cam_dev_buffer *buf;
> +
> +	buf = mtk_cam_vb2_buf_to_dev_buf(vb);

This can be folded into the declaration.

> +
> +	dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> +		__func__,
> +		node->id,
> +		buf->vbb.request_fd,
> +		buf->vbb.vb2_buf.index);
> +
> +	/* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> +	if (vb->vb2_queue->uses_requests)
> +		return;

I'd suggest removing non-request support from this driver. Even if we end up
with a need to provide compatibility for non-request mode, then it should be
built on top of the requests mode, so that the driver itself doesn't have to
deal with two modes.

> +
> +	/* Added the buffer into the tracking list */
> +	spin_lock(&node->slock);
> +	list_add_tail(&buf->list, &node->pending_list);
> +	spin_unlock(&node->slock);
> +
> +	mtk_isp_enqueue(dev, node->desc.dma_port, buf);
> +}
> +
> +static int mtk_cam_vb2_buf_init(struct vb2_buffer *vb)
> +{
> +	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vb->vb2_queue);
> +	struct device *smem_dev = cam_dev->smem_dev;
> +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
> +	struct mtk_cam_dev_buffer *buf;
> +
> +	buf = mtk_cam_vb2_buf_to_dev_buf(vb);
> +	buf->node_id = node->id;
> +	buf->daddr = vb2_dma_contig_plane_dma_addr(&buf->vbb.vb2_buf, 0);
> +	buf->scp_addr = 0;

Just a reminder that this will have to be reworked according to my comments
for the memory allocation patch.

> +
> +	/* scp address is only valid for meta input buffer */
> +	if (node->desc.smem_alloc)
> +		buf->scp_addr = mtk_cam_smem_iova_to_scp_addr(smem_dev,
> +							      buf->daddr);
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vb2_buf_prepare(struct vb2_buffer *vb)
> +{
> +	struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb);
> +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
> +	const struct v4l2_format *fmt = &node->vdev_fmt;
> +	unsigned int size;
> +
> +	if (vb->vb2_queue->type == V4L2_BUF_TYPE_META_OUTPUT ||
> +	    vb->vb2_queue->type == V4L2_BUF_TYPE_META_CAPTURE)
> +		size = fmt->fmt.meta.buffersize;
> +	else
> +		size = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
> +
> +	if (vb2_plane_size(vb, 0) < size)
> +		return -EINVAL;

For OUTPUT buffers we need to check if vb2_get_plane_payload() == size.
Otherwise we could get not enough or invalid data.

> +
> +	v4l2_buf->field = V4L2_FIELD_NONE;
> +	vb2_set_plane_payload(vb, 0, size);

This shouldn't be called on OUTPUT buffers.

> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vb2_queue_setup(struct vb2_queue *vq,
> +				   unsigned int *num_buffers,
> +				   unsigned int *num_planes,
> +				   unsigned int sizes[],
> +				   struct device *alloc_devs[])
> +{
> +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> +	unsigned int max_buffer_count = node->desc.max_buf_count;
> +	const struct v4l2_format *fmt = &node->vdev_fmt;
> +	unsigned int size;
> +
> +	/* Check the limitation of buffer size */
> +	if (max_buffer_count)
> +		*num_buffers = clamp_val(*num_buffers, 1, max_buffer_count);
> +
> +	if (vq->type == V4L2_BUF_TYPE_META_OUTPUT ||
> +	    vq->type == V4L2_BUF_TYPE_META_CAPTURE)
> +		size = fmt->fmt.meta.buffersize;
> +	else
> +		size = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
> +
> +	/* Add for q.create_bufs with fmt.g_sizeimage(p) / 2 test */
> +	if (*num_planes) {

We should also verify that *num_planes == 1, as we don't support more
planes in this driver.

> +		if (sizes[0] < size)
> +			return -EINVAL;
> +	} else {
> +		*num_planes = 1;
> +		sizes[0] = size;
> +	}
> +
> +	return 0;
> +}
> +
> +static void mtk_cam_vb2_return_all_buffers(struct mtk_cam_dev *cam_dev,
> +					   struct mtk_cam_video_device *node,
> +					   enum vb2_buffer_state state)
> +{
> +	struct mtk_cam_dev_buffer *b, *b0;
> +	struct mtk_cam_dev_request *req, *req0;
> +	struct media_request_object *obj, *obj0;
> +	struct vb2_buffer *vb;
> +
> +	dev_dbg(&cam_dev->pdev->dev, "%s: node:%s", __func__, node->vdev.name);
> +
> +	/* Return all buffers */
> +	spin_lock(&node->slock);
> +	list_for_each_entry_safe(b, b0, &node->pending_list, list) {

nit: One would normally call the second argument "prev", or "b_prev".

> +		vb = &b->vbb.vb2_buf;
> +		if (vb->state == VB2_BUF_STATE_ACTIVE)

We shouldn't need to check the buffer state.

> +			vb2_buffer_done(vb, state);
> +		list_del(&b->list);
> +	}
> +	spin_unlock(&node->slock);
> +
> +	spin_lock(&cam_dev->req_lock);
> +	list_for_each_entry_safe(req, req0, &cam_dev->req_list, list) {

nit: Ditto.

> +		list_for_each_entry_safe(obj, obj0, &req->req.objects, list) {

Need to check if the object is a buffer.

> +			vb = container_of(obj, struct vb2_buffer, req_obj);
> +			if (vb->state == VB2_BUF_STATE_ACTIVE)

vb->state shouldn't be accessed directly from the drivers.

Generally, the need to check the state here would suggest that there is
something wrong with how the driver manages the requests. The list that is
being iterated here shouldn't contain any requests that have buffers that
aren't active. That will be achieved if my comments for the request handling
in the DIP driver are applied to this driver as well.

> +				vb2_buffer_done(vb, state);
> +		}
> +		list_del(&req->list);
> +	}
> +	spin_unlock(&cam_dev->req_lock);
> +
> +	if (node->vbq.uses_requests)
> +		mtk_isp_req_flush_buffers(&cam_dev->pdev->dev);
> +}
> +
> +static int mtk_cam_vb2_start_streaming(struct vb2_queue *vq,
> +				       unsigned int count)
> +{
> +	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
> +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> +	struct device *dev = &cam_dev->pdev->dev;
> +	unsigned int node_count = cam_dev->subdev.entity.use_count;
> +	int ret;
> +
> +	if (!node->enabled) {

How is this synchronized with mtk_cam_media_link_setup()?

> +		dev_err(dev, "Node:%d is not enable\n", node->id);
> +		ret = -ENOLINK;
> +		goto fail_no_link;
> +	}
> +
> +	dev_dbg(dev, "%s: count info:%d:%d", __func__,
> +		atomic_read(&cam_dev->streamed_node_count), node_count);
> +
> +	if (atomic_inc_return(&cam_dev->streamed_node_count) < node_count)
> +		return 0;

How do we guarantee that cam_dev->subdev.entity.use_count doesn't change
between calls to this function on different video nodes?

> +
> +	/* Start streaming of the whole pipeline now */
> +	ret = media_pipeline_start(&node->vdev.entity, &cam_dev->pipeline);
> +	if (ret) {
> +		dev_err(dev, "%s: Node:%d failed\n", __func__, node->id);
> +		goto fail_start_pipeline;
> +	}
> +

Related to the above comment: If we start the media pipeline when we start
streaming on the first node, we would naturally prevent the link
configuration changes until the last node stops streaming (as long as the
link is not DYNAMIC). Note that it would only mark the entities as
streaming, but it wouldn't call their s_stream, which I believe is exactly
what we would need to solve the problem above.

> +	/* Stream on sub-devices node */
> +	ret = v4l2_subdev_call(&cam_dev->subdev, video, s_stream, 1);
> +	if (ret) {
> +		dev_err(dev, "Node:%d s_stream on failed:%d\n", node->id, ret);
> +		goto fail_stream_on;
> +	}
> +
> +	return 0;
> +
> +fail_stream_on:
> +	media_pipeline_stop(&node->vdev.entity);
> +fail_start_pipeline:
> +	atomic_dec(&cam_dev->streamed_node_count);
> +fail_no_link:
> +	mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_QUEUED);
> +
> +	return ret;
> +}
> +
> +static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
> +{
> +	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
> +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> +	struct device *dev = &cam_dev->pdev->dev;
> +
> +	if (!node->enabled)
> +		return;

It shouldn't be possible for this to happen, because nobody could have
called start_streaming on a disabled node.

> +
> +	mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_ERROR);

Shouldn't we stop streaming first, so that the hardware operation is
cancelled and any buffers owned by the hardware are released?

> +
> +	dev_dbg(dev, "%s: count info:%d", __func__,
> +		cam_dev->subdev.entity.stream_count);
> +
> +	/* Check the first node to stream-off */
> +	if (!cam_dev->subdev.entity.stream_count)
> +		return;
> +
> +	media_pipeline_stop(&node->vdev.entity);
> +
> +	if (v4l2_subdev_call(&cam_dev->subdev, video, s_stream, 0))
> +		dev_err(dev, "failed to stop streaming\n");
> +}
> +
> +static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
> +{
> +	struct mtk_cam_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
> +
> +	v4l2_ctrl_request_complete(vb->req_obj.req,
> +				   dev->v4l2_dev.ctrl_handler);

This would end up being called multiple times, once for each video node.
Instead, this should be called explicitly by the driver when it completed
the request - perhaps in the frame completion handler?

With that, we probably wouldn't even need this callback.

> +}
> +
> +static int mtk_cam_vidioc_querycap(struct file *file, void *fh,
> +				   struct v4l2_capability *cap)
> +{
> +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> +
> +	strscpy(cap->driver, MTK_CAM_DEV_P1_NAME, sizeof(cap->driver));
> +	strscpy(cap->card, MTK_CAM_DEV_P1_NAME, sizeof(cap->card));

We could just use dev_driver_name(cam_dev->dev) for both.

> +	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
> +		 dev_name(cam_dev->media_dev.dev));

We should just store dev in cam_dev.

> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_enum_fmt(struct file *file, void *fh,
> +				   struct v4l2_fmtdesc *f)
> +{
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +
> +	if (f->index >= node->desc.num_fmts)
> +		return -EINVAL;
> +
> +	f->pixelformat = node->desc.fmts[f->index].fmt.pix_mp.pixelformat;

Is the set of formats available always the same regardless of the sensor
format?

> +	f->flags = 0;

We need f->description too.

> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_g_fmt(struct file *file, void *fh,
> +				struct v4l2_format *f)
> +{
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +
> +	if (!node->desc.num_fmts)
> +		return -EINVAL;

When would that condition happen?

> +
> +	f->fmt = node->vdev_fmt.fmt;
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_try_fmt(struct file *file, void *fh,
> +				  struct v4l2_format *in_fmt)
> +{
> +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +	const struct v4l2_format *dev_fmt;
> +	__u32  width, height;

Don't use __ types in implementation, they are here for UAPI purposes. There
is u32, which you could use instead, but for width and height you don't need
explicit size, so unsigned int should be good.

> +
> +	dev_dbg(&cam_dev->pdev->dev, "%s: fmt:%c%c%c%c, w*h:%u*%u\n",
> +		__func__,
> +		(in_fmt->fmt.pix_mp.pixelformat & 0xFF),
> +		(in_fmt->fmt.pix_mp.pixelformat >> 8) & 0xFF,
> +		(in_fmt->fmt.pix_mp.pixelformat >> 16) & 0xFF,
> +		(in_fmt->fmt.pix_mp.pixelformat >> 24) & 0xFF,
> +		in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height);
> +
> +	width = in_fmt->fmt.pix_mp.width;
> +	height = in_fmt->fmt.pix_mp.height;
> +
> +	dev_fmt = mtk_cam_dev_find_fmt(&node->desc,
> +				       in_fmt->fmt.pix_mp.pixelformat);
> +	if (dev_fmt) {
> +		mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
> +					&in_fmt->fmt.pix_mp,
> +					&dev_fmt->fmt.pix_mp,
> +					node->id);
> +	} else {
> +		mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
> +					     &node->desc, in_fmt);

We shouldn't just load a default format. This function should validate all
the fields one by one and adjust them to something appropriate.

> +	}

CodingStyle: No braces if both if and else bodies have only 1 statement
each.

> +	in_fmt->fmt.pix_mp.width = clamp_t(u32,
> +					   width,
> +					   CAM_MIN_WIDTH,
> +					   in_fmt->fmt.pix_mp.width);

Shouldn't we clamp this with some maximum value too?

> +	in_fmt->fmt.pix_mp.height = clamp_t(u32,
> +					    height,
> +					    CAM_MIN_HEIGHT,
> +					    in_fmt->fmt.pix_mp.height);

Ditto.

> +	mtk_cam_dev_cal_mplane_fmt(&cam_dev->pdev->dev,
> +				   &in_fmt->fmt.pix_mp, node->id);
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_s_fmt(struct file *file, void *fh,
> +				struct v4l2_format *f)
> +{
> +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +
> +	if (cam_dev->streaming)
> +		return -EBUSY;

I think this should rather be something like vb2_queue_is_busy(), which
would prevent format changes if buffers are allocated.

> +
> +	/* Get the valid format */
> +	mtk_cam_vidioc_try_fmt(file, fh, f);
> +
> +	/* Configure to video device */
> +	mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
> +				&node->vdev_fmt.fmt.pix_mp,
> +				&f->fmt.pix_mp,
> +				node->id);
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_enum_input(struct file *file, void *fh,
> +				     struct v4l2_input *input)
> +{
> +	if (input->index)
> +		return -EINVAL;
> +
> +	strscpy(input->name, "camera", sizeof(input->name));
> +	input->type = V4L2_INPUT_TYPE_CAMERA;
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_g_input(struct file *file, void *fh,
> +				  unsigned int *input)
> +{
> +	*input = 0;
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_s_input(struct file *file,
> +				  void *fh, unsigned int input)
> +{
> +	return input == 0 ? 0 : -EINVAL;
> +}
> +
> +static int mtk_cam_vidioc_enum_framesizes(struct file *filp, void *priv,
> +					  struct v4l2_frmsizeenum *sizes)
> +{
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(filp);
> +	const struct v4l2_format *dev_fmt;
> +
> +	dev_fmt = mtk_cam_dev_find_fmt(&node->desc, sizes->pixel_format);
> +	if (!dev_fmt || sizes->index)
> +		return -EINVAL;
> +
> +	sizes->type = node->desc.frmsizes->type;
> +	memcpy(&sizes->stepwise, &node->desc.frmsizes->stepwise,
> +	       sizeof(sizes->stepwise));
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_meta_enum_fmt(struct file *file, void *fh,
> +					struct v4l2_fmtdesc *f)
> +{
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +
> +	if (f->index)
> +		return -EINVAL;
> +
> +	strscpy(f->description, node->desc.description,
> +		sizeof(node->desc.description));
> +	f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
> +	f->flags = 0;
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_vidioc_g_meta_fmt(struct file *file, void *fh,
> +				     struct v4l2_format *f)
> +{
> +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> +
> +	f->fmt.meta.dataformat = node->vdev_fmt.fmt.meta.dataformat;
> +	f->fmt.meta.buffersize = node->vdev_fmt.fmt.meta.buffersize;
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_subdev_core_ops mtk_cam_subdev_core_ops = {
> +	.subscribe_event = mtk_cam_sd_subscribe_event,
> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> +	.s_power = mtk_cam_sd_s_power,
> +};
> +
> +static const struct v4l2_subdev_video_ops mtk_cam_subdev_video_ops = {
> +	.s_stream =  mtk_cam_sd_s_stream,
> +};
> +
> +static const struct v4l2_subdev_ops mtk_cam_subdev_ops = {
> +	.core = &mtk_cam_subdev_core_ops,
> +	.video = &mtk_cam_subdev_video_ops,
> +};
> +
> +static const struct media_entity_operations mtk_cam_media_ops = {

nit: mtk_cam_media_entity_ops?

> +	.link_setup = mtk_cam_media_link_setup,
> +	.link_validate = v4l2_subdev_link_validate,
> +};
> +
> +static const struct vb2_ops mtk_cam_vb2_ops = {
> +	.queue_setup = mtk_cam_vb2_queue_setup,
> +	.wait_prepare = vb2_ops_wait_prepare,
> +	.wait_finish = vb2_ops_wait_finish,
> +	.buf_init = mtk_cam_vb2_buf_init,
> +	.buf_prepare = mtk_cam_vb2_buf_prepare,
> +	.start_streaming = mtk_cam_vb2_start_streaming,
> +	.stop_streaming = mtk_cam_vb2_stop_streaming,
> +	.buf_queue = mtk_cam_vb2_buf_queue,
> +	.buf_request_complete = mtk_cam_vb2_buf_request_complete,
> +};
> +
> +static const struct v4l2_file_operations mtk_cam_v4l2_fops = {
> +	.unlocked_ioctl = video_ioctl2,
> +	.open = mtk_cam_isp_open,
> +	.release = mtk_cam_isp_release,
> +	.poll = vb2_fop_poll,
> +	.mmap = vb2_fop_mmap,
> +#ifdef CONFIG_COMPAT
> +	.compat_ioctl32 = v4l2_compat_ioctl32,
> +#endif
> +};
> +
> +static const struct media_device_ops mtk_cam_media_req_ops = {

nit: Those are media ops, so perhaps just mtk_cam_media_ops?

> +	.link_notify = v4l2_pipeline_link_notify,
> +	.req_alloc = mtk_cam_req_alloc,
> +	.req_free = mtk_cam_req_free,
> +	.req_validate = vb2_request_validate,
> +	.req_queue = mtk_cam_req_queue,
> +};
> +
> +static int mtk_cam_media_register(struct device *dev,
> +				  struct media_device *media_dev)
> +{
> +	media_dev->dev = dev;
> +	strscpy(media_dev->model, MTK_CAM_DEV_P1_NAME,

Could we replace any use of this macro with dev_driver_string(dev) and then
delete the macro? The less name strings in the driver the better, as there
is less change for confusing the userspace if few different names are used
at the same time.

> +		sizeof(media_dev->model));
> +	snprintf(media_dev->bus_info, sizeof(media_dev->bus_info),
> +		 "platform:%s", dev_name(dev));
> +	media_dev->hw_revision = 0;
> +	media_device_init(media_dev);
> +	media_dev->ops = &mtk_cam_media_req_ops;
> +
> +	return media_device_register(media_dev);
> +}
> +
> +static int mtk_cam_video_register_device(struct mtk_cam_dev *cam_dev, u32 i)
> +{
> +	struct device *dev = &cam_dev->pdev->dev;
> +	struct mtk_cam_video_device *node = &cam_dev->vdev_nodes[i];

Would it make sense to pass node as an argument to this function instead of
(or in addition to) i?

> +	struct video_device *vdev = &node->vdev;
> +	struct vb2_queue *vbq = &node->vbq;
> +	u32 output = !cam_dev->vdev_nodes[i].desc.capture;

Why not call it capture instead and avoid the inversion?

> +	u32 link_flags = cam_dev->vdev_nodes[i].desc.link_flags;
> +	int ret;
> +
> +	cam_dev->subdev_pads[i].flags = output ?
> +		MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
> +
> +	/* Initialize media entities */
> +	ret = media_entity_pads_init(&vdev->entity, 1, &node->vdev_pad);
> +	if (ret) {
> +		dev_err(dev, "failed initialize media pad:%d\n", ret);
> +		return ret;
> +	}
> +	node->enabled = false;

Are all the nodes optional? If there is any required node, it should be
always enabled and have the MEDIA_LNK_FL_IMMUTABLE flag set.

> +	node->id = i;
> +	node->vdev_pad.flags = cam_dev->subdev_pads[i].flags;

Hmm, shouldn't the subdev pads have opposite directions (sink vs source)?

> +	mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
> +				     &node->desc,
> +				     &node->vdev_fmt);
> +
> +	/* Initialize vbq */
> +	vbq->type = node->vdev_fmt.type;
> +	if (vbq->type == V4L2_BUF_TYPE_META_OUTPUT)
> +		vbq->io_modes = VB2_MMAP;
> +	else
> +		vbq->io_modes = VB2_MMAP | VB2_DMABUF;
> +
> +	if (node->desc.smem_alloc) {
> +		vbq->bidirectional = 1;
> +		vbq->dev = cam_dev->smem_dev;
> +	} else {
> +		vbq->dev = &cam_dev->pdev->dev;
> +	}
> +
> +	if (vbq->type == V4L2_BUF_TYPE_META_CAPTURE)
> +		vdev->entity.function =
> +			MEDIA_ENT_F_PROC_VIDEO_STATISTICS;

This is a video node, so it's just a DMA, not a processing entity. I believe
all the entities corresponding to video nodes should use MEDIA_ENT_F_IO_V4L.

> +	vbq->ops = &mtk_cam_vb2_ops;
> +	vbq->mem_ops = &vb2_dma_contig_memops;
> +	vbq->buf_struct_size = sizeof(struct mtk_cam_dev_buffer);
> +	vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +	vbq->min_buffers_needed = 0;	/* Can streamon w/o buffers */
> +	/* Put the process hub sub device in the vb2 private data */

What is "process hub" and what "sub device" is this about?

> +	vbq->drv_priv = cam_dev;
> +	vbq->lock = &node->lock;
> +	vbq->supports_requests = true;
> +
> +	ret = vb2_queue_init(vbq);
> +	if (ret) {
> +		dev_err(dev, "failed to init. vb2 queue:%d\n", ret);
> +		goto fail_vb2_queue;
> +	}
> +
> +	/* Initialize vdev */
> +	snprintf(vdev->name, sizeof(vdev->name), "%s %s",
> +		 MTK_CAM_DEV_P1_NAME, node->desc.name);
> +	/* set cap/type/ioctl_ops of the video device */
> +	vdev->device_caps = node->desc.cap | V4L2_CAP_STREAMING;
> +	vdev->ioctl_ops = node->desc.ioctl_ops;
> +	vdev->fops = &mtk_cam_v4l2_fops;
> +	vdev->release = video_device_release_empty;
> +	vdev->lock = &node->lock;
> +	vdev->v4l2_dev = &cam_dev->v4l2_dev;
> +	vdev->queue = &node->vbq;
> +	vdev->vfl_dir = output ? VFL_DIR_TX : VFL_DIR_RX;
> +	vdev->entity.ops = NULL;
> +	/* Enable private control for image video devices */
> +	if (node->desc.image) {
> +		mtk_cam_ctrl_init(cam_dev, &node->ctrl_handler);
> +		vdev->ctrl_handler = &node->ctrl_handler;
> +	}
> +	video_set_drvdata(vdev, cam_dev);
> +	dev_dbg(dev, "register vdev:%d:%s\n", i, vdev->name);
> +
> +	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
> +	if (ret) {
> +		dev_err(dev, "failed to register vde:%d\n", ret);
> +		goto fail_vdev;
> +	}
> +
> +	/* Create link between video node and the subdev pad */
> +	if (output) {
> +		ret = media_create_pad_link(&vdev->entity, 0,
> +					    &cam_dev->subdev.entity,
> +					    i, link_flags);
> +	} else {
> +		ret = media_create_pad_link(&cam_dev->subdev.entity,
> +					    i, &vdev->entity, 0,
> +					    link_flags);
> +	}
> +	if (ret)
> +		goto fail_link;
> +
> +	/* Initialize miscellaneous variables */
> +	mutex_init(&node->lock);
> +	spin_lock_init(&node->slock);
> +	INIT_LIST_HEAD(&node->pending_list);

This should be all initialized before registering the video device.
Otherwise userspace could open the device before these are initialized.

> +
> +	return 0;
> +
> +fail_link:
> +	video_unregister_device(vdev);
> +fail_vdev:
> +	vb2_queue_release(vbq);
> +fail_vb2_queue:
> +	media_entity_cleanup(&vdev->entity);
> +
> +	return ret;
> +}
> +
> +static int mtk_cam_mem2mem2_v4l2_register(struct mtk_cam_dev *cam_dev)

This function doesn't have anything to do with mem2mem. How about
mtk_cam_v4l2_register()?

Perhaps it would make sense to move any media related code into into
mtk_cam_media_register(), keep only V4L2 related code here and have the
caller call the former first and then this one, rather than having such deep
sequence of nested calls, which makes the driver harder to read.

> +{
> +	struct device *dev = &cam_dev->pdev->dev;

How about just storing dev, instead of pdev in the struct? Also, calling the
argument "cam", would make it as short as cam->dev.

> +	/* Total pad numbers is video devices + one seninf pad */
> +	unsigned int num_subdev_pads = MTK_CAM_CIO_PAD_SINK + 1;
> +	unsigned int i;
> +	int ret;
> +
> +	ret = mtk_cam_media_register(dev,
> +				     &cam_dev->media_dev);
> +	if (ret) {
> +		dev_err(dev, "failed to register media device:%d\n", ret);
> +		return ret;
> +	}
> +	dev_info(dev, "Register media device: %s, 0x%pK",
> +		 MTK_CAM_DEV_P1_NAME, cam_dev->media_dev);

An info message should be useful to the user in some way. Printing kernel
pointers isn't useful. Something like "registered media0" could be useful to
let the user know which media device is associated with this driver if there
is more than one in the system.

> +
> +	/* Set up v4l2 device */
> +	cam_dev->v4l2_dev.mdev = &cam_dev->media_dev;
> +	ret = v4l2_device_register(dev, &cam_dev->v4l2_dev);
> +	if (ret) {
> +		dev_err(dev, "failed to register V4L2 device:%d\n", ret);
> +		goto fail_v4l2_dev;

Please call the labels after the cleanup step that needs to be done. It
makes it easier to spot any ordering errors.

> +	}
> +	dev_info(dev, "Register v4l2 device: 0x%pK", cam_dev->v4l2_dev);

Same as above.

> +
> +	/* Initialize subdev media entity */
> +	cam_dev->subdev_pads = devm_kcalloc(dev, num_subdev_pads,
> +					    sizeof(*cam_dev->subdev_pads),
> +					    GFP_KERNEL);
> +	if (!cam_dev->subdev_pads) {
> +		ret = -ENOMEM;
> +		goto fail_subdev_pads;
> +	}
> +
> +	ret = media_entity_pads_init(&cam_dev->subdev.entity,
> +				     num_subdev_pads,
> +				     cam_dev->subdev_pads);
> +	if (ret) {
> +		dev_err(dev, "failed initialize media pads:%d:\n", ret);

Stray ":" at the end of the message.

> +		goto fail_subdev_pads;
> +	}
> +
> +	/* Initialize all pads with MEDIA_PAD_FL_SOURCE */
> +	for (i = 0; i < num_subdev_pads; i++)
> +		cam_dev->subdev_pads[i].flags = MEDIA_PAD_FL_SOURCE;
> +
> +	/* Customize the last one pad as CIO sink pad. */
> +	cam_dev->subdev_pads[MTK_CAM_CIO_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
> +
> +	/* Initialize subdev */
> +	v4l2_subdev_init(&cam_dev->subdev, &mtk_cam_subdev_ops);
> +	cam_dev->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
> +	cam_dev->subdev.entity.ops = &mtk_cam_media_ops;
> +	cam_dev->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
> +				V4L2_SUBDEV_FL_HAS_EVENTS;
> +	snprintf(cam_dev->subdev.name, sizeof(cam_dev->subdev.name),
> +		 "%s", MTK_CAM_DEV_P1_NAME);
> +	v4l2_set_subdevdata(&cam_dev->subdev, cam_dev);
> +
> +	ret = v4l2_device_register_subdev(&cam_dev->v4l2_dev, &cam_dev->subdev);
> +	if (ret) {
> +		dev_err(dev, "failed initialize subdev:%d\n", ret);
> +		goto fail_subdev;
> +	}
> +	dev_info(dev, "register subdev: %s\n", cam_dev->subdev.name);
> +
> +	/* Create video nodes and links */
> +	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++) {
> +		ret = mtk_cam_video_register_device(cam_dev, i);
> +		if (ret)
> +			goto fail_video_register;
> +	}
> +
> +	vb2_dma_contig_set_max_seg_size(dev, DMA_BIT_MASK(32));
> +
> +	return 0;
> +
> +fail_video_register:
> +	i--;

This could be moved into the for clause, as the initialization statement.

> +	for (; i >= 0; i--) {

i is unsigned. Did this compile without warnings?

> +		video_unregister_device(&cam_dev->vdev_nodes[i].vdev);
> +		media_entity_cleanup(&cam_dev->vdev_nodes[i].vdev.entity);
> +		mutex_destroy(&cam_dev->vdev_nodes[i].lock);

Should we move this into mtk_cam_video_unregister_device() to be consistent
with registration?

> +	}
> +fail_subdev:
> +	media_entity_cleanup(&cam_dev->subdev.entity);
> +fail_subdev_pads:
> +	v4l2_device_unregister(&cam_dev->v4l2_dev);
> +fail_v4l2_dev:
> +	dev_err(dev, "fail_v4l2_dev mdev: 0x%pK:%d", &cam_dev->media_dev, ret);

Please print errors only where they actually happen, not at the cleanup.

> +	media_device_unregister(&cam_dev->media_dev);
> +	media_device_cleanup(&cam_dev->media_dev);
> +
> +	return ret;
> +}
> +
> +static int mtk_cam_v4l2_unregister(struct mtk_cam_dev *cam_dev)
> +{
> +	unsigned int i;
> +	struct mtk_cam_video_device *dev;

nit: Move the declaration inside the for loop, since the variable is only
used there.

> +
> +	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++) {
> +		dev = &cam_dev->vdev_nodes[i];
> +		video_unregister_device(&dev->vdev);
> +		media_entity_cleanup(&dev->vdev.entity);
> +		mutex_destroy(&dev->lock);
> +		if (dev->desc.image)
> +			v4l2_ctrl_handler_free(&dev->ctrl_handler);
> +	}
> +
> +	vb2_dma_contig_clear_max_seg_size(&cam_dev->pdev->dev);
> +
> +	v4l2_device_unregister_subdev(&cam_dev->subdev);
> +	media_entity_cleanup(&cam_dev->subdev.entity);
> +	kfree(cam_dev->subdev_pads);

This was allocated using devm_kcalloc(), so no need to free it explicitly.

> +
> +	v4l2_device_unregister(&cam_dev->v4l2_dev);
> +	media_device_unregister(&cam_dev->media_dev);
> +	media_device_cleanup(&cam_dev->media_dev);
> +
> +	return 0;
> +}
> +
> +static int mtk_cam_dev_complete(struct v4l2_async_notifier *notifier)
> +{
> +	struct mtk_cam_dev *cam_dev =
> +		container_of(notifier, struct mtk_cam_dev, notifier);
> +	struct device *dev = &cam_dev->pdev->dev;
> +	int ret;
> +
> +	ret = media_create_pad_link(&cam_dev->seninf->entity,
> +				    MTK_CAM_CIO_PAD_SRC,
> +				    &cam_dev->subdev.entity,
> +				    MTK_CAM_CIO_PAD_SINK,
> +				    0);
> +	if (ret) {
> +		dev_err(dev, "fail to create pad link %s %s err:%d\n",
> +			cam_dev->seninf->entity.name,
> +			cam_dev->subdev.entity.name,
> +			ret);
> +		return ret;
> +	}
> +
> +	dev_info(dev, "Complete the v4l2 registration\n");

dev_dbg()

> +
> +	ret = v4l2_device_register_subdev_nodes(&cam_dev->v4l2_dev);
> +	if (ret) {
> +		dev_err(dev, "failed initialize subdev nodes:%d\n", ret);
> +		return ret;
> +	}
> +
> +	return ret;
> +}

Why not just put the contents of this function inside 
mtk_cam_dev_notifier_complete()?

> +
> +static int mtk_cam_dev_notifier_bound(struct v4l2_async_notifier *notifier,
> +				      struct v4l2_subdev *sd,
> +				      struct v4l2_async_subdev *asd)
> +{
> +	struct mtk_cam_dev *cam_dev =
> +		container_of(notifier, struct mtk_cam_dev, notifier);
> +

Should we somehow check that the entity we got is seninf indeed and there
was no mistake in DT?

> +	cam_dev->seninf = sd;
> +	dev_info(&cam_dev->pdev->dev, "%s is bounded\n", sd->entity.name);

bound

Also please make this dev_dbg().

> +	return 0;
> +}
> +
> +static void mtk_cam_dev_notifier_unbind(struct v4l2_async_notifier *notifier,
> +					struct v4l2_subdev *sd,
> +					struct v4l2_async_subdev *asd)
> +{
> +	struct mtk_cam_dev *cam_dev =
> +		container_of(notifier, struct mtk_cam_dev, notifier);
> +
> +	cam_dev->seninf = NULL;
> +	dev_dbg(&cam_dev->pdev->dev, "%s is unbounded\n", sd->entity.name);

unbound

> +}
> +
> +static int mtk_cam_dev_notifier_complete(struct v4l2_async_notifier *notifier)
> +{
> +	return mtk_cam_dev_complete(notifier);
> +}
> +
> +static const struct v4l2_async_notifier_operations mtk_cam_async_ops = {
> +	.bound = mtk_cam_dev_notifier_bound,
> +	.unbind = mtk_cam_dev_notifier_unbind,
> +	.complete = mtk_cam_dev_notifier_complete,
> +};
> +
> +static int mtk_cam_v4l2_async_register(struct mtk_cam_dev *cam_dev)
> +{
> +	struct device *dev = &cam_dev->pdev->dev;
> +	int ret;
> +
> +	ret = v4l2_async_notifier_parse_fwnode_endpoints(dev,
> +		&cam_dev->notifier, sizeof(struct v4l2_async_subdev),
> +		NULL);
> +	if (ret)
> +		return ret;
> +
> +	if (!cam_dev->notifier.num_subdevs)
> +		return -ENODEV;

Could we print some error messages for the 2 cases above?

> +
> +	cam_dev->notifier.ops = &mtk_cam_async_ops;
> +	dev_info(&cam_dev->pdev->dev, "mtk_cam v4l2_async_notifier_register\n");

dev_dbg()

> +	ret = v4l2_async_notifier_register(&cam_dev->v4l2_dev,
> +					   &cam_dev->notifier);
> +	if (ret) {
> +		dev_err(&cam_dev->pdev->dev,
> +			"failed to register async notifier : %d\n", ret);
> +		v4l2_async_notifier_cleanup(&cam_dev->notifier);
> +	}
> +
> +	return ret;
> +}
> +
> +static void mtk_cam_v4l2_async_unregister(struct mtk_cam_dev *cam_dev)
> +{
> +	v4l2_async_notifier_unregister(&cam_dev->notifier);
> +	v4l2_async_notifier_cleanup(&cam_dev->notifier);
> +}
> +
> +static const struct v4l2_ioctl_ops mtk_cam_v4l2_vcap_ioctl_ops = {
> +	.vidioc_querycap = mtk_cam_vidioc_querycap,
> +	.vidioc_enum_framesizes = mtk_cam_vidioc_enum_framesizes,
> +	.vidioc_enum_fmt_vid_cap_mplane = mtk_cam_vidioc_enum_fmt,
> +	.vidioc_g_fmt_vid_cap_mplane = mtk_cam_vidioc_g_fmt,
> +	.vidioc_s_fmt_vid_cap_mplane = mtk_cam_vidioc_s_fmt,
> +	.vidioc_try_fmt_vid_cap_mplane = mtk_cam_vidioc_try_fmt,
> +	.vidioc_enum_input = mtk_cam_vidioc_enum_input,
> +	.vidioc_g_input = mtk_cam_vidioc_g_input,
> +	.vidioc_s_input = mtk_cam_vidioc_s_input,

I don't think we need vidioc_*_input. At least the current implementation in
this patch doesn't seem to do anything useful.

> +	/* buffer queue management */

Drop this comment, as it's obvious that the callbacks with "buf" in the name
are related to buffers, there are some non-buffer callbacks below too (e.g.
vidioc_subscribe_event) and also the other ops structs below don't have such
comment.

> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> +	.vidioc_streamon = vb2_ioctl_streamon,
> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> +};
> +
> +static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_cap_ioctl_ops = {
> +	.vidioc_querycap = mtk_cam_vidioc_querycap,
> +	.vidioc_enum_fmt_meta_cap = mtk_cam_vidioc_meta_enum_fmt,
> +	.vidioc_g_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
> +	.vidioc_s_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
> +	.vidioc_try_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> +	.vidioc_streamon = vb2_ioctl_streamon,
> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> +};
> +
> +static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_out_ioctl_ops = {
> +	.vidioc_querycap = mtk_cam_vidioc_querycap,
> +	.vidioc_enum_fmt_meta_out = mtk_cam_vidioc_meta_enum_fmt,
> +	.vidioc_g_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
> +	.vidioc_s_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
> +	.vidioc_try_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> +	.vidioc_streamon = vb2_ioctl_streamon,
> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> +};
> +
> +static const struct v4l2_format meta_fmts[] = {
> +	{
> +		.fmt.meta = {
> +			.dataformat = V4L2_META_FMT_MTISP_PARAMS,
> +			.buffersize = 128 * PAGE_SIZE,

PAGE_SIZE is a weird unit for specifying generic buffer sizes. How about
making it 512 * SZ_1K?

That said, it should normally be just sizeof(struct some_struct_used_here).

Same for the other entries below.

> +		},
> +	},
> +	{
> +		.fmt.meta = {
> +			.dataformat = V4L2_META_FMT_MTISP_3A,
> +			.buffersize = 300 * PAGE_SIZE,
> +		},
> +	},
> +	{
> +		.fmt.meta = {
> +			.dataformat = V4L2_META_FMT_MTISP_AF,
> +			.buffersize = 160 * PAGE_SIZE,
> +		},
> +	},
> +	{
> +		.fmt.meta = {
> +			.dataformat = V4L2_META_FMT_MTISP_LCS,
> +			.buffersize = 72 * PAGE_SIZE,
> +		},
> +	},
> +	{
> +		.fmt.meta = {
> +			.dataformat = V4L2_META_FMT_MTISP_LMV,
> +			.buffersize = 256,
> +		},
> +	},
> +};
> +
> +static const struct v4l2_format stream_out_fmts[] = {
> +	{
> +		.fmt.pix_mp = {
> +			.width = IMG_MAX_WIDTH,
> +			.height = IMG_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_B8,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_SRGB,
> +			.num_planes = 1,
> +		},
> +	},
> +	{
> +		.fmt.pix_mp = {
> +			.width = IMG_MAX_WIDTH,
> +			.height = IMG_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_B10,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_SRGB,
> +			.num_planes = 1,
> +		},
> +	},
> +	{
> +		.fmt.pix_mp = {
> +			.width = IMG_MAX_WIDTH,
> +			.height = IMG_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_B12,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_SRGB,
> +			.num_planes = 1,
> +		},
> +	},
> +	{
> +		.fmt.pix_mp = {
> +			.width = IMG_MAX_WIDTH,
> +			.height = IMG_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_B14,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_SRGB,
> +			.num_planes = 1,
> +		},
> +	},
> +};
> +
> +static const struct v4l2_format bin_out_fmts[] = {
> +	{
> +		.fmt.pix_mp = {
> +			.width = RRZ_MAX_WIDTH,
> +			.height = RRZ_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_F8,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_RAW,
> +			.num_planes = 1,
> +		},
> +	},
> +	{
> +		.fmt.pix_mp = {
> +			.width = RRZ_MAX_WIDTH,
> +			.height = RRZ_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_F10,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_RAW,
> +			.num_planes = 1,
> +		},
> +	},
> +	{
> +		.fmt.pix_mp = {
> +			.width = RRZ_MAX_WIDTH,
> +			.height = RRZ_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_F12,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_RAW,
> +			.num_planes = 1,
> +		},
> +	},
> +	{
> +		.fmt.pix_mp = {
> +			.width = RRZ_MAX_WIDTH,
> +			.height = RRZ_MAX_HEIGHT,
> +			.pixelformat = V4L2_PIX_FMT_MTISP_F14,
> +			.field = V4L2_FIELD_NONE,
> +			.colorspace = V4L2_COLORSPACE_RAW,
> +			.num_planes = 1,
> +		},
> +	},
> +};
> +
> +static const struct v4l2_frmsizeenum img_frm_size_nums[] = {
> +	{
> +		.index = 0,
> +		.type = V4L2_FRMSIZE_TYPE_CONTINUOUS,
> +		.stepwise = {
> +			.max_width = IMG_MAX_WIDTH,
> +			.min_width = IMG_MIN_WIDTH,
> +			.max_height = IMG_MAX_HEIGHT,
> +			.min_height = IMG_MIN_HEIGHT,
> +			.step_height = 1,
> +			.step_width = 1,
> +		},
> +	},
> +	{
> +		.index = 0,
> +		.type = V4L2_FRMSIZE_TYPE_CONTINUOUS,
> +		.stepwise = {
> +			.max_width = RRZ_MAX_WIDTH,
> +			.min_width = RRZ_MIN_WIDTH,
> +			.max_height = RRZ_MAX_HEIGHT,
> +			.min_height = RRZ_MIN_HEIGHT,
> +			.step_height = 1,
> +			.step_width = 1,
> +		},
> +	},
> +};
> +
> +static const struct
> +mtk_cam_dev_node_desc output_queues[MTK_CAM_P1_TOTAL_OUTPUT] = {
> +	{
> +		.id = MTK_CAM_P1_META_IN_0,
> +		.name = "meta input",
> +		.description = "ISP tuning parameters",
> +		.cap = V4L2_CAP_META_OUTPUT,
> +		.buf_type = V4L2_BUF_TYPE_META_OUTPUT,
> +		.link_flags = 0,
> +		.capture = false,
> +		.image = false,
> +		.smem_alloc = true,
> +		.fmts = meta_fmts,
> +		.num_fmts = ARRAY_SIZE(meta_fmts),
> +		.default_fmt_idx = 0,
> +		.max_buf_count = 10,
> +		.ioctl_ops = &mtk_cam_v4l2_meta_out_ioctl_ops,
> +	},
> +};
> +
> +static const struct
> +mtk_cam_dev_node_desc capture_queues[MTK_CAM_P1_TOTAL_CAPTURE] = {
> +	{
> +		.id = MTK_CAM_P1_MAIN_STREAM_OUT,
> +		.name = "main stream",
> +		.cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE,
> +		.buf_type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
> +		.link_flags = 0,
> +		.capture = true,
> +		.image = true,
> +		.smem_alloc = false,
> +		.dma_port = R_IMGO,
> +		.fmts = stream_out_fmts,
> +		.num_fmts = ARRAY_SIZE(stream_out_fmts),
> +		.default_fmt_idx = 1,

Why not just make it always 0 and move the default format to the beginning
of stream_out_fmts[]?

> +		.ioctl_ops = &mtk_cam_v4l2_vcap_ioctl_ops,
> +		.frmsizes = &img_frm_size_nums[0],

nit: If you only need to point to these data once, you could define them in
place, as a compound literal, like

>                 .frmsizes = &(struct v4l2_framesizeenum) {
>                         // initialize here
>                 },

> +	},
> +	{
> +		.id = MTK_CAM_P1_PACKED_BIN_OUT,
> +		.name = "packed out",
> +		.cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE,
> +		.buf_type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
> +		.link_flags = 0,
> +		.capture = true,
> +		.image = true,
> +		.smem_alloc = false,
> +		.dma_port = R_RRZO,
> +		.fmts = bin_out_fmts,
> +		.num_fmts = ARRAY_SIZE(bin_out_fmts),
> +		.default_fmt_idx = 1,
> +		.ioctl_ops = &mtk_cam_v4l2_vcap_ioctl_ops,
> +		.frmsizes = &img_frm_size_nums[1],
> +	},
> +	{
> +		.id = MTK_CAM_P1_META_OUT_0,
> +		.name = "partial meta 0",
> +		.description = "AE/AWB histogram",
> +		.cap = V4L2_CAP_META_CAPTURE,
> +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> +		.link_flags = 0,
> +		.capture = true,
> +		.image = false,
> +		.smem_alloc = false,
> +		.dma_port = R_AAO | R_FLKO | R_PSO,
> +		.fmts = meta_fmts,
> +		.num_fmts = ARRAY_SIZE(meta_fmts),
> +		.default_fmt_idx = 1,
> +		.max_buf_count = 5,
> +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> +	},
> +	{
> +		.id = MTK_CAM_P1_META_OUT_1,
> +		.name = "partial meta 1",
> +		.description = "AF histogram",
> +		.cap = V4L2_CAP_META_CAPTURE,
> +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> +		.link_flags = 0,
> +		.capture = true,
> +		.image = false,
> +		.smem_alloc = false,
> +		.dma_port = R_AFO,
> +		.fmts = meta_fmts,
> +		.num_fmts = ARRAY_SIZE(meta_fmts),
> +		.default_fmt_idx = 2,
> +		.max_buf_count = 5,
> +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> +	},
> +	{
> +		.id = MTK_CAM_P1_META_OUT_2,
> +		.name = "partial meta 2",
> +		.description = "Local contrast enhanced statistics",
> +		.cap = V4L2_CAP_META_CAPTURE,
> +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> +		.link_flags = 0,
> +		.capture = true,
> +		.image = false,
> +		.smem_alloc = false,
> +		.dma_port = R_LCSO,
> +		.fmts = meta_fmts,
> +		.num_fmts = ARRAY_SIZE(meta_fmts),
> +		.default_fmt_idx = 3,
> +		.max_buf_count = 10,
> +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> +	},
> +	{
> +		.id = MTK_CAM_P1_META_OUT_3,
> +		.name = "partial meta 3",
> +		.description = "Local motion vector histogram",
> +		.cap = V4L2_CAP_META_CAPTURE,
> +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> +		.link_flags = 0,

link_flags seems to be 0 for all nodes.

> +		.capture = true,

We already know this from .buf_type. We can check using the
V4L2_TYPE_IS_OUTPUT() macro.

> +		.image = false,
> +		.smem_alloc = false,
> +		.dma_port = R_LMVO,
> +		.fmts = meta_fmts,
> +		.num_fmts = ARRAY_SIZE(meta_fmts),

I don't think this is correct. The description suggests one specific format
(local motion vector histogram), so the queue should probably be hardwired
to that format?

> +		.default_fmt_idx = 4,
> +		.max_buf_count = 10,

Where does this number come from?

> +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> +	},
> +};
> +
> +/* The helper to configure the device context */
> +static void mtk_cam_dev_queue_setup(struct mtk_cam_dev *cam_dev)
> +{
> +	unsigned int i, node_idx;
> +
> +	node_idx = 0;
> +
> +	/* Setup the output queue */
> +	for (i = 0; i < MTK_CAM_P1_TOTAL_OUTPUT; i++)

< ARRAY_SIZE(output_queues)

> +		cam_dev->vdev_nodes[node_idx++].desc = output_queues[i];
> +
> +	/* Setup the capture queue */
> +	for (i = 0; i < MTK_CAM_P1_TOTAL_CAPTURE; i++)

< ARRAY_SIZE(capture_queues)

> +		cam_dev->vdev_nodes[node_idx++].desc = capture_queues[i];
> +}
> +
> +int mtk_cam_dev_init(struct platform_device *pdev,
> +		     struct mtk_cam_dev *cam_dev)
> +{
> +	int ret;
> +
> +	cam_dev->pdev = pdev;

Do we need this additional mtk_cam_dev struct? Couldn't we just use
mtk_isp_p1 here?

> +	mtk_cam_dev_queue_setup(cam_dev);
> +	/* v4l2 sub-device registration */
> +
> +	dev_dbg(&cam_dev->pdev->dev, "mem2mem2.name: %s\n",
> +		MTK_CAM_DEV_P1_NAME);

This debugging message doesn't seem very useful. Please remove.

> +	ret = mtk_cam_mem2mem2_v4l2_register(cam_dev);
> +	if (ret)
> +		return ret;
> +
> +	ret = mtk_cam_v4l2_async_register(cam_dev);
> +	if (ret) {
> +		mtk_cam_v4l2_unregister(cam_dev);

Please use an error path on the bottom of the function instead. With goto
labels named after the next clean-up step that needs to be performed.

> +		return ret;
> +	}
> +
> +	spin_lock_init(&cam_dev->req_lock);
> +	INIT_LIST_HEAD(&cam_dev->req_list);
> +	mutex_init(&cam_dev->lock);
> +
> +	return 0;
> +}
> +
> +int mtk_cam_dev_release(struct platform_device *pdev,

"release" is normally used for file_operations. How about "cleanup"?

> +			struct mtk_cam_dev *cam_dev)
> +{
> +	mtk_cam_v4l2_async_unregister(cam_dev);
> +	mtk_cam_v4l2_unregister(cam_dev);
> +
> +	mutex_destroy(&cam_dev->lock);
> +
> +	return 0;
> +}

I'd suggest moving any generic API implementation code (platform_device,
V4L2, VB2, Media Controller, etc.) to mtk_cam.c and then moving any low
level hardware/firmware-related code from mtk_cam.c and mtk_cam-scp.c to
mtk_cam_hw.c.

> +
diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
new file mode 100644
index 000000000000..825cdf20643a
> --- /dev/null
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
@@ -0,0 +1,173 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2018 MediaTek Inc.
> + */
> +
> +#ifndef __MTK_CAM_DEV_V4L2_H__
> +#define __MTK_CAM_DEV_V4L2_H__
> +
> +#include <linux/device.h>
> +#include <linux/types.h>
> +#include <linux/platform_device.h>
> +#include <linux/spinlock.h>
> +#include <linux/videodev2.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-core.h>
> +#include <media/videobuf2-v4l2.h>
> +
> +#define MTK_CAM_DEV_P1_NAME			"MTK-ISP-P1-V4L2"

Maybe it's not a critical thing, but generally it's a good practice to just
explicitly specific this name somewhere, e.g. struct
platform_driver::driver::name and then just refer to dev_driver_name(). It
makes it easier to make sure that the name stays the same everywhere.

> +
> +#define MTK_CAM_P1_META_IN_0			0
> +#define MTK_CAM_P1_TOTAL_OUTPUT		1

Since these are just some utility definitions, we can use enum instead of
assigning the values by hand.

> +
> +#define MTK_CAM_P1_MAIN_STREAM_OUT		1
> +#define MTK_CAM_P1_PACKED_BIN_OUT		2
> +#define MTK_CAM_P1_META_OUT_0			3
> +#define MTK_CAM_P1_META_OUT_1			4
> +#define MTK_CAM_P1_META_OUT_2			5
> +#define MTK_CAM_P1_META_OUT_3			6
> +#define MTK_CAM_P1_TOTAL_CAPTURE		6

Ditto.

> +
> +#define MTK_CAM_P1_TOTAL_NODES			7

Please just add the two totals together rather than manually specifying the
value.

> +
> +struct mtk_cam_dev_request {
> +	struct media_request	req;
> +	struct list_head	list;
> +};
> +
> +struct mtk_cam_dev_buffer {
> +	struct vb2_v4l2_buffer	vbb;
> +	struct list_head	list;
> +	/* Intenal part */
> +	dma_addr_t		daddr;
> +	dma_addr_t		scp_addr;
> +	unsigned int		node_id;
> +};

Could you add kerneldoc comments for the 2 structs?

> +
> +/*
> + * struct mtk_cam_dev_node_desc - node attributes
> + *
> + * @id:		 id of the context queue
> + * @name:	 media entity name
> + * @description: descritpion of node
> + * @cap:	 mapped to V4L2 capabilities
> + * @buf_type:	 mapped to V4L2 buffer type
> + * @dma_port:	 the dma port associated to the buffer
> + * @link_flags:	 default media link flags
> + * @smem_alloc:	 using the cam_smem_drv as alloc ctx or not
> + * @capture:	 true for capture queue (device to user)
> + *		 false for output queue (from user to device)
> + * @image:	 true for image node, false for meta node
> + * @num_fmts:	 the number of supported formats
> + * @default_fmt_idx: default format of this node
> + * @max_buf_count: maximum V4L2 buffer count
> + * @ioctl_ops:  mapped to v4l2_ioctl_ops
> + * @fmts:	supported format
> + * @frmsizes:	supported frame size number
> + *
> + */
> +struct mtk_cam_dev_node_desc {
> +	u8 id;
> +	char *name;
> +	char *description;
> +	u32 cap;
> +	u32 buf_type;
> +	u32 dma_port;
> +	u32 link_flags;
> +	u8 smem_alloc:1;
> +	u8 capture:1;
> +	u8 image:1;
> +	u8 num_fmts;
> +	u8 default_fmt_idx;
> +	u8 max_buf_count;
> +	const struct v4l2_ioctl_ops *ioctl_ops;
> +	const struct v4l2_format *fmts;
> +	const struct v4l2_frmsizeenum *frmsizes;
> +};
> +
> +/*
> + * struct mtk_cam_video_device - Mediatek video device structure.
> + *
> + * @id:		Id for mtk_cam_dev_node_desc or mem2mem2_nodes array
> + * @enabled:	Indicate the device is enabled or not
> + * @vdev_fmt:	The V4L2 format of video device
> + * @vdev_apd:	The media pad graph object of video device

vdev_pad?

> + * @vbq:	A videobuf queue of video device
> + * @desc:	The node attributes of video device
> + * @ctrl_handler:	The control handler of video device
> + * @pending_list:	List for pending buffers before enqueuing into driver
> + * @lock:	Serializes vb2 queue and video device operations.
> + * @slock:	Protect for pending_list.
> + *

Please fix the order of the documentation to match the order of the struct.

> + */
> +struct mtk_cam_video_device {
> +	unsigned int id;
> +	unsigned int enabled;
> +	struct v4l2_format vdev_fmt;
> +	struct mtk_cam_dev_node_desc desc;
> +	struct video_device vdev;

Not documented above.

> +	struct media_pad vdev_pad;
> +	struct vb2_queue vbq;
> +	struct v4l2_ctrl_handler ctrl_handler;
> +	struct list_head pending_list;
> +	/* Used for vbq & vdev */

It's already documented in the kerneldoc comment.

> +	struct mutex lock;
> +	/* protect for pending_list */

It's already documented in the kerneldoc comment.

> +	spinlock_t slock;

How about calling it pending_list_lock?

> +};
> +
> +/*
> + * struct mtk_cam_dev - Mediatek camera device structure.
> + *
> + * @pdev:	Pointer to platform device
> + * @smem_pdev:	Pointer to shared memory platform device
> + * @pipeline:	Media pipeline information
> + * @media_dev:	Media device
> + * @subdev:	The V4L2 sub-device
> + * @v4l2_dev:	The V4L2 device driver
> + * @notifier:	The v4l2_device notifier data
> + * @subdev_pads: Pointer to the number of media pads of this sub-device
> + * @ctrl_handler: The control handler
> + * @vdev_nodes: The array list of mtk_cam_video_device nodes
> + * @seninf:	Pointer to the seninf sub-device
> + * @sensor:	Pointer to the active sensor V4L2 sub-device when streaming on
> + * @lock:       The mutex protecting video device open/release operations
> + * @streaming:	Indicate the overall streaming status is on or off
> + * @streamed_node_count: The number of V4L2 video device nodes are streaming on
> + * @req_list:	Lins to keep media requests before streaming on
> + * @req_lock:	Protect the req_list data
> + *
> + * Below is the graph topology for Camera IO connection.
> + * sensor 1 (main) --> sensor IF --> P1 sub-device
> + * sensor 2 (sub)  -->

This probably isn't the best place for graph topology description. I think
we actually want a separate documentation file for this, similar to
Documentation/media/v4l-drivers/ipu3.rst.

> + *
> + */
> +struct mtk_cam_dev {
> +	struct platform_device *pdev;
> +	struct device *smem_dev;
> +	struct media_pipeline pipeline;
> +	struct media_device media_dev;
> +	struct v4l2_subdev subdev;
> +	struct v4l2_device v4l2_dev;
> +	struct v4l2_async_notifier notifier;
> +	struct media_pad *subdev_pads;
> +	struct v4l2_ctrl_handler ctrl_handler;
> +	struct mtk_cam_video_device vdev_nodes[MTK_CAM_P1_TOTAL_NODES];
> +	struct v4l2_subdev *seninf;
> +	struct v4l2_subdev *sensor;
> +	/* protect video device open/release operations */

It's already documented in the kerneldoc comment.

> +	struct mutex lock;
> +	unsigned int streaming:1;
> +	atomic_t streamed_node_count;
> +	struct list_head req_list;
> +	/* protect for req_list */

It's already documented in the kerneldoc comment.

> +	spinlock_t req_lock;

How about calling it req_list_lock?

Best regards,
Tomasz


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-06-11  3:53 ` [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver Jungo Lin
@ 2019-07-10  9:56   ` Tomasz Figa
  2019-07-20  9:58     ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-10  9:56 UTC (permalink / raw)
  To: Jungo Lin
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi Jungo,

On Tue, Jun 11, 2019 at 11:53:42AM +0800, Jungo Lin wrote:
> This patch adds the Mediatek ISP P1 HW control device driver.
> It handles the ISP HW configuration, provides interrupt handling and
> initializes the V4L2 device nodes and other functions.
> 
> (The current metadata interface used in meta input and partial
> meta nodes is only a temporary solution to kick off the driver
> development and is not ready to be reviewed yet.)
> 
> Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> ---
>  .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
>  .../mtk-isp/isp_50/cam/mtk_cam-regs.h         |  126 ++
>  .../platform/mtk-isp/isp_50/cam/mtk_cam.c     | 1087 +++++++++++++++++
>  .../platform/mtk-isp/isp_50/cam/mtk_cam.h     |  243 ++++
>  4 files changed, 1457 insertions(+)
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
> 

Thanks for the patch! Please see my comments inline.

[snip]

> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
> new file mode 100644
> index 000000000000..9e59a6bfc6b7
> --- /dev/null
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
> @@ -0,0 +1,126 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2018 MediaTek Inc.
> + */
> +
> +#ifndef _CAM_REGS_H
> +#define _CAM_REGS_H
> +
> +/* TG Bit Mask */
> +#define VFDATA_EN_BIT			BIT(0)
> +#define CMOS_EN_BIT			BIT(0)
> +
> +/* normal signal bit */
> +#define VS_INT_ST			BIT(0)
> +#define HW_PASS1_DON_ST		BIT(11)
> +#define SOF_INT_ST			BIT(12)
> +#define SW_PASS1_DON_ST		BIT(30)
> +
> +/* err status bit */
> +#define TG_ERR_ST			BIT(4)
> +#define TG_GBERR_ST			BIT(5)
> +#define CQ_CODE_ERR_ST			BIT(6)
> +#define CQ_APB_ERR_ST			BIT(7)
> +#define CQ_VS_ERR_ST			BIT(8)
> +#define AMX_ERR_ST			BIT(15)
> +#define RMX_ERR_ST			BIT(16)
> +#define BMX_ERR_ST			BIT(17)
> +#define RRZO_ERR_ST			BIT(18)
> +#define AFO_ERR_ST			BIT(19)
> +#define IMGO_ERR_ST			BIT(20)
> +#define AAO_ERR_ST			BIT(21)
> +#define PSO_ERR_ST			BIT(22)
> +#define LCSO_ERR_ST			BIT(23)
> +#define BNR_ERR_ST			BIT(24)
> +#define LSCI_ERR_ST			BIT(25)
> +#define DMA_ERR_ST			BIT(29)
> +
> +/* CAM DMA done status */
> +#define FLKO_DONE_ST			BIT(4)
> +#define AFO_DONE_ST			BIT(5)
> +#define AAO_DONE_ST			BIT(7)
> +#define PSO_DONE_ST			BIT(14)
> +
> +/* IRQ signal mask */
> +#define INT_ST_MASK_CAM		( \
> +					VS_INT_ST |\
> +					SOF_INT_ST |\
> +					HW_PASS1_DON_ST |\
> +					SW_PASS1_DON_ST)
> +
> +/* IRQ Error Mask */
> +#define INT_ST_MASK_CAM_ERR		( \
> +					TG_ERR_ST |\
> +					TG_GBERR_ST |\
> +					CQ_CODE_ERR_ST |\
> +					CQ_APB_ERR_ST |\
> +					CQ_VS_ERR_ST |\
> +					BNR_ERR_ST |\
> +					RMX_ERR_ST |\
> +					BMX_ERR_ST |\
> +					BNR_ERR_ST |\
> +					LSCI_ERR_ST |\
> +					DMA_ERR_ST)
> +
> +/* IRQ Signal Log Mask */
> +#define INT_ST_LOG_MASK_CAM		( \
> +					SOF_INT_ST |\
> +					SW_PASS1_DON_ST |\
> +					HW_PASS1_DON_ST |\
> +					VS_INT_ST |\
> +					TG_ERR_ST |\
> +					TG_GBERR_ST |\
> +					RRZO_ERR_ST |\
> +					AFO_ERR_ST |\
> +					IMGO_ERR_ST |\
> +					AAO_ERR_ST |\
> +					DMA_ERR_ST)
> +
> +/* DMA Event Notification Mask */
> +#define DMA_ST_MASK_CAM		( \
> +					AAO_DONE_ST |\
> +					AFO_DONE_ST)

Could we define the values next to the addresses of registers they apply to?
Also without the _BIT suffix and with the values prefixed with register
names. For example:

#define REG_TG_SEN_MODE		        0x0230
#define TG_SEN_MODE_CMOS_EN		BIT(0)

#define REG_TG_VF_CON			0x0234
#define TG_VF_CON_VFDATA_EN		BIT(0)

> +
> +/* Status check */
> +#define REG_CTL_EN			0x0004
> +#define REG_CTL_DMA_EN			0x0008
> +#define REG_CTL_FMT_SEL		0x0010
> +#define REG_CTL_EN2			0x0018
> +#define REG_CTL_RAW_INT_EN		0x0020
> +#define REG_CTL_RAW_INT_STAT		0x0024
> +#define REG_CTL_RAW_INT2_STAT		0x0034
> +
> +#define REG_TG_SEN_MODE		0x0230
> +#define REG_TG_VF_CON			0x0234
> +
> +#define REG_IMGO_BASE_ADDR		0x1020
> +#define REG_RRZO_BASE_ADDR		0x1050
> +
> +/* Error status log */
> +#define REG_IMGO_ERR_STAT		0x1360
> +#define REG_RRZO_ERR_STAT		0x1364
> +#define REG_AAO_ERR_STAT		0x1368
> +#define REG_AFO_ERR_STAT		0x136c
> +#define REG_LCSO_ERR_STAT		0x1370
> +#define REG_UFEO_ERR_STAT		0x1374
> +#define REG_PDO_ERR_STAT		0x1378
> +#define REG_BPCI_ERR_STAT		0x137c
> +#define REG_LSCI_ERR_STAT		0x1384
> +#define REG_PDI_ERR_STAT		0x138c
> +#define REG_LMVO_ERR_STAT		0x1390
> +#define REG_FLKO_ERR_STAT		0x1394
> +#define REG_PSO_ERR_STAT		0x13a0
> +
> +/* ISP command */
> +#define REG_CQ_THR0_BASEADDR		0x0198
> +#define REG_HW_FRAME_NUM		0x13b8
> +
> +/* META */
> +#define REG_META0_VB2_INDEX		0x14dc
> +#define REG_META1_VB2_INDEX		0x151c

I don't believe these registers are really for VB2 indexes.

> +
> +/* FBC */
> +#define REG_AAO_FBC_STATUS		0x013c
> +#define REG_AFO_FBC_STATUS		0x0134
> +
> +#endif	/* _CAM_REGS_H */
> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
> new file mode 100644
> index 000000000000..c5a3babed69d
> --- /dev/null
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
> @@ -0,0 +1,1087 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (c) 2018 MediaTek Inc.
> +
> +#include <linux/atomic.h>
> +#include <linux/cdev.h>
> +#include <linux/compat.h>
> +#include <linux/fs.h>
> +#include <linux/interrupt.h>
> +#include <linux/jiffies.h>
> +#include <linux/kernel.h>
> +#include <linux/ktime.h>
> +#include <linux/module.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_irq.h>
> +#include <linux/of_address.h>
> +#include <linux/platform_device.h>
> +#include <linux/platform_data/mtk_scp.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/remoteproc.h>
> +#include <linux/sched/clock.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/videodev2.h>
> +#include <linux/vmalloc.h>
> +#include <media/v4l2-event.h>
> +
> +#include "mtk_cam.h"
> +#include "mtk_cam-regs.h"
> +#include "mtk_cam-smem.h"
> +
> +static const struct of_device_id mtk_isp_of_ids[] = {
> +	{.compatible = "mediatek,mt8183-camisp",},
> +	{}
> +};
> +MODULE_DEVICE_TABLE(of, mtk_isp_of_ids);

Please move below. Just above the platform_driver struct where it's used.

> +
> +/* List of clocks required by isp cam */
> +static const char * const mtk_isp_clks[] = {
> +	"camsys_cam_cgpdn", "camsys_camtg_cgpdn"
> +};

Please move inside mtk_isp_probe, as a static const local variable. That
could also let you shorten the name, to clk_names for example.

> +
> +static void isp_dump_dma_status(struct isp_device *isp_dev)
> +{
> +	dev_err(isp_dev->dev,
> +		"IMGO:0x%x, RRZO:0x%x, AAO=0x%x, AFO=0x%x, LMVO=0x%x\n",
> +		readl(isp_dev->regs + REG_IMGO_ERR_STAT),
> +		readl(isp_dev->regs + REG_RRZO_ERR_STAT),
> +		readl(isp_dev->regs + REG_AAO_ERR_STAT),
> +		readl(isp_dev->regs + REG_AFO_ERR_STAT),
> +		readl(isp_dev->regs + REG_LMVO_ERR_STAT));
> +	dev_err(isp_dev->dev,
> +		"LCSO=0x%x, PSO=0x%x, FLKO=0x%x, BPCI:0x%x, LSCI=0x%x\n",
> +		readl(isp_dev->regs + REG_LCSO_ERR_STAT),
> +		readl(isp_dev->regs + REG_PSO_ERR_STAT),
> +		readl(isp_dev->regs + REG_FLKO_ERR_STAT),
> +		readl(isp_dev->regs + REG_BPCI_ERR_STAT),
> +		readl(isp_dev->regs + REG_LSCI_ERR_STAT));
> +}
> +
> +static void mtk_cam_dev_event_frame_sync(struct mtk_cam_dev *cam_dev,
> +					 __u32 frame_seq_no)
> +{
> +	struct v4l2_event event;
> +
> +	memset(&event, 0, sizeof(event));
> +	event.type = V4L2_EVENT_FRAME_SYNC;
> +	event.u.frame_sync.frame_sequence = frame_seq_no;

nit: You can just initialize the structure in the declaration.

> +	v4l2_event_queue(cam_dev->subdev.devnode, &event);
> +}
> +
> +static void mtk_cam_dev_job_finish(struct mtk_isp_p1_ctx *isp_ctx,
> +				   unsigned int request_fd,
> +				   unsigned int frame_seq_no,
> +				   struct list_head *list_buf,
> +				   enum vb2_buffer_state state)
> +{
> +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> +	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
> +	struct mtk_cam_dev_buffer *buf, *b0;
> +	u64    timestamp;

Too many spaces between u64 and timestamp.

> +
> +	if (!cam_dev->streaming)
> +		return;
> +
> +	dev_dbg(&p1_dev->pdev->dev, "%s request fd:%d frame_seq:%d state:%d\n",
> +		__func__, request_fd, frame_seq_no, state);
> +
> +	/*
> +	 * Set the buffer's VB2 status so that the user can dequeue
> +	 * the buffer.
> +	 */
> +	timestamp = ktime_get_ns();
> +	list_for_each_entry_safe(buf, b0, list_buf, list) {

nit: s/b0/buf_prev/

> +		list_del(&buf->list);
> +		buf->vbb.vb2_buf.timestamp = timestamp;
> +		buf->vbb.sequence = frame_seq_no;
> +		if (buf->vbb.vb2_buf.state == VB2_BUF_STATE_ACTIVE)

Any buffer that is not active shouldn't be on this list. If it is then it's
a bug somewhere else in the driver. Could be possibly related to the request
handling issues I pointed out in another comment.

> +			vb2_buffer_done(&buf->vbb.vb2_buf, state);
> +	}
> +}
> +
> +static void isp_deque_frame(struct isp_p1_device *p1_dev,

dequeue

> +			    unsigned int node_id, int vb2_index,
> +			    int frame_seq_no)
> +{
> +	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
> +	struct device *dev = &p1_dev->pdev->dev;
> +	struct mtk_cam_video_device *node = &cam_dev->vdev_nodes[node_id];
> +	struct mtk_cam_dev_buffer *b, *b0;
> +	struct vb2_buffer *vb;
> +
> +	if (!cam_dev->vdev_nodes[node_id].enabled || !cam_dev->streaming)
> +		return;
> +
> +	spin_lock(&node->slock);
> +	b = list_first_entry(&node->pending_list,
> +			     struct mtk_cam_dev_buffer,
> +			     list);
> +	list_for_each_entry_safe(b, b0, &node->pending_list, list) {
> +		vb = &b->vbb.vb2_buf;
> +		if (!vb->vb2_queue->uses_requests &&
> +		    vb->index == vb2_index &&
> +		    vb->state == VB2_BUF_STATE_ACTIVE) {
> +			dev_dbg(dev, "%s:%d:%d", __func__, node_id, vb2_index);
> +			vb->timestamp = ktime_get_ns();
> +			b->vbb.sequence = frame_seq_no;
> +			vb2_buffer_done(vb, VB2_BUF_STATE_DONE);
> +			list_del(&b->list);
> +			break;
> +		}
> +	}
> +	spin_unlock(&node->slock);
> +}
> +
> +static void isp_deque_request_frame(struct isp_p1_device *p1_dev,

dequeue

> +				    int frame_seq_no)
> +{
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct device *dev = &p1_dev->pdev->dev;
> +	struct mtk_isp_queue_job *framejob, *tmp;
> +	struct isp_queue *p1_enqueue_list = &isp_ctx->p1_enqueue_list;
> +
> +	/* Match dequeue work and enqueue frame */
> +	spin_lock(&p1_enqueue_list->lock);
> +	list_for_each_entry_safe(framejob, tmp, &p1_enqueue_list->queue,
> +				 list_entry) {
> +		dev_dbg(dev,
> +			"%s frame_seq_no:%d, target frame_seq_no:%d\n",
> +			__func__,
> +			framejob->frame_seq_no, frame_seq_no);
> +		/* Match by the en-queued request number */
> +		if (framejob->frame_seq_no == frame_seq_no) {
> +			/* Pass to user space */
> +			mtk_cam_dev_job_finish(isp_ctx,
> +					       framejob->request_fd,
> +					       framejob->frame_seq_no,
> +					       &framejob->list_buf,
> +					       VB2_BUF_STATE_DONE);
> +			atomic_dec(&p1_enqueue_list->queue_cnt);
> +			dev_dbg(dev,
> +				"frame_seq_no:%d is done, queue_cnt:%d\n",
> +				framejob->frame_seq_no,
> +				atomic_read(&p1_enqueue_list->queue_cnt));
> +
> +			/* Remove only when frame ready */
> +			list_del(&framejob->list_entry);
> +			kfree(framejob);
> +			break;
> +		} else if (framejob->frame_seq_no < frame_seq_no) {
> +			/* Pass to user space for frame drop */
> +			mtk_cam_dev_job_finish(isp_ctx,
> +					       framejob->request_fd,
> +					       framejob->frame_seq_no,
> +					       &framejob->list_buf,
> +					       VB2_BUF_STATE_ERROR);
> +			atomic_dec(&p1_enqueue_list->queue_cnt);
> +			dev_warn(dev,
> +				 "frame_seq_no:%d drop, queue_cnt:%d\n",
> +				 framejob->frame_seq_no,
> +				 atomic_read(&p1_enqueue_list->queue_cnt));
> +
> +			/* Remove only drop frame */
> +			list_del(&framejob->list_entry);
> +			kfree(framejob);
> +		} else {
> +			break;
> +		}
> +	}
> +	spin_unlock(&p1_enqueue_list->lock);
> +}
> +
> +static int isp_deque_work(void *data)

dequeue

> +{
> +	struct isp_p1_device *p1_dev = (struct isp_p1_device *)data;
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct mtk_cam_dev_stat_event_data event_data;
> +	atomic_t *irq_data_end = &isp_ctx->irq_data_end;
> +	atomic_t *irq_data_start = &isp_ctx->irq_data_start;
> +	unsigned long flags;
> +	int ret, i;
> +
> +	while (1) {
> +		ret = wait_event_interruptible(isp_ctx->isp_deque_thread.wq,
> +					       (atomic_read(irq_data_end) !=
> +					       atomic_read(irq_data_start)) ||
> +					       kthread_should_stop());
> +
> +		if (kthread_should_stop())
> +			break;
> +
> +		spin_lock_irqsave(&isp_ctx->irq_dequeue_lock, flags);
> +		i = atomic_read(&isp_ctx->irq_data_start);
> +		memcpy(&event_data, &isp_ctx->irq_event_datas[i],
> +		       sizeof(event_data));
> +		atomic_set(&isp_ctx->irq_data_start, ++i & 0x3);
> +		spin_unlock_irqrestore(&isp_ctx->irq_dequeue_lock, flags);
> +
> +		if (event_data.irq_status_mask & HW_PASS1_DON_ST &&
> +		    event_data.dma_status_mask & AAO_DONE_ST) {
> +			isp_deque_frame(p1_dev,
> +					MTK_CAM_P1_META_OUT_0,
> +					event_data.meta0_vb2_index,
> +					event_data.frame_seq_no);
> +		}
> +		if (event_data.dma_status_mask & AFO_DONE_ST) {
> +			isp_deque_frame(p1_dev,
> +					MTK_CAM_P1_META_OUT_1,
> +					event_data.meta1_vb2_index,
> +					event_data.frame_seq_no);
> +		}
> +		if (event_data.irq_status_mask & SW_PASS1_DON_ST) {
> +			isp_deque_frame(p1_dev,
> +					MTK_CAM_P1_META_OUT_0,
> +					event_data.meta0_vb2_index,
> +					event_data.frame_seq_no);
> +			isp_deque_frame(p1_dev,
> +					MTK_CAM_P1_META_OUT_1,
> +					event_data.meta1_vb2_index,
> +					event_data.frame_seq_no);
> +			isp_deque_request_frame(p1_dev,
> +						event_data.frame_seq_no);
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int irq_handle_sof(struct isp_device *isp_dev,
> +			  dma_addr_t base_addr,
> +			  unsigned int frame_num)
> +{
> +	unsigned int addr_offset;
> +	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
> +	int cq_num = atomic_read(&p1_dev->isp_ctx.composed_frame_id);
> +
> +	isp_dev->sof_count += 1;
> +
> +	if (cq_num <= frame_num) {
> +		dev_dbg(isp_dev->dev,
> +			"SOF_INT_ST, wait next, cq_num:%d, frame_num:%d",
> +			cq_num, frame_num);
> +		atomic_set(&p1_dev->isp_ctx.composing_frame, 0);
> +		return cq_num;
> +	}
> +	atomic_set(&p1_dev->isp_ctx.composing_frame, cq_num - frame_num);
> +
> +	addr_offset = CQ_ADDRESS_OFFSET * (frame_num % CQ_BUFFER_COUNT);
> +	writel(base_addr + addr_offset, isp_dev->regs + REG_CQ_THR0_BASEADDR);
> +	dev_dbg(isp_dev->dev,
> +		"SOF_INT_ST, update next, cq_num:%d, frame_num:%d cq_addr:0x%x",
> +		cq_num, frame_num, addr_offset);
> +
> +	return cq_num;
> +}
> +
> +static void irq_handle_notify_event(struct isp_device *isp_dev,
> +				    unsigned int irq_status,
> +				    unsigned int dma_status,
> +				    bool sof_only)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct device *dev = isp_dev->dev;
> +	unsigned long flags;
> +	int i;
> +
> +	if (irq_status & VS_INT_ST) {
> +		/* Notify specific HW events to user space */
> +		mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev,
> +					     isp_dev->current_frame);

Shouldn't we call this at SOF_INT_ST and not VS? At least according to the
definition of the V4L2_EVENT_FRAME_SYNC event at
https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-dqevent.html

> +		dev_dbg(dev,
> +			"frame sync is sent:%d:%d\n",
> +			isp_dev->sof_count,
> +			isp_dev->current_frame);
> +		if (sof_only)
> +			return;

If this function can be called only to perform this block, perhaps it should
be split into two functions?

Also, what happens if we get sof_only, but we don't get VS_INT_ST set in
irq_status? Is it expected that in such case the other part of the function
is executed?

> +	}
> +
> +	if (irq_status & SW_PASS1_DON_ST) {
> +		/* Notify TX thread to send if TX frame is blocked */
> +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> +	}
> +
> +	spin_lock_irqsave(&isp_ctx->irq_dequeue_lock, flags);
> +	i = atomic_read(&isp_ctx->irq_data_end);
> +	isp_ctx->irq_event_datas[i].frame_seq_no = isp_dev->current_frame;
> +	isp_ctx->irq_event_datas[i].meta0_vb2_index = isp_dev->meta0_vb2_index;
> +	isp_ctx->irq_event_datas[i].meta1_vb2_index = isp_dev->meta1_vb2_index;
> +	isp_ctx->irq_event_datas[i].irq_status_mask =
> +		(irq_status & INT_ST_MASK_CAM);
> +	isp_ctx->irq_event_datas[i].dma_status_mask =
> +		(dma_status & DMA_ST_MASK_CAM);
> +	atomic_set(&isp_ctx->irq_data_end, ++i & 0x3);
> +	spin_unlock_irqrestore(&isp_ctx->irq_dequeue_lock, flags);
> +
> +	wake_up_interruptible(&isp_ctx->isp_deque_thread.wq);

I can see that all isp_deque_work() does is returning buffers to vb2. I
don't think we need this intricate system to do that, as we could just do
it inside the interrupt handler, in isp_irq_cam() directly.

> +
> +	dev_dbg(dev,
> +		"%s IRQ:0x%x DMA:0x%x seq:%d idx0:%d idx1:%d\n",
> +		__func__,
> +		(irq_status & INT_ST_MASK_CAM),
> +		(dma_status & DMA_ST_MASK_CAM),
> +		isp_dev->current_frame,
> +		isp_dev->meta0_vb2_index,
> +		isp_dev->meta1_vb2_index);
> +}
> +
> +irqreturn_t isp_irq_cam(int irq, void *data)
> +{
> +	struct isp_device *isp_dev = (struct isp_device *)data;
> +	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct device *dev = isp_dev->dev;
> +	unsigned int cam_idx, cq_num, hw_frame_num;
> +	unsigned int meta0_vb2_index, meta1_vb2_index;
> +	unsigned int irq_status, err_status, dma_status;
> +	unsigned int aao_fbc, afo_fbc;
> +	unsigned long flags;
> +
> +	/* Check the streaming is off or not */
> +	if (!p1_dev->cam_dev.streaming)
> +		return IRQ_HANDLED;

This shouldn't be needed. The driver needs to unmask the interrupts in
hardware registers when it starts streaming and mask them when it stops.
Note that I mean the P1 hardware registers, not disable_irq(), which
shouldn't be used.

> +
> +	cam_idx = isp_dev->isp_hw_module - ISP_CAM_A_IDX;
> +	cq_num = 0;
> +
> +	spin_lock_irqsave(&isp_dev->spinlock_irq, flags);
> +	irq_status = readl(isp_dev->regs + REG_CTL_RAW_INT_STAT);
> +	dma_status = readl(isp_dev->regs + REG_CTL_RAW_INT2_STAT);
> +	hw_frame_num = readl(isp_dev->regs + REG_HW_FRAME_NUM);
> +	meta0_vb2_index = readl(isp_dev->regs + REG_META0_VB2_INDEX);
> +	meta1_vb2_index = readl(isp_dev->regs + REG_META1_VB2_INDEX);

Hmm, reading vb2 buffer index from hardware registers? Was this hardware
designed exclusively for V4L2? ;)

Jokes aside, how does the hardware know V4L2 buffer indexes?

> +	aao_fbc = readl(isp_dev->regs + REG_AAO_FBC_STATUS);
> +	afo_fbc = readl(isp_dev->regs + REG_AFO_FBC_STATUS);
> +	spin_unlock_irqrestore(&isp_dev->spinlock_irq, flags);
> +
> +	/* Ignore unnecessary IRQ */
> +	if (!irq_status && (!(dma_status & DMA_ST_MASK_CAM)))
> +		return IRQ_HANDLED;

Unnecessary IRQs should be masked in the hardware IRQ mask registers. If we
get an interrupt without any unmasked hardware IRQs active in the status,
that's an error somewhere and we should return IRQ_NONE.

> +
> +	err_status = irq_status & INT_ST_MASK_CAM_ERR;
> +
> +	/* Sof, done order check */
> +	if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST)) {
> +		dev_dbg(dev, "sof_done block cnt:%d\n", isp_dev->sof_count);
> +
> +		/* Notify IRQ event and enqueue frame */
> +		irq_handle_notify_event(isp_dev, irq_status, dma_status, 0);
> +		isp_dev->current_frame = hw_frame_num;

What exactly is hw_frame_num? Shouldn't we assign it before notifying the
event?

> +		isp_dev->meta0_vb2_index = meta0_vb2_index;
> +		isp_dev->meta1_vb2_index = meta1_vb2_index;
> +	} else {
> +		if (irq_status & SOF_INT_ST) {
> +			isp_dev->current_frame = hw_frame_num;
> +			isp_dev->meta0_vb2_index = meta0_vb2_index;
> +			isp_dev->meta1_vb2_index = meta1_vb2_index;
> +		}
> +		irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
> +	}

The if and else blocks do almost the same things just in different order. Is
it really expected?

> +
> +	if (irq_status & SOF_INT_ST)
> +		cq_num = irq_handle_sof(isp_dev, isp_ctx->scp_mem_iova,
> +					hw_frame_num);
> +
> +	/* Check ISP error status */
> +	if (err_status) {
> +		dev_err(dev,
> +			"raw_int_err:0x%x/0x%x\n",
> +			irq_status, err_status);
> +		/* Show DMA errors in detail */
> +		if (err_status & DMA_ERR_ST)
> +			isp_dump_dma_status(isp_dev);
> +	}
> +
> +	if (irq_status & INT_ST_LOG_MASK_CAM)
> +		dev_dbg(dev, IRQ_STAT_STR,

Please just put that string here, otherwise the reader would have no idea
what message is being printed here.

> +			'A' + cam_idx,
> +			isp_dev->sof_count,
> +			irq_status,
> +			dma_status,
> +			hw_frame_num,
> +			cq_num,
> +			aao_fbc,
> +			afo_fbc);

nit: No need to put each argument in its own line.

> +
> +	return IRQ_HANDLED;
> +}
> +
> +static int isp_setup_scp_rproc(struct isp_p1_device *p1_dev)
> +{
> +	phandle rproc_phandle;
> +	struct device *dev = &p1_dev->pdev->dev;
> +	int ret;
> +
> +	p1_dev->scp_pdev = scp_get_pdev(p1_dev->pdev);
> +	if (!p1_dev->scp_pdev) {
> +		dev_err(dev, "Failed to get scp device\n");
> +		return -ENODEV;
> +	}
> +
> +	ret = of_property_read_u32(dev->of_node, "mediatek,scp",
> +				   &rproc_phandle);
> +	if (ret) {
> +		dev_err(dev, "fail to get rproc_phandle:%d\n", ret);
> +		return -EINVAL;
> +	}
> +
> +	p1_dev->rproc_handle = rproc_get_by_phandle(rproc_phandle);
> +	dev_dbg(dev, "p1 rproc_phandle: 0x%pK\n\n", p1_dev->rproc_handle);
> +	if (!p1_dev->rproc_handle) {
> +		dev_err(dev, "fail to get rproc_handle\n");
> +		return -EINVAL;
> +	}

This look-up should be done once in probe(). Only the rproc_boot() should
happen dynamically.

> +
> +	ret = rproc_boot(p1_dev->rproc_handle);
> +	if (ret) {
> +		/*
> +		 * Return 0 if downloading firmware successfully,
> +		 * otherwise it is failed
> +		 */
> +		return -ENODEV;
> +	}
> +
> +	return 0;
> +}
> +
> +static int isp_init_context(struct isp_p1_device *p1_dev)
> +{
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct device *dev = &p1_dev->pdev->dev;
> +	unsigned int i;
> +
> +	dev_dbg(dev, "init irq work thread\n");
> +	if (!isp_ctx->isp_deque_thread.thread) {
> +		init_waitqueue_head(&isp_ctx->isp_deque_thread.wq);
> +		isp_ctx->isp_deque_thread.thread =
> +			kthread_run(isp_deque_work, (void *)p1_dev,
> +				    "isp_deque_work");
> +		if (IS_ERR(isp_ctx->isp_deque_thread.thread)) {
> +			dev_err(dev, "unable to alloc kthread\n");
> +			isp_ctx->isp_deque_thread.thread = NULL;
> +			return -ENOMEM;
> +		}
> +	}
> +	spin_lock_init(&isp_ctx->irq_dequeue_lock);
> +	mutex_init(&isp_ctx->lock);
> +
> +	INIT_LIST_HEAD(&isp_ctx->p1_enqueue_list.queue);
> +	atomic_set(&isp_ctx->p1_enqueue_list.queue_cnt, 0);
> +
> +	for (i = 0; i < ISP_DEV_NODE_NUM; i++)
> +		spin_lock_init(&p1_dev->isp_devs[i].spinlock_irq);
> +
> +	spin_lock_init(&isp_ctx->p1_enqueue_list.lock);
> +	spin_lock_init(&isp_ctx->composer_txlist.lock);
> +
> +	atomic_set(&isp_ctx->irq_data_end, 0);
> +	atomic_set(&isp_ctx->irq_data_start, 0);
> +
> +	return 0;

Everything here looks like something that should be done once in probe. I
also don't see a point of having a separate mtk_isp_p1_ctx struct for the
data above. It could be just located in p1_dev, at least for now.

If we end up implementing support for multiple contexts, we could isolate
per-context data then, once we know what's really per-context. For now,
let's keep it simple.

> +}
> +
> +static int isp_uninit_context(struct device *dev)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct mtk_isp_queue_job *framejob, *tmp_framejob;
> +
> +	spin_lock_irq(&isp_ctx->p1_enqueue_list.lock);
> +	list_for_each_entry_safe(framejob, tmp_framejob,
> +				 &isp_ctx->p1_enqueue_list.queue, list_entry) {
> +		list_del(&framejob->list_entry);
> +		kfree(framejob);
> +	}
> +	spin_unlock_irq(&isp_ctx->p1_enqueue_list.lock);
> +
> +	if (isp_ctx->isp_deque_thread.thread) {
> +		kthread_stop(isp_ctx->isp_deque_thread.thread);
> +		wake_up_interruptible(&isp_ctx->isp_deque_thread.wq);
> +		isp_ctx->isp_deque_thread.thread = NULL;
> +	}
> +
> +	mutex_destroy(&isp_ctx->lock);
> +
> +	return 0;
> +}
> +
> +static unsigned int get_enabled_dma_ports(struct mtk_cam_dev *cam_dev)
> +{
> +	unsigned int enabled_dma_ports, i;
> +
> +	/* Get the enabled meta DMA ports */
> +	enabled_dma_ports = 0;
> +
> +	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++)
> +		if (cam_dev->vdev_nodes[i].enabled)
> +			enabled_dma_ports |=
> +				cam_dev->vdev_nodes[i].desc.dma_port;
> +
> +	dev_dbg(&cam_dev->pdev->dev, "%s :0x%x", __func__, enabled_dma_ports);
> +
> +	return enabled_dma_ports;
> +}
> +
> +/* Utility functions */
> +static unsigned int get_sensor_pixel_id(unsigned int fmt)
> +{
> +	switch (fmt) {
> +	case MEDIA_BUS_FMT_SBGGR8_1X8:
> +	case MEDIA_BUS_FMT_SBGGR10_1X10:
> +	case MEDIA_BUS_FMT_SBGGR12_1X12:
> +	case MEDIA_BUS_FMT_SBGGR14_1X14:
> +		return RAW_PXL_ID_B;
> +	case MEDIA_BUS_FMT_SGBRG8_1X8:
> +	case MEDIA_BUS_FMT_SGBRG10_1X10:
> +	case MEDIA_BUS_FMT_SGBRG12_1X12:
> +	case MEDIA_BUS_FMT_SGBRG14_1X14:
> +		return RAW_PXL_ID_GB;
> +	case MEDIA_BUS_FMT_SGRBG8_1X8:
> +	case MEDIA_BUS_FMT_SGRBG10_1X10:
> +	case MEDIA_BUS_FMT_SGRBG12_1X12:
> +	case MEDIA_BUS_FMT_SGRBG14_1X14:
> +		return RAW_PXL_ID_GR;
> +	case MEDIA_BUS_FMT_SRGGB8_1X8:
> +	case MEDIA_BUS_FMT_SRGGB10_1X10:
> +	case MEDIA_BUS_FMT_SRGGB12_1X12:
> +	case MEDIA_BUS_FMT_SRGGB14_1X14:
> +		return RAW_PXL_ID_R;
> +	default:

Could we fail here instead?

> +		return RAW_PXL_ID_B;
> +	}
> +}
> +
> +static unsigned int get_sensor_fmt(unsigned int fmt)
> +{
> +	switch (fmt) {
> +	case MEDIA_BUS_FMT_SBGGR8_1X8:
> +	case MEDIA_BUS_FMT_SGBRG8_1X8:
> +	case MEDIA_BUS_FMT_SGRBG8_1X8:
> +	case MEDIA_BUS_FMT_SRGGB8_1X8:
> +		return IMG_FMT_BAYER8;
> +	case MEDIA_BUS_FMT_SBGGR10_1X10:
> +	case MEDIA_BUS_FMT_SGBRG10_1X10:
> +	case MEDIA_BUS_FMT_SGRBG10_1X10:
> +	case MEDIA_BUS_FMT_SRGGB10_1X10:
> +		return IMG_FMT_BAYER10;
> +	case MEDIA_BUS_FMT_SBGGR12_1X12:
> +	case MEDIA_BUS_FMT_SGBRG12_1X12:
> +	case MEDIA_BUS_FMT_SGRBG12_1X12:
> +	case MEDIA_BUS_FMT_SRGGB12_1X12:
> +		return IMG_FMT_BAYER12;
> +	case MEDIA_BUS_FMT_SBGGR14_1X14:
> +	case MEDIA_BUS_FMT_SGBRG14_1X14:
> +	case MEDIA_BUS_FMT_SGRBG14_1X14:
> +	case MEDIA_BUS_FMT_SRGGB14_1X14:
> +		return IMG_FMT_BAYER14;
> +	default:
> +		return IMG_FMT_UNKNOWN;
> +	}
> +}
> +
> +static unsigned int get_img_fmt(unsigned int fourcc)
> +{
> +	switch (fourcc) {
> +	case V4L2_PIX_FMT_MTISP_B8:
> +		return IMG_FMT_BAYER8;
> +	case V4L2_PIX_FMT_MTISP_F8:
> +		return IMG_FMT_FG_BAYER8;
> +	case V4L2_PIX_FMT_MTISP_B10:
> +		return IMG_FMT_BAYER10;
> +	case V4L2_PIX_FMT_MTISP_F10:
> +		return IMG_FMT_FG_BAYER10;
> +	case V4L2_PIX_FMT_MTISP_B12:
> +		return IMG_FMT_BAYER12;
> +	case V4L2_PIX_FMT_MTISP_F12:
> +		return IMG_FMT_FG_BAYER12;
> +	case V4L2_PIX_FMT_MTISP_B14:
> +		return IMG_FMT_BAYER14;
> +	case V4L2_PIX_FMT_MTISP_F14:
> +		return IMG_FMT_FG_BAYER14;
> +	default:
> +		return IMG_FMT_UNKNOWN;
> +	}
> +}
> +
> +static unsigned int get_pixel_byte(unsigned int fourcc)
> +{
> +	switch (fourcc) {
> +	case V4L2_PIX_FMT_MTISP_B8:
> +	case V4L2_PIX_FMT_MTISP_F8:
> +		return 8;
> +	case V4L2_PIX_FMT_MTISP_B10:
> +	case V4L2_PIX_FMT_MTISP_F10:
> +		return 10;
> +	case V4L2_PIX_FMT_MTISP_B12:
> +	case V4L2_PIX_FMT_MTISP_F12:
> +		return 12;
> +	case V4L2_PIX_FMT_MTISP_B14:
> +	case V4L2_PIX_FMT_MTISP_F14:
> +		return 14;
> +	default:

Could we fail here instead, so that we don't mask some potential errors?

> +		return 10;
> +	}
> +}
> +
> +static void config_img_fmt(struct device *dev, struct p1_img_output *out_fmt,
> +			   const struct v4l2_format *in_fmt,
> +			   const struct v4l2_subdev_format *sd_format)
> +{
> +	out_fmt->img_fmt = get_img_fmt(in_fmt->fmt.pix_mp.pixelformat);
> +	out_fmt->pixel_byte = get_pixel_byte(in_fmt->fmt.pix_mp.pixelformat);
> +	out_fmt->size.w = in_fmt->fmt.pix_mp.width;
> +	out_fmt->size.h = in_fmt->fmt.pix_mp.height;
> +
> +	out_fmt->size.stride = in_fmt->fmt.pix_mp.plane_fmt[0].bytesperline;
> +	out_fmt->size.xsize = in_fmt->fmt.pix_mp.plane_fmt[0].bytesperline;

Please group operations on the same field together, i.e. remove the blank
line above size.stride and add one blank line above size.w.

> +
> +	out_fmt->crop.left = 0x0;
> +	out_fmt->crop.top = 0x0;
> +

Remove the blank line.

> +	out_fmt->crop.width = sd_format->format.width;
> +	out_fmt->crop.height = sd_format->format.height;
> +
> +	WARN_ONCE(in_fmt->fmt.pix_mp.width > out_fmt->crop.width ||
> +		  in_fmt->fmt.pix_mp.height > out_fmt->crop.height,
> +		  "img out:%d:%d in:%d:%d",
> +		  in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height,
> +		  out_fmt->crop.width, out_fmt->crop.height);

We should check this earlier and fail the streaming start if there is a
mismatch between sensor and video node configuration.

> +
> +	dev_dbg(dev, "pixel_byte:%d img_fmt:0x%x\n",
> +		out_fmt->pixel_byte,
> +		out_fmt->img_fmt);
> +	dev_dbg(dev,
> +		"param:size=%0dx%0d, stride:%d, xsize:%d, crop=%0dx%0d\n",
> +		out_fmt->size.w, out_fmt->size.h,
> +		out_fmt->size.stride, out_fmt->size.xsize,
> +		out_fmt->crop.width, out_fmt->crop.height);
> +}
> +
> +/* ISP P1 interface functions */
> +int mtk_isp_power_init(struct mtk_cam_dev *cam_dev)
> +{
> +	struct device *dev = &cam_dev->pdev->dev;
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	int ret;
> +
> +	ret = isp_setup_scp_rproc(p1_dev);
> +	if (ret)
> +		return ret;
> +
> +	ret = isp_init_context(p1_dev);
> +	if (ret)
> +		return ret;

The above function doesn't really seem to be related to power management.
Should it be called from subdev stream on?

> +
> +	ret = isp_composer_init(dev);
> +	if (ret)
> +		goto composer_err;

This also doesn't seem to be related to power management.

> +
> +	pm_runtime_get_sync(dev);
> +
> +	/* ISP HW INIT */
> +	isp_ctx->isp_hw_module = ISP_CAM_B_IDX;

There is some amount of code handling various values of isp_hw_module in
this driver. If we're hardcoding ISP_CAM_B_IDX here, it's basically dead
code that can't be tested. Please either add support for all the indexes in
the driver or simplify all the code to just handle CAM_B.

> +	/* Use pure RAW as default HW path */
> +	isp_ctx->isp_raw_path = ISP_PURE_RAW_PATH;
> +	atomic_set(&p1_dev->cam_dev.streamed_node_count, 0);
> +
> +	isp_composer_hw_init(dev);
> +	/* Check enabled DMAs which is configured by media setup */
> +	isp_composer_meta_config(dev, get_enabled_dma_ports(cam_dev));

Hmm, this seems to be also configured by isp_compoer_hw_config(). Are both
necessary?

> +
> +	dev_dbg(dev, "%s done\n", __func__);
> +
> +	return 0;
> +
> +composer_err:
> +	isp_uninit_context(dev);
> +
> +	return ret;
> +}
> +
> +int mtk_isp_power_release(struct device *dev)
> +{
> +	isp_composer_hw_deinit(dev);
> +	isp_uninit_context(dev);

These two don't seem to be related to power either.

Instead, I don't see anything that could undo the rproc_boot() operation
here.

> +
> +	dev_dbg(dev, "%s done\n", __func__);
> +
> +	return 0;
> +}
> +
> +int mtk_isp_config(struct device *dev)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct p1_config_param config_param;
> +	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
> +	struct v4l2_subdev_format sd_fmt;
> +	unsigned int enabled_dma_ports;
> +	struct v4l2_format *img_fmt;
> +	int ret;
> +
> +	p1_dev->isp_devs[isp_ctx->isp_hw_module].current_frame = 0;
> +	p1_dev->isp_devs[isp_ctx->isp_hw_module].sof_count = 0;
> +
> +	isp_ctx->frame_seq_no = 1;
> +	atomic_set(&isp_ctx->composed_frame_id, 0);
> +
> +	/* Get the enabled DMA ports */
> +	enabled_dma_ports = get_enabled_dma_ports(cam_dev);
> +	dev_dbg(dev, "%s enable_dma_ports:0x%x", __func__, enabled_dma_ports);
> +
> +	/* Sensor config */
> +	sd_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
> +	ret = v4l2_subdev_call(cam_dev->sensor, pad, get_fmt, NULL, &sd_fmt);
> +

Unnecessary blank line.

> +	if (ret) {
> +		dev_dbg(dev, "sensor g_fmt on failed:%d\n", ret);
> +		return -EPERM;

return ret?

> +	}
> +
> +	dev_dbg(dev,
> +		"get_fmt ret=%d, w=%d, h=%d, code=0x%x, field=%d, color=%d\n",
> +		ret, sd_fmt.format.width, sd_fmt.format.height,
> +		sd_fmt.format.code, sd_fmt.format.field,
> +		sd_fmt.format.colorspace);
> +
> +	config_param.cfg_in_param.continuous = 0x1;
> +	config_param.cfg_in_param.subsample = 0x0;
> +	/* Fix to one pixel mode in default */
> +	config_param.cfg_in_param.pixel_mode = 0x1;
> +	/* Support normal pattern in default */
> +	config_param.cfg_in_param.data_pattern = 0x0;
> +
> +	config_param.cfg_in_param.crop.left = 0x0;
> +	config_param.cfg_in_param.crop.top = 0x0;

Why hexadecimal numerals? Also, what's the meaning of these values? For
anything boolean, you could just use true and false as a substite of 0 and
1. For anything that has more values, please define the values using macros.

> +
> +	config_param.cfg_in_param.raw_pixel_id =
> +		get_sensor_pixel_id(sd_fmt.format.code);
> +	config_param.cfg_in_param.img_fmt = get_sensor_fmt(sd_fmt.format.code);
> +	config_param.cfg_in_param.crop.width = sd_fmt.format.width;
> +	config_param.cfg_in_param.crop.height = sd_fmt.format.height;

Move the other crop settings from above to here.

> +
> +	config_param.cfg_main_param.bypass = 1;
> +	img_fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_MAIN_STREAM_OUT].vdev_fmt;
> +	if ((enabled_dma_ports & R_IMGO) == R_IMGO) {

No need for the == R_IMGO part.

> +		config_param.cfg_main_param.bypass = 0;
> +		config_param.cfg_main_param.pure_raw = isp_ctx->isp_raw_path;
> +		config_param.cfg_main_param.pure_raw_pack = 1;
> +		config_img_fmt(dev, &config_param.cfg_main_param.output,
> +			       img_fmt, &sd_fmt);
> +	}
> +
> +	config_param.cfg_resize_param.bypass = 1;
> +	img_fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_PACKED_BIN_OUT].vdev_fmt;
> +	if ((enabled_dma_ports & R_RRZO) == R_RRZO) {

Ditto.

> +		config_param.cfg_resize_param.bypass = 0;
> +		config_img_fmt(dev, &config_param.cfg_resize_param.output,
> +			       img_fmt, &sd_fmt);
> +	}
> +
> +	/* Configure meta DMAs info. */
> +	config_param.cfg_meta_param.enabled_meta_dmas = enabled_dma_ports;

Should image DMAs be masked out of this bitmap?

> +
> +	isp_composer_hw_config(dev, &config_param);
> +
> +	dev_dbg(dev, "%s done\n", __func__);
> +
> +	return 0;
> +}
> +
> +void mtk_isp_enqueue(struct device *dev, unsigned int dma_port,
> +		     struct mtk_cam_dev_buffer *buffer)
> +{
> +	struct mtk_isp_scp_p1_cmd frameparams;
> +
> +	memset(&frameparams, 0, sizeof(frameparams));
> +	frameparams.cmd_id = ISP_CMD_ENQUEUE_META;
> +	frameparams.meta_frame.enabled_dma = dma_port;
> +	frameparams.meta_frame.vb_index = buffer->vbb.vb2_buf.index;
> +	frameparams.meta_frame.meta_addr.iova = buffer->daddr;
> +	frameparams.meta_frame.meta_addr.scp_addr = buffer->scp_addr;
> +
> +	isp_composer_enqueue(dev, &frameparams, SCP_ISP_CMD);
> +}
> +
> +void mtk_isp_req_flush_buffers(struct device *dev)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	struct mtk_isp_queue_job *job, *j0;
> +	struct mtk_cam_dev_buffer *buf, *b0;
> +	struct isp_queue *p1_list = &p1_dev->isp_ctx.p1_enqueue_list;
> +
> +	if (!atomic_read(&p1_list->queue_cnt))
> +		return;

Do we need this explicit check? The code below wouldn't do anything if there
isn't anything in the list. IMHO we could even remove queue_cnt completely.

> +
> +	spin_lock(&p1_list->lock);
> +	list_for_each_entry_safe(job, j0, &p1_list->queue, list_entry) {

nit: s/j0/job_prev/

> +		list_for_each_entry_safe(buf, b0, &job->list_buf, list) {

nit: s/b0/buf_pref/

Also, we should be able to replace this with iterating over the generic list
of request objects, rather than this internal list.

> +			list_del(&buf->list);
> +			if (buf->vbb.vb2_buf.state == VB2_BUF_STATE_ACTIVE)

It shouldn't be possible to happen. If you see this check failing, that
means a problem somewhere else in the driver.

> +				vb2_buffer_done(&buf->vbb.vb2_buf,
> +						VB2_BUF_STATE_ERROR);
> +		}
> +		list_del(&job->list_entry);
> +		atomic_dec(&p1_list->queue_cnt);
> +		kfree(job);
> +	}
> +	spin_unlock(&p1_list->lock);
> +}
> +
> +void mtk_isp_req_enqueue(struct device *dev, struct media_request *req)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);

Just pass p1_dev to this function instead of dev.

> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	struct p1_frame_param frameparams;
> +	struct mtk_isp_queue_job *framejob;
> +	struct media_request_object *obj, *obj_safe;
> +	struct vb2_buffer *vb;
> +	struct mtk_cam_dev_buffer *buf;
> +
> +	framejob = kzalloc(sizeof(*framejob), GFP_ATOMIC);

This allocation shouldn't be needed. The structure should be already a part
of the mtk_cam_dev_request struct that wraps media_request. Actually. I'd
even say that the contents of this struct should be just moved to that one
to avoid overabstracting.

> +	memset(framejob, 0, sizeof(*framejob));

Putting the above comment aside, kzalloc() already zeroes the memory.

> +	memset(&frameparams, 0, sizeof(frameparams));
> +	INIT_LIST_HEAD(&framejob->list_buf);
> +
> +	frameparams.frame_seq_no = isp_ctx->frame_seq_no++;
> +	frameparams.sof_idx =
> +		p1_dev->isp_devs[isp_ctx->isp_hw_module].sof_count;

How is this synchronized with the sof_count increment in irq_handle_sof()?

> +	framejob->frame_seq_no = frameparams.frame_seq_no;
> +
> +	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
> +		vb = container_of(obj, struct vb2_buffer, req_obj);

We should check that the object type before assuming it's a vb2_buffer by
calling vb2_request_object_is_buffer().

> +		buf = container_of(vb, struct mtk_cam_dev_buffer, vbb.vb2_buf);
> +		framejob->request_fd = buf->vbb.request_fd;

We shouldn't use request_fd as the key here. We already have req here, which
is the right key to use.

That said, I can see framejob->request_fd only used for printing a debugging
message in mtk_cam_dev_job_finish(). The request API core should already
print something for us once a request is completed, so perhaps that isn't
needed?

> +		frameparams.dma_buffers[buf->node_id].iova = buf->daddr;
> +		frameparams.dma_buffers[buf->node_id].scp_addr = buf->scp_addr;
> +		list_add_tail(&buf->list, &framejob->list_buf);

Why do we need this private list? We could just call exactly the same
list_for_each() over the request objects.

> +	}
> +
> +	spin_lock(&isp_ctx->p1_enqueue_list.lock);
> +	list_add_tail(&framejob->list_entry, &isp_ctx->p1_enqueue_list.queue);

We already have a list head in mtk_cam_dev_request.

> +	atomic_inc(&isp_ctx->p1_enqueue_list.queue_cnt);
> +	spin_unlock(&isp_ctx->p1_enqueue_list.lock);
> +
> +	isp_composer_enqueue(dev, &frameparams, SCP_ISP_FRAME);
> +	dev_dbg(dev, "request fd:%d frame_seq_no:%d is queued cnt:%d\n",
> +		framejob->request_fd,
> +		frameparams.frame_seq_no,
> +		atomic_read(&isp_ctx->p1_enqueue_list.queue_cnt));
> +}
> +
> +static int enable_sys_clock(struct isp_p1_device *p1_dev)
> +{
> +	struct device *dev = &p1_dev->pdev->dev;
> +	int ret;
> +
> +	dev_info(dev, "- %s\n", __func__);

dev_dbg()

> +
> +	ret = clk_bulk_prepare_enable(p1_dev->isp_ctx.num_clks,
> +				      p1_dev->isp_ctx.clk_list);
> +	if (ret)
> +		goto clk_err;
> +
> +	return 0;
> +
> +clk_err:
> +	dev_err(dev, "cannot pre-en isp_cam clock:%d\n", ret);
> +	clk_bulk_disable_unprepare(p1_dev->isp_ctx.num_clks,
> +				   p1_dev->isp_ctx.clk_list);

clk_bulk_prepare_enable() returns without any clocks enabled if it fails, so
this would disable the clocks second time.

> +	return ret;
> +}
> +
> +static void disable_sys_clock(struct isp_p1_device *p1_dev)
> +{
> +	dev_info(&p1_dev->pdev->dev, "- %s\n", __func__);

dev_dbg()

> +	clk_bulk_disable_unprepare(p1_dev->isp_ctx.num_clks,
> +				   p1_dev->isp_ctx.clk_list);
> +}

There is no point in having wrapper functions to just call one function
inside. Please just call clk_bulk_*() directly.

> +
> +static int mtk_isp_suspend(struct device *dev)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	int module = p1_dev->isp_ctx.isp_hw_module;
> +	struct isp_device *isp_dev = &p1_dev->isp_devs[module];
> +	unsigned int reg_val;
> +
> +	dev_dbg(dev, "- %s\n", __func__);
> +

We need to check if the device isn't already runtime suspended. If it is, we
don't have to do anything here and can just return.


We also need to ensure that no new requests are queued to the hardware at
this point. This could be done by replacing any of the kthreads with
workqueues and making all of the workqueues freezable.

> +	isp_dev = &p1_dev->isp_devs[module];
> +	reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
> +	if (reg_val & VFDATA_EN_BIT) {
> +		dev_dbg(dev, "Cam:%d suspend, disable VF\n", module);
> +		/* Disable view finder */
> +		writel((reg_val & (~VFDATA_EN_BIT)),
> +		       isp_dev->regs + REG_TG_VF_CON);
> +		/*
> +		 * After VF enable, the TG frame count will be reset to 0;
> +		 */
> +		reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
> +		writel((reg_val & (~CMOS_EN_BIT)),
> +		       isp_dev->regs +  + REG_TG_SEN_MODE);
> +	}

Are you sure this is the right handling? We need to make sure the hardware
finishes processing the current frame and stops.

> +
> +	disable_sys_clock(p1_dev);
> +
> +	return 0;
> +}
> +
> +static int mtk_isp_resume(struct device *dev)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	int module = p1_dev->isp_ctx.isp_hw_module;
> +	struct isp_device *isp_dev = &p1_dev->isp_devs[module];
> +	unsigned int reg_val;
> +
> +	dev_dbg(dev, "- %s\n", __func__);
> +

We need to check runtime PM status here as well, because if the device was
suspended, there is nothing to do here.

> +	enable_sys_clock(p1_dev);
> +
> +	/* V4L2 stream-on phase & restore HW stream-on status */
> +	if (p1_dev->cam_dev.streaming) {
> +		dev_dbg(dev, "Cam:%d resume,enable VF\n", module);
> +		/* Enable CMOS */
> +		reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
> +		writel((reg_val | CMOS_EN_BIT),
> +		       isp_dev->regs + REG_TG_SEN_MODE);
> +		/* Enable VF */
> +		reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
> +		writel((reg_val | VFDATA_EN_BIT),
> +		       isp_dev->regs + REG_TG_VF_CON);
> +	}

Does the hardware keep all the state, e.g. queued buffers, during suspend?
Would the code above continue all the capture from the next buffer, as
queued by the userspace before the suspend?

> +
> +	return 0;
> +}
> +
> +static int mtk_isp_probe(struct platform_device *pdev)
> +{
> +	struct isp_p1_device *p1_dev;
> +	struct mtk_isp_p1_ctx *isp_ctx;
> +	struct isp_device *isp_dev;
> +	struct device *dev = &pdev->dev;
> +	struct resource *res;
> +	int irq;
> +	int ret;
> +	unsigned int i;
> +
> +	p1_dev = devm_kzalloc(dev, sizeof(*p1_dev), GFP_KERNEL);
> +	if (!p1_dev)
> +		return -ENOMEM;
> +
> +	dev_set_drvdata(dev, p1_dev);
> +	isp_ctx = &p1_dev->isp_ctx;
> +	p1_dev->pdev = pdev;

Perhaps you want to store &pdev->dev instead of pdev? I'm not sure a
reference to platform_device is very useful later in the code.

> +
> +	for (i = ISP_CAMSYS_CONFIG_IDX; i < ISP_DEV_NODE_NUM; i++) {

I think we want to start from 0 here?

> +		isp_dev = &p1_dev->isp_devs[i];
> +		isp_dev->isp_hw_module = i;
> +		isp_dev->dev = dev;

p1_dev already includes a pointer to this.

> +		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
> +		isp_dev->regs = devm_ioremap_resource(dev, res);
> +
> +		dev_dbg(dev, "cam%u, map_addr=0x%lx\n",
> +			i, (unsigned long)isp_dev->regs);
> +
> +		if (!isp_dev->regs)

devm_ioremap_resource() returns ERR_PTR() not NULL on error.

> +			return PTR_ERR(isp_dev->regs);
> +
> +		/* Support IRQ from ISP_CAM_A_IDX */
> +		if (i >= ISP_CAM_A_IDX) {
> +			/* Reg & interrupts index is shifted with 1  */

The reader can already see that it's shifted from the code below. The
comment should say _why_ it is shifted.

> +			irq = platform_get_irq(pdev, i - 1);

The bindings have 3 IRQs, but we only seem to request 2 here. Is that
expected?

> +			if (irq) {

Please invert this if, so that it bails out on error. Also, this should
check for negative errors codes, not 0.

> +				ret = devm_request_irq(dev, irq,
> +						       isp_irq_cam,
> +						       IRQF_SHARED,
> +						       dev_driver_string(dev),

Use dev_name().

> +						       (void *)isp_dev);

No need to cast to void *.

> +				if (ret) {
> +					dev_err(dev,
> +						"req_irq fail, dev:%s irq=%d\n",
> +						dev->of_node->name,
> +						irq);
> +					return ret;
> +				}
> +				dev_dbg(dev, "Registered irq=%d, ISR:%s\n",
> +					irq, dev_driver_string(dev));
> +			}
> +		}
> +		spin_lock_init(&isp_dev->spinlock_irq);
> +	}

We might want to move out the body of this loop to a separate function to
keep this function shorter.

> +
> +	p1_dev->isp_ctx.num_clks = ARRAY_SIZE(mtk_isp_clks);
> +	p1_dev->isp_ctx.clk_list =

nit: "list" is the term commonly used for the list data structure. There is
also a convention to call the length of array XXX num_XXX, so how about
clks and num_clks?

> +		devm_kcalloc(dev,
> +			     p1_dev->isp_ctx.num_clks,
> +			     sizeof(*p1_dev->isp_ctx.clk_list),
> +			     GFP_KERNEL);
> +	if (!p1_dev->isp_ctx.clk_list)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < p1_dev->isp_ctx.num_clks; ++i)
> +		p1_dev->isp_ctx.clk_list->id = mtk_isp_clks[i];

Shouldn't this be clk_list[i].id?

> +
> +	ret = devm_clk_bulk_get(dev,
> +				p1_dev->isp_ctx.num_clks,
> +				p1_dev->isp_ctx.clk_list);
> +	if (ret) {
> +		dev_err(dev, "cannot get isp cam clock:%d\n", ret);
> +		return ret;
> +	}
> +
> +	/* Initialize reserved DMA memory */
> +	ret = mtk_cam_reserved_memory_init(p1_dev);
> +	if (ret) {
> +		dev_err(dev, "failed to configure DMA memory:%d\n", ret);
> +		goto err_init;
> +	}
> +
> +	/* Initialize the v4l2 common part */
> +	ret = mtk_cam_dev_init(pdev, &p1_dev->cam_dev);
> +	if (ret)
> +		goto err_init;
> +
> +	spin_lock_init(&isp_ctx->p1_enqueue_list.lock);
> +	pm_runtime_enable(dev);
> +
> +	return 0;
> +
> +err_init:
> +	if (p1_dev->cam_dev.smem_dev)
> +		device_unregister(p1_dev->cam_dev.smem_dev);
> +
> +	return ret;
> +}
> +
> +static int mtk_isp_remove(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	struct isp_p1_device *p1_dev = dev_get_drvdata(dev);
> +
> +	pm_runtime_disable(dev);
> +	mtk_cam_dev_release(pdev, &p1_dev->cam_dev);
> +
> +	return 0;
> +}
> +
> +static const struct dev_pm_ops mtk_isp_pm_ops = {
> +	SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
> +	SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)

For V4L2 drivers system and runtime PM ops would normally be completely
different. Runtime PM ops would be called when the hardware is idle already
or is about to become active. System PM ops would be called at system power
state change and the hardware might be both idle or active. Please also see
my comments to mtk_isp_suspend() and mtk_isp_resume() above.

> +};
> +
> +static struct platform_driver mtk_isp_driver = {
> +	.probe   = mtk_isp_probe,
> +	.remove  = mtk_isp_remove,
> +	.driver  = {
> +		.name  = "mtk-cam",

Shouldn't this be something like mtk-cam-p1? Please make sure this
reasonably consistent with other drivers.

> +		.of_match_table = of_match_ptr(mtk_isp_of_ids),
> +		.pm     = &mtk_isp_pm_ops,
> +	}
> +};
> +
> +module_platform_driver(mtk_isp_driver);
> +
> +MODULE_DESCRIPTION("Camera ISP driver");

Mediatek Camera P1 ISP driver? Please make sure this is reasonably
consistent with other drivers (SenInf, DIP, FD).

> +MODULE_LICENSE("GPL");

GPL v2

diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
new file mode 100644
index 000000000000..6af3f569664c
> --- /dev/null
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
@@ -0,0 +1,243 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2018 MediaTek Inc.
> + */
> +
> +#ifndef __CAMERA_ISP_H
> +#define __CAMERA_ISP_H
> +
> +#include <linux/cdev.h>
> +#include <linux/clk.h>
> +#include <linux/interrupt.h>
> +#include <linux/ioctl.h>
> +#include <linux/irqreturn.h>
> +#include <linux/miscdevice.h>
> +#include <linux/pm_qos.h>
> +#include <linux/scatterlist.h>
> +
> +#include "mtk_cam-scp.h"
> +#include "mtk_cam-v4l2-util.h"
> +
> +#define CAM_A_MAX_WIDTH		3328
> +#define CAM_A_MAX_HEIGHT		2496
> +#define CAM_B_MAX_WIDTH		5376
> +#define CAM_B_MAX_HEIGHT		4032
> +
> +#define CAM_MIN_WIDTH			80
> +#define CAM_MIN_HEIGHT			60
> +
> +#define IMG_MAX_WIDTH			CAM_B_MAX_WIDTH
> +#define IMG_MAX_HEIGHT			CAM_B_MAX_HEIGHT
> +#define IMG_MIN_WIDTH			CAM_MIN_WIDTH
> +#define IMG_MIN_HEIGHT			CAM_MIN_HEIGHT
> +
> +#define RRZ_MAX_WIDTH			CAM_B_MAX_WIDTH
> +#define RRZ_MAX_HEIGHT			CAM_B_MAX_HEIGHT
> +#define RRZ_MIN_WIDTH			CAM_MIN_WIDTH
> +#define RRZ_MIN_HEIGHT			CAM_MIN_HEIGHT
> +
> +#define R_IMGO				BIT(0)
> +#define R_RRZO				BIT(1)
> +#define R_AAO				BIT(3)
> +#define R_AFO				BIT(4)
> +#define R_LCSO				BIT(5)
> +#define R_PDO				BIT(6)
> +#define R_LMVO				BIT(7)
> +#define R_FLKO				BIT(8)
> +#define R_RSSO				BIT(9)
> +#define R_PSO				BIT(10)
> +
> +#define CQ_BUFFER_COUNT		3
> +#define IRQ_DATA_BUF_SIZE		4
> +#define CQ_ADDRESS_OFFSET		0x640
> +
> +#define ISP_COMPOSING_MAX_NUM		4
> +#define ISP_FRAME_COMPOSING_MAX_NUM	3
> +
> +#define IRQ_STAT_STR	"cam%c, SOF_%d irq(0x%x), " \
> +			"dma(0x%x), frame_num(%d)/cq_num(%d), " \
> +			"fbc1(0x%x), fbc2(0x%x)\n"
> +
> +/*
> + * In order with the sequence of device nodes defined in dtsi rule,
> + * one hardware module should be mapping to one node.
> + */
> +enum isp_dev_node_enum {
> +	ISP_CAMSYS_CONFIG_IDX = 0,
> +	ISP_CAM_UNI_IDX,
> +	ISP_CAM_A_IDX,
> +	ISP_CAM_B_IDX,
> +	ISP_DEV_NODE_NUM
> +};
> +
> +/* Image RAW path for ISP P1 module. */
> +enum isp_raw_path_enum {
> +	ISP_PROCESS_RAW_PATH = 0,
> +	ISP_PURE_RAW_PATH
> +};
> +
> +/* State for struct mtk_isp_p1_ctx: composer_state */
> +enum  {
> +	SCP_ON = 0,
> +	SCP_OFF
> +};

Hmm, looks like bool.

> +
> +enum {
> +	IMG_FMT_UNKNOWN		= 0x0000,
> +	IMG_FMT_RAW_START	= 0x2200,
> +	IMG_FMT_BAYER8		= IMG_FMT_RAW_START,
> +	IMG_FMT_BAYER10,
> +	IMG_FMT_BAYER12,
> +	IMG_FMT_BAYER14,
> +	IMG_FMT_FG_BAYER8,
> +	IMG_FMT_FG_BAYER10,
> +	IMG_FMT_FG_BAYER12,
> +	IMG_FMT_FG_BAYER14,
> +};
> +
> +enum {
> +	RAW_PXL_ID_B = 0,
> +	RAW_PXL_ID_GB,
> +	RAW_PXL_ID_GR,
> +	RAW_PXL_ID_R
> +};

Please use macros with explicitly assigned values for hardware/firmware ABI
definitions.

> +
> +struct isp_queue {
> +	struct list_head queue;
> +	atomic_t queue_cnt;
> +	spinlock_t lock; /* queue attributes protection */
> +};
> +
> +struct isp_thread {
> +	struct task_struct *thread;
> +	wait_queue_head_t wq;
> +};
> +
> +struct mtk_isp_queue_work {
> +	union {
> +		struct mtk_isp_scp_p1_cmd cmd;
> +		struct p1_frame_param frameparams;
> +	};
> +	struct list_head list_entry;
> +	enum mtk_isp_scp_type type;
> +};
> +
> +struct mtk_cam_dev_stat_event_data {
> +	__u32 frame_seq_no;
> +	__u32 meta0_vb2_index;
> +	__u32 meta1_vb2_index;
> +	__u32 irq_status_mask;
> +	__u32 dma_status_mask;
> +};
> +
> +struct mtk_isp_queue_job {
> +	struct list_head list_entry;
> +	struct list_head list_buf;
> +	unsigned int request_fd;
> +	unsigned int frame_seq_no;
> +};

Please document the structs above using kerneldoc.

> +
> +/*
> + * struct isp_device - the ISP device information
> + *
> + * @dev: Pointer to struct device
> + * @regs: Camera ISP base register address
> + * @spinlock_irq: Used to protect register read/write data
> + * @current_frame: Current frame sequence number, set when SOF
> + * @meta0_vb2_index: Meta0 vb2 buffer index, set when SOF
> + * @meta1_vb2_index: Meta1 vb2 buffer index, set when SOF
> + * @sof_count: The accumulated SOF counter
> + * @isp_hw_module: Identity camera A or B
> + *
> + */
> +struct isp_device {

mtk_isp_device?

> +	struct device *dev;
> +	void __iomem *regs;
> +	spinlock_t spinlock_irq; /* ISP reg setting integrity */
> +	unsigned int current_frame;
> +	unsigned int meta0_vb2_index;
> +	unsigned int meta1_vb2_index;
> +	u8 sof_count;
> +	u8 isp_hw_module;
> +};
> +
> +/*
> + * struct mtk_isp_p1_ctx - the ISP device information
> + *
> + * @composer_txlist: Queue for SCP TX data including SCP_ISP_CMD & SCP_ISP_FRAME
> + * @composer_tx_thread: TX Thread for SCP data tranmission
> + * @cmd_queued: The number of SCP_ISP_CMD commands will be sent
> + * @ipi_occupied: The total number of SCP TX data has beent sent
> + * @scp_state: The state of SCP control
> + * @composing_frame: The total number of SCP_ISP_FRAME has beent sent
> + * @composed_frame_id: The ack. frame sequence by SCP
> + * @composer_deinit_thread: The de-initialized thread
> + * @p1_enqueue_list: Queue for ISP frame buffers
> + * @isp_deque_thread: Thread for handling ISP frame buffers dequeue
> + * @irq_event_datas: Ring buffer for struct mtk_cam_dev_stat_event_data data
> + * @irq_data_start: Start index of irq_event_datas ring buffer
> + * @irq_data_end: End index of irq_event_datas ring buffer
> + * @irq_dequeue_lock: Lock to protect irq_event_datas ring buffer
> + * @scp_mem_pa: DMA address for SCP device
> + * @scp_mem_iova: DMA address for ISP HW DMA devices
> + * @frame_seq_no: Sequence number for ISP frame buffer
> + * @isp_hw_module: Active camera HW module
> + * @num_clks: The number of driver's clock
> + * @clk_list: The list of clock data
> + * @lock: Lock to protect context operations
> + *
> + */
> +struct mtk_isp_p1_ctx {
> +	struct isp_queue composer_txlist;
> +	struct isp_thread composer_tx_thread;
> +	atomic_t cmd_queued;
> +	atomic_t ipi_occupied;
> +	atomic_t scp_state;
> +	atomic_t composing_frame;
> +	atomic_t composed_frame_id;
> +	struct isp_thread composer_deinit_thread;
> +	struct isp_queue p1_enqueue_list;
> +	struct isp_thread isp_deque_thread;
> +	struct mtk_cam_dev_stat_event_data irq_event_datas[IRQ_DATA_BUF_SIZE];
> +	atomic_t irq_data_start;
> +	atomic_t irq_data_end;
> +	spinlock_t irq_dequeue_lock; /* ISP frame dequeuq protection */

Already documented in kerneldoc.

> +	dma_addr_t scp_mem_pa;
> +	dma_addr_t scp_mem_iova;
> +	int frame_seq_no;
> +	unsigned int isp_hw_module;
> +	unsigned int isp_raw_path;

Not documented above.

> +	unsigned int num_clks;
> +	struct clk_bulk_data *clk_list;
> +	struct mutex lock; /* Protect context operations */

Already documented in kerneldoc.

> +};
> +
> +struct isp_p1_device {
> +	struct platform_device *pdev;
> +	struct platform_device *scp_pdev;
> +	struct rproc *rproc_handle;
> +	struct mtk_isp_p1_ctx isp_ctx;
> +	struct mtk_cam_dev cam_dev;
> +	struct isp_device isp_devs[ISP_DEV_NODE_NUM];
> +};

Please document in a kerneldoc comment.

> +
> +static inline struct isp_p1_device *
> +p1_ctx_to_dev(const struct mtk_isp_p1_ctx *__p1_ctx)
> +{
> +	return container_of(__p1_ctx, struct isp_p1_device, isp_ctx);
> +}
> +
> +static inline struct isp_p1_device *get_p1_device(struct device *dev)
> +{
> +	return ((struct isp_p1_device *)dev_get_drvdata(dev));

No need to cast. And, I don't think we need a function for this, just call
dev_get_drvdata() directly.

Best regards,
Tomasz


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication
  2019-06-11  3:53 ` [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication Jungo Lin
@ 2019-07-10  9:58   ` Tomasz Figa
  2019-07-21  2:18     ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-10  9:58 UTC (permalink / raw)
  To: Jungo Lin
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi Jungo,

On Tue, Jun 11, 2019 at 11:53:43AM +0800, Jungo Lin wrote:
> This patch adds communication with the co-processor on the SoC
> through the SCP driver. It supports bi-directional commands
> to exchange data and perform command flow control function.
> 
> Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> ---
> This patch depends on "Add support for mt8183 SCP"[1].
> 
> [1] https://patchwork.kernel.org/cover/10972143/
> ---
>  .../platform/mtk-isp/isp_50/cam/Makefile      |   1 +
>  .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.c | 371 ++++++++++++++++++
>  .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.h | 207 ++++++++++
>  3 files changed, 579 insertions(+)
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
>  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
> 

Thanks for the patch! Please see my comments inline.

[snip]

> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
> new file mode 100644
> index 000000000000..04519d0b942f
> --- /dev/null
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
> @@ -0,0 +1,371 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (c) 2018 MediaTek Inc.
> +
> +#include <linux/atomic.h>
> +#include <linux/kthread.h>
> +#include <linux/platform_data/mtk_scp.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/remoteproc.h>
> +#include <linux/sched.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/vmalloc.h>
> +
> +#include "mtk_cam.h"
> +
> +static void isp_composer_deinit(struct mtk_isp_p1_ctx *isp_ctx)
> +{
> +	struct mtk_isp_queue_work *ipi_job, *tmp_ipi_job;
> +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> +
> +	atomic_set(&isp_ctx->cmd_queued, 0);
> +	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
> +	atomic_set(&isp_ctx->composing_frame, 0);
> +	atomic_set(&isp_ctx->ipi_occupied, 0);

Is there any point to set them if we are deinitalizing? Moreover,
isp_composer_init() would set them once we start again.

> +
> +	spin_lock(&isp_ctx->composer_txlist.lock);
> +	list_for_each_entry_safe(ipi_job, tmp_ipi_job,
> +				 &isp_ctx->composer_txlist.queue,
> +				 list_entry) {
> +		list_del(&ipi_job->list_entry);
> +		kfree(ipi_job);
> +	}
> +	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
> +	spin_unlock(&isp_ctx->composer_txlist.lock);
> +
> +	mutex_lock(&isp_ctx->lock);
> +	if (isp_ctx->composer_tx_thread.thread) {
> +		kthread_stop(isp_ctx->composer_tx_thread.thread);

Shouldn't the thread be stopped at this point already? If not, wouldn't the
atomic_set() at the beginning of this function confuse it?

In any case, this should be greatly simplified after we move to a workqueue,
with one work per one task to do, as per other comments.

> +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> +		isp_ctx->composer_tx_thread.thread = NULL;
> +	}
> +
> +	if (isp_ctx->composer_deinit_thread.thread) {
> +		wake_up(&isp_ctx->composer_deinit_thread.wq);
> +		isp_ctx->composer_deinit_thread.thread = NULL;
> +	}
> +	mutex_unlock(&isp_ctx->lock);
> +
> +	pm_runtime_put_sync(&p1_dev->pdev->dev);

No need to use the sync variant.

> +}
> +
> +/*
> + * Two kinds of flow control in isp_composer_tx_work.
> + *
> + * Case 1: IPI commands flow control. The maximum number of command queues is 3.
> + * There are two types of IPI commands (SCP_ISP_CMD/SCP_ISP_FRAME) in P1 driver.
> + * It is controlled by ipi_occupied.

ISP_COMPOSING_MAX_NUM is defined to 4, not 3. Is that expected?

> + * The priority of SCP_ISP_CMD is higher than SCP_ISP_FRAME.

What does it mean and why is it so?

> + *
> + * Case 2: Frame buffers flow control. The maximum number of frame buffers is 3.
> + * It is controlled by composing_frame.
> + * Frame buffer is sent by SCP_ISP_FRAME command.

Case 1 already mentions SCP_ISP_FRAME. What's the difference between that
and case 2?

> + */
> +static int isp_composer_tx_work(void *data)
> +{
> +	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)data;
> +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> +	struct device *dev = &p1_dev->pdev->dev;
> +	struct mtk_isp_queue_work *isp_composer_work, *tmp_ipi_job;
> +	struct isp_queue *composer_txlist = &isp_ctx->composer_txlist;
> +	int ret;
> +
> +	while (1) {
> +		ret = wait_event_interruptible
> +			(isp_ctx->composer_tx_thread.wq,
> +			 (atomic_read(&composer_txlist->queue_cnt) > 0 &&
> +			 atomic_read(&isp_ctx->ipi_occupied)
> +				< ISP_COMPOSING_MAX_NUM &&
> +			 atomic_read(&isp_ctx->composing_frame)
> +				< ISP_FRAME_COMPOSING_MAX_NUM) ||
> +			 (atomic_read(&isp_ctx->cmd_queued) > 0 &&
> +			 atomic_read(&isp_ctx->ipi_occupied)
> +				< ISP_COMPOSING_MAX_NUM) ||
> +			 kthread_should_stop());
> +
> +		if (kthread_should_stop())
> +			break;
> +
> +		spin_lock(&composer_txlist->lock);
> +		if (atomic_read(&isp_ctx->cmd_queued) > 0) {
> +			list_for_each_entry_safe(isp_composer_work, tmp_ipi_job,
> +						 &composer_txlist->queue,
> +						 list_entry) {
> +				if (isp_composer_work->type == SCP_ISP_CMD) {
> +					dev_dbg(dev, "Found a cmd\n");
> +					break;
> +				}
> +			}
> +		} else {
> +			if (atomic_read(&isp_ctx->composing_frame) >=
> +				ISP_FRAME_COMPOSING_MAX_NUM) {
> +				spin_unlock(&composer_txlist->lock);
> +				continue;
> +			}
> +			isp_composer_work =
> +			    list_first_entry_or_null
> +				(&composer_txlist->queue,
> +				 struct mtk_isp_queue_work,
> +				 list_entry);
> +		}

I don't understand why this special handling of CMD vs FRAME is here, so I
might be missing something, but would we really lose anything if we just
simply removed it and queued everything in order?

Moreover, in V4L2, buffer queue and control operations are serialized wrt
each other, so we probably wouldn't even have a chance to hit a case when we
need to prioritize a CMD IPI over a FRAME IPI.

> +
> +		list_del(&isp_composer_work->list_entry);
> +		atomic_dec(&composer_txlist->queue_cnt);
> +		spin_unlock(&composer_txlist->lock);
> +
> +		if (isp_composer_work->type == SCP_ISP_CMD) {
> +			scp_ipi_send
> +				(p1_dev->scp_pdev,
> +				 SCP_IPI_ISP_CMD,
> +				 &isp_composer_work->cmd,
> +				 sizeof(isp_composer_work->cmd),
> +				 0);
> +			atomic_dec(&isp_ctx->cmd_queued);
> +			atomic_inc(&isp_ctx->ipi_occupied);
> +			dev_dbg(dev,
> +				"%s cmd id %d sent, %d ipi buf occupied",
> +				__func__,
> +				isp_composer_work->cmd.cmd_id,
> +				atomic_read(&isp_ctx->ipi_occupied));
> +		} else if (isp_composer_work->type == SCP_ISP_FRAME) {
> +			scp_ipi_send
> +				(p1_dev->scp_pdev,
> +				 SCP_IPI_ISP_FRAME,
> +				 &isp_composer_work->frameparams,
> +				 sizeof(isp_composer_work->frameparams),
> +				 0);
> +			atomic_inc(&isp_ctx->ipi_occupied);
> +			atomic_inc(&isp_ctx->composing_frame);

Why do we need composing frame here, if ipi_occupied already limits us to 3?

> +			dev_dbg(dev,
> +				"%s frame %d sent, %d ipi, %d CQ bufs occupied",
> +				__func__,
> +				isp_composer_work->frameparams.frame_seq_no,
> +				atomic_read(&isp_ctx->ipi_occupied),
> +				atomic_read(&isp_ctx->composing_frame));
> +		} else {
> +			dev_err(dev,
> +				"ignore IPI type: %d!\n",
> +				isp_composer_work->type);
> +		}
> +		kfree(isp_composer_work);
> +	}
> +	return ret;
> +}

The function above is way too complicated than it should be. I'd suggest a
model similar to what we ended up in the DIP driver:
>  - a freezable workqueue created for ISP composing works,
>  - each ISP composing work entry would have a struct work_struct embedded,
>  - isp_composer_enqueue() would enqueue the work_struct to the workqueue
>    above,
>  - the workqueue would keep a queue of works itself, so driver's own list
>    wouldn't be needed anymore,
>  - similarly, each execution of the work func would operate on its own ISP
>    composing work, so things like checking for list emptiness, waiting for
>    work to be queued, etc. wouldn't be needed,
>  - freezability of the workqueue would ensure nice synchonization with
>    system suspend/resume (although one would still need to wait for the
>    hardware/firmware to complete).

WDYT?

> +
> +static int isp_composer_deinit_work(void *data)
> +{
> +	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)data;
> +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(data);
> +	struct device *dev = &p1_dev->pdev->dev;
> +
> +	wait_event_interruptible(isp_ctx->composer_deinit_thread.wq,
> +				 atomic_read(&isp_ctx->scp_state) == SCP_OFF ||
> +				 kthread_should_stop());
> +
> +	dev_dbg(dev, "%s run deinit", __func__);
> +	isp_composer_deinit(isp_ctx);
> +
> +	return 0;
> +}
> +
> +static void isp_composer_handler(void *data, unsigned int len, void *priv)
> +{
> +	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)priv;
> +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> +	struct device *dev = &p1_dev->pdev->dev;
> +	struct mtk_isp_scp_p1_cmd *ipi_msg;
> +
> +	ipi_msg = (struct mtk_isp_scp_p1_cmd *)data;

Should we check that len == sizeof(*ipi_msg)? (Or at least >=, if data could
contain some extra bytes at the end.)

> +
> +	if (ipi_msg->cmd_id != ISP_CMD_ACK)
> +		return;
> +
> +	if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
> +		dev_dbg(dev, "ack frame_num:%d",
> +			ipi_msg->ack_info.frame_seq_no);
> +		atomic_set(&isp_ctx->composed_frame_id,
> +			   ipi_msg->ack_info.frame_seq_no);

I suppose we are expecting here that ipi_msg->ack_info.frame_seq_no would be
just isp_ctx->composed_frame_id + 1, right? If not, we probably dropped some
frames and we should handle that somehow.

> +	} else if (ipi_msg->ack_info.cmd_id == ISP_CMD_DEINIT) {
> +		dev_dbg(dev, "ISP_CMD_DEINIT is acked");
> +		atomic_set(&isp_ctx->scp_state, SCP_OFF);
> +		wake_up_interruptible(&isp_ctx->composer_deinit_thread.wq);
> +	}
> +
> +	atomic_dec_return(&isp_ctx->ipi_occupied);
> +	wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> +}
> +
> +int isp_composer_init(struct device *dev)
> +{
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	int ret;
> +
> +	ret = scp_ipi_register(p1_dev->scp_pdev,
> +			       SCP_IPI_ISP_CMD,
> +			       isp_composer_handler,
> +			       isp_ctx);
> +	if (ret)
> +		return ret;
> +
> +	atomic_set(&isp_ctx->cmd_queued, 0);
> +	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
> +	atomic_set(&isp_ctx->composing_frame, 0);
> +	atomic_set(&isp_ctx->ipi_occupied, 0);
> +	atomic_set(&isp_ctx->scp_state, SCP_ON);
> +
> +	mutex_lock(&isp_ctx->lock);
> +	if (!isp_ctx->composer_tx_thread.thread) {
> +		init_waitqueue_head(&isp_ctx->composer_tx_thread.wq);
> +		INIT_LIST_HEAD(&isp_ctx->composer_txlist.queue);
> +		spin_lock_init(&isp_ctx->composer_txlist.lock);
> +		isp_ctx->composer_tx_thread.thread =
> +			kthread_run(isp_composer_tx_work, isp_ctx,
> +				    "isp_composer_tx");
> +		if (IS_ERR(isp_ctx->composer_tx_thread.thread)) {
> +			dev_err(dev, "unable to start kthread\n");
> +			isp_ctx->composer_tx_thread.thread = NULL;
> +			goto nomem;

Why nomem?

> +		}
> +	} else {
> +		dev_warn(dev, "old tx thread is existed\n");

This shouldn't be possible to happen.

> +	}
> +
> +	if (!isp_ctx->composer_deinit_thread.thread) {
> +		init_waitqueue_head(&isp_ctx->composer_deinit_thread.wq);
> +		isp_ctx->composer_deinit_thread.thread =
> +			kthread_run(isp_composer_deinit_work, isp_ctx,
> +				    "isp_composer_deinit_work");

Why do we need to deinit from another kthread?

> +		if (IS_ERR(isp_ctx->composer_deinit_thread.thread)) {
> +			dev_err(dev, "unable to start kthread\n");
> +			isp_ctx->composer_deinit_thread.thread = NULL;
> +			goto nomem;
> +		}
> +	} else {
> +		dev_warn(dev, "old rx thread is existed\n");

rx? The code above seems to refer to deinit.

> +	}
> +	mutex_unlock(&isp_ctx->lock);
> +
> +	return 0;
> +
> +nomem:
> +	mutex_unlock(&isp_ctx->lock);
> +
> +	return -ENOMEM;

We should return the original error code here.

> +}
> +
> +void isp_composer_enqueue(struct device *dev,
> +			  void *data,
> +			  enum mtk_isp_scp_type type)
> +{
> +	struct mtk_isp_queue_work *isp_composer_work;
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);

Just pass p1_dev to this function instead of dev.

> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +
> +	isp_composer_work = kzalloc(sizeof(*isp_composer_work), GFP_KERNEL);

For most of the cases, it should be possible to preallocate this, e.g.
>  - for FRAME, this could be inside the request struct,
>  - for buffer queue it could be inside the buffer struct.

I'd suggest making the caller responsible for allocating if needed.

> +	isp_composer_work->type = type;
> +
> +	switch (type) {
> +	case SCP_ISP_CMD:
> +		memcpy(&isp_composer_work->cmd, data,
> +		       sizeof(isp_composer_work->cmd));
> +		dev_dbg(dev, "Enq ipi cmd id:%d\n",
> +			isp_composer_work->cmd.cmd_id);
> +
> +		spin_lock(&isp_ctx->composer_txlist.lock);
> +		list_add_tail(&isp_composer_work->list_entry,
> +			      &isp_ctx->composer_txlist.queue);
> +		atomic_inc(&isp_ctx->composer_txlist.queue_cnt);
> +		spin_unlock(&isp_ctx->composer_txlist.lock);
> +
> +		atomic_inc(&isp_ctx->cmd_queued);
> +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> +		break;
> +	case SCP_ISP_FRAME:
> +		memcpy(&isp_composer_work->frameparams, data,
> +		       sizeof(isp_composer_work->frameparams));
> +		dev_dbg(dev, "Enq ipi frame_num:%d\n",
> +			isp_composer_work->frameparams.frame_seq_no);
> +
> +		spin_lock(&isp_ctx->composer_txlist.lock);
> +		list_add_tail(&isp_composer_work->list_entry,
> +			      &isp_ctx->composer_txlist.queue);
> +		atomic_inc(&isp_ctx->composer_txlist.queue_cnt);
> +		spin_unlock(&isp_ctx->composer_txlist.lock);
> +
> +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);

The code in both cases is almost exactly the same. The only difference is
the memcpy destination and size and whether isp_ctx->cmd_queued is
incremented or not.

The memcpy will go away if my comment above is addressed and so that would
go down to making the cmd_queued increment conditional.

> +		break;
> +	default:
> +		break;
> +	}
> +}
> +
> +void isp_composer_hw_init(struct device *dev)
> +{
> +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +
> +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> +	composer_tx_cmd.cmd_id = ISP_CMD_INIT;
> +	composer_tx_cmd.frameparam.hw_module = isp_ctx->isp_hw_module;
> +	composer_tx_cmd.frameparam.cq_addr.iova = isp_ctx->scp_mem_iova;
> +	composer_tx_cmd.frameparam.cq_addr.scp_addr = isp_ctx->scp_mem_pa;

Should we also specify the size of the buffer? Otherwise we could end up
with some undetectable overruns.

> +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> +}
> +
> +void isp_composer_meta_config(struct device *dev,
> +			      unsigned int dma)
> +{
> +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> +
> +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> +	composer_tx_cmd.cmd_id = ISP_CMD_CONFIG_META;
> +	composer_tx_cmd.cfg_meta_out_param.enabled_meta_dmas = dma;
> +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> +}
> +
> +void isp_composer_hw_config(struct device *dev,
> +			    struct p1_config_param *config_param)
> +{
> +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> +
> +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> +	composer_tx_cmd.cmd_id = ISP_CMD_CONFIG;
> +	memcpy(&composer_tx_cmd.config_param, config_param,
> +	       sizeof(*config_param));
> +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> +}
> +
> +void isp_composer_stream(struct device *dev, int on)
> +{
> +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> +
> +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> +	composer_tx_cmd.cmd_id = ISP_CMD_STREAM;
> +	composer_tx_cmd.is_stream_on = on;
> +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> +}
> +
> +void isp_composer_hw_deinit(struct device *dev)
> +{
> +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> +	int ret;
> +
> +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> +	composer_tx_cmd.cmd_id = ISP_CMD_DEINIT;
> +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> +
> +	/* Wait for ISP_CMD_DEINIT command is handled done */
> +	ret = wait_event_timeout(isp_ctx->composer_deinit_thread.wq,
> +				 atomic_read(&isp_ctx->scp_state) == SCP_OFF,
> +				 msecs_to_jiffies(2000));
> +	if (ret)
> +		return;
> +
> +	dev_warn(dev, "Timeout & local de-init\n");
> +	isp_composer_deinit(isp_ctx);
> +}
> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
> new file mode 100644
> index 000000000000..fbd8593e9c2d
> --- /dev/null
> +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
> @@ -0,0 +1,207 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2018 MediaTek Inc.
> + */
> +
> +#ifndef _MTK_ISP_SCP_H
> +#define _MTK_ISP_SCP_H
> +
> +#include <linux/types.h>
> +
> +#include "mtk_cam-v4l2-util.h"
> +
> +/*
> + * struct img_size - image size information.
> + *
> + * @w: image width, the unit is pixel
> + * @h: image height, the unit is pixel
> + * @xsize: bytes per line based on width.
> + * @stride: bytes per line when changing line.
> + *          Normally, calculate new STRIDE based on
> + *          xsize + HW constrain(page or align).
> + *
> + */
> +struct img_size {
> +	__u32 w;
> +	__u32 h;
> +	__u32 xsize;
> +	__u32 stride;
> +} __packed;
> +
> +/*
> + * struct img_buffer - buffer address information.
> + *
> + * @iova: DMA address for external devices.
> + * @scp_addr: SCP address for external co-process unit.
> + *
> + */
> +struct img_buffer {
> +	__u32 iova;
> +	__u32 scp_addr;
> +} __packed;
> +
> +struct p1_img_crop {
> +	__u32 left;
> +	__u32 top;
> +	__u32 width;
> +	__u32 height;
> +} __packed;
> +
> +struct p1_img_output {
> +	struct img_buffer buffer;
> +	struct img_size size;
> +	struct p1_img_crop crop;
> +	__u8 pixel_byte;
> +	__u32 img_fmt;
> +} __packed;

Please document.

> +
> +/*
> + * struct cfg_in_param - image input parameters structure.
> + *                       Normally, it comes from sensor information.
> + *
> + * @continuous: indicate the sensor mode.
> + *              1: continuous
> + *              0: single
> + * @subsample: indicate to enables SOF subsample or not.
> + * @pixel_mode: describe 1/2/4 pixels per clock cycle.
> + * @data_pattern: describe input data pattern.
> + * @raw_pixel_id: bayer sequence.
> + * @tg_fps: the fps rate of TG (time generator).
> + * @img_fmt: the image format of input source.
> + * @p1_img_crop: the crop configuration of input source.
> + *
> + */
> +struct cfg_in_param {
> +	__u8 continuous;
> +	__u8 subsample;
> +	__u8 pixel_mode;
> +	__u8 data_pattern;
> +	__u8 raw_pixel_id;
> +	__u16 tg_fps;
> +	__u32 img_fmt;
> +	struct p1_img_crop crop;
> +} __packed;
> +
> +/*
> + * struct cfg_main_out_param - the image output parameters of main stream.
> + *
> + * @bypass: indicate this device is enabled or disabled or not .

Remove the space before the period.

> + * @pure_raw: indicate the image path control.
> + *            1: pure raw
> + *            0: processing raw
> + * @pure_raw_pack: indicate the image is packed or not.
> + *                 1: packed mode
> + *                 0: unpacked mode
> + * @p1_img_output: the output image information.
> + *
> + */
> +struct cfg_main_out_param {
> +	/* Bypass main out parameters */
> +	__u8 bypass;
> +	/* Control HW image raw path */
> +	__u8 pure_raw;
> +	/* Control HW image pack function */

No need for these inline comments.

> +	__u8 pure_raw_pack;
> +	struct p1_img_output output;
> +} __packed;
> +
> +/*
> + * struct cfg_resize_out_param - the image output parameters of
> + *                               packed out stream.
> + *
> + * @bypass: indicate this device is enabled or disabled or not .

Remove the space before the period.

> + * @p1_img_output: the output image information.
> + *
> + */
> +struct cfg_resize_out_param {
> +	/* Bypass resize parameters */

No need for this inline comment.

> +	__u8 bypass;
> +	struct p1_img_output output;
> +} __packed;
> +
> +/*
> + * struct cfg_meta_out_param - output meta information.
> + *
> + * @enabled_meta_dmas: indicate which meta DMAs are enabled.
> + *
> + */
> +struct cfg_meta_out_param {
> +	__u32 enabled_meta_dmas;
> +} __packed;
> +
> +struct p1_config_param {
> +	/* Sensor/TG info */
> +	struct cfg_in_param cfg_in_param;
> +	/* IMGO DMA */
> +	struct cfg_main_out_param cfg_main_param;
> +	/* RRZO DMA */
> +	struct cfg_resize_out_param cfg_resize_param;
> +	/* 3A DMAs and other. */
> +	struct cfg_meta_out_param cfg_meta_param;

Please change the inline comments to a kerneldoc comment at the top.

> +} __packed;
> +
> +struct p1_frame_param {
> +	/* frame sequence number */
> +	__u32 frame_seq_no;
> +	/* SOF index */
> +	__u32 sof_idx;
> +	/* The memory address of tuning buffer from user space */

Ditto.

> +	struct img_buffer dma_buffers[MTK_CAM_P1_TOTAL_NODES];
> +} __packed;
> +
> +struct P1_meta_frame {
> +	__u32 enabled_dma;
> +	__u32 vb_index;
> +	struct img_buffer meta_addr;
> +} __packed;
> +
> +struct isp_init_info {
> +	__u8 hw_module;
> +	struct img_buffer cq_addr;
> +} __packed;
> +
> +struct isp_ack_info {
> +	__u8 cmd_id;
> +	__u32 frame_seq_no;
> +} __packed;
> +
> +enum mtk_isp_scp_cmds {
> +	ISP_CMD_INIT,
> +	ISP_CMD_CONFIG,
> +	ISP_CMD_STREAM,
> +	ISP_CMD_DEINIT,
> +	ISP_CMD_ACK,
> +	ISP_CMD_FRAME_ACK,
> +	ISP_CMD_CONFIG_META,
> +	ISP_CMD_ENQUEUE_META,
> +	ISP_CMD_RESERVED,
> +};
> +
> +struct mtk_isp_scp_p1_cmd {
> +	__u8 cmd_id;
> +	union {
> +		struct isp_init_info frameparam;
> +		struct p1_config_param config_param;
> +		struct cfg_meta_out_param cfg_meta_out_param;
> +		struct P1_meta_frame meta_frame;
> +		__u8 is_stream_on;
> +		struct isp_ack_info ack_info;
> +	};
> +} __packed;
> +
> +enum mtk_isp_scp_type {
> +	SCP_ISP_CMD = 0,
> +	SCP_ISP_FRAME,
> +};

Please document all the structs and enum above using kerneldoc.

Best regards,
Tomasz


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-10  9:54   ` Tomasz Figa
@ 2019-07-18  4:39     ` Jungo Lin
  2019-07-23 10:21       ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-18  4:39 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi, Tomasz:

On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
> > Implement standard V4L2 video driver that utilizes V4L2
> > and media framework APIs. In this driver, supports one media
> > device, one sub-device and seven video devices during
> > initialization. Moreover, it also connects with sensor and
> > seninf drivers with V4L2 async APIs.
> > 
> > (The current metadata interface used in meta input and partial
> > meta nodes is only a temporary solution to kick off the driver
> > development and is not ready to be reviewed yet.)
> > 
> > Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> > ---
> > This patch depends on "media: support Mediatek sensor interface driver"[1].
> > 
> > ISP P1 sub-device communicates with seninf sub-device with CIO.
> > 
> > [1]. media: support Mediatek sensor interface driver
> > https://patchwork.kernel.org/cover/10979135/
> > ---
> >  .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
> >  .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c    | 1674 +++++++++++++++++
> >  .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h    |  173 ++
> >  3 files changed, 1848 insertions(+)
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
> > 
> 
> Thanks for the patch. Please see my comments inline.
> 
> [snip]
> 

Appreciate your comments on this patch.
Please check my replied inline.

> > +static void mtk_cam_req_try_isp_queue(struct mtk_cam_dev *cam_dev,
> > +				      struct media_request *new_req)
> > +{
> > +	struct mtk_cam_dev_request *req, *req_safe, *cam_dev_req;
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +
> > +	dev_dbg(dev, "%s new req:%d", __func__, !new_req);
> > +
> > +	if (!cam_dev->streaming) {
> > +		cam_dev_req = mtk_cam_req_to_dev_req(new_req);
> > +		spin_lock(&cam_dev->req_lock);
> > +		list_add_tail(&cam_dev_req->list, &cam_dev->req_list);
> > +		spin_unlock(&cam_dev->req_lock);
> > +		dev_dbg(dev, "%s: stream off, no ISP enqueue\n", __func__);
> > +		return;
> > +	}
> > +
> > +	/* Normal enqueue flow */
> > +	if (new_req) {
> > +		mtk_isp_req_enqueue(dev, new_req);
> > +		return;
> > +	}
> > +
> > +	/* Flush all media requests wehen first stream on */
> > +	list_for_each_entry_safe(req, req_safe, &cam_dev->req_list, list) {
> > +		list_del(&req->list);
> > +		mtk_isp_req_enqueue(dev, &req->req);
> > +	}
> > +}
> 
> This will have to be redone, as per the other suggestions, but generally one
> would have a function that tries to queue as much as possible from a list to
> the hardware and another function that adds a request to the list and calls
> the first function.
> 

We revised this function as below.
First to check the en-queue conditions:
a. stream on 
b. The composer buffers in SCP are 3, so we only could has 3 jobs
at the same time.


Second, try to en-queue the frames in the pending job if possible and
move them into running job list if possible.

The request has been inserted into pending job in mtk_cam_req_validate
which is used to validate media_request.

void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam_dev)
{
	struct mtk_cam_dev_request *req, *req_prev;
	struct list_head enqueue_job_list;
	int buffer_cnt = atomic_read(&cam_dev->running_job_count);
	unsigned long flags;

	if (!cam_dev->streaming ||
	    buffer_cnt >= MTK_ISP_MAX_RUNNING_JOBS) {
		dev_dbg(cam_dev->dev, "stream off or buffers are full:%d\n",
			buffer_cnt);
		return;
	}

	INIT_LIST_HEAD(&enqueue_job_list);

	spin_lock(&cam_dev->pending_job_lock);
	list_for_each_entry_safe(req, req_prev,
				 &cam_dev->pending_job_list, list) {
		list_del(&req->list);
		list_add_tail(&req->list, &enqueue_job_list);
		if (atomic_inc_return(&cam_dev->running_job_count) >=
			MTK_ISP_MAX_RUNNING_JOBS)
			break;
	}
	spin_unlock(&cam_dev->pending_job_lock);

	list_for_each_entry_safe(req, req_prev,
				 &enqueue_job_list, list) {
		list_del(&req->list);
		spin_lock_irqsave(&cam_dev->running_job_lock, flags);
		list_add_tail(&req->list, &cam_dev->running_job_list);
		spin_unlock_irqrestore(&cam_dev->running_job_lock, flags);

		mtk_isp_req_enqueue(cam_dev, req);
	}
}


> > +
> > +static void mtk_cam_req_queue(struct media_request *req)
> > +{
> > +	struct mtk_cam_dev *cam_dev = mtk_cam_mdev_to_dev(req->mdev);
> > +
> > +	vb2_request_queue(req);
> > +	mtk_cam_req_try_isp_queue(cam_dev, req);
> 
> Looks like this driver is suffering from versy similar problems in request
> handling as the DIP driver used to.
> 
> I'd prefer to save my time and avoid repeating the same comments, so please
> check my comments for the DIP driver and apply them to this one too:
> 
> https://patchwork.kernel.org/patch/10905223/
> 

Yes, we will follow the same design of DIP and replace this function by
vb2_request_queue and defined new request structure. 

/*
 * struct mtk_cam_dev_request - MTK camera device request.
 *
 * @req: Embedded struct media request.
 * @frame_params: The frame info. & address info. of enabled DMA nodes.
 * @frame_work: work queue entry for frame transmission to SCP.
 * @list: List entry of the object for @struct mtk_cam_dev:
 *        pending_job_list or running_job_list.
 * @buf_count: Buffer count in this request.
 *
 */
struct mtk_cam_dev_request {
	struct media_request req;
	struct mtk_p1_frame_param frame_params;
	struct work_struct frame_work;
	struct list_head list;
	atomic_t buf_count;
};


> > +}
> > +
> > +static struct media_request *mtk_cam_req_alloc(struct media_device *mdev)
> > +{
> > +	struct mtk_cam_dev_request *cam_dev_req;
> > +
> > +	cam_dev_req = kzalloc(sizeof(*cam_dev_req), GFP_KERNEL);
> > +
> > +	return &cam_dev_req->req;
> > +}
> > +
> > +static void mtk_cam_req_free(struct media_request *req)
> > +{
> > +	struct mtk_cam_dev_request *cam_dev_req = mtk_cam_req_to_dev_req(req);
> > +
> > +	kfree(cam_dev_req);
> > +}
> > +
> > +static __u32 img_get_pixel_byte_by_fmt(__u32 pix_fmt)
> 
> Doesn't this function return bits not bytes?
> 

Yes, the unit sould be bits, not bytes.
We will rename this function to get_pixel_bits.

static unsigned int get_pixel_bits(unsigned int pix_fmt)

> > +{
> > +	switch (pix_fmt) {
> > +	case V4L2_PIX_FMT_MTISP_B8:
> > +	case V4L2_PIX_FMT_MTISP_F8:
> > +		return 8;
> > +	case V4L2_PIX_FMT_MTISP_B10:
> > +	case V4L2_PIX_FMT_MTISP_F10:
> > +		return 10;
> > +	case V4L2_PIX_FMT_MTISP_B12:
> > +	case V4L2_PIX_FMT_MTISP_F12:
> > +		return 12;
> > +	case V4L2_PIX_FMT_MTISP_B14:
> > +	case V4L2_PIX_FMT_MTISP_F14:
> > +		return 14;
> > +	default:
> > +		return 0;
> > +	}
> > +}
> > +
> > +static __u32 img_cal_main_stream_stride(struct device *dev, __u32 width,
> > +					__u32 pix_fmt)
> > +{
> > +	__u32 stride;
> > +	__u32 pixel_byte = img_get_pixel_byte_by_fmt(pix_fmt);
> 
> The __ prefixed types should be used only inside UAPI. Please change the
> driver to use the normal ones.
> 

Ok, we will fix our usage in our driver source code.

> > +
> > +	width = ALIGN(width, 4);
> 
> If there is some alignment requirement for width, it should be handled by
> TRY_/S_FMT and here we should already assume everything properly aligned.
> 

We will follow your suggestion to move this code login in TRY_/S_FMT
functions.

> > +	stride = ALIGN(DIV_ROUND_UP(width * pixel_byte, 8), 2);
> > +
> > +	dev_dbg(dev, "main width:%d, stride:%d\n", width, stride);
> > +
> > +	return stride;
> > +}
> > +
> > +static __u32 img_cal_packed_out_stride(struct device *dev, __u32 width,
> > +				       __u32 pix_fmt)
> > +{
> > +	__u32 stride;
> > +	__u32 pixel_byte = img_get_pixel_byte_by_fmt(pix_fmt);
> > +
> > +	width = ALIGN(width, 4);
> 
> Ditto.
> 

Will fix.

> > +	stride = DIV_ROUND_UP(width * 3, 2);
> 
> Could we introduce a local variable for this intermediate value, so that its
> name could explain what the value is?
> 
> > +	stride = DIV_ROUND_UP(stride * pixel_byte, 8);
> > +
> > +	if (pix_fmt == V4L2_PIX_FMT_MTISP_F10)
> > +		stride = ALIGN(stride, 4);
> 
> Is it expected that only the F10 format needs this alignment?
> 

yes, if the pixel bits of image format is 10, the byte alignment of bpl
should be 4. Otherwise, it is 8. We will revise this and add more
comments.

/* 4 bytes alignment for 10 bit other are 8 bytes alignment */
	if (pixel_bits == 10)
		bpl = ALIGN(bpl, 4);
	else
		bpl = ALIGN(bpl, 8);

> > +
> > +	dev_dbg(dev, "packed width:%d, stride:%d\n", width, stride);
> > +
> > +	return stride;
> > +}
> > +
> > +static __u32 img_cal_stride(struct device *dev,
> > +			    int node_id,
> > +			    __u32 width,
> > +			    __u32 pix_fmt)
> > +{
> > +	__u32 bpl;
> > +
> > +	/* Currently, only support one_pixel_mode */
> > +	if (node_id == MTK_CAM_P1_MAIN_STREAM_OUT)
> > +		bpl = img_cal_main_stream_stride(dev, width, pix_fmt);
> > +	else if (node_id == MTK_CAM_P1_PACKED_BIN_OUT)
> > +		bpl = img_cal_packed_out_stride(dev, width, pix_fmt);
> > +
> > +	/* For DIP HW constrained, it needs 4 byte alignment */
> > +	bpl = ALIGN(bpl, 4);
> > +
> > +	return bpl;
> > +}
> > +
> > +static const struct v4l2_format *
> > +mtk_cam_dev_find_fmt(struct mtk_cam_dev_node_desc *desc, u32 format)
> > +{
> > +	unsigned int i;
> > +	const struct v4l2_format *dev_fmt;
> > +
> > +	for (i = 0; i < desc->num_fmts; i++) {
> > +		dev_fmt = &desc->fmts[i];
> > +		if (dev_fmt->fmt.pix_mp.pixelformat == format)
> > +			return dev_fmt;
> > +	}
> > +
> > +	return NULL;
> > +}
> > +
> > +/* Calcuate mplane pix format */
> > +static void
> > +mtk_cam_dev_cal_mplane_fmt(struct device *dev,
> > +			   struct v4l2_pix_format_mplane *dest_fmt,
> > +			   unsigned int node_id)
> > +{
> > +	unsigned int i;
> > +	__u32 bpl, sizeimage, imagsize;
> 
> Perhaps s/sizeimage/plane_size/ and s/imagsize/total_size/?
> 

Has fixed it.
Btw we will only support 1 plane, there is no need for plane_size.

> > +
> > +	imagsize = 0;
> > +	for (i = 0 ; i < dest_fmt->num_planes; ++i) {
> > +		bpl = img_cal_stride(dev,
> > +				     node_id,
> > +				     dest_fmt->width,
> > +				     dest_fmt->pixelformat);
> > +		sizeimage = bpl * dest_fmt->height;
> > +		imagsize += sizeimage;
> > +		dest_fmt->plane_fmt[i].bytesperline = bpl;
> > +		dest_fmt->plane_fmt[i].sizeimage = sizeimage;
> > +		memset(dest_fmt->plane_fmt[i].reserved,
> > +		       0, sizeof(dest_fmt->plane_fmt[i].reserved));
> 
> This memset is not needed. The core clears the reserved fields
> automatically:
> 
> https://elixir.bootlin.com/linux/v5.2/source/drivers/media/v4l2-core/v4l2-ioctl.c#L1559
> 
> (We may want to backport the patch that added that to our 4.19 branch.)
> 

Ok, we will remove both memset functions in this function.


> > +		dev_dbg(dev, "plane:%d,bpl:%d,sizeimage:%u\n",
> > +			i,  bpl, dest_fmt->plane_fmt[i].sizeimage);
> > +	}
> > +
> > +	if (dest_fmt->num_planes == 1)
> > +		dest_fmt->plane_fmt[0].sizeimage = imagsize;
> 
> Hmm, we only seem to support 1 plane raw formats in this driver. Does the
> hardware support any formats with more than 1 plane? If not, all the code
> should be simplified to just assume 1 plane.
> 

No, MTK P1 ISP HW only supports raw formats with 1 plane.
We will revise our source codes to support only 1 plane.

> > +}
> > +
> > +static void
> > +mtk_cam_dev_set_img_fmt(struct device *dev,
> > +			struct v4l2_pix_format_mplane *dest_fmt,
> > +			const struct v4l2_pix_format_mplane *src_fmt,
> > +			unsigned int node_id)
> > +{
> > +	dest_fmt->width = src_fmt->width;
> > +	dest_fmt->height = src_fmt->height;
> > +	dest_fmt->pixelformat = src_fmt->pixelformat;
> > +	dest_fmt->field = src_fmt->field;
> > +	dest_fmt->colorspace = src_fmt->colorspace;
> > +	dest_fmt->num_planes = src_fmt->num_planes;
> > +	/* Use default */
> > +	dest_fmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > +	dest_fmt->quantization = V4L2_QUANTIZATION_DEFAULT;
> > +	dest_fmt->xfer_func =
> > +		V4L2_MAP_XFER_FUNC_DEFAULT(dest_fmt->colorspace);
> > +	memset(dest_fmt->reserved, 0, sizeof(dest_fmt->reserved));
> 
> Given that src_fmt should already be validated and have any fields adjusted
> to match the driver requirements, wouldn't all the lines above be equivalent
> to *dest_fmt = *src_fmt?
> 
> We probably want to move setting all the constant fields to
> mtk_cam_vidioc_try_fmt().
> 

Ok, we will remove mtk_cam_dev_set_img_fmt function
and use *dest_fmt = *src_fmt directly in the caller.
Moreover, all the constant fields are moved to mtk_cam_vidioc_try_fmt()
function.

> > +
> > +	dev_dbg(dev, "%s: Dest Fmt:%c%c%c%c, w*h:%d*%d\n",
> > +		__func__,
> > +		(dest_fmt->pixelformat & 0xFF),
> > +		(dest_fmt->pixelformat >> 8) & 0xFF,
> > +		(dest_fmt->pixelformat >> 16) & 0xFF,
> > +		(dest_fmt->pixelformat >> 24) & 0xFF,
> > +		dest_fmt->width,
> > +		dest_fmt->height);
> > +
> > +	mtk_cam_dev_cal_mplane_fmt(dev, dest_fmt, node_id);
> 
> This should have been called already before this function was called,
> because src_fmt should be already expected to contain valid settings. In
> fact, this is already called in mtk_cam_vidioc_try_fmt().
> 

Ok, we will revise this.

> > +}
> > +
> > +/* Get the default format setting */
> > +static void
> > +mtk_cam_dev_load_default_fmt(struct device *dev,
> 
> Please don't pass struct device pointer around, but instead just the main
> driver data struct, which should be much more convenient for accessing
> various driver data. Please fix the other functions as well.
> 

Ok, we will revise this coding style in our source codes.

> > +			     struct mtk_cam_dev_node_desc *queue_desc,
> > +			     struct v4l2_format *dest)
> > +{
> > +	const struct v4l2_format *default_fmt =
> > +		&queue_desc->fmts[queue_desc->default_fmt_idx];
> > +
> > +	dest->type = queue_desc->buf_type;
> > +
> > +	/* Configure default format based on node type */
> > +	if (queue_desc->image) {
> > +		mtk_cam_dev_set_img_fmt(dev,
> > +					&dest->fmt.pix_mp,
> > +					&default_fmt->fmt.pix_mp,
> > +					queue_desc->id);
> 
> We should probably just call mtk_cam_vidioc_s_fmt() here, with a dummy
> v4l2_format struct and have any incorrect fields replaced by
> mtk_cam_vidioc_try_fmt(), since it's the same logic, as if setting invalid
> v4l2_format at runtime.
> 

Ok, we will revise this.

> > +	} else {
> > +		dest->fmt.meta.dataformat = default_fmt->fmt.meta.dataformat;
> > +		dest->fmt.meta.buffersize = default_fmt->fmt.meta.buffersize;
> > +	}
> > +}
> > +
> > +static int mtk_cam_isp_open(struct file *file)
> > +{
> > +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +	int ret;
> > +
> > +	mutex_lock(&cam_dev->lock);
> > +	ret = v4l2_fh_open(file);
> > +	if (ret)
> > +		goto unlock;
> > +
> > +	ret = v4l2_pipeline_pm_use(&node->vdev.entity, 1);
> 
> Please don't power on open. Normally applications keep the device nodes open
> all the time, so they would keep everything powered on.
> 
> Normally this should be done as late as possible, ideally when starting the
> streaming.
> 

Ok, we will remove this function and just call 4l2_fh_open(file)
function.

> > +	if (ret)
> > +		dev_err(dev, "%s fail:%d", __func__, ret);
> > +
> > +unlock:
> > +	mutex_unlock(&cam_dev->lock);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mtk_cam_isp_release(struct file *file)
> > +{
> > +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +
> > +	mutex_lock(&cam_dev->lock);
> > +	v4l2_pipeline_pm_use(&node->vdev.entity, 0);
> > +	vb2_fop_release(file);
> > +	mutex_unlock(&cam_dev->lock);
> > +
> > +	return 0;
> > +}
> 
> If we remove power handling from open and release, we should be able to just
> use v4l2_fh_open() and vb2_fop_release() directly in the
> v4l2_file_operations struct.
> 

Ok, we will fix this.

> > +
> > +static struct v4l2_subdev *
> > +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> > +{
> > +	struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> > +	struct media_entity *entity;
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	struct v4l2_subdev *sensor;
> 
> This variable would be unitialized if there is no streaming sensor. Was
> there no compiler warning generated for this?
> 

No, there is no compiler warning.
But, we will assign sensor to NULL to avoid unnecessary compiler warning
with different compiler options.

> > +
> > +	media_device_for_each_entity(entity, mdev) {
> > +		dev_dbg(dev, "media entity: %s:0x%x\n",
> > +			entity->name, entity->function);
> > +		if (entity->function == MEDIA_ENT_F_CAM_SENSOR &&
> > +		    entity->stream_count) {
> > +			sensor = media_entity_to_v4l2_subdev(entity);
> > +			dev_dbg(dev, "Sensor found: %s\n", entity->name);
> > +			break;
> > +		}
> > +	}
> > +
> > +	if (!sensor)
> > +		dev_err(dev, "Sensor is not connected\n");
> > +
> > +	return sensor;
> > +}
> > +
> > +static int mtk_cam_cio_stream_on(struct mtk_cam_dev *cam_dev)
> > +{
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	int ret;
> > +
> > +	/* Align vb2_core_streamon design */
> > +	if (cam_dev->streaming) {
> > +		dev_warn(dev, "already streaming\n", dev);
> > +		return 0;
> > +	}
> 
> Could we check this in the caller?
> 

Ok, we will move this logic check in the mtk_cam_sd_s_stream function.

> > +
> > +	if (!cam_dev->seninf) {
> > +		dev_err(dev, "no seninf connected:%d\n", ret);
> > +		return -EPERM;
> 
> I don't think -EPERM is a good error code here. It's about a missing seninf
> device, so perhaps -ENODEV?
> 

Fix it in next patch.

> > +	}
> > +
> > +	/* Get active sensor from graph topology */
> > +	cam_dev->sensor = mtk_cam_cio_get_active_sensor(cam_dev);
> > +	if (!cam_dev->sensor)
> > +		return -EPERM;
> 
> > -ENODEV
> 
> > +
> > +	ret = mtk_isp_config(dev);
> > +	if (ret)
> > +		return -EPERM;
> 
> Maybe just return ret?
> 

Fix it in next patch.

> > +
> > +	/* Seninf must stream on first */
> > +	ret = v4l2_subdev_call(cam_dev->seninf, video, s_stream, 1);
> > +	if (ret) {
> > +		dev_err(dev, "%s stream on failed:%d\n",
> > +			cam_dev->seninf->entity.name, ret);
> > +		return -EPERM;
> 
> return ret?
> 

Fix it in next patch.

> > +	}
> > +
> > +	ret = v4l2_subdev_call(cam_dev->sensor, video, s_stream, 1);
> > +	if (ret) {
> > +		dev_err(dev, "%s stream on failed:%d\n",
> > +			cam_dev->sensor->entity.name, ret);
> > +		goto fail_sensor_on;
> > +	}
> > +
> > +	cam_dev->streaming = true;
> > +	mtk_cam_req_try_isp_queue(cam_dev, NULL);
> > +	isp_composer_stream(dev, 1);
> > +	dev_dbg(dev, "streamed on Pass 1\n");
> > +
> > +	return 0;
> > +
> > +fail_sensor_on:
> > +	v4l2_subdev_call(cam_dev->seninf, video, s_stream, 0);
> > +
> > +	return -EPERM;
> 
> return ret?
> 

Fix it in next patch.

> > +}
> > +
> > +static int mtk_cam_cio_stream_off(struct mtk_cam_dev *cam_dev)
> > +{
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	int ret;
> > +
> > +	if (!cam_dev->streaming) {
> > +		dev_warn(dev, "already stream off");
> > +		return 0;
> > +	}
> 
> Could we check this in the caller?
> 

Ditto.

> > +
> > +	ret = v4l2_subdev_call(cam_dev->sensor, video, s_stream, 0);
> > +	if (ret) {
> > +		dev_err(dev, "%s stream off failed:%d\n",
> > +			cam_dev->sensor->entity.name, ret);
> > +		return -EPERM;
> > +	}
> > +
> > +	ret = v4l2_subdev_call(cam_dev->seninf, video, s_stream, 0);
> > +	if (ret) {
> > +		dev_err(dev, "%s stream off failed:%d\n",
> > +			cam_dev->seninf->entity.name, ret);
> > +		return -EPERM;
> > +	}
> > +
> > +	isp_composer_stream(dev, 0);
> 
> Shouldn't we synchronously wait for the streaming to stop here? Otherwise we
> can't guarantee that the hardware releases all the memory that we're going
> to free once this function returns.
> 

We will add  below functions.
1. Stream off ISP HW
2. Stop ISP HW
3. Clear all pending & running request lists.

	cam_dev->streaming = false;
	mtk_isp_stream(cam_dev, 0);
	mtk_isp_hw_release(cam_dev);
	mtk_cam_dev_req_clear(cam_dev);

	dev_dbg(dev, "streamed off Pass 1\n");

> > +	cam_dev->streaming = false;
> > +	dev_dbg(dev, "streamed off Pass 1\n");
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_sd_s_stream(struct v4l2_subdev *sd, int enable)
> > +{
> > +	struct mtk_cam_dev *cam_dev = mtk_cam_subdev_to_dev(sd);
> > +
> > +	if (enable)
> > +		return mtk_cam_cio_stream_on(cam_dev);
> > +	else
> > +		return mtk_cam_cio_stream_off(cam_dev);
> > +}
> > +
> > +static int mtk_cam_sd_subscribe_event(struct v4l2_subdev *subdev,
> > +				      struct v4l2_fh *fh,
> > +				      struct v4l2_event_subscription *sub)
> > +{
> > +	switch (sub->type) {
> > +	case V4L2_EVENT_FRAME_SYNC:
> > +		return v4l2_event_subscribe(fh, sub, 0, NULL);
> > +	default:
> > +		return -EINVAL;
> > +	}
> > +}
> > +
> > +static int mtk_cam_sd_s_power(struct v4l2_subdev *sd, int on)
> > +{
> > +	struct mtk_cam_dev *cam_dev = mtk_cam_subdev_to_dev(sd);
> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "%s:%d", __func__, on);
> > +
> > +	return on ? mtk_isp_power_init(cam_dev) :
> > +		    mtk_isp_power_release(&cam_dev->pdev->dev);
> 
> s_power is a historical thing and we shouldn't be implementing it. Instead,
> we should use runtime PM and call pm_runtime_get_sync(), pm_runtime_put()
> whenever we start and stop streaming respectively.
> 

Ok, we will remove this callback function.

> > +}
> > +
> > +static int mtk_cam_media_link_setup(struct media_entity *entity,
> > +				    const struct media_pad *local,
> > +				    const struct media_pad *remote, u32 flags)
> > +{
> > +	struct mtk_cam_dev *cam_dev =
> > +		container_of(entity, struct mtk_cam_dev, subdev.entity);
> > +	u32 pad = local->index;
> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "%s: %d -> %d flags:0x%x\n",
> > +		__func__, pad, remote->index, flags);
> > +
> > +	if (pad < MTK_CAM_P1_TOTAL_NODES)
> 
> I assume this check is needed, because the pads with higher indexes are not
> video nodes? If so, a comment would be helpful here.
> 

Yes, we will new comment as below.

	/*
	 * Check video nodes is enabled by link setup.
	 * The pad index of video node should be less than       
         * MTK_CAM_P1_TOTAL_NODES.
	 */
	if (pad < MTK_CAM_P1_TOTAL_NODES)
		cam_dev->vdev_nodes[pad].enabled =
			!!(flags & MEDIA_LNK_FL_ENABLED);

> > +		cam_dev->vdev_nodes[pad].enabled =
> > +			!!(flags & MEDIA_LNK_FL_ENABLED);
> > +
> > +	return 0;
> > +}
> > +
> > +static void mtk_cam_vb2_buf_queue(struct vb2_buffer *vb)
> > +{
> > +	struct mtk_cam_dev *mtk_cam_dev = vb2_get_drv_priv(vb->vb2_queue);
> > +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
> > +	struct device *dev = &mtk_cam_dev->pdev->dev;
> > +	struct mtk_cam_dev_buffer *buf;
> > +
> > +	buf = mtk_cam_vb2_buf_to_dev_buf(vb);
> 
> This can be folded into the declaration.
> 

Fix it in next patch.

> > +
> > +	dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > +		__func__,
> > +		node->id,
> > +		buf->vbb.request_fd,
> > +		buf->vbb.vb2_buf.index);
> > +
> > +	/* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > +	if (vb->vb2_queue->uses_requests)
> > +		return;
> 
> I'd suggest removing non-request support from this driver. Even if we end up
> with a need to provide compatibility for non-request mode, then it should be
> built on top of the requests mode, so that the driver itself doesn't have to
> deal with two modes.
> 

The purpose of non-request function in this driver is needed by
our camera middle-ware design. It needs 3A statistics buffers before
image buffers en-queue. So we need to en-queue 3A statistics with
non-request mode in this driver. After MW got the 3A statistics data, it
will en-queue the images, tuning buffer and other meta buffers with
request mode. Based on this requirement, do you have any suggestion?
For upstream driver, should we only consider request mode?

> > +
> > +	/* Added the buffer into the tracking list */
> > +	spin_lock(&node->slock);
> > +	list_add_tail(&buf->list, &node->pending_list);
> > +	spin_unlock(&node->slock);
> > +
> > +	mtk_isp_enqueue(dev, node->desc.dma_port, buf);
> > +}
> > +
> > +static int mtk_cam_vb2_buf_init(struct vb2_buffer *vb)
> > +{
> > +	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vb->vb2_queue);
> > +	struct device *smem_dev = cam_dev->smem_dev;
> > +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
> > +	struct mtk_cam_dev_buffer *buf;
> > +
> > +	buf = mtk_cam_vb2_buf_to_dev_buf(vb);
> > +	buf->node_id = node->id;
> > +	buf->daddr = vb2_dma_contig_plane_dma_addr(&buf->vbb.vb2_buf, 0);
> > +	buf->scp_addr = 0;
> 
> Just a reminder that this will have to be reworked according to my comments
> for the memory allocation patch.
> 

Yes, we have revised this implementation according to the review of
below patch set.

https://patchwork.kernel.org/patch/10985833/

> > +
> > +	/* scp address is only valid for meta input buffer */
> > +	if (node->desc.smem_alloc)
> > +		buf->scp_addr = mtk_cam_smem_iova_to_scp_addr(smem_dev,
> > +							      buf->daddr);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vb2_buf_prepare(struct vb2_buffer *vb)
> > +{
> > +	struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb);
> > +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vb->vb2_queue);
> > +	const struct v4l2_format *fmt = &node->vdev_fmt;
> > +	unsigned int size;
> > +
> > +	if (vb->vb2_queue->type == V4L2_BUF_TYPE_META_OUTPUT ||
> > +	    vb->vb2_queue->type == V4L2_BUF_TYPE_META_CAPTURE)
> > +		size = fmt->fmt.meta.buffersize;
> > +	else
> > +		size = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
> > +
> > +	if (vb2_plane_size(vb, 0) < size)
> > +		return -EINVAL;
> 
> For OUTPUT buffers we need to check if vb2_get_plane_payload() == size.
> Otherwise we could get not enough or invalid data.
> 

Fixed in next patch.

> > +
> > +	v4l2_buf->field = V4L2_FIELD_NONE;
> > +	vb2_set_plane_payload(vb, 0, size);
> 
> This shouldn't be called on OUTPUT buffers.
> 

Ditto.

> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vb2_queue_setup(struct vb2_queue *vq,
> > +				   unsigned int *num_buffers,
> > +				   unsigned int *num_planes,
> > +				   unsigned int sizes[],
> > +				   struct device *alloc_devs[])
> > +{
> > +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> > +	unsigned int max_buffer_count = node->desc.max_buf_count;
> > +	const struct v4l2_format *fmt = &node->vdev_fmt;
> > +	unsigned int size;
> > +
> > +	/* Check the limitation of buffer size */
> > +	if (max_buffer_count)
> > +		*num_buffers = clamp_val(*num_buffers, 1, max_buffer_count);
> > +
> > +	if (vq->type == V4L2_BUF_TYPE_META_OUTPUT ||
> > +	    vq->type == V4L2_BUF_TYPE_META_CAPTURE)
> > +		size = fmt->fmt.meta.buffersize;
> > +	else
> > +		size = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
> > +
> > +	/* Add for q.create_bufs with fmt.g_sizeimage(p) / 2 test */
> > +	if (*num_planes) {
> 
> We should also verify that *num_planes == 1, as we don't support more
> planes in this driver.
> 

Ok, here is new check logic.

	if (*num_planes) {
		if (sizes[0] < size || *num_planes != 1)
			return -EINVAL;
	} else {
		*num_planes = 1;
		sizes[0] = size;
	}


> > +		if (sizes[0] < size)
> > +			return -EINVAL;
> > +	} else {
> > +		*num_planes = 1;
> > +		sizes[0] = size;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static void mtk_cam_vb2_return_all_buffers(struct mtk_cam_dev *cam_dev,
> > +					   struct mtk_cam_video_device *node,
> > +					   enum vb2_buffer_state state)
> > +{
> > +	struct mtk_cam_dev_buffer *b, *b0;
> > +	struct mtk_cam_dev_request *req, *req0;
> > +	struct media_request_object *obj, *obj0;
> > +	struct vb2_buffer *vb;
> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "%s: node:%s", __func__, node->vdev.name);
> > +
> > +	/* Return all buffers */
> > +	spin_lock(&node->slock);
> > +	list_for_each_entry_safe(b, b0, &node->pending_list, list) {
> 
> nit: One would normally call the second argument "prev", or "b_prev".
> 

Ok, we will follow this coding convention in our source codes. 

> > +		vb = &b->vbb.vb2_buf;
> > +		if (vb->state == VB2_BUF_STATE_ACTIVE)
> 
> We shouldn't need to check the buffer state.
> 

Fixed in next patch.

> > +			vb2_buffer_done(vb, state);
> > +		list_del(&b->list);
> > +	}
> > +	spin_unlock(&node->slock);
> > +
> > +	spin_lock(&cam_dev->req_lock);
> > +	list_for_each_entry_safe(req, req0, &cam_dev->req_list, list) {
> 
> nit: Ditto.
> 

Fixed in next patch.

> > +		list_for_each_entry_safe(obj, obj0, &req->req.objects, list) {
> 
> Need to check if the object is a buffer.
> 

Fixed in next patch.

> > +			vb = container_of(obj, struct vb2_buffer, req_obj);
> > +			if (vb->state == VB2_BUF_STATE_ACTIVE)
> 
> vb->state shouldn't be accessed directly from the drivers.
> 

nit: Ditto.

> Generally, the need to check the state here would suggest that there is
> something wrong with how the driver manages the requests. The list that is
> being iterated here shouldn't contain any requests that have buffers that
> aren't active. That will be achieved if my comments for the request handling
> in the DIP driver are applied to this driver as well.
> 
> > +				vb2_buffer_done(vb, state);
> > +		}
> > +		list_del(&req->list);
> > +	}
> > +	spin_unlock(&cam_dev->req_lock);
> > +
> > +	if (node->vbq.uses_requests)
> > +		mtk_isp_req_flush_buffers(&cam_dev->pdev->dev);
> > +}
> > +
> > +static int mtk_cam_vb2_start_streaming(struct vb2_queue *vq,
> > +				       unsigned int count)
> > +{
> > +	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
> > +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	unsigned int node_count = cam_dev->subdev.entity.use_count;
> > +	int ret;
> > +
> > +	if (!node->enabled) {
> 
> How is this synchronized with mtk_cam_media_link_setup()?
> 

We will follow your suggestion and below is our proposal for this
function.

1. Use !cam_dev->pipeline.streaming_count to decide the first node to
stream-on.
2.a If yes, do the following steps
    2.a-1 Call media_pipeline_start function to prevent the link
configuration changes.
    2.a-2 Call mtk_cam_dev_init_stream function to calculate how many
video nodes are enabled and save it into cam_dev->enabled_node_count.
    2.a-3 Initialize ISP P1 HW in mtk_isp_hw_init function since end
user has called stream-on API
2.b jump step 3.

3. Use cam_dev->streamed_node_count to track how many video nodes are
streamed by user space.
4. Check all enabled video nodes are streamed or not based on
cam_dev->streamed_node_count & cam_dev->enabled_node_count.
5. If yes, call s_stream on for P1 sub-device

Do you think it is reasonable?

> > +		dev_err(dev, "Node:%d is not enable\n", node->id);
> > +		ret = -ENOLINK;
> > +		goto fail_no_link;
> > +	}
> > +
> > +	dev_dbg(dev, "%s: count info:%d:%d", __func__,
> > +		atomic_read(&cam_dev->streamed_node_count), node_count);
> > +
> > +	if (atomic_inc_return(&cam_dev->streamed_node_count) < node_count)
> > +		return 0;
> 
> How do we guarantee that cam_dev->subdev.entity.use_count doesn't change
> between calls to this function on different video nodes?
> 

Ditto.

> > +
> > +	/* Start streaming of the whole pipeline now */
> > +	ret = media_pipeline_start(&node->vdev.entity, &cam_dev->pipeline);
> > +	if (ret) {
> > +		dev_err(dev, "%s: Node:%d failed\n", __func__, node->id);
> > +		goto fail_start_pipeline;
> > +	}
> > +
> 
> Related to the above comment: If we start the media pipeline when we start
> streaming on the first node, we would naturally prevent the link
> configuration changes until the last node stops streaming (as long as the
> link is not DYNAMIC). Note that it would only mark the entities as
> streaming, but it wouldn't call their s_stream, which I believe is exactly
> what we would need to solve the problem above.
> 

Ditto.

> > +	/* Stream on sub-devices node */
> > +	ret = v4l2_subdev_call(&cam_dev->subdev, video, s_stream, 1);
> > +	if (ret) {
> > +		dev_err(dev, "Node:%d s_stream on failed:%d\n", node->id, ret);
> > +		goto fail_stream_on;
> > +	}
> > +
> > +	return 0;
> > +
> > +fail_stream_on:
> > +	media_pipeline_stop(&node->vdev.entity);
> > +fail_start_pipeline:
> > +	atomic_dec(&cam_dev->streamed_node_count);
> > +fail_no_link:
> > +	mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_QUEUED);
> > +
> > +	return ret;
> > +}
> > +
> > +static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
> > +{
> > +	struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
> > +	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +
> > +	if (!node->enabled)
> > +		return;
> 
> It shouldn't be possible for this to happen, because nobody could have
> called start_streaming on a disabled node.
> 

Will remove in next patch.

> > +
> > +	mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_ERROR);
> 
> Shouldn't we stop streaming first, so that the hardware operation is
> cancelled and any buffers owned by the hardware are released?
> 

For this function, below is the new code flow.

1. Check the first node to stream off based on 
cam_dev->streamed_node_count & cam_dev->enabled_node_count.
2. If yes, call all s_stream off for P1 sub-device
3. Call mtk_cam_vb2_return_all_buffers for each node
4. Check the last node to stream off
5. If yes, call media_pipeline_stop to allow user space
to perform link configuration changes, such as disable link.

But, for step 5, is it too late for end user to disable link?
For example, for first node, it has called stream off but
can't call disable link until the last node is stream off?

> > +
> > +	dev_dbg(dev, "%s: count info:%d", __func__,
> > +		cam_dev->subdev.entity.stream_count);
> > +
> > +	/* Check the first node to stream-off */
> > +	if (!cam_dev->subdev.entity.stream_count)
> > +		return;
> > +
> > +	media_pipeline_stop(&node->vdev.entity);
> > +
> > +	if (v4l2_subdev_call(&cam_dev->subdev, video, s_stream, 0))
> > +		dev_err(dev, "failed to stop streaming\n");
> > +}
> > +
> > +static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
> > +{
> > +	struct mtk_cam_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
> > +
> > +	v4l2_ctrl_request_complete(vb->req_obj.req,
> > +				   dev->v4l2_dev.ctrl_handler);
> 
> This would end up being called multiple times, once for each video node.
> Instead, this should be called explicitly by the driver when it completed
> the request - perhaps in the frame completion handler?
> 
> With that, we probably wouldn't even need this callback.
> 

First, if we don't implement this callback function, we will receive
kernel warning as below.

https://elixir.bootlin.com/linux/latest/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L420

Second, this function is only be called in __vb2_queue_cancel function.
Moreover, we will remove cam_dev->v4l2_dev.ctrl_handler in next patch.
So could we just implement dummy empty function?

 * @buf_request_complete: a buffer that was never queued to the driver
but is
 *			associated with a queued request was canceled.
 *			The driver will have to mark associated objects in the
 *			request as completed; required if requests are
 *			supported.


> > +}
> > +
> > +static int mtk_cam_vidioc_querycap(struct file *file, void *fh,
> > +				   struct v4l2_capability *cap)
> > +{
> > +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> > +
> > +	strscpy(cap->driver, MTK_CAM_DEV_P1_NAME, sizeof(cap->driver));
> > +	strscpy(cap->card, MTK_CAM_DEV_P1_NAME, sizeof(cap->card));
> 
> We could just use dev_driver_name(cam_dev->dev) for both.
> 
> > +	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
> > +		 dev_name(cam_dev->media_dev.dev));
> 
> We should just store dev in cam_dev.
> 

Will fix in next patch.

> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_enum_fmt(struct file *file, void *fh,
> > +				   struct v4l2_fmtdesc *f)
> > +{
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +
> > +	if (f->index >= node->desc.num_fmts)
> > +		return -EINVAL;
> > +
> > +	f->pixelformat = node->desc.fmts[f->index].fmt.pix_mp.pixelformat;
> 
> Is the set of formats available always the same regardless of the sensor
> format?
> 

Yes, ISP P1 HW output formats are always available without impact
by sensor formats. 

> > +	f->flags = 0;
> 
> We need f->description too.
> 

For this description, do you suggest 1). we fill this field in this
function or 2). v4l_fill_fmtdesc function in v4l2-ioctl?

https://elixir.bootlin.com/linux/latest/source/drivers/media/v4l2-core/v4l2-ioctl.c#L1152

Basically, we prefer method 1.

> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_g_fmt(struct file *file, void *fh,
> > +				struct v4l2_format *f)
> > +{
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +
> > +	if (!node->desc.num_fmts)
> > +		return -EINVAL;
> 
> When would that condition happen?
> 

Will remove this in next patch.

> > +
> > +	f->fmt = node->vdev_fmt.fmt;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_try_fmt(struct file *file, void *fh,
> > +				  struct v4l2_format *in_fmt)
> > +{
> > +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +	const struct v4l2_format *dev_fmt;
> > +	__u32  width, height;
> 
> Don't use __ types in implementation, they are here for UAPI purposes. There
> is u32, which you could use instead, but for width and height you don't need
> explicit size, so unsigned int should be good.
> 

Ok, we will revise this in next patch.

> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "%s: fmt:%c%c%c%c, w*h:%u*%u\n",
> > +		__func__,
> > +		(in_fmt->fmt.pix_mp.pixelformat & 0xFF),
> > +		(in_fmt->fmt.pix_mp.pixelformat >> 8) & 0xFF,
> > +		(in_fmt->fmt.pix_mp.pixelformat >> 16) & 0xFF,
> > +		(in_fmt->fmt.pix_mp.pixelformat >> 24) & 0xFF,
> > +		in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height);
> > +
> > +	width = in_fmt->fmt.pix_mp.width;
> > +	height = in_fmt->fmt.pix_mp.height;
> > +
> > +	dev_fmt = mtk_cam_dev_find_fmt(&node->desc,
> > +				       in_fmt->fmt.pix_mp.pixelformat);
> > +	if (dev_fmt) {
> > +		mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
> > +					&in_fmt->fmt.pix_mp,
> > +					&dev_fmt->fmt.pix_mp,
> > +					node->id);
> > +	} else {
> > +		mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
> > +					     &node->desc, in_fmt);
> 
> We shouldn't just load a default format. This function should validate all
> the fields one by one and adjust them to something appropriate.
> 

For ISP P1 HW, we only cares these fields of v4l2_pix_format_mplane.
a. width
b. height
c. pixelformat
d. plane_fmt
    - sizeimage
    - bytesperline
e. num_planes
Other fields are consider constant.

So if the user space passes one pixel format with un-supported, we will
apply the default format firstly and adjust width, height, sizeimage,
and bytesperline. We will focus on validate width & height.
Is it ok?

> > +	}
> 
> CodingStyle: No braces if both if and else bodies have only 1 statement
> each.
> 

Will fix coding style in next patch.

> > +	in_fmt->fmt.pix_mp.width = clamp_t(u32,
> > +					   width,
> > +					   CAM_MIN_WIDTH,
> > +					   in_fmt->fmt.pix_mp.width);
> 
> Shouldn't we clamp this with some maximum value too?
> 

Ok, will revise as below:

	try_fmt.fmt.pix_mp.width = clamp_val(f->fmt.pix_mp.width,
					     IMG_MIN_WIDTH, IMG_MAX_WIDTH);
	try_fmt.fmt.pix_mp.height = clamp_val(f->fmt.pix_mp.height,
					      IMG_MAX_HEIGHT, IMG_MIN_HEIGHT);

> > +	in_fmt->fmt.pix_mp.height = clamp_t(u32,
> > +					    height,
> > +					    CAM_MIN_HEIGHT,
> > +					    in_fmt->fmt.pix_mp.height);
> 
> Ditto.
> 

Ditto.

> > +	mtk_cam_dev_cal_mplane_fmt(&cam_dev->pdev->dev,
> > +				   &in_fmt->fmt.pix_mp, node->id);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_s_fmt(struct file *file, void *fh,
> > +				struct v4l2_format *f)
> > +{
> > +	struct mtk_cam_dev *cam_dev = video_drvdata(file);
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +
> > +	if (cam_dev->streaming)
> > +		return -EBUSY;
> 
> I think this should rather be something like vb2_queue_is_busy(), which
> would prevent format changes if buffers are allocated.
> 

Since vb2_queue_is_busy is static function, would we paste its
implementation in this function to check like this?

	if (node->vdev.queue->owner &&
		node->vdev.queue->owner != file->private_data) {
		dev_err(cam_dev->dev, "%s err: buffer allocated\n", __func__);
		return -EBUSY;
	}

> > +
> > +	/* Get the valid format */
> > +	mtk_cam_vidioc_try_fmt(file, fh, f);
> > +
> > +	/* Configure to video device */
> > +	mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
> > +				&node->vdev_fmt.fmt.pix_mp,
> > +				&f->fmt.pix_mp,
> > +				node->id);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_enum_input(struct file *file, void *fh,
> > +				     struct v4l2_input *input)
> > +{
> > +	if (input->index)
> > +		return -EINVAL;
> > +
> > +	strscpy(input->name, "camera", sizeof(input->name));
> > +	input->type = V4L2_INPUT_TYPE_CAMERA;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_g_input(struct file *file, void *fh,
> > +				  unsigned int *input)
> > +{
> > +	*input = 0;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_s_input(struct file *file,
> > +				  void *fh, unsigned int input)
> > +{
> > +	return input == 0 ? 0 : -EINVAL;
> > +}
> > +
> > +static int mtk_cam_vidioc_enum_framesizes(struct file *filp, void *priv,
> > +					  struct v4l2_frmsizeenum *sizes)
> > +{
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(filp);
> > +	const struct v4l2_format *dev_fmt;
> > +
> > +	dev_fmt = mtk_cam_dev_find_fmt(&node->desc, sizes->pixel_format);
> > +	if (!dev_fmt || sizes->index)
> > +		return -EINVAL;
> > +
> > +	sizes->type = node->desc.frmsizes->type;
> > +	memcpy(&sizes->stepwise, &node->desc.frmsizes->stepwise,
> > +	       sizeof(sizes->stepwise));
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_meta_enum_fmt(struct file *file, void *fh,
> > +					struct v4l2_fmtdesc *f)
> > +{
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +
> > +	if (f->index)
> > +		return -EINVAL;
> > +
> > +	strscpy(f->description, node->desc.description,
> > +		sizeof(node->desc.description));
> > +	f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
> > +	f->flags = 0;
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_vidioc_g_meta_fmt(struct file *file, void *fh,
> > +				     struct v4l2_format *f)
> > +{
> > +	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > +
> > +	f->fmt.meta.dataformat = node->vdev_fmt.fmt.meta.dataformat;
> > +	f->fmt.meta.buffersize = node->vdev_fmt.fmt.meta.buffersize;
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct v4l2_subdev_core_ops mtk_cam_subdev_core_ops = {
> > +	.subscribe_event = mtk_cam_sd_subscribe_event,
> > +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> > +	.s_power = mtk_cam_sd_s_power,
> > +};
> > +
> > +static const struct v4l2_subdev_video_ops mtk_cam_subdev_video_ops = {
> > +	.s_stream =  mtk_cam_sd_s_stream,
> > +};
> > +
> > +static const struct v4l2_subdev_ops mtk_cam_subdev_ops = {
> > +	.core = &mtk_cam_subdev_core_ops,
> > +	.video = &mtk_cam_subdev_video_ops,
> > +};
> > +
> > +static const struct media_entity_operations mtk_cam_media_ops = {
> 
> nit: mtk_cam_media_entity_ops?
> 

Rename in next patch.

> > +	.link_setup = mtk_cam_media_link_setup,
> > +	.link_validate = v4l2_subdev_link_validate,
> > +};
> > +
> > +static const struct vb2_ops mtk_cam_vb2_ops = {
> > +	.queue_setup = mtk_cam_vb2_queue_setup,
> > +	.wait_prepare = vb2_ops_wait_prepare,
> > +	.wait_finish = vb2_ops_wait_finish,
> > +	.buf_init = mtk_cam_vb2_buf_init,
> > +	.buf_prepare = mtk_cam_vb2_buf_prepare,
> > +	.start_streaming = mtk_cam_vb2_start_streaming,
> > +	.stop_streaming = mtk_cam_vb2_stop_streaming,
> > +	.buf_queue = mtk_cam_vb2_buf_queue,
> > +	.buf_request_complete = mtk_cam_vb2_buf_request_complete,
> > +};
> > +
> > +static const struct v4l2_file_operations mtk_cam_v4l2_fops = {
> > +	.unlocked_ioctl = video_ioctl2,
> > +	.open = mtk_cam_isp_open,
> > +	.release = mtk_cam_isp_release,
> > +	.poll = vb2_fop_poll,
> > +	.mmap = vb2_fop_mmap,
> > +#ifdef CONFIG_COMPAT
> > +	.compat_ioctl32 = v4l2_compat_ioctl32,
> > +#endif
> > +};
> > +
> > +static const struct media_device_ops mtk_cam_media_req_ops = {
> 
> nit: Those are media ops, so perhaps just mtk_cam_media_ops?
> 

Rename in next patch.

> > +	.link_notify = v4l2_pipeline_link_notify,
> > +	.req_alloc = mtk_cam_req_alloc,
> > +	.req_free = mtk_cam_req_free,
> > +	.req_validate = vb2_request_validate,
> > +	.req_queue = mtk_cam_req_queue,
> > +};
> > +
> > +static int mtk_cam_media_register(struct device *dev,
> > +				  struct media_device *media_dev)
> > +{
> > +	media_dev->dev = dev;
> > +	strscpy(media_dev->model, MTK_CAM_DEV_P1_NAME,
> 
> Could we replace any use of this macro with dev_driver_string(dev) and then
> delete the macro? The less name strings in the driver the better, as there
> is less change for confusing the userspace if few different names are used
> at the same time.
> 

Ok, revised in next patch.

> > +		sizeof(media_dev->model));
> > +	snprintf(media_dev->bus_info, sizeof(media_dev->bus_info),
> > +		 "platform:%s", dev_name(dev));
> > +	media_dev->hw_revision = 0;
> > +	media_device_init(media_dev);
> > +	media_dev->ops = &mtk_cam_media_req_ops;
> > +
> > +	return media_device_register(media_dev);
> > +}
> > +
> > +static int mtk_cam_video_register_device(struct mtk_cam_dev *cam_dev, u32 i)
> > +{
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	struct mtk_cam_video_device *node = &cam_dev->vdev_nodes[i];
> 
> Would it make sense to pass node as an argument to this function instead of
> (or in addition to) i?
> 

Ok, revised in next patch.

> > +	struct video_device *vdev = &node->vdev;
> > +	struct vb2_queue *vbq = &node->vbq;
> > +	u32 output = !cam_dev->vdev_nodes[i].desc.capture;
> 
> Why not call it capture instead and avoid the inversion?
> 

Ok, we will revised as below.

unsigned int output = V4L2_TYPE_IS_OUTPUT(node->desc.buf_type);

> > +	u32 link_flags = cam_dev->vdev_nodes[i].desc.link_flags;
> > +	int ret;
> > +
> > +	cam_dev->subdev_pads[i].flags = output ?
> > +		MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
> > +
> > +	/* Initialize media entities */
> > +	ret = media_entity_pads_init(&vdev->entity, 1, &node->vdev_pad);
> > +	if (ret) {
> > +		dev_err(dev, "failed initialize media pad:%d\n", ret);
> > +		return ret;
> > +	}
> > +	node->enabled = false;
> 
> Are all the nodes optional? If there is any required node, it should be
> always enabled and have the MEDIA_LNK_FL_IMMUTABLE flag set.
> 

Ok, MTK_CAM_P1_MAIN_STREAM_OUT is required node, others are optional.
We will enable TK_CAM_P1_MAIN_STREAM_OUT with MEDIA_LNK_FL_IMMUTABLE |
MEDIA_LNK_FL_ENABLED.

> > +	node->id = i;
> > +	node->vdev_pad.flags = cam_dev->subdev_pads[i].flags;
> 
> Hmm, shouldn't the subdev pads have opposite directions (sink vs source)?
> 

Yes, it is wrong and fix with below statement.

node->vdev_pad.flags = output ? MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;


> > +	mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
> > +				     &node->desc,
> > +				     &node->vdev_fmt);
> > +
> > +	/* Initialize vbq */
> > +	vbq->type = node->vdev_fmt.type;
> > +	if (vbq->type == V4L2_BUF_TYPE_META_OUTPUT)
> > +		vbq->io_modes = VB2_MMAP;
> > +	else
> > +		vbq->io_modes = VB2_MMAP | VB2_DMABUF;
> > +
> > +	if (node->desc.smem_alloc) {
> > +		vbq->bidirectional = 1;
> > +		vbq->dev = cam_dev->smem_dev;
> > +	} else {
> > +		vbq->dev = &cam_dev->pdev->dev;
> > +	}
> > +
> > +	if (vbq->type == V4L2_BUF_TYPE_META_CAPTURE)
> > +		vdev->entity.function =
> > +			MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
> 
> This is a video node, so it's just a DMA, not a processing entity. I believe
> all the entities corresponding to video nodes should use MEDIA_ENT_F_IO_V4L.
> 

Ok, it is fixed.

> > +	vbq->ops = &mtk_cam_vb2_ops;
> > +	vbq->mem_ops = &vb2_dma_contig_memops;
> > +	vbq->buf_struct_size = sizeof(struct mtk_cam_dev_buffer);
> > +	vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> > +	vbq->min_buffers_needed = 0;	/* Can streamon w/o buffers */
> > +	/* Put the process hub sub device in the vb2 private data */
> 
> What is "process hub" and what "sub device" is this about?
> 

We will drop this comment.

> > +	vbq->drv_priv = cam_dev;
> > +	vbq->lock = &node->lock;
> > +	vbq->supports_requests = true;
> > +
> > +	ret = vb2_queue_init(vbq);
> > +	if (ret) {
> > +		dev_err(dev, "failed to init. vb2 queue:%d\n", ret);
> > +		goto fail_vb2_queue;
> > +	}
> > +
> > +	/* Initialize vdev */
> > +	snprintf(vdev->name, sizeof(vdev->name), "%s %s",
> > +		 MTK_CAM_DEV_P1_NAME, node->desc.name);
> > +	/* set cap/type/ioctl_ops of the video device */
> > +	vdev->device_caps = node->desc.cap | V4L2_CAP_STREAMING;
> > +	vdev->ioctl_ops = node->desc.ioctl_ops;
> > +	vdev->fops = &mtk_cam_v4l2_fops;
> > +	vdev->release = video_device_release_empty;
> > +	vdev->lock = &node->lock;
> > +	vdev->v4l2_dev = &cam_dev->v4l2_dev;
> > +	vdev->queue = &node->vbq;
> > +	vdev->vfl_dir = output ? VFL_DIR_TX : VFL_DIR_RX;
> > +	vdev->entity.ops = NULL;
> > +	/* Enable private control for image video devices */
> > +	if (node->desc.image) {
> > +		mtk_cam_ctrl_init(cam_dev, &node->ctrl_handler);
> > +		vdev->ctrl_handler = &node->ctrl_handler;
> > +	}
> > +	video_set_drvdata(vdev, cam_dev);
> > +	dev_dbg(dev, "register vdev:%d:%s\n", i, vdev->name);
> > +
> > +	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
> > +	if (ret) {
> > +		dev_err(dev, "failed to register vde:%d\n", ret);
> > +		goto fail_vdev;
> > +	}
> > +
> > +	/* Create link between video node and the subdev pad */
> > +	if (output) {
> > +		ret = media_create_pad_link(&vdev->entity, 0,
> > +					    &cam_dev->subdev.entity,
> > +					    i, link_flags);
> > +	} else {
> > +		ret = media_create_pad_link(&cam_dev->subdev.entity,
> > +					    i, &vdev->entity, 0,
> > +					    link_flags);
> > +	}
> > +	if (ret)
> > +		goto fail_link;
> > +
> > +	/* Initialize miscellaneous variables */
> > +	mutex_init(&node->lock);
> > +	spin_lock_init(&node->slock);
> > +	INIT_LIST_HEAD(&node->pending_list);
> 
> This should be all initialized before registering the video device.
> Otherwise userspace could open the device before these are initialized.
> 

Fixed in next patch.

> > +
> > +	return 0;
> > +
> > +fail_link:
> > +	video_unregister_device(vdev);
> > +fail_vdev:
> > +	vb2_queue_release(vbq);
> > +fail_vb2_queue:
> > +	media_entity_cleanup(&vdev->entity);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mtk_cam_mem2mem2_v4l2_register(struct mtk_cam_dev *cam_dev)
> 
> This function doesn't have anything to do with mem2mem. How about
> mtk_cam_v4l2_register()?
> 
> Perhaps it would make sense to move any media related code into into
> mtk_cam_media_register(), keep only V4L2 related code here and have the
> caller call the former first and then this one, rather than having such deep
> sequence of nested calls, which makes the driver harder to read.
> 

Fixed in next patch.

> > +{
> > +	struct device *dev = &cam_dev->pdev->dev;
> 
> How about just storing dev, instead of pdev in the struct? Also, calling the
> argument "cam", would make it as short as cam->dev.
> 

Ok, we will revise this in next patch.

> > +	/* Total pad numbers is video devices + one seninf pad */
> > +	unsigned int num_subdev_pads = MTK_CAM_CIO_PAD_SINK + 1;
> > +	unsigned int i;
> > +	int ret;
> > +
> > +	ret = mtk_cam_media_register(dev,
> > +				     &cam_dev->media_dev);
> > +	if (ret) {
> > +		dev_err(dev, "failed to register media device:%d\n", ret);
> > +		return ret;
> > +	}
> > +	dev_info(dev, "Register media device: %s, 0x%pK",
> > +		 MTK_CAM_DEV_P1_NAME, cam_dev->media_dev);
> 
> An info message should be useful to the user in some way. Printing kernel
> pointers isn't useful. Something like "registered media0" could be useful to
> let the user know which media device is associated with this driver if there
> is more than one in the system.
> 

Here is the new log info.

dev_info(dev, "media%d register",cam->media_dev.devnode->minor);


> > +
> > +	/* Set up v4l2 device */
> > +	cam_dev->v4l2_dev.mdev = &cam_dev->media_dev;
> > +	ret = v4l2_device_register(dev, &cam_dev->v4l2_dev);
> > +	if (ret) {
> > +		dev_err(dev, "failed to register V4L2 device:%d\n", ret);
> > +		goto fail_v4l2_dev;
> 
> Please call the labels after the cleanup step that needs to be done. It
> makes it easier to spot any ordering errors.
> 

Will fix in next patch.

> > +	}
> > +	dev_info(dev, "Register v4l2 device: 0x%pK", cam_dev->v4l2_dev);
> 
> Same as above.
> 

Ditto.

dev_info(dev, "Register v4l2 device: %s", cam->v4l2_dev.name);

> > +
> > +	/* Initialize subdev media entity */
> > +	cam_dev->subdev_pads = devm_kcalloc(dev, num_subdev_pads,
> > +					    sizeof(*cam_dev->subdev_pads),
> > +					    GFP_KERNEL);
> > +	if (!cam_dev->subdev_pads) {
> > +		ret = -ENOMEM;
> > +		goto fail_subdev_pads;
> > +	}
> > +
> > +	ret = media_entity_pads_init(&cam_dev->subdev.entity,
> > +				     num_subdev_pads,
> > +				     cam_dev->subdev_pads);
> > +	if (ret) {
> > +		dev_err(dev, "failed initialize media pads:%d:\n", ret);
> 
> Stray ":" at the end of the message.
> 

Fixed in next patch.

> > +		goto fail_subdev_pads;
> > +	}
> > +
> > +	/* Initialize all pads with MEDIA_PAD_FL_SOURCE */
> > +	for (i = 0; i < num_subdev_pads; i++)
> > +		cam_dev->subdev_pads[i].flags = MEDIA_PAD_FL_SOURCE;
> > +
> > +	/* Customize the last one pad as CIO sink pad. */
> > +	cam_dev->subdev_pads[MTK_CAM_CIO_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
> > +
> > +	/* Initialize subdev */
> > +	v4l2_subdev_init(&cam_dev->subdev, &mtk_cam_subdev_ops);
> > +	cam_dev->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
> > +	cam_dev->subdev.entity.ops = &mtk_cam_media_ops;
> > +	cam_dev->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
> > +				V4L2_SUBDEV_FL_HAS_EVENTS;
> > +	snprintf(cam_dev->subdev.name, sizeof(cam_dev->subdev.name),
> > +		 "%s", MTK_CAM_DEV_P1_NAME);
> > +	v4l2_set_subdevdata(&cam_dev->subdev, cam_dev);
> > +
> > +	ret = v4l2_device_register_subdev(&cam_dev->v4l2_dev, &cam_dev->subdev);
> > +	if (ret) {
> > +		dev_err(dev, "failed initialize subdev:%d\n", ret);
> > +		goto fail_subdev;
> > +	}
> > +	dev_info(dev, "register subdev: %s\n", cam_dev->subdev.name);
> > +
> > +	/* Create video nodes and links */
> > +	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++) {
> > +		ret = mtk_cam_video_register_device(cam_dev, i);
> > +		if (ret)
> > +			goto fail_video_register;
> > +	}
> > +
> > +	vb2_dma_contig_set_max_seg_size(dev, DMA_BIT_MASK(32));
> > +
> > +	return 0;
> > +
> > +fail_video_register:
> > +	i--;
> 
> This could be moved into the for clause, as the initialization statement.
> 

Fixed in next patch.

> > +	for (; i >= 0; i--) {
> 
> i is unsigned. Did this compile without warnings?
> 
> > +		video_unregister_device(&cam_dev->vdev_nodes[i].vdev);
> > +		media_entity_cleanup(&cam_dev->vdev_nodes[i].vdev.entity);
> > +		mutex_destroy(&cam_dev->vdev_nodes[i].lock);
> 
> Should we move this into mtk_cam_video_unregister_device() to be consistent
> with registration?
> 

Fixed in next patch.

> > +	}
> > +fail_subdev:
> > +	media_entity_cleanup(&cam_dev->subdev.entity);
> > +fail_subdev_pads:
> > +	v4l2_device_unregister(&cam_dev->v4l2_dev);
> > +fail_v4l2_dev:
> > +	dev_err(dev, "fail_v4l2_dev mdev: 0x%pK:%d", &cam_dev->media_dev, ret);
> 
> Please print errors only where they actually happen, not at the cleanup.
> 

Fixed in next patch.

> > +	media_device_unregister(&cam_dev->media_dev);
> > +	media_device_cleanup(&cam_dev->media_dev);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mtk_cam_v4l2_unregister(struct mtk_cam_dev *cam_dev)
> > +{
> > +	unsigned int i;
> > +	struct mtk_cam_video_device *dev;
> 
> nit: Move the declaration inside the for loop, since the variable is only
> used there.
> 

Fixed in next patch.

> > +
> > +	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++) {
> > +		dev = &cam_dev->vdev_nodes[i];
> > +		video_unregister_device(&dev->vdev);
> > +		media_entity_cleanup(&dev->vdev.entity);
> > +		mutex_destroy(&dev->lock);
> > +		if (dev->desc.image)
> > +			v4l2_ctrl_handler_free(&dev->ctrl_handler);
> > +	}
> > +
> > +	vb2_dma_contig_clear_max_seg_size(&cam_dev->pdev->dev);
> > +
> > +	v4l2_device_unregister_subdev(&cam_dev->subdev);
> > +	media_entity_cleanup(&cam_dev->subdev.entity);
> > +	kfree(cam_dev->subdev_pads);
> 
> This was allocated using devm_kcalloc(), so no need to free it explicitly.
> 

Fixed in next patch.

> > +
> > +	v4l2_device_unregister(&cam_dev->v4l2_dev);
> > +	media_device_unregister(&cam_dev->media_dev);
> > +	media_device_cleanup(&cam_dev->media_dev);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_cam_dev_complete(struct v4l2_async_notifier *notifier)
> > +{
> > +	struct mtk_cam_dev *cam_dev =
> > +		container_of(notifier, struct mtk_cam_dev, notifier);
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	int ret;
> > +
> > +	ret = media_create_pad_link(&cam_dev->seninf->entity,
> > +				    MTK_CAM_CIO_PAD_SRC,
> > +				    &cam_dev->subdev.entity,
> > +				    MTK_CAM_CIO_PAD_SINK,
> > +				    0);
> > +	if (ret) {
> > +		dev_err(dev, "fail to create pad link %s %s err:%d\n",
> > +			cam_dev->seninf->entity.name,
> > +			cam_dev->subdev.entity.name,
> > +			ret);
> > +		return ret;
> > +	}
> > +
> > +	dev_info(dev, "Complete the v4l2 registration\n");
> 
> dev_dbg()
> 

Fixed in next patch.

> > +
> > +	ret = v4l2_device_register_subdev_nodes(&cam_dev->v4l2_dev);
> > +	if (ret) {
> > +		dev_err(dev, "failed initialize subdev nodes:%d\n", ret);
> > +		return ret;
> > +	}
> > +
> > +	return ret;
> > +}
> 
> Why not just put the contents of this function inside 
> mtk_cam_dev_notifier_complete()?
> 

Ok, we will mtk_cam_dev_complete() function and move its content into
mtk_cam_dev_notifier_complete().

> > +
> > +static int mtk_cam_dev_notifier_bound(struct v4l2_async_notifier *notifier,
> > +				      struct v4l2_subdev *sd,
> > +				      struct v4l2_async_subdev *asd)
> > +{
> > +	struct mtk_cam_dev *cam_dev =
> > +		container_of(notifier, struct mtk_cam_dev, notifier);
> > +
> 
> Should we somehow check that the entity we got is seninf indeed and there
> was no mistake in DT?
> 

How about to check the entity function of seninf device?

if (!(sd->entity.function & MEDIA_ENT_F_VID_IF_BRIDGE)) {
	dev_dbg(cam->dev, "No MEDIA_ENT_F_VID_IF_BRIDGE function\n");
		return -ENODEV;
}

If we need to check DT, may we need to implement this in parse_endpoint
callback function of v4l2_async_notifier_parse_fwnode_endpoints?

> > +	cam_dev->seninf = sd;
> > +	dev_info(&cam_dev->pdev->dev, "%s is bounded\n", sd->entity.name);
> 
> bound
> 
> Also please make this dev_dbg().
> 

Fixed in next patch.

> > +	return 0;
> > +}
> > +
> > +static void mtk_cam_dev_notifier_unbind(struct v4l2_async_notifier *notifier,
> > +					struct v4l2_subdev *sd,
> > +					struct v4l2_async_subdev *asd)
> > +{
> > +	struct mtk_cam_dev *cam_dev =
> > +		container_of(notifier, struct mtk_cam_dev, notifier);
> > +
> > +	cam_dev->seninf = NULL;
> > +	dev_dbg(&cam_dev->pdev->dev, "%s is unbounded\n", sd->entity.name);
> 
> unbound
> 

Fixed in next patch.

> > +}
> > +
> > +static int mtk_cam_dev_notifier_complete(struct v4l2_async_notifier *notifier)
> > +{
> > +	return mtk_cam_dev_complete(notifier);
> > +}
> > +
> > +static const struct v4l2_async_notifier_operations mtk_cam_async_ops = {
> > +	.bound = mtk_cam_dev_notifier_bound,
> > +	.unbind = mtk_cam_dev_notifier_unbind,
> > +	.complete = mtk_cam_dev_notifier_complete,
> > +};
> > +
> > +static int mtk_cam_v4l2_async_register(struct mtk_cam_dev *cam_dev)
> > +{
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	int ret;
> > +
> > +	ret = v4l2_async_notifier_parse_fwnode_endpoints(dev,
> > +		&cam_dev->notifier, sizeof(struct v4l2_async_subdev),
> > +		NULL);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (!cam_dev->notifier.num_subdevs)
> > +		return -ENODEV;
> 
> Could we print some error messages for the 2 cases above?
> 

Fixed in next patch.

> > +
> > +	cam_dev->notifier.ops = &mtk_cam_async_ops;
> > +	dev_info(&cam_dev->pdev->dev, "mtk_cam v4l2_async_notifier_register\n");
> 
> dev_dbg()
> 

Fixed in next patch.

> > +	ret = v4l2_async_notifier_register(&cam_dev->v4l2_dev,
> > +					   &cam_dev->notifier);
> > +	if (ret) {
> > +		dev_err(&cam_dev->pdev->dev,
> > +			"failed to register async notifier : %d\n", ret);
> > +		v4l2_async_notifier_cleanup(&cam_dev->notifier);
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static void mtk_cam_v4l2_async_unregister(struct mtk_cam_dev *cam_dev)
> > +{
> > +	v4l2_async_notifier_unregister(&cam_dev->notifier);
> > +	v4l2_async_notifier_cleanup(&cam_dev->notifier);
> > +}
> > +
> > +static const struct v4l2_ioctl_ops mtk_cam_v4l2_vcap_ioctl_ops = {
> > +	.vidioc_querycap = mtk_cam_vidioc_querycap,
> > +	.vidioc_enum_framesizes = mtk_cam_vidioc_enum_framesizes,
> > +	.vidioc_enum_fmt_vid_cap_mplane = mtk_cam_vidioc_enum_fmt,
> > +	.vidioc_g_fmt_vid_cap_mplane = mtk_cam_vidioc_g_fmt,
> > +	.vidioc_s_fmt_vid_cap_mplane = mtk_cam_vidioc_s_fmt,
> > +	.vidioc_try_fmt_vid_cap_mplane = mtk_cam_vidioc_try_fmt,
> > +	.vidioc_enum_input = mtk_cam_vidioc_enum_input,
> > +	.vidioc_g_input = mtk_cam_vidioc_g_input,
> > +	.vidioc_s_input = mtk_cam_vidioc_s_input,
> 
> I don't think we need vidioc_*_input. At least the current implementation in
> this patch doesn't seem to do anything useful.
> 
> > +	/* buffer queue management */
> 
> Drop this comment, as it's obvious that the callbacks with "buf" in the name
> are related to buffers, there are some non-buffer callbacks below too (e.g.
> vidioc_subscribe_event) and also the other ops structs below don't have such
> comment.
> 

Fixed in next patch.

> > +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> > +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> > +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> > +	.vidioc_querybuf = vb2_ioctl_querybuf,
> > +	.vidioc_qbuf = vb2_ioctl_qbuf,
> > +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> > +	.vidioc_streamon = vb2_ioctl_streamon,
> > +	.vidioc_streamoff = vb2_ioctl_streamoff,
> > +	.vidioc_expbuf = vb2_ioctl_expbuf,
> > +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> > +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> > +};
> > +
> > +static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_cap_ioctl_ops = {
> > +	.vidioc_querycap = mtk_cam_vidioc_querycap,
> > +	.vidioc_enum_fmt_meta_cap = mtk_cam_vidioc_meta_enum_fmt,
> > +	.vidioc_g_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
> > +	.vidioc_s_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
> > +	.vidioc_try_fmt_meta_cap = mtk_cam_vidioc_g_meta_fmt,
> > +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> > +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> > +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> > +	.vidioc_querybuf = vb2_ioctl_querybuf,
> > +	.vidioc_qbuf = vb2_ioctl_qbuf,
> > +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> > +	.vidioc_streamon = vb2_ioctl_streamon,
> > +	.vidioc_streamoff = vb2_ioctl_streamoff,
> > +	.vidioc_expbuf = vb2_ioctl_expbuf,
> > +};
> > +
> > +static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_out_ioctl_ops = {
> > +	.vidioc_querycap = mtk_cam_vidioc_querycap,
> > +	.vidioc_enum_fmt_meta_out = mtk_cam_vidioc_meta_enum_fmt,
> > +	.vidioc_g_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
> > +	.vidioc_s_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
> > +	.vidioc_try_fmt_meta_out = mtk_cam_vidioc_g_meta_fmt,
> > +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> > +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> > +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> > +	.vidioc_querybuf = vb2_ioctl_querybuf,
> > +	.vidioc_qbuf = vb2_ioctl_qbuf,
> > +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> > +	.vidioc_streamon = vb2_ioctl_streamon,
> > +	.vidioc_streamoff = vb2_ioctl_streamoff,
> > +	.vidioc_expbuf = vb2_ioctl_expbuf,
> > +};
> > +
> > +static const struct v4l2_format meta_fmts[] = {
> > +	{
> > +		.fmt.meta = {
> > +			.dataformat = V4L2_META_FMT_MTISP_PARAMS,
> > +			.buffersize = 128 * PAGE_SIZE,
> 
> PAGE_SIZE is a weird unit for specifying generic buffer sizes. How about
> making it 512 * SZ_1K?
> 
> That said, it should normally be just sizeof(struct some_struct_used_here).
> 
> Same for the other entries below.
> 

ok, we will change the size unit from PAGE_SIZE to SZ_1K.
If we finalize the meta structures design, we may change to use sizeof.

> > +		},
> > +	},
> > +	{
> > +		.fmt.meta = {
> > +			.dataformat = V4L2_META_FMT_MTISP_3A,
> > +			.buffersize = 300 * PAGE_SIZE,
> > +		},
> > +	},
> > +	{
> > +		.fmt.meta = {
> > +			.dataformat = V4L2_META_FMT_MTISP_AF,
> > +			.buffersize = 160 * PAGE_SIZE,
> > +		},
> > +	},
> > +	{
> > +		.fmt.meta = {
> > +			.dataformat = V4L2_META_FMT_MTISP_LCS,
> > +			.buffersize = 72 * PAGE_SIZE,
> > +		},
> > +	},
> > +	{
> > +		.fmt.meta = {
> > +			.dataformat = V4L2_META_FMT_MTISP_LMV,
> > +			.buffersize = 256,
> > +		},
> > +	},
> > +};
> > +
> > +static const struct v4l2_format stream_out_fmts[] = {
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = IMG_MAX_WIDTH,
> > +			.height = IMG_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_B8,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_SRGB,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = IMG_MAX_WIDTH,
> > +			.height = IMG_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_B10,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_SRGB,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = IMG_MAX_WIDTH,
> > +			.height = IMG_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_B12,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_SRGB,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = IMG_MAX_WIDTH,
> > +			.height = IMG_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_B14,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_SRGB,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +};
> > +
> > +static const struct v4l2_format bin_out_fmts[] = {
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = RRZ_MAX_WIDTH,
> > +			.height = RRZ_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_F8,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_RAW,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = RRZ_MAX_WIDTH,
> > +			.height = RRZ_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_F10,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_RAW,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = RRZ_MAX_WIDTH,
> > +			.height = RRZ_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_F12,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_RAW,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +	{
> > +		.fmt.pix_mp = {
> > +			.width = RRZ_MAX_WIDTH,
> > +			.height = RRZ_MAX_HEIGHT,
> > +			.pixelformat = V4L2_PIX_FMT_MTISP_F14,
> > +			.field = V4L2_FIELD_NONE,
> > +			.colorspace = V4L2_COLORSPACE_RAW,
> > +			.num_planes = 1,
> > +		},
> > +	},
> > +};
> > +
> > +static const struct v4l2_frmsizeenum img_frm_size_nums[] = {
> > +	{
> > +		.index = 0,
> > +		.type = V4L2_FRMSIZE_TYPE_CONTINUOUS,
> > +		.stepwise = {
> > +			.max_width = IMG_MAX_WIDTH,
> > +			.min_width = IMG_MIN_WIDTH,
> > +			.max_height = IMG_MAX_HEIGHT,
> > +			.min_height = IMG_MIN_HEIGHT,
> > +			.step_height = 1,
> > +			.step_width = 1,
> > +		},
> > +	},
> > +	{
> > +		.index = 0,
> > +		.type = V4L2_FRMSIZE_TYPE_CONTINUOUS,
> > +		.stepwise = {
> > +			.max_width = RRZ_MAX_WIDTH,
> > +			.min_width = RRZ_MIN_WIDTH,
> > +			.max_height = RRZ_MAX_HEIGHT,
> > +			.min_height = RRZ_MIN_HEIGHT,
> > +			.step_height = 1,
> > +			.step_width = 1,
> > +		},
> > +	},
> > +};
> > +
> > +static const struct
> > +mtk_cam_dev_node_desc output_queues[MTK_CAM_P1_TOTAL_OUTPUT] = {
> > +	{
> > +		.id = MTK_CAM_P1_META_IN_0,
> > +		.name = "meta input",
> > +		.description = "ISP tuning parameters",
> > +		.cap = V4L2_CAP_META_OUTPUT,
> > +		.buf_type = V4L2_BUF_TYPE_META_OUTPUT,
> > +		.link_flags = 0,
> > +		.capture = false,
> > +		.image = false,
> > +		.smem_alloc = true,
> > +		.fmts = meta_fmts,
> > +		.num_fmts = ARRAY_SIZE(meta_fmts),
> > +		.default_fmt_idx = 0,
> > +		.max_buf_count = 10,
> > +		.ioctl_ops = &mtk_cam_v4l2_meta_out_ioctl_ops,
> > +	},
> > +};
> > +
> > +static const struct
> > +mtk_cam_dev_node_desc capture_queues[MTK_CAM_P1_TOTAL_CAPTURE] = {
> > +	{
> > +		.id = MTK_CAM_P1_MAIN_STREAM_OUT,
> > +		.name = "main stream",
> > +		.cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE,
> > +		.buf_type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
> > +		.link_flags = 0,
> > +		.capture = true,
> > +		.image = true,
> > +		.smem_alloc = false,
> > +		.dma_port = R_IMGO,
> > +		.fmts = stream_out_fmts,
> > +		.num_fmts = ARRAY_SIZE(stream_out_fmts),
> > +		.default_fmt_idx = 1,
> 
> Why not just make it always 0 and move the default format to the beginning
> of stream_out_fmts[]?
> 

Fixed in next patch.

> > +		.ioctl_ops = &mtk_cam_v4l2_vcap_ioctl_ops,
> > +		.frmsizes = &img_frm_size_nums[0],
> 
> nit: If you only need to point to these data once, you could define them in
> place, as a compound literal, like
> 

Fixed in next patch.

> >                 .frmsizes = &(struct v4l2_framesizeenum) {
> >                         // initialize here
> >                 },
> 

Ditto

> > +	},
> > +	{
> > +		.id = MTK_CAM_P1_PACKED_BIN_OUT,
> > +		.name = "packed out",
> > +		.cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE,
> > +		.buf_type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
> > +		.link_flags = 0,
> > +		.capture = true,
> > +		.image = true,
> > +		.smem_alloc = false,
> > +		.dma_port = R_RRZO,
> > +		.fmts = bin_out_fmts,
> > +		.num_fmts = ARRAY_SIZE(bin_out_fmts),
> > +		.default_fmt_idx = 1,
> > +		.ioctl_ops = &mtk_cam_v4l2_vcap_ioctl_ops,
> > +		.frmsizes = &img_frm_size_nums[1],
> > +	},
> > +	{
> > +		.id = MTK_CAM_P1_META_OUT_0,
> > +		.name = "partial meta 0",
> > +		.description = "AE/AWB histogram",
> > +		.cap = V4L2_CAP_META_CAPTURE,
> > +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> > +		.link_flags = 0,
> > +		.capture = true,
> > +		.image = false,
> > +		.smem_alloc = false,
> > +		.dma_port = R_AAO | R_FLKO | R_PSO,
> > +		.fmts = meta_fmts,
> > +		.num_fmts = ARRAY_SIZE(meta_fmts),
> > +		.default_fmt_idx = 1,
> > +		.max_buf_count = 5,
> > +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> > +	},
> > +	{
> > +		.id = MTK_CAM_P1_META_OUT_1,
> > +		.name = "partial meta 1",
> > +		.description = "AF histogram",
> > +		.cap = V4L2_CAP_META_CAPTURE,
> > +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> > +		.link_flags = 0,
> > +		.capture = true,
> > +		.image = false,
> > +		.smem_alloc = false,
> > +		.dma_port = R_AFO,
> > +		.fmts = meta_fmts,
> > +		.num_fmts = ARRAY_SIZE(meta_fmts),
> > +		.default_fmt_idx = 2,
> > +		.max_buf_count = 5,
> > +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> > +	},
> > +	{
> > +		.id = MTK_CAM_P1_META_OUT_2,
> > +		.name = "partial meta 2",
> > +		.description = "Local contrast enhanced statistics",
> > +		.cap = V4L2_CAP_META_CAPTURE,
> > +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> > +		.link_flags = 0,
> > +		.capture = true,
> > +		.image = false,
> > +		.smem_alloc = false,
> > +		.dma_port = R_LCSO,
> > +		.fmts = meta_fmts,
> > +		.num_fmts = ARRAY_SIZE(meta_fmts),
> > +		.default_fmt_idx = 3,
> > +		.max_buf_count = 10,
> > +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> > +	},
> > +	{
> > +		.id = MTK_CAM_P1_META_OUT_3,
> > +		.name = "partial meta 3",
> > +		.description = "Local motion vector histogram",
> > +		.cap = V4L2_CAP_META_CAPTURE,
> > +		.buf_type = V4L2_BUF_TYPE_META_CAPTURE,
> > +		.link_flags = 0,
> 
> link_flags seems to be 0 for all nodes.
> 

Ditto.

> > +		.capture = true,
> 
> We already know this from .buf_type. We can check using the
> V4L2_TYPE_IS_OUTPUT() macro.
> 

Ditto.

> > +		.image = false,
> > +		.smem_alloc = false,
> > +		.dma_port = R_LMVO,
> > +		.fmts = meta_fmts,
> > +		.num_fmts = ARRAY_SIZE(meta_fmts),
> 
> I don't think this is correct. The description suggests one specific format
> (local motion vector histogram), so the queue should probably be hardwired
> to that format?
> 

Yes, we will set num_fmts = 1 for meta nodes.

> > +		.default_fmt_idx = 4,
> > +		.max_buf_count = 10,
> 
> Where does this number come from?
> 

The default maximum VB2 buffer count is 32.
In order to limit memory usage, we like to limit the maximum buffer
counts in the driver layer. The maximum buffer count is estimated
according to our camera MW.

#define VB2_MAX_FRAME	(32)

> > +		.ioctl_ops = &mtk_cam_v4l2_meta_cap_ioctl_ops,
> > +	},
> > +};
> > +
> > +/* The helper to configure the device context */
> > +static void mtk_cam_dev_queue_setup(struct mtk_cam_dev *cam_dev)
> > +{
> > +	unsigned int i, node_idx;
> > +
> > +	node_idx = 0;
> > +
> > +	/* Setup the output queue */
> > +	for (i = 0; i < MTK_CAM_P1_TOTAL_OUTPUT; i++)
> 
> < ARRAY_SIZE(output_queues)
> 

Fixed in next patch.

> > +		cam_dev->vdev_nodes[node_idx++].desc = output_queues[i];
> > +
> > +	/* Setup the capture queue */
> > +	for (i = 0; i < MTK_CAM_P1_TOTAL_CAPTURE; i++)
> 
> < ARRAY_SIZE(capture_queues)
> 

Fixed in next patch.

> > +		cam_dev->vdev_nodes[node_idx++].desc = capture_queues[i];
> > +}
> > +
> > +int mtk_cam_dev_init(struct platform_device *pdev,
> > +		     struct mtk_cam_dev *cam_dev)
> > +{
> > +	int ret;
> > +
> > +	cam_dev->pdev = pdev;
> 
> Do we need this additional mtk_cam_dev struct? Couldn't we just use
> mtk_isp_p1 here?
> 

We remove pdev field and add dev field in mtk_cam_dev struct.

> > +	mtk_cam_dev_queue_setup(cam_dev);
> > +	/* v4l2 sub-device registration */
> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "mem2mem2.name: %s\n",
> > +		MTK_CAM_DEV_P1_NAME);
> 
> This debugging message doesn't seem very useful. Please remove.
> 

Fixed in next patch.

> > +	ret = mtk_cam_mem2mem2_v4l2_register(cam_dev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = mtk_cam_v4l2_async_register(cam_dev);
> > +	if (ret) {
> > +		mtk_cam_v4l2_unregister(cam_dev);
> 
> Please use an error path on the bottom of the function instead. With goto
> labels named after the next clean-up step that needs to be performed.
> 

Fixed in next patch.

> > +		return ret;
> > +	}
> > +
> > +	spin_lock_init(&cam_dev->req_lock);
> > +	INIT_LIST_HEAD(&cam_dev->req_list);
> > +	mutex_init(&cam_dev->lock);
> > +
> > +	return 0;
> > +}
> > +
> > +int mtk_cam_dev_release(struct platform_device *pdev,
> 
> "release" is normally used for file_operations. How about "cleanup"?
> 

Fixed in next patch.

> > +			struct mtk_cam_dev *cam_dev)
> > +{
> > +	mtk_cam_v4l2_async_unregister(cam_dev);
> > +	mtk_cam_v4l2_unregister(cam_dev);
> > +
> > +	mutex_destroy(&cam_dev->lock);
> > +
> > +	return 0;
> > +}
> 
> I'd suggest moving any generic API implementation code (platform_device,
> V4L2, VB2, Media Controller, etc.) to mtk_cam.c and then moving any low
> level hardware/firmware-related code from mtk_cam.c and mtk_cam-scp.c to
> mtk_cam_hw.c.
> 

Fixed in next patch.

> > +
> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
> new file mode 100644
> index 000000000000..825cdf20643a
> > --- /dev/null
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
> @@ -0,0 +1,173 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (c) 2018 MediaTek Inc.
> > + */
> > +
> > +#ifndef __MTK_CAM_DEV_V4L2_H__
> > +#define __MTK_CAM_DEV_V4L2_H__
> > +
> > +#include <linux/device.h>
> > +#include <linux/types.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/spinlock.h>
> > +#include <linux/videodev2.h>
> > +#include <media/v4l2-device.h>
> > +#include <media/v4l2-ctrls.h>
> > +#include <media/v4l2-subdev.h>
> > +#include <media/videobuf2-core.h>
> > +#include <media/videobuf2-v4l2.h>
> > +
> > +#define MTK_CAM_DEV_P1_NAME			"MTK-ISP-P1-V4L2"
> 
> Maybe it's not a critical thing, but generally it's a good practice to just
> explicitly specific this name somewhere, e.g. struct
> platform_driver::driver::name and then just refer to dev_driver_name(). It
> makes it easier to make sure that the name stays the same everywhere.
> 

Fixed in next patch.

> > +
> > +#define MTK_CAM_P1_META_IN_0			0
> > +#define MTK_CAM_P1_TOTAL_OUTPUT		1
> 
> Since these are just some utility definitions, we can use enum instead of
> assigning the values by hand.
> 

We will revise these definitions as below.

/* ID enum value for struct mtk_cam_dev_node_desc:id */
enum  {
	MTK_CAM_P1_META_IN_0 = 0,
	MTK_CAM_P1_MAIN_STREAM_OUT,
	MTK_CAM_P1_PACKED_BIN_OUT,
	MTK_CAM_P1_META_OUT_0,
	MTK_CAM_P1_META_OUT_1,
	MTK_CAM_P1_META_OUT_2,
	MTK_CAM_P1_META_OUT_3,
	MTK_CAM_P1_TOTAL_NODES
};

> > +
> > +#define MTK_CAM_P1_MAIN_STREAM_OUT		1
> > +#define MTK_CAM_P1_PACKED_BIN_OUT		2
> > +#define MTK_CAM_P1_META_OUT_0			3
> > +#define MTK_CAM_P1_META_OUT_1			4
> > +#define MTK_CAM_P1_META_OUT_2			5
> > +#define MTK_CAM_P1_META_OUT_3			6
> > +#define MTK_CAM_P1_TOTAL_CAPTURE		6
> 
> Ditto.
> 

Ditto.

> > +
> > +#define MTK_CAM_P1_TOTAL_NODES			7
> 
> Please just add the two totals together rather than manually specifying the
> value.
> 

Ditto.

> > +
> > +struct mtk_cam_dev_request {
> > +	struct media_request	req;
> > +	struct list_head	list;
> > +};
> > +
> > +struct mtk_cam_dev_buffer {
> > +	struct vb2_v4l2_buffer	vbb;
> > +	struct list_head	list;
> > +	/* Intenal part */
> > +	dma_addr_t		daddr;
> > +	dma_addr_t		scp_addr;
> > +	unsigned int		node_id;
> > +};
> 
> Could you add kerneldoc comments for the 2 structs?
> 

Fixed in next patch.

> > +
> > +/*
> > + * struct mtk_cam_dev_node_desc - node attributes
> > + *
> > + * @id:		 id of the context queue
> > + * @name:	 media entity name
> > + * @description: descritpion of node
> > + * @cap:	 mapped to V4L2 capabilities
> > + * @buf_type:	 mapped to V4L2 buffer type
> > + * @dma_port:	 the dma port associated to the buffer
> > + * @link_flags:	 default media link flags
> > + * @smem_alloc:	 using the cam_smem_drv as alloc ctx or not
> > + * @capture:	 true for capture queue (device to user)
> > + *		 false for output queue (from user to device)
> > + * @image:	 true for image node, false for meta node
> > + * @num_fmts:	 the number of supported formats
> > + * @default_fmt_idx: default format of this node
> > + * @max_buf_count: maximum V4L2 buffer count
> > + * @ioctl_ops:  mapped to v4l2_ioctl_ops
> > + * @fmts:	supported format
> > + * @frmsizes:	supported frame size number
> > + *
> > + */
> > +struct mtk_cam_dev_node_desc {
> > +	u8 id;
> > +	char *name;
> > +	char *description;
> > +	u32 cap;
> > +	u32 buf_type;
> > +	u32 dma_port;
> > +	u32 link_flags;
> > +	u8 smem_alloc:1;
> > +	u8 capture:1;
> > +	u8 image:1;
> > +	u8 num_fmts;
> > +	u8 default_fmt_idx;
> > +	u8 max_buf_count;
> > +	const struct v4l2_ioctl_ops *ioctl_ops;
> > +	const struct v4l2_format *fmts;
> > +	const struct v4l2_frmsizeenum *frmsizes;
> > +};
> > +
> > +/*
> > + * struct mtk_cam_video_device - Mediatek video device structure.
> > + *
> > + * @id:		Id for mtk_cam_dev_node_desc or mem2mem2_nodes array
> > + * @enabled:	Indicate the device is enabled or not
> > + * @vdev_fmt:	The V4L2 format of video device
> > + * @vdev_apd:	The media pad graph object of video device
> 
> vdev_pad?
> 
> > + * @vbq:	A videobuf queue of video device
> > + * @desc:	The node attributes of video device
> > + * @ctrl_handler:	The control handler of video device
> > + * @pending_list:	List for pending buffers before enqueuing into driver
> > + * @lock:	Serializes vb2 queue and video device operations.
> > + * @slock:	Protect for pending_list.
> > + *
> 
> Please fix the order of the documentation to match the order of the struct.
> 

Fixed in next patch.

> > + */
> > +struct mtk_cam_video_device {
> > +	unsigned int id;
> > +	unsigned int enabled;
> > +	struct v4l2_format vdev_fmt;
> > +	struct mtk_cam_dev_node_desc desc;
> > +	struct video_device vdev;
> 
> Not documented above.
> 

Fixed in next patch.

> > +	struct media_pad vdev_pad;
> > +	struct vb2_queue vbq;
> > +	struct v4l2_ctrl_handler ctrl_handler;
> > +	struct list_head pending_list;
> > +	/* Used for vbq & vdev */
> 
> It's already documented in the kerneldoc comment.
> 

Fixed in next patch.
Btw, if we remove this, we will got complain from checkpatch.pl script.

> > +	struct mutex lock;
> > +	/* protect for pending_list */
> 
> It's already documented in the kerneldoc comment.
> 

Ditto.

> > +	spinlock_t slock;
> 
> How about calling it pending_list_lock?
> 

We will rename to buf_list to track all en-queue buffers in this video
node.

struct mtk_cam_video_device {
	unsigned int id;
	unsigned int enabled;
	struct v4l2_format vdev_fmt;
	struct mtk_cam_dev_node_desc desc;
	struct video_device vdev;
	struct media_pad vdev_pad;
	struct vb2_queue vbq;
	struct v4l2_ctrl_handler ctrl_handler;
	struct list_head buf_list;
	struct mutex lock;
	spinlock_t buf_list_lock;
};

> > +};
> > +
> > +/*
> > + * struct mtk_cam_dev - Mediatek camera device structure.
> > + *
> > + * @pdev:	Pointer to platform device
> > + * @smem_pdev:	Pointer to shared memory platform device
> > + * @pipeline:	Media pipeline information
> > + * @media_dev:	Media device
> > + * @subdev:	The V4L2 sub-device
> > + * @v4l2_dev:	The V4L2 device driver
> > + * @notifier:	The v4l2_device notifier data
> > + * @subdev_pads: Pointer to the number of media pads of this sub-device
> > + * @ctrl_handler: The control handler
> > + * @vdev_nodes: The array list of mtk_cam_video_device nodes
> > + * @seninf:	Pointer to the seninf sub-device
> > + * @sensor:	Pointer to the active sensor V4L2 sub-device when streaming on
> > + * @lock:       The mutex protecting video device open/release operations
> > + * @streaming:	Indicate the overall streaming status is on or off
> > + * @streamed_node_count: The number of V4L2 video device nodes are streaming on
> > + * @req_list:	Lins to keep media requests before streaming on
> > + * @req_lock:	Protect the req_list data
> > + *
> > + * Below is the graph topology for Camera IO connection.
> > + * sensor 1 (main) --> sensor IF --> P1 sub-device
> > + * sensor 2 (sub)  -->
> 
> This probably isn't the best place for graph topology description. I think
> we actually want a separate documentation file for this, similar to
> Documentation/media/v4l-drivers/ipu3.rst.
> 

Ok, we will drop our graph topology comment & discuss how to come out
another separate document.

> > + *
> > + */
> > +struct mtk_cam_dev {
> > +	struct platform_device *pdev;
> > +	struct device *smem_dev;
> > +	struct media_pipeline pipeline;
> > +	struct media_device media_dev;
> > +	struct v4l2_subdev subdev;
> > +	struct v4l2_device v4l2_dev;
> > +	struct v4l2_async_notifier notifier;
> > +	struct media_pad *subdev_pads;
> > +	struct v4l2_ctrl_handler ctrl_handler;
> > +	struct mtk_cam_video_device vdev_nodes[MTK_CAM_P1_TOTAL_NODES];
> > +	struct v4l2_subdev *seninf;
> > +	struct v4l2_subdev *sensor;
> > +	/* protect video device open/release operations */
> 
> It's already documented in the kerneldoc comment.
> 

Fixed in next patch.

> > +	struct mutex lock;
> > +	unsigned int streaming:1;
> > +	atomic_t streamed_node_count;
> > +	struct list_head req_list;
> > +	/* protect for req_list */
> 
> It's already documented in the kerneldoc comment.
> 

Fixed in next patch.

> > +	spinlock_t req_lock;
> 
> How about calling it req_list_lock?
> 

Below is new mtk_cam_dev structure.
We will use job to handle request.

struct mtk_cam_dev {
	struct device *dev;
	struct device *smem_dev;
	struct media_pipeline pipeline;
	struct media_device media_dev;
	struct v4l2_subdev subdev;
	struct v4l2_device v4l2_dev;
	struct v4l2_async_notifier notifier;
	struct media_pad *subdev_pads;
	struct v4l2_ctrl_handler ctrl_handler;
	struct mtk_cam_video_device vdev_nodes[MTK_CAM_P1_TOTAL_NODES];
	struct v4l2_subdev *seninf;
	struct v4l2_subdev *sensor;
	struct mutex lock;
	unsigned int streaming:1;
	unsigned int enabled_dmas;
	unsigned int enabled_node_count;
	atomic_t streamed_node_count;
	struct list_head pending_job_list;
	spinlock_t pending_job_lock;
	struct list_head running_job_list;
	spinlock_t running_job_lock;
	atomic_t running_job_count;
};


> Best regards,
> Tomasz
> 

Thanks again for your many inputs on this patch.
It is helpful for us.

Best regards,

Jungo




^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-07-10  9:56   ` Tomasz Figa
@ 2019-07-20  9:58     ` Jungo Lin
  2019-07-25  9:23       ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-20  9:58 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi, Tomasz:

On Wed, 2019-07-10 at 18:56 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Tue, Jun 11, 2019 at 11:53:42AM +0800, Jungo Lin wrote:
> > This patch adds the Mediatek ISP P1 HW control device driver.
> > It handles the ISP HW configuration, provides interrupt handling and
> > initializes the V4L2 device nodes and other functions.
> > 
> > (The current metadata interface used in meta input and partial
> > meta nodes is only a temporary solution to kick off the driver
> > development and is not ready to be reviewed yet.)
> > 
> > Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> > ---
> >  .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
> >  .../mtk-isp/isp_50/cam/mtk_cam-regs.h         |  126 ++
> >  .../platform/mtk-isp/isp_50/cam/mtk_cam.c     | 1087 +++++++++++++++++
> >  .../platform/mtk-isp/isp_50/cam/mtk_cam.h     |  243 ++++
> >  4 files changed, 1457 insertions(+)
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
> > 
> 
> Thanks for the patch! Please see my comments inline.
> 
> [snip]
> 

Thanks for your comments. Please check my replies inline.

[snip]

> > diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
> > new file mode 100644
> > index 000000000000..9e59a6bfc6b7
> > --- /dev/null
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
> > @@ -0,0 +1,126 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (c) 2018 MediaTek Inc.
> > + */
> > +
> > +#ifndef _CAM_REGS_H
> > +#define _CAM_REGS_H
> > +
> > +/* TG Bit Mask */
> > +#define VFDATA_EN_BIT			BIT(0)
> > +#define CMOS_EN_BIT			BIT(0)
> > +
> > +/* normal signal bit */
> > +#define VS_INT_ST			BIT(0)
> > +#define HW_PASS1_DON_ST		BIT(11)
> > +#define SOF_INT_ST			BIT(12)
> > +#define SW_PASS1_DON_ST		BIT(30)
> > +
> > +/* err status bit */
> > +#define TG_ERR_ST			BIT(4)
> > +#define TG_GBERR_ST			BIT(5)
> > +#define CQ_CODE_ERR_ST			BIT(6)
> > +#define CQ_APB_ERR_ST			BIT(7)
> > +#define CQ_VS_ERR_ST			BIT(8)
> > +#define AMX_ERR_ST			BIT(15)
> > +#define RMX_ERR_ST			BIT(16)
> > +#define BMX_ERR_ST			BIT(17)
> > +#define RRZO_ERR_ST			BIT(18)
> > +#define AFO_ERR_ST			BIT(19)
> > +#define IMGO_ERR_ST			BIT(20)
> > +#define AAO_ERR_ST			BIT(21)
> > +#define PSO_ERR_ST			BIT(22)
> > +#define LCSO_ERR_ST			BIT(23)
> > +#define BNR_ERR_ST			BIT(24)
> > +#define LSCI_ERR_ST			BIT(25)
> > +#define DMA_ERR_ST			BIT(29)
> > +
> > +/* CAM DMA done status */
> > +#define FLKO_DONE_ST			BIT(4)
> > +#define AFO_DONE_ST			BIT(5)
> > +#define AAO_DONE_ST			BIT(7)
> > +#define PSO_DONE_ST			BIT(14)
> > +
> > +/* IRQ signal mask */
> > +#define INT_ST_MASK_CAM		( \
> > +					VS_INT_ST |\
> > +					SOF_INT_ST |\
> > +					HW_PASS1_DON_ST |\
> > +					SW_PASS1_DON_ST)
> > +
> > +/* IRQ Error Mask */
> > +#define INT_ST_MASK_CAM_ERR		( \
> > +					TG_ERR_ST |\
> > +					TG_GBERR_ST |\
> > +					CQ_CODE_ERR_ST |\
> > +					CQ_APB_ERR_ST |\
> > +					CQ_VS_ERR_ST |\
> > +					BNR_ERR_ST |\
> > +					RMX_ERR_ST |\
> > +					BMX_ERR_ST |\
> > +					BNR_ERR_ST |\
> > +					LSCI_ERR_ST |\
> > +					DMA_ERR_ST)
> > +
> > +/* IRQ Signal Log Mask */
> > +#define INT_ST_LOG_MASK_CAM		( \
> > +					SOF_INT_ST |\
> > +					SW_PASS1_DON_ST |\
> > +					HW_PASS1_DON_ST |\
> > +					VS_INT_ST |\
> > +					TG_ERR_ST |\
> > +					TG_GBERR_ST |\
> > +					RRZO_ERR_ST |\
> > +					AFO_ERR_ST |\
> > +					IMGO_ERR_ST |\
> > +					AAO_ERR_ST |\
> > +					DMA_ERR_ST)
> > +
> > +/* DMA Event Notification Mask */
> > +#define DMA_ST_MASK_CAM		( \
> > +					AAO_DONE_ST |\
> > +					AFO_DONE_ST)
> 
> Could we define the values next to the addresses of registers they apply to?
> Also without the _BIT suffix and with the values prefixed with register
> names. For example:
> 
> #define REG_TG_SEN_MODE		        0x0230
> #define TG_SEN_MODE_CMOS_EN		BIT(0)
> 
> #define REG_TG_VF_CON			0x0234
> #define TG_VF_CON_VFDATA_EN		BIT(0)
> 

Fix in next patch.

> > +
> > +/* Status check */
> > +#define REG_CTL_EN			0x0004
> > +#define REG_CTL_DMA_EN			0x0008
> > +#define REG_CTL_FMT_SEL		0x0010
> > +#define REG_CTL_EN2			0x0018
> > +#define REG_CTL_RAW_INT_EN		0x0020
> > +#define REG_CTL_RAW_INT_STAT		0x0024
> > +#define REG_CTL_RAW_INT2_STAT		0x0034
> > +
> > +#define REG_TG_SEN_MODE		0x0230
> > +#define REG_TG_VF_CON			0x0234
> > +
> > +#define REG_IMGO_BASE_ADDR		0x1020
> > +#define REG_RRZO_BASE_ADDR		0x1050
> > +
> > +/* Error status log */
> > +#define REG_IMGO_ERR_STAT		0x1360
> > +#define REG_RRZO_ERR_STAT		0x1364
> > +#define REG_AAO_ERR_STAT		0x1368
> > +#define REG_AFO_ERR_STAT		0x136c
> > +#define REG_LCSO_ERR_STAT		0x1370
> > +#define REG_UFEO_ERR_STAT		0x1374
> > +#define REG_PDO_ERR_STAT		0x1378
> > +#define REG_BPCI_ERR_STAT		0x137c
> > +#define REG_LSCI_ERR_STAT		0x1384
> > +#define REG_PDI_ERR_STAT		0x138c
> > +#define REG_LMVO_ERR_STAT		0x1390
> > +#define REG_FLKO_ERR_STAT		0x1394
> > +#define REG_PSO_ERR_STAT		0x13a0
> > +
> > +/* ISP command */
> > +#define REG_CQ_THR0_BASEADDR		0x0198
> > +#define REG_HW_FRAME_NUM		0x13b8
> > +
> > +/* META */
> > +#define REG_META0_VB2_INDEX		0x14dc
> > +#define REG_META1_VB2_INDEX		0x151c
> 
> I don't believe these registers are really for VB2 indexes.
> 

MTK P1 ISP HW supports frame header spare registers for each DMA, such
as CAM_DMA_FH_AAO_SPARE or CAM_DMA_FH_AFO_SPARE. We could save some
frame information in these ISP registers. In this case, we save META0
VB2 index into AAO FH spare register and META1 VB2 index into AFO FH
spare register. These implementation is designed for non-request 3A
DMAs. These VB2 indexes are sent in ISP_CMD_ENQUEUE_META command of
mtk_isp_enqueue function. So we just call CAM_DMA_FH_AAO_SPARE as 
REG_META0_VB2_INDEX for easy understanding. Moreover, if we only need to
support request mode, we should remove this here.

cmd_params.cmd_id = ISP_CMD_ENQUEUE_META;
cmd_params.meta_frame.enabled_dma = dma_port;
cmd_params.meta_frame.vb_index = buffer->vbb.vb2_buf.index;
cmd_params.meta_frame.meta_addr.iova = buffer->daddr;
cmd_params.meta_frame.meta_addr.scp_addr = buffer->scp_addr;

> > +
> > +/* FBC */
> > +#define REG_AAO_FBC_STATUS		0x013c
> > +#define REG_AFO_FBC_STATUS		0x0134
> > +
> > +#endif	/* _CAM_REGS_H */
> > diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
> > new file mode 100644
> > index 000000000000..c5a3babed69d
> > --- /dev/null
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
> > @@ -0,0 +1,1087 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +//
> > +// Copyright (c) 2018 MediaTek Inc.
> > +
> > +#include <linux/atomic.h>
> > +#include <linux/cdev.h>
> > +#include <linux/compat.h>
> > +#include <linux/fs.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/jiffies.h>
> > +#include <linux/kernel.h>
> > +#include <linux/ktime.h>
> > +#include <linux/module.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/of_irq.h>
> > +#include <linux/of_address.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/platform_data/mtk_scp.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/remoteproc.h>
> > +#include <linux/sched/clock.h>
> > +#include <linux/spinlock.h>
> > +#include <linux/types.h>
> > +#include <linux/videodev2.h>
> > +#include <linux/vmalloc.h>
> > +#include <media/v4l2-event.h>
> > +
> > +#include "mtk_cam.h"
> > +#include "mtk_cam-regs.h"
> > +#include "mtk_cam-smem.h"
> > +
> > +static const struct of_device_id mtk_isp_of_ids[] = {
> > +	{.compatible = "mediatek,mt8183-camisp",},
> > +	{}
> > +};
> > +MODULE_DEVICE_TABLE(of, mtk_isp_of_ids);
> 
> Please move below. Just above the platform_driver struct where it's used.
> 

Fix in next patch.

> > +
> > +/* List of clocks required by isp cam */
> > +static const char * const mtk_isp_clks[] = {
> > +	"camsys_cam_cgpdn", "camsys_camtg_cgpdn"
> > +};
> 
> Please move inside mtk_isp_probe, as a static const local variable. That
> could also let you shorten the name, to clk_names for example.
> 

Fix in next patch.

> > +
> > +static void isp_dump_dma_status(struct isp_device *isp_dev)
> > +{
> > +	dev_err(isp_dev->dev,
> > +		"IMGO:0x%x, RRZO:0x%x, AAO=0x%x, AFO=0x%x, LMVO=0x%x\n",
> > +		readl(isp_dev->regs + REG_IMGO_ERR_STAT),
> > +		readl(isp_dev->regs + REG_RRZO_ERR_STAT),
> > +		readl(isp_dev->regs + REG_AAO_ERR_STAT),
> > +		readl(isp_dev->regs + REG_AFO_ERR_STAT),
> > +		readl(isp_dev->regs + REG_LMVO_ERR_STAT));
> > +	dev_err(isp_dev->dev,
> > +		"LCSO=0x%x, PSO=0x%x, FLKO=0x%x, BPCI:0x%x, LSCI=0x%x\n",
> > +		readl(isp_dev->regs + REG_LCSO_ERR_STAT),
> > +		readl(isp_dev->regs + REG_PSO_ERR_STAT),
> > +		readl(isp_dev->regs + REG_FLKO_ERR_STAT),
> > +		readl(isp_dev->regs + REG_BPCI_ERR_STAT),
> > +		readl(isp_dev->regs + REG_LSCI_ERR_STAT));
> > +}
> > +
> > +static void mtk_cam_dev_event_frame_sync(struct mtk_cam_dev *cam_dev,
> > +					 __u32 frame_seq_no)
> > +{
> > +	struct v4l2_event event;
> > +
> > +	memset(&event, 0, sizeof(event));
> > +	event.type = V4L2_EVENT_FRAME_SYNC;
> > +	event.u.frame_sync.frame_sequence = frame_seq_no;
> 
> nit: You can just initialize the structure in the declaration.
> 

Fix in next patch.

void mtk_cam_dev_event_frame_sync(struct mtk_cam_dev *cam,
				  unsigned int frame_seq_no)
{
	struct v4l2_event event = {
		.type = V4L2_EVENT_FRAME_SYNC,
		.u.frame_sync.frame_sequence = frame_seq_no,
	};

	v4l2_event_queue(cam->subdev.devnode, &event);
}

> > +	v4l2_event_queue(cam_dev->subdev.devnode, &event);
> > +}
> > +
> > +static void mtk_cam_dev_job_finish(struct mtk_isp_p1_ctx *isp_ctx,
> > +				   unsigned int request_fd,
> > +				   unsigned int frame_seq_no,
> > +				   struct list_head *list_buf,
> > +				   enum vb2_buffer_state state)
> > +{
> > +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> > +	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
> > +	struct mtk_cam_dev_buffer *buf, *b0;
> > +	u64    timestamp;
> 
> Too many spaces between u64 and timestamp.
> 

Fix in next patch.

> > +
> > +	if (!cam_dev->streaming)
> > +		return;
> > +
> > +	dev_dbg(&p1_dev->pdev->dev, "%s request fd:%d frame_seq:%d state:%d\n",
> > +		__func__, request_fd, frame_seq_no, state);
> > +
> > +	/*
> > +	 * Set the buffer's VB2 status so that the user can dequeue
> > +	 * the buffer.
> > +	 */
> > +	timestamp = ktime_get_ns();
> > +	list_for_each_entry_safe(buf, b0, list_buf, list) {
> 
> nit: s/b0/buf_prev/
> 

Fix in next patch.

> > +		list_del(&buf->list);
> > +		buf->vbb.vb2_buf.timestamp = timestamp;
> > +		buf->vbb.sequence = frame_seq_no;
> > +		if (buf->vbb.vb2_buf.state == VB2_BUF_STATE_ACTIVE)
> 
> Any buffer that is not active shouldn't be on this list. If it is then it's
> a bug somewhere else in the driver. Could be possibly related to the request
> handling issues I pointed out in another comment.
> 

Fix in next patch.

> > +			vb2_buffer_done(&buf->vbb.vb2_buf, state);
> > +	}
> > +}
> > +
> > +static void isp_deque_frame(struct isp_p1_device *p1_dev,
> 
> dequeue
> 

Fix in next patch.

> > +			    unsigned int node_id, int vb2_index,
> > +			    int frame_seq_no)
> > +{
> > +	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	struct mtk_cam_video_device *node = &cam_dev->vdev_nodes[node_id];
> > +	struct mtk_cam_dev_buffer *b, *b0;
> > +	struct vb2_buffer *vb;
> > +
> > +	if (!cam_dev->vdev_nodes[node_id].enabled || !cam_dev->streaming)
> > +		return;
> > +
> > +	spin_lock(&node->slock);
> > +	b = list_first_entry(&node->pending_list,
> > +			     struct mtk_cam_dev_buffer,
> > +			     list);
> > +	list_for_each_entry_safe(b, b0, &node->pending_list, list) {
> > +		vb = &b->vbb.vb2_buf;
> > +		if (!vb->vb2_queue->uses_requests &&
> > +		    vb->index == vb2_index &&
> > +		    vb->state == VB2_BUF_STATE_ACTIVE) {
> > +			dev_dbg(dev, "%s:%d:%d", __func__, node_id, vb2_index);
> > +			vb->timestamp = ktime_get_ns();
> > +			b->vbb.sequence = frame_seq_no;
> > +			vb2_buffer_done(vb, VB2_BUF_STATE_DONE);
> > +			list_del(&b->list);
> > +			break;
> > +		}
> > +	}
> > +	spin_unlock(&node->slock);
> > +}
> > +
> > +static void isp_deque_request_frame(struct isp_p1_device *p1_dev,
> 
> dequeue
> 

Fix in next patch.

> > +				    int frame_seq_no)
> > +{
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	struct mtk_isp_queue_job *framejob, *tmp;
> > +	struct isp_queue *p1_enqueue_list = &isp_ctx->p1_enqueue_list;
> > +
> > +	/* Match dequeue work and enqueue frame */
> > +	spin_lock(&p1_enqueue_list->lock);
> > +	list_for_each_entry_safe(framejob, tmp, &p1_enqueue_list->queue,
> > +				 list_entry) {
> > +		dev_dbg(dev,
> > +			"%s frame_seq_no:%d, target frame_seq_no:%d\n",
> > +			__func__,
> > +			framejob->frame_seq_no, frame_seq_no);
> > +		/* Match by the en-queued request number */
> > +		if (framejob->frame_seq_no == frame_seq_no) {
> > +			/* Pass to user space */
> > +			mtk_cam_dev_job_finish(isp_ctx,
> > +					       framejob->request_fd,
> > +					       framejob->frame_seq_no,
> > +					       &framejob->list_buf,
> > +					       VB2_BUF_STATE_DONE);
> > +			atomic_dec(&p1_enqueue_list->queue_cnt);
> > +			dev_dbg(dev,
> > +				"frame_seq_no:%d is done, queue_cnt:%d\n",
> > +				framejob->frame_seq_no,
> > +				atomic_read(&p1_enqueue_list->queue_cnt));
> > +
> > +			/* Remove only when frame ready */
> > +			list_del(&framejob->list_entry);
> > +			kfree(framejob);
> > +			break;
> > +		} else if (framejob->frame_seq_no < frame_seq_no) {
> > +			/* Pass to user space for frame drop */
> > +			mtk_cam_dev_job_finish(isp_ctx,
> > +					       framejob->request_fd,
> > +					       framejob->frame_seq_no,
> > +					       &framejob->list_buf,
> > +					       VB2_BUF_STATE_ERROR);
> > +			atomic_dec(&p1_enqueue_list->queue_cnt);
> > +			dev_warn(dev,
> > +				 "frame_seq_no:%d drop, queue_cnt:%d\n",
> > +				 framejob->frame_seq_no,
> > +				 atomic_read(&p1_enqueue_list->queue_cnt));
> > +
> > +			/* Remove only drop frame */
> > +			list_del(&framejob->list_entry);
> > +			kfree(framejob);
> > +		} else {
> > +			break;
> > +		}
> > +	}
> > +	spin_unlock(&p1_enqueue_list->lock);
> > +}
> > +
> > +static int isp_deque_work(void *data)
> 
> dequeue
> 

Fix in next patch.

> > +{
> > +	struct isp_p1_device *p1_dev = (struct isp_p1_device *)data;
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct mtk_cam_dev_stat_event_data event_data;
> > +	atomic_t *irq_data_end = &isp_ctx->irq_data_end;
> > +	atomic_t *irq_data_start = &isp_ctx->irq_data_start;
> > +	unsigned long flags;
> > +	int ret, i;
> > +
> > +	while (1) {
> > +		ret = wait_event_interruptible(isp_ctx->isp_deque_thread.wq,
> > +					       (atomic_read(irq_data_end) !=
> > +					       atomic_read(irq_data_start)) ||
> > +					       kthread_should_stop());
> > +
> > +		if (kthread_should_stop())
> > +			break;
> > +
> > +		spin_lock_irqsave(&isp_ctx->irq_dequeue_lock, flags);
> > +		i = atomic_read(&isp_ctx->irq_data_start);
> > +		memcpy(&event_data, &isp_ctx->irq_event_datas[i],
> > +		       sizeof(event_data));
> > +		atomic_set(&isp_ctx->irq_data_start, ++i & 0x3);
> > +		spin_unlock_irqrestore(&isp_ctx->irq_dequeue_lock, flags);
> > +
> > +		if (event_data.irq_status_mask & HW_PASS1_DON_ST &&
> > +		    event_data.dma_status_mask & AAO_DONE_ST) {
> > +			isp_deque_frame(p1_dev,
> > +					MTK_CAM_P1_META_OUT_0,
> > +					event_data.meta0_vb2_index,
> > +					event_data.frame_seq_no);
> > +		}
> > +		if (event_data.dma_status_mask & AFO_DONE_ST) {
> > +			isp_deque_frame(p1_dev,
> > +					MTK_CAM_P1_META_OUT_1,
> > +					event_data.meta1_vb2_index,
> > +					event_data.frame_seq_no);
> > +		}
> > +		if (event_data.irq_status_mask & SW_PASS1_DON_ST) {
> > +			isp_deque_frame(p1_dev,
> > +					MTK_CAM_P1_META_OUT_0,
> > +					event_data.meta0_vb2_index,
> > +					event_data.frame_seq_no);
> > +			isp_deque_frame(p1_dev,
> > +					MTK_CAM_P1_META_OUT_1,
> > +					event_data.meta1_vb2_index,
> > +					event_data.frame_seq_no);
> > +			isp_deque_request_frame(p1_dev,
> > +						event_data.frame_seq_no);
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int irq_handle_sof(struct isp_device *isp_dev,
> > +			  dma_addr_t base_addr,
> > +			  unsigned int frame_num)
> > +{
> > +	unsigned int addr_offset;
> > +	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
> > +	int cq_num = atomic_read(&p1_dev->isp_ctx.composed_frame_id);
> > +
> > +	isp_dev->sof_count += 1;
> > +
> > +	if (cq_num <= frame_num) {
> > +		dev_dbg(isp_dev->dev,
> > +			"SOF_INT_ST, wait next, cq_num:%d, frame_num:%d",
> > +			cq_num, frame_num);
> > +		atomic_set(&p1_dev->isp_ctx.composing_frame, 0);
> > +		return cq_num;
> > +	}
> > +	atomic_set(&p1_dev->isp_ctx.composing_frame, cq_num - frame_num);
> > +
> > +	addr_offset = CQ_ADDRESS_OFFSET * (frame_num % CQ_BUFFER_COUNT);
> > +	writel(base_addr + addr_offset, isp_dev->regs + REG_CQ_THR0_BASEADDR);
> > +	dev_dbg(isp_dev->dev,
> > +		"SOF_INT_ST, update next, cq_num:%d, frame_num:%d cq_addr:0x%x",
> > +		cq_num, frame_num, addr_offset);
> > +
> > +	return cq_num;
> > +}
> > +
> > +static void irq_handle_notify_event(struct isp_device *isp_dev,
> > +				    unsigned int irq_status,
> > +				    unsigned int dma_status,
> > +				    bool sof_only)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct device *dev = isp_dev->dev;
> > +	unsigned long flags;
> > +	int i;
> > +
> > +	if (irq_status & VS_INT_ST) {
> > +		/* Notify specific HW events to user space */
> > +		mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev,
> > +					     isp_dev->current_frame);
> 
> Shouldn't we call this at SOF_INT_ST and not VS? At least according to the
> definition of the V4L2_EVENT_FRAME_SYNC event at
> https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-dqevent.html
> 

Fix in next patch.
We will change to use SOF_INT_ST to avoid misunderstanding.

> > +		dev_dbg(dev,
> > +			"frame sync is sent:%d:%d\n",
> > +			isp_dev->sof_count,
> > +			isp_dev->current_frame);
> > +		if (sof_only)
> > +			return;
> 
> If this function can be called only to perform this block, perhaps it should
> be split into two functions?
> 
> Also, what happens if we get sof_only, but we don't get VS_INT_ST set in
> irq_status? Is it expected that in such case the other part of the function
> is executed?
> 

Ok, we will call mtk_cam_dev_event_frame_sync function when receiving
SOF_INT_ST ISR event in the caller and remove this block.

> > +	}
> > +
> > +	if (irq_status & SW_PASS1_DON_ST) {
> > +		/* Notify TX thread to send if TX frame is blocked */
> > +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> > +	}
> > +
> > +	spin_lock_irqsave(&isp_ctx->irq_dequeue_lock, flags);
> > +	i = atomic_read(&isp_ctx->irq_data_end);
> > +	isp_ctx->irq_event_datas[i].frame_seq_no = isp_dev->current_frame;
> > +	isp_ctx->irq_event_datas[i].meta0_vb2_index = isp_dev->meta0_vb2_index;
> > +	isp_ctx->irq_event_datas[i].meta1_vb2_index = isp_dev->meta1_vb2_index;
> > +	isp_ctx->irq_event_datas[i].irq_status_mask =
> > +		(irq_status & INT_ST_MASK_CAM);
> > +	isp_ctx->irq_event_datas[i].dma_status_mask =
> > +		(dma_status & DMA_ST_MASK_CAM);
> > +	atomic_set(&isp_ctx->irq_data_end, ++i & 0x3);
> > +	spin_unlock_irqrestore(&isp_ctx->irq_dequeue_lock, flags);
> > +
> > +	wake_up_interruptible(&isp_ctx->isp_deque_thread.wq);
> 
> I can see that all isp_deque_work() does is returning buffers to vb2. I
> don't think we need this intricate system to do that, as we could just do
> it inside the interrupt handler, in isp_irq_cam() directly.
> 

Ok, we will move all dequeue function in the ISR function and remove
this dequeue thread and related codes.

> > +
> > +	dev_dbg(dev,
> > +		"%s IRQ:0x%x DMA:0x%x seq:%d idx0:%d idx1:%d\n",
> > +		__func__,
> > +		(irq_status & INT_ST_MASK_CAM),
> > +		(dma_status & DMA_ST_MASK_CAM),
> > +		isp_dev->current_frame,
> > +		isp_dev->meta0_vb2_index,
> > +		isp_dev->meta1_vb2_index);
> > +}
> > +
> > +irqreturn_t isp_irq_cam(int irq, void *data)
> > +{
> > +	struct isp_device *isp_dev = (struct isp_device *)data;
> > +	struct isp_p1_device *p1_dev = get_p1_device(isp_dev->dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct device *dev = isp_dev->dev;
> > +	unsigned int cam_idx, cq_num, hw_frame_num;
> > +	unsigned int meta0_vb2_index, meta1_vb2_index;
> > +	unsigned int irq_status, err_status, dma_status;
> > +	unsigned int aao_fbc, afo_fbc;
> > +	unsigned long flags;
> > +
> > +	/* Check the streaming is off or not */
> > +	if (!p1_dev->cam_dev.streaming)
> > +		return IRQ_HANDLED;
> 
> This shouldn't be needed. The driver needs to unmask the interrupts in
> hardware registers when it starts streaming and mask them when it stops.
> Note that I mean the P1 hardware registers, not disable_irq(), which
> shouldn't be used.
> 

Fix in next patch.

> > +
> > +	cam_idx = isp_dev->isp_hw_module - ISP_CAM_A_IDX;
> > +	cq_num = 0;
> > +
> > +	spin_lock_irqsave(&isp_dev->spinlock_irq, flags);
> > +	irq_status = readl(isp_dev->regs + REG_CTL_RAW_INT_STAT);
> > +	dma_status = readl(isp_dev->regs + REG_CTL_RAW_INT2_STAT);
> > +	hw_frame_num = readl(isp_dev->regs + REG_HW_FRAME_NUM);
> > +	meta0_vb2_index = readl(isp_dev->regs + REG_META0_VB2_INDEX);
> > +	meta1_vb2_index = readl(isp_dev->regs + REG_META1_VB2_INDEX);
> 
> Hmm, reading vb2 buffer index from hardware registers? Was this hardware
> designed exclusively for V4L2? ;)
> 
> Jokes aside, how does the hardware know V4L2 buffer indexes?
> 

This is explained in the above.

> > +	aao_fbc = readl(isp_dev->regs + REG_AAO_FBC_STATUS);
> > +	afo_fbc = readl(isp_dev->regs + REG_AFO_FBC_STATUS);
> > +	spin_unlock_irqrestore(&isp_dev->spinlock_irq, flags);
> > +
> > +	/* Ignore unnecessary IRQ */
> > +	if (!irq_status && (!(dma_status & DMA_ST_MASK_CAM)))
> > +		return IRQ_HANDLED;
> 
> Unnecessary IRQs should be masked in the hardware IRQ mask registers. If we
> get an interrupt without any unmasked hardware IRQs active in the status,
> that's an error somewhere and we should return IRQ_NONE.
> 

Ok, we will check the IRQ EN register firstly and check any unmasked
IRQs for IRQ_NONE case.

> > +
> > +	err_status = irq_status & INT_ST_MASK_CAM_ERR;
> > +
> > +	/* Sof, done order check */
> > +	if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST)) {
> > +		dev_dbg(dev, "sof_done block cnt:%d\n", isp_dev->sof_count);
> > +
> > +		/* Notify IRQ event and enqueue frame */
> > +		irq_handle_notify_event(isp_dev, irq_status, dma_status, 0);
> > +		isp_dev->current_frame = hw_frame_num;
> 
> What exactly is hw_frame_num? Shouldn't we assign it before notifying the
> event?
> 

This is a another spare register for frame sequence number usage.
It comes from struct p1_frame_param:frame_seq_no which is sent by
SCP_ISP_FRAME IPI command. We will rename this to dequeue_frame_seq_no.
Is it a better understanding?

Below is our frame request handling in current design.

1. Buffer preparation
- Combined image buffers (IMGO/RRZO) + meta input buffer (Tuining) +
other meta histogram buffers (LCSO/LMVO) into one request.
- Accumulated one unique frame sequence number to each request and send
this request to the SCP composer to compose CQ (Command queue) buffer
via SCP_ISP_FRAME IPI command.
- CQ buffer is frame registers set. If ISP registers should be updated
per frame, these registers are configured in the CQ buffer, such as
frame sequence number, DMA addresses and tuning ISP registers.
- One frame request will be composed into one CQ buffer.Once CQ buffer
is composed done and kernel driver will receive ISP_CMD_FRAME_ACK with
its corresponding frame sequence number. Based on this, kernel driver
knows which request is ready to be en-queued and save this with
p1_dev->isp_ctx.composed_frame_id.
- The maximum number of CQ buffers in SCP is 3.

2. Buffer en-queue flow
- In order to configure correct CQ buffer setting before next SQF event,
it is depended on by MTK ISP P1 HW CQ mechanism.
- The basic concept of CQ mechanism is loaded ISP CQ buffer settings
when HW_PASS1_DON_ST is received which means DMA output is done.
- Btw, the pre-condition of this, need to tell ISP HW which CQ buffer
address is used. Otherwise, it will loaded one dummy CQ buffer to
bypass.
- So we will check available CQ buffers by comparing composed frame
sequence number & dequeued frame sequence from ISP HW in SOF event.
- If there are available CQ buffers, update the CQ base address to the
next CQ buffer address based on current de-enqueue frame sequence
number. So MTK ISP P1 HW will load this CQ buffer into HW when
HW_PASS1_DON_ST is triggered which is before the next SOF.
- So in next SOF event, ISP HW starts to output DMA buffers with this
request until request is done.
- But, for the first request, it is loaded into HW manually when
streaming is on for better performance.

3. Buffer de-queue flow
- We will use frame sequence number to decide which request is ready to
de-queue.
- We will save some important register setting from ISP HW when SOF is
received. This is because the ISP HW starts to output the data with the
corresponding settings, especially frame sequence number setting.
- When receiving SW_PASS1_DON_ST IRQ event, it means the DMA output is
done. So we could call isp_deque_request_frame with frame sequence
number to de-queue frame to VB2
- For AAO/AFO buffers, it has similar design concept. Sometimes, if only
AAO/AFO non-request buffers are en-queued without request buffers at the
same time, there will be no SW P1 done event for AAO/AFO DMA done.
Needs to depend on other IRQ events, such as AAO/AFO_DONE_EVENT.
- Due to CQ buffer number limitation, if we receive SW_PASS1_DONT_ST,
we may try to send another request to SCP for composing.

Hopefully, my explanation is helpful for better understanding our
implementation. If you still have any questions, please let me know. 

> > +		isp_dev->meta0_vb2_index = meta0_vb2_index;
> > +		isp_dev->meta1_vb2_index = meta1_vb2_index;
> > +	} else {
> > +		if (irq_status & SOF_INT_ST) {
> > +			isp_dev->current_frame = hw_frame_num;
> > +			isp_dev->meta0_vb2_index = meta0_vb2_index;
> > +			isp_dev->meta1_vb2_index = meta1_vb2_index;
> > +		}
> > +		irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
> > +	}
> 
> The if and else blocks do almost the same things just in different order. Is
> it really expected?
> 

If we receive HW_PASS1_DON_ST & SOF_INT_ST IRQ events at the same time,
the correct sequence should be handle HW_PASS1_DON_ST firstly to check
any de-queued frame and update the next frame setting later.
Normally, this is a corner case or system performance issue.

Btw, we will revise the above implementation as below.


if (irq_status & SOF_INT_ST)
	mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev,
					     dequeue_frame_seq_no);

/* Sof, done order check */
if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST))
	dev_warn(dev, "sof_done block cnt:%d\n", p1_dev->sof_count);

/* Notify IRQ event and de-enqueue frame */
irq_handle_notify_event(p1_dev, irq_status, dma_status);

/* Update frame settings & CQ address for frame en-queue */
enqueue_frame_seq_no = 0;
if (irq_status & SOF_INT_ST)
	enqueue_frame_seq_no = irq_handle_sof(p1_dev,
					      dequeue_frame_seq_no,
					      meta0_vb2_index,
					      meta1_vb2_index); 

> > +
> > +	if (irq_status & SOF_INT_ST)
> > +		cq_num = irq_handle_sof(isp_dev, isp_ctx->scp_mem_iova,
> > +					hw_frame_num);
> > +
> > +	/* Check ISP error status */
> > +	if (err_status) {
> > +		dev_err(dev,
> > +			"raw_int_err:0x%x/0x%x\n",
> > +			irq_status, err_status);
> > +		/* Show DMA errors in detail */
> > +		if (err_status & DMA_ERR_ST)
> > +			isp_dump_dma_status(isp_dev);
> > +	}
> > +
> > +	if (irq_status & INT_ST_LOG_MASK_CAM)
> > +		dev_dbg(dev, IRQ_STAT_STR,
> 
> Please just put that string here, otherwise the reader would have no idea
> what message is being printed here.
> 

Fix in next patch.

> > +			'A' + cam_idx,
> > +			isp_dev->sof_count,
> > +			irq_status,
> > +			dma_status,
> > +			hw_frame_num,
> > +			cq_num,
> > +			aao_fbc,
> > +			afo_fbc);
> 
> nit: No need to put each argument in its own line.
> 

Fix in next patch.

> > +
> > +	return IRQ_HANDLED;
> > +}
> > +
> > +static int isp_setup_scp_rproc(struct isp_p1_device *p1_dev)
> > +{
> > +	phandle rproc_phandle;
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	int ret;
> > +
> > +	p1_dev->scp_pdev = scp_get_pdev(p1_dev->pdev);
> > +	if (!p1_dev->scp_pdev) {
> > +		dev_err(dev, "Failed to get scp device\n");
> > +		return -ENODEV;
> > +	}
> > +
> > +	ret = of_property_read_u32(dev->of_node, "mediatek,scp",
> > +				   &rproc_phandle);
> > +	if (ret) {
> > +		dev_err(dev, "fail to get rproc_phandle:%d\n", ret);
> > +		return -EINVAL;
> > +	}
> > +
> > +	p1_dev->rproc_handle = rproc_get_by_phandle(rproc_phandle);
> > +	dev_dbg(dev, "p1 rproc_phandle: 0x%pK\n\n", p1_dev->rproc_handle);
> > +	if (!p1_dev->rproc_handle) {
> > +		dev_err(dev, "fail to get rproc_handle\n");
> > +		return -EINVAL;
> > +	}
> 
> This look-up should be done once in probe(). Only the rproc_boot() should
> happen dynamically.
> 

Fix in next patch.

> > +
> > +	ret = rproc_boot(p1_dev->rproc_handle);
> > +	if (ret) {
> > +		/*
> > +		 * Return 0 if downloading firmware successfully,
> > +		 * otherwise it is failed
> > +		 */
> > +		return -ENODEV;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int isp_init_context(struct isp_p1_device *p1_dev)
> > +{
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	unsigned int i;
> > +
> > +	dev_dbg(dev, "init irq work thread\n");
> > +	if (!isp_ctx->isp_deque_thread.thread) {
> > +		init_waitqueue_head(&isp_ctx->isp_deque_thread.wq);
> > +		isp_ctx->isp_deque_thread.thread =
> > +			kthread_run(isp_deque_work, (void *)p1_dev,
> > +				    "isp_deque_work");
> > +		if (IS_ERR(isp_ctx->isp_deque_thread.thread)) {
> > +			dev_err(dev, "unable to alloc kthread\n");
> > +			isp_ctx->isp_deque_thread.thread = NULL;
> > +			return -ENOMEM;
> > +		}
> > +	}
> > +	spin_lock_init(&isp_ctx->irq_dequeue_lock);
> > +	mutex_init(&isp_ctx->lock);
> > +
> > +	INIT_LIST_HEAD(&isp_ctx->p1_enqueue_list.queue);
> > +	atomic_set(&isp_ctx->p1_enqueue_list.queue_cnt, 0);
> > +
> > +	for (i = 0; i < ISP_DEV_NODE_NUM; i++)
> > +		spin_lock_init(&p1_dev->isp_devs[i].spinlock_irq);
> > +
> > +	spin_lock_init(&isp_ctx->p1_enqueue_list.lock);
> > +	spin_lock_init(&isp_ctx->composer_txlist.lock);
> > +
> > +	atomic_set(&isp_ctx->irq_data_end, 0);
> > +	atomic_set(&isp_ctx->irq_data_start, 0);
> > +
> > +	return 0;
> 
> Everything here looks like something that should be done once in probe. I
> also don't see a point of having a separate mtk_isp_p1_ctx struct for the
> data above. It could be just located in p1_dev, at least for now.
> 
> If we end up implementing support for multiple contexts, we could isolate
> per-context data then, once we know what's really per-context. For now,
> let's keep it simple.
> 

Ok, we will remove isp_ctx structure and move some fields into p1_dev or
cam_dev,

> > +}
> > +
> > +static int isp_uninit_context(struct device *dev)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct mtk_isp_queue_job *framejob, *tmp_framejob;
> > +
> > +	spin_lock_irq(&isp_ctx->p1_enqueue_list.lock);
> > +	list_for_each_entry_safe(framejob, tmp_framejob,
> > +				 &isp_ctx->p1_enqueue_list.queue, list_entry) {
> > +		list_del(&framejob->list_entry);
> > +		kfree(framejob);
> > +	}
> > +	spin_unlock_irq(&isp_ctx->p1_enqueue_list.lock);
> > +
> > +	if (isp_ctx->isp_deque_thread.thread) {
> > +		kthread_stop(isp_ctx->isp_deque_thread.thread);
> > +		wake_up_interruptible(&isp_ctx->isp_deque_thread.wq);
> > +		isp_ctx->isp_deque_thread.thread = NULL;
> > +	}
> > +
> > +	mutex_destroy(&isp_ctx->lock);
> > +
> > +	return 0;
> > +}
> > +
> > +static unsigned int get_enabled_dma_ports(struct mtk_cam_dev *cam_dev)
> > +{
> > +	unsigned int enabled_dma_ports, i;
> > +
> > +	/* Get the enabled meta DMA ports */
> > +	enabled_dma_ports = 0;
> > +
> > +	for (i = 0; i < MTK_CAM_P1_TOTAL_NODES; i++)
> > +		if (cam_dev->vdev_nodes[i].enabled)
> > +			enabled_dma_ports |=
> > +				cam_dev->vdev_nodes[i].desc.dma_port;
> > +
> > +	dev_dbg(&cam_dev->pdev->dev, "%s :0x%x", __func__, enabled_dma_ports);
> > +
> > +	return enabled_dma_ports;
> > +}
> > +
> > +/* Utility functions */
> > +static unsigned int get_sensor_pixel_id(unsigned int fmt)
> > +{
> > +	switch (fmt) {
> > +	case MEDIA_BUS_FMT_SBGGR8_1X8:
> > +	case MEDIA_BUS_FMT_SBGGR10_1X10:
> > +	case MEDIA_BUS_FMT_SBGGR12_1X12:
> > +	case MEDIA_BUS_FMT_SBGGR14_1X14:
> > +		return RAW_PXL_ID_B;
> > +	case MEDIA_BUS_FMT_SGBRG8_1X8:
> > +	case MEDIA_BUS_FMT_SGBRG10_1X10:
> > +	case MEDIA_BUS_FMT_SGBRG12_1X12:
> > +	case MEDIA_BUS_FMT_SGBRG14_1X14:
> > +		return RAW_PXL_ID_GB;
> > +	case MEDIA_BUS_FMT_SGRBG8_1X8:
> > +	case MEDIA_BUS_FMT_SGRBG10_1X10:
> > +	case MEDIA_BUS_FMT_SGRBG12_1X12:
> > +	case MEDIA_BUS_FMT_SGRBG14_1X14:
> > +		return RAW_PXL_ID_GR;
> > +	case MEDIA_BUS_FMT_SRGGB8_1X8:
> > +	case MEDIA_BUS_FMT_SRGGB10_1X10:
> > +	case MEDIA_BUS_FMT_SRGGB12_1X12:
> > +	case MEDIA_BUS_FMT_SRGGB14_1X14:
> > +		return RAW_PXL_ID_R;
> > +	default:
> 
> Could we fail here instead?
> 

Ok, we will return invalid value here, like MTK_CAM_RAW_PXL_ID_UNKNOWN.
We will also check this error in the caller.

> > +		return RAW_PXL_ID_B;
> > +	}
> > +}
> > +
> > +static unsigned int get_sensor_fmt(unsigned int fmt)
> > +{
> > +	switch (fmt) {
> > +	case MEDIA_BUS_FMT_SBGGR8_1X8:
> > +	case MEDIA_BUS_FMT_SGBRG8_1X8:
> > +	case MEDIA_BUS_FMT_SGRBG8_1X8:
> > +	case MEDIA_BUS_FMT_SRGGB8_1X8:
> > +		return IMG_FMT_BAYER8;
> > +	case MEDIA_BUS_FMT_SBGGR10_1X10:
> > +	case MEDIA_BUS_FMT_SGBRG10_1X10:
> > +	case MEDIA_BUS_FMT_SGRBG10_1X10:
> > +	case MEDIA_BUS_FMT_SRGGB10_1X10:
> > +		return IMG_FMT_BAYER10;
> > +	case MEDIA_BUS_FMT_SBGGR12_1X12:
> > +	case MEDIA_BUS_FMT_SGBRG12_1X12:
> > +	case MEDIA_BUS_FMT_SGRBG12_1X12:
> > +	case MEDIA_BUS_FMT_SRGGB12_1X12:
> > +		return IMG_FMT_BAYER12;
> > +	case MEDIA_BUS_FMT_SBGGR14_1X14:
> > +	case MEDIA_BUS_FMT_SGBRG14_1X14:
> > +	case MEDIA_BUS_FMT_SGRBG14_1X14:
> > +	case MEDIA_BUS_FMT_SRGGB14_1X14:
> > +		return IMG_FMT_BAYER14;
> > +	default:
> > +		return IMG_FMT_UNKNOWN;
> > +	}
> > +}
> > +
> > +static unsigned int get_img_fmt(unsigned int fourcc)
> > +{
> > +	switch (fourcc) {
> > +	case V4L2_PIX_FMT_MTISP_B8:
> > +		return IMG_FMT_BAYER8;
> > +	case V4L2_PIX_FMT_MTISP_F8:
> > +		return IMG_FMT_FG_BAYER8;
> > +	case V4L2_PIX_FMT_MTISP_B10:
> > +		return IMG_FMT_BAYER10;
> > +	case V4L2_PIX_FMT_MTISP_F10:
> > +		return IMG_FMT_FG_BAYER10;
> > +	case V4L2_PIX_FMT_MTISP_B12:
> > +		return IMG_FMT_BAYER12;
> > +	case V4L2_PIX_FMT_MTISP_F12:
> > +		return IMG_FMT_FG_BAYER12;
> > +	case V4L2_PIX_FMT_MTISP_B14:
> > +		return IMG_FMT_BAYER14;
> > +	case V4L2_PIX_FMT_MTISP_F14:
> > +		return IMG_FMT_FG_BAYER14;
> > +	default:
> > +		return IMG_FMT_UNKNOWN;
> > +	}
> > +}
> > +
> > +static unsigned int get_pixel_byte(unsigned int fourcc)
> > +{
> > +	switch (fourcc) {
> > +	case V4L2_PIX_FMT_MTISP_B8:
> > +	case V4L2_PIX_FMT_MTISP_F8:
> > +		return 8;
> > +	case V4L2_PIX_FMT_MTISP_B10:
> > +	case V4L2_PIX_FMT_MTISP_F10:
> > +		return 10;
> > +	case V4L2_PIX_FMT_MTISP_B12:
> > +	case V4L2_PIX_FMT_MTISP_F12:
> > +		return 12;
> > +	case V4L2_PIX_FMT_MTISP_B14:
> > +	case V4L2_PIX_FMT_MTISP_F14:
> > +		return 14;
> > +	default:
> 
> Could we fail here instead, so that we don't mask some potential errors?
> 

Ok, we will return MTK_CAM_IMG_FMT_UNKNOWN and check this error in the
caller.

> > +		return 10;
> > +	}
> > +}
> > +
> > +static void config_img_fmt(struct device *dev, struct p1_img_output *out_fmt,
> > +			   const struct v4l2_format *in_fmt,
> > +			   const struct v4l2_subdev_format *sd_format)
> > +{
> > +	out_fmt->img_fmt = get_img_fmt(in_fmt->fmt.pix_mp.pixelformat);
> > +	out_fmt->pixel_byte = get_pixel_byte(in_fmt->fmt.pix_mp.pixelformat);
> > +	out_fmt->size.w = in_fmt->fmt.pix_mp.width;
> > +	out_fmt->size.h = in_fmt->fmt.pix_mp.height;
> > +
> > +	out_fmt->size.stride = in_fmt->fmt.pix_mp.plane_fmt[0].bytesperline;
> > +	out_fmt->size.xsize = in_fmt->fmt.pix_mp.plane_fmt[0].bytesperline;
> 
> Please group operations on the same field together, i.e. remove the blank
> line above size.stride and add one blank line above size.w.
> 

Fix in next patch.

> > +
> > +	out_fmt->crop.left = 0x0;
> > +	out_fmt->crop.top = 0x0;
> > +
> 
> Remove the blank line.
> 

Fix in next patch.

> > +	out_fmt->crop.width = sd_format->format.width;
> > +	out_fmt->crop.height = sd_format->format.height;
> > +
> > +	WARN_ONCE(in_fmt->fmt.pix_mp.width > out_fmt->crop.width ||
> > +		  in_fmt->fmt.pix_mp.height > out_fmt->crop.height,
> > +		  "img out:%d:%d in:%d:%d",
> > +		  in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height,
> > +		  out_fmt->crop.width, out_fmt->crop.height);
> 
> We should check this earlier and fail the streaming start if there is a
> mismatch between sensor and video node configuration.
> 

Fix in next patch.

> > +
> > +	dev_dbg(dev, "pixel_byte:%d img_fmt:0x%x\n",
> > +		out_fmt->pixel_byte,
> > +		out_fmt->img_fmt);
> > +	dev_dbg(dev,
> > +		"param:size=%0dx%0d, stride:%d, xsize:%d, crop=%0dx%0d\n",
> > +		out_fmt->size.w, out_fmt->size.h,
> > +		out_fmt->size.stride, out_fmt->size.xsize,
> > +		out_fmt->crop.width, out_fmt->crop.height);
> > +}
> > +
> > +/* ISP P1 interface functions */
> > +int mtk_isp_power_init(struct mtk_cam_dev *cam_dev)
> > +{
> > +	struct device *dev = &cam_dev->pdev->dev;
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	int ret;
> > +
> > +	ret = isp_setup_scp_rproc(p1_dev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = isp_init_context(p1_dev);
> > +	if (ret)
> > +		return ret;
> 
> The above function doesn't really seem to be related to power management.
> Should it be called from subdev stream on?
> 

We will rename this function to mtk_isp_hw_init.
But, it will be called when the first video node is streamed on.
This is because we need to initialize the HW firstly for sub-device
stream-on performance.  We need to send some IPI commands, such as
ISP_CMD_INIT & ISP_CMD_CONFIG_META & ISP_CMD_ENQUEUE_META in this
timing.

> > +
> > +	ret = isp_composer_init(dev);
> > +	if (ret)
> > +		goto composer_err;
> 
> This also doesn't seem to be related to power management.
> 

Ok, we will rename to mtk_isp_hw_init to avoid misunderstanding.

> > +
> > +	pm_runtime_get_sync(dev);
> > +
> > +	/* ISP HW INIT */
> > +	isp_ctx->isp_hw_module = ISP_CAM_B_IDX;
> 
> There is some amount of code handling various values of isp_hw_module in
> this driver. If we're hardcoding ISP_CAM_B_IDX here, it's basically dead
> code that can't be tested. Please either add support for all the indexes in
> the driver or simplify all the code to just handle CAM_B.
> 

Ok, we will simplify driver code to support single CAM only.
We will remove isp_hw_module usage in the source code.

> > +	/* Use pure RAW as default HW path */
> > +	isp_ctx->isp_raw_path = ISP_PURE_RAW_PATH;
> > +	atomic_set(&p1_dev->cam_dev.streamed_node_count, 0);
> > +
> > +	isp_composer_hw_init(dev);
> > +	/* Check enabled DMAs which is configured by media setup */
> > +	isp_composer_meta_config(dev, get_enabled_dma_ports(cam_dev));
> 
> Hmm, this seems to be also configured by isp_compoer_hw_config(). Are both
> necessary?
> 

Yes, it is necessary for non-request buffers design for Camera 3A video
nodes. For 3A video nodes, we just want to know which 3A video nodes are
enabled in ISP_CMD_CONFIG_META. In this stage, we may not know the image
format from user space. So we just pass the enabled DMA information from
kernel to SCP only. With 3A enabled DMA information, we could configure
related 3A registers in SCP.

> > +
> > +	dev_dbg(dev, "%s done\n", __func__);
> > +
> > +	return 0;
> > +
> > +composer_err:
> > +	isp_uninit_context(dev);
> > +
> > +	return ret;
> > +}
> > +
> > +int mtk_isp_power_release(struct device *dev)
> > +{
> > +	isp_composer_hw_deinit(dev);
> > +	isp_uninit_context(dev);
> 
> These two don't seem to be related to power either.
> 
> Instead, I don't see anything that could undo the rproc_boot() operation
> here.
> 

We will rename this function to mtk_isp_hw_release.
We will also add rproc_shutdown function call here.

int mtk_isp_hw_release(struct mtk_cam_dev *cam)
{
	struct device *dev = cam->dev;
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);

	isp_composer_hw_deinit(p1_dev);
	pm_runtime_put_sync_autosuspend(dev);
	rproc_shutdown(p1_dev->rproc_handle);

	dev_dbg(dev, "%s done\n", __func__);

	return 0;
}

> > +
> > +	dev_dbg(dev, "%s done\n", __func__);
> > +
> > +	return 0;
> > +}
> > +
> > +int mtk_isp_config(struct device *dev)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct p1_config_param config_param;
> > +	struct mtk_cam_dev *cam_dev = &p1_dev->cam_dev;
> > +	struct v4l2_subdev_format sd_fmt;
> > +	unsigned int enabled_dma_ports;
> > +	struct v4l2_format *img_fmt;
> > +	int ret;
> > +
> > +	p1_dev->isp_devs[isp_ctx->isp_hw_module].current_frame = 0;
> > +	p1_dev->isp_devs[isp_ctx->isp_hw_module].sof_count = 0;
> > +
> > +	isp_ctx->frame_seq_no = 1;
> > +	atomic_set(&isp_ctx->composed_frame_id, 0);
> > +
> > +	/* Get the enabled DMA ports */
> > +	enabled_dma_ports = get_enabled_dma_ports(cam_dev);
> > +	dev_dbg(dev, "%s enable_dma_ports:0x%x", __func__, enabled_dma_ports);
> > +
> > +	/* Sensor config */
> > +	sd_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > +	ret = v4l2_subdev_call(cam_dev->sensor, pad, get_fmt, NULL, &sd_fmt);
> > +
> 
> Unnecessary blank line.
> 

Fix in next patch.

> > +	if (ret) {
> > +		dev_dbg(dev, "sensor g_fmt on failed:%d\n", ret);
> > +		return -EPERM;
> 
> return ret?
> 

Fix in next patch.

> > +	}
> > +
> > +	dev_dbg(dev,
> > +		"get_fmt ret=%d, w=%d, h=%d, code=0x%x, field=%d, color=%d\n",
> > +		ret, sd_fmt.format.width, sd_fmt.format.height,
> > +		sd_fmt.format.code, sd_fmt.format.field,
> > +		sd_fmt.format.colorspace);
> > +
> > +	config_param.cfg_in_param.continuous = 0x1;
> > +	config_param.cfg_in_param.subsample = 0x0;
> > +	/* Fix to one pixel mode in default */
> > +	config_param.cfg_in_param.pixel_mode = 0x1;
> > +	/* Support normal pattern in default */
> > +	config_param.cfg_in_param.data_pattern = 0x0;
> > +
> > +	config_param.cfg_in_param.crop.left = 0x0;
> > +	config_param.cfg_in_param.crop.top = 0x0;
> 
> Why hexadecimal numerals? Also, what's the meaning of these values? For
> anything boolean, you could just use true and false as a substitute of 0 and
> 1. For anything that has more values, please define the values using macros.
> 

Fix in next patch.
1. Remove unnecessary hexadecimal number usage
2. Use boolean value if possible
3. Assign non-empty field values.

Below is the fixed version.

	config_param.cfg_in_param.continuous = true;
	/* Fix to one pixel mode in default */
	config_param.cfg_in_param.pixel_mode = MTK_ISP_ONE_PIXEL_MODE;
	config_param.cfg_in_param.crop.width = sd_fmt.format.width;
	config_param.cfg_in_param.crop.height = sd_fmt.format.height;
	config_param.cfg_in_param.raw_pixel_id =
		get_sensor_pixel_id(sd_fmt.format.code);
	config_param.cfg_in_param.img_fmt = get_sensor_fmt(sd_fmt.format.code);
	if (config_param.cfg_in_param.img_fmt == MTK_CAM_IMG_FMT_UNKNOWN ||
	    config_param.cfg_in_param.raw_pixel_id ==
MTK_CAM_RAW_PXL_ID_UNKNOWN) {
		dev_err(dev, "Unknown cfg img fmt or unknown raw pixel id\n");
		return -EINVAL;
	}

> > +
> > +	config_param.cfg_in_param.raw_pixel_id =
> > +		get_sensor_pixel_id(sd_fmt.format.code);
> > +	config_param.cfg_in_param.img_fmt = get_sensor_fmt(sd_fmt.format.code);
> > +	config_param.cfg_in_param.crop.width = sd_fmt.format.width;
> > +	config_param.cfg_in_param.crop.height = sd_fmt.format.height;
> 
> Move the other crop settings from above to here.
> 

Ditto.

> > +
> > +	config_param.cfg_main_param.bypass = 1;
> > +	img_fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_MAIN_STREAM_OUT].vdev_fmt;
> > +	if ((enabled_dma_ports & R_IMGO) == R_IMGO) {
> 
> No need for the == R_IMGO part.
> 

Since R_IMGO is mandatory video node, we will remove this checking.

> > +		config_param.cfg_main_param.bypass = 0;
> > +		config_param.cfg_main_param.pure_raw = isp_ctx->isp_raw_path;
> > +		config_param.cfg_main_param.pure_raw_pack = 1;
> > +		config_img_fmt(dev, &config_param.cfg_main_param.output,
> > +			       img_fmt, &sd_fmt);
> > +	}
> > +
> > +	config_param.cfg_resize_param.bypass = 1;
> > +	img_fmt = &cam_dev->vdev_nodes[MTK_CAM_P1_PACKED_BIN_OUT].vdev_fmt;
> > +	if ((enabled_dma_ports & R_RRZO) == R_RRZO) {
> 
> Ditto.
> 

Fix in next patch as below:

if (enabled_dma_ports & R_RRZO) {
	ret = config_img_fmt(cam,
			     &config_param.cfg_resize_param.output,
			     img_fmt, &sd_fmt);
	if (ret)
		return ret;
} else {
	config_param.cfg_resize_param.bypass = true;
}

> > +		config_param.cfg_resize_param.bypass = 0;
> > +		config_img_fmt(dev, &config_param.cfg_resize_param.output,
> > +			       img_fmt, &sd_fmt);
> > +	}
> > +
> > +	/* Configure meta DMAs info. */
> > +	config_param.cfg_meta_param.enabled_meta_dmas = enabled_dma_ports;
> 
> Should image DMAs be masked out of this bitmap?
> 

We will replace struct cfg_meta_param with enabled_dmas integer.
So we can pass all enabled DMA masks to SCP. 

> > +
> > +	isp_composer_hw_config(dev, &config_param);
> > +
> > +	dev_dbg(dev, "%s done\n", __func__);
> > +
> > +	return 0;
> > +}
> > +
> > +void mtk_isp_enqueue(struct device *dev, unsigned int dma_port,
> > +		     struct mtk_cam_dev_buffer *buffer)
> > +{
> > +	struct mtk_isp_scp_p1_cmd frameparams;
> > +
> > +	memset(&frameparams, 0, sizeof(frameparams));
> > +	frameparams.cmd_id = ISP_CMD_ENQUEUE_META;
> > +	frameparams.meta_frame.enabled_dma = dma_port;
> > +	frameparams.meta_frame.vb_index = buffer->vbb.vb2_buf.index;
> > +	frameparams.meta_frame.meta_addr.iova = buffer->daddr;
> > +	frameparams.meta_frame.meta_addr.scp_addr = buffer->scp_addr;
> > +
> > +	isp_composer_enqueue(dev, &frameparams, SCP_ISP_CMD);
> > +}
> > +
> > +void mtk_isp_req_flush_buffers(struct device *dev)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	struct mtk_isp_queue_job *job, *j0;
> > +	struct mtk_cam_dev_buffer *buf, *b0;
> > +	struct isp_queue *p1_list = &p1_dev->isp_ctx.p1_enqueue_list;
> > +
> > +	if (!atomic_read(&p1_list->queue_cnt))
> > +		return;
> 
> Do we need this explicit check? The code below wouldn't do anything if there
> isn't anything in the list. IMHO we could even remove queue_cnt completely.
> 

Since we have redesign request en-queue mechanism, this function will be
removed.

> > +
> > +	spin_lock(&p1_list->lock);
> > +	list_for_each_entry_safe(job, j0, &p1_list->queue, list_entry) {
> 
> nit: s/j0/job_prev/
> 

Will apply this naming rule in next patch.

> > +		list_for_each_entry_safe(buf, b0, &job->list_buf, list) {
> 
> nit: s/b0/buf_pref/
> 

Ditto.

> Also, we should be able to replace this with iterating over the generic list
> of request objects, rather than this internal list.
> 
> > +			list_del(&buf->list);
> > +			if (buf->vbb.vb2_buf.state == VB2_BUF_STATE_ACTIVE)
> 
> It shouldn't be possible to happen. If you see this check failing, that
> means a problem somewhere else in the driver.
> 

Fix in next patch.

> > +				vb2_buffer_done(&buf->vbb.vb2_buf,
> > +						VB2_BUF_STATE_ERROR);
> > +		}
> > +		list_del(&job->list_entry);
> > +		atomic_dec(&p1_list->queue_cnt);
> > +		kfree(job);
> > +	}
> > +	spin_unlock(&p1_list->lock);
> > +}
> > +
> > +void mtk_isp_req_enqueue(struct device *dev, struct media_request *req)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> 
> Just pass p1_dev to this function instead of dev.
> 

We got your point.
Below is new function declaration.

void mtk_isp_req_enqueue(struct mtk_cam_dev *cam,
			 struct mtk_cam_dev_request *req)

> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	struct p1_frame_param frameparams;
> > +	struct mtk_isp_queue_job *framejob;
> > +	struct media_request_object *obj, *obj_safe;
> > +	struct vb2_buffer *vb;
> > +	struct mtk_cam_dev_buffer *buf;
> > +
> > +	framejob = kzalloc(sizeof(*framejob), GFP_ATOMIC);
> 
> This allocation shouldn't be needed. The structure should be already a part
> of the mtk_cam_dev_request struct that wraps media_request. Actually. I'd
> even say that the contents of this struct should be just moved to that one
> to avoid overabstracting.
> 

For this function, we will apply the new design from P2 driver's
comment. Here is the new implementation.

struct mtk_cam_dev_request {
	struct media_request req;
	struct mtk_p1_frame_param frame_params;
	struct work_struct frame_work;
	struct list_head list;
	atomic_t buf_count;
};

void mtk_isp_req_enqueue(struct mtk_cam_dev *cam,
			 struct mtk_cam_dev_request *req)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);
	int ret;

	req->frame_params.frame_seq_no = ++p1_dev->enqueue_frame_seq_no;
	INIT_WORK(&req->frame_work, isp_tx_frame_worker);
	ret = queue_work(p1_dev->composer_wq, &req->frame_work);
	if (!ret)
		dev_dbg(cam->dev, "frame_no:%d queue_work failed\n",
			req->frame_params.frame_seq_no, ret);
	else
		dev_dbg(cam->dev, "Enqueue fd:%s frame_seq_no:%d job cnt:%d\n",
			req->req.debug_str, req->frame_params.frame_seq_no,
			atomic_read(&cam->running_job_count));
}

> > +	memset(framejob, 0, sizeof(*framejob));
> 
> Putting the above comment aside, kzalloc() already zeroes the memory.
> 

Ditto.

> > +	memset(&frameparams, 0, sizeof(frameparams));
> > +	INIT_LIST_HEAD(&framejob->list_buf);
> > +
> > +	frameparams.frame_seq_no = isp_ctx->frame_seq_no++;
> > +	frameparams.sof_idx =
> > +		p1_dev->isp_devs[isp_ctx->isp_hw_module].sof_count;
> 
> How is this synchronized with the sof_count increment in irq_handle_sof()?
> 

The sof_idx field is only used for debugging purpose.
We will remove this.

> > +	framejob->frame_seq_no = frameparams.frame_seq_no;
> > +
> > +	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
> > +		vb = container_of(obj, struct vb2_buffer, req_obj);
> 
> We should check that the object type before assuming it's a vb2_buffer by
> calling vb2_request_object_is_buffer().
> 

Fix in next patch.

> > +		buf = container_of(vb, struct mtk_cam_dev_buffer, vbb.vb2_buf);
> > +		framejob->request_fd = buf->vbb.request_fd;
> 
> We shouldn't use request_fd as the key here. We already have req here, which
> is the right key to use.
> 
> That said, I can see framejob->request_fd only used for printing a debugging
> message in mtk_cam_dev_job_finish(). The request API core should already
> print something for us once a request is completed, so perhaps that isn't
> needed?
> 

Fix in next patch.

> > +		frameparams.dma_buffers[buf->node_id].iova = buf->daddr;
> > +		frameparams.dma_buffers[buf->node_id].scp_addr = buf->scp_addr;
> > +		list_add_tail(&buf->list, &framejob->list_buf);
> 
> Why do we need this private list? We could just call exactly the same
> list_for_each() over the request objects.
> 

This function is re-designed.

> > +	}
> > +
> > +	spin_lock(&isp_ctx->p1_enqueue_list.lock);
> > +	list_add_tail(&framejob->list_entry, &isp_ctx->p1_enqueue_list.queue);
> 
> We already have a list head in mtk_cam_dev_request.
> 

This function is re-designed.

> > +	atomic_inc(&isp_ctx->p1_enqueue_list.queue_cnt);
> > +	spin_unlock(&isp_ctx->p1_enqueue_list.lock);
> > +
> > +	isp_composer_enqueue(dev, &frameparams, SCP_ISP_FRAME);
> > +	dev_dbg(dev, "request fd:%d frame_seq_no:%d is queued cnt:%d\n",
> > +		framejob->request_fd,
> > +		frameparams.frame_seq_no,
> > +		atomic_read(&isp_ctx->p1_enqueue_list.queue_cnt));
> > +}
> > +
> > +static int enable_sys_clock(struct isp_p1_device *p1_dev)
> > +{
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	int ret;
> > +
> > +	dev_info(dev, "- %s\n", __func__);
> 
> dev_dbg()
> 

Fix in next patch.

> > +
> > +	ret = clk_bulk_prepare_enable(p1_dev->isp_ctx.num_clks,
> > +				      p1_dev->isp_ctx.clk_list);
> > +	if (ret)
> > +		goto clk_err;
> > +
> > +	return 0;
> > +
> > +clk_err:
> > +	dev_err(dev, "cannot pre-en isp_cam clock:%d\n", ret);
> > +	clk_bulk_disable_unprepare(p1_dev->isp_ctx.num_clks,
> > +				   p1_dev->isp_ctx.clk_list);
> 
> clk_bulk_prepare_enable() returns without any clocks enabled if it fails, so
> this would disable the clocks second time.
> 

Fix in next patch.

> > +	return ret;
> > +}
> > +
> > +static void disable_sys_clock(struct isp_p1_device *p1_dev)
> > +{
> > +	dev_info(&p1_dev->pdev->dev, "- %s\n", __func__);
> 
> dev_dbg()
> 

Fix in next patch.

> > +	clk_bulk_disable_unprepare(p1_dev->isp_ctx.num_clks,
> > +				   p1_dev->isp_ctx.clk_list);
> > +}
> 
> There is no point in having wrapper functions to just call one function
> inside. Please just call clk_bulk_*() directly.
> 

Fix in next patch.

> > +
> > +static int mtk_isp_suspend(struct device *dev)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	int module = p1_dev->isp_ctx.isp_hw_module;
> > +	struct isp_device *isp_dev = &p1_dev->isp_devs[module];
> > +	unsigned int reg_val;
> > +
> > +	dev_dbg(dev, "- %s\n", __func__);
> > +
> 
> We need to check if the device isn't already runtime suspended. If it is, we
> don't have to do anything here and can just return.
> 
> 

Add pm_runtime_suspended(dev) to check.

> We also need to ensure that no new requests are queued to the hardware at
> this point. This could be done by replacing any of the kthreads with
> workqueues and making all of the workqueues freezable.
> 

Yes, we will use workqueue to send frame request.
Here is the workqueue's definition.

	p1_dev->composer_wq =
		alloc_ordered_workqueue(dev_name(p1_dev->dev),
					__WQ_LEGACY | WQ_MEM_RECLAIM |
					WQ_FREEZABLE);

> > +	isp_dev = &p1_dev->isp_devs[module];
> > +	reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
> > +	if (reg_val & VFDATA_EN_BIT) {
> > +		dev_dbg(dev, "Cam:%d suspend, disable VF\n", module);
> > +		/* Disable view finder */
> > +		writel((reg_val & (~VFDATA_EN_BIT)),
> > +		       isp_dev->regs + REG_TG_VF_CON);
> > +		/*
> > +		 * After VF enable, the TG frame count will be reset to 0;
> > +		 */
> > +		reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
> > +		writel((reg_val & (~CMOS_EN_BIT)),
> > +		       isp_dev->regs +  + REG_TG_SEN_MODE);
> > +	}
> 
> Are you sure this is the right handling? We need to make sure the hardware
> finishes processing the current frame and stops.
> 

We will revise this handling to make sure the ISP HW is idle before
suspend.

> > +
> > +	disable_sys_clock(p1_dev);
> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_isp_resume(struct device *dev)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	int module = p1_dev->isp_ctx.isp_hw_module;
> > +	struct isp_device *isp_dev = &p1_dev->isp_devs[module];
> > +	unsigned int reg_val;
> > +
> > +	dev_dbg(dev, "- %s\n", __func__);
> > +
> 
> We need to check runtime PM status here as well, because if the device was
> suspended, there is nothing to do here.
> 

Ditto.

> > +	enable_sys_clock(p1_dev);
> > +
> > +	/* V4L2 stream-on phase & restore HW stream-on status */
> > +	if (p1_dev->cam_dev.streaming) {
> > +		dev_dbg(dev, "Cam:%d resume,enable VF\n", module);
> > +		/* Enable CMOS */
> > +		reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
> > +		writel((reg_val | CMOS_EN_BIT),
> > +		       isp_dev->regs + REG_TG_SEN_MODE);
> > +		/* Enable VF */
> > +		reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
> > +		writel((reg_val | VFDATA_EN_BIT),
> > +		       isp_dev->regs + REG_TG_VF_CON);
> > +	}
> 
> Does the hardware keep all the state, e.g. queued buffers, during suspend?
> Would the code above continue all the capture from the next buffer, as
> queued by the userspace before the suspend?
> 

Yes, we will test it.
1. SCP buffers are kept by SCP processor
2. ISP registers are still kept even if ISP clock is disable.

> > +
> > +	return 0;
> > +}
> > +
> > +static int mtk_isp_probe(struct platform_device *pdev)
> > +{
> > +	struct isp_p1_device *p1_dev;
> > +	struct mtk_isp_p1_ctx *isp_ctx;
> > +	struct isp_device *isp_dev;
> > +	struct device *dev = &pdev->dev;
> > +	struct resource *res;
> > +	int irq;
> > +	int ret;
> > +	unsigned int i;
> > +
> > +	p1_dev = devm_kzalloc(dev, sizeof(*p1_dev), GFP_KERNEL);
> > +	if (!p1_dev)
> > +		return -ENOMEM;
> > +
> > +	dev_set_drvdata(dev, p1_dev);
> > +	isp_ctx = &p1_dev->isp_ctx;
> > +	p1_dev->pdev = pdev;
> 
> Perhaps you want to store &pdev->dev instead of pdev? I'm not sure a
> reference to platform_device is very useful later in the code.
> 

Fix in next patch.

> > +
> > +	for (i = ISP_CAMSYS_CONFIG_IDX; i < ISP_DEV_NODE_NUM; i++) {
> 
> I think we want to start from 0 here?
> 

Because of single CAM support, we will revise our DTS tree to support
single CAM only. So we could remove this loop and check the CAM-B HW
information here. Here is below new function.

static int mtk_isp_probe(struct platform_device *pdev)
{
	/* List of clocks required by isp cam */
	static const char * const clk_names[] = {
		"camsys_cam_cgpdn", "camsys_camtg_cgpdn"
	};
	struct mtk_isp_p1_device *p1_dev;
	struct device *dev = &pdev->dev;
	struct resource *res;
	int irq, ret, i;

	p1_dev = devm_kzalloc(dev, sizeof(*p1_dev), GFP_KERNEL);
	if (!p1_dev)
		return -ENOMEM;

	p1_dev->dev = dev;
	dev_set_drvdata(dev, p1_dev);

	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
	p1_dev->regs = devm_ioremap_resource(dev, res);
	if (IS_ERR(p1_dev->regs)) {
		dev_err(dev, "Failed platform resources map\n");
		return PTR_ERR(p1_dev->regs);
	}
	dev_dbg(dev, "cam, map_addr=0x%pK\n", p1_dev->regs);

	irq = platform_get_irq(pdev, 0);
	if (!irq) {
		dev_err(dev, "Missing IRQ resources data\n");
		return -ENODEV;
	}
	ret = devm_request_irq(dev, irq, isp_irq_cam, IRQF_SHARED,
			       dev_name(dev), p1_dev);
	if (ret) {
		dev_err(dev, "req_irq fail, dev:%s irq=%d\n",
			dev->of_node->name, irq);
		return ret;
	}
	dev_dbg(dev, "Reg. irq=%d, isr:%s\n", irq, dev_driver_string(dev));
	spin_lock_init(&p1_dev->spinlock_irq);

	p1_dev->num_clks = ARRAY_SIZE(clk_names);
	p1_dev->clks = devm_kcalloc(dev, p1_dev->num_clks,
				    sizeof(*p1_dev->clks), GFP_KERNEL);
	if (!p1_dev->clks)
		return -ENOMEM;

	for (i = 0; i < p1_dev->num_clks; ++i)
		p1_dev->clks[i].id = clk_names[i];

	ret = devm_clk_bulk_get(dev, p1_dev->num_clks, p1_dev->clks);
	if (ret) {
		dev_err(dev, "cannot get isp cam clock:%d\n", ret);
		return ret;
	}

	ret = isp_setup_scp_rproc(p1_dev, pdev);
	if (ret)
		return ret;

	pm_runtime_enable(dev);

	/* Initialize the v4l2 common part */
	ret = mtk_cam_dev_init(pdev, &p1_dev->cam_dev);
	if (ret)
		return ret;

	return 0;
}

> > +		isp_dev = &p1_dev->isp_devs[i];
> > +		isp_dev->isp_hw_module = i;
> > +		isp_dev->dev = dev;
> 
> p1_dev already includes a pointer to this.
> 

Fix in next patch.

> > +		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
> > +		isp_dev->regs = devm_ioremap_resource(dev, res);
> > +
> > +		dev_dbg(dev, "cam%u, map_addr=0x%lx\n",
> > +			i, (unsigned long)isp_dev->regs);
> > +
> > +		if (!isp_dev->regs)
> 
> devm_ioremap_resource() returns ERR_PTR() not NULL on error.
> 

Fix in next patch.

> > +			return PTR_ERR(isp_dev->regs);
> > +
> > +		/* Support IRQ from ISP_CAM_A_IDX */
> > +		if (i >= ISP_CAM_A_IDX) {
> > +			/* Reg & interrupts index is shifted with 1  */
> 
> The reader can already see that it's shifted from the code below. The
> comment should say _why_ it is shifted.
> 

ok, we will remove this checking after supporting single CAM HW.

> > +			irq = platform_get_irq(pdev, i - 1);
> 
> The bindings have 3 IRQs, but we only seem to request 2 here. Is that
> expected?
> 

ok, we will remove this checking after supporting single CAM HW.

> > +			if (irq) {
> 
> Please invert this if, so that it bails out on error. Also, this should
> check for negative errors codes, not 0.
> 

Fix in next patch.

> > +				ret = devm_request_irq(dev, irq,
> > +						       isp_irq_cam,
> > +						       IRQF_SHARED,
> > +						       dev_driver_string(dev),
> 
> Use dev_name().
> 

Fix in next patch.

> > +						       (void *)isp_dev);
> 
> No need to cast to void *.
> 

Fix in next patch.

> > +				if (ret) {
> > +					dev_err(dev,
> > +						"req_irq fail, dev:%s irq=%d\n",
> > +						dev->of_node->name,
> > +						irq);
> > +					return ret;
> > +				}
> > +				dev_dbg(dev, "Registered irq=%d, ISR:%s\n",
> > +					irq, dev_driver_string(dev));
> > +			}
> > +		}
> > +		spin_lock_init(&isp_dev->spinlock_irq);
> > +	}
> 
> We might want to move out the body of this loop to a separate function to
> keep this function shorter.
> 

Since we have already remove this loop, maybe we don't need to move out
the body into one single function.

> > +
> > +	p1_dev->isp_ctx.num_clks = ARRAY_SIZE(mtk_isp_clks);
> > +	p1_dev->isp_ctx.clk_list =
> 
> nit: "list" is the term commonly used for the list data structure. There is
> also a convention to call the length of array XXX num_XXX, so how about
> clks and num_clks?
> 

Fix in next patch.

> > +		devm_kcalloc(dev,
> > +			     p1_dev->isp_ctx.num_clks,
> > +			     sizeof(*p1_dev->isp_ctx.clk_list),
> > +			     GFP_KERNEL);
> > +	if (!p1_dev->isp_ctx.clk_list)
> > +		return -ENOMEM;
> > +
> > +	for (i = 0; i < p1_dev->isp_ctx.num_clks; ++i)
> > +		p1_dev->isp_ctx.clk_list->id = mtk_isp_clks[i];
> 
> Shouldn't this be clk_list[i].id?
> 

Fix in next patch.

> > +
> > +	ret = devm_clk_bulk_get(dev,
> > +				p1_dev->isp_ctx.num_clks,
> > +				p1_dev->isp_ctx.clk_list);
> > +	if (ret) {
> > +		dev_err(dev, "cannot get isp cam clock:%d\n", ret);
> > +		return ret;
> > +	}
> > +
> > +	/* Initialize reserved DMA memory */
> > +	ret = mtk_cam_reserved_memory_init(p1_dev);
> > +	if (ret) {
> > +		dev_err(dev, "failed to configure DMA memory:%d\n", ret);
> > +		goto err_init;
> > +	}
> > +
> > +	/* Initialize the v4l2 common part */
> > +	ret = mtk_cam_dev_init(pdev, &p1_dev->cam_dev);
> > +	if (ret)
> > +		goto err_init;
> > +
> > +	spin_lock_init(&isp_ctx->p1_enqueue_list.lock);
> > +	pm_runtime_enable(dev);
> > +
> > +	return 0;
> > +
> > +err_init:
> > +	if (p1_dev->cam_dev.smem_dev)
> > +		device_unregister(p1_dev->cam_dev.smem_dev);
> > +
> > +	return ret;
> > +}
> > +
> > +static int mtk_isp_remove(struct platform_device *pdev)
> > +{
> > +	struct device *dev = &pdev->dev;
> > +	struct isp_p1_device *p1_dev = dev_get_drvdata(dev);
> > +
> > +	pm_runtime_disable(dev);
> > +	mtk_cam_dev_release(pdev, &p1_dev->cam_dev);
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct dev_pm_ops mtk_isp_pm_ops = {
> > +	SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
> > +	SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)
> 
> For V4L2 drivers system and runtime PM ops would normally be completely
> different. Runtime PM ops would be called when the hardware is idle already
> or is about to become active. System PM ops would be called at system power
> state change and the hardware might be both idle or active. Please also see
> my comments to mtk_isp_suspend() and mtk_isp_resume() above.
> 

Here is the new implementation. It should be clear to show the
difference between system and runtime PM ops. 

static const struct dev_pm_ops mtk_isp_pm_ops = {
	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
				pm_runtime_force_resume)
	SET_RUNTIME_PM_OPS(mtk_isp_runtime_suspend, mtk_isp_runtime_resume,
NULL)
};

> > +};
> > +
> > +static struct platform_driver mtk_isp_driver = {
> > +	.probe   = mtk_isp_probe,
> > +	.remove  = mtk_isp_remove,
> > +	.driver  = {
> > +		.name  = "mtk-cam",
> 
> Shouldn't this be something like mtk-cam-p1? Please make sure this
> reasonably consistent with other drivers.
> 

Fix in next patch.

> > +		.of_match_table = of_match_ptr(mtk_isp_of_ids),
> > +		.pm     = &mtk_isp_pm_ops,
> > +	}
> > +};
> > +
> > +module_platform_driver(mtk_isp_driver);
> > +
> > +MODULE_DESCRIPTION("Camera ISP driver");
> 
> Mediatek Camera P1 ISP driver? Please make sure this is reasonably
> consistent with other drivers (SenInf, DIP, FD).
> 
> > +MODULE_LICENSE("GPL");
> 
> GPL v2
> 

We will check this naming with other drivers & fix the version.

> diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
> new file mode 100644
> index 000000000000..6af3f569664c
> > --- /dev/null
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
> @@ -0,0 +1,243 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (c) 2018 MediaTek Inc.
> > + */
> > +
> > +#ifndef __CAMERA_ISP_H
> > +#define __CAMERA_ISP_H
> > +
> > +#include <linux/cdev.h>
> > +#include <linux/clk.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/ioctl.h>
> > +#include <linux/irqreturn.h>
> > +#include <linux/miscdevice.h>
> > +#include <linux/pm_qos.h>
> > +#include <linux/scatterlist.h>
> > +
> > +#include "mtk_cam-scp.h"
> > +#include "mtk_cam-v4l2-util.h"
> > +
> > +#define CAM_A_MAX_WIDTH		3328
> > +#define CAM_A_MAX_HEIGHT		2496
> > +#define CAM_B_MAX_WIDTH		5376
> > +#define CAM_B_MAX_HEIGHT		4032
> > +
> > +#define CAM_MIN_WIDTH			80
> > +#define CAM_MIN_HEIGHT			60
> > +
> > +#define IMG_MAX_WIDTH			CAM_B_MAX_WIDTH
> > +#define IMG_MAX_HEIGHT			CAM_B_MAX_HEIGHT
> > +#define IMG_MIN_WIDTH			CAM_MIN_WIDTH
> > +#define IMG_MIN_HEIGHT			CAM_MIN_HEIGHT
> > +
> > +#define RRZ_MAX_WIDTH			CAM_B_MAX_WIDTH
> > +#define RRZ_MAX_HEIGHT			CAM_B_MAX_HEIGHT
> > +#define RRZ_MIN_WIDTH			CAM_MIN_WIDTH
> > +#define RRZ_MIN_HEIGHT			CAM_MIN_HEIGHT
> > +
> > +#define R_IMGO				BIT(0)
> > +#define R_RRZO				BIT(1)
> > +#define R_AAO				BIT(3)
> > +#define R_AFO				BIT(4)
> > +#define R_LCSO				BIT(5)
> > +#define R_PDO				BIT(6)
> > +#define R_LMVO				BIT(7)
> > +#define R_FLKO				BIT(8)
> > +#define R_RSSO				BIT(9)
> > +#define R_PSO				BIT(10)
> > +
> > +#define CQ_BUFFER_COUNT		3
> > +#define IRQ_DATA_BUF_SIZE		4
> > +#define CQ_ADDRESS_OFFSET		0x640
> > +
> > +#define ISP_COMPOSING_MAX_NUM		4
> > +#define ISP_FRAME_COMPOSING_MAX_NUM	3
> > +
> > +#define IRQ_STAT_STR	"cam%c, SOF_%d irq(0x%x), " \
> > +			"dma(0x%x), frame_num(%d)/cq_num(%d), " \
> > +			"fbc1(0x%x), fbc2(0x%x)\n"
> > +
> > +/*
> > + * In order with the sequence of device nodes defined in dtsi rule,
> > + * one hardware module should be mapping to one node.
> > + */
> > +enum isp_dev_node_enum {
> > +	ISP_CAMSYS_CONFIG_IDX = 0,
> > +	ISP_CAM_UNI_IDX,
> > +	ISP_CAM_A_IDX,
> > +	ISP_CAM_B_IDX,
> > +	ISP_DEV_NODE_NUM
> > +};
> > +
> > +/* Image RAW path for ISP P1 module. */
> > +enum isp_raw_path_enum {
> > +	ISP_PROCESS_RAW_PATH = 0,
> > +	ISP_PURE_RAW_PATH
> > +};
> > +
> > +/* State for struct mtk_isp_p1_ctx: composer_state */
> > +enum  {
> > +	SCP_ON = 0,
> > +	SCP_OFF
> > +};
> 
> Hmm, looks like bool.
> 

This will be removed in next patch.

> > +
> > +enum {
> > +	IMG_FMT_UNKNOWN		= 0x0000,
> > +	IMG_FMT_RAW_START	= 0x2200,
> > +	IMG_FMT_BAYER8		= IMG_FMT_RAW_START,
> > +	IMG_FMT_BAYER10,
> > +	IMG_FMT_BAYER12,
> > +	IMG_FMT_BAYER14,
> > +	IMG_FMT_FG_BAYER8,
> > +	IMG_FMT_FG_BAYER10,
> > +	IMG_FMT_FG_BAYER12,
> > +	IMG_FMT_FG_BAYER14,
> > +};
> > +
> > +enum {
> > +	RAW_PXL_ID_B = 0,
> > +	RAW_PXL_ID_GB,
> > +	RAW_PXL_ID_GR,
> > +	RAW_PXL_ID_R
> > +};
> 
> Please use macros with explicitly assigned values for hardware/firmware ABI
> definitions.
> 

Fix in next patch.

> > +
> > +struct isp_queue {
> > +	struct list_head queue;
> > +	atomic_t queue_cnt;
> > +	spinlock_t lock; /* queue attributes protection */
> > +};
> > +
> > +struct isp_thread {
> > +	struct task_struct *thread;
> > +	wait_queue_head_t wq;
> > +};
> > +
> > +struct mtk_isp_queue_work {
> > +	union {
> > +		struct mtk_isp_scp_p1_cmd cmd;
> > +		struct p1_frame_param frameparams;
> > +	};
> > +	struct list_head list_entry;
> > +	enum mtk_isp_scp_type type;
> > +};
> > +
> > +struct mtk_cam_dev_stat_event_data {
> > +	__u32 frame_seq_no;
> > +	__u32 meta0_vb2_index;
> > +	__u32 meta1_vb2_index;
> > +	__u32 irq_status_mask;
> > +	__u32 dma_status_mask;
> > +};
> > +
> > +struct mtk_isp_queue_job {
> > +	struct list_head list_entry;
> > +	struct list_head list_buf;
> > +	unsigned int request_fd;
> > +	unsigned int frame_seq_no;
> > +};
> 
> Please document the structs above using kerneldoc.
> 

These structures are removed in next patch.

> > +
> > +/*
> > + * struct isp_device - the ISP device information
> > + *
> > + * @dev: Pointer to struct device
> > + * @regs: Camera ISP base register address
> > + * @spinlock_irq: Used to protect register read/write data
> > + * @current_frame: Current frame sequence number, set when SOF
> > + * @meta0_vb2_index: Meta0 vb2 buffer index, set when SOF
> > + * @meta1_vb2_index: Meta1 vb2 buffer index, set when SOF
> > + * @sof_count: The accumulated SOF counter
> > + * @isp_hw_module: Identity camera A or B
> > + *
> > + */
> > +struct isp_device {
> 
> mtk_isp_device?
> 

Fix in next patch.

> > +	struct device *dev;
> > +	void __iomem *regs;
> > +	spinlock_t spinlock_irq; /* ISP reg setting integrity */
> > +	unsigned int current_frame;
> > +	unsigned int meta0_vb2_index;
> > +	unsigned int meta1_vb2_index;
> > +	u8 sof_count;
> > +	u8 isp_hw_module;
> > +};
> > +
> > +/*
> > + * struct mtk_isp_p1_ctx - the ISP device information
> > + *
> > + * @composer_txlist: Queue for SCP TX data including SCP_ISP_CMD & SCP_ISP_FRAME
> > + * @composer_tx_thread: TX Thread for SCP data tranmission
> > + * @cmd_queued: The number of SCP_ISP_CMD commands will be sent
> > + * @ipi_occupied: The total number of SCP TX data has beent sent
> > + * @scp_state: The state of SCP control
> > + * @composing_frame: The total number of SCP_ISP_FRAME has beent sent
> > + * @composed_frame_id: The ack. frame sequence by SCP
> > + * @composer_deinit_thread: The de-initialized thread
> > + * @p1_enqueue_list: Queue for ISP frame buffers
> > + * @isp_deque_thread: Thread for handling ISP frame buffers dequeue
> > + * @irq_event_datas: Ring buffer for struct mtk_cam_dev_stat_event_data data
> > + * @irq_data_start: Start index of irq_event_datas ring buffer
> > + * @irq_data_end: End index of irq_event_datas ring buffer
> > + * @irq_dequeue_lock: Lock to protect irq_event_datas ring buffer
> > + * @scp_mem_pa: DMA address for SCP device
> > + * @scp_mem_iova: DMA address for ISP HW DMA devices
> > + * @frame_seq_no: Sequence number for ISP frame buffer
> > + * @isp_hw_module: Active camera HW module
> > + * @num_clks: The number of driver's clock
> > + * @clk_list: The list of clock data
> > + * @lock: Lock to protect context operations
> > + *
> > + */
> > +struct mtk_isp_p1_ctx {
> > +	struct isp_queue composer_txlist;
> > +	struct isp_thread composer_tx_thread;
> > +	atomic_t cmd_queued;
> > +	atomic_t ipi_occupied;
> > +	atomic_t scp_state;
> > +	atomic_t composing_frame;
> > +	atomic_t composed_frame_id;
> > +	struct isp_thread composer_deinit_thread;
> > +	struct isp_queue p1_enqueue_list;
> > +	struct isp_thread isp_deque_thread;
> > +	struct mtk_cam_dev_stat_event_data irq_event_datas[IRQ_DATA_BUF_SIZE];
> > +	atomic_t irq_data_start;
> > +	atomic_t irq_data_end;
> > +	spinlock_t irq_dequeue_lock; /* ISP frame dequeuq protection */
> 
> Already documented in kerneldoc.
> 

Fix in next patch.

> > +	dma_addr_t scp_mem_pa;
> > +	dma_addr_t scp_mem_iova;
> > +	int frame_seq_no;
> > +	unsigned int isp_hw_module;
> > +	unsigned int isp_raw_path;
> 
> Not documented above.
> 

Fix in next patch.

> > +	unsigned int num_clks;
> > +	struct clk_bulk_data *clk_list;
> > +	struct mutex lock; /* Protect context operations */
> 
> Already documented in kerneldoc.
> 

Fix in next patch.

> > +};
> > +
> > +struct isp_p1_device {
> > +	struct platform_device *pdev;
> > +	struct platform_device *scp_pdev;
> > +	struct rproc *rproc_handle;
> > +	struct mtk_isp_p1_ctx isp_ctx;
> > +	struct mtk_cam_dev cam_dev;
> > +	struct isp_device isp_devs[ISP_DEV_NODE_NUM];
> > +};
> 
> Please document in a kerneldoc comment.
> 

Fix in next patch.

> > +
> > +static inline struct isp_p1_device *
> > +p1_ctx_to_dev(const struct mtk_isp_p1_ctx *__p1_ctx)
> > +{
> > +	return container_of(__p1_ctx, struct isp_p1_device, isp_ctx);
> > +}
> > +
> > +static inline struct isp_p1_device *get_p1_device(struct device *dev)
> > +{
> > +	return ((struct isp_p1_device *)dev_get_drvdata(dev));
> 
> No need to cast. And, I don't think we need a function for this, just call
> dev_get_drvdata() directly.
> 

Fix in next patch.

> Best regards,
> Tomasz
> 

Thank you for your valuable comments .

Best regards,


Jungo




^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication
  2019-07-10  9:58   ` Tomasz Figa
@ 2019-07-21  2:18     ` Jungo Lin
  2019-07-25 10:56       ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-21  2:18 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: hverkuil, laurent.pinchart, matthias.bgg, mchehab, linux-media,
	linux-mediatek, linux-arm-kernel, devicetree, srv_heupstream,
	ddavenport, robh, sean.cheng, sj.huang, frederic.chen, ryan.yu,
	rynn.wu, frankie.chiu

Hi Tomasz:

On Wed, 2019-07-10 at 18:58 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Tue, Jun 11, 2019 at 11:53:43AM +0800, Jungo Lin wrote:
> > This patch adds communication with the co-processor on the SoC
> > through the SCP driver. It supports bi-directional commands
> > to exchange data and perform command flow control function.
> > 
> > Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> > ---
> > This patch depends on "Add support for mt8183 SCP"[1].
> > 
> > [1] https://patchwork.kernel.org/cover/10972143/
> > ---
> >  .../platform/mtk-isp/isp_50/cam/Makefile      |   1 +
> >  .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.c | 371 ++++++++++++++++++
> >  .../platform/mtk-isp/isp_50/cam/mtk_cam-scp.h | 207 ++++++++++
> >  3 files changed, 579 insertions(+)
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
> >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
> > 
> 
> Thanks for the patch! Please see my comments inline.
> 
> [snip]
> 

Thank you for your comments. Please check my replies inline.

[snip]

> > diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
> > new file mode 100644
> > index 000000000000..04519d0b942f
> > --- /dev/null
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.c
> > @@ -0,0 +1,371 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +//
> > +// Copyright (c) 2018 MediaTek Inc.
> > +
> > +#include <linux/atomic.h>
> > +#include <linux/kthread.h>
> > +#include <linux/platform_data/mtk_scp.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/remoteproc.h>
> > +#include <linux/sched.h>
> > +#include <linux/spinlock.h>
> > +#include <linux/types.h>
> > +#include <linux/vmalloc.h>
> > +
> > +#include "mtk_cam.h"
> > +
> > +static void isp_composer_deinit(struct mtk_isp_p1_ctx *isp_ctx)
> > +{
> > +	struct mtk_isp_queue_work *ipi_job, *tmp_ipi_job;
> > +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> > +
> > +	atomic_set(&isp_ctx->cmd_queued, 0);
> > +	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
> > +	atomic_set(&isp_ctx->composing_frame, 0);
> > +	atomic_set(&isp_ctx->ipi_occupied, 0);
> 
> Is there any point to set them if we are deinitalizing? Moreover,
> isp_composer_init() would set them once we start again.
> 

We will remove these variable assignments.

> > +
> > +	spin_lock(&isp_ctx->composer_txlist.lock);
> > +	list_for_each_entry_safe(ipi_job, tmp_ipi_job,
> > +				 &isp_ctx->composer_txlist.queue,
> > +				 list_entry) {
> > +		list_del(&ipi_job->list_entry);
> > +		kfree(ipi_job);
> > +	}
> > +	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
> > +	spin_unlock(&isp_ctx->composer_txlist.lock);
> > +
> > +	mutex_lock(&isp_ctx->lock);
> > +	if (isp_ctx->composer_tx_thread.thread) {
> > +		kthread_stop(isp_ctx->composer_tx_thread.thread);
> 
> Shouldn't the thread be stopped at this point already? If not, wouldn't the
> atomic_set() at the beginning of this function confuse it?
> 
> In any case, this should be greatly simplified after we move to a workqueue,
> with one work per one task to do, as per other comments.
> 

We will simplify the IPI sending mechanism and remove these kthread
handling.

> > +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> > +		isp_ctx->composer_tx_thread.thread = NULL;
> > +	}
> > +
> > +	if (isp_ctx->composer_deinit_thread.thread) {
> > +		wake_up(&isp_ctx->composer_deinit_thread.wq);
> > +		isp_ctx->composer_deinit_thread.thread = NULL;
> > +	}
> > +	mutex_unlock(&isp_ctx->lock);
> > +
> > +	pm_runtime_put_sync(&p1_dev->pdev->dev);
> 
> No need to use the sync variant.
> 

We don't get this point. If we will call pm_runtime_get_sync in
mtk_isp_hw_init function, will we need to call
pm_runtime_put_sync_autosuspend in mtk_isp_hw_release in next patch?
As we know, we should call runtime pm functions in pair.

> > +}
> > +
> > +/*
> > + * Two kinds of flow control in isp_composer_tx_work.
> > + *
> > + * Case 1: IPI commands flow control. The maximum number of command queues is 3.
> > + * There are two types of IPI commands (SCP_ISP_CMD/SCP_ISP_FRAME) in P1 driver.
> > + * It is controlled by ipi_occupied.
> 
> ISP_COMPOSING_MAX_NUM is defined to 4, not 3. Is that expected?
> 

In this version, we use async. scp_ipi_send function call with wait = 0.
If kernel sends too many P1 IPI commands in short time, P1 task in SCP
may miss some IPI command due to the IPI command processing time and the
size of command queue in SCP side. In order to avoid this kind of
condition, we use ISP_COMPOSING_MAX_NUM to control the sending flow of
IPI command in kernel side. The ISP_COMPOSING_MAX_NUM is changed to 4
for Chromium EC OS. We just miss to update the comment here.

In new version, we will change to use sync. scp_ipi_send function call
with non-zero wait variable. Based on this, we could remove IPI command
flow control in P1 driver.

> > + * The priority of SCP_ISP_CMD is higher than SCP_ISP_FRAME.
> 
> What does it mean and why is it so?
> 

In the origin design, SCP_ISP_CMD & SCP_ISP_FRAME are sending in the
same command queue by order. However, if we receive ISP_CMD_DEINIT
command, we will like to send this command firstly to SCP before
SCP_ISP_FRAME are queued in the queue. So we need to have one command
prioritize design here. Btw, in the new design, SCP_ISP_CMD &
SCP_ISP_FRAME are sent independent and we can remove this. 

> > + *
> > + * Case 2: Frame buffers flow control. The maximum number of frame buffers is 3.
> > + * It is controlled by composing_frame.
> > + * Frame buffer is sent by SCP_ISP_FRAME command.
> 
> Case 1 already mentions SCP_ISP_FRAME. What's the difference between that
> and case 2?
> 

For case 2, it is related to frame request handling with CQ buffer.
We send frame request data via SCP_ISP_FRAME to compose CQ buffers in
SCP. The maximum CQ buffers in SCP are 3. So in kernel side, we can't
send any SCP_ISP_FRAME command to SCP when the CQ buffers are full in
SCP until ISP HW has output the new frame with the corresponding CQ
buffer.

In the new design, this will be controlled by mtk_cam_dev_req_try_queue
function with MTK_ISP_MAX_RUNNING_JOBS.

void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam)
{
	struct mtk_cam_dev_request *req, *req_prev;
	struct list_head enqueue_job_list;
	int buffer_cnt = atomic_read(&cam->running_job_count);
	unsigned long flags;

	if (!cam->streaming || buffer_cnt >= MTK_ISP_MAX_RUNNING_JOBS) {
		dev_dbg(cam->dev, "stream off or buffers are full:%d\n",
			buffer_cnt);
		return;
	}

	INIT_LIST_HEAD(&enqueue_job_list);

	spin_lock(&cam->pending_job_lock);
	list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list) {
		list_del(&req->list);
		list_add_tail(&req->list, &enqueue_job_list);
		if (atomic_inc_return(&cam->running_job_count) >=
			MTK_ISP_MAX_RUNNING_JOBS)
			break;
	}
	spin_unlock(&cam->pending_job_lock);

	list_for_each_entry_safe(req, req_prev, &enqueue_job_list, list) {
		list_del(&req->list);
		spin_lock_irqsave(&cam->running_job_lock, flags);
		list_add_tail(&req->list, &cam->running_job_list);
		spin_unlock_irqrestore(&cam->running_job_lock, flags);

		mtk_isp_req_enqueue(cam, req);
	}
}

> > + */
> > +static int isp_composer_tx_work(void *data)
> > +{
> > +	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)data;
> > +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	struct mtk_isp_queue_work *isp_composer_work, *tmp_ipi_job;
> > +	struct isp_queue *composer_txlist = &isp_ctx->composer_txlist;
> > +	int ret;
> > +
> > +	while (1) {
> > +		ret = wait_event_interruptible
> > +			(isp_ctx->composer_tx_thread.wq,
> > +			 (atomic_read(&composer_txlist->queue_cnt) > 0 &&
> > +			 atomic_read(&isp_ctx->ipi_occupied)
> > +				< ISP_COMPOSING_MAX_NUM &&
> > +			 atomic_read(&isp_ctx->composing_frame)
> > +				< ISP_FRAME_COMPOSING_MAX_NUM) ||
> > +			 (atomic_read(&isp_ctx->cmd_queued) > 0 &&
> > +			 atomic_read(&isp_ctx->ipi_occupied)
> > +				< ISP_COMPOSING_MAX_NUM) ||
> > +			 kthread_should_stop());
> > +
> > +		if (kthread_should_stop())
> > +			break;
> > +
> > +		spin_lock(&composer_txlist->lock);
> > +		if (atomic_read(&isp_ctx->cmd_queued) > 0) {
> > +			list_for_each_entry_safe(isp_composer_work, tmp_ipi_job,
> > +						 &composer_txlist->queue,
> > +						 list_entry) {
> > +				if (isp_composer_work->type == SCP_ISP_CMD) {
> > +					dev_dbg(dev, "Found a cmd\n");
> > +					break;
> > +				}
> > +			}
> > +		} else {
> > +			if (atomic_read(&isp_ctx->composing_frame) >=
> > +				ISP_FRAME_COMPOSING_MAX_NUM) {
> > +				spin_unlock(&composer_txlist->lock);
> > +				continue;
> > +			}
> > +			isp_composer_work =
> > +			    list_first_entry_or_null
> > +				(&composer_txlist->queue,
> > +				 struct mtk_isp_queue_work,
> > +				 list_entry);
> > +		}
> 
> I don't understand why this special handling of CMD vs FRAME is here, so I
> might be missing something, but would we really lose anything if we just
> simply removed it and queued everything in order?
> 
> Moreover, in V4L2, buffer queue and control operations are serialized wrt
> each other, so we probably wouldn't even have a chance to hit a case when we
> need to prioritize a CMD IPI over a FRAME IPI.
> 

Yes, this implementation is complicated and we will remove
implementation in next patch. We will simplify current implementation by
using:
1. Use sync. scp_ipi_send function call
2. Use workqueue for SCP_ISP_FRAME sending

> > +
> > +		list_del(&isp_composer_work->list_entry);
> > +		atomic_dec(&composer_txlist->queue_cnt);
> > +		spin_unlock(&composer_txlist->lock);
> > +
> > +		if (isp_composer_work->type == SCP_ISP_CMD) {
> > +			scp_ipi_send
> > +				(p1_dev->scp_pdev,
> > +				 SCP_IPI_ISP_CMD,
> > +				 &isp_composer_work->cmd,
> > +				 sizeof(isp_composer_work->cmd),
> > +				 0);
> > +			atomic_dec(&isp_ctx->cmd_queued);
> > +			atomic_inc(&isp_ctx->ipi_occupied);
> > +			dev_dbg(dev,
> > +				"%s cmd id %d sent, %d ipi buf occupied",
> > +				__func__,
> > +				isp_composer_work->cmd.cmd_id,
> > +				atomic_read(&isp_ctx->ipi_occupied));
> > +		} else if (isp_composer_work->type == SCP_ISP_FRAME) {
> > +			scp_ipi_send
> > +				(p1_dev->scp_pdev,
> > +				 SCP_IPI_ISP_FRAME,
> > +				 &isp_composer_work->frameparams,
> > +				 sizeof(isp_composer_work->frameparams),
> > +				 0);
> > +			atomic_inc(&isp_ctx->ipi_occupied);
> > +			atomic_inc(&isp_ctx->composing_frame);
> 
> Why do we need composing frame here, if ipi_occupied already limits us to 3?
> 

If we send SCP_ISP_FRAME command, we need to increase ipi_occupied with
1 for IPI command sending command flow and increase composing_frame with
1 for CQ buffers composing. But this implementation will be removed.

> > +			dev_dbg(dev,
> > +				"%s frame %d sent, %d ipi, %d CQ bufs occupied",
> > +				__func__,
> > +				isp_composer_work->frameparams.frame_seq_no,
> > +				atomic_read(&isp_ctx->ipi_occupied),
> > +				atomic_read(&isp_ctx->composing_frame));
> > +		} else {
> > +			dev_err(dev,
> > +				"ignore IPI type: %d!\n",
> > +				isp_composer_work->type);
> > +		}
> > +		kfree(isp_composer_work);
> > +	}
> > +	return ret;
> > +}
> 
> The function above is way too complicated than it should be. I'd suggest a
> model similar to what we ended up in the DIP driver:
> >  - a freezable workqueue created for ISP composing works,
> >  - each ISP composing work entry would have a struct work_struct embedded,
> >  - isp_composer_enqueue() would enqueue the work_struct to the workqueue
> >    above,
> >  - the workqueue would keep a queue of works itself, so driver's own list
> >    wouldn't be needed anymore,
> >  - similarly, each execution of the work func would operate on its own ISP
> >    composing work, so things like checking for list emptiness, waiting for
> >    work to be queued, etc. wouldn't be needed,
> >  - freezability of the workqueue would ensure nice synchonization with
> >    system suspend/resume (although one would still need to wait for the
> >    hardware/firmware to complete).
> 
> WDYT?
> 

yes, we will adopt your suggestion to re-factor current implementation.
Below is new implementation.

void mtk_isp_req_enqueue(struct mtk_cam_dev *cam,
			 struct mtk_cam_dev_request *req)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);
	int ret;

	/* Accumulated frame sequence number */
	req->frame_params.frame_seq_no = ++p1_dev->enqueue_frame_seq_no;

	INIT_WORK(&req->frame_work, isp_tx_frame_worker);
	ret = queue_work(p1_dev->composer_wq, &req->frame_work);
	if (!ret)
		dev_dbg(cam->dev, "frame_no:%d queue_work failed\n",
			req->frame_params.frame_seq_no, ret);
	else
		dev_dbg(cam->dev, "Enqueue fd:%s frame_seq_no:%d job cnt:%d\n",
			req->req.debug_str, req->frame_params.frame_seq_no,
			atomic_read(&cam->running_job_count));
}

static void isp_tx_frame_worker(struct work_struct *work)
{
	struct mtk_cam_dev_request *req =
		container_of(work, struct mtk_cam_dev_request, frame_work);
	struct mtk_cam_dev *cam =
		container_of(req->req.mdev, struct mtk_cam_dev, media_dev);
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);

	scp_ipi_send(p1_dev->scp_pdev, SCP_IPI_ISP_FRAME, &req->frame_params,
		     sizeof(req->frame_params), MTK_ISP_IPI_SEND_TIMEOUT);
}

> > +
> > +static int isp_composer_deinit_work(void *data)
> > +{
> > +	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)data;
> > +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(data);
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +
> > +	wait_event_interruptible(isp_ctx->composer_deinit_thread.wq,
> > +				 atomic_read(&isp_ctx->scp_state) == SCP_OFF ||
> > +				 kthread_should_stop());
> > +
> > +	dev_dbg(dev, "%s run deinit", __func__);
> > +	isp_composer_deinit(isp_ctx);
> > +
> > +	return 0;
> > +}
> > +
> > +static void isp_composer_handler(void *data, unsigned int len, void *priv)
> > +{
> > +	struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)priv;
> > +	struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> > +	struct device *dev = &p1_dev->pdev->dev;
> > +	struct mtk_isp_scp_p1_cmd *ipi_msg;
> > +
> > +	ipi_msg = (struct mtk_isp_scp_p1_cmd *)data;
> 
> Should we check that len == sizeof(*ipi_msg)? (Or at least >=, if data could
> contain some extra bytes at the end.)
> 

The len parameter is the actual sending bytes from SCP to kernel.
In the runtime, it is only 6 bytes for isp_ack_info command
However, sizeof(*ipi_msg) is large due to struct mtk_isp_scp_p1_cmd is
union structure.

> > +
> > +	if (ipi_msg->cmd_id != ISP_CMD_ACK)
> > +		return;
> > +
> > +	if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
> > +		dev_dbg(dev, "ack frame_num:%d",
> > +			ipi_msg->ack_info.frame_seq_no);
> > +		atomic_set(&isp_ctx->composed_frame_id,
> > +			   ipi_msg->ack_info.frame_seq_no);
> 
> I suppose we are expecting here that ipi_msg->ack_info.frame_seq_no would be
> just isp_ctx->composed_frame_id + 1, right? If not, we probably dropped some
> frames and we should handle that somehow.
> 

No, we use isp_ctx->composed_frame_id to save which frame sequence
number are composed done in SCP. In new design, we will move this from
isp_ctx to p1_dev.

	if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
		atomic_set(&p1_dev->composed_frame_seq_no,
			   ipi_msg->ack_info.frame_seq_no);
		dev_dbg(p1_dev->dev, "ack frame_num:%d\n",
			p1_dev->composed_frame_seq_no);
	}

> > +	} else if (ipi_msg->ack_info.cmd_id == ISP_CMD_DEINIT) {
> > +		dev_dbg(dev, "ISP_CMD_DEINIT is acked");
> > +		atomic_set(&isp_ctx->scp_state, SCP_OFF);
> > +		wake_up_interruptible(&isp_ctx->composer_deinit_thread.wq);
> > +	}
> > +
> > +	atomic_dec_return(&isp_ctx->ipi_occupied);
> > +	wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> > +}
> > +
> > +int isp_composer_init(struct device *dev)
> > +{
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	int ret;
> > +
> > +	ret = scp_ipi_register(p1_dev->scp_pdev,
> > +			       SCP_IPI_ISP_CMD,
> > +			       isp_composer_handler,
> > +			       isp_ctx);
> > +	if (ret)
> > +		return ret;
> > +
> > +	atomic_set(&isp_ctx->cmd_queued, 0);
> > +	atomic_set(&isp_ctx->composer_txlist.queue_cnt, 0);
> > +	atomic_set(&isp_ctx->composing_frame, 0);
> > +	atomic_set(&isp_ctx->ipi_occupied, 0);
> > +	atomic_set(&isp_ctx->scp_state, SCP_ON);
> > +
> > +	mutex_lock(&isp_ctx->lock);
> > +	if (!isp_ctx->composer_tx_thread.thread) {
> > +		init_waitqueue_head(&isp_ctx->composer_tx_thread.wq);
> > +		INIT_LIST_HEAD(&isp_ctx->composer_txlist.queue);
> > +		spin_lock_init(&isp_ctx->composer_txlist.lock);
> > +		isp_ctx->composer_tx_thread.thread =
> > +			kthread_run(isp_composer_tx_work, isp_ctx,
> > +				    "isp_composer_tx");
> > +		if (IS_ERR(isp_ctx->composer_tx_thread.thread)) {
> > +			dev_err(dev, "unable to start kthread\n");
> > +			isp_ctx->composer_tx_thread.thread = NULL;
> > +			goto nomem;
> 
> Why nomem?
> 

It is wrong. Need to correct with
ERR_PTR(isp_ctx->composer_tx_thread.thread).
These kthread handling will be removed in next patch.

> > +		}
> > +	} else {
> > +		dev_warn(dev, "old tx thread is existed\n");
> 
> This shouldn't be possible to happen.
> 

Yes, it should not be happen. Otherwise, there is a bug.

> > +	}
> > +
> > +	if (!isp_ctx->composer_deinit_thread.thread) {
> > +		init_waitqueue_head(&isp_ctx->composer_deinit_thread.wq);
> > +		isp_ctx->composer_deinit_thread.thread =
> > +			kthread_run(isp_composer_deinit_work, isp_ctx,
> > +				    "isp_composer_deinit_work");
> 
> Why do we need to deinit from another kthread?
> 

This code will be removed in next patch.

> > +		if (IS_ERR(isp_ctx->composer_deinit_thread.thread)) {
> > +			dev_err(dev, "unable to start kthread\n");
> > +			isp_ctx->composer_deinit_thread.thread = NULL;
> > +			goto nomem;
> > +		}
> > +	} else {
> > +		dev_warn(dev, "old rx thread is existed\n");
> 
> rx? The code above seems to refer to deinit.
> 

Got it.

> > +	}
> > +	mutex_unlock(&isp_ctx->lock);
> > +
> > +	return 0;
> > +
> > +nomem:
> > +	mutex_unlock(&isp_ctx->lock);
> > +
> > +	return -ENOMEM;
> 
> We should return the original error code here.
> 

Got it.

> > +}
> > +
> > +void isp_composer_enqueue(struct device *dev,
> > +			  void *data,
> > +			  enum mtk_isp_scp_type type)
> > +{
> > +	struct mtk_isp_queue_work *isp_composer_work;
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> 
> Just pass p1_dev to this function instead of dev.
> 

Fix in next patch.

> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +
> > +	isp_composer_work = kzalloc(sizeof(*isp_composer_work), GFP_KERNEL);
> 
> For most of the cases, it should be possible to preallocate this, e.g.
> >  - for FRAME, this could be inside the request struct,
> >  - for buffer queue it could be inside the buffer struct.
> 
> I'd suggest making the caller responsible for allocating if needed.
> 

Fix in next patch.

> > +	isp_composer_work->type = type;
> > +
> > +	switch (type) {
> > +	case SCP_ISP_CMD:
> > +		memcpy(&isp_composer_work->cmd, data,
> > +		       sizeof(isp_composer_work->cmd));
> > +		dev_dbg(dev, "Enq ipi cmd id:%d\n",
> > +			isp_composer_work->cmd.cmd_id);
> > +
> > +		spin_lock(&isp_ctx->composer_txlist.lock);
> > +		list_add_tail(&isp_composer_work->list_entry,
> > +			      &isp_ctx->composer_txlist.queue);
> > +		atomic_inc(&isp_ctx->composer_txlist.queue_cnt);
> > +		spin_unlock(&isp_ctx->composer_txlist.lock);
> > +
> > +		atomic_inc(&isp_ctx->cmd_queued);
> > +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> > +		break;
> > +	case SCP_ISP_FRAME:
> > +		memcpy(&isp_composer_work->frameparams, data,
> > +		       sizeof(isp_composer_work->frameparams));
> > +		dev_dbg(dev, "Enq ipi frame_num:%d\n",
> > +			isp_composer_work->frameparams.frame_seq_no);
> > +
> > +		spin_lock(&isp_ctx->composer_txlist.lock);
> > +		list_add_tail(&isp_composer_work->list_entry,
> > +			      &isp_ctx->composer_txlist.queue);
> > +		atomic_inc(&isp_ctx->composer_txlist.queue_cnt);
> > +		spin_unlock(&isp_ctx->composer_txlist.lock);
> > +
> > +		wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> 
> The code in both cases is almost exactly the same. The only difference is
> the memcpy destination and size and whether isp_ctx->cmd_queued is
> incremented or not.
> 
> The memcpy will go away if my comment above is addressed and so that would
> go down to making the cmd_queued increment conditional.
> 

This function will be removed in next patch.
We will call scp_ipi_send directly in the caller, such as:

void mtk_isp_hw_config(struct mtk_cam_dev *cam,
		       struct p1_config_param *config_param)
{
	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);

	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
	composer_tx_cmd.cmd_id = ISP_CMD_CONFIG;
	memcpy(&composer_tx_cmd.config_param, config_param,
	       sizeof(*config_param));

	scp_ipi_send(p1_dev->scp_pdev, SCP_IPI_ISP_CMD, &composer_tx_cmd,
		     sizeof(composer_tx_cmd), MTK_ISP_IPI_SEND_TIMEOUT);
}

void mtk_isp_stream(struct mtk_cam_dev *cam, int on)
{
	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);

	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
	composer_tx_cmd.cmd_id = ISP_CMD_STREAM;
	composer_tx_cmd.is_stream_on = on;

	scp_ipi_send(p1_dev->scp_pdev, SCP_IPI_ISP_CMD, &composer_tx_cmd,
		     sizeof(composer_tx_cmd), MTK_ISP_IPI_SEND_TIMEOUT);
}

static void isp_composer_hw_deinit(struct mtk_isp_p1_device *p1_dev)
{
	struct mtk_isp_scp_p1_cmd composer_tx_cmd;

	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
	composer_tx_cmd.cmd_id = ISP_CMD_DEINIT;

	scp_ipi_send(p1_dev->scp_pdev, SCP_IPI_ISP_CMD, &composer_tx_cmd,
		     sizeof(composer_tx_cmd), MTK_ISP_IPI_SEND_TIMEOUT);

	isp_composer_uninit(p1_dev);
}


> > +		break;
> > +	default:
> > +		break;
> > +	}
> > +}
> > +
> > +void isp_composer_hw_init(struct device *dev)
> > +{
> > +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +
> > +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> > +	composer_tx_cmd.cmd_id = ISP_CMD_INIT;
> > +	composer_tx_cmd.frameparam.hw_module = isp_ctx->isp_hw_module;
> > +	composer_tx_cmd.frameparam.cq_addr.iova = isp_ctx->scp_mem_iova;
> > +	composer_tx_cmd.frameparam.cq_addr.scp_addr = isp_ctx->scp_mem_pa;
> 
> Should we also specify the size of the buffer? Otherwise we could end up
> with some undetectable overruns.
> 

The size of SCP composer's memory is fixed to 0x200000.
Is it necessary to specify the size of this buffer?

#define MTK_ISP_COMPOSER_MEM_SIZE 0x200000

ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
			MTK_ISP_COMPOSER_MEM_SIZE, &addr, GFP_KERNEL);

> > +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> > +}
> > +
> > +void isp_composer_meta_config(struct device *dev,
> > +			      unsigned int dma)
> > +{
> > +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> > +
> > +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> > +	composer_tx_cmd.cmd_id = ISP_CMD_CONFIG_META;
> > +	composer_tx_cmd.cfg_meta_out_param.enabled_meta_dmas = dma;
> > +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> > +}
> > +
> > +void isp_composer_hw_config(struct device *dev,
> > +			    struct p1_config_param *config_param)
> > +{
> > +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> > +
> > +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> > +	composer_tx_cmd.cmd_id = ISP_CMD_CONFIG;
> > +	memcpy(&composer_tx_cmd.config_param, config_param,
> > +	       sizeof(*config_param));
> > +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> > +}
> > +
> > +void isp_composer_stream(struct device *dev, int on)
> > +{
> > +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> > +
> > +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> > +	composer_tx_cmd.cmd_id = ISP_CMD_STREAM;
> > +	composer_tx_cmd.is_stream_on = on;
> > +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> > +}
> > +
> > +void isp_composer_hw_deinit(struct device *dev)
> > +{
> > +	struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> > +	struct isp_p1_device *p1_dev = get_p1_device(dev);
> > +	struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > +	int ret;
> > +
> > +	memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> > +	composer_tx_cmd.cmd_id = ISP_CMD_DEINIT;
> > +	isp_composer_enqueue(dev, &composer_tx_cmd, SCP_ISP_CMD);
> > +
> > +	/* Wait for ISP_CMD_DEINIT command is handled done */
> > +	ret = wait_event_timeout(isp_ctx->composer_deinit_thread.wq,
> > +				 atomic_read(&isp_ctx->scp_state) == SCP_OFF,
> > +				 msecs_to_jiffies(2000));
> > +	if (ret)
> > +		return;
> > +
> > +	dev_warn(dev, "Timeout & local de-init\n");
> > +	isp_composer_deinit(isp_ctx);
> > +}
> > diff --git a/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
> > new file mode 100644
> > index 000000000000..fbd8593e9c2d
> > --- /dev/null
> > +++ b/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-scp.h
> > @@ -0,0 +1,207 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (c) 2018 MediaTek Inc.
> > + */
> > +
> > +#ifndef _MTK_ISP_SCP_H
> > +#define _MTK_ISP_SCP_H
> > +
> > +#include <linux/types.h>
> > +
> > +#include "mtk_cam-v4l2-util.h"
> > +
> > +/*
> > + * struct img_size - image size information.
> > + *
> > + * @w: image width, the unit is pixel
> > + * @h: image height, the unit is pixel
> > + * @xsize: bytes per line based on width.
> > + * @stride: bytes per line when changing line.
> > + *          Normally, calculate new STRIDE based on
> > + *          xsize + HW constrain(page or align).
> > + *
> > + */
> > +struct img_size {
> > +	__u32 w;
> > +	__u32 h;
> > +	__u32 xsize;
> > +	__u32 stride;
> > +} __packed;
> > +
> > +/*
> > + * struct img_buffer - buffer address information.
> > + *
> > + * @iova: DMA address for external devices.
> > + * @scp_addr: SCP address for external co-process unit.
> > + *
> > + */
> > +struct img_buffer {
> > +	__u32 iova;
> > +	__u32 scp_addr;
> > +} __packed;
> > +
> > +struct p1_img_crop {
> > +	__u32 left;
> > +	__u32 top;
> > +	__u32 width;
> > +	__u32 height;
> > +} __packed;
> > +
> > +struct p1_img_output {
> > +	struct img_buffer buffer;
> > +	struct img_size size;
> > +	struct p1_img_crop crop;
> > +	__u8 pixel_byte;
> > +	__u32 img_fmt;
> > +} __packed;
> 
> Please document.
> 

Fix in next patch.

> > +
> > +/*
> > + * struct cfg_in_param - image input parameters structure.
> > + *                       Normally, it comes from sensor information.
> > + *
> > + * @continuous: indicate the sensor mode.
> > + *              1: continuous
> > + *              0: single
> > + * @subsample: indicate to enables SOF subsample or not.
> > + * @pixel_mode: describe 1/2/4 pixels per clock cycle.
> > + * @data_pattern: describe input data pattern.
> > + * @raw_pixel_id: bayer sequence.
> > + * @tg_fps: the fps rate of TG (time generator).
> > + * @img_fmt: the image format of input source.
> > + * @p1_img_crop: the crop configuration of input source.
> > + *
> > + */
> > +struct cfg_in_param {
> > +	__u8 continuous;
> > +	__u8 subsample;
> > +	__u8 pixel_mode;
> > +	__u8 data_pattern;
> > +	__u8 raw_pixel_id;
> > +	__u16 tg_fps;
> > +	__u32 img_fmt;
> > +	struct p1_img_crop crop;
> > +} __packed;
> > +
> > +/*
> > + * struct cfg_main_out_param - the image output parameters of main stream.
> > + *
> > + * @bypass: indicate this device is enabled or disabled or not .
> 
> Remove the space before the period.
> 

Fix in next patch.

> > + * @pure_raw: indicate the image path control.
> > + *            1: pure raw
> > + *            0: processing raw
> > + * @pure_raw_pack: indicate the image is packed or not.
> > + *                 1: packed mode
> > + *                 0: unpacked mode
> > + * @p1_img_output: the output image information.
> > + *
> > + */
> > +struct cfg_main_out_param {
> > +	/* Bypass main out parameters */
> > +	__u8 bypass;
> > +	/* Control HW image raw path */
> > +	__u8 pure_raw;
> > +	/* Control HW image pack function */
> 
> No need for these inline comments.
> 

Fix in next patch.

> > +	__u8 pure_raw_pack;
> > +	struct p1_img_output output;
> > +} __packed;
> > +
> > +/*
> > + * struct cfg_resize_out_param - the image output parameters of
> > + *                               packed out stream.
> > + *
> > + * @bypass: indicate this device is enabled or disabled or not .
> 
> Remove the space before the period.
> 

Fix in next patch.

> > + * @p1_img_output: the output image information.
> > + *
> > + */
> > +struct cfg_resize_out_param {
> > +	/* Bypass resize parameters */
> 
> No need for this inline comment.
> 

Fix in next patch.

> > +	__u8 bypass;
> > +	struct p1_img_output output;
> > +} __packed;
> > +
> > +/*
> > + * struct cfg_meta_out_param - output meta information.
> > + *
> > + * @enabled_meta_dmas: indicate which meta DMAs are enabled.
> > + *
> > + */
> > +struct cfg_meta_out_param {
> > +	__u32 enabled_meta_dmas;
> > +} __packed;
> > +
> > +struct p1_config_param {
> > +	/* Sensor/TG info */
> > +	struct cfg_in_param cfg_in_param;
> > +	/* IMGO DMA */
> > +	struct cfg_main_out_param cfg_main_param;
> > +	/* RRZO DMA */
> > +	struct cfg_resize_out_param cfg_resize_param;
> > +	/* 3A DMAs and other. */
> > +	struct cfg_meta_out_param cfg_meta_param;
> 
> Please change the inline comments to a kerneldoc comment at the top.
> 

Fix in next patch.

> > +} __packed;
> > +
> > +struct p1_frame_param {
> > +	/* frame sequence number */
> > +	__u32 frame_seq_no;
> > +	/* SOF index */
> > +	__u32 sof_idx;
> > +	/* The memory address of tuning buffer from user space */
> 
> Ditto.
> 

Fix in next patch.

> > +	struct img_buffer dma_buffers[MTK_CAM_P1_TOTAL_NODES];
> > +} __packed;
> > +
> > +struct P1_meta_frame {
> > +	__u32 enabled_dma;
> > +	__u32 vb_index;
> > +	struct img_buffer meta_addr;
> > +} __packed;
> > +
> > +struct isp_init_info {
> > +	__u8 hw_module;
> > +	struct img_buffer cq_addr;
> > +} __packed;
> > +
> > +struct isp_ack_info {
> > +	__u8 cmd_id;
> > +	__u32 frame_seq_no;
> > +} __packed;
> > +
> > +enum mtk_isp_scp_cmds {
> > +	ISP_CMD_INIT,
> > +	ISP_CMD_CONFIG,
> > +	ISP_CMD_STREAM,
> > +	ISP_CMD_DEINIT,
> > +	ISP_CMD_ACK,
> > +	ISP_CMD_FRAME_ACK,
> > +	ISP_CMD_CONFIG_META,
> > +	ISP_CMD_ENQUEUE_META,
> > +	ISP_CMD_RESERVED,
> > +};
> > +
> > +struct mtk_isp_scp_p1_cmd {
> > +	__u8 cmd_id;
> > +	union {
> > +		struct isp_init_info frameparam;
> > +		struct p1_config_param config_param;
> > +		struct cfg_meta_out_param cfg_meta_out_param;
> > +		struct P1_meta_frame meta_frame;
> > +		__u8 is_stream_on;
> > +		struct isp_ack_info ack_info;
> > +	};
> > +} __packed;
> > +
> > +enum mtk_isp_scp_type {
> > +	SCP_ISP_CMD = 0,
> > +	SCP_ISP_FRAME,
> > +};
> 
> Please document all the structs and enum above using kerneldoc.
> 

Fix in next patch.

> Best regards,
> Tomasz
> 

Thank you for your valuable comments.

Best regards,


Jungo 



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-05  7:59         ` Jungo Lin
@ 2019-07-23  7:20           ` Tomasz Figa
  2019-07-23  8:21             ` [RFC, v3 " Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-23  7:20 UTC (permalink / raw)
  To: Jungo Lin
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi Jungo,

On Fri, Jul 5, 2019 at 4:59 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi Tomasz:
>
> On Fri, 2019-07-05 at 13:22 +0900, Tomasz Figa wrote:
> > Hi Jungo,
> >
> > On Fri, Jul 5, 2019 at 12:33 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > >
> > > Hi Tomasz,
>
> [snip]
>
> > > After applying your suggestion in SCP device driver, we could remove
> > > mtk_cam-smem.h/c. Currently, we use dma_alloc_coherent with SCP device
> > > to get SCP address. We could touch the buffer with this SCP address in
> > > SCP processor.
> > >
> > > After that, we use dma_map_page_attrs with P1 device which supports
> > > IOMMU domain to get IOVA address. For this address, we will assign
> > > it to our ISP HW device to proceed.
> > >
> > > Below is the snippet for ISP P1 compose buffer initialization.
> > >
> > >         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
> > >                                  MAX_COMPOSER_SIZE, &addr, GFP_KERNEL);
> > >         if (!ptr) {
> > >                 dev_err(dev, "failed to allocate compose memory\n");
> > >                 return -ENOMEM;
> > >         }
> > >         isp_ctx->scp_mem_pa = addr;
> >
> > addr contains a DMA address, not a physical address. Could we call it
> > scp_mem_dma instead?
> >
> > >         dev_dbg(dev, "scp addr:%pad\n", &addr);
> > >
> > >         /* get iova address */
> > >         addr = dma_map_page_attrs(dev, phys_to_page(addr), 0,
> >
> > addr is a DMA address, so phys_to_page() can't be called on it. The
> > simplest thing here would be to use dma_map_single() with ptr as the
> > CPU address expected.
> >
>
> We have changed to use ma_map_single() with ptr, but encounter IOMMU
> error. From the debug log of iommu_dma_map_page[3], we got
> 0x0000000054800000 instead of expected address: 0x0000000050800000[2].
> There is a address offset(0x4000000). If we change to use
> dma_map_page_attrs with phys_to_page(addr), the address is correct as we
> expected[2]. Do you have any suggestion on this issue? Do we miss
> something?

Sorry for the late reply. Could you show me the code changes you made
to use dma_map_single()? It would sound like the virtual address
passed to dma_map_single() isn't correct.

Best regards,
Tomasz

>
> [1]
> [    1.344786] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
> device_base:0x0000000050000000 dma:0x0000000050800000
> virt_base:ffffff8014000000 va:ffffff8014800000
>
> [    1.346890] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
> va:ffffff8014800000
>
> [    1.347864] iommu_dma_map_page:0x0000000054800000 offset:0
> [    1.348562] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000
>
> [2]
> [    1.346738] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
> device_base:0x0000000050000000 dma:0x0000000050800000
> virt_base:ffffff8014000000 va:ffffff8014800000
> [    1.348841] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
> va:ffffff8014800000
> [    1.349816] iommu_dma_map_page:0x0000000050800000 offset:0
> [    1.350514] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000
>
>
> [3]
> dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
>                 unsigned long offset, size_t size, int prot)
> {
>         phys_addr_t phys = page_to_phys(page);
>         pr_err("iommu_dma_map_page:%pa offset:%lu\n", &phys, offset);
>
>         return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot,
>                         iommu_get_dma_domain(dev));
> }
>
> [snip]
>
> Best regards,
>
> Jungo
>

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC, v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-23  7:20           ` Tomasz Figa
@ 2019-07-23  8:21             ` Jungo Lin
  2019-07-26  5:15               ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-23  8:21 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Mauro Carvalho Chehab, Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, Matthias Brugger, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	ddavenport, Frederic Chen (陳俊元),
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi, Tomasz:

On Tue, 2019-07-23 at 16:20 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Fri, Jul 5, 2019 at 4:59 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> >
> > Hi Tomasz:
> >
> > On Fri, 2019-07-05 at 13:22 +0900, Tomasz Figa wrote:
> > > Hi Jungo,
> > >
> > > On Fri, Jul 5, 2019 at 12:33 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > >
> > > > Hi Tomasz,
> >
> > [snip]
> >
> > > > After applying your suggestion in SCP device driver, we could remove
> > > > mtk_cam-smem.h/c. Currently, we use dma_alloc_coherent with SCP device
> > > > to get SCP address. We could touch the buffer with this SCP address in
> > > > SCP processor.
> > > >
> > > > After that, we use dma_map_page_attrs with P1 device which supports
> > > > IOMMU domain to get IOVA address. For this address, we will assign
> > > > it to our ISP HW device to proceed.
> > > >
> > > > Below is the snippet for ISP P1 compose buffer initialization.
> > > >
> > > >         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
> > > >                                  MAX_COMPOSER_SIZE, &addr, GFP_KERNEL);
> > > >         if (!ptr) {
> > > >                 dev_err(dev, "failed to allocate compose memory\n");
> > > >                 return -ENOMEM;
> > > >         }
> > > >         isp_ctx->scp_mem_pa = addr;
> > >
> > > addr contains a DMA address, not a physical address. Could we call it
> > > scp_mem_dma instead?
> > >
> > > >         dev_dbg(dev, "scp addr:%pad\n", &addr);
> > > >
> > > >         /* get iova address */
> > > >         addr = dma_map_page_attrs(dev, phys_to_page(addr), 0,
> > >
> > > addr is a DMA address, so phys_to_page() can't be called on it. The
> > > simplest thing here would be to use dma_map_single() with ptr as the
> > > CPU address expected.
> > >
> >
> > We have changed to use ma_map_single() with ptr, but encounter IOMMU
> > error. From the debug log of iommu_dma_map_page[3], we got
> > 0x0000000054800000 instead of expected address: 0x0000000050800000[2].
> > There is a address offset(0x4000000). If we change to use
> > dma_map_page_attrs with phys_to_page(addr), the address is correct as we
> > expected[2]. Do you have any suggestion on this issue? Do we miss
> > something?
> 
> Sorry for the late reply. Could you show me the code changes you made
> to use dma_map_single()? It would sound like the virtual address
> passed to dma_map_single() isn't correct.
> 
> Best regards,
> Tomasz
> 


Please check the below code snippet in today's testing.

	p1_dev->cam_dev.smem_dev = &p1_dev->scp_pdev->dev;
	ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
				 MTK_ISP_COMPOSER_MEM_SIZE, &addr, GFP_KERNEL);
	if (!ptr) {
		dev_err(dev, "failed to allocate compose memory\n");
		return -ENOMEM;
	}
	p1_dev->composer_scp_addr = addr;
	p1_dev->composer_virt_addr = ptr;
	dev_info(dev, "scp addr:%pad va:%pK\n", &addr, ptr);

	/* get iova address */
	addr = dma_map_single(dev, ptr, MTK_ISP_COMPOSER_MEM_SIZE,
DMA_BIDIRECTIONAL);
	if (dma_mapping_error(dev, addr)) {
		dma_free_coherent(p1_dev->cam_dev.smem_dev,
				  MTK_ISP_COMPOSER_MEM_SIZE,
				  ptr, p1_dev->composer_scp_addr);
		dev_err(dev, "Failed to map scp iova\n");
		ret = -ENOMEM;
		goto fail_free_mem;
	}
	p1_dev->composer_iova = addr;
	dev_info(dev, "scp iova addr:%pad\n", &addr);

Moreover, below is extracted log[2].

We guess the virtual address which is returned by dma_alloc_coherent
function is not valid kernel logical address. It is actually returned by
memremap() in dma_init_coherent_memory(). Moreover, dma_map_single()
will call virt_to_page() function. For virt_to_page function, it
requires a logical address[1].

[1]https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ch15.html

[2]
  322 [    1.238269] mtk-cam-p1 1a006000.camisp: scp
addr:0x0000000052000000 va:00000000a3adc471
  323 [    1.239582] mtk-cam-p1 1a006000.camisp: scp iova
addr:0x00000000fde00000
 7716 [    1.238963] mtk-cam-p1 1a006000.camisp: scp
addr:0x0000000052000000 va:0000000042ec580f
 7717 [    1.240276] mtk-cam-p1 1a006000.camisp: scp iova
addr:0x00000000fde00000
15088 [    1.239309] mtk-cam-p1 1a006000.camisp: scp
addr:0x0000000052000000 va:000000005e5b3462
15089 [    1.240626] mtk-cam-p1 1a006000.camisp: scp iova
addr:0x00000000fde00000

Best regards,

Jungo

> >
> > [1]
> > [    1.344786] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
> > device_base:0x0000000050000000 dma:0x0000000050800000
> > virt_base:ffffff8014000000 va:ffffff8014800000
> >
> > [    1.346890] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
> > va:ffffff8014800000
> >
> > [    1.347864] iommu_dma_map_page:0x0000000054800000 offset:0
> > [    1.348562] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000
> >
> > [2]
> > [    1.346738] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
> > device_base:0x0000000050000000 dma:0x0000000050800000
> > virt_base:ffffff8014000000 va:ffffff8014800000
> > [    1.348841] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
> > va:ffffff8014800000
> > [    1.349816] iommu_dma_map_page:0x0000000050800000 offset:0
> > [    1.350514] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000
> >
> >
> > [3]
> > dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
> >                 unsigned long offset, size_t size, int prot)
> > {
> >         phys_addr_t phys = page_to_phys(page);
> >         pr_err("iommu_dma_map_page:%pa offset:%lu\n", &phys, offset);
> >
> >         return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot,
> >                         iommu_get_dma_domain(dev));
> > }
> >
> > [snip]
> >
> > Best regards,
> >
> > Jungo
> >
> 
> _______________________________________________
> Linux-mediatek mailing list
> Linux-mediatek@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-mediatek



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-18  4:39     ` Jungo Lin
@ 2019-07-23 10:21       ` Tomasz Figa
  2019-07-24  4:31         ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-23 10:21 UTC (permalink / raw)
  To: Jungo Lin
  Cc: Hans Verkuil, Laurent Pinchart, Matthias Brugger,
	Mauro Carvalho Chehab, Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

Hi Jungo,

On Thu, Jul 18, 2019 at 1:39 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi, Tomasz:
>
> On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> > Hi Jungo,
> >
> > On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
[snip]
> > > +static void mtk_cam_req_try_isp_queue(struct mtk_cam_dev *cam_dev,
> > > +                                 struct media_request *new_req)
> > > +{
> > > +   struct mtk_cam_dev_request *req, *req_safe, *cam_dev_req;
> > > +   struct device *dev = &cam_dev->pdev->dev;
> > > +
> > > +   dev_dbg(dev, "%s new req:%d", __func__, !new_req);
> > > +
> > > +   if (!cam_dev->streaming) {
> > > +           cam_dev_req = mtk_cam_req_to_dev_req(new_req);
> > > +           spin_lock(&cam_dev->req_lock);
> > > +           list_add_tail(&cam_dev_req->list, &cam_dev->req_list);
> > > +           spin_unlock(&cam_dev->req_lock);
> > > +           dev_dbg(dev, "%s: stream off, no ISP enqueue\n", __func__);
> > > +           return;
> > > +   }
> > > +
> > > +   /* Normal enqueue flow */
> > > +   if (new_req) {
> > > +           mtk_isp_req_enqueue(dev, new_req);
> > > +           return;
> > > +   }
> > > +
> > > +   /* Flush all media requests wehen first stream on */
> > > +   list_for_each_entry_safe(req, req_safe, &cam_dev->req_list, list) {
> > > +           list_del(&req->list);
> > > +           mtk_isp_req_enqueue(dev, &req->req);
> > > +   }
> > > +}
> >
> > This will have to be redone, as per the other suggestions, but generally one
> > would have a function that tries to queue as much as possible from a list to
> > the hardware and another function that adds a request to the list and calls
> > the first function.
> >
>
> We revised this function as below.
> First to check the en-queue conditions:
> a. stream on
> b. The composer buffers in SCP are 3, so we only could has 3 jobs
> at the same time.
>
>
> Second, try to en-queue the frames in the pending job if possible and
> move them into running job list if possible.
>
> The request has been inserted into pending job in mtk_cam_req_validate
> which is used to validate media_request.

Thanks for replying to each of the comments, that's very helpful.
Snipped out the parts that I agreed with.

Please note that req_validate is not supposed to change any driver
state. It's only supposed to validate the request. req_queue is the
right callback to insert the request into some internal driver
bookkeeping structures.

>
> void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam_dev)
> {
>         struct mtk_cam_dev_request *req, *req_prev;
>         struct list_head enqueue_job_list;
>         int buffer_cnt = atomic_read(&cam_dev->running_job_count);
>         unsigned long flags;
>
>         if (!cam_dev->streaming ||
>             buffer_cnt >= MTK_ISP_MAX_RUNNING_JOBS) {

Do we have a guarantee that cam_dev->running_job_count doesn't
decrement between the atomic_read() above and this line?

>                 dev_dbg(cam_dev->dev, "stream off or buffers are full:%d\n",
>                         buffer_cnt);
>                 return;
>         }
>
>         INIT_LIST_HEAD(&enqueue_job_list);
>
>         spin_lock(&cam_dev->pending_job_lock);
>         list_for_each_entry_safe(req, req_prev,
>                                  &cam_dev->pending_job_list, list) {
>                 list_del(&req->list);
>                 list_add_tail(&req->list, &enqueue_job_list);

What's the reason to use the second list? Could we just take one job
from pending_job_list, enqueue it and then iterate again?

>                 if (atomic_inc_return(&cam_dev->running_job_count) >=
>                         MTK_ISP_MAX_RUNNING_JOBS)
>                         break;
>         }
>         spin_unlock(&cam_dev->pending_job_lock);
>
>         list_for_each_entry_safe(req, req_prev,
>                                  &enqueue_job_list, list) {
>                 list_del(&req->list);
>                 spin_lock_irqsave(&cam_dev->running_job_lock, flags);
>                 list_add_tail(&req->list, &cam_dev->running_job_list);
>                 spin_unlock_irqrestore(&cam_dev->running_job_lock, flags);
>

Do we have a guarantee that another thread doesn't run the same
function ending up calling mtk_isp_req_enqueue() with another request
before this one and thus making the order of running_job_list
incorrect?

>                 mtk_isp_req_enqueue(cam_dev, req);
>         }
> }
>
[snip]
> > > +   stride = DIV_ROUND_UP(stride * pixel_byte, 8);
> > > +
> > > +   if (pix_fmt == V4L2_PIX_FMT_MTISP_F10)
> > > +           stride = ALIGN(stride, 4);
> >
> > Is it expected that only the F10 format needs this alignment?
> >
>
> yes, if the pixel bits of image format is 10, the byte alignment of bpl
> should be 4. Otherwise, it is 8. We will revise this and add more
> comments.

That means that the B10 format also needs the extra alignment, as
opposed to what the original code did, right?

>
> /* 4 bytes alignment for 10 bit other are 8 bytes alignment */
>         if (pixel_bits == 10)
>                 bpl = ALIGN(bpl, 4);
>         else
>                 bpl = ALIGN(bpl, 8);

SGTM, thanks.

[snip]
> > > +
> > > +static struct v4l2_subdev *
> > > +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> > > +{
> > > +   struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> > > +   struct media_entity *entity;
> > > +   struct device *dev = &cam_dev->pdev->dev;
> > > +   struct v4l2_subdev *sensor;
> >
> > This variable would be unitialized if there is no streaming sensor. Was
> > there no compiler warning generated for this?
> >
>
> No, there is no compiler warning.
> But, we will assign sensor to NULL to avoid unnecessary compiler warning
> with different compiler options.
>

Thanks. It would be useful if you could check why the compiler you're
using doesn't show a warning here. We might be missing other
uninitialized variables.

[snip]
> > > +}
> > > +
> > > +static int mtk_cam_media_link_setup(struct media_entity *entity,
> > > +                               const struct media_pad *local,
> > > +                               const struct media_pad *remote, u32 flags)
> > > +{
> > > +   struct mtk_cam_dev *cam_dev =
> > > +           container_of(entity, struct mtk_cam_dev, subdev.entity);
> > > +   u32 pad = local->index;
> > > +
> > > +   dev_dbg(&cam_dev->pdev->dev, "%s: %d -> %d flags:0x%x\n",
> > > +           __func__, pad, remote->index, flags);
> > > +
> > > +   if (pad < MTK_CAM_P1_TOTAL_NODES)
> >
> > I assume this check is needed, because the pads with higher indexes are not
> > video nodes? If so, a comment would be helpful here.
> >
>
> Yes, we will new comment as below.
>
>         /*
>          * Check video nodes is enabled by link setup.
>          * The pad index of video node should be less than
>          * MTK_CAM_P1_TOTAL_NODES.
>          */
>         if (pad < MTK_CAM_P1_TOTAL_NODES)
>                 cam_dev->vdev_nodes[pad].enabled =
>                         !!(flags & MEDIA_LNK_FL_ENABLED);
>

Could we rephrase this a bit. The comment still doesn't explain why
the index should be less than the constant. Perhaps:

/*
 * The video nodes exposed by the driver have pads indexes
 * from 0 to MTK_CAM_P1_TOTAL_NODES - 1.
 */

[snip]

> > > +
> > > +   dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > > +           __func__,
> > > +           node->id,
> > > +           buf->vbb.request_fd,
> > > +           buf->vbb.vb2_buf.index);
> > > +
> > > +   /* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > > +   if (vb->vb2_queue->uses_requests)
> > > +           return;
> >
> > I'd suggest removing non-request support from this driver. Even if we end up
> > with a need to provide compatibility for non-request mode, then it should be
> > built on top of the requests mode, so that the driver itself doesn't have to
> > deal with two modes.
> >
>
> The purpose of non-request function in this driver is needed by
> our camera middle-ware design. It needs 3A statistics buffers before
> image buffers en-queue. So we need to en-queue 3A statistics with
> non-request mode in this driver. After MW got the 3A statistics data, it
> will en-queue the images, tuning buffer and other meta buffers with
> request mode. Based on this requirement, do you have any suggestion?
> For upstream driver, should we only consider request mode?
>

Where does that requirement come from? Why the timing of queuing of
the buffers to the driver is important?

[snip]
> > > +static int mtk_cam_vb2_start_streaming(struct vb2_queue *vq,
> > > +                                  unsigned int count)
> > > +{
> > > +   struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
> > > +   struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> > > +   struct device *dev = &cam_dev->pdev->dev;
> > > +   unsigned int node_count = cam_dev->subdev.entity.use_count;
> > > +   int ret;
> > > +
> > > +   if (!node->enabled) {
> >
> > How is this synchronized with mtk_cam_media_link_setup()?
> >
>
> We will follow your suggestion and below is our proposal for this
> function.
>
> 1. Use !cam_dev->pipeline.streaming_count to decide the first node to
> stream-on.
> 2.a If yes, do the following steps
>     2.a-1 Call media_pipeline_start function to prevent the link
> configuration changes.
>     2.a-2 Call mtk_cam_dev_init_stream function to calculate how many
> video nodes are enabled and save it into cam_dev->enabled_node_count.
>     2.a-3 Initialize ISP P1 HW in mtk_isp_hw_init function since end
> user has called stream-on API
> 2.b jump step 3.
>
> 3. Use cam_dev->streamed_node_count to track how many video nodes are
> streamed by user space.
> 4. Check all enabled video nodes are streamed or not based on
> cam_dev->streamed_node_count & cam_dev->enabled_node_count.
> 5. If yes, call s_stream on for P1 sub-device
>
> Do you think it is reasonable?
>

That should work indeed.

[snip]
> > > +
> > > +   mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_ERROR);
> >
> > Shouldn't we stop streaming first, so that the hardware operation is
> > cancelled and any buffers owned by the hardware are released?
> >
>
> For this function, below is the new code flow.
>
> 1. Check the first node to stream off based on
> cam_dev->streamed_node_count & cam_dev->enabled_node_count.
> 2. If yes, call all s_stream off for P1 sub-device
> 3. Call mtk_cam_vb2_return_all_buffers for each node
> 4. Check the last node to stream off
> 5. If yes, call media_pipeline_stop to allow user space
> to perform link configuration changes, such as disable link.
>
> But, for step 5, is it too late for end user to disable link?
> For example, for first node, it has called stream off but
> can't call disable link until the last node is stream off?
>

I think that should be okay. From the userspace point of view, having
one of the video nodes streaming implies that the related subdevice
could be streaming as well. The links between the video nodes and the
subdevices don't have the DYNAMIC flag, so the userspace should expect
that it can't change any links connecting to the subdevice when the
subdevice could be streaming.

[snip]
> > > +static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
> > > +{
> > > +   struct mtk_cam_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
> > > +
> > > +   v4l2_ctrl_request_complete(vb->req_obj.req,
> > > +                              dev->v4l2_dev.ctrl_handler);
> >
> > This would end up being called multiple times, once for each video node.
> > Instead, this should be called explicitly by the driver when it completed
> > the request - perhaps in the frame completion handler?
> >
> > With that, we probably wouldn't even need this callback.
> >
>
> First, if we don't implement this callback function, we will receive
> kernel warning as below.
>
> https://elixir.bootlin.com/linux/latest/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L420
>
> Second, this function is only be called in __vb2_queue_cancel function.
> Moreover, we will remove cam_dev->v4l2_dev.ctrl_handler in next patch.
> So could we just implement dummy empty function?
>
>  * @buf_request_complete: a buffer that was never queued to the driver
> but is
>  *                      associated with a queued request was canceled.
>  *                      The driver will have to mark associated objects in the
>  *                      request as completed; required if requests are
>  *                      supported.
>

Good catch, thanks.

Sounds like we may indeed need to implement this callback. In
particular, we may need to remove the request that the buffer was
associated with from the driver queue and return the other buffers
associated to it with an error state. This should be similar to
handling a request failure.
[snip]
> > > +
> > > +   return 0;
> > > +}
> > > +
> > > +static int mtk_cam_vidioc_enum_fmt(struct file *file, void *fh,
> > > +                              struct v4l2_fmtdesc *f)
> > > +{
> > > +   struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > > +
> > > +   if (f->index >= node->desc.num_fmts)
> > > +           return -EINVAL;
> > > +
> > > +   f->pixelformat = node->desc.fmts[f->index].fmt.pix_mp.pixelformat;
> >
> > Is the set of formats available always the same regardless of the sensor
> > format?
> >
>
> Yes, ISP P1 HW output formats are always available without impact
> by sensor formats.
>
> > > +   f->flags = 0;
> >
> > We need f->description too.
> >
>
> For this description, do you suggest 1). we fill this field in this
> function or 2). v4l_fill_fmtdesc function in v4l2-ioctl?
>
> https://elixir.bootlin.com/linux/latest/source/drivers/media/v4l2-core/v4l2-ioctl.c#L1152
>
> Basically, we prefer method 1.
>

That should be v4l_fill_fmtdesc(), as it already includes other
vendor-specific formats.

[snip]
> > > +
> > > +   dev_dbg(&cam_dev->pdev->dev, "%s: fmt:%c%c%c%c, w*h:%u*%u\n",
> > > +           __func__,
> > > +           (in_fmt->fmt.pix_mp.pixelformat & 0xFF),
> > > +           (in_fmt->fmt.pix_mp.pixelformat >> 8) & 0xFF,
> > > +           (in_fmt->fmt.pix_mp.pixelformat >> 16) & 0xFF,
> > > +           (in_fmt->fmt.pix_mp.pixelformat >> 24) & 0xFF,
> > > +           in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height);
> > > +
> > > +   width = in_fmt->fmt.pix_mp.width;
> > > +   height = in_fmt->fmt.pix_mp.height;
> > > +
> > > +   dev_fmt = mtk_cam_dev_find_fmt(&node->desc,
> > > +                                  in_fmt->fmt.pix_mp.pixelformat);
> > > +   if (dev_fmt) {
> > > +           mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
> > > +                                   &in_fmt->fmt.pix_mp,
> > > +                                   &dev_fmt->fmt.pix_mp,
> > > +                                   node->id);
> > > +   } else {
> > > +           mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
> > > +                                        &node->desc, in_fmt);
> >
> > We shouldn't just load a default format. This function should validate all
> > the fields one by one and adjust them to something appropriate.
> >
>
> For ISP P1 HW, we only cares these fields of v4l2_pix_format_mplane.
> a. width
> b. height
> c. pixelformat
> d. plane_fmt
>     - sizeimage
>     - bytesperline
> e. num_planes
> Other fields are consider constant.
>
> So if the user space passes one pixel format with un-supported, we will
> apply the default format firstly and adjust width, height, sizeimage,
> and bytesperline. We will focus on validate width & height.
> Is it ok?

I'm not sure I understand your proposal, but let me describe the
proper behavior here:

if (pixelformat is invalid)
    pixelformat = some valid pixel format;

width = clamp(width, driver min, driver max);
height = clamp(height, driver min, driver max);

num_planes = 1;

calculate_sizeimage_and_bytesperline(fmt);

fill_in_the_remaining_constant_fields(fmt);

Does it make sense?

[snip]
> > > +static int mtk_cam_vidioc_s_fmt(struct file *file, void *fh,
> > > +                           struct v4l2_format *f)
> > > +{
> > > +   struct mtk_cam_dev *cam_dev = video_drvdata(file);
> > > +   struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > > +
> > > +   if (cam_dev->streaming)
> > > +           return -EBUSY;
> >
> > I think this should rather be something like vb2_queue_is_busy(), which
> > would prevent format changes if buffers are allocated.
> >
>
> Since vb2_queue_is_busy is static function, would we paste its
> implementation in this function to check like this?
>
>         if (node->vdev.queue->owner &&
>                 node->vdev.queue->owner != file->private_data) {
>                 dev_err(cam_dev->dev, "%s err: buffer allocated\n", __func__);
>                 return -EBUSY;
>         }
>

Sorry, I mixed up the function name. That should've been vb2_is_busy().

[snip]
> > > +   /* Total pad numbers is video devices + one seninf pad */
> > > +   unsigned int num_subdev_pads = MTK_CAM_CIO_PAD_SINK + 1;
> > > +   unsigned int i;
> > > +   int ret;
> > > +
> > > +   ret = mtk_cam_media_register(dev,
> > > +                                &cam_dev->media_dev);
> > > +   if (ret) {
> > > +           dev_err(dev, "failed to register media device:%d\n", ret);
> > > +           return ret;
> > > +   }
> > > +   dev_info(dev, "Register media device: %s, 0x%pK",
> > > +            MTK_CAM_DEV_P1_NAME, cam_dev->media_dev);
> >
> > An info message should be useful to the user in some way. Printing kernel
> > pointers isn't useful. Something like "registered media0" could be useful to
> > let the user know which media device is associated with this driver if there
> > is more than one in the system.
> >
>
> Here is the new log info.
>
> dev_info(dev, "media%d register",cam->media_dev.devnode->minor);
>

Let's fix the missing space and making a bit more readable:

dev_info(dev, "Registered media%d", cam->media_dev.devnode->minor);

>
> > > +
> > > +   /* Set up v4l2 device */
> > > +   cam_dev->v4l2_dev.mdev = &cam_dev->media_dev;
> > > +   ret = v4l2_device_register(dev, &cam_dev->v4l2_dev);
> > > +   if (ret) {
> > > +           dev_err(dev, "failed to register V4L2 device:%d\n", ret);
> > > +           goto fail_v4l2_dev;
> >
> > Please call the labels after the cleanup step that needs to be done. It
> > makes it easier to spot any ordering errors.
> >
>
> Will fix in next patch.
>
> > > +   }
> > > +   dev_info(dev, "Register v4l2 device: 0x%pK", cam_dev->v4l2_dev);
> >
> > Same as above.
> >
>
> Ditto.
>
> dev_info(dev, "Register v4l2 device: %s", cam->v4l2_dev.name);
>

Perhaps just "Registered %s" to be consistent with the above media log?

[snip]
> > > +
> > > +static int mtk_cam_dev_notifier_bound(struct v4l2_async_notifier *notifier,
> > > +                                 struct v4l2_subdev *sd,
> > > +                                 struct v4l2_async_subdev *asd)
> > > +{
> > > +   struct mtk_cam_dev *cam_dev =
> > > +           container_of(notifier, struct mtk_cam_dev, notifier);
> > > +
> >
> > Should we somehow check that the entity we got is seninf indeed and there
> > was no mistake in DT?
> >
>
> How about to check the entity function of seninf device?
>
> if (!(sd->entity.function & MEDIA_ENT_F_VID_IF_BRIDGE)) {
>         dev_dbg(cam->dev, "No MEDIA_ENT_F_VID_IF_BRIDGE function\n");
>                 return -ENODEV;
> }
>
> If we need to check DT, may we need to implement this in parse_endpoint
> callback function of v4l2_async_notifier_parse_fwnode_endpoints?
>

Yes, checking the entity function is indeed the right way to do this.

[snip]
> > > +           .default_fmt_idx = 4,
> > > +           .max_buf_count = 10,
> >
> > Where does this number come from?
> >
>
> The default maximum VB2 buffer count is 32.
> In order to limit memory usage, we like to limit the maximum buffer
> counts in the driver layer. The maximum buffer count is estimated
> according to our camera MW.
>
> #define VB2_MAX_FRAME   (32)
>

Okay, thanks.

[snip]
> > > +   struct media_pad vdev_pad;
> > > +   struct vb2_queue vbq;
> > > +   struct v4l2_ctrl_handler ctrl_handler;
> > > +   struct list_head pending_list;
> > > +   /* Used for vbq & vdev */
> >
> > It's already documented in the kerneldoc comment.
> >
>
> Fixed in next patch.
> Btw, if we remove this, we will got complain from checkpatch.pl script.
>

Oh really, that's weird. Okay, please keep it then, sorry for the confusion.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-23 10:21       ` Tomasz Figa
@ 2019-07-24  4:31         ` Jungo Lin
  2019-07-26  5:49           ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-24  4:31 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, Laurent Pinchart, Matthias Brugger,
	Mauro Carvalho Chehab, Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

Hi, Tomasz:

On Tue, 2019-07-23 at 19:21 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Thu, Jul 18, 2019 at 1:39 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> >
> > Hi, Tomasz:
> >
> > On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> > > Hi Jungo,
> > >
> > > On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
> [snip]
> > > > +static void mtk_cam_req_try_isp_queue(struct mtk_cam_dev *cam_dev,
> > > > +                                 struct media_request *new_req)
> > > > +{
> > > > +   struct mtk_cam_dev_request *req, *req_safe, *cam_dev_req;
> > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > +
> > > > +   dev_dbg(dev, "%s new req:%d", __func__, !new_req);
> > > > +
> > > > +   if (!cam_dev->streaming) {
> > > > +           cam_dev_req = mtk_cam_req_to_dev_req(new_req);
> > > > +           spin_lock(&cam_dev->req_lock);
> > > > +           list_add_tail(&cam_dev_req->list, &cam_dev->req_list);
> > > > +           spin_unlock(&cam_dev->req_lock);
> > > > +           dev_dbg(dev, "%s: stream off, no ISP enqueue\n", __func__);
> > > > +           return;
> > > > +   }
> > > > +
> > > > +   /* Normal enqueue flow */
> > > > +   if (new_req) {
> > > > +           mtk_isp_req_enqueue(dev, new_req);
> > > > +           return;
> > > > +   }
> > > > +
> > > > +   /* Flush all media requests wehen first stream on */
> > > > +   list_for_each_entry_safe(req, req_safe, &cam_dev->req_list, list) {
> > > > +           list_del(&req->list);
> > > > +           mtk_isp_req_enqueue(dev, &req->req);
> > > > +   }
> > > > +}
> > >
> > > This will have to be redone, as per the other suggestions, but generally one
> > > would have a function that tries to queue as much as possible from a list to
> > > the hardware and another function that adds a request to the list and calls
> > > the first function.
> > >
> >
> > We revised this function as below.
> > First to check the en-queue conditions:
> > a. stream on
> > b. The composer buffers in SCP are 3, so we only could has 3 jobs
> > at the same time.
> >
> >
> > Second, try to en-queue the frames in the pending job if possible and
> > move them into running job list if possible.
> >
> > The request has been inserted into pending job in mtk_cam_req_validate
> > which is used to validate media_request.
> 
> Thanks for replying to each of the comments, that's very helpful.
> Snipped out the parts that I agreed with.
> 
> Please note that req_validate is not supposed to change any driver
> state. It's only supposed to validate the request. req_queue is the
> right callback to insert the request into some internal driver
> bookkeeping structures.
> 

Yes, in req_validate function, we don't change any driver state.
Below is the function's implementation.

a. Call vb2_request_validate(req) to verify media request.
b. Update the buffer internal structure buffer.
c. Insert the request into pending_job_list to prepare en-queue.

static int mtk_cam_req_validate(struct media_request *req)
{
	struct mtk_cam_dev_request *cam_req = mtk_cam_req_to_dev_req(req);
	struct mtk_cam_dev *cam = container_of(req->mdev, struct mtk_cam_dev,
					       media_dev);
	struct media_request_object *req_obj;
	unsigned long flags;
	int ret;

	/* run buffer prepare function to initialize buffer DMA address */
	ret = vb2_request_validate(req);
	if (ret) {
		dev_err(cam->dev, "vb2_request_validate failed:%d\n", ret);
		return ret;
	}

	/* update frame_params */
	list_for_each_entry(req_obj, &req->objects, list) {
		struct vb2_buffer *vb;
		struct mtk_cam_dev_buffer *buf;

		if (!vb2_request_object_is_buffer(req_obj))
			continue;

		vb = container_of(req_obj, struct vb2_buffer, req_obj);
		buf = mtk_cam_vb2_buf_to_dev_buf(vb);
		cam_req->frame_params.dma_bufs[buf->node_id].iova = buf->daddr;
		cam_req->frame_params.dma_bufs[buf->node_id].scp_addr =
			buf->scp_addr;
	}
	atomic_set(&cam_req->buf_count, vb2_request_buffer_cnt(req));

	/* add to pending job list */
	spin_lock_irqsave(&cam->pending_job_lock, flags);
	list_add_tail(&cam_req->list, &cam->pending_job_list);
	spin_unlock_irqrestore(&cam->pending_job_lock, flags);

	return 0;
}

> >
> > void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam_dev)
> > {
> >         struct mtk_cam_dev_request *req, *req_prev;
> >         struct list_head enqueue_job_list;
> >         int buffer_cnt = atomic_read(&cam_dev->running_job_count);
> >         unsigned long flags;
> >
> >         if (!cam_dev->streaming ||
> >             buffer_cnt >= MTK_ISP_MAX_RUNNING_JOBS) {
> 
> Do we have a guarantee that cam_dev->running_job_count doesn't
> decrement between the atomic_read() above and this line?
> 

Ok, we will use cam->pending_job_lock to protect
cam_dev->running_job_count access. Below is the revised version.

void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam)
{
	struct mtk_cam_dev_request *req, *req_prev;
	unsigned long flags;

	if (!cam->streaming) {
		dev_dbg(cam->dev, "stream is off\n");
		return;
	}

	spin_lock_irqsave(&cam->pending_job_lock, flags);
	if (atomic_read(&cam->running_job_count) >= MTK_ISP_MAX_RUNNING_JOBS) {
		dev_dbg(cam->dev, "jobs are full\n");
		spin_unlock_irqrestore(&cam->pending_job_lock, flags);
		return;
	}
	list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list) {
		list_del(&req->list);
		spin_lock_irqsave(&cam->running_job_lock, flags);
		list_add_tail(&req->list, &cam->running_job_list);
		mtk_isp_req_enqueue(cam, req);
		spin_unlock_irqrestore(&cam->running_job_lock, flags);
		if (atomic_inc_return(&cam->running_job_count) >=
			MTK_ISP_MAX_RUNNING_JOBS)
			break;
	}
	spin_unlock_irqrestore(&cam->pending_job_lock, flags);
}

> >                 dev_dbg(cam_dev->dev, "stream off or buffers are full:%d\n",
> >                         buffer_cnt);
> >                 return;
> >         }
> >
> >         INIT_LIST_HEAD(&enqueue_job_list);
> >
> >         spin_lock(&cam_dev->pending_job_lock);
> >         list_for_each_entry_safe(req, req_prev,
> >                                  &cam_dev->pending_job_list, list) {
> >                 list_del(&req->list);
> >                 list_add_tail(&req->list, &enqueue_job_list);
> 
> What's the reason to use the second list? Could we just take one job
> from pending_job_list, enqueue it and then iterate again?
> 

Yes, we could simply the code block to remove enqueue_job_list.

> >                 if (atomic_inc_return(&cam_dev->running_job_count) >=
> >                         MTK_ISP_MAX_RUNNING_JOBS)
> >                         break;
> >         }
> >         spin_unlock(&cam_dev->pending_job_lock);
> >
> >         list_for_each_entry_safe(req, req_prev,
> >                                  &enqueue_job_list, list) {
> >                 list_del(&req->list);
> >                 spin_lock_irqsave(&cam_dev->running_job_lock, flags);
> >                 list_add_tail(&req->list, &cam_dev->running_job_list);
> >                 spin_unlock_irqrestore(&cam_dev->running_job_lock, flags);
> >
> 
> Do we have a guarantee that another thread doesn't run the same
> function ending up calling mtk_isp_req_enqueue() with another request
> before this one and thus making the order of running_job_list
> incorrect?
> 

In the new implementation, we use cam->pending_job_lock to protect this
scenario.

> >                 mtk_isp_req_enqueue(cam_dev, req);
> >         }
> > }
> >
> [snip]
> > > > +   stride = DIV_ROUND_UP(stride * pixel_byte, 8);
> > > > +
> > > > +   if (pix_fmt == V4L2_PIX_FMT_MTISP_F10)
> > > > +           stride = ALIGN(stride, 4);
> > >
> > > Is it expected that only the F10 format needs this alignment?
> > >
> >
> > yes, if the pixel bits of image format is 10, the byte alignment of bpl
> > should be 4. Otherwise, it is 8. We will revise this and add more
> > comments.
> 
> That means that the B10 format also needs the extra alignment, as
> opposed to what the original code did, right?
> 

Sorry for short code snippet.
This alignment checking is only applied to F10, no B10.
If you like to check the full function, you could check this in this
link[1].

static void cal_image_pix_mp(struct mtk_cam_dev *cam, unsigned int
node_id,
			     struct v4l2_pix_format_mplane *mp)
{
	unsigned int bpl, ppl;
	unsigned int pixel_bits = get_pixel_bits(mp->pixelformat);
	unsigned int width = mp->width;

	if (node_id == MTK_CAM_P1_MAIN_STREAM_OUT) {
		/* bayer encoding format & 2 bytes alignment */
		bpl = ALIGN(DIV_ROUND_UP(width * pixel_bits, 8), 2);
	} else if (node_id == MTK_CAM_P1_PACKED_BIN_OUT) {
		/*
		 * The FULL-G encoding format
		 * 1 G component per pixel
		 * 1 R component per 4 pixel
		 * 1 B component per 4 pixel
		 * Total 4G/1R/1B in 4 pixel (pixel per line:ppl)
		 */
		ppl = DIV_ROUND_UP(width * 6, 4);
		bpl = DIV_ROUND_UP(ppl * pixel_bits, 8);

		/* 4 bytes alignment for 10 bit & others are 8 bytes */
		if (pixel_bits == 10)
			bpl = ALIGN(bpl, 4);
		else
			bpl = ALIGN(bpl, 8);
	} 

[1]
https://crrev.com/c/1712885/2/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c#303

> >
> > /* 4 bytes alignment for 10 bit other are 8 bytes alignment */
> >         if (pixel_bits == 10)
> >                 bpl = ALIGN(bpl, 4);
> >         else
> >                 bpl = ALIGN(bpl, 8);
> 
> SGTM, thanks.
> 
> [snip]

Thanks for your review.

> > > > +
> > > > +static struct v4l2_subdev *
> > > > +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> > > > +{
> > > > +   struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> > > > +   struct media_entity *entity;
> > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > +   struct v4l2_subdev *sensor;
> > >
> > > This variable would be unitialized if there is no streaming sensor. Was
> > > there no compiler warning generated for this?
> > >
> >
> > No, there is no compiler warning.
> > But, we will assign sensor to NULL to avoid unnecessary compiler warning
> > with different compiler options.
> >
> 
> Thanks. It would be useful if you could check why the compiler you're
> using doesn't show a warning here. We might be missing other
> uninitialized variables.
> 

We will feedback to your project team to check the possible reason about
compiler warning issue.

> [snip]
> > > > +}
> > > > +
> > > > +static int mtk_cam_media_link_setup(struct media_entity *entity,
> > > > +                               const struct media_pad *local,
> > > > +                               const struct media_pad *remote, u32 flags)
> > > > +{
> > > > +   struct mtk_cam_dev *cam_dev =
> > > > +           container_of(entity, struct mtk_cam_dev, subdev.entity);
> > > > +   u32 pad = local->index;
> > > > +
> > > > +   dev_dbg(&cam_dev->pdev->dev, "%s: %d -> %d flags:0x%x\n",
> > > > +           __func__, pad, remote->index, flags);
> > > > +
> > > > +   if (pad < MTK_CAM_P1_TOTAL_NODES)
> > >
> > > I assume this check is needed, because the pads with higher indexes are not
> > > video nodes? If so, a comment would be helpful here.
> > >
> >
> > Yes, we will new comment as below.
> >
> >         /*
> >          * Check video nodes is enabled by link setup.
> >          * The pad index of video node should be less than
> >          * MTK_CAM_P1_TOTAL_NODES.
> >          */
> >         if (pad < MTK_CAM_P1_TOTAL_NODES)
> >                 cam_dev->vdev_nodes[pad].enabled =
> >                         !!(flags & MEDIA_LNK_FL_ENABLED);
> >
> 
> Could we rephrase this a bit. The comment still doesn't explain why
> the index should be less than the constant. Perhaps:
> 
> /*
>  * The video nodes exposed by the driver have pads indexes
>  * from 0 to MTK_CAM_P1_TOTAL_NODES - 1.
>  */
> 
> [snip]
> 

Thanks for your suggestion.
We will update this.

> > > > +
> > > > +   dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > > > +           __func__,
> > > > +           node->id,
> > > > +           buf->vbb.request_fd,
> > > > +           buf->vbb.vb2_buf.index);
> > > > +
> > > > +   /* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > > > +   if (vb->vb2_queue->uses_requests)
> > > > +           return;
> > >
> > > I'd suggest removing non-request support from this driver. Even if we end up
> > > with a need to provide compatibility for non-request mode, then it should be
> > > built on top of the requests mode, so that the driver itself doesn't have to
> > > deal with two modes.
> > >
> >
> > The purpose of non-request function in this driver is needed by
> > our camera middle-ware design. It needs 3A statistics buffers before
> > image buffers en-queue. So we need to en-queue 3A statistics with
> > non-request mode in this driver. After MW got the 3A statistics data, it
> > will en-queue the images, tuning buffer and other meta buffers with
> > request mode. Based on this requirement, do you have any suggestion?
> > For upstream driver, should we only consider request mode?
> >
> 
> Where does that requirement come from? Why the timing of queuing of
> the buffers to the driver is important?
> 
> [snip]

Basically, this requirement comes from our internal camera
middle-ware/3A hal in user space. Since this is not generic requirement,
we will follow your original suggestion to keep the request mode only
and remove other non-request design in other files. For upstream driver,
it should support request mode only.

> > > > +static int mtk_cam_vb2_start_streaming(struct vb2_queue *vq,
> > > > +                                  unsigned int count)
> > > > +{
> > > > +   struct mtk_cam_dev *cam_dev = vb2_get_drv_priv(vq);
> > > > +   struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > +   unsigned int node_count = cam_dev->subdev.entity.use_count;
> > > > +   int ret;
> > > > +
> > > > +   if (!node->enabled) {
> > >
> > > How is this synchronized with mtk_cam_media_link_setup()?
> > >
> >
> > We will follow your suggestion and below is our proposal for this
> > function.
> >
> > 1. Use !cam_dev->pipeline.streaming_count to decide the first node to
> > stream-on.
> > 2.a If yes, do the following steps
> >     2.a-1 Call media_pipeline_start function to prevent the link
> > configuration changes.
> >     2.a-2 Call mtk_cam_dev_init_stream function to calculate how many
> > video nodes are enabled and save it into cam_dev->enabled_node_count.
> >     2.a-3 Initialize ISP P1 HW in mtk_isp_hw_init function since end
> > user has called stream-on API
> > 2.b jump step 3.
> >
> > 3. Use cam_dev->streamed_node_count to track how many video nodes are
> > streamed by user space.
> > 4. Check all enabled video nodes are streamed or not based on
> > cam_dev->streamed_node_count & cam_dev->enabled_node_count.
> > 5. If yes, call s_stream on for P1 sub-device
> >
> > Do you think it is reasonable?
> >
> 
> That should work indeed.
> 
> [snip]

Ok, thanks for your confirmation.

> > > > +
> > > > +   mtk_cam_vb2_return_all_buffers(cam_dev, node, VB2_BUF_STATE_ERROR);
> > >
> > > Shouldn't we stop streaming first, so that the hardware operation is
> > > cancelled and any buffers owned by the hardware are released?
> > >
> >
> > For this function, below is the new code flow.
> >
> > 1. Check the first node to stream off based on
> > cam_dev->streamed_node_count & cam_dev->enabled_node_count.
> > 2. If yes, call all s_stream off for P1 sub-device
> > 3. Call mtk_cam_vb2_return_all_buffers for each node
> > 4. Check the last node to stream off
> > 5. If yes, call media_pipeline_stop to allow user space
> > to perform link configuration changes, such as disable link.
> >
> > But, for step 5, is it too late for end user to disable link?
> > For example, for first node, it has called stream off but
> > can't call disable link until the last node is stream off?
> >
> 
> I think that should be okay. From the userspace point of view, having
> one of the video nodes streaming implies that the related subdevice
> could be streaming as well. The links between the video nodes and the
> subdevices don't have the DYNAMIC flag, so the userspace should expect
> that it can't change any links connecting to the subdevice when the
> subdevice could be streaming.
> 

Ok, got your point. We will keep this design.

> [snip]
> > > > +static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
> > > > +{
> > > > +   struct mtk_cam_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
> > > > +
> > > > +   v4l2_ctrl_request_complete(vb->req_obj.req,
> > > > +                              dev->v4l2_dev.ctrl_handler);
> > >
> > > This would end up being called multiple times, once for each video node.
> > > Instead, this should be called explicitly by the driver when it completed
> > > the request - perhaps in the frame completion handler?
> > >
> > > With that, we probably wouldn't even need this callback.
> > >
> >
> > First, if we don't implement this callback function, we will receive
> > kernel warning as below.
> >
> > https://elixir.bootlin.com/linux/latest/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L420
> >
> > Second, this function is only be called in __vb2_queue_cancel function.
> > Moreover, we will remove cam_dev->v4l2_dev.ctrl_handler in next patch.
> > So could we just implement dummy empty function?
> >
> >  * @buf_request_complete: a buffer that was never queued to the driver
> > but is
> >  *                      associated with a queued request was canceled.
> >  *                      The driver will have to mark associated objects in the
> >  *                      request as completed; required if requests are
> >  *                      supported.
> >
> 
> Good catch, thanks.
> 
> Sounds like we may indeed need to implement this callback. In
> particular, we may need to remove the request that the buffer was
> associated with from the driver queue and return the other buffers
> associated to it with an error state. This should be similar to
> handling a request failure.
> [snip]

Before calling this callback function, the VB2's stop_streaming has been
called. Normally, we will return the buffers belonged to this vb2 queu
with error state. On other hand, only if the state of request is
MEDIA_REQUEST_STATE_QUEUED, the buf_request_complete will be called in
__vb2_queue_cancel function. It hints this media request has been
validated and inserted into our driver's pending_job_list or
running_job_list. So we will call mtk_cam_dev_req_cleanup() remove these
requests from driver's list when streaming is off. Since we have no
v4l2_ctrl, do we need to do the above things which is already handled in
mtk_cam_vb2_stop_streaming function? Maybe is this callback function
only designed for v4l2_ctrl_request_complete usage?

static void mtk_cam_dev_req_cleanup(struct mtk_cam_dev *cam)
{
	struct mtk_cam_dev_request *req, *req_prev;
	unsigned long flags;

	dev_dbg(cam->dev, "%s\n", __func__);

	spin_lock_irqsave(&cam->pending_job_lock, flags);
	list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list)
		list_del(&req->list);
	spin_unlock_irqrestore(&cam->pending_job_lock, flags);

	spin_lock_irqsave(&cam->running_job_lock, flags);
	list_for_each_entry_safe(req, req_prev, &cam->running_job_list, list)
		list_del(&req->list);
	spin_unlock_irqrestore(&cam->running_job_lock, flags);
}

static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
{
	struct mtk_cam_dev *cam = vb2_get_drv_priv(vq);
	struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
	struct device *dev = cam->dev;

	dev_dbg(dev, "%s node:%d count info:%d", __func__,
		node->id, atomic_read(&cam->stream_count));

	mutex_lock(&cam->op_lock);
	if (atomic_read(&cam->stream_count) == cam->enabled_count)
		if (v4l2_subdev_call(&cam->subdev, video, s_stream, 0))
			dev_err(dev, "failed to stop streaming\n");

	mtk_cam_vb2_return_all_buffers(cam, node, VB2_BUF_STATE_ERROR);

	/* Check the first node to stream-off */
	if (!atomic_dec_and_test(&cam->stream_count)) {
		mutex_unlock(&cam->op_lock);
		return;
	}
	mutex_unlock(&cam->op_lock);

	mtk_cam_dev_req_cleanup(cam);
	media_pipeline_stop(&node->vdev.entity);
}

> > > > +
> > > > +   return 0;
> > > > +}
> > > > +
> > > > +static int mtk_cam_vidioc_enum_fmt(struct file *file, void *fh,
> > > > +                              struct v4l2_fmtdesc *f)
> > > > +{
> > > > +   struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > > > +
> > > > +   if (f->index >= node->desc.num_fmts)
> > > > +           return -EINVAL;
> > > > +
> > > > +   f->pixelformat = node->desc.fmts[f->index].fmt.pix_mp.pixelformat;
> > >
> > > Is the set of formats available always the same regardless of the sensor
> > > format?
> > >
> >
> > Yes, ISP P1 HW output formats are always available without impact
> > by sensor formats.
> >
> > > > +   f->flags = 0;
> > >
> > > We need f->description too.
> > >
> >
> > For this description, do you suggest 1). we fill this field in this
> > function or 2). v4l_fill_fmtdesc function in v4l2-ioctl?
> >
> > https://elixir.bootlin.com/linux/latest/source/drivers/media/v4l2-core/v4l2-ioctl.c#L1152
> >
> > Basically, we prefer method 1.
> >
> 
> That should be v4l_fill_fmtdesc(), as it already includes other
> vendor-specific formats.
> 
> [snip]


Ok, got it. We will follow your suggestion.

> > > > +
> > > > +   dev_dbg(&cam_dev->pdev->dev, "%s: fmt:%c%c%c%c, w*h:%u*%u\n",
> > > > +           __func__,
> > > > +           (in_fmt->fmt.pix_mp.pixelformat & 0xFF),
> > > > +           (in_fmt->fmt.pix_mp.pixelformat >> 8) & 0xFF,
> > > > +           (in_fmt->fmt.pix_mp.pixelformat >> 16) & 0xFF,
> > > > +           (in_fmt->fmt.pix_mp.pixelformat >> 24) & 0xFF,
> > > > +           in_fmt->fmt.pix_mp.width, in_fmt->fmt.pix_mp.height);
> > > > +
> > > > +   width = in_fmt->fmt.pix_mp.width;
> > > > +   height = in_fmt->fmt.pix_mp.height;
> > > > +
> > > > +   dev_fmt = mtk_cam_dev_find_fmt(&node->desc,
> > > > +                                  in_fmt->fmt.pix_mp.pixelformat);
> > > > +   if (dev_fmt) {
> > > > +           mtk_cam_dev_set_img_fmt(&cam_dev->pdev->dev,
> > > > +                                   &in_fmt->fmt.pix_mp,
> > > > +                                   &dev_fmt->fmt.pix_mp,
> > > > +                                   node->id);
> > > > +   } else {
> > > > +           mtk_cam_dev_load_default_fmt(&cam_dev->pdev->dev,
> > > > +                                        &node->desc, in_fmt);
> > >
> > > We shouldn't just load a default format. This function should validate all
> > > the fields one by one and adjust them to something appropriate.
> > >
> >
> > For ISP P1 HW, we only cares these fields of v4l2_pix_format_mplane.
> > a. width
> > b. height
> > c. pixelformat
> > d. plane_fmt
> >     - sizeimage
> >     - bytesperline
> > e. num_planes
> > Other fields are consider constant.
> >
> > So if the user space passes one pixel format with un-supported, we will
> > apply the default format firstly and adjust width, height, sizeimage,
> > and bytesperline. We will focus on validate width & height.
> > Is it ok?
> 
> I'm not sure I understand your proposal, but let me describe the
> proper behavior here:
> 
> if (pixelformat is invalid)
>     pixelformat = some valid pixel format;
> 
> width = clamp(width, driver min, driver max);
> height = clamp(height, driver min, driver max);
> 
> num_planes = 1;
> 
> calculate_sizeimage_and_bytesperline(fmt);
> 
> fill_in_the_remaining_constant_fields(fmt);
> 
> Does it make sense?
> 
> [snip]

Yes, here is our new version.

static int mtk_cam_vidioc_try_fmt(struct file *file, void *fh,
				  struct v4l2_format *f)
{
	struct mtk_cam_dev *cam = video_drvdata(file);
	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
	struct device *dev = cam->dev;
	const struct v4l2_format *dev_fmt;
	struct v4l2_format try_fmt;

	dev_dbg(dev, "%s: fmt:%c%c%c%c, w*h:%u*%u\n",
		__func__,
		(f->fmt.pix_mp.pixelformat & 0xFF),
		(f->fmt.pix_mp.pixelformat >> 8) & 0xFF,
		(f->fmt.pix_mp.pixelformat >> 16) & 0xFF,
		(f->fmt.pix_mp.pixelformat >> 24) & 0xFF,
		f->fmt.pix_mp.width, f->fmt.pix_mp.height);

	memset(&try_fmt, 0, sizeof(try_fmt));
	try_fmt.type = f->type;

	/* Validate pixelformat */
	dev_fmt = mtk_cam_dev_find_fmt(&node->desc, f->fmt.pix_mp.pixelformat);
	if (!dev_fmt) {
		dev_dbg(dev, "unknown fmt:%d\n", f->fmt.pix_mp.pixelformat);
		dev_fmt = &node->desc.fmts[node->desc.default_fmt_idx];
	}
	try_fmt.fmt.pix_mp.pixelformat = dev_fmt->fmt.pix_mp.pixelformat;

	/* Validate image width & height range */
	try_fmt.fmt.pix_mp.width = clamp_val(f->fmt.pix_mp.width,
					     IMG_MIN_WIDTH, IMG_MAX_WIDTH);
	try_fmt.fmt.pix_mp.height = clamp_val(f->fmt.pix_mp.height,
					      IMG_MIN_HEIGHT, IMG_MAX_HEIGHT);

	/* 4 bytes alignment for width */
	try_fmt.fmt.pix_mp.width = ALIGN(try_fmt.fmt.pix_mp.width, 4);

	/* bytesperline & sizeimage calculation */
	cal_image_pix_mp(cam, node->id, &try_fmt.fmt.pix_mp);

	/* Constant format fields */
	try_fmt.fmt.pix_mp.colorspace = V4L2_COLORSPACE_SRGB;
	try_fmt.fmt.pix_mp.field = V4L2_FIELD_NONE;
	try_fmt.fmt.pix_mp.num_planes = 1;
	try_fmt.fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
	try_fmt.fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
	try_fmt.fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_SRGB;

	*f = try_fmt;

	return 0;
}

> > > > +static int mtk_cam_vidioc_s_fmt(struct file *file, void *fh,
> > > > +                           struct v4l2_format *f)
> > > > +{
> > > > +   struct mtk_cam_dev *cam_dev = video_drvdata(file);
> > > > +   struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);
> > > > +
> > > > +   if (cam_dev->streaming)
> > > > +           return -EBUSY;
> > >
> > > I think this should rather be something like vb2_queue_is_busy(), which
> > > would prevent format changes if buffers are allocated.
> > >
> >
> > Since vb2_queue_is_busy is static function, would we paste its
> > implementation in this function to check like this?
> >
> >         if (node->vdev.queue->owner &&
> >                 node->vdev.queue->owner != file->private_data) {
> >                 dev_err(cam_dev->dev, "%s err: buffer allocated\n", __func__);
> >                 return -EBUSY;
> >         }
> >
> 
> Sorry, I mixed up the function name. That should've been vb2_is_busy().
> 
> [snip]

Got it. Thanks for your suggestion.

> > > > +   /* Total pad numbers is video devices + one seninf pad */
> > > > +   unsigned int num_subdev_pads = MTK_CAM_CIO_PAD_SINK + 1;
> > > > +   unsigned int i;
> > > > +   int ret;
> > > > +
> > > > +   ret = mtk_cam_media_register(dev,
> > > > +                                &cam_dev->media_dev);
> > > > +   if (ret) {
> > > > +           dev_err(dev, "failed to register media device:%d\n", ret);
> > > > +           return ret;
> > > > +   }
> > > > +   dev_info(dev, "Register media device: %s, 0x%pK",
> > > > +            MTK_CAM_DEV_P1_NAME, cam_dev->media_dev);
> > >
> > > An info message should be useful to the user in some way. Printing kernel
> > > pointers isn't useful. Something like "registered media0" could be useful to
> > > let the user know which media device is associated with this driver if there
> > > is more than one in the system.
> > >
> >
> > Here is the new log info.
> >
> > dev_info(dev, "media%d register",cam->media_dev.devnode->minor);
> >
> 
> Let's fix the missing space and making a bit more readable:
> 
> dev_info(dev, "Registered media%d", cam->media_dev.devnode->minor);
> 

Ok, we will apply this change.

> >
> > > > +
> > > > +   /* Set up v4l2 device */
> > > > +   cam_dev->v4l2_dev.mdev = &cam_dev->media_dev;
> > > > +   ret = v4l2_device_register(dev, &cam_dev->v4l2_dev);
> > > > +   if (ret) {
> > > > +           dev_err(dev, "failed to register V4L2 device:%d\n", ret);
> > > > +           goto fail_v4l2_dev;
> > >
> > > Please call the labels after the cleanup step that needs to be done. It
> > > makes it easier to spot any ordering errors.
> > >
> >
> > Will fix in next patch.
> >
> > > > +   }
> > > > +   dev_info(dev, "Register v4l2 device: 0x%pK", cam_dev->v4l2_dev);
> > >
> > > Same as above.
> > >
> >
> > Ditto.
> >
> > dev_info(dev, "Register v4l2 device: %s", cam->v4l2_dev.name);
> >
> 
> Perhaps just "Registered %s" to be consistent with the above media log?
> 
> [snip]

Ditto.

> > > > +
> > > > +static int mtk_cam_dev_notifier_bound(struct v4l2_async_notifier *notifier,
> > > > +                                 struct v4l2_subdev *sd,
> > > > +                                 struct v4l2_async_subdev *asd)
> > > > +{
> > > > +   struct mtk_cam_dev *cam_dev =
> > > > +           container_of(notifier, struct mtk_cam_dev, notifier);
> > > > +
> > >
> > > Should we somehow check that the entity we got is seninf indeed and there
> > > was no mistake in DT?
> > >
> >
> > How about to check the entity function of seninf device?
> >
> > if (!(sd->entity.function & MEDIA_ENT_F_VID_IF_BRIDGE)) {
> >         dev_dbg(cam->dev, "No MEDIA_ENT_F_VID_IF_BRIDGE function\n");
> >                 return -ENODEV;
> > }
> >
> > If we need to check DT, may we need to implement this in parse_endpoint
> > callback function of v4l2_async_notifier_parse_fwnode_endpoints?
> >
> 
> Yes, checking the entity function is indeed the right way to do this.
> 
> [snip]

Thanks for your confirm.

> > > > +           .default_fmt_idx = 4,
> > > > +           .max_buf_count = 10,
> > >
> > > Where does this number come from?
> > >
> >
> > The default maximum VB2 buffer count is 32.
> > In order to limit memory usage, we like to limit the maximum buffer
> > counts in the driver layer. The maximum buffer count is estimated
> > according to our camera MW.
> >
> > #define VB2_MAX_FRAME   (32)
> >
> 
> Okay, thanks.
> 
> [snip]
> > > > +   struct media_pad vdev_pad;
> > > > +   struct vb2_queue vbq;
> > > > +   struct v4l2_ctrl_handler ctrl_handler;
> > > > +   struct list_head pending_list;
> > > > +   /* Used for vbq & vdev */
> > >
> > > It's already documented in the kerneldoc comment.
> > >
> >
> > Fixed in next patch.
> > Btw, if we remove this, we will got complain from checkpatch.pl script.
> >
> 
> Oh really, that's weird. Okay, please keep it then, sorry for the confusion.
> 
> Best regards,
> Tomasz

Ok, thanks for your understanding.
We will rollback this change to avoid checkpatch's complains.


Best regards,

Jungo



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-07-20  9:58     ` Jungo Lin
@ 2019-07-25  9:23       ` Tomasz Figa
  2019-07-26  7:23         ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-25  9:23 UTC (permalink / raw)
  To: Jungo Lin
  Cc: Hans Verkuil, Laurent Pinchart, Matthias Brugger,
	Mauro Carvalho Chehab, Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

.Hi Jungo,

On Sat, Jul 20, 2019 at 6:58 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi, Tomasz:
>
> On Wed, 2019-07-10 at 18:56 +0900, Tomasz Figa wrote:
> > Hi Jungo,
> >
> > On Tue, Jun 11, 2019 at 11:53:42AM +0800, Jungo Lin wrote:
> > > This patch adds the Mediatek ISP P1 HW control device driver.
> > > It handles the ISP HW configuration, provides interrupt handling and
> > > initializes the V4L2 device nodes and other functions.
> > >
> > > (The current metadata interface used in meta input and partial
> > > meta nodes is only a temporary solution to kick off the driver
> > > development and is not ready to be reviewed yet.)
> > >
> > > Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> > > ---
> > >  .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
> > >  .../mtk-isp/isp_50/cam/mtk_cam-regs.h         |  126 ++
> > >  .../platform/mtk-isp/isp_50/cam/mtk_cam.c     | 1087 +++++++++++++++++
> > >  .../platform/mtk-isp/isp_50/cam/mtk_cam.h     |  243 ++++
> > >  4 files changed, 1457 insertions(+)
> > >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
> > >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
> > >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
> > >
> >
> > Thanks for the patch! Please see my comments inline.
> >
> > [snip]
> >
>
> Thanks for your comments. Please check my replies inline.
>

Thanks! I'll snip anything I don't have further comments on.

[snip]
> > > +/* META */
> > > +#define REG_META0_VB2_INDEX                0x14dc
> > > +#define REG_META1_VB2_INDEX                0x151c
> >
> > I don't believe these registers are really for VB2 indexes.
> >
>
> MTK P1 ISP HW supports frame header spare registers for each DMA, such
> as CAM_DMA_FH_AAO_SPARE or CAM_DMA_FH_AFO_SPARE. We could save some
> frame information in these ISP registers. In this case, we save META0
> VB2 index into AAO FH spare register and META1 VB2 index into AFO FH
> spare register. These implementation is designed for non-request 3A
> DMAs. These VB2 indexes are sent in ISP_CMD_ENQUEUE_META command of
> mtk_isp_enqueue function. So we just call CAM_DMA_FH_AAO_SPARE as
> REG_META0_VB2_INDEX for easy understanding.

Unfortunately it's not a good idea to mix hardware concepts with
naming specific to the OS the driver is written for. Better to keep
the hardware naming, e.g. CAM_DMA_FH_AAO_SPARE.

> Moreover, if we only need to
> support request mode, we should remove this here.
>
> cmd_params.cmd_id = ISP_CMD_ENQUEUE_META;
> cmd_params.meta_frame.enabled_dma = dma_port;
> cmd_params.meta_frame.vb_index = buffer->vbb.vb2_buf.index;
> cmd_params.meta_frame.meta_addr.iova = buffer->daddr;
> cmd_params.meta_frame.meta_addr.scp_addr = buffer->scp_addr;
>

Okay, removing sounds good to me. Let's keep the code simple.

[snip]
> > > +
> > > +   err_status = irq_status & INT_ST_MASK_CAM_ERR;
> > > +
> > > +   /* Sof, done order check */
> > > +   if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST)) {
> > > +           dev_dbg(dev, "sof_done block cnt:%d\n", isp_dev->sof_count);
> > > +
> > > +           /* Notify IRQ event and enqueue frame */
> > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 0);
> > > +           isp_dev->current_frame = hw_frame_num;
> >
> > What exactly is hw_frame_num? Shouldn't we assign it before notifying the
> > event?
> >
>
> This is a another spare register for frame sequence number usage.
> It comes from struct p1_frame_param:frame_seq_no which is sent by
> SCP_ISP_FRAME IPI command. We will rename this to dequeue_frame_seq_no.
> Is it a better understanding?

I'm sorry, unfortunately it's still not clear to me. Is it the
sequence number of the frame that was just processed and returned to
the kernel or the next frame that is going to be processed from now
on?

>
> Below is our frame request handling in current design.
>
> 1. Buffer preparation
> - Combined image buffers (IMGO/RRZO) + meta input buffer (Tuining) +
> other meta histogram buffers (LCSO/LMVO) into one request.
> - Accumulated one unique frame sequence number to each request and send
> this request to the SCP composer to compose CQ (Command queue) buffer
> via SCP_ISP_FRAME IPI command.
> - CQ buffer is frame registers set. If ISP registers should be updated
> per frame, these registers are configured in the CQ buffer, such as
> frame sequence number, DMA addresses and tuning ISP registers.
> - One frame request will be composed into one CQ buffer.Once CQ buffer
> is composed done and kernel driver will receive ISP_CMD_FRAME_ACK with
> its corresponding frame sequence number. Based on this, kernel driver
> knows which request is ready to be en-queued and save this with
> p1_dev->isp_ctx.composed_frame_id.

Hmm, why do we need to save this in p1_dev->isp_ctx? Wouldn't we
already have a linked lists of requests that are composed and ready to
be enqueued? Also, the request itself would contain its frame ID
inside the driver request struct, right?

> - The maximum number of CQ buffers in SCP is 3.
>
> 2. Buffer en-queue flow
> - In order to configure correct CQ buffer setting before next SQF event,
> it is depended on by MTK ISP P1 HW CQ mechanism.
> - The basic concept of CQ mechanism is loaded ISP CQ buffer settings
> when HW_PASS1_DON_ST is received which means DMA output is done.
> - Btw, the pre-condition of this, need to tell ISP HW which CQ buffer
> address is used. Otherwise, it will loaded one dummy CQ buffer to
> bypass.
> - So we will check available CQ buffers by comparing composed frame
> sequence number & dequeued frame sequence from ISP HW in SOF event.
> - If there are available CQ buffers, update the CQ base address to the
> next CQ buffer address based on current de-enqueue frame sequence
> number. So MTK ISP P1 HW will load this CQ buffer into HW when
> HW_PASS1_DON_ST is triggered which is before the next SOF.
> - So in next SOF event, ISP HW starts to output DMA buffers with this
> request until request is done.
> - But, for the first request, it is loaded into HW manually when
> streaming is on for better performance.
>
> 3. Buffer de-queue flow
> - We will use frame sequence number to decide which request is ready to
> de-queue.
> - We will save some important register setting from ISP HW when SOF is
> received. This is because the ISP HW starts to output the data with the
> corresponding settings, especially frame sequence number setting.

Could you explain a bit more about these important register settings?
When does the hardware update the values in the register to new ones?
At SOF?

> - When receiving SW_PASS1_DON_ST IRQ event, it means the DMA output is
> done. So we could call isp_deque_request_frame with frame sequence
> number to de-queue frame to VB2

What's the difference between HW_PASS1_DON_ST and SW_PASS1_DON_ST?

> - For AAO/AFO buffers, it has similar design concept. Sometimes, if only
> AAO/AFO non-request buffers are en-queued without request buffers at the
> same time, there will be no SW P1 done event for AAO/AFO DMA done.
> Needs to depend on other IRQ events, such as AAO/AFO_DONE_EVENT.

Do we have a case like this? Wouldn't we normally always want to
bundle AAO/AFO buffers with frame buffers?

> - Due to CQ buffer number limitation, if we receive SW_PASS1_DONT_ST,
> we may try to send another request to SCP for composing.

Okay, so basically in SW_PASS1_DONT_ST the CQ completed reading the CQ
buffers, right?

>
> Hopefully, my explanation is helpful for better understanding our
> implementation. If you still have any questions, please let me know.
>

Yes, it's more clear now, thanks. Still some more comments above, though.

> > > +           isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > +           isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > +   } else {
> > > +           if (irq_status & SOF_INT_ST) {
> > > +                   isp_dev->current_frame = hw_frame_num;
> > > +                   isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > +                   isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > +           }
> > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
> > > +   }
> >
> > The if and else blocks do almost the same things just in different order. Is
> > it really expected?
> >
>
> If we receive HW_PASS1_DON_ST & SOF_INT_ST IRQ events at the same time,
> the correct sequence should be handle HW_PASS1_DON_ST firstly to check
> any de-queued frame and update the next frame setting later.
> Normally, this is a corner case or system performance issue.

So it sounds like HW_PASS1_DON_ST means that all data from current
frame has been written, right? If I understand your explanation above
correctly, that would mean following handling of each interrupt:

HW_PASS1_DON_ST:
 - CQ executes with next CQ buffer to prepare for next frame. <- how
is this handled? does the CQ hardware automatically receive this event
from the ISP hadware?
 - return VB2 buffers,
 - complete requests.

SOF_INT_ST:
 - send VSYNC event to userspace,
 - program next CQ buffer to CQ,

SW_PASS1_DON_ST:
 - reclaim CQ buffer and enqueue next frame to composing if available

>
> Btw, we will revise the above implementation as below.
>
>
> if (irq_status & SOF_INT_ST)
>         mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev,
>                                              dequeue_frame_seq_no);
>
> /* Sof, done order check */
> if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST))
>         dev_warn(dev, "sof_done block cnt:%d\n", p1_dev->sof_count);
>
> /* Notify IRQ event and de-enqueue frame */
> irq_handle_notify_event(p1_dev, irq_status, dma_status);

Don't we still need to do this conditionally, only if we got HW_PASS1_DON_ST?

[snip]
> > > +/* ISP P1 interface functions */
> > > +int mtk_isp_power_init(struct mtk_cam_dev *cam_dev)
> > > +{
> > > +   struct device *dev = &cam_dev->pdev->dev;
> > > +   struct isp_p1_device *p1_dev = get_p1_device(dev);
> > > +   struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > > +   int ret;
> > > +
> > > +   ret = isp_setup_scp_rproc(p1_dev);
> > > +   if (ret)
> > > +           return ret;
> > > +
> > > +   ret = isp_init_context(p1_dev);
> > > +   if (ret)
> > > +           return ret;
> >
> > The above function doesn't really seem to be related to power management.
> > Should it be called from subdev stream on?
> >
>
> We will rename this function to mtk_isp_hw_init.
> But, it will be called when the first video node is streamed on.
> This is because we need to initialize the HW firstly for sub-device
> stream-on performance.  We need to send some IPI commands, such as
> ISP_CMD_INIT & ISP_CMD_CONFIG_META & ISP_CMD_ENQUEUE_META in this
> timing.

What performance do you mean here? The time between first video node
stream on and last video node stream on should be really short. Are
you seeing some long delays there?

That said, doing it when the first video node starts streaming is okay.

[snip]
> > > +   /* Use pure RAW as default HW path */
> > > +   isp_ctx->isp_raw_path = ISP_PURE_RAW_PATH;
> > > +   atomic_set(&p1_dev->cam_dev.streamed_node_count, 0);
> > > +
> > > +   isp_composer_hw_init(dev);
> > > +   /* Check enabled DMAs which is configured by media setup */
> > > +   isp_composer_meta_config(dev, get_enabled_dma_ports(cam_dev));
> >
> > Hmm, this seems to be also configured by isp_compoer_hw_config(). Are both
> > necessary?
> >
>
> Yes, it is necessary for non-request buffers design for Camera 3A video
> nodes. For 3A video nodes, we just want to know which 3A video nodes are
> enabled in ISP_CMD_CONFIG_META. In this stage, we may not know the image
> format from user space. So we just pass the enabled DMA information from
> kernel to SCP only. With 3A enabled DMA information, we could configure
> related 3A registers in SCP.

We should try to remove this non-request mode. Let's continue
discussion on the other patch where I brought this topic.

[snip]
> > > +int mtk_isp_power_release(struct device *dev)
> > > +{
> > > +   isp_composer_hw_deinit(dev);
> > > +   isp_uninit_context(dev);
> >
> > These two don't seem to be related to power either.
> >
> > Instead, I don't see anything that could undo the rproc_boot() operation
> > here.
> >
>
> We will rename this function to mtk_isp_hw_release.
> We will also add rproc_shutdown function call here.
>
> int mtk_isp_hw_release(struct mtk_cam_dev *cam)
> {
>         struct device *dev = cam->dev;
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>
>         isp_composer_hw_deinit(p1_dev);
>         pm_runtime_put_sync_autosuspend(dev);

Note that for autosuspend to work correctly, you also need to call
pm_runtime_mark_last_busy() before this one.

[snip]
> > > +   struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > > +   struct p1_frame_param frameparams;
> > > +   struct mtk_isp_queue_job *framejob;
> > > +   struct media_request_object *obj, *obj_safe;
> > > +   struct vb2_buffer *vb;
> > > +   struct mtk_cam_dev_buffer *buf;
> > > +
> > > +   framejob = kzalloc(sizeof(*framejob), GFP_ATOMIC);
> >
> > This allocation shouldn't be needed. The structure should be already a part
> > of the mtk_cam_dev_request struct that wraps media_request. Actually. I'd
> > even say that the contents of this struct should be just moved to that one
> > to avoid overabstracting.
> >
>
> For this function, we will apply the new design from P2 driver's
> comment. Here is the new implementation.
>
> struct mtk_cam_dev_request {
>         struct media_request req;
>         struct mtk_p1_frame_param frame_params;
>         struct work_struct frame_work;
>         struct list_head list;
>         atomic_t buf_count;
> };
>
> void mtk_isp_req_enqueue(struct mtk_cam_dev *cam,
>                          struct mtk_cam_dev_request *req)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);
>         int ret;
>
>         req->frame_params.frame_seq_no = ++p1_dev->enqueue_frame_seq_no;
>         INIT_WORK(&req->frame_work, isp_tx_frame_worker);
>         ret = queue_work(p1_dev->composer_wq, &req->frame_work);
>         if (!ret)
>                 dev_dbg(cam->dev, "frame_no:%d queue_work failed\n",
>                         req->frame_params.frame_seq_no, ret);
>         else
>                 dev_dbg(cam->dev, "Enqueue fd:%s frame_seq_no:%d job cnt:%d\n",
>                         req->req.debug_str, req->frame_params.frame_seq_no,
>                         atomic_read(&cam->running_job_count));

It shouldn't be possible for queue_work() to fail here. We just
received a brand new request from the Request API core and called
INIT_WORK() on the work struct.

[snip]
> > > +   enable_sys_clock(p1_dev);
> > > +
> > > +   /* V4L2 stream-on phase & restore HW stream-on status */
> > > +   if (p1_dev->cam_dev.streaming) {
> > > +           dev_dbg(dev, "Cam:%d resume,enable VF\n", module);
> > > +           /* Enable CMOS */
> > > +           reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
> > > +           writel((reg_val | CMOS_EN_BIT),
> > > +                  isp_dev->regs + REG_TG_SEN_MODE);
> > > +           /* Enable VF */
> > > +           reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
> > > +           writel((reg_val | VFDATA_EN_BIT),
> > > +                  isp_dev->regs + REG_TG_VF_CON);
> > > +   }
> >
> > Does the hardware keep all the state, e.g. queued buffers, during suspend?
> > Would the code above continue all the capture from the next buffer, as
> > queued by the userspace before the suspend?
> >
>
> Yes, we will test it.
> 1. SCP buffers are kept by SCP processor
> 2. ISP registers are still kept even if ISP clock is disable.
>

That said, during system suspend, it would be more than just ISP clock
disabled. I'd expect that the ISP power domain would be powered off.
However, if we ensure that the ISP completes before suspend, I guess
that after the resume the next frame CQ buffer would reprogram all the
registers, right?

Also, would SCP always keep running in system suspend?

[snip]
> > > +
> > > +   for (i = ISP_CAMSYS_CONFIG_IDX; i < ISP_DEV_NODE_NUM; i++) {
> >
> > I think we want to start from 0 here?
> >
>
> Because of single CAM support, we will revise our DTS tree to support
> single CAM only.

Note that DT bindings should describe the hardware not the driver. So
please design the bindings in a way that would cover all the cameras,
even if the driver only takes the information needed to handle 1.

> So we could remove this loop and check the CAM-B HW
> information here. Here is below new function.
>
> static int mtk_isp_probe(struct platform_device *pdev)
> {
>         /* List of clocks required by isp cam */
>         static const char * const clk_names[] = {
>                 "camsys_cam_cgpdn", "camsys_camtg_cgpdn"
>         };
>         struct mtk_isp_p1_device *p1_dev;
>         struct device *dev = &pdev->dev;
>         struct resource *res;
>         int irq, ret, i;
>
>         p1_dev = devm_kzalloc(dev, sizeof(*p1_dev), GFP_KERNEL);
>         if (!p1_dev)
>                 return -ENOMEM;
>
>         p1_dev->dev = dev;
>         dev_set_drvdata(dev, p1_dev);
>
>         res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>         p1_dev->regs = devm_ioremap_resource(dev, res);
>         if (IS_ERR(p1_dev->regs)) {
>                 dev_err(dev, "Failed platform resources map\n");
>                 return PTR_ERR(p1_dev->regs);
>         }
>         dev_dbg(dev, "cam, map_addr=0x%pK\n", p1_dev->regs);
>
>         irq = platform_get_irq(pdev, 0);
>         if (!irq) {
>                 dev_err(dev, "Missing IRQ resources data\n");
>                 return -ENODEV;
>         }
>         ret = devm_request_irq(dev, irq, isp_irq_cam, IRQF_SHARED,
>                                dev_name(dev), p1_dev);
>         if (ret) {
>                 dev_err(dev, "req_irq fail, dev:%s irq=%d\n",
>                         dev->of_node->name, irq);
>                 return ret;
>         }
>         dev_dbg(dev, "Reg. irq=%d, isr:%s\n", irq, dev_driver_string(dev));
>         spin_lock_init(&p1_dev->spinlock_irq);
>
>         p1_dev->num_clks = ARRAY_SIZE(clk_names);
>         p1_dev->clks = devm_kcalloc(dev, p1_dev->num_clks,
>                                     sizeof(*p1_dev->clks), GFP_KERNEL);
>         if (!p1_dev->clks)
>                 return -ENOMEM;
>
>         for (i = 0; i < p1_dev->num_clks; ++i)
>                 p1_dev->clks[i].id = clk_names[i];
>
>         ret = devm_clk_bulk_get(dev, p1_dev->num_clks, p1_dev->clks);
>         if (ret) {
>                 dev_err(dev, "cannot get isp cam clock:%d\n", ret);
>                 return ret;
>         }
>
>         ret = isp_setup_scp_rproc(p1_dev, pdev);
>         if (ret)
>                 return ret;
>
>         pm_runtime_enable(dev);

We also need to call pm_runtime_use_autosuspend() and
pm_runtime_set_autosuspend_delay() before enabling runtime PM. I'd
suggest an autosuspend delay equal to around 2x the time that's needed
to stop and start streaming in total.

[snip]
> > > +static const struct dev_pm_ops mtk_isp_pm_ops = {
> > > +   SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
> > > +   SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)
> >
> > For V4L2 drivers system and runtime PM ops would normally be completely
> > different. Runtime PM ops would be called when the hardware is idle already
> > or is about to become active. System PM ops would be called at system power
> > state change and the hardware might be both idle or active. Please also see
> > my comments to mtk_isp_suspend() and mtk_isp_resume() above.
> >
>
> Here is the new implementation. It should be clear to show the
> difference between system and runtime PM ops.
>
> static const struct dev_pm_ops mtk_isp_pm_ops = {
>         SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>                                 pm_runtime_force_resume)
>         SET_RUNTIME_PM_OPS(mtk_isp_runtime_suspend, mtk_isp_runtime_resume,
> NULL)
> };

That's still not correct. In runtime suspend/resume ops we already are
not streaming anymore, because we call pm_runtime_get/put_*() when
starting and stopping streaming. In system suspend/resume ops we might
be streaming and that's when we need to stop the hardware and wait for
it to finish. Please implement these ops separately.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication
  2019-07-21  2:18     ` Jungo Lin
@ 2019-07-25 10:56       ` Tomasz Figa
  2019-07-26  8:07         ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-25 10:56 UTC (permalink / raw)
  To: Jungo Lin
  Cc: Hans Verkuil, Laurent Pinchart, Matthias Brugger,
	Mauro Carvalho Chehab, Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

Hi Jungo,

On Sun, Jul 21, 2019 at 11:18 AM Jungo Lin <jungo.lin@mediatek.com> wrote:
[snip]
> > > +           wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> > > +           isp_ctx->composer_tx_thread.thread = NULL;
> > > +   }
> > > +
> > > +   if (isp_ctx->composer_deinit_thread.thread) {
> > > +           wake_up(&isp_ctx->composer_deinit_thread.wq);
> > > +           isp_ctx->composer_deinit_thread.thread = NULL;
> > > +   }
> > > +   mutex_unlock(&isp_ctx->lock);
> > > +
> > > +   pm_runtime_put_sync(&p1_dev->pdev->dev);
> >
> > No need to use the sync variant.
> >
>
> We don't get this point. If we will call pm_runtime_get_sync in
> mtk_isp_hw_init function, will we need to call
> pm_runtime_put_sync_autosuspend in mtk_isp_hw_release in next patch?
> As we know, we should call runtime pm functions in pair.
>

My point is that pm_runtime_put_sync() is only needed if one wants the
runtime count to be decremented after the function returns. Normally
there is no need to do so and one would call pm_runtime_put(), or if
autosuspend is used, pm_runtime_put_autosuspend() (note there is no
"sync" in the name).

[snip]
> > +static void isp_composer_handler(void *data, unsigned int len, void *priv)
> > > +{
> > > +   struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)priv;
> > > +   struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> > > +   struct device *dev = &p1_dev->pdev->dev;
> > > +   struct mtk_isp_scp_p1_cmd *ipi_msg;
> > > +
> > > +   ipi_msg = (struct mtk_isp_scp_p1_cmd *)data;
> >
> > Should we check that len == sizeof(*ipi_msg)? (Or at least >=, if data could
> > contain some extra bytes at the end.)
> >
>
> The len parameter is the actual sending bytes from SCP to kernel.
> In the runtime, it is only 6 bytes for isp_ack_info command
> However, sizeof(*ipi_msg) is large due to struct mtk_isp_scp_p1_cmd is
> union structure.
>

That said we still should check if len is enough to cover the data
we're accessing below.

> > > +
> > > +   if (ipi_msg->cmd_id != ISP_CMD_ACK)
> > > +           return;
> > > +
> > > +   if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
> > > +           dev_dbg(dev, "ack frame_num:%d",
> > > +                   ipi_msg->ack_info.frame_seq_no);
> > > +           atomic_set(&isp_ctx->composed_frame_id,
> > > +                      ipi_msg->ack_info.frame_seq_no);
> >
> > I suppose we are expecting here that ipi_msg->ack_info.frame_seq_no would be
> > just isp_ctx->composed_frame_id + 1, right? If not, we probably dropped some
> > frames and we should handle that somehow.
> >
>
> No, we use isp_ctx->composed_frame_id to save which frame sequence
> number are composed done in SCP. In new design, we will move this from
> isp_ctx to p1_dev.

But we compose the frames in order, don't we? Wouldn't every composed
frame would be just previous frame ID + 1?

[snip]
> > > +void isp_composer_hw_init(struct device *dev)
> > > +{
> > > +   struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> > > +   struct isp_p1_device *p1_dev = get_p1_device(dev);
> > > +   struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > > +
> > > +   memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> > > +   composer_tx_cmd.cmd_id = ISP_CMD_INIT;
> > > +   composer_tx_cmd.frameparam.hw_module = isp_ctx->isp_hw_module;
> > > +   composer_tx_cmd.frameparam.cq_addr.iova = isp_ctx->scp_mem_iova;
> > > +   composer_tx_cmd.frameparam.cq_addr.scp_addr = isp_ctx->scp_mem_pa;
> >
> > Should we also specify the size of the buffer? Otherwise we could end up
> > with some undetectable overruns.
> >
>
> The size of SCP composer's memory is fixed to 0x200000.
> Is it necessary to specify the size of this buffer?
>
> #define MTK_ISP_COMPOSER_MEM_SIZE 0x200000
>
> ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
>                         MTK_ISP_COMPOSER_MEM_SIZE, &addr, GFP_KERNEL);
>

Okay, but please add a comment saying that this is an implicit
requirement of the firmware.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC, v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-23  8:21             ` [RFC, v3 " Jungo Lin
@ 2019-07-26  5:15               ` Tomasz Figa
  2019-07-26  7:41                 ` Christoph Hellwig
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-26  5:15 UTC (permalink / raw)
  To: Jungo Lin
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Mauro Carvalho Chehab, Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, Matthias Brugger, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	ddavenport, Frederic Chen (陳俊元),
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

On Tue, Jul 23, 2019 at 5:22 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi, Tomasz:
>
> On Tue, 2019-07-23 at 16:20 +0900, Tomasz Figa wrote:
> > Hi Jungo,
> >
> > On Fri, Jul 5, 2019 at 4:59 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > >
> > > Hi Tomasz:
> > >
> > > On Fri, 2019-07-05 at 13:22 +0900, Tomasz Figa wrote:
> > > > Hi Jungo,
> > > >
> > > > On Fri, Jul 5, 2019 at 12:33 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > > >
> > > > > Hi Tomasz,
> > >
> > > [snip]
> > >
> > > > > After applying your suggestion in SCP device driver, we could remove
> > > > > mtk_cam-smem.h/c. Currently, we use dma_alloc_coherent with SCP device
> > > > > to get SCP address. We could touch the buffer with this SCP address in
> > > > > SCP processor.
> > > > >
> > > > > After that, we use dma_map_page_attrs with P1 device which supports
> > > > > IOMMU domain to get IOVA address. For this address, we will assign
> > > > > it to our ISP HW device to proceed.
> > > > >
> > > > > Below is the snippet for ISP P1 compose buffer initialization.
> > > > >
> > > > >         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
> > > > >                                  MAX_COMPOSER_SIZE, &addr, GFP_KERNEL);
> > > > >         if (!ptr) {
> > > > >                 dev_err(dev, "failed to allocate compose memory\n");
> > > > >                 return -ENOMEM;
> > > > >         }
> > > > >         isp_ctx->scp_mem_pa = addr;
> > > >
> > > > addr contains a DMA address, not a physical address. Could we call it
> > > > scp_mem_dma instead?
> > > >
> > > > >         dev_dbg(dev, "scp addr:%pad\n", &addr);
> > > > >
> > > > >         /* get iova address */
> > > > >         addr = dma_map_page_attrs(dev, phys_to_page(addr), 0,
> > > >
> > > > addr is a DMA address, so phys_to_page() can't be called on it. The
> > > > simplest thing here would be to use dma_map_single() with ptr as the
> > > > CPU address expected.
> > > >
> > >
> > > We have changed to use ma_map_single() with ptr, but encounter IOMMU
> > > error. From the debug log of iommu_dma_map_page[3], we got
> > > 0x0000000054800000 instead of expected address: 0x0000000050800000[2].
> > > There is a address offset(0x4000000). If we change to use
> > > dma_map_page_attrs with phys_to_page(addr), the address is correct as we
> > > expected[2]. Do you have any suggestion on this issue? Do we miss
> > > something?
> >
> > Sorry for the late reply. Could you show me the code changes you made
> > to use dma_map_single()? It would sound like the virtual address
> > passed to dma_map_single() isn't correct.
> >
> > Best regards,
> > Tomasz
> >
>
>
> Please check the below code snippet in today's testing.
>
>         p1_dev->cam_dev.smem_dev = &p1_dev->scp_pdev->dev;
>         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
>                                  MTK_ISP_COMPOSER_MEM_SIZE, &addr, GFP_KERNEL);
>         if (!ptr) {
>                 dev_err(dev, "failed to allocate compose memory\n");
>                 return -ENOMEM;
>         }
>         p1_dev->composer_scp_addr = addr;
>         p1_dev->composer_virt_addr = ptr;
>         dev_info(dev, "scp addr:%pad va:%pK\n", &addr, ptr);
>
>         /* get iova address */
>         addr = dma_map_single(dev, ptr, MTK_ISP_COMPOSER_MEM_SIZE,
> DMA_BIDIRECTIONAL);
>         if (dma_mapping_error(dev, addr)) {
>                 dma_free_coherent(p1_dev->cam_dev.smem_dev,
>                                   MTK_ISP_COMPOSER_MEM_SIZE,
>                                   ptr, p1_dev->composer_scp_addr);
>                 dev_err(dev, "Failed to map scp iova\n");
>                 ret = -ENOMEM;
>                 goto fail_free_mem;
>         }
>         p1_dev->composer_iova = addr;
>         dev_info(dev, "scp iova addr:%pad\n", &addr);
>
> Moreover, below is extracted log[2].
>
> We guess the virtual address which is returned by dma_alloc_coherent
> function is not valid kernel logical address. It is actually returned by
> memremap() in dma_init_coherent_memory(). Moreover, dma_map_single()
> will call virt_to_page() function. For virt_to_page function, it
> requires a logical address[1].
>
> [1]https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ch15.html
>

Indeed virt_to_page() works only with kernel LOWMEM addresses. Whether
virt_to_page() is the right thing to do in dma_map_single() is a good
question, but let's assume it was implemented like this for a reason.

However, you also can't call phys_to_page() on the DMA addresses
returned by dma_alloc_*() either. It works just by luck, because SCP
DMA addresses and CPU physical addresses are numerically the same.

Could you try dma_get_sgtable() with the SCP struct device and then
dma_map_sg() with the P1 struct device?

Best regards,
Tomasz

> [2]
>   322 [    1.238269] mtk-cam-p1 1a006000.camisp: scp
> addr:0x0000000052000000 va:00000000a3adc471
>   323 [    1.239582] mtk-cam-p1 1a006000.camisp: scp iova
> addr:0x00000000fde00000
>  7716 [    1.238963] mtk-cam-p1 1a006000.camisp: scp
> addr:0x0000000052000000 va:0000000042ec580f
>  7717 [    1.240276] mtk-cam-p1 1a006000.camisp: scp iova
> addr:0x00000000fde00000
> 15088 [    1.239309] mtk-cam-p1 1a006000.camisp: scp
> addr:0x0000000052000000 va:000000005e5b3462
> 15089 [    1.240626] mtk-cam-p1 1a006000.camisp: scp iova
> addr:0x00000000fde00000
>
> Best regards,
>
> Jungo
>
> > >
> > > [1]
> > > [    1.344786] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
> > > device_base:0x0000000050000000 dma:0x0000000050800000
> > > virt_base:ffffff8014000000 va:ffffff8014800000
> > >
> > > [    1.346890] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
> > > va:ffffff8014800000
> > >
> > > [    1.347864] iommu_dma_map_page:0x0000000054800000 offset:0
> > > [    1.348562] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000
> > >
> > > [2]
> > > [    1.346738] __dma_alloc_from_coherent: 0x800000 PAGE_SHIFT:12
> > > device_base:0x0000000050000000 dma:0x0000000050800000
> > > virt_base:ffffff8014000000 va:ffffff8014800000
> > > [    1.348841] mtk-cam 1a000000.camisp: scp addr:0x0000000050800000
> > > va:ffffff8014800000
> > > [    1.349816] iommu_dma_map_page:0x0000000050800000 offset:0
> > > [    1.350514] mtk-cam 1a000000.camisp: iova addr:0x00000000fde00000
> > >
> > >
> > > [3]
> > > dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
> > >                 unsigned long offset, size_t size, int prot)
> > > {
> > >         phys_addr_t phys = page_to_phys(page);
> > >         pr_err("iommu_dma_map_page:%pa offset:%lu\n", &phys, offset);
> > >
> > >         return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot,
> > >                         iommu_get_dma_domain(dev));
> > > }
> > >
> > > [snip]
> > >
> > > Best regards,
> > >
> > > Jungo
> > >
> >
> > _______________________________________________
> > Linux-mediatek mailing list
> > Linux-mediatek@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-mediatek
>
>

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-24  4:31         ` Jungo Lin
@ 2019-07-26  5:49           ` Tomasz Figa
  2019-07-29  1:18             ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-26  5:49 UTC (permalink / raw)
  To: Jungo Lin, Hans Verkuil
  Cc: Laurent Pinchart, Matthias Brugger, Mauro Carvalho Chehab,
	Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

On Wed, Jul 24, 2019 at 1:31 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi, Tomasz:
>
> On Tue, 2019-07-23 at 19:21 +0900, Tomasz Figa wrote:
> > Hi Jungo,
> >
> > On Thu, Jul 18, 2019 at 1:39 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > >
> > > Hi, Tomasz:
> > >
> > > On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> > > > Hi Jungo,
> > > >
> > > > On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
> > [snip]
> > > > > +static void mtk_cam_req_try_isp_queue(struct mtk_cam_dev *cam_dev,
> > > > > +                                 struct media_request *new_req)
> > > > > +{
> > > > > +   struct mtk_cam_dev_request *req, *req_safe, *cam_dev_req;
> > > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > > +
> > > > > +   dev_dbg(dev, "%s new req:%d", __func__, !new_req);
> > > > > +
> > > > > +   if (!cam_dev->streaming) {
> > > > > +           cam_dev_req = mtk_cam_req_to_dev_req(new_req);
> > > > > +           spin_lock(&cam_dev->req_lock);
> > > > > +           list_add_tail(&cam_dev_req->list, &cam_dev->req_list);
> > > > > +           spin_unlock(&cam_dev->req_lock);
> > > > > +           dev_dbg(dev, "%s: stream off, no ISP enqueue\n", __func__);
> > > > > +           return;
> > > > > +   }
> > > > > +
> > > > > +   /* Normal enqueue flow */
> > > > > +   if (new_req) {
> > > > > +           mtk_isp_req_enqueue(dev, new_req);
> > > > > +           return;
> > > > > +   }
> > > > > +
> > > > > +   /* Flush all media requests wehen first stream on */
> > > > > +   list_for_each_entry_safe(req, req_safe, &cam_dev->req_list, list) {
> > > > > +           list_del(&req->list);
> > > > > +           mtk_isp_req_enqueue(dev, &req->req);
> > > > > +   }
> > > > > +}
> > > >
> > > > This will have to be redone, as per the other suggestions, but generally one
> > > > would have a function that tries to queue as much as possible from a list to
> > > > the hardware and another function that adds a request to the list and calls
> > > > the first function.
> > > >
> > >
> > > We revised this function as below.
> > > First to check the en-queue conditions:
> > > a. stream on
> > > b. The composer buffers in SCP are 3, so we only could has 3 jobs
> > > at the same time.
> > >
> > >
> > > Second, try to en-queue the frames in the pending job if possible and
> > > move them into running job list if possible.
> > >
> > > The request has been inserted into pending job in mtk_cam_req_validate
> > > which is used to validate media_request.
> >
> > Thanks for replying to each of the comments, that's very helpful.
> > Snipped out the parts that I agreed with.
> >
> > Please note that req_validate is not supposed to change any driver
> > state. It's only supposed to validate the request. req_queue is the
> > right callback to insert the request into some internal driver
> > bookkeeping structures.
> >
>
> Yes, in req_validate function, we don't change any driver state.
> Below is the function's implementation.
>
> a. Call vb2_request_validate(req) to verify media request.
> b. Update the buffer internal structure buffer.
> c. Insert the request into pending_job_list to prepare en-queue.
>

Adding to a list is changing driver state. The callback must not
modify anything else than the request itself.

Queuing to driver's list should happen in req_queue instead.

[snip]
> > >
> > > void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam_dev)
> > > {
> > >         struct mtk_cam_dev_request *req, *req_prev;
> > >         struct list_head enqueue_job_list;
> > >         int buffer_cnt = atomic_read(&cam_dev->running_job_count);
> > >         unsigned long flags;
> > >
> > >         if (!cam_dev->streaming ||
> > >             buffer_cnt >= MTK_ISP_MAX_RUNNING_JOBS) {
> >
> > Do we have a guarantee that cam_dev->running_job_count doesn't
> > decrement between the atomic_read() above and this line?
> >
>
> Ok, we will use cam->pending_job_lock to protect
> cam_dev->running_job_count access. Below is the revised version.
>
> void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam)
> {
>         struct mtk_cam_dev_request *req, *req_prev;
>         unsigned long flags;
>
>         if (!cam->streaming) {
>                 dev_dbg(cam->dev, "stream is off\n");
>                 return;
>         }
>
>         spin_lock_irqsave(&cam->pending_job_lock, flags);
>         if (atomic_read(&cam->running_job_count) >= MTK_ISP_MAX_RUNNING_JOBS) {

If we use a spin_lock to protect the counter, perhaps we don't need
the atomic type anymore?

>                 dev_dbg(cam->dev, "jobs are full\n");
>                 spin_unlock_irqrestore(&cam->pending_job_lock, flags);
>                 return;
>         }
>         list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list) {

Could we instead check the counter here and break if it's >=
MTK_ISP_MAX_RUNNING_JOBS?
Then we could increment it here too to simplify the code.

>                 list_del(&req->list);
>                 spin_lock_irqsave(&cam->running_job_lock, flags);
>                 list_add_tail(&req->list, &cam->running_job_list);
>                 mtk_isp_req_enqueue(cam, req);
>                 spin_unlock_irqrestore(&cam->running_job_lock, flags);
>                 if (atomic_inc_return(&cam->running_job_count) >=
>                         MTK_ISP_MAX_RUNNING_JOBS)
>                         break;

With the above suggestion, this if block would go away.

[snip]
> > >                 mtk_isp_req_enqueue(cam_dev, req);
> > >         }
> > > }
> > >
> > [snip]
> > > > > +   stride = DIV_ROUND_UP(stride * pixel_byte, 8);
> > > > > +
> > > > > +   if (pix_fmt == V4L2_PIX_FMT_MTISP_F10)
> > > > > +           stride = ALIGN(stride, 4);
> > > >
> > > > Is it expected that only the F10 format needs this alignment?
> > > >
> > >
> > > yes, if the pixel bits of image format is 10, the byte alignment of bpl
> > > should be 4. Otherwise, it is 8. We will revise this and add more
> > > comments.
> >
> > That means that the B10 format also needs the extra alignment, as
> > opposed to what the original code did, right?
> >
>
> Sorry for short code snippet.
> This alignment checking is only applied to F10, no B10.
> If you like to check the full function, you could check this in this
> link[1].
>
> static void cal_image_pix_mp(struct mtk_cam_dev *cam, unsigned int
> node_id,
>                              struct v4l2_pix_format_mplane *mp)
> {
>         unsigned int bpl, ppl;
>         unsigned int pixel_bits = get_pixel_bits(mp->pixelformat);
>         unsigned int width = mp->width;
>
>         if (node_id == MTK_CAM_P1_MAIN_STREAM_OUT) {
>                 /* bayer encoding format & 2 bytes alignment */
>                 bpl = ALIGN(DIV_ROUND_UP(width * pixel_bits, 8), 2);
>         } else if (node_id == MTK_CAM_P1_PACKED_BIN_OUT) {
>                 /*
>                  * The FULL-G encoding format
>                  * 1 G component per pixel
>                  * 1 R component per 4 pixel
>                  * 1 B component per 4 pixel
>                  * Total 4G/1R/1B in 4 pixel (pixel per line:ppl)
>                  */
>                 ppl = DIV_ROUND_UP(width * 6, 4);
>                 bpl = DIV_ROUND_UP(ppl * pixel_bits, 8);
>
>                 /* 4 bytes alignment for 10 bit & others are 8 bytes */
>                 if (pixel_bits == 10)
>                         bpl = ALIGN(bpl, 4);
>                 else
>                         bpl = ALIGN(bpl, 8);
>         }
>
> [1]
> https://crrev.com/c/1712885/2/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c#303
>

Got it, thanks!

[snip]
> > > > > +
> > > > > +static struct v4l2_subdev *
> > > > > +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> > > > > +{
> > > > > +   struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> > > > > +   struct media_entity *entity;
> > > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > > +   struct v4l2_subdev *sensor;
> > > >
> > > > This variable would be unitialized if there is no streaming sensor. Was
> > > > there no compiler warning generated for this?
> > > >
> > >
> > > No, there is no compiler warning.
> > > But, we will assign sensor to NULL to avoid unnecessary compiler warning
> > > with different compiler options.
> > >
> >
> > Thanks. It would be useful if you could check why the compiler you're
> > using doesn't show a warning here. We might be missing other
> > uninitialized variables.
> >
>
> We will feedback to your project team to check the possible reason about
> compiler warning issue.
>

Do you mean that it was the Clang toolchain used on Chromium OS (e.g.
emerge chromeos-kernel-4_19)?

[snip]
> > > > > +
> > > > > +   dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > > > > +           __func__,
> > > > > +           node->id,
> > > > > +           buf->vbb.request_fd,
> > > > > +           buf->vbb.vb2_buf.index);
> > > > > +
> > > > > +   /* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > > > > +   if (vb->vb2_queue->uses_requests)
> > > > > +           return;
> > > >
> > > > I'd suggest removing non-request support from this driver. Even if we end up
> > > > with a need to provide compatibility for non-request mode, then it should be
> > > > built on top of the requests mode, so that the driver itself doesn't have to
> > > > deal with two modes.
> > > >
> > >
> > > The purpose of non-request function in this driver is needed by
> > > our camera middle-ware design. It needs 3A statistics buffers before
> > > image buffers en-queue. So we need to en-queue 3A statistics with
> > > non-request mode in this driver. After MW got the 3A statistics data, it
> > > will en-queue the images, tuning buffer and other meta buffers with
> > > request mode. Based on this requirement, do you have any suggestion?
> > > For upstream driver, should we only consider request mode?
> > >
> >
> > Where does that requirement come from? Why the timing of queuing of
> > the buffers to the driver is important?
> >
> > [snip]
>
> Basically, this requirement comes from our internal camera
> middle-ware/3A hal in user space. Since this is not generic requirement,
> we will follow your original suggestion to keep the request mode only
> and remove other non-request design in other files. For upstream driver,
> it should support request mode only.
>

Note that Chromium OS will use the "upstream driver" and we don't want
to diverge, so please make the userspace also use only requests. I
don't see a reason why there would be any need to submit any buffers
outside of a request.

[snip]
> > > > > +static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
> > > > > +{
> > > > > +   struct mtk_cam_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
> > > > > +
> > > > > +   v4l2_ctrl_request_complete(vb->req_obj.req,
> > > > > +                              dev->v4l2_dev.ctrl_handler);
> > > >
> > > > This would end up being called multiple times, once for each video node.
> > > > Instead, this should be called explicitly by the driver when it completed
> > > > the request - perhaps in the frame completion handler?
> > > >
> > > > With that, we probably wouldn't even need this callback.
> > > >
> > >
> > > First, if we don't implement this callback function, we will receive
> > > kernel warning as below.
> > >
> > > https://elixir.bootlin.com/linux/latest/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L420
> > >
> > > Second, this function is only be called in __vb2_queue_cancel function.
> > > Moreover, we will remove cam_dev->v4l2_dev.ctrl_handler in next patch.
> > > So could we just implement dummy empty function?
> > >
> > >  * @buf_request_complete: a buffer that was never queued to the driver
> > > but is
> > >  *                      associated with a queued request was canceled.
> > >  *                      The driver will have to mark associated objects in the
> > >  *                      request as completed; required if requests are
> > >  *                      supported.
> > >
> >
> > Good catch, thanks.
> >
> > Sounds like we may indeed need to implement this callback. In
> > particular, we may need to remove the request that the buffer was
> > associated with from the driver queue and return the other buffers
> > associated to it with an error state. This should be similar to
> > handling a request failure.
> > [snip]
>
> Before calling this callback function, the VB2's stop_streaming has been
> called. Normally, we will return the buffers belonged to this vb2 queu
> with error state. On other hand, only if the state of request is
> MEDIA_REQUEST_STATE_QUEUED, the buf_request_complete will be called in
> __vb2_queue_cancel function. It hints this media request has been
> validated and inserted into our driver's pending_job_list or
> running_job_list. So we will call mtk_cam_dev_req_cleanup() remove these
> requests from driver's list when streaming is off. Since we have no
> v4l2_ctrl, do we need to do the above things which is already handled in
> mtk_cam_vb2_stop_streaming function? Maybe is this callback function
> only designed for v4l2_ctrl_request_complete usage?

Are you sure that this callback can be only called after
stop_streaming? Also wouldn't that be after stop_streaming only on 1
queue? The other queues could still remain streaming, but we still
have to return corresponding buffers I believe.

Hans, could you clarify what exactly this callback is supposed to do?

>
> static void mtk_cam_dev_req_cleanup(struct mtk_cam_dev *cam)
> {
>         struct mtk_cam_dev_request *req, *req_prev;
>         unsigned long flags;
>
>         dev_dbg(cam->dev, "%s\n", __func__);
>
>         spin_lock_irqsave(&cam->pending_job_lock, flags);
>         list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list)
>                 list_del(&req->list);
>         spin_unlock_irqrestore(&cam->pending_job_lock, flags);
>
>         spin_lock_irqsave(&cam->running_job_lock, flags);
>         list_for_each_entry_safe(req, req_prev, &cam->running_job_list, list)
>                 list_del(&req->list);
>         spin_unlock_irqrestore(&cam->running_job_lock, flags);
> }
>
> static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
> {
>         struct mtk_cam_dev *cam = vb2_get_drv_priv(vq);
>         struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
>         struct device *dev = cam->dev;
>
>         dev_dbg(dev, "%s node:%d count info:%d", __func__,
>                 node->id, atomic_read(&cam->stream_count));
>
>         mutex_lock(&cam->op_lock);
>         if (atomic_read(&cam->stream_count) == cam->enabled_count)
>                 if (v4l2_subdev_call(&cam->subdev, video, s_stream, 0))
>                         dev_err(dev, "failed to stop streaming\n");
>
>         mtk_cam_vb2_return_all_buffers(cam, node, VB2_BUF_STATE_ERROR);
>
>         /* Check the first node to stream-off */
>         if (!atomic_dec_and_test(&cam->stream_count)) {
>                 mutex_unlock(&cam->op_lock);
>                 return;
>         }
>         mutex_unlock(&cam->op_lock);
>
>         mtk_cam_dev_req_cleanup(cam);
>         media_pipeline_stop(&node->vdev.entity);
> }

[keeping the context for Hans]

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-07-25  9:23       ` Tomasz Figa
@ 2019-07-26  7:23         ` Jungo Lin
  2019-08-06  9:47           ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-26  7:23 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, Laurent Pinchart, Matthias Brugger,
	Mauro Carvalho Chehab, Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

Hi, Tomasz:

On Thu, 2019-07-25 at 18:23 +0900, Tomasz Figa wrote:
> .Hi Jungo,
> 
> On Sat, Jul 20, 2019 at 6:58 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> >
> > Hi, Tomasz:
> >
> > On Wed, 2019-07-10 at 18:56 +0900, Tomasz Figa wrote:
> > > Hi Jungo,
> > >
> > > On Tue, Jun 11, 2019 at 11:53:42AM +0800, Jungo Lin wrote:
> > > > This patch adds the Mediatek ISP P1 HW control device driver.
> > > > It handles the ISP HW configuration, provides interrupt handling and
> > > > initializes the V4L2 device nodes and other functions.
> > > >
> > > > (The current metadata interface used in meta input and partial
> > > > meta nodes is only a temporary solution to kick off the driver
> > > > development and is not ready to be reviewed yet.)
> > > >
> > > > Signed-off-by: Jungo Lin <jungo.lin@mediatek.com>
> > > > ---
> > > >  .../platform/mtk-isp/isp_50/cam/Makefile      |    1 +
> > > >  .../mtk-isp/isp_50/cam/mtk_cam-regs.h         |  126 ++
> > > >  .../platform/mtk-isp/isp_50/cam/mtk_cam.c     | 1087 +++++++++++++++++
> > > >  .../platform/mtk-isp/isp_50/cam/mtk_cam.h     |  243 ++++
> > > >  4 files changed, 1457 insertions(+)
> > > >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
> > > >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
> > > >  create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
> > > >
> > >
> > > Thanks for the patch! Please see my comments inline.
> > >
> > > [snip]
> > >
> >
> > Thanks for your comments. Please check my replies inline.
> >
> 
> Thanks! I'll snip anything I don't have further comments on.
> 
> [snip]

Ok, got it.

> > > > +/* META */
> > > > +#define REG_META0_VB2_INDEX                0x14dc
> > > > +#define REG_META1_VB2_INDEX                0x151c
> > >
> > > I don't believe these registers are really for VB2 indexes.
> > >
> >
> > MTK P1 ISP HW supports frame header spare registers for each DMA, such
> > as CAM_DMA_FH_AAO_SPARE or CAM_DMA_FH_AFO_SPARE. We could save some
> > frame information in these ISP registers. In this case, we save META0
> > VB2 index into AAO FH spare register and META1 VB2 index into AFO FH
> > spare register. These implementation is designed for non-request 3A
> > DMAs. These VB2 indexes are sent in ISP_CMD_ENQUEUE_META command of
> > mtk_isp_enqueue function. So we just call CAM_DMA_FH_AAO_SPARE as
> > REG_META0_VB2_INDEX for easy understanding.
> 
> Unfortunately it's not a good idea to mix hardware concepts with
> naming specific to the OS the driver is written for. Better to keep
> the hardware naming, e.g. CAM_DMA_FH_AAO_SPARE.
> 

Ok, got your point. We will pay attention in next time.
Moreover, we will remove AAO/AFO non-request design in next patch.
So these codes will also be removed.

> > Moreover, if we only need to
> > support request mode, we should remove this here.
> >
> > cmd_params.cmd_id = ISP_CMD_ENQUEUE_META;
> > cmd_params.meta_frame.enabled_dma = dma_port;
> > cmd_params.meta_frame.vb_index = buffer->vbb.vb2_buf.index;
> > cmd_params.meta_frame.meta_addr.iova = buffer->daddr;
> > cmd_params.meta_frame.meta_addr.scp_addr = buffer->scp_addr;
> >
> 
> Okay, removing sounds good to me. Let's keep the code simple.
> 
> [snip]

Thanks.

> > > > +
> > > > +   err_status = irq_status & INT_ST_MASK_CAM_ERR;
> > > > +
> > > > +   /* Sof, done order check */
> > > > +   if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST)) {
> > > > +           dev_dbg(dev, "sof_done block cnt:%d\n", isp_dev->sof_count);
> > > > +
> > > > +           /* Notify IRQ event and enqueue frame */
> > > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 0);
> > > > +           isp_dev->current_frame = hw_frame_num;
> > >
> > > What exactly is hw_frame_num? Shouldn't we assign it before notifying the
> > > event?
> > >
> >
> > This is a another spare register for frame sequence number usage.
> > It comes from struct p1_frame_param:frame_seq_no which is sent by
> > SCP_ISP_FRAME IPI command. We will rename this to dequeue_frame_seq_no.
> > Is it a better understanding?
> 
> I'm sorry, unfortunately it's still not clear to me. Is it the
> sequence number of the frame that was just processed and returned to
> the kernel or the next frame that is going to be processed from now
> on?
> 

It is the next frame that is going to be proceed. 
We simplify the implementation of isp_irq_cam function. The hw_frame_num
is renamed to dequeue_frame_seq_no and saved this value from HW at
SOF_INT_ST. Since it is obtained in SOF_INI_ST event, it means it is
next frame to be processed. If there is SW_PASS1_DON_ST, it means this
frame is processed done. We use this value to de-queue the frame request
and return buffers to VB2.

The normal IRQ sequence is SOF_INT_ST => SW_PASS1_DON_ST &
HW_PASS1_DON_ST.

a. SW_PASS_DON_ST is designed for DMAs done event.
If there is no available DMA buffers en-queued into HW, there is no
SW_PADD_DON_ST.

b. HW_PASS_DON_ST is designed to trigger CQ buffer load procedure.
It is paired with SOF IRQ event, even if there is no available DMA
buffers.

static void isp_irq_handle_sof(struct mtk_isp_p1_device *p1_dev,
			       unsigned int dequeue_frame_seq_no)
{
	dma_addr_t base_addr = p1_dev->composer_iova;
	int composed_frame_seq_no =
atomic_read(&p1_dev->composed_frame_seq_no);
	unsigned int addr_offset;

	/* Send V4L2_EVENT_FRAME_SYNC event */
	mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev, dequeue_frame_seq_no);
	
	p1_dev->sof_count += 1;
	/* Save dequeue frame information */
	p1_dev->dequeue_frame_seq_no = dequeue_frame_seq_no;

	/* Update CQ base address if needed */
	if (composed_frame_seq_no <= dequeue_frame_seq_no) {
		dev_dbg(p1_dev->dev,
			"SOF_INT_ST, no update, cq_num:%d, frame_seq:%d",
			composed_frame_seq_no, dequeue_frame_seq_no);
		return;
	}
	addr_offset = MTK_ISP_CQ_ADDRESS_OFFSET *
		(dequeue_frame_seq_no % MTK_ISP_CQ_BUFFER_COUNT);
	writel(base_addr + addr_offset, p1_dev->regs + REG_CQ_THR0_BASEADDR);
	dev_dbg(p1_dev->dev,
		"SOF_INT_ST, update next, cq_num:%d, frame_seq:%d cq_addr:0x%x",
		composed_frame_seq_no, dequeue_frame_seq_no, addr_offset);
}

void mtk_cam_dev_dequeue_req_frame(struct mtk_cam_dev *cam,
				   unsigned int frame_seq_no)
{
	struct mtk_cam_dev_request *req, *req_prev;
	unsigned long flags;

	spin_lock_irqsave(&cam->running_job_lock, flags);
	list_for_each_entry_safe(req, req_prev, &cam->running_job_list, list) {
		dev_dbg(cam->dev, "frame_seq:%d, de-queue frame_seq:%d\n",
			req->frame_params.frame_seq_no, frame_seq_no);

		/* Match by the en-queued request number */
		if (req->frame_params.frame_seq_no == frame_seq_no) {
			atomic_dec(&cam->running_job_count);
			/* Pass to user space */
			mtk_cam_dev_job_done(cam, req, VB2_BUF_STATE_DONE);
			list_del(&req->list);
			break;
		} else if (req->frame_params.frame_seq_no < frame_seq_no) {
			atomic_dec(&cam->running_job_count);
			/* Pass to user space for frame drop */
			mtk_cam_dev_job_done(cam, req, VB2_BUF_STATE_ERROR);
			dev_warn(cam->dev, "frame_seq:%d drop\n",
				 req->frame_params.frame_seq_no);
			list_del(&req->list);
		} else {
			break;
		}
	}
	spin_unlock_irqrestore(&cam->running_job_lock, flags);

static irqreturn_t isp_irq_cam(int irq, void *data)
{
	struct mtk_isp_p1_device *p1_dev = (struct mtk_isp_p1_device *)data;
	struct device *dev = p1_dev->dev;
	unsigned int dequeue_frame_seq_no;
	unsigned int irq_status, err_status, dma_status;
	unsigned long flags;

	spin_lock_irqsave(&p1_dev->spinlock_irq, flags);
	irq_status = readl(p1_dev->regs + REG_CTL_RAW_INT_STAT);
	err_status = irq_status & INT_ST_MASK_CAM_ERR;
	dma_status = readl(p1_dev->regs + REG_CTL_RAW_INT2_STAT);
	dequeue_frame_seq_no = readl(p1_dev->regs + REG_FRAME_SEQ_NUM);
	spin_unlock_irqrestore(&p1_dev->spinlock_irq, flags);

	/*
	 * In normal case, the next SOF ISR should come after HW PASS1 DONE
ISR.
	 * If these two ISRs come together, print warning msg to hint.
	 */
	if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST))
		dev_warn(dev, "sof_done block cnt:%d\n", p1_dev->sof_count);

	/* De-queue frame */
	if (irq_status & SW_PASS1_DON_ST) {
		mtk_cam_dev_dequeue_req_frame(&p1_dev->cam_dev,
					      dequeue_frame_seq_no);
		mtk_cam_dev_req_try_queue(&p1_dev->cam_dev);
	}

	/* Save frame info. & update CQ address for frame HW en-queue */
	if (irq_status & SOF_INT_ST)
		isp_irq_handle_sof(p1_dev, dequeue_frame_seq_no);

	/* Check ISP error status */
	if (err_status) {
		dev_err(dev, "int_err:0x%x 0x%x\n", irq_status, err_status);
		/* Show DMA errors in detail */
		if (err_status & DMA_ERR_ST)
			isp_irq_handle_dma_err(p1_dev);
	}

	dev_dbg(dev, "SOF:%d irq:0x%x, dma:0x%x, frame_num:%d",
		p1_dev->sof_count, irq_status, dma_status,
		dequeue_frame_seq_no);

	return IRQ_HANDLED;
}

> >
> > Below is our frame request handling in current design.
> >
> > 1. Buffer preparation
> > - Combined image buffers (IMGO/RRZO) + meta input buffer (Tuining) +
> > other meta histogram buffers (LCSO/LMVO) into one request.
> > - Accumulated one unique frame sequence number to each request and send
> > this request to the SCP composer to compose CQ (Command queue) buffer
> > via SCP_ISP_FRAME IPI command.
> > - CQ buffer is frame registers set. If ISP registers should be updated
> > per frame, these registers are configured in the CQ buffer, such as
> > frame sequence number, DMA addresses and tuning ISP registers.
> > - One frame request will be composed into one CQ buffer.Once CQ buffer
> > is composed done and kernel driver will receive ISP_CMD_FRAME_ACK with
> > its corresponding frame sequence number. Based on this, kernel driver
> > knows which request is ready to be en-queued and save this with
> > p1_dev->isp_ctx.composed_frame_id.
> 
> Hmm, why do we need to save this in p1_dev->isp_ctx? Wouldn't we
> already have a linked lists of requests that are composed and ready to
> be enqueued? Also, the request itself would contain its frame ID
> inside the driver request struct, right?
> 

Below is current implementation for frame request en-queued.
Before en-queued into HW by CQ, the request should be composed by SCP
composer.

a. mtk_cam_dev_req_try_queue()
- Insert the request into p1_dev->running_job_list
b. mtk_isp_req_enqueue()
- Assign new next frame ID to this request.
- Sending to SCP by workqueue
- This request is ready to compose
c. isp_tx_frame_worker()
- Send request to SCP with sync. mode. by SCP_IPI_ISP_FRAME command
- SCP composer will compose the buffer CQ for this request frame based
on struct mtk_p1_frame_param which includes frame ID.
- If scp_ipi_send() is returned, it means the request is composed done.
Or
d. isp_composer_handler()
- If we received the ISP_CMD_FRAME_ACK for SCP_IPI_ISP_FRAME, we save
the frame ID in p1_dev->composed_frame_seq_no which is sent in step C.
- The request is composed done here.
e. isp_irq_handle_sof()
- In SOF timing, we will check there is any available composed CQ
buffers by comparing composed & current de-queued frame ID.

For p1_dev->running_job_list, we can't guarantee the requests are
composed until the end of step c. For step e, we need to know how many
available composed requests are ready to en-queued.

Do you suggest we add another new link-list to save these requests in
step c or we could update p1_dev->composed_frame_seq_no in step c and
remove the implementation in step d[1]?

[1]. isp_composer_handler() is mandatory callback function for SCP
sending API with sync mode design.

static void isp_composer_handler(void *data, unsigned int len, void
*priv)
{
	struct mtk_isp_p1_device *p1_dev = (struct mtk_isp_p1_device *)priv;
	struct mtk_isp_scp_p1_cmd *ipi_msg;

	ipi_msg = (struct mtk_isp_scp_p1_cmd *)data;

	if (ipi_msg->cmd_id != ISP_CMD_ACK)
		return;

	if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
		atomic_set(&p1_dev->composed_frame_seq_no,
			   ipi_msg->ack_info.frame_seq_no);
		dev_dbg(p1_dev->dev, "ack frame_num:%d\n",
			p1_dev->composed_frame_seq_no);
	}
}

> > - The maximum number of CQ buffers in SCP is 3.
> >
> > 2. Buffer en-queue flow
> > - In order to configure correct CQ buffer setting before next SQF event,
> > it is depended on by MTK ISP P1 HW CQ mechanism.
> > - The basic concept of CQ mechanism is loaded ISP CQ buffer settings
> > when HW_PASS1_DON_ST is received which means DMA output is done.
> > - Btw, the pre-condition of this, need to tell ISP HW which CQ buffer
> > address is used. Otherwise, it will loaded one dummy CQ buffer to
> > bypass.
> > - So we will check available CQ buffers by comparing composed frame
> > sequence number & dequeued frame sequence from ISP HW in SOF event.
> > - If there are available CQ buffers, update the CQ base address to the
> > next CQ buffer address based on current de-enqueue frame sequence
> > number. So MTK ISP P1 HW will load this CQ buffer into HW when
> > HW_PASS1_DON_ST is triggered which is before the next SOF.
> > - So in next SOF event, ISP HW starts to output DMA buffers with this
> > request until request is done.
> > - But, for the first request, it is loaded into HW manually when
> > streaming is on for better performance.
> >
> > 3. Buffer de-queue flow
> > - We will use frame sequence number to decide which request is ready to
> > de-queue.
> > - We will save some important register setting from ISP HW when SOF is
> > received. This is because the ISP HW starts to output the data with the
> > corresponding settings, especially frame sequence number setting.
> 
> Could you explain a bit more about these important register settings?
> When does the hardware update the values in the register to new ones?
> At SOF?
> 

Sorry about my words.
In the current implementation, we just save frame ID.


> > - When receiving SW_PASS1_DON_ST IRQ event, it means the DMA output is
> > done. So we could call isp_deque_request_frame with frame sequence
> > number to de-queue frame to VB2
> 
> What's the difference between HW_PASS1_DON_ST and SW_PASS1_DON_ST?
> 

This is explained above.

> > - For AAO/AFO buffers, it has similar design concept. Sometimes, if only
> > AAO/AFO non-request buffers are en-queued without request buffers at the
> > same time, there will be no SW P1 done event for AAO/AFO DMA done.
> > Needs to depend on other IRQ events, such as AAO/AFO_DONE_EVENT.
> 
> Do we have a case like this? Wouldn't we normally always want to
> bundle AAO/AFO buffers with frame buffers?
> 

For upstream driver, we will remove non-request design.

> > - Due to CQ buffer number limitation, if we receive SW_PASS1_DONT_ST,
> > we may try to send another request to SCP for composing.
> 
> Okay, so basically in SW_PASS1_DONT_ST the CQ completed reading the CQ
> buffers, right?
> 

We expected the the life cycle of CQ buffer is same as frame request.
So SW_PASS1_DON_ST is good timing to re-queue the next request to
compose.
For the CQ operations, we will explain later.

> >
> > Hopefully, my explanation is helpful for better understanding our
> > implementation. If you still have any questions, please let me know.
> >
> 
> Yes, it's more clear now, thanks. Still some more comments above, though.
> 
> > > > +           isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > +           isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > +   } else {
> > > > +           if (irq_status & SOF_INT_ST) {
> > > > +                   isp_dev->current_frame = hw_frame_num;
> > > > +                   isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > +                   isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > +           }
> > > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
> > > > +   }
> > >
> > > The if and else blocks do almost the same things just in different order. Is
> > > it really expected?
> > >
> >
> > If we receive HW_PASS1_DON_ST & SOF_INT_ST IRQ events at the same time,
> > the correct sequence should be handle HW_PASS1_DON_ST firstly to check
> > any de-queued frame and update the next frame setting later.
> > Normally, this is a corner case or system performance issue.
> 
> So it sounds like HW_PASS1_DON_ST means that all data from current
> frame has been written, right? If I understand your explanation above
> correctly, that would mean following handling of each interrupt:
> 
> HW_PASS1_DON_ST:
>  - CQ executes with next CQ buffer to prepare for next frame. <- how
> is this handled? does the CQ hardware automatically receive this event
> from the ISP hadware?
>  - return VB2 buffers,
>  - complete requests.
> 
> SOF_INT_ST:
>  - send VSYNC event to userspace,
>  - program next CQ buffer to CQ,
> 
> SW_PASS1_DON_ST:
>  - reclaim CQ buffer and enqueue next frame to composing if available
> 

Sorry for our implementation of HW_PASS1_DON_ST.
It is confusing. 
Below is the revised version based on your conclusion.
So in our new implemmenation, we just handle SOF_INT_ST &
SW_PASS1_DON_ST events. We just add one warning message for
HW_PASS1_DON_ST
 
HW_PASS1_DON_ST:
- CQ executes with next CQ buffer to prepare for next frame.
 
SOF_INT_ST:
- send VSYNC event to userspace,
- program next CQ buffer to CQ,
 
SW_PASS1_DON_ST:
- reclaim CQ buffer and enqueue next frame to composing if available
- return VB2 buffers,
- complete requests.

For CQ HW operations, it is listed below:

a. The CQ buffer has two kinds of information
 - Which ISP registers needs to be updated.
 - Where the corresponding ISP register data to be read.
b. The CQ buffer loading procedure is triggered by HW_PASS1_DONT_ST IRQ
event periodically. 
 - Normally, if the ISP HW receives the completed frame and it will
trigger W_PASS1_DONT_ST IRQ and perform CQ buffer loading immediately.
-  So the CQ buffer loading is performed by ISP HW automatically.
c. The ISP HW will read CQ base address register(REG_CQ_THR0_BASEADDR)
to decide which CQ buffer is loaded.
   - So we configure the next CQ base address in SOF.
d. For CQ buffer loading, CQ will read the ISP registers from CQ buffer
and update the ISP register values into HW.
   - SCP composer will compose one dummy CQ buffer and assign it to
REG_CQ_THR0_BASEADDR of each CQ buffer.
   - Dummy CQ buffer has no updated ISP registers comparing with other
CQ buffers.
   - With this design, if there is no updated new CQ buffer by driver
which may be caused no en-queue frames from user space. The CQ HW will
load dummy CQ buffer and do nothing.
f. The CQ buffer loading is guaranteed by HW to finish before the next
SOF.

> >
> > Btw, we will revise the above implementation as below.
> >
> >
> > if (irq_status & SOF_INT_ST)
> >         mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev,
> >                                              dequeue_frame_seq_no);
> >
> > /* Sof, done order check */
> > if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST))
> >         dev_warn(dev, "sof_done block cnt:%d\n", p1_dev->sof_count);
> >
> > /* Notify IRQ event and de-enqueue frame */
> > irq_handle_notify_event(p1_dev, irq_status, dma_status);
> 
> Don't we still need to do this conditionally, only if we got HW_PASS1_DON_ST?
> 
> [snip]

Yes, in the new version, we will add SW_PASS1_DON_ST check before
calling mtk_cam_dev_dequeue_req_frame function.

> > > > +/* ISP P1 interface functions */
> > > > +int mtk_isp_power_init(struct mtk_cam_dev *cam_dev)
> > > > +{
> > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > +   struct isp_p1_device *p1_dev = get_p1_device(dev);
> > > > +   struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > > > +   int ret;
> > > > +
> > > > +   ret = isp_setup_scp_rproc(p1_dev);
> > > > +   if (ret)
> > > > +           return ret;
> > > > +
> > > > +   ret = isp_init_context(p1_dev);
> > > > +   if (ret)
> > > > +           return ret;
> > >
> > > The above function doesn't really seem to be related to power management.
> > > Should it be called from subdev stream on?
> > >
> >
> > We will rename this function to mtk_isp_hw_init.
> > But, it will be called when the first video node is streamed on.
> > This is because we need to initialize the HW firstly for sub-device
> > stream-on performance.  We need to send some IPI commands, such as
> > ISP_CMD_INIT & ISP_CMD_CONFIG_META & ISP_CMD_ENQUEUE_META in this
> > timing.
> 
> What performance do you mean here? The time between first video node
> stream on and last video node stream on should be really short. Are
> you seeing some long delays there?
> 
> That said, doing it when the first video node starts streaming is okay.
> 
> [snip]

Ok, for IPI command sending performance issue is gone with removing the
design of non-request mode of 3A buffers. We could skip this question.

> > > > +   /* Use pure RAW as default HW path */
> > > > +   isp_ctx->isp_raw_path = ISP_PURE_RAW_PATH;
> > > > +   atomic_set(&p1_dev->cam_dev.streamed_node_count, 0);
> > > > +
> > > > +   isp_composer_hw_init(dev);
> > > > +   /* Check enabled DMAs which is configured by media setup */
> > > > +   isp_composer_meta_config(dev, get_enabled_dma_ports(cam_dev));
> > >
> > > Hmm, this seems to be also configured by isp_compoer_hw_config(). Are both
> > > necessary?
> > >
> >
> > Yes, it is necessary for non-request buffers design for Camera 3A video
> > nodes. For 3A video nodes, we just want to know which 3A video nodes are
> > enabled in ISP_CMD_CONFIG_META. In this stage, we may not know the image
> > format from user space. So we just pass the enabled DMA information from
> > kernel to SCP only. With 3A enabled DMA information, we could configure
> > related 3A registers in SCP.
> 
> We should try to remove this non-request mode. Let's continue
> discussion on the other patch where I brought this topic.
> 
> [snip]

Ok. We will remove the non-request in next patch.

> > > > +int mtk_isp_power_release(struct device *dev)
> > > > +{
> > > > +   isp_composer_hw_deinit(dev);
> > > > +   isp_uninit_context(dev);
> > >
> > > These two don't seem to be related to power either.
> > >
> > > Instead, I don't see anything that could undo the rproc_boot() operation
> > > here.
> > >
> >
> > We will rename this function to mtk_isp_hw_release.
> > We will also add rproc_shutdown function call here.
> >
> > int mtk_isp_hw_release(struct mtk_cam_dev *cam)
> > {
> >         struct device *dev = cam->dev;
> >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> >
> >         isp_composer_hw_deinit(p1_dev);
> >         pm_runtime_put_sync_autosuspend(dev);
> 
> Note that for autosuspend to work correctly, you also need to call
> pm_runtime_mark_last_busy() before this one.
> 
> [snip]

Ok, thanks for your hint.

> > > > +   struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > > > +   struct p1_frame_param frameparams;
> > > > +   struct mtk_isp_queue_job *framejob;
> > > > +   struct media_request_object *obj, *obj_safe;
> > > > +   struct vb2_buffer *vb;
> > > > +   struct mtk_cam_dev_buffer *buf;
> > > > +
> > > > +   framejob = kzalloc(sizeof(*framejob), GFP_ATOMIC);
> > >
> > > This allocation shouldn't be needed. The structure should be already a part
> > > of the mtk_cam_dev_request struct that wraps media_request. Actually. I'd
> > > even say that the contents of this struct should be just moved to that one
> > > to avoid overabstracting.
> > >
> >
> > For this function, we will apply the new design from P2 driver's
> > comment. Here is the new implementation.
> >
> > struct mtk_cam_dev_request {
> >         struct media_request req;
> >         struct mtk_p1_frame_param frame_params;
> >         struct work_struct frame_work;
> >         struct list_head list;
> >         atomic_t buf_count;
> > };
> >
> > void mtk_isp_req_enqueue(struct mtk_cam_dev *cam,
> >                          struct mtk_cam_dev_request *req)
> > {
> >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);
> >         int ret;
> >
> >         req->frame_params.frame_seq_no = ++p1_dev->enqueue_frame_seq_no;
> >         INIT_WORK(&req->frame_work, isp_tx_frame_worker);
> >         ret = queue_work(p1_dev->composer_wq, &req->frame_work);
> >         if (!ret)
> >                 dev_dbg(cam->dev, "frame_no:%d queue_work failed\n",
> >                         req->frame_params.frame_seq_no, ret);
> >         else
> >                 dev_dbg(cam->dev, "Enqueue fd:%s frame_seq_no:%d job cnt:%d\n",
> >                         req->req.debug_str, req->frame_params.frame_seq_no,
> >                         atomic_read(&cam->running_job_count));
> 
> It shouldn't be possible for queue_work() to fail here. We just
> received a brand new request from the Request API core and called
> INIT_WORK() on the work struct.
> 
> [snip]

Ok, got it. We will remove this unnecessary error checking.

> > > > +   enable_sys_clock(p1_dev);
> > > > +
> > > > +   /* V4L2 stream-on phase & restore HW stream-on status */
> > > > +   if (p1_dev->cam_dev.streaming) {
> > > > +           dev_dbg(dev, "Cam:%d resume,enable VF\n", module);
> > > > +           /* Enable CMOS */
> > > > +           reg_val = readl(isp_dev->regs + REG_TG_SEN_MODE);
> > > > +           writel((reg_val | CMOS_EN_BIT),
> > > > +                  isp_dev->regs + REG_TG_SEN_MODE);
> > > > +           /* Enable VF */
> > > > +           reg_val = readl(isp_dev->regs + REG_TG_VF_CON);
> > > > +           writel((reg_val | VFDATA_EN_BIT),
> > > > +                  isp_dev->regs + REG_TG_VF_CON);
> > > > +   }
> > >
> > > Does the hardware keep all the state, e.g. queued buffers, during suspend?
> > > Would the code above continue all the capture from the next buffer, as
> > > queued by the userspace before the suspend?
> > >
> >
> > Yes, we will test it.
> > 1. SCP buffers are kept by SCP processor
> > 2. ISP registers are still kept even if ISP clock is disable.
> >
> 
> That said, during system suspend, it would be more than just ISP clock
> disabled. I'd expect that the ISP power domain would be powered off.
> However, if we ensure that the ISP completes before suspend, I guess
> that after the resume the next frame CQ buffer would reprogram all the
> registers, right?
> 
> Also, would SCP always keep running in system suspend?
> 
> [snip]

Q1. Firstly, we will make sure the ISP complete the current frame before
suspend. For the resume behavior, we will check internally.

Q2. Sorry for my wrong reply. The CQ buffer data should be kept in
memory, not SCP. SCP should also be stopped when system is suspend.

> > > > +
> > > > +   for (i = ISP_CAMSYS_CONFIG_IDX; i < ISP_DEV_NODE_NUM; i++) {
> > >
> > > I think we want to start from 0 here?
> > >
> >
> > Because of single CAM support, we will revise our DTS tree to support
> > single CAM only.
> 
> Note that DT bindings should describe the hardware not the driver. So
> please design the bindings in a way that would cover all the cameras,
> even if the driver only takes the information needed to handle 1.
> 

Ok, we will cover all Camera ISP P1 HW units in the DT.

> > So we could remove this loop and check the CAM-B HW
> > information here. Here is below new function.
> >
> > static int mtk_isp_probe(struct platform_device *pdev)
> > {
> >         /* List of clocks required by isp cam */
> >         static const char * const clk_names[] = {
> >                 "camsys_cam_cgpdn", "camsys_camtg_cgpdn"
> >         };
> >         struct mtk_isp_p1_device *p1_dev;
> >         struct device *dev = &pdev->dev;
> >         struct resource *res;
> >         int irq, ret, i;
> >
> >         p1_dev = devm_kzalloc(dev, sizeof(*p1_dev), GFP_KERNEL);
> >         if (!p1_dev)
> >                 return -ENOMEM;
> >
> >         p1_dev->dev = dev;
> >         dev_set_drvdata(dev, p1_dev);
> >
> >         res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> >         p1_dev->regs = devm_ioremap_resource(dev, res);
> >         if (IS_ERR(p1_dev->regs)) {
> >                 dev_err(dev, "Failed platform resources map\n");
> >                 return PTR_ERR(p1_dev->regs);
> >         }
> >         dev_dbg(dev, "cam, map_addr=0x%pK\n", p1_dev->regs);
> >
> >         irq = platform_get_irq(pdev, 0);
> >         if (!irq) {
> >                 dev_err(dev, "Missing IRQ resources data\n");
> >                 return -ENODEV;
> >         }
> >         ret = devm_request_irq(dev, irq, isp_irq_cam, IRQF_SHARED,
> >                                dev_name(dev), p1_dev);
> >         if (ret) {
> >                 dev_err(dev, "req_irq fail, dev:%s irq=%d\n",
> >                         dev->of_node->name, irq);
> >                 return ret;
> >         }
> >         dev_dbg(dev, "Reg. irq=%d, isr:%s\n", irq, dev_driver_string(dev));
> >         spin_lock_init(&p1_dev->spinlock_irq);
> >
> >         p1_dev->num_clks = ARRAY_SIZE(clk_names);
> >         p1_dev->clks = devm_kcalloc(dev, p1_dev->num_clks,
> >                                     sizeof(*p1_dev->clks), GFP_KERNEL);
> >         if (!p1_dev->clks)
> >                 return -ENOMEM;
> >
> >         for (i = 0; i < p1_dev->num_clks; ++i)
> >                 p1_dev->clks[i].id = clk_names[i];
> >
> >         ret = devm_clk_bulk_get(dev, p1_dev->num_clks, p1_dev->clks);
> >         if (ret) {
> >                 dev_err(dev, "cannot get isp cam clock:%d\n", ret);
> >                 return ret;
> >         }
> >
> >         ret = isp_setup_scp_rproc(p1_dev, pdev);
> >         if (ret)
> >                 return ret;
> >
> >         pm_runtime_enable(dev);
> 
> We also need to call pm_runtime_use_autosuspend() and
> pm_runtime_set_autosuspend_delay() before enabling runtime PM. I'd
> suggest an autosuspend delay equal to around 2x the time that's needed
> to stop and start streaming in total.
> 
> [snip]

O, we will add these function calls.

> > > > +static const struct dev_pm_ops mtk_isp_pm_ops = {
> > > > +   SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
> > > > +   SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)
> > >
> > > For V4L2 drivers system and runtime PM ops would normally be completely
> > > different. Runtime PM ops would be called when the hardware is idle already
> > > or is about to become active. System PM ops would be called at system power
> > > state change and the hardware might be both idle or active. Please also see
> > > my comments to mtk_isp_suspend() and mtk_isp_resume() above.
> > >
> >
> > Here is the new implementation. It should be clear to show the
> > difference between system and runtime PM ops.
> >
> > static const struct dev_pm_ops mtk_isp_pm_ops = {
> >         SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> >                                 pm_runtime_force_resume)
> >         SET_RUNTIME_PM_OPS(mtk_isp_runtime_suspend, mtk_isp_runtime_resume,
> > NULL)
> > };
> 
> That's still not correct. In runtime suspend/resume ops we already are
> not streaming anymore, because we call pm_runtime_get/put_*() when
> starting and stopping streaming. In system suspend/resume ops we might
> be streaming and that's when we need to stop the hardware and wait for
> it to finish. Please implement these ops separately.
> 
> Best regards,
> Tomasz


Ok, got your point.
Below is the new implementation for your review.

static int mtk_isp_pm_suspend(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
	u32 val;
	int ret;

	dev_dbg(dev, "- %s\n", __func__);

	/* Check ISP is streaming or not */
	if (!p1_dev->cam_dev.streaming)
		goto done;

	/* Disable ISP's view finder and wait for TG idle */
	dev_dbg(dev, "Cam suspend, disable VF\n");
	val = readl(p1_dev->regs + REG_TG_VF_CON);
	writel(val & (~TG_VF_CON_VFDATA_EN), p1_dev->regs + REG_TG_VF_CON);
	ret = readl_poll_timeout_atomic(p1_dev->regs + REG_TG_INTER_ST, val,
					(val & TG_CS_MASK) == TG_IDLE_ST,
					USEC_PER_MSEC, MTK_ISP_STOP_HW_TIMEOUT);
	if (ret)
		dev_warn(dev, "can't stop HW:%d:0x%x\n", ret, val);

	/* Disable CMOS */
	val = readl(p1_dev->regs + REG_TG_SEN_MODE);
	writel(val & (~TG_SEN_MODE_CMOS_EN), p1_dev->regs + REG_TG_SEN_MODE);

done:
	/* Force ISP HW to idle */
	ret = pm_runtime_force_suspend(dev);
	if (ret)
		return ret;

	return 0;
}

static int mtk_isp_pm_resume(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
	u32 val;
	int ret;

	dev_dbg(dev, "- %s\n", __func__);

	/* Force ISP HW to resume if needed */
	ret = pm_runtime_force_resume(dev);
	if (ret)
		return ret;

	if (!p1_dev->cam_dev.streaming)
		return 0;

	/* Enable CMOS */
	dev_dbg(dev, "Cam resume, enable CMOS/VF\n");
	val = readl(p1_dev->regs + REG_TG_SEN_MODE);
	writel(val | TG_SEN_MODE_CMOS_EN, p1_dev->regs + REG_TG_SEN_MODE);

	/* Enable VF */
	val = readl(p1_dev->regs + REG_TG_VF_CON);
	writel(val | TG_VF_CON_VFDATA_EN, p1_dev->regs + REG_TG_VF_CON);

	return 0;
}

static int mtk_isp_runtime_suspend(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);

	dev_dbg(dev, "- %s\n", __func__);

	if (pm_runtime_suspended(dev))
		return 0;

	dev_dbg(dev, "%s:disable clock\n", __func__);
	clk_bulk_disable_unprepare(p1_dev->num_clks, p1_dev->clks);

	return 0;
}

static int mtk_isp_runtime_resume(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
	int ret;

	dev_dbg(dev, "- %s\n", __func__);

	if (pm_runtime_suspended(dev))
		return 0;

	dev_dbg(dev, "enable clock\n");
	ret = clk_bulk_prepare_enable(p1_dev->num_clks, p1_dev->clks);
	if (ret) {
		dev_err(dev, "cannot enable clock:%d\n", ret);
		return ret;
	}

	return 0;
}

static const struct dev_pm_ops mtk_isp_pm_ops = {
	SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_pm_suspend, mtk_isp_pm_resume)
	SET_RUNTIME_PM_OPS(mtk_isp_runtime_suspend, mtk_isp_runtime_resume,
			   NULL)
};

Best regards,


Jungo


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC, v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-26  5:15               ` Tomasz Figa
@ 2019-07-26  7:41                 ` Christoph Hellwig
  2019-07-26  7:42                   ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Christoph Hellwig @ 2019-07-26  7:41 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Jungo Lin, devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>, ,
	Linux Media Mailing List

On Fri, Jul 26, 2019 at 02:15:14PM +0900, Tomasz Figa wrote:
> Could you try dma_get_sgtable() with the SCP struct device and then
> dma_map_sg() with the P1 struct device?

Please don't do that.  dma_get_sgtable is a pretty broken API (see
the common near the arm implementation) and we should not add more
users of it.  If you want a piece of memory that can be mapped to
multiple devices allocate it using alloc_pages and then just map
it to each device.

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC, v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-26  7:41                 ` Christoph Hellwig
@ 2019-07-26  7:42                   ` Tomasz Figa
  2019-07-26 11:04                     ` Robin Murphy
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-26  7:42 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Jungo Lin, devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

On Fri, Jul 26, 2019 at 4:41 PM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Fri, Jul 26, 2019 at 02:15:14PM +0900, Tomasz Figa wrote:
> > Could you try dma_get_sgtable() with the SCP struct device and then
> > dma_map_sg() with the P1 struct device?
>
> Please don't do that.  dma_get_sgtable is a pretty broken API (see
> the common near the arm implementation) and we should not add more
> users of it.  If you want a piece of memory that can be mapped to
> multiple devices allocate it using alloc_pages and then just map
> it to each device.

Thanks for taking a look at this thread.

Unfortunately that wouldn't work. We have a specific reserved memory
pool that is the only memory area accessible to one of the devices.
Any idea how to handle this?

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication
  2019-07-25 10:56       ` Tomasz Figa
@ 2019-07-26  8:07         ` Jungo Lin
  0 siblings, 0 replies; 45+ messages in thread
From: Jungo Lin @ 2019-07-26  8:07 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, Laurent Pinchart, Matthias Brugger,
	Mauro Carvalho Chehab, Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

Hi, Tomasz:

On Thu, 2019-07-25 at 19:56 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Sun, Jul 21, 2019 at 11:18 AM Jungo Lin <jungo.lin@mediatek.com> wrote:
> [snip]
> > > > +           wake_up_interruptible(&isp_ctx->composer_tx_thread.wq);
> > > > +           isp_ctx->composer_tx_thread.thread = NULL;
> > > > +   }
> > > > +
> > > > +   if (isp_ctx->composer_deinit_thread.thread) {
> > > > +           wake_up(&isp_ctx->composer_deinit_thread.wq);
> > > > +           isp_ctx->composer_deinit_thread.thread = NULL;
> > > > +   }
> > > > +   mutex_unlock(&isp_ctx->lock);
> > > > +
> > > > +   pm_runtime_put_sync(&p1_dev->pdev->dev);
> > >
> > > No need to use the sync variant.
> > >
> >
> > We don't get this point. If we will call pm_runtime_get_sync in
> > mtk_isp_hw_init function, will we need to call
> > pm_runtime_put_sync_autosuspend in mtk_isp_hw_release in next patch?
> > As we know, we should call runtime pm functions in pair.
> >
> 
> My point is that pm_runtime_put_sync() is only needed if one wants the
> runtime count to be decremented after the function returns. Normally
> there is no need to do so and one would call pm_runtime_put(), or if
> autosuspend is used, pm_runtime_put_autosuspend() (note there is no
> "sync" in the name).
> 
> [snip]

Ok, got your point.
We will change to use pm_runtime_put_autosuspend() which has ASYNC flag.

> > > +static void isp_composer_handler(void *data, unsigned int len, void *priv)
> > > > +{
> > > > +   struct mtk_isp_p1_ctx *isp_ctx = (struct mtk_isp_p1_ctx *)priv;
> > > > +   struct isp_p1_device *p1_dev = p1_ctx_to_dev(isp_ctx);
> > > > +   struct device *dev = &p1_dev->pdev->dev;
> > > > +   struct mtk_isp_scp_p1_cmd *ipi_msg;
> > > > +
> > > > +   ipi_msg = (struct mtk_isp_scp_p1_cmd *)data;
> > >
> > > Should we check that len == sizeof(*ipi_msg)? (Or at least >=, if data could
> > > contain some extra bytes at the end.)
> > >
> >
> > The len parameter is the actual sending bytes from SCP to kernel.
> > In the runtime, it is only 6 bytes for isp_ack_info command
> > However, sizeof(*ipi_msg) is large due to struct mtk_isp_scp_p1_cmd is
> > union structure.
> >
> 
> That said we still should check if len is enough to cover the data
> we're accessing below.
> 

Ok, we will add the len checking before accessing the data.

> > > > +
> > > > +   if (ipi_msg->cmd_id != ISP_CMD_ACK)
> > > > +           return;
> > > > +
> > > > +   if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
> > > > +           dev_dbg(dev, "ack frame_num:%d",
> > > > +                   ipi_msg->ack_info.frame_seq_no);
> > > > +           atomic_set(&isp_ctx->composed_frame_id,
> > > > +                      ipi_msg->ack_info.frame_seq_no);
> > >
> > > I suppose we are expecting here that ipi_msg->ack_info.frame_seq_no would be
> > > just isp_ctx->composed_frame_id + 1, right? If not, we probably dropped some
> > > frames and we should handle that somehow.
> > >
> >
> > No, we use isp_ctx->composed_frame_id to save which frame sequence
> > number are composed done in SCP. In new design, we will move this from
> > isp_ctx to p1_dev.
> 
> But we compose the frames in order, don't we? Wouldn't every composed
> frame would be just previous frame ID + 1?
> 
> [snip]

Yes, we compose the frames in order.
At the same time, we already increased "frame ID + 1" in
mtk_isp_req_enqueue() for each new request before sending to SCP for
composing. After receiving the ACK from SCP, we think the frame ID is
composed done and save by isp_ctx->composed_frame_id(v3).

[RFC v3]
void mtk_isp_req_enqueue(struct device *dev, struct media_request *req)
{
	...
	frameparams.frame_seq_no = isp_ctx->frame_seq_no++;

[RFC v4]
void mtk_isp_req_enqueue(struct mtk_cam_dev *cam,
			 struct mtk_cam_dev_request *req)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(cam->dev);

	/* Accumulated frame sequence number */
	req->frame_params.frame_seq_no = ++p1_dev->enqueue_frame_seq_no;

 

> > > > +void isp_composer_hw_init(struct device *dev)
> > > > +{
> > > > +   struct mtk_isp_scp_p1_cmd composer_tx_cmd;
> > > > +   struct isp_p1_device *p1_dev = get_p1_device(dev);
> > > > +   struct mtk_isp_p1_ctx *isp_ctx = &p1_dev->isp_ctx;
> > > > +
> > > > +   memset(&composer_tx_cmd, 0, sizeof(composer_tx_cmd));
> > > > +   composer_tx_cmd.cmd_id = ISP_CMD_INIT;
> > > > +   composer_tx_cmd.frameparam.hw_module = isp_ctx->isp_hw_module;
> > > > +   composer_tx_cmd.frameparam.cq_addr.iova = isp_ctx->scp_mem_iova;
> > > > +   composer_tx_cmd.frameparam.cq_addr.scp_addr = isp_ctx->scp_mem_pa;
> > >
> > > Should we also specify the size of the buffer? Otherwise we could end up
> > > with some undetectable overruns.
> > >
> >
> > The size of SCP composer's memory is fixed to 0x200000.
> > Is it necessary to specify the size of this buffer?
> >
> > #define MTK_ISP_COMPOSER_MEM_SIZE 0x200000
> >
> > ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
> >                         MTK_ISP_COMPOSER_MEM_SIZE, &addr, GFP_KERNEL);
> >
> 
> Okay, but please add a comment saying that this is an implicit
> requirement of the firmware.
> 
> Best regards,
> Tomasz

Ok, we will add comments.

Best regards,


Jungo





^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC, v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-26  7:42                   ` Tomasz Figa
@ 2019-07-26 11:04                     ` Robin Murphy
  2019-07-26 11:59                       ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Robin Murphy @ 2019-07-26 11:04 UTC (permalink / raw)
  To: Tomasz Figa, Christoph Hellwig
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Mauro Carvalho Chehab, Rynn Wu (吳育恩),
	Linux Media Mailing List, srv_heupstream, Rob Herring,
	Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, Jungo Lin, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	ddavenport, Frederic Chen (陳俊元),
	list@263.net:IOMMU DRIVERS, Joerg Roedel, linux-arm-kernel,
	Matthias Brugger

On 26/07/2019 08:42, Tomasz Figa wrote:
> On Fri, Jul 26, 2019 at 4:41 PM Christoph Hellwig <hch@infradead.org> wrote:
>>
>> On Fri, Jul 26, 2019 at 02:15:14PM +0900, Tomasz Figa wrote:
>>> Could you try dma_get_sgtable() with the SCP struct device and then
>>> dma_map_sg() with the P1 struct device?
>>
>> Please don't do that.  dma_get_sgtable is a pretty broken API (see
>> the common near the arm implementation) and we should not add more
>> users of it.  If you want a piece of memory that can be mapped to
>> multiple devices allocate it using alloc_pages and then just map
>> it to each device.
> 
> Thanks for taking a look at this thread.
> 
> Unfortunately that wouldn't work. We have a specific reserved memory
> pool that is the only memory area accessible to one of the devices.
> Any idea how to handle this?

If it's reserved in the sense of being outside struct-page-backed 
"kernel memory", then provided you have a consistent CPU physical 
address it might be reasonable for other devices to access it via 
dma_map_resource().

Robin.

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC, v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-26 11:04                     ` Robin Murphy
@ 2019-07-26 11:59                       ` Jungo Lin
  2019-07-26 14:04                         ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-26 11:59 UTC (permalink / raw)
  To: Robin Murphy
  Cc: Tomasz Figa, Christoph Hellwig, devicetree,
	Sean Cheng (鄭昇弘),
	Mauro Carvalho Chehab, Rynn Wu (吳育恩),
	Linux Media Mailing List, srv_heupstream, Rob Herring,
	Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, Sj Huang, moderated list:ARM/Mediatek SoC support,
	Laurent Pinchart, ddavenport,
	Frederic Chen (陳俊元),
	list@263.net:IOMMU DRIVERS, Joerg Roedel, linux-arm-kernel,
	Matthias Brugger

Hi Robin:

On Fri, 2019-07-26 at 12:04 +0100, Robin Murphy wrote:
> On 26/07/2019 08:42, Tomasz Figa wrote:
> > On Fri, Jul 26, 2019 at 4:41 PM Christoph Hellwig <hch@infradead.org> wrote:
> >>
> >> On Fri, Jul 26, 2019 at 02:15:14PM +0900, Tomasz Figa wrote:
> >>> Could you try dma_get_sgtable() with the SCP struct device and then
> >>> dma_map_sg() with the P1 struct device?
> >>
> >> Please don't do that.  dma_get_sgtable is a pretty broken API (see
> >> the common near the arm implementation) and we should not add more
> >> users of it.  If you want a piece of memory that can be mapped to
> >> multiple devices allocate it using alloc_pages and then just map
> >> it to each device.
> > 
> > Thanks for taking a look at this thread.
> > 
> > Unfortunately that wouldn't work. We have a specific reserved memory
> > pool that is the only memory area accessible to one of the devices.
> > Any idea how to handle this?
> 
> If it's reserved in the sense of being outside struct-page-backed 
> "kernel memory", then provided you have a consistent CPU physical 
> address it might be reasonable for other devices to access it via 
> dma_map_resource().
> 
> Robin.

Thank you for your suggestion.

After revising to use dma_map_resource(), it is worked. Below is the
current implementation. Pleas kindly help us to check if there is any
misunderstanding.

#define MTK_ISP_COMPOSER_MEM_SIZE		0x200000

	/*
	 * Allocate coherent reserved memory for SCP firmware usage.
	 * The size of SCP composer's memory is fixed to 0x200000
	 * for the requirement of firmware.
	 */
	ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
				 MTK_ISP_COMPOSER_MEM_SIZE, &addr, GFP_KERNEL);
	if (!ptr) {
		dev_err(dev, "failed to allocate compose memory\n");
		return -ENOMEM;
	}
	p1_dev->composer_scp_addr = addr;
	p1_dev->composer_virt_addr = ptr;
	dev_dbg(dev, "scp addr:%pad va:%pK\n", &addr, ptr);

	/*
	 * This reserved memory is also be used by ISP P1 HW.
	 * Need to get iova address for ISP P1 DMA.
	 */
	addr = dma_map_resource(dev, addr, MTK_ISP_COMPOSER_MEM_SIZE,
				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
	if (dma_mapping_error(dev, addr)) {
		dev_err(dev, "Failed to map scp iova\n");
		ret = -ENOMEM;
		goto fail_free_mem;
	}
	p1_dev->composer_iova = addr;
	dev_info(dev, "scp iova addr:%pad\n", &addr);

Moreover, appropriate Tomasz & Christoph's help on this issue.

Best regards,

Jungo


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC, v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device
  2019-07-26 11:59                       ` Jungo Lin
@ 2019-07-26 14:04                         ` Tomasz Figa
  0 siblings, 0 replies; 45+ messages in thread
From: Tomasz Figa @ 2019-07-26 14:04 UTC (permalink / raw)
  To: Jungo Lin, Robin Murphy
  Cc: Christoph Hellwig, devicetree,
	Sean Cheng (鄭昇弘),
	Mauro Carvalho Chehab, Rynn Wu (吳育恩),
	Linux Media Mailing List, srv_heupstream, Rob Herring,
	Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, Sj Huang, moderated list:ARM/Mediatek SoC support,
	Laurent Pinchart, ddavenport,
	Frederic Chen (陳俊元),
	list@263.net:IOMMU DRIVERS, Joerg Roedel,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Matthias Brugger

On Fri, Jul 26, 2019 at 8:59 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi Robin:
>
> On Fri, 2019-07-26 at 12:04 +0100, Robin Murphy wrote:
> > On 26/07/2019 08:42, Tomasz Figa wrote:
> > > On Fri, Jul 26, 2019 at 4:41 PM Christoph Hellwig <hch@infradead.org> wrote:
> > >>
> > >> On Fri, Jul 26, 2019 at 02:15:14PM +0900, Tomasz Figa wrote:
> > >>> Could you try dma_get_sgtable() with the SCP struct device and then
> > >>> dma_map_sg() with the P1 struct device?
> > >>
> > >> Please don't do that.  dma_get_sgtable is a pretty broken API (see
> > >> the common near the arm implementation) and we should not add more
> > >> users of it.  If you want a piece of memory that can be mapped to
> > >> multiple devices allocate it using alloc_pages and then just map
> > >> it to each device.
> > >
> > > Thanks for taking a look at this thread.
> > >
> > > Unfortunately that wouldn't work. We have a specific reserved memory
> > > pool that is the only memory area accessible to one of the devices.
> > > Any idea how to handle this?
> >
> > If it's reserved in the sense of being outside struct-page-backed
> > "kernel memory", then provided you have a consistent CPU physical
> > address it might be reasonable for other devices to access it via
> > dma_map_resource().
> >
> > Robin.
>
> Thank you for your suggestion.
>
> After revising to use dma_map_resource(), it is worked. Below is the
> current implementation. Pleas kindly help us to check if there is any
> misunderstanding.
>
> #define MTK_ISP_COMPOSER_MEM_SIZE               0x200000
>
>         /*
>          * Allocate coherent reserved memory for SCP firmware usage.
>          * The size of SCP composer's memory is fixed to 0x200000
>          * for the requirement of firmware.
>          */
>         ptr = dma_alloc_coherent(p1_dev->cam_dev.smem_dev,
>                                  MTK_ISP_COMPOSER_MEM_SIZE, &addr, GFP_KERNEL);
>         if (!ptr) {
>                 dev_err(dev, "failed to allocate compose memory\n");
>                 return -ENOMEM;
>         }
>         p1_dev->composer_scp_addr = addr;
>         p1_dev->composer_virt_addr = ptr;
>         dev_dbg(dev, "scp addr:%pad va:%pK\n", &addr, ptr);
>
>         /*
>          * This reserved memory is also be used by ISP P1 HW.
>          * Need to get iova address for ISP P1 DMA.
>          */
>         addr = dma_map_resource(dev, addr, MTK_ISP_COMPOSER_MEM_SIZE,
>                                 DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);

This is still incorrect, because addr is a DMA address, but the second
argument to dma_map_resource() is a physical address.

>         if (dma_mapping_error(dev, addr)) {
>                 dev_err(dev, "Failed to map scp iova\n");
>                 ret = -ENOMEM;
>                 goto fail_free_mem;
>         }
>         p1_dev->composer_iova = addr;
>         dev_info(dev, "scp iova addr:%pad\n", &addr);
>
> Moreover, appropriate Tomasz & Christoph's help on this issue.

Robin, the memory is specified using the reserved-memory DT binding
and managed by the coherent DMA pool framework. We can allocate from
it using dma_alloc_coherent(), which gives us a DMA address, not CPU
physial address (although in practice on this platform they are equal
numerically).

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-26  5:49           ` Tomasz Figa
@ 2019-07-29  1:18             ` Jungo Lin
  2019-07-29 10:04               ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-29  1:18 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	ddavenport, Sj Huang, moderated list:ARM/Mediatek SoC support,
	Laurent Pinchart, Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi, Tomasz:

On Fri, 2019-07-26 at 14:49 +0900, Tomasz Figa wrote:
> On Wed, Jul 24, 2019 at 1:31 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> >
> > Hi, Tomasz:
> >
> > On Tue, 2019-07-23 at 19:21 +0900, Tomasz Figa wrote:
> > > Hi Jungo,
> > >
> > > On Thu, Jul 18, 2019 at 1:39 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > >
> > > > Hi, Tomasz:
> > > >
> > > > On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> > > > > Hi Jungo,
> > > > >
> > > > > On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
> > > [snip]
> > > > > > +static void mtk_cam_req_try_isp_queue(struct mtk_cam_dev *cam_dev,
> > > > > > +                                 struct media_request *new_req)
> > > > > > +{
> > > > > > +   struct mtk_cam_dev_request *req, *req_safe, *cam_dev_req;
> > > > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > > > +
> > > > > > +   dev_dbg(dev, "%s new req:%d", __func__, !new_req);
> > > > > > +
> > > > > > +   if (!cam_dev->streaming) {
> > > > > > +           cam_dev_req = mtk_cam_req_to_dev_req(new_req);
> > > > > > +           spin_lock(&cam_dev->req_lock);
> > > > > > +           list_add_tail(&cam_dev_req->list, &cam_dev->req_list);
> > > > > > +           spin_unlock(&cam_dev->req_lock);
> > > > > > +           dev_dbg(dev, "%s: stream off, no ISP enqueue\n", __func__);
> > > > > > +           return;
> > > > > > +   }
> > > > > > +
> > > > > > +   /* Normal enqueue flow */
> > > > > > +   if (new_req) {
> > > > > > +           mtk_isp_req_enqueue(dev, new_req);
> > > > > > +           return;
> > > > > > +   }
> > > > > > +
> > > > > > +   /* Flush all media requests wehen first stream on */
> > > > > > +   list_for_each_entry_safe(req, req_safe, &cam_dev->req_list, list) {
> > > > > > +           list_del(&req->list);
> > > > > > +           mtk_isp_req_enqueue(dev, &req->req);
> > > > > > +   }
> > > > > > +}
> > > > >
> > > > > This will have to be redone, as per the other suggestions, but generally one
> > > > > would have a function that tries to queue as much as possible from a list to
> > > > > the hardware and another function that adds a request to the list and calls
> > > > > the first function.
> > > > >
> > > >
> > > > We revised this function as below.
> > > > First to check the en-queue conditions:
> > > > a. stream on
> > > > b. The composer buffers in SCP are 3, so we only could has 3 jobs
> > > > at the same time.
> > > >
> > > >
> > > > Second, try to en-queue the frames in the pending job if possible and
> > > > move them into running job list if possible.
> > > >
> > > > The request has been inserted into pending job in mtk_cam_req_validate
> > > > which is used to validate media_request.
> > >
> > > Thanks for replying to each of the comments, that's very helpful.
> > > Snipped out the parts that I agreed with.
> > >
> > > Please note that req_validate is not supposed to change any driver
> > > state. It's only supposed to validate the request. req_queue is the
> > > right callback to insert the request into some internal driver
> > > bookkeeping structures.
> > >
> >
> > Yes, in req_validate function, we don't change any driver state.
> > Below is the function's implementation.
> >
> > a. Call vb2_request_validate(req) to verify media request.
> > b. Update the buffer internal structure buffer.
> > c. Insert the request into pending_job_list to prepare en-queue.
> >
> 
> Adding to a list is changing driver state. The callback must not
> modify anything else than the request itself.
> 
> Queuing to driver's list should happen in req_queue instead.
> 
> [snip]

Ok, got your point. We will move these implementation to .req_queue.

static const struct media_device_ops mtk_cam_media_ops = {
	.link_notify = v4l2_pipeline_link_notify,
	.req_alloc = mtk_cam_req_alloc,
	.req_free = mtk_cam_req_free,
	.req_validate = vb2_request_validate,
	.req_queue = mtk_cam_req_queue,
};

static void mtk_cam_req_queue(struct media_request *req)
{
	struct mtk_cam_dev_request *cam_req = mtk_cam_req_to_dev_req(req);
	struct mtk_cam_dev *cam = container_of(req->mdev, struct mtk_cam_dev,
					       media_dev);

	atomic_set(&cam_req->buf_count, vb2_request_buffer_cnt(req));

	/* add to pending job list */
	spin_lock_irq(&cam->pending_job_lock);
	list_add_tail(&cam_req->list, &cam->pending_job_list);
	spin_unlock_irq(&cam->pending_job_lock);

	vb2_request_queue(req);
}

> > > >
> > > > void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam_dev)
> > > > {
> > > >         struct mtk_cam_dev_request *req, *req_prev;
> > > >         struct list_head enqueue_job_list;
> > > >         int buffer_cnt = atomic_read(&cam_dev->running_job_count);
> > > >         unsigned long flags;
> > > >
> > > >         if (!cam_dev->streaming ||
> > > >             buffer_cnt >= MTK_ISP_MAX_RUNNING_JOBS) {
> > >
> > > Do we have a guarantee that cam_dev->running_job_count doesn't
> > > decrement between the atomic_read() above and this line?
> > >
> >
> > Ok, we will use cam->pending_job_lock to protect
> > cam_dev->running_job_count access. Below is the revised version.
> >
> > void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam)
> > {
> >         struct mtk_cam_dev_request *req, *req_prev;
> >         unsigned long flags;
> >
> >         if (!cam->streaming) {
> >                 dev_dbg(cam->dev, "stream is off\n");
> >                 return;
> >         }
> >
> >         spin_lock_irqsave(&cam->pending_job_lock, flags);
> >         if (atomic_read(&cam->running_job_count) >= MTK_ISP_MAX_RUNNING_JOBS) {
> 
> If we use a spin_lock to protect the counter, perhaps we don't need
> the atomic type anymore?
> 

Ok, we will remove atomic type usage.


> >                 dev_dbg(cam->dev, "jobs are full\n");
> >                 spin_unlock_irqrestore(&cam->pending_job_lock, flags);
> >                 return;
> >         }
> >         list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list) {
> 
> Could we instead check the counter here and break if it's >=
> MTK_ISP_MAX_RUNNING_JOBS?
> Then we could increment it here too to simplify the code.
> 

Thanks for your advice.
We simplified this function as below:

void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam)
{
	struct mtk_cam_dev_request *req, *req_prev;
	unsigned long flags;

	if (!cam->streaming) {
		dev_dbg(cam->dev, "stream is off\n");
		return;
	}

	spin_lock_irq(&cam->pending_job_lock);
	spin_lock_irqsave(&cam->running_job_lock, flags);
	list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list) {
		if (cam->running_job_count >= MTK_ISP_MAX_RUNNING_JOBS) {
			dev_dbg(cam->dev, "jobs are full\n");
			break;
		}
		cam->running_job_count++;
		list_del(&req->list);
		list_add_tail(&req->list, &cam->running_job_list);
		mtk_isp_req_enqueue(cam, req);
	}
	spin_unlock_irqrestore(&cam->running_job_lock, flags);
	spin_unlock_irq(&cam->pending_job_lock);
}

> >                 list_del(&req->list);
> >                 spin_lock_irqsave(&cam->running_job_lock, flags);
> >                 list_add_tail(&req->list, &cam->running_job_list);
> >                 mtk_isp_req_enqueue(cam, req);
> >                 spin_unlock_irqrestore(&cam->running_job_lock, flags);
> >                 if (atomic_inc_return(&cam->running_job_count) >=
> >                         MTK_ISP_MAX_RUNNING_JOBS)
> >                         break;
> 
> With the above suggestion, this if block would go away.
> 
> [snip]

Ditto.

> > > >                 mtk_isp_req_enqueue(cam_dev, req);
> > > >         }
> > > > }
> > > >
> > > [snip]
> > > > > > +   stride = DIV_ROUND_UP(stride * pixel_byte, 8);
> > > > > > +
> > > > > > +   if (pix_fmt == V4L2_PIX_FMT_MTISP_F10)
> > > > > > +           stride = ALIGN(stride, 4);
> > > > >
> > > > > Is it expected that only the F10 format needs this alignment?
> > > > >
> > > >
> > > > yes, if the pixel bits of image format is 10, the byte alignment of bpl
> > > > should be 4. Otherwise, it is 8. We will revise this and add more
> > > > comments.
> > >
> > > That means that the B10 format also needs the extra alignment, as
> > > opposed to what the original code did, right?
> > >
> >
> > Sorry for short code snippet.
> > This alignment checking is only applied to F10, no B10.
> > If you like to check the full function, you could check this in this
> > link[1].
> >
> > static void cal_image_pix_mp(struct mtk_cam_dev *cam, unsigned int
> > node_id,
> >                              struct v4l2_pix_format_mplane *mp)
> > {
> >         unsigned int bpl, ppl;
> >         unsigned int pixel_bits = get_pixel_bits(mp->pixelformat);
> >         unsigned int width = mp->width;
> >
> >         if (node_id == MTK_CAM_P1_MAIN_STREAM_OUT) {
> >                 /* bayer encoding format & 2 bytes alignment */
> >                 bpl = ALIGN(DIV_ROUND_UP(width * pixel_bits, 8), 2);
> >         } else if (node_id == MTK_CAM_P1_PACKED_BIN_OUT) {
> >                 /*
> >                  * The FULL-G encoding format
> >                  * 1 G component per pixel
> >                  * 1 R component per 4 pixel
> >                  * 1 B component per 4 pixel
> >                  * Total 4G/1R/1B in 4 pixel (pixel per line:ppl)
> >                  */
> >                 ppl = DIV_ROUND_UP(width * 6, 4);
> >                 bpl = DIV_ROUND_UP(ppl * pixel_bits, 8);
> >
> >                 /* 4 bytes alignment for 10 bit & others are 8 bytes */
> >                 if (pixel_bits == 10)
> >                         bpl = ALIGN(bpl, 4);
> >                 else
> >                         bpl = ALIGN(bpl, 8);
> >         }
> >
> > [1]
> > https://crrev.com/c/1712885/2/drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c#303
> >
> 
> Got it, thanks!
> 
> [snip]
> > > > > > +
> > > > > > +static struct v4l2_subdev *
> > > > > > +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> > > > > > +{
> > > > > > +   struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> > > > > > +   struct media_entity *entity;
> > > > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > > > +   struct v4l2_subdev *sensor;
> > > > >
> > > > > This variable would be unitialized if there is no streaming sensor. Was
> > > > > there no compiler warning generated for this?
> > > > >
> > > >
> > > > No, there is no compiler warning.
> > > > But, we will assign sensor to NULL to avoid unnecessary compiler warning
> > > > with different compiler options.
> > > >
> > >
> > > Thanks. It would be useful if you could check why the compiler you're
> > > using doesn't show a warning here. We might be missing other
> > > uninitialized variables.
> > >
> >
> > We will feedback to your project team to check the possible reason about
> > compiler warning issue.
> >
> 
> Do you mean that it was the Clang toolchain used on Chromium OS (e.g.
> emerge chromeos-kernel-4_19)?

> [snip]

Yes, I checked this comment in the Chromium OS build environment.
But, I think I have made the mistake here. I need to check the build
status in the Mediatek's kernel upstream environment. I will pay
attention in next path set upstream.

> > > > > > +
> > > > > > +   dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > > > > > +           __func__,
> > > > > > +           node->id,
> > > > > > +           buf->vbb.request_fd,
> > > > > > +           buf->vbb.vb2_buf.index);
> > > > > > +
> > > > > > +   /* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > > > > > +   if (vb->vb2_queue->uses_requests)
> > > > > > +           return;
> > > > >
> > > > > I'd suggest removing non-request support from this driver. Even if we end up
> > > > > with a need to provide compatibility for non-request mode, then it should be
> > > > > built on top of the requests mode, so that the driver itself doesn't have to
> > > > > deal with two modes.
> > > > >
> > > >
> > > > The purpose of non-request function in this driver is needed by
> > > > our camera middle-ware design. It needs 3A statistics buffers before
> > > > image buffers en-queue. So we need to en-queue 3A statistics with
> > > > non-request mode in this driver. After MW got the 3A statistics data, it
> > > > will en-queue the images, tuning buffer and other meta buffers with
> > > > request mode. Based on this requirement, do you have any suggestion?
> > > > For upstream driver, should we only consider request mode?
> > > >
> > >
> > > Where does that requirement come from? Why the timing of queuing of
> > > the buffers to the driver is important?
> > >
> > > [snip]
> >
> > Basically, this requirement comes from our internal camera
> > middle-ware/3A hal in user space. Since this is not generic requirement,
> > we will follow your original suggestion to keep the request mode only
> > and remove other non-request design in other files. For upstream driver,
> > it should support request mode only.
> >
> 
> Note that Chromium OS will use the "upstream driver" and we don't want
> to diverge, so please make the userspace also use only requests. I
> don't see a reason why there would be any need to submit any buffers
> outside of a request.
> 
> [snip]

Ok, I have raised your concern to our colleagues and let him to discuss
with you in another communication channel. 

> > > > > > +static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
> > > > > > +{
> > > > > > +   struct mtk_cam_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
> > > > > > +
> > > > > > +   v4l2_ctrl_request_complete(vb->req_obj.req,
> > > > > > +                              dev->v4l2_dev.ctrl_handler);
> > > > >
> > > > > This would end up being called multiple times, once for each video node.
> > > > > Instead, this should be called explicitly by the driver when it completed
> > > > > the request - perhaps in the frame completion handler?
> > > > >
> > > > > With that, we probably wouldn't even need this callback.
> > > > >
> > > >
> > > > First, if we don't implement this callback function, we will receive
> > > > kernel warning as below.
> > > >
> > > > https://elixir.bootlin.com/linux/latest/source/drivers/media/common/videobuf2/videobuf2-v4l2.c#L420
> > > >
> > > > Second, this function is only be called in __vb2_queue_cancel function.
> > > > Moreover, we will remove cam_dev->v4l2_dev.ctrl_handler in next patch.
> > > > So could we just implement dummy empty function?
> > > >
> > > >  * @buf_request_complete: a buffer that was never queued to the driver
> > > > but is
> > > >  *                      associated with a queued request was canceled.
> > > >  *                      The driver will have to mark associated objects in the
> > > >  *                      request as completed; required if requests are
> > > >  *                      supported.
> > > >
> > >
> > > Good catch, thanks.
> > >
> > > Sounds like we may indeed need to implement this callback. In
> > > particular, we may need to remove the request that the buffer was
> > > associated with from the driver queue and return the other buffers
> > > associated to it with an error state. This should be similar to
> > > handling a request failure.
> > > [snip]
> >
> > Before calling this callback function, the VB2's stop_streaming has been
> > called. Normally, we will return the buffers belonged to this vb2 queu
> > with error state. On other hand, only if the state of request is
> > MEDIA_REQUEST_STATE_QUEUED, the buf_request_complete will be called in
> > __vb2_queue_cancel function. It hints this media request has been
> > validated and inserted into our driver's pending_job_list or
> > running_job_list. So we will call mtk_cam_dev_req_cleanup() remove these
> > requests from driver's list when streaming is off. Since we have no
> > v4l2_ctrl, do we need to do the above things which is already handled in
> > mtk_cam_vb2_stop_streaming function? Maybe is this callback function
> > only designed for v4l2_ctrl_request_complete usage?
> 
> Are you sure that this callback can be only called after
> stop_streaming? Also wouldn't that be after stop_streaming only on 1
> queue? The other queues could still remain streaming, but we still
> have to return corresponding buffers I believe.
> 
> Hans, could you clarify what exactly this callback is supposed to do?
> 

Ok, we will look forward Hans' comments on this.

> >
> > static void mtk_cam_dev_req_cleanup(struct mtk_cam_dev *cam)
> > {
> >         struct mtk_cam_dev_request *req, *req_prev;
> >         unsigned long flags;
> >
> >         dev_dbg(cam->dev, "%s\n", __func__);
> >
> >         spin_lock_irqsave(&cam->pending_job_lock, flags);
> >         list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list)
> >                 list_del(&req->list);
> >         spin_unlock_irqrestore(&cam->pending_job_lock, flags);
> >
> >         spin_lock_irqsave(&cam->running_job_lock, flags);
> >         list_for_each_entry_safe(req, req_prev, &cam->running_job_list, list)
> >                 list_del(&req->list);
> >         spin_unlock_irqrestore(&cam->running_job_lock, flags);
> > }
> >
> > static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
> > {
> >         struct mtk_cam_dev *cam = vb2_get_drv_priv(vq);
> >         struct mtk_cam_video_device *node = mtk_cam_vbq_to_vdev(vq);
> >         struct device *dev = cam->dev;
> >
> >         dev_dbg(dev, "%s node:%d count info:%d", __func__,
> >                 node->id, atomic_read(&cam->stream_count));
> >
> >         mutex_lock(&cam->op_lock);
> >         if (atomic_read(&cam->stream_count) == cam->enabled_count)
> >                 if (v4l2_subdev_call(&cam->subdev, video, s_stream, 0))
> >                         dev_err(dev, "failed to stop streaming\n");
> >
> >         mtk_cam_vb2_return_all_buffers(cam, node, VB2_BUF_STATE_ERROR);
> >
> >         /* Check the first node to stream-off */
> >         if (!atomic_dec_and_test(&cam->stream_count)) {
> >                 mutex_unlock(&cam->op_lock);
> >                 return;
> >         }
> >         mutex_unlock(&cam->op_lock);
> >
> >         mtk_cam_dev_req_cleanup(cam);
> >         media_pipeline_stop(&node->vdev.entity);
> > }
> 
> [keeping the context for Hans]
> 
> Best regards,
> Tomasz
> 
> _______________________________________________
> Linux-mediatek mailing list
> Linux-mediatek@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-mediatek


Best regards,


Jungo




^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-29  1:18             ` Jungo Lin
@ 2019-07-29 10:04               ` Tomasz Figa
  2019-07-30  1:44                 ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-07-29 10:04 UTC (permalink / raw)
  To: Jungo Lin
  Cc: Hans Verkuil, devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	ddavenport, Sj Huang, moderated list:ARM/Mediatek SoC support,
	Laurent Pinchart, Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

On Mon, Jul 29, 2019 at 10:18 AM Jungo Lin <jungo.lin@mediatek.com> wrote:
> On Fri, 2019-07-26 at 14:49 +0900, Tomasz Figa wrote:
> > On Wed, Jul 24, 2019 at 1:31 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > On Tue, 2019-07-23 at 19:21 +0900, Tomasz Figa wrote:
> > > > On Thu, Jul 18, 2019 at 1:39 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > > > On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> > > > > > On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
[snip]
> > >                 dev_dbg(cam->dev, "jobs are full\n");
> > >                 spin_unlock_irqrestore(&cam->pending_job_lock, flags);
> > >                 return;
> > >         }
> > >         list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list) {
> >
> > Could we instead check the counter here and break if it's >=
> > MTK_ISP_MAX_RUNNING_JOBS?
> > Then we could increment it here too to simplify the code.
> >
>
> Thanks for your advice.
> We simplified this function as below:
>
> void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam)
> {
>         struct mtk_cam_dev_request *req, *req_prev;
>         unsigned long flags;
>
>         if (!cam->streaming) {
>                 dev_dbg(cam->dev, "stream is off\n");
>                 return;
>         }
>
>         spin_lock_irq(&cam->pending_job_lock);
>         spin_lock_irqsave(&cam->running_job_lock, flags);

Having the inner call spin_lock_irqsave() doesn't really do anything
useful, because the outer spin_lock_irq() disables the IRQs and flags
would always have the IRQ disabled state. Please use irqsave for the
outer call.

[snip]
> > > > > > > +
> > > > > > > +static struct v4l2_subdev *
> > > > > > > +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> > > > > > > +{
> > > > > > > +   struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> > > > > > > +   struct media_entity *entity;
> > > > > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > > > > +   struct v4l2_subdev *sensor;
> > > > > >
> > > > > > This variable would be unitialized if there is no streaming sensor. Was
> > > > > > there no compiler warning generated for this?
> > > > > >
> > > > >
> > > > > No, there is no compiler warning.
> > > > > But, we will assign sensor to NULL to avoid unnecessary compiler warning
> > > > > with different compiler options.
> > > > >
> > > >
> > > > Thanks. It would be useful if you could check why the compiler you're
> > > > using doesn't show a warning here. We might be missing other
> > > > uninitialized variables.
> > > >
> > >
> > > We will feedback to your project team to check the possible reason about
> > > compiler warning issue.
> > >
> >
> > Do you mean that it was the Clang toolchain used on Chromium OS (e.g.
> > emerge chromeos-kernel-4_19)?
>
> > [snip]
>
> Yes, I checked this comment in the Chromium OS build environment.
> But, I think I have made the mistake here. I need to check the build
> status in the Mediatek's kernel upstream environment. I will pay
> attention in next path set upstream.
>

Thanks a lot. I will recheck this in the Chromium OS toolchain too.

> > > > > > > +
> > > > > > > +   dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > > > > > > +           __func__,
> > > > > > > +           node->id,
> > > > > > > +           buf->vbb.request_fd,
> > > > > > > +           buf->vbb.vb2_buf.index);
> > > > > > > +
> > > > > > > +   /* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > > > > > > +   if (vb->vb2_queue->uses_requests)
> > > > > > > +           return;
> > > > > >
> > > > > > I'd suggest removing non-request support from this driver. Even if we end up
> > > > > > with a need to provide compatibility for non-request mode, then it should be
> > > > > > built on top of the requests mode, so that the driver itself doesn't have to
> > > > > > deal with two modes.
> > > > > >
> > > > >
> > > > > The purpose of non-request function in this driver is needed by
> > > > > our camera middle-ware design. It needs 3A statistics buffers before
> > > > > image buffers en-queue. So we need to en-queue 3A statistics with
> > > > > non-request mode in this driver. After MW got the 3A statistics data, it
> > > > > will en-queue the images, tuning buffer and other meta buffers with
> > > > > request mode. Based on this requirement, do you have any suggestion?
> > > > > For upstream driver, should we only consider request mode?
> > > > >
> > > >
> > > > Where does that requirement come from? Why the timing of queuing of
> > > > the buffers to the driver is important?
> > > >
> > > > [snip]
> > >
> > > Basically, this requirement comes from our internal camera
> > > middle-ware/3A hal in user space. Since this is not generic requirement,
> > > we will follow your original suggestion to keep the request mode only
> > > and remove other non-request design in other files. For upstream driver,
> > > it should support request mode only.
> > >
> >
> > Note that Chromium OS will use the "upstream driver" and we don't want
> > to diverge, so please make the userspace also use only requests. I
> > don't see a reason why there would be any need to submit any buffers
> > outside of a request.
> >
> > [snip]
>
> Ok, I have raised your concern to our colleagues and let him to discuss
> with you in another communication channel.
>

Thanks!

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-29 10:04               ` Tomasz Figa
@ 2019-07-30  1:44                 ` Jungo Lin
  2019-08-05  9:59                   ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-07-30  1:44 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	ddavenport, Sj Huang, moderated list:ARM/Mediatek SoC support,
	Laurent Pinchart, Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

On Mon, 2019-07-29 at 19:04 +0900, Tomasz Figa wrote:
> On Mon, Jul 29, 2019 at 10:18 AM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > On Fri, 2019-07-26 at 14:49 +0900, Tomasz Figa wrote:
> > > On Wed, Jul 24, 2019 at 1:31 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > > On Tue, 2019-07-23 at 19:21 +0900, Tomasz Figa wrote:
> > > > > On Thu, Jul 18, 2019 at 1:39 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > > > > On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> > > > > > > On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
> [snip]
> > > >                 dev_dbg(cam->dev, "jobs are full\n");
> > > >                 spin_unlock_irqrestore(&cam->pending_job_lock, flags);
> > > >                 return;
> > > >         }
> > > >         list_for_each_entry_safe(req, req_prev, &cam->pending_job_list, list) {
> > >
> > > Could we instead check the counter here and break if it's >=
> > > MTK_ISP_MAX_RUNNING_JOBS?
> > > Then we could increment it here too to simplify the code.
> > >
> >
> > Thanks for your advice.
> > We simplified this function as below:
> >
> > void mtk_cam_dev_req_try_queue(struct mtk_cam_dev *cam)
> > {
> >         struct mtk_cam_dev_request *req, *req_prev;
> >         unsigned long flags;
> >
> >         if (!cam->streaming) {
> >                 dev_dbg(cam->dev, "stream is off\n");
> >                 return;
> >         }
> >
> >         spin_lock_irq(&cam->pending_job_lock);
> >         spin_lock_irqsave(&cam->running_job_lock, flags);
> 
> Having the inner call spin_lock_irqsave() doesn't really do anything
> useful, because the outer spin_lock_irq() disables the IRQs and flags
> would always have the IRQ disabled state. Please use irqsave for the
> outer call.
> 
> [snip]

Thanks for your comment.
This is a bug which triggers one kernel warning about wrong ISR state as
you said. We have fixed it.

> > > > > > > > +
> > > > > > > > +static struct v4l2_subdev *
> > > > > > > > +mtk_cam_cio_get_active_sensor(struct mtk_cam_dev *cam_dev)
> > > > > > > > +{
> > > > > > > > +   struct media_device *mdev = cam_dev->seninf->entity.graph_obj.mdev;
> > > > > > > > +   struct media_entity *entity;
> > > > > > > > +   struct device *dev = &cam_dev->pdev->dev;
> > > > > > > > +   struct v4l2_subdev *sensor;
> > > > > > >
> > > > > > > This variable would be unitialized if there is no streaming sensor. Was
> > > > > > > there no compiler warning generated for this?
> > > > > > >
> > > > > >
> > > > > > No, there is no compiler warning.
> > > > > > But, we will assign sensor to NULL to avoid unnecessary compiler warning
> > > > > > with different compiler options.
> > > > > >
> > > > >
> > > > > Thanks. It would be useful if you could check why the compiler you're
> > > > > using doesn't show a warning here. We might be missing other
> > > > > uninitialized variables.
> > > > >
> > > >
> > > > We will feedback to your project team to check the possible reason about
> > > > compiler warning issue.
> > > >
> > >
> > > Do you mean that it was the Clang toolchain used on Chromium OS (e.g.
> > > emerge chromeos-kernel-4_19)?
> >
> > > [snip]
> >
> > Yes, I checked this comment in the Chromium OS build environment.
> > But, I think I have made the mistake here. I need to check the build
> > status in the Mediatek's kernel upstream environment. I will pay
> > attention in next path set upstream.
> >
> 
> Thanks a lot. I will recheck this in the Chromium OS toolchain too.
> 

For these complier warnings, we have fixed them in Mediatek upstream
environment[1]. In this build environment, we could observe some
comelier warnings which are not generated by Chromium OS toolchain.

[1]
toolchain/aarch64/usr/bin/aarch64-poky-linux

> > > > > > > > +
> > > > > > > > +   dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > > > > > > > +           __func__,
> > > > > > > > +           node->id,
> > > > > > > > +           buf->vbb.request_fd,
> > > > > > > > +           buf->vbb.vb2_buf.index);
> > > > > > > > +
> > > > > > > > +   /* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > > > > > > > +   if (vb->vb2_queue->uses_requests)
> > > > > > > > +           return;
> > > > > > >
> > > > > > > I'd suggest removing non-request support from this driver. Even if we end up
> > > > > > > with a need to provide compatibility for non-request mode, then it should be
> > > > > > > built on top of the requests mode, so that the driver itself doesn't have to
> > > > > > > deal with two modes.
> > > > > > >
> > > > > >
> > > > > > The purpose of non-request function in this driver is needed by
> > > > > > our camera middle-ware design. It needs 3A statistics buffers before
> > > > > > image buffers en-queue. So we need to en-queue 3A statistics with
> > > > > > non-request mode in this driver. After MW got the 3A statistics data, it
> > > > > > will en-queue the images, tuning buffer and other meta buffers with
> > > > > > request mode. Based on this requirement, do you have any suggestion?
> > > > > > For upstream driver, should we only consider request mode?
> > > > > >
> > > > >
> > > > > Where does that requirement come from? Why the timing of queuing of
> > > > > the buffers to the driver is important?
> > > > >
> > > > > [snip]
> > > >
> > > > Basically, this requirement comes from our internal camera
> > > > middle-ware/3A hal in user space. Since this is not generic requirement,
> > > > we will follow your original suggestion to keep the request mode only
> > > > and remove other non-request design in other files. For upstream driver,
> > > > it should support request mode only.
> > > >
> > >
> > > Note that Chromium OS will use the "upstream driver" and we don't want
> > > to diverge, so please make the userspace also use only requests. I
> > > don't see a reason why there would be any need to submit any buffers
> > > outside of a request.
> > >
> > > [snip]
> >
> > Ok, I have raised your concern to our colleagues and let him to discuss
> > with you in another communication channel.
> >
> 
> Thanks!
> 
> Best regards,
> Tomasz

Our colleague is preparing material to explain the our 3A/MW design. If
he is ready, he will discuss this with you.

In the original plan, we will deliver P1 v4 patch set tomorrow (31th
Jul.). But, there are some comments waiting for other experts' input.
Do you suggest it is better to resolve all comments before v4 patch set
submitting or continue to discuss these comments on v4?

Thanks,


Jungo


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions
  2019-07-30  1:44                 ` Jungo Lin
@ 2019-08-05  9:59                   ` Tomasz Figa
  0 siblings, 0 replies; 45+ messages in thread
From: Tomasz Figa @ 2019-08-05  9:59 UTC (permalink / raw)
  To: Jungo Lin
  Cc: Hans Verkuil, devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	ddavenport, Sj Huang, moderated list:ARM/Mediatek SoC support,
	Laurent Pinchart, Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi Jungo,

On Tue, Jul 30, 2019 at 10:45 AM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> On Mon, 2019-07-29 at 19:04 +0900, Tomasz Figa wrote:
> > On Mon, Jul 29, 2019 at 10:18 AM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > On Fri, 2019-07-26 at 14:49 +0900, Tomasz Figa wrote:
> > > > On Wed, Jul 24, 2019 at 1:31 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > > > On Tue, 2019-07-23 at 19:21 +0900, Tomasz Figa wrote:
> > > > > > On Thu, Jul 18, 2019 at 1:39 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > > > > > On Wed, 2019-07-10 at 18:54 +0900, Tomasz Figa wrote:
> > > > > > > > On Tue, Jun 11, 2019 at 11:53:41AM +0800, Jungo Lin wrote:
[snip]
> > > > > > > > > +
> > > > > > > > > +   dev_dbg(dev, "%s: node:%d fd:%d idx:%d\n",
> > > > > > > > > +           __func__,
> > > > > > > > > +           node->id,
> > > > > > > > > +           buf->vbb.request_fd,
> > > > > > > > > +           buf->vbb.vb2_buf.index);
> > > > > > > > > +
> > > > > > > > > +   /* For request buffers en-queue, handled in mtk_cam_req_try_queue */
> > > > > > > > > +   if (vb->vb2_queue->uses_requests)
> > > > > > > > > +           return;
> > > > > > > >
> > > > > > > > I'd suggest removing non-request support from this driver. Even if we end up
> > > > > > > > with a need to provide compatibility for non-request mode, then it should be
> > > > > > > > built on top of the requests mode, so that the driver itself doesn't have to
> > > > > > > > deal with two modes.
> > > > > > > >
> > > > > > >
> > > > > > > The purpose of non-request function in this driver is needed by
> > > > > > > our camera middle-ware design. It needs 3A statistics buffers before
> > > > > > > image buffers en-queue. So we need to en-queue 3A statistics with
> > > > > > > non-request mode in this driver. After MW got the 3A statistics data, it
> > > > > > > will en-queue the images, tuning buffer and other meta buffers with
> > > > > > > request mode. Based on this requirement, do you have any suggestion?
> > > > > > > For upstream driver, should we only consider request mode?
> > > > > > >
> > > > > >
> > > > > > Where does that requirement come from? Why the timing of queuing of
> > > > > > the buffers to the driver is important?
> > > > > >
> > > > > > [snip]
> > > > >
> > > > > Basically, this requirement comes from our internal camera
> > > > > middle-ware/3A hal in user space. Since this is not generic requirement,
> > > > > we will follow your original suggestion to keep the request mode only
> > > > > and remove other non-request design in other files. For upstream driver,
> > > > > it should support request mode only.
> > > > >
> > > >
> > > > Note that Chromium OS will use the "upstream driver" and we don't want
> > > > to diverge, so please make the userspace also use only requests. I
> > > > don't see a reason why there would be any need to submit any buffers
> > > > outside of a request.
> > > >
> > > > [snip]
> > >
> > > Ok, I have raised your concern to our colleagues and let him to discuss
> > > with you in another communication channel.
> > >
> >
> > Thanks!
> >
> > Best regards,
> > Tomasz
>
> Our colleague is preparing material to explain the our 3A/MW design. If
> he is ready, he will discuss this with you.

Thanks!

>
> In the original plan, we will deliver P1 v4 patch set tomorrow (31th
> Jul.). But, there are some comments waiting for other experts' input.
> Do you suggest it is better to resolve all comments before v4 patch set
> submitting or continue to discuss these comments on v4?

For the remaining v4l2-compliance issues, we can postpone them and
keep on a TODO list in the next version.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-07-26  7:23         ` Jungo Lin
@ 2019-08-06  9:47           ` Tomasz Figa
  2019-08-07  2:11             ` Jungo Lin
  0 siblings, 1 reply; 45+ messages in thread
From: Tomasz Figa @ 2019-08-06  9:47 UTC (permalink / raw)
  To: Jungo Lin
  Cc: Hans Verkuil, Laurent Pinchart, Matthias Brugger,
	Mauro Carvalho Chehab, Linux Media Mailing List,
	moderated list:ARM/Mediatek SoC support,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	devicetree, srv_heupstream, ddavenport, Rob Herring,
	Sean Cheng (鄭昇弘),
	Sj Huang, Frederic Chen (陳俊元),
	Ryan Yu (余孟修),
	Rynn Wu (吳育恩),
	Frankie Chiu (邱文凱)

Hi Jungo,

On Fri, Jul 26, 2019 at 4:24 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi, Tomasz:
>
> On Thu, 2019-07-25 at 18:23 +0900, Tomasz Figa wrote:
> > .Hi Jungo,
> >
> > On Sat, Jul 20, 2019 at 6:58 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > >
> > > Hi, Tomasz:
> > >
> > > On Wed, 2019-07-10 at 18:56 +0900, Tomasz Figa wrote:
> > > > Hi Jungo,
> > > >
> > > > On Tue, Jun 11, 2019 at 11:53:42AM +0800, Jungo Lin wrote:
[snip]
> > > > > +
> > > > > +   err_status = irq_status & INT_ST_MASK_CAM_ERR;
> > > > > +
> > > > > +   /* Sof, done order check */
> > > > > +   if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST)) {
> > > > > +           dev_dbg(dev, "sof_done block cnt:%d\n", isp_dev->sof_count);
> > > > > +
> > > > > +           /* Notify IRQ event and enqueue frame */
> > > > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 0);
> > > > > +           isp_dev->current_frame = hw_frame_num;
> > > >
> > > > What exactly is hw_frame_num? Shouldn't we assign it before notifying the
> > > > event?
> > > >
> > >
> > > This is a another spare register for frame sequence number usage.
> > > It comes from struct p1_frame_param:frame_seq_no which is sent by
> > > SCP_ISP_FRAME IPI command. We will rename this to dequeue_frame_seq_no.
> > > Is it a better understanding?
> >
> > I'm sorry, unfortunately it's still not clear to me. Is it the
> > sequence number of the frame that was just processed and returned to
> > the kernel or the next frame that is going to be processed from now
> > on?
> >
>
> It is the next frame that is going to be proceed.
> We simplify the implementation of isp_irq_cam function. The hw_frame_num
> is renamed to dequeue_frame_seq_no and saved this value from HW at
> SOF_INT_ST. Since it is obtained in SOF_INI_ST event, it means it is
> next frame to be processed. If there is SW_PASS1_DON_ST, it means this
> frame is processed done. We use this value to de-queue the frame request
> and return buffers to VB2.
>
> The normal IRQ sequence is SOF_INT_ST => SW_PASS1_DON_ST &
> HW_PASS1_DON_ST.
>
> a. SW_PASS_DON_ST is designed for DMAs done event.
> If there is no available DMA buffers en-queued into HW, there is no
> SW_PADD_DON_ST.
>
> b. HW_PASS_DON_ST is designed to trigger CQ buffer load procedure.
> It is paired with SOF IRQ event, even if there is no available DMA
> buffers.
>
> static void isp_irq_handle_sof(struct mtk_isp_p1_device *p1_dev,
>                                unsigned int dequeue_frame_seq_no)
> {
>         dma_addr_t base_addr = p1_dev->composer_iova;
>         int composed_frame_seq_no =
> atomic_read(&p1_dev->composed_frame_seq_no);
>         unsigned int addr_offset;
>
>         /* Send V4L2_EVENT_FRAME_SYNC event */
>         mtk_cam_dev_event_frame_sync(&p1_dev->cam_dev, dequeue_frame_seq_no);
>
>         p1_dev->sof_count += 1;
>         /* Save dequeue frame information */
>         p1_dev->dequeue_frame_seq_no = dequeue_frame_seq_no;
>
>         /* Update CQ base address if needed */
>         if (composed_frame_seq_no <= dequeue_frame_seq_no) {
>                 dev_dbg(p1_dev->dev,
>                         "SOF_INT_ST, no update, cq_num:%d, frame_seq:%d",
>                         composed_frame_seq_no, dequeue_frame_seq_no);
>                 return;
>         }
>         addr_offset = MTK_ISP_CQ_ADDRESS_OFFSET *
>                 (dequeue_frame_seq_no % MTK_ISP_CQ_BUFFER_COUNT);
>         writel(base_addr + addr_offset, p1_dev->regs + REG_CQ_THR0_BASEADDR);
>         dev_dbg(p1_dev->dev,
>                 "SOF_INT_ST, update next, cq_num:%d, frame_seq:%d cq_addr:0x%x",
>                 composed_frame_seq_no, dequeue_frame_seq_no, addr_offset);
> }
>
> void mtk_cam_dev_dequeue_req_frame(struct mtk_cam_dev *cam,
>                                    unsigned int frame_seq_no)
> {
>         struct mtk_cam_dev_request *req, *req_prev;
>         unsigned long flags;
>
>         spin_lock_irqsave(&cam->running_job_lock, flags);
>         list_for_each_entry_safe(req, req_prev, &cam->running_job_list, list) {
>                 dev_dbg(cam->dev, "frame_seq:%d, de-queue frame_seq:%d\n",
>                         req->frame_params.frame_seq_no, frame_seq_no);
>
>                 /* Match by the en-queued request number */
>                 if (req->frame_params.frame_seq_no == frame_seq_no) {
>                         atomic_dec(&cam->running_job_count);
>                         /* Pass to user space */
>                         mtk_cam_dev_job_done(cam, req, VB2_BUF_STATE_DONE);
>                         list_del(&req->list);
>                         break;
>                 } else if (req->frame_params.frame_seq_no < frame_seq_no) {
>                         atomic_dec(&cam->running_job_count);
>                         /* Pass to user space for frame drop */
>                         mtk_cam_dev_job_done(cam, req, VB2_BUF_STATE_ERROR);
>                         dev_warn(cam->dev, "frame_seq:%d drop\n",
>                                  req->frame_params.frame_seq_no);
>                         list_del(&req->list);
>                 } else {
>                         break;
>                 }
>         }
>         spin_unlock_irqrestore(&cam->running_job_lock, flags);
>
> static irqreturn_t isp_irq_cam(int irq, void *data)
> {
>         struct mtk_isp_p1_device *p1_dev = (struct mtk_isp_p1_device *)data;
>         struct device *dev = p1_dev->dev;
>         unsigned int dequeue_frame_seq_no;
>         unsigned int irq_status, err_status, dma_status;
>         unsigned long flags;
>
>         spin_lock_irqsave(&p1_dev->spinlock_irq, flags);
>         irq_status = readl(p1_dev->regs + REG_CTL_RAW_INT_STAT);
>         err_status = irq_status & INT_ST_MASK_CAM_ERR;
>         dma_status = readl(p1_dev->regs + REG_CTL_RAW_INT2_STAT);
>         dequeue_frame_seq_no = readl(p1_dev->regs + REG_FRAME_SEQ_NUM);
>         spin_unlock_irqrestore(&p1_dev->spinlock_irq, flags);
>
>         /*
>          * In normal case, the next SOF ISR should come after HW PASS1 DONE
> ISR.
>          * If these two ISRs come together, print warning msg to hint.
>          */
>         if ((irq_status & SOF_INT_ST) && (irq_status & HW_PASS1_DON_ST))
>                 dev_warn(dev, "sof_done block cnt:%d\n", p1_dev->sof_count);
>
>         /* De-queue frame */
>         if (irq_status & SW_PASS1_DON_ST) {
>                 mtk_cam_dev_dequeue_req_frame(&p1_dev->cam_dev,
>                                               dequeue_frame_seq_no);
>                 mtk_cam_dev_req_try_queue(&p1_dev->cam_dev);
>         }
>
>         /* Save frame info. & update CQ address for frame HW en-queue */
>         if (irq_status & SOF_INT_ST)
>                 isp_irq_handle_sof(p1_dev, dequeue_frame_seq_no);
>
>         /* Check ISP error status */
>         if (err_status) {
>                 dev_err(dev, "int_err:0x%x 0x%x\n", irq_status, err_status);
>                 /* Show DMA errors in detail */
>                 if (err_status & DMA_ERR_ST)
>                         isp_irq_handle_dma_err(p1_dev);
>         }
>
>         dev_dbg(dev, "SOF:%d irq:0x%x, dma:0x%x, frame_num:%d",
>                 p1_dev->sof_count, irq_status, dma_status,
>                 dequeue_frame_seq_no);
>
>         return IRQ_HANDLED;
> }

I think I understand this now and the code above also looks good to
me. Thanks a lot!

>
> > >
> > > Below is our frame request handling in current design.
> > >
> > > 1. Buffer preparation
> > > - Combined image buffers (IMGO/RRZO) + meta input buffer (Tuining) +
> > > other meta histogram buffers (LCSO/LMVO) into one request.
> > > - Accumulated one unique frame sequence number to each request and send
> > > this request to the SCP composer to compose CQ (Command queue) buffer
> > > via SCP_ISP_FRAME IPI command.
> > > - CQ buffer is frame registers set. If ISP registers should be updated
> > > per frame, these registers are configured in the CQ buffer, such as
> > > frame sequence number, DMA addresses and tuning ISP registers.
> > > - One frame request will be composed into one CQ buffer.Once CQ buffer
> > > is composed done and kernel driver will receive ISP_CMD_FRAME_ACK with
> > > its corresponding frame sequence number. Based on this, kernel driver
> > > knows which request is ready to be en-queued and save this with
> > > p1_dev->isp_ctx.composed_frame_id.
> >
> > Hmm, why do we need to save this in p1_dev->isp_ctx? Wouldn't we
> > already have a linked lists of requests that are composed and ready to
> > be enqueued? Also, the request itself would contain its frame ID
> > inside the driver request struct, right?
> >
>
> Below is current implementation for frame request en-queued.
> Before en-queued into HW by CQ, the request should be composed by SCP
> composer.
>
> a. mtk_cam_dev_req_try_queue()
> - Insert the request into p1_dev->running_job_list
> b. mtk_isp_req_enqueue()
> - Assign new next frame ID to this request.
> - Sending to SCP by workqueue
> - This request is ready to compose
> c. isp_tx_frame_worker()
> - Send request to SCP with sync. mode. by SCP_IPI_ISP_FRAME command
> - SCP composer will compose the buffer CQ for this request frame based
> on struct mtk_p1_frame_param which includes frame ID.
> - If scp_ipi_send() is returned, it means the request is composed done.
> Or
> d. isp_composer_handler()
> - If we received the ISP_CMD_FRAME_ACK for SCP_IPI_ISP_FRAME, we save
> the frame ID in p1_dev->composed_frame_seq_no which is sent in step C.
> - The request is composed done here.
> e. isp_irq_handle_sof()
> - In SOF timing, we will check there is any available composed CQ
> buffers by comparing composed & current de-queued frame ID.
>
> For p1_dev->running_job_list, we can't guarantee the requests are
> composed until the end of step c. For step e, we need to know how many
> available composed requests are ready to en-queued.
>
> Do you suggest we add another new link-list to save these requests in
> step c or we could update p1_dev->composed_frame_seq_no in step c and
> remove the implementation in step d[1]?

Okay, thanks to your explanation above I think I understood how the
hardware flow behaves and so I think we can indeed keep the
composed_frame_seq counter. Thanks!

>
> [1]. isp_composer_handler() is mandatory callback function for SCP
> sending API with sync mode design.
>
> static void isp_composer_handler(void *data, unsigned int len, void
> *priv)
> {
>         struct mtk_isp_p1_device *p1_dev = (struct mtk_isp_p1_device *)priv;
>         struct mtk_isp_scp_p1_cmd *ipi_msg;
>
>         ipi_msg = (struct mtk_isp_scp_p1_cmd *)data;
>
>         if (ipi_msg->cmd_id != ISP_CMD_ACK)
>                 return;
>
>         if (ipi_msg->ack_info.cmd_id == ISP_CMD_FRAME_ACK) {
>                 atomic_set(&p1_dev->composed_frame_seq_no,
>                            ipi_msg->ack_info.frame_seq_no);
>                 dev_dbg(p1_dev->dev, "ack frame_num:%d\n",
>                         p1_dev->composed_frame_seq_no);
>         }
> }
>
> > > - The maximum number of CQ buffers in SCP is 3.
> > >
> > > 2. Buffer en-queue flow
> > > - In order to configure correct CQ buffer setting before next SQF event,
> > > it is depended on by MTK ISP P1 HW CQ mechanism.
> > > - The basic concept of CQ mechanism is loaded ISP CQ buffer settings
> > > when HW_PASS1_DON_ST is received which means DMA output is done.
> > > - Btw, the pre-condition of this, need to tell ISP HW which CQ buffer
> > > address is used. Otherwise, it will loaded one dummy CQ buffer to
> > > bypass.
> > > - So we will check available CQ buffers by comparing composed frame
> > > sequence number & dequeued frame sequence from ISP HW in SOF event.
> > > - If there are available CQ buffers, update the CQ base address to the
> > > next CQ buffer address based on current de-enqueue frame sequence
> > > number. So MTK ISP P1 HW will load this CQ buffer into HW when
> > > HW_PASS1_DON_ST is triggered which is before the next SOF.
> > > - So in next SOF event, ISP HW starts to output DMA buffers with this
> > > request until request is done.
> > > - But, for the first request, it is loaded into HW manually when
> > > streaming is on for better performance.
> > >
> > > 3. Buffer de-queue flow
> > > - We will use frame sequence number to decide which request is ready to
> > > de-queue.
> > > - We will save some important register setting from ISP HW when SOF is
> > > received. This is because the ISP HW starts to output the data with the
> > > corresponding settings, especially frame sequence number setting.
> >
> > Could you explain a bit more about these important register settings?
> > When does the hardware update the values in the register to new ones?
> > At SOF?
> >
>
> Sorry about my words.
> In the current implementation, we just save frame ID.
>

Ah, okay, makes sense. No worries. :)

>
> > > - When receiving SW_PASS1_DON_ST IRQ event, it means the DMA output is
> > > done. So we could call isp_deque_request_frame with frame sequence
> > > number to de-queue frame to VB2
> >
> > What's the difference between HW_PASS1_DON_ST and SW_PASS1_DON_ST?
> >
>
> This is explained above.
>
> > > - For AAO/AFO buffers, it has similar design concept. Sometimes, if only
> > > AAO/AFO non-request buffers are en-queued without request buffers at the
> > > same time, there will be no SW P1 done event for AAO/AFO DMA done.
> > > Needs to depend on other IRQ events, such as AAO/AFO_DONE_EVENT.
> >
> > Do we have a case like this? Wouldn't we normally always want to
> > bundle AAO/AFO buffers with frame buffers?
> >
>
> For upstream driver, we will remove non-request design.
>

I think we also talked about a thing related to this in the thread for
another patch from this series. Basically on Chrome OS we want to use
the upstream driver, so corresponding userspace changes might be
needed as well.

> > > - Due to CQ buffer number limitation, if we receive SW_PASS1_DONT_ST,
> > > we may try to send another request to SCP for composing.
> >
> > Okay, so basically in SW_PASS1_DONT_ST the CQ completed reading the CQ
> > buffers, right?
> >
>
> We expected the the life cycle of CQ buffer is same as frame request.
> So SW_PASS1_DON_ST is good timing to re-queue the next request to
> compose.
> For the CQ operations, we will explain later.
>
> > >
> > > Hopefully, my explanation is helpful for better understanding our
> > > implementation. If you still have any questions, please let me know.
> > >
> >
> > Yes, it's more clear now, thanks. Still some more comments above, though.
> >
> > > > > +           isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > > +           isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > > +   } else {
> > > > > +           if (irq_status & SOF_INT_ST) {
> > > > > +                   isp_dev->current_frame = hw_frame_num;
> > > > > +                   isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > > +                   isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > > +           }
> > > > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
> > > > > +   }
> > > >
> > > > The if and else blocks do almost the same things just in different order. Is
> > > > it really expected?
> > > >
> > >
> > > If we receive HW_PASS1_DON_ST & SOF_INT_ST IRQ events at the same time,
> > > the correct sequence should be handle HW_PASS1_DON_ST firstly to check
> > > any de-queued frame and update the next frame setting later.
> > > Normally, this is a corner case or system performance issue.
> >
> > So it sounds like HW_PASS1_DON_ST means that all data from current
> > frame has been written, right? If I understand your explanation above
> > correctly, that would mean following handling of each interrupt:
> >
> > HW_PASS1_DON_ST:
> >  - CQ executes with next CQ buffer to prepare for next frame. <- how
> > is this handled? does the CQ hardware automatically receive this event
> > from the ISP hadware?
> >  - return VB2 buffers,
> >  - complete requests.
> >
> > SOF_INT_ST:
> >  - send VSYNC event to userspace,
> >  - program next CQ buffer to CQ,
> >
> > SW_PASS1_DON_ST:
> >  - reclaim CQ buffer and enqueue next frame to composing if available
> >
>
> Sorry for our implementation of HW_PASS1_DON_ST.
> It is confusing.
> Below is the revised version based on your conclusion.
> So in our new implemmenation, we just handle SOF_INT_ST &
> SW_PASS1_DON_ST events. We just add one warning message for
> HW_PASS1_DON_ST
>
> HW_PASS1_DON_ST:
> - CQ executes with next CQ buffer to prepare for next frame.
>
> SOF_INT_ST:
> - send VSYNC event to userspace,
> - program next CQ buffer to CQ,
>
> SW_PASS1_DON_ST:
> - reclaim CQ buffer and enqueue next frame to composing if available
> - return VB2 buffers,
> - complete requests.
>
> For CQ HW operations, it is listed below:
>
> a. The CQ buffer has two kinds of information
>  - Which ISP registers needs to be updated.
>  - Where the corresponding ISP register data to be read.
> b. The CQ buffer loading procedure is triggered by HW_PASS1_DONT_ST IRQ
> event periodically.
>  - Normally, if the ISP HW receives the completed frame and it will
> trigger W_PASS1_DONT_ST IRQ and perform CQ buffer loading immediately.
> -  So the CQ buffer loading is performed by ISP HW automatically.
> c. The ISP HW will read CQ base address register(REG_CQ_THR0_BASEADDR)
> to decide which CQ buffer is loaded.
>    - So we configure the next CQ base address in SOF.
> d. For CQ buffer loading, CQ will read the ISP registers from CQ buffer
> and update the ISP register values into HW.
>    - SCP composer will compose one dummy CQ buffer and assign it to
> REG_CQ_THR0_BASEADDR of each CQ buffer.
>    - Dummy CQ buffer has no updated ISP registers comparing with other
> CQ buffers.
>    - With this design, if there is no updated new CQ buffer by driver
> which may be caused no en-queue frames from user space. The CQ HW will
> load dummy CQ buffer and do nothing.

Does the set of registers programmed by CQ include destination buffer
addresses to? If yes, we would end up overwriting previous frames if
no new buffers are provided.

> f. The CQ buffer loading is guaranteed by HW to finish before the next
> SOF.
>

Okay, thanks a lot for the explanation. This is much more clear now.

[snip]
> > > > > +static const struct dev_pm_ops mtk_isp_pm_ops = {
> > > > > +   SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
> > > > > +   SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)
> > > >
> > > > For V4L2 drivers system and runtime PM ops would normally be completely
> > > > different. Runtime PM ops would be called when the hardware is idle already
> > > > or is about to become active. System PM ops would be called at system power
> > > > state change and the hardware might be both idle or active. Please also see
> > > > my comments to mtk_isp_suspend() and mtk_isp_resume() above.
> > > >
> > >
> > > Here is the new implementation. It should be clear to show the
> > > difference between system and runtime PM ops.
> > >
> > > static const struct dev_pm_ops mtk_isp_pm_ops = {
> > >         SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> > >                                 pm_runtime_force_resume)
> > >         SET_RUNTIME_PM_OPS(mtk_isp_runtime_suspend, mtk_isp_runtime_resume,
> > > NULL)
> > > };
> >
> > That's still not correct. In runtime suspend/resume ops we already are
> > not streaming anymore, because we call pm_runtime_get/put_*() when
> > starting and stopping streaming. In system suspend/resume ops we might
> > be streaming and that's when we need to stop the hardware and wait for
> > it to finish. Please implement these ops separately.
> >
> > Best regards,
> > Tomasz
>
>
> Ok, got your point.
> Below is the new implementation for your review.
>
> static int mtk_isp_pm_suspend(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>         u32 val;
>         int ret;
>
>         dev_dbg(dev, "- %s\n", __func__);
>
>         /* Check ISP is streaming or not */
>         if (!p1_dev->cam_dev.streaming)
>                 goto done;

We would normally check here for pm_runtime_suspended(). Although they
both should be equivalent. Still, there is no need to call
pm_runtime_force_suspend() if the latter is true, so we could just
return 0 instantly.

>
>         /* Disable ISP's view finder and wait for TG idle */
>         dev_dbg(dev, "Cam suspend, disable VF\n");
>         val = readl(p1_dev->regs + REG_TG_VF_CON);
>         writel(val & (~TG_VF_CON_VFDATA_EN), p1_dev->regs + REG_TG_VF_CON);
>         ret = readl_poll_timeout_atomic(p1_dev->regs + REG_TG_INTER_ST, val,
>                                         (val & TG_CS_MASK) == TG_IDLE_ST,
>                                         USEC_PER_MSEC, MTK_ISP_STOP_HW_TIMEOUT);
>         if (ret)
>                 dev_warn(dev, "can't stop HW:%d:0x%x\n", ret, val);
>
>         /* Disable CMOS */
>         val = readl(p1_dev->regs + REG_TG_SEN_MODE);
>         writel(val & (~TG_SEN_MODE_CMOS_EN), p1_dev->regs + REG_TG_SEN_MODE);
>
> done:
>         /* Force ISP HW to idle */
>         ret = pm_runtime_force_suspend(dev);
>         if (ret)
>                 return ret;
>
>         return 0;
> }
>
> static int mtk_isp_pm_resume(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>         u32 val;
>         int ret;
>
>         dev_dbg(dev, "- %s\n", __func__);
>
>         /* Force ISP HW to resume if needed */
>         ret = pm_runtime_force_resume(dev);
>         if (ret)
>                 return ret;

We should do this conditionally based on what pm_runtime_suspended()
returns. If it's non-zero then we can just return 0 instantly.

>
>         if (!p1_dev->cam_dev.streaming)
>                 return 0;
>
>         /* Enable CMOS */
>         dev_dbg(dev, "Cam resume, enable CMOS/VF\n");
>         val = readl(p1_dev->regs + REG_TG_SEN_MODE);
>         writel(val | TG_SEN_MODE_CMOS_EN, p1_dev->regs + REG_TG_SEN_MODE);
>
>         /* Enable VF */
>         val = readl(p1_dev->regs + REG_TG_VF_CON);
>         writel(val | TG_VF_CON_VFDATA_EN, p1_dev->regs + REG_TG_VF_CON);
>
>         return 0;
> }
>
> static int mtk_isp_runtime_suspend(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>
>         dev_dbg(dev, "- %s\n", __func__);
>
>         if (pm_runtime_suspended(dev))
>                 return 0;

Sorry, I guess I wasn't clear in my reply. It's not possible to get
this callback called if the device is already runtime suspended.

>
>         dev_dbg(dev, "%s:disable clock\n", __func__);
>         clk_bulk_disable_unprepare(p1_dev->num_clks, p1_dev->clks);
>
>         return 0;
> }
>
> static int mtk_isp_runtime_resume(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>         int ret;
>
>         dev_dbg(dev, "- %s\n", __func__);
>
>         if (pm_runtime_suspended(dev))
>                 return 0;

In this case the above call would always return non-zero, so the
behavior wouldn't be very good.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-08-06  9:47           ` Tomasz Figa
@ 2019-08-07  2:11             ` Jungo Lin
  2019-08-07 13:25               ` Tomasz Figa
  0 siblings, 1 reply; 45+ messages in thread
From: Jungo Lin @ 2019-08-07  2:11 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

Hi, Tomasz:

On Tue, 2019-08-06 at 18:47 +0900, Tomasz Figa wrote:
> Hi Jungo,
> 
> On Fri, Jul 26, 2019 at 4:24 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> >
> > Hi, Tomasz:
> >
> > On Thu, 2019-07-25 at 18:23 +0900, Tomasz Figa wrote:
> > > .Hi Jungo,
> > >
> > > On Sat, Jul 20, 2019 at 6:58 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > >
> > > > Hi, Tomasz:
> > > >
> > > > On Wed, 2019-07-10 at 18:56 +0900, Tomasz Figa wrote:
> > > > > Hi Jungo,
> > > > >
> > > > > On Tue, Jun 11, 2019 at 11:53:42AM +0800, Jungo Lin wrote:
> [snip]

I just keep some questions to be clarified.
[snip]

> > > > > > +           isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > > > +           isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > > > +   } else {
> > > > > > +           if (irq_status & SOF_INT_ST) {
> > > > > > +                   isp_dev->current_frame = hw_frame_num;
> > > > > > +                   isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > > > +                   isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > > > +           }
> > > > > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
> > > > > > +   }
> > > > >
> > > > > The if and else blocks do almost the same things just in different order. Is
> > > > > it really expected?
> > > > >
> > > >
> > > > If we receive HW_PASS1_DON_ST & SOF_INT_ST IRQ events at the same time,
> > > > the correct sequence should be handle HW_PASS1_DON_ST firstly to check
> > > > any de-queued frame and update the next frame setting later.
> > > > Normally, this is a corner case or system performance issue.
> > >
> > > So it sounds like HW_PASS1_DON_ST means that all data from current
> > > frame has been written, right? If I understand your explanation above
> > > correctly, that would mean following handling of each interrupt:
> > >
> > > HW_PASS1_DON_ST:
> > >  - CQ executes with next CQ buffer to prepare for next frame. <- how
> > > is this handled? does the CQ hardware automatically receive this event
> > > from the ISP hadware?
> > >  - return VB2 buffers,
> > >  - complete requests.
> > >
> > > SOF_INT_ST:
> > >  - send VSYNC event to userspace,
> > >  - program next CQ buffer to CQ,
> > >
> > > SW_PASS1_DON_ST:
> > >  - reclaim CQ buffer and enqueue next frame to composing if available
> > >
> >
> > Sorry for our implementation of HW_PASS1_DON_ST.
> > It is confusing.
> > Below is the revised version based on your conclusion.
> > So in our new implemmenation, we just handle SOF_INT_ST &
> > SW_PASS1_DON_ST events. We just add one warning message for
> > HW_PASS1_DON_ST
> >
> > HW_PASS1_DON_ST:
> > - CQ executes with next CQ buffer to prepare for next frame.
> >
> > SOF_INT_ST:
> > - send VSYNC event to userspace,
> > - program next CQ buffer to CQ,
> >
> > SW_PASS1_DON_ST:
> > - reclaim CQ buffer and enqueue next frame to composing if available
> > - return VB2 buffers,
> > - complete requests.
> >
> > For CQ HW operations, it is listed below:
> >
> > a. The CQ buffer has two kinds of information
> >  - Which ISP registers needs to be updated.
> >  - Where the corresponding ISP register data to be read.
> > b. The CQ buffer loading procedure is triggered by HW_PASS1_DONT_ST IRQ
> > event periodically.
> >  - Normally, if the ISP HW receives the completed frame and it will
> > trigger W_PASS1_DONT_ST IRQ and perform CQ buffer loading immediately.
> > -  So the CQ buffer loading is performed by ISP HW automatically.
> > c. The ISP HW will read CQ base address register(REG_CQ_THR0_BASEADDR)
> > to decide which CQ buffer is loaded.
> >    - So we configure the next CQ base address in SOF.
> > d. For CQ buffer loading, CQ will read the ISP registers from CQ buffer
> > and update the ISP register values into HW.
> >    - SCP composer will compose one dummy CQ buffer and assign it to
> > REG_CQ_THR0_BASEADDR of each CQ buffer.
> >    - Dummy CQ buffer has no updated ISP registers comparing with other
> > CQ buffers.
> >    - With this design, if there is no updated new CQ buffer by driver
> > which may be caused no en-queue frames from user space. The CQ HW will
> > load dummy CQ buffer and do nothing.
> 
> Does the set of registers programmed by CQ include destination buffer
> addresses to? If yes, we would end up overwriting previous frames if
> no new buffers are provided.
> 

Yes, the buffer addresses are changed per frame request. We need to
compose CQ to include these DMA destination addresses. For your concern,
we have DMA flow buffer control (FBC) in HW. If there is no FBC counter
increased due to no buffer for each DMA, the ISP HW doesn't output the
data to the corresponding DMA address.

Below is the simple descriptor of CQ buffer.
a. ISP registers in tuning buffer, including 3A registers.
b. All capture buffers informations.
   - DMA buffer destination address
   - FBC counter
c. Some specif ISP registers for meta DMAs, such as LCE or LMVO.
d. frame sequence number register

> > f. The CQ buffer loading is guaranteed by HW to finish before the next
> > SOF.
> >
> 
> Okay, thanks a lot for the explanation. This is much more clear now.
> 
> [snip]
> > > > > > +static const struct dev_pm_ops mtk_isp_pm_ops = {
> > > > > > +   SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
> > > > > > +   SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)
> > > > >
> > > > > For V4L2 drivers system and runtime PM ops would normally be completely
> > > > > different. Runtime PM ops would be called when the hardware is idle already
> > > > > or is about to become active. System PM ops would be called at system power
> > > > > state change and the hardware might be both idle or active. Please also see
> > > > > my comments to mtk_isp_suspend() and mtk_isp_resume() above.
> > > > >
> > > >
> > > > Here is the new implementation. It should be clear to show the
> > > > difference between system and runtime PM ops.
> > > >
> > > > static const struct dev_pm_ops mtk_isp_pm_ops = {
> > > >         SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> > > >                                 pm_runtime_force_resume)
> > > >         SET_RUNTIME_PM_OPS(mtk_isp_runtime_suspend, mtk_isp_runtime_resume,
> > > > NULL)
> > > > };
> > >
> > > That's still not correct. In runtime suspend/resume ops we already are
> > > not streaming anymore, because we call pm_runtime_get/put_*() when
> > > starting and stopping streaming. In system suspend/resume ops we might
> > > be streaming and that's when we need to stop the hardware and wait for
> > > it to finish. Please implement these ops separately.
> > >
> > > Best regards,
> > > Tomasz
> >
> >
> > Ok, got your point.
> > Below is the new implementation for your review.
> >
> > static int mtk_isp_pm_suspend(struct device *dev)
> > {
> >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> >         u32 val;
> >         int ret;
> >
> >         dev_dbg(dev, "- %s\n", __func__);
> >
> >         /* Check ISP is streaming or not */
> >         if (!p1_dev->cam_dev.streaming)
> >                 goto done;
> 
> We would normally check here for pm_runtime_suspended(). Although they
> both should be equivalent. Still, there is no need to call
> pm_runtime_force_suspend() if the latter is true, so we could just
> return 0 instantly.
> 

Ok, here is the fixed version.

static int mtk_isp_pm_suspend(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
	u32 val;
	int ret;

	dev_dbg(dev, "- %s\n", __func__);

	if (pm_runtime_suspended(dev))
		return 0;

	/* Disable ISP's view finder and wait for TG idle */
	dev_dbg(dev, "cam suspend, disable VF\n");
	val = readl(p1_dev->regs + REG_TG_VF_CON);
	writel(val & (~TG_VF_CON_VFDATA_EN), p1_dev->regs + REG_TG_VF_CON);
	ret = readl_poll_timeout_atomic(p1_dev->regs + REG_TG_INTER_ST, val,
					(val & TG_CS_MASK) == TG_IDLE_ST,
					USEC_PER_MSEC, MTK_ISP_STOP_HW_TIMEOUT);
	if (ret)
		dev_warn(dev, "can't stop HW:%d:0x%x\n", ret, val);

	/* Disable CMOS */
	val = readl(p1_dev->regs + REG_TG_SEN_MODE);
	writel(val & (~TG_SEN_MODE_CMOS_EN), p1_dev->regs + REG_TG_SEN_MODE);

	/* Force ISP HW to idle */
	ret = pm_runtime_force_suspend(dev);
	if (ret)
		return ret;

	return 0;
}
[snip]

> > static int mtk_isp_pm_resume(struct device *dev)
> > {
> >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> >         u32 val;
> >         int ret;
> >
> >         dev_dbg(dev, "- %s\n", __func__);
> >
> >         /* Force ISP HW to resume if needed */
> >         ret = pm_runtime_force_resume(dev);
> >         if (ret)
> >                 return ret;
> 
> We should do this conditionally based on what pm_runtime_suspended()
> returns. If it's non-zero then we can just return 0 instantly.
> 

Ok, here is the fixed version.

static int mtk_isp_pm_resume(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
	u32 val;
	int ret;

	dev_dbg(dev, "- %s\n", __func__);

	if (pm_runtime_suspended(dev))
		return 0;

	/* Force ISP HW to resume */
	ret = pm_runtime_force_resume(dev);
	if (ret)
		return ret;

	/* Enable CMOS */
	dev_dbg(dev, "cam resume, enable CMOS/VF\n");
	val = readl(p1_dev->regs + REG_TG_SEN_MODE);
	writel(val | TG_SEN_MODE_CMOS_EN, p1_dev->regs + REG_TG_SEN_MODE);

	/* Enable VF */
	val = readl(p1_dev->regs + REG_TG_VF_CON);
	writel(val | TG_VF_CON_VFDATA_EN, p1_dev->regs + REG_TG_VF_CON);

	return 0;
}

[snip]

> > static int mtk_isp_runtime_suspend(struct device *dev)
> > {
> >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> >
> >         dev_dbg(dev, "- %s\n", __func__);
> >
> >         if (pm_runtime_suspended(dev))
> >                 return 0;
> 
> Sorry, I guess I wasn't clear in my reply. It's not possible to get
> this callback called if the device is already runtime suspended.
> 

Ok, got it. Need to remove pm_runtime_suspended(dev) checking and move
it into mtk_isp_pm_* functions. If I still don't get your point, could
you kindly provide one sample driver for reference? Based on current
implementation, it is similar to below drivers.
https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/mtk-mdp/mtk_mdp_core.c#L255
https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/exynos4-is/fimc-is-i2c.c#L113


static int mtk_isp_runtime_suspend(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);

	dev_dbg(dev, "%s:disable clock\n", __func__);
	clk_bulk_disable_unprepare(p1_dev->num_clks, p1_dev->clks);

	return 0;
}

[snip]

> > static int mtk_isp_runtime_resume(struct device *dev)
> > {
> >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> >         int ret;
> >
> >         dev_dbg(dev, "- %s\n", __func__);
> >
> >         if (pm_runtime_suspended(dev))
> >                 return 0;
> 
> In this case the above call would always return non-zero, so the
> behavior wouldn't be very good.
> 

Same as above.

static int mtk_isp_runtime_resume(struct device *dev)
{
	struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
	int ret;

	dev_dbg(dev, "%s:enable clock\n", __func__);
	ret = clk_bulk_prepare_enable(p1_dev->num_clks, p1_dev->clks);
	if (ret) {
		dev_err(dev, "failed to enable clock:%d\n", ret);
		return ret;
	}

	return 0;
}

Thanks for your further comments on these issues.

Best regards,

Jugno

> Best regards,
> Tomasz
> 
> _______________________________________________
> Linux-mediatek mailing list
> Linux-mediatek@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-mediatek



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver
  2019-08-07  2:11             ` Jungo Lin
@ 2019-08-07 13:25               ` Tomasz Figa
  0 siblings, 0 replies; 45+ messages in thread
From: Tomasz Figa @ 2019-08-07 13:25 UTC (permalink / raw)
  To: Jungo Lin
  Cc: devicetree, Sean Cheng (鄭昇弘),
	Frederic Chen (陳俊元),
	Rynn Wu (吳育恩),
	srv_heupstream, Rob Herring, Ryan Yu (余孟修),
	Frankie Chiu (邱文凱),
	Hans Verkuil, ddavenport, Sj Huang,
	moderated list:ARM/Mediatek SoC support, Laurent Pinchart,
	Matthias Brugger, Mauro Carvalho Chehab,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>,,
	Linux Media Mailing List

On Wed, Aug 7, 2019 at 11:11 AM Jungo Lin <jungo.lin@mediatek.com> wrote:
>
> Hi, Tomasz:
>
> On Tue, 2019-08-06 at 18:47 +0900, Tomasz Figa wrote:
> > Hi Jungo,
> >
> > On Fri, Jul 26, 2019 at 4:24 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > >
> > > Hi, Tomasz:
> > >
> > > On Thu, 2019-07-25 at 18:23 +0900, Tomasz Figa wrote:
> > > > .Hi Jungo,
> > > >
> > > > On Sat, Jul 20, 2019 at 6:58 PM Jungo Lin <jungo.lin@mediatek.com> wrote:
> > > > >
> > > > > Hi, Tomasz:
> > > > >
> > > > > On Wed, 2019-07-10 at 18:56 +0900, Tomasz Figa wrote:
> > > > > > Hi Jungo,
> > > > > >
> > > > > > On Tue, Jun 11, 2019 at 11:53:42AM +0800, Jungo Lin wrote:
> > [snip]
>
> I just keep some questions to be clarified.
> [snip]
>
> > > > > > > +           isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > > > > +           isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > > > > +   } else {
> > > > > > > +           if (irq_status & SOF_INT_ST) {
> > > > > > > +                   isp_dev->current_frame = hw_frame_num;
> > > > > > > +                   isp_dev->meta0_vb2_index = meta0_vb2_index;
> > > > > > > +                   isp_dev->meta1_vb2_index = meta1_vb2_index;
> > > > > > > +           }
> > > > > > > +           irq_handle_notify_event(isp_dev, irq_status, dma_status, 1);
> > > > > > > +   }
> > > > > >
> > > > > > The if and else blocks do almost the same things just in different order. Is
> > > > > > it really expected?
> > > > > >
> > > > >
> > > > > If we receive HW_PASS1_DON_ST & SOF_INT_ST IRQ events at the same time,
> > > > > the correct sequence should be handle HW_PASS1_DON_ST firstly to check
> > > > > any de-queued frame and update the next frame setting later.
> > > > > Normally, this is a corner case or system performance issue.
> > > >
> > > > So it sounds like HW_PASS1_DON_ST means that all data from current
> > > > frame has been written, right? If I understand your explanation above
> > > > correctly, that would mean following handling of each interrupt:
> > > >
> > > > HW_PASS1_DON_ST:
> > > >  - CQ executes with next CQ buffer to prepare for next frame. <- how
> > > > is this handled? does the CQ hardware automatically receive this event
> > > > from the ISP hadware?
> > > >  - return VB2 buffers,
> > > >  - complete requests.
> > > >
> > > > SOF_INT_ST:
> > > >  - send VSYNC event to userspace,
> > > >  - program next CQ buffer to CQ,
> > > >
> > > > SW_PASS1_DON_ST:
> > > >  - reclaim CQ buffer and enqueue next frame to composing if available
> > > >
> > >
> > > Sorry for our implementation of HW_PASS1_DON_ST.
> > > It is confusing.
> > > Below is the revised version based on your conclusion.
> > > So in our new implemmenation, we just handle SOF_INT_ST &
> > > SW_PASS1_DON_ST events. We just add one warning message for
> > > HW_PASS1_DON_ST
> > >
> > > HW_PASS1_DON_ST:
> > > - CQ executes with next CQ buffer to prepare for next frame.
> > >
> > > SOF_INT_ST:
> > > - send VSYNC event to userspace,
> > > - program next CQ buffer to CQ,
> > >
> > > SW_PASS1_DON_ST:
> > > - reclaim CQ buffer and enqueue next frame to composing if available
> > > - return VB2 buffers,
> > > - complete requests.
> > >
> > > For CQ HW operations, it is listed below:
> > >
> > > a. The CQ buffer has two kinds of information
> > >  - Which ISP registers needs to be updated.
> > >  - Where the corresponding ISP register data to be read.
> > > b. The CQ buffer loading procedure is triggered by HW_PASS1_DONT_ST IRQ
> > > event periodically.
> > >  - Normally, if the ISP HW receives the completed frame and it will
> > > trigger W_PASS1_DONT_ST IRQ and perform CQ buffer loading immediately.
> > > -  So the CQ buffer loading is performed by ISP HW automatically.
> > > c. The ISP HW will read CQ base address register(REG_CQ_THR0_BASEADDR)
> > > to decide which CQ buffer is loaded.
> > >    - So we configure the next CQ base address in SOF.
> > > d. For CQ buffer loading, CQ will read the ISP registers from CQ buffer
> > > and update the ISP register values into HW.
> > >    - SCP composer will compose one dummy CQ buffer and assign it to
> > > REG_CQ_THR0_BASEADDR of each CQ buffer.
> > >    - Dummy CQ buffer has no updated ISP registers comparing with other
> > > CQ buffers.
> > >    - With this design, if there is no updated new CQ buffer by driver
> > > which may be caused no en-queue frames from user space. The CQ HW will
> > > load dummy CQ buffer and do nothing.
> >
> > Does the set of registers programmed by CQ include destination buffer
> > addresses to? If yes, we would end up overwriting previous frames if
> > no new buffers are provided.
> >
>
> Yes, the buffer addresses are changed per frame request. We need to
> compose CQ to include these DMA destination addresses. For your concern,
> we have DMA flow buffer control (FBC) in HW. If there is no FBC counter
> increased due to no buffer for each DMA, the ISP HW doesn't output the
> data to the corresponding DMA address.
>
> Below is the simple descriptor of CQ buffer.
> a. ISP registers in tuning buffer, including 3A registers.
> b. All capture buffers informations.
>    - DMA buffer destination address
>    - FBC counter
> c. Some specif ISP registers for meta DMAs, such as LCE or LMVO.
> d. frame sequence number register
>

Okay, with the FBC counter it sounds fine. Thanks for clarifying.

> > > f. The CQ buffer loading is guaranteed by HW to finish before the next
> > > SOF.
> > >
> >
> > Okay, thanks a lot for the explanation. This is much more clear now.
> >
> > [snip]
> > > > > > > +static const struct dev_pm_ops mtk_isp_pm_ops = {
> > > > > > > +   SET_SYSTEM_SLEEP_PM_OPS(mtk_isp_suspend, mtk_isp_resume)
> > > > > > > +   SET_RUNTIME_PM_OPS(mtk_isp_suspend, mtk_isp_resume, NULL)
> > > > > >
> > > > > > For V4L2 drivers system and runtime PM ops would normally be completely
> > > > > > different. Runtime PM ops would be called when the hardware is idle already
> > > > > > or is about to become active. System PM ops would be called at system power
> > > > > > state change and the hardware might be both idle or active. Please also see
> > > > > > my comments to mtk_isp_suspend() and mtk_isp_resume() above.
> > > > > >
> > > > >
> > > > > Here is the new implementation. It should be clear to show the
> > > > > difference between system and runtime PM ops.
> > > > >
> > > > > static const struct dev_pm_ops mtk_isp_pm_ops = {
> > > > >         SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> > > > >                                 pm_runtime_force_resume)
> > > > >         SET_RUNTIME_PM_OPS(mtk_isp_runtime_suspend, mtk_isp_runtime_resume,
> > > > > NULL)
> > > > > };
> > > >
> > > > That's still not correct. In runtime suspend/resume ops we already are
> > > > not streaming anymore, because we call pm_runtime_get/put_*() when
> > > > starting and stopping streaming. In system suspend/resume ops we might
> > > > be streaming and that's when we need to stop the hardware and wait for
> > > > it to finish. Please implement these ops separately.
> > > >
> > > > Best regards,
> > > > Tomasz
> > >
> > >
> > > Ok, got your point.
> > > Below is the new implementation for your review.
> > >
> > > static int mtk_isp_pm_suspend(struct device *dev)
> > > {
> > >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> > >         u32 val;
> > >         int ret;
> > >
> > >         dev_dbg(dev, "- %s\n", __func__);
> > >
> > >         /* Check ISP is streaming or not */
> > >         if (!p1_dev->cam_dev.streaming)
> > >                 goto done;
> >
> > We would normally check here for pm_runtime_suspended(). Although they
> > both should be equivalent. Still, there is no need to call
> > pm_runtime_force_suspend() if the latter is true, so we could just
> > return 0 instantly.
> >
>
> Ok, here is the fixed version.
>
> static int mtk_isp_pm_suspend(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>         u32 val;
>         int ret;
>
>         dev_dbg(dev, "- %s\n", __func__);
>
>         if (pm_runtime_suspended(dev))
>                 return 0;
>
>         /* Disable ISP's view finder and wait for TG idle */
>         dev_dbg(dev, "cam suspend, disable VF\n");
>         val = readl(p1_dev->regs + REG_TG_VF_CON);
>         writel(val & (~TG_VF_CON_VFDATA_EN), p1_dev->regs + REG_TG_VF_CON);
>         ret = readl_poll_timeout_atomic(p1_dev->regs + REG_TG_INTER_ST, val,
>                                         (val & TG_CS_MASK) == TG_IDLE_ST,
>                                         USEC_PER_MSEC, MTK_ISP_STOP_HW_TIMEOUT);
>         if (ret)
>                 dev_warn(dev, "can't stop HW:%d:0x%x\n", ret, val);

What happens in this case? Is it safe to continue?

>
>         /* Disable CMOS */
>         val = readl(p1_dev->regs + REG_TG_SEN_MODE);
>         writel(val & (~TG_SEN_MODE_CMOS_EN), p1_dev->regs + REG_TG_SEN_MODE);
>
>         /* Force ISP HW to idle */
>         ret = pm_runtime_force_suspend(dev);
>         if (ret)
>                 return ret;

We should probably reenable the hardware if the above failed, so that
we hopefully end up in the same state as before the suspend.

>
>         return 0;
> }
> [snip]
>
> > > static int mtk_isp_pm_resume(struct device *dev)
> > > {
> > >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> > >         u32 val;
> > >         int ret;
> > >
> > >         dev_dbg(dev, "- %s\n", __func__);
> > >
> > >         /* Force ISP HW to resume if needed */
> > >         ret = pm_runtime_force_resume(dev);
> > >         if (ret)
> > >                 return ret;
> >
> > We should do this conditionally based on what pm_runtime_suspended()
> > returns. If it's non-zero then we can just return 0 instantly.
> >
>
> Ok, here is the fixed version.
>
> static int mtk_isp_pm_resume(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>         u32 val;
>         int ret;
>
>         dev_dbg(dev, "- %s\n", __func__);
>
>         if (pm_runtime_suspended(dev))
>                 return 0;
>
>         /* Force ISP HW to resume */
>         ret = pm_runtime_force_resume(dev);
>         if (ret)
>                 return ret;
>
>         /* Enable CMOS */
>         dev_dbg(dev, "cam resume, enable CMOS/VF\n");
>         val = readl(p1_dev->regs + REG_TG_SEN_MODE);
>         writel(val | TG_SEN_MODE_CMOS_EN, p1_dev->regs + REG_TG_SEN_MODE);
>
>         /* Enable VF */
>         val = readl(p1_dev->regs + REG_TG_VF_CON);
>         writel(val | TG_VF_CON_VFDATA_EN, p1_dev->regs + REG_TG_VF_CON);
>
>         return 0;
> }
>
> [snip]
>
> > > static int mtk_isp_runtime_suspend(struct device *dev)
> > > {
> > >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> > >
> > >         dev_dbg(dev, "- %s\n", __func__);
> > >
> > >         if (pm_runtime_suspended(dev))
> > >                 return 0;
> >
> > Sorry, I guess I wasn't clear in my reply. It's not possible to get
> > this callback called if the device is already runtime suspended.
> >
>
> Ok, got it. Need to remove pm_runtime_suspended(dev) checking and move
> it into mtk_isp_pm_* functions. If I still don't get your point, could
> you kindly provide one sample driver for reference?

The above implementation is okay, thanks. :)

> Based on current
> implementation, it is similar to below drivers.
> https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/mtk-mdp/mtk_mdp_core.c#L255
> https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/exynos4-is/fimc-is-i2c.c#L113
>

The first one is an m2m device so it has slightly different rules -
the runtime PM is allowed to suspend between frames if the idle time
is long enough. The second one is a dummy driver for some fake i2c
bus, so it doesn't really have any meaningful implementation.

I think you could take a look at
https://elixir.bootlin.com/linux/v5.3-rc3/source/drivers/media/platform/exynos4-is/fimc-lite.c#L1550
, which is an online capture device too.

>
> static int mtk_isp_runtime_suspend(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>
>         dev_dbg(dev, "%s:disable clock\n", __func__);
>         clk_bulk_disable_unprepare(p1_dev->num_clks, p1_dev->clks);
>
>         return 0;
> }
>
> [snip]
>
> > > static int mtk_isp_runtime_resume(struct device *dev)
> > > {
> > >         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
> > >         int ret;
> > >
> > >         dev_dbg(dev, "- %s\n", __func__);
> > >
> > >         if (pm_runtime_suspended(dev))
> > >                 return 0;
> >
> > In this case the above call would always return non-zero, so the
> > behavior wouldn't be very good.
> >
>
> Same as above.
>
> static int mtk_isp_runtime_resume(struct device *dev)
> {
>         struct mtk_isp_p1_device *p1_dev = dev_get_drvdata(dev);
>         int ret;
>
>         dev_dbg(dev, "%s:enable clock\n", __func__);
>         ret = clk_bulk_prepare_enable(p1_dev->num_clks, p1_dev->clks);
>         if (ret) {
>                 dev_err(dev, "failed to enable clock:%d\n", ret);
>                 return ret;
>         }
>
>         return 0;
> }

Makes sense, thanks!

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 45+ messages in thread

end of thread, other threads:[~2019-08-07 13:26 UTC | newest]

Thread overview: 45+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-11  3:53 [RFC,V3 0/9] media: platform: mtk-isp: Add Mediatek ISP Pass 1 driver Jungo Lin
2019-06-11  3:53 ` [RFC,v3 1/9] dt-bindings: mt8183: Added camera ISP Pass 1 Jungo Lin
2019-06-11  3:53 ` [RFC,v3 2/9] dts: arm64: mt8183: Add ISP Pass 1 nodes Jungo Lin
2019-06-11  3:53 ` [RFC,v3 3/9] media: platform: Add Mediatek ISP Pass 1 driver Kconfig Jungo Lin
2019-06-11  3:53 ` [RFC,v3 4/9] media: platform: Add Mediatek ISP P1 image & meta formats Jungo Lin
2019-06-11  3:53 ` [RFC,v3 5/9] media: platform: Add Mediatek ISP P1 V4L2 control Jungo Lin
2019-07-01  5:50   ` Tomasz Figa
2019-07-02 11:34     ` Jungo Lin
2019-06-11  3:53 ` [RFC,v3 6/9] media: platform: Add Mediatek ISP P1 V4L2 functions Jungo Lin
2019-07-10  9:54   ` Tomasz Figa
2019-07-18  4:39     ` Jungo Lin
2019-07-23 10:21       ` Tomasz Figa
2019-07-24  4:31         ` Jungo Lin
2019-07-26  5:49           ` Tomasz Figa
2019-07-29  1:18             ` Jungo Lin
2019-07-29 10:04               ` Tomasz Figa
2019-07-30  1:44                 ` Jungo Lin
2019-08-05  9:59                   ` Tomasz Figa
2019-06-11  3:53 ` [RFC,v3 7/9] media: platform: Add Mediatek ISP P1 device driver Jungo Lin
2019-07-10  9:56   ` Tomasz Figa
2019-07-20  9:58     ` Jungo Lin
2019-07-25  9:23       ` Tomasz Figa
2019-07-26  7:23         ` Jungo Lin
2019-08-06  9:47           ` Tomasz Figa
2019-08-07  2:11             ` Jungo Lin
2019-08-07 13:25               ` Tomasz Figa
2019-06-11  3:53 ` [RFC,v3 8/9] media: platform: Add Mediatek ISP P1 SCP communication Jungo Lin
2019-07-10  9:58   ` Tomasz Figa
2019-07-21  2:18     ` Jungo Lin
2019-07-25 10:56       ` Tomasz Figa
2019-07-26  8:07         ` Jungo Lin
2019-06-11  3:53 ` [RFC,v3 9/9] media: platform: Add Mediatek ISP P1 shared memory device Jungo Lin
2019-07-01  7:25   ` Tomasz Figa
2019-07-05  3:33     ` Jungo Lin
2019-07-05  4:22       ` Tomasz Figa
2019-07-05  5:44         ` Jungo Lin
2019-07-05  7:59         ` Jungo Lin
2019-07-23  7:20           ` Tomasz Figa
2019-07-23  8:21             ` [RFC, v3 " Jungo Lin
2019-07-26  5:15               ` Tomasz Figa
2019-07-26  7:41                 ` Christoph Hellwig
2019-07-26  7:42                   ` Tomasz Figa
2019-07-26 11:04                     ` Robin Murphy
2019-07-26 11:59                       ` Jungo Lin
2019-07-26 14:04                         ` Tomasz Figa

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).