* [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-18 3:46 ` Yong Zhi
0 siblings, 0 replies; 24+ messages in thread
From: Yong Zhi @ 2017-10-18 3:46 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: jian.xu.zheng, rajmohan.mani, tuukka.toivonen, jerry.w.hu, arnd,
hch, robin.murphy, iommu, Yong Zhi
This patchset adds support for the Intel IPU3 (Image Processing Unit)
ImgU which is essentially a modern memory-to-memory ISP. It implements
raw Bayer to YUV image format conversion as well as a large number of
other pixel processing algorithms for improving the image quality.
Meta data formats are defined for image statistics (3A, i.e. automatic
white balance, exposure and focus, histogram and local area contrast
enhancement) as well as for the pixel processing algorithm parameters.
The documentation for these formats is currently not included in the
patchset but will be added in a future version of this set.
The algorithm parameters need to be considered specific to a given frame
and typically a large number of these parameters change on frame to frame
basis. Additionally, the parameters are highly structured (and not a flat
space of independent configuration primitives). They also reflect the
data structures used by the firmware and the hardware. On top of that,
the algorithms require highly specialized user space to make meaningful
use of them. For these reasons it has been chosen video buffers to pass
the parameters to the device.
On individual patches:
The heart of ImgU is the CSS, or Camera Subsystem, which contains the
image processors and HW accelerators.
The 3A statistics and other firmware parameter computation related
functions are implemented in patch 8.
All h/w programming related code can be found in patch 9.
To access DDR via ImgU's own memory space, IPU3 is also equipped with
its own MMU unit, the driver is implemented in patch 2.
Currently, the MMU driver has dependency on two exported symbols
(iommu_group_ref_get and iommu_group_get_for_dev))to build as ko.
Patch 3 uses above IOMMU driver for DMA mem related functions.
Patch 5-10 are basically IPU3 CSS specific implementations:
6 and 7 provide some utility functions and manage IPU3 fw download and
install.
The firmware which is called ipu3-fw.bin can be downloaded from:
git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
(commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
Patch 9 and 10 are of the same file, the latter implements interface
functions for access fw & hw capabilities defined in patch 8.
Patch 11 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
<URL:https://patchwork.kernel.org/patch/9976293/>
<URL:https://patchwork.kernel.org/patch/9976295/>
Patch 12 uses Kconfig and Makefile created by IPU3 cio2 patch series.
Link to user space implementation:
<URL:https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master>
Device topology:
./media-ctl -d /dev/media0 -p
Media controller API version 4.14.0
Media device information
------------------------
driver ipu3-imgu
model ipu3-imgu
serial
bus info 0000:00:05.0
hw revision 0x0
driver version 4.14.0
Device topology
- entity 1: ipu3-imgu:0 (8 pads, 8 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
<- "input":0 [ENABLED,IMMUTABLE]
pad1: Sink
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
<- "parameters":0 []
pad2: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "output":0 []
pad3: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "viewfinder":0 []
pad4: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "postview":0 []
pad5: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "3a stat":0 []
pad6: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "dvs stat":0 []
pad7: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "lace stat":0 []
- entity 12: input (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Source
-> "ipu3-imgu:0":0 [ENABLED,IMMUTABLE]
- entity 18: parameters (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video1
pad0: Source
-> "ipu3-imgu:0":1 []
- entity 24: output (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video2
pad0: Sink
<- "ipu3-imgu:0":2 []
- entity 30: viewfinder (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video3
pad0: Sink
<- "ipu3-imgu:0":3 []
- entity 36: postview (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video4
pad0: Sink
<- "ipu3-imgu:0":4 []
- entity 42: 3a stat (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video5
pad0: Sink
<- "ipu3-imgu:0":5 []
- entity 48: dvs stat (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video6
pad0: Sink
<- "ipu3-imgu:0":6 []
- entity 54: lace stat (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video7
pad0: Sink
<- "ipu3-imgu:0":7 []
Sample test results on input and 3A video nodes:
localhost # ./v4l2-compliance -d /dev/video0
v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
Driver Info:
Driver name : ipu3-imgu:0
Card type : ipu3-imgu
Bus info : PCI:input
Driver version: 4.14.0
Capabilities : 0x84202000
Video Output Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04202000
Video Output Multiplanar
Streaming
Extended Pix Format
Compliance test for device /dev/video0 (not using libv4l2):
Required ioctls:
test VIDIOC_QUERYCAP: OK
Allow for multiple opens:
test second video open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 1 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Test output 0:
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
test VIDIOC_EXPBUF: OK
Test output 0:
Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
localhost # ./v4l2-compliance -d /dev/video5
v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
Driver Info:
Driver name : ipu3-imgu:0
Card type : ipu3-imgu
Bus info : PCI:3a stat
Driver version: 4.14.0
Capabilities : 0x84A00000
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04A00000
Metadata Capture
Streaming
Extended Pix Format
Compliance test for device /dev/video5 (not using libv4l2):
Required ioctls:
test VIDIOC_QUERYCAP: OK
Allow for multiple opens:
test second video open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
test VIDIOC_EXPBUF: OK
Test input 0:
Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
Note: stream test with -f fails as pre-configuration of sub-devs is required.
===========
= history =
===========
version 4:
- Used V4L2_BUF_TYPE_META_OUTPUT for:
- V4L2_META_FMT_IPU3_STAT_PARAMS
- Used V4L2_BUF_TYPE_META_CAPTURE for:
- V4L2_META_FMT_IPU3_STAT_3A
- V4L2_META_FMT_IPU3_STAT_DVS
- V4L2_META_FMT_IPU3_STAT_LACE
- Supported v4l2 MPLANE format on video nodes.
- ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA lib.
- ipu3-mmu.c: Re-structured the driver:
Removed dependencies on linux/dma-iommu.h
Add dev and dma_dev to struct ipu3_mmu to faciliate the dma_map_ops-less way of
binding between mmu, dmamap and ipu3 driver.
Addressed MMU review comments of v3.
Removed cache flush via setting page table as un-cache for improved performance.
- intel-ipu3.h: Added __padding qualifier for uapi definitions.
- Internal fix: power and performance related issues.
- Fixed v4l2-compliance test failures on video and meta nodes.
- Fixed build failure for x86 with 32bit config.
- Fixed checkpatch.pl errors/warnings/checks.
version 3:
- ipu3-mmu.c and ipu3-dmamap.c:
Tomasz Figa reworked both drivers and updated related files.
- ipu2-abi.h:
update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
use __packed qualifier on structs suggested by Sakari Ailus.
- ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
remove pointer type in firmware blob structs.
fix binary_header array in struct imgu_fw_header.
fix calling ipu3_css_fw_show_binary() before proper checking.
fix logic error for valid length checking of blob name.
- ipu3-css-params.c/ipu3_css_scaler_get_exp():
use lib helper suggested by Andy Shevchenko.
- ipu3-v4l2.c/ipu3_videoc_querycap():
fill device_caps fix suggested by Hans Verkuil.
add VB2_DMABUF suggested by Tomasz Figa.
- ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
- ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
version 2:
This version cherry-picked firmware ABI change and other
fix in order to bring the code up-to-date with our internal release.
I will go over the review comments in v1 and address them in v3 and
future update.
version 1:
- Initial submission
Tomasz Figa (2):
intel-ipu3: Add mmu driver
intel-ipu3: Add IOMMU based dmamap support
Yong Zhi (10):
videodev2.h, v4l2-ioctl: add IPU3 meta buffer format
intel-ipu3: Add user space ABI definitions
intel-ipu3: css: tables
intel-ipu3: css: imgu dma buff pool
intel-ipu3: css: firmware management
intel-ipu3: params: compute and program ccs
intel-ipu3: css hardware setup
intel-ipu3: css pipeline
intel-ipu3: Add imgu v4l2 driver
intel-ipu3: imgu top level pci device
drivers/media/pci/intel/ipu3/Kconfig | 33 +
drivers/media/pci/intel/ipu3/Makefile | 21 +
drivers/media/pci/intel/ipu3/ipu3-abi.h | 1579 ++++
drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 270 +
drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 206 +
drivers/media/pci/intel/ipu3/ipu3-css-params.c | 3161 ++++++++
drivers/media/pci/intel/ipu3/ipu3-css-params.h | 105 +
drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 132 +
drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 54 +
drivers/media/pci/intel/ipu3/ipu3-css.c | 2278 ++++++
drivers/media/pci/intel/ipu3/ipu3-css.h | 225 +
drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 342 +
drivers/media/pci/intel/ipu3/ipu3-dmamap.h | 33 +
drivers/media/pci/intel/ipu3/ipu3-mmu.c | 580 ++
drivers/media/pci/intel/ipu3/ipu3-mmu.h | 26 +
drivers/media/pci/intel/ipu3/ipu3-tables.c | 9621 ++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-tables.h | 82 +
drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1150 +++
drivers/media/pci/intel/ipu3/ipu3.c | 882 +++
drivers/media/pci/intel/ipu3/ipu3.h | 186 +
drivers/media/v4l2-core/v4l2-ioctl.c | 4 +
include/uapi/linux/intel-ipu3.h | 2199 ++++++
include/uapi/linux/videodev2.h | 6 +
23 files changed, 23175 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-abi.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
create mode 100644 include/uapi/linux/intel-ipu3.h
--
2.7.4
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-18 3:46 ` Yong Zhi
0 siblings, 0 replies; 24+ messages in thread
From: Yong Zhi @ 2017-10-18 3:46 UTC (permalink / raw)
To: linux-media-u79uwXL29TY76Z2rM5mHXA, sakari.ailus-VuQAYsv1563Yd54FQh9/CA
Cc: arnd-r2nGTMty4D4, jerry.w.hu-ral2JQCrhuEAvxtiuMwx3w,
jian.xu.zheng-ral2JQCrhuEAvxtiuMwx3w,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
tuukka.toivonen-ral2JQCrhuEAvxtiuMwx3w,
rajmohan.mani-ral2JQCrhuEAvxtiuMwx3w, hch-jcswGhMUV9g, Yong Zhi
This patchset adds support for the Intel IPU3 (Image Processing Unit)
ImgU which is essentially a modern memory-to-memory ISP. It implements
raw Bayer to YUV image format conversion as well as a large number of
other pixel processing algorithms for improving the image quality.
Meta data formats are defined for image statistics (3A, i.e. automatic
white balance, exposure and focus, histogram and local area contrast
enhancement) as well as for the pixel processing algorithm parameters.
The documentation for these formats is currently not included in the
patchset but will be added in a future version of this set.
The algorithm parameters need to be considered specific to a given frame
and typically a large number of these parameters change on frame to frame
basis. Additionally, the parameters are highly structured (and not a flat
space of independent configuration primitives). They also reflect the
data structures used by the firmware and the hardware. On top of that,
the algorithms require highly specialized user space to make meaningful
use of them. For these reasons it has been chosen video buffers to pass
the parameters to the device.
On individual patches:
The heart of ImgU is the CSS, or Camera Subsystem, which contains the
image processors and HW accelerators.
The 3A statistics and other firmware parameter computation related
functions are implemented in patch 8.
All h/w programming related code can be found in patch 9.
To access DDR via ImgU's own memory space, IPU3 is also equipped with
its own MMU unit, the driver is implemented in patch 2.
Currently, the MMU driver has dependency on two exported symbols
(iommu_group_ref_get and iommu_group_get_for_dev))to build as ko.
Patch 3 uses above IOMMU driver for DMA mem related functions.
Patch 5-10 are basically IPU3 CSS specific implementations:
6 and 7 provide some utility functions and manage IPU3 fw download and
install.
The firmware which is called ipu3-fw.bin can be downloaded from:
git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
(commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
Patch 9 and 10 are of the same file, the latter implements interface
functions for access fw & hw capabilities defined in patch 8.
Patch 11 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
<URL:https://patchwork.kernel.org/patch/9976293/>
<URL:https://patchwork.kernel.org/patch/9976295/>
Patch 12 uses Kconfig and Makefile created by IPU3 cio2 patch series.
Link to user space implementation:
<URL:https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master>
Device topology:
./media-ctl -d /dev/media0 -p
Media controller API version 4.14.0
Media device information
------------------------
driver ipu3-imgu
model ipu3-imgu
serial
bus info 0000:00:05.0
hw revision 0x0
driver version 4.14.0
Device topology
- entity 1: ipu3-imgu:0 (8 pads, 8 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
<- "input":0 [ENABLED,IMMUTABLE]
pad1: Sink
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
<- "parameters":0 []
pad2: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "output":0 []
pad3: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "viewfinder":0 []
pad4: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "postview":0 []
pad5: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "3a stat":0 []
pad6: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "dvs stat":0 []
pad7: Source
[fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
-> "lace stat":0 []
- entity 12: input (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Source
-> "ipu3-imgu:0":0 [ENABLED,IMMUTABLE]
- entity 18: parameters (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video1
pad0: Source
-> "ipu3-imgu:0":1 []
- entity 24: output (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video2
pad0: Sink
<- "ipu3-imgu:0":2 []
- entity 30: viewfinder (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video3
pad0: Sink
<- "ipu3-imgu:0":3 []
- entity 36: postview (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video4
pad0: Sink
<- "ipu3-imgu:0":4 []
- entity 42: 3a stat (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video5
pad0: Sink
<- "ipu3-imgu:0":5 []
- entity 48: dvs stat (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video6
pad0: Sink
<- "ipu3-imgu:0":6 []
- entity 54: lace stat (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video7
pad0: Sink
<- "ipu3-imgu:0":7 []
Sample test results on input and 3A video nodes:
localhost # ./v4l2-compliance -d /dev/video0
v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
Driver Info:
Driver name : ipu3-imgu:0
Card type : ipu3-imgu
Bus info : PCI:input
Driver version: 4.14.0
Capabilities : 0x84202000
Video Output Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04202000
Video Output Multiplanar
Streaming
Extended Pix Format
Compliance test for device /dev/video0 (not using libv4l2):
Required ioctls:
test VIDIOC_QUERYCAP: OK
Allow for multiple opens:
test second video open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 1 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Test output 0:
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
test VIDIOC_EXPBUF: OK
Test output 0:
Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
localhost # ./v4l2-compliance -d /dev/video5
v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
Driver Info:
Driver name : ipu3-imgu:0
Card type : ipu3-imgu
Bus info : PCI:3a stat
Driver version: 4.14.0
Capabilities : 0x84A00000
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04A00000
Metadata Capture
Streaming
Extended Pix Format
Compliance test for device /dev/video5 (not using libv4l2):
Required ioctls:
test VIDIOC_QUERYCAP: OK
Allow for multiple opens:
test second video open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
test VIDIOC_EXPBUF: OK
Test input 0:
Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
Note: stream test with -f fails as pre-configuration of sub-devs is required.
===========
= history =
===========
version 4:
- Used V4L2_BUF_TYPE_META_OUTPUT for:
- V4L2_META_FMT_IPU3_STAT_PARAMS
- Used V4L2_BUF_TYPE_META_CAPTURE for:
- V4L2_META_FMT_IPU3_STAT_3A
- V4L2_META_FMT_IPU3_STAT_DVS
- V4L2_META_FMT_IPU3_STAT_LACE
- Supported v4l2 MPLANE format on video nodes.
- ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA lib.
- ipu3-mmu.c: Re-structured the driver:
Removed dependencies on linux/dma-iommu.h
Add dev and dma_dev to struct ipu3_mmu to faciliate the dma_map_ops-less way of
binding between mmu, dmamap and ipu3 driver.
Addressed MMU review comments of v3.
Removed cache flush via setting page table as un-cache for improved performance.
- intel-ipu3.h: Added __padding qualifier for uapi definitions.
- Internal fix: power and performance related issues.
- Fixed v4l2-compliance test failures on video and meta nodes.
- Fixed build failure for x86 with 32bit config.
- Fixed checkpatch.pl errors/warnings/checks.
version 3:
- ipu3-mmu.c and ipu3-dmamap.c:
Tomasz Figa reworked both drivers and updated related files.
- ipu2-abi.h:
update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
use __packed qualifier on structs suggested by Sakari Ailus.
- ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
remove pointer type in firmware blob structs.
fix binary_header array in struct imgu_fw_header.
fix calling ipu3_css_fw_show_binary() before proper checking.
fix logic error for valid length checking of blob name.
- ipu3-css-params.c/ipu3_css_scaler_get_exp():
use lib helper suggested by Andy Shevchenko.
- ipu3-v4l2.c/ipu3_videoc_querycap():
fill device_caps fix suggested by Hans Verkuil.
add VB2_DMABUF suggested by Tomasz Figa.
- ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
- ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
version 2:
This version cherry-picked firmware ABI change and other
fix in order to bring the code up-to-date with our internal release.
I will go over the review comments in v1 and address them in v3 and
future update.
version 1:
- Initial submission
Tomasz Figa (2):
intel-ipu3: Add mmu driver
intel-ipu3: Add IOMMU based dmamap support
Yong Zhi (10):
videodev2.h, v4l2-ioctl: add IPU3 meta buffer format
intel-ipu3: Add user space ABI definitions
intel-ipu3: css: tables
intel-ipu3: css: imgu dma buff pool
intel-ipu3: css: firmware management
intel-ipu3: params: compute and program ccs
intel-ipu3: css hardware setup
intel-ipu3: css pipeline
intel-ipu3: Add imgu v4l2 driver
intel-ipu3: imgu top level pci device
drivers/media/pci/intel/ipu3/Kconfig | 33 +
drivers/media/pci/intel/ipu3/Makefile | 21 +
drivers/media/pci/intel/ipu3/ipu3-abi.h | 1579 ++++
drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 270 +
drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 206 +
drivers/media/pci/intel/ipu3/ipu3-css-params.c | 3161 ++++++++
drivers/media/pci/intel/ipu3/ipu3-css-params.h | 105 +
drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 132 +
drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 54 +
drivers/media/pci/intel/ipu3/ipu3-css.c | 2278 ++++++
drivers/media/pci/intel/ipu3/ipu3-css.h | 225 +
drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 342 +
drivers/media/pci/intel/ipu3/ipu3-dmamap.h | 33 +
drivers/media/pci/intel/ipu3/ipu3-mmu.c | 580 ++
drivers/media/pci/intel/ipu3/ipu3-mmu.h | 26 +
drivers/media/pci/intel/ipu3/ipu3-tables.c | 9621 ++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-tables.h | 82 +
drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1150 +++
drivers/media/pci/intel/ipu3/ipu3.c | 882 +++
drivers/media/pci/intel/ipu3/ipu3.h | 186 +
drivers/media/v4l2-core/v4l2-ioctl.c | 4 +
include/uapi/linux/intel-ipu3.h | 2199 ++++++
include/uapi/linux/videodev2.h | 6 +
23 files changed, 23175 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-abi.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.h
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
create mode 100644 include/uapi/linux/intel-ipu3.h
--
2.7.4
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-18 6:23 ` Christoph Hellwig
0 siblings, 0 replies; 24+ messages in thread
From: Christoph Hellwig @ 2017-10-18 6:23 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, jian.xu.zheng, rajmohan.mani,
tuukka.toivonen, jerry.w.hu, arnd, hch, robin.murphy, iommu
Please keep everyone on CC for all the patches, othervise they are
complete unreviable and will be ignored.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-18 6:23 ` Christoph Hellwig
0 siblings, 0 replies; 24+ messages in thread
From: Christoph Hellwig @ 2017-10-18 6:23 UTC (permalink / raw)
To: Yong Zhi
Cc: arnd-r2nGTMty4D4, jerry.w.hu-ral2JQCrhuEAvxtiuMwx3w,
jian.xu.zheng-ral2JQCrhuEAvxtiuMwx3w,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
tuukka.toivonen-ral2JQCrhuEAvxtiuMwx3w,
rajmohan.mani-ral2JQCrhuEAvxtiuMwx3w, hch-jcswGhMUV9g,
linux-media-u79uwXL29TY76Z2rM5mHXA
Please keep everyone on CC for all the patches, othervise they are
complete unreviable and will be ignored.
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-18 14:42 ` Zhi, Yong
0 siblings, 0 replies; 24+ messages in thread
From: Zhi, Yong @ 2017-10-18 14:42 UTC (permalink / raw)
To: Christoph Hellwig
Cc: linux-media, sakari.ailus, Zheng, Jian Xu, Mani, Rajmohan,
Toivonen, Tuukka, Hu, Jerry W, arnd, robin.murphy, iommu
Hi, Christoph,
Thanks for the message.
> -----Original Message-----
> From: Christoph Hellwig [mailto:hch@lst.de]
> Sent: Tuesday, October 17, 2017 11:23 PM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com; Zheng, Jian
> Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com; iommu@lists.linux-
> foundation.org
> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Please keep everyone on CC for all the patches, othervise they are complete
> unreviable and will be ignored.
Last time, Tomasz instructed me to cc you and other iommu experts only for mmu/dmamap patches(3 of 12), should I resend the 12 patches to both Linux-media and iommu list? or to Linux-media and cc everyone? Or just send the missing patches to iommu list and you folks this time?
Yong
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-18 14:42 ` Zhi, Yong
0 siblings, 0 replies; 24+ messages in thread
From: Zhi, Yong @ 2017-10-18 14:42 UTC (permalink / raw)
To: Christoph Hellwig
Cc: arnd-r2nGTMty4D4, Hu, Jerry W, Zheng, Jian Xu,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Toivonen,
Tuukka, Mani, Rajmohan, linux-media-u79uwXL29TY76Z2rM5mHXA
Hi, Christoph,
Thanks for the message.
> -----Original Message-----
> From: Christoph Hellwig [mailto:hch-jcswGhMUV9g@public.gmane.org]
> Sent: Tuesday, October 17, 2017 11:23 PM
> To: Zhi, Yong <yong.zhi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Cc: linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; sakari.ailus-VuQAYsv1563Yd54FQh9/CA@public.gmane.org; Zheng, Jian
> Xu <jian.xu.zheng-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; Mani, Rajmohan
> <rajmohan.mani-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; Toivonen, Tuukka
> <tuukka.toivonen-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; Hu, Jerry W <jerry.w.hu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>;
> arnd-r2nGTMty4D4@public.gmane.org; hch-jcswGhMUV9g@public.gmane.org; robin.murphy-5wv7dgnIgG8@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
> foundation.org
> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Please keep everyone on CC for all the patches, othervise they are complete
> unreviable and will be ignored.
Last time, Tomasz instructed me to cc you and other iommu experts only for mmu/dmamap patches(3 of 12), should I resend the 12 patches to both Linux-media and iommu list? or to Linux-media and cc everyone? Or just send the missing patches to iommu list and you folks this time?
Yong
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-10-18 3:46 ` Yong Zhi
(?)
(?)
@ 2017-10-20 9:09 ` Sakari Ailus
2017-10-23 21:24 ` Zhi, Yong
-1 siblings, 1 reply; 24+ messages in thread
From: Sakari Ailus @ 2017-10-20 9:09 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, jian.xu.zheng, rajmohan.mani,
tuukka.toivonen, jerry.w.hu, arnd, hch, robin.murphy, iommu
Hi Yong,
On Tue, Oct 17, 2017 at 10:46:48PM -0500, Yong Zhi wrote:
> This patchset adds support for the Intel IPU3 (Image Processing Unit)
> ImgU which is essentially a modern memory-to-memory ISP. It implements
> raw Bayer to YUV image format conversion as well as a large number of
> other pixel processing algorithms for improving the image quality.
>
> Meta data formats are defined for image statistics (3A, i.e. automatic
> white balance, exposure and focus, histogram and local area contrast
> enhancement) as well as for the pixel processing algorithm parameters.
> The documentation for these formats is currently not included in the
> patchset but will be added in a future version of this set.
>
> The algorithm parameters need to be considered specific to a given frame
> and typically a large number of these parameters change on frame to frame
> basis. Additionally, the parameters are highly structured (and not a flat
> space of independent configuration primitives). They also reflect the
> data structures used by the firmware and the hardware. On top of that,
> the algorithms require highly specialized user space to make meaningful
> use of them. For these reasons it has been chosen video buffers to pass
Do you have a to-do list for this patchset? I think it would be useful to
maintain one, in case not all the comments have been addressed.
--
Sakari Ailus
e-mail: sakari.ailus@iki.fi
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-23 21:24 ` Zhi, Yong
0 siblings, 0 replies; 24+ messages in thread
From: Zhi, Yong @ 2017-10-23 21:24 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, sakari.ailus, Zheng, Jian Xu, Mani, Rajmohan,
Toivonen, Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu
Hi, Sakari,
> -----Original Message-----
> From: linux-media-owner@vger.kernel.org [mailto:linux-media-
> owner@vger.kernel.org] On Behalf Of Sakari Ailus
> Sent: Friday, October 20, 2017 2:10 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com; Zheng, Jian
> Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com; iommu@lists.linux-
> foundation.org
> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Hi Yong,
>
> On Tue, Oct 17, 2017 at 10:46:48PM -0500, Yong Zhi wrote:
> > This patchset adds support for the Intel IPU3 (Image Processing Unit)
> > ImgU which is essentially a modern memory-to-memory ISP. It implements
> > raw Bayer to YUV image format conversion as well as a large number of
> > other pixel processing algorithms for improving the image quality.
> >
> > Meta data formats are defined for image statistics (3A, i.e. automatic
> > white balance, exposure and focus, histogram and local area contrast
> > enhancement) as well as for the pixel processing algorithm parameters.
> > The documentation for these formats is currently not included in the
> > patchset but will be added in a future version of this set.
> >
> > The algorithm parameters need to be considered specific to a given
> > frame and typically a large number of these parameters change on frame
> > to frame basis. Additionally, the parameters are highly structured
> > (and not a flat space of independent configuration primitives). They
> > also reflect the data structures used by the firmware and the
> > hardware. On top of that, the algorithms require highly specialized
> > user space to make meaningful use of them. For these reasons it has
> > been chosen video buffers to pass
>
> Do you have a to-do list for this patchset? I think it would be useful to
> maintain one, in case not all the comments have been addressed.
>
Sure, will add in next update.
> --
> Sakari Ailus
> e-mail: sakari.ailus@iki.fi
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2017-10-23 21:24 ` Zhi, Yong
0 siblings, 0 replies; 24+ messages in thread
From: Zhi, Yong @ 2017-10-23 21:24 UTC (permalink / raw)
To: Sakari Ailus
Cc: arnd-r2nGTMty4D4, Hu, Jerry W, Zheng, Jian Xu,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Toivonen,
Tuukka, Mani, Rajmohan, hch-jcswGhMUV9g,
linux-media-u79uwXL29TY76Z2rM5mHXA
Hi, Sakari,
> -----Original Message-----
> From: linux-media-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-media-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Sakari Ailus
> Sent: Friday, October 20, 2017 2:10 AM
> To: Zhi, Yong <yong.zhi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Cc: linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; sakari.ailus-VuQAYsv1563Yd54FQh9/CA@public.gmane.org; Zheng, Jian
> Xu <jian.xu.zheng-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; Mani, Rajmohan
> <rajmohan.mani-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; Toivonen, Tuukka
> <tuukka.toivonen-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; Hu, Jerry W <jerry.w.hu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>;
> arnd-r2nGTMty4D4@public.gmane.org; hch-jcswGhMUV9g@public.gmane.org; robin.murphy-5wv7dgnIgG8@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
> foundation.org
> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Hi Yong,
>
> On Tue, Oct 17, 2017 at 10:46:48PM -0500, Yong Zhi wrote:
> > This patchset adds support for the Intel IPU3 (Image Processing Unit)
> > ImgU which is essentially a modern memory-to-memory ISP. It implements
> > raw Bayer to YUV image format conversion as well as a large number of
> > other pixel processing algorithms for improving the image quality.
> >
> > Meta data formats are defined for image statistics (3A, i.e. automatic
> > white balance, exposure and focus, histogram and local area contrast
> > enhancement) as well as for the pixel processing algorithm parameters.
> > The documentation for these formats is currently not included in the
> > patchset but will be added in a future version of this set.
> >
> > The algorithm parameters need to be considered specific to a given
> > frame and typically a large number of these parameters change on frame
> > to frame basis. Additionally, the parameters are highly structured
> > (and not a flat space of independent configuration primitives). They
> > also reflect the data structures used by the firmware and the
> > hardware. On top of that, the algorithms require highly specialized
> > user space to make meaningful use of them. For these reasons it has
> > been chosen video buffers to pass
>
> Do you have a to-do list for this patchset? I think it would be useful to
> maintain one, in case not all the comments have been addressed.
>
Sure, will add in next update.
> --
> Sakari Ailus
> e-mail: sakari.ailus-X3B1VOXEql0@public.gmane.org
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-10-18 3:46 ` Yong Zhi
` (2 preceding siblings ...)
(?)
@ 2017-11-17 2:58 ` Mani, Rajmohan
2017-11-27 9:55 ` Hans Verkuil
2017-12-20 13:57 ` Mauro Carvalho Chehab
-1 siblings, 2 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2017-11-17 2:58 UTC (permalink / raw)
To: Zhi, Yong, linux-media, sakari.ailus
Cc: Zheng, Jian Xu, Toivonen, Tuukka, Hu, Jerry W, arnd, hch,
robin.murphy, iommu
Hi Sakari and all,
> -----Original Message-----
> From: Zhi, Yong
> Sent: Tuesday, October 17, 2017 8:47 PM
> To: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com
> Cc: Zheng, Jian Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com; iommu@lists.linux-
> foundation.org; Zhi, Yong <yong.zhi@intel.com>
> Subject: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> This patchset adds support for the Intel IPU3 (Image Processing Unit) ImgU
> which is essentially a modern memory-to-memory ISP. It implements raw
> Bayer to YUV image format conversion as well as a large number of other pixel
> processing algorithms for improving the image quality.
>
> Meta data formats are defined for image statistics (3A, i.e. automatic white
> balance, exposure and focus, histogram and local area contrast
> enhancement) as well as for the pixel processing algorithm parameters.
> The documentation for these formats is currently not included in the patchset
> but will be added in a future version of this set.
>
Here is an update on the IPU3 documentation that we are currently working on.
Image processing in IPU3 relies on the following.
1) HW configuration to enable ISP and
2) setting customer specific 3A Tuning / Algorithm Parameters to achieve desired image quality.
We intend to provide documentation on ImgU driver programming interface to help users of this driver to configure and enable ISP HW to meet their needs. This documentation will include details on complete V4L2 Kernel driver interface and IO-Control parameters, except for the ISP internal algorithm and its parameters (which is Intel proprietary IP).
We will also provide an user space library in binary form to help users of this driver, to convert the public 3A tuning parameters to IPU3 algorithm parameters. This tool will be released under NDA to the users of this driver.
> The algorithm parameters need to be considered specific to a given frame and
> typically a large number of these parameters change on frame to frame basis.
> Additionally, the parameters are highly structured (and not a flat space of
> independent configuration primitives). They also reflect the data structures
> used by the firmware and the hardware. On top of that, the algorithms require
> highly specialized user space to make meaningful use of them. For these
> reasons it has been chosen video buffers to pass the parameters to the device.
>
> On individual patches:
>
> The heart of ImgU is the CSS, or Camera Subsystem, which contains the image
> processors and HW accelerators.
>
> The 3A statistics and other firmware parameter computation related functions
> are implemented in patch 8.
>
> All h/w programming related code can be found in patch 9.
>
> To access DDR via ImgU's own memory space, IPU3 is also equipped with its
> own MMU unit, the driver is implemented in patch 2.
>
> Currently, the MMU driver has dependency on two exported symbols
> (iommu_group_ref_get and iommu_group_get_for_dev))to build as ko.
>
> Patch 3 uses above IOMMU driver for DMA mem related functions.
>
> Patch 5-10 are basically IPU3 CSS specific implementations:
>
> 6 and 7 provide some utility functions and manage IPU3 fw download and
> install.
>
> The firmware which is called ipu3-fw.bin can be downloaded from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>
> Patch 9 and 10 are of the same file, the latter implements interface functions
> for access fw & hw capabilities defined in patch 8.
>
> Patch 11 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>
> <URL:https://patchwork.kernel.org/patch/9976293/>
> <URL:https://patchwork.kernel.org/patch/9976295/>
>
> Patch 12 uses Kconfig and Makefile created by IPU3 cio2 patch series.
>
> Link to user space implementation:
>
> <URL:https://chromium.googlesource.com/chromiumos/platform/arc-
> camera/+/master>
>
> Device topology:
>
> ./media-ctl -d /dev/media0 -p
> Media controller API version 4.14.0
>
> Media device information
> ------------------------
> driver ipu3-imgu
> model ipu3-imgu
> serial
> bus info 0000:00:05.0
> hw revision 0x0
> driver version 4.14.0
>
> Device topology
> - entity 1: ipu3-imgu:0 (8 pads, 8 links)
> type V4L2 subdev subtype Unknown flags 0
> device node name /dev/v4l-subdev0
> pad0: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> <- "input":0 [ENABLED,IMMUTABLE]
> pad1: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> <- "parameters":0 []
> pad2: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "output":0 []
> pad3: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "viewfinder":0 []
> pad4: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "postview":0 []
> pad5: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "3a stat":0 []
> pad6: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "dvs stat":0 []
> pad7: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "lace stat":0 []
>
> - entity 12: input (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video0
> pad0: Source
> -> "ipu3-imgu:0":0 [ENABLED,IMMUTABLE]
>
> - entity 18: parameters (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video1
> pad0: Source
> -> "ipu3-imgu:0":1 []
>
> - entity 24: output (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video2
> pad0: Sink
> <- "ipu3-imgu:0":2 []
>
> - entity 30: viewfinder (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video3
> pad0: Sink
> <- "ipu3-imgu:0":3 []
>
> - entity 36: postview (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video4
> pad0: Sink
> <- "ipu3-imgu:0":4 []
>
> - entity 42: 3a stat (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video5
> pad0: Sink
> <- "ipu3-imgu:0":5 []
>
> - entity 48: dvs stat (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video6
> pad0: Sink
> <- "ipu3-imgu:0":6 []
>
> - entity 54: lace stat (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video7
> pad0: Sink
> <- "ipu3-imgu:0":7 []
>
>
> Sample test results on input and 3A video nodes:
>
> localhost # ./v4l2-compliance -d /dev/video0
> v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
>
> Driver Info:
> Driver name : ipu3-imgu:0
> Card type : ipu3-imgu
> Bus info : PCI:input
> Driver version: 4.14.0
> Capabilities : 0x84202000
> Video Output Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04202000
> Video Output Multiplanar
> Streaming
> Extended Pix Format
>
> Compliance test for device /dev/video0 (not using libv4l2):
>
> Required ioctls:
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second video open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 1 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Test output 0:
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not
> Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not
> Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> Test output 0:
>
>
> Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
>
> localhost # ./v4l2-compliance -d /dev/video5
> v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
>
> Driver Info:
> Driver name : ipu3-imgu:0
> Card type : ipu3-imgu
> Bus info : PCI:3a stat
> Driver version: 4.14.0
> Capabilities : 0x84A00000
> Metadata Capture
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04A00000
> Metadata Capture
> Streaming
> Extended Pix Format
>
> Compliance test for device /dev/video5 (not using libv4l2):
>
> Required ioctls:
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second video open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not
> Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not
> Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> Test input 0:
>
>
> Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
>
> Note: stream test with -f fails as pre-configuration of sub-devs is required.
>
> ===========
> = history =
> ===========
>
> version 4:
> - Used V4L2_BUF_TYPE_META_OUTPUT for:
> - V4L2_META_FMT_IPU3_STAT_PARAMS
> - Used V4L2_BUF_TYPE_META_CAPTURE for:
> - V4L2_META_FMT_IPU3_STAT_3A
> - V4L2_META_FMT_IPU3_STAT_DVS
> - V4L2_META_FMT_IPU3_STAT_LACE
> - Supported v4l2 MPLANE format on video nodes.
> - ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA lib.
> - ipu3-mmu.c: Re-structured the driver:
> Removed dependencies on linux/dma-iommu.h
> Add dev and dma_dev to struct ipu3_mmu to faciliate the dma_map_ops-less
> way of
> binding between mmu, dmamap and ipu3 driver.
> Addressed MMU review comments of v3.
> Removed cache flush via setting page table as un-cache for improved
> performance.
> - intel-ipu3.h: Added __padding qualifier for uapi definitions.
> - Internal fix: power and performance related issues.
> - Fixed v4l2-compliance test failures on video and meta nodes.
> - Fixed build failure for x86 with 32bit config.
> - Fixed checkpatch.pl errors/warnings/checks.
>
> version 3:
> - ipu3-mmu.c and ipu3-dmamap.c:
> Tomasz Figa reworked both drivers and updated related files.
> - ipu2-abi.h:
> update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
> use __packed qualifier on structs suggested by Sakari Ailus.
> - ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
> remove pointer type in firmware blob structs.
> fix binary_header array in struct imgu_fw_header.
> fix calling ipu3_css_fw_show_binary() before proper checking.
> fix logic error for valid length checking of blob name.
> - ipu3-css-params.c/ipu3_css_scaler_get_exp():
> use lib helper suggested by Andy Shevchenko.
> - ipu3-v4l2.c/ipu3_videoc_querycap():
> fill device_caps fix suggested by Hans Verkuil.
> add VB2_DMABUF suggested by Tomasz Figa.
> - ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
> - ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
>
> version 2:
> This version cherry-picked firmware ABI change and other fix in order to bring
> the code up-to-date with our internal release.
>
> I will go over the review comments in v1 and address them in v3 and future
> update.
>
> version 1:
> - Initial submission
>
> Tomasz Figa (2):
> intel-ipu3: Add mmu driver
> intel-ipu3: Add IOMMU based dmamap support
>
> Yong Zhi (10):
> videodev2.h, v4l2-ioctl: add IPU3 meta buffer format
> intel-ipu3: Add user space ABI definitions
> intel-ipu3: css: tables
> intel-ipu3: css: imgu dma buff pool
> intel-ipu3: css: firmware management
> intel-ipu3: params: compute and program ccs
> intel-ipu3: css hardware setup
> intel-ipu3: css pipeline
> intel-ipu3: Add imgu v4l2 driver
> intel-ipu3: imgu top level pci device
>
> drivers/media/pci/intel/ipu3/Kconfig | 33 +
> drivers/media/pci/intel/ipu3/Makefile | 21 +
> drivers/media/pci/intel/ipu3/ipu3-abi.h | 1579 ++++
> drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 270 +
> drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 206 +
> drivers/media/pci/intel/ipu3/ipu3-css-params.c | 3161 ++++++++
> drivers/media/pci/intel/ipu3/ipu3-css-params.h | 105 +
> drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 132 +
> drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 54 +
> drivers/media/pci/intel/ipu3/ipu3-css.c | 2278 ++++++
> drivers/media/pci/intel/ipu3/ipu3-css.h | 225 +
> drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 342 +
> drivers/media/pci/intel/ipu3/ipu3-dmamap.h | 33 +
> drivers/media/pci/intel/ipu3/ipu3-mmu.c | 580 ++
> drivers/media/pci/intel/ipu3/ipu3-mmu.h | 26 +
> drivers/media/pci/intel/ipu3/ipu3-tables.c | 9621
> ++++++++++++++++++++++++
> drivers/media/pci/intel/ipu3/ipu3-tables.h | 82 +
> drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1150 +++
> drivers/media/pci/intel/ipu3/ipu3.c | 882 +++
> drivers/media/pci/intel/ipu3/ipu3.h | 186 +
> drivers/media/v4l2-core/v4l2-ioctl.c | 4 +
> include/uapi/linux/intel-ipu3.h | 2199 ++++++
> include/uapi/linux/videodev2.h | 6 +
> 23 files changed, 23175 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-abi.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
> create mode 100644 include/uapi/linux/intel-ipu3.h
>
> --
> 2.7.4
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-11-17 2:58 ` Mani, Rajmohan
@ 2017-11-27 9:55 ` Hans Verkuil
2017-12-05 0:13 ` Mani, Rajmohan
2017-12-20 13:57 ` Mauro Carvalho Chehab
1 sibling, 1 reply; 24+ messages in thread
From: Hans Verkuil @ 2017-11-27 9:55 UTC (permalink / raw)
To: Mani, Rajmohan, Zhi, Yong, linux-media, sakari.ailus
Cc: Zheng, Jian Xu, Toivonen, Tuukka, Hu, Jerry W, arnd, hch,
robin.murphy, iommu
Hi Rajmohan,
On 11/17/2017 03:58 AM, Mani, Rajmohan wrote:
> Hi Sakari and all,
>
>> -----Original Message-----
>> From: Zhi, Yong
>> Sent: Tuesday, October 17, 2017 8:47 PM
>> To: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com
>> Cc: Zheng, Jian Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
>> <rajmohan.mani@intel.com>; Toivonen, Tuukka
>> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
>> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com; iommu@lists.linux-
>> foundation.org; Zhi, Yong <yong.zhi@intel.com>
>> Subject: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>>
>> This patchset adds support for the Intel IPU3 (Image Processing Unit) ImgU
>> which is essentially a modern memory-to-memory ISP. It implements raw
>> Bayer to YUV image format conversion as well as a large number of other pixel
>> processing algorithms for improving the image quality.
>>
>> Meta data formats are defined for image statistics (3A, i.e. automatic white
>> balance, exposure and focus, histogram and local area contrast
>> enhancement) as well as for the pixel processing algorithm parameters.
>> The documentation for these formats is currently not included in the patchset
>> but will be added in a future version of this set.
>>
>
> Here is an update on the IPU3 documentation that we are currently working on.
>
> Image processing in IPU3 relies on the following.
>
> 1) HW configuration to enable ISP and
> 2) setting customer specific 3A Tuning / Algorithm Parameters to achieve desired image quality.
>
> We intend to provide documentation on ImgU driver programming interface to help users of this driver to configure and enable ISP HW to meet their needs. This documentation will include details on complete V4L2 Kernel driver interface and IO-Control parameters, except for the ISP internal algorithm and its parameters (which is Intel proprietary IP).
>
> We will also provide an user space library in binary form to help users of this driver, to convert the public 3A tuning parameters to IPU3 algorithm parameters. This tool will be released under NDA to the users of this driver.
So I discussed this situation with Sakari in Prague during the ELCE. I am not
happy with adding a driver to the kernel that needs an NDA to be usable. I can't
agree to that. It's effectively the same as firmware that's only available under
an NDA and we wouldn't accept such drivers either.
There are a few options:
1) Document the ISP parameters and that format they are stored in allowing for
open source implementations. While this is the ideal, I suspect that this is
a no-go for Intel.
2) The driver can be used without using these ISP parameters and still achieve
'OK' quality. I.e., it's usable for basic webcam usage under normal lighting
conditions. I'm not sure if this is possible at all, though.
3) Make the library available without requiring an NDA.
4) Provide a non-NDA library (ideally open source) that achieves at minimum
the quality as described in 2: i.e. usable for basic webcam.
5) Other solutions I didn't think of?
I think 4 might be the best option, especially if it is open sourced and just
uses non-IP 3A algorithms. It could even be added to the v4l-utils git repo.
A closed source non-NDA library might also work, but you will need to check
what Mauro thinks about that. My problem is that such libraries tend to be
abandoned after a few years and then bit-rot sets in. An open-source solution
is much easier to maintain in the long term.
Regards,
Hans
>
>> The algorithm parameters need to be considered specific to a given frame and
>> typically a large number of these parameters change on frame to frame basis.
>> Additionally, the parameters are highly structured (and not a flat space of
>> independent configuration primitives). They also reflect the data structures
>> used by the firmware and the hardware. On top of that, the algorithms require
>> highly specialized user space to make meaningful use of them. For these
>> reasons it has been chosen video buffers to pass the parameters to the device.
>>
>> On individual patches:
>>
>> The heart of ImgU is the CSS, or Camera Subsystem, which contains the image
>> processors and HW accelerators.
>>
>> The 3A statistics and other firmware parameter computation related functions
>> are implemented in patch 8.
>>
>> All h/w programming related code can be found in patch 9.
>>
>> To access DDR via ImgU's own memory space, IPU3 is also equipped with its
>> own MMU unit, the driver is implemented in patch 2.
>>
>> Currently, the MMU driver has dependency on two exported symbols
>> (iommu_group_ref_get and iommu_group_get_for_dev))to build as ko.
>>
>> Patch 3 uses above IOMMU driver for DMA mem related functions.
>>
>> Patch 5-10 are basically IPU3 CSS specific implementations:
>>
>> 6 and 7 provide some utility functions and manage IPU3 fw download and
>> install.
>>
>> The firmware which is called ipu3-fw.bin can be downloaded from:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
>> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>>
>> Patch 9 and 10 are of the same file, the latter implements interface functions
>> for access fw & hw capabilities defined in patch 8.
>>
>> Patch 11 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>>
>> <URL:https://patchwork.kernel.org/patch/9976293/>
>> <URL:https://patchwork.kernel.org/patch/9976295/>
>>
>> Patch 12 uses Kconfig and Makefile created by IPU3 cio2 patch series.
>>
>> Link to user space implementation:
>>
>> <URL:https://chromium.googlesource.com/chromiumos/platform/arc-
>> camera/+/master>
>>
>> Device topology:
>>
>> ./media-ctl -d /dev/media0 -p
>> Media controller API version 4.14.0
>>
>> Media device information
>> ------------------------
>> driver ipu3-imgu
>> model ipu3-imgu
>> serial
>> bus info 0000:00:05.0
>> hw revision 0x0
>> driver version 4.14.0
>>
>> Device topology
>> - entity 1: ipu3-imgu:0 (8 pads, 8 links)
>> type V4L2 subdev subtype Unknown flags 0
>> device node name /dev/v4l-subdev0
>> pad0: Sink
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> <- "input":0 [ENABLED,IMMUTABLE]
>> pad1: Sink
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> <- "parameters":0 []
>> pad2: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "output":0 []
>> pad3: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "viewfinder":0 []
>> pad4: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "postview":0 []
>> pad5: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "3a stat":0 []
>> pad6: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "dvs stat":0 []
>> pad7: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "lace stat":0 []
>>
>> - entity 12: input (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video0
>> pad0: Source
>> -> "ipu3-imgu:0":0 [ENABLED,IMMUTABLE]
>>
>> - entity 18: parameters (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video1
>> pad0: Source
>> -> "ipu3-imgu:0":1 []
>>
>> - entity 24: output (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video2
>> pad0: Sink
>> <- "ipu3-imgu:0":2 []
>>
>> - entity 30: viewfinder (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video3
>> pad0: Sink
>> <- "ipu3-imgu:0":3 []
>>
>> - entity 36: postview (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video4
>> pad0: Sink
>> <- "ipu3-imgu:0":4 []
>>
>> - entity 42: 3a stat (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video5
>> pad0: Sink
>> <- "ipu3-imgu:0":5 []
>>
>> - entity 48: dvs stat (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video6
>> pad0: Sink
>> <- "ipu3-imgu:0":6 []
>>
>> - entity 54: lace stat (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video7
>> pad0: Sink
>> <- "ipu3-imgu:0":7 []
>>
>>
>> Sample test results on input and 3A video nodes:
>>
>> localhost # ./v4l2-compliance -d /dev/video0
>> v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
>>
>> Driver Info:
>> Driver name : ipu3-imgu:0
>> Card type : ipu3-imgu
>> Bus info : PCI:input
>> Driver version: 4.14.0
>> Capabilities : 0x84202000
>> Video Output Multiplanar
>> Streaming
>> Extended Pix Format
>> Device Capabilities
>> Device Caps : 0x04202000
>> Video Output Multiplanar
>> Streaming
>> Extended Pix Format
>>
>> Compliance test for device /dev/video0 (not using libv4l2):
>>
>> Required ioctls:
>> test VIDIOC_QUERYCAP: OK
>>
>> Allow for multiple opens:
>> test second video open: OK
>> test VIDIOC_QUERYCAP: OK
>> test VIDIOC_G/S_PRIORITY: OK
>> test for unlimited opens: OK
>>
>> Debug ioctls:
>> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>> test VIDIOC_LOG_STATUS: OK (Not Supported)
>>
>> Input ioctls:
>> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>> test VIDIOC_ENUMAUDIO: OK (Not Supported)
>> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
>> test VIDIOC_G/S_AUDIO: OK (Not Supported)
>> Inputs: 0 Audio Inputs: 0 Tuners: 0
>>
>> Output ioctls:
>> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>> test VIDIOC_G/S/ENUMOUTPUT: OK
>> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>> Outputs: 1 Audio Outputs: 0 Modulators: 0
>>
>> Input/Output configuration ioctls:
>> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>> test VIDIOC_G/S_EDID: OK (Not Supported)
>>
>> Test output 0:
>>
>> Control ioctls:
>> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not
>> Supported)
>> test VIDIOC_QUERYCTRL: OK (Not Supported)
>> test VIDIOC_G/S_CTRL: OK (Not Supported)
>> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not
>> Supported)
>> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>> Standard Controls: 0 Private Controls: 0
>>
>> Format ioctls:
>> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>> test VIDIOC_G/S_PARM: OK (Not Supported)
>> test VIDIOC_G_FBUF: OK (Not Supported)
>> test VIDIOC_G_FMT: OK
>> test VIDIOC_TRY_FMT: OK
>> test VIDIOC_S_FMT: OK
>> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>> test Cropping: OK (Not Supported)
>> test Composing: OK (Not Supported)
>> test Scaling: OK
>>
>> Codec ioctls:
>> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
>> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>>
>> Buffer ioctls:
>> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>> test VIDIOC_EXPBUF: OK
>>
>> Test output 0:
>>
>>
>> Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
>>
>> localhost # ./v4l2-compliance -d /dev/video5
>> v4l2-compliance SHA : f71ba5a1779ddb6a5a59562504dcf4fabf5c1de1
>>
>> Driver Info:
>> Driver name : ipu3-imgu:0
>> Card type : ipu3-imgu
>> Bus info : PCI:3a stat
>> Driver version: 4.14.0
>> Capabilities : 0x84A00000
>> Metadata Capture
>> Streaming
>> Extended Pix Format
>> Device Capabilities
>> Device Caps : 0x04A00000
>> Metadata Capture
>> Streaming
>> Extended Pix Format
>>
>> Compliance test for device /dev/video5 (not using libv4l2):
>>
>> Required ioctls:
>> test VIDIOC_QUERYCAP: OK
>>
>> Allow for multiple opens:
>> test second video open: OK
>> test VIDIOC_QUERYCAP: OK
>> test VIDIOC_G/S_PRIORITY: OK
>> test for unlimited opens: OK
>>
>> Debug ioctls:
>> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>> test VIDIOC_LOG_STATUS: OK (Not Supported)
>>
>> Input ioctls:
>> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>> test VIDIOC_ENUMAUDIO: OK (Not Supported)
>> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
>> test VIDIOC_G/S_AUDIO: OK (Not Supported)
>> Inputs: 0 Audio Inputs: 0 Tuners: 0
>>
>> Output ioctls:
>> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
>> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>> Outputs: 0 Audio Outputs: 0 Modulators: 0
>>
>> Input/Output configuration ioctls:
>> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>> test VIDIOC_G/S_EDID: OK (Not Supported)
>>
>> Control ioctls:
>> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not
>> Supported)
>> test VIDIOC_QUERYCTRL: OK (Not Supported)
>> test VIDIOC_G/S_CTRL: OK (Not Supported)
>> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not
>> Supported)
>> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>> Standard Controls: 0 Private Controls: 0
>>
>> Format ioctls:
>> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>> test VIDIOC_G/S_PARM: OK (Not Supported)
>> test VIDIOC_G_FBUF: OK (Not Supported)
>> test VIDIOC_G_FMT: OK
>> test VIDIOC_TRY_FMT: OK
>> test VIDIOC_S_FMT: OK
>> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>> test Cropping: OK (Not Supported)
>> test Composing: OK (Not Supported)
>> test Scaling: OK (Not Supported)
>>
>> Codec ioctls:
>> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
>> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>>
>> Buffer ioctls:
>> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>> test VIDIOC_EXPBUF: OK
>>
>> Test input 0:
>>
>>
>> Total: 43, Succeeded: 43, Failed: 0, Warnings: 0
>>
>> Note: stream test with -f fails as pre-configuration of sub-devs is required.
>>
>> ===========
>> = history =
>> ===========
>>
>> version 4:
>> - Used V4L2_BUF_TYPE_META_OUTPUT for:
>> - V4L2_META_FMT_IPU3_STAT_PARAMS
>> - Used V4L2_BUF_TYPE_META_CAPTURE for:
>> - V4L2_META_FMT_IPU3_STAT_3A
>> - V4L2_META_FMT_IPU3_STAT_DVS
>> - V4L2_META_FMT_IPU3_STAT_LACE
>> - Supported v4l2 MPLANE format on video nodes.
>> - ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA lib.
>> - ipu3-mmu.c: Re-structured the driver:
>> Removed dependencies on linux/dma-iommu.h
>> Add dev and dma_dev to struct ipu3_mmu to faciliate the dma_map_ops-less
>> way of
>> binding between mmu, dmamap and ipu3 driver.
>> Addressed MMU review comments of v3.
>> Removed cache flush via setting page table as un-cache for improved
>> performance.
>> - intel-ipu3.h: Added __padding qualifier for uapi definitions.
>> - Internal fix: power and performance related issues.
>> - Fixed v4l2-compliance test failures on video and meta nodes.
>> - Fixed build failure for x86 with 32bit config.
>> - Fixed checkpatch.pl errors/warnings/checks.
>>
>> version 3:
>> - ipu3-mmu.c and ipu3-dmamap.c:
>> Tomasz Figa reworked both drivers and updated related files.
>> - ipu2-abi.h:
>> update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
>> use __packed qualifier on structs suggested by Sakari Ailus.
>> - ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
>> remove pointer type in firmware blob structs.
>> fix binary_header array in struct imgu_fw_header.
>> fix calling ipu3_css_fw_show_binary() before proper checking.
>> fix logic error for valid length checking of blob name.
>> - ipu3-css-params.c/ipu3_css_scaler_get_exp():
>> use lib helper suggested by Andy Shevchenko.
>> - ipu3-v4l2.c/ipu3_videoc_querycap():
>> fill device_caps fix suggested by Hans Verkuil.
>> add VB2_DMABUF suggested by Tomasz Figa.
>> - ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
>> - ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
>>
>> version 2:
>> This version cherry-picked firmware ABI change and other fix in order to bring
>> the code up-to-date with our internal release.
>>
>> I will go over the review comments in v1 and address them in v3 and future
>> update.
>>
>> version 1:
>> - Initial submission
>>
>> Tomasz Figa (2):
>> intel-ipu3: Add mmu driver
>> intel-ipu3: Add IOMMU based dmamap support
>>
>> Yong Zhi (10):
>> videodev2.h, v4l2-ioctl: add IPU3 meta buffer format
>> intel-ipu3: Add user space ABI definitions
>> intel-ipu3: css: tables
>> intel-ipu3: css: imgu dma buff pool
>> intel-ipu3: css: firmware management
>> intel-ipu3: params: compute and program ccs
>> intel-ipu3: css hardware setup
>> intel-ipu3: css pipeline
>> intel-ipu3: Add imgu v4l2 driver
>> intel-ipu3: imgu top level pci device
>>
>> drivers/media/pci/intel/ipu3/Kconfig | 33 +
>> drivers/media/pci/intel/ipu3/Makefile | 21 +
>> drivers/media/pci/intel/ipu3/ipu3-abi.h | 1579 ++++
>> drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 270 +
>> drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 206 +
>> drivers/media/pci/intel/ipu3/ipu3-css-params.c | 3161 ++++++++
>> drivers/media/pci/intel/ipu3/ipu3-css-params.h | 105 +
>> drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 132 +
>> drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 54 +
>> drivers/media/pci/intel/ipu3/ipu3-css.c | 2278 ++++++
>> drivers/media/pci/intel/ipu3/ipu3-css.h | 225 +
>> drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 342 +
>> drivers/media/pci/intel/ipu3/ipu3-dmamap.h | 33 +
>> drivers/media/pci/intel/ipu3/ipu3-mmu.c | 580 ++
>> drivers/media/pci/intel/ipu3/ipu3-mmu.h | 26 +
>> drivers/media/pci/intel/ipu3/ipu3-tables.c | 9621
>> ++++++++++++++++++++++++
>> drivers/media/pci/intel/ipu3/ipu3-tables.h | 82 +
>> drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1150 +++
>> drivers/media/pci/intel/ipu3/ipu3.c | 882 +++
>> drivers/media/pci/intel/ipu3/ipu3.h | 186 +
>> drivers/media/v4l2-core/v4l2-ioctl.c | 4 +
>> include/uapi/linux/intel-ipu3.h | 2199 ++++++
>> include/uapi/linux/videodev2.h | 6 +
>> 23 files changed, 23175 insertions(+)
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-abi.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.h
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
>> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
>> create mode 100644 include/uapi/linux/intel-ipu3.h
>>
>> --
>> 2.7.4
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-11-27 9:55 ` Hans Verkuil
@ 2017-12-05 0:13 ` Mani, Rajmohan
2017-12-05 3:22 ` Tomasz Figa
0 siblings, 1 reply; 24+ messages in thread
From: Mani, Rajmohan @ 2017-12-05 0:13 UTC (permalink / raw)
To: Hans Verkuil, Zhi, Yong, linux-media, sakari.ailus
Cc: Zheng, Jian Xu, Toivonen, Tuukka, Hu, Jerry W, arnd, hch,
robin.murphy, iommu
Hi Hans,
Thanks for your patience and sharing your thoughts on this.
> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Hi Rajmohan,
>
> On 11/17/2017 03:58 AM, Mani, Rajmohan wrote:
> > Hi Sakari and all,
> >
> >> -----Original Message-----
> >> From: Zhi, Yong
> >> Sent: Tuesday, October 17, 2017 8:47 PM
> >> To: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com
> >> Cc: Zheng, Jian Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
> >> <rajmohan.mani@intel.com>; Toivonen, Tuukka
> >> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
> >> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com; iommu@lists.linux-
> >> foundation.org; Zhi, Yong <yong.zhi@intel.com>
> >> Subject: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> >>
> >> This patchset adds support for the Intel IPU3 (Image Processing Unit)
> >> ImgU which is essentially a modern memory-to-memory ISP. It
> >> implements raw Bayer to YUV image format conversion as well as a
> >> large number of other pixel processing algorithms for improving the image
> quality.
> >>
> >> Meta data formats are defined for image statistics (3A, i.e.
> >> automatic white balance, exposure and focus, histogram and local area
> >> contrast
> >> enhancement) as well as for the pixel processing algorithm parameters.
> >> The documentation for these formats is currently not included in the
> >> patchset but will be added in a future version of this set.
> >>
> >
> > Here is an update on the IPU3 documentation that we are currently working
> on.
> >
> > Image processing in IPU3 relies on the following.
> >
> > 1) HW configuration to enable ISP and
> > 2) setting customer specific 3A Tuning / Algorithm Parameters to achieve
> desired image quality.
> >
> > We intend to provide documentation on ImgU driver programming interface
> to help users of this driver to configure and enable ISP HW to meet their
> needs. This documentation will include details on complete V4L2 Kernel driver
> interface and IO-Control parameters, except for the ISP internal algorithm and
> its parameters (which is Intel proprietary IP).
> >
> > We will also provide an user space library in binary form to help users of this
> driver, to convert the public 3A tuning parameters to IPU3 algorithm
> parameters. This tool will be released under NDA to the users of this driver.
>
> So I discussed this situation with Sakari in Prague during the ELCE. I am not
> happy with adding a driver to the kernel that needs an NDA to be usable. I
> can't agree to that. It's effectively the same as firmware that's only available
> under an NDA and we wouldn't accept such drivers either.
>
Ack
> There are a few options:
>
> 1) Document the ISP parameters and that format they are stored in allowing
> for
> open source implementations. While this is the ideal, I suspect that this is
> a no-go for Intel.
>
Ack
> 2) The driver can be used without using these ISP parameters and still achieve
> 'OK' quality. I.e., it's usable for basic webcam usage under normal lighting
> conditions. I'm not sure if this is possible at all, though.
>
This is something that we have tried and are able to do image capture with
the default ISP parameters using ov5670 sensor.
Additionally we had to set optimal values for the ov5670 sensor's exposure and
gain settings.
Please see if the following image looks to meet the 'OK' quality.
git clone https://github.com/RajmohanMani/ipu3-misc.git
ipu3-misc/ov5670.jpg is the image file.
> 3) Make the library available without requiring an NDA.
>
We are also actively exploring this option to see if this can be done.
> 4) Provide a non-NDA library (ideally open source) that achieves at minimum
> the quality as described in 2: i.e. usable for basic webcam.
>
I see this is the same as option 3) + open sourcing the library.
Open sourcing the library does not look to be an option.
I will reconfirm this.
> 5) Other solutions I didn't think of?
>
> I think 4 might be the best option, especially if it is open sourced and just uses
> non-IP 3A algorithms. It could even be added to the v4l-utils git repo.
>
> A closed source non-NDA library might also work, but you will need to check
> what Mauro thinks about that.
I believe this is option 3) that you are referring here.
Depending on the progress that we make on option 3), we can work on the
next steps.
> My problem is that such libraries tend to be
> abandoned after a few years and then bit-rot sets in. An open-source solution
> is much easier to maintain in the long term.
>
> Regards,
>
> Hans
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-05 0:13 ` Mani, Rajmohan
@ 2017-12-05 3:22 ` Tomasz Figa
2017-12-05 7:46 ` Hans Verkuil
0 siblings, 1 reply; 24+ messages in thread
From: Tomasz Figa @ 2017-12-05 3:22 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Hans Verkuil, Zhi, Yong, linux-media, sakari.ailus, Zheng,
Jian Xu, Toivonen, Tuukka, Hu, Jerry W, arnd, hch, robin.murphy,
iommu
Hi Raj,
On Tue, Dec 5, 2017 at 9:13 AM, Mani, Rajmohan <rajmohan.mani@intel.com> wrote:
> Hi Hans,
>
> Thanks for your patience and sharing your thoughts on this.
>
>> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>>
>> Hi Rajmohan,
>>
>> On 11/17/2017 03:58 AM, Mani, Rajmohan wrote:
>> > Hi Sakari and all,
>> >
>> >> -----Original Message-----
>> >> From: Zhi, Yong
>> >> Sent: Tuesday, October 17, 2017 8:47 PM
>> >> To: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com
>> >> Cc: Zheng, Jian Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
>> >> <rajmohan.mani@intel.com>; Toivonen, Tuukka
>> >> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
>> >> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com; iommu@lists.linux-
>> >> foundation.org; Zhi, Yong <yong.zhi@intel.com>
>> >> Subject: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>> >>
>> >> This patchset adds support for the Intel IPU3 (Image Processing Unit)
>> >> ImgU which is essentially a modern memory-to-memory ISP. It
>> >> implements raw Bayer to YUV image format conversion as well as a
>> >> large number of other pixel processing algorithms for improving the image
>> quality.
>> >>
>> >> Meta data formats are defined for image statistics (3A, i.e.
>> >> automatic white balance, exposure and focus, histogram and local area
>> >> contrast
>> >> enhancement) as well as for the pixel processing algorithm parameters.
>> >> The documentation for these formats is currently not included in the
>> >> patchset but will be added in a future version of this set.
>> >>
>> >
>> > Here is an update on the IPU3 documentation that we are currently working
>> on.
>> >
>> > Image processing in IPU3 relies on the following.
>> >
>> > 1) HW configuration to enable ISP and
>> > 2) setting customer specific 3A Tuning / Algorithm Parameters to achieve
>> desired image quality.
>> >
>> > We intend to provide documentation on ImgU driver programming interface
>> to help users of this driver to configure and enable ISP HW to meet their
>> needs. This documentation will include details on complete V4L2 Kernel driver
>> interface and IO-Control parameters, except for the ISP internal algorithm and
>> its parameters (which is Intel proprietary IP).
>> >
>> > We will also provide an user space library in binary form to help users of this
>> driver, to convert the public 3A tuning parameters to IPU3 algorithm
>> parameters. This tool will be released under NDA to the users of this driver.
>>
>> So I discussed this situation with Sakari in Prague during the ELCE. I am not
>> happy with adding a driver to the kernel that needs an NDA to be usable. I
>> can't agree to that. It's effectively the same as firmware that's only available
>> under an NDA and we wouldn't accept such drivers either.
>>
>
> Ack
>
>> There are a few options:
>>
>> 1) Document the ISP parameters and that format they are stored in allowing
>> for
>> open source implementations. While this is the ideal, I suspect that this is
>> a no-go for Intel.
>>
>
> Ack
>
>> 2) The driver can be used without using these ISP parameters and still achieve
>> 'OK' quality. I.e., it's usable for basic webcam usage under normal lighting
>> conditions. I'm not sure if this is possible at all, though.
>>
>
> This is something that we have tried and are able to do image capture with
> the default ISP parameters using ov5670 sensor.
> Additionally we had to set optimal values for the ov5670 sensor's exposure and
> gain settings.
Does it mean hardcoding some ov5670-specific settings in the ISP
driver? If not, I guess it might be good enough?
>
> Please see if the following image looks to meet the 'OK' quality.
>
> git clone https://github.com/RajmohanMani/ipu3-misc.git
> ipu3-misc/ov5670.jpg is the image file.
>
>> 3) Make the library available without requiring an NDA.
>>
>
> We are also actively exploring this option to see if this can be done.
>
>> 4) Provide a non-NDA library (ideally open source) that achieves at minimum
>> the quality as described in 2: i.e. usable for basic webcam.
>>
>
> I see this is the same as option 3) + open sourcing the library.
> Open sourcing the library does not look to be an option.
> I will reconfirm this.
In my understanding, that could be quite different from option 3). The
open source library would not have to implement all of the
capabilities, just enough to get the "OK" quality and the implemented
part could use some simpler algorithms not covered by IP.
Best regards,
Tomasz
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-05 3:22 ` Tomasz Figa
@ 2017-12-05 7:46 ` Hans Verkuil
2017-12-06 0:46 ` Mani, Rajmohan
0 siblings, 1 reply; 24+ messages in thread
From: Hans Verkuil @ 2017-12-05 7:46 UTC (permalink / raw)
To: Tomasz Figa, Mani, Rajmohan
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu
On 12/05/2017 04:22 AM, Tomasz Figa wrote:
> Hi Raj,
>
> On Tue, Dec 5, 2017 at 9:13 AM, Mani, Rajmohan <rajmohan.mani@intel.com> wrote:
>> Hi Hans,
>>
>> Thanks for your patience and sharing your thoughts on this.
>>
>>> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>>>
>>> Hi Rajmohan,
>>>
>>> On 11/17/2017 03:58 AM, Mani, Rajmohan wrote:
>>>> Hi Sakari and all,
>>>>
>>>>> -----Original Message-----
>>>>> From: Zhi, Yong
>>>>> Sent: Tuesday, October 17, 2017 8:47 PM
>>>>> To: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com
>>>>> Cc: Zheng, Jian Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
>>>>> <rajmohan.mani@intel.com>; Toivonen, Tuukka
>>>>> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
>>>>> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com; iommu@lists.linux-
>>>>> foundation.org; Zhi, Yong <yong.zhi@intel.com>
>>>>> Subject: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>>>>>
>>>>> This patchset adds support for the Intel IPU3 (Image Processing Unit)
>>>>> ImgU which is essentially a modern memory-to-memory ISP. It
>>>>> implements raw Bayer to YUV image format conversion as well as a
>>>>> large number of other pixel processing algorithms for improving the image
>>> quality.
>>>>>
>>>>> Meta data formats are defined for image statistics (3A, i.e.
>>>>> automatic white balance, exposure and focus, histogram and local area
>>>>> contrast
>>>>> enhancement) as well as for the pixel processing algorithm parameters.
>>>>> The documentation for these formats is currently not included in the
>>>>> patchset but will be added in a future version of this set.
>>>>>
>>>>
>>>> Here is an update on the IPU3 documentation that we are currently working
>>> on.
>>>>
>>>> Image processing in IPU3 relies on the following.
>>>>
>>>> 1) HW configuration to enable ISP and
>>>> 2) setting customer specific 3A Tuning / Algorithm Parameters to achieve
>>> desired image quality.
>>>>
>>>> We intend to provide documentation on ImgU driver programming interface
>>> to help users of this driver to configure and enable ISP HW to meet their
>>> needs. This documentation will include details on complete V4L2 Kernel driver
>>> interface and IO-Control parameters, except for the ISP internal algorithm and
>>> its parameters (which is Intel proprietary IP).
>>>>
>>>> We will also provide an user space library in binary form to help users of this
>>> driver, to convert the public 3A tuning parameters to IPU3 algorithm
>>> parameters. This tool will be released under NDA to the users of this driver.
>>>
>>> So I discussed this situation with Sakari in Prague during the ELCE. I am not
>>> happy with adding a driver to the kernel that needs an NDA to be usable. I
>>> can't agree to that. It's effectively the same as firmware that's only available
>>> under an NDA and we wouldn't accept such drivers either.
>>>
>>
>> Ack
>>
>>> There are a few options:
>>>
>>> 1) Document the ISP parameters and that format they are stored in allowing
>>> for
>>> open source implementations. While this is the ideal, I suspect that this is
>>> a no-go for Intel.
>>>
>>
>> Ack
>>
>>> 2) The driver can be used without using these ISP parameters and still achieve
>>> 'OK' quality. I.e., it's usable for basic webcam usage under normal lighting
>>> conditions. I'm not sure if this is possible at all, though.
>>>
>>
>> This is something that we have tried and are able to do image capture with
>> the default ISP parameters using ov5670 sensor.
>> Additionally we had to set optimal values for the ov5670 sensor's exposure and
>> gain settings.
>
> Does it mean hardcoding some ov5670-specific settings in the ISP
> driver? If not, I guess it might be good enough?
It doesn't look too bad, but it's hard to tell from just a single
frame. I think you can work with Sakari on this, if he also thinks it's
good enough, then I am happy with that.
>
>>
>> Please see if the following image looks to meet the 'OK' quality.
>>
>> git clone https://github.com/RajmohanMani/ipu3-misc.git
>> ipu3-misc/ov5670.jpg is the image file.
>>
>>> 3) Make the library available without requiring an NDA.
>>>
>>
>> We are also actively exploring this option to see if this can be done.
>>
>>> 4) Provide a non-NDA library (ideally open source) that achieves at minimum
>>> the quality as described in 2: i.e. usable for basic webcam.
>>>
>>
>> I see this is the same as option 3) + open sourcing the library.
>> Open sourcing the library does not look to be an option.
>> I will reconfirm this.
>
> In my understanding, that could be quite different from option 3). The
> open source library would not have to implement all of the
> capabilities, just enough to get the "OK" quality and the implemented
> part could use some simpler algorithms not covered by IP.
That's correct.
This also solves my concerns about long-term maintenance of closed source
libraries which in my experience tend to suffer from bit-rot after the vendor
no longer sells the hardware it was written for. An open source implementation,
albeit not as powerful as the NDA version, can be added to e.g. the v4l-utils
repo (or at least at a repo on linuxtv.org) and maintained much longer and
independent of the vendor.
Regards,
Hans
>
> Best regards,
> Tomasz
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-05 7:46 ` Hans Verkuil
@ 2017-12-06 0:46 ` Mani, Rajmohan
0 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2017-12-06 0:46 UTC (permalink / raw)
To: Hans Verkuil, Tomasz Figa
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu
Hi Tomasz, Hans,
> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> On 12/05/2017 04:22 AM, Tomasz Figa wrote:
> > Hi Raj,
> >
> > On Tue, Dec 5, 2017 at 9:13 AM, Mani, Rajmohan
> <rajmohan.mani@intel.com> wrote:
> >> Hi Hans,
> >>
> >> Thanks for your patience and sharing your thoughts on this.
> >>
> >>> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> >>>
> >>> Hi Rajmohan,
> >>>
> >>> On 11/17/2017 03:58 AM, Mani, Rajmohan wrote:
> >>>> Hi Sakari and all,
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Zhi, Yong
> >>>>> Sent: Tuesday, October 17, 2017 8:47 PM
> >>>>> To: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com
> >>>>> Cc: Zheng, Jian Xu <jian.xu.zheng@intel.com>; Mani, Rajmohan
> >>>>> <rajmohan.mani@intel.com>; Toivonen, Tuukka
> >>>>> <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
> >>>>> arnd@arndb.de; hch@lst.de; robin.murphy@arm.com;
> >>>>> iommu@lists.linux- foundation.org; Zhi, Yong <yong.zhi@intel.com>
> >>>>> Subject: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> >>>>>
> >>>>> This patchset adds support for the Intel IPU3 (Image Processing
> >>>>> Unit) ImgU which is essentially a modern memory-to-memory ISP. It
> >>>>> implements raw Bayer to YUV image format conversion as well as a
> >>>>> large number of other pixel processing algorithms for improving
> >>>>> the image
> >>> quality.
> >>>>>
> >>>>> Meta data formats are defined for image statistics (3A, i.e.
> >>>>> automatic white balance, exposure and focus, histogram and local
> >>>>> area contrast
> >>>>> enhancement) as well as for the pixel processing algorithm parameters.
> >>>>> The documentation for these formats is currently not included in
> >>>>> the patchset but will be added in a future version of this set.
> >>>>>
> >>>>
> >>>> Here is an update on the IPU3 documentation that we are currently
> >>>> working
> >>> on.
> >>>>
> >>>> Image processing in IPU3 relies on the following.
> >>>>
> >>>> 1) HW configuration to enable ISP and
> >>>> 2) setting customer specific 3A Tuning / Algorithm Parameters to
> >>>> achieve
> >>> desired image quality.
> >>>>
> >>>> We intend to provide documentation on ImgU driver programming
> >>>> interface
> >>> to help users of this driver to configure and enable ISP HW to meet
> >>> their needs. This documentation will include details on complete
> >>> V4L2 Kernel driver interface and IO-Control parameters, except for
> >>> the ISP internal algorithm and its parameters (which is Intel proprietary IP).
> >>>>
> >>>> We will also provide an user space library in binary form to help
> >>>> users of this
> >>> driver, to convert the public 3A tuning parameters to IPU3 algorithm
> >>> parameters. This tool will be released under NDA to the users of this
> driver.
> >>>
> >>> So I discussed this situation with Sakari in Prague during the ELCE.
> >>> I am not happy with adding a driver to the kernel that needs an NDA
> >>> to be usable. I can't agree to that. It's effectively the same as
> >>> firmware that's only available under an NDA and we wouldn't accept such
> drivers either.
> >>>
> >>
> >> Ack
> >>
> >>> There are a few options:
> >>>
> >>> 1) Document the ISP parameters and that format they are stored in
> >>> allowing for
> >>> open source implementations. While this is the ideal, I suspect that this
> is
> >>> a no-go for Intel.
> >>>
> >>
> >> Ack
> >>
> >>> 2) The driver can be used without using these ISP parameters and still
> achieve
> >>> 'OK' quality. I.e., it's usable for basic webcam usage under normal lighting
> >>> conditions. I'm not sure if this is possible at all, though.
> >>>
> >>
> >> This is something that we have tried and are able to do image capture
> >> with the default ISP parameters using ov5670 sensor.
> >> Additionally we had to set optimal values for the ov5670 sensor's
> >> exposure and gain settings.
By the above, I meant to say that optimal values of exposure and gain
settings were set in / through the ov5670 sensor driver.
> >
> > Does it mean hardcoding some ov5670-specific settings in the ISP
> > driver? If not, I guess it might be good enough?
>
No. There is no hardcoding of any sensor specific settings in the ISP driver.
Camera HAL / user space applications are still expected to tune sensor
Specific settings (Auto focus and Auto exposure) for 3A and parameters.
Only Auto white balance is adjusted by default ISP parameters, with which
this image was captured.
> It doesn't look too bad, but it's hard to tell from just a single frame. I think you
> can work with Sakari on this, if he also thinks it's good enough, then I am
> happy with that.
>
Ack
> >
> >>
> >> Please see if the following image looks to meet the 'OK' quality.
> >>
> >> git clone https://github.com/RajmohanMani/ipu3-misc.git
> >> ipu3-misc/ov5670.jpg is the image file.
> >>
> >>> 3) Make the library available without requiring an NDA.
> >>>
> >>
> >> We are also actively exploring this option to see if this can be done.
> >>
> >>> 4) Provide a non-NDA library (ideally open source) that achieves at
> minimum
> >>> the quality as described in 2: i.e. usable for basic webcam.
> >>>
> >>
> >> I see this is the same as option 3) + open sourcing the library.
> >> Open sourcing the library does not look to be an option.
> >> I will reconfirm this.
> >
> > In my understanding, that could be quite different from option 3). The
> > open source library would not have to implement all of the
> > capabilities, just enough to get the "OK" quality and the implemented
> > part could use some simpler algorithms not covered by IP.
>
> That's correct.
>
Thanks for the clarifications here.
We will also be looking into this option.
> This also solves my concerns about long-term maintenance of closed source
> libraries which in my experience tend to suffer from bit-rot after the vendor no
> longer sells the hardware it was written for. An open source implementation,
> albeit not as powerful as the NDA version, can be added to e.g. the v4l-utils
> repo (or at least at a repo on linuxtv.org) and maintained much longer and
> independent of the vendor.
>
Ack
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-11-17 2:58 ` Mani, Rajmohan
2017-11-27 9:55 ` Hans Verkuil
@ 2017-12-20 13:57 ` Mauro Carvalho Chehab
2017-12-26 22:30 ` Mani, Rajmohan
` (4 more replies)
1 sibling, 5 replies; 24+ messages in thread
From: Mauro Carvalho Chehab @ 2017-12-20 13:57 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu
Hi,
Em Fri, 17 Nov 2017 02:58:56 +0000
"Mani, Rajmohan" <rajmohan.mani@intel.com> escreveu:
> Here is an update on the IPU3 documentation that we are currently working on.
>
> Image processing in IPU3 relies on the following.
>
> 1) HW configuration to enable ISP and
> 2) setting customer specific 3A Tuning / Algorithm Parameters to achieve desired image quality.
>
> We intend to provide documentation on ImgU driver programming interface to help users of this driver to configure and enable ISP HW to meet their needs. This documentation will include details on complete V4L2 Kernel driver interface and IO-Control parameters, except for the ISP internal algorithm and its parameters (which is Intel proprietary IP).
Sakari asked me to take a look on this thread, specifically on this
email. I took a look on the other e-mails from this thread that are
discussing about this IP block.
I understand that Intel wants to keep their internal 3A algorithm
protected, just like other vendors protect their own algos. It was never
a requirement to open whatever algorithm are used inside a hardware
(or firmware). The only requirement is that firmwares should be licensed
with redistribution permission, ideally merged at linux-firmware git
tree.
Yet, what I don't understand is why Intel also wants to also protect
the interface for such 3A hardware/firmware algorithm. The parameters
that are passed from an userspace application to Intel ISP logic
doesn't contain the algorithm itself. What's the issue of documenting
the meaning of each parameter?
Thanks,
Mauro
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-20 13:57 ` Mauro Carvalho Chehab
@ 2017-12-26 22:30 ` Mani, Rajmohan
2018-02-07 23:02 ` Mani, Rajmohan
` (3 subsequent siblings)
4 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2017-12-26 22:30 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu
Hi Mauro,
> -----Original Message-----
> From: Mauro Carvalho Chehab [mailto:mchehab@s-opensource.com]
> Sent: Wednesday, December 20, 2017 5:58 AM
> To: Mani, Rajmohan <rajmohan.mani@intel.com>
> Cc: Zhi, Yong <yong.zhi@intel.com>; linux-media@vger.kernel.org;
> sakari.ailus@linux.intel.com; Zheng, Jian Xu <jian.xu.zheng@intel.com>;
> Toivonen, Tuukka <tuukka.toivonen@intel.com>; Hu, Jerry W
> <jerry.w.hu@intel.com>; arnd@arndb.de; hch@lst.de;
> robin.murphy@arm.com; iommu@lists.linux-foundation.org
> Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Hi,
>
> Em Fri, 17 Nov 2017 02:58:56 +0000
> "Mani, Rajmohan" <rajmohan.mani@intel.com> escreveu:
>
> > Here is an update on the IPU3 documentation that we are currently working
> on.
> >
> > Image processing in IPU3 relies on the following.
> >
> > 1) HW configuration to enable ISP and
> > 2) setting customer specific 3A Tuning / Algorithm Parameters to achieve
> desired image quality.
> >
> > We intend to provide documentation on ImgU driver programming interface
> to help users of this driver to configure and enable ISP HW to meet their
> needs. This documentation will include details on complete V4L2 Kernel driver
> interface and IO-Control parameters, except for the ISP internal algorithm and
> its parameters (which is Intel proprietary IP).
>
> Sakari asked me to take a look on this thread, specifically on this email. I took a
> look on the other e-mails from this thread that are discussing about this IP
> block.
>
> I understand that Intel wants to keep their internal 3A algorithm protected,
> just like other vendors protect their own algos. It was never a requirement to
> open whatever algorithm are used inside a hardware (or firmware). The only
> requirement is that firmwares should be licensed with redistribution
> permission, ideally merged at linux-firmware git tree.
>
> Yet, what I don't understand is why Intel also wants to also protect the
> interface for such 3A hardware/firmware algorithm. The parameters that are
> passed from an userspace application to Intel ISP logic doesn't contain the
> algorithm itself. What's the issue of documenting the meaning of each
> parameter?
>
Thanks for looking into this.
To achieve improved image quality using IPU3, 3A (Auto White balance, Auto Focus
and Auto Exposure) Tuning parameters specific to a given camera sensor module,
are converted to Intel ISP algorithm parameters in user space camera HAL using
AIC (Automatic ISP Configuration) library.
As a unique design of Intel ISP, it exposes very detailed algorithm parameters
(~ 10000 parameters) to configure ISP's image processing algorithm per each
image fame in runtime. Typical Camera SW developers (including those at Intel)
are not expected to fully understand and directly set these parameters to
configure the ISP algorithm blocks. Due to the above, a user space AIC library
(in binary form) is provided to generate ISP Algorithm specific parameters, for
a given set of 3A tuning parameters. It significantly reduces the efforts of SW
development in ISP HW configuration.
On the other hand, the ISP algorithm details could be deduced readily through
these detailed parameters by other ISP experts outside Intel. This is the reason
that we want to keep these parameter definitions as Intel proprietary IP.
We are fully aware of your concerns on how to enable open source developers
to use Intel ISP through up-streamed Kernel Driver. Internally, we are working
on the license for this AIC library release now (as Hans said NDA license is not
acceptable). We believe this will be more efficient way to help open source
developers.
This AIC library release would be a binary-only release. This AIC library does
not use any kernel uAPIs directly. The user space Camera HAL that uses kernel
uAPIs is available at
https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master
Thanks
Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-20 13:57 ` Mauro Carvalho Chehab
2017-12-26 22:30 ` Mani, Rajmohan
@ 2018-02-07 23:02 ` Mani, Rajmohan
2018-02-21 1:56 ` Mani, Rajmohan
` (2 subsequent siblings)
4 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2018-02-07 23:02 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu
Hi Mauro,
> -----Original Message-----
> From: Mani, Rajmohan
> Sent: Tuesday, December 26, 2017 2:31 PM
> To: 'Mauro Carvalho Chehab' <mchehab@s-opensource.com>
> Cc: Zhi, Yong <yong.zhi@intel.com>; linux-media@vger.kernel.org;
> sakari.ailus@linux.intel.com; Zheng, Jian Xu <jian.xu.zheng@intel.com>;
> Toivonen, Tuukka <tuukka.toivonen@intel.com>; Hu, Jerry W
> <jerry.w.hu@intel.com>; arnd@arndb.de; hch@lst.de;
> robin.murphy@arm.com; iommu@lists.linux-foundation.org
> Subject: RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Hi Mauro,
>
> > -----Original Message-----
> > From: Mauro Carvalho Chehab [mailto:mchehab@s-opensource.com]
> > Sent: Wednesday, December 20, 2017 5:58 AM
> > To: Mani, Rajmohan <rajmohan.mani@intel.com>
> > Cc: Zhi, Yong <yong.zhi@intel.com>; linux-media@vger.kernel.org;
> > sakari.ailus@linux.intel.com; Zheng, Jian Xu
> > <jian.xu.zheng@intel.com>; Toivonen, Tuukka
> > <tuukka.toivonen@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
> > arnd@arndb.de; hch@lst.de; robin.murphy@arm.com;
> > iommu@lists.linux-foundation.org
> > Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> >
> > Hi,
> >
> > Em Fri, 17 Nov 2017 02:58:56 +0000
> > "Mani, Rajmohan" <rajmohan.mani@intel.com> escreveu:
> >
> > > Here is an update on the IPU3 documentation that we are currently
> > > working
> > on.
> > >
> > > Image processing in IPU3 relies on the following.
> > >
> > > 1) HW configuration to enable ISP and
> > > 2) setting customer specific 3A Tuning / Algorithm Parameters to
> > > achieve
> > desired image quality.
> > >
> > > We intend to provide documentation on ImgU driver programming
> > > interface
> > to help users of this driver to configure and enable ISP HW to meet
> > their needs. This documentation will include details on complete V4L2
> > Kernel driver interface and IO-Control parameters, except for the ISP
> > internal algorithm and its parameters (which is Intel proprietary IP).
> >
> > Sakari asked me to take a look on this thread, specifically on this
> > email. I took a look on the other e-mails from this thread that are
> > discussing about this IP block.
> >
> > I understand that Intel wants to keep their internal 3A algorithm
> > protected, just like other vendors protect their own algos. It was
> > never a requirement to open whatever algorithm are used inside a
> > hardware (or firmware). The only requirement is that firmwares should
> > be licensed with redistribution permission, ideally merged at linux-firmware
> git tree.
> >
> > Yet, what I don't understand is why Intel also wants to also protect
> > the interface for such 3A hardware/firmware algorithm. The parameters
> > that are passed from an userspace application to Intel ISP logic
> > doesn't contain the algorithm itself. What's the issue of documenting
> > the meaning of each parameter?
> >
>
> Thanks for looking into this.
>
> To achieve improved image quality using IPU3, 3A (Auto White balance, Auto
> Focus and Auto Exposure) Tuning parameters specific to a given camera sensor
> module, are converted to Intel ISP algorithm parameters in user space camera
> HAL using AIC (Automatic ISP Configuration) library.
>
> As a unique design of Intel ISP, it exposes very detailed algorithm parameters
> (~ 10000 parameters) to configure ISP's image processing algorithm per each
> image fame in runtime. Typical Camera SW developers (including those at
> Intel) are not expected to fully understand and directly set these parameters to
> configure the ISP algorithm blocks. Due to the above, a user space AIC library
> (in binary form) is provided to generate ISP Algorithm specific parameters, for a
> given set of 3A tuning parameters. It significantly reduces the efforts of SW
> development in ISP HW configuration.
>
> On the other hand, the ISP algorithm details could be deduced readily through
> these detailed parameters by other ISP experts outside Intel. This is the reason
> that we want to keep these parameter definitions as Intel proprietary IP.
>
> We are fully aware of your concerns on how to enable open source developers
> to use Intel ISP through up-streamed Kernel Driver. Internally, we are working
> on the license for this AIC library release now (as Hans said NDA license is not
> acceptable). We believe this will be more efficient way to help open source
> developers.
>
> This AIC library release would be a binary-only release. This AIC library does not
> use any kernel uAPIs directly. The user space Camera HAL that uses kernel
> uAPIs is available at
> https://chromium.googlesource.com/chromiumos/platform/arc-
> camera/+/master
>
Just pinging to know your thoughts on this.
Thanks
Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-20 13:57 ` Mauro Carvalho Chehab
@ 2018-02-21 1:56 ` Mani, Rajmohan
2018-02-07 23:02 ` Mani, Rajmohan
` (3 subsequent siblings)
4 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2018-02-21 1:56 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu
Hi Mauro,
> > > Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> > >
> > > Hi,
> > >
> > > Em Fri, 17 Nov 2017 02:58:56 +0000
> > > "Mani, Rajmohan" <rajmohan.mani@intel.com> escreveu:
> > >
> > > > Here is an update on the IPU3 documentation that we are currently
> > > > working
> > > on.
> > > >
> > > > Image processing in IPU3 relies on the following.
> > > >
> > > > 1) HW configuration to enable ISP and
> > > > 2) setting customer specific 3A Tuning / Algorithm Parameters to
> > > > achieve
> > > desired image quality.
> > > >
> > > > We intend to provide documentation on ImgU driver programming
> > > > interface
> > > to help users of this driver to configure and enable ISP HW to meet
> > > their needs. This documentation will include details on complete
> > > V4L2 Kernel driver interface and IO-Control parameters, except for
> > > the ISP internal algorithm and its parameters (which is Intel proprietary IP).
> > >
> > > Sakari asked me to take a look on this thread, specifically on this
> > > email. I took a look on the other e-mails from this thread that are
> > > discussing about this IP block.
> > >
> > > I understand that Intel wants to keep their internal 3A algorithm
> > > protected, just like other vendors protect their own algos. It was
> > > never a requirement to open whatever algorithm are used inside a
> > > hardware (or firmware). The only requirement is that firmwares
> > > should be licensed with redistribution permission, ideally merged at
> > > linux-firmware
> > git tree.
> > >
> > > Yet, what I don't understand is why Intel also wants to also protect
> > > the interface for such 3A hardware/firmware algorithm. The
> > > parameters that are passed from an userspace application to Intel
> > > ISP logic doesn't contain the algorithm itself. What's the issue of
> > > documenting the meaning of each parameter?
> > >
> >
> > Thanks for looking into this.
> >
> > To achieve improved image quality using IPU3, 3A (Auto White balance,
> > Auto Focus and Auto Exposure) Tuning parameters specific to a given
> > camera sensor module, are converted to Intel ISP algorithm parameters
> > in user space camera HAL using AIC (Automatic ISP Configuration) library.
> >
> > As a unique design of Intel ISP, it exposes very detailed algorithm
> > parameters (~ 10000 parameters) to configure ISP's image processing
> > algorithm per each image fame in runtime. Typical Camera SW developers
> > (including those at
> > Intel) are not expected to fully understand and directly set these
> > parameters to configure the ISP algorithm blocks. Due to the above, a
> > user space AIC library (in binary form) is provided to generate ISP
> > Algorithm specific parameters, for a given set of 3A tuning
> > parameters. It significantly reduces the efforts of SW development in ISP HW
> configuration.
> >
> > On the other hand, the ISP algorithm details could be deduced readily
> > through these detailed parameters by other ISP experts outside Intel.
> > This is the reason that we want to keep these parameter definitions as Intel
> proprietary IP.
> >
> > We are fully aware of your concerns on how to enable open source
> > developers to use Intel ISP through up-streamed Kernel Driver.
> > Internally, we are working on the license for this AIC library release
> > now (as Hans said NDA license is not acceptable). We believe this will
> > be more efficient way to help open source developers.
> >
> > This AIC library release would be a binary-only release. This AIC
> > library does not use any kernel uAPIs directly. The user space Camera
> > HAL that uses kernel uAPIs is available at
> > https://chromium.googlesource.com/chromiumos/platform/arc-
> > camera/+/master
> >
The AIC library (in binary form) is available here.
https://storage.googleapis.com/chromeos-localmirror/distfiles/intel-3a-libs-bin-2017.09.27.tbz2
Licensing information can be found in ./LICENSE.intel_3a_library file after unzipping the tar file.
>
> Just pinging to know your thoughts on this.
>
> Thanks
> Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2018-02-21 1:56 ` Mani, Rajmohan
0 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2018-02-21 1:56 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: arnd-r2nGTMty4D4, Hu, Jerry W, Zheng, Jian Xu,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Toivonen,
Tuukka, Zhi, Yong, hch-jcswGhMUV9g,
linux-media-u79uwXL29TY76Z2rM5mHXA
Hi Mauro,
> > > Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> > >
> > > Hi,
> > >
> > > Em Fri, 17 Nov 2017 02:58:56 +0000
> > > "Mani, Rajmohan" <rajmohan.mani-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> escreveu:
> > >
> > > > Here is an update on the IPU3 documentation that we are currently
> > > > working
> > > on.
> > > >
> > > > Image processing in IPU3 relies on the following.
> > > >
> > > > 1) HW configuration to enable ISP and
> > > > 2) setting customer specific 3A Tuning / Algorithm Parameters to
> > > > achieve
> > > desired image quality.
> > > >
> > > > We intend to provide documentation on ImgU driver programming
> > > > interface
> > > to help users of this driver to configure and enable ISP HW to meet
> > > their needs. This documentation will include details on complete
> > > V4L2 Kernel driver interface and IO-Control parameters, except for
> > > the ISP internal algorithm and its parameters (which is Intel proprietary IP).
> > >
> > > Sakari asked me to take a look on this thread, specifically on this
> > > email. I took a look on the other e-mails from this thread that are
> > > discussing about this IP block.
> > >
> > > I understand that Intel wants to keep their internal 3A algorithm
> > > protected, just like other vendors protect their own algos. It was
> > > never a requirement to open whatever algorithm are used inside a
> > > hardware (or firmware). The only requirement is that firmwares
> > > should be licensed with redistribution permission, ideally merged at
> > > linux-firmware
> > git tree.
> > >
> > > Yet, what I don't understand is why Intel also wants to also protect
> > > the interface for such 3A hardware/firmware algorithm. The
> > > parameters that are passed from an userspace application to Intel
> > > ISP logic doesn't contain the algorithm itself. What's the issue of
> > > documenting the meaning of each parameter?
> > >
> >
> > Thanks for looking into this.
> >
> > To achieve improved image quality using IPU3, 3A (Auto White balance,
> > Auto Focus and Auto Exposure) Tuning parameters specific to a given
> > camera sensor module, are converted to Intel ISP algorithm parameters
> > in user space camera HAL using AIC (Automatic ISP Configuration) library.
> >
> > As a unique design of Intel ISP, it exposes very detailed algorithm
> > parameters (~ 10000 parameters) to configure ISP's image processing
> > algorithm per each image fame in runtime. Typical Camera SW developers
> > (including those at
> > Intel) are not expected to fully understand and directly set these
> > parameters to configure the ISP algorithm blocks. Due to the above, a
> > user space AIC library (in binary form) is provided to generate ISP
> > Algorithm specific parameters, for a given set of 3A tuning
> > parameters. It significantly reduces the efforts of SW development in ISP HW
> configuration.
> >
> > On the other hand, the ISP algorithm details could be deduced readily
> > through these detailed parameters by other ISP experts outside Intel.
> > This is the reason that we want to keep these parameter definitions as Intel
> proprietary IP.
> >
> > We are fully aware of your concerns on how to enable open source
> > developers to use Intel ISP through up-streamed Kernel Driver.
> > Internally, we are working on the license for this AIC library release
> > now (as Hans said NDA license is not acceptable). We believe this will
> > be more efficient way to help open source developers.
> >
> > This AIC library release would be a binary-only release. This AIC
> > library does not use any kernel uAPIs directly. The user space Camera
> > HAL that uses kernel uAPIs is available at
> > https://chromium.googlesource.com/chromiumos/platform/arc-
> > camera/+/master
> >
The AIC library (in binary form) is available here.
https://storage.googleapis.com/chromeos-localmirror/distfiles/intel-3a-libs-bin-2017.09.27.tbz2
Licensing information can be found in ./LICENSE.intel_3a_library file after unzipping the tar file.
>
> Just pinging to know your thoughts on this.
>
> Thanks
> Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-20 13:57 ` Mauro Carvalho Chehab
@ 2018-03-28 3:14 ` Mani, Rajmohan
2018-02-07 23:02 ` Mani, Rajmohan
` (3 subsequent siblings)
4 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2018-03-28 3:14 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu, tfiga
Adding Tomasz...
> -----Original Message-----
> From: Mani, Rajmohan
> Sent: Tuesday, February 20, 2018 5:56 PM
> To: 'Mauro Carvalho Chehab' <mchehab@s-opensource.com>
> Cc: Zhi, Yong <yong.zhi@intel.com>; 'linux-media@vger.kernel.org' <linux-
> media@vger.kernel.org>; 'sakari.ailus@linux.intel.com'
> <sakari.ailus@linux.intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>;
> Toivonen, Tuukka <tuukka.toivonen@intel.com>; Hu, Jerry W
> <jerry.w.hu@intel.com>; 'arnd@arndb.de' <arnd@arndb.de>; 'hch@lst.de'
> <hch@lst.de>; 'robin.murphy@arm.com' <robin.murphy@arm.com>;
> 'iommu@lists.linux-foundation.org' <iommu@lists.linux-foundation.org>
> Subject: RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Hi Mauro,
>
> > > > Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> > > >
> > > > Hi,
> > > >
> > > > Em Fri, 17 Nov 2017 02:58:56 +0000 "Mani, Rajmohan"
> > > > <rajmohan.mani@intel.com> escreveu:
> > > >
> > > > > Here is an update on the IPU3 documentation that we are
> > > > > currently working
> > > > on.
> > > > >
> > > > > Image processing in IPU3 relies on the following.
> > > > >
> > > > > 1) HW configuration to enable ISP and
> > > > > 2) setting customer specific 3A Tuning / Algorithm Parameters to
> > > > > achieve
> > > > desired image quality.
> > > > >
> > > > > We intend to provide documentation on ImgU driver programming
> > > > > interface
> > > > to help users of this driver to configure and enable ISP HW to
> > > > meet their needs. This documentation will include details on
> > > > complete
> > > > V4L2 Kernel driver interface and IO-Control parameters, except for
> > > > the ISP internal algorithm and its parameters (which is Intel proprietary
> IP).
> > > >
> > > > Sakari asked me to take a look on this thread, specifically on
> > > > this email. I took a look on the other e-mails from this thread
> > > > that are discussing about this IP block.
> > > >
> > > > I understand that Intel wants to keep their internal 3A algorithm
> > > > protected, just like other vendors protect their own algos. It was
> > > > never a requirement to open whatever algorithm are used inside a
> > > > hardware (or firmware). The only requirement is that firmwares
> > > > should be licensed with redistribution permission, ideally merged
> > > > at linux-firmware
> > > git tree.
> > > >
> > > > Yet, what I don't understand is why Intel also wants to also
> > > > protect the interface for such 3A hardware/firmware algorithm. The
> > > > parameters that are passed from an userspace application to Intel
> > > > ISP logic doesn't contain the algorithm itself. What's the issue
> > > > of documenting the meaning of each parameter?
> > > >
> > >
> > > Thanks for looking into this.
> > >
> > > To achieve improved image quality using IPU3, 3A (Auto White
> > > balance, Auto Focus and Auto Exposure) Tuning parameters specific to
> > > a given camera sensor module, are converted to Intel ISP algorithm
> > > parameters in user space camera HAL using AIC (Automatic ISP
> Configuration) library.
> > >
> > > As a unique design of Intel ISP, it exposes very detailed algorithm
> > > parameters (~ 10000 parameters) to configure ISP's image processing
> > > algorithm per each image fame in runtime. Typical Camera SW
> > > developers (including those at
> > > Intel) are not expected to fully understand and directly set these
> > > parameters to configure the ISP algorithm blocks. Due to the above,
> > > a user space AIC library (in binary form) is provided to generate
> > > ISP Algorithm specific parameters, for a given set of 3A tuning
> > > parameters. It significantly reduces the efforts of SW development
> > > in ISP HW
> > configuration.
> > >
> > > On the other hand, the ISP algorithm details could be deduced
> > > readily through these detailed parameters by other ISP experts outside
> Intel.
> > > This is the reason that we want to keep these parameter definitions
> > > as Intel
> > proprietary IP.
> > >
> > > We are fully aware of your concerns on how to enable open source
> > > developers to use Intel ISP through up-streamed Kernel Driver.
> > > Internally, we are working on the license for this AIC library
> > > release now (as Hans said NDA license is not acceptable). We believe
> > > this will be more efficient way to help open source developers.
> > >
> > > This AIC library release would be a binary-only release. This AIC
> > > library does not use any kernel uAPIs directly. The user space
> > > Camera HAL that uses kernel uAPIs is available at
> > > https://chromium.googlesource.com/chromiumos/platform/arc-
> > > camera/+/master
> > >
>
> The AIC library (in binary form) is available here.
> https://storage.googleapis.com/chromeos-localmirror/distfiles/intel-3a-libs-
> bin-2017.09.27.tbz2
>
> Licensing information can be found in ./LICENSE.intel_3a_library file after
> unzipping the tar file.
>
> >
> > Just pinging to know your thoughts on this.
> >
> > Thanks
> > Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2018-03-28 3:14 ` Mani, Rajmohan
0 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2018-03-28 3:14 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: arnd-r2nGTMty4D4, Hu, Jerry W, Zheng, Jian Xu,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Toivonen,
Tuukka, Zhi, Yong, hch-jcswGhMUV9g,
linux-media-u79uwXL29TY76Z2rM5mHXA
Adding Tomasz...
> -----Original Message-----
> From: Mani, Rajmohan
> Sent: Tuesday, February 20, 2018 5:56 PM
> To: 'Mauro Carvalho Chehab' <mchehab-JsYNTwtnfakRB7SZvlqPiA@public.gmane.org>
> Cc: Zhi, Yong <yong.zhi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; 'linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org' <linux-
> media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>; 'sakari.ailus-VuQAYsv1563Yd54FQh9/CA@public.gmane.org'
> <sakari.ailus-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>; Zheng, Jian Xu <jian.xu.zheng-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>;
> Toivonen, Tuukka <tuukka.toivonen-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; Hu, Jerry W
> <jerry.w.hu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; 'arnd-r2nGTMty4D4@public.gmane.org' <arnd-r2nGTMty4D4@public.gmane.org>; 'hch-jcswGhMUV9g@public.gmane.org'
> <hch-jcswGhMUV9g@public.gmane.org>; 'robin.murphy-5wv7dgnIgG8@public.gmane.org' <robin.murphy-5wv7dgnIgG8@public.gmane.org>;
> 'iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org' <iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org>
> Subject: RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
>
> Hi Mauro,
>
> > > > Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> > > >
> > > > Hi,
> > > >
> > > > Em Fri, 17 Nov 2017 02:58:56 +0000 "Mani, Rajmohan"
> > > > <rajmohan.mani-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> escreveu:
> > > >
> > > > > Here is an update on the IPU3 documentation that we are
> > > > > currently working
> > > > on.
> > > > >
> > > > > Image processing in IPU3 relies on the following.
> > > > >
> > > > > 1) HW configuration to enable ISP and
> > > > > 2) setting customer specific 3A Tuning / Algorithm Parameters to
> > > > > achieve
> > > > desired image quality.
> > > > >
> > > > > We intend to provide documentation on ImgU driver programming
> > > > > interface
> > > > to help users of this driver to configure and enable ISP HW to
> > > > meet their needs. This documentation will include details on
> > > > complete
> > > > V4L2 Kernel driver interface and IO-Control parameters, except for
> > > > the ISP internal algorithm and its parameters (which is Intel proprietary
> IP).
> > > >
> > > > Sakari asked me to take a look on this thread, specifically on
> > > > this email. I took a look on the other e-mails from this thread
> > > > that are discussing about this IP block.
> > > >
> > > > I understand that Intel wants to keep their internal 3A algorithm
> > > > protected, just like other vendors protect their own algos. It was
> > > > never a requirement to open whatever algorithm are used inside a
> > > > hardware (or firmware). The only requirement is that firmwares
> > > > should be licensed with redistribution permission, ideally merged
> > > > at linux-firmware
> > > git tree.
> > > >
> > > > Yet, what I don't understand is why Intel also wants to also
> > > > protect the interface for such 3A hardware/firmware algorithm. The
> > > > parameters that are passed from an userspace application to Intel
> > > > ISP logic doesn't contain the algorithm itself. What's the issue
> > > > of documenting the meaning of each parameter?
> > > >
> > >
> > > Thanks for looking into this.
> > >
> > > To achieve improved image quality using IPU3, 3A (Auto White
> > > balance, Auto Focus and Auto Exposure) Tuning parameters specific to
> > > a given camera sensor module, are converted to Intel ISP algorithm
> > > parameters in user space camera HAL using AIC (Automatic ISP
> Configuration) library.
> > >
> > > As a unique design of Intel ISP, it exposes very detailed algorithm
> > > parameters (~ 10000 parameters) to configure ISP's image processing
> > > algorithm per each image fame in runtime. Typical Camera SW
> > > developers (including those at
> > > Intel) are not expected to fully understand and directly set these
> > > parameters to configure the ISP algorithm blocks. Due to the above,
> > > a user space AIC library (in binary form) is provided to generate
> > > ISP Algorithm specific parameters, for a given set of 3A tuning
> > > parameters. It significantly reduces the efforts of SW development
> > > in ISP HW
> > configuration.
> > >
> > > On the other hand, the ISP algorithm details could be deduced
> > > readily through these detailed parameters by other ISP experts outside
> Intel.
> > > This is the reason that we want to keep these parameter definitions
> > > as Intel
> > proprietary IP.
> > >
> > > We are fully aware of your concerns on how to enable open source
> > > developers to use Intel ISP through up-streamed Kernel Driver.
> > > Internally, we are working on the license for this AIC library
> > > release now (as Hans said NDA license is not acceptable). We believe
> > > this will be more efficient way to help open source developers.
> > >
> > > This AIC library release would be a binary-only release. This AIC
> > > library does not use any kernel uAPIs directly. The user space
> > > Camera HAL that uses kernel uAPIs is available at
> > > https://chromium.googlesource.com/chromiumos/platform/arc-
> > > camera/+/master
> > >
>
> The AIC library (in binary form) is available here.
> https://storage.googleapis.com/chromeos-localmirror/distfiles/intel-3a-libs-
> bin-2017.09.27.tbz2
>
> Licensing information can be found in ./LICENSE.intel_3a_library file after
> unzipping the tar file.
>
> >
> > Just pinging to know your thoughts on this.
> >
> > Thanks
> > Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
2017-12-20 13:57 ` Mauro Carvalho Chehab
@ 2018-04-04 1:07 ` Mani, Rajmohan
2018-02-07 23:02 ` Mani, Rajmohan
` (3 subsequent siblings)
4 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2018-04-04 1:07 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Zhi, Yong, linux-media, sakari.ailus, Zheng, Jian Xu, Toivonen,
Tuukka, Hu, Jerry W, arnd, hch, robin.murphy, iommu, tfiga
Hi Mauro and all,
> > Subject: RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> >
> > Hi Mauro,
> >
> > > > > Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> > > > >
> > > > > Hi,
> > > > >
> > > > > Em Fri, 17 Nov 2017 02:58:56 +0000 "Mani, Rajmohan"
> > > > > <rajmohan.mani@intel.com> escreveu:
> > > > >
> > > > > > Here is an update on the IPU3 documentation that we are
> > > > > > currently working
> > > > > on.
> > > > > >
> > > > > > Image processing in IPU3 relies on the following.
> > > > > >
> > > > > > 1) HW configuration to enable ISP and
> > > > > > 2) setting customer specific 3A Tuning / Algorithm Parameters
> > > > > > to achieve
> > > > > desired image quality.
> > > > > >
> > > > > > We intend to provide documentation on ImgU driver programming
> > > > > > interface
> > > > > to help users of this driver to configure and enable ISP HW to
> > > > > meet their needs. This documentation will include details on
> > > > > complete
> > > > > V4L2 Kernel driver interface and IO-Control parameters, except
> > > > > for the ISP internal algorithm and its parameters (which is
> > > > > Intel proprietary
> > IP).
> > > > >
> > > > > Sakari asked me to take a look on this thread, specifically on
> > > > > this email. I took a look on the other e-mails from this thread
> > > > > that are discussing about this IP block.
> > > > >
> > > > > I understand that Intel wants to keep their internal 3A
> > > > > algorithm protected, just like other vendors protect their own
> > > > > algos. It was never a requirement to open whatever algorithm are
> > > > > used inside a hardware (or firmware). The only requirement is
> > > > > that firmwares should be licensed with redistribution
> > > > > permission, ideally merged at linux-firmware
> > > > git tree.
> > > > >
> > > > > Yet, what I don't understand is why Intel also wants to also
> > > > > protect the interface for such 3A hardware/firmware algorithm.
> > > > > The parameters that are passed from an userspace application to
> > > > > Intel ISP logic doesn't contain the algorithm itself. What's the
> > > > > issue of documenting the meaning of each parameter?
> > > > >
> > > >
> > > > Thanks for looking into this.
> > > >
> > > > To achieve improved image quality using IPU3, 3A (Auto White
> > > > balance, Auto Focus and Auto Exposure) Tuning parameters specific
> > > > to a given camera sensor module, are converted to Intel ISP
> > > > algorithm parameters in user space camera HAL using AIC (Automatic
> > > > ISP
> > Configuration) library.
> > > >
> > > > As a unique design of Intel ISP, it exposes very detailed
> > > > algorithm parameters (~ 10000 parameters) to configure ISP's image
> > > > processing algorithm per each image fame in runtime. Typical
> > > > Camera SW developers (including those at
> > > > Intel) are not expected to fully understand and directly set these
> > > > parameters to configure the ISP algorithm blocks. Due to the
> > > > above, a user space AIC library (in binary form) is provided to
> > > > generate ISP Algorithm specific parameters, for a given set of 3A
> > > > tuning parameters. It significantly reduces the efforts of SW
> > > > development in ISP HW
> > > configuration.
> > > >
> > > > On the other hand, the ISP algorithm details could be deduced
> > > > readily through these detailed parameters by other ISP experts
> > > > outside
> > Intel.
> > > > This is the reason that we want to keep these parameter
> > > > definitions as Intel
> > > proprietary IP.
> > > >
> > > > We are fully aware of your concerns on how to enable open source
> > > > developers to use Intel ISP through up-streamed Kernel Driver.
> > > > Internally, we are working on the license for this AIC library
> > > > release now (as Hans said NDA license is not acceptable). We
> > > > believe this will be more efficient way to help open source developers.
> > > >
> > > > This AIC library release would be a binary-only release. This AIC
> > > > library does not use any kernel uAPIs directly. The user space
> > > > Camera HAL that uses kernel uAPIs is available at
> > > > https://chromium.googlesource.com/chromiumos/platform/arc-
> > > > camera/+/master
> > > >
> >
> > The AIC library (in binary form) is available here.
> > https://storage.googleapis.com/chromeos-localmirror/distfiles/intel-3a
> > -libs-
> > bin-2017.09.27.tbz2
> >
> > Licensing information can be found in ./LICENSE.intel_3a_library file
> > after unzipping the tar file.
> >
> > >
> > > Just pinging to know your thoughts on this.
> > >
To follow up further on this, we have posted the preliminary documentation on
bnr (bayer noise reduction) parameters as below.
https://www.mail-archive.com/linux-media@vger.kernel.org/msg128683.html
We would like hear your early feedback and apply the same towards the
documentation of remaining IPU3 parameters.
Thanks
Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
@ 2018-04-04 1:07 ` Mani, Rajmohan
0 siblings, 0 replies; 24+ messages in thread
From: Mani, Rajmohan @ 2018-04-04 1:07 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: arnd-r2nGTMty4D4, Hu, Jerry W, Zheng, Jian Xu,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Toivonen,
Tuukka, Zhi, Yong, hch-jcswGhMUV9g,
linux-media-u79uwXL29TY76Z2rM5mHXA
Hi Mauro and all,
> > Subject: RE: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> >
> > Hi Mauro,
> >
> > > > > Subject: Re: [PATCH v4 00/12] Intel IPU3 ImgU patchset
> > > > >
> > > > > Hi,
> > > > >
> > > > > Em Fri, 17 Nov 2017 02:58:56 +0000 "Mani, Rajmohan"
> > > > > <rajmohan.mani-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> escreveu:
> > > > >
> > > > > > Here is an update on the IPU3 documentation that we are
> > > > > > currently working
> > > > > on.
> > > > > >
> > > > > > Image processing in IPU3 relies on the following.
> > > > > >
> > > > > > 1) HW configuration to enable ISP and
> > > > > > 2) setting customer specific 3A Tuning / Algorithm Parameters
> > > > > > to achieve
> > > > > desired image quality.
> > > > > >
> > > > > > We intend to provide documentation on ImgU driver programming
> > > > > > interface
> > > > > to help users of this driver to configure and enable ISP HW to
> > > > > meet their needs. This documentation will include details on
> > > > > complete
> > > > > V4L2 Kernel driver interface and IO-Control parameters, except
> > > > > for the ISP internal algorithm and its parameters (which is
> > > > > Intel proprietary
> > IP).
> > > > >
> > > > > Sakari asked me to take a look on this thread, specifically on
> > > > > this email. I took a look on the other e-mails from this thread
> > > > > that are discussing about this IP block.
> > > > >
> > > > > I understand that Intel wants to keep their internal 3A
> > > > > algorithm protected, just like other vendors protect their own
> > > > > algos. It was never a requirement to open whatever algorithm are
> > > > > used inside a hardware (or firmware). The only requirement is
> > > > > that firmwares should be licensed with redistribution
> > > > > permission, ideally merged at linux-firmware
> > > > git tree.
> > > > >
> > > > > Yet, what I don't understand is why Intel also wants to also
> > > > > protect the interface for such 3A hardware/firmware algorithm.
> > > > > The parameters that are passed from an userspace application to
> > > > > Intel ISP logic doesn't contain the algorithm itself. What's the
> > > > > issue of documenting the meaning of each parameter?
> > > > >
> > > >
> > > > Thanks for looking into this.
> > > >
> > > > To achieve improved image quality using IPU3, 3A (Auto White
> > > > balance, Auto Focus and Auto Exposure) Tuning parameters specific
> > > > to a given camera sensor module, are converted to Intel ISP
> > > > algorithm parameters in user space camera HAL using AIC (Automatic
> > > > ISP
> > Configuration) library.
> > > >
> > > > As a unique design of Intel ISP, it exposes very detailed
> > > > algorithm parameters (~ 10000 parameters) to configure ISP's image
> > > > processing algorithm per each image fame in runtime. Typical
> > > > Camera SW developers (including those at
> > > > Intel) are not expected to fully understand and directly set these
> > > > parameters to configure the ISP algorithm blocks. Due to the
> > > > above, a user space AIC library (in binary form) is provided to
> > > > generate ISP Algorithm specific parameters, for a given set of 3A
> > > > tuning parameters. It significantly reduces the efforts of SW
> > > > development in ISP HW
> > > configuration.
> > > >
> > > > On the other hand, the ISP algorithm details could be deduced
> > > > readily through these detailed parameters by other ISP experts
> > > > outside
> > Intel.
> > > > This is the reason that we want to keep these parameter
> > > > definitions as Intel
> > > proprietary IP.
> > > >
> > > > We are fully aware of your concerns on how to enable open source
> > > > developers to use Intel ISP through up-streamed Kernel Driver.
> > > > Internally, we are working on the license for this AIC library
> > > > release now (as Hans said NDA license is not acceptable). We
> > > > believe this will be more efficient way to help open source developers.
> > > >
> > > > This AIC library release would be a binary-only release. This AIC
> > > > library does not use any kernel uAPIs directly. The user space
> > > > Camera HAL that uses kernel uAPIs is available at
> > > > https://chromium.googlesource.com/chromiumos/platform/arc-
> > > > camera/+/master
> > > >
> >
> > The AIC library (in binary form) is available here.
> > https://storage.googleapis.com/chromeos-localmirror/distfiles/intel-3a
> > -libs-
> > bin-2017.09.27.tbz2
> >
> > Licensing information can be found in ./LICENSE.intel_3a_library file
> > after unzipping the tar file.
> >
> > >
> > > Just pinging to know your thoughts on this.
> > >
To follow up further on this, we have posted the preliminary documentation on
bnr (bayer noise reduction) parameters as below.
https://www.mail-archive.com/linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg128683.html
We would like hear your early feedback and apply the same towards the
documentation of remaining IPU3 parameters.
Thanks
Raj
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2018-04-04 1:07 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-10-18 3:46 [PATCH v4 00/12] Intel IPU3 ImgU patchset Yong Zhi
2017-10-18 3:46 ` Yong Zhi
2017-10-18 6:23 ` Christoph Hellwig
2017-10-18 6:23 ` Christoph Hellwig
2017-10-18 14:42 ` Zhi, Yong
2017-10-18 14:42 ` Zhi, Yong
2017-10-20 9:09 ` Sakari Ailus
2017-10-23 21:24 ` Zhi, Yong
2017-10-23 21:24 ` Zhi, Yong
2017-11-17 2:58 ` Mani, Rajmohan
2017-11-27 9:55 ` Hans Verkuil
2017-12-05 0:13 ` Mani, Rajmohan
2017-12-05 3:22 ` Tomasz Figa
2017-12-05 7:46 ` Hans Verkuil
2017-12-06 0:46 ` Mani, Rajmohan
2017-12-20 13:57 ` Mauro Carvalho Chehab
2017-12-26 22:30 ` Mani, Rajmohan
2018-02-07 23:02 ` Mani, Rajmohan
2018-02-21 1:56 ` Mani, Rajmohan
2018-02-21 1:56 ` Mani, Rajmohan
2018-03-28 3:14 ` Mani, Rajmohan
2018-03-28 3:14 ` Mani, Rajmohan
2018-04-04 1:07 ` Mani, Rajmohan
2018-04-04 1:07 ` Mani, Rajmohan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.