[RFC/WIP 0/4] HEIC image encoder

[RFC/WIP 0/4] HEIC image encoder

[Stanimir Varbanov]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 3611 bytes --]

Hi,

HEIC (High-Efficiency Image Container) is a variant of HEIF (High
Efficiency Image File Format) where HEVC/h265 codec is used to encode
images.  For more info see [1].

In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC.
The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is
also an option).

There are two encoding modes which should be selectable by clients:
    1. Regular image encoding
    2. Grid image encoding

1. Regular image encoding

Propose to reuse stateful video encoder spec [2].

- queuing one OUTPUT buffer will produce one CAPTURE buffer.  The
client could trigger Drain sequence at any point of time.

2. Grid image encoding

Propose to reuse stateful video encoder spec [2].

- queuing one OUTPUT buffer will produce a number of grids CAPTURE
buffers.  The client could trigger Drain sequence at any point of time.

This image encoding mode is used when the input image resolution is
bigger then the hardware can support and/or for compatibility reasons
(for exmaple, the HEIC decoding hardware is not capable to decode higher
than VGA resolutions).

In this mode the input image is divided on square blocks (we call them grids)
and every block is encoded separately (the Venus driver presently supports 
grid size of 512x512 but that could be changed in the future).

To set the grid size we use a new v4l2 control.

The side effect of this mode is that the client have to set the v4l2
control and thus enable grid encoding before setting the formats on
CAPTURE and OUTPUT queues, because the grid size reflects on the
selected resolutions. Also the horizontal and vertical strides will
also be affected because thеy have to be aligned to the grid size
in order to satisfy DMA alignment restrictions.

Using of v4l2 control to set up Grid mode and Grid size above looks
inpractical and somehow breaks the v4l2 and v4l2 control rules, so
I'd give one more option. 

Encoding the Grid mode/size in the new proposed HEIC pixel format:

   V4L2_PIX_FMT_HEIC - Regular HEIC image encoding
   V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 
   and so on ...

Comments and suggestions are welcome!

regards,
Stan

[1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format
[2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html


Stanimir Varbanov (4):
  media: Add HEIC compressed pixel format
  v4l2-ctrls: Add HEIC grid size control
  venus: helpers: Add a new helper for buffer processing
  venus: Add HEIC image encoder

 .../media/v4l/pixfmt-compressed.rst           |   12 +
 drivers/media/platform/qcom/venus/Makefile    |    2 +
 drivers/media/platform/qcom/venus/core.h      |   10 +
 drivers/media/platform/qcom/venus/helpers.c   |   20 +
 drivers/media/platform/qcom/venus/helpers.h   |    1 +
 drivers/media/platform/qcom/venus/hfi_cmds.c  |   10 +-
 .../media/platform/qcom/venus/hfi_helper.h    |    5 +
 drivers/media/platform/qcom/venus/ienc.c      | 1348 +++++++++++++++++
 drivers/media/platform/qcom/venus/ienc.h      |   14 +
 .../media/platform/qcom/venus/ienc_ctrls.c    |   83 +
 drivers/media/v4l2-core/v4l2-ctrls.c          |    3 +
 drivers/media/v4l2-core/v4l2-ioctl.c          |    1 +
 include/uapi/linux/v4l2-controls.h            |    1 +
 include/uapi/linux/videodev2.h                |    1 +
 14 files changed, 1510 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/platform/qcom/venus/ienc.c
 create mode 100644 drivers/media/platform/qcom/venus/ienc.h
 create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c

-- 
2.25.1

[RFC/WIP 1/4] media: Add HEIC compressed pixel format

[Stanimir Varbanov]

Add HEIC (High-Efficiency Image Container) pixel format. This an
image container which use HEVC codec to encoded images.

Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
---
 .../userspace-api/media/v4l/pixfmt-compressed.rst    | 12 ++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c                 |  1 +
 include/uapi/linux/videodev2.h                       |  1 +
 3 files changed, 14 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst b/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
index ba6c0c961204..246bff90dcac 100644
--- a/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
+++ b/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
@@ -186,6 +186,18 @@ Compressed Formats
 	If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM``
 	then the decoder has no	requirements since it can parse all the
 	information from the raw bytestream.
+    * .. _V4L2-PIX-FMT-HEIC:
+
+      - ``V4L2_PIX_FMT_HEIC``
+      - 'HEIC'
+      - High Efficiency Image Container is an image container file format which
+        uses HEVC encoding and it is a variant of HEIF (High Efficiency Image File)
+        format.
+	The decoder expects one Access Unit per buffer.
+	The encoder generates one Access Unit per buffer.
+	If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM``
+	then the decoder has no	requirements since it can parse all the
+	information from the raw bytestream.
     * .. _V4L2-PIX-FMT-HEVC-SLICE:
 
       - ``V4L2_PIX_FMT_HEVC_SLICE``
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 31d1342e61e8..3a1b4c3a76c8 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1454,6 +1454,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		case V4L2_PIX_FMT_S5C_UYVY_JPG:	descr = "S5C73MX interleaved UYVY/JPEG"; break;
 		case V4L2_PIX_FMT_MT21C:	descr = "Mediatek Compressed Format"; break;
 		case V4L2_PIX_FMT_SUNXI_TILED_NV12: descr = "Sunxi Tiled NV12 Format"; break;
+		case V4L2_PIX_FMT_HEIC:		descr = "HEIC Image Format"; break;
 		default:
 			if (fmt->description[0])
 				return;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 79dbde3bcf8d..2153b5c31d46 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -699,6 +699,7 @@ struct v4l2_pix_format {
 #define V4L2_PIX_FMT_FWHT     v4l2_fourcc('F', 'W', 'H', 'T') /* Fast Walsh Hadamard Transform (vicodec) */
 #define V4L2_PIX_FMT_FWHT_STATELESS     v4l2_fourcc('S', 'F', 'W', 'H') /* Stateless FWHT (vicodec) */
 #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
+#define V4L2_PIX_FMT_HEIC	v4l2_fourcc('H', 'E', 'I', 'C') /* HEIC HEVC image format */
 
 /*  Vendor-specific formats   */
 #define V4L2_PIX_FMT_CPIA1    v4l2_fourcc('C', 'P', 'I', 'A') /* cpia1 YUV */
-- 
2.25.1

[RFC/WIP 2/4] v4l2-ctrls: Add HEIC grid size control

[Stanimir Varbanov]

Add a control to set HEIC image grid size. This sets a grid (partition)
size for image encoding.  It will be used to instruct image encoder to
produce CAPTURE buffers for any grid of the input image, and it is
applicable when the input YUV buffer resolution is bigger then the
hardware can support.

Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 3 +++
 include/uapi/linux/v4l2-controls.h   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index ca50e21e2838..6b443f946906 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1034,6 +1034,8 @@ const char *v4l2_ctrl_get_name(u32 id)
 	case V4L2_CID_MPEG_VIDEO_HEVC_DECODE_MODE:		return "HEVC Decode Mode";
 	case V4L2_CID_MPEG_VIDEO_HEVC_START_CODE:		return "HEVC Start Code";
 
+	case V4L2_CID_MPEG_VIDEO_HEIC_GRID_SIZE:		return "HEIC Grid Size";
+
 	/* CAMERA controls */
 	/* Keep the order of the 'case's the same as in v4l2-controls.h! */
 	case V4L2_CID_CAMERA_CLASS:		return "Camera Controls";
@@ -1282,6 +1284,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
 	case V4L2_CID_MPEG_VIDEO_MV_H_SEARCH_RANGE:
 	case V4L2_CID_MPEG_VIDEO_MV_V_SEARCH_RANGE:
 	case V4L2_CID_MPEG_VIDEO_DEC_DISPLAY_DELAY:
+	case V4L2_CID_MPEG_VIDEO_HEIC_GRID_SIZE:
 		*type = V4L2_CTRL_TYPE_INTEGER;
 		break;
 	case V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME:
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index 6f8c08507bf4..6ae6cf3c815c 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -800,6 +800,7 @@ enum v4l2_mpeg_video_frame_skip_mode {
 
 #define V4L2_CID_MPEG_VIDEO_DEC_DISPLAY_DELAY		(V4L2_CID_CODEC_BASE + 653)
 #define V4L2_CID_MPEG_VIDEO_DEC_DISPLAY_DELAY_ENABLE	(V4L2_CID_CODEC_BASE + 654)
+#define V4L2_CID_MPEG_VIDEO_HEIC_GRID_SIZE		(V4L2_CID_CODEC_BASE + 655)
 
 /*  MPEG-class control IDs specific to the CX2341x driver as defined by V4L2 */
 #define V4L2_CID_CODEC_CX2341X_BASE				(V4L2_CTRL_CLASS_CODEC | 0x1000)
-- 
2.25.1

[RFC/WIP 3/4] venus: helpers: Add a new helper for buffer processing

[Stanimir Varbanov]

The new helper will be used from encoder and decoder drivers
to enqueue buffers for processing by firmware.

Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
---
 drivers/media/platform/qcom/venus/helpers.c | 20 ++++++++++++++++++++
 drivers/media/platform/qcom/venus/helpers.h |  1 +
 2 files changed, 21 insertions(+)

diff --git a/drivers/media/platform/qcom/venus/helpers.c b/drivers/media/platform/qcom/venus/helpers.c
index dc8ef13d0c95..8e7f2ec0cd01 100644
--- a/drivers/media/platform/qcom/venus/helpers.c
+++ b/drivers/media/platform/qcom/venus/helpers.c
@@ -1450,6 +1450,26 @@ void venus_helper_vb2_buf_queue(struct vb2_buffer *vb)
 }
 EXPORT_SYMBOL_GPL(venus_helper_vb2_buf_queue);
 
+void venus_helper_process_buf(struct vb2_buffer *vb)
+{
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct venus_inst *inst = vb2_get_drv_priv(vb->vb2_queue);
+	int ret;
+
+	cache_payload(inst, vb);
+
+	if (vb2_start_streaming_called(vb->vb2_queue)) {
+		ret = is_buf_refed(inst, vbuf);
+		if (ret)
+			return;
+
+		ret = session_process_buf(inst, vbuf);
+		if (ret)
+			return_buf_error(inst, vbuf);
+	}
+}
+EXPORT_SYMBOL_GPL(venus_helper_process_buf);
+
 void venus_helper_buffers_done(struct venus_inst *inst, unsigned int type,
 			       enum vb2_buffer_state state)
 {
diff --git a/drivers/media/platform/qcom/venus/helpers.h b/drivers/media/platform/qcom/venus/helpers.h
index e6269b4be3af..c7cea33e2803 100644
--- a/drivers/media/platform/qcom/venus/helpers.h
+++ b/drivers/media/platform/qcom/venus/helpers.h
@@ -19,6 +19,7 @@ void venus_helper_buffers_done(struct venus_inst *inst, unsigned int type,
 int venus_helper_vb2_buf_init(struct vb2_buffer *vb);
 int venus_helper_vb2_buf_prepare(struct vb2_buffer *vb);
 void venus_helper_vb2_buf_queue(struct vb2_buffer *vb);
+void venus_helper_process_buf(struct vb2_buffer *vb);
 void venus_helper_vb2_stop_streaming(struct vb2_queue *q);
 int venus_helper_vb2_start_streaming(struct venus_inst *inst);
 void venus_helper_m2m_device_run(void *priv);
-- 
2.25.1

[RFC/WIP 4/4] venus: Add HEIC image encoder

[Stanimir Varbanov]

This adds support for HEIC image encoder to Venus driver as a
separate device video node.

Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
---
 drivers/media/platform/qcom/venus/Makefile    |    2 +
 drivers/media/platform/qcom/venus/core.h      |   10 +
 drivers/media/platform/qcom/venus/hfi_cmds.c  |   10 +-
 .../media/platform/qcom/venus/hfi_helper.h    |    5 +
 drivers/media/platform/qcom/venus/ienc.c      | 1348 +++++++++++++++++
 drivers/media/platform/qcom/venus/ienc.h      |   14 +
 .../media/platform/qcom/venus/ienc_ctrls.c    |   83 +
 7 files changed, 1471 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/platform/qcom/venus/ienc.c
 create mode 100644 drivers/media/platform/qcom/venus/ienc.h
 create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c

diff --git a/drivers/media/platform/qcom/venus/Makefile b/drivers/media/platform/qcom/venus/Makefile
index 91ee6be10292..1277e08b73f4 100644
--- a/drivers/media/platform/qcom/venus/Makefile
+++ b/drivers/media/platform/qcom/venus/Makefile
@@ -9,7 +9,9 @@ venus-core-objs += core.o helpers.o firmware.o \
 
 venus-dec-objs += vdec.o vdec_ctrls.o
 venus-enc-objs += venc.o venc_ctrls.o
+venus-ienc-objs += ienc.o ienc_ctrls.o
 
 obj-$(CONFIG_VIDEO_QCOM_VENUS) += venus-core.o
 obj-$(CONFIG_VIDEO_QCOM_VENUS) += venus-dec.o
 obj-$(CONFIG_VIDEO_QCOM_VENUS) += venus-enc.o
+obj-$(CONFIG_VIDEO_QCOM_VENUS) += venus-ienc.o
diff --git a/drivers/media/platform/qcom/venus/core.h b/drivers/media/platform/qcom/venus/core.h
index 6de596b5a9d1..c5919724f18e 100644
--- a/drivers/media/platform/qcom/venus/core.h
+++ b/drivers/media/platform/qcom/venus/core.h
@@ -150,11 +150,13 @@ struct venus_core {
 	struct reset_control *resets[VIDC_RESETS_NUM_MAX];
 	struct video_device *vdev_dec;
 	struct video_device *vdev_enc;
+	struct video_device *vdev_ienc;
 	struct v4l2_device v4l2_dev;
 	const struct venus_resources *res;
 	struct device *dev;
 	struct device *dev_dec;
 	struct device *dev_enc;
+	struct device *dev_ienc;
 	unsigned int use_tz;
 	struct video_firmware {
 		struct device *dev;
@@ -261,6 +263,13 @@ struct venc_controls {
 	u32 base_priority_id;
 };
 
+struct ienc_controls {
+	u32 profile;
+	u32 level;
+	u32 image_quality;
+	u32 grid_size;
+};
+
 struct venus_buffer {
 	struct vb2_v4l2_buffer vb;
 	struct list_head list;
@@ -370,6 +379,7 @@ struct venus_inst {
 	union {
 		struct vdec_controls dec;
 		struct venc_controls enc;
+		struct ienc_controls ienc;
 	} controls;
 	struct v4l2_fh fh;
 	unsigned int streamon_cap, streamon_out;
diff --git a/drivers/media/platform/qcom/venus/hfi_cmds.c b/drivers/media/platform/qcom/venus/hfi_cmds.c
index fb64046d1e35..48232db4e491 100644
--- a/drivers/media/platform/qcom/venus/hfi_cmds.c
+++ b/drivers/media/platform/qcom/venus/hfi_cmds.c
@@ -1268,7 +1268,15 @@ pkt_session_set_property_6xx(struct hfi_session_set_property_pkt *pkt,
 		cq->frame_quality = in->frame_quality;
 		pkt->shdr.hdr.size += sizeof(u32) + sizeof(*cq);
 		break;
-	} default:
+	}
+	case HFI_PROPERTY_CONFIG_HEIC_GRID_ENABLE: {
+		struct hfi_heic_grid_enable *in = pdata, *grid = prop_data;
+
+		grid->grid_enable = in->grid_enable;
+		pkt->shdr.hdr.size += sizeof(u32) + sizeof(*grid);
+		break;
+	}
+	default:
 		return pkt_session_set_property_4xx(pkt, cookie, ptype, pdata);
 	}
 
diff --git a/drivers/media/platform/qcom/venus/hfi_helper.h b/drivers/media/platform/qcom/venus/hfi_helper.h
index f367f43c9fb7..3836dedd61b0 100644
--- a/drivers/media/platform/qcom/venus/hfi_helper.h
+++ b/drivers/media/platform/qcom/venus/hfi_helper.h
@@ -534,6 +534,7 @@
 #define HFI_PROPERTY_CONFIG_VENC_LTRPERIOD			0x200600c
 #define HFI_PROPERTY_CONFIG_VENC_PERF_MODE			0x200600e
 #define HFI_PROPERTY_CONFIG_HEIC_FRAME_QUALITY			0x2006014
+#define HFI_PROPERTY_CONFIG_HEIC_GRID_ENABLE			0x2006015
 
 /*
  * HFI_PROPERTY_PARAM_VPE_COMMON_START
@@ -772,6 +773,10 @@ struct hfi_heic_frame_quality {
 	u32 reserved[3];
 };
 
+struct hfi_heic_grid_enable {
+	u32 grid_enable;
+};
+
 struct hfi_quantization {
 	u32 qp_i;
 	u32 qp_p;
diff --git a/drivers/media/platform/qcom/venus/ienc.c b/drivers/media/platform/qcom/venus/ienc.c
new file mode 100644
index 000000000000..a54a3d666233
--- /dev/null
+++ b/drivers/media/platform/qcom/venus/ienc.c
@@ -0,0 +1,1348 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2012-2016, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2017 Linaro Ltd.
+ */
+#include <linux/clk.h>
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/slab.h>
+#include <media/v4l2-mem2mem.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ctrls.h>
+
+#include "hfi_venus_io.h"
+#include "hfi_parser.h"
+#include "core.h"
+#include "helpers.h"
+#include "ienc.h"
+#include "pm_helpers.h"
+
+/*
+ * Three resons to keep MPLANE formats (despite that the number of planes
+ * currently is one):
+ * - the MPLANE formats allow only one plane to be used
+ * - the downstream driver use MPLANE formats too
+ * - future firmware versions could add support for >1 planes
+ */
+static const struct venus_format ienc_formats[] = {
+	{
+		.pixfmt = V4L2_PIX_FMT_NV12,
+		.num_planes = 1,
+		.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE,
+	}, {
+		.pixfmt = V4L2_PIX_FMT_HEIC,
+		.num_planes = 1,
+		.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
+	},
+};
+
+static const struct venus_format *
+find_format(struct venus_inst *inst, u32 pixfmt, u32 type)
+{
+	const struct venus_format *fmt = ienc_formats;
+	unsigned int size = ARRAY_SIZE(ienc_formats);
+	unsigned int i;
+	u32 check_pixfmt;
+
+	for (i = 0; i < size; i++) {
+		if (fmt[i].pixfmt == pixfmt)
+			break;
+	}
+
+	if (i == size || fmt[i].type != type)
+		return NULL;
+
+	check_pixfmt = fmt[i].pixfmt;
+
+	if (check_pixfmt == V4L2_PIX_FMT_HEIC)
+		check_pixfmt = V4L2_PIX_FMT_HEVC;
+
+	if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE &&
+	    !venus_helper_check_codec(inst, check_pixfmt))
+		return NULL;
+
+	return &fmt[i];
+}
+
+static const struct venus_format *
+find_format_by_index(struct venus_inst *inst, unsigned int index, u32 type)
+{
+	const struct venus_format *fmt = ienc_formats;
+	unsigned int size = ARRAY_SIZE(ienc_formats);
+	unsigned int i, k = 0;
+	u32 check_pixfmt;
+
+	if (index > size)
+		return NULL;
+
+	for (i = 0; i < size; i++) {
+		bool valid;
+
+		if (fmt[i].type != type)
+			continue;
+
+		check_pixfmt = fmt[i].pixfmt;
+		if (check_pixfmt == V4L2_PIX_FMT_HEIC)
+			check_pixfmt = V4L2_PIX_FMT_HEVC;
+
+		valid = type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE ||
+			venus_helper_check_codec(inst, check_pixfmt);
+		if (k == index && valid)
+			break;
+		if (valid)
+			k++;
+	}
+
+	if (i == size)
+		return NULL;
+
+	return &fmt[i];
+}
+
+static int
+ienc_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
+{
+	strscpy(cap->driver, "qcom-venus", sizeof(cap->driver));
+	strscpy(cap->card, "Qualcomm Venus image encoder", sizeof(cap->card));
+	strscpy(cap->bus_info, "platform:qcom-venus", sizeof(cap->bus_info));
+
+	return 0;
+}
+
+static int ienc_enum_fmt(struct file *file, void *fh, struct v4l2_fmtdesc *f)
+{
+	struct venus_inst *inst = to_inst(file);
+	const struct venus_format *fmt;
+
+	fmt = find_format_by_index(inst, f->index, f->type);
+
+	memset(f->reserved, 0, sizeof(f->reserved));
+
+	if (!fmt)
+		return -EINVAL;
+
+	f->pixelformat = fmt->pixfmt;
+
+	return 0;
+}
+
+static const struct venus_format *
+ienc_try_fmt_common(struct venus_inst *inst, struct v4l2_format *f)
+{
+	struct v4l2_pix_format_mplane *pixmp = &f->fmt.pix_mp;
+	struct v4l2_plane_pix_format *pfmt = pixmp->plane_fmt;
+	const struct venus_format *fmt;
+	u32 sizeimage;
+
+	memset(pfmt[0].reserved, 0, sizeof(pfmt[0].reserved));
+	memset(pixmp->reserved, 0, sizeof(pixmp->reserved));
+
+	fmt = find_format(inst, pixmp->pixelformat, f->type);
+	if (!fmt) {
+		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+			pixmp->pixelformat = V4L2_PIX_FMT_HEIC;
+		else if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+			pixmp->pixelformat = V4L2_PIX_FMT_NV12;
+		else
+			return NULL;
+		fmt = find_format(inst, pixmp->pixelformat, f->type);
+	}
+
+	pixmp->width = clamp(pixmp->width, frame_width_min(inst),
+			     frame_width_max(inst));
+	pixmp->height = clamp(pixmp->height, frame_height_min(inst),
+			      frame_height_max(inst));
+
+	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+		pixmp->height = ALIGN(pixmp->height, 32);
+
+	pixmp->width = ALIGN(pixmp->width, 2);
+	pixmp->height = ALIGN(pixmp->height, 2);
+
+	if (pixmp->field == V4L2_FIELD_ANY)
+		pixmp->field = V4L2_FIELD_NONE;
+	pixmp->num_planes = fmt->num_planes;
+	pixmp->flags = 0;
+
+	sizeimage = venus_helper_get_framesz(pixmp->pixelformat,
+					     pixmp->width,
+					     pixmp->height);
+	pfmt[0].sizeimage = max(ALIGN(pfmt[0].sizeimage, SZ_4K), sizeimage);
+
+	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+		pfmt[0].bytesperline = ALIGN(pixmp->width, 128);
+	else
+		pfmt[0].bytesperline = 0;
+
+	return fmt;
+}
+
+static int ienc_try_fmt(struct file *file, void *fh, struct v4l2_format *f)
+{
+	struct venus_inst *inst = to_inst(file);
+
+	ienc_try_fmt_common(inst, f);
+
+	return 0;
+}
+
+static int ienc_s_fmt(struct file *file, void *fh, struct v4l2_format *f)
+{
+	struct venus_inst *inst = to_inst(file);
+	struct v4l2_pix_format_mplane *pixmp = &f->fmt.pix_mp;
+	struct v4l2_pix_format_mplane orig_pixmp;
+	const struct venus_format *fmt;
+	struct v4l2_format format;
+	u32 pixfmt_out = 0, pixfmt_cap = 0;
+	struct vb2_queue *q;
+
+	q = v4l2_m2m_get_vq(inst->m2m_ctx, f->type);
+	if (!q)
+		return -EINVAL;
+
+	if (vb2_is_busy(q))
+		return -EBUSY;
+
+	orig_pixmp = *pixmp;
+
+	fmt = ienc_try_fmt_common(inst, f);
+	if (!fmt)
+		return -EINVAL;
+
+	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+		pixfmt_out = pixmp->pixelformat;
+		pixfmt_cap = inst->fmt_cap->pixfmt;
+	} else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+		pixfmt_cap = pixmp->pixelformat;
+		pixfmt_out = inst->fmt_out->pixfmt;
+	}
+
+	memset(&format, 0, sizeof(format));
+
+	format.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+	format.fmt.pix_mp.pixelformat = pixfmt_out;
+	format.fmt.pix_mp.width = orig_pixmp.width;
+	format.fmt.pix_mp.height = orig_pixmp.height;
+	ienc_try_fmt_common(inst, &format);
+
+	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+		inst->out_width = format.fmt.pix_mp.width;
+		inst->out_height = format.fmt.pix_mp.height;
+		inst->colorspace = pixmp->colorspace;
+		inst->ycbcr_enc = pixmp->ycbcr_enc;
+		inst->quantization = pixmp->quantization;
+		inst->xfer_func = pixmp->xfer_func;
+	}
+
+	memset(&format, 0, sizeof(format));
+
+	format.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+	format.fmt.pix_mp.pixelformat = pixfmt_cap;
+	format.fmt.pix_mp.width = orig_pixmp.width;
+	format.fmt.pix_mp.height = orig_pixmp.height;
+	ienc_try_fmt_common(inst, &format);
+
+	inst->width = format.fmt.pix_mp.width;
+	inst->height = format.fmt.pix_mp.height;
+
+	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+		inst->fmt_out = fmt;
+	} else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+		inst->fmt_cap = fmt;
+		inst->output_buf_size = pixmp->plane_fmt[0].sizeimage;
+	}
+
+	return 0;
+}
+
+static int ienc_g_fmt(struct file *file, void *fh, struct v4l2_format *f)
+{
+	struct v4l2_pix_format_mplane *pixmp = &f->fmt.pix_mp;
+	struct venus_inst *inst = to_inst(file);
+	const struct venus_format *fmt;
+
+	if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+		fmt = inst->fmt_cap;
+	else if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+		fmt = inst->fmt_out;
+	else
+		return -EINVAL;
+
+	pixmp->pixelformat = fmt->pixfmt;
+
+	if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+		pixmp->width = inst->width;
+		pixmp->height = inst->height;
+		pixmp->colorspace = inst->colorspace;
+		pixmp->ycbcr_enc = inst->ycbcr_enc;
+		pixmp->quantization = inst->quantization;
+		pixmp->xfer_func = inst->xfer_func;
+	} else if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+		pixmp->width = inst->out_width;
+		pixmp->height = inst->out_height;
+	}
+
+	ienc_try_fmt_common(inst, f);
+
+	return 0;
+}
+
+static int
+ienc_g_selection(struct file *file, void *fh, struct v4l2_selection *s)
+{
+	struct venus_inst *inst = to_inst(file);
+
+	if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
+		return -EINVAL;
+
+	switch (s->target) {
+	case V4L2_SEL_TGT_CROP_DEFAULT:
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		s->r.width = inst->out_width;
+		s->r.height = inst->out_height;
+		break;
+	case V4L2_SEL_TGT_CROP:
+		s->r.width = inst->width;
+		s->r.height = inst->height;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	s->r.top = 0;
+	s->r.left = 0;
+
+	return 0;
+}
+
+static int
+ienc_s_selection(struct file *file, void *fh, struct v4l2_selection *s)
+{
+	struct venus_inst *inst = to_inst(file);
+
+	if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
+		return -EINVAL;
+
+	if (s->r.width > inst->out_width ||
+	    s->r.height > inst->out_height)
+		return -EINVAL;
+
+	s->r.width = ALIGN(s->r.width, 2);
+	s->r.height = ALIGN(s->r.height, 2);
+
+	switch (s->target) {
+	case V4L2_SEL_TGT_CROP:
+		s->r.top = 0;
+		s->r.left = 0;
+		inst->width = s->r.width;
+		inst->height = s->r.height;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int ienc_s_parm(struct file *file, void *fh, struct v4l2_streamparm *a)
+{
+	struct venus_inst *inst = to_inst(file);
+	struct v4l2_outputparm *out = &a->parm.output;
+	struct v4l2_fract *timeperframe = &out->timeperframe;
+	u64 us_per_frame, fps;
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE &&
+	    a->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+		return -EINVAL;
+
+	memset(out->reserved, 0, sizeof(out->reserved));
+
+	if (!timeperframe->denominator)
+		timeperframe->denominator = inst->timeperframe.denominator;
+	if (!timeperframe->numerator)
+		timeperframe->numerator = inst->timeperframe.numerator;
+
+	out->capability = V4L2_CAP_TIMEPERFRAME;
+
+	us_per_frame = timeperframe->numerator * (u64)USEC_PER_SEC;
+	do_div(us_per_frame, timeperframe->denominator);
+
+	if (!us_per_frame)
+		return -EINVAL;
+
+	fps = (u64)USEC_PER_SEC;
+	do_div(fps, us_per_frame);
+
+	inst->timeperframe = *timeperframe;
+	inst->fps = fps;
+
+	return 0;
+}
+
+static int ienc_g_parm(struct file *file, void *fh, struct v4l2_streamparm *a)
+{
+	struct venus_inst *inst = to_inst(file);
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE &&
+	    a->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+		return -EINVAL;
+
+	a->parm.output.capability |= V4L2_CAP_TIMEPERFRAME;
+	a->parm.output.timeperframe = inst->timeperframe;
+
+	return 0;
+}
+
+static int ienc_enum_framesizes(struct file *file, void *fh,
+				struct v4l2_frmsizeenum *fsize)
+{
+	struct venus_inst *inst = to_inst(file);
+	const struct venus_format *fmt;
+
+	fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
+
+	fmt = find_format(inst, fsize->pixel_format,
+			  V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+	if (!fmt) {
+		fmt = find_format(inst, fsize->pixel_format,
+				  V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE);
+		if (!fmt)
+			return -EINVAL;
+	}
+
+	if (fsize->index)
+		return -EINVAL;
+
+	fsize->stepwise.min_width = frame_width_min(inst);
+	fsize->stepwise.max_width = frame_width_max(inst);
+	fsize->stepwise.step_width = frame_width_step(inst);
+	fsize->stepwise.min_height = frame_height_min(inst);
+	fsize->stepwise.max_height = frame_height_max(inst);
+	fsize->stepwise.step_height = frame_height_step(inst);
+
+	return 0;
+}
+
+static int ienc_enum_frameintervals(struct file *file, void *fh,
+				    struct v4l2_frmivalenum *fival)
+{
+	struct venus_inst *inst = to_inst(file);
+	const struct venus_format *fmt;
+	unsigned int framerate_factor = 1;
+
+	fival->type = V4L2_FRMIVAL_TYPE_STEPWISE;
+
+	fmt = find_format(inst, fival->pixel_format,
+			  V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+	if (!fmt) {
+		fmt = find_format(inst, fival->pixel_format,
+				  V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE);
+		if (!fmt)
+			return -EINVAL;
+	}
+
+	if (fival->index)
+		return -EINVAL;
+
+	if (!fival->width || !fival->height)
+		return -EINVAL;
+
+	if (fival->width > frame_width_max(inst) ||
+	    fival->width < frame_width_min(inst) ||
+	    fival->height > frame_height_max(inst) ||
+	    fival->height < frame_height_min(inst))
+		return -EINVAL;
+
+	if (IS_V1(inst->core)) {
+		/* framerate is reported in 1/65535 fps unit */
+		framerate_factor = (1 << 16);
+	}
+
+	fival->stepwise.min.numerator = 1;
+	fival->stepwise.min.denominator = frate_max(inst) / framerate_factor;
+	fival->stepwise.max.numerator = 1;
+	fival->stepwise.max.denominator = frate_min(inst) / framerate_factor;
+	fival->stepwise.step.numerator = 1;
+	fival->stepwise.step.denominator = frate_max(inst) / framerate_factor;
+
+	return 0;
+}
+
+static int ienc_encoder_cmd(struct file *file, void *fh,
+			    struct v4l2_encoder_cmd *ec)
+{
+	struct venus_inst *inst = to_inst(file);
+	struct v4l2_m2m_ctx *m2m_ctx = inst->m2m_ctx;
+	struct hfi_frame_data fdata = {0};
+	int ret = 0;
+
+	ret = v4l2_m2m_ioctl_try_encoder_cmd(file, fh, ec);
+	if (ret < 0)
+		return ret;
+
+	mutex_lock(&inst->lock);
+
+	if (!vb2_is_streaming(&m2m_ctx->cap_q_ctx.q) ||
+	    !vb2_is_streaming(&m2m_ctx->out_q_ctx.q))
+		goto unlock;
+
+	if (m2m_ctx->is_draining) {
+		ret = -EBUSY;
+		goto unlock;
+	}
+
+	if (ec->cmd == V4L2_ENC_CMD_STOP) {
+		if (v4l2_m2m_has_stopped(m2m_ctx)) {
+			ret = 0;
+			goto unlock;
+		}
+
+		m2m_ctx->is_draining = true;
+
+		fdata.buffer_type = HFI_BUFFER_INPUT;
+		fdata.flags |= HFI_BUFFERFLAG_EOS;
+		fdata.device_addr = 0;
+		fdata.clnt_data = (u32)-1;
+
+		ret = hfi_session_process_buf(inst, &fdata);
+		if (ret)
+			goto unlock;
+	}
+
+	if (ec->cmd == V4L2_ENC_CMD_START && v4l2_m2m_has_stopped(m2m_ctx)) {
+		vb2_clear_last_buffer_dequeued(&m2m_ctx->cap_q_ctx.q);
+		inst->m2m_ctx->has_stopped = false;
+		venus_helper_process_initial_out_bufs(inst);
+		venus_helper_process_initial_cap_bufs(inst);
+	}
+
+unlock:
+	mutex_unlock(&inst->lock);
+	return ret;
+}
+
+static const struct v4l2_ioctl_ops ienc_ioctl_ops = {
+	.vidioc_querycap = ienc_querycap,
+	.vidioc_enum_fmt_vid_cap = ienc_enum_fmt,
+	.vidioc_enum_fmt_vid_out = ienc_enum_fmt,
+	.vidioc_s_fmt_vid_cap_mplane = ienc_s_fmt,
+	.vidioc_s_fmt_vid_out_mplane = ienc_s_fmt,
+	.vidioc_g_fmt_vid_cap_mplane = ienc_g_fmt,
+	.vidioc_g_fmt_vid_out_mplane = ienc_g_fmt,
+	.vidioc_try_fmt_vid_cap_mplane = ienc_try_fmt,
+	.vidioc_try_fmt_vid_out_mplane = ienc_try_fmt,
+	.vidioc_g_selection = ienc_g_selection,
+	.vidioc_s_selection = ienc_s_selection,
+	.vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,
+	.vidioc_querybuf = v4l2_m2m_ioctl_querybuf,
+	.vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,
+	.vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf,
+	.vidioc_qbuf = v4l2_m2m_ioctl_qbuf,
+	.vidioc_expbuf = v4l2_m2m_ioctl_expbuf,
+	.vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,
+	.vidioc_streamon = v4l2_m2m_ioctl_streamon,
+	.vidioc_streamoff = v4l2_m2m_ioctl_streamoff,
+	.vidioc_s_parm = ienc_s_parm,
+	.vidioc_g_parm = ienc_g_parm,
+	.vidioc_enum_framesizes = ienc_enum_framesizes,
+	.vidioc_enum_frameintervals = ienc_enum_frameintervals,
+	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
+	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+	.vidioc_try_encoder_cmd = v4l2_m2m_ioctl_try_encoder_cmd,
+	.vidioc_encoder_cmd = ienc_encoder_cmd,
+};
+
+static int ienc_set_properties(struct venus_inst *inst)
+{
+	struct ienc_controls *ctr = &inst->controls.ienc;
+	struct hfi_multi_slice_control mslice;
+	struct hfi_intra_period intra_period;
+	struct hfi_ltr_mode ltr_mode;
+	struct hfi_framerate frate;
+	struct hfi_intra_refresh ir;
+	struct hfi_enable en;
+	u32 ptype, rate_control;
+	u32 profile, level, hier_p;
+	int ret;
+
+	ret = venus_helper_set_work_mode(inst);
+	if (ret)
+		return ret;
+
+	ptype = HFI_PROPERTY_CONFIG_FRAME_RATE;
+	frate.buffer_type = HFI_BUFFER_OUTPUT;
+	frate.framerate = inst->fps * (1 << 16);
+
+	ret = hfi_session_set_property(inst, ptype, &frate);
+	if (ret)
+		return ret;
+
+	ptype = HFI_PROPERTY_PARAM_VENC_RATE_CONTROL;
+	rate_control = HFI_RATE_CONTROL_CQ;
+
+	ret = hfi_session_set_property(inst, ptype, &rate_control);
+	if (ret)
+		return ret;
+
+	if (rate_control == HFI_RATE_CONTROL_CQ && ctr->image_quality) {
+		struct hfi_heic_frame_quality quality = {0};
+
+		ptype = HFI_PROPERTY_CONFIG_HEIC_FRAME_QUALITY;
+		quality.frame_quality = ctr->image_quality;
+
+		ret = hfi_session_set_property(inst, ptype, &quality);
+		if (ret)
+			return ret;
+	}
+
+	if (ctr->grid_size) {
+		struct hfi_heic_grid_enable grid = {};
+
+		ptype = HFI_PROPERTY_CONFIG_HEIC_GRID_ENABLE;
+		grid.grid_enable = 1;
+
+		ret = hfi_session_set_property(inst, ptype, &grid);
+		if (ret)
+			return ret;
+	}
+
+	/* disable multi slice mode aka set single mode */
+	ptype = HFI_PROPERTY_PARAM_VENC_MULTI_SLICE_CONTROL;
+	mslice.multi_slice = HFI_MULTI_SLICE_OFF;
+	mslice.slice_size = 0;
+
+	ret = hfi_session_set_property(inst, ptype, &mslice);
+	if (ret)
+		return ret;
+
+	/* disable LTR */
+	ptype = HFI_PROPERTY_PARAM_VENC_LTRMODE;
+	ltr_mode.ltr_mode = HFI_LTR_MODE_DISABLE;
+	ltr_mode.ltr_count = 0;
+	ltr_mode.trust_mode = 0;
+
+	ret = hfi_session_set_property(inst, ptype, &ltr_mode);
+	if (ret)
+		return ret;
+
+	/* disable layer encoding */
+	ptype = HFI_PROPERTY_CONFIG_VENC_HIER_P_ENH_LAYER;
+	hier_p = 0;
+
+	ret = hfi_session_set_property(inst, ptype, &hier_p);
+	if (ret)
+		return ret;
+
+	/* disable IR */
+	ptype = HFI_PROPERTY_PARAM_VENC_INTRA_REFRESH;
+	ir.mode = HFI_INTRA_REFRESH_NONE;
+	ir.cir_mbs = 0;
+
+	ret = hfi_session_set_property(inst, ptype, &ir);
+	if (ret)
+		return ret;
+
+	/* Reset P & B frames */
+	ptype = HFI_PROPERTY_CONFIG_VENC_INTRA_PERIOD;
+	intra_period.pframes = 0;
+	intra_period.bframes = 0;
+
+	ret = hfi_session_set_property(inst, ptype, &intra_period);
+	if (ret)
+		return ret;
+
+	switch (inst->hfi_codec) {
+	case HFI_VIDEO_CODEC_HEVC:
+		profile = ctr->profile;
+		level = ctr->level;
+		break;
+	default:
+		profile = 0;
+		level = 0;
+		break;
+	}
+
+	ret = venus_helper_set_profile_level(inst, profile, level);
+	if (ret)
+		return ret;
+
+	/* Prepend SPS and PPS with every I/IDR frame */
+	ptype = HFI_PROPERTY_CONFIG_VENC_SYNC_FRAME_SEQUENCE_HEADER;
+	en.enable = 1;
+	ret = hfi_session_set_property(inst, ptype, &en);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int ienc_init_session(struct venus_inst *inst)
+{
+	struct ienc_controls *ctr = &inst->controls.ienc;
+	u32 width, height;
+	int ret;
+
+	ret = venus_helper_session_init(inst);
+	if (ret == -EALREADY)
+		return 0;
+	else if (ret)
+		return ret;
+
+	ret = venus_helper_set_input_resolution(inst, inst->width,
+						inst->height);
+	if (ret)
+		goto deinit;
+
+	width = inst->width;
+	height = inst->height;
+
+	if (ctr->grid_size) {
+		width = ctr->grid_size;//512;
+		height = ctr->grid_size;//512;
+	}
+
+	ret = venus_helper_set_output_resolution(inst, width, height,
+						 HFI_BUFFER_OUTPUT);
+	if (ret)
+		goto deinit;
+
+	ret = venus_helper_set_color_format(inst, inst->fmt_out->pixfmt);
+	if (ret)
+		goto deinit;
+
+	ret = ienc_set_properties(inst);
+	if (ret)
+		goto deinit;
+
+	return 0;
+deinit:
+	hfi_session_deinit(inst);
+	return ret;
+}
+
+static int ienc_out_num_buffers(struct venus_inst *inst, unsigned int *num)
+{
+	struct hfi_buffer_requirements bufreq;
+	int ret;
+
+	ret = venus_helper_get_bufreq(inst, HFI_BUFFER_INPUT, &bufreq);
+	if (ret)
+		return ret;
+
+	*num = bufreq.count_actual;
+
+	return 0;
+}
+
+static int ienc_queue_setup(struct vb2_queue *q,
+			    unsigned int *num_buffers, unsigned int *num_planes,
+			    unsigned int sizes[], struct device *alloc_devs[])
+{
+	struct venus_inst *inst = vb2_get_drv_priv(q);
+	unsigned int num, min = 4;
+	int ret;
+
+	if (*num_planes) {
+		if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE &&
+		    *num_planes != inst->fmt_out->num_planes)
+			return -EINVAL;
+
+		if (q->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE &&
+		    *num_planes != inst->fmt_cap->num_planes)
+			return -EINVAL;
+
+		if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE &&
+		    sizes[0] < inst->input_buf_size)
+			return -EINVAL;
+
+		if (q->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE &&
+		    sizes[0] < inst->output_buf_size)
+			return -EINVAL;
+
+		return 0;
+	}
+
+	mutex_lock(&inst->lock);
+	ret = ienc_init_session(inst);
+	mutex_unlock(&inst->lock);
+
+	if (ret)
+		return ret;
+
+	switch (q->type) {
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		*num_planes = inst->fmt_out->num_planes;
+
+		ret = ienc_out_num_buffers(inst, &num);
+		if (ret)
+			break;
+
+		num = max(num, min);
+		*num_buffers = max(*num_buffers, num);
+		inst->num_input_bufs = *num_buffers;
+
+		sizes[0] = venus_helper_get_framesz(inst->fmt_out->pixfmt,
+						    inst->width,
+						    inst->height);
+		inst->input_buf_size = sizes[0];
+		break;
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		*num_planes = inst->fmt_cap->num_planes;
+		*num_buffers = max(*num_buffers, min);
+		inst->num_output_bufs = *num_buffers;
+		sizes[0] = venus_helper_get_framesz(inst->fmt_cap->pixfmt,
+						    inst->width,
+						    inst->height);
+		sizes[0] = max(sizes[0], inst->output_buf_size);
+		inst->output_buf_size = sizes[0];
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+
+	return ret;
+}
+
+static int ienc_buf_init(struct vb2_buffer *vb)
+{
+	struct venus_inst *inst = vb2_get_drv_priv(vb->vb2_queue);
+
+	inst->buf_count++;
+
+	return venus_helper_vb2_buf_init(vb);
+}
+
+static void ienc_release_session(struct venus_inst *inst)
+{
+	int ret;
+
+	mutex_lock(&inst->lock);
+
+	ret = hfi_session_deinit(inst);
+	if (ret || inst->session_error)
+		hfi_session_abort(inst);
+
+	mutex_unlock(&inst->lock);
+
+	venus_pm_load_scale(inst);
+	INIT_LIST_HEAD(&inst->registeredbufs);
+	venus_pm_release_core(inst);
+}
+
+static void ienc_buf_cleanup(struct vb2_buffer *vb)
+{
+	struct venus_inst *inst = vb2_get_drv_priv(vb->vb2_queue);
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct venus_buffer *buf = to_venus_buffer(vbuf);
+
+	mutex_lock(&inst->lock);
+	if (vb->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+		if (!list_empty(&inst->registeredbufs))
+			list_del_init(&buf->reg_list);
+	mutex_unlock(&inst->lock);
+
+	inst->buf_count--;
+	if (!inst->buf_count)
+		ienc_release_session(inst);
+}
+
+static int ienc_verify_conf(struct venus_inst *inst)
+{
+	enum hfi_version ver = inst->core->res->hfi_version;
+	struct hfi_buffer_requirements bufreq;
+	int ret;
+
+	if (!inst->num_input_bufs || !inst->num_output_bufs)
+		return -EINVAL;
+
+	ret = venus_helper_get_bufreq(inst, HFI_BUFFER_OUTPUT, &bufreq);
+	if (ret)
+		return ret;
+
+	if (inst->num_output_bufs < bufreq.count_actual ||
+	    inst->num_output_bufs < HFI_BUFREQ_COUNT_MIN(&bufreq, ver))
+		return -EINVAL;
+
+	ret = venus_helper_get_bufreq(inst, HFI_BUFFER_INPUT, &bufreq);
+	if (ret)
+		return ret;
+
+	if (inst->num_input_bufs < bufreq.count_actual ||
+	    inst->num_input_bufs < HFI_BUFREQ_COUNT_MIN(&bufreq, ver))
+		return -EINVAL;
+
+	return 0;
+}
+
+static int ienc_start_streaming(struct vb2_queue *q, unsigned int count)
+{
+	struct venus_inst *inst = vb2_get_drv_priv(q);
+	struct v4l2_m2m_ctx *m2m_ctx = inst->m2m_ctx;
+	int ret;
+
+	mutex_lock(&inst->lock);
+
+	v4l2_m2m_update_start_streaming_state(m2m_ctx, q);
+
+	if (V4L2_TYPE_IS_OUTPUT(q->type))
+		inst->streamon_out = 1;
+	else
+		inst->streamon_cap = 1;
+
+	if (inst->streamon_out && inst->streamon_cap &&
+	    inst->state == INST_INIT) {
+		venus_helper_init_instance(inst);
+
+		inst->sequence_cap = 0;
+		inst->sequence_out = 0;
+
+		ret = ienc_init_session(inst);
+		if (ret)
+			goto bufs_done;
+
+		ret = venus_pm_acquire_core(inst);
+		if (ret)
+			goto deinit_sess;
+
+		ret = ienc_set_properties(inst);
+		if (ret)
+			goto deinit_sess;
+
+		ret = ienc_verify_conf(inst);
+		if (ret)
+			goto deinit_sess;
+
+		ret = venus_helper_set_num_bufs(inst, inst->num_input_bufs,
+						inst->num_output_bufs, 0);
+		if (ret)
+			goto deinit_sess;
+
+		ret = venus_helper_vb2_start_streaming(inst);
+		if (ret)
+			goto deinit_sess;
+
+		venus_helper_process_initial_out_bufs(inst);
+		venus_helper_process_initial_cap_bufs(inst);
+	}  else if (V4L2_TYPE_IS_CAPTURE(q->type) && inst->streamon_cap &&
+		    inst->streamon_out) {
+		ret = venus_helper_vb2_start_streaming(inst);
+		if (ret)
+			goto bufs_done;
+
+		venus_helper_process_initial_out_bufs(inst);
+		venus_helper_process_initial_cap_bufs(inst);
+	}
+
+	mutex_unlock(&inst->lock);
+
+	return 0;
+
+deinit_sess:
+	hfi_session_deinit(inst);
+bufs_done:
+	venus_helper_buffers_done(inst, q->type, VB2_BUF_STATE_QUEUED);
+	if (V4L2_TYPE_IS_OUTPUT(q->type))
+		inst->streamon_out = 0;
+	else
+		inst->streamon_cap = 0;
+	mutex_unlock(&inst->lock);
+	return ret;
+}
+
+static void ienc_stop_streaming(struct vb2_queue *q)
+{
+	struct venus_inst *inst = vb2_get_drv_priv(q);
+	struct v4l2_m2m_ctx *m2m_ctx = inst->m2m_ctx;
+	int ret = -EINVAL;
+
+	mutex_lock(&inst->lock);
+
+	v4l2_m2m_clear_state(m2m_ctx);
+
+	if (V4L2_TYPE_IS_CAPTURE(q->type)) {
+		ret = hfi_session_stop(inst);
+		ret |= hfi_session_unload_res(inst);
+		ret |= venus_helper_unregister_bufs(inst);
+		ret |= venus_helper_intbufs_free(inst);
+	}
+
+	venus_helper_buffers_done(inst, q->type, VB2_BUF_STATE_ERROR);
+
+	if (V4L2_TYPE_IS_OUTPUT(q->type))
+		inst->streamon_out = 0;
+	else
+		inst->streamon_cap = 0;
+
+	mutex_unlock(&inst->lock);
+}
+
+static void ienc_vb2_buf_queue(struct vb2_buffer *vb)
+{
+	struct venus_inst *inst = vb2_get_drv_priv(vb->vb2_queue);
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct v4l2_m2m_ctx *m2m_ctx = inst->m2m_ctx;
+
+	mutex_lock(&inst->lock);
+
+	v4l2_m2m_buf_queue(m2m_ctx, vbuf);
+
+	if (!(inst->streamon_out && inst->streamon_cap))
+		goto unlock;
+
+	if (v4l2_m2m_has_stopped(m2m_ctx))
+		goto unlock;
+
+	venus_helper_process_buf(vb);
+
+unlock:
+	mutex_unlock(&inst->lock);
+}
+
+static const struct vb2_ops ienc_vb2_ops = {
+	.queue_setup = ienc_queue_setup,
+	.buf_init = ienc_buf_init,
+	.buf_cleanup = ienc_buf_cleanup,
+	.buf_prepare = venus_helper_vb2_buf_prepare,
+	.start_streaming = ienc_start_streaming,
+	.stop_streaming = ienc_stop_streaming,
+	.buf_queue = ienc_vb2_buf_queue,
+};
+
+static void ienc_buf_done(struct venus_inst *inst, unsigned int buf_type,
+			  u32 tag, u32 bytesused, u32 data_offset, u32 flags,
+			  u32 hfi_flags, u64 timestamp_us)
+{
+	struct vb2_v4l2_buffer *vbuf;
+	struct vb2_buffer *vb;
+	unsigned int type;
+	struct v4l2_m2m_ctx *m2m_ctx = inst->m2m_ctx;
+
+	if (buf_type == HFI_BUFFER_INPUT)
+		type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+	else
+		type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+
+	vbuf = venus_helper_find_buf(inst, type, tag);
+	if (!vbuf)
+		return;
+
+	vbuf->flags = flags;
+	vb = &vbuf->vb2_buf;
+
+	if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+		vb = &vbuf->vb2_buf;
+		vb2_set_plane_payload(vb, 0, bytesused + data_offset);
+		vb->planes[0].data_offset = data_offset;
+		vb->timestamp = timestamp_us * NSEC_PER_USEC;
+		vbuf->sequence = inst->sequence_cap++;
+
+		if ((!bytesused && m2m_ctx->is_draining) ||
+		    (vbuf->flags & V4L2_BUF_FLAG_LAST)) {
+			vbuf->flags |= V4L2_BUF_FLAG_LAST;
+			v4l2_m2m_mark_stopped(inst->m2m_ctx);
+		}
+	} else {
+		vbuf->sequence = inst->sequence_out++;
+	}
+
+	v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_DONE);
+}
+
+static void ienc_event_notify(struct venus_inst *inst, u32 event,
+			      struct hfi_event_data *data)
+{
+	struct device *dev = inst->core->dev_enc;
+
+	if (event == EVT_SESSION_ERROR) {
+		inst->session_error = true;
+		dev_err(dev, "enc: event session error %x\n", inst->error);
+	}
+}
+
+static const struct hfi_inst_ops ienc_hfi_ops = {
+	.buf_done = ienc_buf_done,
+	.event_notify = ienc_event_notify,
+};
+
+static void ienc_m2m_device_run(void *priv)
+{
+}
+
+static const struct v4l2_m2m_ops ienc_m2m_ops = {
+	.device_run = ienc_m2m_device_run,
+	.job_abort = venus_helper_m2m_job_abort,
+};
+
+static int m2m_queue_init(void *priv, struct vb2_queue *src_vq,
+			  struct vb2_queue *dst_vq)
+{
+	struct venus_inst *inst = priv;
+	int ret;
+
+	src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+	src_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
+	src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+	src_vq->ops = &ienc_vb2_ops;
+	src_vq->mem_ops = &vb2_dma_contig_memops;
+	src_vq->drv_priv = inst;
+	src_vq->buf_struct_size = sizeof(struct venus_buffer);
+	src_vq->allow_zero_bytesused = 1;
+	src_vq->min_buffers_needed = 1;
+	src_vq->dev = inst->core->dev;
+	if (inst->core->res->hfi_version == HFI_VERSION_1XX)
+		src_vq->bidirectional = 1;
+	ret = vb2_queue_init(src_vq);
+	if (ret)
+		return ret;
+
+	dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+	dst_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
+	dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+	dst_vq->ops = &ienc_vb2_ops;
+	dst_vq->mem_ops = &vb2_dma_contig_memops;
+	dst_vq->drv_priv = inst;
+	dst_vq->buf_struct_size = sizeof(struct venus_buffer);
+	dst_vq->allow_zero_bytesused = 1;
+	dst_vq->min_buffers_needed = 1;
+	dst_vq->dev = inst->core->dev;
+	return vb2_queue_init(dst_vq);
+}
+
+static void ienc_inst_init(struct venus_inst *inst)
+{
+	inst->fmt_cap = &ienc_formats[1];
+	inst->fmt_out = &ienc_formats[0];
+	inst->width = 1280;
+	inst->height = ALIGN(720, 32);
+	inst->out_width = 1280;
+	inst->out_height = 720;
+	inst->fps = 1;
+	inst->timeperframe.numerator = 1;
+	inst->timeperframe.denominator = 1;
+	inst->hfi_codec = HFI_VIDEO_CODEC_HEVC;
+}
+
+static int ienc_open(struct file *file)
+{
+	struct venus_core *core = video_drvdata(file);
+	struct venus_inst *inst;
+	int ret;
+
+	inst = kzalloc(sizeof(*inst), GFP_KERNEL);
+	if (!inst)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&inst->dpbbufs);
+	INIT_LIST_HEAD(&inst->registeredbufs);
+	INIT_LIST_HEAD(&inst->internalbufs);
+	INIT_LIST_HEAD(&inst->list);
+	mutex_init(&inst->lock);
+
+	inst->core = core;
+	inst->session_type = VIDC_SESSION_TYPE_ENC;
+	inst->clk_data.core_id = VIDC_CORE_ID_DEFAULT;
+	inst->core_acquired = false;
+
+	venus_helper_init_instance(inst);
+
+	ret = pm_runtime_get_sync(core->dev_enc);
+	if (ret < 0)
+		goto err_put_sync;
+
+	ret = ienc_ctrl_init(inst);
+	if (ret)
+		goto err_put_sync;
+
+	ret = hfi_session_create(inst, &ienc_hfi_ops);
+	if (ret)
+		goto err_ctrl_deinit;
+
+	ienc_inst_init(inst);
+
+	/*
+	 * create m2m device for every instance, the m2m context scheduling
+	 * is made by firmware side so we do not need to care about.
+	 */
+	inst->m2m_dev = v4l2_m2m_init(&ienc_m2m_ops);
+	if (IS_ERR(inst->m2m_dev)) {
+		ret = PTR_ERR(inst->m2m_dev);
+		goto err_session_destroy;
+	}
+
+	inst->m2m_ctx = v4l2_m2m_ctx_init(inst->m2m_dev, inst, m2m_queue_init);
+	if (IS_ERR(inst->m2m_ctx)) {
+		ret = PTR_ERR(inst->m2m_ctx);
+		goto err_m2m_release;
+	}
+
+	v4l2_fh_init(&inst->fh, core->vdev_ienc);
+
+	inst->fh.ctrl_handler = &inst->ctrl_handler;
+	v4l2_fh_add(&inst->fh);
+	inst->fh.m2m_ctx = inst->m2m_ctx;
+	file->private_data = &inst->fh;
+
+	return 0;
+
+err_m2m_release:
+	v4l2_m2m_release(inst->m2m_dev);
+err_session_destroy:
+	hfi_session_destroy(inst);
+err_ctrl_deinit:
+	ienc_ctrl_deinit(inst);
+err_put_sync:
+	pm_runtime_put_sync(core->dev_enc);
+	kfree(inst);
+	return ret;
+}
+
+static int ienc_close(struct file *file)
+{
+	struct venus_inst *inst = to_inst(file);
+
+	v4l2_m2m_ctx_release(inst->m2m_ctx);
+	v4l2_m2m_release(inst->m2m_dev);
+	ienc_ctrl_deinit(inst);
+	hfi_session_destroy(inst);
+	mutex_destroy(&inst->lock);
+	v4l2_fh_del(&inst->fh);
+	v4l2_fh_exit(&inst->fh);
+
+	pm_runtime_put_sync(inst->core->dev_enc);
+
+	kfree(inst);
+	return 0;
+}
+
+static const struct v4l2_file_operations ienc_fops = {
+	.owner = THIS_MODULE,
+	.open = ienc_open,
+	.release = ienc_close,
+	.unlocked_ioctl = video_ioctl2,
+	.poll = v4l2_m2m_fop_poll,
+	.mmap = v4l2_m2m_fop_mmap,
+};
+
+static int ienc_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct video_device *vdev;
+	struct venus_core *core;
+	int ret;
+
+	if (!dev->parent)
+		return -EPROBE_DEFER;
+
+	core = dev_get_drvdata(dev->parent);
+	if (!core)
+		return -EPROBE_DEFER;
+
+	if (!IS_V6(core))
+		return -ENODEV;
+
+	platform_set_drvdata(pdev, core);
+
+	if (core->pm_ops->venc_get) {
+		ret = core->pm_ops->venc_get(dev);
+		if (ret)
+			return ret;
+	}
+
+	vdev = video_device_alloc();
+	if (!vdev)
+		return -ENOMEM;
+
+	strscpy(vdev->name, "qcom-venus-image-encoder", sizeof(vdev->name));
+	vdev->release = video_device_release;
+	vdev->fops = &ienc_fops;
+	vdev->ioctl_ops = &ienc_ioctl_ops;
+	vdev->vfl_dir = VFL_DIR_M2M;
+	vdev->v4l2_dev = &core->v4l2_dev;
+	vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+
+	ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+	if (ret)
+		goto err_vdev_release;
+
+	core->vdev_ienc = vdev;
+	core->dev_ienc = dev;
+
+	video_set_drvdata(vdev, core);
+	pm_runtime_enable(dev);
+
+	return 0;
+
+err_vdev_release:
+	video_device_release(vdev);
+	return ret;
+}
+
+static int ienc_remove(struct platform_device *pdev)
+{
+	struct venus_core *core = dev_get_drvdata(pdev->dev.parent);
+
+	video_unregister_device(core->vdev_ienc);
+	pm_runtime_disable(core->dev_ienc);
+
+	if (core->pm_ops->venc_put)
+		core->pm_ops->venc_put(core->dev_ienc);
+
+	return 0;
+}
+
+static __maybe_unused int ienc_runtime_suspend(struct device *dev)
+{
+	struct venus_core *core = dev_get_drvdata(dev);
+	const struct venus_pm_ops *pm_ops = core->pm_ops;
+	int ret = 0;
+
+	if (pm_ops->venc_power)
+		ret = pm_ops->venc_power(dev, POWER_OFF);
+
+	return ret;
+}
+
+static __maybe_unused int ienc_runtime_resume(struct device *dev)
+{
+	struct venus_core *core = dev_get_drvdata(dev);
+	const struct venus_pm_ops *pm_ops = core->pm_ops;
+	int ret = 0;
+
+	if (pm_ops->venc_power)
+		ret = pm_ops->venc_power(dev, POWER_ON);
+
+	return ret;
+}
+
+static const struct dev_pm_ops ienc_pm_ops = {
+	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+				pm_runtime_force_resume)
+	SET_RUNTIME_PM_OPS(ienc_runtime_suspend, ienc_runtime_resume, NULL)
+};
+
+static const struct of_device_id ienc_dt_match[] = {
+	{ .compatible = "venus-image-encoder" },
+	{ }
+};
+MODULE_DEVICE_TABLE(of, ienc_dt_match);
+
+static struct platform_driver qcom_venus_ienc_driver = {
+	.probe = ienc_probe,
+	.remove = ienc_remove,
+	.driver = {
+		.name = "qcom-venus-image-encoder",
+		.of_match_table = ienc_dt_match,
+		.pm = &ienc_pm_ops,
+	},
+};
+module_platform_driver(qcom_venus_ienc_driver);
+
+MODULE_ALIAS("platform:qcom-venus-image-encoder");
+MODULE_DESCRIPTION("Qualcomm Venus image encoder driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/media/platform/qcom/venus/ienc.h b/drivers/media/platform/qcom/venus/ienc.h
new file mode 100644
index 000000000000..6d050e0c3b7f
--- /dev/null
+++ b/drivers/media/platform/qcom/venus/ienc.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2012-2016, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2017 Linaro Ltd.
+ */
+#ifndef __VENUS_VENC_H__
+#define __VENUS_VENC_H__
+
+struct venus_inst;
+
+int ienc_ctrl_init(struct venus_inst *inst);
+void ienc_ctrl_deinit(struct venus_inst *inst);
+
+#endif
diff --git a/drivers/media/platform/qcom/venus/ienc_ctrls.c b/drivers/media/platform/qcom/venus/ienc_ctrls.c
new file mode 100644
index 000000000000..e925b20834b0
--- /dev/null
+++ b/drivers/media/platform/qcom/venus/ienc_ctrls.c
@@ -0,0 +1,83 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2012-2016, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2017 Linaro Ltd.
+ */
+#include <linux/types.h>
+#include <media/v4l2-ctrls.h>
+
+#include "core.h"
+#include "ienc.h"
+
+static int ienc_op_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct venus_inst *inst = ctrl_to_inst(ctrl);
+	struct ienc_controls *ctr = &inst->controls.ienc;
+
+	switch (ctrl->id) {
+	case V4L2_CID_MPEG_VIDEO_HEVC_PROFILE:
+		ctr->profile = ctrl->val;
+		break;
+	case V4L2_CID_MPEG_VIDEO_HEVC_LEVEL:
+		ctr->level = ctrl->val;
+		break;
+	case V4L2_CID_MPEG_VIDEO_CONSTANT_QUALITY:
+		ctr->image_quality = ctrl->val;
+		break;
+	case V4L2_CID_MPEG_VIDEO_HEIC_GRID_SIZE:
+		ctr->grid_size = ctrl->val;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops ienc_ctrl_ops = {
+	.s_ctrl = ienc_op_s_ctrl,
+};
+
+int ienc_ctrl_init(struct venus_inst *inst)
+{
+	int ret;
+
+	ret = v4l2_ctrl_handler_init(&inst->ctrl_handler, 3);
+	if (ret)
+		return ret;
+
+	v4l2_ctrl_new_std_menu(&inst->ctrl_handler, &ienc_ctrl_ops,
+		V4L2_CID_MPEG_VIDEO_HEVC_PROFILE,
+		V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN_STILL_PICTURE,
+		~(1 << V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN_STILL_PICTURE),
+		V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN_STILL_PICTURE);
+
+	v4l2_ctrl_new_std_menu(&inst->ctrl_handler, &ienc_ctrl_ops,
+		V4L2_CID_MPEG_VIDEO_HEVC_LEVEL,
+		V4L2_MPEG_VIDEO_HEVC_LEVEL_6_2,
+		0, V4L2_MPEG_VIDEO_HEVC_LEVEL_1);
+
+	v4l2_ctrl_new_std(&inst->ctrl_handler, &ienc_ctrl_ops,
+			  V4L2_CID_MPEG_VIDEO_CONSTANT_QUALITY, 0, 100, 1, 80);
+
+	v4l2_ctrl_new_std(&inst->ctrl_handler, &ienc_ctrl_ops,
+			  V4L2_CID_MPEG_VIDEO_HEIC_GRID_SIZE, 0, 512, 512, 0);
+
+	ret = inst->ctrl_handler.error;
+	if (ret)
+		goto err;
+
+	ret = v4l2_ctrl_handler_setup(&inst->ctrl_handler);
+	if (ret)
+		goto err;
+
+	return 0;
+err:
+	v4l2_ctrl_handler_free(&inst->ctrl_handler);
+	return ret;
+}
+
+void ienc_ctrl_deinit(struct venus_inst *inst)
+{
+	v4l2_ctrl_handler_free(&inst->ctrl_handler);
+}
-- 
2.25.1

Re: [RFC/WIP 0/4] HEIC image encoder

[Nicolas Dufresne]

Le jeudi 29 avril 2021 à 16:28 +0300, Stanimir Varbanov a écrit :
> Hi,
> 
> HEIC (High-Efficiency Image Container) is a variant of HEIF (High
> Efficiency Image File Format) where HEVC/h265 codec is used to encode
> images.  For more info see [1].
> 
> In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC.
> The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is
> also an option).
> 
> There are two encoding modes which should be selectable by clients:
>     1. Regular image encoding
>     2. Grid image encoding
> 
> 1. Regular image encoding
> 
> Propose to reuse stateful video encoder spec [2].
> 
> - queuing one OUTPUT buffer will produce one CAPTURE buffer.  The
> client could trigger Drain sequence at any point of time.

Do you know the rationale why the normal HEVC encoder isn't used and then muxed
into the HEIF container ? It is these days quite atypical for HW to handle the
container part. Perhaps they hacked the header, I am not so familiar with these
new ISOMBF based image format (AV1 got something very similar, though less
convoluted).

> 
> 2. Grid image encoding
> 
> Propose to reuse stateful video encoder spec [2].
> 
> - queuing one OUTPUT buffer will produce a number of grids CAPTURE
> buffers.  The client could trigger Drain sequence at any point of time.
> 
> This image encoding mode is used when the input image resolution is
> bigger then the hardware can support and/or for compatibility reasons
> (for exmaple, the HEIC decoding hardware is not capable to decode higher
> than VGA resolutions).
> 
> In this mode the input image is divided on square blocks (we call them grids)
> and every block is encoded separately (the Venus driver presently supports 
> grid size of 512x512 but that could be changed in the future).

Are the blocks always power of two ? Or multiple of 16 ? How do you expose this
information to application ? It sounds like an HW decoder would also need to
expose these restrictions. Userspace will need to be able to check without
trying if the HW decoder handles the grid side to be able to fallback to
software decoding.

> 
> To set the grid size we use a new v4l2 control.
> 
> The side effect of this mode is that the client have to set the v4l2
> control and thus enable grid encoding before setting the formats on
> CAPTURE and OUTPUT queues, because the grid size reflects on the
> selected resolutions. Also the horizontal and vertical strides will
> also be affected because thеy have to be aligned to the grid size
> in order to satisfy DMA alignment restrictions.
> 
> Using of v4l2 control to set up Grid mode and Grid size above looks
> inpractical and somehow breaks the v4l2 and v4l2 control rules, so
> I'd give one more option. 

The stasteless decoders uses a control that must be set after the OUTPUT format,
but before enumerating capture formats. Not exactly the same, but there is a
control that interact with the format negotiation.

> 
> Encoding the Grid mode/size in the new proposed HEIC pixel format:
> 
>    V4L2_PIX_FMT_HEIC - Regular HEIC image encoding
>    V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 
>    and so on ...

I would be worry of the about of formats that may generates.

> 
> Comments and suggestions are welcome!
> 
> regards,
> Stan
> 
> [1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format
> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html
> 
> 
> Stanimir Varbanov (4):
>   media: Add HEIC compressed pixel format
>   v4l2-ctrls: Add HEIC grid size control
>   venus: helpers: Add a new helper for buffer processing
>   venus: Add HEIC image encoder
> 
>  .../media/v4l/pixfmt-compressed.rst           |   12 +
>  drivers/media/platform/qcom/venus/Makefile    |    2 +
>  drivers/media/platform/qcom/venus/core.h      |   10 +
>  drivers/media/platform/qcom/venus/helpers.c   |   20 +
>  drivers/media/platform/qcom/venus/helpers.h   |    1 +
>  drivers/media/platform/qcom/venus/hfi_cmds.c  |   10 +-
>  .../media/platform/qcom/venus/hfi_helper.h    |    5 +
>  drivers/media/platform/qcom/venus/ienc.c      | 1348 +++++++++++++++++
>  drivers/media/platform/qcom/venus/ienc.h      |   14 +
>  .../media/platform/qcom/venus/ienc_ctrls.c    |   83 +
>  drivers/media/v4l2-core/v4l2-ctrls.c          |    3 +
>  drivers/media/v4l2-core/v4l2-ioctl.c          |    1 +
>  include/uapi/linux/v4l2-controls.h            |    1 +
>  include/uapi/linux/videodev2.h                |    1 +
>  14 files changed, 1510 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/platform/qcom/venus/ienc.c
>  create mode 100644 drivers/media/platform/qcom/venus/ienc.h
>  create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c
> 

Re: [RFC/WIP 1/4] media: Add HEIC compressed pixel format

[Nicolas Dufresne]

Le jeudi 29 avril 2021 à 16:28 +0300, Stanimir Varbanov a écrit :
> Add HEIC (High-Efficiency Image Container) pixel format. This an
> image container which use HEVC codec to encoded images.
> 
> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
> ---
>  .../userspace-api/media/v4l/pixfmt-compressed.rst    | 12 ++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c                 |  1 +
>  include/uapi/linux/videodev2.h                       |  1 +
>  3 files changed, 14 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst b/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
> index ba6c0c961204..246bff90dcac 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
> @@ -186,6 +186,18 @@ Compressed Formats
>  	If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM``
>  	then the decoder has no	requirements since it can parse all the
>  	information from the raw bytestream.
> +    * .. _V4L2-PIX-FMT-HEIC:
> +
> +      - ``V4L2_PIX_FMT_HEIC``
> +      - 'HEIC'
> +      - High Efficiency Image Container is an image container file format which
> +        uses HEVC encoding and it is a variant of HEIF (High Efficiency Image File)
> +        format.

Can you clarify, is it expected that an HEIF container be compatible or not ?
Assuming this exist. The HEIC being a brand name, and not really a standard
seems rather confusing. Is this is right name, or should you introduce HEIF with
variant control, similar to HEVC profile control.

Speaking of profile, does it inherit anything from HEVC ? So we need to set HEVC
pofile/level ? Is there some way's to affect the quality or is it the HEVC QP
controls ?

> 
> 
> +	The decoder expects one Access Unit per buffer.
> +	The encoder generates one Access Unit per buffer.
> +	If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM``
> +	then the decoder has no	requirements since it can parse all the
> +	information from the raw bytestream.
>      * .. _V4L2-PIX-FMT-HEVC-SLICE:
>  
>        - ``V4L2_PIX_FMT_HEVC_SLICE``
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 31d1342e61e8..3a1b4c3a76c8 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1454,6 +1454,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  		case V4L2_PIX_FMT_S5C_UYVY_JPG:	descr = "S5C73MX interleaved UYVY/JPEG"; break;
>  		case V4L2_PIX_FMT_MT21C:	descr = "Mediatek Compressed Format"; break;
>  		case V4L2_PIX_FMT_SUNXI_TILED_NV12: descr = "Sunxi Tiled NV12 Format"; break;
> +		case V4L2_PIX_FMT_HEIC:		descr = "HEIC Image Format"; break;
>  		default:
>  			if (fmt->description[0])
>  				return;
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 79dbde3bcf8d..2153b5c31d46 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -699,6 +699,7 @@ struct v4l2_pix_format {
>  #define V4L2_PIX_FMT_FWHT     v4l2_fourcc('F', 'W', 'H', 'T') /* Fast Walsh Hadamard Transform (vicodec) */
>  #define V4L2_PIX_FMT_FWHT_STATELESS     v4l2_fourcc('S', 'F', 'W', 'H') /* Stateless FWHT (vicodec) */
>  #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
> +#define V4L2_PIX_FMT_HEIC	v4l2_fourcc('H', 'E', 'I', 'C') /* HEIC HEVC image format */
>  
>  /*  Vendor-specific formats   */
>  #define V4L2_PIX_FMT_CPIA1    v4l2_fourcc('C', 'P', 'I', 'A') /* cpia1 YUV */

Re: [RFC/WIP 0/4] HEIC image encoder

[Stanimir Varbanov]

Hi Nicolas,

Thanks for comments!

On 5/18/21 8:07 PM, Nicolas Dufresne wrote:
> Le jeudi 29 avril 2021 à 16:28 +0300, Stanimir Varbanov a écrit :
>> Hi,
>>
>> HEIC (High-Efficiency Image Container) is a variant of HEIF (High
>> Efficiency Image File Format) where HEVC/h265 codec is used to encode
>> images.  For more info see [1].
>>
>> In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC.
>> The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is
>> also an option).
>>
>> There are two encoding modes which should be selectable by clients:
>>     1. Regular image encoding
>>     2. Grid image encoding
>>
>> 1. Regular image encoding
>>
>> Propose to reuse stateful video encoder spec [2].
>>
>> - queuing one OUTPUT buffer will produce one CAPTURE buffer.  The
>> client could trigger Drain sequence at any point of time.
> 
> Do you know the rationale why the normal HEVC encoder isn't used and then muxed
> into the HEIF container ? It is these days quite atypical for HW to handle the
> container part. Perhaps they hacked the header, I am not so familiar with these
> new ISOMBF based image format (AV1 got something very similar, though less
> convoluted).

Maybe I did not explain well, but the Venus (and all the knowledge I
have is based on it) does not implement the container part. The
container part is implemented in client. For example I used libheif to
create .heic image file from Venus encoded hevc bitstream.

>>
>> 2. Grid image encoding
>>
>> Propose to reuse stateful video encoder spec [2].
>>
>> - queuing one OUTPUT buffer will produce a number of grids CAPTURE
>> buffers.  The client could trigger Drain sequence at any point of time.
>>
>> This image encoding mode is used when the input image resolution is
>> bigger then the hardware can support and/or for compatibility reasons
>> (for exmaple, the HEIC decoding hardware is not capable to decode higher
>> than VGA resolutions).
>>
>> In this mode the input image is divided on square blocks (we call them grids)
>> and every block is encoded separately (the Venus driver presently supports 
>> grid size of 512x512 but that could be changed in the future).
> 
> Are the blocks always power of two ? Or multiple of 16 ? How do you expose this

I guess the blocks will definitely be a power of two. As far as for
multiple of 16, I guess grid size should be multiple of CTUs (32x32 or
64x64) in case of HEVC. It might be different for the other codecs.

> information to application ? It sounds like an HW decoder would also need to
> expose these restrictions. Userspace will need to be able to check without
> trying if the HW decoder handles the grid side to be able to fallback to
> software decoding.

Depending on what we will decide :
 - use v4l2 control for setting the grid size then we can use
VIDIOC_QUERYCTRL
 - if we decide to set the grid size on the CAPTURE queue SFMT the we
can use VIDIOC_ENUM_FMT

> 
>>
>> To set the grid size we use a new v4l2 control.
>>
>> The side effect of this mode is that the client have to set the v4l2
>> control and thus enable grid encoding before setting the formats on
>> CAPTURE and OUTPUT queues, because the grid size reflects on the
>> selected resolutions. Also the horizontal and vertical strides will
>> also be affected because thеy have to be aligned to the grid size
>> in order to satisfy DMA alignment restrictions.
>>
>> Using of v4l2 control to set up Grid mode and Grid size above looks
>> inpractical and somehow breaks the v4l2 and v4l2 control rules, so
>> I'd give one more option. 
> 
> The stasteless decoders uses a control that must be set after the OUTPUT format,
> but before enumerating capture formats. Not exactly the same, but there is a
> control that interact with the format negotiation.

I'd try to avoid such control if possible.

> 
>>
>> Encoding the Grid mode/size in the new proposed HEIC pixel format:
>>
>>    V4L2_PIX_FMT_HEIC - Regular HEIC image encoding
>>    V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 
>>    and so on ...
> 
> I would be worry of the about of formats that may generates.
> 
>>
>> Comments and suggestions are welcome!
>>
>> regards,
>> Stan
>>
>> [1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format
>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html
>>
>>
>> Stanimir Varbanov (4):
>>   media: Add HEIC compressed pixel format
>>   v4l2-ctrls: Add HEIC grid size control
>>   venus: helpers: Add a new helper for buffer processing
>>   venus: Add HEIC image encoder
>>
>>  .../media/v4l/pixfmt-compressed.rst           |   12 +
>>  drivers/media/platform/qcom/venus/Makefile    |    2 +
>>  drivers/media/platform/qcom/venus/core.h      |   10 +
>>  drivers/media/platform/qcom/venus/helpers.c   |   20 +
>>  drivers/media/platform/qcom/venus/helpers.h   |    1 +
>>  drivers/media/platform/qcom/venus/hfi_cmds.c  |   10 +-
>>  .../media/platform/qcom/venus/hfi_helper.h    |    5 +
>>  drivers/media/platform/qcom/venus/ienc.c      | 1348 +++++++++++++++++
>>  drivers/media/platform/qcom/venus/ienc.h      |   14 +
>>  .../media/platform/qcom/venus/ienc_ctrls.c    |   83 +
>>  drivers/media/v4l2-core/v4l2-ctrls.c          |    3 +
>>  drivers/media/v4l2-core/v4l2-ioctl.c          |    1 +
>>  include/uapi/linux/v4l2-controls.h            |    1 +
>>  include/uapi/linux/videodev2.h                |    1 +
>>  14 files changed, 1510 insertions(+), 1 deletion(-)
>>  create mode 100644 drivers/media/platform/qcom/venus/ienc.c
>>  create mode 100644 drivers/media/platform/qcom/venus/ienc.h
>>  create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c
>>
> 
> 

-- 
regards,
Stan

Re: [RFC/WIP 1/4] media: Add HEIC compressed pixel format

[Stanimir Varbanov]

On 5/18/21 8:11 PM, Nicolas Dufresne wrote:
> Le jeudi 29 avril 2021 à 16:28 +0300, Stanimir Varbanov a écrit :
>> Add HEIC (High-Efficiency Image Container) pixel format. This an
>> image container which use HEVC codec to encoded images.
>>
>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
>> ---
>>  .../userspace-api/media/v4l/pixfmt-compressed.rst    | 12 ++++++++++++
>>  drivers/media/v4l2-core/v4l2-ioctl.c                 |  1 +
>>  include/uapi/linux/videodev2.h                       |  1 +
>>  3 files changed, 14 insertions(+)
>>
>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst b/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
>> index ba6c0c961204..246bff90dcac 100644
>> --- a/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst
>> @@ -186,6 +186,18 @@ Compressed Formats
>>  	If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM``
>>  	then the decoder has no	requirements since it can parse all the
>>  	information from the raw bytestream.
>> +    * .. _V4L2-PIX-FMT-HEIC:
>> +
>> +      - ``V4L2_PIX_FMT_HEIC``
>> +      - 'HEIC'
>> +      - High Efficiency Image Container is an image container file format which
>> +        uses HEVC encoding and it is a variant of HEIF (High Efficiency Image File)
>> +        format.
> 
> Can you clarify, is it expected that an HEIF container be compatible or not ?
> Assuming this exist. The HEIC being a brand name, and not really a standard
> seems rather confusing. Is this is right name, or should you introduce HEIF with
> variant control, similar to HEVC profile control.


V4L2_PIX_FMT_HFIF_HEVC is a good option, I guess.

> 
> Speaking of profile, does it inherit anything from HEVC ? So we need to set HEVC
> pofile/level ? Is there some way's to affect the quality or is it the HEVC QP
> controls ?

V4L2_PIX_FMT_HFIF_HEVC will accept
V4L2_MPEG_VIDEO_HEVC_PROFILE_MAIN_STILL_PICTURE profile and the image
quality is set through V4L2_CID_MPEG_VIDEO_CONSTANT_QUALITY

> 
>>
>>
>> +	The decoder expects one Access Unit per buffer.
>> +	The encoder generates one Access Unit per buffer.
>> +	If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM``
>> +	then the decoder has no	requirements since it can parse all the
>> +	information from the raw bytestream.
>>      * .. _V4L2-PIX-FMT-HEVC-SLICE:
>>  
>>        - ``V4L2_PIX_FMT_HEVC_SLICE``
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index 31d1342e61e8..3a1b4c3a76c8 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -1454,6 +1454,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>>  		case V4L2_PIX_FMT_S5C_UYVY_JPG:	descr = "S5C73MX interleaved UYVY/JPEG"; break;
>>  		case V4L2_PIX_FMT_MT21C:	descr = "Mediatek Compressed Format"; break;
>>  		case V4L2_PIX_FMT_SUNXI_TILED_NV12: descr = "Sunxi Tiled NV12 Format"; break;
>> +		case V4L2_PIX_FMT_HEIC:		descr = "HEIC Image Format"; break;
>>  		default:
>>  			if (fmt->description[0])
>>  				return;
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index 79dbde3bcf8d..2153b5c31d46 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -699,6 +699,7 @@ struct v4l2_pix_format {
>>  #define V4L2_PIX_FMT_FWHT     v4l2_fourcc('F', 'W', 'H', 'T') /* Fast Walsh Hadamard Transform (vicodec) */
>>  #define V4L2_PIX_FMT_FWHT_STATELESS     v4l2_fourcc('S', 'F', 'W', 'H') /* Stateless FWHT (vicodec) */
>>  #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
>> +#define V4L2_PIX_FMT_HEIC	v4l2_fourcc('H', 'E', 'I', 'C') /* HEIC HEVC image format */
>>  
>>  /*  Vendor-specific formats   */
>>  #define V4L2_PIX_FMT_CPIA1    v4l2_fourcc('C', 'P', 'I', 'A') /* cpia1 YUV */
> 
> 

-- 
regards,
Stan

Re: [RFC/WIP 0/4] HEIC image encoder

[Hans Verkuil]

Hi Stanimir,

On 29/04/2021 15:28, Stanimir Varbanov wrote:
> Hi,
> 
> HEIC (High-Efficiency Image Container) is a variant of HEIF (High
> Efficiency Image File Format) where HEVC/h265 codec is used to encode
> images.  For more info see [1].
> 
> In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC.
> The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is
> also an option).
> 
> There are two encoding modes which should be selectable by clients:
>     1. Regular image encoding
>     2. Grid image encoding
> 
> 1. Regular image encoding
> 
> Propose to reuse stateful video encoder spec [2].
> 
> - queuing one OUTPUT buffer will produce one CAPTURE buffer.  The
> client could trigger Drain sequence at any point of time.
> 
> 2. Grid image encoding
> 
> Propose to reuse stateful video encoder spec [2].
> 
> - queuing one OUTPUT buffer will produce a number of grids CAPTURE
> buffers.  The client could trigger Drain sequence at any point of time.
> 
> This image encoding mode is used when the input image resolution is
> bigger then the hardware can support and/or for compatibility reasons
> (for exmaple, the HEIC decoding hardware is not capable to decode higher
> than VGA resolutions).

Is grid image encoding part of the spec for this format? Is this something
that the venus hardware needs due to image resolution limitations as
described above?

Would it be possible for the driver to handle this internally? I.e.,
if it detects that it needs to switch to grid mode, can it just encode
each grid and copy it in the capture buffer? This assumes that there is
metadata that can be used by a decoder do find and decode each grid.

> 
> In this mode the input image is divided on square blocks (we call them grids)
> and every block is encoded separately (the Venus driver presently supports 
> grid size of 512x512 but that could be changed in the future).
> 
> To set the grid size we use a new v4l2 control.

Can the driver make a choice of the grid size, and the control just
reports the grid size? I.e., does it make sense for userspace to set
this?

The wiki page [1] doesn't mention grids, so where does this come from?
Is it part of some spec? Or is it a venus-specific feature?

> 
> The side effect of this mode is that the client have to set the v4l2
> control and thus enable grid encoding before setting the formats on
> CAPTURE and OUTPUT queues, because the grid size reflects on the
> selected resolutions. Also the horizontal and vertical strides will
> also be affected because thеy have to be aligned to the grid size
> in order to satisfy DMA alignment restrictions.
> 
> Using of v4l2 control to set up Grid mode and Grid size above looks
> inpractical and somehow breaks the v4l2 and v4l2 control rules, so
> I'd give one more option. 
> 
> Encoding the Grid mode/size in the new proposed HEIC pixel format:
> 
>    V4L2_PIX_FMT_HEIC - Regular HEIC image encoding
>    V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 
>    and so on ...
> 
> Comments and suggestions are welcome!

I notice that this RFC just talks about the encoder, does venus also
support a decoder? How would a HW decoder handle grids?

Regards,

	Hans

> 
> regards,
> Stan
> 
> [1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format
> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html
> 
> 
> Stanimir Varbanov (4):
>   media: Add HEIC compressed pixel format
>   v4l2-ctrls: Add HEIC grid size control
>   venus: helpers: Add a new helper for buffer processing
>   venus: Add HEIC image encoder
> 
>  .../media/v4l/pixfmt-compressed.rst           |   12 +
>  drivers/media/platform/qcom/venus/Makefile    |    2 +
>  drivers/media/platform/qcom/venus/core.h      |   10 +
>  drivers/media/platform/qcom/venus/helpers.c   |   20 +
>  drivers/media/platform/qcom/venus/helpers.h   |    1 +
>  drivers/media/platform/qcom/venus/hfi_cmds.c  |   10 +-
>  .../media/platform/qcom/venus/hfi_helper.h    |    5 +
>  drivers/media/platform/qcom/venus/ienc.c      | 1348 +++++++++++++++++
>  drivers/media/platform/qcom/venus/ienc.h      |   14 +
>  .../media/platform/qcom/venus/ienc_ctrls.c    |   83 +
>  drivers/media/v4l2-core/v4l2-ctrls.c          |    3 +
>  drivers/media/v4l2-core/v4l2-ioctl.c          |    1 +
>  include/uapi/linux/v4l2-controls.h            |    1 +
>  include/uapi/linux/videodev2.h                |    1 +
>  14 files changed, 1510 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/platform/qcom/venus/ienc.c
>  create mode 100644 drivers/media/platform/qcom/venus/ienc.h
>  create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c
> 

Re: [RFC/WIP 0/4] HEIC image encoder

[Stanimir Varbanov]

Hi Hans,

On 5/27/21 10:54 AM, Hans Verkuil wrote:
> Hi Stanimir,
> 
> On 29/04/2021 15:28, Stanimir Varbanov wrote:
>> Hi,
>>
>> HEIC (High-Efficiency Image Container) is a variant of HEIF (High
>> Efficiency Image File Format) where HEVC/h265 codec is used to encode
>> images.  For more info see [1].
>>
>> In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC.
>> The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is
>> also an option).
>>
>> There are two encoding modes which should be selectable by clients:
>>     1. Regular image encoding
>>     2. Grid image encoding
>>
>> 1. Regular image encoding
>>
>> Propose to reuse stateful video encoder spec [2].
>>
>> - queuing one OUTPUT buffer will produce one CAPTURE buffer.  The
>> client could trigger Drain sequence at any point of time.
>>
>> 2. Grid image encoding
>>
>> Propose to reuse stateful video encoder spec [2].
>>
>> - queuing one OUTPUT buffer will produce a number of grids CAPTURE
>> buffers.  The client could trigger Drain sequence at any point of time.
>>
>> This image encoding mode is used when the input image resolution is
>> bigger then the hardware can support and/or for compatibility reasons
>> (for exmaple, the HEIC decoding hardware is not capable to decode higher
>> than VGA resolutions).
> 
> Is grid image encoding part of the spec for this format? Is this something
> that the venus hardware needs due to image resolution limitations as
> described above?

Yes, it is part of the ISO/IEC 23008-12 (2017). The spec defines Image
grid derivation, where each tile is a separate image and associated with
derived image of type _grid_ which reconstruct all tiles into a single
image for display.

> 
> Would it be possible for the driver to handle this internally? I.e.,
> if it detects that it needs to switch to grid mode, can it just encode
> each grid and copy it in the capture buffer? This assumes that there is
> metadata that can be used by a decoder do find and decode each grid.
> 

In case that is is part of the spec I don't think we have to do it.
Something more, when each tile is separate image the decoding process
could be done in parallel.

>>
>> In this mode the input image is divided on square blocks (we call them grids)
>> and every block is encoded separately (the Venus driver presently supports 
>> grid size of 512x512 but that could be changed in the future).
>>
>> To set the grid size we use a new v4l2 control.
> 
> Can the driver make a choice of the grid size, and the control just
> reports the grid size? I.e., does it make sense for userspace to set
> this?
> 

I'm not familiar with userspace implementations so far, but my feeling
is that the userspace should configure that - at least this will give
clients flexibility. References with more information [1] - [5].

> The wiki page [1] doesn't mention grids, so where does this come from?
> Is it part of some spec? Or is it a venus-specific feature?
> 
>>
>> The side effect of this mode is that the client have to set the v4l2
>> control and thus enable grid encoding before setting the formats on
>> CAPTURE and OUTPUT queues, because the grid size reflects on the
>> selected resolutions. Also the horizontal and vertical strides will
>> also be affected because thеy have to be aligned to the grid size
>> in order to satisfy DMA alignment restrictions.
>>
>> Using of v4l2 control to set up Grid mode and Grid size above looks
>> inpractical and somehow breaks the v4l2 and v4l2 control rules, so
>> I'd give one more option. 
>>
>> Encoding the Grid mode/size in the new proposed HEIC pixel format:
>>
>>    V4L2_PIX_FMT_HEIC - Regular HEIC image encoding
>>    V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 
>>    and so on ...
>>
>> Comments and suggestions are welcome!
> 
> I notice that this RFC just talks about the encoder, does venus also
> support a decoder? How would a HW decoder handle grids?

AFAIK the decoding part is not doing something special and
reconstructing the whole image from tiles is done by the userspace
client [6].

> 
> Regards,
> 
> 	Hans

-- 
regards,
Stan

[1] https://0xc0000054.github.io/pdn-avif/using-image-grids.html#fnref:3
[2] https://nokiatech.github.io/heif/technical.html
[3] https://github.com/lclevy/canon_cr3/blob/master/heif.md
[4]
https://github.com/nokiatech/heif/blob/master/srcs/api-cpp/GridImageItem.cpp
[5]
https://github.com/strukturag/libheif/blob/master/libheif/heif_context.cc#L163
[6]
https://github.com/strukturag/libheif/blob/master/libheif/heif_context.cc#L1317

Re: [RFC/WIP 0/4] HEIC image encoder

[Nicolas Dufresne]

Le vendredi 11 juin 2021 à 16:12 +0300, Stanimir Varbanov a écrit :
> > 
> > Would it be possible for the driver to handle this internally? I.e.,
> > if it detects that it needs to switch to grid mode, can it just encode
> > each grid and copy it in the capture buffer? This assumes that there is
> > metadata that can be used by a decoder do find and decode each grid.
> > 
> 
> In case that is is part of the spec I don't think we have to do it.
> Something more, when each tile is separate image the decoding process
> could be done in parallel.

Does it means there is no userspace (or at least no Open Source) userspace for
it ?

Re: [RFC/WIP 0/4] HEIC image encoder

[Stanimir Varbanov]

On 6/11/21 5:46 PM, Nicolas Dufresne wrote:
> Le vendredi 11 juin 2021 à 16:12 +0300, Stanimir Varbanov a écrit :
>>>
>>> Would it be possible for the driver to handle this internally? I.e.,
>>> if it detects that it needs to switch to grid mode, can it just encode
>>> each grid and copy it in the capture buffer? This assumes that there is
>>> metadata that can be used by a decoder do find and decode each grid.
>>>
>>
>> In case that is is part of the spec I don't think we have to do it.
>> Something more, when each tile is separate image the decoding process
>> could be done in parallel.
> 
> Does it means there is no userspace (or at least no Open Source) userspace for
> it ?
> 
> 

My searching came to two libraries:
 * the open source library [1].
 * library from Nokia [2].

-- 
regards,
Stan

[1] https://github.com/strukturag/libheif
[2] https://github.com/nokiatech/heif