[PATCH v2 00/14] Clean H264 stateless uAPI

[PATCH v2 00/14] Clean H264 stateless uAPI

[Ezequiel Garcia]

Here's a new round for the H.264 uAPI cleanup, which as discussed
aims at being stabilized and promoted as a first-class public uAPI soon.

It should be noted that there is already GStreamer native
support for this interface, which will be part of 1.18,
once it's released.

I have pushed a branch porting GStreamer to
support these interface changes:

https://gitlab.freedesktop.org/ezequielgarcia/gst-plugins-bad/-/commits/for_h264_uapi_v3

As was discussed the SLICE_PARAMS control is now clarified
to work for one-slice-per-request operation, using CAPTURE
buffer holding features. This is how Cedrus driver is implemented.

The other drivers currently supported Hantro and Rockchip VDEC,
as well as the MTK stateless decoder posted by Alex Courbot
operate in frame-based mode.

These "frame-based" devices parse the slice headers in hardware,
and therefore shall not support SLICE_PARAMS. To that extent,
the redundant bitstream fields are now part of the DECODE_PARAMS
control.

Hopefully now the specification documentation is clear enough.
GStreamer, Chromium and FFmpeg (which I'm sure will be implemented
as soon as we stabilize the API) should serve as reference examples
on how the API is consumed.

Changelog:

v1->v2:
* Clean SLICE_PARAMS documentation, which we don't
  expect to be part of an array anymore. 
* Clarify how frame-based and slice-based modes
  are expected to work.
* Add Cedrus patches to fix field references,
  as requested by Jernej.
* Fix wrongly removed SPS in rkvdec.
* Fix rkvdec DPB reference implementation.
* Fix missing Cedrus and missing control member,
  for prediction weight table control.
* Say "raster scan" instead of "matrix" in the docs.
* Drop duplicated macros and use v4l2_h264_dpb_reference
  for the DPB reference signalling.

RFC->v1: 
* Split prediction weight table to a separate control.
* Increase size of first_mb_in_slice field.
* Cleanup DPB entry interface, to support field coding.
* Increase of DPB entry pic_num field.
* Move slice invariant fields to the per-frame control.

Ezequiel Garcia (10):
  media: uapi: h264: Further clarify scaling lists order
  media: uapi: h264: Split prediction weight parameters
  media: uapi: h264: Increase size of 'first_mb_in_slice' field
  media: uapi: h264: Clean DPB entry interface
  media: uapi: h264: Increase size of DPB entry pic_num
  media: uapi: h264: Drop SLICE_PARAMS 'size' field
  media: uapi: h264: Clarify SLICE_BASED mode
  media: uapi: h264: Clean slice invariants syntax elements
  media: hantro: Don't require unneeded H264_SLICE_PARAMS
  media: rkvdec: Don't require unneeded H264_SLICE_PARAMS

Jernej Skrabec (3):
  media: uapi: h264: Update reference lists
  media: cedrus: h264: Properly configure reference field
  media: cedrus: h264: Fix frame list construction

Philipp Zabel (1):
  media: uapi: h264: Clarify pic_order_cnt_bit_size field

 .../media/v4l/ext-ctrls-codec.rst             | 224 ++++++++++--------
 drivers/media/v4l2-core/v4l2-ctrls.c          |  28 +++
 drivers/media/v4l2-core/v4l2-h264.c           |  12 +-
 drivers/staging/media/hantro/hantro_drv.c     |   5 -
 .../staging/media/hantro/hantro_g1_h264_dec.c |  21 +-
 drivers/staging/media/hantro/hantro_h264.c    |   8 +-
 drivers/staging/media/hantro/hantro_hw.h      |   2 -
 drivers/staging/media/rkvdec/rkvdec-h264.c    |  27 +--
 drivers/staging/media/rkvdec/rkvdec.c         |   5 -
 drivers/staging/media/sunxi/cedrus/cedrus.c   |   7 +
 drivers/staging/media/sunxi/cedrus/cedrus.h   |   1 +
 .../staging/media/sunxi/cedrus/cedrus_dec.c   |   2 +
 .../staging/media/sunxi/cedrus/cedrus_h264.c  |  49 ++--
 include/media/h264-ctrls.h                    |  80 ++++---
 include/media/v4l2-ctrls.h                    |   2 +
 include/media/v4l2-h264.h                     |   3 +-
 16 files changed, 257 insertions(+), 219 deletions(-)

-- 
2.27.0

[PATCH v2 01/14] media: uapi: h264: Update reference lists

[Ezequiel Garcia]

From: Jernej Skrabec <jernej.skrabec@siol.net>

When dealing with with interlaced frames, reference lists must tell if
each particular reference is meant for top or bottom field. This info
is currently not provided at all in the H264 related controls.

Make reference lists hold a structure which will also hold an
enumerator type along index into DPB array. The enumerator must
be used to specify if reference is for top or bottom field.

Currently the only user of these lists is Cedrus which is just compile
fixed here. Actual usage of will come in a following commit.

Signed-off-by: Jernej Skrabec <jernej.skrabec@siol.net>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
v2:
* As pointed out by Jonas, enum v4l2_h264_dpb_reference here.
---
 .../media/v4l/ext-ctrls-codec.rst             | 44 ++++++++++++++++++-
 .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 +--
 include/media/h264-ctrls.h                    | 23 +++++++---
 3 files changed, 62 insertions(+), 11 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index d0d506a444b1..f2b2a381369f 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1843,10 +1843,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - __u32
       - ``slice_group_change_cycle``
       -
-    * - __u8
+    * - struct :c:type:`v4l2_h264_reference`
       - ``ref_pic_list0[32]``
       - Reference picture list after applying the per-slice modifications
-    * - __u8
+    * - struct :c:type:`v4l2_h264_reference`
       - ``ref_pic_list1[32]``
       - Reference picture list after applying the per-slice modifications
     * - __u32
@@ -1926,6 +1926,46 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
       - ``chroma_offset[32][2]``
       -
 
+``Picture Reference``
+
+.. c:type:: v4l2_h264_reference
+
+.. cssclass:: longtable
+
+.. flat-table:: struct v4l2_h264_reference
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - enum :c:type:`v4l2_h264_dpb_reference`
+      - ``reference``
+      - Specifies how the DPB entry is referenced.
+    * - __u8
+      - ``index``
+      - Index into the :c:type:`v4l2_ctrl_h264_decode_params`.dpb array.
+
+.. c:type:: v4l2_h264_dpb_reference
+
+.. cssclass:: longtable
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - ``V4L2_H264_DPB_TOP_REF``
+      - 0x1
+      - The top field in field pair is used for
+        short-term reference.
+    * - ``V4L2_H264_DPB_BOTTOM_REF``
+      - 0x2
+     - The bottom field in field pair is used for
+        short-term reference.
+    * - ``V4L2_H264_DPB_FRAME_REF``
+      - 0x3
+      - The frame (or the top/bottom fields, if it's a field pair)
+        is used for short-term reference.
+
 ``V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (struct)``
     Specifies the decode parameters (as extracted from the bitstream)
     for the associated H264 slice data. This includes the necessary
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
index 54ee2aa423e2..cce527bbdf86 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
@@ -166,8 +166,8 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
 
 static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
 				   struct cedrus_run *run,
-				   const u8 *ref_list, u8 num_ref,
-				   enum cedrus_h264_sram_off sram)
+				   const struct v4l2_h264_reference *ref_list,
+				   u8 num_ref, enum cedrus_h264_sram_off sram)
 {
 	const struct v4l2_ctrl_h264_decode_params *decode = run->h264.decode_params;
 	struct vb2_queue *cap_q;
@@ -188,7 +188,7 @@ static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
 		int buf_idx;
 		u8 dpb_idx;
 
-		dpb_idx = ref_list[i];
+		dpb_idx = ref_list[i].index;
 		dpb = &decode->dpb[dpb_idx];
 
 		if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE))
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index 080fd1293c42..4c0bb7f5fb05 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -19,6 +19,8 @@
  */
 #define V4L2_H264_NUM_DPB_ENTRIES 16
 
+#define V4L2_H264_REF_LIST_LEN (2 * V4L2_H264_NUM_DPB_ENTRIES)
+
 /* Our pixel format isn't stable at the moment */
 #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
 
@@ -140,6 +142,19 @@ struct v4l2_h264_pred_weight_table {
 #define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED	0x04
 #define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH		0x08
 
+enum v4l2_h264_dpb_reference {
+	V4L2_H264_DPB_TOP_REF = 0x1,
+	V4L2_H264_DPB_BOTTOM_REF = 0x2,
+	V4L2_H264_DPB_FRAME_REF = 0x3,
+};
+
+struct v4l2_h264_reference {
+	enum v4l2_h264_dpb_reference fields;
+
+	/* Index into v4l2_ctrl_h264_decode_params.dpb[] */
+	__u8 index;
+};
+
 struct v4l2_ctrl_h264_slice_params {
 	/* Size in bytes, including header */
 	__u32 size;
@@ -178,12 +193,8 @@ struct v4l2_ctrl_h264_slice_params {
 	__u8 num_ref_idx_l1_active_minus1;
 	__u32 slice_group_change_cycle;
 
-	/*
-	 * Entries on each list are indices into
-	 * v4l2_ctrl_h264_decode_params.dpb[].
-	 */
-	__u8 ref_pic_list0[32];
-	__u8 ref_pic_list1[32];
+	struct v4l2_h264_reference ref_pic_list0[V4L2_H264_REF_LIST_LEN];
+	struct v4l2_h264_reference ref_pic_list1[V4L2_H264_REF_LIST_LEN];
 
 	__u32 flags;
 };
-- 
2.27.0

[PATCH v2 02/14] media: uapi: h264: Further clarify scaling lists order

[Ezequiel Garcia]

Commit 0b0393d59eb4a ("media: uapi: h264: clarify
expected scaling_list_4x4/8x8 order") improved the
documentation on H264 scaling lists order.

This commit improves the documentation by clarifying
that the lists themselves are expected in raster scan order.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
v2:
* Use "raster scan order" instead of "matrix order"
---
 Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index f2b2a381369f..d1438b1e259f 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1725,12 +1725,14 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
       - ``scaling_list_4x4[6][16]``
       - Scaling matrix after applying the inverse scanning process.
         Expected list order is Intra Y, Intra Cb, Intra Cr, Inter Y,
-        Inter Cb, Inter Cr.
+        Inter Cb, Inter Cr. The values on each scaling list are
+        expected in raster scan order.
     * - __u8
       - ``scaling_list_8x8[6][64]``
       - Scaling matrix after applying the inverse scanning process.
         Expected list order is Intra Y, Inter Y, Intra Cb, Inter Cb,
-        Intra Cr, Inter Cr.
+        Intra Cr, Inter Cr. The values on each scaling list are
+        expected in raster scan order.
 
 ``V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS (struct)``
     Specifies the slice parameters (as extracted from the bitstream)
-- 
2.27.0

[PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

The prediction weight parameters are only required under
certain conditions, which depend on slice header parameters.

As specified in section 7.3.3 Slice header syntax, the prediction
weight table is present if:

((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
(weighted_bipred_idc == 1 && slice_type == B))

Given its size, it makes sense to move this table to its control,
so applications can avoid passing it if the slice doesn't specify it.

Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
is 772 bytes.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
v2: Fix missing Cedrus changes and mssing control declaration,
    as noted by Hans and Jernej.
---
 .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
 drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
 drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
 drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
 .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
 .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
 include/media/h264-ctrls.h                    |  5 +++--
 include/media/v4l2-ctrls.h                    |  2 ++
 8 files changed, 37 insertions(+), 13 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index d1438b1e259f..c36ce5a95fc5 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1879,18 +1879,23 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
       - 0x00000008
       -
 
-``Prediction Weight Table``
+``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
+    Prediction weight table defined according to :ref:`h264`,
+    section 7.4.3.2 "Prediction Weight Table Semantics".
+    The prediction weight table must be passed by applications
+    under the conditions explained in section 7.3.3 "Slice header
+    syntax".
 
-    The bitstream parameters are defined according to :ref:`h264`,
-    section 7.4.3.2 "Prediction Weight Table Semantics". For further
-    documentation, refer to the above specification, unless there is
-    an explicit comment stating otherwise.
+    .. note::
+
+       This compound control is not yet part of the public kernel API and
+       it is expected to change.
 
-.. c:type:: v4l2_h264_pred_weight_table
+.. c:type:: v4l2_ctrl_h264_pred_weights
 
 .. cssclass:: longtable
 
-.. flat-table:: struct v4l2_h264_pred_weight_table
+.. flat-table:: struct v4l2_ctrl_h264_pred_weights
     :header-rows:  0
     :stub-columns: 0
     :widths:       1 1 2
diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 3f3fbcd60cc6..76c8dc8fb31c 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
 	case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:		return "H264 Decode Parameters";
 	case V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:		return "H264 Decode Mode";
 	case V4L2_CID_MPEG_VIDEO_H264_START_CODE:		return "H264 Start Code";
+	case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:		return "H264 Prediction Weight Table";
 	case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:			return "MPEG2 Level";
 	case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:			return "MPEG2 Profile";
 	case V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:		return "MPEG4 I-Frame QP Value";
@@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
 	case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
 		*type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
 		break;
+	case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
+		*type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
+		break;
 	case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
 		*type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
 		break;
@@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
 	case V4L2_CTRL_TYPE_H264_SPS:
 	case V4L2_CTRL_TYPE_H264_PPS:
 	case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
+	case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
 	case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
 	case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
 		break;
@@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
 	case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
 		elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
 		break;
+	case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
+		elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
+		break;
 	case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
 		elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
 		break;
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c
index bc27f9430eeb..027cdd1be5a0 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
@@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[] = {
 		.codec		= CEDRUS_CODEC_H264,
 		.required	= true,
 	},
+	{
+		.cfg = {
+			.id	= V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
+		},
+		.codec		= CEDRUS_CODEC_H264,
+		.required	= true,
+	},
 	{
 		.cfg = {
 			.id	= V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE,
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.h b/drivers/staging/media/sunxi/cedrus/cedrus.h
index 96765555ab8a..93c843ae14bb 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus.h
+++ b/drivers/staging/media/sunxi/cedrus/cedrus.h
@@ -62,6 +62,7 @@ struct cedrus_h264_run {
 	const struct v4l2_ctrl_h264_scaling_matrix	*scaling_matrix;
 	const struct v4l2_ctrl_h264_slice_params	*slice_params;
 	const struct v4l2_ctrl_h264_sps			*sps;
+	const struct v4l2_ctrl_h264_pred_weights	*pred_weights;
 };
 
 struct cedrus_mpeg2_run {
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
index 58c48e4fdfe9..6385026d1b6b 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
@@ -57,6 +57,8 @@ void cedrus_device_run(void *priv)
 			V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS);
 		run.h264.sps = cedrus_find_control_data(ctx,
 			V4L2_CID_MPEG_VIDEO_H264_SPS);
+		run.h264.pred_weights = cedrus_find_control_data(ctx,
+			V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS);
 		break;
 
 	case V4L2_PIX_FMT_HEVC_SLICE:
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
index cce527bbdf86..a9ba78b15907 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
@@ -256,10 +256,8 @@ static void cedrus_write_scaling_lists(struct cedrus_ctx *ctx,
 static void cedrus_write_pred_weight_table(struct cedrus_ctx *ctx,
 					   struct cedrus_run *run)
 {
-	const struct v4l2_ctrl_h264_slice_params *slice =
-		run->h264.slice_params;
-	const struct v4l2_h264_pred_weight_table *pred_weight =
-		&slice->pred_weight_table;
+	const struct v4l2_ctrl_h264_pred_weights *pred_weight =
+		run->h264.pred_weights;
 	struct cedrus_dev *dev = ctx->dev;
 	int i, j, k;
 
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index 4c0bb7f5fb05..54cd9bec0b23 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -36,6 +36,7 @@
 #define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS	(V4L2_CID_MPEG_BASE+1004)
 #define V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE	(V4L2_CID_MPEG_BASE+1005)
 #define V4L2_CID_MPEG_VIDEO_H264_START_CODE	(V4L2_CID_MPEG_BASE+1006)
+#define V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS	(V4L2_CID_MPEG_BASE+1007)
 
 /* enum v4l2_ctrl_type type values */
 #define V4L2_CTRL_TYPE_H264_SPS			0x0110
@@ -43,6 +44,7 @@
 #define V4L2_CTRL_TYPE_H264_SCALING_MATRIX	0x0112
 #define V4L2_CTRL_TYPE_H264_SLICE_PARAMS	0x0113
 #define V4L2_CTRL_TYPE_H264_DECODE_PARAMS	0x0114
+#define V4L2_CTRL_TYPE_H264_PRED_WEIGHTS	0x0115
 
 enum v4l2_mpeg_video_h264_decode_mode {
 	V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED,
@@ -125,7 +127,7 @@ struct v4l2_h264_weight_factors {
 	__s16 chroma_offset[32][2];
 };
 
-struct v4l2_h264_pred_weight_table {
+struct v4l2_ctrl_h264_pred_weights {
 	__u16 luma_log2_weight_denom;
 	__u16 chroma_log2_weight_denom;
 	struct v4l2_h264_weight_factors weight_factors[2];
@@ -177,7 +179,6 @@ struct v4l2_ctrl_h264_slice_params {
 	__s32 delta_pic_order_cnt0;
 	__s32 delta_pic_order_cnt1;
 
-	struct v4l2_h264_pred_weight_table pred_weight_table;
 	/* Size in bits of dec_ref_pic_marking() syntax element. */
 	__u32 dec_ref_pic_marking_bit_size;
 	/* Size in bits of pic order count syntax. */
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index f40e2cbb21d3..cb25f345e9ad 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -51,6 +51,7 @@ struct video_device;
  * @p_h264_scaling_matrix:	Pointer to a struct v4l2_ctrl_h264_scaling_matrix.
  * @p_h264_slice_params:	Pointer to a struct v4l2_ctrl_h264_slice_params.
  * @p_h264_decode_params:	Pointer to a struct v4l2_ctrl_h264_decode_params.
+ * @p_h264_pred_weights:	Pointer to a struct v4l2_ctrl_h264_pred_weights.
  * @p_vp8_frame_header:		Pointer to a VP8 frame header structure.
  * @p_hevc_sps:			Pointer to an HEVC sequence parameter set structure.
  * @p_hevc_pps:			Pointer to an HEVC picture parameter set structure.
@@ -74,6 +75,7 @@ union v4l2_ctrl_ptr {
 	struct v4l2_ctrl_h264_scaling_matrix *p_h264_scaling_matrix;
 	struct v4l2_ctrl_h264_slice_params *p_h264_slice_params;
 	struct v4l2_ctrl_h264_decode_params *p_h264_decode_params;
+	struct v4l2_ctrl_h264_pred_weights *p_h264_pred_weights;
 	struct v4l2_ctrl_vp8_frame_header *p_vp8_frame_header;
 	struct v4l2_ctrl_hevc_sps *p_hevc_sps;
 	struct v4l2_ctrl_hevc_pps *p_hevc_pps;
-- 
2.27.0

[PATCH v2 04/14] media: uapi: h264: Clarify pic_order_cnt_bit_size field

[Ezequiel Garcia]

From: Philipp Zabel <p.zabel@pengutronix.de>

Since pic_order_cnt_bit_size is not a syntax element itself, explicitly
state that it is the total size in bits of the pic_order_cnt_lsb,
delta_pic_order_cnt_bottom, delta_pic_order_cnt[0], and
delta_pic_order_cnt[1] syntax elements contained in the slice.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
[Ezequiel: rebase]
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
---
 Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index c36ce5a95fc5..e4dec3f9b71a 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1815,7 +1815,9 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
       - Size in bits of the dec_ref_pic_marking() syntax element.
     * - __u32
       - ``pic_order_cnt_bit_size``
-      -
+      - Combined size in bits of the picture order count related syntax
+        elements: pic_order_cnt_lsb, delta_pic_order_cnt_bottom,
+        delta_pic_order_cnt0, and delta_pic_order_cnt1.
     * - __u8
       - ``cabac_init_idc``
       -
-- 
2.27.0

[PATCH v2 05/14] media: uapi: h264: Increase size of 'first_mb_in_slice' field

[Ezequiel Garcia]

Slice header syntax element 'first_mb_in_slice' can point
to the last macroblock, currently the field can only reference
65536 macroblocks which is insufficient for 8K videos.

Although unlikely, a 8192x4320 video (where macroblocks are 16x16),
would contain 138240 macroblocks on a frame.

As per the H264 specification, 'first_mb_in_slice' can be up to
PicSizeInMbs - 1, so increase the size of the field to 32-bits.

Note that v4l2_ctrl_h264_slice_params struct will be modified
in a follow-up commit, and so we defer its 64-bit padding.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 2 +-
 include/media/h264-ctrls.h                                | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index e4dec3f9b71a..141c9249d528 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1774,7 +1774,7 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - __u32
       - ``header_bit_size``
       -
-    * - __u16
+    * - __u32
       - ``first_mb_in_slice``
       -
     * - __u8
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index 54cd9bec0b23..66327ca55088 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -167,7 +167,8 @@ struct v4l2_ctrl_h264_slice_params {
 	/* Offset in bits to slice_data() from the beginning of this slice. */
 	__u32 header_bit_size;
 
-	__u16 first_mb_in_slice;
+	__u32 first_mb_in_slice;
+
 	__u8 slice_type;
 	__u8 pic_parameter_set_id;
 	__u8 colour_plane_id;
-- 
2.27.0

[PATCH v2 06/14] media: uapi: h264: Clean DPB entry interface

[Ezequiel Garcia]

As discussed recently, the current interface for the
Decoded Picture Buffer is not enough to properly
support field coding.

This commit introduces enough semantics to support
frame and field coding, and to signal how DPB entries
are "used for reference".

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
v2:
* Fix rkvdec usage of fields flags as noted by Jonas.
---
 .../media/v4l/ext-ctrls-codec.rst             | 24 ++++++-------------
 drivers/media/v4l2-core/v4l2-h264.c           |  4 ++--
 drivers/staging/media/rkvdec/rkvdec-h264.c    | 17 ++++++-------
 include/media/h264-ctrls.h                    |  2 +-
 4 files changed, 19 insertions(+), 28 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index 141c9249d528..966b52ddcf4c 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -2063,6 +2063,9 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - __s32
       - ``bottom_field_order_cnt``
       -
+    * - enum :c:type:`v4l2_h264_dpb_reference`
+      - ``reference``
+      - Specifies how the DPB entry is referenced.
     * - __u32
       - ``flags``
       - See :ref:`DPB Entry Flags <h264_dpb_flags>`
@@ -2080,29 +2083,16 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
 
     * - ``V4L2_H264_DPB_ENTRY_FLAG_VALID``
       - 0x00000001
-      - The DPB entry is valid and should be considered
+      - The DPB entry is valid (non-empty) and should be considered.
     * - ``V4L2_H264_DPB_ENTRY_FLAG_ACTIVE``
       - 0x00000002
-      - The DPB entry is currently being used as a reference frame
+      - The DPB entry is used for reference.
     * - ``V4L2_H264_DPB_ENTRY_FLAG_LONG_TERM``
       - 0x00000004
-      - The DPB entry is a long term reference frame
+      - The DPB entry is used for long-term reference.
     * - ``V4L2_H264_DPB_ENTRY_FLAG_FIELD``
       - 0x00000008
-      - The DPB entry is a field reference, which means only one of the field
-        will be used when decoding the new frame/field. When not set the DPB
-        entry is a frame reference (both fields will be used). Note that this
-        flag does not say anything about the number of fields contained in the
-        reference frame, it just describes the one used to decode the new
-        field/frame
-    * - ``V4L2_H264_DPB_ENTRY_FLAG_BOTTOM_FIELD``
-      - 0x00000010
-      - The DPB entry is a bottom field reference (only the bottom field of the
-        reference frame is needed to decode the new frame/field). Only valid if
-        V4L2_H264_DPB_ENTRY_FLAG_FIELD is set. When
-        V4L2_H264_DPB_ENTRY_FLAG_FIELD is set but
-        V4L2_H264_DPB_ENTRY_FLAG_BOTTOM_FIELD is not, that means the
-        DPB entry is a top field reference
+      - The DPB entry is a single field or a complementary field pair.
 
 ``V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE (enum)``
     Specifies the decoding mode to use. Currently exposes slice-based and
diff --git a/drivers/media/v4l2-core/v4l2-h264.c b/drivers/media/v4l2-core/v4l2-h264.c
index edf6225f0522..1432810c1ce2 100644
--- a/drivers/media/v4l2-core/v4l2-h264.c
+++ b/drivers/media/v4l2-core/v4l2-h264.c
@@ -66,10 +66,10 @@ v4l2_h264_init_reflist_builder(struct v4l2_h264_reflist_builder *b,
 		else
 			b->refs[i].frame_num = dpb[i].frame_num;
 
-		if (!(dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_FIELD))
+		if (dpb[i].reference == V4L2_H264_DPB_FRAME_REF)
 			pic_order_count = min(dpb[i].top_field_order_cnt,
 					      dpb[i].bottom_field_order_cnt);
-		else if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_BOTTOM_FIELD)
+		else if (dpb[i].reference & V4L2_H264_DPB_BOTTOM_REF)
 			pic_order_count = dpb[i].bottom_field_order_cnt;
 		else
 			pic_order_count = dpb[i].top_field_order_cnt;
diff --git a/drivers/staging/media/rkvdec/rkvdec-h264.c b/drivers/staging/media/rkvdec/rkvdec-h264.c
index 7b66e2743a4f..7e555f18e357 100644
--- a/drivers/staging/media/rkvdec/rkvdec-h264.c
+++ b/drivers/staging/media/rkvdec/rkvdec-h264.c
@@ -949,16 +949,17 @@ static void config_registers(struct rkvdec_ctx *ctx,
 	for (i = 0; i < ARRAY_SIZE(dec_params->dpb); i++) {
 		struct vb2_buffer *vb_buf = get_ref_buf(ctx, run, i);
 
-		refer_addr = vb2_dma_contig_plane_dma_addr(vb_buf, 0) |
-			     RKVDEC_COLMV_USED_FLAG_REF;
+		refer_addr = vb2_dma_contig_plane_dma_addr(vb_buf, 0);
 
-		if (!(dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_FIELD))
-			refer_addr |= RKVDEC_TOPFIELD_USED_REF |
-				      RKVDEC_BOTFIELD_USED_REF;
-		else if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_BOTTOM_FIELD)
-			refer_addr |= RKVDEC_BOTFIELD_USED_REF;
-		else
+		if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE)
+			refer_addr |= RKVDEC_COLMV_USED_FLAG_REF;
+		if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_FIELD)
+			refer_addr |= RKVDEC_FIELD_REF;
+
+		if (dpb[i].reference & V4L2_H264_DPB_TOP_REF)
 			refer_addr |= RKVDEC_TOPFIELD_USED_REF;
+		if (dpb[i].reference & V4L2_H264_DPB_BOTTOM_REF)
+			refer_addr |= RKVDEC_BOTFIELD_USED_REF;
 
 		writel_relaxed(dpb[i].top_field_order_cnt,
 			       rkvdec->regs +  poc_reg_tbl_top_field[i]);
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index 66327ca55088..6a95d39f4d15 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -205,7 +205,6 @@ struct v4l2_ctrl_h264_slice_params {
 #define V4L2_H264_DPB_ENTRY_FLAG_ACTIVE		0x02
 #define V4L2_H264_DPB_ENTRY_FLAG_LONG_TERM	0x04
 #define V4L2_H264_DPB_ENTRY_FLAG_FIELD		0x08
-#define V4L2_H264_DPB_ENTRY_FLAG_BOTTOM_FIELD	0x10
 
 struct v4l2_h264_dpb_entry {
 	__u64 reference_ts;
@@ -214,6 +213,7 @@ struct v4l2_h264_dpb_entry {
 	/* Note that field is indicated by v4l2_buffer.field */
 	__s32 top_field_order_cnt;
 	__s32 bottom_field_order_cnt;
+	enum v4l2_h264_dpb_reference reference;
 	__u32 flags; /* V4L2_H264_DPB_ENTRY_FLAG_* */
 };
 
-- 
2.27.0

[PATCH v2 07/14] media: uapi: h264: Increase size of DPB entry pic_num

[Ezequiel Garcia]

DPB entry PicNum maximum value is 2*MaxFrameNum for interlaced
content (field_pic_flag=1).

As specified, MaxFrameNum is 2^(log2_max_frame_num_minus4 + 4)
and log2_max_frame_num_minus4 is in the range of 0 to 12,
which means pic_num should be a 32-bit field.

The v4l2_h264_dpb_entry struct needs to be padded to avoid a hole,
which might be also useful to allow future uAPI extensions.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 .../userspace-api/media/v4l/ext-ctrls-codec.rst     |  5 ++++-
 drivers/media/v4l2-core/v4l2-ctrls.c                | 13 +++++++++++++
 include/media/h264-ctrls.h                          |  3 ++-
 include/media/v4l2-h264.h                           |  2 +-
 4 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index 966b52ddcf4c..427fc5727ec0 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -2054,7 +2054,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - __u16
       - ``frame_num``
       -
-    * - __u16
+    * - __u8
+      - ``reserved[2]``
+      - Applications and drivers must set this to zero.
+    * - __u32
       - ``pic_num``
       -
     * - __s32
diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 76c8dc8fb31c..b9457789fa55 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1725,6 +1725,8 @@ static void std_log(const struct v4l2_ctrl *ctrl)
 
 #define zero_padding(s) \
 	memset(&(s).padding, 0, sizeof((s).padding))
+#define zero_reserved(s) \
+	memset(&(s).reserved, 0, sizeof((s).reserved))
 
 /*
  * Compound controls validation requires setting unused fields/flags to zero
@@ -1735,6 +1737,7 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
 {
 	struct v4l2_ctrl_mpeg2_slice_params *p_mpeg2_slice_params;
 	struct v4l2_ctrl_vp8_frame_header *p_vp8_frame_header;
+	struct v4l2_ctrl_h264_decode_params *p_h264_dec_params;
 	struct v4l2_ctrl_hevc_sps *p_hevc_sps;
 	struct v4l2_ctrl_hevc_pps *p_hevc_pps;
 	struct v4l2_ctrl_hevc_slice_params *p_hevc_slice_params;
@@ -1796,7 +1799,17 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
 	case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
 	case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
 	case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
+		break;
+
 	case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
+		p_h264_dec_params = p;
+
+		for (i = 0; i < V4L2_H264_NUM_DPB_ENTRIES; i++) {
+			struct v4l2_h264_dpb_entry *dpb_entry =
+				&p_h264_dec_params->dpb[i];
+
+			zero_reserved(*dpb_entry);
+		}
 		break;
 
 	case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index 6a95d39f4d15..4f05ee265997 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -209,7 +209,8 @@ struct v4l2_ctrl_h264_slice_params {
 struct v4l2_h264_dpb_entry {
 	__u64 reference_ts;
 	__u16 frame_num;
-	__u16 pic_num;
+	__u8 reserved[2];
+	__u32 pic_num;
 	/* Note that field is indicated by v4l2_buffer.field */
 	__s32 top_field_order_cnt;
 	__s32 bottom_field_order_cnt;
diff --git a/include/media/v4l2-h264.h b/include/media/v4l2-h264.h
index bc9ebb560ccf..1a5f26fc2a9a 100644
--- a/include/media/v4l2-h264.h
+++ b/include/media/v4l2-h264.h
@@ -33,7 +33,7 @@ struct v4l2_h264_reflist_builder {
 	struct {
 		s32 pic_order_count;
 		int frame_num;
-		u16 pic_num;
+		u32 pic_num;
 		u16 longterm : 1;
 	} refs[V4L2_H264_NUM_DPB_ENTRIES];
 	s32 cur_pic_order_count;
-- 
2.27.0

[PATCH v2 08/14] media: uapi: h264: Drop SLICE_PARAMS 'size' field

[Ezequiel Garcia]

The SLICE_PARAMS control is intended for slice-based
devices. In this mode, the OUTPUT buffer contains
a single slice, and so the buffer's plane payload size
can be used to query the slice size.

To reduce the API surface drop the size from the
SLICE_PARAMS control.

A follow-up change will remove other members in SLICE_PARAMS
so we don't need to add padding fields here.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 3 ---
 drivers/staging/media/sunxi/cedrus/cedrus_h264.c          | 7 +++----
 include/media/h264-ctrls.h                                | 3 ---
 3 files changed, 3 insertions(+), 10 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index 427fc5727ec0..fff74b7bf32a 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1760,9 +1760,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     :stub-columns: 0
     :widths:       1 1 2
 
-    * - __u32
-      - ``size``
-      -
     * - __u32
       - ``start_byte_offset``
         Offset (in bytes) from the beginning of the OUTPUT buffer to the start
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
index a9ba78b15907..8b6f05aadbe8 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
@@ -324,17 +324,16 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
 	struct vb2_buffer *src_buf = &run->src->vb2_buf;
 	struct cedrus_dev *dev = ctx->dev;
 	dma_addr_t src_buf_addr;
-	u32 len = slice->size * 8;
+	size_t slice_bytes = vb2_get_plane_payload(src_buf, 0);
 	unsigned int pic_width_in_mbs;
 	bool mbaff_pic;
 	u32 reg;
 
-	cedrus_write(dev, VE_H264_VLD_LEN, len);
+	cedrus_write(dev, VE_H264_VLD_LEN, slice_bytes * 8);
 	cedrus_write(dev, VE_H264_VLD_OFFSET, 0);
 
 	src_buf_addr = vb2_dma_contig_plane_dma_addr(src_buf, 0);
-	cedrus_write(dev, VE_H264_VLD_END,
-		     src_buf_addr + vb2_get_plane_payload(src_buf, 0));
+	cedrus_write(dev, VE_H264_VLD_END, src_buf_addr + slice_bytes);
 	cedrus_write(dev, VE_H264_VLD_ADDR,
 		     VE_H264_VLD_ADDR_VAL(src_buf_addr) |
 		     VE_H264_VLD_ADDR_FIRST | VE_H264_VLD_ADDR_VALID |
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index 4f05ee265997..f74736fcfa00 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -158,9 +158,6 @@ struct v4l2_h264_reference {
 };
 
 struct v4l2_ctrl_h264_slice_params {
-	/* Size in bytes, including header */
-	__u32 size;
-
 	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
 	__u32 start_byte_offset;
 
-- 
2.27.0

[PATCH v2 09/14] media: uapi: h264: Clarify SLICE_BASED mode

[Ezequiel Garcia]

Currently, the SLICE_BASED and FRAME_BASED modes documentation
is misleading and not matching the intended use-cases.

Drop non-required fields SLICE_PARAMS 'start_byte_offset' and
DECODE_PARAMS 'num_slices' and clarify the decoding modes in the
documentation.

On SLICE_BASED mode, a single slice is expected per OUTPUT buffer,
and therefore 'start_byte_offset' is not needed (since the offset
to the slice is the start of the buffer).

This mode requires the use of CAPTURE buffer holding, and so
the number of slices shall not be required.

On FRAME_BASED mode, the devices are expected to take care of slice
parsing. Neither SLICE_PARAMS are required (and shouldn't be
exposed by frame-based drivers), nor the number of slices.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 .../media/v4l/ext-ctrls-codec.rst             | 39 +++++--------------
 include/media/h264-ctrls.h                    |  4 --
 2 files changed, 10 insertions(+), 33 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index fff74b7bf32a..b3963c1acc2c 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1748,9 +1748,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
        This compound control is not yet part of the public kernel API
        and it is expected to change.
 
-       This structure is expected to be passed as an array, with one
-       entry for each slice included in the bitstream buffer.
-
 .. c:type:: v4l2_ctrl_h264_slice_params
 
 .. cssclass:: longtable
@@ -1760,17 +1757,9 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     :stub-columns: 0
     :widths:       1 1 2
 
-    * - __u32
-      - ``start_byte_offset``
-        Offset (in bytes) from the beginning of the OUTPUT buffer to the start
-        of the slice. If the slice starts with a start code, then this is the
-        offset to such start code. When operating in slice-based decoding mode
-        (see :c:type:`v4l2_mpeg_video_h264_decode_mode`), this field should
-        be set to 0. When operating in frame-based decoding mode, this field
-        should be 0 for the first slice.
     * - __u32
       - ``header_bit_size``
-      -
+      - Offset in bits to slice_data() from the beginning of this slice.
     * - __u32
       - ``first_mb_in_slice``
       -
@@ -1998,12 +1987,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - struct :c:type:`v4l2_h264_dpb_entry`
       - ``dpb[16]``
       -
-    * - __u16
-      - ``num_slices``
-      - Number of slices needed to decode the current frame/field. When
-        operating in slice-based decoding mode (see
-        :c:type:`v4l2_mpeg_video_h264_decode_mode`), this field
-        should always be set to one.
     * - __u16
       - ``nal_ref_idc``
       - NAL reference ID value coming from the NAL Unit header
@@ -2121,22 +2104,20 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - ``V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED``
       - 0
       - Decoding is done at the slice granularity.
-        In this mode, ``num_slices`` field in struct
-        :c:type:`v4l2_ctrl_h264_decode_params` should be set to 1,
-        and ``start_byte_offset`` in struct
-        :c:type:`v4l2_ctrl_h264_slice_params` should be set to 0.
         The OUTPUT buffer must contain a single slice.
+        When this mode is selected, the ``V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS``
+        control shall be set. When multiple slices compose a frame,
+        use of ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` flag
+        is required.
     * - ``V4L2_MPEG_VIDEO_H264_DECODE_MODE_FRAME_BASED``
       - 1
-      - Decoding is done at the frame granularity.
-        In this mode, ``num_slices`` field in struct
-        :c:type:`v4l2_ctrl_h264_decode_params` should be set to the number
-        of slices in the frame, and ``start_byte_offset`` in struct
-        :c:type:`v4l2_ctrl_h264_slice_params` should be set accordingly
-        for each slice. For the first slice, ``start_byte_offset`` should
-        be zero.
+      - Decoding is done at the frame granularity,
         The OUTPUT buffer must contain all slices needed to decode the
         frame. The OUTPUT buffer must also contain both fields.
+        This mode will be supported by devices that
+        parse the slice(s) header(s) in hardware. When this mode is
+        selected, the ``V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS``
+        control shall not be set.
 
 ``V4L2_CID_MPEG_VIDEO_H264_START_CODE (enum)``
     Specifies the H264 slice start code expected for each slice.
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index f74736fcfa00..ea8c8c93305b 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -158,9 +158,6 @@ struct v4l2_h264_reference {
 };
 
 struct v4l2_ctrl_h264_slice_params {
-	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
-	__u32 start_byte_offset;
-
 	/* Offset in bits to slice_data() from the beginning of this slice. */
 	__u32 header_bit_size;
 
@@ -219,7 +216,6 @@ struct v4l2_h264_dpb_entry {
 
 struct v4l2_ctrl_h264_decode_params {
 	struct v4l2_h264_dpb_entry dpb[V4L2_H264_NUM_DPB_ENTRIES];
-	__u16 num_slices;
 	__u16 nal_ref_idc;
 	__s32 top_field_order_cnt;
 	__s32 bottom_field_order_cnt;
-- 
2.27.0

[PATCH v2 10/14] media: uapi: h264: Clean slice invariants syntax elements

[Ezequiel Garcia]

The H.264 specification requires in section 7.4.3 "Slice header semantics",
that the following values shall be the same in all slice headers:

  pic_parameter_set_id
  frame_num
  field_pic_flag
  bottom_field_flag
  idr_pic_id
  pic_order_cnt_lsb
  delta_pic_order_cnt_bottom
  delta_pic_order_cnt[ 0 ]
  delta_pic_order_cnt[ 1 ]
  sp_for_switch_flag
  slice_group_change_cycle

These bitstream fields are part of the slice header, and therefore
passed redundantly on each slice. The purpose of the redundancy
is to make the codec fault-tolerant in network scenarios.

This is of course not needed to be reflected in the V4L2 controls,
given the bitstream has already been parsed by applications.
Therefore, move the redundant fields to the per-frame decode
parameters control (DECODE_PARAMS).

Field 'pic_parameter_set_id' is simply removed in this case,
because the PPS control must currently contain the active PPS.

Syntax elements dec_ref_pic_marking() and those related
to pic order count, remain invariant as well, and therefore,
the fields dec_ref_pic_marking_bit_size and pic_order_cnt_bit_size
are also common to all slices.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
---
 .../media/v4l/ext-ctrls-codec.rst             | 86 +++++++++----------
 drivers/media/v4l2-core/v4l2-ctrls.c          |  7 ++
 drivers/media/v4l2-core/v4l2-h264.c           |  8 +-
 .../staging/media/hantro/hantro_g1_h264_dec.c | 21 +++--
 drivers/staging/media/hantro/hantro_h264.c    |  3 +-
 drivers/staging/media/rkvdec/rkvdec-h264.c    |  6 +-
 .../staging/media/sunxi/cedrus/cedrus_h264.c  |  9 +-
 include/media/h264-ctrls.h                    | 39 +++++----
 include/media/v4l2-h264.h                     |  1 -
 9 files changed, 90 insertions(+), 90 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
index b3963c1acc2c..8761e6a4e130 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
@@ -1766,44 +1766,12 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - __u8
       - ``slice_type``
       -
-    * - __u8
-      - ``pic_parameter_set_id``
-      -
     * - __u8
       - ``colour_plane_id``
       -
     * - __u8
       - ``redundant_pic_cnt``
       -
-    * - __u16
-      - ``frame_num``
-      -
-    * - __u16
-      - ``idr_pic_id``
-      -
-    * - __u16
-      - ``pic_order_cnt_lsb``
-      -
-    * - __s32
-      - ``delta_pic_order_cnt_bottom``
-      -
-    * - __s32
-      - ``delta_pic_order_cnt0``
-      -
-    * - __s32
-      - ``delta_pic_order_cnt1``
-      -
-    * - struct :c:type:`v4l2_h264_pred_weight_table`
-      - ``pred_weight_table``
-      -
-    * - __u32
-      - ``dec_ref_pic_marking_bit_size``
-      - Size in bits of the dec_ref_pic_marking() syntax element.
-    * - __u32
-      - ``pic_order_cnt_bit_size``
-      - Combined size in bits of the picture order count related syntax
-        elements: pic_order_cnt_lsb, delta_pic_order_cnt_bottom,
-        delta_pic_order_cnt0, and delta_pic_order_cnt1.
     * - __u8
       - ``cabac_init_idc``
       -
@@ -1830,9 +1798,9 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
       - ``num_ref_idx_l1_active_minus1``
       - If num_ref_idx_active_override_flag is not set, this field must be
         set to the value of num_ref_idx_l1_default_active_minus1.
-    * - __u32
-      - ``slice_group_change_cycle``
-      -
+    * - __u8
+      - ``reserved``
+      - Applications and drivers must set this to zero.
     * - struct :c:type:`v4l2_h264_reference`
       - ``ref_pic_list0[32]``
       - Reference picture list after applying the per-slice modifications
@@ -1854,17 +1822,11 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     :stub-columns: 0
     :widths:       1 1 2
 
-    * - ``V4L2_H264_SLICE_FLAG_FIELD_PIC``
-      - 0x00000001
-      -
-    * - ``V4L2_H264_SLICE_FLAG_BOTTOM_FIELD``
-      - 0x00000002
-      -
     * - ``V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED``
-      - 0x00000004
+      - 0x00000001
       -
     * - ``V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH``
-      - 0x00000008
+      - 0x00000002
       -
 
 ``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
@@ -1990,12 +1952,44 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - __u16
       - ``nal_ref_idc``
       - NAL reference ID value coming from the NAL Unit header
+    * - __u16
+      - ``frame_num``
+      -
     * - __s32
       - ``top_field_order_cnt``
       - Picture Order Count for the coded top field
     * - __s32
       - ``bottom_field_order_cnt``
       - Picture Order Count for the coded bottom field
+    * - __u16
+      - ``idr_pic_id``
+      -
+    * - __u16
+      - ``pic_order_cnt_lsb``
+      -
+    * - __s32
+      - ``delta_pic_order_cnt_bottom``
+      -
+    * - __s32
+      - ``delta_pic_order_cnt0``
+      -
+    * - __s32
+      - ``delta_pic_order_cnt1``
+      -
+    * - __u32
+      - ``dec_ref_pic_marking_bit_size``
+      - Size in bits of the dec_ref_pic_marking() syntax element.
+    * - __u32
+      - ``pic_order_cnt_bit_size``
+      - Combined size in bits of the picture order count related syntax
+        elements: pic_order_cnt_lsb, delta_pic_order_cnt_bottom,
+        delta_pic_order_cnt0, and delta_pic_order_cnt1.
+    * - __u32
+      - ``slice_group_change_cycle``
+      -
+    * - __u32
+      - ``reserved``
+      - Applications and drivers must set this to zero.
     * - __u32
       - ``flags``
       - See :ref:`Decode Parameters Flags <h264_decode_params_flags>`
@@ -2014,6 +2008,12 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
     * - ``V4L2_H264_DECODE_PARAM_FLAG_IDR_PIC``
       - 0x00000001
       - That picture is an IDR picture
+    * - ``V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC``
+      - 0x00000002
+      -
+    * - ``V4L2_H264_DECODE_PARAM_FLAG_BOTTOM_FIELD``
+      - 0x00000004
+      -
 
 .. c:type:: v4l2_h264_dpb_entry
 
diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index b9457789fa55..b846f5b089c9 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1737,6 +1737,7 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
 {
 	struct v4l2_ctrl_mpeg2_slice_params *p_mpeg2_slice_params;
 	struct v4l2_ctrl_vp8_frame_header *p_vp8_frame_header;
+	struct v4l2_ctrl_h264_slice_params *p_h264_slice_params;
 	struct v4l2_ctrl_h264_decode_params *p_h264_dec_params;
 	struct v4l2_ctrl_hevc_sps *p_hevc_sps;
 	struct v4l2_ctrl_hevc_pps *p_hevc_pps;
@@ -1798,7 +1799,12 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
 	case V4L2_CTRL_TYPE_H264_PPS:
 	case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
 	case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
+		break;
+
 	case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
+		p_h264_slice_params = p;
+
+		zero_reserved(*p_h264_slice_params);
 		break;
 
 	case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
@@ -1810,6 +1816,7 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
 
 			zero_reserved(*dpb_entry);
 		}
+		zero_reserved(*p_h264_dec_params);
 		break;
 
 	case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
diff --git a/drivers/media/v4l2-core/v4l2-h264.c b/drivers/media/v4l2-core/v4l2-h264.c
index 1432810c1ce2..90d08de221ed 100644
--- a/drivers/media/v4l2-core/v4l2-h264.c
+++ b/drivers/media/v4l2-core/v4l2-h264.c
@@ -18,14 +18,12 @@
  *
  * @b: the builder context to initialize
  * @dec_params: decode parameters control
- * @slice_params: first slice parameters control
  * @sps: SPS control
  * @dpb: DPB to use when creating the reference list
  */
 void
 v4l2_h264_init_reflist_builder(struct v4l2_h264_reflist_builder *b,
 		const struct v4l2_ctrl_h264_decode_params *dec_params,
-		const struct v4l2_ctrl_h264_slice_params *slice_params,
 		const struct v4l2_ctrl_h264_sps *sps,
 		const struct v4l2_h264_dpb_entry dpb[V4L2_H264_NUM_DPB_ENTRIES])
 {
@@ -33,13 +31,13 @@ v4l2_h264_init_reflist_builder(struct v4l2_h264_reflist_builder *b,
 	unsigned int i;
 
 	max_frame_num = 1 << (sps->log2_max_frame_num_minus4 + 4);
-	cur_frame_num = slice_params->frame_num;
+	cur_frame_num = dec_params->frame_num;
 
 	memset(b, 0, sizeof(*b));
-	if (!(slice_params->flags & V4L2_H264_SLICE_FLAG_FIELD_PIC))
+	if (!(dec_params->flags & V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC))
 		b->cur_pic_order_count = min(dec_params->bottom_field_order_cnt,
 					     dec_params->top_field_order_cnt);
-	else if (slice_params->flags & V4L2_H264_SLICE_FLAG_BOTTOM_FIELD)
+	else if (dec_params->flags & V4L2_H264_DECODE_PARAM_FLAG_BOTTOM_FIELD)
 		b->cur_pic_order_count = dec_params->bottom_field_order_cnt;
 	else
 		b->cur_pic_order_count = dec_params->top_field_order_cnt;
diff --git a/drivers/staging/media/hantro/hantro_g1_h264_dec.c b/drivers/staging/media/hantro/hantro_g1_h264_dec.c
index 424c648ce9fc..f9839e9c6da5 100644
--- a/drivers/staging/media/hantro/hantro_g1_h264_dec.c
+++ b/drivers/staging/media/hantro/hantro_g1_h264_dec.c
@@ -23,7 +23,6 @@ static void set_params(struct hantro_ctx *ctx)
 {
 	const struct hantro_h264_dec_ctrls *ctrls = &ctx->h264_dec.ctrls;
 	const struct v4l2_ctrl_h264_decode_params *dec_param = ctrls->decode;
-	const struct v4l2_ctrl_h264_slice_params *slices = ctrls->slices;
 	const struct v4l2_ctrl_h264_sps *sps = ctrls->sps;
 	const struct v4l2_ctrl_h264_pps *pps = ctrls->pps;
 	struct vb2_v4l2_buffer *src_buf = hantro_get_src_buf(ctx);
@@ -42,11 +41,11 @@ static void set_params(struct hantro_ctx *ctx)
 
 	if (!(sps->flags & V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY) &&
 	    (sps->flags & V4L2_H264_SPS_FLAG_MB_ADAPTIVE_FRAME_FIELD ||
-	     slices[0].flags & V4L2_H264_SLICE_FLAG_FIELD_PIC))
+	     dec_param->flags & V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC))
 		reg |= G1_REG_DEC_CTRL0_PIC_INTERLACE_E;
-	if (slices[0].flags & V4L2_H264_SLICE_FLAG_FIELD_PIC)
+	if (dec_param->flags & V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC)
 		reg |= G1_REG_DEC_CTRL0_PIC_FIELDMODE_E;
-	if (!(slices[0].flags & V4L2_H264_SLICE_FLAG_BOTTOM_FIELD))
+	if (!(dec_param->flags & V4L2_H264_DECODE_PARAM_FLAG_BOTTOM_FIELD))
 		reg |= G1_REG_DEC_CTRL0_PIC_TOPFIELD_E;
 	vdpu_write_relaxed(vpu, reg, G1_REG_DEC_CTRL0);
 
@@ -75,7 +74,7 @@ static void set_params(struct hantro_ctx *ctx)
 
 	/* Decoder control register 4. */
 	reg = G1_REG_DEC_CTRL4_FRAMENUM_LEN(sps->log2_max_frame_num_minus4 + 4) |
-	      G1_REG_DEC_CTRL4_FRAMENUM(slices[0].frame_num) |
+	      G1_REG_DEC_CTRL4_FRAMENUM(dec_param->frame_num) |
 	      G1_REG_DEC_CTRL4_WEIGHT_BIPR_IDC(pps->weighted_bipred_idc);
 	if (pps->flags & V4L2_H264_PPS_FLAG_ENTROPY_CODING_MODE)
 		reg |= G1_REG_DEC_CTRL4_CABAC_E;
@@ -88,8 +87,8 @@ static void set_params(struct hantro_ctx *ctx)
 	vdpu_write_relaxed(vpu, reg, G1_REG_DEC_CTRL4);
 
 	/* Decoder control register 5. */
-	reg = G1_REG_DEC_CTRL5_REFPIC_MK_LEN(slices[0].dec_ref_pic_marking_bit_size) |
-	      G1_REG_DEC_CTRL5_IDR_PIC_ID(slices[0].idr_pic_id);
+	reg = G1_REG_DEC_CTRL5_REFPIC_MK_LEN(dec_param->dec_ref_pic_marking_bit_size) |
+	      G1_REG_DEC_CTRL5_IDR_PIC_ID(dec_param->idr_pic_id);
 	if (pps->flags & V4L2_H264_PPS_FLAG_CONSTRAINED_INTRA_PRED)
 		reg |= G1_REG_DEC_CTRL5_CONST_INTRA_E;
 	if (pps->flags & V4L2_H264_PPS_FLAG_DEBLOCKING_FILTER_CONTROL_PRESENT)
@@ -103,10 +102,10 @@ static void set_params(struct hantro_ctx *ctx)
 	vdpu_write_relaxed(vpu, reg, G1_REG_DEC_CTRL5);
 
 	/* Decoder control register 6. */
-	reg = G1_REG_DEC_CTRL6_PPS_ID(slices[0].pic_parameter_set_id) |
+	reg = G1_REG_DEC_CTRL6_PPS_ID(pps->pic_parameter_set_id) |
 	      G1_REG_DEC_CTRL6_REFIDX0_ACTIVE(pps->num_ref_idx_l0_default_active_minus1 + 1) |
 	      G1_REG_DEC_CTRL6_REFIDX1_ACTIVE(pps->num_ref_idx_l1_default_active_minus1 + 1) |
-	      G1_REG_DEC_CTRL6_POC_LENGTH(slices[0].pic_order_cnt_bit_size);
+	      G1_REG_DEC_CTRL6_POC_LENGTH(dec_param->pic_order_cnt_bit_size);
 	vdpu_write_relaxed(vpu, reg, G1_REG_DEC_CTRL6);
 
 	/* Error concealment register. */
@@ -246,7 +245,7 @@ static void set_buffers(struct hantro_ctx *ctx)
 	/* Destination (decoded frame) buffer. */
 	dst_dma = hantro_get_dec_buf_addr(ctx, &dst_buf->vb2_buf);
 	/* Adjust dma addr to start at second line for bottom field */
-	if (ctrls->slices[0].flags & V4L2_H264_SLICE_FLAG_BOTTOM_FIELD)
+	if (ctrls->decode->flags & V4L2_H264_DECODE_PARAM_FLAG_BOTTOM_FIELD)
 		offset = ALIGN(ctx->src_fmt.width, MB_DIM);
 	vdpu_write_relaxed(vpu, dst_dma + offset, G1_REG_ADDR_DST);
 
@@ -265,7 +264,7 @@ static void set_buffers(struct hantro_ctx *ctx)
 		 * DMV buffer is split in two for field encoded frames,
 		 * adjust offset for bottom field
 		 */
-		if (ctrls->slices[0].flags & V4L2_H264_SLICE_FLAG_BOTTOM_FIELD)
+		if (ctrls->decode->flags & V4L2_H264_DECODE_PARAM_FLAG_BOTTOM_FIELD)
 			offset += 32 * MB_WIDTH(ctx->src_fmt.width) *
 				  MB_HEIGHT(ctx->src_fmt.height);
 		vdpu_write_relaxed(vpu, dst_dma + offset, G1_REG_ADDR_DIR_MV);
diff --git a/drivers/staging/media/hantro/hantro_h264.c b/drivers/staging/media/hantro/hantro_h264.c
index 194d05848077..0cbe514dc79a 100644
--- a/drivers/staging/media/hantro/hantro_h264.c
+++ b/drivers/staging/media/hantro/hantro_h264.c
@@ -372,8 +372,7 @@ int hantro_h264_dec_prepare_run(struct hantro_ctx *ctx)
 
 	/* Build the P/B{0,1} ref lists. */
 	v4l2_h264_init_reflist_builder(&reflist_builder, ctrls->decode,
-				       &ctrls->slices[0], ctrls->sps,
-				       ctx->h264_dec.dpb);
+				       ctrls->sps, ctx->h264_dec.dpb);
 	v4l2_h264_build_p_ref_list(&reflist_builder, h264_ctx->reflists.p);
 	v4l2_h264_build_b_ref_lists(&reflist_builder, h264_ctx->reflists.b0,
 				    h264_ctx->reflists.b1);
diff --git a/drivers/staging/media/rkvdec/rkvdec-h264.c b/drivers/staging/media/rkvdec/rkvdec-h264.c
index 7e555f18e357..33e1d10f6253 100644
--- a/drivers/staging/media/rkvdec/rkvdec-h264.c
+++ b/drivers/staging/media/rkvdec/rkvdec-h264.c
@@ -730,7 +730,6 @@ static void assemble_hw_rps(struct rkvdec_ctx *ctx,
 			    struct rkvdec_h264_run *run)
 {
 	const struct v4l2_ctrl_h264_decode_params *dec_params = run->decode_params;
-	const struct v4l2_ctrl_h264_slice_params *sl_params = &run->slices_params[0];
 	const struct v4l2_h264_dpb_entry *dpb = dec_params->dpb;
 	struct rkvdec_h264_ctx *h264_ctx = ctx->priv;
 	const struct v4l2_ctrl_h264_sps *sps = run->sps;
@@ -754,7 +753,7 @@ static void assemble_hw_rps(struct rkvdec_ctx *ctx,
 			continue;
 
 		if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_LONG_TERM ||
-		    dpb[i].frame_num < sl_params->frame_num) {
+		    dpb[i].frame_num < dec_params->frame_num) {
 			p[i] = dpb[i].frame_num;
 			continue;
 		}
@@ -1094,8 +1093,7 @@ static int rkvdec_h264_run(struct rkvdec_ctx *ctx)
 
 	/* Build the P/B{0,1} ref lists. */
 	v4l2_h264_init_reflist_builder(&reflist_builder, run.decode_params,
-				       &run.slices_params[0], run.sps,
-				       run.decode_params->dpb);
+				       run.sps, run.decode_params->dpb);
 	h264_ctx->reflists.num_valid = reflist_builder.num_valid;
 	v4l2_h264_build_p_ref_list(&reflist_builder, h264_ctx->reflists.p);
 	v4l2_h264_build_b_ref_lists(&reflist_builder, h264_ctx->reflists.b0,
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
index 8b6f05aadbe8..8e31f81358b7 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
@@ -95,7 +95,6 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
 {
 	struct cedrus_h264_sram_ref_pic pic_list[CEDRUS_H264_FRAME_NUM];
 	const struct v4l2_ctrl_h264_decode_params *decode = run->h264.decode_params;
-	const struct v4l2_ctrl_h264_slice_params *slice = run->h264.slice_params;
 	const struct v4l2_ctrl_h264_sps *sps = run->h264.sps;
 	struct vb2_queue *cap_q;
 	struct cedrus_buffer *output_buf;
@@ -144,7 +143,7 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
 	output_buf = vb2_to_cedrus_buffer(&run->dst->vb2_buf);
 	output_buf->codec.h264.position = position;
 
-	if (slice->flags & V4L2_H264_SLICE_FLAG_FIELD_PIC)
+	if (decode->flags & V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC)
 		output_buf->codec.h264.pic_type = CEDRUS_H264_PIC_TYPE_FIELD;
 	else if (sps->flags & V4L2_H264_SPS_FLAG_MB_ADAPTIVE_FRAME_FIELD)
 		output_buf->codec.h264.pic_type = CEDRUS_H264_PIC_TYPE_MBAFF;
@@ -411,7 +410,7 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
 		reg |= VE_H264_SPS_DIRECT_8X8_INFERENCE;
 	cedrus_write(dev, VE_H264_SPS, reg);
 
-	mbaff_pic = !(slice->flags & V4L2_H264_SLICE_FLAG_FIELD_PIC) &&
+	mbaff_pic = !(decode->flags & V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC) &&
 		    (sps->flags & V4L2_H264_SPS_FLAG_MB_ADAPTIVE_FRAME_FIELD);
 	pic_width_in_mbs = sps->pic_width_in_mbs_minus1 + 1;
 
@@ -425,9 +424,9 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
 	reg |= slice->cabac_init_idc & 0x3;
 	if (ctx->fh.m2m_ctx->new_frame)
 		reg |= VE_H264_SHS_FIRST_SLICE_IN_PIC;
-	if (slice->flags & V4L2_H264_SLICE_FLAG_FIELD_PIC)
+	if (decode->flags & V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC)
 		reg |= VE_H264_SHS_FIELD_PIC;
-	if (slice->flags & V4L2_H264_SLICE_FLAG_BOTTOM_FIELD)
+	if (decode->flags & V4L2_H264_DECODE_PARAM_FLAG_BOTTOM_FIELD)
 		reg |= VE_H264_SHS_BOTTOM_FIELD;
 	if (slice->flags & V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED)
 		reg |= VE_H264_SHS_DIRECT_SPATIAL_MV_PRED;
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index ea8c8c93305b..fa5663876e73 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -139,10 +139,8 @@ struct v4l2_ctrl_h264_pred_weights {
 #define V4L2_H264_SLICE_TYPE_SP				3
 #define V4L2_H264_SLICE_TYPE_SI				4
 
-#define V4L2_H264_SLICE_FLAG_FIELD_PIC			0x01
-#define V4L2_H264_SLICE_FLAG_BOTTOM_FIELD		0x02
-#define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED	0x04
-#define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH		0x08
+#define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED	0x01
+#define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH		0x02
 
 enum v4l2_h264_dpb_reference {
 	V4L2_H264_DPB_TOP_REF = 0x1,
@@ -164,21 +162,8 @@ struct v4l2_ctrl_h264_slice_params {
 	__u32 first_mb_in_slice;
 
 	__u8 slice_type;
-	__u8 pic_parameter_set_id;
 	__u8 colour_plane_id;
 	__u8 redundant_pic_cnt;
-	__u16 frame_num;
-	__u16 idr_pic_id;
-	__u16 pic_order_cnt_lsb;
-	__s32 delta_pic_order_cnt_bottom;
-	__s32 delta_pic_order_cnt0;
-	__s32 delta_pic_order_cnt1;
-
-	/* Size in bits of dec_ref_pic_marking() syntax element. */
-	__u32 dec_ref_pic_marking_bit_size;
-	/* Size in bits of pic order count syntax. */
-	__u32 pic_order_cnt_bit_size;
-
 	__u8 cabac_init_idc;
 	__s8 slice_qp_delta;
 	__s8 slice_qs_delta;
@@ -187,7 +172,8 @@ struct v4l2_ctrl_h264_slice_params {
 	__s8 slice_beta_offset_div2;
 	__u8 num_ref_idx_l0_active_minus1;
 	__u8 num_ref_idx_l1_active_minus1;
-	__u32 slice_group_change_cycle;
+
+	__u8 reserved;
 
 	struct v4l2_h264_reference ref_pic_list0[V4L2_H264_REF_LIST_LEN];
 	struct v4l2_h264_reference ref_pic_list1[V4L2_H264_REF_LIST_LEN];
@@ -212,13 +198,28 @@ struct v4l2_h264_dpb_entry {
 	__u32 flags; /* V4L2_H264_DPB_ENTRY_FLAG_* */
 };
 
-#define V4L2_H264_DECODE_PARAM_FLAG_IDR_PIC	0x01
+#define V4L2_H264_DECODE_PARAM_FLAG_IDR_PIC		0x01
+#define V4L2_H264_DECODE_PARAM_FLAG_FIELD_PIC		0x02
+#define V4L2_H264_DECODE_PARAM_FLAG_BOTTOM_FIELD	0x04
 
 struct v4l2_ctrl_h264_decode_params {
 	struct v4l2_h264_dpb_entry dpb[V4L2_H264_NUM_DPB_ENTRIES];
 	__u16 nal_ref_idc;
+	__u16 frame_num;
 	__s32 top_field_order_cnt;
 	__s32 bottom_field_order_cnt;
+	__u16 idr_pic_id;
+	__u16 pic_order_cnt_lsb;
+	__s32 delta_pic_order_cnt_bottom;
+	__s32 delta_pic_order_cnt0;
+	__s32 delta_pic_order_cnt1;
+	/* Size in bits of dec_ref_pic_marking() syntax element. */
+	__u32 dec_ref_pic_marking_bit_size;
+	/* Size in bits of pic order count syntax. */
+	__u32 pic_order_cnt_bit_size;
+	__u32 slice_group_change_cycle;
+
+	__u32 reserved;
 	__u32 flags; /* V4L2_H264_DECODE_PARAM_FLAG_* */
 };
 
diff --git a/include/media/v4l2-h264.h b/include/media/v4l2-h264.h
index 1a5f26fc2a9a..f08ba181263d 100644
--- a/include/media/v4l2-h264.h
+++ b/include/media/v4l2-h264.h
@@ -44,7 +44,6 @@ struct v4l2_h264_reflist_builder {
 void
 v4l2_h264_init_reflist_builder(struct v4l2_h264_reflist_builder *b,
 		const struct v4l2_ctrl_h264_decode_params *dec_params,
-		const struct v4l2_ctrl_h264_slice_params *slice_params,
 		const struct v4l2_ctrl_h264_sps *sps,
 		const struct v4l2_h264_dpb_entry dpb[V4L2_H264_NUM_DPB_ENTRIES]);
 
-- 
2.27.0

[PATCH v2 11/14] media: hantro: Don't require unneeded H264_SLICE_PARAMS

[Ezequiel Garcia]

Now that slice invariant parameters have been moved,
the driver no longer needs this control, so drop it.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/staging/media/hantro/hantro_drv.c  | 5 -----
 drivers/staging/media/hantro/hantro_h264.c | 5 -----
 drivers/staging/media/hantro/hantro_hw.h   | 2 --
 3 files changed, 12 deletions(-)

diff --git a/drivers/staging/media/hantro/hantro_drv.c b/drivers/staging/media/hantro/hantro_drv.c
index 34797507f214..3cd00cc0a364 100644
--- a/drivers/staging/media/hantro/hantro_drv.c
+++ b/drivers/staging/media/hantro/hantro_drv.c
@@ -306,11 +306,6 @@ static const struct hantro_ctrl controls[] = {
 		.cfg = {
 			.id = V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS,
 		},
-	}, {
-		.codec = HANTRO_H264_DECODER,
-		.cfg = {
-			.id = V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS,
-		},
 	}, {
 		.codec = HANTRO_H264_DECODER,
 		.cfg = {
diff --git a/drivers/staging/media/hantro/hantro_h264.c b/drivers/staging/media/hantro/hantro_h264.c
index 0cbe514dc79a..6887318ed4d8 100644
--- a/drivers/staging/media/hantro/hantro_h264.c
+++ b/drivers/staging/media/hantro/hantro_h264.c
@@ -349,11 +349,6 @@ int hantro_h264_dec_prepare_run(struct hantro_ctx *ctx)
 	if (WARN_ON(!ctrls->decode))
 		return -EINVAL;
 
-	ctrls->slices =
-		hantro_get_ctrl(ctx, V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS);
-	if (WARN_ON(!ctrls->slices))
-		return -EINVAL;
-
 	ctrls->sps =
 		hantro_get_ctrl(ctx, V4L2_CID_MPEG_VIDEO_H264_SPS);
 	if (WARN_ON(!ctrls->sps))
diff --git a/drivers/staging/media/hantro/hantro_hw.h b/drivers/staging/media/hantro/hantro_hw.h
index f066de6b592d..219283a06f52 100644
--- a/drivers/staging/media/hantro/hantro_hw.h
+++ b/drivers/staging/media/hantro/hantro_hw.h
@@ -56,14 +56,12 @@ struct hantro_jpeg_enc_hw_ctx {
  * struct hantro_h264_dec_ctrls
  * @decode:	Decode params
  * @scaling:	Scaling info
- * @slice:	Slice params
  * @sps:	SPS info
  * @pps:	PPS info
  */
 struct hantro_h264_dec_ctrls {
 	const struct v4l2_ctrl_h264_decode_params *decode;
 	const struct v4l2_ctrl_h264_scaling_matrix *scaling;
-	const struct v4l2_ctrl_h264_slice_params *slices;
 	const struct v4l2_ctrl_h264_sps *sps;
 	const struct v4l2_ctrl_h264_pps *pps;
 };
-- 
2.27.0

[PATCH v2 12/14] media: rkvdec: Don't require unneeded H264_SLICE_PARAMS

[Ezequiel Garcia]

Now that slice invariant parameters have been moved,
the driver no longer needs this control, so drop it.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
Reviewed-by: Jonas Karlman <jonas@kwiboo.se>
---
v2:
* Fix wrongly removed SPS.
---
 drivers/staging/media/rkvdec/rkvdec-h264.c | 4 ----
 drivers/staging/media/rkvdec/rkvdec.c      | 5 -----
 2 files changed, 9 deletions(-)

diff --git a/drivers/staging/media/rkvdec/rkvdec-h264.c b/drivers/staging/media/rkvdec/rkvdec-h264.c
index 33e1d10f6253..fdb4fc2157d3 100644
--- a/drivers/staging/media/rkvdec/rkvdec-h264.c
+++ b/drivers/staging/media/rkvdec/rkvdec-h264.c
@@ -109,7 +109,6 @@ struct rkvdec_h264_reflists {
 struct rkvdec_h264_run {
 	struct rkvdec_run base;
 	const struct v4l2_ctrl_h264_decode_params *decode_params;
-	const struct v4l2_ctrl_h264_slice_params *slices_params;
 	const struct v4l2_ctrl_h264_sps *sps;
 	const struct v4l2_ctrl_h264_pps *pps;
 	const struct v4l2_ctrl_h264_scaling_matrix *scaling_matrix;
@@ -1066,9 +1065,6 @@ static void rkvdec_h264_run_preamble(struct rkvdec_ctx *ctx,
 	ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 			      V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS);
 	run->decode_params = ctrl ? ctrl->p_cur.p : NULL;
-	ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
-			      V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS);
-	run->slices_params = ctrl ? ctrl->p_cur.p : NULL;
 	ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl,
 			      V4L2_CID_MPEG_VIDEO_H264_SPS);
 	run->sps = ctrl ? ctrl->p_cur.p : NULL;
diff --git a/drivers/staging/media/rkvdec/rkvdec.c b/drivers/staging/media/rkvdec/rkvdec.c
index c8151328fb70..7c5129593921 100644
--- a/drivers/staging/media/rkvdec/rkvdec.c
+++ b/drivers/staging/media/rkvdec/rkvdec.c
@@ -59,11 +59,6 @@ static const struct rkvdec_ctrl_desc rkvdec_h264_ctrl_descs[] = {
 		.mandatory = true,
 		.cfg.id = V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS,
 	},
-	{
-		.per_request = true,
-		.mandatory = true,
-		.cfg.id = V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS,
-	},
 	{
 		.per_request = true,
 		.mandatory = true,
-- 
2.27.0

[PATCH v2 13/14] media: cedrus: h264: Properly configure reference field

[Ezequiel Garcia]

From: Jernej Skrabec <jernej.skrabec@siol.net>

When interlaced H264 content is being decoded, references must indicate
which field is being referenced. Currently this was done by checking
capture buffer flags. However, that is not correct because capture
buffer may hold both fields.

Fix this by checking newly introduced flags in reference lists.

Signed-off-by: Jernej Skrabec <jernej.skrabec@siol.net>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
---
 drivers/staging/media/sunxi/cedrus/cedrus_h264.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
index 8e31f81358b7..91d88a96badc 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
@@ -182,7 +182,6 @@ static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
 	for (i = 0; i < num_ref; i++) {
 		const struct v4l2_h264_dpb_entry *dpb;
 		const struct cedrus_buffer *cedrus_buf;
-		const struct vb2_v4l2_buffer *ref_buf;
 		unsigned int position;
 		int buf_idx;
 		u8 dpb_idx;
@@ -197,12 +196,11 @@ static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
 		if (buf_idx < 0)
 			continue;
 
-		ref_buf = to_vb2_v4l2_buffer(cap_q->bufs[buf_idx]);
-		cedrus_buf = vb2_v4l2_to_cedrus_buffer(ref_buf);
+		cedrus_buf = vb2_to_cedrus_buffer(cap_q->bufs[buf_idx]);
 		position = cedrus_buf->codec.h264.position;
 
 		sram_array[i] |= position << 1;
-		if (ref_buf->field == V4L2_FIELD_BOTTOM)
+		if (ref_list[i].fields & V4L2_H264_DPB_BOTTOM_REF)
 			sram_array[i] |= BIT(0);
 	}
 
-- 
2.27.0

[PATCH v2 14/14] media: cedrus: h264: Fix frame list construction

[Ezequiel Garcia]

From: Jernej Skrabec <jernej.skrabec@siol.net>

Current frame list construction algorithm assumes that decoded image
will be output into its own buffer. That is true for progressive content
but not for interlaced where each field is decoded separately into same
buffer.

Fix that by checking if capture buffer is listed in DPB. If it is, reuse
it.

Signed-off-by: Jernej Skrabec <jernej.skrabec@siol.net>
---
 drivers/staging/media/sunxi/cedrus/cedrus_h264.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
index 91d88a96badc..7b2169d185b8 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
@@ -101,7 +101,7 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
 	struct cedrus_dev *dev = ctx->dev;
 	unsigned long used_dpbs = 0;
 	unsigned int position;
-	unsigned int output = 0;
+	int output = -1;
 	unsigned int i;
 
 	cap_q = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE);
@@ -124,6 +124,11 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
 		position = cedrus_buf->codec.h264.position;
 		used_dpbs |= BIT(position);
 
+		if (run->dst->vb2_buf.timestamp == dpb->reference_ts) {
+			output = position;
+			continue;
+		}
+
 		if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE))
 			continue;
 
@@ -131,13 +136,11 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
 				    dpb->top_field_order_cnt,
 				    dpb->bottom_field_order_cnt,
 				    &pic_list[position]);
-
-		output = max(position, output);
 	}
 
-	position = find_next_zero_bit(&used_dpbs, CEDRUS_H264_FRAME_NUM,
-				      output);
-	if (position >= CEDRUS_H264_FRAME_NUM)
+	if (output >= 0)
+		position = output;
+	else
 		position = find_first_zero_bit(&used_dpbs, CEDRUS_H264_FRAME_NUM);
 
 	output_buf = vb2_to_cedrus_buffer(&run->dst->vb2_buf);
-- 
2.27.0

Re: [PATCH v2 01/14] media: uapi: h264: Update reference lists

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 6553 bytes --]

Hi,

On Thu 06 Aug 20, 12:12, Ezequiel Garcia wrote:
> From: Jernej Skrabec <jernej.skrabec@siol.net>
> 
> When dealing with with interlaced frames, reference lists must tell if
> each particular reference is meant for top or bottom field. This info
> is currently not provided at all in the H264 related controls.
> 
> Make reference lists hold a structure which will also hold an
> enumerator type along index into DPB array. The enumerator must
> be used to specify if reference is for top or bottom field.
> 
> Currently the only user of these lists is Cedrus which is just compile
> fixed here. Actual usage of will come in a following commit.

Is there a particular reason we are adding this to the ref_pic_list[0-1]
instead of the DPB entries directly?

It feels nicer to avoid making the lists structs when the entries they are
referring to are already in a struct. I think this is the approach Kwiboo took
when adding support for fields in references some time ago.

What do you think?

Cheers,

Paul

> Signed-off-by: Jernej Skrabec <jernej.skrabec@siol.net>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
> v2:
> * As pointed out by Jonas, enum v4l2_h264_dpb_reference here.
> ---
>  .../media/v4l/ext-ctrls-codec.rst             | 44 ++++++++++++++++++-
>  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 +--
>  include/media/h264-ctrls.h                    | 23 +++++++---
>  3 files changed, 62 insertions(+), 11 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> index d0d506a444b1..f2b2a381369f 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> @@ -1843,10 +1843,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>      * - __u32
>        - ``slice_group_change_cycle``
>        -
> -    * - __u8
> +    * - struct :c:type:`v4l2_h264_reference`
>        - ``ref_pic_list0[32]``
>        - Reference picture list after applying the per-slice modifications
> -    * - __u8
> +    * - struct :c:type:`v4l2_h264_reference`
>        - ``ref_pic_list1[32]``
>        - Reference picture list after applying the per-slice modifications
>      * - __u32
> @@ -1926,6 +1926,46 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>        - ``chroma_offset[32][2]``
>        -
>  
> +``Picture Reference``
> +
> +.. c:type:: v4l2_h264_reference
> +
> +.. cssclass:: longtable
> +
> +.. flat-table:: struct v4l2_h264_reference
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - enum :c:type:`v4l2_h264_dpb_reference`
> +      - ``reference``
> +      - Specifies how the DPB entry is referenced.
> +    * - __u8
> +      - ``index``
> +      - Index into the :c:type:`v4l2_ctrl_h264_decode_params`.dpb array.
> +
> +.. c:type:: v4l2_h264_dpb_reference
> +
> +.. cssclass:: longtable
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - ``V4L2_H264_DPB_TOP_REF``
> +      - 0x1
> +      - The top field in field pair is used for
> +        short-term reference.
> +    * - ``V4L2_H264_DPB_BOTTOM_REF``
> +      - 0x2
> +     - The bottom field in field pair is used for
> +        short-term reference.
> +    * - ``V4L2_H264_DPB_FRAME_REF``
> +      - 0x3
> +      - The frame (or the top/bottom fields, if it's a field pair)
> +        is used for short-term reference.
> +
>  ``V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (struct)``
>      Specifies the decode parameters (as extracted from the bitstream)
>      for the associated H264 slice data. This includes the necessary
> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> index 54ee2aa423e2..cce527bbdf86 100644
> --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> @@ -166,8 +166,8 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
>  
>  static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
>  				   struct cedrus_run *run,
> -				   const u8 *ref_list, u8 num_ref,
> -				   enum cedrus_h264_sram_off sram)
> +				   const struct v4l2_h264_reference *ref_list,
> +				   u8 num_ref, enum cedrus_h264_sram_off sram)
>  {
>  	const struct v4l2_ctrl_h264_decode_params *decode = run->h264.decode_params;
>  	struct vb2_queue *cap_q;
> @@ -188,7 +188,7 @@ static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
>  		int buf_idx;
>  		u8 dpb_idx;
>  
> -		dpb_idx = ref_list[i];
> +		dpb_idx = ref_list[i].index;
>  		dpb = &decode->dpb[dpb_idx];
>  
>  		if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE))
> diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> index 080fd1293c42..4c0bb7f5fb05 100644
> --- a/include/media/h264-ctrls.h
> +++ b/include/media/h264-ctrls.h
> @@ -19,6 +19,8 @@
>   */
>  #define V4L2_H264_NUM_DPB_ENTRIES 16
>  
> +#define V4L2_H264_REF_LIST_LEN (2 * V4L2_H264_NUM_DPB_ENTRIES)
> +
>  /* Our pixel format isn't stable at the moment */
>  #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
>  
> @@ -140,6 +142,19 @@ struct v4l2_h264_pred_weight_table {
>  #define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED	0x04
>  #define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH		0x08
>  
> +enum v4l2_h264_dpb_reference {
> +	V4L2_H264_DPB_TOP_REF = 0x1,
> +	V4L2_H264_DPB_BOTTOM_REF = 0x2,
> +	V4L2_H264_DPB_FRAME_REF = 0x3,
> +};
> +
> +struct v4l2_h264_reference {
> +	enum v4l2_h264_dpb_reference fields;
> +
> +	/* Index into v4l2_ctrl_h264_decode_params.dpb[] */
> +	__u8 index;
> +};
> +
>  struct v4l2_ctrl_h264_slice_params {
>  	/* Size in bytes, including header */
>  	__u32 size;
> @@ -178,12 +193,8 @@ struct v4l2_ctrl_h264_slice_params {
>  	__u8 num_ref_idx_l1_active_minus1;
>  	__u32 slice_group_change_cycle;
>  
> -	/*
> -	 * Entries on each list are indices into
> -	 * v4l2_ctrl_h264_decode_params.dpb[].
> -	 */
> -	__u8 ref_pic_list0[32];
> -	__u8 ref_pic_list1[32];
> +	struct v4l2_h264_reference ref_pic_list0[V4L2_H264_REF_LIST_LEN];
> +	struct v4l2_h264_reference ref_pic_list1[V4L2_H264_REF_LIST_LEN];
>  
>  	__u32 flags;
>  };
> -- 
> 2.27.0
> 

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 08/14] media: uapi: h264: Drop SLICE_PARAMS 'size' field

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 3502 bytes --]

Hi,

On Thu 06 Aug 20, 12:13, Ezequiel Garcia wrote:
> The SLICE_PARAMS control is intended for slice-based
> devices. In this mode, the OUTPUT buffer contains
> a single slice, and so the buffer's plane payload size
> can be used to query the slice size.

If we later extend the API for supporting multiple slices with dynamic array
controls, I guess we'll need to know the size of each slice in each control
elements. So I'd rather keep that even if it's indeed redundant with
vb2_get_plane_payload in single-slice mode.

What do you think?

Paul

> To reduce the API surface drop the size from the
> SLICE_PARAMS control.
> 
> A follow-up change will remove other members in SLICE_PARAMS
> so we don't need to add padding fields here.
> 
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 3 ---
>  drivers/staging/media/sunxi/cedrus/cedrus_h264.c          | 7 +++----
>  include/media/h264-ctrls.h                                | 3 ---
>  3 files changed, 3 insertions(+), 10 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> index 427fc5727ec0..fff74b7bf32a 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> @@ -1760,9 +1760,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>      :stub-columns: 0
>      :widths:       1 1 2
>  
> -    * - __u32
> -      - ``size``
> -      -
>      * - __u32
>        - ``start_byte_offset``
>          Offset (in bytes) from the beginning of the OUTPUT buffer to the start
> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> index a9ba78b15907..8b6f05aadbe8 100644
> --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> @@ -324,17 +324,16 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
>  	struct vb2_buffer *src_buf = &run->src->vb2_buf;
>  	struct cedrus_dev *dev = ctx->dev;
>  	dma_addr_t src_buf_addr;
> -	u32 len = slice->size * 8;
> +	size_t slice_bytes = vb2_get_plane_payload(src_buf, 0);
>  	unsigned int pic_width_in_mbs;
>  	bool mbaff_pic;
>  	u32 reg;
>  
> -	cedrus_write(dev, VE_H264_VLD_LEN, len);
> +	cedrus_write(dev, VE_H264_VLD_LEN, slice_bytes * 8);
>  	cedrus_write(dev, VE_H264_VLD_OFFSET, 0);
>  
>  	src_buf_addr = vb2_dma_contig_plane_dma_addr(src_buf, 0);
> -	cedrus_write(dev, VE_H264_VLD_END,
> -		     src_buf_addr + vb2_get_plane_payload(src_buf, 0));
> +	cedrus_write(dev, VE_H264_VLD_END, src_buf_addr + slice_bytes);
>  	cedrus_write(dev, VE_H264_VLD_ADDR,
>  		     VE_H264_VLD_ADDR_VAL(src_buf_addr) |
>  		     VE_H264_VLD_ADDR_FIRST | VE_H264_VLD_ADDR_VALID |
> diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> index 4f05ee265997..f74736fcfa00 100644
> --- a/include/media/h264-ctrls.h
> +++ b/include/media/h264-ctrls.h
> @@ -158,9 +158,6 @@ struct v4l2_h264_reference {
>  };
>  
>  struct v4l2_ctrl_h264_slice_params {
> -	/* Size in bytes, including header */
> -	__u32 size;
> -
>  	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
>  	__u32 start_byte_offset;
>  
> -- 
> 2.27.0
> 

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 09/14] media: uapi: h264: Clarify SLICE_BASED mode

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 6130 bytes --]

Hi,

On Thu 06 Aug 20, 12:13, Ezequiel Garcia wrote:
> Currently, the SLICE_BASED and FRAME_BASED modes documentation
> is misleading and not matching the intended use-cases.
> 
> Drop non-required fields SLICE_PARAMS 'start_byte_offset' and
> DECODE_PARAMS 'num_slices' and clarify the decoding modes in the
> documentation.
> 
> On SLICE_BASED mode, a single slice is expected per OUTPUT buffer,
> and therefore 'start_byte_offset' is not needed (since the offset
> to the slice is the start of the buffer).

Same comment as for the size: how do we plan on managing multiple slices
in a single output buffer later on after removing this?

Cheers,

Paul

> This mode requires the use of CAPTURE buffer holding, and so
> the number of slices shall not be required.
> 
> On FRAME_BASED mode, the devices are expected to take care of slice
> parsing. Neither SLICE_PARAMS are required (and shouldn't be
> exposed by frame-based drivers), nor the number of slices.
> 
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  .../media/v4l/ext-ctrls-codec.rst             | 39 +++++--------------
>  include/media/h264-ctrls.h                    |  4 --
>  2 files changed, 10 insertions(+), 33 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> index fff74b7bf32a..b3963c1acc2c 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> @@ -1748,9 +1748,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>         This compound control is not yet part of the public kernel API
>         and it is expected to change.
>  
> -       This structure is expected to be passed as an array, with one
> -       entry for each slice included in the bitstream buffer.
> -
>  .. c:type:: v4l2_ctrl_h264_slice_params
>  
>  .. cssclass:: longtable
> @@ -1760,17 +1757,9 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>      :stub-columns: 0
>      :widths:       1 1 2
>  
> -    * - __u32
> -      - ``start_byte_offset``
> -        Offset (in bytes) from the beginning of the OUTPUT buffer to the start
> -        of the slice. If the slice starts with a start code, then this is the
> -        offset to such start code. When operating in slice-based decoding mode
> -        (see :c:type:`v4l2_mpeg_video_h264_decode_mode`), this field should
> -        be set to 0. When operating in frame-based decoding mode, this field
> -        should be 0 for the first slice.
>      * - __u32
>        - ``header_bit_size``
> -      -
> +      - Offset in bits to slice_data() from the beginning of this slice.
>      * - __u32
>        - ``first_mb_in_slice``
>        -
> @@ -1998,12 +1987,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>      * - struct :c:type:`v4l2_h264_dpb_entry`
>        - ``dpb[16]``
>        -
> -    * - __u16
> -      - ``num_slices``
> -      - Number of slices needed to decode the current frame/field. When
> -        operating in slice-based decoding mode (see
> -        :c:type:`v4l2_mpeg_video_h264_decode_mode`), this field
> -        should always be set to one.
>      * - __u16
>        - ``nal_ref_idc``
>        - NAL reference ID value coming from the NAL Unit header
> @@ -2121,22 +2104,20 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>      * - ``V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED``
>        - 0
>        - Decoding is done at the slice granularity.
> -        In this mode, ``num_slices`` field in struct
> -        :c:type:`v4l2_ctrl_h264_decode_params` should be set to 1,
> -        and ``start_byte_offset`` in struct
> -        :c:type:`v4l2_ctrl_h264_slice_params` should be set to 0.
>          The OUTPUT buffer must contain a single slice.
> +        When this mode is selected, the ``V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS``
> +        control shall be set. When multiple slices compose a frame,
> +        use of ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` flag
> +        is required.
>      * - ``V4L2_MPEG_VIDEO_H264_DECODE_MODE_FRAME_BASED``
>        - 1
> -      - Decoding is done at the frame granularity.
> -        In this mode, ``num_slices`` field in struct
> -        :c:type:`v4l2_ctrl_h264_decode_params` should be set to the number
> -        of slices in the frame, and ``start_byte_offset`` in struct
> -        :c:type:`v4l2_ctrl_h264_slice_params` should be set accordingly
> -        for each slice. For the first slice, ``start_byte_offset`` should
> -        be zero.
> +      - Decoding is done at the frame granularity,
>          The OUTPUT buffer must contain all slices needed to decode the
>          frame. The OUTPUT buffer must also contain both fields.
> +        This mode will be supported by devices that
> +        parse the slice(s) header(s) in hardware. When this mode is
> +        selected, the ``V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS``
> +        control shall not be set.
>  
>  ``V4L2_CID_MPEG_VIDEO_H264_START_CODE (enum)``
>      Specifies the H264 slice start code expected for each slice.
> diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> index f74736fcfa00..ea8c8c93305b 100644
> --- a/include/media/h264-ctrls.h
> +++ b/include/media/h264-ctrls.h
> @@ -158,9 +158,6 @@ struct v4l2_h264_reference {
>  };
>  
>  struct v4l2_ctrl_h264_slice_params {
> -	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
> -	__u32 start_byte_offset;
> -
>  	/* Offset in bits to slice_data() from the beginning of this slice. */
>  	__u32 header_bit_size;
>  
> @@ -219,7 +216,6 @@ struct v4l2_h264_dpb_entry {
>  
>  struct v4l2_ctrl_h264_decode_params {
>  	struct v4l2_h264_dpb_entry dpb[V4L2_H264_NUM_DPB_ENTRIES];
> -	__u16 num_slices;
>  	__u16 nal_ref_idc;
>  	__s32 top_field_order_cnt;
>  	__s32 bottom_field_order_cnt;
> -- 
> 2.27.0
> 

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 01/14] media: uapi: h264: Update reference lists

[Jernej Škrabec]

Hi!

Dne četrtek, 06. avgust 2020 ob 17:47:07 CEST je Paul Kocialkowski napisal(a):
> Hi,
> 
> On Thu 06 Aug 20, 12:12, Ezequiel Garcia wrote:
> > From: Jernej Skrabec <jernej.skrabec@siol.net>
> > 
> > When dealing with with interlaced frames, reference lists must tell if
> > each particular reference is meant for top or bottom field. This info
> > is currently not provided at all in the H264 related controls.
> > 
> > Make reference lists hold a structure which will also hold an
> > enumerator type along index into DPB array. The enumerator must
> > be used to specify if reference is for top or bottom field.
> > 
> > Currently the only user of these lists is Cedrus which is just compile
> > fixed here. Actual usage of will come in a following commit.
> 
> Is there a particular reason we are adding this to the ref_pic_list[0-1]
> instead of the DPB entries directly?

Yes, it is.

> 
> It feels nicer to avoid making the lists structs when the entries they are
> referring to are already in a struct. I think this is the approach Kwiboo
> took when adding support for fields in references some time ago.
> 
> What do you think?

It's different thing. I tried using that, but image wasn't decoded correctly. 
IMO this is also the same reason why VAAPI doesn't have indices to DPB and 
instead have full VAPictureH264 structure array for RefPicList0 and 
RefPicList1. VAAPI has also note here "/* See 8.2.4.2 */" but I need to check 
it...

Best regards,
Jernej 

> 
> Cheers,
> 
> Paul
> 
> > Signed-off-by: Jernej Skrabec <jernej.skrabec@siol.net>
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> > v2:
> > * As pointed out by Jonas, enum v4l2_h264_dpb_reference here.
> > ---
> > 
> >  .../media/v4l/ext-ctrls-codec.rst             | 44 ++++++++++++++++++-
> >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 +--
> >  include/media/h264-ctrls.h                    | 23 +++++++---
> >  3 files changed, 62 insertions(+), 11 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
> > d0d506a444b1..f2b2a381369f 100644
> > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > @@ -1843,10 +1843,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type
> > -> 
> >      * - __u32
> >      
> >        - ``slice_group_change_cycle``
> >        -
> > 
> > -    * - __u8
> > +    * - struct :c:type:`v4l2_h264_reference`
> > 
> >        - ``ref_pic_list0[32]``
> >        - Reference picture list after applying the per-slice modifications
> > 
> > -    * - __u8
> > +    * - struct :c:type:`v4l2_h264_reference`
> > 
> >        - ``ref_pic_list1[32]``
> >        - Reference picture list after applying the per-slice modifications
> >      
> >      * - __u32
> > 
> > @@ -1926,6 +1926,46 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type
> > -
> > 
> >        - ``chroma_offset[32][2]``
> >        -
> > 
> > +``Picture Reference``
> > +
> > +.. c:type:: v4l2_h264_reference
> > +
> > +.. cssclass:: longtable
> > +
> > +.. flat-table:: struct v4l2_h264_reference
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths:       1 1 2
> > +
> > +    * - enum :c:type:`v4l2_h264_dpb_reference`
> > +      - ``reference``
> > +      - Specifies how the DPB entry is referenced.
> > +    * - __u8
> > +      - ``index``
> > +      - Index into the :c:type:`v4l2_ctrl_h264_decode_params`.dpb array.
> > +
> > +.. c:type:: v4l2_h264_dpb_reference
> > +
> > +.. cssclass:: longtable
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths:       1 1 2
> > +
> > +    * - ``V4L2_H264_DPB_TOP_REF``
> > +      - 0x1
> > +      - The top field in field pair is used for
> > +        short-term reference.
> > +    * - ``V4L2_H264_DPB_BOTTOM_REF``
> > +      - 0x2
> > +     - The bottom field in field pair is used for
> > +        short-term reference.
> > +    * - ``V4L2_H264_DPB_FRAME_REF``
> > +      - 0x3
> > +      - The frame (or the top/bottom fields, if it's a field pair)
> > +        is used for short-term reference.
> > +
> > 
> >  ``V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (struct)``
> >  
> >      Specifies the decode parameters (as extracted from the bitstream)
> >      for the associated H264 slice data. This includes the necessary
> > 
> > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c index
> > 54ee2aa423e2..cce527bbdf86 100644
> > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > @@ -166,8 +166,8 @@ static void cedrus_write_frame_list(struct cedrus_ctx
> > *ctx,> 
> >  static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
> >  
> >  				   struct cedrus_run *run,
> > 
> > -				   const u8 *ref_list, u8 
num_ref,
> > -				   enum cedrus_h264_sram_off sram)
> > +				   const struct 
v4l2_h264_reference *ref_list,
> > +				   u8 num_ref, enum 
cedrus_h264_sram_off sram)
> > 
> >  {
> >  
> >  	const struct v4l2_ctrl_h264_decode_params *decode =
> >  	run->h264.decode_params; struct vb2_queue *cap_q;
> > 
> > @@ -188,7 +188,7 @@ static void _cedrus_write_ref_list(struct cedrus_ctx
> > *ctx,> 
> >  		int buf_idx;
> >  		u8 dpb_idx;
> > 
> > -		dpb_idx = ref_list[i];
> > +		dpb_idx = ref_list[i].index;
> > 
> >  		dpb = &decode->dpb[dpb_idx];
> >  		
> >  		if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE))
> > 
> > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > index 080fd1293c42..4c0bb7f5fb05 100644
> > --- a/include/media/h264-ctrls.h
> > +++ b/include/media/h264-ctrls.h
> > @@ -19,6 +19,8 @@
> > 
> >   */
> >  
> >  #define V4L2_H264_NUM_DPB_ENTRIES 16
> > 
> > +#define V4L2_H264_REF_LIST_LEN (2 * V4L2_H264_NUM_DPB_ENTRIES)
> > +
> > 
> >  /* Our pixel format isn't stable at the moment */
> >  #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264
> >  parsed slices */> 
> > @@ -140,6 +142,19 @@ struct v4l2_h264_pred_weight_table {
> > 
> >  #define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED	0x04
> >  #define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH		0x08
> > 
> > +enum v4l2_h264_dpb_reference {
> > +	V4L2_H264_DPB_TOP_REF = 0x1,
> > +	V4L2_H264_DPB_BOTTOM_REF = 0x2,
> > +	V4L2_H264_DPB_FRAME_REF = 0x3,
> > +};
> > +
> > +struct v4l2_h264_reference {
> > +	enum v4l2_h264_dpb_reference fields;
> > +
> > +	/* Index into v4l2_ctrl_h264_decode_params.dpb[] */
> > +	__u8 index;
> > +};
> > +
> > 
> >  struct v4l2_ctrl_h264_slice_params {
> >  
> >  	/* Size in bytes, including header */
> >  	__u32 size;
> > 
> > @@ -178,12 +193,8 @@ struct v4l2_ctrl_h264_slice_params {
> > 
> >  	__u8 num_ref_idx_l1_active_minus1;
> >  	__u32 slice_group_change_cycle;
> > 
> > -	/*
> > -	 * Entries on each list are indices into
> > -	 * v4l2_ctrl_h264_decode_params.dpb[].
> > -	 */
> > -	__u8 ref_pic_list0[32];
> > -	__u8 ref_pic_list1[32];
> > +	struct v4l2_h264_reference ref_pic_list0[V4L2_H264_REF_LIST_LEN];
> > +	struct v4l2_h264_reference ref_pic_list1[V4L2_H264_REF_LIST_LEN];
> > 
> >  	__u32 flags;
> >  
> >  };

Re: [PATCH v2 01/14] media: uapi: h264: Update reference lists

[Ezequiel Garcia]

On Thu, 2020-08-06 at 17:47 +0200, Paul Kocialkowski wrote:
> Hi,
> 
> On Thu 06 Aug 20, 12:12, Ezequiel Garcia wrote:
> > From: Jernej Skrabec <jernej.skrabec@siol.net>
> > 
> > When dealing with with interlaced frames, reference lists must tell if
> > each particular reference is meant for top or bottom field. This info
> > is currently not provided at all in the H264 related controls.
> > 
> > Make reference lists hold a structure which will also hold an
> > enumerator type along index into DPB array. The enumerator must
> > be used to specify if reference is for top or bottom field.
> > 
> > Currently the only user of these lists is Cedrus which is just compile
> > fixed here. Actual usage of will come in a following commit.
> 
> Is there a particular reason we are adding this to the ref_pic_list[0-1]
> instead of the DPB entries directly?
> 

I actually asked the same question to Jernej's original series.

Given the references are per-field, which is why there are twice
as many references as DPB entries, the semantics proposed here
are required for slice-based decoders.

See Nicolas' reply:

https://www.spinics.net/lists/arm-kernel/msg812277.html

As Jernej's already reply, this is required to fix Cedrus.

Thanks!
Ezequiel

> Cheers,
> 
> Paul
> 
> > Signed-off-by: Jernej Skrabec <jernej.skrabec@siol.net>
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> > v2:
> > * As pointed out by Jonas, enum v4l2_h264_dpb_reference here.
> > ---
> >  .../media/v4l/ext-ctrls-codec.rst             | 44 ++++++++++++++++++-
> >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 +--
> >  include/media/h264-ctrls.h                    | 23 +++++++---
> >  3 files changed, 62 insertions(+), 11 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > index d0d506a444b1..f2b2a381369f 100644
> > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > @@ -1843,10 +1843,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
> >      * - __u32
> >        - ``slice_group_change_cycle``
> >        -
> > -    * - __u8
> > +    * - struct :c:type:`v4l2_h264_reference`
> >        - ``ref_pic_list0[32]``
> >        - Reference picture list after applying the per-slice modifications
> > -    * - __u8
> > +    * - struct :c:type:`v4l2_h264_reference`
> >        - ``ref_pic_list1[32]``
> >        - Reference picture list after applying the per-slice modifications
> >      * - __u32
> > @@ -1926,6 +1926,46 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
> >        - ``chroma_offset[32][2]``
> >        -
> >  
> > +``Picture Reference``
> > +
> > +.. c:type:: v4l2_h264_reference
> > +
> > +.. cssclass:: longtable
> > +
> > +.. flat-table:: struct v4l2_h264_reference
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths:       1 1 2
> > +
> > +    * - enum :c:type:`v4l2_h264_dpb_reference`
> > +      - ``reference``
> > +      - Specifies how the DPB entry is referenced.
> > +    * - __u8
> > +      - ``index``
> > +      - Index into the :c:type:`v4l2_ctrl_h264_decode_params`.dpb array.
> > +
> > +.. c:type:: v4l2_h264_dpb_reference
> > +
> > +.. cssclass:: longtable
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths:       1 1 2
> > +
> > +    * - ``V4L2_H264_DPB_TOP_REF``
> > +      - 0x1
> > +      - The top field in field pair is used for
> > +        short-term reference.
> > +    * - ``V4L2_H264_DPB_BOTTOM_REF``
> > +      - 0x2
> > +     - The bottom field in field pair is used for
> > +        short-term reference.
> > +    * - ``V4L2_H264_DPB_FRAME_REF``
> > +      - 0x3
> > +      - The frame (or the top/bottom fields, if it's a field pair)
> > +        is used for short-term reference.
> > +
> >  ``V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (struct)``
> >      Specifies the decode parameters (as extracted from the bitstream)
> >      for the associated H264 slice data. This includes the necessary
> > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > index 54ee2aa423e2..cce527bbdf86 100644
> > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > @@ -166,8 +166,8 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
> >  
> >  static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
> >  				   struct cedrus_run *run,
> > -				   const u8 *ref_list, u8 num_ref,
> > -				   enum cedrus_h264_sram_off sram)
> > +				   const struct v4l2_h264_reference *ref_list,
> > +				   u8 num_ref, enum cedrus_h264_sram_off sram)
> >  {
> >  	const struct v4l2_ctrl_h264_decode_params *decode = run->h264.decode_params;
> >  	struct vb2_queue *cap_q;
> > @@ -188,7 +188,7 @@ static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
> >  		int buf_idx;
> >  		u8 dpb_idx;
> >  
> > -		dpb_idx = ref_list[i];
> > +		dpb_idx = ref_list[i].index;
> >  		dpb = &decode->dpb[dpb_idx];
> >  
> >  		if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE))
> > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > index 080fd1293c42..4c0bb7f5fb05 100644
> > --- a/include/media/h264-ctrls.h
> > +++ b/include/media/h264-ctrls.h
> > @@ -19,6 +19,8 @@
> >   */
> >  #define V4L2_H264_NUM_DPB_ENTRIES 16
> >  
> > +#define V4L2_H264_REF_LIST_LEN (2 * V4L2_H264_NUM_DPB_ENTRIES)
> > +
> >  /* Our pixel format isn't stable at the moment */
> >  #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
> >  
> > @@ -140,6 +142,19 @@ struct v4l2_h264_pred_weight_table {
> >  #define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED	0x04
> >  #define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH		0x08
> >  
> > +enum v4l2_h264_dpb_reference {
> > +	V4L2_H264_DPB_TOP_REF = 0x1,
> > +	V4L2_H264_DPB_BOTTOM_REF = 0x2,
> > +	V4L2_H264_DPB_FRAME_REF = 0x3,
> > +};
> > +
> > +struct v4l2_h264_reference {
> > +	enum v4l2_h264_dpb_reference fields;
> > +
> > +	/* Index into v4l2_ctrl_h264_decode_params.dpb[] */
> > +	__u8 index;
> > +};
> > +
> >  struct v4l2_ctrl_h264_slice_params {
> >  	/* Size in bytes, including header */
> >  	__u32 size;
> > @@ -178,12 +193,8 @@ struct v4l2_ctrl_h264_slice_params {
> >  	__u8 num_ref_idx_l1_active_minus1;
> >  	__u32 slice_group_change_cycle;
> >  
> > -	/*
> > -	 * Entries on each list are indices into
> > -	 * v4l2_ctrl_h264_decode_params.dpb[].
> > -	 */
> > -	__u8 ref_pic_list0[32];
> > -	__u8 ref_pic_list1[32];
> > +	struct v4l2_h264_reference ref_pic_list0[V4L2_H264_REF_LIST_LEN];
> > +	struct v4l2_h264_reference ref_pic_list1[V4L2_H264_REF_LIST_LEN];
> >  
> >  	__u32 flags;
> >  };
> > -- 
> > 2.27.0
> > 

Re: [PATCH v2 08/14] media: uapi: h264: Drop SLICE_PARAMS 'size' field

[Ezequiel Garcia]

On Thu, 2020-08-06 at 17:50 +0200, Paul Kocialkowski wrote:
> Hi,
> 
> On Thu 06 Aug 20, 12:13, Ezequiel Garcia wrote:
> > The SLICE_PARAMS control is intended for slice-based
> > devices. In this mode, the OUTPUT buffer contains
> > a single slice, and so the buffer's plane payload size
> > can be used to query the slice size.
> 
> If we later extend the API for supporting multiple slices with dynamic array
> controls, I guess we'll need to know the size of each slice in each control
> elements. So I'd rather keep that even if it's indeed redundant with
> vb2_get_plane_payload in single-slice mode.
> 

If we later extend the API, another control (possibly
another decoding mode?) shall be introduced.

This API covers single-slice-per-request as specified
and documented in patch 9/14 "Clarify SLICE_BASED mode".

This is along the lines of the proposal drafted by Nicolas,
see my reply: https://lkml.org/lkml/2020/8/5/791.

This applies to num_slices, slice size and slice start offset.

There are multiple ways of doing this.

Thanks!
Ezequiel

> What do you think?
> 
> Paul
> 
> > To reduce the API surface drop the size from the
> > SLICE_PARAMS control.
> > 
> > A follow-up change will remove other members in SLICE_PARAMS
> > so we don't need to add padding fields here.
> > 
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> >  Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 3 ---
> >  drivers/staging/media/sunxi/cedrus/cedrus_h264.c          | 7 +++----
> >  include/media/h264-ctrls.h                                | 3 ---
> >  3 files changed, 3 insertions(+), 10 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > index 427fc5727ec0..fff74b7bf32a 100644
> > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > @@ -1760,9 +1760,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
> >      :stub-columns: 0
> >      :widths:       1 1 2
> >  
> > -    * - __u32
> > -      - ``size``
> > -      -
> >      * - __u32
> >        - ``start_byte_offset``
> >          Offset (in bytes) from the beginning of the OUTPUT buffer to the start
> > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > index a9ba78b15907..8b6f05aadbe8 100644
> > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > @@ -324,17 +324,16 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
> >  	struct vb2_buffer *src_buf = &run->src->vb2_buf;
> >  	struct cedrus_dev *dev = ctx->dev;
> >  	dma_addr_t src_buf_addr;
> > -	u32 len = slice->size * 8;
> > +	size_t slice_bytes = vb2_get_plane_payload(src_buf, 0);
> >  	unsigned int pic_width_in_mbs;
> >  	bool mbaff_pic;
> >  	u32 reg;
> >  
> > -	cedrus_write(dev, VE_H264_VLD_LEN, len);
> > +	cedrus_write(dev, VE_H264_VLD_LEN, slice_bytes * 8);
> >  	cedrus_write(dev, VE_H264_VLD_OFFSET, 0);
> >  
> >  	src_buf_addr = vb2_dma_contig_plane_dma_addr(src_buf, 0);
> > -	cedrus_write(dev, VE_H264_VLD_END,
> > -		     src_buf_addr + vb2_get_plane_payload(src_buf, 0));
> > +	cedrus_write(dev, VE_H264_VLD_END, src_buf_addr + slice_bytes);
> >  	cedrus_write(dev, VE_H264_VLD_ADDR,
> >  		     VE_H264_VLD_ADDR_VAL(src_buf_addr) |
> >  		     VE_H264_VLD_ADDR_FIRST | VE_H264_VLD_ADDR_VALID |
> > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > index 4f05ee265997..f74736fcfa00 100644
> > --- a/include/media/h264-ctrls.h
> > +++ b/include/media/h264-ctrls.h
> > @@ -158,9 +158,6 @@ struct v4l2_h264_reference {
> >  };
> >  
> >  struct v4l2_ctrl_h264_slice_params {
> > -	/* Size in bytes, including header */
> > -	__u32 size;
> > -
> >  	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
> >  	__u32 start_byte_offset;
> >  
> > -- 
> > 2.27.0
> > 

Re: [PATCH v2 01/14] media: uapi: h264: Update reference lists

[Ezequiel Garcia]

On Thu, 6 Aug 2020 at 14:38, Ezequiel Garcia <ezequiel@collabora.com> wrote:
>
> From: Jernej Skrabec <jernej.skrabec@siol.net>
>
> When dealing with with interlaced frames, reference lists must tell if
> each particular reference is meant for top or bottom field. This info
> is currently not provided at all in the H264 related controls.
>
> Make reference lists hold a structure which will also hold an
> enumerator type along index into DPB array. The enumerator must
> be used to specify if reference is for top or bottom field.
>
> Currently the only user of these lists is Cedrus which is just compile
> fixed here. Actual usage of will come in a following commit.
>
> Signed-off-by: Jernej Skrabec <jernej.skrabec@siol.net>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
> v2:
> * As pointed out by Jonas, enum v4l2_h264_dpb_reference here.
> ---
>  .../media/v4l/ext-ctrls-codec.rst             | 44 ++++++++++++++++++-
>  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 +--
>  include/media/h264-ctrls.h                    | 23 +++++++---
>  3 files changed, 62 insertions(+), 11 deletions(-)
>
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> index d0d506a444b1..f2b2a381369f 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> @@ -1843,10 +1843,10 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>      * - __u32
>        - ``slice_group_change_cycle``
>        -
> -    * - __u8
> +    * - struct :c:type:`v4l2_h264_reference`
>        - ``ref_pic_list0[32]``
>        - Reference picture list after applying the per-slice modifications
> -    * - __u8
> +    * - struct :c:type:`v4l2_h264_reference`
>        - ``ref_pic_list1[32]``
>        - Reference picture list after applying the per-slice modifications
>      * - __u32
> @@ -1926,6 +1926,46 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>        - ``chroma_offset[32][2]``
>        -
>
> +``Picture Reference``
> +
> +.. c:type:: v4l2_h264_reference
> +
> +.. cssclass:: longtable
> +
> +.. flat-table:: struct v4l2_h264_reference
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - enum :c:type:`v4l2_h264_dpb_reference`
> +      - ``reference``
> +      - Specifies how the DPB entry is referenced.

Ah, a small overlook on my side. Given this struct
is used for DPB entries and slice references,
we should name this using something more generic,
and not mention DPB to avoid confusions.

Perhaps "struct v4l2_h264_field_reference".

Thanks!
Ezequiel


> +    * - __u8
> +      - ``index``
> +      - Index into the :c:type:`v4l2_ctrl_h264_decode_params`.dpb array.
> +
> +.. c:type:: v4l2_h264_dpb_reference
> +
> +.. cssclass:: longtable
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - ``V4L2_H264_DPB_TOP_REF``
> +      - 0x1
> +      - The top field in field pair is used for
> +        short-term reference.
> +    * - ``V4L2_H264_DPB_BOTTOM_REF``
> +      - 0x2
> +     - The bottom field in field pair is used for
> +        short-term reference.
> +    * - ``V4L2_H264_DPB_FRAME_REF``
> +      - 0x3
> +      - The frame (or the top/bottom fields, if it's a field pair)
> +        is used for short-term reference.
> +
>  ``V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS (struct)``
>      Specifies the decode parameters (as extracted from the bitstream)
>      for the associated H264 slice data. This includes the necessary
> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> index 54ee2aa423e2..cce527bbdf86 100644
> --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> @@ -166,8 +166,8 @@ static void cedrus_write_frame_list(struct cedrus_ctx *ctx,
>
>  static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
>                                    struct cedrus_run *run,
> -                                  const u8 *ref_list, u8 num_ref,
> -                                  enum cedrus_h264_sram_off sram)
> +                                  const struct v4l2_h264_reference *ref_list,
> +                                  u8 num_ref, enum cedrus_h264_sram_off sram)
>  {
>         const struct v4l2_ctrl_h264_decode_params *decode = run->h264.decode_params;
>         struct vb2_queue *cap_q;
> @@ -188,7 +188,7 @@ static void _cedrus_write_ref_list(struct cedrus_ctx *ctx,
>                 int buf_idx;
>                 u8 dpb_idx;
>
> -               dpb_idx = ref_list[i];
> +               dpb_idx = ref_list[i].index;
>                 dpb = &decode->dpb[dpb_idx];
>
>                 if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE))
> diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> index 080fd1293c42..4c0bb7f5fb05 100644
> --- a/include/media/h264-ctrls.h
> +++ b/include/media/h264-ctrls.h
> @@ -19,6 +19,8 @@
>   */
>  #define V4L2_H264_NUM_DPB_ENTRIES 16
>
> +#define V4L2_H264_REF_LIST_LEN (2 * V4L2_H264_NUM_DPB_ENTRIES)
> +
>  /* Our pixel format isn't stable at the moment */
>  #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
>
> @@ -140,6 +142,19 @@ struct v4l2_h264_pred_weight_table {
>  #define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED    0x04
>  #define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH             0x08
>
> +enum v4l2_h264_dpb_reference {
> +       V4L2_H264_DPB_TOP_REF = 0x1,
> +       V4L2_H264_DPB_BOTTOM_REF = 0x2,
> +       V4L2_H264_DPB_FRAME_REF = 0x3,
> +};
> +
> +struct v4l2_h264_reference {
> +       enum v4l2_h264_dpb_reference fields;
> +
> +       /* Index into v4l2_ctrl_h264_decode_params.dpb[] */
> +       __u8 index;
> +};
> +
>  struct v4l2_ctrl_h264_slice_params {
>         /* Size in bytes, including header */
>         __u32 size;
> @@ -178,12 +193,8 @@ struct v4l2_ctrl_h264_slice_params {
>         __u8 num_ref_idx_l1_active_minus1;
>         __u32 slice_group_change_cycle;
>
> -       /*
> -        * Entries on each list are indices into
> -        * v4l2_ctrl_h264_decode_params.dpb[].
> -        */
> -       __u8 ref_pic_list0[32];
> -       __u8 ref_pic_list1[32];
> +       struct v4l2_h264_reference ref_pic_list0[V4L2_H264_REF_LIST_LEN];
> +       struct v4l2_h264_reference ref_pic_list1[V4L2_H264_REF_LIST_LEN];
>
>         __u32 flags;
>  };
> --
> 2.27.0
>

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Jonas Karlman]

On 2020-08-06 17:12, Ezequiel Garcia wrote:
> The prediction weight parameters are only required under
> certain conditions, which depend on slice header parameters.
> 
> As specified in section 7.3.3 Slice header syntax, the prediction
> weight table is present if:
> 
> ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> (weighted_bipred_idc == 1 && slice_type == B))

Maybe a macro can be added to help check this contition?

Something like this maybe:

#define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
	((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
	 ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
	   (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
	 ((pps)->weighted_bipred_idc == 1 && \
	  (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))

> 
> Given its size, it makes sense to move this table to its control,
> so applications can avoid passing it if the slice doesn't specify it.
> 
> Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
> is 772 bytes.
> 
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
> v2: Fix missing Cedrus changes and mssing control declaration,
>     as noted by Hans and Jernej.
> ---
>  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
>  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
>  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
>  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
>  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
>  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
>  include/media/h264-ctrls.h                    |  5 +++--
>  include/media/v4l2-ctrls.h                    |  2 ++
>  8 files changed, 37 insertions(+), 13 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> index d1438b1e259f..c36ce5a95fc5 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> @@ -1879,18 +1879,23 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
>        - 0x00000008
>        -
>  
> -``Prediction Weight Table``
> +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> +    Prediction weight table defined according to :ref:`h264`,
> +    section 7.4.3.2 "Prediction Weight Table Semantics".
> +    The prediction weight table must be passed by applications
> +    under the conditions explained in section 7.3.3 "Slice header
> +    syntax".
>  
> -    The bitstream parameters are defined according to :ref:`h264`,
> -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
> -    documentation, refer to the above specification, unless there is
> -    an explicit comment stating otherwise.
> +    .. note::
> +
> +       This compound control is not yet part of the public kernel API and
> +       it is expected to change.
>  
> -.. c:type:: v4l2_h264_pred_weight_table
> +.. c:type:: v4l2_ctrl_h264_pred_weights
>  
>  .. cssclass:: longtable
>  
> -.. flat-table:: struct v4l2_h264_pred_weight_table
> +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
>      :header-rows:  0
>      :stub-columns: 0
>      :widths:       1 1 2
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
> index 3f3fbcd60cc6..76c8dc8fb31c 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
>  	case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:		return "H264 Decode Parameters";
>  	case V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:		return "H264 Decode Mode";
>  	case V4L2_CID_MPEG_VIDEO_H264_START_CODE:		return "H264 Start Code";
> +	case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:		return "H264 Prediction Weight Table";
>  	case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:			return "MPEG2 Level";
>  	case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:			return "MPEG2 Profile";
>  	case V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:		return "MPEG4 I-Frame QP Value";
> @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
>  	case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
>  		*type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
>  		break;
> +	case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> +		*type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> +		break;
>  	case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
>  		*type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
>  		break;
> @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
>  	case V4L2_CTRL_TYPE_H264_SPS:
>  	case V4L2_CTRL_TYPE_H264_PPS:
>  	case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> +	case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
>  	case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
>  	case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
>  		break;
> @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
>  	case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
>  		elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
>  		break;
> +	case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> +		elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
> +		break;
>  	case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
>  		elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
>  		break;
> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c
> index bc27f9430eeb..027cdd1be5a0 100644
> --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[] = {
>  		.codec		= CEDRUS_CODEC_H264,
>  		.required	= true,
>  	},
> +	{
> +		.cfg = {
> +			.id	= V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> +		},
> +		.codec		= CEDRUS_CODEC_H264,
> +		.required	= true,

This should probably be false if this control is to be optional as implied
by the commit message.

Best regards,
Jonas

> +	},
>  	{
>  		.cfg = {
>  			.id	= V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE,
> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.h b/drivers/staging/media/sunxi/cedrus/cedrus.h
> index 96765555ab8a..93c843ae14bb 100644
> --- a/drivers/staging/media/sunxi/cedrus/cedrus.h
> +++ b/drivers/staging/media/sunxi/cedrus/cedrus.h
> @@ -62,6 +62,7 @@ struct cedrus_h264_run {
>  	const struct v4l2_ctrl_h264_scaling_matrix	*scaling_matrix;
>  	const struct v4l2_ctrl_h264_slice_params	*slice_params;
>  	const struct v4l2_ctrl_h264_sps			*sps;
> +	const struct v4l2_ctrl_h264_pred_weights	*pred_weights;
>  };
>  
>  struct cedrus_mpeg2_run {
> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> index 58c48e4fdfe9..6385026d1b6b 100644
> --- a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> +++ b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> @@ -57,6 +57,8 @@ void cedrus_device_run(void *priv)
>  			V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS);
>  		run.h264.sps = cedrus_find_control_data(ctx,
>  			V4L2_CID_MPEG_VIDEO_H264_SPS);
> +		run.h264.pred_weights = cedrus_find_control_data(ctx,
> +			V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS);
>  		break;
>  
>  	case V4L2_PIX_FMT_HEVC_SLICE:
> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> index cce527bbdf86..a9ba78b15907 100644
> --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> @@ -256,10 +256,8 @@ static void cedrus_write_scaling_lists(struct cedrus_ctx *ctx,
>  static void cedrus_write_pred_weight_table(struct cedrus_ctx *ctx,
>  					   struct cedrus_run *run)
>  {
> -	const struct v4l2_ctrl_h264_slice_params *slice =
> -		run->h264.slice_params;
> -	const struct v4l2_h264_pred_weight_table *pred_weight =
> -		&slice->pred_weight_table;
> +	const struct v4l2_ctrl_h264_pred_weights *pred_weight =
> +		run->h264.pred_weights;
>  	struct cedrus_dev *dev = ctx->dev;
>  	int i, j, k;
>  
> diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> index 4c0bb7f5fb05..54cd9bec0b23 100644
> --- a/include/media/h264-ctrls.h
> +++ b/include/media/h264-ctrls.h
> @@ -36,6 +36,7 @@
>  #define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS	(V4L2_CID_MPEG_BASE+1004)
>  #define V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE	(V4L2_CID_MPEG_BASE+1005)
>  #define V4L2_CID_MPEG_VIDEO_H264_START_CODE	(V4L2_CID_MPEG_BASE+1006)
> +#define V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS	(V4L2_CID_MPEG_BASE+1007)
>  
>  /* enum v4l2_ctrl_type type values */
>  #define V4L2_CTRL_TYPE_H264_SPS			0x0110
> @@ -43,6 +44,7 @@
>  #define V4L2_CTRL_TYPE_H264_SCALING_MATRIX	0x0112
>  #define V4L2_CTRL_TYPE_H264_SLICE_PARAMS	0x0113
>  #define V4L2_CTRL_TYPE_H264_DECODE_PARAMS	0x0114
> +#define V4L2_CTRL_TYPE_H264_PRED_WEIGHTS	0x0115
>  
>  enum v4l2_mpeg_video_h264_decode_mode {
>  	V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED,
> @@ -125,7 +127,7 @@ struct v4l2_h264_weight_factors {
>  	__s16 chroma_offset[32][2];
>  };
>  
> -struct v4l2_h264_pred_weight_table {
> +struct v4l2_ctrl_h264_pred_weights {
>  	__u16 luma_log2_weight_denom;
>  	__u16 chroma_log2_weight_denom;
>  	struct v4l2_h264_weight_factors weight_factors[2];
> @@ -177,7 +179,6 @@ struct v4l2_ctrl_h264_slice_params {
>  	__s32 delta_pic_order_cnt0;
>  	__s32 delta_pic_order_cnt1;
>  
> -	struct v4l2_h264_pred_weight_table pred_weight_table;
>  	/* Size in bits of dec_ref_pic_marking() syntax element. */
>  	__u32 dec_ref_pic_marking_bit_size;
>  	/* Size in bits of pic order count syntax. */
> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> index f40e2cbb21d3..cb25f345e9ad 100644
> --- a/include/media/v4l2-ctrls.h
> +++ b/include/media/v4l2-ctrls.h
> @@ -51,6 +51,7 @@ struct video_device;
>   * @p_h264_scaling_matrix:	Pointer to a struct v4l2_ctrl_h264_scaling_matrix.
>   * @p_h264_slice_params:	Pointer to a struct v4l2_ctrl_h264_slice_params.
>   * @p_h264_decode_params:	Pointer to a struct v4l2_ctrl_h264_decode_params.
> + * @p_h264_pred_weights:	Pointer to a struct v4l2_ctrl_h264_pred_weights.
>   * @p_vp8_frame_header:		Pointer to a VP8 frame header structure.
>   * @p_hevc_sps:			Pointer to an HEVC sequence parameter set structure.
>   * @p_hevc_pps:			Pointer to an HEVC picture parameter set structure.
> @@ -74,6 +75,7 @@ union v4l2_ctrl_ptr {
>  	struct v4l2_ctrl_h264_scaling_matrix *p_h264_scaling_matrix;
>  	struct v4l2_ctrl_h264_slice_params *p_h264_slice_params;
>  	struct v4l2_ctrl_h264_decode_params *p_h264_decode_params;
> +	struct v4l2_ctrl_h264_pred_weights *p_h264_pred_weights;
>  	struct v4l2_ctrl_vp8_frame_header *p_vp8_frame_header;
>  	struct v4l2_ctrl_hevc_sps *p_hevc_sps;
>  	struct v4l2_ctrl_hevc_pps *p_hevc_pps;
> 

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
>
> On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > The prediction weight parameters are only required under
> > certain conditions, which depend on slice header parameters.
> >
> > As specified in section 7.3.3 Slice header syntax, the prediction
> > weight table is present if:
> >
> > ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> > (weighted_bipred_idc == 1 && slice_type == B))
>
> Maybe a macro can be added to help check this contition?
>
> Something like this maybe:
>
> #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
>         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
>          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
>            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
>          ((pps)->weighted_bipred_idc == 1 && \
>           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
>

Yeah, that could make sense.

Note that the biggest value in having the prediction weight table
separated is to allow  applications to skip setting this largish control,
reducing the amount of data that needs to be passed from userspace
-- especially when not needed :-)

> >
> > Given its size, it makes sense to move this table to its control,
> > so applications can avoid passing it if the slice doesn't specify it.
> >
> > Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> > With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
> > is 772 bytes.
> >
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> > v2: Fix missing Cedrus changes and mssing control declaration,
> >     as noted by Hans and Jernej.
> > ---
> >  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
> >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> >  include/media/h264-ctrls.h                    |  5 +++--
> >  include/media/v4l2-ctrls.h                    |  2 ++
> >  8 files changed, 37 insertions(+), 13 deletions(-)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > index d1438b1e259f..c36ce5a95fc5 100644
> > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > @@ -1879,18 +1879,23 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
> >        - 0x00000008
> >        -
> >
> > -``Prediction Weight Table``
> > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > +    Prediction weight table defined according to :ref:`h264`,
> > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > +    The prediction weight table must be passed by applications
> > +    under the conditions explained in section 7.3.3 "Slice header
> > +    syntax".
> >
> > -    The bitstream parameters are defined according to :ref:`h264`,
> > -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
> > -    documentation, refer to the above specification, unless there is
> > -    an explicit comment stating otherwise.
> > +    .. note::
> > +
> > +       This compound control is not yet part of the public kernel API and
> > +       it is expected to change.
> >
> > -.. c:type:: v4l2_h264_pred_weight_table
> > +.. c:type:: v4l2_ctrl_h264_pred_weights
> >
> >  .. cssclass:: longtable
> >
> > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> >      :header-rows:  0
> >      :stub-columns: 0
> >      :widths:       1 1 2
> > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
> > index 3f3fbcd60cc6..76c8dc8fb31c 100644
> > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return "H264 Decode Parameters";
> >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return "H264 Decode Mode";
> >       case V4L2_CID_MPEG_VIDEO_H264_START_CODE:               return "H264 Start Code";
> > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return "H264 Prediction Weight Table";
> >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return "MPEG2 Level";
> >       case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:                 return "MPEG2 Profile";
> >       case V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return "MPEG4 I-Frame QP Value";
> > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
> >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> >               break;
> > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > +             break;
> >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> >               break;
> > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
> >       case V4L2_CTRL_TYPE_H264_SPS:
> >       case V4L2_CTRL_TYPE_H264_PPS:
> >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> >               break;
> > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
> >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> >               elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
> >               break;
> > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > +             elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
> > +             break;
> >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> >               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
> >               break;
> > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > index bc27f9430eeb..027cdd1be5a0 100644
> > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[] = {
> >               .codec          = CEDRUS_CODEC_H264,
> >               .required       = true,
> >       },
> > +     {
> > +             .cfg = {
> > +                     .id     = V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > +             },
> > +             .codec          = CEDRUS_CODEC_H264,
> > +             .required       = true,
>
> This should probably be false if this control is to be optional as implied
> by the commit message.
>

Well, the control is optional if the driver implements it as optional,
which Cedrus isn't currently doing :-)

I have been reluctant to change Cedrus much, since I am not
testing there. Perhaps we can address this with a follow-up patch,
adding the macro you suggest and changing the control to optional?

Speaking of optional controls, note that v4l2 controls are cached
per-fd (per context). This means for instance, that the active PPS
and SPS need only be passed to drivers when activated,
and not on each frame.

Similarly, another optimization to reduce the amount of
data passed is to have default scaling matrices in the kernel,
and allow applications to not pass them, and have the drivers
fallback to default (default or previously set?) in that case.

I have tested these things, but haven't posted patches because
I'm not entirely sure how well-defined and specified this behavior
currently is. It is something to keep in mind though, as a future
optimization.

Thanks!
Ezequiel



> Best regards,
> Jonas
>
> > +     },
> >       {
> >               .cfg = {
> >                       .id     = V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE,
> > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.h b/drivers/staging/media/sunxi/cedrus/cedrus.h
> > index 96765555ab8a..93c843ae14bb 100644
> > --- a/drivers/staging/media/sunxi/cedrus/cedrus.h
> > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.h
> > @@ -62,6 +62,7 @@ struct cedrus_h264_run {
> >       const struct v4l2_ctrl_h264_scaling_matrix      *scaling_matrix;
> >       const struct v4l2_ctrl_h264_slice_params        *slice_params;
> >       const struct v4l2_ctrl_h264_sps                 *sps;
> > +     const struct v4l2_ctrl_h264_pred_weights        *pred_weights;
> >  };
> >
> >  struct cedrus_mpeg2_run {
> > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> > index 58c48e4fdfe9..6385026d1b6b 100644
> > --- a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> > @@ -57,6 +57,8 @@ void cedrus_device_run(void *priv)
> >                       V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS);
> >               run.h264.sps = cedrus_find_control_data(ctx,
> >                       V4L2_CID_MPEG_VIDEO_H264_SPS);
> > +             run.h264.pred_weights = cedrus_find_control_data(ctx,
> > +                     V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS);
> >               break;
> >
> >       case V4L2_PIX_FMT_HEVC_SLICE:
> > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > index cce527bbdf86..a9ba78b15907 100644
> > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > @@ -256,10 +256,8 @@ static void cedrus_write_scaling_lists(struct cedrus_ctx *ctx,
> >  static void cedrus_write_pred_weight_table(struct cedrus_ctx *ctx,
> >                                          struct cedrus_run *run)
> >  {
> > -     const struct v4l2_ctrl_h264_slice_params *slice =
> > -             run->h264.slice_params;
> > -     const struct v4l2_h264_pred_weight_table *pred_weight =
> > -             &slice->pred_weight_table;
> > +     const struct v4l2_ctrl_h264_pred_weights *pred_weight =
> > +             run->h264.pred_weights;
> >       struct cedrus_dev *dev = ctx->dev;
> >       int i, j, k;
> >
> > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > index 4c0bb7f5fb05..54cd9bec0b23 100644
> > --- a/include/media/h264-ctrls.h
> > +++ b/include/media/h264-ctrls.h
> > @@ -36,6 +36,7 @@
> >  #define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS       (V4L2_CID_MPEG_BASE+1004)
> >  #define V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE (V4L2_CID_MPEG_BASE+1005)
> >  #define V4L2_CID_MPEG_VIDEO_H264_START_CODE  (V4L2_CID_MPEG_BASE+1006)
> > +#define V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS        (V4L2_CID_MPEG_BASE+1007)
> >
> >  /* enum v4l2_ctrl_type type values */
> >  #define V4L2_CTRL_TYPE_H264_SPS                      0x0110
> > @@ -43,6 +44,7 @@
> >  #define V4L2_CTRL_TYPE_H264_SCALING_MATRIX   0x0112
> >  #define V4L2_CTRL_TYPE_H264_SLICE_PARAMS     0x0113
> >  #define V4L2_CTRL_TYPE_H264_DECODE_PARAMS    0x0114
> > +#define V4L2_CTRL_TYPE_H264_PRED_WEIGHTS     0x0115
> >
> >  enum v4l2_mpeg_video_h264_decode_mode {
> >       V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED,
> > @@ -125,7 +127,7 @@ struct v4l2_h264_weight_factors {
> >       __s16 chroma_offset[32][2];
> >  };
> >
> > -struct v4l2_h264_pred_weight_table {
> > +struct v4l2_ctrl_h264_pred_weights {
> >       __u16 luma_log2_weight_denom;
> >       __u16 chroma_log2_weight_denom;
> >       struct v4l2_h264_weight_factors weight_factors[2];
> > @@ -177,7 +179,6 @@ struct v4l2_ctrl_h264_slice_params {
> >       __s32 delta_pic_order_cnt0;
> >       __s32 delta_pic_order_cnt1;
> >
> > -     struct v4l2_h264_pred_weight_table pred_weight_table;
> >       /* Size in bits of dec_ref_pic_marking() syntax element. */
> >       __u32 dec_ref_pic_marking_bit_size;
> >       /* Size in bits of pic order count syntax. */
> > diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> > index f40e2cbb21d3..cb25f345e9ad 100644
> > --- a/include/media/v4l2-ctrls.h
> > +++ b/include/media/v4l2-ctrls.h
> > @@ -51,6 +51,7 @@ struct video_device;
> >   * @p_h264_scaling_matrix:   Pointer to a struct v4l2_ctrl_h264_scaling_matrix.
> >   * @p_h264_slice_params:     Pointer to a struct v4l2_ctrl_h264_slice_params.
> >   * @p_h264_decode_params:    Pointer to a struct v4l2_ctrl_h264_decode_params.
> > + * @p_h264_pred_weights:     Pointer to a struct v4l2_ctrl_h264_pred_weights.
> >   * @p_vp8_frame_header:              Pointer to a VP8 frame header structure.
> >   * @p_hevc_sps:                      Pointer to an HEVC sequence parameter set structure.
> >   * @p_hevc_pps:                      Pointer to an HEVC picture parameter set structure.
> > @@ -74,6 +75,7 @@ union v4l2_ctrl_ptr {
> >       struct v4l2_ctrl_h264_scaling_matrix *p_h264_scaling_matrix;
> >       struct v4l2_ctrl_h264_slice_params *p_h264_slice_params;
> >       struct v4l2_ctrl_h264_decode_params *p_h264_decode_params;
> > +     struct v4l2_ctrl_h264_pred_weights *p_h264_pred_weights;
> >       struct v4l2_ctrl_vp8_frame_header *p_vp8_frame_header;
> >       struct v4l2_ctrl_hevc_sps *p_hevc_sps;
> >       struct v4l2_ctrl_hevc_pps *p_hevc_pps;
> >

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Jernej Škrabec]

Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia napisal(a):
> On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > The prediction weight parameters are only required under
> > > certain conditions, which depend on slice header parameters.
> > > 
> > > As specified in section 7.3.3 Slice header syntax, the prediction
> > > weight table is present if:
> > > 
> > > ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> > > (weighted_bipred_idc == 1 && slice_type == B))
> > 
> > Maybe a macro can be added to help check this contition?
> > 
> > Something like this maybe:
> > 
> > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > 
> >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> >         
> >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> >          
> >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> >          
> >          ((pps)->weighted_bipred_idc == 1 && \
> >          
> >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> 
> Yeah, that could make sense.
> 
> Note that the biggest value in having the prediction weight table
> separated is to allow  applications to skip setting this largish control,
> reducing the amount of data that needs to be passed from userspace
> -- especially when not needed :-)
> 
> > > Given its size, it makes sense to move this table to its control,
> > > so applications can avoid passing it if the slice doesn't specify it.
> > > 
> > > Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> > > With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
> > > is 772 bytes.
> > > 
> > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > ---
> > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > 
> > >     as noted by Hans and Jernej.
> > > 
> > > ---
> > > 
> > >  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
> > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > >  include/media/h264-ctrls.h                    |  5 +++--
> > >  include/media/v4l2-ctrls.h                    |  2 ++
> > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
> > > d1438b1e259f..c36ce5a95fc5 100644
> > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > @@ -1879,18 +1879,23 @@ enum
> > > v4l2_mpeg_video_h264_hierarchical_coding_type -> > 
> > >        - 0x00000008
> > >        -
> > > 
> > > -``Prediction Weight Table``
> > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > +    Prediction weight table defined according to :ref:`h264`,
> > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > +    The prediction weight table must be passed by applications
> > > +    under the conditions explained in section 7.3.3 "Slice header
> > > +    syntax".
> > > 
> > > -    The bitstream parameters are defined according to :ref:`h264`,
> > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
> > > -    documentation, refer to the above specification, unless there is
> > > -    an explicit comment stating otherwise.
> > > +    .. note::
> > > +
> > > +       This compound control is not yet part of the public kernel API
> > > and
> > > +       it is expected to change.
> > > 
> > > -.. c:type:: v4l2_h264_pred_weight_table
> > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > 
> > >  .. cssclass:: longtable
> > > 
> > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > 
> > >      :header-rows:  0
> > >      :stub-columns: 0
> > >      :widths:       1 1 2
> > > 
> > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > b/drivers/media/v4l2-core/v4l2-ctrls.c index 3f3fbcd60cc6..76c8dc8fb31c
> > > 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > 
> > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
> > >       "H264 Decode Parameters"; case
> > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return "H264
> > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:          
> > >           return "H264 Start Code";> > 
> > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
> > > "H264 Prediction Weight Table";> > 
> > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
> > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:           
> > >            return "MPEG2 Profile"; case
> > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return "MPEG4
> > >       I-Frame QP Value";> > 
> > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
> > > enum v4l2_ctrl_type *type,> > 
> > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > >               break;
> > > 
> > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > +             break;
> > > 
> > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > >               break;
> > > 
> > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
> > > v4l2_ctrl *ctrl, u32 idx,> > 
> > >       case V4L2_CTRL_TYPE_H264_SPS:
> > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > 
> > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > >       
> > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > >               break;
> > > 
> > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
> > > v4l2_ctrl_handler *hdl,> > 
> > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > >               elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
> > >               break;
> > > 
> > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > +             elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
> > > +             break;
> > > 
> > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > >               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
> > >               break;
> > > 
> > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > bc27f9430eeb..027cdd1be5a0 100644
> > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[]
> > > = {
> > > 
> > >               .codec          = CEDRUS_CODEC_H264,
> > >               .required       = true,
> > >       
> > >       },
> > > 
> > > +     {
> > > +             .cfg = {
> > > +                     .id     = V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > +             },
> > > +             .codec          = CEDRUS_CODEC_H264,
> > > +             .required       = true,
> > 
> > This should probably be false if this control is to be optional as implied
> > by the commit message.
> 
> Well, the control is optional if the driver implements it as optional,
> which Cedrus isn't currently doing :-)

Why do you think so? Prediction weights are filled only when they are 
needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/
sunxi/cedrus/cedrus_h264.c#L370

Best regards,
Jernej

> 
> I have been reluctant to change Cedrus much, since I am not
> testing there. Perhaps we can address this with a follow-up patch,
> adding the macro you suggest and changing the control to optional?
> 
> Speaking of optional controls, note that v4l2 controls are cached
> per-fd (per context). This means for instance, that the active PPS
> and SPS need only be passed to drivers when activated,
> and not on each frame.
> 
> Similarly, another optimization to reduce the amount of
> data passed is to have default scaling matrices in the kernel,
> and allow applications to not pass them, and have the drivers
> fallback to default (default or previously set?) in that case.
> 
> I have tested these things, but haven't posted patches because
> I'm not entirely sure how well-defined and specified this behavior
> currently is. It is something to keep in mind though, as a future
> optimization.
> 
> Thanks!
> Ezequiel
> 
> > Best regards,
> > Jonas
> > 
> > > +     },
> > > 
> > >       {
> > >       
> > >               .cfg = {
> > >               
> > >                       .id     = V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE,
> > > 
> > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.h
> > > b/drivers/staging/media/sunxi/cedrus/cedrus.h index
> > > 96765555ab8a..93c843ae14bb 100644
> > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.h
> > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.h
> > > @@ -62,6 +62,7 @@ struct cedrus_h264_run {
> > > 
> > >       const struct v4l2_ctrl_h264_scaling_matrix      *scaling_matrix;
> > >       const struct v4l2_ctrl_h264_slice_params        *slice_params;
> > >       const struct v4l2_ctrl_h264_sps                 *sps;
> > > 
> > > +     const struct v4l2_ctrl_h264_pred_weights        *pred_weights;
> > > 
> > >  };
> > >  
> > >  struct cedrus_mpeg2_run {
> > > 
> > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> > > b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c index
> > > 58c48e4fdfe9..6385026d1b6b 100644
> > > --- a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c
> > > @@ -57,6 +57,8 @@ void cedrus_device_run(void *priv)
> > > 
> > >                       V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS);
> > >               
> > >               run.h264.sps = cedrus_find_control_data(ctx,
> > >               
> > >                       V4L2_CID_MPEG_VIDEO_H264_SPS);
> > > 
> > > +             run.h264.pred_weights = cedrus_find_control_data(ctx,
> > > +                     V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS);
> > > 
> > >               break;
> > >       
> > >       case V4L2_PIX_FMT_HEVC_SLICE:
> > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c index
> > > cce527bbdf86..a9ba78b15907 100644
> > > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > @@ -256,10 +256,8 @@ static void cedrus_write_scaling_lists(struct
> > > cedrus_ctx *ctx,> > 
> > >  static void cedrus_write_pred_weight_table(struct cedrus_ctx *ctx,
> > >  
> > >                                          struct cedrus_run *run)
> > >  
> > >  {
> > > 
> > > -     const struct v4l2_ctrl_h264_slice_params *slice =
> > > -             run->h264.slice_params;
> > > -     const struct v4l2_h264_pred_weight_table *pred_weight =
> > > -             &slice->pred_weight_table;
> > > +     const struct v4l2_ctrl_h264_pred_weights *pred_weight =
> > > +             run->h264.pred_weights;
> > > 
> > >       struct cedrus_dev *dev = ctx->dev;
> > >       int i, j, k;
> > > 
> > > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > > index 4c0bb7f5fb05..54cd9bec0b23 100644
> > > --- a/include/media/h264-ctrls.h
> > > +++ b/include/media/h264-ctrls.h
> > > @@ -36,6 +36,7 @@
> > > 
> > >  #define V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS      
> > >  (V4L2_CID_MPEG_BASE+1004) #define V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE
> > >  (V4L2_CID_MPEG_BASE+1005) #define V4L2_CID_MPEG_VIDEO_H264_START_CODE 
> > >  (V4L2_CID_MPEG_BASE+1006)> > 
> > > +#define V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS       
> > > (V4L2_CID_MPEG_BASE+1007)> > 
> > >  /* enum v4l2_ctrl_type type values */
> > >  #define V4L2_CTRL_TYPE_H264_SPS                      0x0110
> > > 
> > > @@ -43,6 +44,7 @@
> > > 
> > >  #define V4L2_CTRL_TYPE_H264_SCALING_MATRIX   0x0112
> > >  #define V4L2_CTRL_TYPE_H264_SLICE_PARAMS     0x0113
> > >  #define V4L2_CTRL_TYPE_H264_DECODE_PARAMS    0x0114
> > > 
> > > +#define V4L2_CTRL_TYPE_H264_PRED_WEIGHTS     0x0115
> > > 
> > >  enum v4l2_mpeg_video_h264_decode_mode {
> > >  
> > >       V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED,
> > > 
> > > @@ -125,7 +127,7 @@ struct v4l2_h264_weight_factors {
> > > 
> > >       __s16 chroma_offset[32][2];
> > >  
> > >  };
> > > 
> > > -struct v4l2_h264_pred_weight_table {
> > > +struct v4l2_ctrl_h264_pred_weights {
> > > 
> > >       __u16 luma_log2_weight_denom;
> > >       __u16 chroma_log2_weight_denom;
> > >       struct v4l2_h264_weight_factors weight_factors[2];
> > > 
> > > @@ -177,7 +179,6 @@ struct v4l2_ctrl_h264_slice_params {
> > > 
> > >       __s32 delta_pic_order_cnt0;
> > >       __s32 delta_pic_order_cnt1;
> > > 
> > > -     struct v4l2_h264_pred_weight_table pred_weight_table;
> > > 
> > >       /* Size in bits of dec_ref_pic_marking() syntax element. */
> > >       __u32 dec_ref_pic_marking_bit_size;
> > >       /* Size in bits of pic order count syntax. */
> > > 
> > > diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> > > index f40e2cbb21d3..cb25f345e9ad 100644
> > > --- a/include/media/v4l2-ctrls.h
> > > +++ b/include/media/v4l2-ctrls.h
> > > @@ -51,6 +51,7 @@ struct video_device;
> > > 
> > >   * @p_h264_scaling_matrix:   Pointer to a struct
> > >   v4l2_ctrl_h264_scaling_matrix. * @p_h264_slice_params:     Pointer to
> > >   a struct v4l2_ctrl_h264_slice_params. * @p_h264_decode_params:   
> > >   Pointer to a struct v4l2_ctrl_h264_decode_params.> > 
> > > + * @p_h264_pred_weights:     Pointer to a struct
> > > v4l2_ctrl_h264_pred_weights.> > 
> > >   * @p_vp8_frame_header:              Pointer to a VP8 frame header
> > >   structure. * @p_hevc_sps:                      Pointer to an HEVC
> > >   sequence parameter set structure. * @p_hevc_pps:                     
> > >   Pointer to an HEVC picture parameter set structure.> > 
> > > @@ -74,6 +75,7 @@ union v4l2_ctrl_ptr {
> > > 
> > >       struct v4l2_ctrl_h264_scaling_matrix *p_h264_scaling_matrix;
> > >       struct v4l2_ctrl_h264_slice_params *p_h264_slice_params;
> > >       struct v4l2_ctrl_h264_decode_params *p_h264_decode_params;
> > > 
> > > +     struct v4l2_ctrl_h264_pred_weights *p_h264_pred_weights;
> > > 
> > >       struct v4l2_ctrl_vp8_frame_header *p_vp8_frame_header;
> > >       struct v4l2_ctrl_hevc_sps *p_hevc_sps;
> > >       struct v4l2_ctrl_hevc_pps *p_hevc_pps;

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
> Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia napisal(a):
> > On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > > The prediction weight parameters are only required under
> > > > certain conditions, which depend on slice header parameters.
> > > > 
> > > > As specified in section 7.3.3 Slice header syntax, the prediction
> > > > weight table is present if:
> > > > 
> > > > ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> > > > (weighted_bipred_idc == 1 && slice_type == B))
> > > 
> > > Maybe a macro can be added to help check this contition?
> > > 
> > > Something like this maybe:
> > > 
> > > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > > 
> > >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> > >         
> > >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> > >          
> > >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> > >          
> > >          ((pps)->weighted_bipred_idc == 1 && \
> > >          
> > >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> > 
> > Yeah, that could make sense.
> > 
> > Note that the biggest value in having the prediction weight table
> > separated is to allow  applications to skip setting this largish control,
> > reducing the amount of data that needs to be passed from userspace
> > -- especially when not needed :-)
> > 
> > > > Given its size, it makes sense to move this table to its control,
> > > > so applications can avoid passing it if the slice doesn't specify it.
> > > > 
> > > > Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> > > > With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
> > > > is 772 bytes.
> > > > 
> > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > ---
> > > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > > 
> > > >     as noted by Hans and Jernej.
> > > > 
> > > > ---
> > > > 
> > > >  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
> > > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > > >  include/media/h264-ctrls.h                    |  5 +++--
> > > >  include/media/v4l2-ctrls.h                    |  2 ++
> > > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
> > > > d1438b1e259f..c36ce5a95fc5 100644
> > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > @@ -1879,18 +1879,23 @@ enum
> > > > v4l2_mpeg_video_h264_hierarchical_coding_type -> > 
> > > >        - 0x00000008
> > > >        -
> > > > 
> > > > -``Prediction Weight Table``
> > > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > > +    Prediction weight table defined according to :ref:`h264`,
> > > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > > +    The prediction weight table must be passed by applications
> > > > +    under the conditions explained in section 7.3.3 "Slice header
> > > > +    syntax".
> > > > 
> > > > -    The bitstream parameters are defined according to :ref:`h264`,
> > > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
> > > > -    documentation, refer to the above specification, unless there is
> > > > -    an explicit comment stating otherwise.
> > > > +    .. note::
> > > > +
> > > > +       This compound control is not yet part of the public kernel API
> > > > and
> > > > +       it is expected to change.
> > > > 
> > > > -.. c:type:: v4l2_h264_pred_weight_table
> > > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > > 
> > > >  .. cssclass:: longtable
> > > > 
> > > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > > 
> > > >      :header-rows:  0
> > > >      :stub-columns: 0
> > > >      :widths:       1 1 2
> > > > 
> > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > b/drivers/media/v4l2-core/v4l2-ctrls.c index 3f3fbcd60cc6..76c8dc8fb31c
> > > > 100644
> > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > > 
> > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
> > > >       "H264 Decode Parameters"; case
> > > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return "H264
> > > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:          
> > > >           return "H264 Start Code";> > 
> > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
> > > > "H264 Prediction Weight Table";> > 
> > > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
> > > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:           
> > > >            return "MPEG2 Profile"; case
> > > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return "MPEG4
> > > >       I-Frame QP Value";> > 
> > > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
> > > > enum v4l2_ctrl_type *type,> > 
> > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > > >               break;
> > > > 
> > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > > +             break;
> > > > 
> > > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > > >               break;
> > > > 
> > > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
> > > > v4l2_ctrl *ctrl, u32 idx,> > 
> > > >       case V4L2_CTRL_TYPE_H264_SPS:
> > > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > > 
> > > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > > >       
> > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > >               break;
> > > > 
> > > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
> > > > v4l2_ctrl_handler *hdl,> > 
> > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > >               elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
> > > >               break;
> > > > 
> > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > +             elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
> > > > +             break;
> > > > 
> > > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > > >               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
> > > >               break;
> > > > 
> > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > > bc27f9430eeb..027cdd1be5a0 100644
> > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[]
> > > > = {
> > > > 
> > > >               .codec          = CEDRUS_CODEC_H264,
> > > >               .required       = true,
> > > >       
> > > >       },
> > > > 
> > > > +     {
> > > > +             .cfg = {
> > > > +                     .id     = V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > +             },
> > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > +             .required       = true,
> > > 
> > > This should probably be false if this control is to be optional as implied
> > > by the commit message.
> > 
> > Well, the control is optional if the driver implements it as optional,
> > which Cedrus isn't currently doing :-)
> 
> Why do you think so? Prediction weights are filled only when they are 
> needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/
> sunxi/cedrus/cedrus_h264.c#L370
> 

Right, but that should be changed to be really optional.
How does the driver reject/fail the request if the table is NULL?

In any case, I don't think it's necessarily something we need
to tackle now.

Thanks,
Ezequiel

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Jonas Karlman]

On 2020-08-10 14:57, Ezequiel Garcia wrote:
> On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
>> Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia napisal(a):
>>> On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
>>>> On 2020-08-06 17:12, Ezequiel Garcia wrote:
>>>>> The prediction weight parameters are only required under
>>>>> certain conditions, which depend on slice header parameters.
>>>>>
>>>>> As specified in section 7.3.3 Slice header syntax, the prediction
>>>>> weight table is present if:
>>>>>
>>>>> ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
>>>>> (weighted_bipred_idc == 1 && slice_type == B))
>>>>
>>>> Maybe a macro can be added to help check this contition?
>>>>
>>>> Something like this maybe:
>>>>
>>>> #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
>>>>
>>>>         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
>>>>         
>>>>          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
>>>>          
>>>>            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
>>>>          
>>>>          ((pps)->weighted_bipred_idc == 1 && \
>>>>          
>>>>           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
>>>
>>> Yeah, that could make sense.
>>>
>>> Note that the biggest value in having the prediction weight table
>>> separated is to allow  applications to skip setting this largish control,
>>> reducing the amount of data that needs to be passed from userspace
>>> -- especially when not needed :-)
>>>
>>>>> Given its size, it makes sense to move this table to its control,
>>>>> so applications can avoid passing it if the slice doesn't specify it.
>>>>>
>>>>> Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
>>>>> With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
>>>>> is 772 bytes.
>>>>>
>>>>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>>>>> ---
>>>>> v2: Fix missing Cedrus changes and mssing control declaration,
>>>>>
>>>>>     as noted by Hans and Jernej.
>>>>>
>>>>> ---
>>>>>
>>>>>  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
>>>>>  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
>>>>>  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
>>>>>  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
>>>>>  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
>>>>>  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
>>>>>  include/media/h264-ctrls.h                    |  5 +++--
>>>>>  include/media/v4l2-ctrls.h                    |  2 ++
>>>>>  8 files changed, 37 insertions(+), 13 deletions(-)
>>>>>
>>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>> b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
>>>>> d1438b1e259f..c36ce5a95fc5 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>> @@ -1879,18 +1879,23 @@ enum
>>>>> v4l2_mpeg_video_h264_hierarchical_coding_type -> > 
>>>>>        - 0x00000008
>>>>>        -
>>>>>
>>>>> -``Prediction Weight Table``
>>>>> +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
>>>>> +    Prediction weight table defined according to :ref:`h264`,
>>>>> +    section 7.4.3.2 "Prediction Weight Table Semantics".
>>>>> +    The prediction weight table must be passed by applications
>>>>> +    under the conditions explained in section 7.3.3 "Slice header
>>>>> +    syntax".
>>>>>
>>>>> -    The bitstream parameters are defined according to :ref:`h264`,
>>>>> -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
>>>>> -    documentation, refer to the above specification, unless there is
>>>>> -    an explicit comment stating otherwise.
>>>>> +    .. note::
>>>>> +
>>>>> +       This compound control is not yet part of the public kernel API
>>>>> and
>>>>> +       it is expected to change.
>>>>>
>>>>> -.. c:type:: v4l2_h264_pred_weight_table
>>>>> +.. c:type:: v4l2_ctrl_h264_pred_weights
>>>>>
>>>>>  .. cssclass:: longtable
>>>>>
>>>>> -.. flat-table:: struct v4l2_h264_pred_weight_table
>>>>> +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
>>>>>
>>>>>      :header-rows:  0
>>>>>      :stub-columns: 0
>>>>>      :widths:       1 1 2
>>>>>
>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
>>>>> b/drivers/media/v4l2-core/v4l2-ctrls.c index 3f3fbcd60cc6..76c8dc8fb31c
>>>>> 100644
>>>>> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
>>>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
>>>>> @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
>>>>>
>>>>>       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
>>>>>       "H264 Decode Parameters"; case
>>>>>       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return "H264
>>>>>       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:          
>>>>>           return "H264 Start Code";> > 
>>>>> +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
>>>>> "H264 Prediction Weight Table";> > 
>>>>>       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
>>>>>       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:           
>>>>>            return "MPEG2 Profile"; case
>>>>>       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return "MPEG4
>>>>>       I-Frame QP Value";> > 
>>>>> @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
>>>>> enum v4l2_ctrl_type *type,> > 
>>>>>       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
>>>>>               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
>>>>>               break;
>>>>>
>>>>> +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
>>>>> +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
>>>>> +             break;
>>>>>
>>>>>       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
>>>>>               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
>>>>>               break;
>>>>>
>>>>> @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
>>>>> v4l2_ctrl *ctrl, u32 idx,> > 
>>>>>       case V4L2_CTRL_TYPE_H264_SPS:
>>>>>       case V4L2_CTRL_TYPE_H264_PPS:
>>>>>
>>>>>       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
>>>>> +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
>>>>>       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
>>>>>       
>>>>>       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
>>>>>               break;
>>>>>
>>>>> @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
>>>>> v4l2_ctrl_handler *hdl,> > 
>>>>>       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
>>>>>               elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
>>>>>               break;
>>>>>
>>>>> +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
>>>>> +             elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
>>>>> +             break;
>>>>>
>>>>>       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
>>>>>               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
>>>>>               break;
>>>>>
>>>>> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
>>>>> b/drivers/staging/media/sunxi/cedrus/cedrus.c index
>>>>> bc27f9430eeb..027cdd1be5a0 100644
>>>>> --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
>>>>> +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
>>>>> @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[]
>>>>> = {
>>>>>
>>>>>               .codec          = CEDRUS_CODEC_H264,
>>>>>               .required       = true,
>>>>>       
>>>>>       },
>>>>>
>>>>> +     {
>>>>> +             .cfg = {
>>>>> +                     .id     = V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
>>>>> +             },
>>>>> +             .codec          = CEDRUS_CODEC_H264,
>>>>> +             .required       = true,
>>>>
>>>> This should probably be false if this control is to be optional as implied
>>>> by the commit message.
>>>
>>> Well, the control is optional if the driver implements it as optional,
>>> which Cedrus isn't currently doing :-)
>>
>> Why do you think so? Prediction weights are filled only when they are 
>> needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/
>> sunxi/cedrus/cedrus_h264.c#L370
>>
> 
> Right, but that should be changed to be really optional.
> How does the driver reject/fail the request if the table is NULL?
> 
> In any case, I don't think it's necessarily something we need
> to tackle now.

I do not fully follow, the commit message state following:

  Note that the biggest value in having the prediction weight table
  separated is to allow applications to skip setting this largish control

Yet the driver still require this new control to be included in the request
thanks to the .required = true statement. (if i understand the code correctly)

So applications still need to set this largish control?

Best regards,
Jonas

> 
> Thanks,
> Ezequiel
> 

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

On Mon, 2020-08-10 at 14:55 +0000, Jonas Karlman wrote:
> On 2020-08-10 14:57, Ezequiel Garcia wrote:
> > On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
> > > Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia napisal(a):
> > > > On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > > > > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > > > > The prediction weight parameters are only required under
> > > > > > certain conditions, which depend on slice header parameters.
> > > > > > 
> > > > > > As specified in section 7.3.3 Slice header syntax, the prediction
> > > > > > weight table is present if:
> > > > > > 
> > > > > > ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> > > > > > (weighted_bipred_idc == 1 && slice_type == B))
> > > > > 
> > > > > Maybe a macro can be added to help check this contition?
> > > > > 
> > > > > Something like this maybe:
> > > > > 
> > > > > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > > > > 
> > > > >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> > > > >         
> > > > >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> > > > >          
> > > > >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> > > > >          
> > > > >          ((pps)->weighted_bipred_idc == 1 && \
> > > > >          
> > > > >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> > > > 
> > > > Yeah, that could make sense.
> > > > 
> > > > Note that the biggest value in having the prediction weight table
> > > > separated is to allow  applications to skip setting this largish control,
> > > > reducing the amount of data that needs to be passed from userspace
> > > > -- especially when not needed :-)
> > > > 
> > > > > > Given its size, it makes sense to move this table to its control,
> > > > > > so applications can avoid passing it if the slice doesn't specify it.
> > > > > > 
> > > > > > Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> > > > > > With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
> > > > > > is 772 bytes.
> > > > > > 
> > > > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > > ---
> > > > > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > > > > 
> > > > > >     as noted by Hans and Jernej.
> > > > > > 
> > > > > > ---
> > > > > > 
> > > > > >  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
> > > > > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > > > > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > > > > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > > > > >  include/media/h264-ctrls.h                    |  5 +++--
> > > > > >  include/media/v4l2-ctrls.h                    |  2 ++
> > > > > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > > > > 
> > > > > > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
> > > > > > d1438b1e259f..c36ce5a95fc5 100644
> > > > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > @@ -1879,18 +1879,23 @@ enum
> > > > > > v4l2_mpeg_video_h264_hierarchical_coding_type -> > 
> > > > > >        - 0x00000008
> > > > > >        -
> > > > > > 
> > > > > > -``Prediction Weight Table``
> > > > > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > > > > +    Prediction weight table defined according to :ref:`h264`,
> > > > > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > > > > +    The prediction weight table must be passed by applications
> > > > > > +    under the conditions explained in section 7.3.3 "Slice header
> > > > > > +    syntax".
> > > > > > 
> > > > > > -    The bitstream parameters are defined according to :ref:`h264`,
> > > > > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
> > > > > > -    documentation, refer to the above specification, unless there is
> > > > > > -    an explicit comment stating otherwise.
> > > > > > +    .. note::
> > > > > > +
> > > > > > +       This compound control is not yet part of the public kernel API
> > > > > > and
> > > > > > +       it is expected to change.
> > > > > > 
> > > > > > -.. c:type:: v4l2_h264_pred_weight_table
> > > > > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > > > > 
> > > > > >  .. cssclass:: longtable
> > > > > > 
> > > > > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > > > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > > > > 
> > > > > >      :header-rows:  0
> > > > > >      :stub-columns: 0
> > > > > >      :widths:       1 1 2
> > > > > > 
> > > > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > b/drivers/media/v4l2-core/v4l2-ctrls.c index 3f3fbcd60cc6..76c8dc8fb31c
> > > > > > 100644
> > > > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > > > > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
> > > > > >       "H264 Decode Parameters"; case
> > > > > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return "H264
> > > > > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:          
> > > > > >           return "H264 Start Code";> > 
> > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
> > > > > > "H264 Prediction Weight Table";> > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
> > > > > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:           
> > > > > >            return "MPEG2 Profile"; case
> > > > > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return "MPEG4
> > > > > >       I-Frame QP Value";> > 
> > > > > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
> > > > > > enum v4l2_ctrl_type *type,> > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > > > > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > > > > >               break;
> > > > > > 
> > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > > > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > > > > +             break;
> > > > > > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > > > > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > > > > >               break;
> > > > > > 
> > > > > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
> > > > > > v4l2_ctrl *ctrl, u32 idx,> > 
> > > > > >       case V4L2_CTRL_TYPE_H264_SPS:
> > > > > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > > > > 
> > > > > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > > > > >       
> > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > >               break;
> > > > > > 
> > > > > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
> > > > > > v4l2_ctrl_handler *hdl,> > 
> > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > >               elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
> > > > > >               break;
> > > > > > 
> > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > +             elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
> > > > > > +             break;
> > > > > > 
> > > > > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > > > > >               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
> > > > > >               break;
> > > > > > 
> > > > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > > > > bc27f9430eeb..027cdd1be5a0 100644
> > > > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[]
> > > > > > = {
> > > > > > 
> > > > > >               .codec          = CEDRUS_CODEC_H264,
> > > > > >               .required       = true,
> > > > > >       
> > > > > >       },
> > > > > > 
> > > > > > +     {
> > > > > > +             .cfg = {
> > > > > > +                     .id     = V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > > > +             },
> > > > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > > > +             .required       = true,
> > > > > 
> > > > > This should probably be false if this control is to be optional as implied
> > > > > by the commit message.
> > > > 
> > > > Well, the control is optional if the driver implements it as optional,
> > > > which Cedrus isn't currently doing :-)
> > > 
> > > Why do you think so? Prediction weights are filled only when they are 
> > > needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/
> > > sunxi/cedrus/cedrus_h264.c#L370
> > > 
> > 
> > Right, but that should be changed to be really optional.
> > How does the driver reject/fail the request if the table is NULL?
> > 
> > In any case, I don't think it's necessarily something we need
> > to tackle now.
> 
> I do not fully follow, the commit message state following:
> 
>   Note that the biggest value in having the prediction weight table
>   separated is to allow applications to skip setting this largish control
> 

This is not exactly what the commit message says, but yeah
that's the idea.

> Yet the driver still require this new control to be included in the request
> thanks to the .required = true statement. (if i understand the code correctly)
> 
> So applications still need to set this largish control?
> 

This is a uAPI change that paves the way for Cedrus to make the control
optional. The series doesn't take care of it, but it prepares the road
for it.

Since we are not stabilizing the uAPI (yet), I think we still have
some room to make this change in steps: first we merge the new control
and then we add the needed changes to Cedrus?

Does that make sense?

> Best regards,
> Jonas
> 
> > Thanks,
> > Ezequiel
> > 

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Jonas Karlman]

On 2020-08-10 17:28, Ezequiel Garcia wrote:
> On Mon, 2020-08-10 at 14:55 +0000, Jonas Karlman wrote:
>> On 2020-08-10 14:57, Ezequiel Garcia wrote:
>>> On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
>>>> Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia napisal(a):
>>>>> On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
>>>>>> On 2020-08-06 17:12, Ezequiel Garcia wrote:
>>>>>>> The prediction weight parameters are only required under
>>>>>>> certain conditions, which depend on slice header parameters.
>>>>>>>
>>>>>>> As specified in section 7.3.3 Slice header syntax, the prediction
>>>>>>> weight table is present if:
>>>>>>>
>>>>>>> ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
>>>>>>> (weighted_bipred_idc == 1 && slice_type == B))
>>>>>>
>>>>>> Maybe a macro can be added to help check this contition?
>>>>>>
>>>>>> Something like this maybe:
>>>>>>
>>>>>> #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
>>>>>>
>>>>>>         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
>>>>>>         
>>>>>>          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
>>>>>>          
>>>>>>            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
>>>>>>          
>>>>>>          ((pps)->weighted_bipred_idc == 1 && \
>>>>>>          
>>>>>>           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
>>>>>
>>>>> Yeah, that could make sense.
>>>>>
>>>>> Note that the biggest value in having the prediction weight table
>>>>> separated is to allow  applications to skip setting this largish control,
>>>>> reducing the amount of data that needs to be passed from userspace
>>>>> -- especially when not needed :-)
>>>>>
>>>>>>> Given its size, it makes sense to move this table to its control,
>>>>>>> so applications can avoid passing it if the slice doesn't specify it.
>>>>>>>
>>>>>>> Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
>>>>>>> With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
>>>>>>> is 772 bytes.
>>>>>>>
>>>>>>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>>>>>>> ---
>>>>>>> v2: Fix missing Cedrus changes and mssing control declaration,
>>>>>>>
>>>>>>>     as noted by Hans and Jernej.
>>>>>>>
>>>>>>> ---
>>>>>>>
>>>>>>>  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
>>>>>>>  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
>>>>>>>  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
>>>>>>>  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
>>>>>>>  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
>>>>>>>  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
>>>>>>>  include/media/h264-ctrls.h                    |  5 +++--
>>>>>>>  include/media/v4l2-ctrls.h                    |  2 ++
>>>>>>>  8 files changed, 37 insertions(+), 13 deletions(-)
>>>>>>>
>>>>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>>>> b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
>>>>>>> d1438b1e259f..c36ce5a95fc5 100644
>>>>>>> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>>>> @@ -1879,18 +1879,23 @@ enum
>>>>>>> v4l2_mpeg_video_h264_hierarchical_coding_type -> > 
>>>>>>>        - 0x00000008
>>>>>>>        -
>>>>>>>
>>>>>>> -``Prediction Weight Table``
>>>>>>> +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
>>>>>>> +    Prediction weight table defined according to :ref:`h264`,
>>>>>>> +    section 7.4.3.2 "Prediction Weight Table Semantics".
>>>>>>> +    The prediction weight table must be passed by applications
>>>>>>> +    under the conditions explained in section 7.3.3 "Slice header
>>>>>>> +    syntax".
>>>>>>>
>>>>>>> -    The bitstream parameters are defined according to :ref:`h264`,
>>>>>>> -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
>>>>>>> -    documentation, refer to the above specification, unless there is
>>>>>>> -    an explicit comment stating otherwise.
>>>>>>> +    .. note::
>>>>>>> +
>>>>>>> +       This compound control is not yet part of the public kernel API
>>>>>>> and
>>>>>>> +       it is expected to change.
>>>>>>>
>>>>>>> -.. c:type:: v4l2_h264_pred_weight_table
>>>>>>> +.. c:type:: v4l2_ctrl_h264_pred_weights
>>>>>>>
>>>>>>>  .. cssclass:: longtable
>>>>>>>
>>>>>>> -.. flat-table:: struct v4l2_h264_pred_weight_table
>>>>>>> +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
>>>>>>>
>>>>>>>      :header-rows:  0
>>>>>>>      :stub-columns: 0
>>>>>>>      :widths:       1 1 2
>>>>>>>
>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
>>>>>>> b/drivers/media/v4l2-core/v4l2-ctrls.c index 3f3fbcd60cc6..76c8dc8fb31c
>>>>>>> 100644
>>>>>>> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
>>>>>>> @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
>>>>>>>
>>>>>>>       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
>>>>>>>       "H264 Decode Parameters"; case
>>>>>>>       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return "H264
>>>>>>>       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:          
>>>>>>>           return "H264 Start Code";> > 
>>>>>>> +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
>>>>>>> "H264 Prediction Weight Table";> > 
>>>>>>>       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
>>>>>>>       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:           
>>>>>>>            return "MPEG2 Profile"; case
>>>>>>>       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return "MPEG4
>>>>>>>       I-Frame QP Value";> > 
>>>>>>> @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
>>>>>>> enum v4l2_ctrl_type *type,> > 
>>>>>>>       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
>>>>>>>               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
>>>>>>>               break;
>>>>>>>
>>>>>>> +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
>>>>>>> +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
>>>>>>> +             break;
>>>>>>>
>>>>>>>       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
>>>>>>>               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
>>>>>>>               break;
>>>>>>>
>>>>>>> @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
>>>>>>> v4l2_ctrl *ctrl, u32 idx,> > 
>>>>>>>       case V4L2_CTRL_TYPE_H264_SPS:
>>>>>>>       case V4L2_CTRL_TYPE_H264_PPS:
>>>>>>>
>>>>>>>       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
>>>>>>> +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
>>>>>>>       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
>>>>>>>       
>>>>>>>       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
>>>>>>>               break;
>>>>>>>
>>>>>>> @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
>>>>>>> v4l2_ctrl_handler *hdl,> > 
>>>>>>>       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
>>>>>>>               elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
>>>>>>>               break;
>>>>>>>
>>>>>>> +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
>>>>>>> +             elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
>>>>>>> +             break;
>>>>>>>
>>>>>>>       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
>>>>>>>               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
>>>>>>>               break;
>>>>>>>
>>>>>>> diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
>>>>>>> b/drivers/staging/media/sunxi/cedrus/cedrus.c index
>>>>>>> bc27f9430eeb..027cdd1be5a0 100644
>>>>>>> --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
>>>>>>> +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
>>>>>>> @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[]
>>>>>>> = {
>>>>>>>
>>>>>>>               .codec          = CEDRUS_CODEC_H264,
>>>>>>>               .required       = true,
>>>>>>>       
>>>>>>>       },
>>>>>>>
>>>>>>> +     {
>>>>>>> +             .cfg = {
>>>>>>> +                     .id     = V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
>>>>>>> +             },
>>>>>>> +             .codec          = CEDRUS_CODEC_H264,
>>>>>>> +             .required       = true,
>>>>>>
>>>>>> This should probably be false if this control is to be optional as implied
>>>>>> by the commit message.
>>>>>
>>>>> Well, the control is optional if the driver implements it as optional,
>>>>> which Cedrus isn't currently doing :-)
>>>>
>>>> Why do you think so? Prediction weights are filled only when they are 
>>>> needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/
>>>> sunxi/cedrus/cedrus_h264.c#L370
>>>>
>>>
>>> Right, but that should be changed to be really optional.
>>> How does the driver reject/fail the request if the table is NULL?
>>>
>>> In any case, I don't think it's necessarily something we need
>>> to tackle now.
>>
>> I do not fully follow, the commit message state following:
>>
>>   Note that the biggest value in having the prediction weight table
>>   separated is to allow applications to skip setting this largish control
>>
> 
> This is not exactly what the commit message says, but yeah
> that's the idea.

Hehe, I copied your reply instead of commit message / doc changes :-)

> 
>> Yet the driver still require this new control to be included in the request
>> thanks to the .required = true statement. (if i understand the code correctly)
>>
>> So applications still need to set this largish control?
>>
> 
> This is a uAPI change that paves the way for Cedrus to make the control
> optional. The series doesn't take care of it, but it prepares the road
> for it.
> 
> Since we are not stabilizing the uAPI (yet), I think we still have
> some room to make this change in steps: first we merge the new control
> and then we add the needed changes to Cedrus?
> 
> Does that make sense?

Sure, make sense, I will rephrase it as questions instead :-)

What is not fully clear to me is if this new ctrl should be considered
optional or required from userspace point of view.

Will there be a way for userspace to know if a ctrl is optional or not?

If I implement uapi changes as suggested in this patch in ffmpeg,
i.e. only set weight table ctrl when data if it is present in slice header,
then decoding stops working for cedrus when weight table is not present.

Should I just play it safe and continue to always set the new ctrl for slice
based decoding and treat it as required? Or are we saying that it should be
optional and cedrus is just not following the uapi after this series?

Best regards,
Jonas

> 
>> Best regards,
>> Jonas
>>
>>> Thanks,
>>> Ezequiel
>>>
> 
> 

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Jernej Škrabec]

Dne ponedeljek, 10. avgust 2020 ob 14:57:17 CEST je Ezequiel Garcia 
napisal(a):
> On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
> > Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia 
napisal(a):
> > > On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > > > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > > > The prediction weight parameters are only required under
> > > > > certain conditions, which depend on slice header parameters.
> > > > > 
> > > > > As specified in section 7.3.3 Slice header syntax, the prediction
> > > > > weight table is present if:
> > > > > 
> > > > > ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> > > > > (weighted_bipred_idc == 1 && slice_type == B))
> > > > 
> > > > Maybe a macro can be added to help check this contition?
> > > > 
> > > > Something like this maybe:
> > > > 
> > > > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > > > 
> > > >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> > > >         
> > > >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> > > >          
> > > >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> > > >          
> > > >          ((pps)->weighted_bipred_idc == 1 && \
> > > >          
> > > >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> > > 
> > > Yeah, that could make sense.
> > > 
> > > Note that the biggest value in having the prediction weight table
> > > separated is to allow  applications to skip setting this largish
> > > control,
> > > reducing the amount of data that needs to be passed from userspace
> > > -- especially when not needed :-)
> > > 
> > > > > Given its size, it makes sense to move this table to its control,
> > > > > so applications can avoid passing it if the slice doesn't specify
> > > > > it.
> > > > > 
> > > > > Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> > > > > With this change, it's 188 bytes and struct
> > > > > v4l2_ctrl_h264_pred_weight
> > > > > is 772 bytes.
> > > > > 
> > > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > ---
> > > > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > > > 
> > > > >     as noted by Hans and Jernej.
> > > > > 
> > > > > ---
> > > > > 
> > > > >  .../media/v4l/ext-ctrls-codec.rst             | 19
> > > > >  ++++++++++++-------
> > > > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > > > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > > > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > > > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > > > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > > > >  include/media/h264-ctrls.h                    |  5 +++--
> > > > >  include/media/v4l2-ctrls.h                    |  2 ++
> > > > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > > > 
> > > > > diff --git
> > > > > a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
> > > > > d1438b1e259f..c36ce5a95fc5 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > @@ -1879,18 +1879,23 @@ enum
> > > > > v4l2_mpeg_video_h264_hierarchical_coding_type -> >
> > > > > 
> > > > >        - 0x00000008
> > > > >        -
> > > > > 
> > > > > -``Prediction Weight Table``
> > > > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > > > +    Prediction weight table defined according to :ref:`h264`,
> > > > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > > > +    The prediction weight table must be passed by applications
> > > > > +    under the conditions explained in section 7.3.3 "Slice header
> > > > > +    syntax".
> > > > > 
> > > > > -    The bitstream parameters are defined according to :ref:`h264`,
> > > > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For
> > > > > further
> > > > > -    documentation, refer to the above specification, unless there
> > > > > is
> > > > > -    an explicit comment stating otherwise.
> > > > > +    .. note::
> > > > > +
> > > > > +       This compound control is not yet part of the public kernel
> > > > > API
> > > > > and
> > > > > +       it is expected to change.
> > > > > 
> > > > > -.. c:type:: v4l2_h264_pred_weight_table
> > > > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > > > 
> > > > >  .. cssclass:: longtable
> > > > > 
> > > > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > > > 
> > > > >      :header-rows:  0
> > > > >      :stub-columns: 0
> > > > >      :widths:       1 1 2
> > > > > 
> > > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > b/drivers/media/v4l2-core/v4l2-ctrls.c index
> > > > > 3f3fbcd60cc6..76c8dc8fb31c
> > > > > 100644
> > > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > > > 
> > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
> > > > >       "H264 Decode Parameters"; case
> > > > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return
> > > > >       "H264
> > > > >       
> > > > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:
> > > > >           return "H264 Start Code";> >
> > > > > 
> > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
> > > > > "H264 Prediction Weight Table";> >
> > > > > 
> > > > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
> > > > >       
> > > > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:
> > > > >            return "MPEG2 Profile"; case
> > > > >       
> > > > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return
> > > > >       "MPEG4
> > > > >       I-Frame QP Value";> >
> > > > > 
> > > > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
> > > > > enum v4l2_ctrl_type *type,> >
> > > > > 
> > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > > > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > > > >               break;
> > > > > 
> > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > > > +             break;
> > > > > 
> > > > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > > > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > > > >               break;
> > > > > 
> > > > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
> > > > > v4l2_ctrl *ctrl, u32 idx,> >
> > > > > 
> > > > >       case V4L2_CTRL_TYPE_H264_SPS:
> > > > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > > > 
> > > > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > > > >       
> > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > >               break;
> > > > > 
> > > > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
> > > > > v4l2_ctrl_handler *hdl,> >
> > > > > 
> > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > >               elem_size = sizeof(struct
> > > > >               v4l2_ctrl_h264_decode_params);
> > > > >               break;
> > > > > 
> > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > +             elem_size = sizeof(struct
> > > > > v4l2_ctrl_h264_pred_weights);
> > > > > +             break;
> > > > > 
> > > > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > > > >               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
> > > > >               break;
> > > > > 
> > > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > > > bc27f9430eeb..027cdd1be5a0 100644
> > > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > @@ -78,6 +78,13 @@ static const struct cedrus_control
> > > > > cedrus_controls[]
> > > > > = {
> > > > > 
> > > > >               .codec          = CEDRUS_CODEC_H264,
> > > > >               .required       = true,
> > > > >       
> > > > >       },
> > > > > 
> > > > > +     {
> > > > > +             .cfg = {
> > > > > +                     .id     =
> > > > > V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > > +             },
> > > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > > +             .required       = true,
> > > > 
> > > > This should probably be false if this control is to be optional as
> > > > implied
> > > > by the commit message.
> > > 
> > > Well, the control is optional if the driver implements it as optional,
> > > which Cedrus isn't currently doing :-)
> > 
> > Why do you think so? Prediction weights are filled only when they are
> > needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/medi
> > a/ sunxi/cedrus/cedrus_h264.c#L370
> 
> Right, but that should be changed to be really optional.
> How does the driver reject/fail the request if the table is NULL?

It's my understanding that pointer to this table can't be NULL. NULL would 
mean that there is no control with that ID registered in the driver.

Best regards,
Jernej

> 
> In any case, I don't think it's necessarily something we need
> to tackle now.
> 
> Thanks,
> Ezequiel

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

On Mon, 2020-08-10 at 19:05 +0200, Jernej Škrabec wrote:
> Dne ponedeljek, 10. avgust 2020 ob 14:57:17 CEST je Ezequiel Garcia 
> napisal(a):
> > On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
> > > Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia 
> napisal(a):
> > > > On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > > > > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > > > > The prediction weight parameters are only required under
> > > > > > certain conditions, which depend on slice header parameters.
> > > > > > 
> > > > > > As specified in section 7.3.3 Slice header syntax, the prediction
> > > > > > weight table is present if:
> > > > > > 
> > > > > > ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> > > > > > (weighted_bipred_idc == 1 && slice_type == B))
> > > > > 
> > > > > Maybe a macro can be added to help check this contition?
> > > > > 
> > > > > Something like this maybe:
> > > > > 
> > > > > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > > > > 
> > > > >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> > > > >         
> > > > >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> > > > >          
> > > > >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> > > > >          
> > > > >          ((pps)->weighted_bipred_idc == 1 && \
> > > > >          
> > > > >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> > > > 
> > > > Yeah, that could make sense.
> > > > 
> > > > Note that the biggest value in having the prediction weight table
> > > > separated is to allow  applications to skip setting this largish
> > > > control,
> > > > reducing the amount of data that needs to be passed from userspace
> > > > -- especially when not needed :-)
> > > > 
> > > > > > Given its size, it makes sense to move this table to its control,
> > > > > > so applications can avoid passing it if the slice doesn't specify
> > > > > > it.
> > > > > > 
> > > > > > Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> > > > > > With this change, it's 188 bytes and struct
> > > > > > v4l2_ctrl_h264_pred_weight
> > > > > > is 772 bytes.
> > > > > > 
> > > > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > > ---
> > > > > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > > > > 
> > > > > >     as noted by Hans and Jernej.
> > > > > > 
> > > > > > ---
> > > > > > 
> > > > > >  .../media/v4l/ext-ctrls-codec.rst             | 19
> > > > > >  ++++++++++++-------
> > > > > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > > > > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > > > > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > > > > >  include/media/h264-ctrls.h                    |  5 +++--
> > > > > >  include/media/v4l2-ctrls.h                    |  2 ++
> > > > > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > > > > 
> > > > > > diff --git
> > > > > > a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
> > > > > > d1438b1e259f..c36ce5a95fc5 100644
> > > > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > @@ -1879,18 +1879,23 @@ enum
> > > > > > v4l2_mpeg_video_h264_hierarchical_coding_type -> >
> > > > > > 
> > > > > >        - 0x00000008
> > > > > >        -
> > > > > > 
> > > > > > -``Prediction Weight Table``
> > > > > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > > > > +    Prediction weight table defined according to :ref:`h264`,
> > > > > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > > > > +    The prediction weight table must be passed by applications
> > > > > > +    under the conditions explained in section 7.3.3 "Slice header
> > > > > > +    syntax".
> > > > > > 
> > > > > > -    The bitstream parameters are defined according to :ref:`h264`,
> > > > > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For
> > > > > > further
> > > > > > -    documentation, refer to the above specification, unless there
> > > > > > is
> > > > > > -    an explicit comment stating otherwise.
> > > > > > +    .. note::
> > > > > > +
> > > > > > +       This compound control is not yet part of the public kernel
> > > > > > API
> > > > > > and
> > > > > > +       it is expected to change.
> > > > > > 
> > > > > > -.. c:type:: v4l2_h264_pred_weight_table
> > > > > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > > > > 
> > > > > >  .. cssclass:: longtable
> > > > > > 
> > > > > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > > > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > > > > 
> > > > > >      :header-rows:  0
> > > > > >      :stub-columns: 0
> > > > > >      :widths:       1 1 2
> > > > > > 
> > > > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > b/drivers/media/v4l2-core/v4l2-ctrls.c index
> > > > > > 3f3fbcd60cc6..76c8dc8fb31c
> > > > > > 100644
> > > > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > > > > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
> > > > > >       "H264 Decode Parameters"; case
> > > > > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return
> > > > > >       "H264
> > > > > >       
> > > > > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:
> > > > > >           return "H264 Start Code";> >
> > > > > > 
> > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
> > > > > > "H264 Prediction Weight Table";> >
> > > > > > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
> > > > > >       
> > > > > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:
> > > > > >            return "MPEG2 Profile"; case
> > > > > >       
> > > > > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return
> > > > > >       "MPEG4
> > > > > >       I-Frame QP Value";> >
> > > > > > 
> > > > > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
> > > > > > enum v4l2_ctrl_type *type,> >
> > > > > > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > > > > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > > > > >               break;
> > > > > > 
> > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > > > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > > > > +             break;
> > > > > > 
> > > > > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > > > > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > > > > >               break;
> > > > > > 
> > > > > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
> > > > > > v4l2_ctrl *ctrl, u32 idx,> >
> > > > > > 
> > > > > >       case V4L2_CTRL_TYPE_H264_SPS:
> > > > > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > > > > 
> > > > > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > > > > >       
> > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > >               break;
> > > > > > 
> > > > > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
> > > > > > v4l2_ctrl_handler *hdl,> >
> > > > > > 
> > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > >               elem_size = sizeof(struct
> > > > > >               v4l2_ctrl_h264_decode_params);
> > > > > >               break;
> > > > > > 
> > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > +             elem_size = sizeof(struct
> > > > > > v4l2_ctrl_h264_pred_weights);
> > > > > > +             break;
> > > > > > 
> > > > > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > > > > >               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
> > > > > >               break;
> > > > > > 
> > > > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > > > > bc27f9430eeb..027cdd1be5a0 100644
> > > > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > @@ -78,6 +78,13 @@ static const struct cedrus_control
> > > > > > cedrus_controls[]
> > > > > > = {
> > > > > > 
> > > > > >               .codec          = CEDRUS_CODEC_H264,
> > > > > >               .required       = true,
> > > > > >       
> > > > > >       },
> > > > > > 
> > > > > > +     {
> > > > > > +             .cfg = {
> > > > > > +                     .id     =
> > > > > > V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > > > +             },
> > > > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > > > +             .required       = true,
> > > > > 
> > > > > This should probably be false if this control is to be optional as
> > > > > implied
> > > > > by the commit message.
> > > > 
> > > > Well, the control is optional if the driver implements it as optional,
> > > > which Cedrus isn't currently doing :-)
> > > 
> > > Why do you think so? Prediction weights are filled only when they are
> > > needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/medi
> > > a/ sunxi/cedrus/cedrus_h264.c#L370
> > 
> > Right, but that should be changed to be really optional.
> > How does the driver reject/fail the request if the table is NULL?
> 
> It's my understanding that pointer to this table can't be NULL. NULL would 
> mean that there is no control with that ID registered in the driver.
> 

Hm, I'm starting to think you are right. So, does this mean
the default quantization matrix here is bogus?

        if (quantization && quantization->load_intra_quantiser_matrix)
                matrix = quantization->intra_quantiser_matrix;
        else
                matrix = intra_quantization_matrix_default;

Thanks,
Ezequiel

> Best regards,
> Jernej
> 
> > In any case, I don't think it's necessarily something we need
> > to tackle now.
> > 
> > Thanks,
> > Ezequiel
> 
> 
> 

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Jernej Škrabec]

Dne ponedeljek, 10. avgust 2020 ob 21:30:59 CEST je Ezequiel Garcia 
napisal(a):
> On Mon, 2020-08-10 at 19:05 +0200, Jernej Škrabec wrote:
> > Dne ponedeljek, 10. avgust 2020 ob 14:57:17 CEST je Ezequiel Garcia
> > 
> > napisal(a):
> > > On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
> > > > Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia
> > 
> > napisal(a):
> > > > > On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > > > > > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > > > > > The prediction weight parameters are only required under
> > > > > > > certain conditions, which depend on slice header parameters.
> > > > > > > 
> > > > > > > As specified in section 7.3.3 Slice header syntax, the
> > > > > > > prediction
> > > > > > > weight table is present if:
> > > > > > > 
> > > > > > > ((weighted_pred_flag && (slice_type == P || slice_type == SP))
> > > > > > > || \
> > > > > > > (weighted_bipred_idc == 1 && slice_type == B))
> > > > > > 
> > > > > > Maybe a macro can be added to help check this contition?
> > > > > > 
> > > > > > Something like this maybe:
> > > > > > 
> > > > > > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > > > > > 
> > > > > >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> > > > > >         
> > > > > >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> > > > > >          
> > > > > >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> > > > > >          
> > > > > >          ((pps)->weighted_bipred_idc == 1 && \
> > > > > >          
> > > > > >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> > > > > 
> > > > > Yeah, that could make sense.
> > > > > 
> > > > > Note that the biggest value in having the prediction weight table
> > > > > separated is to allow  applications to skip setting this largish
> > > > > control,
> > > > > reducing the amount of data that needs to be passed from userspace
> > > > > -- especially when not needed :-)
> > > > > 
> > > > > > > Given its size, it makes sense to move this table to its
> > > > > > > control,
> > > > > > > so applications can avoid passing it if the slice doesn't
> > > > > > > specify
> > > > > > > it.
> > > > > > > 
> > > > > > > Before this change struct v4l2_ctrl_h264_slice_params was 960
> > > > > > > bytes.
> > > > > > > With this change, it's 188 bytes and struct
> > > > > > > v4l2_ctrl_h264_pred_weight
> > > > > > > is 772 bytes.
> > > > > > > 
> > > > > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > > > ---
> > > > > > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > > > > > 
> > > > > > >     as noted by Hans and Jernej.
> > > > > > > 
> > > > > > > ---
> > > > > > > 
> > > > > > >  .../media/v4l/ext-ctrls-codec.rst             | 19
> > > > > > >  ++++++++++++-------
> > > > > > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > > > > > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > > > > > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > > > > > >  include/media/h264-ctrls.h                    |  5 +++--
> > > > > > >  include/media/v4l2-ctrls.h                    |  2 ++
> > > > > > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > > > > > 
> > > > > > > diff --git
> > > > > > > a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > index
> > > > > > > d1438b1e259f..c36ce5a95fc5 100644
> > > > > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > @@ -1879,18 +1879,23 @@ enum
> > > > > > > v4l2_mpeg_video_h264_hierarchical_coding_type -> >
> > > > > > > 
> > > > > > >        - 0x00000008
> > > > > > >        -
> > > > > > > 
> > > > > > > -``Prediction Weight Table``
> > > > > > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > > > > > +    Prediction weight table defined according to :ref:`h264`,
> > > > > > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > > > > > +    The prediction weight table must be passed by applications
> > > > > > > +    under the conditions explained in section 7.3.3 "Slice
> > > > > > > header
> > > > > > > +    syntax".
> > > > > > > 
> > > > > > > -    The bitstream parameters are defined according to
> > > > > > > :ref:`h264`,
> > > > > > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For
> > > > > > > further
> > > > > > > -    documentation, refer to the above specification, unless
> > > > > > > there
> > > > > > > is
> > > > > > > -    an explicit comment stating otherwise.
> > > > > > > +    .. note::
> > > > > > > +
> > > > > > > +       This compound control is not yet part of the public
> > > > > > > kernel
> > > > > > > API
> > > > > > > and
> > > > > > > +       it is expected to change.
> > > > > > > 
> > > > > > > -.. c:type:: v4l2_h264_pred_weight_table
> > > > > > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > > > > > 
> > > > > > >  .. cssclass:: longtable
> > > > > > > 
> > > > > > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > > > > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > > > > > 
> > > > > > >      :header-rows:  0
> > > > > > >      :stub-columns: 0
> > > > > > >      :widths:       1 1 2
> > > > > > > 
> > > > > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > b/drivers/media/v4l2-core/v4l2-ctrls.c index
> > > > > > > 3f3fbcd60cc6..76c8dc8fb31c
> > > > > > > 100644
> > > > > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > > > > > 
> > > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:           
> > > > > > >       return
> > > > > > >       "H264 Decode Parameters"; case
> > > > > > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return
> > > > > > >       "H264
> > > > > > >       
> > > > > > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:
> > > > > > >           return "H264 Start Code";> >
> > > > > > > 
> > > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:            
> > > > > > > return
> > > > > > > "H264 Prediction Weight Table";> >
> > > > > > > 
> > > > > > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                  
> > > > > > >       return
> > > > > > >       
> > > > > > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:
> > > > > > >            return "MPEG2 Profile"; case
> > > > > > >       
> > > > > > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return
> > > > > > >       "MPEG4
> > > > > > >       I-Frame QP Value";> >
> > > > > > > 
> > > > > > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char
> > > > > > > **name,
> > > > > > > enum v4l2_ctrl_type *type,> >
> > > > > > > 
> > > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > > > > > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > > > > > >               break;
> > > > > > > 
> > > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > > > > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > > > > > +             break;
> > > > > > > 
> > > > > > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > > > > > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > > > > > >               break;
> > > > > > > 
> > > > > > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const
> > > > > > > struct
> > > > > > > v4l2_ctrl *ctrl, u32 idx,> >
> > > > > > > 
> > > > > > >       case V4L2_CTRL_TYPE_H264_SPS:
> > > > > > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > > > > > 
> > > > > > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > > > > > >       
> > > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > > >               break;
> > > > > > > 
> > > > > > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl
> > > > > > > *v4l2_ctrl_new(struct
> > > > > > > v4l2_ctrl_handler *hdl,> >
> > > > > > > 
> > > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > > >               elem_size = sizeof(struct
> > > > > > >               v4l2_ctrl_h264_decode_params);
> > > > > > >               break;
> > > > > > > 
> > > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > > +             elem_size = sizeof(struct
> > > > > > > v4l2_ctrl_h264_pred_weights);
> > > > > > > +             break;
> > > > > > > 
> > > > > > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > > > > > >               elem_size = sizeof(struct
> > > > > > >               v4l2_ctrl_vp8_frame_header);
> > > > > > >               break;
> > > > > > > 
> > > > > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > > > > > bc27f9430eeb..027cdd1be5a0 100644
> > > > > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > @@ -78,6 +78,13 @@ static const struct cedrus_control
> > > > > > > cedrus_controls[]
> > > > > > > = {
> > > > > > > 
> > > > > > >               .codec          = CEDRUS_CODEC_H264,
> > > > > > >               .required       = true,
> > > > > > >       
> > > > > > >       },
> > > > > > > 
> > > > > > > +     {
> > > > > > > +             .cfg = {
> > > > > > > +                     .id     =
> > > > > > > V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > > > > +             },
> > > > > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > > > > +             .required       = true,
> > > > > > 
> > > > > > This should probably be false if this control is to be optional as
> > > > > > implied
> > > > > > by the commit message.
> > > > > 
> > > > > Well, the control is optional if the driver implements it as
> > > > > optional,
> > > > > which Cedrus isn't currently doing :-)
> > > > 
> > > > Why do you think so? Prediction weights are filled only when they are
> > > > needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/
> > > > medi
> > > > a/ sunxi/cedrus/cedrus_h264.c#L370
> > > 
> > > Right, but that should be changed to be really optional.
> > > How does the driver reject/fail the request if the table is NULL?
> > 
> > It's my understanding that pointer to this table can't be NULL. NULL would
> > mean that there is no control with that ID registered in the driver.
> 
> Hm, I'm starting to think you are right. So, does this mean
> the default quantization matrix here is bogus?
> 
>         if (quantization && quantization->load_intra_quantiser_matrix)
>                 matrix = quantization->intra_quantiser_matrix;
>         else
>                 matrix = intra_quantization_matrix_default;

No, not really. Userspace can set load_intra_quantiser_matrix flag to false.

Best regards,
Jernej

> 
> Thanks,
> Ezequiel
> 
> > Best regards,
> > Jernej
> > 
> > > In any case, I don't think it's necessarily something we need
> > > to tackle now.
> > > 
> > > Thanks,
> > > Ezequiel

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

On Mon, 2020-08-10 at 16:57 +0000, Jonas Karlman wrote:
> On 2020-08-10 17:28, Ezequiel Garcia wrote:
> > On Mon, 2020-08-10 at 14:55 +0000, Jonas Karlman wrote:
> > > On 2020-08-10 14:57, Ezequiel Garcia wrote:
> > > > On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
> > > > > Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia napisal(a):
> > > > > > On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > > > > > > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > > > > > > The prediction weight parameters are only required under
> > > > > > > > certain conditions, which depend on slice header parameters.
> > > > > > > > 
> > > > > > > > As specified in section 7.3.3 Slice header syntax, the prediction
> > > > > > > > weight table is present if:
> > > > > > > > 
> > > > > > > > ((weighted_pred_flag && (slice_type == P || slice_type == SP)) || \
> > > > > > > > (weighted_bipred_idc == 1 && slice_type == B))
> > > > > > > 
> > > > > > > Maybe a macro can be added to help check this contition?
> > > > > > > 
> > > > > > > Something like this maybe:
> > > > > > > 
> > > > > > > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > > > > > > 
> > > > > > >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> > > > > > >         
> > > > > > >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> > > > > > >          
> > > > > > >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> > > > > > >          
> > > > > > >          ((pps)->weighted_bipred_idc == 1 && \
> > > > > > >          
> > > > > > >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> > > > > > 
> > > > > > Yeah, that could make sense.
> > > > > > 
> > > > > > Note that the biggest value in having the prediction weight table
> > > > > > separated is to allow  applications to skip setting this largish control,
> > > > > > reducing the amount of data that needs to be passed from userspace
> > > > > > -- especially when not needed :-)
> > > > > > 
> > > > > > > > Given its size, it makes sense to move this table to its control,
> > > > > > > > so applications can avoid passing it if the slice doesn't specify it.
> > > > > > > > 
> > > > > > > > Before this change struct v4l2_ctrl_h264_slice_params was 960 bytes.
> > > > > > > > With this change, it's 188 bytes and struct v4l2_ctrl_h264_pred_weight
> > > > > > > > is 772 bytes.
> > > > > > > > 
> > > > > > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > > > > ---
> > > > > > > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > > > > > > 
> > > > > > > >     as noted by Hans and Jernej.
> > > > > > > > 
> > > > > > > > ---
> > > > > > > > 
> > > > > > > >  .../media/v4l/ext-ctrls-codec.rst             | 19 ++++++++++++-------
> > > > > > > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > > > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > > > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > > > > > > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > > > > > > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > > > > > > >  include/media/h264-ctrls.h                    |  5 +++--
> > > > > > > >  include/media/v4l2-ctrls.h                    |  2 ++
> > > > > > > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > > > > > > 
> > > > > > > > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index
> > > > > > > > d1438b1e259f..c36ce5a95fc5 100644
> > > > > > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > > @@ -1879,18 +1879,23 @@ enum
> > > > > > > > v4l2_mpeg_video_h264_hierarchical_coding_type -> > 
> > > > > > > >        - 0x00000008
> > > > > > > >        -
> > > > > > > > 
> > > > > > > > -``Prediction Weight Table``
> > > > > > > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > > > > > > +    Prediction weight table defined according to :ref:`h264`,
> > > > > > > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > > > > > > +    The prediction weight table must be passed by applications
> > > > > > > > +    under the conditions explained in section 7.3.3 "Slice header
> > > > > > > > +    syntax".
> > > > > > > > 
> > > > > > > > -    The bitstream parameters are defined according to :ref:`h264`,
> > > > > > > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For further
> > > > > > > > -    documentation, refer to the above specification, unless there is
> > > > > > > > -    an explicit comment stating otherwise.
> > > > > > > > +    .. note::
> > > > > > > > +
> > > > > > > > +       This compound control is not yet part of the public kernel API
> > > > > > > > and
> > > > > > > > +       it is expected to change.
> > > > > > > > 
> > > > > > > > -.. c:type:: v4l2_h264_pred_weight_table
> > > > > > > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > > > > > > 
> > > > > > > >  .. cssclass:: longtable
> > > > > > > > 
> > > > > > > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > > > > > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > > > > > > 
> > > > > > > >      :header-rows:  0
> > > > > > > >      :stub-columns: 0
> > > > > > > >      :widths:       1 1 2
> > > > > > > > 
> > > > > > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > > b/drivers/media/v4l2-core/v4l2-ctrls.c index 3f3fbcd60cc6..76c8dc8fb31c
> > > > > > > > 100644
> > > > > > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > > > > > > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:            return
> > > > > > > >       "H264 Decode Parameters"; case
> > > > > > > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return "H264
> > > > > > > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:          
> > > > > > > >           return "H264 Start Code";> > 
> > > > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:             return
> > > > > > > > "H264 Prediction Weight Table";> > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                   return
> > > > > > > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:           
> > > > > > > >            return "MPEG2 Profile"; case
> > > > > > > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return "MPEG4
> > > > > > > >       I-Frame QP Value";> > 
> > > > > > > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char **name,
> > > > > > > > enum v4l2_ctrl_type *type,> > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > > > > > > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > > > > > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > > > > > > +             break;
> > > > > > > > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > > > > > > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const struct
> > > > > > > > v4l2_ctrl *ctrl, u32 idx,> > 
> > > > > > > >       case V4L2_CTRL_TYPE_H264_SPS:
> > > > > > > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > > > > > > 
> > > > > > > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > > > > > > >       
> > > > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct
> > > > > > > > v4l2_ctrl_handler *hdl,> > 
> > > > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > > > >               elem_size = sizeof(struct v4l2_ctrl_h264_decode_params);
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > > > +             elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
> > > > > > > > +             break;
> > > > > > > > 
> > > > > > > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > > > > > > >               elem_size = sizeof(struct v4l2_ctrl_vp8_frame_header);
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > > > > > > bc27f9430eeb..027cdd1be5a0 100644
> > > > > > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > > @@ -78,6 +78,13 @@ static const struct cedrus_control cedrus_controls[]
> > > > > > > > = {
> > > > > > > > 
> > > > > > > >               .codec          = CEDRUS_CODEC_H264,
> > > > > > > >               .required       = true,
> > > > > > > >       
> > > > > > > >       },
> > > > > > > > 
> > > > > > > > +     {
> > > > > > > > +             .cfg = {
> > > > > > > > +                     .id     = V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > > > > > +             },
> > > > > > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > > > > > +             .required       = true,
> > > > > > > 
> > > > > > > This should probably be false if this control is to be optional as implied
> > > > > > > by the commit message.
> > > > > > 
> > > > > > Well, the control is optional if the driver implements it as optional,
> > > > > > which Cedrus isn't currently doing :-)
> > > > > 
> > > > > Why do you think so? Prediction weights are filled only when they are 
> > > > > needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/
> > > > > sunxi/cedrus/cedrus_h264.c#L370
> > > > > 
> > > > 
> > > > Right, but that should be changed to be really optional.
> > > > How does the driver reject/fail the request if the table is NULL?
> > > > 
> > > > In any case, I don't think it's necessarily something we need
> > > > to tackle now.
> > > 
> > > I do not fully follow, the commit message state following:
> > > 
> > >   Note that the biggest value in having the prediction weight table
> > >   separated is to allow applications to skip setting this largish control
> > > 
> > 
> > This is not exactly what the commit message says, but yeah
> > that's the idea.
> 
> Hehe, I copied your reply instead of commit message / doc changes :-)
> 
> > > Yet the driver still require this new control to be included in the request
> > > thanks to the .required = true statement. (if i understand the code correctly)
> > > 
> > > So applications still need to set this largish control?
> > > 
> > 
> > This is a uAPI change that paves the way for Cedrus to make the control
> > optional. The series doesn't take care of it, but it prepares the road
> > for it.
> > 
> > Since we are not stabilizing the uAPI (yet), I think we still have
> > some room to make this change in steps: first we merge the new control
> > and then we add the needed changes to Cedrus?
> > 
> > Does that make sense?
> 
> Sure, make sense, I will rephrase it as questions instead :-)
> 
> What is not fully clear to me is if this new ctrl should be considered
> optional or required from userspace point of view.
> 
> Will there be a way for userspace to know if a ctrl is optional or not?
> 

We don't currently have a way for applications to query if a control
is mandatory for a given request. However... 

> If I implement uapi changes as suggested in this patch in ffmpeg,
> i.e. only set weight table ctrl when data if it is present in slice header,
> then decoding stops working for cedrus when weight table is not present.
> 
> Should I just play it safe and continue to always set the new ctrl for slice
> based decoding and treat it as required? Or are we saying that it should be
> optional and cedrus is just not following the uapi after this series?
> 

... as Jernej pointed-out, if the control is initialized then it'll never
be NULL. It's initialized in std_init_compound.

So we can just go ahead and add this patch to the series?

It will be the application responsability to fill the
control when needed, and applications are now able to not pass
the control when not needed, just as expected.

diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c
index 027cdd1be5a0..826324faad7e 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
@@ -83,7 +83,7 @@ static const struct cedrus_control cedrus_controls[] = {
 			.id	= V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
 		},
 		.codec		= CEDRUS_CODEC_H264,
-		.required	= true,
+		.required	= false,
 	},
 	{
 		.cfg = {
diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
index 7b2169d185b8..57bad8733aed 100644
--- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
+++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
@@ -364,11 +364,7 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
 
 	cedrus_skip_bits(dev, slice->header_bit_size);
 
-	if (((pps->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) &&
-	     (slice->slice_type == V4L2_H264_SLICE_TYPE_P ||
-	      slice->slice_type == V4L2_H264_SLICE_TYPE_SP)) ||
-	    (pps->weighted_bipred_idc == 1 &&
-	     slice->slice_type == V4L2_H264_SLICE_TYPE_B))
+	if (V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice))
 		cedrus_write_pred_weight_table(ctx, run);
 
 	if ((slice->slice_type == V4L2_H264_SLICE_TYPE_P) ||
diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
index fa5663876e73..7d1a6f97eb12 100644
--- a/include/media/h264-ctrls.h
+++ b/include/media/h264-ctrls.h
@@ -127,6 +127,13 @@ struct v4l2_h264_weight_factors {
 	__s16 chroma_offset[32][2];
 };
 
+#define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
+	((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
+	 ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
+	  (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
+	 ((pps)->weighted_bipred_idc == 1 && \
+	  (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
+
 struct v4l2_ctrl_h264_pred_weights {
 	__u16 luma_log2_weight_denom;
 	__u16 chroma_log2_weight_denom;

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

On Mon, 2020-08-10 at 21:34 +0200, Jernej Škrabec wrote:
> Dne ponedeljek, 10. avgust 2020 ob 21:30:59 CEST je Ezequiel Garcia 
> napisal(a):
> > On Mon, 2020-08-10 at 19:05 +0200, Jernej Škrabec wrote:
> > > Dne ponedeljek, 10. avgust 2020 ob 14:57:17 CEST je Ezequiel Garcia
> > > 
> > > napisal(a):
> > > > On Sun, 2020-08-09 at 23:11 +0200, Jernej Škrabec wrote:
> > > > > Dne nedelja, 09. avgust 2020 ob 15:55:50 CEST je Ezequiel Garcia
> > > 
> > > napisal(a):
> > > > > > On Sat, 8 Aug 2020 at 18:01, Jonas Karlman <jonas@kwiboo.se> wrote:
> > > > > > > On 2020-08-06 17:12, Ezequiel Garcia wrote:
> > > > > > > > The prediction weight parameters are only required under
> > > > > > > > certain conditions, which depend on slice header parameters.
> > > > > > > > 
> > > > > > > > As specified in section 7.3.3 Slice header syntax, the
> > > > > > > > prediction
> > > > > > > > weight table is present if:
> > > > > > > > 
> > > > > > > > ((weighted_pred_flag && (slice_type == P || slice_type == SP))
> > > > > > > > > > \
> > > > > > > > (weighted_bipred_idc == 1 && slice_type == B))
> > > > > > > 
> > > > > > > Maybe a macro can be added to help check this contition?
> > > > > > > 
> > > > > > > Something like this maybe:
> > > > > > > 
> > > > > > > #define V4L2_H264_CTRL_PRED_WEIGHTS_REQUIRED(pps, slice) \
> > > > > > > 
> > > > > > >         ((((pps)->flags & V4L2_H264_PPS_FLAG_WEIGHTED_PRED) && \
> > > > > > >         
> > > > > > >          ((slice)->slice_type == V4L2_H264_SLICE_TYPE_P || \
> > > > > > >          
> > > > > > >            (slice)->slice_type == V4L2_H264_SLICE_TYPE_SP)) || \
> > > > > > >          
> > > > > > >          ((pps)->weighted_bipred_idc == 1 && \
> > > > > > >          
> > > > > > >           (slice)->slice_type == V4L2_H264_SLICE_TYPE_B))
> > > > > > 
> > > > > > Yeah, that could make sense.
> > > > > > 
> > > > > > Note that the biggest value in having the prediction weight table
> > > > > > separated is to allow  applications to skip setting this largish
> > > > > > control,
> > > > > > reducing the amount of data that needs to be passed from userspace
> > > > > > -- especially when not needed :-)
> > > > > > 
> > > > > > > > Given its size, it makes sense to move this table to its
> > > > > > > > control,
> > > > > > > > so applications can avoid passing it if the slice doesn't
> > > > > > > > specify
> > > > > > > > it.
> > > > > > > > 
> > > > > > > > Before this change struct v4l2_ctrl_h264_slice_params was 960
> > > > > > > > bytes.
> > > > > > > > With this change, it's 188 bytes and struct
> > > > > > > > v4l2_ctrl_h264_pred_weight
> > > > > > > > is 772 bytes.
> > > > > > > > 
> > > > > > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > > > > ---
> > > > > > > > v2: Fix missing Cedrus changes and mssing control declaration,
> > > > > > > > 
> > > > > > > >     as noted by Hans and Jernej.
> > > > > > > > 
> > > > > > > > ---
> > > > > > > > 
> > > > > > > >  .../media/v4l/ext-ctrls-codec.rst             | 19
> > > > > > > >  ++++++++++++-------
> > > > > > > >  drivers/media/v4l2-core/v4l2-ctrls.c          |  8 ++++++++
> > > > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.c   |  7 +++++++
> > > > > > > >  drivers/staging/media/sunxi/cedrus/cedrus.h   |  1 +
> > > > > > > >  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 ++
> > > > > > > >  .../staging/media/sunxi/cedrus/cedrus_h264.c  |  6 ++----
> > > > > > > >  include/media/h264-ctrls.h                    |  5 +++--
> > > > > > > >  include/media/v4l2-ctrls.h                    |  2 ++
> > > > > > > >  8 files changed, 37 insertions(+), 13 deletions(-)
> > > > > > > > 
> > > > > > > > diff --git
> > > > > > > > a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > > b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > > index
> > > > > > > > d1438b1e259f..c36ce5a95fc5 100644
> > > > > > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > > > > @@ -1879,18 +1879,23 @@ enum
> > > > > > > > v4l2_mpeg_video_h264_hierarchical_coding_type -> >
> > > > > > > > 
> > > > > > > >        - 0x00000008
> > > > > > > >        -
> > > > > > > > 
> > > > > > > > -``Prediction Weight Table``
> > > > > > > > +``V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS (struct)``
> > > > > > > > +    Prediction weight table defined according to :ref:`h264`,
> > > > > > > > +    section 7.4.3.2 "Prediction Weight Table Semantics".
> > > > > > > > +    The prediction weight table must be passed by applications
> > > > > > > > +    under the conditions explained in section 7.3.3 "Slice
> > > > > > > > header
> > > > > > > > +    syntax".
> > > > > > > > 
> > > > > > > > -    The bitstream parameters are defined according to
> > > > > > > > :ref:`h264`,
> > > > > > > > -    section 7.4.3.2 "Prediction Weight Table Semantics". For
> > > > > > > > further
> > > > > > > > -    documentation, refer to the above specification, unless
> > > > > > > > there
> > > > > > > > is
> > > > > > > > -    an explicit comment stating otherwise.
> > > > > > > > +    .. note::
> > > > > > > > +
> > > > > > > > +       This compound control is not yet part of the public
> > > > > > > > kernel
> > > > > > > > API
> > > > > > > > and
> > > > > > > > +       it is expected to change.
> > > > > > > > 
> > > > > > > > -.. c:type:: v4l2_h264_pred_weight_table
> > > > > > > > +.. c:type:: v4l2_ctrl_h264_pred_weights
> > > > > > > > 
> > > > > > > >  .. cssclass:: longtable
> > > > > > > > 
> > > > > > > > -.. flat-table:: struct v4l2_h264_pred_weight_table
> > > > > > > > +.. flat-table:: struct v4l2_ctrl_h264_pred_weights
> > > > > > > > 
> > > > > > > >      :header-rows:  0
> > > > > > > >      :stub-columns: 0
> > > > > > > >      :widths:       1 1 2
> > > > > > > > 
> > > > > > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > > b/drivers/media/v4l2-core/v4l2-ctrls.c index
> > > > > > > > 3f3fbcd60cc6..76c8dc8fb31c
> > > > > > > > 100644
> > > > > > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > > > > > > @@ -897,6 +897,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > > > > > > > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:           
> > > > > > > >       return
> > > > > > > >       "H264 Decode Parameters"; case
> > > > > > > >       V4L2_CID_MPEG_VIDEO_H264_DECODE_MODE:              return
> > > > > > > >       "H264
> > > > > > > >       
> > > > > > > >       Decode Mode"; case V4L2_CID_MPEG_VIDEO_H264_START_CODE:
> > > > > > > >           return "H264 Start Code";> >
> > > > > > > > 
> > > > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:            
> > > > > > > > return
> > > > > > > > "H264 Prediction Weight Table";> >
> > > > > > > > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL:                  
> > > > > > > >       return
> > > > > > > >       
> > > > > > > >       "MPEG2 Level"; case V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE:
> > > > > > > >            return "MPEG2 Profile"; case
> > > > > > > >       
> > > > > > > >       V4L2_CID_MPEG_VIDEO_MPEG4_I_FRAME_QP:              return
> > > > > > > >       "MPEG4
> > > > > > > >       I-Frame QP Value";> >
> > > > > > > > 
> > > > > > > > @@ -1412,6 +1413,9 @@ void v4l2_ctrl_fill(u32 id, const char
> > > > > > > > **name,
> > > > > > > > enum v4l2_ctrl_type *type,> >
> > > > > > > > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_H264_DECODE_PARAMS:
> > > > > > > >               *type = V4L2_CTRL_TYPE_H264_DECODE_PARAMS;
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > +     case V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS:
> > > > > > > > +             *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> > > > > > > > +             break;
> > > > > > > > 
> > > > > > > >       case V4L2_CID_MPEG_VIDEO_VP8_FRAME_HEADER:
> > > > > > > >               *type = V4L2_CTRL_TYPE_VP8_FRAME_HEADER;
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > @@ -1790,6 +1794,7 @@ static int std_validate_compound(const
> > > > > > > > struct
> > > > > > > > v4l2_ctrl *ctrl, u32 idx,> >
> > > > > > > > 
> > > > > > > >       case V4L2_CTRL_TYPE_H264_SPS:
> > > > > > > >       case V4L2_CTRL_TYPE_H264_PPS:
> > > > > > > > 
> > > > > > > >       case V4L2_CTRL_TYPE_H264_SCALING_MATRIX:
> > > > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > > >       case V4L2_CTRL_TYPE_H264_SLICE_PARAMS:
> > > > > > > >       
> > > > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > @@ -2553,6 +2558,9 @@ static struct v4l2_ctrl
> > > > > > > > *v4l2_ctrl_new(struct
> > > > > > > > v4l2_ctrl_handler *hdl,> >
> > > > > > > > 
> > > > > > > >       case V4L2_CTRL_TYPE_H264_DECODE_PARAMS:
> > > > > > > >               elem_size = sizeof(struct
> > > > > > > >               v4l2_ctrl_h264_decode_params);
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > +     case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> > > > > > > > +             elem_size = sizeof(struct
> > > > > > > > v4l2_ctrl_h264_pred_weights);
> > > > > > > > +             break;
> > > > > > > > 
> > > > > > > >       case V4L2_CTRL_TYPE_VP8_FRAME_HEADER:
> > > > > > > >               elem_size = sizeof(struct
> > > > > > > >               v4l2_ctrl_vp8_frame_header);
> > > > > > > >               break;
> > > > > > > > 
> > > > > > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > > b/drivers/staging/media/sunxi/cedrus/cedrus.c index
> > > > > > > > bc27f9430eeb..027cdd1be5a0 100644
> > > > > > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c
> > > > > > > > @@ -78,6 +78,13 @@ static const struct cedrus_control
> > > > > > > > cedrus_controls[]
> > > > > > > > = {
> > > > > > > > 
> > > > > > > >               .codec          = CEDRUS_CODEC_H264,
> > > > > > > >               .required       = true,
> > > > > > > >       
> > > > > > > >       },
> > > > > > > > 
> > > > > > > > +     {
> > > > > > > > +             .cfg = {
> > > > > > > > +                     .id     =
> > > > > > > > V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > > > > > +             },
> > > > > > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > > > > > +             .required       = true,
> > > > > > > 
> > > > > > > This should probably be false if this control is to be optional as
> > > > > > > implied
> > > > > > > by the commit message.
> > > > > > 
> > > > > > Well, the control is optional if the driver implements it as
> > > > > > optional,
> > > > > > which Cedrus isn't currently doing :-)
> > > > > 
> > > > > Why do you think so? Prediction weights are filled only when they are
> > > > > needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/
> > > > > medi
> > > > > a/ sunxi/cedrus/cedrus_h264.c#L370
> > > > 
> > > > Right, but that should be changed to be really optional.
> > > > How does the driver reject/fail the request if the table is NULL?
> > > 
> > > It's my understanding that pointer to this table can't be NULL. NULL would
> > > mean that there is no control with that ID registered in the driver.
> > 
> > Hm, I'm starting to think you are right. So, does this mean
> > the default quantization matrix here is bogus?
> > 
> >         if (quantization && quantization->load_intra_quantiser_matrix)
> >                 matrix = quantization->intra_quantiser_matrix;
> >         else
> >                 matrix = intra_quantization_matrix_default;
> 
> No, not really. Userspace can set load_intra_quantiser_matrix flag to false.
> 

Right. But quantization can never be NULL.

I think this if (quantization ...) has been confusing me
into thinking a control can be NULL.

:-)

Thanks for the pointers, this is useful.

Ezequiel

> Best regards,
> Jernej
> 
> > Thanks,
> > Ezequiel
> > 
> > > Best regards,
> > > Jernej
> > > 
> > > > In any case, I don't think it's necessarily something we need
> > > > to tackle now.
> > > > 
> > > > Thanks,
> > > > Ezequiel
> 
> 
> 

Re: [PATCH v2 00/14] Clean H264 stateless uAPI

[Jernej Škrabec]

Hi!

Dne četrtek, 06. avgust 2020 ob 17:12:56 CEST je Ezequiel Garcia napisal(a):
> Here's a new round for the H.264 uAPI cleanup, which as discussed
> aims at being stabilized and promoted as a first-class public uAPI soon.
> 
> It should be noted that there is already GStreamer native
> support for this interface, which will be part of 1.18,
> once it's released.
> 
> I have pushed a branch porting GStreamer to
> support these interface changes:
> 
> https://gitlab.freedesktop.org/ezequielgarcia/gst-plugins-bad/-/commits/for_
> h264_uapi_v3
> 
> As was discussed the SLICE_PARAMS control is now clarified
> to work for one-slice-per-request operation, using CAPTURE
> buffer holding features. This is how Cedrus driver is implemented.
> 
> The other drivers currently supported Hantro and Rockchip VDEC,
> as well as the MTK stateless decoder posted by Alex Courbot
> operate in frame-based mode.
> 
> These "frame-based" devices parse the slice headers in hardware,
> and therefore shall not support SLICE_PARAMS. To that extent,
> the redundant bitstream fields are now part of the DECODE_PARAMS
> control.
> 
> Hopefully now the specification documentation is clear enough.
> GStreamer, Chromium and FFmpeg (which I'm sure will be implemented
> as soon as we stabilize the API) should serve as reference examples
> on how the API is consumed.
> 

I tested this series using https://github.com/Kwiboo/FFmpeg/commits/v4l2-request-hwaccel-4.3.1 on Cedrus (Allwinner H6) using additional patch 
contained in discussion around patch 3 and I couldn't find any issue.

You can add to whole series:
Tested-by: Jernej Skrabec <jernej.skrabec@siol.net>

Best regards,
Jernej

Re: [PATCH v2 03/14] media: uapi: h264: Split prediction weight parameters

[Ezequiel Garcia]

Hi Jonas, Jernej and everyone :)

> > > > > > > > +     {
> > > > > > > > +             .cfg = {
> > > > > > > > +                     .id     =
> > > > > > > > V4L2_CID_MPEG_VIDEO_H264_PRED_WEIGHTS,
> > > > > > > > +             },
> > > > > > > > +             .codec          = CEDRUS_CODEC_H264,
> > > > > > > > +             .required       = true,
> > > > > > > 
> > > > > > > This should probably be false if this control is to be optional as
> > > > > > > implied
> > > > > > > by the commit message.
> > > > > > 
> > > > > > Well, the control is optional if the driver implements it as
> > > > > > optional,
> > > > > > which Cedrus isn't currently doing :-)
> > > > > 
> > > > > Why do you think so? Prediction weights are filled only when they are
> > > > > needed:https://elixir.bootlin.com/linux/latest/source/drivers/staging/
> > > > > medi
> > > > > a/ sunxi/cedrus/cedrus_h264.c#L370
> > > > 
> > > > Right, but that should be changed to be really optional.
> > > > How does the driver reject/fail the request if the table is NULL?
> > > 
> > > It's my understanding that pointer to this table can't be NULL. NULL would
> > > mean that there is no control with that ID registered in the driver.
> > 
> > Hm, I'm starting to think you are right. So, does this mean
> > the default quantization matrix here is bogus?
> > 
> >         if (quantization && quantization->load_intra_quantiser_matrix)
> >                 matrix = quantization->intra_quantiser_matrix;
> >         else
> >                 matrix = intra_quantization_matrix_default;
> 
> No, not really. Userspace can set load_intra_quantiser_matrix flag to false.
> 

The above made me revisit the current H264 semantics
for the picture scaling matrix.

As you can see, we currently have V4L2_H264_PPS_FLAG_PIC_SCALING_MATRIX_PRESENT,
which we were expecting to match 1:1 the H264 PPS syntax element
"pic_scaling_matrix_present_flag".

However, after a bit of reflection and discussion with Nicolas, I believe
it's not appropriate to have this flag as a 1:1 match with the PPS syntax element.

A H264 scaling matrix can be first specified by the SPS and then modified
by the PPS. We can expect the modification process to be solved by userspace.
All we need in the uAPI is a flag that indicates
if V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX should be used or not.

(As Jernej already pointed out, a initialized control shall never be NULL,
so we want to flag if the control should be used or not) [1].

Applications are expected to fill V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX
if a scaling matrix needs to be passed, which is the case is:

sps->scaling_matrix_present_flag || pps->pic_scaling_matrix_present_flag

So that is the meaning of the flag we want. [2]

Moreover, Baseline, Main and Extended profiles are specified to have
neither SPS scaling_matrix_present_flag nor PPS pic_scaling_matrix_present_flag
syntax elements, so it makes sense to allow applications _not_ setting
V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX in a request.

On the uAPI side, the only change needed is:

-#define V4L2_H264_PPS_FLAG_PIC_SCALING_MATRIX_PRESENT                  0x0080
+#define V4L2_H264_PPS_FLAG_SCALING_MATRIX_PRESENT                      0x0080

(just to avoid confusing the flag with the syntax element)

Together with proper documentation to clarify what the flag is.

Drivers can use this flag as (rkvdec as an example):

-       /* always use the matrix sent from userspace */
-       WRITE_PPS(1, SCALING_LIST_ENABLE_FLAG);
-
+       WRITE_PPS(!!(pps->flags & V4L2_H264_PPS_FLAG_SCALING_MATRIX_PRESENT),
+                 SCALING_LIST_ENABLE_FLAG);

Which also means the scaling matrix control is optional and won't be programmed
to the hardware when the not present. [3]

Thanks!
Ezequiel

[1] We may also check if a control is part of a request or not,
but that seems more complex and more obscure than just checking a flag.

[2] In theory, the uAPI could also have semantics to flag
seq_scaling_list_present_flag[i] and pic_scaling_list_present_flag[i],
for each scaling list. I think this makes things overly complicated.

[3] We could add Flat_4x4_16 and Flat_8x8_16 in the kernel,
but all the drivers we support, have a flag for scaling-matrix-not-present,
so there's no need.

Re: [PATCH v2 08/14] media: uapi: h264: Drop SLICE_PARAMS 'size' field

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 4818 bytes --]

Hi,

On Fri 07 Aug 20, 11:44, Ezequiel Garcia wrote:
> On Thu, 2020-08-06 at 17:50 +0200, Paul Kocialkowski wrote:
> > Hi,
> > 
> > On Thu 06 Aug 20, 12:13, Ezequiel Garcia wrote:
> > > The SLICE_PARAMS control is intended for slice-based
> > > devices. In this mode, the OUTPUT buffer contains
> > > a single slice, and so the buffer's plane payload size
> > > can be used to query the slice size.
> > 
> > If we later extend the API for supporting multiple slices with dynamic array
> > controls, I guess we'll need to know the size of each slice in each control
> > elements. So I'd rather keep that even if it's indeed redundant with
> > vb2_get_plane_payload in single-slice mode.
> > 
> 
> If we later extend the API, another control (possibly
> another decoding mode?) shall be introduced.
> 
> This API covers single-slice-per-request as specified
> and documented in patch 9/14 "Clarify SLICE_BASED mode".
> 
> This is along the lines of the proposal drafted by Nicolas,
> see my reply: https://lkml.org/lkml/2020/8/5/791.
> 
> This applies to num_slices, slice size and slice start offset.
> 
> There are multiple ways of doing this.

If feels a bit problematic to remove these fields without a clear plan yet
on how to support multiple slices in the future. These may need to be added
again later, except that it will be too late and new controls will need to be
introduced.

Also, could we consider adding more reserved fields to handle such future needs?

Cheers,

Paul

> Thanks!
> Ezequiel
> 
> > What do you think?
> > 
> > Paul
> > 
> > > To reduce the API surface drop the size from the
> > > SLICE_PARAMS control.
> > > 
> > > A follow-up change will remove other members in SLICE_PARAMS
> > > so we don't need to add padding fields here.
> > > 
> > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > ---
> > >  Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 3 ---
> > >  drivers/staging/media/sunxi/cedrus/cedrus_h264.c          | 7 +++----
> > >  include/media/h264-ctrls.h                                | 3 ---
> > >  3 files changed, 3 insertions(+), 10 deletions(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > index 427fc5727ec0..fff74b7bf32a 100644
> > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > @@ -1760,9 +1760,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
> > >      :stub-columns: 0
> > >      :widths:       1 1 2
> > >  
> > > -    * - __u32
> > > -      - ``size``
> > > -      -
> > >      * - __u32
> > >        - ``start_byte_offset``
> > >          Offset (in bytes) from the beginning of the OUTPUT buffer to the start
> > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > index a9ba78b15907..8b6f05aadbe8 100644
> > > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > @@ -324,17 +324,16 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
> > >  	struct vb2_buffer *src_buf = &run->src->vb2_buf;
> > >  	struct cedrus_dev *dev = ctx->dev;
> > >  	dma_addr_t src_buf_addr;
> > > -	u32 len = slice->size * 8;
> > > +	size_t slice_bytes = vb2_get_plane_payload(src_buf, 0);
> > >  	unsigned int pic_width_in_mbs;
> > >  	bool mbaff_pic;
> > >  	u32 reg;
> > >  
> > > -	cedrus_write(dev, VE_H264_VLD_LEN, len);
> > > +	cedrus_write(dev, VE_H264_VLD_LEN, slice_bytes * 8);
> > >  	cedrus_write(dev, VE_H264_VLD_OFFSET, 0);
> > >  
> > >  	src_buf_addr = vb2_dma_contig_plane_dma_addr(src_buf, 0);
> > > -	cedrus_write(dev, VE_H264_VLD_END,
> > > -		     src_buf_addr + vb2_get_plane_payload(src_buf, 0));
> > > +	cedrus_write(dev, VE_H264_VLD_END, src_buf_addr + slice_bytes);
> > >  	cedrus_write(dev, VE_H264_VLD_ADDR,
> > >  		     VE_H264_VLD_ADDR_VAL(src_buf_addr) |
> > >  		     VE_H264_VLD_ADDR_FIRST | VE_H264_VLD_ADDR_VALID |
> > > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > > index 4f05ee265997..f74736fcfa00 100644
> > > --- a/include/media/h264-ctrls.h
> > > +++ b/include/media/h264-ctrls.h
> > > @@ -158,9 +158,6 @@ struct v4l2_h264_reference {
> > >  };
> > >  
> > >  struct v4l2_ctrl_h264_slice_params {
> > > -	/* Size in bytes, including header */
> > > -	__u32 size;
> > > -
> > >  	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
> > >  	__u32 start_byte_offset;
> > >  
> > > -- 
> > > 2.27.0
> > > 
> 
> 

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 08/14] media: uapi: h264: Drop SLICE_PARAMS 'size' field

[Ezequiel Garcia]

Hi Paul,

On Wed, 2020-08-19 at 15:54 +0200, Paul Kocialkowski wrote:
> Hi,
> 
> On Fri 07 Aug 20, 11:44, Ezequiel Garcia wrote:
> > On Thu, 2020-08-06 at 17:50 +0200, Paul Kocialkowski wrote:
> > > Hi,
> > > 
> > > On Thu 06 Aug 20, 12:13, Ezequiel Garcia wrote:
> > > > The SLICE_PARAMS control is intended for slice-based
> > > > devices. In this mode, the OUTPUT buffer contains
> > > > a single slice, and so the buffer's plane payload size
> > > > can be used to query the slice size.
> > > 
> > > If we later extend the API for supporting multiple slices with dynamic array
> > > controls, I guess we'll need to know the size of each slice in each control
> > > elements. So I'd rather keep that even if it's indeed redundant with
> > > vb2_get_plane_payload in single-slice mode.
> > > 
> > 
> > If we later extend the API, another control (possibly
> > another decoding mode?) shall be introduced.
> > 
> > This API covers single-slice-per-request as specified
> > and documented in patch 9/14 "Clarify SLICE_BASED mode".
> > 
> > This is along the lines of the proposal drafted by Nicolas,
> > see my reply: https://lkml.org/lkml/2020/8/5/791.
> > 
> > This applies to num_slices, slice size and slice start offset.
> > 
> > There are multiple ways of doing this.
> 
> If feels a bit problematic to remove these fields without a clear plan yet
> on how to support multiple slices in the future. These may need to be added
> again later, except that it will be too late and new controls will need to be
> introduced.
> 

As Nicolas and I have repeatedly mentioned we do have a plan :)

Nothing prevents us from implementing this now, but since it seems
we don't have any need since the 1-slice-per-buffer is working well,
there is no motivation for it.

As you well mention, it does require new controls. This is
totally expected since new decoding semantics will need new controls.

Here's my version of the plan:

enum v4l2_mpeg_video_h264_decode_mode {
        V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED,
        V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_ARRAY_BASED,
        V4L2_MPEG_VIDEO_H264_DECODE_MODE_FRAME_BASED,
};

#define V4L2_CID_MPEG_VIDEO_H264_SLICE_ARRAY    (V4L2_CID_MPEG_BASE+1008)

struct v4l2_ctrl_h264_slice_array {
        __u16 num_slices;

        struct v4l2_ctrl_h264_slice slices[16];
}

struct v4l2_ctrl_h264_slice {
        __u32 size;
        __u32 start_byte_offset;
        struct v4l2_ctrl_h264_slice_params params;
}

Now, the _specific_ way this will be done is not under
discussion at the moment.

> Also, could we consider adding more reserved fields to handle such future needs?
> 

I have considered that (on each control), but I fail
to see the reason for it. The same may be said of all controls,
yet I don't think we bloat each one just in case. 

Thanks,
Ezequiel

> Cheers,
> 
> Paul
> 
> > Thanks!
> > Ezequiel
> > 
> > > What do you think?
> > > 
> > > Paul
> > > 
> > > > To reduce the API surface drop the size from the
> > > > SLICE_PARAMS control.
> > > > 
> > > > A follow-up change will remove other members in SLICE_PARAMS
> > > > so we don't need to add padding fields here.
> > > > 
> > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > ---
> > > >  Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 3 ---
> > > >  drivers/staging/media/sunxi/cedrus/cedrus_h264.c          | 7 +++----
> > > >  include/media/h264-ctrls.h                                | 3 ---
> > > >  3 files changed, 3 insertions(+), 10 deletions(-)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > index 427fc5727ec0..fff74b7bf32a 100644
> > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > @@ -1760,9 +1760,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
> > > >      :stub-columns: 0
> > > >      :widths:       1 1 2
> > > >  
> > > > -    * - __u32
> > > > -      - ``size``
> > > > -      -
> > > >      * - __u32
> > > >        - ``start_byte_offset``
> > > >          Offset (in bytes) from the beginning of the OUTPUT buffer to the start
> > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > > index a9ba78b15907..8b6f05aadbe8 100644
> > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > > @@ -324,17 +324,16 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
> > > >  	struct vb2_buffer *src_buf = &run->src->vb2_buf;
> > > >  	struct cedrus_dev *dev = ctx->dev;
> > > >  	dma_addr_t src_buf_addr;
> > > > -	u32 len = slice->size * 8;
> > > > +	size_t slice_bytes = vb2_get_plane_payload(src_buf, 0);
> > > >  	unsigned int pic_width_in_mbs;
> > > >  	bool mbaff_pic;
> > > >  	u32 reg;
> > > >  
> > > > -	cedrus_write(dev, VE_H264_VLD_LEN, len);
> > > > +	cedrus_write(dev, VE_H264_VLD_LEN, slice_bytes * 8);
> > > >  	cedrus_write(dev, VE_H264_VLD_OFFSET, 0);
> > > >  
> > > >  	src_buf_addr = vb2_dma_contig_plane_dma_addr(src_buf, 0);
> > > > -	cedrus_write(dev, VE_H264_VLD_END,
> > > > -		     src_buf_addr + vb2_get_plane_payload(src_buf, 0));
> > > > +	cedrus_write(dev, VE_H264_VLD_END, src_buf_addr + slice_bytes);
> > > >  	cedrus_write(dev, VE_H264_VLD_ADDR,
> > > >  		     VE_H264_VLD_ADDR_VAL(src_buf_addr) |
> > > >  		     VE_H264_VLD_ADDR_FIRST | VE_H264_VLD_ADDR_VALID |
> > > > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > > > index 4f05ee265997..f74736fcfa00 100644
> > > > --- a/include/media/h264-ctrls.h
> > > > +++ b/include/media/h264-ctrls.h
> > > > @@ -158,9 +158,6 @@ struct v4l2_h264_reference {
> > > >  };
> > > >  
> > > >  struct v4l2_ctrl_h264_slice_params {
> > > > -	/* Size in bytes, including header */
> > > > -	__u32 size;
> > > > -
> > > >  	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
> > > >  	__u32 start_byte_offset;
> > > >  
> > > > -- 
> > > > 2.27.0
> > > > 

Re: [PATCH v2 08/14] media: uapi: h264: Drop SLICE_PARAMS 'size' field

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 6829 bytes --]

Le jeudi 20 août 2020 à 04:32 -0300, Ezequiel Garcia a écrit :
> Hi Paul,
> 
> On Wed, 2020-08-19 at 15:54 +0200, Paul Kocialkowski wrote:
> > Hi,
> > 
> > On Fri 07 Aug 20, 11:44, Ezequiel Garcia wrote:
> > > On Thu, 2020-08-06 at 17:50 +0200, Paul Kocialkowski wrote:
> > > > Hi,
> > > > 
> > > > On Thu 06 Aug 20, 12:13, Ezequiel Garcia wrote:
> > > > > The SLICE_PARAMS control is intended for slice-based
> > > > > devices. In this mode, the OUTPUT buffer contains
> > > > > a single slice, and so the buffer's plane payload size
> > > > > can be used to query the slice size.
> > > > 
> > > > If we later extend the API for supporting multiple slices with dynamic array
> > > > controls, I guess we'll need to know the size of each slice in each control
> > > > elements. So I'd rather keep that even if it's indeed redundant with
> > > > vb2_get_plane_payload in single-slice mode.
> > > > 
> > > 
> > > If we later extend the API, another control (possibly
> > > another decoding mode?) shall be introduced.
> > > 
> > > This API covers single-slice-per-request as specified
> > > and documented in patch 9/14 "Clarify SLICE_BASED mode".
> > > 
> > > This is along the lines of the proposal drafted by Nicolas,
> > > see my reply: https://lkml.org/lkml/2020/8/5/791.
> > > 
> > > This applies to num_slices, slice size and slice start offset.
> > > 
> > > There are multiple ways of doing this.
> > 
> > If feels a bit problematic to remove these fields without a clear plan yet
> > on how to support multiple slices in the future. These may need to be added
> > again later, except that it will be too late and new controls will need to be
> > introduced.
> > 
> 
> As Nicolas and I have repeatedly mentioned we do have a plan :)
> 
> Nothing prevents us from implementing this now, but since it seems
> we don't have any need since the 1-slice-per-buffer is working well,
> there is no motivation for it.
> 
> As you well mention, it does require new controls. This is
> totally expected since new decoding semantics will need new controls.
> 
> Here's my version of the plan:
> 
> enum v4l2_mpeg_video_h264_decode_mode {
>         V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_BASED,
>         V4L2_MPEG_VIDEO_H264_DECODE_MODE_SLICE_ARRAY_BASED,
>         V4L2_MPEG_VIDEO_H264_DECODE_MODE_FRAME_BASED,
> };
> 
> #define V4L2_CID_MPEG_VIDEO_H264_SLICE_ARRAY    (V4L2_CID_MPEG_BASE+1008)
> 
> struct v4l2_ctrl_h264_slice_array {
>         __u16 num_slices;
> 
>         struct v4l2_ctrl_h264_slice slices[16];
> }

Well, ideally the new dynamic array control format can hold this
information for us, avoiding the 16 slice limit again, but that is
inded the plan.

> 
> struct v4l2_ctrl_h264_slice {
>         __u32 size;
>         __u32 start_byte_offset;
>         struct v4l2_ctrl_h264_slice_params params;
> }
> 
> Now, the _specific_ way this will be done is not under
> discussion at the moment.
> 
> > Also, could we consider adding more reserved fields to handle such future needs?
> > 
> 
> I have considered that (on each control), but I fail
> to see the reason for it. The same may be said of all controls,
> yet I don't think we bloat each one just in case. 
> 
> Thanks,
> Ezequiel
> 
> > Cheers,
> > 
> > Paul
> > 
> > > Thanks!
> > > Ezequiel
> > > 
> > > > What do you think?
> > > > 
> > > > Paul
> > > > 
> > > > > To reduce the API surface drop the size from the
> > > > > SLICE_PARAMS control.
> > > > > 
> > > > > A follow-up change will remove other members in SLICE_PARAMS
> > > > > so we don't need to add padding fields here.
> > > > > 
> > > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > ---
> > > > >  Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 3 ---
> > > > >  drivers/staging/media/sunxi/cedrus/cedrus_h264.c          | 7 +++----
> > > > >  include/media/h264-ctrls.h                                | 3 ---
> > > > >  3 files changed, 3 insertions(+), 10 deletions(-)
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > index 427fc5727ec0..fff74b7bf32a 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> > > > > @@ -1760,9 +1760,6 @@ enum v4l2_mpeg_video_h264_hierarchical_coding_type -
> > > > >      :stub-columns: 0
> > > > >      :widths:       1 1 2
> > > > >  
> > > > > -    * - __u32
> > > > > -      - ``size``
> > > > > -      -
> > > > >      * - __u32
> > > > >        - ``start_byte_offset``
> > > > >          Offset (in bytes) from the beginning of the OUTPUT buffer to the start
> > > > > diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > > > index a9ba78b15907..8b6f05aadbe8 100644
> > > > > --- a/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > > > +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h264.c
> > > > > @@ -324,17 +324,16 @@ static void cedrus_set_params(struct cedrus_ctx *ctx,
> > > > >  	struct vb2_buffer *src_buf = &run->src->vb2_buf;
> > > > >  	struct cedrus_dev *dev = ctx->dev;
> > > > >  	dma_addr_t src_buf_addr;
> > > > > -	u32 len = slice->size * 8;
> > > > > +	size_t slice_bytes = vb2_get_plane_payload(src_buf, 0);
> > > > >  	unsigned int pic_width_in_mbs;
> > > > >  	bool mbaff_pic;
> > > > >  	u32 reg;
> > > > >  
> > > > > -	cedrus_write(dev, VE_H264_VLD_LEN, len);
> > > > > +	cedrus_write(dev, VE_H264_VLD_LEN, slice_bytes * 8);
> > > > >  	cedrus_write(dev, VE_H264_VLD_OFFSET, 0);
> > > > >  
> > > > >  	src_buf_addr = vb2_dma_contig_plane_dma_addr(src_buf, 0);
> > > > > -	cedrus_write(dev, VE_H264_VLD_END,
> > > > > -		     src_buf_addr + vb2_get_plane_payload(src_buf, 0));
> > > > > +	cedrus_write(dev, VE_H264_VLD_END, src_buf_addr + slice_bytes);
> > > > >  	cedrus_write(dev, VE_H264_VLD_ADDR,
> > > > >  		     VE_H264_VLD_ADDR_VAL(src_buf_addr) |
> > > > >  		     VE_H264_VLD_ADDR_FIRST | VE_H264_VLD_ADDR_VALID |
> > > > > diff --git a/include/media/h264-ctrls.h b/include/media/h264-ctrls.h
> > > > > index 4f05ee265997..f74736fcfa00 100644
> > > > > --- a/include/media/h264-ctrls.h
> > > > > +++ b/include/media/h264-ctrls.h
> > > > > @@ -158,9 +158,6 @@ struct v4l2_h264_reference {
> > > > >  };
> > > > >  
> > > > >  struct v4l2_ctrl_h264_slice_params {
> > > > > -	/* Size in bytes, including header */
> > > > > -	__u32 size;
> > > > > -
> > > > >  	/* Offset in bytes to the start of slice in the OUTPUT buffer. */
> > > > >  	__u32 start_byte_offset;
> > > > >  
> > > > > -- 
> > > > > 2.27.0
> > > > > 
> 
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]