From: Nicolas Dufresne <nicolas.dufresne@collabora.com>
To: unlisted-recipients:; (no To-header on input)
Cc: nicolas@ndufresne.ca, linux-media@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: [PATCH v4 00/24] H.264 Field Decoding Support for Frame-based Decoders
Date: Tue, 26 Apr 2022 08:57:26 -0400 [thread overview]
Message-ID: <20220426125751.108293-1-nicolas.dufresne@collabora.com> (raw)
Until now, only Cedrus (a slice base decoder) supported interlaced
decoding. In order to support field decoding in our frame-based decoder,
the v4l2-h264 library needed adaptation to produce the appropriate
reference lists.
This patch extends the v4l2-h264 library to produce the larger references
list needed to represent fields separately. Hantro, MTK-VCODEC and RKVDEC
drivers have been adapted to accommodate the larger lists. Though, only
Hantro and RKVDEC actually have HW support for field decoding. So only
these two have been updated to make use of the larger lists. All this work
has been done using the H.264 specification, LibreELEC downstream kernel
patches, Rockchip MPP reference software and Hantro reference software.
For reviewers, the following is the map of all commit. Patches that could
be merge independently of this serie are marked as independent. Note that
the test results do depend on the generic fixes.
01 : Documentation fix (independent)
02-03 : Improving some generic traces (independent)
04 : Minor v4l2-h264 fix (independent)
05-11 : v4l2-h264 field decoding support
12-16 : rkvdec h.264 generic fixes (independent)
17-20 : rkvdec h.264 field decoding support
21-24 : hantro h.264 field decoding support
All this work have been tested using GStreamer mainline implementation
but also with FFMPEG LibreELEC fork using the testing tool fluster
running through the ITU-T H.264 (2016-02) AVCv2 set of bitstream. Before
this patch, the scores were:
Hantro:
FFMPEG: 88/135
GSteamer: 90/135
RKVDEC:
FFMPEG: 73/135
GSteamer: 77/135
And after these changes:
Hantro:
FFMPEG: 118/135
GSteamer: 129/135
RKVDEC:
FFMPEG: 118/135
GSteamer: 129/135
Note that a bug in FFMPEG / LibreELEC fork was noticed and fixed with the
following change:
diff --git a/libavcodec/v4l2_request_h264.c b/libavcodec/v4l2_request_h264.c
index 88da8f0a2d..394bae0550 100644
--- a/libavcodec/v4l2_request_h264.c
+++ b/libavcodec/v4l2_request_h264.c
@@ -66,7 +66,7 @@ static void fill_dpb_entry(struct v4l2_h264_dpb_entry *entry, const H264Picture
{
entry->reference_ts = ff_v4l2_request_get_capture_timestamp(pic->f);
entry->pic_num = pic->pic_id;
- entry->frame_num = pic->frame_num;
+ entry->frame_num = pic->long_ref ? pic->pic_id : pic->frame_num;
entry->fields = pic->reference & V4L2_H264_FRAME_REF;
entry->flags = V4L2_H264_DPB_ENTRY_FLAG_VALID;
if (entry->fields)
Some useful links:
Detailed Hantro Results: https://gitlab.freedesktop.org/-/snippets/5189
Detailed RKVDEC Results: https://gitlab.freedesktop.org/-/snippets/5253
ITU-T H.264 (2016-02) AVCv2: https://www.itu.int/net/itu-t/sigdb/spevideo/VideoForm-s.aspx?val=102002641
Fluster: https://github.com/fluendo/fluster
GStreamer: https://gitlab.freedesktop.org/gstreamer/gstreamer/
FFMPEG Fork: https://github.com/jernejsk/FFmpeg/tree/v4l2-request-hwaccel-4.4
Rockchip MPP: https://github.com/rockchip-linux/mpp
Changes in v4:
- Fixed mtk-vcodec port
- Ported tegra-vde driver (compiled tested only)
- Applied Hans' commit message, comment and documentation suggestions
Changes in v3:
- Improved debug message on timestamp miss-match
- Moved H264 SPS validation into rkvdec-h264
- Added more comments around H264 SPS validation
- Also validate at streamon (rkvdec start())
- Applied more Review-by and Fixes tag
- Fixed Signed-off-by chain in Jonas patch
Changes in v2:
- Applied most of Sebastian's suggestion in comments and commit messages.
- Use a bool for dpb_valid and dpb_bottom in rkvdec
- Dropped one wrong typo fix (media: v4l2-mem2mem: Fix typo in trace message)
- Dropped Alex fix (media: rkvdec-h264: Don't hardcode SPS/PPS parameters
+ I will carry this one later, it seems cosmetic
Jonas Karlman (5):
media: rkvdec: h264: Fix bit depth wrap in pps packet
media: rkvdec: h264: Validate and use pic width and height in mbs
media: rkvdec: h264: Fix reference frame_num wrap for second field
media: rkvdec: Ensure decoded resolution fit coded resolution
media: hantro: h264: Make dpb entry management more robust
Nicolas Dufresne (18):
media: doc: Document dual use of H.264 pic_num/frame_num
media: v4l2-mem2mem: Trace on implicit un-hold
media: h264: Avoid wrapping long_term_frame_idx
media: h264: Use v4l2_h264_reference for reflist
media: h264: Increase reference lists size to 32
media: h264: Store current picture fields
media: h264: Store all fields into the unordered list
media: v4l2: Trace calculated p/b0/b1 initial reflist
media: h264: Sort p/b reflist using frame_num
media: v4l2: Reorder field reflist
media: rkvdec: Stop overclocking the decoder
media: rkvdec: h264: Fix dpb_valid implementation
media: rkvdec: Move H264 SPS validation in rkvdec-h264
media: rkvdec-h264: Add field decoding support
media: rkvdec: Enable capture buffer holding for H264
media: hantro: Stop using H.264 parameter pic_num
media: hantro: Add H.264 field decoding support
media: hantro: Enable HOLD_CAPTURE_BUF for H.264
Sebastian Fricke (1):
media: videobuf2-v4l2: Warn on holding buffers without support
.../media/v4l/ext-ctrls-codec-stateless.rst | 10 +-
.../media/common/videobuf2/videobuf2-v4l2.c | 7 +-
.../mediatek/vcodec/vdec/vdec_h264_req_if.c | 36 ++-
.../media/platform/nvidia/tegra-vde/h264.c | 19 +-
drivers/media/v4l2-core/v4l2-h264.c | 275 +++++++++++++++---
drivers/media/v4l2-core/v4l2-mem2mem.c | 1 +
.../staging/media/hantro/hantro_g1_h264_dec.c | 38 +--
drivers/staging/media/hantro/hantro_h264.c | 134 +++++++--
drivers/staging/media/hantro/hantro_hw.h | 7 +-
drivers/staging/media/hantro/hantro_v4l2.c | 25 ++
.../media/hantro/rockchip_vpu2_hw_h264_dec.c | 98 +++----
drivers/staging/media/rkvdec/rkvdec-h264.c | 157 +++++++---
drivers/staging/media/rkvdec/rkvdec.c | 35 +--
drivers/staging/media/rkvdec/rkvdec.h | 2 +
include/media/v4l2-h264.h | 31 +-
15 files changed, 639 insertions(+), 236 deletions(-)
--
2.34.1
next reply other threads:[~2022-04-26 12:58 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-26 12:57 Nicolas Dufresne [this message]
2022-04-26 12:57 ` [PATCH v4 01/24] media: doc: Document dual use of H.264 pic_num/frame_num Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 02/24] media: v4l2-mem2mem: Trace on implicit un-hold Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 03/24] media: videobuf2-v4l2: Warn on holding buffers without support Nicolas Dufresne
2022-04-27 4:31 ` Tomasz Figa
2022-04-27 15:08 ` Nicolas Dufresne
2022-04-28 6:12 ` Hans Verkuil
2022-04-28 13:09 ` Nicolas Dufresne
2022-11-24 8:57 ` Hans Verkuil
2022-04-26 12:57 ` [PATCH v4 04/24] media: h264: Avoid wrapping long_term_frame_idx Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 05/24] media: h264: Use v4l2_h264_reference for reflist Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-05-05 0:42 ` Dmitry Osipenko
2022-05-05 0:42 ` Dmitry Osipenko
2022-05-05 0:42 ` Dmitry Osipenko
2022-05-05 0:42 ` Dmitry Osipenko
2022-04-26 12:57 ` [PATCH v4 06/24] media: h264: Increase reference lists size to 32 Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 07/24] media: h264: Store current picture fields Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 08/24] media: h264: Store all fields into the unordered list Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 09/24] media: v4l2: Trace calculated p/b0/b1 initial reflist Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 10/24] media: h264: Sort p/b reflist using frame_num Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 11/24] media: v4l2: Reorder field reflist Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 12/24] media: rkvdec: Stop overclocking the decoder Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 13/24] media: rkvdec: h264: Fix dpb_valid implementation Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 14/24] media: rkvdec: h264: Fix bit depth wrap in pps packet Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 15/24] media: rkvdec: Move H264 SPS validation in rkvdec-h264 Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 16/24] media: rkvdec: h264: Validate and use pic width and height in mbs Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 17/24] media: rkvdec: h264: Fix reference frame_num wrap for second field Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 18/24] media: rkvdec: Ensure decoded resolution fit coded resolution Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 19/24] media: rkvdec-h264: Add field decoding support Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 20/24] media: rkvdec: Enable capture buffer holding for H264 Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 21/24] media: hantro: Stop using H.264 parameter pic_num Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 22/24] media: hantro: h264: Make dpb entry management more robust Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 23/24] media: hantro: Add H.264 field decoding support Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
2022-04-28 8:50 ` Hans Verkuil
2022-04-28 8:50 ` Hans Verkuil
2022-04-28 13:10 ` Nicolas Dufresne
2022-04-28 13:10 ` Nicolas Dufresne
2022-04-26 12:57 ` [PATCH v4 24/24] media: hantro: Enable HOLD_CAPTURE_BUF for H.264 Nicolas Dufresne
2022-04-26 12:57 ` Nicolas Dufresne
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220426125751.108293-1-nicolas.dufresne@collabora.com \
--to=nicolas.dufresne@collabora.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=nicolas@ndufresne.ca \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.