From: Tomasz Figa <tfiga@chromium.org>
To: Stanimir Varbanov <stanimir.varbanov@linaro.org>
Cc: Hans Verkuil <hverkuil-cisco@xs4all.nl>,
Linux Media Mailing List <linux-media@vger.kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Philipp Zabel <p.zabel@pengutronix.de>,
Ezequiel Garcia <ezequiel@collabora.com>,
Paul Kocialkowski <paul.kocialkowski@bootlin.com>,
Nicolas Dufresne <nicolas@ndufresne.ca>
Subject: Re: [PATCH] v4l2-ctrl: add control for thumnails
Date: Tue, 13 Oct 2020 15:12:46 +0200 [thread overview]
Message-ID: <CAAFQd5A7MdFXsYWVx+3DY-dt=HZTg_Kt1uhntoEzzjp+e02R=Q@mail.gmail.com> (raw)
In-Reply-To: <de40cdf1-0b84-a252-0137-a550f078d28b@linaro.org>
On Tue, Oct 13, 2020 at 2:52 PM Stanimir Varbanov
<stanimir.varbanov@linaro.org> wrote:
>
> Hi,
>
> On 6/4/20 3:57 PM, Tomasz Figa wrote:
> > On Thu, Jun 4, 2020 at 2:56 PM Hans Verkuil <hverkuil-cisco@xs4all.nl> wrote:
> >>
> >> On 04/06/2020 14:34, Stanimir Varbanov wrote:
> >>> Hi Hans,
> >>>
> >>> On 6/4/20 12:08 PM, Hans Verkuil wrote:
> >>>> On 04/06/2020 11:02, Stanimir Varbanov wrote:
> >>>>> Hi Hans,
> >>>>>
> >>>>> On 5/27/20 12:53 AM, Stanimir Varbanov wrote:
> >>>>>> Hi Hans,
> >>>>>>
> >>>>>> On 5/26/20 3:04 PM, Hans Verkuil wrote:
> >>>>>>> On 26/05/2020 10:54, Stanimir Varbanov wrote:
> >>>>>>>> Add v4l2 control for decoder thumbnail.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
> >>>>>>>> ---
> >>>>>>>> Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 7 +++++++
> >>>>>>>> drivers/media/v4l2-core/v4l2-ctrls.c | 2 ++
> >>>>>>>> include/uapi/linux/v4l2-controls.h | 2 ++
> >>>>>>>> 3 files changed, 11 insertions(+)
> >>>>>>>>
> >>>>>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> >>>>>>>> index d0d506a444b1..e838e410651b 100644
> >>>>>>>> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> >>>>>>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
> >>>>>>>> @@ -3726,6 +3726,13 @@ enum v4l2_mpeg_video_hevc_size_of_length_field -
> >>>>>>>> disables generating SPS and PPS at every IDR. Setting it to one enables
> >>>>>>>> generating SPS and PPS at every IDR.
> >>>>>>>>
> >>>>>>>> +``V4L2_CID_MPEG_VIDEO_DECODER_THUMBNAIL (button)``
> >>>>>>>> + Instructs the decoder to produce immediate output. The decoder should
> >>>>>>>> + consume first input buffer for progressive stream (or first two buffers
> >>>>>>>> + for interlace). Decoder should not allocate more output buffers that it
> >>>>>>>> + is required to consume one input frame. Usually the decoder input
> >>>>>>>> + buffers will contain only I/IDR frames but it is not mandatory.
> >>>>>>>
> >>>>>>> This is very vague. It doesn't explain why the control is called 'THUMBNAIL',
> >>>>>>> but more importantly it doesn't explain how this relates to normal decoding.
> >>>>>>
> >>>>>> If in the normal decode the capture queue buffers are 5, in the
> >>>>>> thumbnail mode the number of buffers will be only 1 (if the bitstream is
> >>>>>> progressive) and this will guarantee low memory usage. The other
> >>>>>> difference is that the decoder will produce decoded frames (without
> >>>>>> errors) only for I/IDR (sync frames).
> >>>>
> >>>> Isn't this really a "DECODE_SYNC_FRAMES_ONLY" control? That's what it does,
> >>>> right? Skip any B/P frames and only decode sync frames.
> >>>
> >>> Yes, it is.
> >>> To me V4L2_CID_MPEG_VIDEO_DECODE_SYNC_FRAMES sounds better. If you are
> >>> fine I can send a new patch.
> >>>
> >>> The definition of "sync frames" is a bit difficult for codec-agnostic
> >>> controls. Is it sound better "INTRA", DECODE_INTRA_FRAMES (ONLY)?
> >>
> >> INTRA is better. DECODE_INTRA_FRAMES_ONLY is a good name, I think.
> >>
> >> Thumbnail creation can be given as an example in the description of the
> >> control, but that's just a use-case.
> >
> > How about the other aspect of the behavior?
> >
> > "Decoder should not allocate more output buffers that it
> > is required to consume one input frame."
> >
>
> In fact I have to refine this; It looks like that picture type decode vs
> thumbnail are two different modes.
>
> Thumbnail mode - first frame decode without additional memory (one input
> buffer and one output buffer). The first frame can be even non-intra
> frame as well.
How comes it can be decoded without additional memory to store the
reference frames then?
> Also no matter frame parser is sending, the decoder will
> try to produce output for thumbnail generation.
Well, thinking of it now, actually DECODE_INTRA_FRAMES_ONLY makes
sense here - if it is set, only 1 CAPTURE buffer could be allowed
indeed, because no reference frames are needed for decoding.
If inter frames are needed, I believe full DPB needs to be allocated,
because it all depends on the stream how the references are set, so
this is equivalent to normal decoding.
best regards,
Tomasz
>
> > Best regards,
> > Tomasz
> >
> >>
> >> Regards,
> >>
> >> Hans
> >>
> >>>
> >>>>
> >>>> That this is useful for creating thumbnails is just a specific use-case.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Hans
> >>>>
> >>>>>>
> >>>>>>>
> >>>>>>> I.e. if you are decoding and 'press' this control, what happens then?
> >>>>>>
> >>>>>> Might be the button type wasn't great idea. In fact the control should
> >>>>>> be set before streamon so that the driver returns min_capture_bufs 1.
> >>>>>>
> >>>>>>>
> >>>>>>> What exactly is the use-case?
> >>>>>>
> >>>>>> It could be used to generate thumbnails of all video clips in a folder
> >>>>>> or when you open a Gallery application on your phone.
> >>>>>>
> >>>>>
> >>>>> What is your opinion on that control? I could consider to make it Venus
> >>>>> custom control but from the use-case it looks other drivers also can
> >>>>> benefit of it.
> >>>>>
> >>>>> I tried to make more generic one [1] but it looks it will be too difficult.
> >>>>>
> >>>>
> >>>
> >>
>
> --
> regards,
> Stan
prev parent reply other threads:[~2020-10-13 13:13 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-26 8:54 [PATCH] v4l2-ctrl: add control for thumnails Stanimir Varbanov
2020-05-26 12:04 ` Hans Verkuil
2020-05-26 21:53 ` Stanimir Varbanov
2020-06-04 9:02 ` Stanimir Varbanov
2020-06-04 9:08 ` Hans Verkuil
2020-06-04 12:34 ` Stanimir Varbanov
2020-06-04 12:56 ` Hans Verkuil
2020-06-04 12:57 ` Tomasz Figa
2020-10-13 12:52 ` Stanimir Varbanov
2020-10-13 13:12 ` Tomasz Figa [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAAFQd5A7MdFXsYWVx+3DY-dt=HZTg_Kt1uhntoEzzjp+e02R=Q@mail.gmail.com' \
--to=tfiga@chromium.org \
--cc=ezequiel@collabora.com \
--cc=hverkuil-cisco@xs4all.nl \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=nicolas@ndufresne.ca \
--cc=p.zabel@pengutronix.de \
--cc=paul.kocialkowski@bootlin.com \
--cc=stanimir.varbanov@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).