decode sync frames only

decode sync frames only

[Stanimir Varbanov]

Hi,

I need to port a decoder v4l2 control in mainline Venus driver which
instructs the decoder to decode sync frames only (I frame/ IDR frame).
In practice the usage of such control is to generate thumbnails for
particular video.

To do that I researched what we have currently in v4l2-controls.h and
found something similar but for encoders:

V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME
"Force a key frame for the next queued buffer. Applicable to encoders.
This is a general, codec-agnostic keyframe control."

I think I have two options:

1. reuse V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME for the decoder and document it

2. create a new v4l control V4L2_CID_MPEG_VIDC_VIDEO_SYNC_FRAME_DECODE

Any suggestions?

-- 
regards,
Stan

Re: decode sync frames only

[Hans Verkuil]

On 20/04/2020 16:38, Stanimir Varbanov wrote:
> Hi,
> 
> I need to port a decoder v4l2 control in mainline Venus driver which
> instructs the decoder to decode sync frames only (I frame/ IDR frame).
> In practice the usage of such control is to generate thumbnails for
> particular video.
> 
> To do that I researched what we have currently in v4l2-controls.h and
> found something similar but for encoders:
> 
> V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME
> "Force a key frame for the next queued buffer. Applicable to encoders.
> This is a general, codec-agnostic keyframe control."
> 
> I think I have two options:
> 
> 1. reuse V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME for the decoder and document it
> 
> 2. create a new v4l control V4L2_CID_MPEG_VIDC_VIDEO_SYNC_FRAME_DECODE

Make a new v4l control. The encoder control is for something quite different.

How about V4L2_CID_MPEG_VIDEO_DEC_KEY_FRAMES_ONLY?

Regards,

	Hans

Re: decode sync frames only

[Stanimir Varbanov]

Hi Hans,

On 4/20/20 5:48 PM, Hans Verkuil wrote:
> On 20/04/2020 16:38, Stanimir Varbanov wrote:
>> Hi,
>>
>> I need to port a decoder v4l2 control in mainline Venus driver which
>> instructs the decoder to decode sync frames only (I frame/ IDR frame).
>> In practice the usage of such control is to generate thumbnails for
>> particular video.
>>
>> To do that I researched what we have currently in v4l2-controls.h and
>> found something similar but for encoders:
>>
>> V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME
>> "Force a key frame for the next queued buffer. Applicable to encoders.
>> This is a general, codec-agnostic keyframe control."
>>
>> I think I have two options:
>>
>> 1. reuse V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME for the decoder and document it
>>
>> 2. create a new v4l control V4L2_CID_MPEG_VIDC_VIDEO_SYNC_FRAME_DECODE
> 
> Make a new v4l control. The encoder control is for something quite different.
> 
> How about V4L2_CID_MPEG_VIDEO_DEC_KEY_FRAMES_ONLY?

What you mean by "_DEC_" in the name - DECODER or DECODE?

I've found
V4L2_CID_MPEG_VIDEO_DECODER_SLICE_INTERFACE and
V4L2_CID_MPEG_VIDEO_DECODER_MPEG4_DEBLOCK_FILTER

in v4l2-controls.h which use the whole word "DECODER", thus I wonder
what is proffered word to follow the v4l2-controls.h naming style.

-- 
regards,
Stan

Re: decode sync frames only

[Hans Verkuil]

On 20/04/2020 17:39, Stanimir Varbanov wrote:
> Hi Hans,
> 
> On 4/20/20 5:48 PM, Hans Verkuil wrote:
>> On 20/04/2020 16:38, Stanimir Varbanov wrote:
>>> Hi,
>>>
>>> I need to port a decoder v4l2 control in mainline Venus driver which
>>> instructs the decoder to decode sync frames only (I frame/ IDR frame).
>>> In practice the usage of such control is to generate thumbnails for
>>> particular video.
>>>
>>> To do that I researched what we have currently in v4l2-controls.h and
>>> found something similar but for encoders:
>>>
>>> V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME
>>> "Force a key frame for the next queued buffer. Applicable to encoders.
>>> This is a general, codec-agnostic keyframe control."
>>>
>>> I think I have two options:
>>>
>>> 1. reuse V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME for the decoder and document it
>>>
>>> 2. create a new v4l control V4L2_CID_MPEG_VIDC_VIDEO_SYNC_FRAME_DECODE
>>
>> Make a new v4l control. The encoder control is for something quite different.
>>
>> How about V4L2_CID_MPEG_VIDEO_DEC_KEY_FRAMES_ONLY?
> 
> What you mean by "_DEC_" in the name - DECODER or DECODE?

DECODER

> 
> I've found
> V4L2_CID_MPEG_VIDEO_DECODER_SLICE_INTERFACE and
> V4L2_CID_MPEG_VIDEO_DECODER_MPEG4_DEBLOCK_FILTER
> 
> in v4l2-controls.h which use the whole word "DECODER", thus I wonder
> what is proffered word to follow the v4l2-controls.h naming style.
> 

It's a bit long, but DECODER is unambiguous, so let's use that.

Regards,

	Hans

Re: decode sync frames only

[Nicolas Dufresne]

Hi Stanimir,

Le lundi 20 avril 2020 à 17:38 +0300, Stanimir Varbanov a écrit :
> Hi,
> 
> I need to port a decoder v4l2 control in mainline Venus driver which
> instructs the decoder to decode sync frames only (I frame/ IDR frame).
> In practice the usage of such control is to generate thumbnails for
> particular video.
> 
> To do that I researched what we have currently in v4l2-controls.h and
> found something similar but for encoders:
> 
> V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME
> "Force a key frame for the next queued buffer. Applicable to encoders.
> This is a general, codec-agnostic keyframe control."
> 
> I think I have two options:
> 
> 1. reuse V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME for the decoder and document it
> 
> 2. create a new v4l control V4L2_CID_MPEG_VIDC_VIDEO_SYNC_FRAME_DECODE

Seems analogue to GStreamer flag GST_SEEK_FLAG_TRICKMODE_KEY_UNIT, if that helps
your inspiration. It's used to speed up fast-forward (ore thumbnails
generation). What the decoder will effectively do is to skip the P and the B
frames. Some decoder offers fine grain control on that.

That being said, in GStreamer (and probably most framework) frames are skipped
in userspace. But decoder tend to wait for the next key frame to output the
first one, so we endup having to do more work to force the frame out (like
CMD_STOP/START cycle, or crafts an early AUD). Would be nice to document if that
mode will help in this regard (or not). 

For protocols like DASH, when doing fast-forward we don't even download the non-
key-frames from the server, so the skipping behaviour does not provide any gain
any way.

> 
> Any suggestions?
> 

Could be as Hans suggested, or something less generic but that covers more
trickmodes ?

V4L2_CID_MPEG_VIDEO_DECODER_H264_TRICKMODE
  - No skip
  - Skip B Frames
  - I frame only

Or maybe try and construct something base on what other do ?

FFMPEG software decoder has the following:
  (0): Skip nothing     - 0
  (1): Skip B-frames    - 1
  (2): Skip IDCT/Dequantization - 2
  (5): Skip everything  - 5

CODA from Chips&Media is fancier, and propose some variants that have
implication how muchs state is maintained, hence allowing or not the transition
back to non-skipping outside IRAP boundary.
  NORMAL
  THUMBNAIL
  Skip nonIRAP
  Skip nonREF-PIC
  Skip Temporal Layer

I didn't find such feature for Samsung MFC, but Amlogic is likely one candidate
having this type of skipping feature (as it also has HW demuxers). Hope this
helps getting the big picture, hence proposing uAPI that isn't too limited.

regards,
Nicolas

Re: decode sync frames only

[Stanimir Varbanov]

Hi Nicolas,

On 4/21/20 1:09 AM, Nicolas Dufresne wrote:
> Hi Stanimir,
> 
> Le lundi 20 avril 2020 à 17:38 +0300, Stanimir Varbanov a écrit :
>> Hi,
>>
>> I need to port a decoder v4l2 control in mainline Venus driver which
>> instructs the decoder to decode sync frames only (I frame/ IDR frame).
>> In practice the usage of such control is to generate thumbnails for
>> particular video.
>>
>> To do that I researched what we have currently in v4l2-controls.h and
>> found something similar but for encoders:
>>
>> V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME
>> "Force a key frame for the next queued buffer. Applicable to encoders.
>> This is a general, codec-agnostic keyframe control."
>>
>> I think I have two options:
>>
>> 1. reuse V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME for the decoder and document it
>>
>> 2. create a new v4l control V4L2_CID_MPEG_VIDC_VIDEO_SYNC_FRAME_DECODE
> 
> Seems analogue to GStreamer flag GST_SEEK_FLAG_TRICKMODE_KEY_UNIT, if that helps
> your inspiration. It's used to speed up fast-forward (ore thumbnails
> generation). What the decoder will effectively do is to skip the P and the B
> frames. Some decoder offers fine grain control on that.
> 
> That being said, in GStreamer (and probably most framework) frames are skipped
> in userspace. But decoder tend to wait for the next key frame to output the
> first one, so we endup having to do more work to force the frame out (like
> CMD_STOP/START cycle, or crafts an early AUD). Would be nice to document if that
> mode will help in this regard (or not). 
> 
> For protocols like DASH, when doing fast-forward we don't even download the non-
> key-frames from the server, so the skipping behaviour does not provide any gain
> any way.
> 
>>
>> Any suggestions?
>>
> 
> Could be as Hans suggested, or something less generic but that covers more
> trickmodes ?
> 
> V4L2_CID_MPEG_VIDEO_DECODER_H264_TRICKMODE
>   - No skip
>   - Skip B Frames
>   - I frame only
> 
> Or maybe try and construct something base on what other do ?
> 
> FFMPEG software decoder has the following:
>   (0): Skip nothing     - 0
>   (1): Skip B-frames    - 1
>   (2): Skip IDCT/Dequantization - 2
>   (5): Skip everything  - 5
> 
> CODA from Chips&Media is fancier, and propose some variants that have
> implication how muchs state is maintained, hence allowing or not the transition
> back to non-skipping outside IRAP boundary.
>   NORMAL
>   THUMBNAIL
>   Skip nonIRAP
>   Skip nonREF-PIC
>   Skip Temporal Layer
> 
> I didn't find such feature for Samsung MFC, but Amlogic is likely one candidate
> having this type of skipping feature (as it also has HW demuxers). Hope this
> helps getting the big picture, hence proposing uAPI that isn't too limited.
> 
> regards,
> Nicolas
> 

I like your idea to have more generic one and codec-agnostic. I guess
something like below menu control:

V4L2_CID_MPEG_VIDEO_DECODER_DECODE_PICTURE_TYPE
"Normal"	- decode everything applicable to the codec (default)
"Thumbnail"	- decode sync frames (pictures?) with low memory usage
"Sync frames"	- decode sync frames without memory constraints
"Skip B frames"	- decode sync frames and unidirectional frames
"Skip Temporal layer(s)" - decode everything but skip temporal layers
above base layer

-- 
regards,
Stan

Re: decode sync frames only

[Nicolas Dufresne]

Le mercredi 22 avril 2020 à 14:43 +0300, Stanimir Varbanov a écrit :
> Hi Nicolas,
> 
> On 4/21/20 1:09 AM, Nicolas Dufresne wrote:
> > Hi Stanimir,
> > 
> > Le lundi 20 avril 2020 à 17:38 +0300, Stanimir Varbanov a écrit :
> > > Hi,
> > > 
> > > I need to port a decoder v4l2 control in mainline Venus driver which
> > > instructs the decoder to decode sync frames only (I frame/ IDR frame).
> > > In practice the usage of such control is to generate thumbnails for
> > > particular video.
> > > 
> > > To do that I researched what we have currently in v4l2-controls.h and
> > > found something similar but for encoders:
> > > 
> > > V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME
> > > "Force a key frame for the next queued buffer. Applicable to encoders.
> > > This is a general, codec-agnostic keyframe control."
> > > 
> > > I think I have two options:
> > > 
> > > 1. reuse V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME for the decoder and document it
> > > 
> > > 2. create a new v4l control V4L2_CID_MPEG_VIDC_VIDEO_SYNC_FRAME_DECODE
> > 
> > Seems analogue to GStreamer flag GST_SEEK_FLAG_TRICKMODE_KEY_UNIT, if that helps
> > your inspiration. It's used to speed up fast-forward (ore thumbnails
> > generation). What the decoder will effectively do is to skip the P and the B
> > frames. Some decoder offers fine grain control on that.
> > 
> > That being said, in GStreamer (and probably most framework) frames are skipped
> > in userspace. But decoder tend to wait for the next key frame to output the
> > first one, so we endup having to do more work to force the frame out (like
> > CMD_STOP/START cycle, or crafts an early AUD). Would be nice to document if that
> > mode will help in this regard (or not). 
> > 
> > For protocols like DASH, when doing fast-forward we don't even download the non-
> > key-frames from the server, so the skipping behaviour does not provide any gain
> > any way.
> > 
> > > Any suggestions?
> > > 
> > 
> > Could be as Hans suggested, or something less generic but that covers more
> > trickmodes ?
> > 
> > V4L2_CID_MPEG_VIDEO_DECODER_H264_TRICKMODE
> >   - No skip
> >   - Skip B Frames
> >   - I frame only
> > 
> > Or maybe try and construct something base on what other do ?
> > 
> > FFMPEG software decoder has the following:
> >   (0): Skip nothing     - 0
> >   (1): Skip B-frames    - 1
> >   (2): Skip IDCT/Dequantization - 2
> >   (5): Skip everything  - 5
> > 
> > CODA from Chips&Media is fancier, and propose some variants that have
> > implication how muchs state is maintained, hence allowing or not the transition
> > back to non-skipping outside IRAP boundary.
> >   NORMAL
> >   THUMBNAIL
> >   Skip nonIRAP
> >   Skip nonREF-PIC
> >   Skip Temporal Layer
> > 
> > I didn't find such feature for Samsung MFC, but Amlogic is likely one candidate
> > having this type of skipping feature (as it also has HW demuxers). Hope this
> > helps getting the big picture, hence proposing uAPI that isn't too limited.
> > 
> > regards,
> > Nicolas
> > 
> 
> I like your idea to have more generic one and codec-agnostic. I guess
> something like below menu control:
> 
> V4L2_CID_MPEG_VIDEO_DECODER_DECODE_PICTURE_TYPE
> "Normal"	- decode everything applicable to the codec (default)
> "Thumbnail"	- decode sync frames (pictures?) with low memory usage
> "Sync frames"	- decode sync frames without memory constraints
> "Skip B frames"	- decode sync frames and unidirectional frames
> "Skip Temporal layer(s)" - decode everything but skip temporal layers
> above base layer

That's promising, of course it will be hard to claim that "Skip B-
frame" is codec-agnostic. But it could be acceptable if we come up with
a proper story for enumeration. If I may make the analogy with the
enum-framerate, the answer from that can vary depending on the
configured format. The enumeration of the menu could behave similarly I
think.

Nicolas