[RFC] V4L2 unified low-level decoder API

[RFC] V4L2 unified low-level decoder API

[Hugues FRUCHET]

Hi,

This RFC aims to start discussions in order to define the codec specific 
controls structures to fulfill the low-level decoder API needed by non 
"Stream API" based decoders ("stateless" or "Frame API" based decoders).
Several implementation exists now which runs on several SoC and various 
software frameworks.
The idea is to find the communalities between all those implementations 
and SoC to define a single unified interface in V4L2 includes.
Even if "Request API" is needed to pass those codec specific controls 
from userspace down to kernel on a per-buffer basis, we can start 
discussions and define the controls in parallel of its development.
We can even propose some implementations based on existing V4L2 control 
framework (which doesn't support "per-frame" basis) by ensuring 
atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint 
can then be relaxed when "Request API" is merged.

I would like to propose to work on a "per-codec" basis, having at least 
2 different SoC and 2 different frameworks to test and validate controls.
To do so, I have tried to identify some people that have worked on this 
subject and have proposed some implementations, feel free to correct me 
and enhance the list if needed:
* MPEG2/MPEG4
    - Florent Revest for Allwinner A13 CedarX support [1] tested with 
VLC -> libVA + sunxi-cedrus-drv-video -> V4L2
    - Myself for STMicroelectronics Delta support [2] tested with 
GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2

* VP8
- Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with 
Chromium -> V4L2
- Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the 
framework used>

* H264
- Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with 
Chromium -> V4L2
- Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? -> 
libVA + rockchip-va-driver -> V4L2
VLC? -> libVDPAU + rockchip-va-driver -> V4L2

I can work to define MPEG2/MPEG4 controls and propose functional 
implementations for those codecs, and will be glad to co-work with you 
Florent.
I can help on H264 on a code review basis based on the functional H264 
setup I have in-house and codec knowledge, but I cannot provide 
implementation in a reasonable timeframe, same for VP8.

Apart of very details of each codec, we have also to state about generic 
concerns such as:
- new pixel format introduction (VP8 => VP8F, H264 => S264, MPG2 => 
MG2F, MPG4 => MG4F)
- new device caps to indicate that driver requires extra headers ? maybe 
not needed because redundant with new pixel format
- continue to modify v4l2-controls.h ? or do we add some new specific 
header files (H264 is huge!) ?
- how to manage sequence header & picture header, optional/extended 
controls (MPEG2 sequence/picture extensions, H264 SEI, ...). Personally 
I have added flags inside a single control structure, H264 is done in a 
different way using several controls (SPS/PPS/SLICE/DECODE/...)

Thanks you to all of you for your attention and feel free to react on 
this topic if you are interested to work on this subject.

Best regards,
Hugues.

[0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 
11https://www.spinics.net/lists/linux-media/msg106699.html
[1] MPEG2 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104823.html
[1] MPEG4 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104817.html
[2] MPEG2 STi4xx 
Deltahttp://www.spinics.net/lists/linux-media/msg106240.html
[2] MPEG4 STi4xx Delta is also supported but not yet pushed
[3] VP8 Rockchip RK3288, RK3399? 
VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch 
[4] VP8 Rockchip RK3288 
VPUhttp://www.spinics.net/lists/linux-media/msg97997.html
[5] H264 Rockchip RK3288, RK3399? 
VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
[6] H264 Rockchip RK3288 
VPUhttp://www.spinics.net/lists/linux-media/msg105095.html

Re: [RFC] V4L2 unified low-level decoder API

[Hans Verkuil]

On 10/27/2016 09:42 AM, Hugues FRUCHET wrote:
> Hi,
> 
> This RFC aims to start discussions in order to define the codec specific 
> controls structures to fulfill the low-level decoder API needed by non 
> "Stream API" based decoders ("stateless" or "Frame API" based decoders).

Let's refer to this as 'stateful' decoders and 'stateless' decoders. This
is the preferred terminology (and much more descriptive than 'Stream' vs
'Frame'). It's also not really a new API, although it does rely on the
Request API.

> Several implementation exists now which runs on several SoC and various 
> software frameworks.
> The idea is to find the communalities between all those implementations 
> and SoC to define a single unified interface in V4L2 includes.
> Even if "Request API" is needed to pass those codec specific controls 
> from userspace down to kernel on a per-buffer basis, we can start 
> discussions and define the controls in parallel of its development.
> We can even propose some implementations based on existing V4L2 control 
> framework (which doesn't support "per-frame" basis) by ensuring 
> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint 
> can then be relaxed when "Request API" is merged.
> 
> I would like to propose to work on a "per-codec" basis, having at least 
> 2 different SoC and 2 different frameworks to test and validate controls.
> To do so, I have tried to identify some people that have worked on this 
> subject and have proposed some implementations, feel free to correct me 
> and enhance the list if needed:
> * MPEG2/MPEG4
>     - Florent Revest for Allwinner A13 CedarX support [1] tested with 
> VLC -> libVA + sunxi-cedrus-drv-video -> V4L2
>     - Myself for STMicroelectronics Delta support [2] tested with 
> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
> 
> * VP8
> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with 
> Chromium -> V4L2
> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the 
> framework used>
> 
> * H264
> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with 
> Chromium -> V4L2
> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? -> 
> libVA + rockchip-va-driver -> V4L2
> VLC? -> libVDPAU + rockchip-va-driver -> V4L2
> 
> I can work to define MPEG2/MPEG4 controls and propose functional 
> implementations for those codecs, and will be glad to co-work with you 
> Florent.
> I can help on H264 on a code review basis based on the functional H264 
> setup I have in-house and codec knowledge, but I cannot provide 
> implementation in a reasonable timeframe, same for VP8.
> 
> Apart of very details of each codec, we have also to state about generic 
> concerns such as:
> - new pixel format introduction (VP8 => VP8F, H264 => S264, MPG2 => 
> MG2F, MPG4 => MG4F)
> - new device caps to indicate that driver requires extra headers ? maybe 
> not needed because redundant with new pixel format

That's indeed typically signaled through the pixelformat.

> - continue to modify v4l2-controls.h ? or do we add some new specific 
> header files (H264 is huge!) ?
> - how to manage sequence header & picture header, optional/extended 
> controls (MPEG2 sequence/picture extensions, H264 SEI, ...). Personally 
> I have added flags inside a single control structure, H264 is done in a 
> different way using several controls (SPS/PPS/SLICE/DECODE/...)
> 
> Thanks you to all of you for your attention and feel free to react on 
> this topic if you are interested to work on this subject.

As long as the V4L2 driver underpins the various solutions I am happy :-)

I do think that having a libv4l plugin will be useful since it will make
it easy for applications that support the stateful decoder to use the
same code for a stateless decoder by seamlessly using the plugin.

This does not prevent other approaches at the same time, of course.

Regards,

	Hans

> 
> Best regards,
> Hugues.
> 
> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 
> 11https://www.spinics.net/lists/linux-media/msg106699.html
> [1] MPEG2 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104823.html
> [1] MPEG4 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104817.html
> [2] MPEG2 STi4xx 
> Deltahttp://www.spinics.net/lists/linux-media/msg106240.html
> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
> [3] VP8 Rockchip RK3288, RK3399? 
> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch 
> [4] VP8 Rockchip RK3288 
> VPUhttp://www.spinics.net/lists/linux-media/msg97997.html
> [5] H264 Rockchip RK3288, RK3399? 
> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
> [6] H264 Rockchip RK3288 
> VPUhttp://www.spinics.net/lists/linux-media/msg105095.html
> N�����r��y���b�X��ǧv�^�)޺{.n�+����{���bj)���w*\x1fjg���\x1e�����ݢj/���z�ޖ��2�ޙ���&�)ߡ�a��\x7f��\x1e�G���h�\x0f�j:+v���w�٥
> 

Re: [RFC] V4L2 unified low-level decoder API

[Hugues FRUCHET]

Hi,

Here is an update regarding MPEG-2 implementation based on ST video decoder:
* MPEG-2 API + DELTA kernel driver: 
http://www.mail-archive.com/linux-media@vger.kernel.org/msg107405.html
* libv4l-codecparsers plugin including MPEG-2 back-end: 
http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html

Please note that this is implemented & functional using currently 
available V4L2 control framework (no Request API), assuming that user 
side keeps unicity of S_EXT_CTRL() / QBUF(OUTPUT) pair.
Request API will remove this constraint, but the point here is to define 
control interface, as far as I have understood code, Request API will 
not affect those control definitions.

Some updates inline thereafter regarding activities on this subject; me 
for MPEG-2 on ST platform and Randy Li, Nicolas Dufresne, Ayaka for H264 
on Rockchip platform:


On 11/14/2016 10:55 AM, Hans Verkuil wrote:
> On 10/27/2016 09:42 AM, Hugues FRUCHET wrote:
>> Hi,
>>
>> This RFC aims to start discussions in order to define the codec specific
>> controls structures to fulfill the low-level decoder API needed by non
>> "Stream API" based decoders ("stateless" or "Frame API" based decoders).
>
> Let's refer to this as 'stateful' decoders and 'stateless' decoders. This
> is the preferred terminology (and much more descriptive than 'Stream' vs
> 'Frame'). It's also not really a new API, although it does rely on the
> Request API.
>
>> Several implementation exists now which runs on several SoC and various
>> software frameworks.
>> The idea is to find the communalities between all those implementations
>> and SoC to define a single unified interface in V4L2 includes.
>> Even if "Request API" is needed to pass those codec specific controls
>> from userspace down to kernel on a per-buffer basis, we can start
>> discussions and define the controls in parallel of its development.
>> We can even propose some implementations based on existing V4L2 control
>> framework (which doesn't support "per-frame" basis) by ensuring
>> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint
>> can then be relaxed when "Request API" is merged.
>>
>> I would like to propose to work on a "per-codec" basis, having at least
>> 2 different SoC and 2 different frameworks to test and validate controls.
>> To do so, I have tried to identify some people that have worked on this
>> subject and have proposed some implementations, feel free to correct me
>> and enhance the list if needed:
>> * MPEG2/MPEG4
>>     - Florent Revest for Allwinner A13 CedarX support [1] tested with
>> VLC -> libVA + sunxi-cedrus-drv-video -> V4L2
>>     - Myself for STMicroelectronics Delta support [2] tested with
>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
Available on ST platform with [2] & [2.1] patchset series.

>>
>> * VP8
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with
>> Chromium -> V4L2
>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>> framework used>
>>
>> * H264
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with
>> Chromium -> V4L2
>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>> libVA + rockchip-va-driver -> V4L2
>> VLC? -> libVDPAU + rockchip-va-driver -> V4L2
Tested with Gstreamer -> VA-API element -> Rockchip VA-API driver -> V4L2
https://github.com/rockchip-linux/libvdpau-rockchip

Study on-going for H264 userland/kernel partitioning in this thread:
Request API: stateless VPU: the buffer mechanism and DPB management: 
http://www.mail-archive.com/linux-media@vger.kernel.org/msg107165.html

>>
>> I can work to define MPEG2/MPEG4 controls and propose functional
>> implementations for those codecs, and will be glad to co-work with you
>> Florent.
>> I can help on H264 on a code review basis based on the functional H264
>> setup I have in-house and codec knowledge, but I cannot provide
>> implementation in a reasonable timeframe, same for VP8.
>>
>> Apart of very details of each codec, we have also to state about generic
>> concerns such as:
>> - new pixel format introduction (VP8 => VP8F, H264 => S264, MPG2 =>
>> MG2F, MPG4 => MG4F)
>> - new device caps to indicate that driver requires extra headers ? maybe
>> not needed because redundant with new pixel format
>
> That's indeed typically signaled through the pixelformat.
>

[2] is implemented this way in [media] v4l: add parsed MPEG-2 support, 
two new pixel format MG1P and MG2P ("P" for "Parsed") have been introduced:
+#define V4L2_PIX_FMT_MPEG1_PARSED v4l2_fourcc('M', 'G', '1', 'P') /*
MPEG1 with parsing metadata given through controls */
+#define V4L2_PIX_FMT_MPEG2_PARSED v4l2_fourcc('M', 'G', '2', 'P') /* 
MPEG2 with parsing metadata given through controls */

libv4l plugin [2.1] intercepts the pixel format information negotiated 
between user and driver (enum_fmt/try_fmt/get_fmt/s_fmt) and selects the 
right parser to use to convert MPEG-2 compressed format to new "MPEG-2 
parsed" compressed format required by kernel driver.
Plugin is designed to support several formats.


>> - continue to modify v4l2-controls.h ? or do we add some new specific
>> header files (H264 is huge!) ?
>> - how to manage sequence header & picture header, optional/extended
>> controls (MPEG2 sequence/picture extensions, H264 SEI, ...). Personally
>> I have added flags inside a single control structure, H264 is done in a
>> different way using several controls (SPS/PPS/SLICE/DECODE/...)
>>
>> Thanks you to all of you for your attention and feel free to react on
>> this topic if you are interested to work on this subject.
>
> As long as the V4L2 driver underpins the various solutions I am happy :-)
>
> I do think that having a libv4l plugin will be useful since it will make
> it easy for applications that support the stateful decoder to use the
> same code for a stateless decoder by seamlessly using the plugin.
>
> This does not prevent other approaches at the same time, of course.
>
> Regards,
>
> 	Hans
>
[2] implements new extended controls for MPEG-2 in [media] v4l: add 
parsed MPEG-2 support 
http://www.mail-archive.com/linux-media@vger.kernel.org/msg107406.html
This has been done in uapi/linux/v4l2-controls.h as extension of already 
existing MPEG video controls, defining one control per "header" (so not 
using a single control with selection flag):
#define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_HDR
#define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_EXT
#define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_DISPLAY_EXT
#define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_MATRIX_EXT
#define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_HDR
#define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_EXT

Those controls and their associated data structure have been defined 
based on MPEG-2 standard ISO/IEC 13818-2.

>>
>> Best regards,
>> Hugues.
>>
>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 &
>> 11https://www.spinics.net/lists/linux-media/msg106699.html
>> [1] MPEG2 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104823.html
>> [1] MPEG4 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104817.html
>> [2] MPEG2 STi4xx
>> Deltahttp://www.spinics.net/lists/linux-media/msg106240.html
[2] MPEG2 DELTA kernel driver: http://www.mail-archive.com/linux-
media@vger.kernel.org/msg107405.html
[2.1] MPEG2 libv4l plugin: 
http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html

>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>> [3] VP8 Rockchip RK3288, RK3399?
>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>> [4] VP8 Rockchip RK3288
>> VPUhttp://www.spinics.net/lists/linux-media/msg97997.html
>> [5] H264 Rockchip RK3288, RK3399?
>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>> [6] H264 Rockchip RK3288
>> VPUhttp://www.spinics.net/lists/linux-media/msg105095.html
https://github.com/rockchip-linux/libvdpau-rockchip
https://github.com/rockchip-linux/gstreamer-rockchip
https://github.com/rockchip-linux/mpp

>> N�����r��y���b�X��ǧv�^�)޺{.n�+����{���bj)���w*\x1fjg���\x1e�����ݢj/���z�ޖ��2�ޙ���&�)ߡ�a��\x7f��\x1e�G���h�\x0f�j:+v���w�٥
>>

Re: [RFC] V4L2 unified low-level decoder API

[Hugues FRUCHET]

Hi all,

Here is the latest st-delta MPEG2 code:
[PATCH v6 0/3] Add support for MPEG-2 in DELTA video decoder 
http://www.mail-archive.com/linux-media@vger.kernel.org/msg112067.html
[PATCH v2 0/3] Add a libv4l plugin for video bitstream parsing 
http://www.mail-archive.com/linux-media@vger.kernel.org/msg112076.html

Before merging this work Hans would like to have feedback from peers, in 
order to be sure that this is inline with other SoC vendors drivers 
expectations.

Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip 
driver ?
Laurent, could you give your view regarding Renesas driver ?

I have also added in appendice [7] the materials presented by Laurent at 
ELC 2017 in Portland to introduce stateless video codecs and V4L2 
request API, thanks for this presentation Laurent.


Best regards,
Hugues.

> On 02/07/2017 08:21 AM, Hugues FRUCHET wrote:
> Hi,
> 
> Here is an update regarding MPEG-2 implementation based on ST video decoder:
> * MPEG-2 API + DELTA kernel driver:
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107405.html
> * libv4l-codecparsers plugin including MPEG-2 back-end:
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
> 
> Please note that this is implemented & functional using currently available V4L2 control framework (no Request API), assuming that user side keeps unicity of S_EXT_CTRL() / QBUF(OUTPUT) pair.
> Request API will remove this constraint, but the point here is to define control interface, as far as I have understood code, Request API will not affect those control definitions.
> 
> Some updates inline thereafter regarding activities on this subject; me for MPEG-2 on ST platform and Randy Li, Nicolas Dufresne, Ayaka for H264 on Rockchip platform:
> 
> 
> On 11/14/2016 10:55 AM, Hans Verkuil wrote:
>> On 10/27/2016 09:42 AM, Hugues FRUCHET wrote:
>>> Hi,
>>>
>>> This RFC aims to start discussions in order to define the codec
>>> specific controls structures to fulfill the low-level decoder API
>>> needed by non "Stream API" based decoders ("stateless" or "Frame API" based decoders).
>>
>> Let's refer to this as 'stateful' decoders and 'stateless' decoders.
>> This is the preferred terminology (and much more descriptive than
>> 'Stream' vs 'Frame'). It's also not really a new API, although it does
>> rely on the Request API.
>>
>>> Several implementation exists now which runs on several SoC and
>>> various software frameworks.
>>> The idea is to find the communalities between all those
>>> implementations and SoC to define a single unified interface in V4L2 includes.
>>> Even if "Request API" is needed to pass those codec specific controls
>>> from userspace down to kernel on a per-buffer basis, we can start
>>> discussions and define the controls in parallel of its development.
>>> We can even propose some implementations based on existing V4L2
>>> control framework (which doesn't support "per-frame" basis) by
>>> ensuring atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]).
>>> Constraint can then be relaxed when "Request API" is merged.
>>>
>>> I would like to propose to work on a "per-codec" basis, having at
>>> least
>>> 2 different SoC and 2 different frameworks to test and validate controls.
>>> To do so, I have tried to identify some people that have worked on
>>> this subject and have proposed some implementations, feel free to
>>> correct me and enhance the list if needed:
>>> * MPEG2/MPEG4
>>>      - Florent Revest for Allwinner A13 CedarX support [1] tested with
>>> VLC -> libVA + sunxi-cedrus-drv-video -> V4L2
>>>      - Myself for STMicroelectronics Delta support [2] tested with
>>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
> Available on ST platform with [2] & [2.1] patchset series.
> 
>>>
>>> * VP8
>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested
>>> with Chromium -> V4L2
>>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>>> framework used>
>>>
>>> * H264
>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested
>>> with Chromium -> V4L2
>>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>>> libVA + rockchip-va-driver -> V4L2 VLC? -> libVDPAU +
>>> rockchip-va-driver -> V4L2
> Tested with Gstreamer -> VA-API element -> Rockchip VA-API driver -> V4L2 https://github.com/rockchip-linux/libvdpau-rockchip
> 
> Study on-going for H264 userland/kernel partitioning in this thread:
> Request API: stateless VPU: the buffer mechanism and DPB management:
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107165.html
> 
>>>
>>> I can work to define MPEG2/MPEG4 controls and propose functional
>>> implementations for those codecs, and will be glad to co-work with
>>> you Florent.
>>> I can help on H264 on a code review basis based on the functional
>>> H264 setup I have in-house and codec knowledge, but I cannot provide
>>> implementation in a reasonable timeframe, same for VP8.
>>>
>>> Apart of very details of each codec, we have also to state about
>>> generic concerns such as:
>>> - new pixel format introduction (VP8 => VP8F, H264 => S264, MPG2 =>
>>> MG2F, MPG4 => MG4F)
>>> - new device caps to indicate that driver requires extra headers ?
>>> maybe not needed because redundant with new pixel format
>>
>> That's indeed typically signaled through the pixelformat.
>>
> 
> [2] is implemented this way in [media] v4l: add parsed MPEG-2 support, two new pixel format MG1P and MG2P ("P" for "Parsed") have been introduced:
> +#define V4L2_PIX_FMT_MPEG1_PARSED v4l2_fourcc('M', 'G', '1', 'P') /*
> MPEG1 with parsing metadata given through controls */
> +#define V4L2_PIX_FMT_MPEG2_PARSED v4l2_fourcc('M', 'G', '2', 'P') /*
> MPEG2 with parsing metadata given through controls */
> 
> libv4l plugin [2.1] intercepts the pixel format information negotiated
> between user and driver (enum_fmt/try_fmt/get_fmt/s_fmt) and selects the
> right parser to use to convert MPEG-2 compressed format to new "MPEG-2
> parsed" compressed format required by kernel driver.
> Plugin is designed to support several formats.
> 
> 
>>> - continue to modify v4l2-controls.h ? or do we add some new specific
>>> header files (H264 is huge!) ?
>>> - how to manage sequence header & picture header, optional/extended
>>> controls (MPEG2 sequence/picture extensions, H264 SEI, ...). Personally
>>> I have added flags inside a single control structure, H264 is done in a
>>> different way using several controls (SPS/PPS/SLICE/DECODE/...)
>>>
>>> Thanks you to all of you for your attention and feel free to react on
>>> this topic if you are interested to work on this subject.
>>
>> As long as the V4L2 driver underpins the various solutions I am happy :-)
>>
>> I do think that having a libv4l plugin will be useful since it will make
>> it easy for applications that support the stateful decoder to use the
>> same code for a stateless decoder by seamlessly using the plugin.
>>
>> This does not prevent other approaches at the same time, of course.
>>
>> Regards,
>>
>> Hans
>>
> [2] implements new extended controls for MPEG-2 in [media] v4l: add
> parsed MPEG-2 support
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107406.html
> This has been done in uapi/linux/v4l2-controls.h as extension of already
> existing MPEG video controls, defining one control per "header" (so not
> using a single control with selection flag):
> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_HDR
> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_EXT
> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_DISPLAY_EXT
> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_MATRIX_EXT
> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_HDR
> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_EXT
> 
> Those controls and their associated data structure have been defined
> based on MPEG-2 standard ISO/IEC 13818-2.
> 
>>>
>>> Best regards,
>>> Hugues.
>>>
>>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 &
>>> 11https://www.spinics.net/lists/linux-media/msg106699.html
>>> [1] MPEG2 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104823.html
>>> [1] MPEG4 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104817.html
>>> [2] MPEG2 STi4xx
>>> Deltahttp://www.spinics.net/lists/linux-media/msg106240.html
> [2] MPEG2 DELTA kernel driver: http://www.mail-archive.com/linux-
> media@vger.kernel.org/msg107405.html
> [2.1] MPEG2 libv4l plugin:
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
> 
>>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>> [3] VP8 Rockchip RK3288, RK3399?
>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>> [4] VP8 Rockchip RK3288
>>> VPUhttp://www.spinics.net/lists/linux-media/msg97997.html
>>> [5] H264 Rockchip RK3288, RK3399?
>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>> [6] H264 Rockchip RK3288
>>> VPUhttp://www.spinics.net/lists/linux-media/msg105095.html
> https://github.com/rockchip-linux/libvdpau-rockchip
> https://github.com/rockchip-linux/gstreamer-rockchip
> https://github.com/rockchip-linux/mpp
[7] "2017 is the Year of the Linux Video Codec Drivers" presentation 
done by Laurent Pinchart @ ELC 2017 Portland
https://www.youtube.com/watch?v=Y5P8CE9RtFs
http://events.linuxfoundation.org/sites/events/files/slides/20170223-elc.pdf

> 

Re: [RFC] V4L2 unified low-level decoder API

[Randy Li]

On 05/19/2017 04:08 PM, Hugues FRUCHET wrote:
> Hi all,
> 
> Here is the latest st-delta MPEG2 code:
> [PATCH v6 0/3] Add support for MPEG-2 in DELTA video decoder
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg112067.html
> [PATCH v2 0/3] Add a libv4l plugin for video bitstream parsing
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg112076.html
> 
I would review it.
> Before merging this work Hans would like to have feedback from peers, in
> order to be sure that this is inline with other SoC vendors drivers
> expectations.
> 
> Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip
> driver ?
The work of the rockchip just re-start a few weeks age, I have just 
finished the driver probing type as I decide to make a clean beginning. 
The video IP of the rockchip is too complext with a different combine.

The pixel format will begin in JPEG then AVC. I am now more familiar 
with those codec now.
> Laurent, could you give your view regarding Renesas driver ?
> 
> I have also added in appendice [7] the materials presented by Laurent at
> ELC 2017 in Portland to introduce stateless video codecs and V4L2
> request API, thanks for this presentation Laurent.
> 
> 
> Best regards,
> Hugues.
> 
>> On 02/07/2017 08:21 AM, Hugues FRUCHET wrote:
>> Hi,
>>
>> Here is an update regarding MPEG-2 implementation based on ST video decoder:
>> * MPEG-2 API + DELTA kernel driver:
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107405.html
>> * libv4l-codecparsers plugin including MPEG-2 back-end:
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
>>
>> Please note that this is implemented & functional using currently available V4L2 control framework (no Request API), assuming that user side keeps unicity of S_EXT_CTRL() / QBUF(OUTPUT) pair.
>> Request API will remove this constraint, but the point here is to define control interface, as far as I have understood code, Request API will not affect those control definitions.
>>
>> Some updates inline thereafter regarding activities on this subject; me for MPEG-2 on ST platform and Randy Li, Nicolas Dufresne, Ayaka for H264 on Rockchip platform:
>>
>>
>> On 11/14/2016 10:55 AM, Hans Verkuil wrote:
>>> On 10/27/2016 09:42 AM, Hugues FRUCHET wrote:
>>>> Hi,
>>>>
>>>> This RFC aims to start discussions in order to define the codec
>>>> specific controls structures to fulfill the low-level decoder API
>>>> needed by non "Stream API" based decoders ("stateless" or "Frame API" based decoders).
>>>
>>> Let's refer to this as 'stateful' decoders and 'stateless' decoders.
>>> This is the preferred terminology (and much more descriptive than
>>> 'Stream' vs 'Frame'). It's also not really a new API, although it does
>>> rely on the Request API.
>>>
>>>> Several implementation exists now which runs on several SoC and
>>>> various software frameworks.
>>>> The idea is to find the communalities between all those
>>>> implementations and SoC to define a single unified interface in V4L2 includes.
>>>> Even if "Request API" is needed to pass those codec specific controls
>>>> from userspace down to kernel on a per-buffer basis, we can start
>>>> discussions and define the controls in parallel of its development.
>>>> We can even propose some implementations based on existing V4L2
>>>> control framework (which doesn't support "per-frame" basis) by
>>>> ensuring atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]).
>>>> Constraint can then be relaxed when "Request API" is merged.
>>>>
>>>> I would like to propose to work on a "per-codec" basis, having at
>>>> least
>>>> 2 different SoC and 2 different frameworks to test and validate controls.
>>>> To do so, I have tried to identify some people that have worked on
>>>> this subject and have proposed some implementations, feel free to
>>>> correct me and enhance the list if needed:
>>>> * MPEG2/MPEG4
>>>>       - Florent Revest for Allwinner A13 CedarX support [1] tested with
>>>> VLC -> libVA + sunxi-cedrus-drv-video -> V4L2
>>>>       - Myself for STMicroelectronics Delta support [2] tested with
>>>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>> Available on ST platform with [2] & [2.1] patchset series.
>>
>>>>
>>>> * VP8
>>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested
>>>> with Chromium -> V4L2
>>>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>>>> framework used>
>>>>
>>>> * H264
>>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested
>>>> with Chromium -> V4L2
>>>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>>>> libVA + rockchip-va-driver -> V4L2 VLC? -> libVDPAU +
>>>> rockchip-va-driver -> V4L2
>> Tested with Gstreamer -> VA-API element -> Rockchip VA-API driver -> V4L2 https://github.com/rockchip-linux/libvdpau-rockchip
>>
>> Study on-going for H264 userland/kernel partitioning in this thread:
>> Request API: stateless VPU: the buffer mechanism and DPB management:
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107165.html
>>
>>>>
>>>> I can work to define MPEG2/MPEG4 controls and propose functional
>>>> implementations for those codecs, and will be glad to co-work with
>>>> you Florent.
>>>> I can help on H264 on a code review basis based on the functional
>>>> H264 setup I have in-house and codec knowledge, but I cannot provide
>>>> implementation in a reasonable timeframe, same for VP8.
>>>>
>>>> Apart of very details of each codec, we have also to state about
>>>> generic concerns such as:
>>>> - new pixel format introduction (VP8 => VP8F, H264 => S264, MPG2 =>
>>>> MG2F, MPG4 => MG4F)
>>>> - new device caps to indicate that driver requires extra headers ?
>>>> maybe not needed because redundant with new pixel format
>>>
>>> That's indeed typically signaled through the pixelformat.
>>>
>>
>> [2] is implemented this way in [media] v4l: add parsed MPEG-2 support, two new pixel format MG1P and MG2P ("P" for "Parsed") have been introduced:
>> +#define V4L2_PIX_FMT_MPEG1_PARSED v4l2_fourcc('M', 'G', '1', 'P') /*
>> MPEG1 with parsing metadata given through controls */
>> +#define V4L2_PIX_FMT_MPEG2_PARSED v4l2_fourcc('M', 'G', '2', 'P') /*
>> MPEG2 with parsing metadata given through controls */
>>
>> libv4l plugin [2.1] intercepts the pixel format information negotiated
>> between user and driver (enum_fmt/try_fmt/get_fmt/s_fmt) and selects the
>> right parser to use to convert MPEG-2 compressed format to new "MPEG-2
>> parsed" compressed format required by kernel driver.
>> Plugin is designed to support several formats.
>>
>>
>>>> - continue to modify v4l2-controls.h ? or do we add some new specific
>>>> header files (H264 is huge!) ?
>>>> - how to manage sequence header & picture header, optional/extended
>>>> controls (MPEG2 sequence/picture extensions, H264 SEI, ...). Personally
>>>> I have added flags inside a single control structure, H264 is done in a
>>>> different way using several controls (SPS/PPS/SLICE/DECODE/...)
>>>>
>>>> Thanks you to all of you for your attention and feel free to react on
>>>> this topic if you are interested to work on this subject.
>>>
>>> As long as the V4L2 driver underpins the various solutions I am happy :-)
>>>
>>> I do think that having a libv4l plugin will be useful since it will make
>>> it easy for applications that support the stateful decoder to use the
>>> same code for a stateless decoder by seamlessly using the plugin.
>>>
>>> This does not prevent other approaches at the same time, of course.
>>>
>>> Regards,
>>>
>>> Hans
>>>
>> [2] implements new extended controls for MPEG-2 in [media] v4l: add
>> parsed MPEG-2 support
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107406.html
>> This has been done in uapi/linux/v4l2-controls.h as extension of already
>> existing MPEG video controls, defining one control per "header" (so not
>> using a single control with selection flag):
>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_HDR
>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_EXT
>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_DISPLAY_EXT
>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_MATRIX_EXT
>> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_HDR
>> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_EXT
>>
>> Those controls and their associated data structure have been defined
>> based on MPEG-2 standard ISO/IEC 13818-2.
>>
>>>>
>>>> Best regards,
>>>> Hugues.
>>>>
>>>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 &
>>>> 11https://www.spinics.net/lists/linux-media/msg106699.html
>>>> [1] MPEG2 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104823.html
>>>> [1] MPEG4 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104817.html
>>>> [2] MPEG2 STi4xx
>>>> Deltahttp://www.spinics.net/lists/linux-media/msg106240.html
>> [2] MPEG2 DELTA kernel driver: http://www.mail-archive.com/linux-
>> media@vger.kernel.org/msg107405.html
>> [2.1] MPEG2 libv4l plugin:
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
>>
>>>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>>> [3] VP8 Rockchip RK3288, RK3399?
>>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>>> [4] VP8 Rockchip RK3288
>>>> VPUhttp://www.spinics.net/lists/linux-media/msg97997.html
>>>> [5] H264 Rockchip RK3288, RK3399?
>>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>>> [6] H264 Rockchip RK3288
>>>> VPUhttp://www.spinics.net/lists/linux-media/msg105095.html
>> https://github.com/rockchip-linux/libvdpau-rockchip
>> https://github.com/rockchip-linux/gstreamer-rockchip
>> https://github.com/rockchip-linux/mpp
> [7] "2017 is the Year of the Linux Video Codec Drivers" presentation
> done by Laurent Pinchart @ ELC 2017 Portland
> https://www.youtube.com/watch?v=Y5P8CE9RtFs
> http://events.linuxfoundation.org/sites/events/files/slides/20170223-elc.pdf
> 

-- 
Randy Li

Re: [RFC] V4L2 unified low-level decoder API

[Hugues FRUCHET]

Hi Randy,

Did you get a chance to review interface ?
I would really appreciate your feedback in order that we move forward on 
this topic and get at least one implementation merged.

Best regards,
Hugues.

On 05/19/2017 10:15 AM, Randy Li wrote:
> 
> 
> On 05/19/2017 04:08 PM, Hugues FRUCHET wrote:
>> Hi all,
>>
>> Here is the latest st-delta MPEG2 code:
>> [PATCH v6 0/3] Add support for MPEG-2 in DELTA video decoder
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg112067.html
>> [PATCH v2 0/3] Add a libv4l plugin for video bitstream parsing
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg112076.html
>>
> I would review it.
>> Before merging this work Hans would like to have feedback from peers, in
>> order to be sure that this is inline with other SoC vendors drivers
>> expectations.
>>
>> Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip
>> driver ?
> The work of the rockchip just re-start a few weeks age, I have just
> finished the driver probing type as I decide to make a clean beginning.
> The video IP of the rockchip is too complext with a different combine.
> 
> The pixel format will begin in JPEG then AVC. I am now more familiar
> with those codec now.
>> Laurent, could you give your view regarding Renesas driver ?
>>
>> I have also added in appendice [7] the materials presented by Laurent at
>> ELC 2017 in Portland to introduce stateless video codecs and V4L2
>> request API, thanks for this presentation Laurent.
>>
>>
>> Best regards,
>> Hugues.
>>
>>> On 02/07/2017 08:21 AM, Hugues FRUCHET wrote:
>>> Hi,
>>>
>>> Here is an update regarding MPEG-2 implementation based on ST video decoder:
>>> * MPEG-2 API + DELTA kernel driver:
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107405.html
>>> * libv4l-codecparsers plugin including MPEG-2 back-end:
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
>>>
>>> Please note that this is implemented & functional using currently available V4L2 control framework (no Request API), assuming that user side keeps unicity of S_EXT_CTRL() / QBUF(OUTPUT) pair.
>>> Request API will remove this constraint, but the point here is to define control interface, as far as I have understood code, Request API will not affect those control definitions.
>>>
>>> Some updates inline thereafter regarding activities on this subject; me for MPEG-2 on ST platform and Randy Li, Nicolas Dufresne, Ayaka for H264 on Rockchip platform:
>>>
>>>
>>> On 11/14/2016 10:55 AM, Hans Verkuil wrote:
>>>> On 10/27/2016 09:42 AM, Hugues FRUCHET wrote:
>>>>> Hi,
>>>>>
>>>>> This RFC aims to start discussions in order to define the codec
>>>>> specific controls structures to fulfill the low-level decoder API
>>>>> needed by non "Stream API" based decoders ("stateless" or "Frame API" based decoders).
>>>>
>>>> Let's refer to this as 'stateful' decoders and 'stateless' decoders.
>>>> This is the preferred terminology (and much more descriptive than
>>>> 'Stream' vs 'Frame'). It's also not really a new API, although it does
>>>> rely on the Request API.
>>>>
>>>>> Several implementation exists now which runs on several SoC and
>>>>> various software frameworks.
>>>>> The idea is to find the communalities between all those
>>>>> implementations and SoC to define a single unified interface in V4L2 includes.
>>>>> Even if "Request API" is needed to pass those codec specific controls
>>>>> from userspace down to kernel on a per-buffer basis, we can start
>>>>> discussions and define the controls in parallel of its development.
>>>>> We can even propose some implementations based on existing V4L2
>>>>> control framework (which doesn't support "per-frame" basis) by
>>>>> ensuring atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]).
>>>>> Constraint can then be relaxed when "Request API" is merged.
>>>>>
>>>>> I would like to propose to work on a "per-codec" basis, having at
>>>>> least
>>>>> 2 different SoC and 2 different frameworks to test and validate controls.
>>>>> To do so, I have tried to identify some people that have worked on
>>>>> this subject and have proposed some implementations, feel free to
>>>>> correct me and enhance the list if needed:
>>>>> * MPEG2/MPEG4
>>>>>        - Florent Revest for Allwinner A13 CedarX support [1] tested with
>>>>> VLC -> libVA + sunxi-cedrus-drv-video -> V4L2
>>>>>        - Myself for STMicroelectronics Delta support [2] tested with
>>>>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>>> Available on ST platform with [2] & [2.1] patchset series.
>>>
>>>>>
>>>>> * VP8
>>>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested
>>>>> with Chromium -> V4L2
>>>>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>>>>> framework used>
>>>>>
>>>>> * H264
>>>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested
>>>>> with Chromium -> V4L2
>>>>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>>>>> libVA + rockchip-va-driver -> V4L2 VLC? -> libVDPAU +
>>>>> rockchip-va-driver -> V4L2
>>> Tested with Gstreamer -> VA-API element -> Rockchip VA-API driver -> V4L2 https://github.com/rockchip-linux/libvdpau-rockchip
>>>
>>> Study on-going for H264 userland/kernel partitioning in this thread:
>>> Request API: stateless VPU: the buffer mechanism and DPB management:
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107165.html
>>>
>>>>>
>>>>> I can work to define MPEG2/MPEG4 controls and propose functional
>>>>> implementations for those codecs, and will be glad to co-work with
>>>>> you Florent.
>>>>> I can help on H264 on a code review basis based on the functional
>>>>> H264 setup I have in-house and codec knowledge, but I cannot provide
>>>>> implementation in a reasonable timeframe, same for VP8.
>>>>>
>>>>> Apart of very details of each codec, we have also to state about
>>>>> generic concerns such as:
>>>>> - new pixel format introduction (VP8 => VP8F, H264 => S264, MPG2 =>
>>>>> MG2F, MPG4 => MG4F)
>>>>> - new device caps to indicate that driver requires extra headers ?
>>>>> maybe not needed because redundant with new pixel format
>>>>
>>>> That's indeed typically signaled through the pixelformat.
>>>>
>>>
>>> [2] is implemented this way in [media] v4l: add parsed MPEG-2 support, two new pixel format MG1P and MG2P ("P" for "Parsed") have been introduced:
>>> +#define V4L2_PIX_FMT_MPEG1_PARSED v4l2_fourcc('M', 'G', '1', 'P') /*
>>> MPEG1 with parsing metadata given through controls */
>>> +#define V4L2_PIX_FMT_MPEG2_PARSED v4l2_fourcc('M', 'G', '2', 'P') /*
>>> MPEG2 with parsing metadata given through controls */
>>>
>>> libv4l plugin [2.1] intercepts the pixel format information negotiated
>>> between user and driver (enum_fmt/try_fmt/get_fmt/s_fmt) and selects the
>>> right parser to use to convert MPEG-2 compressed format to new "MPEG-2
>>> parsed" compressed format required by kernel driver.
>>> Plugin is designed to support several formats.
>>>
>>>
>>>>> - continue to modify v4l2-controls.h ? or do we add some new specific
>>>>> header files (H264 is huge!) ?
>>>>> - how to manage sequence header & picture header, optional/extended
>>>>> controls (MPEG2 sequence/picture extensions, H264 SEI, ...). Personally
>>>>> I have added flags inside a single control structure, H264 is done in a
>>>>> different way using several controls (SPS/PPS/SLICE/DECODE/...)
>>>>>
>>>>> Thanks you to all of you for your attention and feel free to react on
>>>>> this topic if you are interested to work on this subject.
>>>>
>>>> As long as the V4L2 driver underpins the various solutions I am happy :-)
>>>>
>>>> I do think that having a libv4l plugin will be useful since it will make
>>>> it easy for applications that support the stateful decoder to use the
>>>> same code for a stateless decoder by seamlessly using the plugin.
>>>>
>>>> This does not prevent other approaches at the same time, of course.
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>>
>>> [2] implements new extended controls for MPEG-2 in [media] v4l: add
>>> parsed MPEG-2 support
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107406.html
>>> This has been done in uapi/linux/v4l2-controls.h as extension of already
>>> existing MPEG video controls, defining one control per "header" (so not
>>> using a single control with selection flag):
>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_HDR
>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_EXT
>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_DISPLAY_EXT
>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_MATRIX_EXT
>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_HDR
>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_EXT
>>>
>>> Those controls and their associated data structure have been defined
>>> based on MPEG-2 standard ISO/IEC 13818-2.
>>>
>>>>>
>>>>> Best regards,
>>>>> Hugues.
>>>>>
>>>>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 &
>>>>> 11https://www.spinics.net/lists/linux-media/msg106699.html
>>>>> [1] MPEG2 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104823.html
>>>>> [1] MPEG4 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104817.html
>>>>> [2] MPEG2 STi4xx
>>>>> Deltahttp://www.spinics.net/lists/linux-media/msg106240.html
>>> [2] MPEG2 DELTA kernel driver: http://www.mail-archive.com/linux-
>>> media@vger.kernel.org/msg107405.html
>>> [2.1] MPEG2 libv4l plugin:
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
>>>
>>>>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>>>> [3] VP8 Rockchip RK3288, RK3399?
>>>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>>>> [4] VP8 Rockchip RK3288
>>>>> VPUhttp://www.spinics.net/lists/linux-media/msg97997.html
>>>>> [5] H264 Rockchip RK3288, RK3399?
>>>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>>>> [6] H264 Rockchip RK3288
>>>>> VPUhttp://www.spinics.net/lists/linux-media/msg105095.html
>>> https://github.com/rockchip-linux/libvdpau-rockchip
>>> https://github.com/rockchip-linux/gstreamer-rockchip
>>> https://github.com/rockchip-linux/mpp
>> [7] "2017 is the Year of the Linux Video Codec Drivers" presentation
>> done by Laurent Pinchart @ ELC 2017 Portland
>> https://www.youtube.com/watch?v=Y5P8CE9RtFs
>> http://events.linuxfoundation.org/sites/events/files/slides/20170223-elc.pdf
>>
> 

Re: [RFC] V4L2 unified low-level decoder API

[Randy Li]

On 06/06/2017 03:59 PM, Hugues FRUCHET wrote:
> Hi Randy,
> 
> Did you get a chance to review interface ?
Oh, I have had look a quick view on it yesterday. The video IP of that 
platform doesn't come from the On2, right?
> I would really appreciate your feedback in order that we move forward on
> this topic and get at least one implementation merged.
I am a little busy recently, I will give you a feedback before the next 
Monday.
btw, only the MPEG-2 is supported?
I am not very familiar with MPEG-2 standard, I am more familiar with 
MPEG-4 PART 10 or HEVC. As the MPEG-2 is more simple, I may not meet any 
problem to understand it.
> 
> Best regards,
> Hugues.
> 
> On 05/19/2017 10:15 AM, Randy Li wrote:
>>
>>
>> On 05/19/2017 04:08 PM, Hugues FRUCHET wrote:
>>> Hi all,
>>>
>>> Here is the latest st-delta MPEG2 code:
>>> [PATCH v6 0/3] Add support for MPEG-2 in DELTA video decoder
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg112067.html
>>> [PATCH v2 0/3] Add a libv4l plugin for video bitstream parsing
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg112076.html
>>>
>> I would review it.
>>> Before merging this work Hans would like to have feedback from peers, in
>>> order to be sure that this is inline with other SoC vendors drivers
>>> expectations.
>>>
>>> Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip
>>> driver ?
>> The work of the rockchip just re-start a few weeks age, I have just
>> finished the driver probing type as I decide to make a clean beginning.
>> The video IP of the rockchip is too complext with a different combine.
>>
>> The pixel format will begin in JPEG then AVC. I am now more familiar
>> with those codec now.
>>> Laurent, could you give your view regarding Renesas driver ?
>>>
>>> I have also added in appendice [7] the materials presented by Laurent at
>>> ELC 2017 in Portland to introduce stateless video codecs and V4L2
>>> request API, thanks for this presentation Laurent.
>>>
>>>
>>> Best regards,
>>> Hugues.
>>>
>>>> On 02/07/2017 08:21 AM, Hugues FRUCHET wrote:
>>>> Hi,
>>>>
>>>> Here is an update regarding MPEG-2 implementation based on ST video decoder:
>>>> * MPEG-2 API + DELTA kernel driver:
>>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107405.html
>>>> * libv4l-codecparsers plugin including MPEG-2 back-end:
>>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
>>>>
>>>> Please note that this is implemented & functional using currently available V4L2 control framework (no Request API), assuming that user side keeps unicity of S_EXT_CTRL() / QBUF(OUTPUT) pair.
>>>> Request API will remove this constraint, but the point here is to define control interface, as far as I have understood code, Request API will not affect those control definitions.
>>>>
>>>> Some updates inline thereafter regarding activities on this subject; me for MPEG-2 on ST platform and Randy Li, Nicolas Dufresne, Ayaka for H264 on Rockchip platform:
>>>>
>>>>
>>>> On 11/14/2016 10:55 AM, Hans Verkuil wrote:
>>>>> On 10/27/2016 09:42 AM, Hugues FRUCHET wrote:
>>>>>> Hi,
>>>>>>
>>>>>> This RFC aims to start discussions in order to define the codec
>>>>>> specific controls structures to fulfill the low-level decoder API
>>>>>> needed by non "Stream API" based decoders ("stateless" or "Frame API" based decoders).
>>>>>
>>>>> Let's refer to this as 'stateful' decoders and 'stateless' decoders.
>>>>> This is the preferred terminology (and much more descriptive than
>>>>> 'Stream' vs 'Frame'). It's also not really a new API, although it does
>>>>> rely on the Request API.
>>>>>
>>>>>> Several implementation exists now which runs on several SoC and
>>>>>> various software frameworks.
>>>>>> The idea is to find the communalities between all those
>>>>>> implementations and SoC to define a single unified interface in V4L2 includes.
>>>>>> Even if "Request API" is needed to pass those codec specific controls
>>>>>> from userspace down to kernel on a per-buffer basis, we can start
>>>>>> discussions and define the controls in parallel of its development.
>>>>>> We can even propose some implementations based on existing V4L2
>>>>>> control framework (which doesn't support "per-frame" basis) by
>>>>>> ensuring atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]).
>>>>>> Constraint can then be relaxed when "Request API" is merged.
>>>>>>
>>>>>> I would like to propose to work on a "per-codec" basis, having at
>>>>>> least
>>>>>> 2 different SoC and 2 different frameworks to test and validate controls.
>>>>>> To do so, I have tried to identify some people that have worked on
>>>>>> this subject and have proposed some implementations, feel free to
>>>>>> correct me and enhance the list if needed:
>>>>>> * MPEG2/MPEG4
>>>>>>         - Florent Revest for Allwinner A13 CedarX support [1] tested with
>>>>>> VLC -> libVA + sunxi-cedrus-drv-video -> V4L2
>>>>>>         - Myself for STMicroelectronics Delta support [2] tested with
>>>>>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>>>> Available on ST platform with [2] & [2.1] patchset series.
>>>>
>>>>>>
>>>>>> * VP8
>>>>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested
>>>>>> with Chromium -> V4L2
>>>>>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>>>>>> framework used>
>>>>>>
>>>>>> * H264
>>>>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested
>>>>>> with Chromium -> V4L2
>>>>>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>>>>>> libVA + rockchip-va-driver -> V4L2 VLC? -> libVDPAU +
>>>>>> rockchip-va-driver -> V4L2
>>>> Tested with Gstreamer -> VA-API element -> Rockchip VA-API driver -> V4L2 https://github.com/rockchip-linux/libvdpau-rockchip
>>>>
>>>> Study on-going for H264 userland/kernel partitioning in this thread:
>>>> Request API: stateless VPU: the buffer mechanism and DPB management:
>>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107165.html
>>>>
>>>>>>
>>>>>> I can work to define MPEG2/MPEG4 controls and propose functional
>>>>>> implementations for those codecs, and will be glad to co-work with
>>>>>> you Florent.
>>>>>> I can help on H264 on a code review basis based on the functional
>>>>>> H264 setup I have in-house and codec knowledge, but I cannot provide
>>>>>> implementation in a reasonable timeframe, same for VP8.
>>>>>>
>>>>>> Apart of very details of each codec, we have also to state about
>>>>>> generic concerns such as:
>>>>>> - new pixel format introduction (VP8 => VP8F, H264 => S264, MPG2 =>
>>>>>> MG2F, MPG4 => MG4F)
>>>>>> - new device caps to indicate that driver requires extra headers ?
>>>>>> maybe not needed because redundant with new pixel format
>>>>>
>>>>> That's indeed typically signaled through the pixelformat.
>>>>>
>>>>
>>>> [2] is implemented this way in [media] v4l: add parsed MPEG-2 support, two new pixel format MG1P and MG2P ("P" for "Parsed") have been introduced:
>>>> +#define V4L2_PIX_FMT_MPEG1_PARSED v4l2_fourcc('M', 'G', '1', 'P') /*
>>>> MPEG1 with parsing metadata given through controls */
>>>> +#define V4L2_PIX_FMT_MPEG2_PARSED v4l2_fourcc('M', 'G', '2', 'P') /*
>>>> MPEG2 with parsing metadata given through controls */
>>>>
>>>> libv4l plugin [2.1] intercepts the pixel format information negotiated
>>>> between user and driver (enum_fmt/try_fmt/get_fmt/s_fmt) and selects the
>>>> right parser to use to convert MPEG-2 compressed format to new "MPEG-2
>>>> parsed" compressed format required by kernel driver.
>>>> Plugin is designed to support several formats.
>>>>
>>>>
>>>>>> - continue to modify v4l2-controls.h ? or do we add some new specific
>>>>>> header files (H264 is huge!) ?
>>>>>> - how to manage sequence header & picture header, optional/extended
>>>>>> controls (MPEG2 sequence/picture extensions, H264 SEI, ...). Personally
>>>>>> I have added flags inside a single control structure, H264 is done in a
>>>>>> different way using several controls (SPS/PPS/SLICE/DECODE/...)
>>>>>>
>>>>>> Thanks you to all of you for your attention and feel free to react on
>>>>>> this topic if you are interested to work on this subject.
>>>>>
>>>>> As long as the V4L2 driver underpins the various solutions I am happy :-)
>>>>>
>>>>> I do think that having a libv4l plugin will be useful since it will make
>>>>> it easy for applications that support the stateful decoder to use the
>>>>> same code for a stateless decoder by seamlessly using the plugin.
>>>>>
>>>>> This does not prevent other approaches at the same time, of course.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Hans
>>>>>
>>>> [2] implements new extended controls for MPEG-2 in [media] v4l: add
>>>> parsed MPEG-2 support
>>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107406.html
>>>> This has been done in uapi/linux/v4l2-controls.h as extension of already
>>>> existing MPEG video controls, defining one control per "header" (so not
>>>> using a single control with selection flag):
>>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_HDR
>>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_EXT
>>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_DISPLAY_EXT
>>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_SEQ_MATRIX_EXT
>>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_HDR
>>>> #define V4L2_CID_MPEG_VIDEO_MPEG2_PIC_EXT
>>>>
>>>> Those controls and their associated data structure have been defined
>>>> based on MPEG-2 standard ISO/IEC 13818-2.
>>>>
>>>>>>
>>>>>> Best regards,
>>>>>> Hugues.
>>>>>>
>>>>>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 &
>>>>>> 11https://www.spinics.net/lists/linux-media/msg106699.html
>>>>>> [1] MPEG2 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104823.html
>>>>>> [1] MPEG4 A13 CedarXhttp://www.spinics.net/lists/linux-media/msg104817.html
>>>>>> [2] MPEG2 STi4xx
>>>>>> Deltahttp://www.spinics.net/lists/linux-media/msg106240.html
>>>> [2] MPEG2 DELTA kernel driver: http://www.mail-archive.com/linux-
>>>> media@vger.kernel.org/msg107405.html
>>>> [2.1] MPEG2 libv4l plugin:
>>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg107812.html
>>>>
>>>>>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>>>>> [3] VP8 Rockchip RK3288, RK3399?
>>>>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>>>>> [4] VP8 Rockchip RK3288
>>>>>> VPUhttp://www.spinics.net/lists/linux-media/msg97997.html
>>>>>> [5] H264 Rockchip RK3288, RK3399?
>>>>>> VPUhttps://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>>>>> [6] H264 Rockchip RK3288
>>>>>> VPUhttp://www.spinics.net/lists/linux-media/msg105095.html
>>>> https://github.com/rockchip-linux/libvdpau-rockchip
>>>> https://github.com/rockchip-linux/gstreamer-rockchip
>>>> https://github.com/rockchip-linux/mpp
>>> [7] "2017 is the Year of the Linux Video Codec Drivers" presentation
>>> done by Laurent Pinchart @ ELC 2017 Portland
>>> https://www.youtube.com/watch?v=Y5P8CE9RtFs
>>> http://events.linuxfoundation.org/sites/events/files/slides/20170223-elc.pdf
>>>

-- 
Randy Li

Re: [RFC] V4L2 unified low-level decoder API

[Pawel Osciak]

Hi,

On Fri, May 19, 2017 at 1:08 AM, Hugues FRUCHET <hugues.fruchet@st.com> wrote:
> Before merging this work Hans would like to have feedback from peers, in
> order to be sure that this is inline with other SoC vendors drivers
> expectations.
>
> Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip
> driver ?

The drivers for Rockchip codecs are submitted to the public Chromium OS kernel
and working on our RK-based platforms. We have also since added a VP9 API as
well, which is also working on devices that support it. This gives us
a set of H.264,
VP8 and VP9 APIs on both kernel and userspace side (in the open source
Chromium browser) that are working currently and can be used for
further testing.

We are interested in merging the API patches as well as these drivers upstream
(they were posted on this list two years ago), however we've been blocked by the
progress of request API, which is required for this. Alexandre Courbot
is currently
looking into creating a minimal version of the request API that would provide
enough functionality for stateless codecs, and also plans to further work on
re-submitting the particular codec API patches, once the request API
is finalized.

Thanks,
Pawel

Re: [RFC] V4L2 unified low-level decoder API

[Alexandre Courbot]

On Thu, Jun 8, 2017 at 5:56 PM, Pawel Osciak <posciak@chromium.org> wrote:
> Hi,
>
> On Fri, May 19, 2017 at 1:08 AM, Hugues FRUCHET <hugues.fruchet@st.com> wrote:
>> Before merging this work Hans would like to have feedback from peers, in
>> order to be sure that this is inline with other SoC vendors drivers
>> expectations.
>>
>> Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip
>> driver ?
>
> The drivers for Rockchip codecs are submitted to the public Chromium OS kernel
> and working on our RK-based platforms. We have also since added a VP9 API as
> well, which is also working on devices that support it. This gives us
> a set of H.264,
> VP8 and VP9 APIs on both kernel and userspace side (in the open source
> Chromium browser) that are working currently and can be used for
> further testing.
>
> We are interested in merging the API patches as well as these drivers upstream
> (they were posted on this list two years ago), however we've been blocked by the
> progress of request API, which is required for this. Alexandre Courbot
> is currently
> looking into creating a minimal version of the request API that would provide
> enough functionality for stateless codecs, and also plans to further work on
> re-submitting the particular codec API patches, once the request API
> is finalized.

Hi everyone,

It is a bit scary to start hacking on V4L with something as disruptive
as the request API, so please bear with me as I will inevitably post
code that will go from cutely clueless to downright offensive.

Thankfully Pawel is not too far from my desk, and we (Pawel, Tomasz
and myself) had a very fruitful in-person brainstorming session last
week with Laurent, so this should limit the damage.

In any case, I think that everybody agrees that this needs to be
pushed forward. Chromium OS in particular has a big interest to see
this feature landing upstream, so I will dedicate some cycles to this.

>From reading the meetings reports (e.g.
https://www.spinics.net/lists/linux-media/msg106699.html) I understand
that we want to support several use-cases with this and we already
have several proposals with code. Chromium in a first time is
interested in stateless codecs support, and this use-case also seems
to be the simplest to implement, so we would like to start pushing in
that direction first. This should give us a reasonably sized code base
to rely upon and implement the other use-cases as we see clearer
through this.

I still need to study a bit the existing proposals and to clearly lay
out the conclusions of our meeting with Laurent, but the general idea
has not changed too much, except maybe that we thought it may be nice
to make state management less explicit to userspace by default. I
would be interested in knowing whether there are updated versions of
the implementations mentioned in the meeting report above, and/or a
merge of these work? Also, if someone is actively working on this at
the moment, we will definitely want to sync on IRC or anywhere else.

Excited to work with you all! :)

Cheers,
Alex.

Re: [RFC] V4L2 unified low-level decoder API

[Hans Verkuil]

Hi Alexandre,

On 08/06/17 11:59, Alexandre Courbot wrote:
> On Thu, Jun 8, 2017 at 5:56 PM, Pawel Osciak <posciak@chromium.org> wrote:
>> Hi,
>>
>> On Fri, May 19, 2017 at 1:08 AM, Hugues FRUCHET <hugues.fruchet@st.com> wrote:
>>> Before merging this work Hans would like to have feedback from peers, in
>>> order to be sure that this is inline with other SoC vendors drivers
>>> expectations.
>>>
>>> Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip
>>> driver ?
>>
>> The drivers for Rockchip codecs are submitted to the public Chromium OS kernel
>> and working on our RK-based platforms. We have also since added a VP9 API as
>> well, which is also working on devices that support it. This gives us
>> a set of H.264,
>> VP8 and VP9 APIs on both kernel and userspace side (in the open source
>> Chromium browser) that are working currently and can be used for
>> further testing.
>>
>> We are interested in merging the API patches as well as these drivers upstream
>> (they were posted on this list two years ago), however we've been blocked by the
>> progress of request API, which is required for this. Alexandre Courbot
>> is currently
>> looking into creating a minimal version of the request API that would provide
>> enough functionality for stateless codecs, and also plans to further work on
>> re-submitting the particular codec API patches, once the request API
>> is finalized.
> 
> Hi everyone,
> 
> It is a bit scary to start hacking on V4L with something as disruptive
> as the request API, so please bear with me as I will inevitably post
> code that will go from cutely clueless to downright offensive.

Yeah, you went straight into the deep end of the pool :-)

I am very, very pleased to see Google picking up this work. We need more
core V4L2 developers, so welcome!

> Thankfully Pawel is not too far from my desk, and we (Pawel, Tomasz
> and myself) had a very fruitful in-person brainstorming session last
> week with Laurent, so this should limit the damage.
> 
> In any case, I think that everybody agrees that this needs to be
> pushed forward. Chromium OS in particular has a big interest to see
> this feature landing upstream, so I will dedicate some cycles to this.

Absolutely!

> From reading the meetings reports (e.g.
> https://www.spinics.net/lists/linux-media/msg106699.html) I understand
> that we want to support several use-cases with this and we already
> have several proposals with code. Chromium in a first time is
> interested in stateless codecs support, and this use-case also seems
> to be the simplest to implement, so we would like to start pushing in
> that direction first. This should give us a reasonably sized code base
> to rely upon and implement the other use-cases as we see clearer
> through this.
> 
> I still need to study a bit the existing proposals and to clearly lay
> out the conclusions of our meeting with Laurent, but the general idea
> has not changed too much, except maybe that we thought it may be nice
> to make state management less explicit to userspace by default. I
> would be interested in knowing whether there are updated versions of
> the implementations mentioned in the meeting report above, and/or a
> merge of these work? Also, if someone is actively working on this at
> the moment, we will definitely want to sync on IRC or anywhere else.

Laurent has been the last one working on this, but his code doesn't have
the control handling :-(

My latest (well, December 2015) tree with the control request code
is here:

https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=requests2

It's AFAIK a slightly newer version from what ChromeOS uses.

> Excited to work with you all! :)

Looking forward to your code!

Regards,

	Hans

Re: [RFC] V4L2 unified low-level decoder API

[ayaka]

On 06/08/2017 06:56 PM, Hans Verkuil wrote:
> Hi Alexandre,
>
> On 08/06/17 11:59, Alexandre Courbot wrote:
>> On Thu, Jun 8, 2017 at 5:56 PM, Pawel Osciak <posciak@chromium.org> wrote:
>>> Hi,
>>>
>>> On Fri, May 19, 2017 at 1:08 AM, Hugues FRUCHET <hugues.fruchet@st.com> wrote:
>>>> Before merging this work Hans would like to have feedback from peers, in
>>>> order to be sure that this is inline with other SoC vendors drivers
>>>> expectations.
>>>>
>>>> Thomasz, Pawel, could you give your view regarding ChromeOS and Rockchip
>>>> driver ?
>>> The drivers for Rockchip codecs are submitted to the public Chromium OS kernel
>>> and working on our RK-based platforms. We have also since added a VP9 API as
That driver still lacks a number of feature comparing to the rockchip 
android driver, since google never requests them. Also the performance 
is not as good as the android one.
That is why I start to write a new driver myself.
Also the rockchip platform is very complex, that driver won't be work on 
all the SoCs without a large of modification(now only two chips are 
supported)
>>> well, which is also working on devices that support it. This gives us
>>> a set of H.264,
>>> VP8 and VP9 APIs on both kernel and userspace side (in the open source
>>> Chromium browser) that are working currently and can be used for
>>> further testing.
>>>
>>> We are interested in merging the API patches as well as these drivers upstream
>>> (they were posted on this list two years ago), however we've been blocked by the
>>> progress of request API, which is required for this. Alexandre Courbot
>>> is currently
Well the request API looks fine for me, I just can't understand why it 
is not merged except those are a few functions have a reference problem 
stopping build v4l2 as a module.
>>> looking into creating a minimal version of the request API that would provide
>>> enough functionality for stateless codecs, and also plans to further work on
>>> re-submitting the particular codec API patches, once the request API
>>> is finalized.
>> Hi everyone,
>>
>> It is a bit scary to start hacking on V4L with something as disruptive
>> as the request API, so please bear with me as I will inevitably post
>> code that will go from cutely clueless to downright offensive.
> Yeah, you went straight into the deep end of the pool :-)
>
> I am very, very pleased to see Google picking up this work. We need more
> core V4L2 developers, so welcome!
>
>> Thankfully Pawel is not too far from my desk, and we (Pawel, Tomasz
>> and myself) had a very fruitful in-person brainstorming session last
>> week with Laurent, so this should limit the damage.
>>
>> In any case, I think that everybody agrees that this needs to be
>> pushed forward. Chromium OS in particular has a big interest to see
>> this feature landing upstream, so I will dedicate some cycles to this.
> Absolutely!
>
>>  From reading the meetings reports (e.g.
>> https://www.spinics.net/lists/linux-media/msg106699.html) I understand
>> that we want to support several use-cases with this and we already
>> have several proposals with code. Chromium in a first time is
>> interested in stateless codecs support, and this use-case also seems
>> to be the simplest to implement, so we would like to start pushing in
>> that direction first. This should give us a reasonably sized code base
>> to rely upon and implement the other use-cases as we see clearer
>> through this.
>>
>> I still need to study a bit the existing proposals and to clearly lay
>> out the conclusions of our meeting with Laurent, but the general idea
>> has not changed too much, except maybe that we thought it may be nice
>> to make state management less explicit to userspace by default. I
>> would be interested in knowing whether there are updated versions of
>> the implementations mentioned in the meeting report above, and/or a
>> merge of these work? Also, if someone is actively working on this at
>> the moment, we will definitely want to sync on IRC or anywhere else.
> Laurent has been the last one working on this, but his code doesn't have
> the control handling :-(
>
> My latest (well, December 2015) tree with the control request code
> is here:
>
> https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=requests2
>
> It's AFAIK a slightly newer version from what ChromeOS uses.
>
>> Excited to work with you all! :)
> Looking forward to your code!
>
> Regards,
>
> 	Hans
>
>

Re: [RFC] V4L2 unified low-level decoder API

[Tomasz Figa]

On Thu, Jun 8, 2017 at 8:11 PM, ayaka <ayaka@soulik.info> wrote:
>
>
> On 06/08/2017 06:56 PM, Hans Verkuil wrote:
>>
>> Hi Alexandre,
>>
>> On 08/06/17 11:59, Alexandre Courbot wrote:
>>>
>>> On Thu, Jun 8, 2017 at 5:56 PM, Pawel Osciak <posciak@chromium.org>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Fri, May 19, 2017 at 1:08 AM, Hugues FRUCHET <hugues.fruchet@st.com>
>>>> wrote:
>>>>>
>>>>> Before merging this work Hans would like to have feedback from peers,
>>>>> in
>>>>> order to be sure that this is inline with other SoC vendors drivers
>>>>> expectations.
>>>>>
>>>>> Thomasz, Pawel, could you give your view regarding ChromeOS and
>>>>> Rockchip
>>>>> driver ?
>>>>
>>>> The drivers for Rockchip codecs are submitted to the public Chromium OS
>>>> kernel
>>>> and working on our RK-based platforms. We have also since added a VP9
>>>> API as
>
> That driver still lacks a number of feature comparing to the rockchip
> android driver, since google never requests them. Also the performance is
> not as good as the android one.

I'm not sure exactly what features or performance problems you're
mentioning, but that's not the point. It's a reasonably simple driver
without too much complexity, written with good V4L2 codec API
compliance in mind and proven in the industry. I see it as a good
starting point.

> That is why I start to write a new driver myself.

I think it would make sense to work on incrementally adding those
missing features and performance optimizations, instead of spending
time on unnecessary effort of doing the same basic work, which is
already done in our driver.

> Also the rockchip platform is very complex, that driver won't be work on all
> the SoCs without a large of modification(now only two chips are supported)

This is another story. The rockchip-vpu driver in ChromeOS 4.4 should
be actually refactored into generic V4L2 stateless codec helpers (most
of the shared code in current driver) + few separate drivers using
those helpers, one for each physical IP block. Current design is there
only because we thought there is more similarity between those IP
blocks and we didn't have more time to actually clean it up later,
after we realized that the assumption was false.

Best regards,
Tomasz

Re: [RFC] V4L2 unified low-level decoder API

[Randy Li]

On 11/04/2016 09:55 PM, Hugues FRUCHET wrote:
> Hi Randy,
>
> thanks for reply, some comments below:
>
>
> On 10/27/2016 03:08 AM, Randy Li wrote:
>>
>>
>> On 10/26/2016 11:09 PM, Hugues FRUCHET wrote:
>>> Hi,
>>>
>>>
>>>
>>> This RFC aims to start discussions in order to define the codec specific
>>> controls structures to fulfill the low-level decoder API needed by non
>>> “Stream API” based decoders (“stateless” or “Frame API” based decoders).
>>>
>>> Several implementation exists now which runs on several SoC and various
>>> software frameworks.
>>>
>>> The idea is to find the communalities between all those implementations
>>> and SoC to define a single unified interface in V4L2 includes.
>>>
>>> Even if “Request API” is needed to pass those codec specific controls
>>> from userspace down to kernel on a per-buffer basis, we can start
>>> discussions and define the controls in parallel of its development.
>> Yes, I have sent a one for H.264 decoder and JPEG encoder.
>>>
>>> We can even propose some implementations based on existing V4L2 control
>>> framework (which doesn’t support “per-frame” basis) by ensuring
>>> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint
>>> can then be relaxed when “Request API” is merged.
>>>
>>>
>>>
>>> I would like to propose to work on a “per-codec” basis, having at least
>>> 2 different SoC and 2 different frameworks to test and validate controls.
>>>
>>> To do so, I have tried to identify some people that have worked on this
>>> subject and have proposed some implementations, feel free to correct me
>>> and enhance the list if needed:
>>>
>>> * MPEG2/MPEG4
>>>
>>>    - Florent Revest for Allwinner A13 CedarX support [1] tested with VLC
>>> -> libVA + sunxi-cedrus-drv-video -> V4L2
>>>
>>>    - Myself for STMicroelectronics Delta support [2] tested with
>>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>>>
>>>
>>>
>>> * VP8
>>>
>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with
>>> Chromium -> V4L2
>>>
>>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>>> framework used>
>> There is rockchip VDPAU driver supporting it, but it is .
>
> Could you point out the code that is used ? Which application is used on
> top of VDPAU ?
https://github.com/rockchip-linux/libvdpau-rockchip
>
>>>
>>>
>>>
>>> * H264
>>>
>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with
>>> Chromium -> V4L2
>>>
>>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>>> libVA + rockchip-va-driver -> V4L2
>> I only tested it with Gstreamer -> VA-API element -> Rockchip VA-API
>> driver -> V4L2
>
> OK got it, thks !
>
>>>
>>>                                                                                                                          VLC?
>>> -> libVDPAU + rockchip-va-driver -> V4L2
>>>
>>> I can work to define MPEG2/MPEG4 controls and propose functional
>>> implementations for those codecs, and will be glad to co-work with you
>>> Florent.
>> But it may not work with Rockchip's SoC, you may check the following branch
>> https://github.com/hizukiayaka/rockchip-video-driver/tree/rk_v4l2_mix
>
> I have checked code and I have only found H264 support, do I miss
> something ?
No, I have said above, only H264 decoder and JPEG encoder are supported 
in currently Rockchip VA-API driver. And H264 decoder depends on a 
Rockchip H264 parser. The rk_v4l2_mix just a branch make that clearly, 
it could get what the VA-API doesn't offer from code.
>
>>>
>>> I can help on H264 on a code review basis based on the functional H264
>>> setup I have in-house and codec knowledge, but I cannot provide
>>> implementation in a reasonable timeframe, same for VP8.
>>>
>>>
>>>
>>> Apart of very details of each codec, we have also to state about generic
>>> concerns such as:
>>>
>>> -          new pixel format introduction (VP8 => VP8F, H264 => S264,
>>> MPG2 => MG2F, MPG4 => MG4F)
>> I don't think it is necessary.
>
> But currently it is done that way in all patches proposals I have seen
> so far, including rockchip:
> rockchip_decoder_v4l2.c:{VAProfileH264Baseline,V4L2_PIX_FMT_H264_SLICE},
It is Google's idea, it would be removed with new version kernel driver 
of mine. Also I don't like multiplanes image format from Google driver.
>
> We have to state about it all together. Seems natural to me to do this
> way instead of device caps.
> Doing so user knows that the driver is based on "Frame API" -so
> additional headers are required to decode input stream- and not
> on "Stream API" -H264 stream can be decoded directly-.
 > We should probably use something else then "STREAMING" in the
 > capabilities instead of duplicating all the encoding formats (exception
 > to H264 byte-stream and H264 AVC, that also applies to streaming
 > drivers and there is not easy way to introduce stream-format in the API
 > atm). Other then that, this solution works, so it could just be
 > considered the right way, I just find it less elegant personally.
I agree with Nicolas.
>
>

>>>
>>> Best regards,
>>>
>>> Hugues.
>>>
>>>
>>>
>>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 11
>>> https://www.spinics.net/lists/linux-media/msg106699.html
>>>
>>> [1] MPEG2 A13 CedarX http://www.spinics.net/lists/linux-media/msg104823.html
>>>
>>> [1] MPEG4 A13 CedarX http://www.spinics.net/lists/linux-media/msg104817.html
>>>
>>> [2] MPEG2 STi4xx Delta
>>> http://www.spinics.net/lists/linux-media/msg106240.html
>>>
>>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>>
>>> [3] VP8 Rockchip RK3288, RK3399? VPU
>>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>>
>>>
>>> [4] VP8 Rockchip RK3288 VPU
>>> http://www.spinics.net/lists/linux-media/msg97997.html
>>>
>>> [5] H264 Rockchip RK3288, RK3399? VPU
>>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>>
>>> [6] H264 Rockchip RK3288 VPU
>>> http://www.spinics.net/lists/linux-media/msg105095.html
>>>

-- 
Randy Li
The third produce department

Re: [RFC] V4L2 unified low-level decoder API

[Randy Li]

On 11/04/2016 09:55 PM, Hugues FRUCHET wrote:
> Hi Randy,
>
> thanks for reply, some comments below:
>
>
> On 10/27/2016 03:08 AM, Randy Li wrote:
>>
>>
>> On 10/26/2016 11:09 PM, Hugues FRUCHET wrote:
>>> Hi,
>>>
>>>
>>>
>>> This RFC aims to start discussions in order to define the codec specific
>>> controls structures to fulfill the low-level decoder API needed by non
>>> “Stream API” based decoders (“stateless” or “Frame API” based decoders).
>>>
>>> Several implementation exists now which runs on several SoC and various
>>> software frameworks.
>>>
>>> The idea is to find the communalities between all those implementations
>>> and SoC to define a single unified interface in V4L2 includes.
>>>
>>> Even if “Request API” is needed to pass those codec specific controls
>>> from userspace down to kernel on a per-buffer basis, we can start
>>> discussions and define the controls in parallel of its development.
>> Yes, I have sent a one for H.264 decoder and JPEG encoder.
>>>
>>> We can even propose some implementations based on existing V4L2 control
>>> framework (which doesn’t support “per-frame” basis) by ensuring
>>> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint
>>> can then be relaxed when “Request API” is merged.
>>>
>>>
>>>
>>> I would like to propose to work on a “per-codec” basis, having at least
>>> 2 different SoC and 2 different frameworks to test and validate controls.
>>>
>>> To do so, I have tried to identify some people that have worked on this
>>> subject and have proposed some implementations, feel free to correct me
>>> and enhance the list if needed:
>>>
>>> * MPEG2/MPEG4
>>>
>>>    - Florent Revest for Allwinner A13 CedarX support [1] tested with VLC
>>> -> libVA + sunxi-cedrus-drv-video -> V4L2
>>>
>>>    - Myself for STMicroelectronics Delta support [2] tested with
>>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>>>
>>>
>>>
>>> * VP8
>>>
>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with
>>> Chromium -> V4L2
>>>
>>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>>> framework used>
>> There is rockchip VDPAU driver supporting it, but it is .
>
> Could you point out the code that is used ? Which application is used on
> top of VDPAU ?
https://github.com/rockchip-linux/libvdpau-rockchip
>
>>>
>>>
>>>
>>> * H264
>>>
>>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with
>>> Chromium -> V4L2
>>>
>>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>>> libVA + rockchip-va-driver -> V4L2
>> I only tested it with Gstreamer -> VA-API element -> Rockchip VA-API
>> driver -> V4L2
>
> OK got it, thks !
>
>>>
>>>                                                                                                                          VLC?
>>> -> libVDPAU + rockchip-va-driver -> V4L2
>>>
>>> I can work to define MPEG2/MPEG4 controls and propose functional
>>> implementations for those codecs, and will be glad to co-work with you
>>> Florent.
>> But it may not work with Rockchip's SoC, you may check the following branch
>> https://github.com/hizukiayaka/rockchip-video-driver/tree/rk_v4l2_mix
>
> I have checked code and I have only found H264 support, do I miss
> something ?
No, I have said above, only H264 decoder and JPEG encoder are supported 
in currently Rockchip VA-API driver. And H264 decoder depends on a 
Rockchip H264 parser. The rk_v4l2_mix just a branch make that clearly, 
it could get what the VA-API doesn't offer from code.
>
>>>
>>> I can help on H264 on a code review basis based on the functional H264
>>> setup I have in-house and codec knowledge, but I cannot provide
>>> implementation in a reasonable timeframe, same for VP8.
>>>
>>>
>>>
>>> Apart of very details of each codec, we have also to state about generic
>>> concerns such as:
>>>
>>> -          new pixel format introduction (VP8 => VP8F, H264 => S264,
>>> MPG2 => MG2F, MPG4 => MG4F)
>> I don't think it is necessary.
>
> But currently it is done that way in all patches proposals I have seen
> so far, including rockchip:
> rockchip_decoder_v4l2.c:{VAProfileH264Baseline,V4L2_PIX_FMT_H264_SLICE},
It is Google's idea, it would be removed with new version kernel driver 
of mine. Also I don't like multiplanes image format from Google driver.
>
> We have to state about it all together. Seems natural to me to do this
> way instead of device caps.
> Doing so user knows that the driver is based on "Frame API" -so
> additional headers are required to decode input stream- and not
> on "Stream API" -H264 stream can be decoded directly-.
 > We should probably use something else then "STREAMING" in the
 > capabilities instead of duplicating all the encoding formats (exception
 > to H264 byte-stream and H264 AVC, that also applies to streaming
 > drivers and there is not easy way to introduce stream-format in the API
 > atm). Other then that, this solution works, so it could just be
 > considered the right way, I just find it less elegant personally.
I agree with Nicolas.
>
>

>>>
>>> Best regards,
>>>
>>> Hugues.
>>>
>>>
>>>
>>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 11
>>> https://www.spinics.net/lists/linux-media/msg106699.html
>>>
>>> [1] MPEG2 A13 CedarX http://www.spinics.net/lists/linux-media/msg104823.html
>>>
>>> [1] MPEG4 A13 CedarX http://www.spinics.net/lists/linux-media/msg104817.html
>>>
>>> [2] MPEG2 STi4xx Delta
>>> http://www.spinics.net/lists/linux-media/msg106240.html
>>>
>>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>>
>>> [3] VP8 Rockchip RK3288, RK3399? VPU
>>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>>
>>>
>>> [4] VP8 Rockchip RK3288 VPU
>>> http://www.spinics.net/lists/linux-media/msg97997.html
>>>
>>> [5] H264 Rockchip RK3288, RK3399? VPU
>>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>>
>>> [6] H264 Rockchip RK3288 VPU
>>> http://www.spinics.net/lists/linux-media/msg105095.html
>>>

-- 
Randy Li
The third produce department

Re: [RFC] V4L2 unified low-level decoder API

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

Le vendredi 04 novembre 2016 à 14:55 +0100, Hugues FRUCHET a écrit :
> >>
> >> I can help on H264 on a code review basis based on the functional H264
> >> setup I have in-house and codec knowledge, but I cannot provide
> >> implementation in a reasonable timeframe, same for VP8.
> >>
> >>
> >>
> >> Apart of very details of each codec, we have also to state about generic
> >> concerns such as:
> >>
> >> -          new pixel format introduction (VP8 => VP8F, H264 => S264,
> >> MPG2 => MG2F, MPG4 => MG4F)
> > I don't think it is necessary.
> 
> But currently it is done that way in all patches proposals I have seen 
> so far, including rockchip:
> rockchip_decoder_v4l2.c:{VAProfileH264Baseline,V4L2_PIX_FMT_H264_SLICE},
> 
> We have to state about it all together. Seems natural to me to do this 
> way instead of device caps.
> Doing so user knows that the driver is based on "Frame API" -so 
> additional headers are required to decode input stream- and not
> on "Stream API" -H264 stream can be decoded directly-.

We should probably use something else then "STREAMING" in the
capabilities instead of duplicating all the encoding formats (exception
to H264 byte-stream and H264 AVC, that also applies to streaming
drivers and there is not easy way to introduce stream-format in the API
atm). Other then that, this solution works, so it could just be
considered the right way, I just find it less elegant personally.

my two cents,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFC] V4L2 unified low-level decoder API

[Hugues FRUCHET]

Hi Randy,

thanks for reply, some comments below:


On 10/27/2016 03:08 AM, Randy Li wrote:
>
>
> On 10/26/2016 11:09 PM, Hugues FRUCHET wrote:
>> Hi,
>>
>>
>>
>> This RFC aims to start discussions in order to define the codec specific
>> controls structures to fulfill the low-level decoder API needed by non
>> “Stream API” based decoders (“stateless” or “Frame API” based decoders).
>>
>> Several implementation exists now which runs on several SoC and various
>> software frameworks.
>>
>> The idea is to find the communalities between all those implementations
>> and SoC to define a single unified interface in V4L2 includes.
>>
>> Even if “Request API” is needed to pass those codec specific controls
>> from userspace down to kernel on a per-buffer basis, we can start
>> discussions and define the controls in parallel of its development.
> Yes, I have sent a one for H.264 decoder and JPEG encoder.
>>
>> We can even propose some implementations based on existing V4L2 control
>> framework (which doesn’t support “per-frame” basis) by ensuring
>> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint
>> can then be relaxed when “Request API” is merged.
>>
>>
>>
>> I would like to propose to work on a “per-codec” basis, having at least
>> 2 different SoC and 2 different frameworks to test and validate controls.
>>
>> To do so, I have tried to identify some people that have worked on this
>> subject and have proposed some implementations, feel free to correct me
>> and enhance the list if needed:
>>
>> * MPEG2/MPEG4
>>
>>    - Florent Revest for Allwinner A13 CedarX support [1] tested with VLC
>> -> libVA + sunxi-cedrus-drv-video -> V4L2
>>
>>    - Myself for STMicroelectronics Delta support [2] tested with
>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>>
>>
>>
>> * VP8
>>
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with
>> Chromium -> V4L2
>>
>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>> framework used>
> There is rockchip VDPAU driver supporting it, but it is .

Could you point out the code that is used ? Which application is used on 
top of VDPAU ?

>>
>>
>>
>> * H264
>>
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with
>> Chromium -> V4L2
>>
>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>> libVA + rockchip-va-driver -> V4L2
> I only tested it with Gstreamer -> VA-API element -> Rockchip VA-API
> driver -> V4L2

OK got it, thks !

>>
>>                                                                                                                          VLC?
>> -> libVDPAU + rockchip-va-driver -> V4L2
>>
>> I can work to define MPEG2/MPEG4 controls and propose functional
>> implementations for those codecs, and will be glad to co-work with you
>> Florent.
> But it may not work with Rockchip's SoC, you may check the following branch
> https://github.com/hizukiayaka/rockchip-video-driver/tree/rk_v4l2_mix

I have checked code and I have only found H264 support, do I miss 
something ?

>>
>> I can help on H264 on a code review basis based on the functional H264
>> setup I have in-house and codec knowledge, but I cannot provide
>> implementation in a reasonable timeframe, same for VP8.
>>
>>
>>
>> Apart of very details of each codec, we have also to state about generic
>> concerns such as:
>>
>> -          new pixel format introduction (VP8 => VP8F, H264 => S264,
>> MPG2 => MG2F, MPG4 => MG4F)
> I don't think it is necessary.

But currently it is done that way in all patches proposals I have seen 
so far, including rockchip:
rockchip_decoder_v4l2.c:{VAProfileH264Baseline,V4L2_PIX_FMT_H264_SLICE},

We have to state about it all together. Seems natural to me to do this 
way instead of device caps.
Doing so user knows that the driver is based on "Frame API" -so 
additional headers are required to decode input stream- and not
on "Stream API" -H264 stream can be decoded directly-.


>>
>> -          new device caps to indicate that driver requires extra
>> headers ? maybe not needed because redundant with new pixel format
> I prefer this one.
>>
>> -          continue to modify v4l2-controls.h ? or do we add some new
>> specific header files (H264 is huge!) ?
> Not huge. You could check rockchip's kernel.

Checking 
https://github.com/hizukiayaka/rockchip-video-driver/blob/rk_v4l2_mix/librkdec/include/linux/v4l2-controls.h
it's already 400 lines of types definition for H264, VP8 and VP9.


>>
>> -          how to manage sequence header & picture header,
>> optional/extended controls (MPEG2 sequence/picture extensions, H264 SEI,
>> …). Personally I have added flags inside a single control structure,
>> H264 is done in a different way using several controls
>> (SPS/PPS/SLICE/DECODE/…)
> the last one is dpb, except the dpb, it would have the same numbers of
> controls to those structures defined in VA-API H264 decoder.

I'm a bit lost because of differences I see between implementations [5] 
and [6].
Digging into rk_v4l2_mix, I see that implementation is done using [5] 
interface:
https://github.com/hizukiayaka/rockchip-video-driver/blob/rk_v4l2_mix/librkdec/include/linux/v4l2-controls.h
and not the one pushed on V4L2 mailing list ([6] 
http://www.spinics.net/lists/linux-media/msg105095.html).


>>
>>
>> Thanks you to all of you for your attention and feel free to react on
>> this topic if you are interested to work on this subject.
> Currently, I have to pause the process of VA-API drive, and moving to
> the other idea I have said before, creating a new API in userspace

Could you point out the thread which gives details on this ?
Does this break the H264 V4L2 decoder settings defined in [5] ?

> (but won't archive the goal I set before in this step). There are some
> shortages in VA-API I have said in last email making the performance in
> 4K video and extending the Gstreamer VA-API is a little difficult job
> and need more time.
> And the development for the new VPU driver for rockchip would pause a
> while as well.
>
> It would not be a long time(a few weeks) and I am still available in my
> free time(at home). It is good to know the wheel begin to roll. And do
> feel free to assign job to me.

Thanks Randy, let's continue discussion and try to involve more people 
on subject in order that we converge.

>>
>>
>>
>> Best regards,
>>
>> Hugues.
>>
>>
>>
>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 11
>> https://www.spinics.net/lists/linux-media/msg106699.html
>>
>> [1] MPEG2 A13 CedarX http://www.spinics.net/lists/linux-media/msg104823.html
>>
>> [1] MPEG4 A13 CedarX http://www.spinics.net/lists/linux-media/msg104817.html
>>
>> [2] MPEG2 STi4xx Delta
>> http://www.spinics.net/lists/linux-media/msg106240.html
>>
>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>
>> [3] VP8 Rockchip RK3288, RK3399? VPU
>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>
>>
>> [4] VP8 Rockchip RK3288 VPU
>> http://www.spinics.net/lists/linux-media/msg97997.html
>>
>> [5] H264 Rockchip RK3288, RK3399? VPU
>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>
>> [6] H264 Rockchip RK3288 VPU
>> http://www.spinics.net/lists/linux-media/msg105095.html
>>
>>
>>
>

Re: [RFC] V4L2 unified low-level decoder API

[Hugues FRUCHET]

Hi Randy,

thanks for reply, some comments below:


On 10/27/2016 03:08 AM, Randy Li wrote:
>
>
> On 10/26/2016 11:09 PM, Hugues FRUCHET wrote:
>> Hi,
>>
>>
>>
>> This RFC aims to start discussions in order to define the codec specific
>> controls structures to fulfill the low-level decoder API needed by non
>> “Stream API” based decoders (“stateless” or “Frame API” based decoders).
>>
>> Several implementation exists now which runs on several SoC and various
>> software frameworks.
>>
>> The idea is to find the communalities between all those implementations
>> and SoC to define a single unified interface in V4L2 includes.
>>
>> Even if “Request API” is needed to pass those codec specific controls
>> from userspace down to kernel on a per-buffer basis, we can start
>> discussions and define the controls in parallel of its development.
> Yes, I have sent a one for H.264 decoder and JPEG encoder.
>>
>> We can even propose some implementations based on existing V4L2 control
>> framework (which doesn’t support “per-frame” basis) by ensuring
>> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint
>> can then be relaxed when “Request API” is merged.
>>
>>
>>
>> I would like to propose to work on a “per-codec” basis, having at least
>> 2 different SoC and 2 different frameworks to test and validate controls.
>>
>> To do so, I have tried to identify some people that have worked on this
>> subject and have proposed some implementations, feel free to correct me
>> and enhance the list if needed:
>>
>> * MPEG2/MPEG4
>>
>>    - Florent Revest for Allwinner A13 CedarX support [1] tested with VLC
>> -> libVA + sunxi-cedrus-drv-video -> V4L2
>>
>>    - Myself for STMicroelectronics Delta support [2] tested with
>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>>
>>
>>
>> * VP8
>>
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with
>> Chromium -> V4L2
>>
>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>> framework used>
> There is rockchip VDPAU driver supporting it, but it is .

Could you point out the code that is used ? Which application is used on 
top of VDPAU ?

>>
>>
>>
>> * H264
>>
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with
>> Chromium -> V4L2
>>
>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>> libVA + rockchip-va-driver -> V4L2
> I only tested it with Gstreamer -> VA-API element -> Rockchip VA-API
> driver -> V4L2

OK got it, thks !

>>
>>                                                                                                                          VLC?
>> -> libVDPAU + rockchip-va-driver -> V4L2
>>
>> I can work to define MPEG2/MPEG4 controls and propose functional
>> implementations for those codecs, and will be glad to co-work with you
>> Florent.
> But it may not work with Rockchip's SoC, you may check the following branch
> https://github.com/hizukiayaka/rockchip-video-driver/tree/rk_v4l2_mix

I have checked code and I have only found H264 support, do I miss 
something ?

>>
>> I can help on H264 on a code review basis based on the functional H264
>> setup I have in-house and codec knowledge, but I cannot provide
>> implementation in a reasonable timeframe, same for VP8.
>>
>>
>>
>> Apart of very details of each codec, we have also to state about generic
>> concerns such as:
>>
>> -          new pixel format introduction (VP8 => VP8F, H264 => S264,
>> MPG2 => MG2F, MPG4 => MG4F)
> I don't think it is necessary.

But currently it is done that way in all patches proposals I have seen 
so far, including rockchip:
rockchip_decoder_v4l2.c:{VAProfileH264Baseline,V4L2_PIX_FMT_H264_SLICE},

We have to state about it all together. Seems natural to me to do this 
way instead of device caps.
Doing so user knows that the driver is based on "Frame API" -so 
additional headers are required to decode input stream- and not
on "Stream API" -H264 stream can be decoded directly-.


>>
>> -          new device caps to indicate that driver requires extra
>> headers ? maybe not needed because redundant with new pixel format
> I prefer this one.
>>
>> -          continue to modify v4l2-controls.h ? or do we add some new
>> specific header files (H264 is huge!) ?
> Not huge. You could check rockchip's kernel.

Checking 
https://github.com/hizukiayaka/rockchip-video-driver/blob/rk_v4l2_mix/librkdec/include/linux/v4l2-controls.h
it's already 400 lines of types definition for H264, VP8 and VP9.


>>
>> -          how to manage sequence header & picture header,
>> optional/extended controls (MPEG2 sequence/picture extensions, H264 SEI,
>> …). Personally I have added flags inside a single control structure,
>> H264 is done in a different way using several controls
>> (SPS/PPS/SLICE/DECODE/…)
> the last one is dpb, except the dpb, it would have the same numbers of
> controls to those structures defined in VA-API H264 decoder.

I'm a bit lost because of differences I see between implementations [5] 
and [6].
Digging into rk_v4l2_mix, I see that implementation is done using [5] 
interface:
https://github.com/hizukiayaka/rockchip-video-driver/blob/rk_v4l2_mix/librkdec/include/linux/v4l2-controls.h
and not the one pushed on V4L2 mailing list ([6] 
http://www.spinics.net/lists/linux-media/msg105095.html).


>>
>>
>> Thanks you to all of you for your attention and feel free to react on
>> this topic if you are interested to work on this subject.
> Currently, I have to pause the process of VA-API drive, and moving to
> the other idea I have said before, creating a new API in userspace

Could you point out the thread which gives details on this ?
Does this break the H264 V4L2 decoder settings defined in [5] ?

> (but won't archive the goal I set before in this step). There are some
> shortages in VA-API I have said in last email making the performance in
> 4K video and extending the Gstreamer VA-API is a little difficult job
> and need more time.
> And the development for the new VPU driver for rockchip would pause a
> while as well.
>
> It would not be a long time(a few weeks) and I am still available in my
> free time(at home). It is good to know the wheel begin to roll. And do
> feel free to assign job to me.

Thanks Randy, let's continue discussion and try to involve more people 
on subject in order that we converge.

>>
>>
>>
>> Best regards,
>>
>> Hugues.
>>
>>
>>
>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 11
>> https://www.spinics.net/lists/linux-media/msg106699.html
>>
>> [1] MPEG2 A13 CedarX http://www.spinics.net/lists/linux-media/msg104823.html
>>
>> [1] MPEG4 A13 CedarX http://www.spinics.net/lists/linux-media/msg104817.html
>>
>> [2] MPEG2 STi4xx Delta
>> http://www.spinics.net/lists/linux-media/msg106240.html
>>
>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>
>> [3] VP8 Rockchip RK3288, RK3399? VPU
>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>
>>
>> [4] VP8 Rockchip RK3288 VPU
>> http://www.spinics.net/lists/linux-media/msg97997.html
>>
>> [5] H264 Rockchip RK3288, RK3399? VPU
>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>
>> [6] H264 Rockchip RK3288 VPU
>> http://www.spinics.net/lists/linux-media/msg105095.html
>>
>>
>>
>
_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] V4L2 unified low-level decoder API

[Randy Li]

On 10/26/2016 11:09 PM, Hugues FRUCHET wrote:
> Hi,
>
>
>
> This RFC aims to start discussions in order to define the codec specific
> controls structures to fulfill the low-level decoder API needed by non
> “Stream API” based decoders (“stateless” or “Frame API” based decoders).
>
> Several implementation exists now which runs on several SoC and various
> software frameworks.
>
> The idea is to find the communalities between all those implementations
> and SoC to define a single unified interface in V4L2 includes.
>
> Even if “Request API” is needed to pass those codec specific controls
> from userspace down to kernel on a per-buffer basis, we can start
> discussions and define the controls in parallel of its development.
Yes, I have sent a one for H.264 decoder and JPEG encoder.
>
> We can even propose some implementations based on existing V4L2 control
> framework (which doesn’t support “per-frame” basis) by ensuring
> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint
> can then be relaxed when “Request API” is merged.
>
>
>
> I would like to propose to work on a “per-codec” basis, having at least
> 2 different SoC and 2 different frameworks to test and validate controls.
>
> To do so, I have tried to identify some people that have worked on this
> subject and have proposed some implementations, feel free to correct me
> and enhance the list if needed:
>
> * MPEG2/MPEG4
>
>    - Florent Revest for Allwinner A13 CedarX support [1] tested with VLC
> -> libVA + sunxi-cedrus-drv-video -> V4L2
>
>    - Myself for STMicroelectronics Delta support [2] tested with
> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>
>
>
> * VP8
>
> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with
> Chromium -> V4L2
>
> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
> framework used>
There is rockchip VDPAU driver supporting it, but it is .
>
>
>
> * H264
>
> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with
> Chromium -> V4L2
>
> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
> libVA + rockchip-va-driver -> V4L2
I only tested it with Gstreamer -> VA-API element -> Rockchip VA-API 
driver -> V4L2
>
>                                                                                                                          VLC?
> -> libVDPAU + rockchip-va-driver -> V4L2
>
> I can work to define MPEG2/MPEG4 controls and propose functional
> implementations for those codecs, and will be glad to co-work with you
> Florent.
But it may not work with Rockchip's SoC, you may check the following branch
https://github.com/hizukiayaka/rockchip-video-driver/tree/rk_v4l2_mix
>
> I can help on H264 on a code review basis based on the functional H264
> setup I have in-house and codec knowledge, but I cannot provide
> implementation in a reasonable timeframe, same for VP8.
>
>
>
> Apart of very details of each codec, we have also to state about generic
> concerns such as:
>
> -          new pixel format introduction (VP8 => VP8F, H264 => S264,
> MPG2 => MG2F, MPG4 => MG4F)
I don't think it is necessary.
>
> -          new device caps to indicate that driver requires extra
> headers ? maybe not needed because redundant with new pixel format
I prefer this one.
>
> -          continue to modify v4l2-controls.h ? or do we add some new
> specific header files (H264 is huge!) ?
Not huge. You could check rockchip's kernel.
>
> -          how to manage sequence header & picture header,
> optional/extended controls (MPEG2 sequence/picture extensions, H264 SEI,
> …). Personally I have added flags inside a single control structure,
> H264 is done in a different way using several controls
> (SPS/PPS/SLICE/DECODE/…)
the last one is dpb, except the dpb, it would have the same numbers of 
controls to those structures defined in VA-API H264 decoder.
>
>
> Thanks you to all of you for your attention and feel free to react on
> this topic if you are interested to work on this subject.
Currently, I have to pause the process of VA-API drive, and moving to 
the other idea I have said before, creating a new API in userspace(but 
won't archive the goal I set before in this step). There are some 
shortages in VA-API I have said in last email making the performance in 
4K video and extending the Gstreamer VA-API is a little difficult job 
and need more time.
And the development for the new VPU driver for rockchip would pause a 
while as well.

It would not be a long time(a few weeks) and I am still available in my 
free time(at home). It is good to know the wheel begin to roll. And do 
feel free to assign job to me.
>
>
>
> Best regards,
>
> Hugues.
>
>
>
> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 11
> https://www.spinics.net/lists/linux-media/msg106699.html
>
> [1] MPEG2 A13 CedarX http://www.spinics.net/lists/linux-media/msg104823.html
>
> [1] MPEG4 A13 CedarX http://www.spinics.net/lists/linux-media/msg104817.html
>
> [2] MPEG2 STi4xx Delta
> http://www.spinics.net/lists/linux-media/msg106240.html
>
> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>
> [3] VP8 Rockchip RK3288, RK3399? VPU
> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>
>
> [4] VP8 Rockchip RK3288 VPU
> http://www.spinics.net/lists/linux-media/msg97997.html
>
> [5] H264 Rockchip RK3288, RK3399? VPU
> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>
> [6] H264 Rockchip RK3288 VPU
> http://www.spinics.net/lists/linux-media/msg105095.html
>
>
>

-- 
Randy Li
The third produce department