Re: [PATCH v5 2/4] staging: media: Introduce NVIDIA Tegra video decoder driver

Re: [PATCH v5 2/4] staging: media: Introduce NVIDIA Tegra video decoder driver

[Ezequiel Garcia]

Hi Dmitry,

On Mon, 11 Dec 2017 at 21:27, Dmitry Osipenko <digetx@gmail.com> wrote:
>
> NVIDIA Tegra20/30/114/124/132 SoC's have video decoder engine that
> supports standard set of video formats like H.264 / MPEG-4 / WMV / VC1.
>
> Signed-off-by: Dmitry Osipenko <digetx@gmail.com>
> ---
>  MAINTAINERS                                 |    9 +
>  drivers/staging/media/Kconfig               |    2 +
>  drivers/staging/media/Makefile              |    1 +
>  drivers/staging/media/tegra-vde/Kconfig     |    7 +
>  drivers/staging/media/tegra-vde/Makefile    |    1 +
>  drivers/staging/media/tegra-vde/TODO        |    4 +
>  drivers/staging/media/tegra-vde/tegra-vde.c | 1213 +++++++++++++++++++++++++++
>  drivers/staging/media/tegra-vde/uapi.h      |   78 ++
>  8 files changed, 1315 insertions(+)
>  create mode 100644 drivers/staging/media/tegra-vde/Kconfig
>  create mode 100644 drivers/staging/media/tegra-vde/Makefile
>  create mode 100644 drivers/staging/media/tegra-vde/TODO
>  create mode 100644 drivers/staging/media/tegra-vde/tegra-vde.c
>  create mode 100644 drivers/staging/media/tegra-vde/uapi.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 7d195739f892..7f7c24949a06 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -8706,6 +8706,15 @@ T:       git git://linuxtv.org/media_tree.git
>  S:     Maintained
>  F:     drivers/media/dvb-frontends/stv6111*
>
> +MEDIA DRIVERS FOR NVIDIA TEGRA - VDE
> +M:     Dmitry Osipenko <digetx@gmail.com>
> +L:     linux-media@vger.kernel.org
> +L:     linux-tegra@vger.kernel.org
> +T:     git git://linuxtv.org/media_tree.git
> +S:     Maintained
> +F:     Documentation/devicetree/bindings/media/nvidia,tegra-vde.txt
> +F:     drivers/staging/media/tegra-vde/
> +
>  MEDIA INPUT INFRASTRUCTURE (V4L/DVB)
>  M:     Mauro Carvalho Chehab <mchehab@s-opensource.com>
>  M:     Mauro Carvalho Chehab <mchehab@kernel.org>
> diff --git a/drivers/staging/media/Kconfig b/drivers/staging/media/Kconfig
> index 3a09140700e6..227437f22acf 100644
> --- a/drivers/staging/media/Kconfig
> +++ b/drivers/staging/media/Kconfig
> @@ -31,4 +31,6 @@ source "drivers/staging/media/imx/Kconfig"
>
>  source "drivers/staging/media/omap4iss/Kconfig"
>
> +source "drivers/staging/media/tegra-vde/Kconfig"
> +
>  endif
> diff --git a/drivers/staging/media/Makefile b/drivers/staging/media/Makefile
> index f25327163c67..59a47f69884f 100644
> --- a/drivers/staging/media/Makefile
> +++ b/drivers/staging/media/Makefile
> @@ -5,3 +5,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA)   += imx/
>  obj-$(CONFIG_VIDEO_DM365_VPFE) += davinci_vpfe/
>  obj-$(CONFIG_VIDEO_OMAP4)      += omap4iss/
>  obj-$(CONFIG_INTEL_ATOMISP)     += atomisp/
> +obj-$(CONFIG_TEGRA_VDE)                += tegra-vde/
> diff --git a/drivers/staging/media/tegra-vde/Kconfig b/drivers/staging/media/tegra-vde/Kconfig
> new file mode 100644
> index 000000000000..ec3ddddebdaa
> --- /dev/null
> +++ b/drivers/staging/media/tegra-vde/Kconfig
> @@ -0,0 +1,7 @@
> +config TEGRA_VDE
> +       tristate "NVIDIA Tegra Video Decoder Engine driver"
> +       depends on ARCH_TEGRA || COMPILE_TEST
> +       select SRAM
> +       help
> +           Say Y here to enable support for the NVIDIA Tegra video decoder
> +           driver.
> diff --git a/drivers/staging/media/tegra-vde/Makefile b/drivers/staging/media/tegra-vde/Makefile
> new file mode 100644
> index 000000000000..444c1d62daa1
> --- /dev/null
> +++ b/drivers/staging/media/tegra-vde/Makefile
> @@ -0,0 +1 @@
> +obj-$(CONFIG_TEGRA_VDE)        += tegra-vde.o
> diff --git a/drivers/staging/media/tegra-vde/TODO b/drivers/staging/media/tegra-vde/TODO
> new file mode 100644
> index 000000000000..31aaa3e66d80
> --- /dev/null
> +++ b/drivers/staging/media/tegra-vde/TODO
> @@ -0,0 +1,4 @@
> +TODO:
> +       - Implement V4L2 API once it gains support for stateless decoders.
> +
> +Contact: Dmitry Osipenko <digetx@gmail.com>

The API for H264 stateless decoding is ready.
See https://lkml.org/lkml/2020/11/18/795.

One minor comment below.

> diff --git a/drivers/staging/media/tegra-vde/uapi.h b/drivers/staging/media/tegra-vde/uapi.h
> new file mode 100644
> index 000000000000..a50c7bcae057
> --- /dev/null
> +++ b/drivers/staging/media/tegra-vde/uapi.h
> @@ -0,0 +1,78 @@
> +/*
> + * Copyright (C) 2016-2017 Dmitry Osipenko <digetx@gmail.com>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +
> +#ifndef _UAPI_TEGRA_VDE_H_
> +#define _UAPI_TEGRA_VDE_H_
> +
> +#include <linux/types.h>
> +#include <asm/ioctl.h>
> +
> +#define FLAG_B_FRAME           (1 << 0)
> +#define FLAG_REFERENCE         (1 << 1)
> +
> +struct tegra_vde_h264_frame {
> +       __s32 y_fd;
> +       __s32 cb_fd;
> +       __s32 cr_fd;
> +       __s32 aux_fd;
> +       __u32 y_offset;
> +       __u32 cb_offset;
> +       __u32 cr_offset;
> +       __u32 aux_offset;
> +       __u32 frame_num;
> +       __u32 flags;
> +
> +       __u32 reserved;
> +} __attribute__((packed));
> +
> +struct tegra_vde_h264_decoder_ctx {
> +       __s32 bitstream_data_fd;
> +       __u32 bitstream_data_offset;
> +
> +       __u64 dpb_frames_ptr;
> +       __u8  dpb_frames_nb;
> +       __u8  dpb_ref_frames_with_earlier_poc_nb;
> +
> +       // SPS
> +       __u8  baseline_profile;
> +       __u8  level_idc;
> +       __u8  log2_max_pic_order_cnt_lsb;
> +       __u8  log2_max_frame_num;
> +       __u8  pic_order_cnt_type;
> +       __u8  direct_8x8_inference_flag;
> +       __u8  pic_width_in_mbs;
> +       __u8  pic_height_in_mbs;
> +
> +       // PPS
> +       __u8  pic_init_qp;
> +       __u8  deblocking_filter_control_present_flag;
> +       __u8  constrained_intra_pred_flag;
> +       __u8  chroma_qp_index_offset;
> +       __u8  pic_order_present_flag;
> +

This seems to be bottom_field_pic_order_in_frame_present_flag,
as there is no "pic_order_present_flag" syntax element.

Thanks,
Ezequiel
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

Re: [PATCH v5 2/4] staging: media: Introduce NVIDIA Tegra video decoder driver

[Dmitry Osipenko]

22.11.2020 04:02, Ezequiel Garcia пишет:
> Hi Dmitry,
> 
...
>> +++ b/drivers/staging/media/tegra-vde/TODO
>> @@ -0,0 +1,4 @@
>> +TODO:
>> +       - Implement V4L2 API once it gains support for stateless decoders.
>> +
>> +Contact: Dmitry Osipenko <digetx@gmail.com>
> 
> The API for H264 stateless decoding is ready.
> See https://lkml.org/lkml/2020/11/18/795.

Hello Ezequiel,

Thank you for the notification! My last attempt at implementing V4L API
support was about a year ago and it stopped once I realized that there
is no userspace which uses that API. FFMPEG and chromium browser had
some kind of V4L support, but it all was oriented at downstream driver
stacks, and thus, not usable. Do you know what is the current status?

> One minor comment below.
> 
...
>> +       // PPS
>> +       __u8  pic_init_qp;
>> +       __u8  deblocking_filter_control_present_flag;
>> +       __u8  constrained_intra_pred_flag;
>> +       __u8  chroma_qp_index_offset;
>> +       __u8  pic_order_present_flag;
>> +
> 
> This seems to be bottom_field_pic_order_in_frame_present_flag,
> as there is no "pic_order_present_flag" syntax element.

Correct, looks like I borrowed that name from the libvdpau API.

https://vdpau.pages.freedesktop.org/libvdpau/struct_vdp_picture_info_h264.html#a405f7ef26ea76bb2c446e151062fc001
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

Re: [PATCH v5 2/4] staging: media: Introduce NVIDIA Tegra video decoder driver

[Ezequiel Garcia]

On Sat, 21 Nov 2020 at 23:01, Dmitry Osipenko <digetx@gmail.com> wrote:
>
> 22.11.2020 04:02, Ezequiel Garcia пишет:
> > Hi Dmitry,
> >
> ...
> >> +++ b/drivers/staging/media/tegra-vde/TODO
> >> @@ -0,0 +1,4 @@
> >> +TODO:
> >> +       - Implement V4L2 API once it gains support for stateless decoders.
> >> +
> >> +Contact: Dmitry Osipenko <digetx@gmail.com>
> >
> > The API for H264 stateless decoding is ready.
> > See https://lkml.org/lkml/2020/11/18/795.
>
> Hello Ezequiel,
>
> Thank you for the notification! My last attempt at implementing V4L API
> support was about a year ago and it stopped once I realized that there
> is no userspace which uses that API. FFMPEG and chromium browser had
> some kind of V4L support, but it all was oriented at downstream driver
> stacks, and thus, not usable. Do you know what is the current status?
>

The bulk of the API, which relies on the stateless decoder interface [1],
and H264 stateless V4L2 controls has been ready for some time now,
and there are various implementations supporting it.

Chromium supports it [2], and I've tested it on chromebooks,
through chromeos builds. We haven't tried a non-chromeos build,
and I would say it's quite some work.

GStreamer support is available as well. See [3] which should
work for the latest H264 controls (the ones being moved out of staging).

LibreELEC developers maintain an Ffmpeg branch [4], I expect it will
be updated for the latest H264 controls soon, and hopefully merged
in mainline Ffmpeg.

GStreamer and Ffmpeg are relatively straightforward to build and test.

Thanks,
Ezequiel

[1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
[2] https://github.com/chromium/chromium/tree/master/media/gpu/v4l2
[3] https://gitlab.freedesktop.org/ezequielgarcia/gst-plugins-bad/-/tree/h264_stable_uapi
[4] https://github.com/Kwiboo/FFmpeg/tree/v4l2-request-hwaccel-4.3.


> > One minor comment below.
> >
> ...
> >> +       // PPS
> >> +       __u8  pic_init_qp;
> >> +       __u8  deblocking_filter_control_present_flag;
> >> +       __u8  constrained_intra_pred_flag;
> >> +       __u8  chroma_qp_index_offset;
> >> +       __u8  pic_order_present_flag;
> >> +
> >
> > This seems to be bottom_field_pic_order_in_frame_present_flag,
> > as there is no "pic_order_present_flag" syntax element.
>
> Correct, looks like I borrowed that name from the libvdpau API.
>
> https://vdpau.pages.freedesktop.org/libvdpau/struct_vdp_picture_info_h264.html#a405f7ef26ea76bb2c446e151062fc001
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

Re: [PATCH v5 2/4] staging: media: Introduce NVIDIA Tegra video decoder driver

[Dmitry Osipenko]

23.11.2020 06:07, Ezequiel Garcia пишет:
> On Sat, 21 Nov 2020 at 23:01, Dmitry Osipenko <digetx@gmail.com> wrote:
>>
>> 22.11.2020 04:02, Ezequiel Garcia пишет:
>>> Hi Dmitry,
>>>
>> ...
>>>> +++ b/drivers/staging/media/tegra-vde/TODO
>>>> @@ -0,0 +1,4 @@
>>>> +TODO:
>>>> +       - Implement V4L2 API once it gains support for stateless decoders.
>>>> +
>>>> +Contact: Dmitry Osipenko <digetx@gmail.com>
>>>
>>> The API for H264 stateless decoding is ready.
>>> See https://lkml.org/lkml/2020/11/18/795.
>>
>> Hello Ezequiel,
>>
>> Thank you for the notification! My last attempt at implementing V4L API
>> support was about a year ago and it stopped once I realized that there
>> is no userspace which uses that API. FFMPEG and chromium browser had
>> some kind of V4L support, but it all was oriented at downstream driver
>> stacks, and thus, not usable. Do you know what is the current status?
>>
> 
> The bulk of the API, which relies on the stateless decoder interface [1],
> and H264 stateless V4L2 controls has been ready for some time now,
> and there are various implementations supporting it.
> 
> Chromium supports it [2], and I've tested it on chromebooks,
> through chromeos builds. We haven't tried a non-chromeos build,
> and I would say it's quite some work.
> 
> GStreamer support is available as well. See [3] which should
> work for the latest H264 controls (the ones being moved out of staging).
> 
> LibreELEC developers maintain an Ffmpeg branch [4], I expect it will
> be updated for the latest H264 controls soon, and hopefully merged
> in mainline Ffmpeg.
> 
> GStreamer and Ffmpeg are relatively straightforward to build and test.
> 
> Thanks,
> Ezequiel
> 
> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
> [2] https://github.com/chromium/chromium/tree/master/media/gpu/v4l2
> [3] https://gitlab.freedesktop.org/ezequielgarcia/gst-plugins-bad/-/tree/h264_stable_uapi
> [4] https://github.com/Kwiboo/FFmpeg/tree/v4l2-request-hwaccel-4.3.

Alright, thank you.

_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel