gstreamer: v4l2videodec plugin

gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

Hi,

I'm working on QCOM v4l2 video decoder/encoder driver and in order to
test its functionalities I'm using gstreamer v4l2videodec plugin. I am
able to use the v4l2videodec plugin with MMAP, now I want to try the
dmabuf export from v4l2 and import dmabuf buffers to glimagesink. I
upgraded gst to 1.7.91 so that I have the dmabuf support in glimagesink.
Mesa version is 11.1.2.

I'm using the following pipeline:

GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
$GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
capture-io-mode=dmabuf ! glimagesink

I stalled on this error:

eglimagememory
gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimageallocator0>
eglCreateImage failed: EGL_BAD_MATCH

which in Mesa is:

libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
dri2_create_image_khr_texture

Do someone know how the dmabuf import is tested when the support has
been added to glimagesink? Or some pointers how to continue with debugging?

Thanks for the answers.

-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

adding gstreamer-devel

On 04/11/2016 03:03 PM, Stanimir Varbanov wrote:
> Hi,
> 
> I'm working on QCOM v4l2 video decoder/encoder driver and in order to
> test its functionalities I'm using gstreamer v4l2videodec plugin. I am
> able to use the v4l2videodec plugin with MMAP, now I want to try the
> dmabuf export from v4l2 and import dmabuf buffers to glimagesink. I
> upgraded gst to 1.7.91 so that I have the dmabuf support in glimagesink.
> Mesa version is 11.1.2.
> 
> I'm using the following pipeline:
> 
> GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
> $GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
> capture-io-mode=dmabuf ! glimagesink
> 
> I stalled on this error:
> 
> eglimagememory
> gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimageallocator0>
> eglCreateImage failed: EGL_BAD_MATCH
> 
> which in Mesa is:
> 
> libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
> dri2_create_image_khr_texture
> 
> Do someone know how the dmabuf import is tested when the support has
> been added to glimagesink? Or some pointers how to continue with debugging?
> 
> Thanks for the answers.
> 


-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Víctor M. Jáquez L.]

On 04/11/16 at 03:11pm, Stanimir Varbanov wrote:
> adding gstreamer-devel
> 
> On 04/11/2016 03:03 PM, Stanimir Varbanov wrote:
> > Hi,
> > 
> > I'm working on QCOM v4l2 video decoder/encoder driver and in order to
> > test its functionalities I'm using gstreamer v4l2videodec plugin. I am
> > able to use the v4l2videodec plugin with MMAP, now I want to try the
> > dmabuf export from v4l2 and import dmabuf buffers to glimagesink. I
> > upgraded gst to 1.7.91 so that I have the dmabuf support in glimagesink.
> > Mesa version is 11.1.2.
> > 
> > I'm using the following pipeline:
> > 
> > GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
> > $GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
> > capture-io-mode=dmabuf ! glimagesink
> > 
> > I stalled on this error:
> > 
> > eglimagememory
> > gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimageallocator0>
> > eglCreateImage failed: EGL_BAD_MATCH
> > 
> > which in Mesa is:
> > 
> > libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
> > dri2_create_image_khr_texture
> > 
> > Do someone know how the dmabuf import is tested when the support has
> > been added to glimagesink? Or some pointers how to continue with
> > debugging?

Perhaps this is not useful for your case, but there's a kmssink (a simple
video sink that uses KMS/DRM kernel API)[1]. It supports dmabuf import and
rendering, and the way it does it is heavily inspired on how glimagesink does
it, obviously without the EGL burden, just the kernel's PRIME API.

1. https://bugzilla.gnome.org/show_bug.cgi?id=761059

Re: gstreamer: v4l2videodec plugin

[Nicolas Dufresne]

Le lundi 11 avril 2016 à 15:11 +0300, Stanimir Varbanov a écrit :
> adding gstreamer-devel
> 
> On 04/11/2016 03:03 PM, Stanimir Varbanov wrote:
> > 
> > Hi,
> > 
> > I'm working on QCOM v4l2 video decoder/encoder driver and in order
> > to
> > test its functionalities I'm using gstreamer v4l2videodec plugin. I
> > am
> > able to use the v4l2videodec plugin with MMAP, now I want to try
> > the
> > dmabuf export from v4l2 and import dmabuf buffers to glimagesink. I
> > upgraded gst to 1.7.91 so that I have the dmabuf support in
> > glimagesink.
> > Mesa version is 11.1.2.

I'm very happy to see this report. So far, we only had report that this
element works on Freescale IMX.6 (CODA) and Exynos 4/5.

> > 
> > I'm using the following pipeline:
> > 
> > GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
> > $GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
> > capture-io-mode=dmabuf ! glimagesink
> > 
> > I stalled on this error:
> > 
> > eglimagememory
> > gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimagea
> > llocator0>
> > eglCreateImage failed: EGL_BAD_MATCH
> > 
> > which in Mesa is:
> > 
> > libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
> > dri2_create_image_khr_texture
> > 
> > Do someone know how the dmabuf import is tested when the support
> > has
> > been added to glimagesink? Or some pointers how to continue with
> > debugging?

So far the DMABuf support in glimagesink has been tested on Intel/Mesa
and libMALI. There is work in progress in Gallium/Mesa, but until
recently there was no support for offset in imported buffer, which
would result in BAD_MATCH error. I cannot guaranty this is the exact
reason here, BAD_MATCH is used for a very wide variety of reason in
those extensions. The right place to dig into this issue would be
through the Mesa list and/or Mesa code. Find out what is missing for
you driver in Mesa and then I may help you further.

For the reference, the importation strategy we use in GStreamer has
been inspired of Kodi (xmbc). It consist of importing each YUV plane
seperatly using R8 and RG88 textures and doing the color conversion
using shaders. Though, if the frame is allocated as a single DMABuf,
this requires using offset to access the frame data, and that support
had only been recently added in Gallium base code and in Radeon driver
recently. I don't know if Freedreno, VC4 have that, and I know nouveau
don't.

cheers,
Nicolas

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

Hi Victor,

On 04/11/2016 03:55 PM, Víctor M. Jáquez L. wrote:
> On 04/11/16 at 03:11pm, Stanimir Varbanov wrote:
>> adding gstreamer-devel
>>
>> On 04/11/2016 03:03 PM, Stanimir Varbanov wrote:
>>> Hi,
>>>
>>> I'm working on QCOM v4l2 video decoder/encoder driver and in order to
>>> test its functionalities I'm using gstreamer v4l2videodec plugin. I am
>>> able to use the v4l2videodec plugin with MMAP, now I want to try the
>>> dmabuf export from v4l2 and import dmabuf buffers to glimagesink. I
>>> upgraded gst to 1.7.91 so that I have the dmabuf support in glimagesink.
>>> Mesa version is 11.1.2.
>>>
>>> I'm using the following pipeline:
>>>
>>> GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
>>> $GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
>>> capture-io-mode=dmabuf ! glimagesink
>>>
>>> I stalled on this error:
>>>
>>> eglimagememory
>>> gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimageallocator0>
>>> eglCreateImage failed: EGL_BAD_MATCH
>>>
>>> which in Mesa is:
>>>
>>> libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
>>> dri2_create_image_khr_texture
>>>
>>> Do someone know how the dmabuf import is tested when the support has
>>> been added to glimagesink? Or some pointers how to continue with
>>> debugging?
> 
> Perhaps this is not useful for your case, but there's a kmssink (a simple
> video sink that uses KMS/DRM kernel API)[1]. It supports dmabuf import and
> rendering, and the way it does it is heavily inspired on how glimagesink does
> it, obviously without the EGL burden, just the kernel's PRIME API.

Thanks for the info, I've searched few times for such an element in
gstreamer. I find it useful and will give it a try when it is merged.


-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

Hi Nicolas,

On 04/11/2016 07:25 PM, Nicolas Dufresne wrote:
> Le lundi 11 avril 2016 à 15:11 +0300, Stanimir Varbanov a écrit :
>> adding gstreamer-devel
>>
>> On 04/11/2016 03:03 PM, Stanimir Varbanov wrote:
>>>
>>> Hi,
>>>
>>> I'm working on QCOM v4l2 video decoder/encoder driver and in order
>>> to
>>> test its functionalities I'm using gstreamer v4l2videodec plugin. I
>>> am
>>> able to use the v4l2videodec plugin with MMAP, now I want to try
>>> the
>>> dmabuf export from v4l2 and import dmabuf buffers to glimagesink. I
>>> upgraded gst to 1.7.91 so that I have the dmabuf support in
>>> glimagesink.
>>> Mesa version is 11.1.2.
> 
> I'm very happy to see this report. So far, we only had report that this
> element works on Freescale IMX.6 (CODA) and Exynos 4/5.

In this context, I would be very happy to see v4l2videoenc merged soon :)

> 
>>>
>>> I'm using the following pipeline:
>>>
>>> GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
>>> $GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
>>> capture-io-mode=dmabuf ! glimagesink
>>>
>>> I stalled on this error:
>>>
>>> eglimagememory
>>> gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimagea
>>> llocator0>
>>> eglCreateImage failed: EGL_BAD_MATCH
>>>
>>> which in Mesa is:
>>>
>>> libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
>>> dri2_create_image_khr_texture
>>>
>>> Do someone know how the dmabuf import is tested when the support
>>> has
>>> been added to glimagesink? Or some pointers how to continue with
>>> debugging?
> 
> So far the DMABuf support in glimagesink has been tested on Intel/Mesa
> and libMALI. There is work in progress in Gallium/Mesa, but until
> recently there was no support for offset in imported buffer, which
> would result in BAD_MATCH error. I cannot guaranty this is the exact
> reason here, BAD_MATCH is used for a very wide variety of reason in
> those extensions. The right place to dig into this issue would be
> through the Mesa list and/or Mesa code. Find out what is missing for
> you driver in Mesa and then I may help you further.

I came down to these conditions

https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/dri/dri2.c?h=11.2#n1063

but I don't know how this is related. The gstreamer
(gst_egl_image_memory_from_dmabuf) doesn't set this "level" so it will
be zero.

> 
> For the reference, the importation strategy we use in GStreamer has
> been inspired of Kodi (xmbc). It consist of importing each YUV plane
> seperatly using R8 and RG88 textures and doing the color conversion
> using shaders. Though, if the frame is allocated as a single DMABuf,
> this requires using offset to access the frame data, and that support

Yep that is my case, the driver capture buffers has one plain, hence one
dmabuf will be exported per buffer.

> had only been recently added in Gallium base code and in Radeon driver
> recently. I don't know if Freedreno, VC4 have that, and I know nouveau
> don't.

Rob, do we need to add something in Freedreno Gallium driver to handle
dmabuf import?

-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

<snip>

>>>>
>>>> I'm using the following pipeline:
>>>>
>>>> GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
>>>> $GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
>>>> capture-io-mode=dmabuf ! glimagesink
>>>>
>>>> I stalled on this error:
>>>>
>>>> eglimagememory
>>>> gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimagea
>>>> llocator0>
>>>> eglCreateImage failed: EGL_BAD_MATCH
>>>>
>>>> which in Mesa is:
>>>>
>>>> libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
>>>> dri2_create_image_khr_texture
>>>>
>>>> Do someone know how the dmabuf import is tested when the support
>>>> has
>>>> been added to glimagesink? Or some pointers how to continue with
>>>> debugging?
>>
>> So far the DMABuf support in glimagesink has been tested on Intel/Mesa
>> and libMALI. There is work in progress in Gallium/Mesa, but until
>> recently there was no support for offset in imported buffer, which
>> would result in BAD_MATCH error. I cannot guaranty this is the exact
>> reason here, BAD_MATCH is used for a very wide variety of reason in
>> those extensions. The right place to dig into this issue would be
>> through the Mesa list and/or Mesa code. Find out what is missing for
>> you driver in Mesa and then I may help you further.
> 
> I came down to these conditions
> 
> https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/dri/dri2.c?h=11.2#n1063
> 
> but I don't know how this is related. The gstreamer
> (gst_egl_image_memory_from_dmabuf) doesn't set this "level" so it will
> be zero.

That was wrong assumption, the error comes from another Mesa function.
I'm not sure still which one dri2_from_dma_bufs() or
dri2_create_image_dma_buf(). Will try to rebuild Mesa.

-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

Le mardi 12 avril 2016 à 11:57 +0300, Stanimir Varbanov a écrit :
> > I'm very happy to see this report. So far, we only had report that
> this
> > element works on Freescale IMX.6 (CODA) and Exynos 4/5.
> 
> In this context, I would be very happy to see v4l2videoenc merged
> soon :)

That will happen when all review comments are resolved.

cheers,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: gstreamer: v4l2videodec plugin

[Rob Clark]

On Tue, Apr 12, 2016 at 4:57 AM, Stanimir Varbanov
<stanimir.varbanov@linaro.org> wrote:
> Hi Nicolas,
>
> On 04/11/2016 07:25 PM, Nicolas Dufresne wrote:
>> Le lundi 11 avril 2016 à 15:11 +0300, Stanimir Varbanov a écrit :
>>> adding gstreamer-devel
>>>
>>> On 04/11/2016 03:03 PM, Stanimir Varbanov wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm working on QCOM v4l2 video decoder/encoder driver and in order
>>>> to
>>>> test its functionalities I'm using gstreamer v4l2videodec plugin. I
>>>> am
>>>> able to use the v4l2videodec plugin with MMAP, now I want to try
>>>> the
>>>> dmabuf export from v4l2 and import dmabuf buffers to glimagesink. I
>>>> upgraded gst to 1.7.91 so that I have the dmabuf support in
>>>> glimagesink.
>>>> Mesa version is 11.1.2.
>>
>> I'm very happy to see this report. So far, we only had report that this
>> element works on Freescale IMX.6 (CODA) and Exynos 4/5.
>
> In this context, I would be very happy to see v4l2videoenc merged soon :)
>
>>
>>>>
>>>> I'm using the following pipeline:
>>>>
>>>> GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 $GSTDEBUG
>>>> $GSTFILESRC ! qtdemux name=m m.video_0 ! h264parse ! v4l2video32dec
>>>> capture-io-mode=dmabuf ! glimagesink
>>>>
>>>> I stalled on this error:
>>>>
>>>> eglimagememory
>>>> gsteglimagememory.c:473:gst_egl_image_memory_from_dmabuf:<eglimagea
>>>> llocator0>
>>>> eglCreateImage failed: EGL_BAD_MATCH
>>>>
>>>> which in Mesa is:
>>>>
>>>> libEGL debug: EGL user error 0x3009 (EGL_BAD_MATCH) in
>>>> dri2_create_image_khr_texture
>>>>
>>>> Do someone know how the dmabuf import is tested when the support
>>>> has
>>>> been added to glimagesink? Or some pointers how to continue with
>>>> debugging?
>>
>> So far the DMABuf support in glimagesink has been tested on Intel/Mesa
>> and libMALI. There is work in progress in Gallium/Mesa, but until
>> recently there was no support for offset in imported buffer, which
>> would result in BAD_MATCH error. I cannot guaranty this is the exact
>> reason here, BAD_MATCH is used for a very wide variety of reason in
>> those extensions. The right place to dig into this issue would be
>> through the Mesa list and/or Mesa code. Find out what is missing for
>> you driver in Mesa and then I may help you further.
>
> I came down to these conditions
>
> https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/dri/dri2.c?h=11.2#n1063
>
> but I don't know how this is related. The gstreamer
> (gst_egl_image_memory_from_dmabuf) doesn't set this "level" so it will
> be zero.
>
>>
>> For the reference, the importation strategy we use in GStreamer has
>> been inspired of Kodi (xmbc). It consist of importing each YUV plane
>> seperatly using R8 and RG88 textures and doing the color conversion
>> using shaders. Though, if the frame is allocated as a single DMABuf,
>> this requires using offset to access the frame data, and that support
>
> Yep that is my case, the driver capture buffers has one plain, hence one
> dmabuf will be exported per buffer.
>
>> had only been recently added in Gallium base code and in Radeon driver
>> recently. I don't know if Freedreno, VC4 have that, and I know nouveau
>> don't.
>
> Rob, do we need to add something in Freedreno Gallium driver to handle
> dmabuf import?

The issue is probably the YUV format, which we cannot really deal with
properly in gallium..  it's a similar issue to multi-planer even if it
is in a single buffer.

The best way to handle this would be to import the same dmabuf fd
twice, with appropriate offsets, to create one GL_RED eglimage for Y
and one GL_RG eglimage for UV, and then combine them in shader in a
similar way to how you'd handle separate Y and UV planes..

BR,
-R

> --
> regards,
> Stan

Re: gstreamer: v4l2videodec plugin

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Le vendredi 15 avril 2016 à 11:58 -0400, Rob Clark a écrit :
> The issue is probably the YUV format, which we cannot really deal
> with
> properly in gallium..  it's a similar issue to multi-planer even if
> it
> is in a single buffer.
> 
> The best way to handle this would be to import the same dmabuf fd
> twice, with appropriate offsets, to create one GL_RED eglimage for Y
> and one GL_RG eglimage for UV, and then combine them in shader in a
> similar way to how you'd handle separate Y and UV planes..

That's the strategy we use in GStreamer, as very few GL stack support
implicit color conversions. For that to work you need to implement the
"offset" field in winsys_handle, that was added recently, and make sure
you have R8 and RG88 support (usually this is just mapping).

cheers,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: gstreamer: v4l2videodec plugin

[Rob Clark]

On Fri, Apr 15, 2016 at 12:09 PM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> Le vendredi 15 avril 2016 à 11:58 -0400, Rob Clark a écrit :
>> The issue is probably the YUV format, which we cannot really deal
>> with
>> properly in gallium..  it's a similar issue to multi-planer even if
>> it
>> is in a single buffer.
>>
>> The best way to handle this would be to import the same dmabuf fd
>> twice, with appropriate offsets, to create one GL_RED eglimage for Y
>> and one GL_RG eglimage for UV, and then combine them in shader in a
>> similar way to how you'd handle separate Y and UV planes..
>
> That's the strategy we use in GStreamer, as very few GL stack support
> implicit color conversions. For that to work you need to implement the
> "offset" field in winsys_handle, that was added recently, and make sure
> you have R8 and RG88 support (usually this is just mapping).

oh, heh, looks like nobody bothered to add this yet:

---------
diff --git a/src/gallium/drivers/freedreno/freedreno_resource.c
b/src/gallium/drivers/freedreno/freedreno_resource.c
index 9aded3b..fab78ab 100644
--- a/src/gallium/drivers/freedreno/freedreno_resource.c
+++ b/src/gallium/drivers/freedreno/freedreno_resource.c
@@ -669,6 +669,7 @@ fd_resource_from_handle(struct pipe_screen *pscreen,
     rsc->base.vtbl = &fd_resource_vtbl;
     rsc->cpp = util_format_get_blocksize(tmpl->format);
     slice->pitch /= rsc->cpp;
+    slice->offset = handle->offset;

     assert(rsc->cpp);
---------


> cheers,
> Nicolas

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

On 04/15/2016 07:09 PM, Nicolas Dufresne wrote:
> Le vendredi 15 avril 2016 à 11:58 -0400, Rob Clark a écrit :
>> The issue is probably the YUV format, which we cannot really deal
>> with
>> properly in gallium..  it's a similar issue to multi-planer even if
>> it
>> is in a single buffer.
>>
>> The best way to handle this would be to import the same dmabuf fd
>> twice, with appropriate offsets, to create one GL_RED eglimage for Y
>> and one GL_RG eglimage for UV, and then combine them in shader in a
>> similar way to how you'd handle separate Y and UV planes..
> 
> That's the strategy we use in GStreamer, as very few GL stack support
> implicit color conversions. For that to work you need to implement the
> "offset" field in winsys_handle, that was added recently, and make sure
> you have R8 and RG88 support (usually this is just mapping).

Thanks,

OK, I have made the relevant changes in Mesa and now I have image but
the U and V components are swapped (the format is NV12, the UV plane is
at the same buffer but at offset). Digging further and tracing gstreamer
with apitrace I'm observing something weird.

The gst import dmabuf with following call:

eglCreateImageKHR(dpy = 0x7fa8013030, ctx = NULL, target =
EGL_LINUX_DMA_BUF_EXT, buffer = NULL, attrib_list = {EGL_WIDTH, 640,
EGL_HEIGHT, 360, EGL_LINUX_DRM_FOURCC_EXT, 943215175,
EGL_DMA_BUF_PLANE0_FD_EXT, 29, EGL_DMA_BUF_PLANE0_OFFSET_EXT, 942080,
EGL_DMA_BUF_PLANE0_PITCH_EXT, 1280, EGL_NONE}) = 0x7f980027d0

the fourcc format is DRM_FORMAT_GR88 (943215175 decimal).

after that:

glTexImage2D(target = GL_TEXTURE_2D, level = 0, internalformat = GL_RG8,
width = 640, height = 360, border = 0, format = GL_RG, type =
GL_UNSIGNED_BYTE, pixels = NULL)

and finally on the fragment shader we have:

yuv.x=texture2D(Ytex, texcoord * tex_scale0).r;
yuv.yz=texture2D(UVtex, texcoord * tex_scale1).rg;

I was expecting to see DRM_FORMAT_RG88 / GL_RG and shader sampling
y <- r
z <- g

or DRM_FORMAT_GR88 / GL_RG and shader sampling
y <- g
z <- r

Also, browsing the code in Mesa for Intel i965 dri driver I found where
the __DRI_IMAGE_FORMAT_GR88 becomes MESA_FORMAT_R8G8_UNORM [1].

So I'm wondering is that intensional?

Depending on the answer I should make the same in the Gallium dri2 in
dri2_from_dma_bufs().

-- 
regards,
Stan

[1]
https://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/common/dri_util.c#n878

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

cc: mesa-dev ML

On 04/28/2016 02:33 PM, Stanimir Varbanov wrote:
> On 04/15/2016 07:09 PM, Nicolas Dufresne wrote:
>> Le vendredi 15 avril 2016 à 11:58 -0400, Rob Clark a écrit :
>>> The issue is probably the YUV format, which we cannot really deal
>>> with
>>> properly in gallium..  it's a similar issue to multi-planer even if
>>> it
>>> is in a single buffer.
>>>
>>> The best way to handle this would be to import the same dmabuf fd
>>> twice, with appropriate offsets, to create one GL_RED eglimage for Y
>>> and one GL_RG eglimage for UV, and then combine them in shader in a
>>> similar way to how you'd handle separate Y and UV planes..
>>
>> That's the strategy we use in GStreamer, as very few GL stack support
>> implicit color conversions. For that to work you need to implement the
>> "offset" field in winsys_handle, that was added recently, and make sure
>> you have R8 and RG88 support (usually this is just mapping).
> 
> Thanks,
> 
> OK, I have made the relevant changes in Mesa and now I have image but
> the U and V components are swapped (the format is NV12, the UV plane is
> at the same buffer but at offset). Digging further and tracing gstreamer
> with apitrace I'm observing something weird.
> 
> The gst import dmabuf with following call:
> 
> eglCreateImageKHR(dpy = 0x7fa8013030, ctx = NULL, target =
> EGL_LINUX_DMA_BUF_EXT, buffer = NULL, attrib_list = {EGL_WIDTH, 640,
> EGL_HEIGHT, 360, EGL_LINUX_DRM_FOURCC_EXT, 943215175,
> EGL_DMA_BUF_PLANE0_FD_EXT, 29, EGL_DMA_BUF_PLANE0_OFFSET_EXT, 942080,
> EGL_DMA_BUF_PLANE0_PITCH_EXT, 1280, EGL_NONE}) = 0x7f980027d0
> 
> the fourcc format is DRM_FORMAT_GR88 (943215175 decimal).
> 
> after that:
> 
> glTexImage2D(target = GL_TEXTURE_2D, level = 0, internalformat = GL_RG8,
> width = 640, height = 360, border = 0, format = GL_RG, type =
> GL_UNSIGNED_BYTE, pixels = NULL)
> 
> and finally on the fragment shader we have:
> 
> yuv.x=texture2D(Ytex, texcoord * tex_scale0).r;
> yuv.yz=texture2D(UVtex, texcoord * tex_scale1).rg;
> 
> I was expecting to see DRM_FORMAT_RG88 / GL_RG and shader sampling
> y <- r
> z <- g
> 
> or DRM_FORMAT_GR88 / GL_RG and shader sampling
> y <- g
> z <- r
> 
> Also, browsing the code in Mesa for Intel i965 dri driver I found where
> the __DRI_IMAGE_FORMAT_GR88 becomes MESA_FORMAT_R8G8_UNORM [1].
> 
> So I'm wondering is that intensional?
> 
> Depending on the answer I should make the same in the Gallium dri2 in
> dri2_from_dma_bufs().
> 

Re: gstreamer: v4l2videodec plugin

[Rob Clark]

On Thu, Apr 28, 2016 at 7:33 AM, Stanimir Varbanov
<stanimir.varbanov@linaro.org> wrote:
> On 04/15/2016 07:09 PM, Nicolas Dufresne wrote:
>> Le vendredi 15 avril 2016 à 11:58 -0400, Rob Clark a écrit :
>>> The issue is probably the YUV format, which we cannot really deal
>>> with
>>> properly in gallium..  it's a similar issue to multi-planer even if
>>> it
>>> is in a single buffer.
>>>
>>> The best way to handle this would be to import the same dmabuf fd
>>> twice, with appropriate offsets, to create one GL_RED eglimage for Y
>>> and one GL_RG eglimage for UV, and then combine them in shader in a
>>> similar way to how you'd handle separate Y and UV planes..
>>
>> That's the strategy we use in GStreamer, as very few GL stack support
>> implicit color conversions. For that to work you need to implement the
>> "offset" field in winsys_handle, that was added recently, and make sure
>> you have R8 and RG88 support (usually this is just mapping).
>
> Thanks,
>
> OK, I have made the relevant changes in Mesa and now I have image but
> the U and V components are swapped (the format is NV12, the UV plane is
> at the same buffer but at offset). Digging further and tracing gstreamer
> with apitrace I'm observing something weird.
>
> The gst import dmabuf with following call:
>
> eglCreateImageKHR(dpy = 0x7fa8013030, ctx = NULL, target =
> EGL_LINUX_DMA_BUF_EXT, buffer = NULL, attrib_list = {EGL_WIDTH, 640,
> EGL_HEIGHT, 360, EGL_LINUX_DRM_FOURCC_EXT, 943215175,
> EGL_DMA_BUF_PLANE0_FD_EXT, 29, EGL_DMA_BUF_PLANE0_OFFSET_EXT, 942080,
> EGL_DMA_BUF_PLANE0_PITCH_EXT, 1280, EGL_NONE}) = 0x7f980027d0
>
> the fourcc format is DRM_FORMAT_GR88 (943215175 decimal).
>
> after that:
>
> glTexImage2D(target = GL_TEXTURE_2D, level = 0, internalformat = GL_RG8,
> width = 640, height = 360, border = 0, format = GL_RG, type =
> GL_UNSIGNED_BYTE, pixels = NULL)
>
> and finally on the fragment shader we have:
>
> yuv.x=texture2D(Ytex, texcoord * tex_scale0).r;
> yuv.yz=texture2D(UVtex, texcoord * tex_scale1).rg;
>
> I was expecting to see DRM_FORMAT_RG88 / GL_RG and shader sampling
> y <- r
> z <- g
>
> or DRM_FORMAT_GR88 / GL_RG and shader sampling
> y <- g
> z <- r

IIRC you are using gles?  Could you recompile glimagesink to use
desktop GL?  I'm wondering a bit, but just speculation since I don't
have a way to step through it, but the 'if (_mesa_is_gles())' case in
st_ChooseTextureFormat..  normally for gles the driver is more free to
choose the corresponding internal-format, which is maybe not the right
thing to do for textures which are imported eglimages.

(if recompiling mesa is easier, you could just change that to 'if (0)'
and see if it "fixes" things.. that ofc is not the proper fix, but it
would confirm whether this is what is going on..)

BR,
-R

> Also, browsing the code in Mesa for Intel i965 dri driver I found where
> the __DRI_IMAGE_FORMAT_GR88 becomes MESA_FORMAT_R8G8_UNORM [1].
>
> So I'm wondering is that intensional?
>
> Depending on the answer I should make the same in the Gallium dri2 in
> dri2_from_dma_bufs().
>
> --
> regards,
> Stan
>
> [1]
> https://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/common/dri_util.c#n878
>

Re: gstreamer: v4l2videodec plugin

[Nicolas Dufresne]

Le jeudi 28 avril 2016 à 14:33 +0300, Stanimir Varbanov a écrit :
> So I'm wondering is that intensional?
> 
> Depending on the answer I should make the same in the Gallium dri2 in
> dri2_from_dma_bufs().

It's DRI format that are confusing. In GStreamer DRI_FORMAT_GR88 would
be named RG88 (if it existed). That's because DRI API present it in the
way it would look on the CPU after loading that 16bit word into a
register. In GStreamer instead, we expose is as the way you'll find it
in the data array. Let's see it this way, DRI present the information
in a way people writing rasterizer see it.


Nicolas

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

On 05/02/2016 12:48 AM, Nicolas Dufresne wrote:
> Le jeudi 28 avril 2016 à 14:33 +0300, Stanimir Varbanov a écrit :
>> So I'm wondering is that intensional?
>>
>> Depending on the answer I should make the same in the Gallium dri2 in
>> dri2_from_dma_bufs().
> 
> It's DRI format that are confusing. In GStreamer DRI_FORMAT_GR88 would
> be named RG88 (if it existed). That's because DRI API present it in the
> way it would look on the CPU after loading that 16bit word into a
> register. In GStreamer instead, we expose is as the way you'll find it
> in the data array. Let's see it this way, DRI present the information
> in a way people writing rasterizer see it.

Thanks Nicolas,

I will cook up patches based on this for Gallium dri2 and will see the
comments.

-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

cc: ayaka

On 04/12/2016 06:57 PM, Nicolas Dufresne wrote:
> Le mardi 12 avril 2016 à 11:57 +0300, Stanimir Varbanov a écrit :
>>> I'm very happy to see this report. So far, we only had report that
>> this
>>> element works on Freescale IMX.6 (CODA) and Exynos 4/5.
>>
>> In this context, I would be very happy to see v4l2videoenc merged
>> soon :)
> 
> That will happen when all review comments are resolved.

yes, of course :)

Just FYI, I applied the WIP patches on my side and I'm very happy to
report that they just works. So If you need some testing on qcom video
accelerator just ping me.

One thing which bothers me is how the extra-controls property working,
i.e. I failed to change the h264 profile for example with below construct:

extra-controls="controls,h264_profile=4;"


-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 1002 bytes --]

Le vendredi 13 mai 2016 à 11:45 +0300, Stanimir Varbanov a écrit :
> yes, of course :)
> 
> Just FYI, I applied the WIP patches on my side and I'm very happy to
> report that they just works. So If you need some testing on qcom
> video
> accelerator just ping me.
> 
> One thing which bothers me is how the extra-controls property
> working,
> i.e. I failed to change the h264 profile for example with below
> construct:
> 
> extra-controls="controls,h264_profile=4;"

While I got you. I would be very interested to know who QCOM driver
interpreted the width and height expose on capture side of the decoder.
I'm adding Philippe Zabel in CC here (he's maintaining the
CODA/Freescale decoder). In Samsung MFC driver, the width and height
expose by the driver is the display size. Though, recently, Philippe is
reporting that his driver is exposing the coded size. Fixing one in
GStreamer will break the other, so I was wondering what other drivers
are doing.

cheers,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: gstreamer: v4l2videodec plugin

[Stanimir Varbanov]

On 05/14/2016 01:59 PM, Nicolas Dufresne wrote:
> Le vendredi 13 mai 2016 à 11:45 +0300, Stanimir Varbanov a écrit :
>> yes, of course :)
>>
>> Just FYI, I applied the WIP patches on my side and I'm very happy to
>> report that they just works. So If you need some testing on qcom
>> video
>> accelerator just ping me.
>>
>> One thing which bothers me is how the extra-controls property
>> working,
>> i.e. I failed to change the h264 profile for example with below
>> construct:
>>
>> extra-controls="controls,h264_profile=4;"
> 
> While I got you. I would be very interested to know who QCOM driver
> interpreted the width and height expose on capture side of the decoder.
> I'm adding Philippe Zabel in CC here (he's maintaining the
> CODA/Freescale decoder). In Samsung MFC driver, the width and height
> expose by the driver is the display size. Though, recently, Philippe is
> reporting that his driver is exposing the coded size. Fixing one in
> GStreamer will break the other, so I was wondering what other drivers
> are doing.

In qcom driver s_fmt on capture queue will return bigger or the same as
coded resolution depending on the width/height. This is related to hw
alignment restrictions i.e 1280x720 on output queue will become 1280x736
on capture queue. The the userspace can/must call g_crop to receive the
active resolution for displaying.

-- 
regards,
Stan

Re: gstreamer: v4l2videodec plugin

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

Le vendredi 13 mai 2016 à 11:45 +0300, Stanimir Varbanov a écrit :
> One thing which bothers me is how the extra-controls property
> working,
> i.e. I failed to change the h264 profile for example with below
> construct:
> 
> extra-controls="controls,h264_profile=4;"

Yes, and profile should be negotiated with downstream in GStreamer. For
common controls, like bitrate, it should be exposed as separate
properties instead.

Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: gstreamer: v4l2videodec plugin

[ayaka]

在 2016年05月15日 15:41, Nicolas Dufresne 写道:
> Le vendredi 13 mai 2016 à 11:45 +0300, Stanimir Varbanov a écrit :
>> One thing which bothers me is how the extra-controls property
>> working,
>> i.e. I failed to change the h264 profile for example with below
>> construct:
>>
>> extra-controls="controls,h264_profile=4;"
> Yes, and profile should be negotiated with downstream in GStreamer. For
> common controls, like bitrate, it should be exposed as separate
> properties instead.
Please try the new patches in
https://github.com/hizukiayaka/gst-plugins-good
And look at this commit
https://github.com/hizukiayaka/gst-plugins-good/commit/92b99ba9322cf8a1039b877315b12bc9813e95cf

NOTE: even you could set those extra controls and driver accepted. It 
doesn't mean it would work.
I looked at the s5p-mfc driver, it just set it then leave it alone. I 
may fix this bug in the few weeks.
>
> Nicolas

Re: gstreamer: v4l2videodec plugin

[Philipp Zabel]

Am Samstag, den 14.05.2016, 15:07 +0300 schrieb Stanimir Varbanov:
[...]
> > While I got you. I would be very interested to know who QCOM driver
> > interpreted the width and height expose on capture side of the decoder.
> > I'm adding Philippe Zabel in CC here (he's maintaining the
> > CODA/Freescale decoder). In Samsung MFC driver, the width and height
> > expose by the driver is the display size. Though, recently, Philippe is
> > reporting that his driver is exposing the coded size. Fixing one in
> > GStreamer will break the other, so I was wondering what other drivers
> > are doing.
> 
> In qcom driver s_fmt on capture queue will return bigger or the same as
> coded resolution depending on the width/height. This is related to hw
> alignment restrictions i.e 1280x720 on output queue will become 1280x736
> on capture queue. The the userspace can/must call g_crop to receive the
> active resolution for displaying.

Since in that case the input video is nominally 1280x720, and we are not
cropping away anything from that, this shouldn't use G_CROP (which maps
to G_SELECTION with V4L2_SEL_TGT_CROP_ACTIVE), but G_SELECTION with
V4L2_SEL_TGT_COMPOSE_DEFAULT on the capture queue.

For mem2mem devices, cropping should happen at the output queue, and
composing at the capture queue. For devices that don't scale, such as
decoders, the output crop rectangle should have the same size as the
capture compose rectangle.

Is there any reason not to update the MFC driver to use G_SELECTION? The
g_crop implementation could be kept for backwards compatibility.

regards
Philipp

Re: gstreamer: v4l2videodec plugin

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

Le jeudi 19 mai 2016 à 15:25 +0200, Philipp Zabel a écrit :
> Is there any reason not to update the MFC driver to use G_SELECTION?
> The
> g_crop implementation could be kept for backwards compatibility.

Videobuf2 already provide this backward compatible implementation, so
there is no reason not to port that driver (if this is not done
already, maybe ask Kamil ?).

Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]