imx6: webkit html5 video with hardware acceleration

imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

Dear list,

I have been trying to get hardware accelerated html5 video working on
the imx6 for a some time now. After trying 'daisy' and also the more
recent alpha and beta branches, I have come to the conclusion that
this does not work (and never has). Looking at the webkit source and
also googling around only confirm my suspicions.

Does anyone out there have this working?

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Gerard Bucas]

Hi Richard

We have trying EXACTLY the same thing & although we see a slight improvement on the latest beta BSP, I agree with your conclusion. 

This is a serious problem for practical applications of the I.MX6 - any application that requires HTML5 playback or a browser in general. 

See the thread to this list that I started yesterday:

Re: [meta-freescale] Any hope of vblank-synchronized rendering (from vivante) on i.MX6 soon?

Glad we have yet another concerned user!! 😃

Regards

Gerard Bucas



> On 23 Sep 2014, at 14:13, Richard Cochran <richardcochran@gmail.com> wrote:
> 
> Dear list,
> 
> I have been trying to get hardware accelerated html5 video working on
> the imx6 for a some time now. After trying 'daisy' and also the more
> recent alpha and beta branches, I have come to the conclusion that
> this does not work (and never has). Looking at the webkit source and
> also googling around only confirm my suspicions.
> 
> Does anyone out there have this working?
> 
> Thanks,
> Richard
> -- 
> _______________________________________________
> meta-freescale mailing list
> meta-freescale@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-freescale

Re: imx6: webkit html5 video with hardware acceleration

[Bin Lin]

Hi Richard,

We tried to get the HTML5 playback working with Qt5.2.1 webkit on daisy 
branch.  It hangs when it is trying to play.  We tried it on both the 
youtubeview example of Qt5 webkitqml and the browser example of 
webkitwidgets, eglfs setup.  The results are the same.

What are the steps you followed to make the HTML5 at least to play under 
webkit?

Regards,

Bin

On 9/23/2014 11:33 AM, Gerard Bucas wrote:
> Hi Richard
>
> We have trying EXACTLY the same thing & although we see a slight improvement on the latest beta BSP, I agree with your conclusion.
>
> This is a serious problem for practical applications of the I.MX6 - any application that requires HTML5 playback or a browser in general.
>
> See the thread to this list that I started yesterday:
>
> Re: [meta-freescale] Any hope of vblank-synchronized rendering (from vivante) on i.MX6 soon?
>
> Glad we have yet another concerned user!! 😃
>
> Regards
>
> Gerard Bucas
>
>
>
>> On 23 Sep 2014, at 14:13, Richard Cochran <richardcochran@gmail.com> wrote:
>>
>> Dear list,
>>
>> I have been trying to get hardware accelerated html5 video working on
>> the imx6 for a some time now. After trying 'daisy' and also the more
>> recent alpha and beta branches, I have come to the conclusion that
>> this does not work (and never has). Looking at the webkit source and
>> also googling around only confirm my suspicions.
>>
>> Does anyone out there have this working?
>>
>> Thanks,
>> Richard
>> -- 
>> _______________________________________________
>> meta-freescale mailing list
>> meta-freescale@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/meta-freescale

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

[-- Attachment #1: Type: text/plain, Size: 3185 bytes --]

This is because by default, WebKit will use GStreamer for decoding (and 
therefore automatically use hw accelerated decoding if an appropriate 
plugin is installed), but will copy the decoded pixels with the CPU. 
Efficient transmitting of decoded frames to the display is just as 
important as hw accelerated decoding, if not more. Yet, many people only 
care about the decoding part. On embedded platforms like the imx6, this 
means that zerocopy must be enforced, that is, ensure that _no_ copies 
of frames are made with the CPU. If copies are necessary, they must be 
done by using DMA transfers only.

With stock WebKit, you can pretty much forget about that. Look at this 
source: 
https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/VideoSinkGStreamer.cpp#L138 
note how it copies pixels with the CPU.

There is apparently a GL-based alternative, but I haven't seen it in 
action yet. Also, if TEXTURE_MAPPER_GL implies Desktop OpenGL (without 
being able to choose OpenGL ES instead), it means it will use the wrong 
API for embedded devices (*never* use the regular OpenGL, since it is 
made for desktops - use OpenGL ES instead). With OpenGL ES, there is a 
Vivante GPU specific extension that allows for zerocopy. But this has to 
be integrated into WebKit, which is probably rather difficult.

On 2014-09-23 20:55, Bin Lin wrote:
> Hi Richard,
>
> We tried to get the HTML5 playback working with Qt5.2.1 webkit on 
> daisy branch.  It hangs when it is trying to play.  We tried it on 
> both the youtubeview example of Qt5 webkitqml and the browser example 
> of webkitwidgets, eglfs setup.  The results are the same.
>
> What are the steps you followed to make the HTML5 at least to play 
> under webkit?
>
> Regards,
>
> Bin
>
> On 9/23/2014 11:33 AM, Gerard Bucas wrote:
>> Hi Richard
>>
>> We have trying EXACTLY the same thing & although we see a slight 
>> improvement on the latest beta BSP, I agree with your conclusion.
>>
>> This is a serious problem for practical applications of the I.MX6 - 
>> any application that requires HTML5 playback or a browser in general.
>>
>> See the thread to this list that I started yesterday:
>>
>> Re: [meta-freescale] Any hope of vblank-synchronized rendering (from 
>> vivante) on i.MX6 soon?
>>
>> Glad we have yet another concerned user!! 😃
>>
>> Regards
>>
>> Gerard Bucas
>>
>>
>>
>>> On 23 Sep 2014, at 14:13, Richard Cochran <richardcochran@gmail.com> 
>>> wrote:
>>>
>>> Dear list,
>>>
>>> I have been trying to get hardware accelerated html5 video working on
>>> the imx6 for a some time now. After trying 'daisy' and also the more
>>> recent alpha and beta branches, I have come to the conclusion that
>>> this does not work (and never has). Looking at the webkit source and
>>> also googling around only confirm my suspicions.
>>>
>>> Does anyone out there have this working?
>>>
>>> Thanks,
>>> Richard
>>> -- 
>>> _______________________________________________
>>> meta-freescale mailing list
>>> meta-freescale@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/meta-freescale
>


[-- Attachment #2: Type: text/html, Size: 5113 bytes --]

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 11:55:55AM -0700, Bin Lin wrote:
> We tried to get the HTML5 playback working with Qt5.2.1 webkit on
> daisy branch.  It hangs when it is trying to play.  We tried it on
> both the youtubeview example of Qt5 webkitqml and the browser
> example of webkitwidgets, eglfs setup.  The results are the same.

Right, both Qt4 and Qt5 webkits also don't work.

> What are the steps you followed to make the HTML5 at least to play
> under webkit?

That is the point. It doesn't play videos.

The 'daisy' branch has midori, and I was able to build midori without
any issues, and this builds webkit.

IIRC, you can see that webkit-gtk will start gstreamer, but there is
no picture at all.

Sorry,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 09:02:03PM +0200, Carlos Rafael Giani wrote:
> 
> With stock WebKit, you can pretty much forget about that. Look at
> this source: https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/VideoSinkGStreamer.cpp#L138
> note how it copies pixels with the CPU.

Right, I saw that, too.

For zero copy, you would need a imx6-specific hack^H^H^H^H modification.

So the first question becomes is this.

Q. Does Freescale provide any kind of support for hardware accelerated
   HTML5 video on the imx6?

A. No.

I already knew this. The *next* question, which I am trying to ask
here, is, does anyone have this working at all, like with some kind of
unoffical or alpha-quality patch?

> There is apparently a GL-based alternative, but I haven't seen it in
> action yet.

GL is something completely else entirely.

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

On 2014-09-23 21:06, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 11:55:55AM -0700, Bin Lin wrote:
>> We tried to get the HTML5 playback working with Qt5.2.1 webkit on
>> daisy branch.  It hangs when it is trying to play.  We tried it on
>> both the youtubeview example of Qt5 webkitqml and the browser
>> example of webkitwidgets, eglfs setup.  The results are the same.
> Right, both Qt4 and Qt5 webkits also don't work.
>
>> What are the steps you followed to make the HTML5 at least to play
>> under webkit?
> That is the point. It doesn't play videos.
>
> The 'daisy' branch has midori, and I was able to build midori without
> any issues, and this builds webkit.
>
> IIRC, you can see that webkit-gtk will start gstreamer, but there is
> no picture at all.
>
> Sorry,
> Richard

Oh that is probably because it is using GStreamer 0.10. The 0.10 
Freescale plugins for the VPU are known to not work with WebKit. See 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 and 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4156. Check if it was 
built with its GStreamer 0.10 or 1.0 backend. With 1.0 , you can use 
gstreamer-imx for decoding.

[-- Attachment #2: Type: text/html, Size: 2033 bytes --]

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

On 2014-09-23 21:14, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 09:02:03PM +0200, Carlos Rafael Giani wrote:
>> With stock WebKit, you can pretty much forget about that. Look at
>> this source: https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/VideoSinkGStreamer.cpp#L138
>> note how it copies pixels with the CPU.
> Right, I saw that, too.
>
> For zero copy, you would need a imx6-specific hack^H^H^H^H modification.
>
> So the first question becomes is this.
>
> Q. Does Freescale provide any kind of support for hardware accelerated
>     HTML5 video on the imx6?
>
> A. No.
>
> I already knew this. The *next* question, which I am trying to ask
> here, is, does anyone have this working at all, like with some kind of
> unoffical or alpha-quality patch?

It is working for Chromium. I added patches for that. WebKit though, I 
don't know.

>
>> There is apparently a GL-based alternative, but I haven't seen it in
>> action yet.
> GL is something completely else entirely.
>
>

Not necessarily. Keep in mind what I wrote about the direct textures. If 
WebKit has a GLES-based rendering backend, these textures could be used 
for smooth video playback. It'd have to extract the physical address out 
of the decoded GStreamer frame (possible by using the appsink GStreamer 
element) and pass it on to the texture, and release the GStreamer frame 
once that video frame has been shown and the next frame is to be displayed.

I also use the direct textures for smooth zerocopy playback in Chromium.

One remark: Power consumption wise, using a 3D API (and therefore the 
GPU's 3D core) for 2D operations isn't so efficient. But, adding a 
G2D-based backend for example would be much more work. (G2D is a 
highlevel API for controlling the GPU's 2D core, which uses less power.)

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 09:15:46PM +0200, Carlos Rafael Giani wrote:
> 
> Oh that is probably because it is using GStreamer 0.10. The 0.10
> Freescale plugins for the VPU are known to not work with WebKit. See
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 and
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4156.

Yes, I saw those already, thanks. They give zero information as to why
the VPU cannot work with webkit. I should think that it could work, if
it were only implemented!

> Check if it
> was built with its GStreamer 0.10 or 1.0 backend.

It uses 0.10.

> With 1.0 , you can
> use gstreamer-imx for decoding.

Well, you *can* use 1.0, but does it actually work?

If so, can you please share your yocto configuration that builds 
midori/webkit/gstreamer to work with hardware acceleration?

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Bin Lin]

[-- Attachment #1: Type: text/plain, Size: 3620 bytes --]

We are able to play hw accelerated local video under the Qt5 multimedia 
qmlvideo example and believe that it is using Gstreamer as well.  
Looking for a way to do the same under Webkit.  Any connections between 
the two on video playback?

Regards,

Bin




On 9/23/2014 12:02 PM, Carlos Rafael Giani wrote:
> This is because by default, WebKit will use GStreamer for decoding 
> (and therefore automatically use hw accelerated decoding if an 
> appropriate plugin is installed), but will copy the decoded pixels 
> with the CPU. Efficient transmitting of decoded frames to the display 
> is just as important as hw accelerated decoding, if not more. Yet, 
> many people only care about the decoding part. On embedded platforms 
> like the imx6, this means that zerocopy must be enforced, that is, 
> ensure that _no_ copies of frames are made with the CPU. If copies are 
> necessary, they must be done by using DMA transfers only.
>
> With stock WebKit, you can pretty much forget about that. Look at this 
> source: 
> https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/VideoSinkGStreamer.cpp#L138 
> note how it copies pixels with the CPU.
>
> There is apparently a GL-based alternative, but I haven't seen it in 
> action yet. Also, if TEXTURE_MAPPER_GL implies Desktop OpenGL (without 
> being able to choose OpenGL ES instead), it means it will use the 
> wrong API for embedded devices (*never* use the regular OpenGL, since 
> it is made for desktops - use OpenGL ES instead). With OpenGL ES, 
> there is a Vivante GPU specific extension that allows for zerocopy. 
> But this has to be integrated into WebKit, which is probably rather 
> difficult.
>
> On 2014-09-23 20:55, Bin Lin wrote:
>> Hi Richard,
>>
>> We tried to get the HTML5 playback working with Qt5.2.1 webkit on 
>> daisy branch.  It hangs when it is trying to play.  We tried it on 
>> both the youtubeview example of Qt5 webkitqml and the browser example 
>> of webkitwidgets, eglfs setup.  The results are the same.
>>
>> What are the steps you followed to make the HTML5 at least to play 
>> under webkit?
>>
>> Regards,
>>
>> Bin
>>
>> On 9/23/2014 11:33 AM, Gerard Bucas wrote:
>>> Hi Richard
>>>
>>> We have trying EXACTLY the same thing & although we see a slight 
>>> improvement on the latest beta BSP, I agree with your conclusion.
>>>
>>> This is a serious problem for practical applications of the I.MX6 - 
>>> any application that requires HTML5 playback or a browser in general.
>>>
>>> See the thread to this list that I started yesterday:
>>>
>>> Re: [meta-freescale] Any hope of vblank-synchronized rendering (from 
>>> vivante) on i.MX6 soon?
>>>
>>> Glad we have yet another concerned user!! 😃
>>>
>>> Regards
>>>
>>> Gerard Bucas
>>>
>>>
>>>
>>>> On 23 Sep 2014, at 14:13, Richard Cochran 
>>>> <richardcochran@gmail.com> wrote:
>>>>
>>>> Dear list,
>>>>
>>>> I have been trying to get hardware accelerated html5 video working on
>>>> the imx6 for a some time now. After trying 'daisy' and also the more
>>>> recent alpha and beta branches, I have come to the conclusion that
>>>> this does not work (and never has). Looking at the webkit source and
>>>> also googling around only confirm my suspicions.
>>>>
>>>> Does anyone out there have this working?
>>>>
>>>> Thanks,
>>>> Richard
>>>> -- 
>>>> _______________________________________________
>>>> meta-freescale mailing list
>>>> meta-freescale@yoctoproject.org
>>>> https://lists.yoctoproject.org/listinfo/meta-freescale
>>
>
>
>

[-- Attachment #2: Type: text/html, Size: 5983 bytes --]

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

On 2014-09-23 21:23, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 09:15:46PM +0200, Carlos Rafael Giani wrote:
>> Oh that is probably because it is using GStreamer 0.10. The 0.10
>> Freescale plugins for the VPU are known to not work with WebKit. See
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4155 and
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4156.
> Yes, I saw those already, thanks. They give zero information as to why
> the VPU cannot work with webkit. I should think that it could work, if
> it were only implemented!
>
>> Check if it
>> was built with its GStreamer 0.10 or 1.0 backend.
> It uses 0.10.
>
>> With 1.0 , you can
>> use gstreamer-imx for decoding.
> Well, you *can* use 1.0, but does it actually work?
>
> If so, can you please share your yocto configuration that builds
> midori/webkit/gstreamer to work with hardware acceleration?
>
> Thanks,
> Richard

Well, I saw it when I used qtwebkit. It was rather strange, though. When 
the internal WebKit copy of qtwebkit detected GStreamer 1.0 , it 
switched to 1.0 as the backend. Otherwise, is used 0.10. This is strange 
because Qt is (or was? I am not up to date) firmly 0.10 based, at least 
QtMultimedia, and it is generally not possible to use both 0.10 and 1.0 
in the same process. I guess it worked because WebKit was built in the 
multi-process mode.

When I get the time, I'll try again, and see if gstreamer-imx is being 
used for decoding. With any luck, I'll be able to see any frames. You 
can try this: in Yocto, first build gstreamer 1.0 (and 
-base/good/bad/ugly) explicitely. Meaning, call "bitbake gstreamer1.0 
gstreamer1.0-plugins-base gstreamer1.0-plugins-good 
gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav". 
*Then*, try to build WebKit. Use bitbake -c configure, so you get the 
chance to look into the configure log, to see if it is using 0.10 or 
1.0. (If you want to be sure it doesn't use 0.10, first remove 0.10 from 
your Yocto build by running "bitbake -c cleansstate gstreamer 
gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly 
gst-ffmpeg".)

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

[-- Attachment #1: Type: text/plain, Size: 3818 bytes --]

There is none. They use completely different media pipelines and video 
sinks.


> We are able to play hw accelerated local video under the Qt5 
> multimedia qmlvideo example and believe that it is using Gstreamer as 
> well.  Looking for a way to do the same under Webkit.  Any connections 
> between the two on video playback?
>
> Regards,
>
> Bin
>
>
>
>
> On 9/23/2014 12:02 PM, Carlos Rafael Giani wrote:
>> This is because by default, WebKit will use GStreamer for decoding 
>> (and therefore automatically use hw accelerated decoding if an 
>> appropriate plugin is installed), but will copy the decoded pixels 
>> with the CPU. Efficient transmitting of decoded frames to the display 
>> is just as important as hw accelerated decoding, if not more. Yet, 
>> many people only care about the decoding part. On embedded platforms 
>> like the imx6, this means that zerocopy must be enforced, that is, 
>> ensure that _no_ copies of frames are made with the CPU. If copies 
>> are necessary, they must be done by using DMA transfers only.
>>
>> With stock WebKit, you can pretty much forget about that. Look at 
>> this source: 
>> https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/VideoSinkGStreamer.cpp#L138 
>> note how it copies pixels with the CPU.
>>
>> There is apparently a GL-based alternative, but I haven't seen it in 
>> action yet. Also, if TEXTURE_MAPPER_GL implies Desktop OpenGL 
>> (without being able to choose OpenGL ES instead), it means it will 
>> use the wrong API for embedded devices (*never* use the regular 
>> OpenGL, since it is made for desktops - use OpenGL ES instead). With 
>> OpenGL ES, there is a Vivante GPU specific extension that allows for 
>> zerocopy. But this has to be integrated into WebKit, which is 
>> probably rather difficult.
>>
>> On 2014-09-23 20:55, Bin Lin wrote:
>>> Hi Richard,
>>>
>>> We tried to get the HTML5 playback working with Qt5.2.1 webkit on 
>>> daisy branch.  It hangs when it is trying to play.  We tried it on 
>>> both the youtubeview example of Qt5 webkitqml and the browser 
>>> example of webkitwidgets, eglfs setup.  The results are the same.
>>>
>>> What are the steps you followed to make the HTML5 at least to play 
>>> under webkit?
>>>
>>> Regards,
>>>
>>> Bin
>>>
>>> On 9/23/2014 11:33 AM, Gerard Bucas wrote:
>>>> Hi Richard
>>>>
>>>> We have trying EXACTLY the same thing & although we see a slight 
>>>> improvement on the latest beta BSP, I agree with your conclusion.
>>>>
>>>> This is a serious problem for practical applications of the I.MX6 - 
>>>> any application that requires HTML5 playback or a browser in general.
>>>>
>>>> See the thread to this list that I started yesterday:
>>>>
>>>> Re: [meta-freescale] Any hope of vblank-synchronized rendering 
>>>> (from vivante) on i.MX6 soon?
>>>>
>>>> Glad we have yet another concerned user!! 😃
>>>>
>>>> Regards
>>>>
>>>> Gerard Bucas
>>>>
>>>>
>>>>
>>>>> On 23 Sep 2014, at 14:13, Richard Cochran 
>>>>> <richardcochran@gmail.com> wrote:
>>>>>
>>>>> Dear list,
>>>>>
>>>>> I have been trying to get hardware accelerated html5 video working on
>>>>> the imx6 for a some time now. After trying 'daisy' and also the more
>>>>> recent alpha and beta branches, I have come to the conclusion that
>>>>> this does not work (and never has). Looking at the webkit source and
>>>>> also googling around only confirm my suspicions.
>>>>>
>>>>> Does anyone out there have this working?
>>>>>
>>>>> Thanks,
>>>>> Richard
>>>>> -- 
>>>>> _______________________________________________
>>>>> meta-freescale mailing list
>>>>> meta-freescale@yoctoproject.org
>>>>> https://lists.yoctoproject.org/listinfo/meta-freescale
>>>
>>
>>
>>
>
>


[-- Attachment #2: Type: text/html, Size: 6670 bytes --]

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 09:20:52PM +0200, Carlos Rafael Giani wrote:
> It is working for Chromium. I added patches for that.

Where are your patches? chromium.org git?

Reference please, Tag/branch/sha etc?

(I can't really run chromium on my target hardware, too much bloat,
but still it would be interesting to see it on the sabre board.)

> Not necessarily. Keep in mind what I wrote about the direct
> textures. If WebKit has a GLES-based rendering backend, these
> textures could be used for smooth video playback. It'd have to
> extract the physical address out of the decoded GStreamer frame
> (possible by using the appsink GStreamer element) and pass it on to
> the texture, and release the GStreamer frame once that video frame
> has been shown and the next frame is to be displayed.

Sounds like this still need to be implemented. Anyone working on it?

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

https://github.com/Freescale/chromium-imx

This is already integrated in meta-fsl-arm . Check out 
https://github.com/Freescale/meta-fsl-arm/blob/master/browser-layer/recipes-browser/chromium/chromium-imx.inc

On 2014-09-23 21:30, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 09:20:52PM +0200, Carlos Rafael Giani wrote:
>> It is working for Chromium. I added patches for that.
> Where are your patches? chromium.org git?
>
> Reference please, Tag/branch/sha etc?
>
> (I can't really run chromium on my target hardware, too much bloat,
> but still it would be interesting to see it on the sabre board.)
>
>> Not necessarily. Keep in mind what I wrote about the direct
>> textures. If WebKit has a GLES-based rendering backend, these
>> textures could be used for smooth video playback. It'd have to
>> extract the physical address out of the decoded GStreamer frame
>> (possible by using the appsink GStreamer element) and pass it on to
>> the texture, and release the GStreamer frame once that video frame
>> has been shown and the next frame is to be displayed.
> Sounds like this still need to be implemented. Anyone working on it?
>
> Thanks,
> Richard

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 09:29:13PM +0200, Carlos Rafael Giani wrote:
 
> When I get the time, I'll try again, and see if gstreamer-imx is
> being used for decoding. With any luck, I'll be able to see any
> frames. You can try this: in Yocto, first build gstreamer 1.0 (and
> -base/good/bad/ugly) explicitely.

...

> *Then*, try to build WebKit.

Thanks for the tip. Maybe I'll give it a try.

[ This is just the kind of manual bitbake tinkering that makes
  the FSL Yocto such a joy to work with. ]

However, I won't be betting that gstreamer 1.0 will somehow work
better. After all, webkit copies into its Cairo drawable in the sink.
That alone will cause a too high CPU load with a full screen video.

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Bin Lin]

We are currently using GStreamer 0.10.  Will give a try on 1.0.

Thanks,

Bin




On 9/23/2014 12:39 PM, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 09:29:13PM +0200, Carlos Rafael Giani wrote:
>   
>> When I get the time, I'll try again, and see if gstreamer-imx is
>> being used for decoding. With any luck, I'll be able to see any
>> frames. You can try this: in Yocto, first build gstreamer 1.0 (and
>> -base/good/bad/ugly) explicitely.
> ...
>
>> *Then*, try to build WebKit.
> Thanks for the tip. Maybe I'll give it a try.
>
> [ This is just the kind of manual bitbake tinkering that makes
>    the FSL Yocto such a joy to work with. ]
>
> However, I won't be betting that gstreamer 1.0 will somehow work
> better. After all, webkit copies into its Cairo drawable in the sink.
> That alone will cause a too high CPU load with a full screen video.
>
> Thanks,
> Richard

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 09:32:09PM +0200, Carlos Rafael Giani wrote:
> https://github.com/Freescale/chromium-imx
> 
> This is already integrated in meta-fsl-arm . Check out https://github.com/Freescale/meta-fsl-arm/blob/master/browser-layer/recipes-browser/chromium/chromium-imx.inc

Just for the record, this work is quite new, isn't it?

It looks to me like this work is from late August. Earlier this year
(March?) there was some talk about Chromium, and the FSL Yocto UG (Rev
L3.10.31_1.1.0-alpha, 05/2014) also mentions Chromium with HW
acceleration (although I couldn't get it working).

Can you explain the relationship between your patches and the earlier
work, if there is any?

Also, to your knowledge, no one has worked on webkit-gtk to enable HW
acceleration, is that right?

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

On 2014-09-23 21:52, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 09:32:09PM +0200, Carlos Rafael Giani wrote:
>> https://github.com/Freescale/chromium-imx
>>
>> This is already integrated in meta-fsl-arm . Check out https://github.com/Freescale/meta-fsl-arm/blob/master/browser-layer/recipes-browser/chromium/chromium-imx.inc
> Just for the record, this work is quite new, isn't it?
>
> It looks to me like this work is from late August. Earlier this year
> (March?) there was some talk about Chromium, and the FSL Yocto UG (Rev
> L3.10.31_1.1.0-alpha, 05/2014) also mentions Chromium with HW
> acceleration (although I couldn't get it working).
>
> Can you explain the relationship between your patches and the earlier
> work, if there is any?
>
> Also, to your knowledge, no one has worked on webkit-gtk to enable HW
> acceleration, is that right?
>
> Thanks,
> Richard
>
>

Yes, this is quite new. Its main target version is 37.0.2062.94, which 
is also the current stable version of Chromium in meta-browser.

These talks were related regarding some ideas for the implementation, 
but the code itself is completely new.

As for WebKit-GTK, I have heard of some efforts for it, but this was 
told to me offline, and it was months ago. I don't follow its development.

Re: imx6: webkit html5 video with hardware acceleration

[Otavio Salvador]

On Tue, Sep 23, 2014 at 4:52 PM, Richard Cochran
<richardcochran@gmail.com> wrote:
> On Tue, Sep 23, 2014 at 09:32:09PM +0200, Carlos Rafael Giani wrote:
>> https://github.com/Freescale/chromium-imx
>>
>> This is already integrated in meta-fsl-arm . Check out https://github.com/Freescale/meta-fsl-arm/blob/master/browser-layer/recipes-browser/chromium/chromium-imx.inc
>
> Just for the record, this work is quite new, isn't it?
>
> It looks to me like this work is from late August. Earlier this year
> (March?) there was some talk about Chromium, and the FSL Yocto UG (Rev
> L3.10.31_1.1.0-alpha, 05/2014) also mentions Chromium with HW
> acceleration (although I couldn't get it working).
>
> Can you explain the relationship between your patches and the earlier
> work, if there is any?
>
> Also, to your knowledge, no one has worked on webkit-gtk to enable HW
> acceleration, is that right?

Carlos has already made clear it is in the community. It is available
in master branches of Community so use those or port them.

We have several customers using this already and been working fine.

Regarding Qt5, I added the patch for Qt Multimedia to work with
acceleration in Daisy and it has been integrated in 5.3 which is in
use in master.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 04:57:19PM -0300, Otavio Salvador wrote:
> On Tue, Sep 23, 2014 at 4:52 PM, Richard Cochran <richardcochran@gmail.com> wrote:
> > Also, to your knowledge, no one has worked on webkit-gtk to enable HW
> > acceleration, is that right?
> 
> Carlos has already made clear it is in the community. It is available
> in master branches of Community so use those or port them.

Sorry, it really is not clear, at least not to me. If you look at
the other posts in this thread, you will see that I am not the only
one.

So please forgive my ignorance and kindly explain what you mean by
"master branches of Community". In order to port anything, I need to
know 1) the git repositories and 2) the specific commits.

Is there a tag or sha that you can name which contains webkit-gtk with
imx6 HW acceleration support?

> We have several customers using this already and been working fine.

Good for them. I would also like to see it working. Does this mean I
will have to order commercial support from you?

> Regarding Qt5, I added the patch for Qt Multimedia to work with
> acceleration in Daisy and it has been integrated in 5.3 which is in
> use in master.

Well, the Qt5 video player does work, but that is something totally
different, as Carlos pointed out.

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Otavio Salvador]

On Tue, Sep 23, 2014 at 5:12 PM, Richard Cochran
<richardcochran@gmail.com> wrote:
> On Tue, Sep 23, 2014 at 04:57:19PM -0300, Otavio Salvador wrote:
>> On Tue, Sep 23, 2014 at 4:52 PM, Richard Cochran <richardcochran@gmail.com> wrote:
>> > Also, to your knowledge, no one has worked on webkit-gtk to enable HW
>> > acceleration, is that right?
>>
>> Carlos has already made clear it is in the community. It is available
>> in master branches of Community so use those or port them.
>
> Sorry, it really is not clear, at least not to me. If you look at
> the other posts in this thread, you will see that I am not the only
> one.
>
> So please forgive my ignorance and kindly explain what you mean by
> "master branches of Community". In order to port anything, I need to
> know 1) the git repositories and 2) the specific commits.

Are you using FSL Community BSP or FSL Yocto release? It is available
in master branches of the first. More precisely in master branch of
meta-fsl-arm.

> Is there a tag or sha that you can name which contains webkit-gtk with
> imx6 HW acceleration support?

WebKit-Gtk no. I know some work has been started in past but didn't
see any progress.

It is supported in Chromium as said above.

>> We have several customers using this already and been working fine.
>
> Good for them. I would also like to see it working. Does this mean I
> will have to order commercial support from you?

It is public so you can try to do it yourself if you want; if you want
our professional services, yes.

>> Regarding Qt5, I added the patch for Qt Multimedia to work with
>> acceleration in Daisy and it has been integrated in 5.3 which is in
>> use in master.
>
> Well, the Qt5 video player does work, but that is something totally
> different, as Carlos pointed out.

Yes; this was not intended for you explicitly but someone else were
asking about Qt5 use with GStreamer. So I commented as well.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 05:27:08PM -0300, Otavio Salvador wrote:
> 
> WebKit-Gtk no. I know some work has been started in past but didn't
> see any progress.

Okay, thanks for clarifying that.

Do you remember who was working on this?

(We may decide to go ahead and implement this ourselves, but we don't
want to re-invent the wheel.)
 
> It is supported in Chromium as said above.

Right. Chromium triggers the OOM killer on my target board :(

Oh, well.

> Yes; this was not intended for you explicitly but someone else were
> asking about Qt5 use with GStreamer. So I commented as well.

But does HW acceleration work in Qt5-webkit via a HTML5 page?

We found that it does *not* work, but if it did, it would be at least
an alternative to midori/webkit-gtk.

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Carlos Rafael Giani]

On 2014-09-23 22:42, Richard Cochran wrote:
> Right. Chromium triggers the OOM killer on my target board :(
>
> Oh, well.
>

Merely starting Chromium triggers the OOM killer already?
How much RAM does your board have? And how much *free* RAM was available 
when you started Chromium?
Also, are we talking about Chromium 37.0.2062.94 here?

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 10:46:42PM +0200, Carlos Rafael Giani wrote:
> Merely starting Chromium triggers the OOM killer already?

Yup.

> How much RAM does your board have?

256 MB.

> And how much *free* RAM was
> available when you started Chromium?

The Freescale kernel hackery is really hurting us. With the kernel in
the alpha/beta branch, we need 144 MB for CMA. I am sure this is due
to something dumb in the thousands of new FSL kernel patches, but I
haven't had the time to track this down.

The daisy kernel worked on that board (with full screen video) with a
CMA of 64 MB, but unfortunately the newer user space does *not* work
with the daisy kernel. Brilliant stuff, that.

> Also, are we talking about Chromium 37.0.2062.94 here?

It was 35, IIRC.

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Tue, Sep 23, 2014 at 05:27:08PM -0300, Otavio Salvador wrote:
> 
> Are you using FSL Community BSP or FSL Yocto release?

Hm, up until now I didn't realize that these are two different
things. For daisy, I used fsl-community-bsp-platform, but for the
beta, I used fsl-arm-yocto-bsp.

So, if I understood correctly,

  repo init -u git://git.freescale.com/imx/fsl-arm-yocto-bsp.git

gets me the "offical" FSL project, but

  repo init -u https://github.com/Freescale/fsl-community-bsp-platform

gets me the "community" project. Is that right?

Moreover, it looks like both repo manifests share most of the same git
repositories. Can you explain to me what the differences are?

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Otavio Salvador]

On Wed, Sep 24, 2014 at 5:58 AM, Richard Cochran
<richardcochran@gmail.com> wrote:
> On Tue, Sep 23, 2014 at 05:27:08PM -0300, Otavio Salvador wrote:
>>
>> Are you using FSL Community BSP or FSL Yocto release?
>
> Hm, up until now I didn't realize that these are two different
> things. For daisy, I used fsl-community-bsp-platform, but for the
> beta, I used fsl-arm-yocto-bsp.
>
> So, if I understood correctly,
>
>   repo init -u git://git.freescale.com/imx/fsl-arm-yocto-bsp.git
>
> gets me the "offical" FSL project, but
>
>   repo init -u https://github.com/Freescale/fsl-community-bsp-platform
>
> gets me the "community" project. Is that right?
>
> Moreover, it looks like both repo manifests share most of the same git
> repositories. Can you explain to me what the differences are?

The FSL one is a code drop which is used by Freescale for providing a
point on time with tested code. This is the one they test internally
with their QA process and is static so no bugfixes, security updates
and so on.

Once this is done, the updates are ported to the FSL Community BSP
one. This is to provide a reference BSP and platform for people to
build products upon it.

The goal of both are different. I summarised my view of it at
https://community.freescale.com/message/402940#402940

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Wed, Sep 24, 2014 at 08:21:13AM -0300, Otavio Salvador wrote:
> 
> The goal of both are different. I summarised my view of it at
> https://community.freescale.com/message/402940#402940

That does help a lot.

Thanks,
Richard

Re: imx6: webkit html5 video with hardware acceleration

[Daiane Angolini]

So, in order to summarize this issue, I need help to determine 1) the
bugs can be fixed 2) the known issues, version or package limitation.

I don´t want to have the very same discussion in the future without
have this documented somewhere.

Please, help me to list those and i´m going to document it.


Daiane

Re: imx6: webkit html5 video with hardware acceleration

[Bin Lin]

On 9/24/2014 1:58 AM, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 05:27:08PM -0300, Otavio Salvador wrote:
>> Are you using FSL Community BSP or FSL Yocto release?
> Hm, up until now I didn't realize that these are two different
> things. For daisy, I used fsl-community-bsp-platform, but for the
> beta, I used fsl-arm-yocto-bsp.
>
> So, if I understood correctly,
>
>    repo init -u git://git.freescale.com/imx/fsl-arm-yocto-bsp.git
>
> gets me the "offical" FSL project, but
>
>    repo init -u https://github.com/Freescale/fsl-community-bsp-platform
>
> gets me the "community" project. Is that right?
>
> Moreover, it looks like both repo manifests share most of the same git
> repositories. Can you explain to me what the differences are?
This link explains the difference:

https://community.freescale.com/thread/323782

> Thanks,
> Richard
>
>
>
>

Re: imx6: webkit html5 video with hardware acceleration

[Eric Nelson]

Hi Richard,

On 09/23/2014 09:50 PM, Richard Cochran wrote:
> On Tue, Sep 23, 2014 at 10:46:42PM +0200, Carlos Rafael Giani wrote:
>> Merely starting Chromium triggers the OOM killer already?
> 
> Yup.
> 
>> How much RAM does your board have?
> 
> 256 MB.
> 
>> And how much *free* RAM was
>> available when you started Chromium?
> 
> The Freescale kernel hackery is really hurting us. With the kernel in
> the alpha/beta branch, we need 144 MB for CMA. I am sure this is due
> to something dumb in the thousands of new FSL kernel patches, but I
> haven't had the time to track this down.
> 
> The daisy kernel worked on that board (with full screen video) with a
> CMA of 64 MB, but unfortunately the newer user space does *not* work
> with the daisy kernel. Brilliant stuff, that.
> 

You can tweak this through a kernel parameter:

	root@nitrogen:~# cat /sys/module/galcore/parameters/contiguousSize
	134217728

So if you add "galcore.contiguousSize=64M" to the kernel command-line,
you should get the old default.

Re: imx6: webkit html5 video with hardware acceleration

[Richard Cochran]

On Wed, Sep 24, 2014 at 09:42:37AM -0700, Eric Nelson wrote:
> So if you add "galcore.contiguousSize=64M" to the kernel command-line,
> you should get the old default.

IIRC, galcore is only one of many drivers using the CMA pool.

I was using galcore.contiguousSize=4194304.  It looks like the other
drivers/libs are using way more and larger buffers.

Anyhow, the FSL beta kernel is unusable for me, because the serial
console is broken.

Thanks,
Richard