Re: Chromium acceleration

Re: Chromium acceleration

[Becue Paul]

[-- Attachment #1: Type: text/plain, Size: 2068 bytes --]

Hello
I am working on a device with an iMX6 Solo processor for making a kind of web-kiosk with touch interface on Yocto.
To learn Yocto I bought your book a month ago. It helped me a lot. Not too long and to the point. Thank you for it.
I am building it with Yocto Linux - X11 - chromium.I added the chromium-imx layer from the freescale layer to my build. I builded for daisy and dizzy.
But I can't get the hardware acceleration on for chromium. When I look at chrome://gpu I get:
Graphics Feature Status Canvas: Software only, hardware acceleration unavailable Flash: Software only, hardware acceleration unavailable Flash Stage3D: Software only, hardware acceleration unavailable Flash Stage3D Baseline profile: Software only, hardware acceleration unavailable Compositing: Software only, hardware acceleration unavailable Rasterization: Software only, hardware acceleration unavailable Threaded Rasterization: Unavailable Video Decode: Software only, hardware acceleration unavailable Video Encode: Software only, hardware acceleration unavailable WebGL: Unavailable
and on the command line:
ERROR:browser_main_loop.cc(162)] Running without the SUID sandbox! See https://code.google.com/p/chromium/wiki/LinuxSUIDSandboxDevelopment for more information on developing with the sandbox on.
ERROR:gl_implementation_x11.cc(86)] glxGetProcAddress not found.
ERROR:gpu_child_thread.cc(143)] Exiting GPU process due to errors during initialization
ERROR: v4l2 capture: slave not found!
ERROR:renderer_main.cc(204)] Running without renderer sandbox
ERROR:desktop_window_tree_host_x11.cc(1547)] Not implemented reached in void views::DesktopWindowTreeHostX11::MapWindow(ui::WindowShowState)
ERROR:renderer_main.cc(204)] Running without renderer sandbox
ERROR:renderer_main.cc(204)] Running without renderer sandbox
ERROR:renderer_main.cc(204)] Running without renderer sandbox
ERROR:renderer_main.cc(204)] Running without renderer sandbox
Did I something wrong or is it simply not possible with the GPU of the iMX6?
Thank you in advance.

Paul Becue.


[-- Attachment #2: Type: text/html, Size: 6114 bytes --]

Re: Chromium acceleration

[Daiane Angolini]

On Thu, Nov 27, 2014 at 8:14 AM, Becue Paul <P.Becue@televic.com> wrote:
> Hello
>
> I am working on a device with an iMX6 Solo processor for making a kind of
> web-kiosk with touch interface on Yocto.
> To learn Yocto I bought your book a month ago. It helped me a lot. Not too
> long and to the point. Thank you for it.

Thanks a lot. I'm glad to hear that.

> I am building it with Yocto Linux - X11 - chromium.I added the chromium-imx
> layer from the freescale layer to my build. I builded for daisy and dizzy.
> But I can't get the hardware acceleration on for chromium. When I look at
> chrome://gpu I get:
> Graphics Feature Status Canvas: Software only, hardware acceleration
> unavailable Flash: Software only, hardware acceleration unavailable Flash
> Stage3D: Software only, hardware acceleration unavailable Flash Stage3D
> Baseline profile: Software only, hardware acceleration unavailable
> Compositing: Software only, hardware acceleration unavailable Rasterization:
> Software only, hardware acceleration unavailable Threaded Rasterization:
> Unavailable Video Decode: Software only, hardware acceleration unavailable
> Video Encode: Software only, hardware acceleration unavailable WebGL:
> Unavailable
> and on the command line:
> ERROR:browser_main_loop.cc(162)] Running without the SUID sandbox! See
> https://code.google.com/p/chromium/wiki/LinuxSUIDSandboxDevelopment for more
> information on developing with the sandbox on.
> ERROR:gl_implementation_x11.cc(86)] glxGetProcAddress not found.
> ERROR:gpu_child_thread.cc(143)] Exiting GPU process due to errors during
> initialization
> ERROR: v4l2 capture: slave not found!
> ERROR:renderer_main.cc(204)] Running without renderer sandbox
> ERROR:desktop_window_tree_host_x11.cc(1547)] Not implemented reached in void
> views::DesktopWindowTreeHostX11::MapWindow(ui::WindowShowState)
> ERROR:renderer_main.cc(204)] Running without renderer sandbox
> ERROR:renderer_main.cc(204)] Running without renderer sandbox
> ERROR:renderer_main.cc(204)] Running without renderer sandbox
> ERROR:renderer_main.cc(204)] Running without renderer sandbox

I tested cromium sometime ago with dizzy and it worked fine, I only
used 4.8 for toolchain.

> Did I something wrong or is it simply not possible with the GPU of the iMX6?
> Thank you in advance.

What are you using? Freescale Official Release or FSL Community BSP?



Daiane
>
>
>
> Paul Becue.
>
>
>
>
> --
> _______________________________________________
> meta-freescale mailing list
> meta-freescale@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-freescale
>

Re: Chromium acceleration

[Otavio Salvador]

On Mon, Dec 1, 2014 at 12:34 PM, Daiane Angolini <daiane.list@gmail.com> wrote:
> On Thu, Nov 27, 2014 at 8:14 AM, Becue Paul <P.Becue@televic.com> wrote:
>> I am working on a device with an iMX6 Solo processor for making a kind of
>> web-kiosk with touch interface on Yocto.
>> To learn Yocto I bought your book a month ago. It helped me a lot. Not too
>> long and to the point. Thank you for it.
>
> Thanks a lot. I'm glad to hear that.

Me too :-)

>> I am building it with Yocto Linux - X11 - chromium.I added the chromium-imx
>> layer from the freescale layer to my build. I builded for daisy and dizzy.
>> But I can't get the hardware acceleration on for chromium. When I look at
>> chrome://gpu I get:
>> Graphics Feature Status Canvas: Software only, hardware acceleration
>> unavailable Flash: Software only, hardware acceleration unavailable Flash
>> Stage3D: Software only, hardware acceleration unavailable Flash Stage3D
>> Baseline profile: Software only, hardware acceleration unavailable
>> Compositing: Software only, hardware acceleration unavailable Rasterization:
>> Software only, hardware acceleration unavailable Threaded Rasterization:
>> Unavailable Video Decode: Software only, hardware acceleration unavailable
>> Video Encode: Software only, hardware acceleration unavailable WebGL:
>> Unavailable
>> and on the command line:
>> ERROR:browser_main_loop.cc(162)] Running without the SUID sandbox! See
>> https://code.google.com/p/chromium/wiki/LinuxSUIDSandboxDevelopment for more
>> information on developing with the sandbox on.
>> ERROR:gl_implementation_x11.cc(86)] glxGetProcAddress not found.
>> ERROR:gpu_child_thread.cc(143)] Exiting GPU process due to errors during
>> initialization
>> ERROR: v4l2 capture: slave not found!
>> ERROR:renderer_main.cc(204)] Running without renderer sandbox
>> ERROR:desktop_window_tree_host_x11.cc(1547)] Not implemented reached in void
>> views::DesktopWindowTreeHostX11::MapWindow(ui::WindowShowState)
>> ERROR:renderer_main.cc(204)] Running without renderer sandbox
>> ERROR:renderer_main.cc(204)] Running without renderer sandbox
>> ERROR:renderer_main.cc(204)] Running without renderer sandbox
>> ERROR:renderer_main.cc(204)] Running without renderer sandbox
>
> I tested cromium sometime ago with dizzy and it worked fine, I only
> used 4.8 for toolchain.

I am using it with several customers (with different levels of
customization) with 4.9 toolchain and it is working fine.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Chromium acceleration

[Boszormenyi Zoltan]

[-- Attachment #1: Type: text/plain, Size: 1796 bytes --]

2014-04-02 14:28 keltezéssel, Carlos Rafael Giani írta:
> On 2014-04-02 14:02, Christian Betz wrote:
>>
>>         The VPU part could be because of missing firmware. Check if the vpu files are
>>         present in /lib/firmware.
>>
>>     As for MP4, this is a known problem. You are building Chromium, not Chrome. MP4
>>     support is part of the restricted feature set, which is included in Chrome but not
>>     Chromium. Try a WebM file for example.
>>
>>
>> this apparently can be worked around with gyp options:
>>
>> "proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome to allow Chrome to play 
>> h.264 content, which is the only codec VAVDA knows about today."
>>
>> this is described on a wiki page setting up hw video decode on **intel** processors:
>>
>> https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode
>>
>> note: i haven't actually tried this! (but i would like to)
>
> We tried that back then. The Chrome branding enabled a million other things , which 
> caused all sorts of difficulties. Also note that enabling the Chrome branding might have 
> legal repercussions.

A comment on that page says to use these flags (Chrome branding is not needed, only 
chromeos=1):

export GYP_DEFINES="chromeos=1 proprietary_codecs=1 ffmpeg_branding=Chrome"

do_configure succeeds but do_compile fails with "brlapi.h" missing. A google search
gave me this: https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/StpTzFnf8ok
Supposedly, running "build/install-build-deps.sh" fixes it but it turned out it only fixes
things for building Chromium for the host and the script wants an Ubuntu host OS.

Is there a ready to use recipe for brlapi/brltty somewhere or should I make one?

Thanks in advance,
Zoltán Böszörményi


[-- Attachment #2: Type: text/html, Size: 3956 bytes --]

Re: Chromium acceleration

[Eric Nelson]

Hi Zoltan,

On 04/02/2014 03:29 AM, Boszormenyi Zoltan wrote:
> Hi,
>
> 2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>>
>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>> build currently in meta-browser.
>>
>> Additional notes to follow, but this appears to achieve HTML5 video
>> against Webm/Ogg videos when chromium is started with these command
>> line arguments:
>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>
> you meant "--use-gl=egl".
>

Probably right.

> I have rebuilt Chromium with these patches. The result cannot play a
> short MP4 video from a file:// URL.
> The console output is very suspicious, libGAL complains about a missing
> ID file that actually exists.
> The image about the console is attached.
>
> This is a copy of the original message, I cancelled waiting for moderation.
> The original image exceeded the list limits so I reduced it to 16-color
> greyscale.
>

There's a lesson here that you should be putting console output
either directly into your e-mail (trimmed to important points).
Putting the output into an image makes it slow to read and tedious
to quote.

 From what I can tell, you aren't running on an image that otherwise
supports GPU or VPU acceleration. That's a pre-requisite for this,
and I recommend using fsl-image-gui or some other known good
starting point.

Also note that we sent these patches not to imply that they're
fully functional, but to allow others to have a working example
for exploring the code.

Regards,


Eric

Re: Chromium acceleration

[Carlos Rafael Giani]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On 2014-04-02 14:02, Christian Betz wrote:
>
>         The VPU part could be because of missing firmware. Check if
>         the vpu files are present in /lib/firmware.
>
>     As for MP4, this is a known problem. You are building Chromium,
>     not Chrome. MP4 support is part of the restricted feature set,
>     which is included in Chrome but not Chromium. Try a WebM file for
>     example.
>
>
> this apparently can be worked around with gyp options:
>
> "proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome to allow 
> Chrome to play h.264 content, which is the only codec VAVDA knows 
> about today."
>
> this is described on a wiki page setting up hw video decode on 
> **intel** processors:
>
> https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode
>
> note: i haven't actually tried this! (but i would like to)

We tried that back then. The Chrome branding enabled a million other 
things , which caused all sorts of difficulties. Also note that enabling 
the Chrome branding might have legal repercussions.

[-- Attachment #2: Type: text/html, Size: 2597 bytes --]

Re: Chromium acceleration

[Christian Betz]

[-- Attachment #1: Type: text/plain, Size: 758 bytes --]

>
> The VPU part could be because of missing firmware. Check if the vpu files
>> are present in /lib/firmware.
>>
> As for MP4, this is a known problem. You are building Chromium, not
> Chrome. MP4 support is part of the restricted feature set, which is
> included in Chrome but not Chromium. Try a WebM file for example.
>

this apparently can be worked around with gyp options:

"proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome to allow
Chrome to play h.264 content, which is the only codec VAVDA knows about
today."

this is described on a wiki page setting up hw video decode on **intel**
processors:

https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode

note: i haven't actually tried this! (but i would like to)

[-- Attachment #2: Type: text/html, Size: 1408 bytes --]

Re: Chromium acceleration

[Boszormenyi Zoltan]

2014-04-02 12:23 keltezéssel, Carlos Rafael Giani írta:
> On 2014-04-02 12:21, Boszormenyi Zoltan wrote:
>> Hi,
>>
>> 2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>>>
>>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>>> build currently in meta-browser.
>>>
>>> Additional notes to follow, but this appears to achieve HTML5 video
>>> against Webm/Ogg videos when chromium is started with these command
>>> line arguments:
>>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>>
>> you meant "--use-gl=egl".
>>
>> I have rebuilt Chromium with these patches. The result cannot play a short MP4 video 
>> from a file:// URL.
>> The console output is very suspicious, libGAL complains about a missing ID file that 
>> actually exists.
>> The image about the console is attached.
>>
>> Best regards,
>> Zoltán Böszörményi
>>
>
> The VPU part could be because of missing firmware. Check if the vpu files are present in 
> /lib/firmware.

I have /lib/firmware/vpu/vpu_fw_imxq6.bin, size 253968, the last 6 digits of md5sum is e8debf.

> As for MP4, this is a known problem. You are building Chromium, not Chrome. MP4 support 
> is part of the restricted feature set, which is included in Chrome but not Chromium. Try 
> a WebM file for example.

I tried these:

1. http://www.webmfiles.org/demo-files/
The webm examples play nicely, even scaled to full screen.

2. http://devfiles.myopera.com/articles/1891/custom-controls-webm-720p.html
It's the same "Elephants Dream" as one of the examples in (1) but at 720p.
It pays nicely in the browser control but it's a slideshow as full screen.

On the other hand, gstreamer 0.10 with gst-fsl-plugin and its gst-fsl-plugin-gplay
subpackage, gplay plays a 720p and a 1080p H.264 test file very well without tearing.

Best regards,
Zoltán Böszörményi

Re: Chromium acceleration

[Carlos Rafael Giani]

On 2014-04-02 12:28, Gary Thomas wrote:
> On 2014-04-02 04:21, Carlos Rafael Giani wrote:
>> Keep in mind what I wrote. This version of VPU acceleration is very basic (it will copy frames with the CPU), and fulfilled the customer's immediate needs back then, but can be
>> done much better. I am currently looking into a better approach.
> Are these patches available somewhere?

No, not yet, there is no finished code at the moment. If I am right, the 
necessary changes are not big though. The actual bottleneck here is time :)

Re: Chromium acceleration

[Boszormenyi Zoltan]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

Hi,

2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>
> Mahyar updated these patches to apply against the chromium-35.0.1883.0
> build currently in meta-browser.
>
> Additional notes to follow, but this appears to achieve HTML5 video
> against Webm/Ogg videos when chromium is started with these command
> line arguments:
>      --ignore-gpu-blacklist --enable-gpu --usegl-egl

you meant "--use-gl=egl".

I have rebuilt Chromium with these patches. The result cannot play a short MP4 video from 
a file:// URL.
The console output is very suspicious, libGAL complains about a missing ID file that 
actually exists.
The image about the console is attached.

This is a copy of the original message, I cancelled waiting for moderation.
The original image exceeded the list limits so I reduced it to 16-color greyscale.

Best regards,
Zoltán Böszörményi


[-- Attachment #2: 20140402_121516_800x600.png --]
[-- Type: image/png, Size: 113651 bytes --]

Re: Chromium acceleration

[Gary Thomas]

On 2014-04-02 04:21, Carlos Rafael Giani wrote:
> Keep in mind what I wrote. This version of VPU acceleration is very basic (it will copy frames with the CPU), and fulfilled the customer's immediate needs back then, but can be
> done much better. I am currently looking into a better approach.
> 
> On 2014-04-01 21:22, Eric Nelson wrote:
>> Thanks again Carlos,
>>
>> On 03/20/2014 04:19 PM, Carlos Rafael Giani wrote:
>> >
>> > <snip>
>>>
>>> Back then we needed some hw decoding quickly, so we did not look further
>>> into this, since we had spent a lot of time getting the 2D acceleration
>>> stable already. I could only briefly look at the exynos accelerator
>>> code, but it does look like the right direction, although the VPU uses
>>> physical addresses directly instead of dmabuf FDs. I am currently
>>> writing a frontend for the libfslvpuwrap, which takes care of certain
>>> complexities (like being able to pass userdata pointers to frames, to
>>> make it possible to associate input and output frames, which is
>>> essential for correct timestamping, especially when things like h264
>>> frame reordering are active). This frontend will hide these things in a
>>> future gstreamer-imx release to make the decoder code clearer and easier
>>> to maintain, and is intended to be reusable, for example for Chromium
>>> integration, or XBMC. Beyond that, my patches are probably not that
>>> helpful anymore, since they were based on the vpx decoder way of
>>> decoding, and I agree that the exynos code is a better starting point.
>>> Nevertheless, I attached the patch. Keep in mind that it is very basic,
>>> old, and wasn't further developed on because it was "good enough" for
>>> the project.
>>>
>>> But here are some additional notes from our efforts to accelerate
>>> Chromium on imx6 (keep in mind these apply to version 32.0.1664.3 ) :
>>>
>>> 1. We started the google-chrome binary with these switches:
>>> --ignore-gpu-blacklist --enable-gpu --use-gl=egl
>>> --enable-accelerated-2d-canvas
>>>
>>> 2. To make building Chromium easier, we switched to component build (by
>>> default, it is all linked into one enormous binary). I attached a patch
>>> that was necessary to fix some minor issues. Perhaps it is not necessary
>>> anymore.
>>>
>>> 3. Chromium tried to load libEGL with dlopen() , which caused problems,
>>> because with the Vivante libraries, libEGL also needs libGAL. Usually,
>>> build scripts just add -lEGL -lGAL to the linker line. With dlopen()
>>> this wasn't possible of course. We patched the gyp scripts to link
>>> against this libraries during building instead. I attached this patch.
>>> It is really a hack, because it circumvents the sandboxing process (this
>>> is why Chromium loads EGL and GLESv2 with dlopen() ).
>>>
>>
>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>> build currently in meta-browser.

Are these patches available somewhere?

>> Additional notes to follow, but this appears to achieve HTML5 video
>> against Webm/Ogg videos when chromium is started with these command
>> line arguments:
>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Chromium acceleration

[Carlos Rafael Giani]

On 2014-04-02 12:21, Boszormenyi Zoltan wrote:
> Hi,
>
> 2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>>
>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>> build currently in meta-browser.
>>
>> Additional notes to follow, but this appears to achieve HTML5 video
>> against Webm/Ogg videos when chromium is started with these command
>> line arguments:
>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>
> you meant "--use-gl=egl".
>
> I have rebuilt Chromium with these patches. The result cannot play a 
> short MP4 video from a file:// URL.
> The console output is very suspicious, libGAL complains about a 
> missing ID file that actually exists.
> The image about the console is attached.
>
> Best regards,
> Zoltán Böszörményi
>

The VPU part could be because of missing firmware. Check if the vpu 
files are present in /lib/firmware.
As for MP4, this is a known problem. You are building Chromium, not 
Chrome. MP4 support is part of the restricted feature set, which is 
included in Chrome but not Chromium. Try a WebM file for example.

Re: Chromium acceleration

[Carlos Rafael Giani]

Keep in mind what I wrote. This version of VPU acceleration is very 
basic (it will copy frames with the CPU), and fulfilled the customer's 
immediate needs back then, but can be done much better. I am currently 
looking into a better approach.

On 2014-04-01 21:22, Eric Nelson wrote:
> Thanks again Carlos,
>
> On 03/20/2014 04:19 PM, Carlos Rafael Giani wrote:
> >
> > <snip>
>>
>> Back then we needed some hw decoding quickly, so we did not look further
>> into this, since we had spent a lot of time getting the 2D acceleration
>> stable already. I could only briefly look at the exynos accelerator
>> code, but it does look like the right direction, although the VPU uses
>> physical addresses directly instead of dmabuf FDs. I am currently
>> writing a frontend for the libfslvpuwrap, which takes care of certain
>> complexities (like being able to pass userdata pointers to frames, to
>> make it possible to associate input and output frames, which is
>> essential for correct timestamping, especially when things like h264
>> frame reordering are active). This frontend will hide these things in a
>> future gstreamer-imx release to make the decoder code clearer and easier
>> to maintain, and is intended to be reusable, for example for Chromium
>> integration, or XBMC. Beyond that, my patches are probably not that
>> helpful anymore, since they were based on the vpx decoder way of
>> decoding, and I agree that the exynos code is a better starting point.
>> Nevertheless, I attached the patch. Keep in mind that it is very basic,
>> old, and wasn't further developed on because it was "good enough" for
>> the project.
>>
>> But here are some additional notes from our efforts to accelerate
>> Chromium on imx6 (keep in mind these apply to version 32.0.1664.3 ) :
>>
>> 1. We started the google-chrome binary with these switches:
>> --ignore-gpu-blacklist --enable-gpu --use-gl=egl
>> --enable-accelerated-2d-canvas
>>
>> 2. To make building Chromium easier, we switched to component build (by
>> default, it is all linked into one enormous binary). I attached a patch
>> that was necessary to fix some minor issues. Perhaps it is not necessary
>> anymore.
>>
>> 3. Chromium tried to load libEGL with dlopen() , which caused problems,
>> because with the Vivante libraries, libEGL also needs libGAL. Usually,
>> build scripts just add -lEGL -lGAL to the linker line. With dlopen()
>> this wasn't possible of course. We patched the gyp scripts to link
>> against this libraries during building instead. I attached this patch.
>> It is really a hack, because it circumvents the sandboxing process (this
>> is why Chromium loads EGL and GLESv2 with dlopen() ).
>>
>
> Mahyar updated these patches to apply against the chromium-35.0.1883.0
> build currently in meta-browser.
>
> Additional notes to follow, but this appears to achieve HTML5 video
> against Webm/Ogg videos when chromium is started with these command
> line arguments:
>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>
> Regards,
>
>
> Eric
>

Re: Chromium acceleration

[Eric Nelson]

[-- Attachment #1: Type: text/plain, Size: 2727 bytes --]

Thanks again Carlos,

On 03/20/2014 04:19 PM, Carlos Rafael Giani wrote:
 >
 > <snip>
>
> Back then we needed some hw decoding quickly, so we did not look further
> into this, since we had spent a lot of time getting the 2D acceleration
> stable already. I could only briefly look at the exynos accelerator
> code, but it does look like the right direction, although the VPU uses
> physical addresses directly instead of dmabuf FDs. I am currently
> writing a frontend for the libfslvpuwrap, which takes care of certain
> complexities (like being able to pass userdata pointers to frames, to
> make it possible to associate input and output frames, which is
> essential for correct timestamping, especially when things like h264
> frame reordering are active). This frontend will hide these things in a
> future gstreamer-imx release to make the decoder code clearer and easier
> to maintain, and is intended to be reusable, for example for Chromium
> integration, or XBMC. Beyond that, my patches are probably not that
> helpful anymore, since they were based on the vpx decoder way of
> decoding, and I agree that the exynos code is a better starting point.
> Nevertheless, I attached the patch. Keep in mind that it is very basic,
> old, and wasn't further developed on because it was "good enough" for
> the project.
>
> But here are some additional notes from our efforts to accelerate
> Chromium on imx6 (keep in mind these apply to version 32.0.1664.3 ) :
>
> 1. We started the google-chrome binary with these switches:
> --ignore-gpu-blacklist --enable-gpu --use-gl=egl
> --enable-accelerated-2d-canvas
>
> 2. To make building Chromium easier, we switched to component build (by
> default, it is all linked into one enormous binary). I attached a patch
> that was necessary to fix some minor issues. Perhaps it is not necessary
> anymore.
>
> 3. Chromium tried to load libEGL with dlopen() , which caused problems,
> because with the Vivante libraries, libEGL also needs libGAL. Usually,
> build scripts just add -lEGL -lGAL to the linker line. With dlopen()
> this wasn't possible of course. We patched the gyp scripts to link
> against this libraries during building instead. I attached this patch.
> It is really a hack, because it circumvents the sandboxing process (this
> is why Chromium loads EGL and GLESv2 with dlopen() ).
>

Mahyar updated these patches to apply against the chromium-35.0.1883.0
build currently in meta-browser.

Additional notes to follow, but this appears to achieve HTML5 video
against Webm/Ogg videos when chromium is started with these command
line arguments:
	 --ignore-gpu-blacklist --enable-gpu --usegl-egl

Regards,


Eric


[-- Attachment #2: 0001-Modified-gl.gyp-to-link-libEGL-and-libGAL-at-build-t.patch --]
[-- Type: text/x-diff, Size: 1025 bytes --]

From 03924737a9378db934da9355bb38c916b5e3fa45 Mon Sep 17 00:00:00 2001
From: Eric Nelson <eric.nelson@boundarydevices.com>
Date: Sun, 23 Mar 2014 07:10:36 -0700
Subject: [PATCH 1/3] Modified gl.gyp to link libEGL and libGAL at build time

Signed-off-by: Eric Nelson <eric.nelson@boundarydevices.com>
---
 ui/gl/gl.gyp | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/ui/gl/gl.gyp b/ui/gl/gl.gyp
index 2ec6d76..f7ee1a4 100644
--- a/ui/gl/gl.gyp
+++ b/ui/gl/gl.gyp
@@ -123,6 +123,12 @@
         '<(gl_binding_output_dir)/gl_bindings_autogen_osmesa.cc',
         '<(gl_binding_output_dir)/gl_bindings_autogen_osmesa.h',
       ],
+      'link_settings': {
+        'libraries': [
+          '-lEGL',
+          '-lGAL',
+        ],
+      },
       # hard_dependency is necessary for this target because it has actions
       # that generate header files included by dependent targets. The header
       # files must be generated before the dependents are compiled. The usual
-- 
1.8.3.2


[-- Attachment #3: 0002-Added-hardware-accelerated-decoding-with-the-i.MX-VP.patch --]
[-- Type: text/x-diff, Size: 29768 bytes --]

From 44ccb7f8a920389897e60e30e3f0ee3319f7bf7d Mon Sep 17 00:00:00 2001
From: Eric Nelson <eric.nelson@boundarydevices.com>
Date: Sun, 23 Mar 2014 07:17:08 -0700
Subject: [PATCH 2/3] Added hardware accelerated decoding with the i.MX VPU

---
 content/renderer/media/webmediaplayer_impl.cc |   5 +
 media/filters/vpu_framebuffers.cc             | 106 +++++
 media/filters/vpu_framebuffers.h              |  36 ++
 media/filters/vpu_video_decoder.cc            | 614 ++++++++++++++++++++++++++
 media/filters/vpu_video_decoder.h             |  74 ++++
 media/media.gyp                               |  28 ++
 6 files changed, 863 insertions(+)
 create mode 100644 media/filters/vpu_framebuffers.cc
 create mode 100644 media/filters/vpu_framebuffers.h
 create mode 100644 media/filters/vpu_video_decoder.cc
 create mode 100644 media/filters/vpu_video_decoder.h

diff --git a/content/renderer/media/webmediaplayer_impl.cc b/content/renderer/media/webmediaplayer_impl.cc
index b7094f6..7ac7318 100644
--- a/content/renderer/media/webmediaplayer_impl.cc
+++ b/content/renderer/media/webmediaplayer_impl.cc
@@ -57,6 +57,7 @@
 #include "media/filters/opus_audio_decoder.h"
 #include "media/filters/video_renderer_impl.h"
 #include "media/filters/vpx_video_decoder.h"
+#include "media/filters/vpu_video_decoder.h"
 #include "third_party/WebKit/public/platform/WebContentDecryptionModule.h"
 #include "third_party/WebKit/public/platform/WebMediaSource.h"
 #include "third_party/WebKit/public/platform/WebRect.h"
@@ -1187,6 +1188,10 @@ void WebMediaPlayerImpl::StartPipeline() {
         new media::GpuVideoDecoder(gpu_factories_, media_log_));
   }
 
+#if !defined(MEDIA_DISABLE_IMXVPU)
+  video_decoders.push_back(new media::VpuVideoDecoder(media_loop_));
+#endif  // !defined(MEDIA_DISABLE_IMXVPU)
+
 #if !defined(MEDIA_DISABLE_LIBVPX)
   video_decoders.push_back(new media::VpxVideoDecoder(media_loop_));
 #endif  // !defined(MEDIA_DISABLE_LIBVPX)
diff --git a/media/filters/vpu_framebuffers.cc b/media/filters/vpu_framebuffers.cc
new file mode 100644
index 0000000..fce43fc
--- /dev/null
+++ b/media/filters/vpu_framebuffers.cc
@@ -0,0 +1,106 @@
+#include <string.h>
+#include <stdint.h>
+#include "vpu_framebuffers.h"
+#include "base/logging.h"
+
+
+#define ALIGN_VAL_TO(LENGTH, ALIGN_SIZE)  ( ((uintptr_t)((LENGTH) + (ALIGN_SIZE) - 1) / (ALIGN_SIZE)) * (ALIGN_SIZE) )
+#define FRAME_ALIGN 16
+
+
+namespace media {
+
+
+VPUFramebuffers::VPUFramebuffers() {
+}
+
+
+void VPUFramebuffers::init(VpuDecHandle handle, VpuDecInitInfo const &init_info) {
+  framebuffers.resize(init_info.nMinFrameBufferCount);
+
+  y_stride = ALIGN_VAL_TO(init_info.nPicWidth, FRAME_ALIGN);
+  if (init_info.nInterlace)
+    y_size = y_stride * ALIGN_VAL_TO(init_info.nPicHeight, (2 * FRAME_ALIGN));
+  else
+    y_size = y_stride * ALIGN_VAL_TO(init_info.nPicHeight, FRAME_ALIGN);
+
+  uv_stride = y_stride / 2;
+  u_size = v_size = mv_size = y_size / 4;
+
+  int alignment = init_info.nAddressAlignment;
+  if (alignment > 1)
+  {
+    y_size = ALIGN_VAL_TO(y_size, alignment);
+    u_size = ALIGN_VAL_TO(u_size, alignment);
+    v_size = ALIGN_VAL_TO(v_size, alignment);
+    mv_size = ALIGN_VAL_TO(mv_size, alignment);
+  }
+
+  pic_width = init_info.nPicWidth;
+  pic_height = init_info.nPicHeight;
+
+  total_size = y_size + u_size + v_size + mv_size + alignment;
+
+  LOG(INFO) << "Num VPU framebuffers: " << framebuffers.size();
+  LOG(INFO) << "VPU framebuffer memory block size:"
+    << "  total: " << total_size
+    << "  Y " << y_size
+    << "  U: " << u_size
+    << "  V: " << v_size
+    << "  Mv: " << mv_size
+    << "  alignment: " << alignment
+    ;
+
+  for (Framebuffers::iterator iter = framebuffers.begin(); iter != framebuffers.end(); ++iter) {
+    unsigned char *virt_ptr, *phys_ptr;
+
+    VpuFrameBuffer &framebuffer = *iter;
+
+    VpuMemDesc mem_desc;
+    memset(&mem_desc, 0, sizeof(VpuMemDesc));
+    mem_desc.nSize = total_size;
+    VPU_DecGetMem(&mem_desc);
+    phys_mem_blocks.push_back(mem_desc);
+
+    virt_ptr = (unsigned char*)(mem_desc.nVirtAddr);
+    phys_ptr = (unsigned char*)(mem_desc.nPhyAddr);
+
+    if (alignment > 1) {
+      phys_ptr = (unsigned char*)ALIGN_VAL_TO(phys_ptr, alignment);
+      virt_ptr = (unsigned char*)ALIGN_VAL_TO(virt_ptr, alignment);
+    }
+
+    framebuffer.nStrideY = y_stride;
+    framebuffer.nStrideC = uv_stride;
+
+    /* fill phy addr*/
+    framebuffer.pbufY     = phys_ptr;
+    framebuffer.pbufCb    = phys_ptr + y_size;
+    framebuffer.pbufCr    = phys_ptr + y_size + u_size;
+    framebuffer.pbufMvCol = phys_ptr + y_size + u_size + v_size;
+
+    /* fill virt addr */
+    framebuffer.pbufVirtY     = virt_ptr;
+    framebuffer.pbufVirtCb    = virt_ptr + y_size;
+    framebuffer.pbufVirtCr    = virt_ptr + y_size + u_size;
+    framebuffer.pbufVirtMvCol = virt_ptr + y_size + u_size + v_size;
+
+    framebuffer.pbufY_tilebot = 0;
+    framebuffer.pbufCb_tilebot = 0;
+    framebuffer.pbufVirtY_tilebot = 0;
+    framebuffer.pbufVirtCb_tilebot = 0;
+  }
+
+  VPU_DecRegisterFrameBuffer(handle, &(framebuffers[0]), framebuffers.size());
+}
+
+
+VPUFramebuffers::~VPUFramebuffers() {
+  LOG(INFO) << "Freeing VPU framebuffer memory";
+  for (PhysMemBlocks::iterator iter = phys_mem_blocks.begin(); iter != phys_mem_blocks.end(); ++iter)
+    VPU_DecFreeMem(&(*iter));
+}
+
+
+}
+
diff --git a/media/filters/vpu_framebuffers.h b/media/filters/vpu_framebuffers.h
new file mode 100644
index 0000000..bb9b021
--- /dev/null
+++ b/media/filters/vpu_framebuffers.h
@@ -0,0 +1,36 @@
+#ifndef MEDIA_FILTERS_VPU_FRAMEBUFFERS_H_
+#define MEDIA_FILTERS_VPU_FRAMEBUFFERS_H_
+
+#include <list>
+#include <vector>
+#include <vpu_wrapper.h>
+
+
+namespace media {
+
+
+class VPUFramebuffers
+{
+public:
+  explicit VPUFramebuffers();
+  ~VPUFramebuffers();
+
+  void init(VpuDecHandle handle, VpuDecInitInfo const &init_info);
+
+private:
+  typedef std::vector < VpuFrameBuffer > Framebuffers;
+  typedef std::list < VpuMemDesc > PhysMemBlocks;
+
+  PhysMemBlocks phys_mem_blocks;
+  Framebuffers framebuffers;
+  int y_stride, uv_stride;
+  int y_size, u_size, v_size, mv_size;
+  int total_size;
+  unsigned int pic_width, pic_height;
+};
+
+
+}
+
+
+#endif
diff --git a/media/filters/vpu_video_decoder.cc b/media/filters/vpu_video_decoder.cc
new file mode 100644
index 0000000..396d4f1
--- /dev/null
+++ b/media/filters/vpu_video_decoder.cc
@@ -0,0 +1,614 @@
+#include <list>
+#include <vector>
+#include <stddef.h>
+#include <stdint.h>
+#include <algorithm>
+#include <string>
+
+#include <string.h>
+
+#include "media/filters/vpu_video_decoder.h"
+
+#include "base/bind.h"
+#include "base/callback_helpers.h"
+#include "base/command_line.h"
+#include "base/location.h"
+#include "base/logging.h"
+#include "base/synchronization/lock.h"
+#include "base/message_loop/message_loop_proxy.h"
+#include "base/strings/string_number_conversions.h"
+#include "base/sys_byteorder.h"
+#include "media/base/bind_to_loop.h"
+#include "media/base/decoder_buffer.h"
+#include "media/base/demuxer_stream.h"
+#include "media/base/media_switches.h"
+#include "media/base/pipeline.h"
+#include "media/base/video_decoder_config.h"
+#include "media/base/video_frame.h"
+#include "media/base/video_util.h"
+
+#include "media/filters/vpu_framebuffers.h"
+
+#include <vpu_wrapper.h>
+
+
+#define ALIGN_VAL_TO(LENGTH, ALIGN_SIZE)  ( ((uintptr_t)((LENGTH) + (ALIGN_SIZE) - 1) / (ALIGN_SIZE)) * (ALIGN_SIZE) )
+
+
+
+namespace media {
+
+
+namespace {
+
+
+char const *VpuStrerror(VpuDecRetCode const code) {
+  switch (code) {
+    case VPU_DEC_RET_SUCCESS: return "success";
+    case VPU_DEC_RET_FAILURE: return "failure";
+    case VPU_DEC_RET_INVALID_PARAM: return "invalid param";
+    case VPU_DEC_RET_INVALID_HANDLE: return "invalid handle";
+    case VPU_DEC_RET_INVALID_FRAME_BUFFER: return "invalid frame buffer";
+    case VPU_DEC_RET_INSUFFICIENT_FRAME_BUFFERS: return "insufficient frame buffers";
+    case VPU_DEC_RET_INVALID_STRIDE: return "invalid stride";
+    case VPU_DEC_RET_WRONG_CALL_SEQUENCE: return "wrong call sequence";
+    case VPU_DEC_RET_FAILURE_TIMEOUT: return "failure timeout";
+    default:
+      return NULL;
+  }
+}
+
+
+long inst_count = 0;
+base::Lock vpu_load_lock;
+
+
+void LoadDecoder() {
+  VpuDecRetCode ret;
+
+  base::AutoLock alock(vpu_load_lock);
+
+  if (inst_count == 0) {
+    ret = VPU_DecLoad();
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "VPU loading failed: " << VpuStrerror(ret);
+      return;
+    }
+
+  }
+
+  ++inst_count;
+
+  if (inst_count == 1) {
+    VpuVersionInfo version;
+    VpuWrapperVersionInfo wrapper_version;
+
+    ret = VPU_DecGetVersionInfo(&version);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Getting version info failed: " << VpuStrerror(ret);
+      return;
+    }
+
+    ret = VPU_DecGetWrapperVersionInfo(&wrapper_version);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Getting wrapper version info failed: " << VpuStrerror(ret);
+      return;
+    }
+
+    LOG(INFO) << "VPU Loaded";
+    LOG(INFO) << "VPU firmware version " << version.nFwMajor << "." << version.nFwMinor << "." << version.nFwRelease << "_r" << version.nFwCode;
+    LOG(INFO) << "VPU library version " << version.nLibMajor << "." << version.nLibMinor << "." << version.nLibRelease;
+    LOG(INFO) << "VPU wrapper version " << wrapper_version.nMajor << "." << wrapper_version.nMinor << "." << wrapper_version.nRelease << " " << wrapper_version.pBinary;
+  }
+}
+
+
+void UnloadDecoder() {
+  base::AutoLock alock(vpu_load_lock);
+
+  if (inst_count == 0)
+    return;
+
+  --inst_count;
+
+  if (inst_count == 0)
+    VPU_DecUnLoad();
+}
+
+
+}
+
+
+struct VpuVideoDecoder::Internal
+{
+  typedef std::vector < uint8_t > VirtMemBlock;
+  typedef std::list < VirtMemBlock > VirtMemBlocks;
+
+  typedef std::list < VpuMemDesc > PhysMemBlocks;
+
+  VpuDecHandle handle;
+
+  VpuDecInitInfo init_info;
+  VpuMemInfo mem_info;
+
+  bool vpu_opened;
+
+  VirtMemBlocks virt_mem_blocks;
+  PhysMemBlocks phys_mem_blocks;
+
+  VPUFramebuffers framebuffers;
+
+  Internal()
+    : vpu_opened(false)
+  {
+  }
+
+  ~Internal()
+  {
+    FreeMemBlocks();
+  }
+
+  bool AllocMemBlocks()
+  {
+    VpuDecRetCode ret;
+
+    memset(&mem_info, 0, sizeof(VpuMemInfo));
+    ret = VPU_DecQueryMem(&mem_info);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get VPU memory information: " << VpuStrerror(ret);
+      return false;
+    }
+
+    for (int i = 0; i < mem_info.nSubBlockNum; ++i)
+    {
+      int size = mem_info.MemSubBlock[i].nAlignment + mem_info.MemSubBlock[i].nSize;
+
+      LOG(INFO) << "Sub block " << i << "  type: " << ((mem_info.MemSubBlock[i].MemType == VPU_MEM_VIRT) ? "virtual" : "phys") << "  size: " << size;
+
+      if (mem_info.MemSubBlock[i].MemType == VPU_MEM_VIRT)
+      {
+        unsigned char *virt_ptr;
+        virt_mem_blocks.push_back(VirtMemBlock());
+        VirtMemBlock &blk = virt_mem_blocks.back();
+        blk.resize(size);
+        virt_ptr = reinterpret_cast < unsigned char* > (&blk[0]);
+
+        mem_info.MemSubBlock[i].pVirtAddr = (unsigned char *)ALIGN_VAL_TO(virt_ptr, mem_info.MemSubBlock[i].nAlignment);
+      }
+      else if (mem_info.MemSubBlock[i].MemType == VPU_MEM_PHY)
+      {
+        VpuMemDesc mem_desc;
+        memset(&mem_desc, 0, sizeof(VpuMemDesc));
+        mem_desc.nSize = size;
+        if (VPU_DecGetMem(&mem_desc) != VPU_DEC_RET_SUCCESS) {
+          LOG(ERROR) << "Unable to allocate " << size << " bytes of physical memory";
+          return false;
+        }
+
+        phys_mem_blocks.push_back(mem_desc);
+
+        mem_info.MemSubBlock[i].pVirtAddr = (unsigned char *)ALIGN_VAL_TO((unsigned char*)(mem_desc.nVirtAddr), mem_info.MemSubBlock[i].nAlignment);
+        mem_info.MemSubBlock[i].pPhyAddr = (unsigned char *)ALIGN_VAL_TO((unsigned char*)(mem_desc.nPhyAddr), mem_info.MemSubBlock[i].nAlignment);
+      }
+      else
+        LOG(WARNING) << "Sub block " << i << " type is unknown - skipping";
+    }
+
+    LOG(INFO) << "VPU memory blocks allocated";
+
+    return true;
+  }
+
+  void FreeMemBlocks()
+  {
+    LOG(INFO) << "Freeing VPU memory blocks";
+
+    for (PhysMemBlocks::iterator iter = phys_mem_blocks.begin(); iter != phys_mem_blocks.end(); ++iter)
+      VPU_DecFreeMem(&(*iter));
+  }
+};
+
+
+
+void CopyVpuFramebufferTo(VpuFrameBuffer *framebuffer, scoped_refptr<VideoFrame>* video_frame, VpuDecInitInfo &init_info, const VideoDecoderConfig& config);
+
+
+
+
+VpuVideoDecoder::VpuVideoDecoder(
+    const scoped_refptr<base::MessageLoopProxy>& message_loop)
+    : message_loop_(message_loop),
+      weak_factory_(this),
+      state_(kUninitialized) {
+    LoadDecoder();
+    internal = new Internal;
+}
+
+VpuVideoDecoder::~VpuVideoDecoder() {
+  DCHECK_EQ(kUninitialized, state_);
+  CloseDecoder();
+  delete internal;
+  UnloadDecoder();
+}
+
+void VpuVideoDecoder::Initialize(const VideoDecoderConfig& config,
+                                 const PipelineStatusCB& status_cb) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK(config.IsValidConfig());
+  DCHECK(!config.is_encrypted());
+  DCHECK(decode_cb_.is_null());
+  DCHECK(reset_cb_.is_null());
+
+  weak_this_ = weak_factory_.GetWeakPtr();
+
+  if (!internal->AllocMemBlocks()) {
+    status_cb.Run(PIPELINE_ERROR_INITIALIZATION_FAILED);
+    return;
+  }
+
+  if (!ConfigureDecoder(config)) {
+    status_cb.Run(DECODER_ERROR_NOT_SUPPORTED);
+    return;
+  }
+
+  // Success!
+  LOG(INFO) << "VPU decoder initialization succeeded";
+  config_ = config;
+  state_ = kNormal;
+  status_cb.Run(PIPELINE_OK);
+}
+
+bool VpuVideoDecoder::ConfigureDecoder(const VideoDecoderConfig& config) {
+  bool can_handle = false;
+
+  if ((config.format() == VideoFrame::YV12) || (config.format() == VideoFrame::YV12A) || (config.format() == VideoFrame::I420)) {
+    switch (config.codec()) {
+      case kCodecH264:
+      case kCodecVC1:
+      case kCodecMPEG2:
+      case kCodecMPEG4:
+      case kCodecVP8:
+        can_handle = true;
+      default:
+        break;
+    }
+  }
+
+  if (!can_handle) {
+    LOG(INFO) << "Cannot handle video config: " << config.AsHumanReadableString();
+    return false;
+  }
+
+  CloseDecoder();
+  return OpenDecoder(config);
+}
+
+bool VpuVideoDecoder::OpenDecoder(const VideoDecoderConfig& config) {
+  if (internal->vpu_opened)
+    return true;
+
+  VpuDecOpenParam open_param;
+  VpuDecRetCode ret;
+
+  memset(&open_param, 0, sizeof(open_param));
+
+  switch (config.codec())
+  {
+    case kCodecH264:
+      open_param.CodecFormat = VPU_V_AVC;
+      open_param.nReorderEnable = 1;
+      LOG(INFO) << "Setting h.264 as stream format";
+      break;
+    case kCodecVC1:
+      open_param.CodecFormat = VPU_V_VC1_AP;
+      LOG(INFO) << "Setting VC1 as stream format";
+      break;
+    case kCodecMPEG2:
+      open_param.CodecFormat = VPU_V_MPEG2;
+      LOG(INFO) << "Setting MPEG-2 as stream format";
+      break;
+    case kCodecMPEG4:
+      open_param.CodecFormat = VPU_V_MPEG4;
+      LOG(INFO) << "Setting MPEG-4 as stream format";
+      break;
+    case kCodecVP8:
+      open_param.CodecFormat = VPU_V_VP8;
+      LOG(INFO) << "Setting VP8 as stream format";
+      break;
+    default:
+      LOG(ERROR) << "Could not set unknown stream format";
+      break;
+  }
+
+  open_param.nChromaInterleave = 0;
+  open_param.nMapType = 0;
+  open_param.nTiled2LinearEnable = 0;
+  open_param.nEnableFileMode = 0;
+  open_param.nPicWidth = config.coded_size().width();
+  open_param.nPicHeight = config.coded_size().height();
+
+  ret = VPU_DecOpen(&(internal->handle), &open_param, &(internal->mem_info));
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Opening new VPU handle failed: " << VpuStrerror(ret);
+    return false;
+  }
+
+  int config_param;
+
+  config_param = VPU_DEC_SKIPNONE;
+  ret = VPU_DecConfig(internal->handle, VPU_DEC_CONF_SKIPMODE, &config_param);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Could not configure skip mode: " << VpuStrerror(ret);
+    return false;
+  }
+
+  config_param = 0;
+  ret = VPU_DecConfig(internal->handle, VPU_DEC_CONF_BUFDELAY, &config_param);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Could not configure buffer delay: " << VpuStrerror(ret);
+    return false;
+  }
+
+  config_param = VPU_DEC_IN_NORMAL;
+  ret = VPU_DecConfig(internal->handle, VPU_DEC_CONF_INPUTTYPE, &config_param);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Could not configure input type: " << VpuStrerror(ret);
+    return false;
+  }
+
+  internal->vpu_opened = true;
+
+  return true;
+}
+
+void VpuVideoDecoder::CloseDecoder() {
+  VpuDecRetCode ret;
+
+  if (!(internal->vpu_opened))
+    return;
+
+  ret = VPU_DecFlushAll(internal->handle);
+  if (ret != VPU_DEC_RET_SUCCESS)
+    LOG(ERROR) << "Flushing decoder failed: " << VpuStrerror(ret);
+  VPU_DecClose(internal->handle);
+  if (ret != VPU_DEC_RET_SUCCESS)
+    LOG(ERROR) << "Closing decoder failed: " << VpuStrerror(ret);
+
+  internal->vpu_opened = false;
+}
+
+void VpuVideoDecoder::Decode(const scoped_refptr<DecoderBuffer>& buffer,
+                             const DecodeCB& decode_cb) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK(!decode_cb.is_null());
+  CHECK_NE(state_, kUninitialized);
+  CHECK(decode_cb_.is_null()) << "Overlapping decodes are not supported.";
+
+  decode_cb_ = BindToCurrentLoop(decode_cb);
+
+  if (state_ == kError) {
+    base::ResetAndReturn(&decode_cb_).Run(kDecodeError, NULL);
+    return;
+  }
+
+  // Return empty frames if decoding has finished.
+  if (state_ == kDecodeFinished) {
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    return;
+  }
+
+  DecodeBuffer(buffer);
+}
+
+void VpuVideoDecoder::Reset(const base::Closure& closure) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK(reset_cb_.is_null());
+  reset_cb_ = BindToCurrentLoop(closure);
+
+  // Defer the reset if a decode is pending.
+  if (!decode_cb_.is_null())
+    return;
+
+  DoReset();
+}
+
+void VpuVideoDecoder::Stop(const base::Closure& closure) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  base::ScopedClosureRunner runner(BindToCurrentLoop(closure));
+
+  if (state_ == kUninitialized)
+    return;
+
+  if (!decode_cb_.is_null()) {
+    base::ResetAndReturn(&decode_cb_).Run(kOk, NULL);
+    // Reset is pending only when decode is pending.
+    if (!reset_cb_.is_null())
+      base::ResetAndReturn(&reset_cb_).Run();
+  }
+
+  state_ = kUninitialized;
+}
+
+bool VpuVideoDecoder::HasAlpha() const {
+  return false;
+}
+
+void VpuVideoDecoder::DecodeBuffer(const scoped_refptr<DecoderBuffer>& buffer) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK_NE(state_, kUninitialized);
+  DCHECK_NE(state_, kDecodeFinished);
+  DCHECK_NE(state_, kError);
+  DCHECK(reset_cb_.is_null());
+  DCHECK(!decode_cb_.is_null());
+  DCHECK(buffer);
+
+  // Transition to kDecodeFinished on the first end of stream buffer.
+  if (state_ == kNormal && buffer->end_of_stream()) {
+    state_ = kDecodeFinished;
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    return;
+  }
+
+  scoped_refptr<VideoFrame> video_frame;
+  if (!VpuDecode(buffer, &video_frame)) {
+    state_ = kError;
+    base::ResetAndReturn(&decode_cb_).Run(kDecodeError, NULL);
+    return;
+  }
+
+  // If we didn't get a frame we need more data.
+  if (!video_frame.get()) {
+    base::ResetAndReturn(&decode_cb_).Run(kNotEnoughData, NULL);
+    return;
+  }
+
+  base::ResetAndReturn(&decode_cb_).Run(kOk, video_frame);
+}
+
+bool VpuVideoDecoder::VpuDecode(const scoped_refptr<DecoderBuffer>& buffer,
+                                scoped_refptr<VideoFrame>* video_frame) {
+  DCHECK(video_frame);
+  DCHECK(!buffer->end_of_stream());
+
+  int buffer_ret_code;
+  VpuBufferNode in_data;
+  VpuDecRetCode ret;
+
+  memset(&in_data, 0, sizeof(in_data));
+  in_data.pVirAddr = (unsigned char *)(buffer->data());
+  in_data.nSize = buffer->data_size();
+
+  ret = VPU_DecDecodeBuf(internal->handle, &in_data, &buffer_ret_code);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Failed to decode frame: " << VpuStrerror(ret);
+    return false;
+  }
+
+  VLOG(1) << "VPU_DecDecodeBuf buffer ret code is 0x" << std::hex << buffer_ret_code << std::dec;
+
+  if (buffer_ret_code & VPU_DEC_INIT_OK) {
+    ret = VPU_DecGetInitialInfo(internal->handle, &(internal->init_info));
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get init info: " << VpuStrerror(ret);
+      return false;
+    }
+    internal->framebuffers.init(internal->handle, internal->init_info);
+  }
+
+  if (buffer_ret_code & VPU_DEC_FLUSH) {
+    ret = VPU_DecFlushAll(internal->handle);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Flushing VPU failed: " << VpuStrerror(ret);
+      return false;
+    }
+
+    *video_frame = NULL;
+
+    return true;
+  }
+
+  if (buffer_ret_code & VPU_DEC_NO_ENOUGH_INBUF) {
+    LOG(INFO) << "Need more input";
+    return true;
+  }
+
+  if (buffer_ret_code & VPU_DEC_ONE_FRM_CONSUMED) {
+    VpuDecFrameLengthInfo dec_framelen_info;
+
+    ret = VPU_DecGetConsumedFrameInfo(internal->handle, &dec_framelen_info);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get information about consumed frame: " << VpuStrerror(ret);
+    } else {
+      LOG(INFO) << "One frame got consumed:"
+        << "  framebuffer addr: 0x" << std::hex << uintptr_t(dec_framelen_info.pFrame) << std::dec
+        << "  stuff length: " << dec_framelen_info.nStuffLength
+        << "  frame length: " << dec_framelen_info.nFrameLength
+        ;
+    }
+  }
+
+  if (buffer_ret_code & VPU_DEC_OUTPUT_DIS) {
+    VpuDecOutFrameInfo out_frame_info;
+    int64 timestamp = buffer->timestamp().InMicroseconds();
+
+    ret = VPU_DecGetOutputFrame(internal->handle, &out_frame_info);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get decoded output frame: " << VpuStrerror(ret);
+      return false;
+    }
+
+    LOG(INFO) << "Output frame:"
+      << "  pic width: " << internal->init_info.nPicWidth
+      << "  pic height: " << internal->init_info.nPicHeight
+      << "  pic type: " << out_frame_info.ePicType
+      << "  Y stride: " << out_frame_info.pDisplayFrameBuf->nStrideY
+      << "  CbCr stride: " << out_frame_info.pDisplayFrameBuf->nStrideC
+      ;
+
+    CopyVpuFramebufferTo(out_frame_info.pDisplayFrameBuf, video_frame, internal->init_info, config_);
+    (*video_frame)->SetTimestamp(base::TimeDelta::FromMicroseconds(timestamp));  
+
+    ret = VPU_DecOutFrameDisplayed(internal->handle, out_frame_info.pDisplayFrameBuf);
+    if (ret != VPU_DEC_RET_SUCCESS)
+      LOG(ERROR) << "Clearing display framebuffer failed: " << VpuStrerror(ret);
+  }
+  else
+    *video_frame = NULL;
+
+  return true;
+}
+
+void VpuVideoDecoder::DoReset() {
+  DCHECK(decode_cb_.is_null());
+
+  state_ = kNormal;
+  reset_cb_.Run();
+  reset_cb_.Reset();
+}
+
+void CopyVpuFramebufferTo(VpuFrameBuffer *framebuffer, scoped_refptr<VideoFrame>* video_frame, VpuDecInitInfo &init_info, const VideoDecoderConfig& config) {
+  CHECK(framebuffer);
+
+  gfx::Size size(init_info.nPicWidth, init_info.nPicHeight);
+
+  *video_frame = VideoFrame::CreateFrame(
+      config.format(),
+      size,
+      gfx::Rect(size),
+      config.natural_size(),
+      kNoTimestamp());
+
+  CopyYPlane(framebuffer->pbufVirtY,
+             framebuffer->nStrideY,
+             init_info.nPicHeight,
+             video_frame->get());
+  if (config.format() == VideoFrame::I420) {
+    CopyUPlane(framebuffer->pbufVirtCb,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+    CopyVPlane(framebuffer->pbufVirtCr,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+  } else { /* YV12 and YV12A */
+    /* U and V swapped */
+    CopyUPlane(framebuffer->pbufVirtCb,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+    CopyVPlane(framebuffer->pbufVirtCr,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+
+    if (config.format() == VideoFrame::YV12A) {
+      MakeOpaqueAPlane(framebuffer->nStrideY,
+                       init_info.nPicHeight,
+                       video_frame->get()); 
+    }
+  }
+}
+
+
+}  // namespace media
diff --git a/media/filters/vpu_video_decoder.h b/media/filters/vpu_video_decoder.h
new file mode 100644
index 0000000..d7016aa
--- /dev/null
+++ b/media/filters/vpu_video_decoder.h
@@ -0,0 +1,74 @@
+#ifndef MEDIA_FILTERS_VPU_VIDEO_DECODER_H_
+#define MEDIA_FILTERS_VPU_VIDEO_DECODER_H_
+
+#include "base/callback.h"
+#include "base/memory/weak_ptr.h"
+#include "media/base/demuxer_stream.h"
+#include "media/base/video_decoder.h"
+#include "media/base/video_decoder_config.h"
+#include "media/base/video_frame.h"
+
+namespace base {
+class MessageLoopProxy;
+}
+
+namespace media {
+
+class MEDIA_EXPORT VpuVideoDecoder : public VideoDecoder {
+ public:
+  explicit VpuVideoDecoder(
+      const scoped_refptr<base::MessageLoopProxy>& message_loop);
+  virtual ~VpuVideoDecoder();
+
+  // VideoDecoder implementation.
+  virtual void Initialize(const VideoDecoderConfig& config,
+                          const PipelineStatusCB& status_cb) OVERRIDE;
+  virtual void Decode(const scoped_refptr<DecoderBuffer>& buffer,
+                      const DecodeCB& decode_cb) OVERRIDE;
+  virtual void Reset(const base::Closure& closure) OVERRIDE;
+  virtual void Stop(const base::Closure& closure) OVERRIDE;
+  virtual bool HasAlpha() const OVERRIDE;
+
+ private:
+  enum DecoderState {
+    kUninitialized,
+    kNormal,
+    kFlushCodec,
+    kDecodeFinished,
+    kError
+  };
+
+  // Handles (re-)initializing the decoder with a (new) config.
+  // Returns true when initialization was successful.
+  bool ConfigureDecoder(const VideoDecoderConfig& config);
+
+  bool OpenDecoder(const VideoDecoderConfig& config);
+  void CloseDecoder();
+
+  void DecodeBuffer(const scoped_refptr<DecoderBuffer>& buffer);
+  bool VpuDecode(const scoped_refptr<DecoderBuffer>& buffer,
+                 scoped_refptr<VideoFrame>* video_frame);
+
+  // Reset decoder and call |reset_cb_|.
+  void DoReset();
+
+  scoped_refptr<base::MessageLoopProxy> message_loop_;
+  base::WeakPtrFactory<VpuVideoDecoder> weak_factory_;
+  base::WeakPtr<VpuVideoDecoder> weak_this_;
+
+  DecoderState state_;
+
+  DecodeCB decode_cb_;
+  base::Closure reset_cb_;
+
+  VideoDecoderConfig config_;
+
+  struct Internal;
+  Internal *internal;
+
+  DISALLOW_COPY_AND_ASSIGN(VpuVideoDecoder);
+};
+
+}  // namespace media
+
+#endif  // MEDIA_FILTERS_VPU_VIDEO_DECODER_H_
diff --git a/media/media.gyp b/media/media.gyp
index 4782741..ec53626 100644
--- a/media/media.gyp
+++ b/media/media.gyp
@@ -11,6 +11,7 @@
     # (DT_NEEDED) instead of using dlopen. This helps with automated
     # detection of ABI mismatches and prevents silent errors.
     'linux_link_pulseaudio%': 0,
+    'media_use_imxvpu%': 1,
     'conditions': [
       ['OS=="android"', {
         # Android doesn't use ffmpeg.
@@ -404,6 +405,10 @@
         'filters/video_frame_painter.h',
         'filters/video_renderer_impl.cc',
         'filters/video_renderer_impl.h',
+        'filters/vpu_framebuffers.cc',
+        'filters/vpu_framebuffers.h',
+        'filters/vpu_video_decoder.cc',
+        'filters/vpu_video_decoder.h',
         'filters/vpx_video_decoder.cc',
         'filters/vpx_video_decoder.h',
         'filters/webvtt_util.h',
@@ -658,6 +663,29 @@
                 'DISABLE_USER_INPUT_MONITOR',
               ],
             }],
+            ['media_use_imxvpu==1', {
+              'cflags': [
+                '<!@(<(pkg-config) --cflags libfslvpuwrap)',
+              ],
+              'link_settings': {
+                'libraries': [
+                  '<!@(<(pkg-config) --libs libfslvpuwrap)',
+                ],
+              },
+            }, {  # media_use_imxvpu==0
+              'direct_dependent_settings': {
+                'defines': [
+                  'MEDIA_DISABLE_IMXVPU',
+                ],
+              },
+              # Exclude the VPU sources.
+              'sources!': [
+                'filters/vpu_framebuffers.cc',
+                'filters/vpu_framebuffers.h',
+                'filters/vpu_video_decoder.cc',
+                'filters/vpu_video_decoder.h',
+              ],
+            }],
             ['use_cras==1', {
               'cflags': [
                 '<!@(<(pkg-config) --cflags libcras)',
-- 
1.8.3.2


[-- Attachment #4: 0003-Update-vpu_video_decoder.cc-to-match-chromium-35.0.1883.patch --]
[-- Type: text/x-diff, Size: 1813 bytes --]

From f059d6501b17ab8aed618f53f7e4255cf8d8326b Mon Sep 17 00:00:00 2001
From: Mahyar Yaghmaee <mahyar@boundarydevices.com>
Date: Fri, 28 Mar 2014 14:06:40 -0700
Subject: [PATCH 3/3] Update vpu_video_decoder.cc to match chromium-35.0.1883

Signed-off-by: Mahyar Yaghmaee <mahyar@boundarydevices.com>
---
 media/filters/vpu_video_decoder.cc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/media/filters/vpu_video_decoder.cc b/media/filters/vpu_video_decoder.cc
index 396d4f1..9024484 100644
--- a/media/filters/vpu_video_decoder.cc
+++ b/media/filters/vpu_video_decoder.cc
@@ -18,7 +18,7 @@
 #include "base/message_loop/message_loop_proxy.h"
 #include "base/strings/string_number_conversions.h"
 #include "base/sys_byteorder.h"
-#include "media/base/bind_to_loop.h"
+#include "media/base/bind_to_current_loop.h"
 #include "media/base/decoder_buffer.h"
 #include "media/base/demuxer_stream.h"
 #include "media/base/media_switches.h"
@@ -393,7 +393,7 @@ void VpuVideoDecoder::Decode(const scoped_refptr<DecoderBuffer>& buffer,
 
   // Return empty frames if decoding has finished.
   if (state_ == kDecodeFinished) {
-    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEOSFrame());
     return;
   }
 
@@ -445,7 +445,7 @@ void VpuVideoDecoder::DecodeBuffer(const scoped_refptr<DecoderBuffer>& buffer) {
   // Transition to kDecodeFinished on the first end of stream buffer.
   if (state_ == kNormal && buffer->end_of_stream()) {
     state_ = kDecodeFinished;
-    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEOSFrame());
     return;
   }
 
-- 
1.8.3.2

Re: Chromium acceleration

[Boszormenyi Zoltan]

2014-03-25 10:46 keltezéssel, Marco Trillo írta:
> Hi,
>
> On 03/25/2014 10:32 AM, zboszor@pr.hu wrote:
>> Hi,
>>
>> how can I add the meta-browser tree to the set of recipes? I did:
>>
>> $ cd fsl-community-bsp/sources
>> $ git clone https://github.com/OSSystems/meta-browser.git
>> $ cd ..
>> $ . ./setup-environment build-master-next/
>> $ bitbake chromium
>> Loading cache: 100%
>> |###########################################################################################################################| 
>>
>> ETA:  00:00:00
>> Loaded 1806 entries from dependency cache.
>> ERROR: Nothing PROVIDES 'chromium'
>>
>> Summary: There was 1 WARNING message shown.
>> Summary: There was 1 ERROR message shown, returning a non-zero exit code.
>> [zozo@localhost build-master-next]$
>>
>> Where should I move the meta-browser directory from the toplevel "sources"?
>> What else should be done?
>
> You need to add the "meta-browser" layer to your bblayers.conf (in build/conf/), like this:
>
> BBLAYERS += " ${BSPDIR}/sources/meta-browser "
>
> You will probably need other to add other layers such as "meta-gnome".

Thanks for the answer. I also added "meta-gnome" to BBLAYERS and
I needed these set in local.conf:

LICENSE_FLAGS_WHITELIST = "commercial_libav commercial_x264"
PREFERRED_PROVIDER_jpeg = "libjpeg"
PREFERRED_PROVIDER_jpeg-native = "libjpeg-native"

Using libjpeg-turbo fails with:

ERROR: Fetcher failure: Fetch command failed with exit code 1, output:
svn: E210002: Unable to connect to a repository at URL 
'svn://svn.code.sf.net/p/libjpeg-turbo/code/trunk'
svn: E210002: Network connection closed unexpectedly

ERROR: Function failed: Fetcher failure for URL: 
'svn://svn.code.sf.net/p/libjpeg-turbo/code;protocol=svn;module=trunk'. Unable to fetch 
URL from any source.
ERROR: Logfile of failure stored in: 
/home/zozo/fsl-community-bsp/build-master-next/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/libjpeg-turbo/8d+1.3.0-r0/temp/log.do_fetch.23323
ERROR: Task 949 
(/home/zozo/fsl-community-bsp/sources/meta-openembedded/meta-oe/recipes-core/jpeg/libjpeg-turbo_svn.bb, 
do_fetch) failed with exit code '1'

Best regards,
Zoltán Böszörményi

Re: Chromium acceleration

[Marco Trillo]

Hi,

On 03/25/2014 10:32 AM, zboszor@pr.hu wrote:
> Hi,
>
> how can I add the meta-browser tree to the set of recipes? I did:
>
> $ cd fsl-community-bsp/sources
> $ git clone https://github.com/OSSystems/meta-browser.git
> $ cd ..
> $ . ./setup-environment build-master-next/
> $ bitbake chromium
> Loading cache: 100%
> |###########################################################################################################################|
> ETA:  00:00:00
> Loaded 1806 entries from dependency cache.
> ERROR: Nothing PROVIDES 'chromium'
>
> Summary: There was 1 WARNING message shown.
> Summary: There was 1 ERROR message shown, returning a non-zero exit code.
> [zozo@localhost build-master-next]$
>
> Where should I move the meta-browser directory from the toplevel "sources"?
> What else should be done?

You need to add the "meta-browser" layer to your bblayers.conf (in 
build/conf/), like this:

BBLAYERS += " ${BSPDIR}/sources/meta-browser "

You will probably need other to add other layers such as "meta-gnome".

Hope it helps,
Marco.

Re: Chromium acceleration

[Eric Bénard]

Hi Zoltán,

Le Tue, 25 Mar 2014 10:32:22 +0100,
zboszor@pr.hu a écrit :

> Hi,
> 
> how can I add the meta-browser tree to the set of recipes? I did:
> 
> $ cd fsl-community-bsp/sources
> $ git clone https://github.com/OSSystems/meta-browser.git
> $ cd ..
> $ . ./setup-environment build-master-next/
> $ bitbake chromium
> Loading cache: 100%
> |###########################################################################################################################|
> ETA:  00:00:00
> Loaded 1806 entries from dependency cache.
> ERROR: Nothing PROVIDES 'chromium'
> 
> Summary: There was 1 WARNING message shown.
> Summary: There was 1 ERROR message shown, returning a non-zero exit code.
> [zozo@localhost build-master-next]$
> 
> Where should I move the meta-browser directory from the toplevel "sources"?
> What else should be done?
> 
http://www.yoctoproject.org/docs/1.5.1/dev-manual/dev-manual.html#enabling-your-layer

Eric

Re: Chromium acceleration

[zboszor]

Hi,

how can I add the meta-browser tree to the set of recipes? I did:

$ cd fsl-community-bsp/sources
$ git clone https://github.com/OSSystems/meta-browser.git
$ cd ..
$ . ./setup-environment build-master-next/
$ bitbake chromium
Loading cache: 100%
|###########################################################################################################################|
ETA:  00:00:00
Loaded 1806 entries from dependency cache.
ERROR: Nothing PROVIDES 'chromium'

Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
[zozo@localhost build-master-next]$

Where should I move the meta-browser directory from the toplevel "sources"?
What else should be done?

Thanks in advance,
Zoltán Böszörményi

Re: Chromium acceleration

[Eric Nelson]

Hi Diego,

On 03/21/2014 04:49 AM, Diego wrote:
> Otavio Salvador wrote:
>> Eric Nelson wrote:
>>> A simplistic test shows that it's really straightforward to
>>> add both Chromium and Firefox into fsl-image-gui by pulling
>>> in meta-browser.
>>
>> The biggest problem adding more stuff is maintenance. This increases
>> build time and tests needed for a good coverage.
>>
>> What people think?
>
> Hi Otavio and Eric,
>
> I think we can sum up your discussion in two points:
> - web applications and in general browser usage is really important nowadays
> in general, and it is taking its share of interest also in the embedded world;
> - firefox and chromium require quite a lot of resources both in terms of
> maintenance and in terms of build time and computation.
>
> I can confirm both points, as we have a couple of projects using Chromium. So
> on one hand it is something important to support and test regularly (and that
> would help finding build breakages), on the other hand build takes a lot time,
> and an overwhelming amount of RAM. I used to build images with Firefox and
> Chromium 29 with 6 bitbake build threads and it worked on a 4GB of RAM and 2GB
> swap, but now with Chromium 35 I had to pump up the VM to 12GB of RAM (8GB
> still killed the final ld linking process, which is the real "RAM eater").
>
> So while testing browsers would be nice, I think the best option would be to
> test it somehow separately from the fsl-image-gui. Do you think defining an
> image in meta-browser would make sense?
>

I wasn't suggesting that Chromium or Firefox be included in
fsl-image-gui. Only that meta-browser (and I think meta-gnome as a
dependency) be included in the default manifest.

This will prevent the need to "git clone" the repositories separately,
and allow inclusion through local.conf.

Regards,


Eric

Re: Chromium acceleration

[Dmitriy B.]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

>
> 3. Chromium tried to load libEGL with dlopen() , which caused problems,
> because with the Vivante libraries, libEGL also needs libGAL. Usually,
> build scripts just add -lEGL -lGAL to the linker line.
>

Vivante finally fixed that in the latest EGL binaries release so now you
dont need to manually add libGAL everytime. libEGL/GLESv1-2/GL now depend
on libGAL properly (you can check that with ldd).

Looking forward to test Chromium with your imx6 patches soon!

Best Regards,
Dmitriy Beykun

[-- Attachment #2: Type: text/html, Size: 833 bytes --]

Re: Chromium acceleration

[Diego]

Otavio Salvador wrote:
> Eric Nelson wrote:
> > A simplistic test shows that it's really straightforward to
> > add both Chromium and Firefox into fsl-image-gui by pulling
> > in meta-browser.
> 
> The biggest problem adding more stuff is maintenance. This increases
> build time and tests needed for a good coverage.
> 
> What people think?

Hi Otavio and Eric,

I think we can sum up your discussion in two points:
- web applications and in general browser usage is really important nowadays 
in general, and it is taking its share of interest also in the embedded world;
- firefox and chromium require quite a lot of resources both in terms of 
maintenance and in terms of build time and computation.

I can confirm both points, as we have a couple of projects using Chromium. So 
on one hand it is something important to support and test regularly (and that 
would help finding build breakages), on the other hand build takes a lot time, 
and an overwhelming amount of RAM. I used to build images with Firefox and 
Chromium 29 with 6 bitbake build threads and it worked on a 4GB of RAM and 2GB 
swap, but now with Chromium 35 I had to pump up the VM to 12GB of RAM (8GB 
still killed the final ld linking process, which is the real "RAM eater").

So while testing browsers would be nice, I think the best option would be to 
test it somehow separately from the fsl-image-gui. Do you think defining an 
image in meta-browser would make sense?

Bests,
Diego

Re: Chromium acceleration

[Carlos Rafael Giani]

[-- Attachment #1.1: Type: text/plain, Size: 3898 bytes --]

On 2014-03-20 14:46, Christian Betz wrote:
>
>     As for the decoder itself: I implemented it in the Chromium media
>     framework, in media/ . I simply took the vpx decoder code, copied
>     it, and modified it to use the VPU. I had VP8, MPEG2, MPEG4, and
>     h264 decoding working. It wasnt much code, but unfortunately, the
>     interfaces tend to change between versions, so the code would have
>     to be updated for the newest Chromium version. It is not much
>     code, I will try to clean it up a bit and post it. In addition, I
>     will ask the colleagues for the other patches for accelerating the
>     2D and WebGL rendering.
>
>
> thanks carlos!
>
> my analysis suggested that my team should reproduce these chunks of 
> code which are for samsung exynos:
>
> https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/exynos_video_decode_accelerator.cc
>
> to oversimplify greatly: we want to port 
> 'exynos_video_decode_accelerator' to a new 
> 'imx_video_decode_accelerator'. in this module we would make use of 
> the same direct texture techniques as eglvivsink from gstreamer-imx.
>
> this is definitely not the route you have taken, from what I can tell. 
> is this something you considered and steered away from? have you 
> examined these code paths at all? there are a few other special 
> modules besides exynos too.
>
> i might be wrong, but it could be the best chance of getting something 
> upstreamed would be do it "like everyone else" (i.e. samsung). but i'm 
> probably thinking *way* too far ahead here.

Back then we needed some hw decoding quickly, so we did not look further 
into this, since we had spent a lot of time getting the 2D acceleration 
stable already. I could only briefly look at the exynos accelerator 
code, but it does look like the right direction, although the VPU uses 
physical addresses directly instead of dmabuf FDs. I am currently 
writing a frontend for the libfslvpuwrap, which takes care of certain 
complexities (like being able to pass userdata pointers to frames, to 
make it possible to associate input and output frames, which is 
essential for correct timestamping, especially when things like h264 
frame reordering are active). This frontend will hide these things in a 
future gstreamer-imx release to make the decoder code clearer and easier 
to maintain, and is intended to be reusable, for example for Chromium 
integration, or XBMC. Beyond that, my patches are probably not that 
helpful anymore, since they were based on the vpx decoder way of 
decoding, and I agree that the exynos code is a better starting point. 
Nevertheless, I attached the patch. Keep in mind that it is very basic, 
old, and wasn't further developed on because it was "good enough" for 
the project.

But here are some additional notes from our efforts to accelerate 
Chromium on imx6 (keep in mind these apply to version 32.0.1664.3 ) :

1. We started the google-chrome binary with these switches: 
--ignore-gpu-blacklist --enable-gpu --use-gl=egl 
--enable-accelerated-2d-canvas

2. To make building Chromium easier, we switched to component build (by 
default, it is all linked into one enormous binary). I attached a patch 
that was necessary to fix some minor issues. Perhaps it is not necessary 
anymore.

3. Chromium tried to load libEGL with dlopen() , which caused problems, 
because with the Vivante libraries, libEGL also needs libGAL. Usually, 
build scripts just add -lEGL -lGAL to the linker line. With dlopen() 
this wasn't possible of course. We patched the gyp scripts to link 
against this libraries during building instead. I attached this patch. 
It is really a hack, because it circumvents the sandboxing process (this 
is why Chromium loads EGL and GLESv2 with dlopen() ).


Carlos

[-- Attachment #1.2: Type: text/html, Size: 5719 bytes --]

[-- Attachment #2: 0002-Fix-issue-with-component-linking.patch --]
[-- Type: text/x-patch, Size: 823 bytes --]

--- a/build/common.gypi	2013-09-05 12:33:58.000000000 +0200
+++ b/build/common.gypi	2013-09-13 09:29:24.185329621 +0200
@@ -2840,11 +2840,6 @@
                   '-Wl,--as-needed',
                 ],
               }],
-              ['OS=="linux" and target_arch=="ia32"', {
-                'ldflags': [
-                  '-Wl,--no-as-needed',
-                ],
-              }],
             ],
           },
           'Release_Base': {
@@ -3460,6 +3455,10 @@
               # gets the right path.
               '-B<(PRODUCT_DIR)/../../third_party/gold',
             ],
+	    }, {
+	            'ldflags': [
+	            '-Wl,--no-as-needed',
+		 ],
           }],
           ['native_discardable_memory', {
             'defines': ['DISCARDABLE_MEMORY_ALWAYS_SUPPORTED_NATIVELY'],



[-- Attachment #3: 0004-Modified-gl.gyp-to-link-libEGL-and-libGAL-at-build-t.patch --]
[-- Type: text/x-patch, Size: 636 bytes --]

diff --git a/ui/gl/gl.gyp b/ui/gl/gl.gyp
index 274065b..e857f9c 100644
--- a/ui/gl/gl.gyp
+++ b/ui/gl/gl.gyp
@@ -123,6 +123,12 @@
         '<(gl_binding_output_dir)/gl_bindings_autogen_osmesa.h',
         '<(gl_binding_output_dir)/gl_interface_autogen_gl.h',
       ],
+      'link_settings': {
+        'libraries': [
+          '-lEGL',
+          '-lGAL',
+        ],
+      },
       # hard_dependency is necessary for this target because it has actions
       # that generate header files included by dependent targets. The header
       # files must be generated before the dependents are compiled. The usual



[-- Attachment #4: 0005-Added-hardware-accelerated-decoding-with-the-i.MX-VP.patch --]
[-- Type: text/x-patch, Size: 28854 bytes --]

diff --git a/content/renderer/media/webmediaplayer_impl.cc b/content/renderer/media/webmediaplayer_impl.cc
index a99c3ae..1ec8bfa 100644
--- a/content/renderer/media/webmediaplayer_impl.cc
+++ b/content/renderer/media/webmediaplayer_impl.cc
@@ -47,6 +47,7 @@
 #include "media/filters/gpu_video_decoder.h"
 #include "media/filters/opus_audio_decoder.h"
 #include "media/filters/video_renderer_base.h"
+#include "media/filters/vpu_video_decoder.h"
 #include "media/filters/vpx_video_decoder.h"
 #include "third_party/WebKit/public/platform/WebMediaSource.h"
 #include "third_party/WebKit/public/platform/WebRect.h"
@@ -1103,6 +1104,10 @@ void WebMediaPlayerImpl::StartPipeline() {
         new media::GpuVideoDecoder(gpu_factories_, media_log_));
   }
 
+#if !defined(MEDIA_DISABLE_IMXVPU)
+  video_decoders.push_back(new media::VpuVideoDecoder(media_loop_));
+#endif  // !defined(MEDIA_DISABLE_IMXVPU)
+
   // TODO(phajdan.jr): Remove ifdefs when libvpx with vp9 support is released
   // (http://crbug.com/174287) .
 #if !defined(MEDIA_DISABLE_LIBVPX)
diff --git a/media/filters/vpu_framebuffers.cc b/media/filters/vpu_framebuffers.cc
new file mode 100644
index 0000000..fce43fc
--- /dev/null
+++ b/media/filters/vpu_framebuffers.cc
@@ -0,0 +1,106 @@
+#include <string.h>
+#include <stdint.h>
+#include "vpu_framebuffers.h"
+#include "base/logging.h"
+
+
+#define ALIGN_VAL_TO(LENGTH, ALIGN_SIZE)  ( ((uintptr_t)((LENGTH) + (ALIGN_SIZE) - 1) / (ALIGN_SIZE)) * (ALIGN_SIZE) )
+#define FRAME_ALIGN 16
+
+
+namespace media {
+
+
+VPUFramebuffers::VPUFramebuffers() {
+}
+
+
+void VPUFramebuffers::init(VpuDecHandle handle, VpuDecInitInfo const &init_info) {
+  framebuffers.resize(init_info.nMinFrameBufferCount);
+
+  y_stride = ALIGN_VAL_TO(init_info.nPicWidth, FRAME_ALIGN);
+  if (init_info.nInterlace)
+    y_size = y_stride * ALIGN_VAL_TO(init_info.nPicHeight, (2 * FRAME_ALIGN));
+  else
+    y_size = y_stride * ALIGN_VAL_TO(init_info.nPicHeight, FRAME_ALIGN);
+
+  uv_stride = y_stride / 2;
+  u_size = v_size = mv_size = y_size / 4;
+
+  int alignment = init_info.nAddressAlignment;
+  if (alignment > 1)
+  {
+    y_size = ALIGN_VAL_TO(y_size, alignment);
+    u_size = ALIGN_VAL_TO(u_size, alignment);
+    v_size = ALIGN_VAL_TO(v_size, alignment);
+    mv_size = ALIGN_VAL_TO(mv_size, alignment);
+  }
+
+  pic_width = init_info.nPicWidth;
+  pic_height = init_info.nPicHeight;
+
+  total_size = y_size + u_size + v_size + mv_size + alignment;
+
+  LOG(INFO) << "Num VPU framebuffers: " << framebuffers.size();
+  LOG(INFO) << "VPU framebuffer memory block size:"
+    << "  total: " << total_size
+    << "  Y " << y_size
+    << "  U: " << u_size
+    << "  V: " << v_size
+    << "  Mv: " << mv_size
+    << "  alignment: " << alignment
+    ;
+
+  for (Framebuffers::iterator iter = framebuffers.begin(); iter != framebuffers.end(); ++iter) {
+    unsigned char *virt_ptr, *phys_ptr;
+
+    VpuFrameBuffer &framebuffer = *iter;
+
+    VpuMemDesc mem_desc;
+    memset(&mem_desc, 0, sizeof(VpuMemDesc));
+    mem_desc.nSize = total_size;
+    VPU_DecGetMem(&mem_desc);
+    phys_mem_blocks.push_back(mem_desc);
+
+    virt_ptr = (unsigned char*)(mem_desc.nVirtAddr);
+    phys_ptr = (unsigned char*)(mem_desc.nPhyAddr);
+
+    if (alignment > 1) {
+      phys_ptr = (unsigned char*)ALIGN_VAL_TO(phys_ptr, alignment);
+      virt_ptr = (unsigned char*)ALIGN_VAL_TO(virt_ptr, alignment);
+    }
+
+    framebuffer.nStrideY = y_stride;
+    framebuffer.nStrideC = uv_stride;
+
+    /* fill phy addr*/
+    framebuffer.pbufY     = phys_ptr;
+    framebuffer.pbufCb    = phys_ptr + y_size;
+    framebuffer.pbufCr    = phys_ptr + y_size + u_size;
+    framebuffer.pbufMvCol = phys_ptr + y_size + u_size + v_size;
+
+    /* fill virt addr */
+    framebuffer.pbufVirtY     = virt_ptr;
+    framebuffer.pbufVirtCb    = virt_ptr + y_size;
+    framebuffer.pbufVirtCr    = virt_ptr + y_size + u_size;
+    framebuffer.pbufVirtMvCol = virt_ptr + y_size + u_size + v_size;
+
+    framebuffer.pbufY_tilebot = 0;
+    framebuffer.pbufCb_tilebot = 0;
+    framebuffer.pbufVirtY_tilebot = 0;
+    framebuffer.pbufVirtCb_tilebot = 0;
+  }
+
+  VPU_DecRegisterFrameBuffer(handle, &(framebuffers[0]), framebuffers.size());
+}
+
+
+VPUFramebuffers::~VPUFramebuffers() {
+  LOG(INFO) << "Freeing VPU framebuffer memory";
+  for (PhysMemBlocks::iterator iter = phys_mem_blocks.begin(); iter != phys_mem_blocks.end(); ++iter)
+    VPU_DecFreeMem(&(*iter));
+}
+
+
+}
+
diff --git a/media/filters/vpu_framebuffers.h b/media/filters/vpu_framebuffers.h
new file mode 100644
index 0000000..bb9b021
--- /dev/null
+++ b/media/filters/vpu_framebuffers.h
@@ -0,0 +1,36 @@
+#ifndef MEDIA_FILTERS_VPU_FRAMEBUFFERS_H_
+#define MEDIA_FILTERS_VPU_FRAMEBUFFERS_H_
+
+#include <list>
+#include <vector>
+#include <vpu_wrapper.h>
+
+
+namespace media {
+
+
+class VPUFramebuffers
+{
+public:
+  explicit VPUFramebuffers();
+  ~VPUFramebuffers();
+
+  void init(VpuDecHandle handle, VpuDecInitInfo const &init_info);
+
+private:
+  typedef std::vector < VpuFrameBuffer > Framebuffers;
+  typedef std::list < VpuMemDesc > PhysMemBlocks;
+
+  PhysMemBlocks phys_mem_blocks;
+  Framebuffers framebuffers;
+  int y_stride, uv_stride;
+  int y_size, u_size, v_size, mv_size;
+  int total_size;
+  unsigned int pic_width, pic_height;
+};
+
+
+}
+
+
+#endif
diff --git a/media/filters/vpu_video_decoder.cc b/media/filters/vpu_video_decoder.cc
new file mode 100644
index 0000000..396d4f1
--- /dev/null
+++ b/media/filters/vpu_video_decoder.cc
@@ -0,0 +1,614 @@
+#include <list>
+#include <vector>
+#include <stddef.h>
+#include <stdint.h>
+#include <algorithm>
+#include <string>
+
+#include <string.h>
+
+#include "media/filters/vpu_video_decoder.h"
+
+#include "base/bind.h"
+#include "base/callback_helpers.h"
+#include "base/command_line.h"
+#include "base/location.h"
+#include "base/logging.h"
+#include "base/synchronization/lock.h"
+#include "base/message_loop/message_loop_proxy.h"
+#include "base/strings/string_number_conversions.h"
+#include "base/sys_byteorder.h"
+#include "media/base/bind_to_loop.h"
+#include "media/base/decoder_buffer.h"
+#include "media/base/demuxer_stream.h"
+#include "media/base/media_switches.h"
+#include "media/base/pipeline.h"
+#include "media/base/video_decoder_config.h"
+#include "media/base/video_frame.h"
+#include "media/base/video_util.h"
+
+#include "media/filters/vpu_framebuffers.h"
+
+#include <vpu_wrapper.h>
+
+
+#define ALIGN_VAL_TO(LENGTH, ALIGN_SIZE)  ( ((uintptr_t)((LENGTH) + (ALIGN_SIZE) - 1) / (ALIGN_SIZE)) * (ALIGN_SIZE) )
+
+
+
+namespace media {
+
+
+namespace {
+
+
+char const *VpuStrerror(VpuDecRetCode const code) {
+  switch (code) {
+    case VPU_DEC_RET_SUCCESS: return "success";
+    case VPU_DEC_RET_FAILURE: return "failure";
+    case VPU_DEC_RET_INVALID_PARAM: return "invalid param";
+    case VPU_DEC_RET_INVALID_HANDLE: return "invalid handle";
+    case VPU_DEC_RET_INVALID_FRAME_BUFFER: return "invalid frame buffer";
+    case VPU_DEC_RET_INSUFFICIENT_FRAME_BUFFERS: return "insufficient frame buffers";
+    case VPU_DEC_RET_INVALID_STRIDE: return "invalid stride";
+    case VPU_DEC_RET_WRONG_CALL_SEQUENCE: return "wrong call sequence";
+    case VPU_DEC_RET_FAILURE_TIMEOUT: return "failure timeout";
+    default:
+      return NULL;
+  }
+}
+
+
+long inst_count = 0;
+base::Lock vpu_load_lock;
+
+
+void LoadDecoder() {
+  VpuDecRetCode ret;
+
+  base::AutoLock alock(vpu_load_lock);
+
+  if (inst_count == 0) {
+    ret = VPU_DecLoad();
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "VPU loading failed: " << VpuStrerror(ret);
+      return;
+    }
+
+  }
+
+  ++inst_count;
+
+  if (inst_count == 1) {
+    VpuVersionInfo version;
+    VpuWrapperVersionInfo wrapper_version;
+
+    ret = VPU_DecGetVersionInfo(&version);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Getting version info failed: " << VpuStrerror(ret);
+      return;
+    }
+
+    ret = VPU_DecGetWrapperVersionInfo(&wrapper_version);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Getting wrapper version info failed: " << VpuStrerror(ret);
+      return;
+    }
+
+    LOG(INFO) << "VPU Loaded";
+    LOG(INFO) << "VPU firmware version " << version.nFwMajor << "." << version.nFwMinor << "." << version.nFwRelease << "_r" << version.nFwCode;
+    LOG(INFO) << "VPU library version " << version.nLibMajor << "." << version.nLibMinor << "." << version.nLibRelease;
+    LOG(INFO) << "VPU wrapper version " << wrapper_version.nMajor << "." << wrapper_version.nMinor << "." << wrapper_version.nRelease << " " << wrapper_version.pBinary;
+  }
+}
+
+
+void UnloadDecoder() {
+  base::AutoLock alock(vpu_load_lock);
+
+  if (inst_count == 0)
+    return;
+
+  --inst_count;
+
+  if (inst_count == 0)
+    VPU_DecUnLoad();
+}
+
+
+}
+
+
+struct VpuVideoDecoder::Internal
+{
+  typedef std::vector < uint8_t > VirtMemBlock;
+  typedef std::list < VirtMemBlock > VirtMemBlocks;
+
+  typedef std::list < VpuMemDesc > PhysMemBlocks;
+
+  VpuDecHandle handle;
+
+  VpuDecInitInfo init_info;
+  VpuMemInfo mem_info;
+
+  bool vpu_opened;
+
+  VirtMemBlocks virt_mem_blocks;
+  PhysMemBlocks phys_mem_blocks;
+
+  VPUFramebuffers framebuffers;
+
+  Internal()
+    : vpu_opened(false)
+  {
+  }
+
+  ~Internal()
+  {
+    FreeMemBlocks();
+  }
+
+  bool AllocMemBlocks()
+  {
+    VpuDecRetCode ret;
+
+    memset(&mem_info, 0, sizeof(VpuMemInfo));
+    ret = VPU_DecQueryMem(&mem_info);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get VPU memory information: " << VpuStrerror(ret);
+      return false;
+    }
+
+    for (int i = 0; i < mem_info.nSubBlockNum; ++i)
+    {
+      int size = mem_info.MemSubBlock[i].nAlignment + mem_info.MemSubBlock[i].nSize;
+
+      LOG(INFO) << "Sub block " << i << "  type: " << ((mem_info.MemSubBlock[i].MemType == VPU_MEM_VIRT) ? "virtual" : "phys") << "  size: " << size;
+
+      if (mem_info.MemSubBlock[i].MemType == VPU_MEM_VIRT)
+      {
+        unsigned char *virt_ptr;
+        virt_mem_blocks.push_back(VirtMemBlock());
+        VirtMemBlock &blk = virt_mem_blocks.back();
+        blk.resize(size);
+        virt_ptr = reinterpret_cast < unsigned char* > (&blk[0]);
+
+        mem_info.MemSubBlock[i].pVirtAddr = (unsigned char *)ALIGN_VAL_TO(virt_ptr, mem_info.MemSubBlock[i].nAlignment);
+      }
+      else if (mem_info.MemSubBlock[i].MemType == VPU_MEM_PHY)
+      {
+        VpuMemDesc mem_desc;
+        memset(&mem_desc, 0, sizeof(VpuMemDesc));
+        mem_desc.nSize = size;
+        if (VPU_DecGetMem(&mem_desc) != VPU_DEC_RET_SUCCESS) {
+          LOG(ERROR) << "Unable to allocate " << size << " bytes of physical memory";
+          return false;
+        }
+
+        phys_mem_blocks.push_back(mem_desc);
+
+        mem_info.MemSubBlock[i].pVirtAddr = (unsigned char *)ALIGN_VAL_TO((unsigned char*)(mem_desc.nVirtAddr), mem_info.MemSubBlock[i].nAlignment);
+        mem_info.MemSubBlock[i].pPhyAddr = (unsigned char *)ALIGN_VAL_TO((unsigned char*)(mem_desc.nPhyAddr), mem_info.MemSubBlock[i].nAlignment);
+      }
+      else
+        LOG(WARNING) << "Sub block " << i << " type is unknown - skipping";
+    }
+
+    LOG(INFO) << "VPU memory blocks allocated";
+
+    return true;
+  }
+
+  void FreeMemBlocks()
+  {
+    LOG(INFO) << "Freeing VPU memory blocks";
+
+    for (PhysMemBlocks::iterator iter = phys_mem_blocks.begin(); iter != phys_mem_blocks.end(); ++iter)
+      VPU_DecFreeMem(&(*iter));
+  }
+};
+
+
+
+void CopyVpuFramebufferTo(VpuFrameBuffer *framebuffer, scoped_refptr<VideoFrame>* video_frame, VpuDecInitInfo &init_info, const VideoDecoderConfig& config);
+
+
+
+
+VpuVideoDecoder::VpuVideoDecoder(
+    const scoped_refptr<base::MessageLoopProxy>& message_loop)
+    : message_loop_(message_loop),
+      weak_factory_(this),
+      state_(kUninitialized) {
+    LoadDecoder();
+    internal = new Internal;
+}
+
+VpuVideoDecoder::~VpuVideoDecoder() {
+  DCHECK_EQ(kUninitialized, state_);
+  CloseDecoder();
+  delete internal;
+  UnloadDecoder();
+}
+
+void VpuVideoDecoder::Initialize(const VideoDecoderConfig& config,
+                                 const PipelineStatusCB& status_cb) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK(config.IsValidConfig());
+  DCHECK(!config.is_encrypted());
+  DCHECK(decode_cb_.is_null());
+  DCHECK(reset_cb_.is_null());
+
+  weak_this_ = weak_factory_.GetWeakPtr();
+
+  if (!internal->AllocMemBlocks()) {
+    status_cb.Run(PIPELINE_ERROR_INITIALIZATION_FAILED);
+    return;
+  }
+
+  if (!ConfigureDecoder(config)) {
+    status_cb.Run(DECODER_ERROR_NOT_SUPPORTED);
+    return;
+  }
+
+  // Success!
+  LOG(INFO) << "VPU decoder initialization succeeded";
+  config_ = config;
+  state_ = kNormal;
+  status_cb.Run(PIPELINE_OK);
+}
+
+bool VpuVideoDecoder::ConfigureDecoder(const VideoDecoderConfig& config) {
+  bool can_handle = false;
+
+  if ((config.format() == VideoFrame::YV12) || (config.format() == VideoFrame::YV12A) || (config.format() == VideoFrame::I420)) {
+    switch (config.codec()) {
+      case kCodecH264:
+      case kCodecVC1:
+      case kCodecMPEG2:
+      case kCodecMPEG4:
+      case kCodecVP8:
+        can_handle = true;
+      default:
+        break;
+    }
+  }
+
+  if (!can_handle) {
+    LOG(INFO) << "Cannot handle video config: " << config.AsHumanReadableString();
+    return false;
+  }
+
+  CloseDecoder();
+  return OpenDecoder(config);
+}
+
+bool VpuVideoDecoder::OpenDecoder(const VideoDecoderConfig& config) {
+  if (internal->vpu_opened)
+    return true;
+
+  VpuDecOpenParam open_param;
+  VpuDecRetCode ret;
+
+  memset(&open_param, 0, sizeof(open_param));
+
+  switch (config.codec())
+  {
+    case kCodecH264:
+      open_param.CodecFormat = VPU_V_AVC;
+      open_param.nReorderEnable = 1;
+      LOG(INFO) << "Setting h.264 as stream format";
+      break;
+    case kCodecVC1:
+      open_param.CodecFormat = VPU_V_VC1_AP;
+      LOG(INFO) << "Setting VC1 as stream format";
+      break;
+    case kCodecMPEG2:
+      open_param.CodecFormat = VPU_V_MPEG2;
+      LOG(INFO) << "Setting MPEG-2 as stream format";
+      break;
+    case kCodecMPEG4:
+      open_param.CodecFormat = VPU_V_MPEG4;
+      LOG(INFO) << "Setting MPEG-4 as stream format";
+      break;
+    case kCodecVP8:
+      open_param.CodecFormat = VPU_V_VP8;
+      LOG(INFO) << "Setting VP8 as stream format";
+      break;
+    default:
+      LOG(ERROR) << "Could not set unknown stream format";
+      break;
+  }
+
+  open_param.nChromaInterleave = 0;
+  open_param.nMapType = 0;
+  open_param.nTiled2LinearEnable = 0;
+  open_param.nEnableFileMode = 0;
+  open_param.nPicWidth = config.coded_size().width();
+  open_param.nPicHeight = config.coded_size().height();
+
+  ret = VPU_DecOpen(&(internal->handle), &open_param, &(internal->mem_info));
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Opening new VPU handle failed: " << VpuStrerror(ret);
+    return false;
+  }
+
+  int config_param;
+
+  config_param = VPU_DEC_SKIPNONE;
+  ret = VPU_DecConfig(internal->handle, VPU_DEC_CONF_SKIPMODE, &config_param);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Could not configure skip mode: " << VpuStrerror(ret);
+    return false;
+  }
+
+  config_param = 0;
+  ret = VPU_DecConfig(internal->handle, VPU_DEC_CONF_BUFDELAY, &config_param);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Could not configure buffer delay: " << VpuStrerror(ret);
+    return false;
+  }
+
+  config_param = VPU_DEC_IN_NORMAL;
+  ret = VPU_DecConfig(internal->handle, VPU_DEC_CONF_INPUTTYPE, &config_param);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Could not configure input type: " << VpuStrerror(ret);
+    return false;
+  }
+
+  internal->vpu_opened = true;
+
+  return true;
+}
+
+void VpuVideoDecoder::CloseDecoder() {
+  VpuDecRetCode ret;
+
+  if (!(internal->vpu_opened))
+    return;
+
+  ret = VPU_DecFlushAll(internal->handle);
+  if (ret != VPU_DEC_RET_SUCCESS)
+    LOG(ERROR) << "Flushing decoder failed: " << VpuStrerror(ret);
+  VPU_DecClose(internal->handle);
+  if (ret != VPU_DEC_RET_SUCCESS)
+    LOG(ERROR) << "Closing decoder failed: " << VpuStrerror(ret);
+
+  internal->vpu_opened = false;
+}
+
+void VpuVideoDecoder::Decode(const scoped_refptr<DecoderBuffer>& buffer,
+                             const DecodeCB& decode_cb) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK(!decode_cb.is_null());
+  CHECK_NE(state_, kUninitialized);
+  CHECK(decode_cb_.is_null()) << "Overlapping decodes are not supported.";
+
+  decode_cb_ = BindToCurrentLoop(decode_cb);
+
+  if (state_ == kError) {
+    base::ResetAndReturn(&decode_cb_).Run(kDecodeError, NULL);
+    return;
+  }
+
+  // Return empty frames if decoding has finished.
+  if (state_ == kDecodeFinished) {
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    return;
+  }
+
+  DecodeBuffer(buffer);
+}
+
+void VpuVideoDecoder::Reset(const base::Closure& closure) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK(reset_cb_.is_null());
+  reset_cb_ = BindToCurrentLoop(closure);
+
+  // Defer the reset if a decode is pending.
+  if (!decode_cb_.is_null())
+    return;
+
+  DoReset();
+}
+
+void VpuVideoDecoder::Stop(const base::Closure& closure) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  base::ScopedClosureRunner runner(BindToCurrentLoop(closure));
+
+  if (state_ == kUninitialized)
+    return;
+
+  if (!decode_cb_.is_null()) {
+    base::ResetAndReturn(&decode_cb_).Run(kOk, NULL);
+    // Reset is pending only when decode is pending.
+    if (!reset_cb_.is_null())
+      base::ResetAndReturn(&reset_cb_).Run();
+  }
+
+  state_ = kUninitialized;
+}
+
+bool VpuVideoDecoder::HasAlpha() const {
+  return false;
+}
+
+void VpuVideoDecoder::DecodeBuffer(const scoped_refptr<DecoderBuffer>& buffer) {
+  DCHECK(message_loop_->BelongsToCurrentThread());
+  DCHECK_NE(state_, kUninitialized);
+  DCHECK_NE(state_, kDecodeFinished);
+  DCHECK_NE(state_, kError);
+  DCHECK(reset_cb_.is_null());
+  DCHECK(!decode_cb_.is_null());
+  DCHECK(buffer);
+
+  // Transition to kDecodeFinished on the first end of stream buffer.
+  if (state_ == kNormal && buffer->end_of_stream()) {
+    state_ = kDecodeFinished;
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    return;
+  }
+
+  scoped_refptr<VideoFrame> video_frame;
+  if (!VpuDecode(buffer, &video_frame)) {
+    state_ = kError;
+    base::ResetAndReturn(&decode_cb_).Run(kDecodeError, NULL);
+    return;
+  }
+
+  // If we didn't get a frame we need more data.
+  if (!video_frame.get()) {
+    base::ResetAndReturn(&decode_cb_).Run(kNotEnoughData, NULL);
+    return;
+  }
+
+  base::ResetAndReturn(&decode_cb_).Run(kOk, video_frame);
+}
+
+bool VpuVideoDecoder::VpuDecode(const scoped_refptr<DecoderBuffer>& buffer,
+                                scoped_refptr<VideoFrame>* video_frame) {
+  DCHECK(video_frame);
+  DCHECK(!buffer->end_of_stream());
+
+  int buffer_ret_code;
+  VpuBufferNode in_data;
+  VpuDecRetCode ret;
+
+  memset(&in_data, 0, sizeof(in_data));
+  in_data.pVirAddr = (unsigned char *)(buffer->data());
+  in_data.nSize = buffer->data_size();
+
+  ret = VPU_DecDecodeBuf(internal->handle, &in_data, &buffer_ret_code);
+  if (ret != VPU_DEC_RET_SUCCESS) {
+    LOG(ERROR) << "Failed to decode frame: " << VpuStrerror(ret);
+    return false;
+  }
+
+  VLOG(1) << "VPU_DecDecodeBuf buffer ret code is 0x" << std::hex << buffer_ret_code << std::dec;
+
+  if (buffer_ret_code & VPU_DEC_INIT_OK) {
+    ret = VPU_DecGetInitialInfo(internal->handle, &(internal->init_info));
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get init info: " << VpuStrerror(ret);
+      return false;
+    }
+    internal->framebuffers.init(internal->handle, internal->init_info);
+  }
+
+  if (buffer_ret_code & VPU_DEC_FLUSH) {
+    ret = VPU_DecFlushAll(internal->handle);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Flushing VPU failed: " << VpuStrerror(ret);
+      return false;
+    }
+
+    *video_frame = NULL;
+
+    return true;
+  }
+
+  if (buffer_ret_code & VPU_DEC_NO_ENOUGH_INBUF) {
+    LOG(INFO) << "Need more input";
+    return true;
+  }
+
+  if (buffer_ret_code & VPU_DEC_ONE_FRM_CONSUMED) {
+    VpuDecFrameLengthInfo dec_framelen_info;
+
+    ret = VPU_DecGetConsumedFrameInfo(internal->handle, &dec_framelen_info);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get information about consumed frame: " << VpuStrerror(ret);
+    } else {
+      LOG(INFO) << "One frame got consumed:"
+        << "  framebuffer addr: 0x" << std::hex << uintptr_t(dec_framelen_info.pFrame) << std::dec
+        << "  stuff length: " << dec_framelen_info.nStuffLength
+        << "  frame length: " << dec_framelen_info.nFrameLength
+        ;
+    }
+  }
+
+  if (buffer_ret_code & VPU_DEC_OUTPUT_DIS) {
+    VpuDecOutFrameInfo out_frame_info;
+    int64 timestamp = buffer->timestamp().InMicroseconds();
+
+    ret = VPU_DecGetOutputFrame(internal->handle, &out_frame_info);
+    if (ret != VPU_DEC_RET_SUCCESS) {
+      LOG(ERROR) << "Could not get decoded output frame: " << VpuStrerror(ret);
+      return false;
+    }
+
+    LOG(INFO) << "Output frame:"
+      << "  pic width: " << internal->init_info.nPicWidth
+      << "  pic height: " << internal->init_info.nPicHeight
+      << "  pic type: " << out_frame_info.ePicType
+      << "  Y stride: " << out_frame_info.pDisplayFrameBuf->nStrideY
+      << "  CbCr stride: " << out_frame_info.pDisplayFrameBuf->nStrideC
+      ;
+
+    CopyVpuFramebufferTo(out_frame_info.pDisplayFrameBuf, video_frame, internal->init_info, config_);
+    (*video_frame)->SetTimestamp(base::TimeDelta::FromMicroseconds(timestamp));  
+
+    ret = VPU_DecOutFrameDisplayed(internal->handle, out_frame_info.pDisplayFrameBuf);
+    if (ret != VPU_DEC_RET_SUCCESS)
+      LOG(ERROR) << "Clearing display framebuffer failed: " << VpuStrerror(ret);
+  }
+  else
+    *video_frame = NULL;
+
+  return true;
+}
+
+void VpuVideoDecoder::DoReset() {
+  DCHECK(decode_cb_.is_null());
+
+  state_ = kNormal;
+  reset_cb_.Run();
+  reset_cb_.Reset();
+}
+
+void CopyVpuFramebufferTo(VpuFrameBuffer *framebuffer, scoped_refptr<VideoFrame>* video_frame, VpuDecInitInfo &init_info, const VideoDecoderConfig& config) {
+  CHECK(framebuffer);
+
+  gfx::Size size(init_info.nPicWidth, init_info.nPicHeight);
+
+  *video_frame = VideoFrame::CreateFrame(
+      config.format(),
+      size,
+      gfx::Rect(size),
+      config.natural_size(),
+      kNoTimestamp());
+
+  CopyYPlane(framebuffer->pbufVirtY,
+             framebuffer->nStrideY,
+             init_info.nPicHeight,
+             video_frame->get());
+  if (config.format() == VideoFrame::I420) {
+    CopyUPlane(framebuffer->pbufVirtCb,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+    CopyVPlane(framebuffer->pbufVirtCr,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+  } else { /* YV12 and YV12A */
+    /* U and V swapped */
+    CopyUPlane(framebuffer->pbufVirtCb,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+    CopyVPlane(framebuffer->pbufVirtCr,
+               framebuffer->nStrideC,
+               init_info.nPicHeight / 2,
+               video_frame->get());
+
+    if (config.format() == VideoFrame::YV12A) {
+      MakeOpaqueAPlane(framebuffer->nStrideY,
+                       init_info.nPicHeight,
+                       video_frame->get()); 
+    }
+  }
+}
+
+
+}  // namespace media
diff --git a/media/filters/vpu_video_decoder.h b/media/filters/vpu_video_decoder.h
new file mode 100644
index 0000000..d7016aa
--- /dev/null
+++ b/media/filters/vpu_video_decoder.h
@@ -0,0 +1,74 @@
+#ifndef MEDIA_FILTERS_VPU_VIDEO_DECODER_H_
+#define MEDIA_FILTERS_VPU_VIDEO_DECODER_H_
+
+#include "base/callback.h"
+#include "base/memory/weak_ptr.h"
+#include "media/base/demuxer_stream.h"
+#include "media/base/video_decoder.h"
+#include "media/base/video_decoder_config.h"
+#include "media/base/video_frame.h"
+
+namespace base {
+class MessageLoopProxy;
+}
+
+namespace media {
+
+class MEDIA_EXPORT VpuVideoDecoder : public VideoDecoder {
+ public:
+  explicit VpuVideoDecoder(
+      const scoped_refptr<base::MessageLoopProxy>& message_loop);
+  virtual ~VpuVideoDecoder();
+
+  // VideoDecoder implementation.
+  virtual void Initialize(const VideoDecoderConfig& config,
+                          const PipelineStatusCB& status_cb) OVERRIDE;
+  virtual void Decode(const scoped_refptr<DecoderBuffer>& buffer,
+                      const DecodeCB& decode_cb) OVERRIDE;
+  virtual void Reset(const base::Closure& closure) OVERRIDE;
+  virtual void Stop(const base::Closure& closure) OVERRIDE;
+  virtual bool HasAlpha() const OVERRIDE;
+
+ private:
+  enum DecoderState {
+    kUninitialized,
+    kNormal,
+    kFlushCodec,
+    kDecodeFinished,
+    kError
+  };
+
+  // Handles (re-)initializing the decoder with a (new) config.
+  // Returns true when initialization was successful.
+  bool ConfigureDecoder(const VideoDecoderConfig& config);
+
+  bool OpenDecoder(const VideoDecoderConfig& config);
+  void CloseDecoder();
+
+  void DecodeBuffer(const scoped_refptr<DecoderBuffer>& buffer);
+  bool VpuDecode(const scoped_refptr<DecoderBuffer>& buffer,
+                 scoped_refptr<VideoFrame>* video_frame);
+
+  // Reset decoder and call |reset_cb_|.
+  void DoReset();
+
+  scoped_refptr<base::MessageLoopProxy> message_loop_;
+  base::WeakPtrFactory<VpuVideoDecoder> weak_factory_;
+  base::WeakPtr<VpuVideoDecoder> weak_this_;
+
+  DecoderState state_;
+
+  DecodeCB decode_cb_;
+  base::Closure reset_cb_;
+
+  VideoDecoderConfig config_;
+
+  struct Internal;
+  Internal *internal;
+
+  DISALLOW_COPY_AND_ASSIGN(VpuVideoDecoder);
+};
+
+}  // namespace media
+
+#endif  // MEDIA_FILTERS_VPU_VIDEO_DECODER_H_
diff --git a/media/media.gyp b/media/media.gyp
index a25571d..b12d386 100644
--- a/media/media.gyp
+++ b/media/media.gyp
@@ -11,6 +11,7 @@
     # (DT_NEEDED) instead of using dlopen. This helps with automated
     # detection of ABI mismatches and prevents silent errors.
     'linux_link_pulseaudio%': 0,
+    'media_use_imxvpu%': 1,
     'conditions': [
       ['OS=="android"', {
         # Android doesn't use ffmpeg.
@@ -377,6 +378,10 @@
         'filters/video_frame_stream.h',
         'filters/video_renderer_base.cc',
         'filters/video_renderer_base.h',
+        'filters/vpu_framebuffers.cc',
+        'filters/vpu_framebuffers.h',
+        'filters/vpu_video_decoder.cc',
+        'filters/vpu_video_decoder.h',
         'filters/vpx_video_decoder.cc',
         'filters/vpx_video_decoder.h',
         'filters/wsola_internals.cc',
@@ -592,6 +597,29 @@
                 'DISABLE_USER_INPUT_MONITOR',
               ],
             }],
+            ['media_use_imxvpu==1', {
+              'cflags': [
+                '<!@(<(pkg-config) --cflags libfslvpuwrap)',
+              ],
+              'link_settings': {
+                'libraries': [
+                  '<!@(<(pkg-config) --libs libfslvpuwrap)',
+                ],
+              },
+            }, {  # media_use_imxvpu==0
+              'direct_dependent_settings': {
+                'defines': [
+                  'MEDIA_DISABLE_IMXVPU',
+                ],
+              },
+              # Exclude the VPU sources.
+              'sources!': [
+                'filters/vpu_framebuffers.cc',
+                'filters/vpu_framebuffers.h',
+                'filters/vpu_video_decoder.cc',
+                'filters/vpu_video_decoder.h',
+              ],
+            }],
             ['use_cras==1', {
               'cflags': [
                 '<!@(<(pkg-config) --cflags libcras)',

Re: Chromium acceleration

[Otavio Salvador]

On Thu, Mar 20, 2014 at 11:07 AM, Christian Betz
<christian.betz@gmail.com> wrote:
>> \m/ bear on mind I pushed Chromium 35 in meta-browser ;-)
>
>
> this is great news. with any luck we'll let you know how it builds and works
> for us sometime next week. thank you!
>
> i assume you got it built successfully and have it running in some fashion.
> In case we have trouble, can you share what your exact setup was? I'm just
> assuming you used a imx6qsabresd machine with code from fsl master-next or
> master-branches (i.e. latest GPU drivers).

Gary did.

> did you look at chrome://gpu and confirm various acceleration modes are
> enabled? i attached a screenshot from my **intel/nvidia** system for
> reference.
>
> I think lauren is curious if webgl is working for you with latest vivante
> drivers. And *I* am curious about 'accelerated compositing'.

I didn't build the new chromium for a while (working for a customer
with a way older version for a specific version).

> btw everyone: m35 is a very ambitious release where google has ripped out
> GTK and replaced it with 'Aura' (a combined widget and compositing toolkit
> apparently). i just wanted to mention that.

Yes; this also opens the door for Wayland support in future ;-)

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Chromium acceleration

[Otavio Salvador]

On Thu, Mar 20, 2014 at 11:29 AM, Eric Nelson
<eric.nelson@boundarydevices.com> wrote:
> Thanks Otavio,
>
>
> On 03/20/2014 05:22 AM, Otavio Salvador wrote:
>>
>> On Thu, Mar 20, 2014 at 5:30 AM, Carlos Rafael Giani
>> <dv@pseudoterminal.org> wrote:
>>>
>>> Hi Eric and Christian,
>>>
>>> <snip>
>
>>>
>>> As for the decoder itself: I implemented it in the Chromium media
>>> framework,
>>> in media/ . I simply took the vpx decoder code, copied it, and modified
>>> it
>>> to use the VPU. I had VP8, MPEG2, MPEG4, and h264 decoding working. It
>>> wasnt
>>> much code, but unfortunately, the interfaces tend to change between
>>> versions, so the code would have to be updated for the newest Chromium
>>> version. It is not much code, I will try to clean it up a bit and post
>>> it.
>>> In addition, I will ask the colleagues for the other patches for
>>> accelerating the 2D and WebGL rendering.
>>
>>
>> \m/ bear on mind I pushed Chromium 35 in meta-browser ;-)
>>
>
> I sent an RFC patch yesterday regarding inclusion of meta-qt5 into
> fsl-community-bsp-platform.
>
> Is there any interest or conflict in the same for meta-browser?
>
> A simplistic test shows that it's really straightforward to
> add both Chromium and Firefox into fsl-image-gui by pulling
> in meta-browser.

The biggest problem adding more stuff is maintenance. This increases
build time and tests needed for a good coverage.

What people think?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Chromium acceleration

[Eric Nelson]

Thanks Otavio,

On 03/20/2014 05:22 AM, Otavio Salvador wrote:
> On Thu, Mar 20, 2014 at 5:30 AM, Carlos Rafael Giani
> <dv@pseudoterminal.org> wrote:
>> Hi Eric and Christian,
>>
 >> <snip>
>>
>> As for the decoder itself: I implemented it in the Chromium media framework,
>> in media/ . I simply took the vpx decoder code, copied it, and modified it
>> to use the VPU. I had VP8, MPEG2, MPEG4, and h264 decoding working. It wasnt
>> much code, but unfortunately, the interfaces tend to change between
>> versions, so the code would have to be updated for the newest Chromium
>> version. It is not much code, I will try to clean it up a bit and post it.
>> In addition, I will ask the colleagues for the other patches for
>> accelerating the 2D and WebGL rendering.
>
> \m/ bear on mind I pushed Chromium 35 in meta-browser ;-)
>

I sent an RFC patch yesterday regarding inclusion of meta-qt5 into
fsl-community-bsp-platform.

Is there any interest or conflict in the same for meta-browser?

A simplistic test shows that it's really straightforward to
add both Chromium and Firefox into fsl-image-gui by pulling
in meta-browser.

Regards,


Eric

Re: Chromium acceleration

[Lauren Post]

Yes we are using that version for our chromium.


-----Original Message-----
From: Eric Nelson [mailto:eric.nelson@boundarydevices.com] 
Sent: Thursday, March 20, 2014 9:25 AM
To: Post Lauren-RAA013; meta-freescale@yoctoproject.org
Subject: Re: Chromium acceleration

Thanks Lauren,

On 03/20/2014 05:47 AM, Lauren Post wrote:
> My original statement about acceleration was for webGL in the browser.
>

Are you working in the context of OSSystems/meta-browser (i.e. Otavio's push of Chromium 35)?

Re: Chromium acceleration

[Eric Nelson]

Thanks Lauren,

On 03/20/2014 05:47 AM, Lauren Post wrote:
> My original statement about acceleration was for webGL in the browser.
>

Are you working in the context of OSSystems/meta-browser
(i.e. Otavio's push of Chromium 35)?

Re: Chromium acceleration

[Christian Betz]

[-- Attachment #1.1: Type: text/plain, Size: 955 bytes --]

>
> \m/ bear on mind I pushed Chromium 35 in meta-browser ;-)
>

this is great news. with any luck we'll let you know how it builds and
works for us sometime next week. thank you!

i assume you got it built successfully and have it running in some fashion.
In case we have trouble, can you share what your exact setup was? I'm just
assuming you used a imx6qsabresd machine with code from fsl master-next or
master-branches (i.e. latest GPU drivers).

did you look at chrome://gpu and confirm various acceleration modes are
enabled? i attached a screenshot from my **intel/nvidia** system for
reference.

I think lauren is curious if webgl is working for you with latest vivante
drivers. And *I* am curious about 'accelerated compositing'.

btw everyone: m35 is a very ambitious release where google has ripped out
GTK and replaced it with 'Aura' (a combined widget and compositing toolkit
apparently). i just wanted to mention that.

[-- Attachment #1.2: Type: text/html, Size: 1428 bytes --]

[-- Attachment #2: about_gpu_nvidia_x86_64.png --]
[-- Type: image/png, Size: 70457 bytes --]

Re: Chromium acceleration

[Christian Betz]

[-- Attachment #1: Type: text/plain, Size: 1507 bytes --]

>
> As for the decoder itself: I implemented it in the Chromium media
> framework, in media/ . I simply took the vpx decoder code, copied it, and
> modified it to use the VPU. I had VP8, MPEG2, MPEG4, and h264 decoding
> working. It wasnt much code, but unfortunately, the interfaces tend to
> change between versions, so the code would have to be updated for the
> newest Chromium version. It is not much code, I will try to clean it up a
> bit and post it. In addition, I will ask the colleagues for the other
> patches for accelerating the 2D and WebGL rendering.


thanks carlos!

my analysis suggested that my team should reproduce these chunks of code
which are for samsung exynos:

https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/exynos_video_decode_accelerator.cc

to oversimplify greatly: we want to port 'exynos_video_decode_accelerator'
to a new 'imx_video_decode_accelerator'. in this module we would make use
of the same direct texture techniques as eglvivsink from gstreamer-imx.

this is definitely not the route you have taken, from what I can tell. is
this something you considered and steered away from? have you examined
these code paths at all? there are a few other special modules besides
exynos too.

i might be wrong, but it could be the best chance of getting something
upstreamed would be do it "like everyone else" (i.e. samsung). but i'm
probably thinking *way* too far ahead here.

[-- Attachment #2: Type: text/html, Size: 2088 bytes --]

Re: Chromium acceleration

[Lauren Post]

My original statement about acceleration was for webGL in the browser.

ffmpeg-mt is not hardware accelerated with the VPU although it is possible to update it to integrate with VPU.   As other emails have mentioned removing data movement and the CSC conversion is the key to improving performance with ffmpeg with vpu acceleration.  It is not something we have in our upcoming release.

Lauren

-----Original Message-----
From: Eric Nelson [mailto:eric.nelson@boundarydevices.com] 
Sent: Wednesday, March 19, 2014 3:50 PM
To: Post Lauren-RAA013; meta-freescale@yoctoproject.org
Subject: Chromium acceleration

Hi Lauren,

A while back, you mentioned that 3.10.17-ga would help provide support for "Chromium acceleration" but didn't provide details.

Did you mean WebGL support (i.e. GPU acceleration), or video acceleration (VPU support)?

My understanding is that Chromium is using ffmpeg-mt as the basis for accelerated video:
	http://www.chromium.org/developers/design-documents/video

Is anyone currently working on support for ffmpeg-mt and i.MX video?

Is anyone interested?

Please advise,


Eric

Re: Chromium acceleration

[Otavio Salvador]

On Thu, Mar 20, 2014 at 5:30 AM, Carlos Rafael Giani
<dv@pseudoterminal.org> wrote:
> Hi Eric and Christian,
>
> I wanted to avoid _any_ copy involving the CPU, including any texture
> upload. The Vivante GLES direct texture extension can color-convert, yes,
> but it can also read pixels directly from a DMA buffer, without any copy.
> This is what I use for the GLES-based sink in my GStreamer plugins, and
> allows me to play 1080p videos with less than 6% CPU usage.
> v4lsink would be another idea, but I am a bit skeptical, because it wouldn't
> play well with any X11 or Wayland compositor, since it would essentially act
> like an overlay. This is even worse with the isink (if the regular,
> non-overlay layer is used), because then, the sink simply overwrites pixels
> drawn by X11. What the isink does is to make sure the pixels are blit in the
> area of its output window, so it does not overwrite anything outside. But
> again, with desktop compositing this would break down. (Also, the isink is
> quite slow.)
>
> I would consider this one texture upload only if absolutely necessary.
>
> My idea was to actually produce the texture in the decoder myself, and just
> send the texture GL name around to the renderer. This would require that
> everything is drawn with GLES of course - what would a software renderer do
> with a GL name?
>
> As for the decoder itself: I implemented it in the Chromium media framework,
> in media/ . I simply took the vpx decoder code, copied it, and modified it
> to use the VPU. I had VP8, MPEG2, MPEG4, and h264 decoding working. It wasnt
> much code, but unfortunately, the interfaces tend to change between
> versions, so the code would have to be updated for the newest Chromium
> version. It is not much code, I will try to clean it up a bit and post it.
> In addition, I will ask the colleagues for the other patches for
> accelerating the 2D and WebGL rendering.

\m/ bear on mind I pushed Chromium 35 in meta-browser ;-)

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Chromium acceleration

[Carlos Rafael Giani]

Hi Eric and Christian,

I wanted to avoid _any_ copy involving the CPU, including any texture 
upload. The Vivante GLES direct texture extension can color-convert, 
yes, but it can also read pixels directly from a DMA buffer, without any 
copy. This is what I use for the GLES-based sink in my GStreamer 
plugins, and allows me to play 1080p videos with less than 6% CPU usage.
v4lsink would be another idea, but I am a bit skeptical, because it 
wouldn't play well with any X11 or Wayland compositor, since it would 
essentially act like an overlay. This is even worse with the isink (if 
the regular, non-overlay layer is used), because then, the sink simply 
overwrites pixels drawn by X11. What the isink does is to make sure the 
pixels are blit in the area of its output window, so it does not 
overwrite anything outside. But again, with desktop compositing this 
would break down. (Also, the isink is quite slow.)

I would consider this one texture upload only if absolutely necessary.

My idea was to actually produce the texture in the decoder myself, and 
just send the texture GL name around to the renderer. This would require 
that everything is drawn with GLES of course - what would a software 
renderer do with a GL name?

As for the decoder itself: I implemented it in the Chromium media 
framework, in media/ . I simply took the vpx decoder code, copied it, 
and modified it to use the VPU. I had VP8, MPEG2, MPEG4, and h264 
decoding working. It wasnt much code, but unfortunately, the interfaces 
tend to change between versions, so the code would have to be updated 
for the newest Chromium version. It is not much code, I will try to 
clean it up a bit and post it. In addition, I will ask the colleagues 
for the other patches for accelerating the 2D and WebGL rendering.

Carlos


On 2014-03-20 03:50, Eric Nelson wrote:
> Thanks Christian,
>
> On 03/19/2014 06:29 PM, Christian Betz wrote:
>>     The Vivante libraries can map the physical buffers produced by
>>     the VPU directly, so they could do format conversion on their
>>     way to the graphics stack if the Chromium bindings have access
>>     to that (the "single copy" I referred to above).
>>
>>
>> in my experience "zero-copy" really means zero memcpy() (or similar
>> system call, or really anything that involves the cpu to move around
>> image data. /metadata /such as buffer pointers is generally allowed). in
>> other words having the GPU do format conversion isn't considered a copy.
>>
>> here's a great article on this general technique implemented in
>> webkit/gtk+:
>> http://blogs.igalia.com/vjaquez/2013/07/26/composited-video-support-in-webkitgtk/ 
>>
>>
>
> That's a nice article, and if Webkit (Blink) retains the same mapping,
> video buffers from the VPU would end up hitting a RenderObject and
> the only "copy" would be in rendering that to the Framebuffer.
>
> Since we're on the fat end of the pipe here, that copy can be expensive
> and might limit things to 1x1080P decode, but I'd be happy with that.
>
> In the mfw_v4lsink case at least, this can be bypassed by having
> the V4L2 buffer filled by the VPU and also displayed by the IPU
> (such that copy), but aiming there would keep the video out of the
> rest of the RenderLayer tree (I think).
>
> This is what I was so awkwardly trying to suggest.
>
>> moving beyond semantics... let me tell you what i know about this
>> particular issue:
>>
>> there is source code in chromium to accomplish a similar goal.
>> specifically there is a "generic" android driver, which works out of the
>> box with i.mx6 because freescale implements the standard android video
>> decode interface that allows VPU -> GPU texture uploading (don't quote
>> me on this, i'm just reading the source code:
>> https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/android_video_decode_accelerator.cc) 
>>
>>
>> sadly there is no such standard interface on GNU/Linux. outside of
>> android (i.e. chromeos), there are drivers for particular chipsets, like
>> samsung exynos. here is the source for that:
>> https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/exynos_video_decode_accelerator.cc 
>>
>>
>> the group should also be aware of the current situation regarding these
>> modules /only/ working when certain flags to enable "chromeos" are used
>> (explained here:
>> https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode). in other
>> words: this stuff doesn't work on your x86 with the chrome you download
>> from google.
>>
>
> Again, thanks for the info.
>
> Regards,
>
>
> Eric

Re: Chromium acceleration

[Eric Nelson]

Thanks Christian,

On 03/19/2014 06:29 PM, Christian Betz wrote:
>     The Vivante libraries can map the physical buffers produced by
>     the VPU directly, so they could do format conversion on their
>     way to the graphics stack if the Chromium bindings have access
>     to that (the "single copy" I referred to above).
>
>
> in my experience "zero-copy" really means zero memcpy() (or similar
> system call, or really anything that involves the cpu to move around
> image data. /metadata /such as buffer pointers is generally allowed). in
> other words having the GPU do format conversion isn't considered a copy.
>
> here's a great article on this general technique implemented in
> webkit/gtk+:
> http://blogs.igalia.com/vjaquez/2013/07/26/composited-video-support-in-webkitgtk/
>

That's a nice article, and if Webkit (Blink) retains the same mapping,
video buffers from the VPU would end up hitting a RenderObject and
the only "copy" would be in rendering that to the Framebuffer.

Since we're on the fat end of the pipe here, that copy can be expensive
and might limit things to 1x1080P decode, but I'd be happy with that.

In the mfw_v4lsink case at least, this can be bypassed by having
the V4L2 buffer filled by the VPU and also displayed by the IPU
(such that copy), but aiming there would keep the video out of the
rest of the RenderLayer tree (I think).

This is what I was so awkwardly trying to suggest.

> moving beyond semantics... let me tell you what i know about this
> particular issue:
>
> there is source code in chromium to accomplish a similar goal.
> specifically there is a "generic" android driver, which works out of the
> box with i.mx6 because freescale implements the standard android video
> decode interface that allows VPU -> GPU texture uploading (don't quote
> me on this, i'm just reading the source code:
> https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/android_video_decode_accelerator.cc)
>
> sadly there is no such standard interface on GNU/Linux. outside of
> android (i.e. chromeos), there are drivers for particular chipsets, like
> samsung exynos. here is the source for that:
> https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/exynos_video_decode_accelerator.cc
>
> the group should also be aware of the current situation regarding these
> modules /only/ working when certain flags to enable "chromeos" are used
> (explained here:
> https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode). in other
> words: this stuff doesn't work on your x86 with the chrome you download
> from google.
>

Again, thanks for the info.

Regards,


Eric

Re: Chromium acceleration

[Christian Betz]

[-- Attachment #1: Type: text/plain, Size: 2199 bytes --]

>
> The Vivante libraries can map the physical buffers produced by
> the VPU directly, so they could do format conversion on their
> way to the graphics stack if the Chromium bindings have access
> to that (the "single copy" I referred to above).
>

in my experience "zero-copy" really means zero memcpy() (or similar system
call, or really anything that involves the cpu to move around image data.
*metadata *such as buffer pointers is generally allowed). in other words
having the GPU do format conversion isn't considered a copy.

here's a great article on this general technique implemented in
webkit/gtk+:
http://blogs.igalia.com/vjaquez/2013/07/26/composited-video-support-in-webkitgtk/

moving beyond semantics... let me tell you what i know about this
particular issue:

there is source code in chromium to accomplish a similar goal. specifically
there is a "generic" android driver, which works out of the box with i.mx6
because freescale implements the standard android video decode interface
that allows VPU -> GPU texture uploading (don't quote me on this, i'm just
reading the source code:
https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/android_video_decode_accelerator.cc
)

sadly there is no such standard interface on GNU/Linux. outside of android
(i.e. chromeos), there are drivers for particular chipsets, like samsung
exynos. here is the source for that:
https://chromium.googlesource.com/chromium/src/+/a361fce28da709ea872062f30fb4b65fcc37b695/content/common/gpu/media/exynos_video_decode_accelerator.cc

the group should also be aware of the current situation regarding these
modules *only* working when certain flags to enable "chromeos" are used
(explained here: https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode).
in other words: this stuff doesn't work on your x86 with the chrome you
download from google.

@carlos: have you implemented a module along the lines of this exynos
accelerator or did you take another route? a team member is actually about
to attack this problem next week so it'd be great to see what you have
before we reinvent your wheel.

[-- Attachment #2: Type: text/html, Size: 3243 bytes --]

Re: Chromium acceleration

[Eric Nelson]

Hi Carlos,

On 03/19/2014 02:00 PM, Carlos Rafael Giani wrote:
> Hello,
>
> I wrote a patch for VPU acceleration in Chromium months ago for a
> company project. I got the OK from the customer to cleanup and
> eventually opensource the patches, but didn't have the time yet.

You're the man!

What can we do to help find you the time?

> Together with other colleagues we also got HW-accelerated WebGL to work,
> though some other areas weren't running so well (multitouch for
> example). Also, Canvas remained unaccelerated. As the Chromium
> developers explained, there are conceptual problems with the Canvas API
> that make acceleration difficult.
>

Nice! I have to admit being a bit less interested in WebGL than in
video acceleration, but I'm sure others really want this.

 >
> What is also missing is a zerocopy method for displaying video frames.
 >
Do we really want zero copy? It seems that at some level, having
a single copy into a GPU accelerated rendering stack is more
convenient.

> In theory, the direct textures from the Vivante GPU could be used;
> however, this requires passing the physical buffers through to a
> modified renderer somehow.
 >
 > Chromium can use OpenGL ES for rendering everything, but does not
 > contain anything to introduce special video textures yet.
>

I'm not quite sure I'm grokking this.

The Vivante libraries can map the physical buffers produced by
the VPU directly, so they could do format conversion on their
way to the graphics stack if the Chromium bindings have access
to that (the "single copy" I referred to above).

Or, (he says instead of finding the code), is this handled by
ffmpeg-mt?

Using an IPU overlay (a.la. mfw_v4lsink) is nice, but makes
doing overlays and such more difficult, and you probably know
how web-developers can be ;).

Regards,


Eric

Re: Chromium acceleration

[Carlos Rafael Giani]

Hello,

I wrote a patch for VPU acceleration in Chromium months ago for a 
company project. I got the OK from the customer to cleanup and 
eventually opensource the patches, but didn't have the time yet. 
Together with other colleagues we also got HW-accelerated WebGL to work, 
though some other areas weren't running so well (multitouch for 
example). Also, Canvas remained unaccelerated. As the Chromium 
developers explained, there are conceptual problems with the Canvas API 
that make acceleration difficult.

What is also missing is a zerocopy method for displaying video frames. 
In theory, the direct textures from the Vivante GPU could be used; 
however, this requires passing the physical buffers through to a 
modified renderer somehow. Chromium can use OpenGL ES for rendering 
everything, but does not contain anything to introduce special video 
textures yet.

Carlos

On 03/19/2014 09:49 PM, Eric Nelson wrote:
> Hi Lauren,
>
> A while back, you mentioned that 3.10.17-ga would help provide
> support for "Chromium acceleration" but didn't provide details.
>
> Did you mean WebGL support (i.e. GPU acceleration), or video
> acceleration (VPU support)?
>
> My understanding is that Chromium is using ffmpeg-mt as the
> basis for accelerated video:
>     http://www.chromium.org/developers/design-documents/video
>
> Is anyone currently working on support for ffmpeg-mt and i.MX
> video?
>
> Is anyone interested?
>
> Please advise,
>
>
> Eric

Chromium acceleration

[Eric Nelson]

Hi Lauren,

A while back, you mentioned that 3.10.17-ga would help provide
support for "Chromium acceleration" but didn't provide details.

Did you mean WebGL support (i.e. GPU acceleration), or video
acceleration (VPU support)?

My understanding is that Chromium is using ffmpeg-mt as the
basis for accelerated video:
	http://www.chromium.org/developers/design-documents/video

Is anyone currently working on support for ffmpeg-mt and i.MX
video?

Is anyone interested?

Please advise,


Eric