rockchip-vpu driver usability

rockchip-vpu driver usability

[Baruch Siach]

Hi Tomasz,

I'm trying to get the RK3288 VPU do h.264 encoding on the Tinkerboard platform 
with latest mainline kernel. I found the v4l2 based rockchip-vpu driver in the 
vendor kernel. But I could not get it work even in the context of that kernel. 
Is this driver in usable state? If not, how far it is from being usable?

Another option is to forward port the rockchip-vcodec shim driver for the MPP 
framework. I'm aware of Myy Miouyouyou's work[1] in this area. But this driver 
seems unusable as well. Testing v4.17-rc6, the PM code fails to enable the 
video power domain (the hvec power domain enables correctly, though). The 
error is:

  rockchip-pm-domain ff730000.power-management:power-controller: failed to get ack on domain 'pd_video', val=0x8021c

With the TinkerOS provided kernel I managed to make the VPU encode to h.264 
using the gstreamer MPP plugin. But I would rather not use that kernel if 
possible, since that would mean backporting other v4l2 code that I need.

Do you have any other idea/direction for me to explore?

Thanks,
baruch

[1] https://github.com/Miouyouyou/rockchip-vcodec

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.52.368.4656, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Tomasz Figa]

Hi Baruch,

On Thu, May 24, 2018 at 7:34 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:

> Hi Tomasz,

> I'm trying to get the RK3288 VPU do h.264 encoding on the Tinkerboard
platform
> with latest mainline kernel. I found the v4l2 based rockchip-vpu driver
in the
> vendor kernel. But I could not get it work even in the context of that
kernel.
> Is this driver in usable state? If not, how far it is from being usable?

The driver in chromeos-3.14 kernel [1] is running in production on RK3288
systems, while the one in chromeos-4.4 [2] on RK3399 systems.

[1]
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.14/drivers/media/platform/rk3288-vpu/
[2]
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-4.4/drivers/media/platform/rockchip-vpu/

However, it doesn't implement a regular stateful V4L2 encoder API, since
the hardware is stateless and does only part of the encoding. For other
things, like bitstream generation and bitrate control, we have libv4l-based
plugins for rk3288 [3] and rk3399 [4].

[3]
https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip/
[4]
https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip_v2/

Do you have any details on what was going wrong when you tried to "get it
work even in the context of that kernel"?


> Another option is to forward port the rockchip-vcodec shim driver for the
MPP
> framework. I'm aware of Myy Miouyouyou's work[1] in this area. But this
driver
> seems unusable as well. Testing v4.17-rc6, the PM code fails to enable the
> video power domain (the hvec power domain enables correctly, though). The
> error is:

>    rockchip-pm-domain ff730000.power-management:power-controller: failed
to get ack on domain 'pd_video', val=0x8021c

Hmm, this looks like some problem with power domains. I don't think this is
related to the codec driver itself. Could be something wrong with power
domain driver or some data in device tree, such as power domain clocks.


> With the TinkerOS provided kernel I managed to make the VPU encode to
h.264
> using the gstreamer MPP plugin. But I would rather not use that kernel if
> possible, since that would mean backporting other v4l2 code that I need.

> Do you have any other idea/direction for me to explore?

I think you should be able to get our driver to work, but ideally we should
create a proper stateless encoder API, based on V4L2 Request API, similar
to what's being proposed for decoders.

Best regards,
Tomasz


> Thanks,
> baruch

> [1] https://github.com/Miouyouyou/rockchip-vcodec

> --
>       http://baruch.siach.name/blog/                  ~. .~   Tk Open
Systems

=}------------------------------------------------ooO--U--Ooo------------{=
>     - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.52.368.4656, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Myy]

Le 24/05/2018 à 12:34, Baruch Siach a écrit :
> Another option is to forward port the rockchip-vcodec shim driver for the MPP
> framework. I'm aware of Myy Miouyouyou's work[1] in this area. But this driver
> seems unusable as well. Testing v4.17-rc6, the PM code fails to enable the
> video power domain (the hvec power domain enables correctly, though). The
> error is:
>
>    rockchip-pm-domain ff730000.power-management:power-controller: failed to get ack on domain 'pd_video', val=0x8021c
I currently didn't test this driver on that version. Though, the latest 
"non-crashing" branch was the branch "retry" .

Anyway, it took me a lot of (too much) time to understand how the driver 
worked at the first place (What MMIO address it is writing, which data 
comes where, ...).
The pages of TRM providing the useful registers description being 
removed to avoid infringing some NDA with the VPU chipmaker (at least it 
was the explanation I got), I didn't make it far by analysing the VPU 
code, which is closely related to MPP.

IIRC (I'm at work, cannot check the code right now)
The way that driver worked, in the 4.4 kernel, was :
* MPP sets up a big data structure corresponding to the VPU registers 
and sends it to the VPU driver, though an ioctl.
* The VPU driver receives the data structure and interprets it as a 
uint32_t table, corresponding to the registers and do some magic things 
with some indices, setups the IOMMU so that the VPU can write the 
resulting frame, through a DMA operation, to some memory buffer provided 
by the user, and starts the decoding process.
* MPP sends an ioctl to the VPU in order to know the ETA of the VPU job.
* The VPU sends back the ETA to MPP in return to that ioctl call.

Now, that driver has a lot of weird IOMMU code that should be there. For 
example, it does DMA SG tables remapping by hand, taking into account 
IOMMU alignment restrictions... That code is now deprecated in the new 
kernels which should use the proper IOMMU helpers put in place.

Also, if you look at the rockchip kernel, there's another V4L2 driver 
written by part of the Chromium and Rockchip team, which describes 
everything in a clear fashion so now I'm aiming to retest the whole 
thing, based on these information.
If that works, I'll try to mimic that V4L2 driver, in a way that works 
with mainline kernels (which moved the whole video formats handling into 
the user-space library it seems).
Here's the file describing the registers 
https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/platform/rockchip-vpu/rk3288_vpu_regs.h 
and the whole new code is here : 
https://github.com/rockchip-linux/kernel/tree/release-4.4/drivers/media/platform/rockchip-vpu

-- 
Myy

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: rockchip-vpu driver usability

[Baruch Siach]

Hi Tomasz,

On Thu, May 24, 2018 at 08:46:35PM +0900, Tomasz Figa wrote:
> On Thu, May 24, 2018 at 7:34 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > I'm trying to get the RK3288 VPU do h.264 encoding on the Tinkerboard
> > platform with latest mainline kernel. I found the v4l2 based rockchip-vpu 
> > driver in the vendor kernel. But I could not get it work even in the 
> > context of that kernel. Is this driver in usable state? If not, how far it 
> > is from being usable?
> 
> The driver in chromeos-3.14 kernel [1] is running in production on RK3288
> systems, while the one in chromeos-4.4 [2] on RK3399 systems.
> 
> [1]
> https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.14/drivers/media/platform/rk3288-vpu/
> [2]
> https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-4.4/drivers/media/platform/rockchip-vpu/
> 
> However, it doesn't implement a regular stateful V4L2 encoder API, since
> the hardware is stateless and does only part of the encoding. For other
> things, like bitstream generation and bitrate control, we have libv4l-based
> plugins for rk3288 [3] and rk3399 [4].
> 
> [3]
> https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip/
> [4]
> https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip_v2/

This is interesting, thanks.

It looks like only v2 supports h.264 encoding. Does the v2 plugin work with 
the rk3288?

A recent discussion on the linux-media list indicated that gstreamer and 
libv4l2 are not a good match. Which application would you recommend for doing 
h.264 encoding?

> Do you have any details on what was going wrong when you tried to "get it
> work even in the context of that kernel"?

I used the code from the 'develop' branch of the TinkerOS kernel source 
tree[1]. I had to build the latest gstreamer version 1.14.0 to get the 
v4l2h264enc plugin. On startup I got a lot of "must be set encoding output 
size" error messages from try_fmt. Shortly after that I got a NULL dereference 
BUG message.

But since you say that the driver does not implement the regular API, I can't 
expect gstreamer native V4L2 code to work, right?

[1] https://github.com/TinkerBoard/debian_kernel

> > Another option is to forward port the rockchip-vcodec shim driver for the
> > MPP
> > framework. I'm aware of Myy Miouyouyou's work[1] in this area. But this
> > driver
> > seems unusable as well. Testing v4.17-rc6, the PM code fails to enable the
> > video power domain (the hvec power domain enables correctly, though). The
> > error is:
> 
> >    rockchip-pm-domain ff730000.power-management:power-controller: failed
> to get ack on domain 'pd_video', val=0x8021c
> 
> Hmm, this looks like some problem with power domains. I don't think this is
> related to the codec driver itself. Could be something wrong with power
> domain driver or some data in device tree, such as power domain clocks.
> 
> 
> > With the TinkerOS provided kernel I managed to make the VPU encode to
> h.264
> > using the gstreamer MPP plugin. But I would rather not use that kernel if
> > possible, since that would mean backporting other v4l2 code that I need.
> 
> > Do you have any other idea/direction for me to explore?
> 
> I think you should be able to get our driver to work, but ideally we should
> create a proper stateless encoder API, based on V4L2 Request API, similar
> to what's being proposed for decoders.

Thanks for your detailed and helpful response.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.52.368.4656, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Tomasz Figa]

On Thu, May 24, 2018 at 9:18 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:

> Hi Tomasz,

> On Thu, May 24, 2018 at 08:46:35PM +0900, Tomasz Figa wrote:
> > On Thu, May 24, 2018 at 7:34 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > I'm trying to get the RK3288 VPU do h.264 encoding on the Tinkerboard
> > > platform with latest mainline kernel. I found the v4l2 based
rockchip-vpu
> > > driver in the vendor kernel. But I could not get it work even in the
> > > context of that kernel. Is this driver in usable state? If not, how
far it
> > > is from being usable?
> >
> > The driver in chromeos-3.14 kernel [1] is running in production on
RK3288
> > systems, while the one in chromeos-4.4 [2] on RK3399 systems.
> >
> > [1]
> >
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.14/drivers/media/platform/rk3288-vpu/
> > [2]
> >
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-4.4/drivers/media/platform/rockchip-vpu/
> >
> > However, it doesn't implement a regular stateful V4L2 encoder API, since
> > the hardware is stateless and does only part of the encoding. For other
> > things, like bitstream generation and bitrate control, we have
libv4l-based
> > plugins for rk3288 [3] and rk3399 [4].
> >
> > [3]
> >
https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip/
> > [4]
> >
https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip_v2/

> This is interesting, thanks.

> It looks like only v2 supports h.264 encoding. Does the v2 plugin work
with
> the rk3288?

Hmm, looks like I missed the part about h264 in your original message (only
noticed encoding). We didn't have a need to encode h264 on rk3288 devices.
The need showed up only later when we were working on rk3399. This made
both the driver and plugin for rk3288 lack h264 support.

AFAICT the hardware programming interface differs between rk3288 and
rk3399, so v2 might not work, but maybe you just need to reshuffle some
registers.

It's also possible that Rockchip has some code for it. Jeffy, do you know
if there is h264 driver code and plugin for rk3288 available somewhere
(based on the driver developed for Chrome OS 3.14 kernel)?


> A recent discussion on the linux-media list indicated that gstreamer and
> libv4l2 are not a good match. Which application would you recommend for
doing
> h.264 encoding?

> > Do you have any details on what was going wrong when you tried to "get
it
> > work even in the context of that kernel"?

> I used the code from the 'develop' branch of the TinkerOS kernel source
> tree[1]. I had to build the latest gstreamer version 1.14.0 to get the
> v4l2h264enc plugin. On startup I got a lot of "must be set encoding output
> size" error messages from try_fmt. Shortly after that I got a NULL
dereference
> BUG message.

> But since you say that the driver does not implement the regular API, I
can't
> expect gstreamer native V4L2 code to work, right?

Nope. GStreamer would be expecting the normal stateful encoder API.


> [1] https://github.com/TinkerBoard/debian_kernel

> > > Another option is to forward port the rockchip-vcodec shim driver for
the
> > > MPP
> > > framework. I'm aware of Myy Miouyouyou's work[1] in this area. But
this
> > > driver
> > > seems unusable as well. Testing v4.17-rc6, the PM code fails to
enable the
> > > video power domain (the hvec power domain enables correctly, though).
The
> > > error is:
> >
> > >    rockchip-pm-domain ff730000.power-management:power-controller:
failed
> > to get ack on domain 'pd_video', val=0x8021c
> >
> > Hmm, this looks like some problem with power domains. I don't think
this is
> > related to the codec driver itself. Could be something wrong with power
> > domain driver or some data in device tree, such as power domain clocks.
> >
> >
> > > With the TinkerOS provided kernel I managed to make the VPU encode to
> > h.264
> > > using the gstreamer MPP plugin. But I would rather not use that
kernel if
> > > possible, since that would mean backporting other v4l2 code that I
need.
> >
> > > Do you have any other idea/direction for me to explore?
> >
> > I think you should be able to get our driver to work, but ideally we
should
> > create a proper stateless encoder API, based on V4L2 Request API,
similar
> > to what's being proposed for decoders.

> Thanks for your detailed and helpful response.

Well, not so helpful in the end. I feel like we could have done better with
pushing this driver upstream. Hopefully the Request API finalizes soon and
we can try to push this forward.

Best regards,
Tomasz

Re: rockchip-vpu driver usability

[JeffyChen]

Hi,

On 05/25/2018 02:10 PM, Tomasz Figa wrote:
> On Thu, May 24, 2018 at 9:18 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
>
>> Hi Tomasz,
>
>> On Thu, May 24, 2018 at 08:46:35PM +0900, Tomasz Figa wrote:
>>> On Thu, May 24, 2018 at 7:34 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
>>>> I'm trying to get the RK3288 VPU do h.264 encoding on the Tinkerboard
>>>> platform with latest mainline kernel. I found the v4l2 based
> rockchip-vpu
>>>> driver in the vendor kernel. But I could not get it work even in the
>>>> context of that kernel. Is this driver in usable state? If not, how
> far it
>>>> is from being usable?
>>>
>>> The driver in chromeos-3.14 kernel [1] is running in production on
> RK3288
>>> systems, while the one in chromeos-4.4 [2] on RK3399 systems.
>>>
>>> [1]
>>>
> https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.14/drivers/media/platform/rk3288-vpu/
>>> [2]
>>>
> https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-4.4/drivers/media/platform/rockchip-vpu/
>>>
>>> However, it doesn't implement a regular stateful V4L2 encoder API, since
>>> the hardware is stateless and does only part of the encoding. For other
>>> things, like bitstream generation and bitrate control, we have
> libv4l-based
>>> plugins for rk3288 [3] and rk3399 [4].
>>>
>>> [3]
>>>
> https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip/
>>> [4]
>>>
> https://chromium.googlesource.com/chromiumos/third_party/libv4lplugins/+/master/libv4l-rockchip_v2/
>
>> This is interesting, thanks.
>
>> It looks like only v2 supports h.264 encoding. Does the v2 plugin work
> with
>> the rk3288?
>
> Hmm, looks like I missed the part about h264 in your original message (only
> noticed encoding). We didn't have a need to encode h264 on rk3288 devices.
> The need showed up only later when we were working on rk3399. This made
> both the driver and plugin for rk3288 lack h264 support.
>
> AFAICT the hardware programming interface differs between rk3288 and
> rk3399, so v2 might not work, but maybe you just need to reshuffle some
> registers.
>
> It's also possible that Rockchip has some code for it. Jeffy, do you know
> if there is h264 driver code and plugin for rk3288 available somewhere
> (based on the driver developed for Chrome OS 3.14 kernel)?
>

well, i do saw a h264e driver for rk3288 in cros kernel:
chromiumos/src/third_party/kernel/local/drivers/media/platform/rockchip-vpu# 
ls rk3288_vpu_hw_h264e.c

and the related struct defined in v2 plugin:
libv4l-rockchip_v2/libvepu/common/rk_venc.h:120:struct 
rk3288_h264e_reg_params {


>
>> A recent discussion on the linux-media list indicated that gstreamer and
>> libv4l2 are not a good match. Which application would you recommend for
> doing
>> h.264 encoding?
>
>>> Do you have any details on what was going wrong when you tried to "get
> it
>>> work even in the context of that kernel"?
>
>> I used the code from the 'develop' branch of the TinkerOS kernel source
>> tree[1]. I had to build the latest gstreamer version 1.14.0 to get the
>> v4l2h264enc plugin. On startup I got a lot of "must be set encoding output
>> size" error messages from try_fmt. Shortly after that I got a NULL
> dereference
>> BUG message.
>
>> But since you say that the driver does not implement the regular API, I
> can't
>> expect gstreamer native V4L2 code to work, right?
>
> Nope. GStreamer would be expecting the normal stateful encoder API.
>
>
>> [1] https://github.com/TinkerBoard/debian_kernel
>
>>>> Another option is to forward port the rockchip-vcodec shim driver for
> the
>>>> MPP
>>>> framework. I'm aware of Myy Miouyouyou's work[1] in this area. But
> this
>>>> driver
>>>> seems unusable as well. Testing v4.17-rc6, the PM code fails to
> enable the
>>>> video power domain (the hvec power domain enables correctly, though).
> The
>>>> error is:
>>>
>>>>     rockchip-pm-domain ff730000.power-management:power-controller:
> failed
>>> to get ack on domain 'pd_video', val=0x8021c
>>>
>>> Hmm, this looks like some problem with power domains. I don't think
> this is
>>> related to the codec driver itself. Could be something wrong with power
>>> domain driver or some data in device tree, such as power domain clocks.
>>>
>>>
>>>> With the TinkerOS provided kernel I managed to make the VPU encode to
>>> h.264
>>>> using the gstreamer MPP plugin. But I would rather not use that
> kernel if
>>>> possible, since that would mean backporting other v4l2 code that I
> need.
>>>
>>>> Do you have any other idea/direction for me to explore?
>>>
>>> I think you should be able to get our driver to work, but ideally we
> should
>>> create a proper stateless encoder API, based on V4L2 Request API,
> similar
>>> to what's being proposed for decoders.
>
>> Thanks for your detailed and helpful response.
>
> Well, not so helpful in the end. I feel like we could have done better with
> pushing this driver upstream. Hopefully the Request API finalizes soon and
> we can try to push this forward.
>
> Best regards,
> Tomasz
>
>
>

Re: rockchip-vpu driver usability

[Baruch Siach]

Hi Jeffy,

On Fri, May 25, 2018 at 05:52:57PM +0800, JeffyChen wrote:
> On 05/25/2018 02:10 PM, Tomasz Figa wrote:
> > On Thu, May 24, 2018 at 9:18 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > It looks like only v2 supports h.264 encoding. Does the v2 plugin work
> > > with the rk3288?
> > 
> > Hmm, looks like I missed the part about h264 in your original message (only
> > noticed encoding). We didn't have a need to encode h264 on rk3288 devices.
> > The need showed up only later when we were working on rk3399. This made
> > both the driver and plugin for rk3288 lack h264 support.
> > 
> > AFAICT the hardware programming interface differs between rk3288 and
> > rk3399, so v2 might not work, but maybe you just need to reshuffle some
> > registers.
> > 
> > It's also possible that Rockchip has some code for it. Jeffy, do you know
> > if there is h264 driver code and plugin for rk3288 available somewhere
> > (based on the driver developed for Chrome OS 3.14 kernel)?
> 
> well, i do saw a h264e driver for rk3288 in cros kernel:
> chromiumos/src/third_party/kernel/local/drivers/media/platform/rockchip-vpu#
> ls rk3288_vpu_hw_h264e.c
> 
> and the related struct defined in v2 plugin:
> libv4l-rockchip_v2/libvepu/common/rk_venc.h:120:struct
> rk3288_h264e_reg_params {

Thanks for the reference.

I'm trying to build the v4l plugin stand alone. But in the repo I only see a 
Makefile.am file. I guess it integrates with the higher level build system of 
CromiumOS.

Do you have any idea how to build this v4l plugin code stand-alone?

Thanks,
baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.52.368.4656, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Tomasz Figa]

On Mon, Jul 2, 2018 at 4:25 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
>
> Hi Jeffy,
>
> On Fri, May 25, 2018 at 05:52:57PM +0800, JeffyChen wrote:
> > On 05/25/2018 02:10 PM, Tomasz Figa wrote:
> > > On Thu, May 24, 2018 at 9:18 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > > It looks like only v2 supports h.264 encoding. Does the v2 plugin work
> > > > with the rk3288?
> > >
> > > Hmm, looks like I missed the part about h264 in your original message (only
> > > noticed encoding). We didn't have a need to encode h264 on rk3288 devices.
> > > The need showed up only later when we were working on rk3399. This made
> > > both the driver and plugin for rk3288 lack h264 support.
> > >
> > > AFAICT the hardware programming interface differs between rk3288 and
> > > rk3399, so v2 might not work, but maybe you just need to reshuffle some
> > > registers.
> > >
> > > It's also possible that Rockchip has some code for it. Jeffy, do you know
> > > if there is h264 driver code and plugin for rk3288 available somewhere
> > > (based on the driver developed for Chrome OS 3.14 kernel)?
> >
> > well, i do saw a h264e driver for rk3288 in cros kernel:
> > chromiumos/src/third_party/kernel/local/drivers/media/platform/rockchip-vpu#
> > ls rk3288_vpu_hw_h264e.c
> >
> > and the related struct defined in v2 plugin:
> > libv4l-rockchip_v2/libvepu/common/rk_venc.h:120:struct
> > rk3288_h264e_reg_params {
>
> Thanks for the reference.
>
> I'm trying to build the v4l plugin stand alone. But in the repo I only see a
> Makefile.am file. I guess it integrates with the higher level build system of
> CromiumOS.
>
> Do you have any idea how to build this v4l plugin code stand-alone?

It's supposed to be built as a part of v4l-utils
(http://git.linuxtv.org/v4l-utils.git), which includes libv4l. We
build it with 1.6.0, but I guess newer versions may work too.

For reference, here's our ebuild (for Gentoo portage) to build it:
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/libv4lplugins/libv4lplugins-9999.ebuild

Best regards,
Tomasz

Re: rockchip-vpu driver usability

[Baruch Siach]

Hi Tomasz,

On Mon, Jul 02, 2018 at 04:30:52PM +0900, Tomasz Figa wrote:
> On Mon, Jul 2, 2018 at 4:25 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > On Fri, May 25, 2018 at 05:52:57PM +0800, JeffyChen wrote:
> > > On 05/25/2018 02:10 PM, Tomasz Figa wrote:
> > > > On Thu, May 24, 2018 at 9:18 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > > > It looks like only v2 supports h.264 encoding. Does the v2 plugin work
> > > > > with the rk3288?
> > > >
> > > > Hmm, looks like I missed the part about h264 in your original message (only
> > > > noticed encoding). We didn't have a need to encode h264 on rk3288 devices.
> > > > The need showed up only later when we were working on rk3399. This made
> > > > both the driver and plugin for rk3288 lack h264 support.
> > > >
> > > > AFAICT the hardware programming interface differs between rk3288 and
> > > > rk3399, so v2 might not work, but maybe you just need to reshuffle some
> > > > registers.
> > > >
> > > > It's also possible that Rockchip has some code for it. Jeffy, do you know
> > > > if there is h264 driver code and plugin for rk3288 available somewhere
> > > > (based on the driver developed for Chrome OS 3.14 kernel)?
> > >
> > > well, i do saw a h264e driver for rk3288 in cros kernel:
> > > chromiumos/src/third_party/kernel/local/drivers/media/platform/rockchip-vpu#
> > > ls rk3288_vpu_hw_h264e.c
> > >
> > > and the related struct defined in v2 plugin:
> > > libv4l-rockchip_v2/libvepu/common/rk_venc.h:120:struct
> > > rk3288_h264e_reg_params {
> >
> > Thanks for the reference.
> >
> > I'm trying to build the v4l plugin stand alone. But in the repo I only see a
> > Makefile.am file. I guess it integrates with the higher level build system of
> > CromiumOS.
> >
> > Do you have any idea how to build this v4l plugin code stand-alone?
> 
> It's supposed to be built as a part of v4l-utils
> (http://git.linuxtv.org/v4l-utils.git), which includes libv4l. We
> build it with 1.6.0, but I guess newer versions may work too.
> 
> For reference, here's our ebuild (for Gentoo portage) to build it:
> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/libv4lplugins/libv4lplugins-9999.ebuild

The rockchip plugin code refers to the 'config_store' field in v4l2_buffer and 
v4l2_ext_controls. This requires a special version of videodev2.h that is in 
the chromeos-4.4 kernel (featuring commits 3b6d3bc13e572 and 86be552490a2a). 

The code of libv4l2 also requires patching since it refers to the 'reserved2' 
field of v4l2_buffer since before version 1.6.0.

I could not find these patches in the libv4l2 ebuild[1]. Where can I find 
them? Is there some other version of libv4l2 that the rockchip plugin builds 
against?

Thanks,
baruch

[1] https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/libv4l/

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.52.368.4656, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Tomasz Figa]

Hi Baruch,

On Mon, Jul 2, 2018 at 11:57 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
>
> Hi Tomasz,
>
> On Mon, Jul 02, 2018 at 04:30:52PM +0900, Tomasz Figa wrote:
> > On Mon, Jul 2, 2018 at 4:25 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > On Fri, May 25, 2018 at 05:52:57PM +0800, JeffyChen wrote:
> > > > On 05/25/2018 02:10 PM, Tomasz Figa wrote:
> > > > > On Thu, May 24, 2018 at 9:18 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > > > > It looks like only v2 supports h.264 encoding. Does the v2 plugin work
> > > > > > with the rk3288?
> > > > >
> > > > > Hmm, looks like I missed the part about h264 in your original message (only
> > > > > noticed encoding). We didn't have a need to encode h264 on rk3288 devices.
> > > > > The need showed up only later when we were working on rk3399. This made
> > > > > both the driver and plugin for rk3288 lack h264 support.
> > > > >
> > > > > AFAICT the hardware programming interface differs between rk3288 and
> > > > > rk3399, so v2 might not work, but maybe you just need to reshuffle some
> > > > > registers.
> > > > >
> > > > > It's also possible that Rockchip has some code for it. Jeffy, do you know
> > > > > if there is h264 driver code and plugin for rk3288 available somewhere
> > > > > (based on the driver developed for Chrome OS 3.14 kernel)?
> > > >
> > > > well, i do saw a h264e driver for rk3288 in cros kernel:
> > > > chromiumos/src/third_party/kernel/local/drivers/media/platform/rockchip-vpu#
> > > > ls rk3288_vpu_hw_h264e.c
> > > >
> > > > and the related struct defined in v2 plugin:
> > > > libv4l-rockchip_v2/libvepu/common/rk_venc.h:120:struct
> > > > rk3288_h264e_reg_params {
> > >
> > > Thanks for the reference.
> > >
> > > I'm trying to build the v4l plugin stand alone. But in the repo I only see a
> > > Makefile.am file. I guess it integrates with the higher level build system of
> > > CromiumOS.
> > >
> > > Do you have any idea how to build this v4l plugin code stand-alone?
> >
> > It's supposed to be built as a part of v4l-utils
> > (http://git.linuxtv.org/v4l-utils.git), which includes libv4l. We
> > build it with 1.6.0, but I guess newer versions may work too.
> >
> > For reference, here's our ebuild (for Gentoo portage) to build it:
> > https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/libv4lplugins/libv4lplugins-9999.ebuild
>
> The rockchip plugin code refers to the 'config_store' field in v4l2_buffer and
> v4l2_ext_controls. This requires a special version of videodev2.h that is in
> the chromeos-4.4 kernel (featuring commits 3b6d3bc13e572 and 86be552490a2a).
>
> The code of libv4l2 also requires patching since it refers to the 'reserved2'
> field of v4l2_buffer since before version 1.6.0.
>
> I could not find these patches in the libv4l2 ebuild[1]. Where can I find
> them? Is there some other version of libv4l2 that the rockchip plugin builds
> against?

The code is based on Chrome OS kernel headers [1], which are based on
Chrome OS 4.4 kernel [2], which includes some downstream changes.
'config_store' comes from the V4L2 configuration store work in
progress series by Hans Verkuil, which eventually evolved into what we
know today as Request API (still not merged, though). The purpose of
configuration stores was to bind specific V4L2 control values with
particular buffers, since Rockchip encoder needs a number of low level
encoding parameters for every frame [3].

Another caveat is that the encoder API we have implemented in that
driver is just based on hardware registers (excluding the security
sensitive registers, which the driver deals with directly). The plugin
generates registers values and the driver just writes them to the
hardware. This needs to be replaced with a proper stateless encoder
API, based on Request API, which we're working on.

[1] https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/sys-kernel/linux-headers/linux-headers-4.4.ebuild
[2] https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-4.4/
[3] https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-4.4/drivers/media/platform/rockchip-vpu/rockchip_vpu_hw.h#117

Best regards,
Tomasz

>
> Thanks,
> baruch
>
> [1] https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/libv4l/
>
> --
>      http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
> =}------------------------------------------------ooO--U--Ooo------------{=
>    - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.52.368.4656, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Baruch Siach]

Hi Tomasz,

On Tue, Jul 03, 2018 at 11:29:03PM +0900, Tomasz Figa wrote:
> On Mon, Jul 2, 2018 at 11:57 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > On Mon, Jul 02, 2018 at 04:30:52PM +0900, Tomasz Figa wrote:
> > > On Mon, Jul 2, 2018 at 4:25 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > > I'm trying to build the v4l plugin stand alone. But in the repo I only 
> > > > see a Makefile.am file. I guess it integrates with the higher level 
> > > > build system of CromiumOS.
> > > >
> > > > Do you have any idea how to build this v4l plugin code stand-alone?
> > >
> > > It's supposed to be built as a part of v4l-utils
> > > (http://git.linuxtv.org/v4l-utils.git), which includes libv4l. We
> > > build it with 1.6.0, but I guess newer versions may work too.
> > >
> > > For reference, here's our ebuild (for Gentoo portage) to build it:
> > > https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/libv4lplugins/libv4lplugins-9999.ebuild
> >
> > The rockchip plugin code refers to the 'config_store' field in v4l2_buffer and
> > v4l2_ext_controls. This requires a special version of videodev2.h that is in
> > the chromeos-4.4 kernel (featuring commits 3b6d3bc13e572 and 86be552490a2a).
> >
> > The code of libv4l2 also requires patching since it refers to the 'reserved2'
> > field of v4l2_buffer since before version 1.6.0.
> >
> > I could not find these patches in the libv4l2 ebuild[1]. Where can I find
> > them? Is there some other version of libv4l2 that the rockchip plugin builds
> > against?
> 
> The code is based on Chrome OS kernel headers [1], which are based on
> Chrome OS 4.4 kernel [2], which includes some downstream changes.
> 'config_store' comes from the V4L2 configuration store work in
> progress series by Hans Verkuil, which eventually evolved into what we
> know today as Request API (still not merged, though). The purpose of
> configuration stores was to bind specific V4L2 control values with
> particular buffers, since Rockchip encoder needs a number of low level
> encoding parameters for every frame [3].
> 
> Another caveat is that the encoder API we have implemented in that
> driver is just based on hardware registers (excluding the security
> sensitive registers, which the driver deals with directly). The plugin
> generates registers values and the driver just writes them to the
> hardware. This needs to be replaced with a proper stateless encoder
> API, based on Request API, which we're working on.

So to get the rockchip-vpu driver working with current kernel I need to also 
forward-port the embryonic Request API as well. I'll look into this.

Are you aware of any other dependencies?

The alternative for me is to back-port rkisp1 video capture driver to vendor 
provided 4.4 kernel so I can use the MPP shim driver. I hope to not get there.

The easiest option, of course, is to just wait for the Request API and the new 
codec driver. I'm not sure I can afford to wait though.

Thanks for your support in sorting that out.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.2.679.5364, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Tomasz Figa]

On Wed, Jul 4, 2018 at 5:55 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
>
> Hi Tomasz,
>
> On Tue, Jul 03, 2018 at 11:29:03PM +0900, Tomasz Figa wrote:
> > On Mon, Jul 2, 2018 at 11:57 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > On Mon, Jul 02, 2018 at 04:30:52PM +0900, Tomasz Figa wrote:
> > > > On Mon, Jul 2, 2018 at 4:25 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > > > I'm trying to build the v4l plugin stand alone. But in the repo I only
> > > > > see a Makefile.am file. I guess it integrates with the higher level
> > > > > build system of CromiumOS.
> > > > >
> > > > > Do you have any idea how to build this v4l plugin code stand-alone?
> > > >
> > > > It's supposed to be built as a part of v4l-utils
> > > > (http://git.linuxtv.org/v4l-utils.git), which includes libv4l. We
> > > > build it with 1.6.0, but I guess newer versions may work too.
> > > >
> > > > For reference, here's our ebuild (for Gentoo portage) to build it:
> > > > https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/libv4lplugins/libv4lplugins-9999.ebuild
> > >
> > > The rockchip plugin code refers to the 'config_store' field in v4l2_buffer and
> > > v4l2_ext_controls. This requires a special version of videodev2.h that is in
> > > the chromeos-4.4 kernel (featuring commits 3b6d3bc13e572 and 86be552490a2a).
> > >
> > > The code of libv4l2 also requires patching since it refers to the 'reserved2'
> > > field of v4l2_buffer since before version 1.6.0.
> > >
> > > I could not find these patches in the libv4l2 ebuild[1]. Where can I find
> > > them? Is there some other version of libv4l2 that the rockchip plugin builds
> > > against?
> >
> > The code is based on Chrome OS kernel headers [1], which are based on
> > Chrome OS 4.4 kernel [2], which includes some downstream changes.
> > 'config_store' comes from the V4L2 configuration store work in
> > progress series by Hans Verkuil, which eventually evolved into what we
> > know today as Request API (still not merged, though). The purpose of
> > configuration stores was to bind specific V4L2 control values with
> > particular buffers, since Rockchip encoder needs a number of low level
> > encoding parameters for every frame [3].
> >
> > Another caveat is that the encoder API we have implemented in that
> > driver is just based on hardware registers (excluding the security
> > sensitive registers, which the driver deals with directly). The plugin
> > generates registers values and the driver just writes them to the
> > hardware. This needs to be replaced with a proper stateless encoder
> > API, based on Request API, which we're working on.
>
> So to get the rockchip-vpu driver working with current kernel I need to also
> forward-port the embryonic Request API as well. I'll look into this.
>
> Are you aware of any other dependencies?

I think V4L2 compound controls, but that should be already in
upstream. I'll try to look up relevant Chromium kernel patches for you
a bit later.

>
> The alternative for me is to back-port rkisp1 video capture driver to vendor
> provided 4.4 kernel so I can use the MPP shim driver. I hope to not get there.
>
> The easiest option, of course, is to just wait for the Request API and the new
> codec driver. I'm not sure I can afford to wait though.

I think you could also try to remove the use of config stores and make
the driver latch control values in QBUF(OUTPUT). You would also need
to adjust the plugin code accordingly.

Best regards,
Tomasz

Re: rockchip-vpu driver usability

[Baruch Siach]

Hi Tomasz,

On Wed, Jul 04, 2018 at 06:04:35PM +0900, Tomasz Figa wrote:
> On Wed, Jul 4, 2018 at 5:55 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > So to get the rockchip-vpu driver working with current kernel I need to 
> > also forward-port the embryonic Request API as well. I'll look into this.
> >
> > Are you aware of any other dependencies?
> 
> I think V4L2 compound controls, but that should be already in
> upstream. I'll try to look up relevant Chromium kernel patches for you
> a bit later.
> 
> > The alternative for me is to back-port rkisp1 video capture driver to vendor
> > provided 4.4 kernel so I can use the MPP shim driver. I hope to not get there.
> >
> > The easiest option, of course, is to just wait for the Request API and the new
> > codec driver. I'm not sure I can afford to wait though.
> 
> I think you could also try to remove the use of config stores and make
> the driver latch control values in QBUF(OUTPUT). You would also need
> to adjust the plugin code accordingly.

I'm trying to wrap me head around the config_store implementation in the 
'chromeos-4.4' kernel branch (commit a36974eae81a). I think I get the general 
idea. Still need to figure out the details.

I'm not sure I get your QBUF(OUTPUT) suggestion, though. Does "latch" means 
that the driver attaches the same controls to all buffers, essentially 
behaving like a traditional V4L2 device? Something else?

The limitation of being able to encode only one stream at a time is acceptable 
for my application.

Thanks,
baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org - tel: +972.52.368.4656, http://www.tkos.co.il -

Re: rockchip-vpu driver usability

[Tomasz Figa]

Hi Baruch,

Sorry, it looks like this message has been stuck somewhere in my
mailbox for quite a long time.

On Thu, Jul 5, 2018 at 9:14 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
>
> Hi Tomasz,
>
> On Wed, Jul 04, 2018 at 06:04:35PM +0900, Tomasz Figa wrote:
> > On Wed, Jul 4, 2018 at 5:55 PM Baruch Siach <baruch-NswTu9S1W3P6gbPvEgmw2w@public.gmane.org> wrote:
> > > So to get the rockchip-vpu driver working with current kernel I need to
> > > also forward-port the embryonic Request API as well. I'll look into this.
> > >
> > > Are you aware of any other dependencies?
> >
> > I think V4L2 compound controls, but that should be already in
> > upstream. I'll try to look up relevant Chromium kernel patches for you
> > a bit later.
> >
> > > The alternative for me is to back-port rkisp1 video capture driver to vendor
> > > provided 4.4 kernel so I can use the MPP shim driver. I hope to not get there.
> > >
> > > The easiest option, of course, is to just wait for the Request API and the new
> > > codec driver. I'm not sure I can afford to wait though.
> >
> > I think you could also try to remove the use of config stores and make
> > the driver latch control values in QBUF(OUTPUT). You would also need
> > to adjust the plugin code accordingly.
>
> I'm trying to wrap me head around the config_store implementation in the
> 'chromeos-4.4' kernel branch (commit a36974eae81a). I think I get the general
> idea. Still need to figure out the details.
>
> I'm not sure I get your QBUF(OUTPUT) suggestion, though. Does "latch" means
> that the driver attaches the same controls to all buffers, essentially
> behaving like a traditional V4L2 device? Something else?
>
> The limitation of being able to encode only one stream at a time is acceptable
> for my application.

The point is that userspace needs to specify a number of controls
exactly for particular frame. Let's consider following sequence in
user space:

1) S_CTRL(control X, 1)
2) QBUF(buffer 1, OUTPUT)
3) S_CTRL(control X, 2)
4) QBUF(buffer 2, OUTPUT)
5) S_CTRL(control X, 3)
6) QBUF(buffer 3, OUTPUT)

Traditional V4L2 device would start processing buffer 1 after 2)
completes, but all the further ioctls would just execute
asynchronously, possibly leading to buffer 2 being processed with
control X set to 3 (not 2) or even corrupting processing results of
buffer 1, since control X changes during the processing.

Latching would mean that rather than setting the active control value
in S_CTRL, the driver would keep the last recently set value and write
it to per-buffer data in QBUF.

Obviously, if your user space waits for processing to end between 2)
and 3) (and 4) and 5)), each buffer would be processed with correct
control value even with traditional behavior.

Best regards,
Tomasz