media: rockchip: the memory layout of multiplanes buffer for DMA address

media: rockchip: the memory layout of multiplanes buffer for DMA address

[Ayaka]

Hello all

I am writing the v4l2 decoder driver for rockchip. Although I hear the suggest from the Hans before, it is ok for decoder to use single plane buffer format, but I still decide to the multi planes format.

There is not a register for vdpau1 and vdpau2 setting the chroma starting address in both pixel image output(it has one only applied for jpeg) and reference. And the second plane should follow the first plane. It sounds pretty fix for the single plane, but the single plane can’t describe offset of the second plane, which is not the result of bytes per line multiples the height.

There are two different memory access steps in those rockchip device, 16bytes for vdpau1 and vdpau2, 64bytes for rkvdec and 128bytes for rkvdec with a high resolution. Although those access steps can be adjusted by the memory cache registers. So it is hard to describe the pixel format with the single plane formats or userspace would need to do more work.

Currently, I use the dma-contig allocator in my driver, it would allocate the second plane first, which results that the second plane has a lower address than first plane, which device would request the second plane follows the first plane. I increase the sizeimage of the first plane to solve this problem now and let device can continue to run, but I need a way to solve this problem.

Is there a way to control how does dma-contig allocate a buffer? I have not figured out the internal flow of the videobuf2. I know how to allocate two planes in different memory region which the s5p-mfc does with two alloc_devs, but that is not what I want.

Last time in FOSDEM, kwiboo and I talk some problems of the request API and stateless decoder, I say the a method to describe a buffer with many offsets as the buffer meta data would solve the most of  problems we talked, but I have no idea on how to implement it. And I think a buffer meta describing a buffer with many offsets would solve this problem as well.

Sent from my iPad

Re: media: rockchip: the memory layout of multiplanes buffer for DMA address

[Nicolas Dufresne]

Hi Ayaka,

Le mercredi 27 février 2019 à 23:13 +0800, Ayaka a écrit :
> Last time in FOSDEM, kwiboo and I talk some problems of the request
> API and stateless decoder, I say the a method to describe a buffer
> with many offsets as the buffer meta data would solve the most of 
> problems we talked, but I have no idea on how to implement it. And I
> think a buffer meta describing a buffer with many offsets would solve
> this problem as well.

for single allocation case, the only supported in-between plane padding
is an evenly distributed padding. This padding is communicated to
userspace through S_FMT, by extending the width and height. Userspace
then reads the display width/height through G_SELECTION API.

For anything else, the MPLANE structure with one of the multi-plane
format can "express" such buffer, though from userspace they are
exposed as two memory pointer or DMABuf FDs (which make importation
complicated if the buffer should initially be within a single
allocation). I'll leave to kernel dev the task to explain what is
feasible there (e.g. sub-buffering, etc.)

Nicolas

Re: media: rockchip: the memory layout of multiplanes buffer for DMA address

[Ayaka]

> On Feb 28, 2019, at 5:07 AM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> 
> Hi Ayaka,
> 
>> Le mercredi 27 février 2019 à 23:13 +0800, Ayaka a écrit :
>> Last time in FOSDEM, kwiboo and I talk some problems of the request
>> API and stateless decoder, I say the a method to describe a buffer
>> with many offsets as the buffer meta data would solve the most of 
>> problems we talked, but I have no idea on how to implement it. And I
>> think a buffer meta describing a buffer with many offsets would solve
>> this problem as well.
> 
> for single allocation case, the only supported in-between plane padding
> is an evenly distributed padding. This padding is communicated to
> userspace through S_FMT, by extending the width and height. Userspace
> then reads the display width/height through G_SELECTION API.
It can solve the partly problem, it is hard to use only width and height to describe a buffer. For hevc and rkvdec decoder in 128bytes mode, it is aligned with 128 bytes plus 128bytes. Sometimes, the padding data may. just less than a component. In that case, only offset would solve this problem.
> 
> For anything else, the MPLANE structure with one of the multi-plane
> format can "express" such buffer, though from userspace they are
> exposed as two memory pointer or DMABuf FDs (which make importation
The video output(VOP) supports two address for luma and chroma and the new generation of rkvdec supports that as well. But for the general situation, we should think we can only set one DMA address to the device.
> complicated if the buffer should initially be within a single
> allocation). I'll leave to kernel dev the task to explain what is
> feasible there (e.g. sub-buffering, etc.)
I think it can use the same fd but with a different data_offset in struct v4l2_plane(with a larger number of byteused in the second plane).

If I can find a solution to solve this problem, it would be hard to future develop on the stateless media device. Please help on this problem.
> 
> Nicolas
> 

Re: media: rockchip: the memory layout of multiplanes buffer for DMA address

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 2805 bytes --]

Le jeudi 28 février 2019 à 09:12 +0800, Ayaka a écrit :
> > On Feb 28, 2019, at 5:07 AM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > 
> > Hi Ayaka,
> > 
> > > Le mercredi 27 février 2019 à 23:13 +0800, Ayaka a écrit :
> > > Last time in FOSDEM, kwiboo and I talk some problems of the request
> > > API and stateless decoder, I say the a method to describe a buffer
> > > with many offsets as the buffer meta data would solve the most of 
> > > problems we talked, but I have no idea on how to implement it. And I
> > > think a buffer meta describing a buffer with many offsets would solve
> > > this problem as well.
> > 
> > for single allocation case, the only supported in-between plane padding
> > is an evenly distributed padding. This padding is communicated to
> > userspace through S_FMT, by extending the width and height. Userspace
> > then reads the display width/height through G_SELECTION API.
> It can solve the partly problem, it is hard to use only width and height to describe a buffer. For hevc and rkvdec decoder in 128bytes mode, it is aligned with 128 bytes plus 128bytes. Sometimes, the padding data may. just less than a component. In that case, only offset would solve this problem.
> > For anything else, the MPLANE structure with one of the multi-plane
> > format can "express" such buffer, though from userspace they are
> > exposed as two memory pointer or DMABuf FDs (which make importation
> The video output(VOP) supports two address for luma and chroma and the new generation of rkvdec supports that as well. But for the general situation, we should think we can only set one DMA address to the device.
> > complicated if the buffer should initially be within a single
> > allocation). I'll leave to kernel dev the task to explain what is
> > feasible there (e.g. sub-buffering, etc.)
> I think it can use the same fd but with a different data_offset in struct v4l2_plane(with a larger number of byteused in the second plane).

I think Hans said there is problem (something against the spec) in
using data_offset that way. The GStreamer implementation assumes that,
but got told recently that this might not be correct. I'd like Hans to
comment, since I haven't understood the reason yet.

> 
> If I can find a solution to solve this problem, it would be hard to future develop on the stateless media device. Please help on this problem.

I don't think this is specific to state less CODEC. While more complex,
when dealing with CMA, the kernel can do that math to figure-out if two
memory pointers are from the same allocation. It quite easy if you know
in advance the spacing.

Without being identical, the framework does similar things when trying
to import USERPTR in a CMA based V4L2 driver.

> > Nicolas
> > 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: media: rockchip: the memory layout of multiplanes buffer for DMA address

[Ayaka]

Sent from my iPad

> On Mar 1, 2019, at 12:21 AM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> 
> Le jeudi 28 février 2019 à 09:12 +0800, Ayaka a écrit :
>>> On Feb 28, 2019, at 5:07 AM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>>> 
>>> Hi Ayaka,
>>> 
>>>> Le mercredi 27 février 2019 à 23:13 +0800, Ayaka a écrit :
>>>> Last time in FOSDEM, kwiboo and I talk some problems of the request
>>>> API and stateless decoder, I say the a method to describe a buffer
>>>> with many offsets as the buffer meta data would solve the most of 
>>>> problems we talked, but I have no idea on how to implement it. And I
>>>> think a buffer meta describing a buffer with many offsets would solve
>>>> this problem as well.
>>> 
>>> for single allocation case, the only supported in-between plane padding
>>> is an evenly distributed padding. This padding is communicated to
>>> userspace through S_FMT, by extending the width and height. Userspace
>>> then reads the display width/height through G_SELECTION API.
>> It can solve the partly problem, it is hard to use only width and height to describe a buffer. For hevc and rkvdec decoder in 128bytes mode, it is aligned with 128 bytes plus 128bytes. Sometimes, the padding data may. just less than a component. In that case, only offset would solve this problem.
>>> For anything else, the MPLANE structure with one of the multi-plane
>>> format can "express" such buffer, though from userspace they are
>>> exposed as two memory pointer or DMABuf FDs (which make importation
>> The video output(VOP) supports two address for luma and chroma and the new generation of rkvdec supports that as well. But for the general situation, we should think we can only set one DMA address to the device.
>>> complicated if the buffer should initially be within a single
>>> allocation). I'll leave to kernel dev the task to explain what is
>>> feasible there (e.g. sub-buffering, etc.)
>> I think it can use the same fd but with a different data_offset in struct v4l2_plane(with a larger number of byteused in the second plane).
> 
> I think Hans said there is problem (something against the spec) in
> using data_offset that way. The GStreamer implementation assumes that,
> but got told recently that this might not be correct. I'd like Hans to
> comment, since I haven't understood the reason yet.
> 
>> 
>> If I can find a solution to solve this problem, it would be hard to future develop on the stateless media device. Please help on this problem.
> 
> I don't think this is specific to state less CODEC. While more complex,
> when dealing with CMA, the kernel can do that math to figure-out if two
> memory pointers are from the same allocation. It quite easy if you know
> in advance the spacing.
> 
I check only the s5p-mfc use CMA only in media. Also there are two register for the luma and chroma start address. Which is different to the current problem in rockchip platform.
> Without being identical, the framework does similar things when trying
> to import USERPTR in a CMA based V4L2 driver.
> 
>>> Nicolas
>>> 

Re: media: rockchip: the memory layout of multiplanes buffer for DMA address

[Tomasz Figa]

On Thu, Feb 28, 2019 at 12:13 AM Ayaka <ayaka@soulik.info> wrote:
>
> Hello all
>
> I am writing the v4l2 decoder driver for rockchip. Although I hear the suggest from the Hans before, it is ok for decoder to use single plane buffer format, but I still decide to the multi planes format.
>
> There is not a register for vdpau1 and vdpau2 setting the chroma starting address in both pixel image output(it has one only applied for jpeg) and reference. And the second plane should follow the first plane. It sounds pretty fix for the single plane, but the single plane can’t describe offset of the second plane, which is not the result of bytes per line multiples the height.
>
> There are two different memory access steps in those rockchip device, 16bytes for vdpau1 and vdpau2, 64bytes for rkvdec and 128bytes for rkvdec with a high resolution. Although those access steps can be adjusted by the memory cache registers. So it is hard to describe the pixel format with the single plane formats or userspace would need to do more work.

Why not just adjust the bytes per line to a multiple of 16/64/128
bytes? Most of the platforms allocate buffers this way for performance
reasons anyway.

>
> Currently, I use the dma-contig allocator in my driver, it would allocate the second plane first, which results that the second plane has a lower address than first plane, which device would request the second plane follows the first plane. I increase the sizeimage of the first plane to solve this problem now and let device can continue to run, but I need a way to solve this problem.
>
> Is there a way to control how does dma-contig allocate a buffer? I have not figured out the internal flow of the videobuf2. I know how to allocate two planes in different memory region which the s5p-mfc does with two alloc_devs, but that is not what I want.
>
> Last time in FOSDEM, kwiboo and I talk some problems of the request API and stateless decoder, I say the a method to describe a buffer with many offsets as the buffer meta data would solve the most of  problems we talked, but I have no idea on how to implement it. And I think a buffer meta describing a buffer with many offsets would solve this problem as well.
>
> Sent from my iPad

P.S. Please fix your email client to properly wrap the lines around
the ~75 column line.

Best regards,
Tomasz

Re: media: rockchip: the memory layout of multiplanes buffer for DMA address

[ayaka]

On 3/12/19 4:22 PM, Tomasz Figa wrote:
> On Thu, Feb 28, 2019 at 12:13 AM Ayaka <ayaka@soulik.info> wrote:
>> Hello all
>>
>> I am writing the v4l2 decoder driver for rockchip. Although I hear the suggest from the Hans before, it is ok for decoder to use single plane buffer format, but I still decide to the multi planes format.
>>
>> There is not a register for vdpau1 and vdpau2 setting the chroma starting address in both pixel image output(it has one only applied for jpeg) and reference. And the second plane should follow the first plane. It sounds pretty fix for the single plane, but the single plane can’t describe offset of the second plane, which is not the result of bytes per line multiples the height.
>>
>> There are two different memory access steps in those rockchip device, 16bytes for vdpau1 and vdpau2, 64bytes for rkvdec and 128bytes for rkvdec with a high resolution. Although those access steps can be adjusted by the memory cache registers. So it is hard to describe the pixel format with the single plane formats or userspace would need to do more work.
> Why not just adjust the bytes per line to a multiple of 16/64/128
> bytes? Most of the platforms allocate buffers this way for performance
> reasons anyway.

I can't, it depends on the device design. Also I need extra empty 
line(128/256 bytes) in each lines for the rkvdec. Maybe the device for 
chrome only request an alignment with 16 bytes per line and at vertical 
line, it is ok for the H.264 under the 1920x1080 resolution,  but above 
that resolution would suffer a poor performance as well UHD.

I made a mistake in the previous mail, the current suggest alignment for 
rkvdec is 256 bytes and plus an extra line.

Also it requests  a different alignment way for those chroma subsample 
are not 4:2:0. And it is not easy to align with a width value while its 
bitdepth is large than 8 bits.

>> Currently, I use the dma-contig allocator in my driver, it would allocate the second plane first, which results that the second plane has a lower address than first plane, which device would request the second plane follows the first plane. I increase the sizeimage of the first plane to solve this problem now and let device can continue to run, but I need a way to solve this problem.
>>
>> Is there a way to control how does dma-contig allocate a buffer? I have not figured out the internal flow of the videobuf2. I know how to allocate two planes in different memory region which the s5p-mfc does with two alloc_devs, but that is not what I want.
>>
>> Last time in FOSDEM, kwiboo and I talk some problems of the request API and stateless decoder, I say the a method to describe a buffer with many offsets as the buffer meta data would solve the most of  problems we talked, but I have no idea on how to implement it. And I think a buffer meta describing a buffer with many offsets would solve this problem as well.
>>
>> Sent from my iPad
> P.S. Please fix your email client to properly wrap the lines around
> the ~75 column line.
Blame the thuderbird.
>
> Best regards,
> Tomasz