RFC: Request API and memory-to-memory devices

RFC: Request API and memory-to-memory devices

[Hans Verkuil]

Memory-to-memory devices have one video node, one internal control handler
but two vb2_queues (DMA engines). While often there is one buffer produced
for every buffer consumed, but this is by no means standard. E.g. deinterlacers
will produce on buffer for every two buffers consumed. Codecs that receive
a bit stream and can parse it to discover the framing may have no relation
between the number of buffers consumed and the number of buffers produced.

This poses a few problems for the Request API. Requiring that a request
contains the buffers for both output and capture queue will be difficult
to implement, especially in the latter case where there is no relationship
between the number of consumed and produced buffers.

In addition, userspace can make two requests: one for the capture queue,
one for the output queue, each with associated controls. But since the
controls are shared between capture and output there is an issue of
what to do when the same control is set in both requests.

I propose to restrict the usage of requests for m2m drivers to the output
queue only. This keeps things simple for both kernel and userspace and
avoids complex solutions.

Requests only make sense if there is actually configuration you can apply
for each buffer, and while that's true for the output queue, on the capture
queue you just capture the result of whatever the device delivers. I don't
believe there is much if anything you can or want to control per-buffer.

Am I missing something? Comments?

Regards,

	Hans

Re: RFC: Request API and memory-to-memory devices

[Tomasz Figa]

On Thu, May 24, 2018 at 5:44 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:

> Memory-to-memory devices have one video node, one internal control handler
> but two vb2_queues (DMA engines). While often there is one buffer produced
> for every buffer consumed, but this is by no means standard. E.g.
deinterlacers
> will produce on buffer for every two buffers consumed. Codecs that receive
> a bit stream and can parse it to discover the framing may have no relation
> between the number of buffers consumed and the number of buffers produced.

> This poses a few problems for the Request API. Requiring that a request
> contains the buffers for both output and capture queue will be difficult
> to implement, especially in the latter case where there is no relationship
> between the number of consumed and produced buffers.

> In addition, userspace can make two requests: one for the capture queue,
> one for the output queue, each with associated controls. But since the
> controls are shared between capture and output there is an issue of
> what to do when the same control is set in both requests.

> I propose to restrict the usage of requests for m2m drivers to the output
> queue only. This keeps things simple for both kernel and userspace and
> avoids complex solutions.

> Requests only make sense if there is actually configuration you can apply
> for each buffer, and while that's true for the output queue, on the
capture
> queue you just capture the result of whatever the device delivers. I don't
> believe there is much if anything you can or want to control per-buffer.

> Am I missing something? Comments?

That sounds fair to me, should make kernel code simpler (no 2 requests per
job, as we saw in vim2m in some earlier RFC) and shouldn't complicate
userspace (it has to dequeue requests and buffers anyway, so it always
knows which buffer belongs to which request for the real 1:1 M2M cases).

Best regards,
Tomasz

Re: RFC: Request API and memory-to-memory devices

[Sakari Ailus]

Hi Hans,

On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote:
> Memory-to-memory devices have one video node, one internal control handler
> but two vb2_queues (DMA engines). While often there is one buffer produced
> for every buffer consumed, but this is by no means standard. E.g. deinterlacers
> will produce on buffer for every two buffers consumed. Codecs that receive
> a bit stream and can parse it to discover the framing may have no relation
> between the number of buffers consumed and the number of buffers produced.

Do you have examples of such devices? I presume they're not supported in
the current m2m API either, are they?

> 
> This poses a few problems for the Request API. Requiring that a request
> contains the buffers for both output and capture queue will be difficult
> to implement, especially in the latter case where there is no relationship
> between the number of consumed and produced buffers.
> 
> In addition, userspace can make two requests: one for the capture queue,
> one for the output queue, each with associated controls. But since the
> controls are shared between capture and output there is an issue of
> what to do when the same control is set in both requests.

As I commented on v13, the two requests need to be handled separately in
this case. Mem-to-mem devices are rather special in this respect; there's
an established practice of matching buffers in the order they arrive from
the queues, but that's not how the request API is intended to work: the
buffers are associated to the request, and a request is processed
independently of other requests.

While that approach might work for mem-to-mem devices at least in some use
cases, it is not a feasible approach for other devices. As a consequence,
will have different API semantics between mem2mem devices and the rest. I'd
like to avoid that if possible: this will be similarly visible in the user
applications as well.

> 
> I propose to restrict the usage of requests for m2m drivers to the output
> queue only. This keeps things simple for both kernel and userspace and
> avoids complex solutions.

If there's a convincing reason to use different API semantics, such as the
relationship between different buffers being unknown to the user, then
there very likely is a need to associate non-request data with
request-bound data in the driver. But it'd be better to limit it to where
it's really needed.

> 
> Requests only make sense if there is actually configuration you can apply
> for each buffer, and while that's true for the output queue, on the capture
> queue you just capture the result of whatever the device delivers. I don't
> believe there is much if anything you can or want to control per-buffer.

May there be controls associated with the capture queue buffers?

-- 
Sakari Ailus
sakari.ailus@linux.intel.com

Re: RFC: Request API and memory-to-memory devices

[Tomasz Figa]

On Fri, May 25, 2018 at 11:17 PM Sakari Ailus <sakari.ailus@linux.intel.com>
wrote:

> Hi Hans,

> On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote:
> > Memory-to-memory devices have one video node, one internal control
handler
> > but two vb2_queues (DMA engines). While often there is one buffer
produced
> > for every buffer consumed, but this is by no means standard. E.g.
deinterlacers
> > will produce on buffer for every two buffers consumed. Codecs that
receive
> > a bit stream and can parse it to discover the framing may have no
relation
> > between the number of buffers consumed and the number of buffers
produced.

> Do you have examples of such devices? I presume they're not supported in
> the current m2m API either, are they?

All stateful video decoders work like this and this means all video
decoders currently in mainline (e.g. s5p-mfc, coda, mtk-vcodec), since we
don't support stateless decoders yet. This is due to the nature of encoded
bitstreams, such as H264, which can have frame reordering, long reference
frame range and other tricks.

I wouldn't say there is something like m2m API, unless you mean the m2m
helpers, but that's not mandatory. Stateful decoders work according to V4L2
Codec API, which is a further specification of V4L2 (or something like
application notes?).


> >
> > This poses a few problems for the Request API. Requiring that a request
> > contains the buffers for both output and capture queue will be difficult
> > to implement, especially in the latter case where there is no
relationship
> > between the number of consumed and produced buffers.
> >
> > In addition, userspace can make two requests: one for the capture queue,
> > one for the output queue, each with associated controls. But since the
> > controls are shared between capture and output there is an issue of
> > what to do when the same control is set in both requests.

> As I commented on v13, the two requests need to be handled separately in
> this case. Mem-to-mem devices are rather special in this respect; there's
> an established practice of matching buffers in the order they arrive from
> the queues, but that's not how the request API is intended to work: the
> buffers are associated to the request, and a request is processed
> independently of other requests.

> While that approach might work for mem-to-mem devices at least in some use
> cases, it is not a feasible approach for other devices. As a consequence,
> will have different API semantics between mem2mem devices and the rest.
I'd
> like to avoid that if possible: this will be similarly visible in the user
> applications as well.


I'd say that the semantics for m2m devices are already significantly
different from non-m2m devices even without Request API, e.g. we have codec
API for codecs. Everyone is aware of this special treatment and it doesn't
seem to be a problem for anyone.

> >
> > I propose to restrict the usage of requests for m2m drivers to the
output
> > queue only. This keeps things simple for both kernel and userspace and
> > avoids complex solutions.

> If there's a convincing reason to use different API semantics, such as the
> relationship between different buffers being unknown to the user, then
> there very likely is a need to associate non-request data with
> request-bound data in the driver. But it'd be better to limit it to where
> it's really needed.

> >
> > Requests only make sense if there is actually configuration you can
apply
> > for each buffer, and while that's true for the output queue, on the
capture
> > queue you just capture the result of whatever the device delivers. I
don't
> > believe there is much if anything you can or want to control per-buffer.

> May there be controls associated with the capture queue buffers?

What would be the value in having such association? Don't we want to
associate thing with particular hardware job (which is represented by a
request) rather than buffer?

Best regards,
Tomasz

Re: RFC: Request API and memory-to-memory devices

[Hans Verkuil]

On 25/05/18 16:16, Sakari Ailus wrote:
> Hi Hans,
> 
> On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote:
>> Memory-to-memory devices have one video node, one internal control handler
>> but two vb2_queues (DMA engines). While often there is one buffer produced
>> for every buffer consumed, but this is by no means standard. E.g. deinterlacers
>> will produce on buffer for every two buffers consumed. Codecs that receive
>> a bit stream and can parse it to discover the framing may have no relation
>> between the number of buffers consumed and the number of buffers produced.
> 
> Do you have examples of such devices? I presume they're not supported in
> the current m2m API either, are they?
> 
>>
>> This poses a few problems for the Request API. Requiring that a request
>> contains the buffers for both output and capture queue will be difficult
>> to implement, especially in the latter case where there is no relationship
>> between the number of consumed and produced buffers.
>>
>> In addition, userspace can make two requests: one for the capture queue,
>> one for the output queue, each with associated controls. But since the
>> controls are shared between capture and output there is an issue of
>> what to do when the same control is set in both requests.
> 
> As I commented on v13, the two requests need to be handled separately in
> this case. Mem-to-mem devices are rather special in this respect; there's
> an established practice of matching buffers in the order they arrive from
> the queues, but that's not how the request API is intended to work: the
> buffers are associated to the request, and a request is processed
> independently of other requests.
> 
> While that approach might work for mem-to-mem devices at least in some use
> cases, it is not a feasible approach for other devices. As a consequence,
> will have different API semantics between mem2mem devices and the rest. I'd
> like to avoid that if possible: this will be similarly visible in the user
> applications as well.
> 
>>
>> I propose to restrict the usage of requests for m2m drivers to the output
>> queue only. This keeps things simple for both kernel and userspace and
>> avoids complex solutions.
> 
> If there's a convincing reason to use different API semantics, such as the
> relationship between different buffers being unknown to the user, then
> there very likely is a need to associate non-request data with
> request-bound data in the driver. But it'd be better to limit it to where
> it's really needed.
> 
>>
>> Requests only make sense if there is actually configuration you can apply
>> for each buffer, and while that's true for the output queue, on the capture
>> queue you just capture the result of whatever the device delivers. I don't
>> believe there is much if anything you can or want to control per-buffer.
> 
> May there be controls associated with the capture queue buffers?
> 

In theory that could happen for m2m devices, but those controls would be different
from controls associated with the output queue.

The core problem is that if there is no clear relationship between capture
and output buffers, then you cannot add a capture and an output buffer to
the same request. That simply wouldn't work.

How to signal this to the user? For m2m devices we could just specify that
in the spec and check this in the core. As Tomasz said, m2m devices are
already sufficiently special that I don't think this is a problem. But in
the more generic case (complex pipelines) I cannot off-hand think of something
elegant.

I guess I would have to sleep on this a bit.

Regards,

	Hans

Re: RFC: Request API and memory-to-memory devices

[Hans Verkuil]

On 05/25/2018 05:26 PM, Hans Verkuil wrote:
> On 25/05/18 16:16, Sakari Ailus wrote:
>> Hi Hans,
>>
>> On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote:
>>> Memory-to-memory devices have one video node, one internal control handler
>>> but two vb2_queues (DMA engines). While often there is one buffer produced
>>> for every buffer consumed, but this is by no means standard. E.g. deinterlacers
>>> will produce on buffer for every two buffers consumed. Codecs that receive
>>> a bit stream and can parse it to discover the framing may have no relation
>>> between the number of buffers consumed and the number of buffers produced.
>>
>> Do you have examples of such devices? I presume they're not supported in
>> the current m2m API either, are they?
>>
>>>
>>> This poses a few problems for the Request API. Requiring that a request
>>> contains the buffers for both output and capture queue will be difficult
>>> to implement, especially in the latter case where there is no relationship
>>> between the number of consumed and produced buffers.
>>>
>>> In addition, userspace can make two requests: one for the capture queue,
>>> one for the output queue, each with associated controls. But since the
>>> controls are shared between capture and output there is an issue of
>>> what to do when the same control is set in both requests.
>>
>> As I commented on v13, the two requests need to be handled separately in
>> this case. Mem-to-mem devices are rather special in this respect; there's
>> an established practice of matching buffers in the order they arrive from
>> the queues, but that's not how the request API is intended to work: the
>> buffers are associated to the request, and a request is processed
>> independently of other requests.
>>
>> While that approach might work for mem-to-mem devices at least in some use
>> cases, it is not a feasible approach for other devices. As a consequence,
>> will have different API semantics between mem2mem devices and the rest. I'd
>> like to avoid that if possible: this will be similarly visible in the user
>> applications as well.
>>
>>>
>>> I propose to restrict the usage of requests for m2m drivers to the output
>>> queue only. This keeps things simple for both kernel and userspace and
>>> avoids complex solutions.
>>
>> If there's a convincing reason to use different API semantics, such as the
>> relationship between different buffers being unknown to the user, then
>> there very likely is a need to associate non-request data with
>> request-bound data in the driver. But it'd be better to limit it to where
>> it's really needed.
>>
>>>
>>> Requests only make sense if there is actually configuration you can apply
>>> for each buffer, and while that's true for the output queue, on the capture
>>> queue you just capture the result of whatever the device delivers. I don't
>>> believe there is much if anything you can or want to control per-buffer.
>>
>> May there be controls associated with the capture queue buffers?
>>
> 
> In theory that could happen for m2m devices, but those controls would be different
> from controls associated with the output queue.
> 
> The core problem is that if there is no clear relationship between capture
> and output buffers, then you cannot add a capture and an output buffer to
> the same request. That simply wouldn't work.
> 
> How to signal this to the user? For m2m devices we could just specify that
> in the spec and check this in the core. As Tomasz said, m2m devices are
> already sufficiently special that I don't think this is a problem. But in
> the more generic case (complex pipelines) I cannot off-hand think of something
> elegant.
> 
> I guess I would have to sleep on this a bit.

After thinking this over I believe that the only way this can be exposed to
userspace is via the media controller. How exactly I do not know, but since this
depends on the topology the MC is the only place you can provide such information
to the user.

For now I will just disable the use of requests for the capture queue of an m2m
device. This can always be relaxed in the future once we figured out how to
present this in the MC.

Regards,

	Hans