* RFC: Request API and memory-to-memory devices @ 2018-05-24 8:44 Hans Verkuil 2018-05-25 4:15 ` Tomasz Figa 2018-05-25 14:16 ` Sakari Ailus 0 siblings, 2 replies; 6+ messages in thread From: Hans Verkuil @ 2018-05-24 8:44 UTC (permalink / raw) To: Linux Media Mailing List, Tomasz Figa, Sakari Ailus, Alexandre Courbot, Laurent Pinchart, Nicolas Dufresne Memory-to-memory devices have one video node, one internal control handler but two vb2_queues (DMA engines). While often there is one buffer produced for every buffer consumed, but this is by no means standard. E.g. deinterlacers will produce on buffer for every two buffers consumed. Codecs that receive a bit stream and can parse it to discover the framing may have no relation between the number of buffers consumed and the number of buffers produced. This poses a few problems for the Request API. Requiring that a request contains the buffers for both output and capture queue will be difficult to implement, especially in the latter case where there is no relationship between the number of consumed and produced buffers. In addition, userspace can make two requests: one for the capture queue, one for the output queue, each with associated controls. But since the controls are shared between capture and output there is an issue of what to do when the same control is set in both requests. I propose to restrict the usage of requests for m2m drivers to the output queue only. This keeps things simple for both kernel and userspace and avoids complex solutions. Requests only make sense if there is actually configuration you can apply for each buffer, and while that's true for the output queue, on the capture queue you just capture the result of whatever the device delivers. I don't believe there is much if anything you can or want to control per-buffer. Am I missing something? Comments? Regards, Hans ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: Request API and memory-to-memory devices 2018-05-24 8:44 RFC: Request API and memory-to-memory devices Hans Verkuil @ 2018-05-25 4:15 ` Tomasz Figa 2018-05-25 14:16 ` Sakari Ailus 1 sibling, 0 replies; 6+ messages in thread From: Tomasz Figa @ 2018-05-25 4:15 UTC (permalink / raw) To: Hans Verkuil Cc: Linux Media Mailing List, Sakari Ailus, Alexandre Courbot, Laurent Pinchart, nicolas On Thu, May 24, 2018 at 5:44 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: > Memory-to-memory devices have one video node, one internal control handler > but two vb2_queues (DMA engines). While often there is one buffer produced > for every buffer consumed, but this is by no means standard. E.g. deinterlacers > will produce on buffer for every two buffers consumed. Codecs that receive > a bit stream and can parse it to discover the framing may have no relation > between the number of buffers consumed and the number of buffers produced. > This poses a few problems for the Request API. Requiring that a request > contains the buffers for both output and capture queue will be difficult > to implement, especially in the latter case where there is no relationship > between the number of consumed and produced buffers. > In addition, userspace can make two requests: one for the capture queue, > one for the output queue, each with associated controls. But since the > controls are shared between capture and output there is an issue of > what to do when the same control is set in both requests. > I propose to restrict the usage of requests for m2m drivers to the output > queue only. This keeps things simple for both kernel and userspace and > avoids complex solutions. > Requests only make sense if there is actually configuration you can apply > for each buffer, and while that's true for the output queue, on the capture > queue you just capture the result of whatever the device delivers. I don't > believe there is much if anything you can or want to control per-buffer. > Am I missing something? Comments? That sounds fair to me, should make kernel code simpler (no 2 requests per job, as we saw in vim2m in some earlier RFC) and shouldn't complicate userspace (it has to dequeue requests and buffers anyway, so it always knows which buffer belongs to which request for the real 1:1 M2M cases). Best regards, Tomasz ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: Request API and memory-to-memory devices 2018-05-24 8:44 RFC: Request API and memory-to-memory devices Hans Verkuil 2018-05-25 4:15 ` Tomasz Figa @ 2018-05-25 14:16 ` Sakari Ailus 2018-05-25 14:40 ` Tomasz Figa 2018-05-25 15:26 ` Hans Verkuil 1 sibling, 2 replies; 6+ messages in thread From: Sakari Ailus @ 2018-05-25 14:16 UTC (permalink / raw) To: Hans Verkuil Cc: Linux Media Mailing List, Tomasz Figa, Alexandre Courbot, Laurent Pinchart, Nicolas Dufresne Hi Hans, On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote: > Memory-to-memory devices have one video node, one internal control handler > but two vb2_queues (DMA engines). While often there is one buffer produced > for every buffer consumed, but this is by no means standard. E.g. deinterlacers > will produce on buffer for every two buffers consumed. Codecs that receive > a bit stream and can parse it to discover the framing may have no relation > between the number of buffers consumed and the number of buffers produced. Do you have examples of such devices? I presume they're not supported in the current m2m API either, are they? > > This poses a few problems for the Request API. Requiring that a request > contains the buffers for both output and capture queue will be difficult > to implement, especially in the latter case where there is no relationship > between the number of consumed and produced buffers. > > In addition, userspace can make two requests: one for the capture queue, > one for the output queue, each with associated controls. But since the > controls are shared between capture and output there is an issue of > what to do when the same control is set in both requests. As I commented on v13, the two requests need to be handled separately in this case. Mem-to-mem devices are rather special in this respect; there's an established practice of matching buffers in the order they arrive from the queues, but that's not how the request API is intended to work: the buffers are associated to the request, and a request is processed independently of other requests. While that approach might work for mem-to-mem devices at least in some use cases, it is not a feasible approach for other devices. As a consequence, will have different API semantics between mem2mem devices and the rest. I'd like to avoid that if possible: this will be similarly visible in the user applications as well. > > I propose to restrict the usage of requests for m2m drivers to the output > queue only. This keeps things simple for both kernel and userspace and > avoids complex solutions. If there's a convincing reason to use different API semantics, such as the relationship between different buffers being unknown to the user, then there very likely is a need to associate non-request data with request-bound data in the driver. But it'd be better to limit it to where it's really needed. > > Requests only make sense if there is actually configuration you can apply > for each buffer, and while that's true for the output queue, on the capture > queue you just capture the result of whatever the device delivers. I don't > believe there is much if anything you can or want to control per-buffer. May there be controls associated with the capture queue buffers? -- Sakari Ailus sakari.ailus@linux.intel.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: Request API and memory-to-memory devices 2018-05-25 14:16 ` Sakari Ailus @ 2018-05-25 14:40 ` Tomasz Figa 2018-05-25 15:26 ` Hans Verkuil 1 sibling, 0 replies; 6+ messages in thread From: Tomasz Figa @ 2018-05-25 14:40 UTC (permalink / raw) To: Sakari Ailus Cc: Hans Verkuil, Linux Media Mailing List, Alexandre Courbot, Laurent Pinchart, nicolas On Fri, May 25, 2018 at 11:17 PM Sakari Ailus <sakari.ailus@linux.intel.com> wrote: > Hi Hans, > On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote: > > Memory-to-memory devices have one video node, one internal control handler > > but two vb2_queues (DMA engines). While often there is one buffer produced > > for every buffer consumed, but this is by no means standard. E.g. deinterlacers > > will produce on buffer for every two buffers consumed. Codecs that receive > > a bit stream and can parse it to discover the framing may have no relation > > between the number of buffers consumed and the number of buffers produced. > Do you have examples of such devices? I presume they're not supported in > the current m2m API either, are they? All stateful video decoders work like this and this means all video decoders currently in mainline (e.g. s5p-mfc, coda, mtk-vcodec), since we don't support stateless decoders yet. This is due to the nature of encoded bitstreams, such as H264, which can have frame reordering, long reference frame range and other tricks. I wouldn't say there is something like m2m API, unless you mean the m2m helpers, but that's not mandatory. Stateful decoders work according to V4L2 Codec API, which is a further specification of V4L2 (or something like application notes?). > > > > This poses a few problems for the Request API. Requiring that a request > > contains the buffers for both output and capture queue will be difficult > > to implement, especially in the latter case where there is no relationship > > between the number of consumed and produced buffers. > > > > In addition, userspace can make two requests: one for the capture queue, > > one for the output queue, each with associated controls. But since the > > controls are shared between capture and output there is an issue of > > what to do when the same control is set in both requests. > As I commented on v13, the two requests need to be handled separately in > this case. Mem-to-mem devices are rather special in this respect; there's > an established practice of matching buffers in the order they arrive from > the queues, but that's not how the request API is intended to work: the > buffers are associated to the request, and a request is processed > independently of other requests. > While that approach might work for mem-to-mem devices at least in some use > cases, it is not a feasible approach for other devices. As a consequence, > will have different API semantics between mem2mem devices and the rest. I'd > like to avoid that if possible: this will be similarly visible in the user > applications as well. I'd say that the semantics for m2m devices are already significantly different from non-m2m devices even without Request API, e.g. we have codec API for codecs. Everyone is aware of this special treatment and it doesn't seem to be a problem for anyone. > > > > I propose to restrict the usage of requests for m2m drivers to the output > > queue only. This keeps things simple for both kernel and userspace and > > avoids complex solutions. > If there's a convincing reason to use different API semantics, such as the > relationship between different buffers being unknown to the user, then > there very likely is a need to associate non-request data with > request-bound data in the driver. But it'd be better to limit it to where > it's really needed. > > > > Requests only make sense if there is actually configuration you can apply > > for each buffer, and while that's true for the output queue, on the capture > > queue you just capture the result of whatever the device delivers. I don't > > believe there is much if anything you can or want to control per-buffer. > May there be controls associated with the capture queue buffers? What would be the value in having such association? Don't we want to associate thing with particular hardware job (which is represented by a request) rather than buffer? Best regards, Tomasz ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: Request API and memory-to-memory devices 2018-05-25 14:16 ` Sakari Ailus 2018-05-25 14:40 ` Tomasz Figa @ 2018-05-25 15:26 ` Hans Verkuil 2018-06-04 11:42 ` Hans Verkuil 1 sibling, 1 reply; 6+ messages in thread From: Hans Verkuil @ 2018-05-25 15:26 UTC (permalink / raw) To: Sakari Ailus Cc: Linux Media Mailing List, Tomasz Figa, Alexandre Courbot, Laurent Pinchart, Nicolas Dufresne On 25/05/18 16:16, Sakari Ailus wrote: > Hi Hans, > > On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote: >> Memory-to-memory devices have one video node, one internal control handler >> but two vb2_queues (DMA engines). While often there is one buffer produced >> for every buffer consumed, but this is by no means standard. E.g. deinterlacers >> will produce on buffer for every two buffers consumed. Codecs that receive >> a bit stream and can parse it to discover the framing may have no relation >> between the number of buffers consumed and the number of buffers produced. > > Do you have examples of such devices? I presume they're not supported in > the current m2m API either, are they? > >> >> This poses a few problems for the Request API. Requiring that a request >> contains the buffers for both output and capture queue will be difficult >> to implement, especially in the latter case where there is no relationship >> between the number of consumed and produced buffers. >> >> In addition, userspace can make two requests: one for the capture queue, >> one for the output queue, each with associated controls. But since the >> controls are shared between capture and output there is an issue of >> what to do when the same control is set in both requests. > > As I commented on v13, the two requests need to be handled separately in > this case. Mem-to-mem devices are rather special in this respect; there's > an established practice of matching buffers in the order they arrive from > the queues, but that's not how the request API is intended to work: the > buffers are associated to the request, and a request is processed > independently of other requests. > > While that approach might work for mem-to-mem devices at least in some use > cases, it is not a feasible approach for other devices. As a consequence, > will have different API semantics between mem2mem devices and the rest. I'd > like to avoid that if possible: this will be similarly visible in the user > applications as well. > >> >> I propose to restrict the usage of requests for m2m drivers to the output >> queue only. This keeps things simple for both kernel and userspace and >> avoids complex solutions. > > If there's a convincing reason to use different API semantics, such as the > relationship between different buffers being unknown to the user, then > there very likely is a need to associate non-request data with > request-bound data in the driver. But it'd be better to limit it to where > it's really needed. > >> >> Requests only make sense if there is actually configuration you can apply >> for each buffer, and while that's true for the output queue, on the capture >> queue you just capture the result of whatever the device delivers. I don't >> believe there is much if anything you can or want to control per-buffer. > > May there be controls associated with the capture queue buffers? > In theory that could happen for m2m devices, but those controls would be different from controls associated with the output queue. The core problem is that if there is no clear relationship between capture and output buffers, then you cannot add a capture and an output buffer to the same request. That simply wouldn't work. How to signal this to the user? For m2m devices we could just specify that in the spec and check this in the core. As Tomasz said, m2m devices are already sufficiently special that I don't think this is a problem. But in the more generic case (complex pipelines) I cannot off-hand think of something elegant. I guess I would have to sleep on this a bit. Regards, Hans ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: Request API and memory-to-memory devices 2018-05-25 15:26 ` Hans Verkuil @ 2018-06-04 11:42 ` Hans Verkuil 0 siblings, 0 replies; 6+ messages in thread From: Hans Verkuil @ 2018-06-04 11:42 UTC (permalink / raw) To: Sakari Ailus Cc: Linux Media Mailing List, Tomasz Figa, Alexandre Courbot, Laurent Pinchart, Nicolas Dufresne On 05/25/2018 05:26 PM, Hans Verkuil wrote: > On 25/05/18 16:16, Sakari Ailus wrote: >> Hi Hans, >> >> On Thu, May 24, 2018 at 10:44:13AM +0200, Hans Verkuil wrote: >>> Memory-to-memory devices have one video node, one internal control handler >>> but two vb2_queues (DMA engines). While often there is one buffer produced >>> for every buffer consumed, but this is by no means standard. E.g. deinterlacers >>> will produce on buffer for every two buffers consumed. Codecs that receive >>> a bit stream and can parse it to discover the framing may have no relation >>> between the number of buffers consumed and the number of buffers produced. >> >> Do you have examples of such devices? I presume they're not supported in >> the current m2m API either, are they? >> >>> >>> This poses a few problems for the Request API. Requiring that a request >>> contains the buffers for both output and capture queue will be difficult >>> to implement, especially in the latter case where there is no relationship >>> between the number of consumed and produced buffers. >>> >>> In addition, userspace can make two requests: one for the capture queue, >>> one for the output queue, each with associated controls. But since the >>> controls are shared between capture and output there is an issue of >>> what to do when the same control is set in both requests. >> >> As I commented on v13, the two requests need to be handled separately in >> this case. Mem-to-mem devices are rather special in this respect; there's >> an established practice of matching buffers in the order they arrive from >> the queues, but that's not how the request API is intended to work: the >> buffers are associated to the request, and a request is processed >> independently of other requests. >> >> While that approach might work for mem-to-mem devices at least in some use >> cases, it is not a feasible approach for other devices. As a consequence, >> will have different API semantics between mem2mem devices and the rest. I'd >> like to avoid that if possible: this will be similarly visible in the user >> applications as well. >> >>> >>> I propose to restrict the usage of requests for m2m drivers to the output >>> queue only. This keeps things simple for both kernel and userspace and >>> avoids complex solutions. >> >> If there's a convincing reason to use different API semantics, such as the >> relationship between different buffers being unknown to the user, then >> there very likely is a need to associate non-request data with >> request-bound data in the driver. But it'd be better to limit it to where >> it's really needed. >> >>> >>> Requests only make sense if there is actually configuration you can apply >>> for each buffer, and while that's true for the output queue, on the capture >>> queue you just capture the result of whatever the device delivers. I don't >>> believe there is much if anything you can or want to control per-buffer. >> >> May there be controls associated with the capture queue buffers? >> > > In theory that could happen for m2m devices, but those controls would be different > from controls associated with the output queue. > > The core problem is that if there is no clear relationship between capture > and output buffers, then you cannot add a capture and an output buffer to > the same request. That simply wouldn't work. > > How to signal this to the user? For m2m devices we could just specify that > in the spec and check this in the core. As Tomasz said, m2m devices are > already sufficiently special that I don't think this is a problem. But in > the more generic case (complex pipelines) I cannot off-hand think of something > elegant. > > I guess I would have to sleep on this a bit. After thinking this over I believe that the only way this can be exposed to userspace is via the media controller. How exactly I do not know, but since this depends on the topology the MC is the only place you can provide such information to the user. For now I will just disable the use of requests for the capture queue of an m2m device. This can always be relaxed in the future once we figured out how to present this in the MC. Regards, Hans ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-06-04 11:42 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-24 8:44 RFC: Request API and memory-to-memory devices Hans Verkuil 2018-05-25 4:15 ` Tomasz Figa 2018-05-25 14:16 ` Sakari Ailus 2018-05-25 14:40 ` Tomasz Figa 2018-05-25 15:26 ` Hans Verkuil 2018-06-04 11:42 ` Hans Verkuil
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.