Overlay support in the i.MX7 display

Overlay support in the i.MX7 display

[Laurent Pinchart]

Hello,

I'm looking at the available options to support overlays in the display
pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
overlays, the feature being implemented in the PXP. A driver for the PXP
is available but only supports older SoCs whose PXP doesn't support
overlays. This driver is implemented as a V4L2 mem2mem driver, which
makes support of additional input channels impossible.

Here are the options I can envision:

- Extend the existing PXP driver to support multiple channels. This is
  technically feasible, but will require moving away from the V4L2
  mem2mem framework, which would break userspace. I don't think this
  path could lead anywhere.

- Write a new PXP driver for the i.MX7, still using V4L2, but with
  multiple video nodes. This would allow blending multiple layers, but
  would require writing the output to memory, while the PXP has support
  for direct connections to the LCDIF (through small SRAM buffers).
  Performances would thus be suboptimal. The API would also be awkward,
  as using the PXP for display would require usage of V4L2 in
  applications.

- Extend the mxsfb driver with PXP support, and expose the PXP inputs as
  KMS planes. The PXP would only be used when available, and would be
  transparent to applications. This would however prevent using it
  separately from the display (to perform multi-pass alpha blending for
  instance).

What would be the best option going forward ? Would any of you, by any
chance, have already started work in this area ?

-- 
Regards,

Laurent Pinchart

Re: Overlay support in the i.MX7 display

[Stefan Agner]

Hi Laurent,

On 2019-11-01 09:43, Laurent Pinchart wrote:
> Hello,
> 
> I'm looking at the available options to support overlays in the display
> pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> overlays, the feature being implemented in the PXP. A driver for the PXP
> is available but only supports older SoCs whose PXP doesn't support
> overlays. This driver is implemented as a V4L2 mem2mem driver, which
> makes support of additional input channels impossible.

Thanks for bringing this up, it is a topic I have wondered too:
Interaction between PXP and mxsfb.

I am not very familiar with the V4L2 subsystem so take my opinions with
a grain of salt.

> 
> Here are the options I can envision:
> 
> - Extend the existing PXP driver to support multiple channels. This is
>   technically feasible, but will require moving away from the V4L2
>   mem2mem framework, which would break userspace. I don't think this
>   path could lead anywhere.
> 
> - Write a new PXP driver for the i.MX7, still using V4L2, but with
>   multiple video nodes. This would allow blending multiple layers, but
>   would require writing the output to memory, while the PXP has support
>   for direct connections to the LCDIF (through small SRAM buffers).
>   Performances would thus be suboptimal. The API would also be awkward,
>   as using the PXP for display would require usage of V4L2 in
>   applications.

So the video nodes would be sinks? I would expect overlays to be usable
through KMS, I guess that would then not work, correct?

> 
> - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
>   KMS planes. The PXP would only be used when available, and would be
>   transparent to applications. This would however prevent using it
>   separately from the display (to perform multi-pass alpha blending for
>   instance).

KMS planes are well defined and are well integrated with the KMS API, so
I prefer this option. But is this compatible with the currently
supported video use-case? E.g. could we make PXP available through V4L2
and through DRM/mxsfb?

Not sure what your use case is exactly, but when playing a video I
wonder where is the higher value using PXP: Color conversion and scaling
or compositing...? I would expect higher value in the former use case.

> 
> What would be the best option going forward ? Would any of you, by any
> chance, have already started work in this area ?

I have not worked in that area, so feel free!

--
Stefan

Re: Overlay support in the i.MX7 display

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 3172 bytes --]

On Sun, 03 Nov 2019 19:15:49 +0100
Stefan Agner <stefan@agner.ch> wrote:

> Hi Laurent,
> 
> On 2019-11-01 09:43, Laurent Pinchart wrote:
> > Hello,
> > 
> > I'm looking at the available options to support overlays in the display
> > pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> > overlays, the feature being implemented in the PXP. A driver for the PXP
> > is available but only supports older SoCs whose PXP doesn't support
> > overlays. This driver is implemented as a V4L2 mem2mem driver, which
> > makes support of additional input channels impossible.  
> 
> Thanks for bringing this up, it is a topic I have wondered too:
> Interaction between PXP and mxsfb.
> 
> I am not very familiar with the V4L2 subsystem so take my opinions with
> a grain of salt.
> 
> > 
> > Here are the options I can envision:
> > 
> > - Extend the existing PXP driver to support multiple channels. This is
> >   technically feasible, but will require moving away from the V4L2
> >   mem2mem framework, which would break userspace. I don't think this
> >   path could lead anywhere.
> > 
> > - Write a new PXP driver for the i.MX7, still using V4L2, but with
> >   multiple video nodes. This would allow blending multiple layers, but
> >   would require writing the output to memory, while the PXP has support
> >   for direct connections to the LCDIF (through small SRAM buffers).
> >   Performances would thus be suboptimal. The API would also be awkward,
> >   as using the PXP for display would require usage of V4L2 in
> >   applications.  
> 
> So the video nodes would be sinks? I would expect overlays to be usable
> through KMS, I guess that would then not work, correct?
> 
> > 
> > - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
> >   KMS planes. The PXP would only be used when available, and would be
> >   transparent to applications. This would however prevent using it
> >   separately from the display (to perform multi-pass alpha blending for
> >   instance).  
> 
> KMS planes are well defined and are well integrated with the KMS API, so
> I prefer this option. But is this compatible with the currently
> supported video use-case? E.g. could we make PXP available through V4L2
> and through DRM/mxsfb?
> 
> Not sure what your use case is exactly, but when playing a video I
> wonder where is the higher value using PXP: Color conversion and scaling
> or compositing...? I would expect higher value in the former use case.

Hi,

mind, with Wayland architecture, color conversion and scaling could be
at the same level/step as compositing, in the display server instead of
an application. Hence if the PXP capabilities were advertised as KMS
planes, there should be nothing to patch in Wayland-designed
applications to make use of them, assuming the applications did not
already rely on V4L2 M2M devices.

Would it not be possible to expose PXP through both uAPI interfaces? At
least KMS atomic's TEST_ONLY feature would make it easy to say "no" to
userspace if another bit of userspace already reserved the device via
e.g. V4L2.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Overlay support in the i.MX7 display

[Laurent Pinchart]

Hello,

On Mon, Nov 04, 2019 at 10:09:47AM +0200, Pekka Paalanen wrote:
> On Sun, 03 Nov 2019 19:15:49 +0100 Stefan Agner wrote:
> > On 2019-11-01 09:43, Laurent Pinchart wrote:
> > > Hello,
> > > 
> > > I'm looking at the available options to support overlays in the display
> > > pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> > > overlays, the feature being implemented in the PXP. A driver for the PXP
> > > is available but only supports older SoCs whose PXP doesn't support
> > > overlays. This driver is implemented as a V4L2 mem2mem driver, which
> > > makes support of additional input channels impossible.  
> > 
> > Thanks for bringing this up, it is a topic I have wondered too:
> > Interaction between PXP and mxsfb.
> > 
> > I am not very familiar with the V4L2 subsystem so take my opinions with
> > a grain of salt.
> > 
> > > Here are the options I can envision:
> > > 
> > > - Extend the existing PXP driver to support multiple channels. This is
> > >   technically feasible, but will require moving away from the V4L2
> > >   mem2mem framework, which would break userspace. I don't think this
> > >   path could lead anywhere.
> > > 
> > > - Write a new PXP driver for the i.MX7, still using V4L2, but with
> > >   multiple video nodes. This would allow blending multiple layers, but
> > >   would require writing the output to memory, while the PXP has support
> > >   for direct connections to the LCDIF (through small SRAM buffers).
> > >   Performances would thus be suboptimal. The API would also be awkward,
> > >   as using the PXP for display would require usage of V4L2 in
> > >   applications.  
> > 
> > So the video nodes would be sinks? I would expect overlays to be usable
> > through KMS, I guess that would then not work, correct?

There would be sink video nodes for the PXP inputs, and one source video
node for the PXP output. The PXP can be used stand-alone, in
memory-to-memory mode, and V4L2 is a good fit for that.

> > > 
> > > - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
> > >   KMS planes. The PXP would only be used when available, and would be
> > >   transparent to applications. This would however prevent using it
> > >   separately from the display (to perform multi-pass alpha blending for
> > >   instance).  
> > 
> > KMS planes are well defined and are well integrated with the KMS API, so
> > I prefer this option. But is this compatible with the currently
> > supported video use-case? E.g. could we make PXP available through V4L2
> > and through DRM/mxsfb?

That's the issue, it's not easily doable. I think we could do so, but
how to ensure mutual exclusion between the two APIs needs to be
researched. I fear it will result in an awkward solution with fuzzy
semantics. A module parameter could be an option, but wouldn't be very
flexible.

> > Not sure what your use case is exactly, but when playing a video I
> > wonder where is the higher value using PXP: Color conversion and scaling
> > or compositing...? I would expect higher value in the former use case.

I think it's highly use-case-dependent.

> mind, with Wayland architecture, color conversion and scaling could be
> at the same level/step as compositing, in the display server instead of
> an application. Hence if the PXP capabilities were advertised as KMS
> planes, there should be nothing to patch in Wayland-designed
> applications to make use of them, assuming the applications did not
> already rely on V4L2 M2M devices.
> 
> Would it not be possible to expose PXP through both uAPI interfaces? At
> least KMS atomic's TEST_ONLY feature would make it easy to say "no" to
> userspace if another bit of userspace already reserved the device via
> e.g. V4L2.

We would also need to figure out how to do it the other way around,
reporting properly through V4L2 that the device is busy. I think it's
feasible, but I doubt it would result in anything usable for userspace.
If the KMS device exposes multiple planes unconditionally and fails the
atomic commit if the PXP is used through V4L2, I think it would be hard
for Wayland to use this consistently. Given that I expect the PXP to be
mostly used for display purpose I'm tempted to allocate it for display
unconditionally, or, possibly, decide how to expose it through a module
parameter.

We have a similar situation on Renesas R-Car Gen3 platforms, with a
memory-to-memory compositor called VSP. Some VSP instances are connected
to the display controller, and we allocate them for display
unconditionally. Other VSP instances are exposed as V4L2 devices. We
haven't heard of anyone who wanted to use the display VSP instances for
unrelated purposes. If such a use case arose, exposing those instances
through V4L2 would just be a matter of flipping one bit in the driver
(all the infrastructure is in place), which we would likely expose as a
module parameter.

-- 
Regards,

Laurent Pinchart

Re: Overlay support in the i.MX7 display

[Daniel Vetter]

On Mon, Nov 04, 2019 at 02:58:29PM +0200, Laurent Pinchart wrote:
> Hello,
> 
> On Mon, Nov 04, 2019 at 10:09:47AM +0200, Pekka Paalanen wrote:
> > On Sun, 03 Nov 2019 19:15:49 +0100 Stefan Agner wrote:
> > > On 2019-11-01 09:43, Laurent Pinchart wrote:
> > > > Hello,
> > > > 
> > > > I'm looking at the available options to support overlays in the display
> > > > pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> > > > overlays, the feature being implemented in the PXP. A driver for the PXP
> > > > is available but only supports older SoCs whose PXP doesn't support
> > > > overlays. This driver is implemented as a V4L2 mem2mem driver, which
> > > > makes support of additional input channels impossible.  
> > > 
> > > Thanks for bringing this up, it is a topic I have wondered too:
> > > Interaction between PXP and mxsfb.
> > > 
> > > I am not very familiar with the V4L2 subsystem so take my opinions with
> > > a grain of salt.
> > > 
> > > > Here are the options I can envision:
> > > > 
> > > > - Extend the existing PXP driver to support multiple channels. This is
> > > >   technically feasible, but will require moving away from the V4L2
> > > >   mem2mem framework, which would break userspace. I don't think this
> > > >   path could lead anywhere.
> > > > 
> > > > - Write a new PXP driver for the i.MX7, still using V4L2, but with
> > > >   multiple video nodes. This would allow blending multiple layers, but
> > > >   would require writing the output to memory, while the PXP has support
> > > >   for direct connections to the LCDIF (through small SRAM buffers).
> > > >   Performances would thus be suboptimal. The API would also be awkward,
> > > >   as using the PXP for display would require usage of V4L2 in
> > > >   applications.  
> > > 
> > > So the video nodes would be sinks? I would expect overlays to be usable
> > > through KMS, I guess that would then not work, correct?
> 
> There would be sink video nodes for the PXP inputs, and one source video
> node for the PXP output. The PXP can be used stand-alone, in
> memory-to-memory mode, and V4L2 is a good fit for that.
> 
> > > > 
> > > > - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
> > > >   KMS planes. The PXP would only be used when available, and would be
> > > >   transparent to applications. This would however prevent using it
> > > >   separately from the display (to perform multi-pass alpha blending for
> > > >   instance).  
> > > 
> > > KMS planes are well defined and are well integrated with the KMS API, so
> > > I prefer this option. But is this compatible with the currently
> > > supported video use-case? E.g. could we make PXP available through V4L2
> > > and through DRM/mxsfb?
> 
> That's the issue, it's not easily doable. I think we could do so, but
> how to ensure mutual exclusion between the two APIs needs to be
> researched. I fear it will result in an awkward solution with fuzzy
> semantics. A module parameter could be an option, but wouldn't be very
> flexible.
> 
> > > Not sure what your use case is exactly, but when playing a video I
> > > wonder where is the higher value using PXP: Color conversion and scaling
> > > or compositing...? I would expect higher value in the former use case.
> 
> I think it's highly use-case-dependent.
> 
> > mind, with Wayland architecture, color conversion and scaling could be
> > at the same level/step as compositing, in the display server instead of
> > an application. Hence if the PXP capabilities were advertised as KMS
> > planes, there should be nothing to patch in Wayland-designed
> > applications to make use of them, assuming the applications did not
> > already rely on V4L2 M2M devices.
> > 
> > Would it not be possible to expose PXP through both uAPI interfaces? At
> > least KMS atomic's TEST_ONLY feature would make it easy to say "no" to
> > userspace if another bit of userspace already reserved the device via
> > e.g. V4L2.
> 
> We would also need to figure out how to do it the other way around,
> reporting properly through V4L2 that the device is busy. I think it's
> feasible, but I doubt it would result in anything usable for userspace.
> If the KMS device exposes multiple planes unconditionally and fails the
> atomic commit if the PXP is used through V4L2, I think it would be hard
> for Wayland to use this consistently. Given that I expect the PXP to be
> mostly used for display purpose I'm tempted to allocate it for display
> unconditionally, or, possibly, decide how to expose it through a module
> parameter.

KMS should be fine if planes are missing, userspace is supposed to be able
to cope with that. Not all userspace does, but welp.
 
I figured the bigger issue will be on the v4l side, since "device
temporarily gone" is not something v4l understands as a concept?

But even then having one device for userspace would be best I think, just
a lot more reasonable (insert wish for unified video/display subsystem
here).

> We have a similar situation on Renesas R-Car Gen3 platforms, with a
> memory-to-memory compositor called VSP. Some VSP instances are connected
> to the display controller, and we allocate them for display
> unconditionally. Other VSP instances are exposed as V4L2 devices. We
> haven't heard of anyone who wanted to use the display VSP instances for
> unrelated purposes. If such a use case arose, exposing those instances
> through V4L2 would just be a matter of flipping one bit in the driver
> (all the infrastructure is in place), which we would likely expose as a
> module parameter.

Hm yeah I guess we could just assign them, if the use-cases are clear-cut
enough. Are you thinking of doing these links with dt (so at least it
would be patchable)?
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: Overlay support in the i.MX7 display

[Philipp Zabel]

Hi Laurent,

On Fri, 2019-11-01 at 10:43 +0200, Laurent Pinchart wrote:
> Hello,
> 
> I'm looking at the available options to support overlays in the display
> pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> overlays, the feature being implemented in the PXP. A driver for the PXP
> is available but only supports older SoCs whose PXP doesn't support
> overlays. This driver is implemented as a V4L2 mem2mem driver, which
> makes support of additional input channels impossible.
> 
> Here are the options I can envision:
> 
> - Extend the existing PXP driver to support multiple channels. This is
>   technically feasible, but will require moving away from the V4L2
>   mem2mem framework, which would break userspace. I don't think this
>   path could lead anywhere.

I may be biased, but please don't break the V4L2 mem2mem usecase :)

> - Write a new PXP driver for the i.MX7, still using V4L2, but with
>   multiple video nodes. This would allow blending multiple layers, but
>   would require writing the output to memory, while the PXP has support
>   for direct connections to the LCDIF (through small SRAM buffers).
>   Performances would thus be suboptimal. The API would also be awkward,
>   as using the PXP for display would require usage of V4L2 in
>   applications.

I'm not sure V4L2 is the best API for multi-pass 2D composition,
especially as the PXP is able to blit an overlay onto a background in
place.

> - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
>   KMS planes. The PXP would only be used when available, and would be
>   transparent to applications. This would however prevent using it
>   separately from the display (to perform multi-pass alpha blending for
>   instance).

For the SRAM block row buffer path to the LCDIF, I think the KMS plane
abstraction is the way to go. The DRM and V4L2 drivers could be made to
use a shared backend, such that only one of plane composition and V4L2
scaling/CSC functions can work at the same time.

> What would be the best option going forward ? Would any of you, by any
> chance, have already started work in this area ?

I have not worked on this.

regards
Philipp

Re: Overlay support in the i.MX7 display

[Nicolas Dufresne]

Le lundi 04 novembre 2019 à 10:09 +0200, Pekka Paalanen a écrit :
> On Sun, 03 Nov 2019 19:15:49 +0100
> Stefan Agner <stefan@agner.ch> wrote:
> 
> > Hi Laurent,
> > 
> > On 2019-11-01 09:43, Laurent Pinchart wrote:
> > > Hello,
> > > 
> > > I'm looking at the available options to support overlays in the display
> > > pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> > > overlays, the feature being implemented in the PXP. A driver for the PXP
> > > is available but only supports older SoCs whose PXP doesn't support
> > > overlays. This driver is implemented as a V4L2 mem2mem driver, which
> > > makes support of additional input channels impossible.  
> > 
> > Thanks for bringing this up, it is a topic I have wondered too:
> > Interaction between PXP and mxsfb.
> > 
> > I am not very familiar with the V4L2 subsystem so take my opinions with
> > a grain of salt.
> > 
> > > Here are the options I can envision:
> > > 
> > > - Extend the existing PXP driver to support multiple channels. This is
> > >   technically feasible, but will require moving away from the V4L2
> > >   mem2mem framework, which would break userspace. I don't think this
> > >   path could lead anywhere.
> > > 
> > > - Write a new PXP driver for the i.MX7, still using V4L2, but with
> > >   multiple video nodes. This would allow blending multiple layers, but
> > >   would require writing the output to memory, while the PXP has support
> > >   for direct connections to the LCDIF (through small SRAM buffers).
> > >   Performances would thus be suboptimal. The API would also be awkward,
> > >   as using the PXP for display would require usage of V4L2 in
> > >   applications.  
> > 
> > So the video nodes would be sinks? I would expect overlays to be usable
> > through KMS, I guess that would then not work, correct?
> > 
> > > - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
> > >   KMS planes. The PXP would only be used when available, and would be
> > >   transparent to applications. This would however prevent using it
> > >   separately from the display (to perform multi-pass alpha blending for
> > >   instance).  
> > 
> > KMS planes are well defined and are well integrated with the KMS API, so
> > I prefer this option. But is this compatible with the currently
> > supported video use-case? E.g. could we make PXP available through V4L2
> > and through DRM/mxsfb?
> > 
> > Not sure what your use case is exactly, but when playing a video I
> > wonder where is the higher value using PXP: Color conversion and scaling
> > or compositing...? I would expect higher value in the former use case.
> 
> Hi,
> 
> mind, with Wayland architecture, color conversion and scaling could be
> at the same level/step as compositing, in the display server instead of
> an application. Hence if the PXP capabilities were advertised as KMS
> planes, there should be nothing to patch in Wayland-designed
> applications to make use of them, assuming the applications did not
> already rely on V4L2 M2M devices.

The PXP can already be used with GStreamer v4l2convert element, for CSC
and scaling.

> 
> Would it not be possible to expose PXP through both uAPI interfaces? At
> least KMS atomic's TEST_ONLY feature would make it easy to say "no" to
> userspace if another bit of userspace already reserved the device via
> e.g. V4L2.

Same exist for decoders with fixed number of streams/instances I think.

> 
> 
> Thanks,
> pq

Re: Overlay support in the i.MX7 display

[Nicolas Dufresne]

Le lundi 04 novembre 2019 à 14:58 +0200, Laurent Pinchart a écrit :
> Hello,
> 
> On Mon, Nov 04, 2019 at 10:09:47AM +0200, Pekka Paalanen wrote:
> > On Sun, 03 Nov 2019 19:15:49 +0100 Stefan Agner wrote:
> > > On 2019-11-01 09:43, Laurent Pinchart wrote:
> > > > Hello,
> > > > 
> > > > I'm looking at the available options to support overlays in the display
> > > > pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> > > > overlays, the feature being implemented in the PXP. A driver for the PXP
> > > > is available but only supports older SoCs whose PXP doesn't support
> > > > overlays. This driver is implemented as a V4L2 mem2mem driver, which
> > > > makes support of additional input channels impossible.  
> > > 
> > > Thanks for bringing this up, it is a topic I have wondered too:
> > > Interaction between PXP and mxsfb.
> > > 
> > > I am not very familiar with the V4L2 subsystem so take my opinions with
> > > a grain of salt.
> > > 
> > > > Here are the options I can envision:
> > > > 
> > > > - Extend the existing PXP driver to support multiple channels. This is
> > > >   technically feasible, but will require moving away from the V4L2
> > > >   mem2mem framework, which would break userspace. I don't think this
> > > >   path could lead anywhere.
> > > > 
> > > > - Write a new PXP driver for the i.MX7, still using V4L2, but with
> > > >   multiple video nodes. This would allow blending multiple layers, but
> > > >   would require writing the output to memory, while the PXP has support
> > > >   for direct connections to the LCDIF (through small SRAM buffers).
> > > >   Performances would thus be suboptimal. The API would also be awkward,
> > > >   as using the PXP for display would require usage of V4L2 in
> > > >   applications.  
> > > 
> > > So the video nodes would be sinks? I would expect overlays to be usable
> > > through KMS, I guess that would then not work, correct?
> 
> There would be sink video nodes for the PXP inputs, and one source video
> node for the PXP output. The PXP can be used stand-alone, in
> memory-to-memory mode, and V4L2 is a good fit for that.
> 
> > > > - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
> > > >   KMS planes. The PXP would only be used when available, and would be
> > > >   transparent to applications. This would however prevent using it
> > > >   separately from the display (to perform multi-pass alpha blending for
> > > >   instance).  
> > > 
> > > KMS planes are well defined and are well integrated with the KMS API, so
> > > I prefer this option. But is this compatible with the currently
> > > supported video use-case? E.g. could we make PXP available through V4L2
> > > and through DRM/mxsfb?
> 
> That's the issue, it's not easily doable. I think we could do so, but
> how to ensure mutual exclusion between the two APIs needs to be
> researched. I fear it will result in an awkward solution with fuzzy
> semantics. A module parameter could be an option, but wouldn't be very
> flexible.
> 
> > > Not sure what your use case is exactly, but when playing a video I
> > > wonder where is the higher value using PXP: Color conversion and scaling
> > > or compositing...? I would expect higher value in the former use case.
> 
> I think it's highly use-case-dependent.
> 
> > mind, with Wayland architecture, color conversion and scaling could be
> > at the same level/step as compositing, in the display server instead of
> > an application. Hence if the PXP capabilities were advertised as KMS
> > planes, there should be nothing to patch in Wayland-designed
> > applications to make use of them, assuming the applications did not
> > already rely on V4L2 M2M devices.
> > 
> > Would it not be possible to expose PXP through both uAPI interfaces? At
> > least KMS atomic's TEST_ONLY feature would make it easy to say "no" to
> > userspace if another bit of userspace already reserved the device via
> > e.g. V4L2.
> 
> We would also need to figure out how to do it the other way around,
> reporting properly through V4L2 that the device is busy. I think it's
> feasible, but I doubt it would result in anything usable for userspace.

We already have this needs for decoders with fixed number of streams.

> If the KMS device exposes multiple planes unconditionally and fails the
> atomic commit if the PXP is used through V4L2, I think it would be hard
> for Wayland to use this consistently. Given that I expect the PXP to be
> mostly used for display purpose I'm tempted to allocate it for display
> unconditionally, or, possibly, decide how to expose it through a module
> parameter.

It's a strange statement "mostly used for display purpose", considering
the upstream driver exist for video scaling and color conversion, and
no one have yet implement the "display purpose" driver.

My impression is that the complication is kernel specific (the fact we
very have two subsystems for the same IPs). Since software wise,
sharing and allocating resources seems pretty simple.

> 
> We have a similar situation on Renesas R-Car Gen3 platforms, with a
> memory-to-memory compositor called VSP. Some VSP instances are connected
> to the display controller, and we allocate them for display
> unconditionally. Other VSP instances are exposed as V4L2 devices. We
> haven't heard of anyone who wanted to use the display VSP instances for
> unrelated purposes. If such a use case arose, exposing those instances
> through V4L2 would just be a matter of flipping one bit in the driver
> (all the infrastructure is in place), which we would likely expose as a
> module parameter.

Re: Overlay support in the i.MX7 display

[Nicolas Dufresne]

Le lundi 04 novembre 2019 à 19:24 +0100, Daniel Vetter a écrit :
> On Mon, Nov 04, 2019 at 02:58:29PM +0200, Laurent Pinchart wrote:
> > Hello,
> > 
> > On Mon, Nov 04, 2019 at 10:09:47AM +0200, Pekka Paalanen wrote:
> > > On Sun, 03 Nov 2019 19:15:49 +0100 Stefan Agner wrote:
> > > > On 2019-11-01 09:43, Laurent Pinchart wrote:
> > > > > Hello,
> > > > > 
> > > > > I'm looking at the available options to support overlays in the display
> > > > > pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> > > > > overlays, the feature being implemented in the PXP. A driver for the PXP
> > > > > is available but only supports older SoCs whose PXP doesn't support
> > > > > overlays. This driver is implemented as a V4L2 mem2mem driver, which
> > > > > makes support of additional input channels impossible.  
> > > > 
> > > > Thanks for bringing this up, it is a topic I have wondered too:
> > > > Interaction between PXP and mxsfb.
> > > > 
> > > > I am not very familiar with the V4L2 subsystem so take my opinions with
> > > > a grain of salt.
> > > > 
> > > > > Here are the options I can envision:
> > > > > 
> > > > > - Extend the existing PXP driver to support multiple channels. This is
> > > > >   technically feasible, but will require moving away from the V4L2
> > > > >   mem2mem framework, which would break userspace. I don't think this
> > > > >   path could lead anywhere.
> > > > > 
> > > > > - Write a new PXP driver for the i.MX7, still using V4L2, but with
> > > > >   multiple video nodes. This would allow blending multiple layers, but
> > > > >   would require writing the output to memory, while the PXP has support
> > > > >   for direct connections to the LCDIF (through small SRAM buffers).
> > > > >   Performances would thus be suboptimal. The API would also be awkward,
> > > > >   as using the PXP for display would require usage of V4L2 in
> > > > >   applications.  
> > > > 
> > > > So the video nodes would be sinks? I would expect overlays to be usable
> > > > through KMS, I guess that would then not work, correct?
> > 
> > There would be sink video nodes for the PXP inputs, and one source video
> > node for the PXP output. The PXP can be used stand-alone, in
> > memory-to-memory mode, and V4L2 is a good fit for that.
> > 
> > > > > - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
> > > > >   KMS planes. The PXP would only be used when available, and would be
> > > > >   transparent to applications. This would however prevent using it
> > > > >   separately from the display (to perform multi-pass alpha blending for
> > > > >   instance).  
> > > > 
> > > > KMS planes are well defined and are well integrated with the KMS API, so
> > > > I prefer this option. But is this compatible with the currently
> > > > supported video use-case? E.g. could we make PXP available through V4L2
> > > > and through DRM/mxsfb?
> > 
> > That's the issue, it's not easily doable. I think we could do so, but
> > how to ensure mutual exclusion between the two APIs needs to be
> > researched. I fear it will result in an awkward solution with fuzzy
> > semantics. A module parameter could be an option, but wouldn't be very
> > flexible.
> > 
> > > > Not sure what your use case is exactly, but when playing a video I
> > > > wonder where is the higher value using PXP: Color conversion and scaling
> > > > or compositing...? I would expect higher value in the former use case.
> > 
> > I think it's highly use-case-dependent.
> > 
> > > mind, with Wayland architecture, color conversion and scaling could be
> > > at the same level/step as compositing, in the display server instead of
> > > an application. Hence if the PXP capabilities were advertised as KMS
> > > planes, there should be nothing to patch in Wayland-designed
> > > applications to make use of them, assuming the applications did not
> > > already rely on V4L2 M2M devices.
> > > 
> > > Would it not be possible to expose PXP through both uAPI interfaces? At
> > > least KMS atomic's TEST_ONLY feature would make it easy to say "no" to
> > > userspace if another bit of userspace already reserved the device via
> > > e.g. V4L2.
> > 
> > We would also need to figure out how to do it the other way around,
> > reporting properly through V4L2 that the device is busy. I think it's
> > feasible, but I doubt it would result in anything usable for userspace.
> > If the KMS device exposes multiple planes unconditionally and fails the
> > atomic commit if the PXP is used through V4L2, I think it would be hard
> > for Wayland to use this consistently. Given that I expect the PXP to be
> > mostly used for display purpose I'm tempted to allocate it for display
> > unconditionally, or, possibly, decide how to expose it through a module
> > parameter.
> 
> KMS should be fine if planes are missing, userspace is supposed to be able
> to cope with that. Not all userspace does, but welp.
>  
> I figured the bigger issue will be on the v4l side, since "device
> temporarily gone" is not something v4l understands as a concept?

m2m are different, since each time you open the device you get a new
instance. This is handled for decoders already in userspace, I really
don't see the difference. I'm guessing application that wants to
multiplex the PXP, will have to avoid the PXP planes for this HW, this
might be a little tricky and seems HW specific.

> 
> But even then having one device for userspace would be best I think, just
> a lot more reasonable (insert wish for unified video/display subsystem
> here).
> 
> > We have a similar situation on Renesas R-Car Gen3 platforms, with a
> > memory-to-memory compositor called VSP. Some VSP instances are connected
> > to the display controller, and we allocate them for display
> > unconditionally. Other VSP instances are exposed as V4L2 devices. We
> > haven't heard of anyone who wanted to use the display VSP instances for
> > unrelated purposes. If such a use case arose, exposing those instances
> > through V4L2 would just be a matter of flipping one bit in the driver
> > (all the infrastructure is in place), which we would likely expose as a
> > module parameter.
> 
> Hm yeah I guess we could just assign them, if the use-cases are clear-cut
> enough. Are you thinking of doing these links with dt (so at least it
> would be patchable)?
> -Daniel

Re: Overlay support in the i.MX7 display

[Nicolas Dufresne]

Le mardi 05 novembre 2019 à 10:17 +0100, Philipp Zabel a écrit :
> Hi Laurent,
> 
> On Fri, 2019-11-01 at 10:43 +0200, Laurent Pinchart wrote:
> > Hello,
> > 
> > I'm looking at the available options to support overlays in the display
> > pipeline of the i.MX7. The LCDIF itself unfortunaltey doesn't support
> > overlays, the feature being implemented in the PXP. A driver for the PXP
> > is available but only supports older SoCs whose PXP doesn't support
> > overlays. This driver is implemented as a V4L2 mem2mem driver, which
> > makes support of additional input channels impossible.
> > 
> > Here are the options I can envision:
> > 
> > - Extend the existing PXP driver to support multiple channels. This is
> >   technically feasible, but will require moving away from the V4L2
> >   mem2mem framework, which would break userspace. I don't think this
> >   path could lead anywhere.
> 
> I may be biased, but please don't break the V4L2 mem2mem usecase :)
> 
> > - Write a new PXP driver for the i.MX7, still using V4L2, but with
> >   multiple video nodes. This would allow blending multiple layers, but
> >   would require writing the output to memory, while the PXP has support
> >   for direct connections to the LCDIF (through small SRAM buffers).
> >   Performances would thus be suboptimal. The API would also be awkward,
> >   as using the PXP for display would require usage of V4L2 in
> >   applications.
> 
> I'm not sure V4L2 is the best API for multi-pass 2D composition,
> especially as the PXP is able to blit an overlay onto a background in
> place.

There was some userspace (GStreamer element) doing exactly that with
v4l2 m2m using the imx6 driver. The API was fine, even though fences
would have made programming it easier.

https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/308

(not merge as we don't have an agreement on kernel side, notably we
don't even have a way to control the blend function, so the result is
likely dependant on the use case the driver was written for)

The real limitation was that these IP usually supports more then just
blit/blend over another surface, and as you said, supports background.
And to support this use case, we'd need an m2m driver with multiple queues per direction. That was discussed in that last workshop at ELCE, and applies to other m2m IP like muxers and demuxers which exist on STB kind of SoC.

> 
> > - Extend the mxsfb driver with PXP support, and expose the PXP inputs as
> >   KMS planes. The PXP would only be used when available, and would be
> >   transparent to applications. This would however prevent using it
> >   separately from the display (to perform multi-pass alpha blending for
> >   instance).
> 
> For the SRAM block row buffer path to the LCDIF, I think the KMS plane
> abstraction is the way to go. The DRM and V4L2 drivers could be made to
> use a shared backend, such that only one of plane composition and V4L2
> scaling/CSC functions can work at the same time.
> 
> > What would be the best option going forward ? Would any of you, by any
> > chance, have already started work in this area ?
> 
> I have not worked on this.
> 
> regards
> Philipp
>