Re: [RFC v2 00/22] Add Support for Plane Color Lut and CSC features

[Pekka Paalanen <ppaalanen@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 9704 bytes --]

On Tue, 26 Oct 2021 11:02:31 -0400
Harry Wentland <harry.wentland@amd.com> wrote:

> On 2021-10-12 17:01, Shankar, Uma wrote:
> > 
> >   
> >> -----Original Message-----
> >> From: Pekka Paalanen <ppaalanen@gmail.com>
> >> Sent: Tuesday, October 12, 2021 5:25 PM
> >> To: Shankar, Uma <uma.shankar@intel.com>
> >> Cc: intel-gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org;
> >> harry.wentland@amd.com; ville.syrjala@linux.intel.com; brian.starkey@arm.com;
> >> sebastian@sebastianwick.net; Shashank.Sharma@amd.com
> >> Subject: Re: [RFC v2 00/22] Add Support for Plane Color Lut and CSC features
> >>
> >> On Tue,  7 Sep 2021 03:08:42 +0530
> >> Uma Shankar <uma.shankar@intel.com> wrote:
> >>  
> >>> This is how a typical display color hardware pipeline looks like:
> >>>  +-------------------------------------------+
> >>>  |                RAM                        |
> >>>  |  +------+    +---------+    +---------+   |
> >>>  |  | FB 1 |    |  FB 2   |    | FB N    |   |
> >>>  |  +------+    +---------+    +---------+   |
> >>>  +-------------------------------------------+
> >>>        |  Plane Color Hardware Block |
> >>> +--------------------------------------------+
> >>>  | +---v-----+   +---v-------+   +---v------+ |
> >>>  | | Plane A |   | Plane B   |   | Plane N  | |
> >>>  | | DeGamma |   | Degamma   |   | Degamma  | |
> >>>  | +---+-----+   +---+-------+   +---+------+ |
> >>>  |     |             |               |        |
> >>>  | +---v-----+   +---v-------+   +---v------+ |
> >>>  | |Plane A  |   | Plane B   |   | Plane N  | |
> >>>  | |CSC/CTM  |   | CSC/CTM   |   | CSC/CTM  | |
> >>>  | +---+-----+   +----+------+   +----+-----+ |
> >>>  |     |              |               |       |
> >>>  | +---v-----+   +----v------+   +----v-----+ |
> >>>  | | Plane A |   | Plane B   |   | Plane N  | |
> >>>  | | Gamma   |   | Gamma     |   | Gamma    | |
> >>>  | +---+-----+   +----+------+   +----+-----+ |
> >>>  |     |              |               |       |
> >>>  +--------------------------------------------+
> >>> +------v--------------v---------------v-------|
> >>> ||                                           ||
> >>> ||           Pipe Blender                    ||
> >>> +--------------------+------------------------+
> >>> |                    |                        |
> >>> |        +-----------v----------+             |
> >>> |        |  Pipe DeGamma        |             |
> >>> |        |                      |             |
> >>> |        +-----------+----------+             |
> >>> |                    |            Pipe Color  |
> >>> |        +-----------v----------+ Hardware    |
> >>> |        |  Pipe CSC/CTM        |             |
> >>> |        |                      |             |
> >>> |        +-----------+----------+             |
> >>> |                    |                        |
> >>> |        +-----------v----------+             |
> >>> |        |  Pipe Gamma          |             |
> >>> |        |                      |             |
> >>> |        +-----------+----------+             |
> >>> |                    |                        |
> >>> +---------------------------------------------+
> >>>                      |
> >>>                      v
> >>>                Pipe Output
> >>>
> >>> This patch series adds properties for plane color features. It adds
> >>> properties for degamma used to linearize data and CSC used for gamut
> >>> conversion. It also includes Gamma support used to again non-linearize
> >>> data as per panel supported color space. These can be utilize by user
> >>> space to convert planes from one format to another, one color space to
> >>> another etc.
> >>>
> >>> Userspace can take smart blending decisions and utilize these hardware
> >>> supported plane color features to get accurate color profile. The same
> >>> can help in consistent color quality from source to panel taking
> >>> advantage of advanced color features in hardware.
> >>>
> >>> These patches add the property interfaces and enable helper functions.
> >>> This series adds Intel's XE_LPD hw specific plane gamma feature. We
> >>> can build up and add other platform/hardware specific implementation
> >>> on top of this series.
> >>>
> >>> Credits: Special mention and credits to Ville Syrjala for coming up
> >>> with a design for this feature and inputs. This series is based on his
> >>> original design and idea.
> >>>
> >>> Note: Userspace support for this new UAPI will be done on Chrome in
> >>> alignment with weston and general opensource community.
> >>> Discussion ongoing with Harry Wentland, Pekka and community on color
> >>> pipeline and UAPI design. Harry's RFC below:  
> >>> https://patchwork.freedesktop.org/series/89506/>>>> We need to converge on a common UAPI interface which caters to all the  
> >>> modern color hardware pipelines.
> >>>
> >>> ToDo: State readout for this feature will be added next.
> >>>
> >>> v2: Added UAPI description and added change in the rfc section of
> >>> kernel Documentation folder  
> >>
> >> Hi,
> >>
> >> thank you for this. I do believe the KMS UAPI should expose what hardware can do
> >> (prescribed operations) rather than how they would be often used (to realize a
> >> conversion from one space description to another). This proposal fits quite nicely
> >> with what I have envisioned for Weston.  
> >   
> 
> It's taken me a while but I am starting to agree with the prescriptive approach to
> expose HW functionality. One thing we'll want to be careful of is to make sure
> this isn't tied to specific HW more than it needs to be. I'll comment in other
> places of this patchset to elaborate.
> 
> What's making me come around, i.e. change from a prescriptive (these are the
> input/output/blending spaces/formats) to a descriptive (these are the LUTs and
> CTMs) approach?
> 
> 1) The prescriptive way has no good way of dealing with gamut and tone mapping.
>    To do so we would need explicit OOTFs and CTMs or 3D LUTs anyways.
> 
> 2) The prescriptive way provides no semblance of guarantee that transforms
>    are equivalent when the compositor uses shaders transforms and composition
>    vs when the compositor uses KMS transforms and composition.
> 
> 3) Policy about treatment of surfaces/planes and blending is best left with
>    the compositor (for the above reasons).

Hi Harry,

I think we might have confusion there about prescriptive vs.
descriptive, but I understand what you mean, and yes, those are my
points too.

> >> I mainly went over the big picture by commenting in detail on the proposal
> >> document, and not looking too carefully at the other documentation or UAPI details
> >> at this time.  
> > 
> > Thanks Pekka for the feedback.
> >   
> >> Unfortunately I was unable to decipher how userspace is supposed to use the
> >> XE_LPD special gamma features.  
> > 
> > I will include the details on how userspace should actually get this through a sample
> > IGT reference, that should help make this clear.
> >   
> 
> It looks like with your current definitions each userspace compositor (Weston, kwin,
> mutter, wlroots, Chrome's compositor, Android's compositor, etc.) would need to learn
> how to program the XE_LPD LUTs as well as AMD LUTs. Would these definitions change
> in future Intel HW generations? Would this mean all compositors would need to learn
> again how to program the future LUT format?
> 
> Other options would be to give userspace a generic LUT with 4k FP16 entries and then
> re-map that to the HW LUT in the kernel driver.

Do you want to run the optimisation algorithm to map the generic LUT to
hardware specific optimised representation in the kernel driver?
Wouldn't that be too much?

To me this seems like the perfect starting point for a libcamera
equivalent to KMS. Feed the generic LUT to the lib, and get whatever
the hardware-driver-specific KMS UAPI needs. Perhaps even with an error
measure, so that a compositor can evaluate whether the mapping is
accurate enough or if it needs to fall back.

> I might be missing some of the nuances of the XE_LPD LUT but it seems to me that the
> main difference between different PWL implementations is the distribution of the
> points used to define the LUT. Maybe a more generic PWL implementation could have
> a kernel driver report one (or more) PWL point distributions. We could encode these
> as enums and pre-defined arrays in a UAPI header. That way the compositor could have
> a single, generic implementation of programming PWL in FP16 and the kernel driver
> would only need to remap the FP16 to the HW-internal format, which is a trivial
> conversion. Using this approach compositors would implement PWL support once and won't
> have to touch it again in the future. Is there anything that would make this approach
> a bad idea for Intel HW (or other HW)? (Credit for this idea goes to Vitaly)

I think that is also a workable solution. If that happens, it seems
likely that a library for that optimisation will appear.

It has just one shortcoming: the compositor has no idea of the
precision as there is no way to get an error measure for how much the
FP16->internal conversion loses precision.

But maybe that is never significant? Or could we have some precision
guarantees otherwise?

I'm thinking of hard-core professional color users who wouldn't
normally trust off-loading to KMS at all, at least not on-demand
off-loading where you are never sure if you are looking at SW or KMS
composition.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]