[RFC PATCH 0/3] A drm_plane API to support HDR planes

[RFC PATCH 0/3] A drm_plane API to support HDR planes

[Harry Wentland]

## Introduction

We are looking to enable HDR support for a couple of single-plane and multi-plane scenarios. To do this effectively we recommend new interfaces to drm_plane. Below I'll give a bit of background on HDR and why we propose these interfaces.


## Defining a pixel's luminance

Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms.

Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows.

A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits.

Using a content that's defined in PQ space we can approximate the real world in a much better way.

Here are some examples of real-life objects and their approximate luminance values:

| Object            | Luminance in nits |
| ----------------- | ----------------- |
| Sun               | 1.6 million       |
| Fluorescent light | 10,000            |
| Highlights        | 1,000 - sunlight  |
| White Objects     | 250 - 1,000       |
| Typical objects   | 1 - 250           |
| Shadows           | 0.01 - 1          |
| Ultra Blacks      | 0 - 0.0005        |


## Describing the luminance space

**We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are:

| EOTF      | Description                                                               |
| --------- |:------------------------------------------------------------------------- |
| Gamma 2.2 | a simple 2.2 gamma                                                        |
| sRGB      | 2.4 gamma with small initial linear section                               |
| PQ 2084   | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support |
| Linear    | Linear relationship between pixel value and luminance value               |


## Mastering Luminances

Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.

| Display                  | Luminance range in nits |
| ------------------------ | ----------------------- |
| Typical PC display       | 0.3 - 200               |
| Excellent LCD HDTV       | 0.3 - 400               |
| HDR LCD w/ local dimming | 0.05 - 1,500            |

Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.

The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.

It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.


## SDR Luminance

Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable.

**We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space.


## Let There Be Color

So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value.

A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue.

Examples of color spaces are:

| Color Space | Description                                |
| ----------- | ------------------------------------------ |
| BT 601      | similar to BT 709                          |
| BT 709      | used by sRGB content; ~53% of BT 2020      |
| DCI-P3      | used by most HDR displays; ~72% of BT 2020 |
| BT 2020     | standard for most HDR content              |

The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane. 

**We propose to add definitions for the RGB variants of the BT color spaces.**


## Color Primaries and White Point

Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut.


## Displays and Tonemapping

External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.

Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.


## Pixel Formats

The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF.


## Proposed use-cases

Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases:

**One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set.

**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.

**One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane.

In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed.


## Summary of proposed interface changes

per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color space definitions
- new transfer function property
- new SDR maximum white level property


## References

[1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer


## Further Reading

https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst
http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf
https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf


Bhawanpreet Lakha (3):
  drm/color: Add RGB Color encodings
  drm/color: Add Color transfer functions for HDR/SDR
  drm/color: Add sdr boost property

 .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |  4 +-
 .../gpu/drm/arm/display/komeda/komeda_plane.c |  4 +-
 drivers/gpu/drm/arm/malidp_planes.c           |  4 +-
 drivers/gpu/drm/armada/armada_overlay.c       |  4 +-
 drivers/gpu/drm/drm_atomic_uapi.c             |  8 ++
 drivers/gpu/drm/drm_color_mgmt.c              | 84 +++++++++++++++++--
 drivers/gpu/drm/i915/display/intel_sprite.c   |  4 +-
 .../drm/i915/display/skl_universal_plane.c    |  4 +-
 drivers/gpu/drm/nouveau/dispnv04/overlay.c    |  4 +-
 drivers/gpu/drm/omapdrm/omap_plane.c          |  4 +-
 drivers/gpu/drm/sun4i/sun8i_vi_layer.c        |  4 +-
 drivers/gpu/drm/tidss/tidss_plane.c           |  6 +-
 include/drm/drm_color_mgmt.h                  | 25 +++++-
 include/drm/drm_plane.h                       | 30 +++++++
 14 files changed, 173 insertions(+), 16 deletions(-)

-- 
2.31.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Harry Wentland]

From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>

Add the following color encodings
- RGB versions for BT601, BT709, BT2020
- DCI-P3: Used for digital movies

Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
Signed-off-by: Harry Wentland <harry.wentland@amd.com>
---
 drivers/gpu/drm/drm_color_mgmt.c | 4 ++++
 include/drm/drm_color_mgmt.h     | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
index bb14f488c8f6..a183ebae2941 100644
--- a/drivers/gpu/drm/drm_color_mgmt.c
+++ b/drivers/gpu/drm/drm_color_mgmt.c
@@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = {
 	[DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr",
 	[DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr",
 	[DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
+	[DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
+	[DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
+	[DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
+	[DRM_COLOR_P3] = "DCI-P3",
 };
 
 static const char * const color_range_name[] = {
diff --git a/include/drm/drm_color_mgmt.h b/include/drm/drm_color_mgmt.h
index 81c298488b0c..3043dd73480c 100644
--- a/include/drm/drm_color_mgmt.h
+++ b/include/drm/drm_color_mgmt.h
@@ -78,6 +78,10 @@ enum drm_color_encoding {
 	DRM_COLOR_YCBCR_BT601,
 	DRM_COLOR_YCBCR_BT709,
 	DRM_COLOR_YCBCR_BT2020,
+	DRM_COLOR_RGB_BT601,
+	DRM_COLOR_RGB_BT709,
+	DRM_COLOR_RGB_BT2020,
+	DRM_COLOR_P3,
 	DRM_COLOR_ENCODING_MAX,
 };
 
-- 
2.31.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Ville Syrjälä]

On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
> From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
> 
> Add the following color encodings
> - RGB versions for BT601, BT709, BT2020
> - DCI-P3: Used for digital movies
> 
> Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
> Signed-off-by: Harry Wentland <harry.wentland@amd.com>
> ---
>  drivers/gpu/drm/drm_color_mgmt.c | 4 ++++
>  include/drm/drm_color_mgmt.h     | 4 ++++
>  2 files changed, 8 insertions(+)
> 
> diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
> index bb14f488c8f6..a183ebae2941 100644
> --- a/drivers/gpu/drm/drm_color_mgmt.c
> +++ b/drivers/gpu/drm/drm_color_mgmt.c
> @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = {
>  	[DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr",
>  	[DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr",
>  	[DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
> +	[DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
> +	[DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
> +	[DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
> +	[DRM_COLOR_P3] = "DCI-P3",

These are a totally different thing than the YCbCr stuff.
The YCbCr stuff just specifies the YCbCr<->RGB converison matrix,
whereas these are I guess supposed to specify the primaries/whitepoint?
But without specifying what we're converting *to* these mean absolutely
nothing. Ie. I don't think they belong in this property.

The previous proposals around this topic have suggested a new
property to specify a conversion matrix either explicitly, or
via a separate enum (which would specify both the src and dst
colorspaces). I've always argued the enum approach is needed
anyway since not all hardware has a programmable matrix for
this. But a fully programmable matrix could be nice for tone
mapping purposes/etc, so we may want to make sure both are
possible.

As for the transfer func, the proposals so far have mostly just
been to expose a programmable degamma/gamma LUTs for each plane.
But considering how poor the current gamma uapi is we've thrown
around some ideas how to allow the kernel to properly expose the
hw capabilities. This is one of those ideas:
https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html
I think that basic idea could be also be extended to allow fixed
curves in case the hw doesn't have a fully programmable LUT. But
dunno if that's relevant for your hw.

-- 
Ville Syrjälä
Intel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Harry Wentland]

On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
> On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
>> From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
>>
>> Add the following color encodings
>> - RGB versions for BT601, BT709, BT2020
>> - DCI-P3: Used for digital movies
>>
>> Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
>> Signed-off-by: Harry Wentland <harry.wentland@amd.com>
>> ---
>>   drivers/gpu/drm/drm_color_mgmt.c | 4 ++++
>>   include/drm/drm_color_mgmt.h     | 4 ++++
>>   2 files changed, 8 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
>> index bb14f488c8f6..a183ebae2941 100644
>> --- a/drivers/gpu/drm/drm_color_mgmt.c
>> +++ b/drivers/gpu/drm/drm_color_mgmt.c
>> @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = {
>>   	[DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr",
>>   	[DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr",
>>   	[DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
>> +	[DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
>> +	[DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
>> +	[DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
>> +	[DRM_COLOR_P3] = "DCI-P3",
> 
> These are a totally different thing than the YCbCr stuff.
> The YCbCr stuff just specifies the YCbCr<->RGB converison matrix,
> whereas these are I guess supposed to specify the primaries/whitepoint?
> But without specifying what we're converting *to* these mean absolutely
> nothing. Ie. I don't think they belong in this property.
> 

If this is the intention I don't see it documented.

I might have overlooked something but do we have a way to explicitly 
specify today what *to* format the YCbCr color encodings convert into? 
Would that be a combination of the output color encoding specified via 
colorspace_property and the color space encoded in the primaries and 
whitepoint of the hdr_output_metadata?

Fundamentally I don't see how the use of this property differs, whether 
you translate from YCbCr or from RGB. In either case you're converting 
from the defined input color space and pixel format to an output color 
space and pixel format.

> The previous proposals around this topic have suggested a new
> property to specify a conversion matrix either explicitly, or
> via a separate enum (which would specify both the src and dst
> colorspaces). I've always argued the enum approach is needed
> anyway since not all hardware has a programmable matrix for
> this. But a fully programmable matrix could be nice for tone
> mapping purposes/etc, so we may want to make sure both are
> possible.
> 
> As for the transfer func, the proposals so far have mostly just
> been to expose a programmable degamma/gamma LUTs for each plane.
> But considering how poor the current gamma uapi is we've thrown
> around some ideas how to allow the kernel to properly expose the
> hw capabilities. This is one of those ideas:
> https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html>> I think that basic idea could be also be extended to allow fixed
> curves in case the hw doesn't have a fully programmable LUT. But
> dunno if that's relevant for your hw.
> 

The problem with exposing gamma, whether per-plane or per-crtc, is that 
it is hard to define an API that works for all the HW out there. The 
capabilities for different HW differ a lot, not just between vendors but 
also between generations of a vendor's HW.

Another reason I'm proposing to define the color space (and gamma) of a 
plane is to make this explicit. Up until the color space and gamma of a 
plane or framebuffer are not well defined, which leads to drivers 
assuming the color space and gamma of a buffer (for blending and other 
purposes) and might lead to sub-optimal outcomes.

Harry

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Ville Syrjälä]

On Mon, Apr 26, 2021 at 02:56:26PM -0400, Harry Wentland wrote:
> On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
> > On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
> >> From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
> >>
> >> Add the following color encodings
> >> - RGB versions for BT601, BT709, BT2020
> >> - DCI-P3: Used for digital movies
> >>
> >> Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
> >> Signed-off-by: Harry Wentland <harry.wentland@amd.com>
> >> ---
> >>   drivers/gpu/drm/drm_color_mgmt.c | 4 ++++
> >>   include/drm/drm_color_mgmt.h     | 4 ++++
> >>   2 files changed, 8 insertions(+)
> >>
> >> diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
> >> index bb14f488c8f6..a183ebae2941 100644
> >> --- a/drivers/gpu/drm/drm_color_mgmt.c
> >> +++ b/drivers/gpu/drm/drm_color_mgmt.c
> >> @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = {
> >>   	[DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr",
> >>   	[DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr",
> >>   	[DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
> >> +	[DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
> >> +	[DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
> >> +	[DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
> >> +	[DRM_COLOR_P3] = "DCI-P3",
> > 
> > These are a totally different thing than the YCbCr stuff.
> > The YCbCr stuff just specifies the YCbCr<->RGB converison matrix,
> > whereas these are I guess supposed to specify the primaries/whitepoint?
> > But without specifying what we're converting *to* these mean absolutely
> > nothing. Ie. I don't think they belong in this property.
> > 
> 
> If this is the intention I don't see it documented.
> 
> I might have overlooked something but do we have a way to explicitly 
> specify today what *to* format the YCbCr color encodings convert into? 

These just specific which YCbCr<->RGB matrix to use as specificed
in the relevant standards. The primaries/whitepoint/etc. don't
change at all.

> Would that be a combination of the output color encoding specified via 
> colorspace_property and the color space encoded in the primaries and 
> whitepoint of the hdr_output_metadata?

Those propertis only affect the infoframes. They don't apply any
color processing to the data.

> 
> Fundamentally I don't see how the use of this property differs, whether 
> you translate from YCbCr or from RGB. In either case you're converting 
> from the defined input color space and pixel format to an output color 
> space and pixel format.

The gamut does not change when you do YCbCr<->RGB conversion.

> 
> > The previous proposals around this topic have suggested a new
> > property to specify a conversion matrix either explicitly, or
> > via a separate enum (which would specify both the src and dst
> > colorspaces). I've always argued the enum approach is needed
> > anyway since not all hardware has a programmable matrix for
> > this. But a fully programmable matrix could be nice for tone
> > mapping purposes/etc, so we may want to make sure both are
> > possible.
> > 
> > As for the transfer func, the proposals so far have mostly just
> > been to expose a programmable degamma/gamma LUTs for each plane.
> > But considering how poor the current gamma uapi is we've thrown
> > around some ideas how to allow the kernel to properly expose the
> > hw capabilities. This is one of those ideas:
> > https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html>> I think that basic idea could be also be extended to allow fixed
> > curves in case the hw doesn't have a fully programmable LUT. But
> > dunno if that's relevant for your hw.
> > 
> 
> The problem with exposing gamma, whether per-plane or per-crtc, is that 
> it is hard to define an API that works for all the HW out there. The 
> capabilities for different HW differ a lot, not just between vendors but 
> also between generations of a vendor's HW.
> 
> Another reason I'm proposing to define the color space (and gamma) of a 
> plane is to make this explicit. Up until the color space and gamma of a 
> plane or framebuffer are not well defined, which leads to drivers 
> assuming the color space and gamma of a buffer (for blending and other 
> purposes) and might lead to sub-optimal outcomes.

The current state is that things just get passed through as is
(apart from the crtc LUTs/CTM).

-- 
Ville Syrjälä
Intel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 7720 bytes --]

On Mon, 26 Apr 2021 22:08:55 +0300
Ville Syrjälä <ville.syrjala@linux.intel.com> wrote:

> On Mon, Apr 26, 2021 at 02:56:26PM -0400, Harry Wentland wrote:
> > On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:  
> > > On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:  
> > >> From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
> > >>
> > >> Add the following color encodings
> > >> - RGB versions for BT601, BT709, BT2020
> > >> - DCI-P3: Used for digital movies
> > >>
> > >> Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
> > >> Signed-off-by: Harry Wentland <harry.wentland@amd.com>
> > >> ---
> > >>   drivers/gpu/drm/drm_color_mgmt.c | 4 ++++
> > >>   include/drm/drm_color_mgmt.h     | 4 ++++
> > >>   2 files changed, 8 insertions(+)
> > >>
> > >> diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
> > >> index bb14f488c8f6..a183ebae2941 100644
> > >> --- a/drivers/gpu/drm/drm_color_mgmt.c
> > >> +++ b/drivers/gpu/drm/drm_color_mgmt.c
> > >> @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = {
> > >>   	[DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr",
> > >>   	[DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr",
> > >>   	[DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
> > >> +	[DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
> > >> +	[DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
> > >> +	[DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
> > >> +	[DRM_COLOR_P3] = "DCI-P3",  
> > > 
> > > These are a totally different thing than the YCbCr stuff.
> > > The YCbCr stuff just specifies the YCbCr<->RGB converison matrix,
> > > whereas these are I guess supposed to specify the primaries/whitepoint?
> > > But without specifying what we're converting *to* these mean absolutely
> > > nothing. Ie. I don't think they belong in this property.
> > >   
> > 
> > If this is the intention I don't see it documented.
> > 
> > I might have overlooked something but do we have a way to explicitly 
> > specify today what *to* format the YCbCr color encodings convert into?   
> 
> These just specific which YCbCr<->RGB matrix to use as specificed
> in the relevant standards. The primaries/whitepoint/etc. don't
> change at all.

Ville is correct here.

> > Would that be a combination of the output color encoding specified via 
> > colorspace_property and the color space encoded in the primaries and 
> > whitepoint of the hdr_output_metadata?  

Conversion between YCbCR and RGB is not a color space conversion in the
sense of color spaces (chromaticity of primaries and white point). It
is a color model conversion or a color encoding conversion more like.

A benefit of YCbCr is that you can use less bandwidth to transmit the
same image and people won't realise that you lost anything: chroma
sub-sampling. Sub-sampling with RGB wouldn't work that well. It's a
lossy compression technique, but different standards use different
compression algorithms (well, matrices) to balance what gets lost.

> Those propertis only affect the infoframes. They don't apply any
> color processing to the data.

Indeed.

An example:

You start with YUV video you want to display. That means you have YCbCr
data using color space X and EOTF Foo. When you convert that to RGB,
the RGB data still has color space X and EOTF Foo. Then you use the
infoframe to tell your monitor that the data is in color space X and
using EOTF Foo.

At no point in that pipeline there is a color space transformation,
until the data actually reaches the monitor which may do magic things
to map color space X and EOTF Foo into what it can actually make as
light.

Or as with the traditional way, you don't care what color space or EOTF
your video uses or your monitor has. You just hope they are close
enough to look good enough that people don't see anything wrong. Close
your eyes and sing a happy song. With HDR and WCG, that totally breaks
down.

> > Fundamentally I don't see how the use of this property differs, whether 
> > you translate from YCbCr or from RGB. In either case you're converting 
> > from the defined input color space and pixel format to an output color 
> > space and pixel format.  
> 
> The gamut does not change when you do YCbCr<->RGB conversion.

Right. Neither does dynamic range.

> > > The previous proposals around this topic have suggested a new
> > > property to specify a conversion matrix either explicitly, or
> > > via a separate enum (which would specify both the src and dst
> > > colorspaces). I've always argued the enum approach is needed
> > > anyway since not all hardware has a programmable matrix for
> > > this. But a fully programmable matrix could be nice for tone
> > > mapping purposes/etc, so we may want to make sure both are
> > > possible.
> > > 
> > > As for the transfer func, the proposals so far have mostly just
> > > been to expose a programmable degamma/gamma LUTs for each plane.
> > > But considering how poor the current gamma uapi is we've thrown
> > > around some ideas how to allow the kernel to properly expose the
> > > hw capabilities. This is one of those ideas:
> > > https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html>> I think that basic idea could be also be extended to allow fixed
> > > curves in case the hw doesn't have a fully programmable LUT. But
> > > dunno if that's relevant for your hw.
> > >   
> > 
> > The problem with exposing gamma, whether per-plane or per-crtc, is that 
> > it is hard to define an API that works for all the HW out there. The 
> > capabilities for different HW differ a lot, not just between vendors but 
> > also between generations of a vendor's HW.
> > 
> > Another reason I'm proposing to define the color space (and gamma) of a 
> > plane is to make this explicit. Up until the color space and gamma of a 
> > plane or framebuffer are not well defined, which leads to drivers 
> > assuming the color space and gamma of a buffer (for blending and other 
> > purposes) and might lead to sub-optimal outcomes.  
> 
> The current state is that things just get passed through as is
> (apart from the crtc LUTs/CTM).

Right. I would claim that the kernel does not even want to know about
color spaces or EOTFs. Instead, the kernel should offer userspace ways
to program the hardware to do the color *transformations* it wants to
do. The color_encoding KMS property is already like this: it defines
the conversion matrix, not what the input or output are.

Infoframes being sent to displays are a different thing. They just tell
the monitor what kind of image data userspace has configured KMS to
send it, but does not change what KMS actually does with pixels.


Also, please, let's talk about EOTF and EOTF^-1 instead of gamma when
appropriate.

Electro-optical transfer function (EOTF) is very clear in what it
means: it is the mapping from electrical values (the non-linear pixel
values you are used to, good for storage and transmission) to optical
values (values that are linear in light intensity, therefore good for
things like blending and filtering).

Gamma is kind of the same, but when you use it in sentences it easily
becomes ambiguous. Like if you have "gamma corrected pixels", what does
that mean. Are they electrical values or optical values or maybe
electrical values with a different EOTF. What EOTF.

However, in KMS "gamma LUT" is kind of standardised terminology, and it
does not need to be an EOTF or inverse-EOTF. One can use a gamma LUT
for EETF, mapping from one EOTF to another, e.g. from data with content
EOTF to data with monitor EOTF.


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Sebastian Wick]

On 2021-04-26 20:56, Harry Wentland wrote:
> On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
>> On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
>>> From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
>>> 
>>> Add the following color encodings
>>> - RGB versions for BT601, BT709, BT2020
>>> - DCI-P3: Used for digital movies
>>> 
>>> Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
>>> Signed-off-by: Harry Wentland <harry.wentland@amd.com>
>>> ---
>>>   drivers/gpu/drm/drm_color_mgmt.c | 4 ++++
>>>   include/drm/drm_color_mgmt.h     | 4 ++++
>>>   2 files changed, 8 insertions(+)
>>> 
>>> diff --git a/drivers/gpu/drm/drm_color_mgmt.c 
>>> b/drivers/gpu/drm/drm_color_mgmt.c
>>> index bb14f488c8f6..a183ebae2941 100644
>>> --- a/drivers/gpu/drm/drm_color_mgmt.c
>>> +++ b/drivers/gpu/drm/drm_color_mgmt.c
>>> @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] 
>>> = {
>>>   	[DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr",
>>>   	[DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr",
>>>   	[DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
>>> +	[DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
>>> +	[DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
>>> +	[DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
>>> +	[DRM_COLOR_P3] = "DCI-P3",
>> 
>> These are a totally different thing than the YCbCr stuff.
>> The YCbCr stuff just specifies the YCbCr<->RGB converison matrix,
>> whereas these are I guess supposed to specify the 
>> primaries/whitepoint?
>> But without specifying what we're converting *to* these mean 
>> absolutely
>> nothing. Ie. I don't think they belong in this property.
>> 
> 
> If this is the intention I don't see it documented.
> 
> I might have overlooked something but do we have a way to explicitly
> specify today what *to* format the YCbCr color encodings convert into?
> Would that be a combination of the output color encoding specified via
> colorspace_property and the color space encoded in the primaries and
> whitepoint of the hdr_output_metadata?
> 
> Fundamentally I don't see how the use of this property differs,
> whether you translate from YCbCr or from RGB. In either case you're
> converting from the defined input color space and pixel format to an
> output color space and pixel format.
> 
>> The previous proposals around this topic have suggested a new
>> property to specify a conversion matrix either explicitly, or
>> via a separate enum (which would specify both the src and dst
>> colorspaces). I've always argued the enum approach is needed
>> anyway since not all hardware has a programmable matrix for
>> this. But a fully programmable matrix could be nice for tone
>> mapping purposes/etc, so we may want to make sure both are
>> possible.
>> 
>> As for the transfer func, the proposals so far have mostly just
>> been to expose a programmable degamma/gamma LUTs for each plane.
>> But considering how poor the current gamma uapi is we've thrown
>> around some ideas how to allow the kernel to properly expose the
>> hw capabilities. This is one of those ideas:
>> https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html>> 
>> I think that basic idea could be also be extended to allow fixed
>> curves in case the hw doesn't have a fully programmable LUT. But
>> dunno if that's relevant for your hw.
>> 
> 
> The problem with exposing gamma, whether per-plane or per-crtc, is
> that it is hard to define an API that works for all the HW out there.
> The capabilities for different HW differ a lot, not just between
> vendors but also between generations of a vendor's HW.

Introducing another API if hardware is sufficiently different doesn't
seem like the worst idea. At least it sounds a lot more tractable than
teaching the kernel about all the different use cases, opinions and
nuances that arise from color management.

In the end generic user space must always be able to fall back to
software so the worst case is that it won't be able to offload an
operation if it doesn't know about a new API.

> Another reason I'm proposing to define the color space (and gamma) of
> a plane is to make this explicit. Up until the color space and gamma
> of a plane or framebuffer are not well defined, which leads to drivers
> assuming the color space and gamma of a buffer (for blending and other
> purposes) and might lead to sub-optimal outcomes.

Blending only is "correct" with linear light so that property of the
color space is important. However, why does the kernel have to be
involved here? As long as user space knows that for correct blending the
data must represent linear light and knows when in the pipeline blending
happens it can make sure that the data at that point in the pipeline
contains linear light.

What other purposes are there?

In general I agree with the others that user space only wants a pipeline
of transformations where the mechanism, the order and ideally the
precision is defined.

> Harry
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Harry Wentland]

On 2021-04-30 8:53 p.m., Sebastian Wick wrote:
> On 2021-04-26 20:56, Harry Wentland wrote:
>> On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
>>> On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
>>>> From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
>>>>
>>>> Add the following color encodings
>>>> - RGB versions for BT601, BT709, BT2020
>>>> - DCI-P3: Used for digital movies
>>>>
>>>> Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
>>>> Signed-off-by: Harry Wentland <harry.wentland@amd.com>
>>>> ---
>>>>   drivers/gpu/drm/drm_color_mgmt.c | 4 ++++
>>>>   include/drm/drm_color_mgmt.h     | 4 ++++
>>>>   2 files changed, 8 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
>>>> index bb14f488c8f6..a183ebae2941 100644
>>>> --- a/drivers/gpu/drm/drm_color_mgmt.c
>>>> +++ b/drivers/gpu/drm/drm_color_mgmt.c
>>>> @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = {
>>>>       [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr",
>>>>       [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr",
>>>>       [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
>>>> +    [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
>>>> +    [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
>>>> +    [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
>>>> +    [DRM_COLOR_P3] = "DCI-P3",
>>>
>>> These are a totally different thing than the YCbCr stuff.
>>> The YCbCr stuff just specifies the YCbCr<->RGB converison matrix,
>>> whereas these are I guess supposed to specify the primaries/whitepoint?
>>> But without specifying what we're converting *to* these mean absolutely
>>> nothing. Ie. I don't think they belong in this property.
>>>
>>
>> If this is the intention I don't see it documented.
>>
>> I might have overlooked something but do we have a way to explicitly
>> specify today what *to* format the YCbCr color encodings convert into?
>> Would that be a combination of the output color encoding specified via
>> colorspace_property and the color space encoded in the primaries and
>> whitepoint of the hdr_output_metadata?
>>
>> Fundamentally I don't see how the use of this property differs,
>> whether you translate from YCbCr or from RGB. In either case you're
>> converting from the defined input color space and pixel format to an
>> output color space and pixel format.
>>
>>> The previous proposals around this topic have suggested a new
>>> property to specify a conversion matrix either explicitly, or
>>> via a separate enum (which would specify both the src and dst
>>> colorspaces). I've always argued the enum approach is needed
>>> anyway since not all hardware has a programmable matrix for
>>> this. But a fully programmable matrix could be nice for tone
>>> mapping purposes/etc, so we may want to make sure both are
>>> possible.
>>>
>>> As for the transfer func, the proposals so far have mostly just
>>> been to expose a programmable degamma/gamma LUTs for each plane.
>>> But considering how poor the current gamma uapi is we've thrown
>>> around some ideas how to allow the kernel to properly expose the
>>> hw capabilities. This is one of those ideas:
>>> https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html I think that basic idea could be also be extended to allow fixed
>>> curves in case the hw doesn't have a fully programmable LUT. But
>>> dunno if that's relevant for your hw.
>>>
>>
>> The problem with exposing gamma, whether per-plane or per-crtc, is
>> that it is hard to define an API that works for all the HW out there.
>> The capabilities for different HW differ a lot, not just between
>> vendors but also between generations of a vendor's HW.
> 
> Introducing another API if hardware is sufficiently different doesn't
> seem like the worst idea. At least it sounds a lot more tractable than
> teaching the kernel about all the different use cases, opinions and
> nuances that arise from color management.
> 
> In the end generic user space must always be able to fall back to
> software so the worst case is that it won't be able to offload an
> operation if it doesn't know about a new API.
> 
>> Another reason I'm proposing to define the color space (and gamma) of
>> a plane is to make this explicit. Up until the color space and gamma
>> of a plane or framebuffer are not well defined, which leads to drivers
>> assuming the color space and gamma of a buffer (for blending and other
>> purposes) and might lead to sub-optimal outcomes.
> 
> Blending only is "correct" with linear light so that property of the
> color space is important. However, why does the kernel have to be
> involved here? As long as user space knows that for correct blending the
> data must represent linear light and knows when in the pipeline blending
> happens it can make sure that the data at that point in the pipeline
> contains linear light.
> 

The only reason the kernel needs to be involved is to take full advantage
of the available HW without requiring KMS clients to be aware of
the difference in display HW.

Harry

> What other purposes are there?
> 
> In general I agree with the others that user space only wants a pipeline
> of transformations where the mechanism, the order and ideally the
> precision is defined.
> 
>> Harry
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx>

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

On Fri, 14 May 2021 17:04:51 -0400
Harry Wentland <harry.wentland@amd.com> wrote:

> On 2021-04-30 8:53 p.m., Sebastian Wick wrote:
> > On 2021-04-26 20:56, Harry Wentland wrote:  

...

> >> Another reason I'm proposing to define the color space (and gamma) of
> >> a plane is to make this explicit. Up until the color space and gamma
> >> of a plane or framebuffer are not well defined, which leads to drivers
> >> assuming the color space and gamma of a buffer (for blending and other
> >> purposes) and might lead to sub-optimal outcomes.  
> > 
> > Blending only is "correct" with linear light so that property of the
> > color space is important. However, why does the kernel have to be
> > involved here? As long as user space knows that for correct blending the
> > data must represent linear light and knows when in the pipeline blending
> > happens it can make sure that the data at that point in the pipeline
> > contains linear light.
> >   
> 
> The only reason the kernel needs to be involved is to take full advantage
> of the available HW without requiring KMS clients to be aware of
> the difference in display HW.

Can you explain in more tangible examples, why you think so, please?

Is it because hardware does not fit the KMS UAPI model of the abstract
pixel pipeline?

Or is it because you have fixed-function hardware elements that you can
only make use of when userspace uses an enum-based UAPI?

I would totally agree that the driver does not want to be analysing LUT
entries to decipher if it could use a fixed-function element or not. It
would introduce uncertainty in the UAPI. So fixed-function elements
would need their own properties, but I don't know if that is feasible
as generic UAPI or if it should be driver-specific (and so left unused
by generic userspace).


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Harry Wentland]

On 2021-05-17 4:34 a.m., Pekka Paalanen wrote:
> On Fri, 14 May 2021 17:04:51 -0400
> Harry Wentland <harry.wentland@amd.com> wrote:
> 
>> On 2021-04-30 8:53 p.m., Sebastian Wick wrote:
>>> On 2021-04-26 20:56, Harry Wentland wrote:  
> 
> ...
> 
>>>> Another reason I'm proposing to define the color space (and gamma) of
>>>> a plane is to make this explicit. Up until the color space and gamma
>>>> of a plane or framebuffer are not well defined, which leads to drivers
>>>> assuming the color space and gamma of a buffer (for blending and other
>>>> purposes) and might lead to sub-optimal outcomes.  
>>>
>>> Blending only is "correct" with linear light so that property of the
>>> color space is important. However, why does the kernel have to be
>>> involved here? As long as user space knows that for correct blending the
>>> data must represent linear light and knows when in the pipeline blending
>>> happens it can make sure that the data at that point in the pipeline
>>> contains linear light.
>>>   
>>
>> The only reason the kernel needs to be involved is to take full advantage
>> of the available HW without requiring KMS clients to be aware of
>> the difference in display HW.
> 
> Can you explain in more tangible examples, why you think so, please?
> 
> Is it because hardware does not fit the KMS UAPI model of the abstract
> pixel pipeline?
> 

I'd wager no HW is designed to meet KMS UAPI, rather KMS UAPI is designed
to abstract HW.

> Or is it because you have fixed-function hardware elements that you can
> only make use of when userspace uses an enum-based UAPI?
> 

One example is our degamma on our latest generation HW, where we have
fixed-function "degamma" (rather de-EOTF):

https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_dpp.c#L166

> I would totally agree that the driver does not want to be analysing LUT
> entries to decipher if it could use a fixed-function element or not. It
> would introduce uncertainty in the UAPI. So fixed-function elements
> would need their own properties, but I don't know if that is feasible
> as generic UAPI or if it should be driver-specific (and so left unused
> by generic userspace).
> 


For the CRTC LUTs we actually do a linearity check to program the
HW into bypass when the LUT is linear since the KMS LUT definition
doesn't map well onto the LUT definition used by our HW and leads
to rounding errors and failing IGT kms_color tests (if I remember
this correctly).

https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c#L330

Hence the suggestion to define pre-defined TFs right at a KMS level
for usecases where we can assume the display will tonemap the 
content.

Harry

> 
> Thanks,
> pq
> 

Re: [RFC PATCH 1/3] drm/color: Add RGB Color encodings

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 3925 bytes --]

On Tue, 18 May 2021 10:32:48 -0400
Harry Wentland <harry.wentland@amd.com> wrote:

> On 2021-05-17 4:34 a.m., Pekka Paalanen wrote:
> > On Fri, 14 May 2021 17:04:51 -0400
> > Harry Wentland <harry.wentland@amd.com> wrote:
> >   
> >> On 2021-04-30 8:53 p.m., Sebastian Wick wrote:  
> >>> On 2021-04-26 20:56, Harry Wentland wrote:    
> > 
> > ...
> >   
> >>>> Another reason I'm proposing to define the color space (and gamma) of
> >>>> a plane is to make this explicit. Up until the color space and gamma
> >>>> of a plane or framebuffer are not well defined, which leads to drivers
> >>>> assuming the color space and gamma of a buffer (for blending and other
> >>>> purposes) and might lead to sub-optimal outcomes.    
> >>>
> >>> Blending only is "correct" with linear light so that property of the
> >>> color space is important. However, why does the kernel have to be
> >>> involved here? As long as user space knows that for correct blending the
> >>> data must represent linear light and knows when in the pipeline blending
> >>> happens it can make sure that the data at that point in the pipeline
> >>> contains linear light.
> >>>     
> >>
> >> The only reason the kernel needs to be involved is to take full advantage
> >> of the available HW without requiring KMS clients to be aware of
> >> the difference in display HW.  
> > 
> > Can you explain in more tangible examples, why you think so, please?
> > 
> > Is it because hardware does not fit the KMS UAPI model of the abstract
> > pixel pipeline?
> >   
> 
> I'd wager no HW is designed to meet KMS UAPI, rather KMS UAPI is designed
> to abstract HW.

Of course, but you are in big trouble in any case if there is a
fundamental mismatch. You may have to declare that all existing KMS
properties for this stuff will be mutually exclusive with your new
properties, so that you can introduce a new generic abstract pipeline
in KMS.

By mutually exclusive I mean that a driver must advertise only one or
the other set of properties and never both. If you want to support
userspace that doesn't understand the alternative set, maybe you also
need a drm client cap to switch to the alternative set per-drm-client.

> > Or is it because you have fixed-function hardware elements that you can
> > only make use of when userspace uses an enum-based UAPI?
> >   
> 
> One example is our degamma on our latest generation HW, where we have
> fixed-function "degamma" (rather de-EOTF):
> 
> https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_dpp.c#L166

Ok.

> > I would totally agree that the driver does not want to be analysing LUT
> > entries to decipher if it could use a fixed-function element or not. It
> > would introduce uncertainty in the UAPI. So fixed-function elements
> > would need their own properties, but I don't know if that is feasible
> > as generic UAPI or if it should be driver-specific (and so left unused
> > by generic userspace).
> >   
> 
> 
> For the CRTC LUTs we actually do a linearity check to program the
> HW into bypass when the LUT is linear since the KMS LUT definition
> doesn't map well onto the LUT definition used by our HW and leads
> to rounding errors and failing IGT kms_color tests (if I remember
> this correctly).
> 
> https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c#L330
> 
> Hence the suggestion to define pre-defined TFs right at a KMS level
> for usecases where we can assume the display will tonemap the 
> content.

Right. Explaining this would have been a good introduction in your
cover letter.

Maybe you want to define new KMS properties that shall be mutually
exclusive with the existing KMS GAMMA/CTM/DEGAMMA properties and
clearly document them as such.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[RFC PATCH 2/3] drm/color: Add Color transfer functions for HDR/SDR

[Harry Wentland]

From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>

Due to the way displays and human vision work it is most effective to
encode luminance information in a non-linear space.

For SDR this non-linear mapping is assumed to roughly use a gamma 2.2 curve.
This was due to the way CRTs worked and was fine for SDR content with a
low luminance range.

The large luminance range (0-10,000 nits) for HDR exposes some
short-comings of a simple gamma curve that have been addressed
through various Electro-Optical Transfer Functions (EOTFs).

Rather than assuming how framebuffer content is encoded we want to
make sure userspace presenting HDR content is explicit about the
EOTF of the content, so a driver can decide whether the content
can be supported or not.

This Patch adds common color transfer functions for SDR/HDR. These can
be used to communicate with the driver regarding the transformation to
use for a given plane.

enums added:
	DRM_COLOR_TF_SRGB
		roughly 2.4 gamma with initial linear section
	DRM_COLOR_TF_BT709
		Similar to Gamma 2.2-2.8
	DRM_COLOR_TF_PQ2084
		most common tf used for HDR video (HDR10/Dolby). Can support up
		to 10,000 nits

The usage is similar to color_encoding and color_range where the driver
can specify the default and supported tfs and pass it into
drm_plane_create_color_properties().

Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
Signed-off-by: Harry Wentland <harry.wentland@amd.com>
---
 .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |  4 +-
 .../gpu/drm/arm/display/komeda/komeda_plane.c |  4 +-
 drivers/gpu/drm/arm/malidp_planes.c           |  4 +-
 drivers/gpu/drm/armada/armada_overlay.c       |  4 +-
 drivers/gpu/drm/drm_atomic_uapi.c             |  4 ++
 drivers/gpu/drm/drm_color_mgmt.c              | 63 +++++++++++++++++--
 drivers/gpu/drm/i915/display/intel_sprite.c   |  4 +-
 .../drm/i915/display/skl_universal_plane.c    |  4 +-
 drivers/gpu/drm/nouveau/dispnv04/overlay.c    |  4 +-
 drivers/gpu/drm/omapdrm/omap_plane.c          |  4 +-
 drivers/gpu/drm/sun4i/sun8i_vi_layer.c        |  4 +-
 drivers/gpu/drm/tidss/tidss_plane.c           |  6 +-
 include/drm/drm_color_mgmt.h                  | 15 ++++-
 include/drm/drm_plane.h                       | 16 +++++
 14 files changed, 124 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index 6985362c367d..28481540327f 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -7127,7 +7127,9 @@ static int amdgpu_dm_plane_init(struct amdgpu_display_manager *dm,
 			BIT(DRM_COLOR_YCBCR_BT2020),
 			BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) |
 			BIT(DRM_COLOR_YCBCR_FULL_RANGE),
-			DRM_COLOR_YCBCR_BT709, DRM_COLOR_YCBCR_LIMITED_RANGE);
+			BIT(DRM_COLOR_TF_SRGB),
+			DRM_COLOR_YCBCR_BT709, DRM_COLOR_YCBCR_LIMITED_RANGE,
+			DRM_COLOR_TF_SRGB);
 	}
 
 	supported_rotations =
diff --git a/drivers/gpu/drm/arm/display/komeda/komeda_plane.c b/drivers/gpu/drm/arm/display/komeda/komeda_plane.c
index 2d5066ea270c..eb8ccf805408 100644
--- a/drivers/gpu/drm/arm/display/komeda/komeda_plane.c
+++ b/drivers/gpu/drm/arm/display/komeda/komeda_plane.c
@@ -299,8 +299,10 @@ static int komeda_plane_add(struct komeda_kms_dev *kms,
 			BIT(DRM_COLOR_YCBCR_BT2020),
 			BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) |
 			BIT(DRM_COLOR_YCBCR_FULL_RANGE),
+			BIT(DRM_COLOR_TF_SRGB),
 			DRM_COLOR_YCBCR_BT601,
-			DRM_COLOR_YCBCR_LIMITED_RANGE);
+			DRM_COLOR_YCBCR_LIMITED_RANGE,
+			DRM_COLOR_TF_SRGB);
 	if (err)
 		goto cleanup;
 
diff --git a/drivers/gpu/drm/arm/malidp_planes.c b/drivers/gpu/drm/arm/malidp_planes.c
index 351a85088d0e..0d77a2c0c829 100644
--- a/drivers/gpu/drm/arm/malidp_planes.c
+++ b/drivers/gpu/drm/arm/malidp_planes.c
@@ -1020,7 +1020,9 @@ int malidp_de_planes_init(struct drm_device *drm)
 					BIT(DRM_COLOR_YCBCR_BT2020),
 					BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) | \
 					BIT(DRM_COLOR_YCBCR_FULL_RANGE),
-					enc, range);
+					BIT(DRM_COLOR_SRGB),
+					enc, range,
+					DRM_COLOR_SRGB);
 			if (!ret)
 				/* program the HW registers */
 				malidp_de_set_color_encoding(plane, enc, range);
diff --git a/drivers/gpu/drm/armada/armada_overlay.c b/drivers/gpu/drm/armada/armada_overlay.c
index 6346b890279a..ad3a743242f0 100644
--- a/drivers/gpu/drm/armada/armada_overlay.c
+++ b/drivers/gpu/drm/armada/armada_overlay.c
@@ -589,8 +589,10 @@ int armada_overlay_plane_create(struct drm_device *dev, unsigned long crtcs)
 						BIT(DRM_COLOR_YCBCR_BT601) |
 						BIT(DRM_COLOR_YCBCR_BT709),
 						BIT(DRM_COLOR_YCBCR_LIMITED_RANGE),
+						BIT(DRM_COLOR_TF_SRGB),
 						DEFAULT_ENCODING,
-						DRM_COLOR_YCBCR_LIMITED_RANGE);
+						DRM_COLOR_YCBCR_LIMITED_RANGE,
+						DRM_COLOR_TF_SRGB);
 
 	return ret;
 }
diff --git a/drivers/gpu/drm/drm_atomic_uapi.c b/drivers/gpu/drm/drm_atomic_uapi.c
index 268bb69c2e2f..ea95c1224253 100644
--- a/drivers/gpu/drm/drm_atomic_uapi.c
+++ b/drivers/gpu/drm/drm_atomic_uapi.c
@@ -595,6 +595,8 @@ static int drm_atomic_plane_set_property(struct drm_plane *plane,
 		state->color_encoding = val;
 	} else if (property == plane->color_range_property) {
 		state->color_range = val;
+	} else if (property == plane->color_tf_property) {
+		state->color_tf = val;
 	} else if (property == config->prop_fb_damage_clips) {
 		ret = drm_atomic_replace_property_blob_from_id(dev,
 					&state->fb_damage_clips,
@@ -661,6 +663,8 @@ drm_atomic_plane_get_property(struct drm_plane *plane,
 		*val = state->color_encoding;
 	} else if (property == plane->color_range_property) {
 		*val = state->color_range;
+	} else if (property == plane->color_tf_property) {
+		*val = state->color_tf;
 	} else if (property == config->prop_fb_damage_clips) {
 		*val = (state->fb_damage_clips) ?
 			state->fb_damage_clips->base.id : 0;
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
index a183ebae2941..2404b07046c5 100644
--- a/drivers/gpu/drm/drm_color_mgmt.c
+++ b/drivers/gpu/drm/drm_color_mgmt.c
@@ -106,6 +106,11 @@
  * 	Optional plane enum property to support different non RGB
  * 	color parameter ranges. The driver can provide a subset of
  * 	standard enum values supported by the DRM plane.
+ *
+ * "COLOR_TRANFER_FUNCTION":
+ * 	Optional plane enum property to support different
+ * 	color luminance mappings. The driver can provide a subset of
+ * 	standard enum values supported by the DRM plane.
  */
 
 /**
@@ -480,6 +485,10 @@ static const char * const color_range_name[] = {
 	[DRM_COLOR_YCBCR_LIMITED_RANGE] = "YCbCr limited range",
 };
 
+static const char * const color_tf_name[] = {
+	[DRM_COLOR_TF_SRGB] = "sRGB",
+	[DRM_COLOR_TF_PQ2084] = "PQ2084",
+};
 /**
  * drm_get_color_encoding_name - return a string for color encoding
  * @encoding: color encoding to compute name of
@@ -510,30 +519,49 @@ const char *drm_get_color_range_name(enum drm_color_range range)
 	return color_range_name[range];
 }
 
+/**
+ * drm_get_color_transfer_function - return a string for color transfer function
+ * @tf: transfer function to compute name of
+ *
+ * In contrast to the other drm_get_*_name functions this one here returns a
+ * const pointer and hence is threadsafe.
+ */
+const char *drm_get_color_transfer_function_name(enum drm_color_transfer_function tf)
+{
+	if (WARN_ON(tf >= ARRAY_SIZE(color_tf_name)))
+		return "unknown";
+
+	return color_tf_name[tf];
+}
 /**
  * drm_plane_create_color_properties - color encoding related plane properties
  * @plane: plane object
  * @supported_encodings: bitfield indicating supported color encodings
  * @supported_ranges: bitfileld indicating supported color ranges
+ * @supported_tfs: bitfileld indicating supported color transfer functions
  * @default_encoding: default color encoding
  * @default_range: default color range
+ * @default_tf: default color transfer function
  *
- * Create and attach plane specific COLOR_ENCODING and COLOR_RANGE
- * properties to @plane. The supported encodings and ranges should
- * be provided in supported_encodings and supported_ranges bitmasks.
+ * Create and attach plane specific COLOR_ENCODING, COLOR_RANGE and COLOR_TRANSFER_FUNCTION
+ * properties to @plane. The supported encodings, ranges  and tfs should
+ * be provided in supported_encodings, supported_ranges and supported_tfs bitmasks.
  * Each bit set in the bitmask indicates that its number as enum
  * value is supported.
  */
 int drm_plane_create_color_properties(struct drm_plane *plane,
 				      u32 supported_encodings,
 				      u32 supported_ranges,
+				      u32 supported_tfs,
 				      enum drm_color_encoding default_encoding,
-				      enum drm_color_range default_range)
+				      enum drm_color_range default_range,
+				      enum drm_color_transfer_function default_tf)
 {
 	struct drm_device *dev = plane->dev;
 	struct drm_property *prop;
 	struct drm_prop_enum_list enum_list[max_t(int, DRM_COLOR_ENCODING_MAX,
-						       DRM_COLOR_RANGE_MAX)];
+						       max_t(int, DRM_COLOR_RANGE_MAX,
+							     DRM_COLOR_TF_MAX))];
 	int i, len;
 
 	if (WARN_ON(supported_encodings == 0 ||
@@ -546,6 +574,11 @@ int drm_plane_create_color_properties(struct drm_plane *plane,
 		    (supported_ranges & BIT(default_range)) == 0))
 		return -EINVAL;
 
+	if (WARN_ON(supported_tfs == 0 ||
+		    (supported_tfs & -BIT(DRM_COLOR_TF_MAX)) != 0 ||
+		    (supported_tfs & BIT(default_tf)) == 0))
+		return -EINVAL;
+
 	len = 0;
 	for (i = 0; i < DRM_COLOR_ENCODING_MAX; i++) {
 		if ((supported_encodings & BIT(i)) == 0)
@@ -584,6 +617,26 @@ int drm_plane_create_color_properties(struct drm_plane *plane,
 	if (plane->state)
 		plane->state->color_range = default_range;
 
+
+	len = 0;
+	for (i = 0; i < DRM_COLOR_TF_MAX; i++) {
+		if ((supported_tfs & BIT(i)) == 0)
+			continue;
+
+		enum_list[len].type = i;
+		enum_list[len].name = color_tf_name[i];
+		len++;
+	}
+
+	prop = drm_property_create_enum(dev, 0, "COLOR_TRANSFER_FUNCTION",
+					enum_list, len);
+	if (!prop)
+		return -ENOMEM;
+	plane->color_tf_property = prop;
+	drm_object_attach_property(&plane->base, prop, default_tf);
+	if (plane->state)
+		plane->state->color_tf = default_tf;
+
 	return 0;
 }
 EXPORT_SYMBOL(drm_plane_create_color_properties);
diff --git a/drivers/gpu/drm/i915/display/intel_sprite.c b/drivers/gpu/drm/i915/display/intel_sprite.c
index 4cbdb8fd4bb1..2208c3d82246 100644
--- a/drivers/gpu/drm/i915/display/intel_sprite.c
+++ b/drivers/gpu/drm/i915/display/intel_sprite.c
@@ -1881,8 +1881,10 @@ intel_sprite_plane_create(struct drm_i915_private *dev_priv,
 					  BIT(DRM_COLOR_YCBCR_BT709),
 					  BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) |
 					  BIT(DRM_COLOR_YCBCR_FULL_RANGE),
+					  BIT(DRM_COLOR_TF_SRGB),
 					  DRM_COLOR_YCBCR_BT709,
-					  DRM_COLOR_YCBCR_LIMITED_RANGE);
+					  DRM_COLOR_YCBCR_LIMITED_RANGE,
+					  DRM_COLOR_TF_SRGB);
 
 	zpos = sprite + 1;
 	drm_plane_create_zpos_immutable_property(&plane->base, zpos);
diff --git a/drivers/gpu/drm/i915/display/skl_universal_plane.c b/drivers/gpu/drm/i915/display/skl_universal_plane.c
index 1f335cb09149..c2d64ef9ae3b 100644
--- a/drivers/gpu/drm/i915/display/skl_universal_plane.c
+++ b/drivers/gpu/drm/i915/display/skl_universal_plane.c
@@ -2091,8 +2091,10 @@ skl_universal_plane_create(struct drm_i915_private *dev_priv,
 					  supported_csc,
 					  BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) |
 					  BIT(DRM_COLOR_YCBCR_FULL_RANGE),
+					  BIT(DRM_COLOR_TF_SRGB),
 					  DRM_COLOR_YCBCR_BT709,
-					  DRM_COLOR_YCBCR_LIMITED_RANGE);
+					  DRM_COLOR_YCBCR_LIMITED_RANGE,
+					  DRM_COLOR_TF_SRGB);
 
 	drm_plane_create_alpha_property(&plane->base);
 	drm_plane_create_blend_mode_property(&plane->base,
diff --git a/drivers/gpu/drm/nouveau/dispnv04/overlay.c b/drivers/gpu/drm/nouveau/dispnv04/overlay.c
index 37e63e98cd08..fe40e24469d1 100644
--- a/drivers/gpu/drm/nouveau/dispnv04/overlay.c
+++ b/drivers/gpu/drm/nouveau/dispnv04/overlay.c
@@ -345,8 +345,10 @@ nv10_overlay_init(struct drm_device *device)
 					  BIT(DRM_COLOR_YCBCR_BT601) |
 					  BIT(DRM_COLOR_YCBCR_BT709),
 					  BIT(DRM_COLOR_YCBCR_LIMITED_RANGE),
+					  BIT(DRM_COLOR_TF_SRGB),
 					  DRM_COLOR_YCBCR_BT601,
-					  DRM_COLOR_YCBCR_LIMITED_RANGE);
+					  DRM_COLOR_YCBCR_LIMITED_RANGE,
+					  DRM_COLOR_TF_SRGB);
 
 	plane->set_params = nv10_set_params;
 	nv10_set_params(plane);
diff --git a/drivers/gpu/drm/omapdrm/omap_plane.c b/drivers/gpu/drm/omapdrm/omap_plane.c
index 51dc24acea73..0929e1c39653 100644
--- a/drivers/gpu/drm/omapdrm/omap_plane.c
+++ b/drivers/gpu/drm/omapdrm/omap_plane.c
@@ -319,8 +319,10 @@ struct drm_plane *omap_plane_init(struct drm_device *dev,
 						  BIT(DRM_COLOR_YCBCR_BT709),
 						  BIT(DRM_COLOR_YCBCR_FULL_RANGE) |
 						  BIT(DRM_COLOR_YCBCR_LIMITED_RANGE),
+						  BIT(DRM_COLOR_TF_SRGB),
 						  DRM_COLOR_YCBCR_BT601,
-						  DRM_COLOR_YCBCR_FULL_RANGE);
+						  DRM_COLOR_YCBCR_FULL_RANGE,
+						  DRM_COLOR_TF_SRGB);
 
 	return plane;
 
diff --git a/drivers/gpu/drm/sun4i/sun8i_vi_layer.c b/drivers/gpu/drm/sun4i/sun8i_vi_layer.c
index 8abb59e2f0c0..d15267db2274 100644
--- a/drivers/gpu/drm/sun4i/sun8i_vi_layer.c
+++ b/drivers/gpu/drm/sun4i/sun8i_vi_layer.c
@@ -584,8 +584,10 @@ struct sun8i_vi_layer *sun8i_vi_layer_init_one(struct drm_device *drm,
 	ret = drm_plane_create_color_properties(&layer->plane,
 						supported_encodings,
 						supported_ranges,
+						BIT(DRM_COLOR_TF_SRGB),
 						DRM_COLOR_YCBCR_BT709,
-						DRM_COLOR_YCBCR_LIMITED_RANGE);
+						DRM_COLOR_YCBCR_LIMITED_RANGE,
+						DRM_COLOR_TF_SRGB);
 	if (ret) {
 		dev_err(drm->dev, "Couldn't add encoding and range properties!\n");
 		return ERR_PTR(ret);
diff --git a/drivers/gpu/drm/tidss/tidss_plane.c b/drivers/gpu/drm/tidss/tidss_plane.c
index 35067ae674ea..e50857cfd212 100644
--- a/drivers/gpu/drm/tidss/tidss_plane.c
+++ b/drivers/gpu/drm/tidss/tidss_plane.c
@@ -179,8 +179,10 @@ struct tidss_plane *tidss_plane_create(struct tidss_device *tidss,
 			       BIT(DRM_COLOR_YCBCR_BT709));
 	u32 color_ranges = (BIT(DRM_COLOR_YCBCR_FULL_RANGE) |
 			    BIT(DRM_COLOR_YCBCR_LIMITED_RANGE));
+	u32 color_tfs = BIT(DRM_COLOR_TF_SRGB;
 	u32 default_encoding = DRM_COLOR_YCBCR_BT601;
 	u32 default_range = DRM_COLOR_YCBCR_FULL_RANGE;
+	u32 default_tf = DRM_COLOR_TF_SRGB;;
 	u32 blend_modes = (BIT(DRM_MODE_BLEND_PREMULTI) |
 			   BIT(DRM_MODE_BLEND_COVERAGE));
 	int ret;
@@ -210,8 +212,10 @@ struct tidss_plane *tidss_plane_create(struct tidss_device *tidss,
 	ret = drm_plane_create_color_properties(&tplane->plane,
 						color_encodings,
 						color_ranges,
+						color_tfs,
 						default_encoding,
-						default_range);
+						default_range,
+						default_tf);
 	if (ret)
 		goto err;
 
diff --git a/include/drm/drm_color_mgmt.h b/include/drm/drm_color_mgmt.h
index 3043dd73480c..f59806366855 100644
--- a/include/drm/drm_color_mgmt.h
+++ b/include/drm/drm_color_mgmt.h
@@ -91,11 +91,24 @@ enum drm_color_range {
 	DRM_COLOR_RANGE_MAX,
 };
 
+/**
+ * enum drm_color_transfer_function - common transfer function used for sdr/hdr formats
+ *
+ * DRM_COLOR_TF_SRGB - Based on gamma curve and is used for printer/monitors/web
+ * DRM_COLOR_TF_PQ2084 - Used for HDR and allows for up to 10,000 nit support.
+*/
+enum drm_color_transfer_function {
+	DRM_COLOR_TF_SRGB,
+	DRM_COLOR_TF_PQ2084,
+	DRM_COLOR_TF_MAX,
+};
 int drm_plane_create_color_properties(struct drm_plane *plane,
 				      u32 supported_encodings,
 				      u32 supported_ranges,
+				      u32 supported_tf,
 				      enum drm_color_encoding default_encoding,
-				      enum drm_color_range default_range);
+				      enum drm_color_range default_range,
+				      enum drm_color_transfer_function default_tf);
 
 /**
  * enum drm_color_lut_tests - hw-specific LUT tests to perform
diff --git a/include/drm/drm_plane.h b/include/drm/drm_plane.h
index 95ab14a4336a..c85c59501a7a 100644
--- a/include/drm/drm_plane.h
+++ b/include/drm/drm_plane.h
@@ -179,6 +179,14 @@ struct drm_plane_state {
 	 */
 	enum drm_color_range color_range;
 
+	/**
+	 * @color_transfer_function:
+	 *
+	 * Transfer function for HDR color/luminance mapping. This will allow the
+	 * driver to know what transfer function should be used to for the current
+	 * format for a proper HDR color/luminance output.
+	 */
+	enum drm_color_transfer_function color_tf;
 	/**
 	 * @fb_damage_clips:
 	 *
@@ -741,6 +749,14 @@ struct drm_plane {
 	 * See drm_plane_create_color_properties().
 	 */
 	struct drm_property *color_range_property;
+	/**
+	 * @color_tf_property:
+	 *
+	 * Optional "COLOR_TRANSFER_FUNCTION" enum property for specifying
+	 * color transfer function for non RGB formats, mostly used for HDR.
+	 * See drm_plane_create_color_properties().
+	 */
+	struct drm_property *color_tf_property;
 
 	/**
 	 * @scaling_filter_property: property to apply a particular filter while
-- 
2.31.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC PATCH 3/3] drm/color: Add sdr boost property

[Harry Wentland]

From: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>

SDR is typically mastered at 200 nits and HDR is mastered at up to 10,000
nits. Due to this luminance range difference if we blend a SDR and
HDR plane together, we can run into problems where the HDR plane is too
bright or the SDR plane is too dim

A common solution to this problem is to boost the SDR plane so that its
not too dim.

This patch introduces a "sdr_white_level" property, this property can be
used by the userspace to boost the SDR content luminance. The boost
value is a explicit luiminance value in nits. This allows the userspace
to set the maximum white level for the SDR plane.

Signed-off-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
Signed-off-by: Harry Wentland <harry.wentland@amd.com>
---
 drivers/gpu/drm/drm_atomic_uapi.c |  4 ++++
 drivers/gpu/drm/drm_color_mgmt.c  | 17 +++++++++++++++++
 include/drm/drm_color_mgmt.h      |  6 ++++++
 include/drm/drm_plane.h           | 14 ++++++++++++++
 4 files changed, 41 insertions(+)

diff --git a/drivers/gpu/drm/drm_atomic_uapi.c b/drivers/gpu/drm/drm_atomic_uapi.c
index ea95c1224253..b3b6de7b74aa 100644
--- a/drivers/gpu/drm/drm_atomic_uapi.c
+++ b/drivers/gpu/drm/drm_atomic_uapi.c
@@ -597,6 +597,8 @@ static int drm_atomic_plane_set_property(struct drm_plane *plane,
 		state->color_range = val;
 	} else if (property == plane->color_tf_property) {
 		state->color_tf = val;
+	} else if (property == plane->sdr_white_level_property) {
+		state->sdr_white_level = val;
 	} else if (property == config->prop_fb_damage_clips) {
 		ret = drm_atomic_replace_property_blob_from_id(dev,
 					&state->fb_damage_clips,
@@ -665,6 +667,8 @@ drm_atomic_plane_get_property(struct drm_plane *plane,
 		*val = state->color_range;
 	} else if (property == plane->color_tf_property) {
 		*val = state->color_tf;
+	} else if (property == plane->sdr_white_level_property) {
+		*val = state->sdr_white_level;
 	} else if (property == config->prop_fb_damage_clips) {
 		*val = (state->fb_damage_clips) ?
 			state->fb_damage_clips->base.id : 0;
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c
index 2404b07046c5..a0b77d7d0565 100644
--- a/drivers/gpu/drm/drm_color_mgmt.c
+++ b/drivers/gpu/drm/drm_color_mgmt.c
@@ -519,6 +519,23 @@ const char *drm_get_color_range_name(enum drm_color_range range)
 	return color_range_name[range];
 }
 
+int drm_plane_create_sdr_white_level_property(struct drm_plane *plane){
+
+	struct drm_property *prop;
+
+	prop = drm_property_create_range(plane->dev, 0, "SDR_WHITE_LEVEL", 0, UINT_MAX);
+
+	if (!prop)
+		return -ENOMEM;
+
+	plane->sdr_white_level_property = prop;
+	drm_object_attach_property(&plane->base, prop, DRM_DEFAULT_SDR_WHITE_LEVEL);
+
+	if (plane->state)
+		plane->state->sdr_white_level = DRM_DEFAULT_SDR_WHITE_LEVEL;
+
+	return 0;
+}
 /**
  * drm_get_color_transfer_function - return a string for color transfer function
  * @tf: transfer function to compute name of
diff --git a/include/drm/drm_color_mgmt.h b/include/drm/drm_color_mgmt.h
index f59806366855..a020346b1747 100644
--- a/include/drm/drm_color_mgmt.h
+++ b/include/drm/drm_color_mgmt.h
@@ -26,6 +26,12 @@
 #include <linux/ctype.h>
 #include <drm/drm_property.h>
 
+/**
+ * Default SDR white level in nits. Although there is no standard SDR nit level, 200
+ * is chosen as the default since that is the generally accepted value.
+ */
+#define DRM_DEFAULT_SDR_WHITE_LEVEL 200
+
 struct drm_crtc;
 struct drm_plane;
 
diff --git a/include/drm/drm_plane.h b/include/drm/drm_plane.h
index c85c59501a7a..fad8b7dd430c 100644
--- a/include/drm/drm_plane.h
+++ b/include/drm/drm_plane.h
@@ -187,6 +187,11 @@ struct drm_plane_state {
 	 * format for a proper HDR color/luminance output.
 	 */
 	enum drm_color_transfer_function color_tf;
+	/**
+	 * @sdr_white_level:
+	 * SDR white level boost for HDR+SDR multi plane usecases. max white level in nits
+	 */
+	unsigned int sdr_white_level;
 	/**
 	 * @fb_damage_clips:
 	 *
@@ -757,6 +762,15 @@ struct drm_plane {
 	 * See drm_plane_create_color_properties().
 	 */
 	struct drm_property *color_tf_property;
+	/**
+	 * @sdr_white_level:
+	 *
+	 * Optional sdr_white_level. When HDR and SDR are combined in multi plane
+	 * overlay cases, the sdr plane will be very dim. This property allows
+	 * the driver to boost the sdr plane's white level. The value should be
+	 * max white level in nits.
+	 */
+	struct drm_property *sdr_white_level_property;
 
 	/**
 	 * @scaling_filter_property: property to apply a particular filter while
-- 
2.31.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Daniel Vetter]

On Mon, Apr 26, 2021 at 01:38:49PM -0400, Harry Wentland wrote:
> 
> ## Introduction
> 
> We are looking to enable HDR support for a couple of single-plane and
> multi-plane scenarios. To do this effectively we recommend new
> interfaces to drm_plane. Below I'll give a bit of background on HDR and
> why we propose these interfaces.

I think this is on of the topics that would tremendously benefit from the
uapi rfc process, with lots of compositor people involved.

https://dri.freedesktop.org/docs/drm/gpu/rfc/

Also for this I think we really do need a pretty solid understanding of
the involve compositor protocols, otherwise the kernel uapi is going to be
for naught.
-Daniel

> 
> 
> ## Defining a pixel's luminance
> 
> Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms.
> 
> Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows.
> 
> A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits.
> 
> Using a content that's defined in PQ space we can approximate the real world in a much better way.
> 
> Here are some examples of real-life objects and their approximate luminance values:
> 
> | Object            | Luminance in nits |
> | ----------------- | ----------------- |
> | Sun               | 1.6 million       |
> | Fluorescent light | 10,000            |
> | Highlights        | 1,000 - sunlight  |
> | White Objects     | 250 - 1,000       |
> | Typical objects   | 1 - 250           |
> | Shadows           | 0.01 - 1          |
> | Ultra Blacks      | 0 - 0.0005        |
> 
> 
> ## Describing the luminance space
> 
> **We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are:
> 
> | EOTF      | Description                                                               |
> | --------- |:------------------------------------------------------------------------- |
> | Gamma 2.2 | a simple 2.2 gamma                                                        |
> | sRGB      | 2.4 gamma with small initial linear section                               |
> | PQ 2084   | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support |
> | Linear    | Linear relationship between pixel value and luminance value               |
> 
> 
> ## Mastering Luminances
> 
> Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
> 
> | Display                  | Luminance range in nits |
> | ------------------------ | ----------------------- |
> | Typical PC display       | 0.3 - 200               |
> | Excellent LCD HDTV       | 0.3 - 400               |
> | HDR LCD w/ local dimming | 0.05 - 1,500            |
> 
> Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
> 
> The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
> 
> It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
> 
> 
> ## SDR Luminance
> 
> Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable.
> 
> **We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space.
> 
> 
> ## Let There Be Color
> 
> So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value.
> 
> A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue.
> 
> Examples of color spaces are:
> 
> | Color Space | Description                                |
> | ----------- | ------------------------------------------ |
> | BT 601      | similar to BT 709                          |
> | BT 709      | used by sRGB content; ~53% of BT 2020      |
> | DCI-P3      | used by most HDR displays; ~72% of BT 2020 |
> | BT 2020     | standard for most HDR content              |
> 
> The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane. 
> 
> **We propose to add definitions for the RGB variants of the BT color spaces.**
> 
> 
> ## Color Primaries and White Point
> 
> Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut.
> 
> 
> ## Displays and Tonemapping
> 
> External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
> 
> Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
> 
> 
> ## Pixel Formats
> 
> The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF.
> 
> 
> ## Proposed use-cases
> 
> Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases:
> 
> **One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set.
> 
> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
> 
> **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane.
> 
> In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed.
> 
> 
> ## Summary of proposed interface changes
> 
> per drm_plane:
> - new RGB color space definitions, mirroring the existing YUV color space definitions
> - new transfer function property
> - new SDR maximum white level property
> 
> 
> ## References
> 
> [1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
> 
> 
> ## Further Reading
> 
> https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst
> http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf
> https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
> 
> 
> Bhawanpreet Lakha (3):
>   drm/color: Add RGB Color encodings
>   drm/color: Add Color transfer functions for HDR/SDR
>   drm/color: Add sdr boost property
> 
>  .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |  4 +-
>  .../gpu/drm/arm/display/komeda/komeda_plane.c |  4 +-
>  drivers/gpu/drm/arm/malidp_planes.c           |  4 +-
>  drivers/gpu/drm/armada/armada_overlay.c       |  4 +-
>  drivers/gpu/drm/drm_atomic_uapi.c             |  8 ++
>  drivers/gpu/drm/drm_color_mgmt.c              | 84 +++++++++++++++++--
>  drivers/gpu/drm/i915/display/intel_sprite.c   |  4 +-
>  .../drm/i915/display/skl_universal_plane.c    |  4 +-
>  drivers/gpu/drm/nouveau/dispnv04/overlay.c    |  4 +-
>  drivers/gpu/drm/omapdrm/omap_plane.c          |  4 +-
>  drivers/gpu/drm/sun4i/sun8i_vi_layer.c        |  4 +-
>  drivers/gpu/drm/tidss/tidss_plane.c           |  6 +-
>  include/drm/drm_color_mgmt.h                  | 25 +++++-
>  include/drm/drm_plane.h                       | 30 +++++++
>  14 files changed, 173 insertions(+), 16 deletions(-)
> 
> -- 
> 2.31.0
> 
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 16407 bytes --]

On Mon, 26 Apr 2021 13:38:49 -0400
Harry Wentland <harry.wentland@amd.com> wrote:

> ## Introduction
> 
> We are looking to enable HDR support for a couple of single-plane and
> multi-plane scenarios. To do this effectively we recommend new
> interfaces to drm_plane. Below I'll give a bit of background on HDR
> and why we propose these interfaces.
> 
> 
> ## Defining a pixel's luminance
> 
> Currently the luminance space of pixels in a framebuffer/plane
> presented to the display is not well defined. It's usually assumed to
> be in a 2.2 or 2.4 gamma space and has no mapping to an absolute
> luminance value but is interpreted in relative terms.
> 
> Luminance can be measured and described in absolute terms as candela
> per meter squared, or cd/m2, or nits. Even though a pixel value can
> be mapped to luminance in a linear fashion to do so without losing a
> lot of detail requires 16-bpc color depth. The reason for this is
> that human perception can distinguish roughly between a 0.5-1%
> luminance delta. A linear representation is suboptimal, wasting
> precision in the highlights and losing precision in the shadows.
> 
> A gamma curve is a decent approximation to a human's perception of
> luminance, but the PQ (perceptual quantizer) function [1] improves on
> it. It also defines the luminance values in absolute terms, with the
> highest value being 10,000 nits and the lowest 0.0005 nits.
> 
> Using a content that's defined in PQ space we can approximate the
> real world in a much better way.
> 
> Here are some examples of real-life objects and their approximate
> luminance values:
> 
> | Object            | Luminance in nits |
> | ----------------- | ----------------- |
> | Sun               | 1.6 million       |
> | Fluorescent light | 10,000            |
> | Highlights        | 1,000 - sunlight  |
> | White Objects     | 250 - 1,000       |
> | Typical objects   | 1 - 250           |
> | Shadows           | 0.01 - 1          |
> | Ultra Blacks      | 0 - 0.0005        |
> 
> 
> ## Describing the luminance space
> 
> **We propose a new drm_plane property to describe the Eletro-Optical
> Transfer Function (EOTF) with which its framebuffer was composed.**
> Examples of EOTF are:
> 
> | EOTF      | Description                                                               |
> | --------- |:------------------------------------------------------------------------- |
> | Gamma 2.2 | a simple 2.2 gamma                                                        |
> | sRGB      | 2.4 gamma with small initial linear section                               |
> | PQ 2084   | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support |
> | Linear    | Linear relationship between pixel value and luminance value               |
> 

The definitions agree with what I have learnt so far. However, with
these EOTF definitions, only PQ defines absolute luminance values
while the others do not. So this is not enough information to blend
planes together if they do not all use the same EOTF with the same
dynamic range. More below.


> 
> ## Mastering Luminances
> 
> Now we are able to use the PQ 2084 EOTF to define the luminance of
> pixels in absolute terms. Unfortunately we're again presented with
> physical limitations of the display technologies on the market today.
> Here are a few examples of luminance ranges of displays.
> 
> | Display                  | Luminance range in nits |
> | ------------------------ | ----------------------- |
> | Typical PC display       | 0.3 - 200               |
> | Excellent LCD HDTV       | 0.3 - 400               |
> | HDR LCD w/ local dimming | 0.05 - 1,500            |
> 
> Since no display can currently show the full 0.0005 to 10,000 nits
> luminance range the display will need to tonemap the HDR content, i.e
> to fit the content within a display's capabilities. To assist with
> tonemapping HDR content is usually accompanied with a metadata that
> describes (among other things) the minimum and maximum mastering
> luminance, i.e. the maximum and minimum luminance of the display that
> was used to master the HDR content.
> 
> The HDR metadata is currently defined on the drm_connector via the
> hdr_output_metadata blob property.
> 
> It might be useful to define per-plane hdr metadata, as different
> planes might have been mastered differently.

I don't think this would directly help with the dynamic range blending
problem. You still need to establish the mapping between the optical
values from two different EOTFs and dynamic ranges. Or can you know
which optical values match the mastering display maximum and minimum
luminances for not-PQ?


> ## SDR Luminance
> 
> Since SDR covers a smaller luminance range than HDR, an SDR plane
> might look dark when blended with HDR content. Since the max HDR
> luminance can be quite variable (200-1,500 nits on actual displays)
> it is best to make the SDR maximum luminance value configurable.
> 
> **We propose a drm_plane property to specfy the desired maximum
> luminance of the SDR plane in nits.** This allows us to map the SDR
> content predictably into HDR's absolute luminance space.

What would be the mapping? Simple linear scaling? A more complicated
tone mapping?

Rather than "SDR luminance", do you perhaps intend this to configure
the dynamic range of the non-absolute-luminance EOTFs?
In that case maybe you'd need a black luminance level too?


> ## Let There Be Color
> 
> So far we've only talked about luminance, ignoring colors altogether.
> Just like in the luminance space, traditionally the color space of
> display outputs has not been well defined. Similar to how an EOTF
> defines a mapping of pixel data to an absolute luminance value, the
> color space maps color information for each pixel onto the CIE 1931
> chromaticity space. This can be thought of as a mapping to an
> absolute, real-life, color value.
> 
> A color space is defined by its primaries and white point. The
> primaries and white point are expressed as coordinates in the CIE
> 1931 color space. Think of the red primary as the reddest red that
> can be displayed within the color space. Same for green and blue.
> 
> Examples of color spaces are:
> 
> | Color Space | Description                                |
> | ----------- | ------------------------------------------ |
> | BT 601      | similar to BT 709                          |
> | BT 709      | used by sRGB content; ~53% of BT 2020      |
> | DCI-P3      | used by most HDR displays; ~72% of BT 2020 |
> | BT 2020     | standard for most HDR content              |
> 
> The color space is defined in DRM for YCbCr planes via the
> color_encoding property of the drm_plane. 

I don't think that is quite right.

As far I understand, COLOR_ENCODING property controls which matrix is
used to convert from YCbCr to RGB, but that is all it does. It is not
used for the actual color space. So while these BT standards do
specify the chromaticities, they also specify the YCbCr encoding which
is the part used in this property.

YCbCr and RGB are color models. They are not color spaces. RGB is an
additive color model while YCbCr is not, AFAIU. Blending requires an
additive color model and linear luminance encoding.

You need two color space definitions to create one color space
transformation: source color space and destination color space. You
also need an idea *how* the two color spaces should be mapped, which is
called "rendering intent". You can't do anything with just one color
space definition, except to pass it on along with the pixels.

To be able to blend planes together, all planes need to be converted to
the same color space first: the blending color space, whatever you
choose it to be. I do not see where KMS would do this color space
conversion, or where it would get the definition of the blending color
space.

> **We propose to add definitions for the RGB variants of the BT color
> spaces.**

Therefore I'm not sure this makes sense.


> ## Color Primaries and White Point
> 
> Just like displays can currently not represent the entire 0.0005 -
> 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not
> capable of representing the entire BT.2020 color Gamut. For this
> reason video content will often specify the color primaries and white
> point used to master the video, in order to allow displays to be able
> to map the image as best as possible onto the display's gamut.
> 
> 
> ## Displays and Tonemapping
> 
> External displays are able to do their own tone and color mapping,
> based on the mastering luminance, color primaries, and white space
> defined in the HDR metadata.
> 
> Internal panels (which are currently few and far between) usually
> don't include the complex HW to do tone and color mapping on their
> own and will require the display driver to perform appropriate
> mapping.

FWIW, when designing Weston's color management, we are aiming for
the latter "simple" panels foremost, because that gives us full control
of all color conversions and tone mappings.

OTOH, if Weston has to present to a display which only accepts e.g.
BT.2020/PQ signal, the display might always mangle the image in
unexpected ways. Therefore I expect that by default Weston will do
everything it can to try to make the display not apply anything magic
image enhancement: trust that EDID description of the display gamut and
dynamic range are correct, and use HDR metadata to tell the display
that those values are exactly what we are using. And we use them.

IMO, a display doing its tone mapping magically is only useful when you
want to be able to use "simple" playback devices that cannot adapt to
the display they are driving. Magic tone mapping is also a way for
hardware vendors to differentiate, which from the color management
perspective is harmful as it makes it more difficult or impossible to
predict the display behaviour or to keep it consistent.

So there are two opposing goals:

- Traditional color management wants absolute control of the display,
  leaving nothing unpredictable and preferably also nothing undefined.
  Undefined behaviour can always be measured (profiled) which makes it
  predictable and known. The viewing environment is controlled and
  constant.

- Entertainment wants the most visually impressive image quality by
  dynamically adapting to both displayed content and to the viewing
  environment conditions.

> ## Pixel Formats
> 
> The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are
> unrelated to color space and EOTF definitions. HDR pixels can be
> formatted in different ways but in order to not lose precision HDR
> content requires at least 10 bpc precision. For this reason
> ARGB2101010, P010, and FP16 are the obvious candidates for HDR.
> ARGB2101010 and P010 have the advantage of requiring only half the
> bandwidth as FP16, while FP16 has the advantage of enough precision
> to operate in a linear space, i.e. without EOTF.

Right.

> ## Proposed use-cases
> 
> Although the userspace side of this work is still in the early stages
> it is clear that we will want to support the following two use-cases:
> 
> **One XRGB2101010 HDR Plane:** A single, composited plane of HDR
> content. The use-case is a video player on a desktop with the
> compositor owning the composition of SDR and HDR content. The content
> shall be PQ BT.2020 formatted. The drm_connector's
> hdr_output_metadata shall be set.

This use case is already possible, right?

> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc
> desktop plane, with a P010 HDR video plane underlayed. The HDR plane
> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR
> boost value. The drm_connector's hdr_output_metadata shall be set.

This use case requires blending in KMS, so is the primary goal I
suppose.

> **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth
> transition we recommend an OS that supports HDR output to provide the
> hdr_output_metadata on the drm_connector to configure the output for
> HDR, even when the content is only SDR. This will allow for a smooth
> transition between SDR-only and HDR content. In this use-case the SDR
> max luminance value should be provided on the drm_plane.

I think this might be already possible by crafting a CRTC GAMMA LUT? Not
sure about precision.

> In DCN we will de-PQ or de-Gamma all input in order to blend in
> linear space. For SDR content we will also apply any desired boost
> before blending. After blending we will then re-apply the PQ EOTF and
> do RGB to YCbCr conversion if needed.

This assumes the same color space over everything.

> 
> ## Summary of proposed interface changes
> 
> per drm_plane:
> - new RGB color space definitions, mirroring the existing YUV color
> space definitions
> - new transfer function property
> - new SDR maximum white level property

How will these new KMS properties interact with per-plane DEGAMMA, CTM
and/or GAMMA properties?

Why go with your proposal instead of per-plane CTM and LUT?

I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs
both per-plane and on CRTC, would be:

plane:
	FB -> M1 -> LUT1 -> M2 -> blending input

CRTC:
	blending output -> LUT2 -> M3 -> connector

FB: framebuffer
M1: matrix transform, capable of converting e.g. YCbCr to RGB
LUT1: 1D LUT for content EOTF, to produce light-linear RGB
M2: matrix transform for color space transformation

LUT2: 1D LUT for applying monitor EOTF^-1
M3: matrix transform, e.g. if you need to push YCbCr on the connector

We also need to know where and how clipping happens.

I think this scheme would allow implementing everything you want, and
it would not be tied to rigid enumerations, and it won't have any
magical conversions done under the hood as you would need to do to
convert from one enum space to another. It leaves the render intent to
be defined by the userspace compositor, rather than building a fixed
policy in the kernel.

Userspace would be setting transformation operators, not color spaces,
to the kernel, allowing the blending space to be chosen by userspace.
In Weston we aim to choose then blending color space to be the same as
the output color space, except in optical (linear) encoding. The output
color space can be non-standard, e.g. measured with a display profiler
equipment.

I would expect gamut mapping, dynamic range mapping and tone mapping to
be places where most experimentation and innovation happens, so
implementing them in the kernel with just few or no controllable
parameters at this time seems like it could become useless fast.


Thanks,
pq

> ## References
> 
> [1]
> https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
> 
> 
> ## Further Reading
> 
> https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst
> http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf
> https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
> 
> 
> Bhawanpreet Lakha (3):
>   drm/color: Add RGB Color encodings
>   drm/color: Add Color transfer functions for HDR/SDR
>   drm/color: Add sdr boost property
> 
>  .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |  4 +-
>  .../gpu/drm/arm/display/komeda/komeda_plane.c |  4 +-
>  drivers/gpu/drm/arm/malidp_planes.c           |  4 +-
>  drivers/gpu/drm/armada/armada_overlay.c       |  4 +-
>  drivers/gpu/drm/drm_atomic_uapi.c             |  8 ++
>  drivers/gpu/drm/drm_color_mgmt.c              | 84
> +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c   |
> 4 +- .../drm/i915/display/skl_universal_plane.c    |  4 +-
>  drivers/gpu/drm/nouveau/dispnv04/overlay.c    |  4 +-
>  drivers/gpu/drm/omapdrm/omap_plane.c          |  4 +-
>  drivers/gpu/drm/sun4i/sun8i_vi_layer.c        |  4 +-
>  drivers/gpu/drm/tidss/tidss_plane.c           |  6 +-
>  include/drm/drm_color_mgmt.h                  | 25 +++++-
>  include/drm/drm_plane.h                       | 30 +++++++
>  14 files changed, 173 insertions(+), 16 deletions(-)
> 


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Shashank Sharma]

Hello Harry,

Many of us in the mail chain have discussed this before, on what is the right way to blend and tone map a SDR and a HDR buffer from same/different color spaces, and what kind of DRM plane properties will be needed.

As you can see from the previous comments, that the majority of the decision making will happen in the Compositor, as it's the only SW unit, which has the overall picture clear.

Reference: (https://lists.freedesktop.org/archives/wayland-devel/2019-January/039808.html )

If we see a systematic approach of how do we make such blending policy, it will look like:


- Compositor needs to understand the following values of each of the buffer:

    - Color space or Gamut: BT2020/SRGB/DCI-P3/BT709/BT601 etc

    - Color format (RGB/YCBCR) and subsampling (444/422/420)

    - Tone (SDR/HDR_A/HDR_B)


- Then the Compositor needs to understand the capabilities of the output display, as this will be a clamping value

    - Output Gamut support (BT2020/SRGB/DCIP3)

    - Output max Luminance of the monitor in Nits (even in case of HDR content to HDR display)

  

Based of all this information above, the compositor needs to set a blending target, which contains the following:

    - Output Colorspace of the blended output: say BT2020

    - Output Luminance of the blended output: Match content, if monitor can support it

    - Output Color format of the blended output: Say YCBCR4:2:0


Let's assume compositor prepares a blending policy with output as:

    - Output Luminance: HDR 500 Nits

    - Output color space: BT2020

    - Output color format: RGB888

    - Output curve: ST2084

  

Assuming these details, A compositor will look for DRM color properties like these:

1. Degamma plane property : To make buffers linear for Gamut mapping

2. Gamut mapping plane property:  To gamut map SRGB buffer to BT2020 colorspace

3. Color space conversion plane property: To convert from YCBCR->RGB

4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H

5. Gamma plane/CRTC property: to re-apply the output ST2084 curve


We will also need connector/CRTC properties to set AVI info-frames accordingly.

A high level block diagram for blending on a generic HW should look like this:

/*
 *  SDR 200Nits┌────────────────┐ SDR 200 Nits  ┌────────────────┐ SDR 200 ┌──────────────────┐HDR 500┌────────────────┐ HDR 500
 *   BT709     │                │ BT709         │                │ BT2020  │                  │BT2020 │                │ BT2020
 *   ────────► │   Degamma      ├─────────────► │ Gamut Mapping  ├────────►│  Tone mapping    ├──────►│  Gamma         │
 *  RGB888     │     2.2        │ RGB888        │  709->2020     │ RGB888  │    S2H           │RGB888 │  ST2084        │ RGB888
 *  Non Linear │                │ Linear        │                │ Linear  │   200->500       │Linear │                │ ST2084
 *             └────────────────┘               └────────────────┘         └──────────────────┘       └────────────────┘
 *
 *
 *
 *
 *
 *
 *
 *
 *             ┌─────────────────┐             ┌─────────────────┐           ┌─────────────────┐       ┌────────────────┐
 * HDR 600 Nits│                 │HDR 600 Nits │                 │HDR600     │                 │HDR500 │                │ HDR500
 *   ────────► │  Degamma        ├────────────►│  Color space    ├──────────►│  Tone mapping   ├──────►│  Gamma         │
 * BT2020      │  OETF ST2084    │ BT2020      │  conversion     │BT2020     │   H2H           │BT2020 │  ST2084        │ BT2020
 * YCBCR420    │                 │ YCBCR420    │ YCBCR->RGB      │RGB88      │   600->500      │RGB888 │                │ RGB888
 * Non Linear  └─────────────────┘ Linear      └─────────────────┘Linear     └─────────────────┘Linear └────────────────┘ ST2084
 */


Hope this helps to refine the series.


Regards

Shashank

On 27/04/21 20:20, Pekka Paalanen wrote:
> On Mon, 26 Apr 2021 13:38:49 -0400
> Harry Wentland <harry.wentland@amd.com> wrote:
>
>> ## Introduction
>>
>> We are looking to enable HDR support for a couple of single-plane and
>> multi-plane scenarios. To do this effectively we recommend new
>> interfaces to drm_plane. Below I'll give a bit of background on HDR
>> and why we propose these interfaces.
>>
>>
>> ## Defining a pixel's luminance
>>
>> Currently the luminance space of pixels in a framebuffer/plane
>> presented to the display is not well defined. It's usually assumed to
>> be in a 2.2 or 2.4 gamma space and has no mapping to an absolute
>> luminance value but is interpreted in relative terms.
>>
>> Luminance can be measured and described in absolute terms as candela
>> per meter squared, or cd/m2, or nits. Even though a pixel value can
>> be mapped to luminance in a linear fashion to do so without losing a
>> lot of detail requires 16-bpc color depth. The reason for this is
>> that human perception can distinguish roughly between a 0.5-1%
>> luminance delta. A linear representation is suboptimal, wasting
>> precision in the highlights and losing precision in the shadows.
>>
>> A gamma curve is a decent approximation to a human's perception of
>> luminance, but the PQ (perceptual quantizer) function [1] improves on
>> it. It also defines the luminance values in absolute terms, with the
>> highest value being 10,000 nits and the lowest 0.0005 nits.
>>
>> Using a content that's defined in PQ space we can approximate the
>> real world in a much better way.
>>
>> Here are some examples of real-life objects and their approximate
>> luminance values:
>>
>> | Object            | Luminance in nits |
>> | ----------------- | ----------------- |
>> | Sun               | 1.6 million       |
>> | Fluorescent light | 10,000            |
>> | Highlights        | 1,000 - sunlight  |
>> | White Objects     | 250 - 1,000       |
>> | Typical objects   | 1 - 250           |
>> | Shadows           | 0.01 - 1          |
>> | Ultra Blacks      | 0 - 0.0005        |
>>
>>
>> ## Describing the luminance space
>>
>> **We propose a new drm_plane property to describe the Eletro-Optical
>> Transfer Function (EOTF) with which its framebuffer was composed.**
>> Examples of EOTF are:
>>
>> | EOTF      | Description                                                               |
>> | --------- |:------------------------------------------------------------------------- |
>> | Gamma 2.2 | a simple 2.2 gamma                                                        |
>> | sRGB      | 2.4 gamma with small initial linear section                               |
>> | PQ 2084   | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support |
>> | Linear    | Linear relationship between pixel value and luminance value               |
>>
> The definitions agree with what I have learnt so far. However, with
> these EOTF definitions, only PQ defines absolute luminance values
> while the others do not. So this is not enough information to blend
> planes together if they do not all use the same EOTF with the same
> dynamic range. More below.
>
>
>> ## Mastering Luminances
>>
>> Now we are able to use the PQ 2084 EOTF to define the luminance of
>> pixels in absolute terms. Unfortunately we're again presented with
>> physical limitations of the display technologies on the market today.
>> Here are a few examples of luminance ranges of displays.
>>
>> | Display                  | Luminance range in nits |
>> | ------------------------ | ----------------------- |
>> | Typical PC display       | 0.3 - 200               |
>> | Excellent LCD HDTV       | 0.3 - 400               |
>> | HDR LCD w/ local dimming | 0.05 - 1,500            |
>>
>> Since no display can currently show the full 0.0005 to 10,000 nits
>> luminance range the display will need to tonemap the HDR content, i.e
>> to fit the content within a display's capabilities. To assist with
>> tonemapping HDR content is usually accompanied with a metadata that
>> describes (among other things) the minimum and maximum mastering
>> luminance, i.e. the maximum and minimum luminance of the display that
>> was used to master the HDR content.
>>
>> The HDR metadata is currently defined on the drm_connector via the
>> hdr_output_metadata blob property.
>>
>> It might be useful to define per-plane hdr metadata, as different
>> planes might have been mastered differently.
> I don't think this would directly help with the dynamic range blending
> problem. You still need to establish the mapping between the optical
> values from two different EOTFs and dynamic ranges. Or can you know
> which optical values match the mastering display maximum and minimum
> luminances for not-PQ?
>
>
>> ## SDR Luminance
>>
>> Since SDR covers a smaller luminance range than HDR, an SDR plane
>> might look dark when blended with HDR content. Since the max HDR
>> luminance can be quite variable (200-1,500 nits on actual displays)
>> it is best to make the SDR maximum luminance value configurable.
>>
>> **We propose a drm_plane property to specfy the desired maximum
>> luminance of the SDR plane in nits.** This allows us to map the SDR
>> content predictably into HDR's absolute luminance space.
> What would be the mapping? Simple linear scaling? A more complicated
> tone mapping?
>
> Rather than "SDR luminance", do you perhaps intend this to configure
> the dynamic range of the non-absolute-luminance EOTFs?
> In that case maybe you'd need a black luminance level too?
>
>
>> ## Let There Be Color
>>
>> So far we've only talked about luminance, ignoring colors altogether.
>> Just like in the luminance space, traditionally the color space of
>> display outputs has not been well defined. Similar to how an EOTF
>> defines a mapping of pixel data to an absolute luminance value, the
>> color space maps color information for each pixel onto the CIE 1931
>> chromaticity space. This can be thought of as a mapping to an
>> absolute, real-life, color value.
>>
>> A color space is defined by its primaries and white point. The
>> primaries and white point are expressed as coordinates in the CIE
>> 1931 color space. Think of the red primary as the reddest red that
>> can be displayed within the color space. Same for green and blue.
>>
>> Examples of color spaces are:
>>
>> | Color Space | Description                                |
>> | ----------- | ------------------------------------------ |
>> | BT 601      | similar to BT 709                          |
>> | BT 709      | used by sRGB content; ~53% of BT 2020      |
>> | DCI-P3      | used by most HDR displays; ~72% of BT 2020 |
>> | BT 2020     | standard for most HDR content              |
>>
>> The color space is defined in DRM for YCbCr planes via the
>> color_encoding property of the drm_plane. 
> I don't think that is quite right.
>
> As far I understand, COLOR_ENCODING property controls which matrix is
> used to convert from YCbCr to RGB, but that is all it does. It is not
> used for the actual color space. So while these BT standards do
> specify the chromaticities, they also specify the YCbCr encoding which
> is the part used in this property.
>
> YCbCr and RGB are color models. They are not color spaces. RGB is an
> additive color model while YCbCr is not, AFAIU. Blending requires an
> additive color model and linear luminance encoding.
>
> You need two color space definitions to create one color space
> transformation: source color space and destination color space. You
> also need an idea *how* the two color spaces should be mapped, which is
> called "rendering intent". You can't do anything with just one color
> space definition, except to pass it on along with the pixels.
>
> To be able to blend planes together, all planes need to be converted to
> the same color space first: the blending color space, whatever you
> choose it to be. I do not see where KMS would do this color space
> conversion, or where it would get the definition of the blending color
> space.
>
>> **We propose to add definitions for the RGB variants of the BT color
>> spaces.**
> Therefore I'm not sure this makes sense.
>
>
>> ## Color Primaries and White Point
>>
>> Just like displays can currently not represent the entire 0.0005 -
>> 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not
>> capable of representing the entire BT.2020 color Gamut. For this
>> reason video content will often specify the color primaries and white
>> point used to master the video, in order to allow displays to be able
>> to map the image as best as possible onto the display's gamut.
>>
>>
>> ## Displays and Tonemapping
>>
>> External displays are able to do their own tone and color mapping,
>> based on the mastering luminance, color primaries, and white space
>> defined in the HDR metadata.
>>
>> Internal panels (which are currently few and far between) usually
>> don't include the complex HW to do tone and color mapping on their
>> own and will require the display driver to perform appropriate
>> mapping.
> FWIW, when designing Weston's color management, we are aiming for
> the latter "simple" panels foremost, because that gives us full control
> of all color conversions and tone mappings.
>
> OTOH, if Weston has to present to a display which only accepts e.g.
> BT.2020/PQ signal, the display might always mangle the image in
> unexpected ways. Therefore I expect that by default Weston will do
> everything it can to try to make the display not apply anything magic
> image enhancement: trust that EDID description of the display gamut and
> dynamic range are correct, and use HDR metadata to tell the display
> that those values are exactly what we are using. And we use them.
>
> IMO, a display doing its tone mapping magically is only useful when you
> want to be able to use "simple" playback devices that cannot adapt to
> the display they are driving. Magic tone mapping is also a way for
> hardware vendors to differentiate, which from the color management
> perspective is harmful as it makes it more difficult or impossible to
> predict the display behaviour or to keep it consistent.
>
> So there are two opposing goals:
>
> - Traditional color management wants absolute control of the display,
>   leaving nothing unpredictable and preferably also nothing undefined.
>   Undefined behaviour can always be measured (profiled) which makes it
>   predictable and known. The viewing environment is controlled and
>   constant.
>
> - Entertainment wants the most visually impressive image quality by
>   dynamically adapting to both displayed content and to the viewing
>   environment conditions.
>
>> ## Pixel Formats
>>
>> The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are
>> unrelated to color space and EOTF definitions. HDR pixels can be
>> formatted in different ways but in order to not lose precision HDR
>> content requires at least 10 bpc precision. For this reason
>> ARGB2101010, P010, and FP16 are the obvious candidates for HDR.
>> ARGB2101010 and P010 have the advantage of requiring only half the
>> bandwidth as FP16, while FP16 has the advantage of enough precision
>> to operate in a linear space, i.e. without EOTF.
> Right.
>
>> ## Proposed use-cases
>>
>> Although the userspace side of this work is still in the early stages
>> it is clear that we will want to support the following two use-cases:
>>
>> **One XRGB2101010 HDR Plane:** A single, composited plane of HDR
>> content. The use-case is a video player on a desktop with the
>> compositor owning the composition of SDR and HDR content. The content
>> shall be PQ BT.2020 formatted. The drm_connector's
>> hdr_output_metadata shall be set.
> This use case is already possible, right?
>
>> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc
>> desktop plane, with a P010 HDR video plane underlayed. The HDR plane
>> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR
>> boost value. The drm_connector's hdr_output_metadata shall be set.
> This use case requires blending in KMS, so is the primary goal I
> suppose.
>
>> **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth
>> transition we recommend an OS that supports HDR output to provide the
>> hdr_output_metadata on the drm_connector to configure the output for
>> HDR, even when the content is only SDR. This will allow for a smooth
>> transition between SDR-only and HDR content. In this use-case the SDR
>> max luminance value should be provided on the drm_plane.
> I think this might be already possible by crafting a CRTC GAMMA LUT? Not
> sure about precision.
>
>> In DCN we will de-PQ or de-Gamma all input in order to blend in
>> linear space. For SDR content we will also apply any desired boost
>> before blending. After blending we will then re-apply the PQ EOTF and
>> do RGB to YCbCr conversion if needed.
> This assumes the same color space over everything.
>
>> ## Summary of proposed interface changes
>>
>> per drm_plane:
>> - new RGB color space definitions, mirroring the existing YUV color
>> space definitions
>> - new transfer function property
>> - new SDR maximum white level property
> How will these new KMS properties interact with per-plane DEGAMMA, CTM
> and/or GAMMA properties?
>
> Why go with your proposal instead of per-plane CTM and LUT?
>
> I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs
> both per-plane and on CRTC, would be:
>
> plane:
> 	FB -> M1 -> LUT1 -> M2 -> blending input
>
> CRTC:
> 	blending output -> LUT2 -> M3 -> connector
>
> FB: framebuffer
> M1: matrix transform, capable of converting e.g. YCbCr to RGB
> LUT1: 1D LUT for content EOTF, to produce light-linear RGB
> M2: matrix transform for color space transformation
>
> LUT2: 1D LUT for applying monitor EOTF^-1
> M3: matrix transform, e.g. if you need to push YCbCr on the connector
>
> We also need to know where and how clipping happens.
>
> I think this scheme would allow implementing everything you want, and
> it would not be tied to rigid enumerations, and it won't have any
> magical conversions done under the hood as you would need to do to
> convert from one enum space to another. It leaves the render intent to
> be defined by the userspace compositor, rather than building a fixed
> policy in the kernel.
>
> Userspace would be setting transformation operators, not color spaces,
> to the kernel, allowing the blending space to be chosen by userspace.
> In Weston we aim to choose then blending color space to be the same as
> the output color space, except in optical (linear) encoding. The output
> color space can be non-standard, e.g. measured with a display profiler
> equipment.
>
> I would expect gamut mapping, dynamic range mapping and tone mapping to
> be places where most experimentation and innovation happens, so
> implementing them in the kernel with just few or no controllable
> parameters at this time seems like it could become useless fast.
>
>
> Thanks,
> pq
>
>> ## References
>>
>> [1]
>> https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
>>
>>
>> ## Further Reading
>>
>> https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst
>> http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf
>> https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
>>
>>
>> Bhawanpreet Lakha (3):
>>   drm/color: Add RGB Color encodings
>>   drm/color: Add Color transfer functions for HDR/SDR
>>   drm/color: Add sdr boost property
>>
>>  .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |  4 +-
>>  .../gpu/drm/arm/display/komeda/komeda_plane.c |  4 +-
>>  drivers/gpu/drm/arm/malidp_planes.c           |  4 +-
>>  drivers/gpu/drm/armada/armada_overlay.c       |  4 +-
>>  drivers/gpu/drm/drm_atomic_uapi.c             |  8 ++
>>  drivers/gpu/drm/drm_color_mgmt.c              | 84
>> +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c   |
>> 4 +- .../drm/i915/display/skl_universal_plane.c    |  4 +-
>>  drivers/gpu/drm/nouveau/dispnv04/overlay.c    |  4 +-
>>  drivers/gpu/drm/omapdrm/omap_plane.c          |  4 +-
>>  drivers/gpu/drm/sun4i/sun8i_vi_layer.c        |  4 +-
>>  drivers/gpu/drm/tidss/tidss_plane.c           |  6 +-
>>  include/drm/drm_color_mgmt.h                  | 25 +++++-
>>  include/drm/drm_plane.h                       | 30 +++++++
>>  14 files changed, 173 insertions(+), 16 deletions(-)
>>
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 3059 bytes --]

On Wed, 28 Apr 2021 13:24:27 +0530
Shashank Sharma <shashank.sharma@amd.com> wrote:

> Assuming these details, A compositor will look for DRM color properties like these:
> 
> 1. Degamma plane property : To make buffers linear for Gamut mapping
> 
> 2. Gamut mapping plane property:  To gamut map SRGB buffer to BT2020 colorspace
> 
> 3. Color space conversion plane property: To convert from YCBCR->RGB
> 
> 4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H
> 
> 5. Gamma plane/CRTC property: to re-apply the output ST2084 curve
> 
> 

...

>  *
>  *
>  *
>  *             ┌─────────────────┐             ┌─────────────────┐           ┌─────────────────┐       ┌────────────────┐
>  * HDR 600 Nits│                 │HDR 600 Nits │                 │HDR600     │                 │HDR500 │                │ HDR500
>  *   ────────► │  Degamma        ├────────────►│  Color space    ├──────────►│  Tone mapping   ├──────►│  Gamma         │
>  * BT2020      │  OETF ST2084    │ BT2020      │  conversion     │BT2020     │   H2H           │BT2020 │  ST2084        │ BT2020
>  * YCBCR420    │                 │ YCBCR420    │ YCBCR->RGB      │RGB88      │   600->500      │RGB888 │                │ RGB888
>  * Non Linear  └─────────────────┘ Linear      └─────────────────┘Linear     └─────────────────┘Linear └────────────────┘ ST2084
>  */

Hi Shashank,

I think you might have degamma and color model conversion reversed, or
is that a new thing in the HDR specs?

Usually the YCbCr/RGB conversion matrix applies to non-linear values
AFAIU.

There is also confusion with OETF vs. EOTF. I got that initially wrong
too. OETF is not just a name for inverse-EOTF but it is used in a
different context. Though here it seems to be just a typo.

OETF is inherent to a camera when it converts light into
electrical signals. EOTF is inherent to a monitor when it converts
electrical signals to light. Depending on what the electrical signals
have been defined to be in each step of a broadcasting chain, you might
need OETF or EOTF or their inverse or a different OETF or EOTF or their
inverse.

As we are talking about displays and likely assuming display-referred
content (not scene-referred content), we probably have no use for OETF,
but we could have several different EOTFs.


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Shashank Sharma]

Hello Pekka,

On 30/04/21 15:13, Pekka Paalanen wrote:
> On Wed, 28 Apr 2021 13:24:27 +0530
> Shashank Sharma <shashank.sharma@amd.com> wrote:
>
>> Assuming these details, A compositor will look for DRM color properties like these:
>>
>> 1. Degamma plane property : To make buffers linear for Gamut mapping
>>
>> 2. Gamut mapping plane property:  To gamut map SRGB buffer to BT2020 colorspace
>>
>> 3. Color space conversion plane property: To convert from YCBCR->RGB
>>
>> 4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H
>>
>> 5. Gamma plane/CRTC property: to re-apply the output ST2084 curve
>>
>>
> ...
>
>>  *
>>  *
>>  *
>>  *             ┌─────────────────┐             ┌─────────────────┐           ┌─────────────────┐       ┌────────────────┐
>>  * HDR 600 Nits│                 │HDR 600 Nits │                 │HDR600     │                 │HDR500 │                │ HDR500
>>  *   ────────► │  Degamma        ├────────────►│  Color space    ├──────────►│  Tone mapping   ├──────►│  Gamma         │
>>  * BT2020      │  OETF ST2084    │ BT2020      │  conversion     │BT2020     │   H2H           │BT2020 │  ST2084        │ BT2020
>>  * YCBCR420    │                 │ YCBCR420    │ YCBCR->RGB      │RGB88      │   600->500      │RGB888 │                │ RGB888
>>  * Non Linear  └─────────────────┘ Linear      └─────────────────┘Linear     └─────────────────┘Linear └────────────────┘ ST2084
>>  */
> Hi Shashank,
>
> I think you might have degamma and color model conversion reversed, or
> is that a new thing in the HDR specs?
>
> Usually the YCbCr/RGB conversion matrix applies to non-linear values
> AFAIU.
Ah, that was due to the Gamut mapping block. You are right, color format conversion can happen on non-linear data (doesn't mean it can't happen on linear), but in the sequential block above, there was gamut mapping (color space conversion), which needs to be done on Linear space, and I was a bit too lazy to create separate blocks, so I just re[placed the block titles  :D.
> There is also confusion with OETF vs. EOTF. I got that initially wrong
> too. OETF is not just a name for inverse-EOTF but it is used in a
> different context. Though here it seems to be just a typo.
> OETF is inherent to a camera when it converts light into
> electrical signals. EOTF is inherent to a monitor when it converts
> electrical signals to light. Depending on what the electrical signals
> have been defined to be in each step of a broadcasting chain, you might
> need OETF or EOTF or their inverse or a different OETF or EOTF or their
> inverse.

Yes, that was a typo. The intention was to call it inverse curve for HDR encoded buffers. It's almost 4 years (and 2 companies) since I last did HDR, so I am a bit rusty on the topic ;) .

- Shashank

>
> As we are talking about displays and likely assuming display-referred
> content (not scene-referred content), we probably have no use for OETF,
> but we could have several different EOTFs.
>
>
> Thanks,
> pq
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Harry Wentland]

On 2021-04-30 6:39 a.m., Shashank Sharma wrote:
> Hello Pekka,
> 
> On 30/04/21 15:13, Pekka Paalanen wrote:
>> On Wed, 28 Apr 2021 13:24:27 +0530
>> Shashank Sharma <shashank.sharma@amd.com> wrote:
>>
>>> Assuming these details, A compositor will look for DRM color properties like these:
>>>
>>> 1. Degamma plane property : To make buffers linear for Gamut mapping
>>>
>>> 2. Gamut mapping plane property:  To gamut map SRGB buffer to BT2020 colorspace
>>>
>>> 3. Color space conversion plane property: To convert from YCBCR->RGB
>>>
>>> 4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H
>>>
>>> 5. Gamma plane/CRTC property: to re-apply the output ST2084 curve
>>>
>>>
>> ...
>>
>>>  *
>>>  *
>>>  *
>>>  *             ┌─────────────────┐             ┌─────────────────┐           ┌─────────────────┐       ┌────────────────┐
>>>  * HDR 600 Nits│                 │HDR 600 Nits │                 │HDR600     │                 │HDR500 │                │ HDR500
>>>  *   ────────► │  Degamma        ├────────────►│  Color space    ├──────────►│  Tone mapping   ├──────►│  Gamma         │
>>>  * BT2020      │  OETF ST2084    │ BT2020      │  conversion     │BT2020     │   H2H           │BT2020 │  ST2084        │ BT2020
>>>  * YCBCR420    │                 │ YCBCR420    │ YCBCR->RGB      │RGB88      │   600->500      │RGB888 │                │ RGB888
>>>  * Non Linear  └─────────────────┘ Linear      └─────────────────┘Linear     └─────────────────┘Linear └────────────────┘ ST2084
>>>  */
>> Hi Shashank,
>>
>> I think you might have degamma and color model conversion reversed, or
>> is that a new thing in the HDR specs?
>>
>> Usually the YCbCr/RGB conversion matrix applies to non-linear values
>> AFAIU.
> Ah, that was due to the Gamut mapping block. You are right, color format conversion can happen on non-linear data (doesn't mean it can't happen on linear), but in the sequential block above, there was gamut mapping (color space conversion), which needs to be done on Linear space, and I was a bit too lazy to create separate blocks, so I just re[placed the block titles  :D.
>> There is also confusion with OETF vs. EOTF. I got that initially wrong
>> too. OETF is not just a name for inverse-EOTF but it is used in a
>> different context. Though here it seems to be just a typo.
>> OETF is inherent to a camera when it converts light into
>> electrical signals. EOTF is inherent to a monitor when it converts
>> electrical signals to light. Depending on what the electrical signals
>> have been defined to be in each step of a broadcasting chain, you might
>> need OETF or EOTF or their inverse or a different OETF or EOTF or their
>> inverse.
> 
> Yes, that was a typo. The intention was to call it inverse curve for HDR encoded buffers. It's almost 4 years (and 2 companies) since I last did HDR, so I am a bit rusty on the topic ;) .
> 
> - Shashank
> 

Thanks, Ville and Shashank. This is indeed helpful. I apologize for the late
response but I needed to take some time to do more reading and internalize some
of the HDR and CM concepts. I will send out a v2 of my patchset but realize
that it is only a small step toward the right KMS interface for HDR and CM.

Harry

>>
>> As we are talking about displays and likely assuming display-referred
>> content (not scene-referred content), we probably have no use for OETF,
>> but we could have several different EOTFs.
>>
>>
>> Thanks,
>> pq

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Harry Wentland]

On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
> On Mon, 26 Apr 2021 13:38:49 -0400
> Harry Wentland <harry.wentland@amd.com> wrote:
> 
>> ## Introduction
>>
>> We are looking to enable HDR support for a couple of single-plane and
>> multi-plane scenarios. To do this effectively we recommend new
>> interfaces to drm_plane. Below I'll give a bit of background on HDR
>> and why we propose these interfaces.
>>
>>
>> ## Defining a pixel's luminance
>>
>> Currently the luminance space of pixels in a framebuffer/plane
>> presented to the display is not well defined. It's usually assumed to
>> be in a 2.2 or 2.4 gamma space and has no mapping to an absolute
>> luminance value but is interpreted in relative terms.
>>
>> Luminance can be measured and described in absolute terms as candela
>> per meter squared, or cd/m2, or nits. Even though a pixel value can
>> be mapped to luminance in a linear fashion to do so without losing a
>> lot of detail requires 16-bpc color depth. The reason for this is
>> that human perception can distinguish roughly between a 0.5-1%
>> luminance delta. A linear representation is suboptimal, wasting
>> precision in the highlights and losing precision in the shadows.
>>
>> A gamma curve is a decent approximation to a human's perception of
>> luminance, but the PQ (perceptual quantizer) function [1] improves on
>> it. It also defines the luminance values in absolute terms, with the
>> highest value being 10,000 nits and the lowest 0.0005 nits.
>>
>> Using a content that's defined in PQ space we can approximate the
>> real world in a much better way.
>>
>> Here are some examples of real-life objects and their approximate
>> luminance values:
>>
>> | Object            | Luminance in nits |
>> | ----------------- | ----------------- |
>> | Sun               | 1.6 million       |
>> | Fluorescent light | 10,000            |
>> | Highlights        | 1,000 - sunlight  |
>> | White Objects     | 250 - 1,000       |
>> | Typical objects   | 1 - 250           |
>> | Shadows           | 0.01 - 1          |
>> | Ultra Blacks      | 0 - 0.0005        |
>>
>>
>> ## Describing the luminance space
>>
>> **We propose a new drm_plane property to describe the Eletro-Optical
>> Transfer Function (EOTF) with which its framebuffer was composed.**
>> Examples of EOTF are:
>>
>> | EOTF      | Description                                                               |
>> | --------- |:------------------------------------------------------------------------- |
>> | Gamma 2.2 | a simple 2.2 gamma                                                        |
>> | sRGB      | 2.4 gamma with small initial linear section                               |
>> | PQ 2084   | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support |
>> | Linear    | Linear relationship between pixel value and luminance value               |
>>
> 
> The definitions agree with what I have learnt so far. However, with
> these EOTF definitions, only PQ defines absolute luminance values
> while the others do not. So this is not enough information to blend
> planes together if they do not all use the same EOTF with the same
> dynamic range. More below.
> 

Good point.

> 
>>
>> ## Mastering Luminances
>>
>> Now we are able to use the PQ 2084 EOTF to define the luminance of
>> pixels in absolute terms. Unfortunately we're again presented with
>> physical limitations of the display technologies on the market today.
>> Here are a few examples of luminance ranges of displays.
>>
>> | Display                  | Luminance range in nits |
>> | ------------------------ | ----------------------- |
>> | Typical PC display       | 0.3 - 200               |
>> | Excellent LCD HDTV       | 0.3 - 400               |
>> | HDR LCD w/ local dimming | 0.05 - 1,500            |
>>
>> Since no display can currently show the full 0.0005 to 10,000 nits
>> luminance range the display will need to tonemap the HDR content, i.e
>> to fit the content within a display's capabilities. To assist with
>> tonemapping HDR content is usually accompanied with a metadata that
>> describes (among other things) the minimum and maximum mastering
>> luminance, i.e. the maximum and minimum luminance of the display that
>> was used to master the HDR content.
>>
>> The HDR metadata is currently defined on the drm_connector via the
>> hdr_output_metadata blob property.
>>
>> It might be useful to define per-plane hdr metadata, as different
>> planes might have been mastered differently.
> 
> I don't think this would directly help with the dynamic range blending
> problem. You still need to establish the mapping between the optical
> values from two different EOTFs and dynamic ranges. Or can you know
> which optical values match the mastering display maximum and minimum
> luminances for not-PQ?
> 

My understanding of this is probably best illustrated by this example:

Assume HDR was mastered on a display with a maximum white level of 500
nits and played back on a display that supports a max white level of 400
nits. If you know the mastering white level of 500 you know that this is
the maximum value you need to compress down to 400 nits, allowing you to
use the full extent of the 400 nits panel.

If you do not know the mastering luminance is 500 nits you would
have to compress 10,000 nits down to 400 (assuming PQ), losing quite
a bit of the full 400 nits available as you'll need room to map the 500
to 10,000 nits range which in reality is completely unused. You might end
up with mapping 500 nits to 350 nits, instead of mapping it to 400.

> 
>> ## SDR Luminance
>>
>> Since SDR covers a smaller luminance range than HDR, an SDR plane
>> might look dark when blended with HDR content. Since the max HDR
>> luminance can be quite variable (200-1,500 nits on actual displays)
>> it is best to make the SDR maximum luminance value configurable.
>>
>> **We propose a drm_plane property to specfy the desired maximum
>> luminance of the SDR plane in nits.** This allows us to map the SDR
>> content predictably into HDR's absolute luminance space.
> 
> What would be the mapping? Simple linear scaling? A more complicated
> tone mapping?
> 
> Rather than "SDR luminance", do you perhaps intend this to configure
> the dynamic range of the non-absolute-luminance EOTFs?
> In that case maybe you'd need a black luminance level too?
> 

SDR luminance is essentially how AMD HW lets us deal with this, but
conceptually you're right, this could extend beyond SDR to any
non-absolute-luminance (NAL?) EOTF.

I don't think AMD HW currently has a way to program a black luminance
level unless we turn the white/black luminance into a LUT.

> 
>> ## Let There Be Color
>>
>> So far we've only talked about luminance, ignoring colors altogether.
>> Just like in the luminance space, traditionally the color space of
>> display outputs has not been well defined. Similar to how an EOTF
>> defines a mapping of pixel data to an absolute luminance value, the
>> color space maps color information for each pixel onto the CIE 1931
>> chromaticity space. This can be thought of as a mapping to an
>> absolute, real-life, color value.
>>
>> A color space is defined by its primaries and white point. The
>> primaries and white point are expressed as coordinates in the CIE
>> 1931 color space. Think of the red primary as the reddest red that
>> can be displayed within the color space. Same for green and blue.
>>
>> Examples of color spaces are:
>>
>> | Color Space | Description                                |
>> | ----------- | ------------------------------------------ |
>> | BT 601      | similar to BT 709                          |
>> | BT 709      | used by sRGB content; ~53% of BT 2020      |
>> | DCI-P3      | used by most HDR displays; ~72% of BT 2020 |
>> | BT 2020     | standard for most HDR content              |
>>
>> The color space is defined in DRM for YCbCr planes via the
>> color_encoding property of the drm_plane. 
> 
> I don't think that is quite right.
> 
> As far I understand, COLOR_ENCODING property controls which matrix is
> used to convert from YCbCr to RGB, but that is all it does. It is not
> used for the actual color space. So while these BT standards do
> specify the chromaticities, they also specify the YCbCr encoding which
> is the part used in this property.
> 
> YCbCr and RGB are color models. They are not color spaces. RGB is an
> additive color model while YCbCr is not, AFAIU. Blending requires an
> additive color model and linear luminance encoding.
> 
> You need two color space definitions to create one color space
> transformation: source color space and destination color space. You
> also need an idea *how* the two color spaces should be mapped, which is
> called "rendering intent". You can't do anything with just one color
> space definition, except to pass it on along with the pixels.
> 
> To be able to blend planes together, all planes need to be converted to
> the same color space first: the blending color space, whatever you
> choose it to be. I do not see where KMS would do this color space
> conversion, or where it would get the definition of the blending color
> space.
> 
>> **We propose to add definitions for the RGB variants of the BT color
>> spaces.**
> 
> Therefore I'm not sure this makes sense.
> 

You and Ville are absolutely correct and a reading of
https://en.wikipedia.org/wiki/YCbCr set me straight. :)

> 
>> ## Color Primaries and White Point
>>
>> Just like displays can currently not represent the entire 0.0005 -
>> 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not
>> capable of representing the entire BT.2020 color Gamut. For this
>> reason video content will often specify the color primaries and white
>> point used to master the video, in order to allow displays to be able
>> to map the image as best as possible onto the display's gamut.
>>
>>
>> ## Displays and Tonemapping
>>
>> External displays are able to do their own tone and color mapping,
>> based on the mastering luminance, color primaries, and white space
>> defined in the HDR metadata.
>>
>> Internal panels (which are currently few and far between) usually
>> don't include the complex HW to do tone and color mapping on their
>> own and will require the display driver to perform appropriate
>> mapping.
> 
> FWIW, when designing Weston's color management, we are aiming for
> the latter "simple" panels foremost, because that gives us full control
> of all color conversions and tone mappings.
> 
> OTOH, if Weston has to present to a display which only accepts e.g.
> BT.2020/PQ signal, the display might always mangle the image in
> unexpected ways. Therefore I expect that by default Weston will do
> everything it can to try to make the display not apply anything magic
> image enhancement: trust that EDID description of the display gamut and
> dynamic range are correct, and use HDR metadata to tell the display
> that those values are exactly what we are using. And we use them.
> 
> IMO, a display doing its tone mapping magically is only useful when you
> want to be able to use "simple" playback devices that cannot adapt to
> the display they are driving. Magic tone mapping is also a way for
> hardware vendors to differentiate, which from the color management
> perspective is harmful as it makes it more difficult or impossible to
> predict the display behaviour or to keep it consistent.
> 
> So there are two opposing goals:
> 
> - Traditional color management wants absolute control of the display,
>   leaving nothing unpredictable and preferably also nothing undefined.
>   Undefined behaviour can always be measured (profiled) which makes it
>   predictable and known. The viewing environment is controlled and
>   constant.
> 
> - Entertainment wants the most visually impressive image quality by
>   dynamically adapting to both displayed content and to the viewing
>   environment conditions.
> 

I feel much of the mess around HDR and CM is all the magic that everyone
tries to do to differentiate themselves. I'm not a fan of that.

One thing to watch out for is that the definitions for some of these
things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair
bit between HW vendors and we'd want to find a way to hide these
complexities to KMS clients.

>> ## Pixel Formats
>>
>> The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are
>> unrelated to color space and EOTF definitions. HDR pixels can be
>> formatted in different ways but in order to not lose precision HDR
>> content requires at least 10 bpc precision. For this reason
>> ARGB2101010, P010, and FP16 are the obvious candidates for HDR.
>> ARGB2101010 and P010 have the advantage of requiring only half the
>> bandwidth as FP16, while FP16 has the advantage of enough precision
>> to operate in a linear space, i.e. without EOTF.
> 
> Right.
> 
>> ## Proposed use-cases
>>
>> Although the userspace side of this work is still in the early stages
>> it is clear that we will want to support the following two use-cases:
>>
>> **One XRGB2101010 HDR Plane:** A single, composited plane of HDR
>> content. The use-case is a video player on a desktop with the
>> compositor owning the composition of SDR and HDR content. The content
>> shall be PQ BT.2020 formatted. The drm_connector's
>> hdr_output_metadata shall be set.
> 
> This use case is already possible, right?
> 

Yes, not sure how well tested it is but the AMD driver at least is programming
the LUTs into bypass if we don't have a degamma or regamma LUT.

>> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc
>> desktop plane, with a P010 HDR video plane underlayed. The HDR plane
>> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR
>> boost value. The drm_connector's hdr_output_metadata shall be set.
> 
> This use case requires blending in KMS, so is the primary goal I
> suppose.
> 

Correct, my understanding is that the key here is to ensure blending in KMS
is done in same blending space (linear RGB) as blending using GPU shaders and
SDR boost is applied in the same fashion.

>> **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth
>> transition we recommend an OS that supports HDR output to provide the
>> hdr_output_metadata on the drm_connector to configure the output for
>> HDR, even when the content is only SDR. This will allow for a smooth
>> transition between SDR-only and HDR content. In this use-case the SDR
>> max luminance value should be provided on the drm_plane.
> 
> I think this might be already possible by crafting a CRTC GAMMA LUT? Not
> sure about precision.
> 
>> In DCN we will de-PQ or de-Gamma all input in order to blend in
>> linear space. For SDR content we will also apply any desired boost
>> before blending. After blending we will then re-apply the PQ EOTF and
>> do RGB to YCbCr conversion if needed.
> 
> This assumes the same color space over everything.
> 
>>
>> ## Summary of proposed interface changes
>>
>> per drm_plane:
>> - new RGB color space definitions, mirroring the existing YUV color
>> space definitions
>> - new transfer function property
>> - new SDR maximum white level property
> 
> How will these new KMS properties interact with per-plane DEGAMMA, CTM
> and/or GAMMA properties?
> 
> Why go with your proposal instead of per-plane CTM and LUT?
> 
> I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs
> both per-plane and on CRTC, would be:
> 
> plane:
> 	FB -> M1 -> LUT1 -> M2 -> blending input
> 
> CRTC:
> 	blending output -> LUT2 -> M3 -> connector
> 
> FB: framebuffer
> M1: matrix transform, capable of converting e.g. YCbCr to RGB
> LUT1: 1D LUT for content EOTF, to produce light-linear RGB
> M2: matrix transform for color space transformation
> 
> LUT2: 1D LUT for applying monitor EOTF^-1
> M3: matrix transform, e.g. if you need to push YCbCr on the connector
> 
> We also need to know where and how clipping happens.
> 
> I think this scheme would allow implementing everything you want, and
> it would not be tied to rigid enumerations, and it won't have any
> magical conversions done under the hood as you would need to do to
> convert from one enum space to another. It leaves the render intent to
> be defined by the userspace compositor, rather than building a fixed
> policy in the kernel.
> 
> Userspace would be setting transformation operators, not color spaces,
> to the kernel, allowing the blending space to be chosen by userspace.
> In Weston we aim to choose then blending color space to be the same as
> the output color space, except in optical (linear) encoding. The output
> color space can be non-standard, e.g. measured with a display profiler
> equipment.
> 
> I would expect gamut mapping, dynamic range mapping and tone mapping to
> be places where most experimentation and innovation happens, so
> implementing them in the kernel with just few or no controllable
> parameters at this time seems like it could become useless fast.
> 

How important from a Wayland compositor perspective would it be to have fully
configurable LUTs per plane, as opposed to providing KMS with a named transfer
function, such as PQ or BT709, for LUT1 and LUT2?

M1 on AMD's driver is currently hidden in our driver and essentially always
set to BT709 at the moment, though we should look at color_encoding to pick
the right M1. Are the color_encoding and color_range properties enough for
what you'd like to achieve for M1?

M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be
applied at M2 by AMD's driver. I need to dig into this a bit more. It would
make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane).

I did a bunch of reading and thinking since your comments and created a v2 of
the patchset. I don't think it's where we need it to be but I hope it's a
step in the right direction. I will send it shortly.

Harry


> 
> Thanks,
> pq
> 
>> ## References
>>
>> [1]
>> https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
>>
>>
>> ## Further Reading
>>
>> https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst
>> http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf
>> https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
>>
>>
>> Bhawanpreet Lakha (3):
>>   drm/color: Add RGB Color encodings
>>   drm/color: Add Color transfer functions for HDR/SDR
>>   drm/color: Add sdr boost property
>>
>>  .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |  4 +-
>>  .../gpu/drm/arm/display/komeda/komeda_plane.c |  4 +-
>>  drivers/gpu/drm/arm/malidp_planes.c           |  4 +-
>>  drivers/gpu/drm/armada/armada_overlay.c       |  4 +-
>>  drivers/gpu/drm/drm_atomic_uapi.c             |  8 ++
>>  drivers/gpu/drm/drm_color_mgmt.c              | 84
>> +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c   |
>> 4 +- .../drm/i915/display/skl_universal_plane.c    |  4 +-
>>  drivers/gpu/drm/nouveau/dispnv04/overlay.c    |  4 +-
>>  drivers/gpu/drm/omapdrm/omap_plane.c          |  4 +-
>>  drivers/gpu/drm/sun4i/sun8i_vi_layer.c        |  4 +-
>>  drivers/gpu/drm/tidss/tidss_plane.c           |  6 +-
>>  include/drm/drm_color_mgmt.h                  | 25 +++++-
>>  include/drm/drm_plane.h                       | 30 +++++++
>>  14 files changed, 173 insertions(+), 16 deletions(-)
>>
> 

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 14665 bytes --]

On Fri, 14 May 2021 17:05:11 -0400
Harry Wentland <harry.wentland@amd.com> wrote:

> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
> > On Mon, 26 Apr 2021 13:38:49 -0400
> > Harry Wentland <harry.wentland@amd.com> wrote:

...

> >> ## Mastering Luminances
> >>
> >> Now we are able to use the PQ 2084 EOTF to define the luminance of
> >> pixels in absolute terms. Unfortunately we're again presented with
> >> physical limitations of the display technologies on the market today.
> >> Here are a few examples of luminance ranges of displays.
> >>
> >> | Display                  | Luminance range in nits |
> >> | ------------------------ | ----------------------- |
> >> | Typical PC display       | 0.3 - 200               |
> >> | Excellent LCD HDTV       | 0.3 - 400               |
> >> | HDR LCD w/ local dimming | 0.05 - 1,500            |
> >>
> >> Since no display can currently show the full 0.0005 to 10,000 nits
> >> luminance range the display will need to tonemap the HDR content, i.e
> >> to fit the content within a display's capabilities. To assist with
> >> tonemapping HDR content is usually accompanied with a metadata that
> >> describes (among other things) the minimum and maximum mastering
> >> luminance, i.e. the maximum and minimum luminance of the display that
> >> was used to master the HDR content.
> >>
> >> The HDR metadata is currently defined on the drm_connector via the
> >> hdr_output_metadata blob property.
> >>
> >> It might be useful to define per-plane hdr metadata, as different
> >> planes might have been mastered differently.  
> > 
> > I don't think this would directly help with the dynamic range blending
> > problem. You still need to establish the mapping between the optical
> > values from two different EOTFs and dynamic ranges. Or can you know
> > which optical values match the mastering display maximum and minimum
> > luminances for not-PQ?
> >   
> 
> My understanding of this is probably best illustrated by this example:
> 
> Assume HDR was mastered on a display with a maximum white level of 500
> nits and played back on a display that supports a max white level of 400
> nits. If you know the mastering white level of 500 you know that this is
> the maximum value you need to compress down to 400 nits, allowing you to
> use the full extent of the 400 nits panel.

Right, but in the kernel, where do you get these nits values from?

hdr_output_metadata blob is infoframe data to the monitor. I think this
should be independent of the metadata used for color transformations in
the display pipeline before the monitor.

EDID may tell us the monitor HDR metadata, but again what is used in
the color transformations should be independent, because EDIDs lie,
lighting environments change, and users have different preferences.

What about black levels?

Do you want to do black level adjustment?

How exactly should the compression work?

Where do you map the mid-tones?

What if the end user wants something different?

> If you do not know the mastering luminance is 500 nits you would
> have to compress 10,000 nits down to 400 (assuming PQ), losing quite
> a bit of the full 400 nits available as you'll need room to map the 500
> to 10,000 nits range which in reality is completely unused. You might end
> up with mapping 500 nits to 350 nits, instead of mapping it to 400.

The quality of the result depends on the compression (tone-mapping)
algorithm. I believe no-one will ever do a simple linear compression of
ranges.

Instead, you probably do something smooth in the black end, keep
mid-tones roughly as they are, and the again do a smooth saturation to
some "reasonable" level that goes well with the monitor in the current
lighting environment without introducing coloring artifacts, and just
clip the huge overshoot of the full PQ-range.

There are many big and small decisions to be made in how to map
out-of-gamut or out-of-brightness values into the displayable range,
and no algorithm fits all use cases. I believe this is why e.g. ICC
has several different "render intents", some of which are so vaguely
defined that they are practically undefined - just like what "a good
picture" means. You have essentially three dimensions: luminance, hue,
and saturation. Which one will you sacrifice, shift or emphasize and to
what degree to fit the square content peg into the round display hole?

A naive example: Let's say content has 300 nits red. Your display can
show max 400 nits white or max 180 nits red, and anything in between.
What will you show?

The problem is, that if UAPI does not define exactly what happens, then
taking advantage of these hardware capabilities means you have no idea
what happens to your content. This may be fine for closed systems, where
the software has been carefully built for the specific hardware
revision and the use cases of the complete system have been
pre-defined, so that it can assume what should and will happen. But
should that be exposed as a generic UAPI, when generic userspace has no
chance of knowing what it will do?


...

> >> ## Displays and Tonemapping
> >>
> >> External displays are able to do their own tone and color mapping,
> >> based on the mastering luminance, color primaries, and white space
> >> defined in the HDR metadata.
> >>
> >> Internal panels (which are currently few and far between) usually
> >> don't include the complex HW to do tone and color mapping on their
> >> own and will require the display driver to perform appropriate
> >> mapping.  
> > 
> > FWIW, when designing Weston's color management, we are aiming for
> > the latter "simple" panels foremost, because that gives us full control
> > of all color conversions and tone mappings.
> > 
> > OTOH, if Weston has to present to a display which only accepts e.g.
> > BT.2020/PQ signal, the display might always mangle the image in
> > unexpected ways. Therefore I expect that by default Weston will do
> > everything it can to try to make the display not apply anything magic
> > image enhancement: trust that EDID description of the display gamut and
> > dynamic range are correct, and use HDR metadata to tell the display
> > that those values are exactly what we are using. And we use them.
> > 
> > IMO, a display doing its tone mapping magically is only useful when you
> > want to be able to use "simple" playback devices that cannot adapt to
> > the display they are driving. Magic tone mapping is also a way for
> > hardware vendors to differentiate, which from the color management
> > perspective is harmful as it makes it more difficult or impossible to
> > predict the display behaviour or to keep it consistent.
> > 
> > So there are two opposing goals:
> > 
> > - Traditional color management wants absolute control of the display,
> >   leaving nothing unpredictable and preferably also nothing undefined.
> >   Undefined behaviour can always be measured (profiled) which makes it
> >   predictable and known. The viewing environment is controlled and
> >   constant.
> > 
> > - Entertainment wants the most visually impressive image quality by
> >   dynamically adapting to both displayed content and to the viewing
> >   environment conditions.
> >   
> 
> I feel much of the mess around HDR and CM is all the magic that everyone
> tries to do to differentiate themselves. I'm not a fan of that.
> 
> One thing to watch out for is that the definitions for some of these
> things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair
> bit between HW vendors and we'd want to find a way to hide these
> complexities to KMS clients.

How could we hide them?

Actually, what do you mean with "hide"?


...

> >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc
> >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane
> >> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR
> >> boost value. The drm_connector's hdr_output_metadata shall be set.  
> > 
> > This use case requires blending in KMS, so is the primary goal I
> > suppose.
> >   
> 
> Correct, my understanding is that the key here is to ensure blending in KMS
> is done in same blending space (linear RGB) as blending using GPU shaders and
> SDR boost is applied in the same fashion.

Yes, but this is also something I think cannot be achieved by telling
the kernel about color spaces and letting the driver come up with a
color transformation. I believe this can only be achieved by telling
the kernel the color transformation directly, and also describing to
userspace what kind of color transformations are possible (1D LUT, 3D
LUT, other; precision...).

...

> >> ## Summary of proposed interface changes
> >>
> >> per drm_plane:
> >> - new RGB color space definitions, mirroring the existing YUV color
> >> space definitions
> >> - new transfer function property
> >> - new SDR maximum white level property  
> > 
> > How will these new KMS properties interact with per-plane DEGAMMA, CTM
> > and/or GAMMA properties?
> > 
> > Why go with your proposal instead of per-plane CTM and LUT?
> > 
> > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs
> > both per-plane and on CRTC, would be:
> > 
> > plane:
> > 	FB -> M1 -> LUT1 -> M2 -> blending input
> > 
> > CRTC:
> > 	blending output -> LUT2 -> M3 -> connector
> > 
> > FB: framebuffer
> > M1: matrix transform, capable of converting e.g. YCbCr to RGB
> > LUT1: 1D LUT for content EOTF, to produce light-linear RGB
> > M2: matrix transform for color space transformation
> > 
> > LUT2: 1D LUT for applying monitor EOTF^-1
> > M3: matrix transform, e.g. if you need to push YCbCr on the connector
> > 
> > We also need to know where and how clipping happens.
> > 
> > I think this scheme would allow implementing everything you want, and
> > it would not be tied to rigid enumerations, and it won't have any
> > magical conversions done under the hood as you would need to do to
> > convert from one enum space to another. It leaves the render intent to
> > be defined by the userspace compositor, rather than building a fixed
> > policy in the kernel.
> > 
> > Userspace would be setting transformation operators, not color spaces,
> > to the kernel, allowing the blending space to be chosen by userspace.
> > In Weston we aim to choose then blending color space to be the same as
> > the output color space, except in optical (linear) encoding. The output
> > color space can be non-standard, e.g. measured with a display profiler
> > equipment.
> > 
> > I would expect gamut mapping, dynamic range mapping and tone mapping to
> > be places where most experimentation and innovation happens, so
> > implementing them in the kernel with just few or no controllable
> > parameters at this time seems like it could become useless fast.
> >   
> 
> How important from a Wayland compositor perspective would it be to have fully
> configurable LUTs per plane, as opposed to providing KMS with a named transfer
> function, such as PQ or BT709, for LUT1 and LUT2?

For Wayland desktop compositors, absolutely essential in my opinion.

I would strongly advice against any desktop compositor from using the
"named transfer function" API and instead do everything in GPU shaders
if no other KMS way exists, unless the KMS UAPI definition gives the
exact mathematical formula of what that step will do.

For entertainment devices using Wayland as an internal detail, who
cares. The system manufacturer makes things work any way they want, and
you cannot run non-curated apps on those devices, so there are no
compatibility issues.

Then there would be systems between those two extremes, which would
likely use a mixture of those approaches, e.g. Kodi on bare hardware I
imagine (no Wayland there, but Wayland is kind of unrelated except for
the ideology).

Mind, I'm not talking only about EOTFs here, but thinking of the whole
color management. EOTFs per se are simple. Tone-mapping, including gamut
mapping and luminance mapping, are hard because there is no one correct
formula.

However, if we are talking about *only* EOTFs, then an enumeration of
the curves could be usable. But you were also talking about configuring
dynamic ranges and doing luminance range mapping in the driver, which I
think won't work with enums, but could perhaps be accounted for in M2
by userspace. Then again, if you have enums and not a LUT, you cannot
implement custom non-linear per-channel mapping which might be useful
for e.g. adapting SDR content for a HDR blending space. Or HDR content
to a different HDR blending space.

See, a LUT can be more than just EOTF: it can apply a part of the
tone-mapping too. An enum cannot.

I have a feeling that most tone-mapping curves would be non-linear with
some sort of S-shape and clipping of black and white ends. But that is
also a naive view, because tone-mapping is not a 1D-curve, it's a
mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do
600 nit red for instance. On a 600 nit monitor you can do exactly one
color with 600 nits: white.

> M1 on AMD's driver is currently hidden in our driver and essentially always
> set to BT709 at the moment, though we should look at color_encoding to pick
> the right M1. Are the color_encoding and color_range properties enough for
> what you'd like to achieve for M1?

Probably yes, for standard format content, until a new standard comes up
again and then you have to add another enum value in the UAPI. M1 is
probably the best place where an enum fits in the pipeline though, and
for M3 as well because non-RGB inputs to monitors are quite
standardised, and at those steps I would assume there are no parts of
tone-mapping done.

> M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be
> applied at M2 by AMD's driver. I need to dig into this a bit more. It would
> make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane).

Good, it's really important to make sure that all drivers agree on the
pipeline layout that is defined(???) in the KMS UAPI specification.

> I did a bunch of reading and thinking since your comments and created a v2 of
> the patchset. I don't think it's where we need it to be but I hope it's a
> step in the right direction. I will send it shortly.

I have to say I'm still very sceptical about this enum-based approach
as you have noticed. :-)


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Sebastian Wick]

On 2021-05-17 10:57, Pekka Paalanen wrote:
> On Fri, 14 May 2021 17:05:11 -0400
> Harry Wentland <harry.wentland@amd.com> wrote:
> 
>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
>> > On Mon, 26 Apr 2021 13:38:49 -0400
>> > Harry Wentland <harry.wentland@amd.com> wrote:
> 
> ...
> 
>> >> ## Mastering Luminances
>> >>
>> >> Now we are able to use the PQ 2084 EOTF to define the luminance of
>> >> pixels in absolute terms. Unfortunately we're again presented with
>> >> physical limitations of the display technologies on the market today.
>> >> Here are a few examples of luminance ranges of displays.
>> >>
>> >> | Display                  | Luminance range in nits |
>> >> | ------------------------ | ----------------------- |
>> >> | Typical PC display       | 0.3 - 200               |
>> >> | Excellent LCD HDTV       | 0.3 - 400               |
>> >> | HDR LCD w/ local dimming | 0.05 - 1,500            |
>> >>
>> >> Since no display can currently show the full 0.0005 to 10,000 nits
>> >> luminance range the display will need to tonemap the HDR content, i.e
>> >> to fit the content within a display's capabilities. To assist with
>> >> tonemapping HDR content is usually accompanied with a metadata that
>> >> describes (among other things) the minimum and maximum mastering
>> >> luminance, i.e. the maximum and minimum luminance of the display that
>> >> was used to master the HDR content.
>> >>
>> >> The HDR metadata is currently defined on the drm_connector via the
>> >> hdr_output_metadata blob property.
>> >>
>> >> It might be useful to define per-plane hdr metadata, as different
>> >> planes might have been mastered differently.
>> >
>> > I don't think this would directly help with the dynamic range blending
>> > problem. You still need to establish the mapping between the optical
>> > values from two different EOTFs and dynamic ranges. Or can you know
>> > which optical values match the mastering display maximum and minimum
>> > luminances for not-PQ?
>> >
>> 
>> My understanding of this is probably best illustrated by this example:
>> 
>> Assume HDR was mastered on a display with a maximum white level of 500
>> nits and played back on a display that supports a max white level of 
>> 400
>> nits. If you know the mastering white level of 500 you know that this 
>> is
>> the maximum value you need to compress down to 400 nits, allowing you 
>> to
>> use the full extent of the 400 nits panel.
> 
> Right, but in the kernel, where do you get these nits values from?
> 
> hdr_output_metadata blob is infoframe data to the monitor. I think this
> should be independent of the metadata used for color transformations in
> the display pipeline before the monitor.
> 
> EDID may tell us the monitor HDR metadata, but again what is used in
> the color transformations should be independent, because EDIDs lie,
> lighting environments change, and users have different preferences.
> 
> What about black levels?
> 
> Do you want to do black level adjustment?
> 
> How exactly should the compression work?
> 
> Where do you map the mid-tones?
> 
> What if the end user wants something different?

I suspect that this is not about tone mapping at all. The use cases
listed always have the display in PQ mode and just assume that no
content exceeds the PQ limitations. Then you can simply bring all
content to the color space with a matrix multiplication and then map the
linear light content somewhere into the PQ range. Tone mapping is
performed in the display only.

 From a generic wayland compositor point of view this is uninteresting.

I completely agree with what you said below though. I would even argue
that all generic KMS abstract pipeline elements must have a well defined
place in the pipeline and follow an exact specified formula.

> 
>> If you do not know the mastering luminance is 500 nits you would
>> have to compress 10,000 nits down to 400 (assuming PQ), losing quite
>> a bit of the full 400 nits available as you'll need room to map the 
>> 500
>> to 10,000 nits range which in reality is completely unused. You might 
>> end
>> up with mapping 500 nits to 350 nits, instead of mapping it to 400.
> 
> The quality of the result depends on the compression (tone-mapping)
> algorithm. I believe no-one will ever do a simple linear compression of
> ranges.
> 
> Instead, you probably do something smooth in the black end, keep
> mid-tones roughly as they are, and the again do a smooth saturation to
> some "reasonable" level that goes well with the monitor in the current
> lighting environment without introducing coloring artifacts, and just
> clip the huge overshoot of the full PQ-range.
> 
> There are many big and small decisions to be made in how to map
> out-of-gamut or out-of-brightness values into the displayable range,
> and no algorithm fits all use cases. I believe this is why e.g. ICC
> has several different "render intents", some of which are so vaguely
> defined that they are practically undefined - just like what "a good
> picture" means. You have essentially three dimensions: luminance, hue,
> and saturation. Which one will you sacrifice, shift or emphasize and to
> what degree to fit the square content peg into the round display hole?
> 
> A naive example: Let's say content has 300 nits red. Your display can
> show max 400 nits white or max 180 nits red, and anything in between.
> What will you show?
> 
> The problem is, that if UAPI does not define exactly what happens, then
> taking advantage of these hardware capabilities means you have no idea
> what happens to your content. This may be fine for closed systems, 
> where
> the software has been carefully built for the specific hardware
> revision and the use cases of the complete system have been
> pre-defined, so that it can assume what should and will happen. But
> should that be exposed as a generic UAPI, when generic userspace has no
> chance of knowing what it will do?
> 
> 
> ...
> 
>> >> ## Displays and Tonemapping
>> >>
>> >> External displays are able to do their own tone and color mapping,
>> >> based on the mastering luminance, color primaries, and white space
>> >> defined in the HDR metadata.
>> >>
>> >> Internal panels (which are currently few and far between) usually
>> >> don't include the complex HW to do tone and color mapping on their
>> >> own and will require the display driver to perform appropriate
>> >> mapping.
>> >
>> > FWIW, when designing Weston's color management, we are aiming for
>> > the latter "simple" panels foremost, because that gives us full control
>> > of all color conversions and tone mappings.
>> >
>> > OTOH, if Weston has to present to a display which only accepts e.g.
>> > BT.2020/PQ signal, the display might always mangle the image in
>> > unexpected ways. Therefore I expect that by default Weston will do
>> > everything it can to try to make the display not apply anything magic
>> > image enhancement: trust that EDID description of the display gamut and
>> > dynamic range are correct, and use HDR metadata to tell the display
>> > that those values are exactly what we are using. And we use them.
>> >
>> > IMO, a display doing its tone mapping magically is only useful when you
>> > want to be able to use "simple" playback devices that cannot adapt to
>> > the display they are driving. Magic tone mapping is also a way for
>> > hardware vendors to differentiate, which from the color management
>> > perspective is harmful as it makes it more difficult or impossible to
>> > predict the display behaviour or to keep it consistent.
>> >
>> > So there are two opposing goals:
>> >
>> > - Traditional color management wants absolute control of the display,
>> >   leaving nothing unpredictable and preferably also nothing undefined.
>> >   Undefined behaviour can always be measured (profiled) which makes it
>> >   predictable and known. The viewing environment is controlled and
>> >   constant.
>> >
>> > - Entertainment wants the most visually impressive image quality by
>> >   dynamically adapting to both displayed content and to the viewing
>> >   environment conditions.
>> >
>> 
>> I feel much of the mess around HDR and CM is all the magic that 
>> everyone
>> tries to do to differentiate themselves. I'm not a fan of that.
>> 
>> One thing to watch out for is that the definitions for some of these
>> things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair
>> bit between HW vendors and we'd want to find a way to hide these
>> complexities to KMS clients.
> 
> How could we hide them?
> 
> Actually, what do you mean with "hide"?
> 
> 
> ...
> 
>> >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc
>> >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane
>> >> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR
>> >> boost value. The drm_connector's hdr_output_metadata shall be set.
>> >
>> > This use case requires blending in KMS, so is the primary goal I
>> > suppose.
>> >
>> 
>> Correct, my understanding is that the key here is to ensure blending 
>> in KMS
>> is done in same blending space (linear RGB) as blending using GPU 
>> shaders and
>> SDR boost is applied in the same fashion.
> 
> Yes, but this is also something I think cannot be achieved by telling
> the kernel about color spaces and letting the driver come up with a
> color transformation. I believe this can only be achieved by telling
> the kernel the color transformation directly, and also describing to
> userspace what kind of color transformations are possible (1D LUT, 3D
> LUT, other; precision...).
> 
> ...
> 
>> >> ## Summary of proposed interface changes
>> >>
>> >> per drm_plane:
>> >> - new RGB color space definitions, mirroring the existing YUV color
>> >> space definitions
>> >> - new transfer function property
>> >> - new SDR maximum white level property
>> >
>> > How will these new KMS properties interact with per-plane DEGAMMA, CTM
>> > and/or GAMMA properties?
>> >
>> > Why go with your proposal instead of per-plane CTM and LUT?
>> >
>> > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs
>> > both per-plane and on CRTC, would be:
>> >
>> > plane:
>> > 	FB -> M1 -> LUT1 -> M2 -> blending input
>> >
>> > CRTC:
>> > 	blending output -> LUT2 -> M3 -> connector
>> >
>> > FB: framebuffer
>> > M1: matrix transform, capable of converting e.g. YCbCr to RGB
>> > LUT1: 1D LUT for content EOTF, to produce light-linear RGB
>> > M2: matrix transform for color space transformation
>> >
>> > LUT2: 1D LUT for applying monitor EOTF^-1
>> > M3: matrix transform, e.g. if you need to push YCbCr on the connector
>> >
>> > We also need to know where and how clipping happens.
>> >
>> > I think this scheme would allow implementing everything you want, and
>> > it would not be tied to rigid enumerations, and it won't have any
>> > magical conversions done under the hood as you would need to do to
>> > convert from one enum space to another. It leaves the render intent to
>> > be defined by the userspace compositor, rather than building a fixed
>> > policy in the kernel.
>> >
>> > Userspace would be setting transformation operators, not color spaces,
>> > to the kernel, allowing the blending space to be chosen by userspace.
>> > In Weston we aim to choose then blending color space to be the same as
>> > the output color space, except in optical (linear) encoding. The output
>> > color space can be non-standard, e.g. measured with a display profiler
>> > equipment.
>> >
>> > I would expect gamut mapping, dynamic range mapping and tone mapping to
>> > be places where most experimentation and innovation happens, so
>> > implementing them in the kernel with just few or no controllable
>> > parameters at this time seems like it could become useless fast.
>> >
>> 
>> How important from a Wayland compositor perspective would it be to 
>> have fully
>> configurable LUTs per plane, as opposed to providing KMS with a named 
>> transfer
>> function, such as PQ or BT709, for LUT1 and LUT2?
> 
> For Wayland desktop compositors, absolutely essential in my opinion.
> 
> I would strongly advice against any desktop compositor from using the
> "named transfer function" API and instead do everything in GPU shaders
> if no other KMS way exists, unless the KMS UAPI definition gives the
> exact mathematical formula of what that step will do.
> 
> For entertainment devices using Wayland as an internal detail, who
> cares. The system manufacturer makes things work any way they want, and
> you cannot run non-curated apps on those devices, so there are no
> compatibility issues.
> 
> Then there would be systems between those two extremes, which would
> likely use a mixture of those approaches, e.g. Kodi on bare hardware I
> imagine (no Wayland there, but Wayland is kind of unrelated except for
> the ideology).
> 
> Mind, I'm not talking only about EOTFs here, but thinking of the whole
> color management. EOTFs per se are simple. Tone-mapping, including 
> gamut
> mapping and luminance mapping, are hard because there is no one correct
> formula.
> 
> However, if we are talking about *only* EOTFs, then an enumeration of
> the curves could be usable. But you were also talking about configuring
> dynamic ranges and doing luminance range mapping in the driver, which I
> think won't work with enums, but could perhaps be accounted for in M2
> by userspace. Then again, if you have enums and not a LUT, you cannot
> implement custom non-linear per-channel mapping which might be useful
> for e.g. adapting SDR content for a HDR blending space. Or HDR content
> to a different HDR blending space.
> 
> See, a LUT can be more than just EOTF: it can apply a part of the
> tone-mapping too. An enum cannot.
> 
> I have a feeling that most tone-mapping curves would be non-linear with
> some sort of S-shape and clipping of black and white ends. But that is
> also a naive view, because tone-mapping is not a 1D-curve, it's a
> mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do
> 600 nit red for instance. On a 600 nit monitor you can do exactly one
> color with 600 nits: white.
> 
>> M1 on AMD's driver is currently hidden in our driver and essentially 
>> always
>> set to BT709 at the moment, though we should look at color_encoding to 
>> pick
>> the right M1. Are the color_encoding and color_range properties enough 
>> for
>> what you'd like to achieve for M1?
> 
> Probably yes, for standard format content, until a new standard comes 
> up
> again and then you have to add another enum value in the UAPI. M1 is
> probably the best place where an enum fits in the pipeline though, and
> for M3 as well because non-RGB inputs to monitors are quite
> standardised, and at those steps I would assume there are no parts of
> tone-mapping done.
> 
>> M3, currently the only matrix exposed by KMS (on drm_crtc) currently 
>> seems to be
>> applied at M2 by AMD's driver. I need to dig into this a bit more. It 
>> would
>> make sense to expose M2 and M3 as part of KMS (on drm_crtc and 
>> drm_plane).
> 
> Good, it's really important to make sure that all drivers agree on the
> pipeline layout that is defined(???) in the KMS UAPI specification.
> 
>> I did a bunch of reading and thinking since your comments and created 
>> a v2 of
>> the patchset. I don't think it's where we need it to be but I hope 
>> it's a
>> step in the right direction. I will send it shortly.
> 
> I have to say I'm still very sceptical about this enum-based approach
> as you have noticed. :-)
> 
> 
> Thanks,
> pq

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Vitaly Prosyak]

On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
> On 2021-05-17 10:57, Pekka Paalanen wrote:
>> On Fri, 14 May 2021 17:05:11 -0400
>> Harry Wentland <harry.wentland@amd.com> wrote:
>>
>>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
>>> > On Mon, 26 Apr 2021 13:38:49 -0400
>>> > Harry Wentland <harry.wentland@amd.com> wrote:
>>
>> ...
>>
>>> >> ## Mastering Luminances
>>> >>
>>> >> Now we are able to use the PQ 2084 EOTF to define the luminance of
>>> >> pixels in absolute terms. Unfortunately we're again presented with
>>> >> physical limitations of the display technologies on the market 
>>> today.
>>> >> Here are a few examples of luminance ranges of displays.
>>> >>
>>> >> | Display                  | Luminance range in nits |
>>> >> | ------------------------ | ----------------------- |
>>> >> | Typical PC display       | 0.3 - 200 |
>>> >> | Excellent LCD HDTV       | 0.3 - 400 |
>>> >> | HDR LCD w/ local dimming | 0.05 - 1,500 |
>>> >>
>>> >> Since no display can currently show the full 0.0005 to 10,000 nits
>>> >> luminance range the display will need to tonemap the HDR content, 
>>> i.e
>>> >> to fit the content within a display's capabilities. To assist with
>>> >> tonemapping HDR content is usually accompanied with a metadata that
>>> >> describes (among other things) the minimum and maximum mastering
>>> >> luminance, i.e. the maximum and minimum luminance of the display 
>>> that
>>> >> was used to master the HDR content.
>>> >>
>>> >> The HDR metadata is currently defined on the drm_connector via the
>>> >> hdr_output_metadata blob property.
>>> >>
>>> >> It might be useful to define per-plane hdr metadata, as different
>>> >> planes might have been mastered differently.
>>> >
>>> > I don't think this would directly help with the dynamic range 
>>> blending
>>> > problem. You still need to establish the mapping between the optical
>>> > values from two different EOTFs and dynamic ranges. Or can you know
>>> > which optical values match the mastering display maximum and minimum
>>> > luminances for not-PQ?
>>> >
>>>
>>> My understanding of this is probably best illustrated by this example:
>>>
>>> Assume HDR was mastered on a display with a maximum white level of 500
>>> nits and played back on a display that supports a max white level of 
>>> 400
>>> nits. If you know the mastering white level of 500 you know that 
>>> this is
>>> the maximum value you need to compress down to 400 nits, allowing 
>>> you to
>>> use the full extent of the 400 nits panel.
>>
>> Right, but in the kernel, where do you get these nits values from?
>>
>> hdr_output_metadata blob is infoframe data to the monitor. I think this
>> should be independent of the metadata used for color transformations in
>> the display pipeline before the monitor.
>>
>> EDID may tell us the monitor HDR metadata, but again what is used in
>> the color transformations should be independent, because EDIDs lie,
>> lighting environments change, and users have different preferences.
>>
>> What about black levels?
>>
>> Do you want to do black level adjustment?
>>
>> How exactly should the compression work?
>>
>> Where do you map the mid-tones?
>>
>> What if the end user wants something different?
>
> I suspect that this is not about tone mapping at all. The use cases
> listed always have the display in PQ mode and just assume that no
> content exceeds the PQ limitations. Then you can simply bring all
> content to the color space with a matrix multiplication and then map the
> linear light content somewhere into the PQ range. Tone mapping is
> performed in the display only.
>
> From a generic wayland compositor point of view this is uninteresting.
>
It a compositor's decision to provide or not the metadata property to 
the kernel. The metadata can be available from one or multiple clients 
or most likely not available at all.

Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM 
occurs in display ) or NATIVE mode and do not attach any metadata to the 
connector and do TM in compositor.

It is all about user preference or compositor design, or a combination 
of both options.


> I completely agree with what you said below though. I would even argue
> that all generic KMS abstract pipeline elements must have a well defined
> place in the pipeline and follow an exact specified formula.
>
>>
>>> If you do not know the mastering luminance is 500 nits you would
>>> have to compress 10,000 nits down to 400 (assuming PQ), losing quite
>>> a bit of the full 400 nits available as you'll need room to map the 500
>>> to 10,000 nits range which in reality is completely unused. You 
>>> might end
>>> up with mapping 500 nits to 350 nits, instead of mapping it to 400.
>>
>> The quality of the result depends on the compression (tone-mapping)
>> algorithm. I believe no-one will ever do a simple linear compression of
>> ranges.
>>
>> Instead, you probably do something smooth in the black end, keep
>> mid-tones roughly as they are, and the again do a smooth saturation to
>> some "reasonable" level that goes well with the monitor in the current
>> lighting environment without introducing coloring artifacts, and just
>> clip the huge overshoot of the full PQ-range.
>>
>> There are many big and small decisions to be made in how to map
>> out-of-gamut or out-of-brightness values into the displayable range,
>> and no algorithm fits all use cases. I believe this is why e.g. ICC
>> has several different "render intents", some of which are so vaguely
>> defined that they are practically undefined - just like what "a good
>> picture" means. You have essentially three dimensions: luminance, hue,
>> and saturation. Which one will you sacrifice, shift or emphasize and to
>> what degree to fit the square content peg into the round display hole?
>>
>> A naive example: Let's say content has 300 nits red. Your display can
>> show max 400 nits white or max 180 nits red, and anything in between.
>> What will you show?
>>
>> The problem is, that if UAPI does not define exactly what happens, then
>> taking advantage of these hardware capabilities means you have no idea
>> what happens to your content. This may be fine for closed systems, where
>> the software has been carefully built for the specific hardware
>> revision and the use cases of the complete system have been
>> pre-defined, so that it can assume what should and will happen. But
>> should that be exposed as a generic UAPI, when generic userspace has no
>> chance of knowing what it will do?
>>
>>
>> ...
>>
>>> >> ## Displays and Tonemapping
>>> >>
>>> >> External displays are able to do their own tone and color mapping,
>>> >> based on the mastering luminance, color primaries, and white space
>>> >> defined in the HDR metadata.
>>> >>
>>> >> Internal panels (which are currently few and far between) usually
>>> >> don't include the complex HW to do tone and color mapping on their
>>> >> own and will require the display driver to perform appropriate
>>> >> mapping.
>>> >
>>> > FWIW, when designing Weston's color management, we are aiming for
>>> > the latter "simple" panels foremost, because that gives us full 
>>> control
>>> > of all color conversions and tone mappings.
>>> >
>>> > OTOH, if Weston has to present to a display which only accepts e.g.
>>> > BT.2020/PQ signal, the display might always mangle the image in
>>> > unexpected ways. Therefore I expect that by default Weston will do
>>> > everything it can to try to make the display not apply anything magic
>>> > image enhancement: trust that EDID description of the display 
>>> gamut and
>>> > dynamic range are correct, and use HDR metadata to tell the display
>>> > that those values are exactly what we are using. And we use them.
>>> >
>>> > IMO, a display doing its tone mapping magically is only useful 
>>> when you
>>> > want to be able to use "simple" playback devices that cannot adapt to
>>> > the display they are driving. Magic tone mapping is also a way for
>>> > hardware vendors to differentiate, which from the color management
>>> > perspective is harmful as it makes it more difficult or impossible to
>>> > predict the display behaviour or to keep it consistent.
>>> >
>>> > So there are two opposing goals:
>>> >
>>> > - Traditional color management wants absolute control of the display,
>>> >   leaving nothing unpredictable and preferably also nothing 
>>> undefined.
>>> >   Undefined behaviour can always be measured (profiled) which 
>>> makes it
>>> >   predictable and known. The viewing environment is controlled and
>>> >   constant.
>>> >
>>> > - Entertainment wants the most visually impressive image quality by
>>> >   dynamically adapting to both displayed content and to the viewing
>>> >   environment conditions.
>>> >
>>>
>>> I feel much of the mess around HDR and CM is all the magic that 
>>> everyone
>>> tries to do to differentiate themselves. I'm not a fan of that. occures
>>>
>>> One thing to watch out for is that the definitions for some of these
>>> things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair
>>> bit between HW vendors and we'd want to find a way to hide these
>>> complexities to KMS clients.
>>
>> How could we hide them?
>>
>> Actually, what do you mean with "hide"?
>>
>>
>> ...
>>
>>> >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc
>>> >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane
>>> >> shall be PQ BT.2020 formatted. The desktop plane shall specify an 
>>> SDR
>>> >> boost value. The drm_connector's hdr_output_metadata shall be set.
>>> >
>>> > This use case requires blending in KMS, so is the primary goal I
>>> > suppose.
>>> >
>>>
>>> Correct, my understanding is that the key here is to ensure blending 
>>> in KMS
>>> is done in same blending space (linear RGB) as blending using GPU 
>>> shaders and
>>> SDR boost is applied in the same fashion.
>>
>> Yes, but this is also something I think cannot be achieved by telling
>> the kernel about color spaces and letting the driver come up with a
>> color transformation. I believe this can only be achieved by telling
>> the kernel the color transformation directly, and also describing to
>> userspace what kind of color transformations are possible (1D LUT, 3D
>> LUT, other; precision...).
>>
>> ...
>>
>>> >> ## Summary of proposed interface changes
>>> >>
>>> >> per drm_plane:
>>> >> - new RGB color space definitions, mirroring the existing YUV color
>>> >> space definitions
>>> >> - new transfer function property
>>> >> - new SDR maximum white level property
>>> >
>>> > How will these new KMS properties interact with per-plane DEGAMMA, 
>>> CTM
>>> > and/or GAMMA properties?
>>> >
>>> > Why go with your proposal instead of per-plane CTM and LUT?
>>> >
>>> > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs
>>> > both per-plane and on CRTC, would be:
>>> >
>>> > plane:
>>> >     FB -> M1 -> LUT1 -> M2 -> blending input
>>> >
>>> > CRTC:
>>> >     blending output -> LUT2 -> M3 -> connector
>>> >
>>> > FB: framebuffer
>>> > M1: matrix transform, capable of converting e.g. YCbCr to RGB
>>> > LUT1: 1D LUT for content EOTF, to produce light-linear RGB
>>> > M2: matrix transform for color space transformation
>>> >
>>> > LUT2: 1D LUT for applying monitor EOTF^-1
>>> > M3: matrix transform, e.g. if you need to push YCbCr on the connector
>>> >
>>> > We also need to know where and how clipping happens.
>>> >
>>> > I think this scheme would allow implementing everything you want, and
>>> > it would not be tied to rigid enumerations, and it won't have any
>>> > magical conversions done under the hood as you would need to do to
>>> > convert from one enum space to another. It leaves the render 
>>> intent to
>>> > be defined by the userspace compositor, rather than building a fixed
>>> > policy in the kernel.
>>> >
>>> > Userspace would be setting transformation operators, not color 
>>> spaces,
>>> > to the kernel, allowing the blending space to be chosen by userspace.
>>> > In Weston we aim to choose then blending color space to be the 
>>> same as
>>> > the output color space, except in optical (linear) encoding. The 
>>> output
>>> > color space can be non-standard, e.g. measured with a display 
>>> profiler
>>> > equipment.
>>> >
>>> > I would expect gamut mapping, dynamic range mapping and tone 
>>> mapping to
>>> > be places where most experimentation and innovation happens, so
>>> > implementing them in the kernel with just few or no controllable
>>> > parameters at this time seems like it could become useless fast.
>>> >
>>>
>>> How important from a Wayland compositor perspective would it be to 
>>> have fully
>>> configurable LUTs per plane, as opposed to providing KMS with a 
>>> named transfer
>>> function, such as PQ or BT709, for LUT1 and LUT2?
>>
>> For Wayland desktop compositors, absolutely essential in my opinion.
>>
>> I would strongly advice against any desktop compositor from using the
>> "named transfer function" API and instead do everything in GPU shaders
>> if no other KMS way exists, unless the KMS UAPI definition gives the
>> exact mathematical formula of what that step will do.
>>
>> For entertainment devices using Wayland as an internal detail, who
>> cares. The system manufacturer makes things work any way they want, and
>> you cannot run non-curated apps on those devices, so there are no
>> compatibility issues.
>>
>> Then there would be systems between those two extremes, which would
>> likely use a mixture of those approaches, e.g. Kodi on bare hardware I
>> imagine (no Wayland there, but Wayland is kind of unrelated except for
>> the ideology).
>>
>> Mind, I'm not talking only about EOTFs here, but thinking of the whole
>> color management. EOTFs per se are simple. Tone-mapping, including gamut
>> mapping and luminance mapping, are hard because there is no one correct
>> formula.
>>
>> However, if we are talking about *only* EOTFs, then an enumeration of
>> the curves could be usable. But you were also talking about configuring
>> dynamic ranges and doing luminance range mapping in the driver, which I
>> think won't work with enums, but could perhaps be accounted for in M2
>> by userspace. Then again, if you have enums and not a LUT, you cannot
>> implement custom non-linear per-channel mapping which might be useful
>> for e.g. adapting SDR content for a HDR blending space. Or HDR content
>> to a different HDR blending space.
>>
>> See, a LUT can be more than just EOTF: it can apply a part of the
>> tone-mapping too. An enum cannot.
>>
>> I have a feeling that most tone-mapping curves would be non-linear with
>> some sort of S-shape and clipping of black and white ends. But that is
>> also a naive view, because tone-mapping is not a 1D-curve, it's a
>> mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do
>> 600 nit red for instance. On a 600 nit monitor you can do exactly one
>> color with 600 nits: white.
>>
>>> M1 on AMD's driver is currently hidden in our driver and essentially 
>>> always
>>> set to BT709 at the moment, though we should look at color_encoding 
>>> to pick
>>> the right M1. Are the color_encoding and color_range properties 
>>> enough for
>>> what you'd like to achieve for M1?
>>
>> Probably yes, for standard format content, until a new standard comes up
>> again and then you have to add another enum value in the UAPI. M1 is
>> probably the best place where an enum fits in the pipeline though, and
>> for M3 as well because non-RGB inputs to monitors are quite
>> standardised, and at those steps I would assume there are no parts of
>> tone-mapping done.
>>
>>> M3, currently the only matrix exposed by KMS (on drm_crtc) currently 
>>> seems to be
>>> applied at M2 by AMD's driver. I need to dig into this a bit more. 
>>> It would
>>> make sense to expose M2 and M3 as part of KMS (on drm_crtc and 
>>> drm_plane).
>>
>> Good, it's really important to make sure that all drivers agree on the
>> pipeline layout that is defined(???) in the KMS UAPI specification.
>>
>>> I did a bunch of reading and thinking since your comments and 
>>> created a v2 of
>>> the patchset. I don't think it's where we need it to be but I hope 
>>> it's a
>>> step in the right direction. I will send it shortly.
>>
>> I have to say I'm still very sceptical about this enum-based approach
>> as you have noticed. :-)
>>
>>
>> Thanks,
>> pq

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 5859 bytes --]

On Mon, 17 May 2021 15:39:03 -0400
Vitaly Prosyak <vitaly.prosyak@amd.com> wrote:

> On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
> > On 2021-05-17 10:57, Pekka Paalanen wrote:  
> >> On Fri, 14 May 2021 17:05:11 -0400
> >> Harry Wentland <harry.wentland@amd.com> wrote:
> >>  
> >>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:  
> >>> > On Mon, 26 Apr 2021 13:38:49 -0400
> >>> > Harry Wentland <harry.wentland@amd.com> wrote:  
> >>
> >> ...
> >>  
> >>> >> ## Mastering Luminances
> >>> >>
> >>> >> Now we are able to use the PQ 2084 EOTF to define the luminance of
> >>> >> pixels in absolute terms. Unfortunately we're again presented with
> >>> >> physical limitations of the display technologies on the market   
> >>> today.  
> >>> >> Here are a few examples of luminance ranges of displays.
> >>> >>
> >>> >> | Display                  | Luminance range in nits |
> >>> >> | ------------------------ | ----------------------- |
> >>> >> | Typical PC display       | 0.3 - 200 |
> >>> >> | Excellent LCD HDTV       | 0.3 - 400 |
> >>> >> | HDR LCD w/ local dimming | 0.05 - 1,500 |
> >>> >>
> >>> >> Since no display can currently show the full 0.0005 to 10,000 nits
> >>> >> luminance range the display will need to tonemap the HDR content,   
> >>> i.e  
> >>> >> to fit the content within a display's capabilities. To assist with
> >>> >> tonemapping HDR content is usually accompanied with a metadata that
> >>> >> describes (among other things) the minimum and maximum mastering
> >>> >> luminance, i.e. the maximum and minimum luminance of the display   
> >>> that  
> >>> >> was used to master the HDR content.
> >>> >>
> >>> >> The HDR metadata is currently defined on the drm_connector via the
> >>> >> hdr_output_metadata blob property.
> >>> >>
> >>> >> It might be useful to define per-plane hdr metadata, as different
> >>> >> planes might have been mastered differently.  
> >>> >
> >>> > I don't think this would directly help with the dynamic range   
> >>> blending  
> >>> > problem. You still need to establish the mapping between the optical
> >>> > values from two different EOTFs and dynamic ranges. Or can you know
> >>> > which optical values match the mastering display maximum and minimum
> >>> > luminances for not-PQ?
> >>> >  
> >>>
> >>> My understanding of this is probably best illustrated by this example:
> >>>
> >>> Assume HDR was mastered on a display with a maximum white level of 500
> >>> nits and played back on a display that supports a max white level of 
> >>> 400
> >>> nits. If you know the mastering white level of 500 you know that 
> >>> this is
> >>> the maximum value you need to compress down to 400 nits, allowing 
> >>> you to
> >>> use the full extent of the 400 nits panel.  
> >>
> >> Right, but in the kernel, where do you get these nits values from?
> >>
> >> hdr_output_metadata blob is infoframe data to the monitor. I think this
> >> should be independent of the metadata used for color transformations in
> >> the display pipeline before the monitor.
> >>
> >> EDID may tell us the monitor HDR metadata, but again what is used in
> >> the color transformations should be independent, because EDIDs lie,
> >> lighting environments change, and users have different preferences.
> >>
> >> What about black levels?
> >>
> >> Do you want to do black level adjustment?
> >>
> >> How exactly should the compression work?
> >>
> >> Where do you map the mid-tones?
> >>
> >> What if the end user wants something different?  
> >
> > I suspect that this is not about tone mapping at all. The use cases
> > listed always have the display in PQ mode and just assume that no
> > content exceeds the PQ limitations. Then you can simply bring all
> > content to the color space with a matrix multiplication and then map the
> > linear light content somewhere into the PQ range. Tone mapping is
> > performed in the display only.

The use cases do use the word "desktop" though. Harry, could you expand
on this, are you seeking a design that is good for generic desktop
compositors too, or one that is more tailored to "embedded" video
player systems taking the most advantage of (potentially
fixed-function) hardware?

What matrix would one choose? Which render intent would it
correspond to?

If you need to adapt different dynamic ranges into the blending dynamic
range, would a simple linear transformation really be enough?

> > From a generic wayland compositor point of view this is uninteresting.
> >  
> It a compositor's decision to provide or not the metadata property to 
> the kernel. The metadata can be available from one or multiple clients 
> or most likely not available at all.
> 
> Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM 
> occurs in display ) or NATIVE mode and do not attach any metadata to the 
> connector and do TM in compositor.
> 
> It is all about user preference or compositor design, or a combination 
> of both options.

Indeed. The thing here is that you cannot just add KMS UAPI, you also
need to have the FOSS userspace to go with it. So you need to have your
audience defined, userspace patches written and reviewed and agreed
to be a good idea. I'm afraid this particular UAPI design would be
difficult to justify with Weston. Maybe Kodi is a better audience?

But then again, one also needs to consider whether it is enough for a
new UAPI to satisfy only part of the possible audience and then need
yet another new UAPI to satisfy the rest. Adding new UAPI requires
defining the interactions with all existing UAPI as well.

Maybe we do need several different UAPIs for the "same" things if the
hardware designs are too different to cater with just one.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Harry Wentland]

On 2021-05-18 3:56 a.m., Pekka Paalanen wrote:
> On Mon, 17 May 2021 15:39:03 -0400
> Vitaly Prosyak <vitaly.prosyak@amd.com> wrote:
> 
>> On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
>>> On 2021-05-17 10:57, Pekka Paalanen wrote:  
>>>> On Fri, 14 May 2021 17:05:11 -0400
>>>> Harry Wentland <harry.wentland@amd.com> wrote:
>>>>  
>>>>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:  
>>>>>> On Mon, 26 Apr 2021 13:38:49 -0400
>>>>>> Harry Wentland <harry.wentland@amd.com> wrote:  
>>>>
>>>> ...
>>>>  
>>>>>>> ## Mastering Luminances
>>>>>>>
>>>>>>> Now we are able to use the PQ 2084 EOTF to define the luminance of
>>>>>>> pixels in absolute terms. Unfortunately we're again presented with
>>>>>>> physical limitations of the display technologies on the market   
>>>>> today.  
>>>>>>> Here are a few examples of luminance ranges of displays.
>>>>>>>
>>>>>>> | Display                  | Luminance range in nits |
>>>>>>> | ------------------------ | ----------------------- |
>>>>>>> | Typical PC display       | 0.3 - 200 |
>>>>>>> | Excellent LCD HDTV       | 0.3 - 400 |
>>>>>>> | HDR LCD w/ local dimming | 0.05 - 1,500 |
>>>>>>>
>>>>>>> Since no display can currently show the full 0.0005 to 10,000 nits
>>>>>>> luminance range the display will need to tonemap the HDR content,   
>>>>> i.e  
>>>>>>> to fit the content within a display's capabilities. To assist with
>>>>>>> tonemapping HDR content is usually accompanied with a metadata that
>>>>>>> describes (among other things) the minimum and maximum mastering
>>>>>>> luminance, i.e. the maximum and minimum luminance of the display   
>>>>> that  
>>>>>>> was used to master the HDR content.
>>>>>>>
>>>>>>> The HDR metadata is currently defined on the drm_connector via the
>>>>>>> hdr_output_metadata blob property.
>>>>>>>
>>>>>>> It might be useful to define per-plane hdr metadata, as different
>>>>>>> planes might have been mastered differently.  
>>>>>>
>>>>>> I don't think this would directly help with the dynamic range   
>>>>> blending  
>>>>>> problem. You still need to establish the mapping between the optical
>>>>>> values from two different EOTFs and dynamic ranges. Or can you know
>>>>>> which optical values match the mastering display maximum and minimum
>>>>>> luminances for not-PQ?
>>>>>>  
>>>>>
>>>>> My understanding of this is probably best illustrated by this example:
>>>>>
>>>>> Assume HDR was mastered on a display with a maximum white level of 500
>>>>> nits and played back on a display that supports a max white level of 
>>>>> 400
>>>>> nits. If you know the mastering white level of 500 you know that 
>>>>> this is
>>>>> the maximum value you need to compress down to 400 nits, allowing 
>>>>> you to
>>>>> use the full extent of the 400 nits panel.  
>>>>
>>>> Right, but in the kernel, where do you get these nits values from?
>>>>
>>>> hdr_output_metadata blob is infoframe data to the monitor. I think this
>>>> should be independent of the metadata used for color transformations in
>>>> the display pipeline before the monitor.
>>>>
>>>> EDID may tell us the monitor HDR metadata, but again what is used in
>>>> the color transformations should be independent, because EDIDs lie,
>>>> lighting environments change, and users have different preferences.
>>>>
>>>> What about black levels?
>>>>
>>>> Do you want to do black level adjustment?
>>>>
>>>> How exactly should the compression work?
>>>>
>>>> Where do you map the mid-tones?
>>>>
>>>> What if the end user wants something different?  
>>>
>>> I suspect that this is not about tone mapping at all. The use cases
>>> listed always have the display in PQ mode and just assume that no
>>> content exceeds the PQ limitations. Then you can simply bring all
>>> content to the color space with a matrix multiplication and then map the
>>> linear light content somewhere into the PQ range. Tone mapping is
>>> performed in the display only.
> 
> The use cases do use the word "desktop" though. Harry, could you expand
> on this, are you seeking a design that is good for generic desktop
> compositors too, or one that is more tailored to "embedded" video
> player systems taking the most advantage of (potentially
> fixed-function) hardware?
> 

The goal is to enable this on a generic desktop, such as generic Wayland
implementations or ChromeOS. We're not looking for a custom solution for
some embedded systems, though the solution we end up with should obviously
not prevent an implementation on embedded video players.

> What matrix would one choose? Which render intent would it
> correspond to?
> 
> If you need to adapt different dynamic ranges into the blending dynamic
> range, would a simple linear transformation really be enough?
> 
>>> From a generic wayland compositor point of view this is uninteresting.
>>>  
>> It a compositor's decision to provide or not the metadata property to 
>> the kernel. The metadata can be available from one or multiple clients 
>> or most likely not available at all.
>>
>> Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM 
>> occurs in display ) or NATIVE mode and do not attach any metadata to the 
>> connector and do TM in compositor.
>>
>> It is all about user preference or compositor design, or a combination 
>> of both options.
> 
> Indeed. The thing here is that you cannot just add KMS UAPI, you also
> need to have the FOSS userspace to go with it. So you need to have your
> audience defined, userspace patches written and reviewed and agreed
> to be a good idea. I'm afraid this particular UAPI design would be
> difficult to justify with Weston. Maybe Kodi is a better audience?
> 

I'm not sure designing a UAPI for Kodi that's not going to work for
Wayland-compositors is the right thing. From a KMS driver maintainer
standpoint I don't want an API for each userspace.

The idea here is to do design and discussion in public so we can eventually
arrive at a UAPI for HDR and CM that works for Wayland and by extension
for every other userspace.

> But then again, one also needs to consider whether it is enough for a
> new UAPI to satisfy only part of the possible audience and then need
> yet another new UAPI to satisfy the rest. Adding new UAPI requires
> defining the interactions with all existing UAPI as well.
> 
> Maybe we do need several different UAPIs for the "same" things if the
> hardware designs are too different to cater with just one.
> 

I feel we should have a section in the RFC that sketches how different HW
deals with this currently. It would be good if we can arrive at a UAPI that
captures at least the common functionality of various HW.

Harry

> 
> Thanks,
> pq
> 

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Sebastian Wick]

On 2021-05-18 16:19, Harry Wentland wrote:
> On 2021-05-18 3:56 a.m., Pekka Paalanen wrote:
>> On Mon, 17 May 2021 15:39:03 -0400
>> Vitaly Prosyak <vitaly.prosyak@amd.com> wrote:
>> 
>>> On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
>>>> On 2021-05-17 10:57, Pekka Paalanen wrote:
>>>>> On Fri, 14 May 2021 17:05:11 -0400
>>>>> Harry Wentland <harry.wentland@amd.com> wrote:
>>>>> 
>>>>>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
>>>>>>> On Mon, 26 Apr 2021 13:38:49 -0400
>>>>>>> Harry Wentland <harry.wentland@amd.com> wrote:
>>>>> 
>>>>> ...
>>>>> 
>>>>>>>> ## Mastering Luminances
>>>>>>>> 
>>>>>>>> Now we are able to use the PQ 2084 EOTF to define the luminance 
>>>>>>>> of
>>>>>>>> pixels in absolute terms. Unfortunately we're again presented 
>>>>>>>> with
>>>>>>>> physical limitations of the display technologies on the market
>>>>>> today.
>>>>>>>> Here are a few examples of luminance ranges of displays.
>>>>>>>> 
>>>>>>>> | Display                  | Luminance range in nits |
>>>>>>>> | ------------------------ | ----------------------- |
>>>>>>>> | Typical PC display       | 0.3 - 200 |
>>>>>>>> | Excellent LCD HDTV       | 0.3 - 400 |
>>>>>>>> | HDR LCD w/ local dimming | 0.05 - 1,500 |
>>>>>>>> 
>>>>>>>> Since no display can currently show the full 0.0005 to 10,000 
>>>>>>>> nits
>>>>>>>> luminance range the display will need to tonemap the HDR 
>>>>>>>> content,
>>>>>> i.e
>>>>>>>> to fit the content within a display's capabilities. To assist 
>>>>>>>> with
>>>>>>>> tonemapping HDR content is usually accompanied with a metadata 
>>>>>>>> that
>>>>>>>> describes (among other things) the minimum and maximum mastering
>>>>>>>> luminance, i.e. the maximum and minimum luminance of the display
>>>>>> that
>>>>>>>> was used to master the HDR content.
>>>>>>>> 
>>>>>>>> The HDR metadata is currently defined on the drm_connector via 
>>>>>>>> the
>>>>>>>> hdr_output_metadata blob property.
>>>>>>>> 
>>>>>>>> It might be useful to define per-plane hdr metadata, as 
>>>>>>>> different
>>>>>>>> planes might have been mastered differently.
>>>>>>> 
>>>>>>> I don't think this would directly help with the dynamic range
>>>>>> blending
>>>>>>> problem. You still need to establish the mapping between the 
>>>>>>> optical
>>>>>>> values from two different EOTFs and dynamic ranges. Or can you 
>>>>>>> know
>>>>>>> which optical values match the mastering display maximum and 
>>>>>>> minimum
>>>>>>> luminances for not-PQ?
>>>>>>> 
>>>>>> 
>>>>>> My understanding of this is probably best illustrated by this 
>>>>>> example:
>>>>>> 
>>>>>> Assume HDR was mastered on a display with a maximum white level of 
>>>>>> 500
>>>>>> nits and played back on a display that supports a max white level 
>>>>>> of
>>>>>> 400
>>>>>> nits. If you know the mastering white level of 500 you know that
>>>>>> this is
>>>>>> the maximum value you need to compress down to 400 nits, allowing
>>>>>> you to
>>>>>> use the full extent of the 400 nits panel.
>>>>> 
>>>>> Right, but in the kernel, where do you get these nits values from?
>>>>> 
>>>>> hdr_output_metadata blob is infoframe data to the monitor. I think 
>>>>> this
>>>>> should be independent of the metadata used for color 
>>>>> transformations in
>>>>> the display pipeline before the monitor.
>>>>> 
>>>>> EDID may tell us the monitor HDR metadata, but again what is used 
>>>>> in
>>>>> the color transformations should be independent, because EDIDs lie,
>>>>> lighting environments change, and users have different preferences.
>>>>> 
>>>>> What about black levels?
>>>>> 
>>>>> Do you want to do black level adjustment?
>>>>> 
>>>>> How exactly should the compression work?
>>>>> 
>>>>> Where do you map the mid-tones?
>>>>> 
>>>>> What if the end user wants something different?
>>>> 
>>>> I suspect that this is not about tone mapping at all. The use cases
>>>> listed always have the display in PQ mode and just assume that no
>>>> content exceeds the PQ limitations. Then you can simply bring all
>>>> content to the color space with a matrix multiplication and then map 
>>>> the
>>>> linear light content somewhere into the PQ range. Tone mapping is
>>>> performed in the display only.
>> 
>> The use cases do use the word "desktop" though. Harry, could you 
>> expand
>> on this, are you seeking a design that is good for generic desktop
>> compositors too, or one that is more tailored to "embedded" video
>> player systems taking the most advantage of (potentially
>> fixed-function) hardware?
>> 
> 
> The goal is to enable this on a generic desktop, such as generic 
> Wayland
> implementations or ChromeOS. We're not looking for a custom solution 
> for
> some embedded systems, though the solution we end up with should 
> obviously
> not prevent an implementation on embedded video players.
> 
>> What matrix would one choose? Which render intent would it
>> correspond to?
>> 
>> If you need to adapt different dynamic ranges into the blending 
>> dynamic
>> range, would a simple linear transformation really be enough?
>> 
>>>> From a generic wayland compositor point of view this is 
>>>> uninteresting.
>>>> 
>>> It a compositor's decision to provide or not the metadata property to
>>> the kernel. The metadata can be available from one or multiple 
>>> clients
>>> or most likely not available at all.
>>> 
>>> Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM
>>> occurs in display ) or NATIVE mode and do not attach any metadata to 
>>> the
>>> connector and do TM in compositor.
>>> 
>>> It is all about user preference or compositor design, or a 
>>> combination
>>> of both options.
>> 
>> Indeed. The thing here is that you cannot just add KMS UAPI, you also
>> need to have the FOSS userspace to go with it. So you need to have 
>> your
>> audience defined, userspace patches written and reviewed and agreed
>> to be a good idea. I'm afraid this particular UAPI design would be
>> difficult to justify with Weston. Maybe Kodi is a better audience?
>> 
> 
> I'm not sure designing a UAPI for Kodi that's not going to work for
> Wayland-compositors is the right thing. From a KMS driver maintainer
> standpoint I don't want an API for each userspace.
> 
> The idea here is to do design and discussion in public so we can 
> eventually
> arrive at a UAPI for HDR and CM that works for Wayland and by extension
> for every other userspace.

Eventually we want to be able to drive displays in PQ mode in weston (I
think?) where the TF property could be used in some cases. So that
property seems good to me. The SDR boost property then might also be
useful but it depends on the details of the formula/mechanism that you
can come up with.

But I want to stress again that we're going to drive all display in
their native color space and dynamic range if possible where none of
those properties discussed are useful and without some kind of 3D LUT in
the plane's pixel pipeline we won't be able to make use of them at all.

The Kodi folks can probably give you better feedback and an actual
implementation in reasonable time because they want to drive the display
in PQ mode.

>> But then again, one also needs to consider whether it is enough for a
>> new UAPI to satisfy only part of the possible audience and then need
>> yet another new UAPI to satisfy the rest. Adding new UAPI requires
>> defining the interactions with all existing UAPI as well.
>> 
>> Maybe we do need several different UAPIs for the "same" things if the
>> hardware designs are too different to cater with just one.
>> 
> 
> I feel we should have a section in the RFC that sketches how different 
> HW
> deals with this currently. It would be good if we can arrive at a UAPI 
> that
> captures at least the common functionality of various HW.
> 
> Harry
> 
>> 
>> Thanks,
>> pq
>> 

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 6230 bytes --]

On Tue, 18 May 2021 10:19:25 -0400
Harry Wentland <harry.wentland@amd.com> wrote:

> On 2021-05-18 3:56 a.m., Pekka Paalanen wrote:
> > On Mon, 17 May 2021 15:39:03 -0400
> > Vitaly Prosyak <vitaly.prosyak@amd.com> wrote:
> >   
> >> On 2021-05-17 12:48 p.m., Sebastian Wick wrote:  

...

> >>> I suspect that this is not about tone mapping at all. The use cases
> >>> listed always have the display in PQ mode and just assume that no
> >>> content exceeds the PQ limitations. Then you can simply bring all
> >>> content to the color space with a matrix multiplication and then map the
> >>> linear light content somewhere into the PQ range. Tone mapping is
> >>> performed in the display only.  
> > 
> > The use cases do use the word "desktop" though. Harry, could you expand
> > on this, are you seeking a design that is good for generic desktop
> > compositors too, or one that is more tailored to "embedded" video
> > player systems taking the most advantage of (potentially
> > fixed-function) hardware?
> >   
> 
> The goal is to enable this on a generic desktop, such as generic Wayland
> implementations or ChromeOS. We're not looking for a custom solution for
> some embedded systems, though the solution we end up with should obviously
> not prevent an implementation on embedded video players.

(There is a TL;DR: at the end.)

Echoing a little bit what Sebastian already said, I believe there are
two sides to this again:
- color management in the traditional sense
- modern standardised display technology

It was perhaps too harsh to say that generic Wayland compositors cannot
use enum-based color-related UAPI. Sometimes they could, sometimes it
won't be good enough.

Traditional color management assumes that no two monitors are the same,
even if they are the same make, model, and manufacturing batch, and are
driven exactly the same way. Hence, all monitors may require
calibration (adjusting monitor knobs), and/or they may require
profiling (measuring the light emission with a special hardware device
designed for that). Also the viewing environment has an effect.

For profiling to be at all meaningful, calibration must be fixed. This
means that there must be no dynamic on-the-fly adaptation done in the
monitor, in the display hardware, or in the kernel. That is a tall
order that I guess is going to be less and less achievable, especially
with HDR monitors.

The other side is where the end user trusts the standards, and trusts
that the drivers and the hardware do what they are specified to do.
This is where you can trust that the monitor does the tone-mapping magic
right.

Weston needs to support both approaches, because we want to prove our
new approach to traditional color management, but we also want to
support HDR, and if possible, do both at the same time. Doing both at
the same time is what we think foremost, because it's also the hardest
thing to achieve. If that can be done, then everything else works out
too.

However, this should not exclude the possibility to trust standards and
monitor magic, when the end user wants it.

It's also possible that a monitor simply doesn't support a mode that
would enable fully color managed HDR, so Weston will need to be able to
drive monitors with e.g. BT.2020/PQ data eventually. It's just not the
first goal we have.

This debate is a little bit ironic. The Wayland approach to traditional
color management is that end users should trust the display server to
do the right thing, where before people only trusted the individual
apps using a specific CMS implementation. The display server was the
untrusted one that should just get out of the way and not touch
anything. Now I'm arguing that I don't want to trust monitor magic, who
knows what atrocities it does to my picture! But take the next logical
step, and one would be arguing that end users should trust also
monitors to do the right thing. :-)

The above has two catches:

- Do you actually trust hardware manufacturers and marketers and EDID?
  Monitors have secret sauce you can't inspect nor change.

- You feed a single video stream to a monitor, in a single format,
  encoding and color space. The display server OTOH gets an arbitrary
  number of input video streams in arbitrary formats, encodings, and
  color spaces, and it needs to composite them into one.

Composition is hard. It's not enough to know what kind of signals you
take in and what kind of signal you must output. You also need to know
what the end user wants from the result: the render intent.

Even if we trust the monitor magic to do the right thing in
interpreting and displaying our output signal, we still need to know
what the end user wants from the composition, and we need to control
the composition formula to achieve that.

TL;DR:

I would summarise my comments so far into these:

- Telling the kernel the color spaces and letting it come up with
  whatever color transformation formula from those is not enough,
  because it puts the render intent policy decision in the kernel.

- Telling the kernel what color transformations need to be done is
  good, if it is clearly defined.

- Using an enum-based UAPI to tell the kernel what color
  transformations needs to be done (e.g. which EOTF or EOTF^-1 to apply
  at a step in the abstract pipeline) is very likely ok for many
  Wayland compositors in most cases, but may not be sufficient for all
  use cases. Of course, one is always bound by what hardware can do, so
  not a big deal.

- You may need to define mutually exclusive KMS properties (referring
  to my email in another branch of this email tree).

- I'm not sure I (we?) can meaningfully review things like "SDR boost"
  property until we know ourselves how to composite different types of
  content together. Maybe someone else could.

Does this help or raise thoughts?

The work on Weston CM&HDR right now is aiming to get it up to a point
where we can start nicely testing different compositing approaches and
methods and parameters, and I expect that will also feed back into the
Wayland CM&HDR protocol design as well.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH 0/3] A drm_plane API to support HDR planes

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 1977 bytes --]

On Wed, 19 May 2021 11:53:37 +0300
Pekka Paalanen <ppaalanen@gmail.com> wrote:

...

> TL;DR:
> 
> I would summarise my comments so far into these:
> 
> - Telling the kernel the color spaces and letting it come up with
>   whatever color transformation formula from those is not enough,
>   because it puts the render intent policy decision in the kernel.
> 
> - Telling the kernel what color transformations need to be done is
>   good, if it is clearly defined.
> 
> - Using an enum-based UAPI to tell the kernel what color
>   transformations needs to be done (e.g. which EOTF or EOTF^-1 to apply
>   at a step in the abstract pipeline) is very likely ok for many
>   Wayland compositors in most cases, but may not be sufficient for all
>   use cases. Of course, one is always bound by what hardware can do, so
>   not a big deal.
> 
> - You may need to define mutually exclusive KMS properties (referring
>   to my email in another branch of this email tree).
> 
> - I'm not sure I (we?) can meaningfully review things like "SDR boost"
>   property until we know ourselves how to composite different types of
>   content together. Maybe someone else could.
> 
> Does this help or raise thoughts?
> 
> The work on Weston CM&HDR right now is aiming to get it up to a point
> where we can start nicely testing different compositing approaches and
> methods and parameters, and I expect that will also feed back into the
> Wayland CM&HDR protocol design as well.

I have forgot to mention one important thing:

Generic Wayland compositors will be using KMS planes opportunistically.
The compositor will be switching between GL and KMS compositing
on-demand, refresh by refresh. This means that both GL and KMS
compositing must produce identical results, or users will be seeing
"color flicks" on switch.

This is a practical reason why we really want to know in full detail
how the KMS pipeline processes pixels.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]