From: Harry Wentland <harry.wentland@amd.com>
To: Simon Ser <contact@emersion.fr>,
DRI Development <dri-devel@lists.freedesktop.org>
Cc: "Pekka Paalanen" <pekka.paalanen@collabora.com>,
"Jonas Ådahl" <jadahl@redhat.com>,
"xaver.hugl@gmail.com" <xaver.hugl@gmail.com>,
"Melissa Wen" <mwen@igalia.com>,
wayland-devel <wayland-devel@lists.freedesktop.org>,
"Michel Dänzer" <mdaenzer@redhat.com>,
"Uma Shankar" <uma.shankar@intel.com>,
"Victoria Brekenfeld" <victoria@system76.com>,
"Aleix Pol" <aleixpol@kde.org>,
"Sebastian Wick" <sebastian.wick@redhat.com>,
"Joshua Ashton" <joshua@froggi.es>
Subject: Re: [RFC] Plane color pipeline KMS uAPI
Date: Thu, 4 May 2023 17:10:46 -0400 [thread overview]
Message-ID: <67191ae5-460c-53d0-24c2-06fd6d20fcb1@amd.com> (raw)
In-Reply-To: <QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr>
On 5/4/23 11:22, Simon Ser wrote:
> Hi all,
>
> The goal of this RFC is to expose a generic KMS uAPI to configure the color
> pipeline before blending, ie. after a pixel is tapped from a plane's
> framebuffer and before it's blended with other planes. With this new uAPI we
> aim to reduce the battery life impact of color management and HDR on mobile
> devices, to improve performance and to decrease latency by skipping
> composition on the 3D engine. This proposal is the result of discussions at
> the Red Hat HDR hackfest [1] which took place a few days ago. Engineers
> familiar with the AMD, Intel and NVIDIA hardware have participated in the
> discussion.
>
Thanks for typing this up. It does a great job describing the vision.
> This proposal takes a prescriptive approach instead of a descriptive approach.
> Drivers describe the available hardware blocks in terms of low-level
> mathematical operations, then user-space configures each block. We decided
> against a descriptive approach where user-space would provide a high-level
> description of the colorspace and other parameters: we want to give more
> control and flexibility to user-space, e.g. to be able to replicate exactly the
> color pipeline with shaders and switch between shaders and KMS pipelines
> seamlessly, and to avoid forcing user-space into a particular color management
> policy.
>
> We've decided against mirroring the existing CRTC properties
> DEGAMMA_LUT/CTM/GAMMA_LUT onto KMS planes. Indeed, the color management
> pipeline can significantly differ between vendors and this approach cannot
> accurately abstract all hardware. In particular, the availability, ordering and
> capabilities of hardware blocks is different on each display engine. So, we've
> decided to go for a highly detailed hardware capability discovery.
>
> This new uAPI should not be in conflict with existing standard KMS properties,
> since there are none which control the pre-blending color pipeline at the
> moment. It does conflict with any vendor-specific properties like
> NV_INPUT_COLORSPACE or the patches on the mailing list adding AMD-specific
> properties. Drivers will need to either reject atomic commits configuring both
> uAPIs, or alternatively we could add a DRM client cap which hides the vendor
> properties and shows the new generic properties when enabled.
>
> To use this uAPI, first user-space needs to discover hardware capabilities via
> KMS objects and properties, then user-space can configure the hardware via an
> atomic commit. This works similarly to the existing KMS uAPI, e.g. planes.
>
> Our proposal introduces a new "color_pipeline" plane property, and a new KMS
> object type, "COLOROP" (short for color operation). The "color_pipeline" plane
> property is an enum, each enum entry represents a color pipeline supported by
> the hardware. The special zero entry indicates that the pipeline is in
> "bypass"/"no-op" mode. For instance, the following plane properties describe a
> primary plane with 2 supported pipelines but currently configured in bypass
> mode:
>
> Plane 10
> ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> ├─ …
> └─ "color_pipeline": enum {0, 42, 52} = 0
>
> The non-zero entries describe color pipelines as a linked list of COLOROP KMS
> objects. The entry value is an object ID pointing to the head of the linked
> list (the first operation in the color pipeline).
>
> The new COLOROP objects also expose a number of KMS properties. Each has a
> type, a reference to the next COLOROP object in the linked list, and other
> type-specific properties. Here is an example for a 1D LUT operation:
>
> Color operation 42
> ├─ "type": enum {Bypass, 1D curve} = 1D curve
> ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
> ├─ "lut_size": immutable range = 4096
> ├─ "lut_data": blob
> └─ "next": immutable color operation ID = 43
>
> To configure this hardware block, user-space can fill a KMS blob with 4096 u32
> entries, then set "lut_data" to the blob ID. Other color operation types might
> have different properties.
>
> Here is another example with a 3D LUT:
>
> Color operation 42
> ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> ├─ "lut_size": immutable range = 33
> ├─ "lut_data": blob
> └─ "next": immutable color operation ID = 43
>
> And one last example with a matrix:
>
> Color operation 42
> ├─ "type": enum {Bypass, Matrix} = Matrix
> ├─ "matrix_data": blob
> └─ "next": immutable color operation ID = 43
>
> [Simon note: having "Bypass" in the "type" enum, and making "type" mutable is
> a bit weird. Maybe we can just add an "active"/"bypass" boolean property on
> blocks which can be bypassed instead.]
I would favor a "bypass" boolean property.
>
> [Jonas note: perhaps a single "data" property for both LUTs and matrices
> would make more sense. And a "size" prop for both 1D and 3D LUTs.]
>
I concur. We'll probably want to document for which types a property
applies.
> If some hardware supports re-ordering operations in the color pipeline, the
> driver can expose multiple pipelines with different operation ordering, and
> user-space can pick the ordering it prefers by selecting the right pipeline.
> The same scheme can be used to expose hardware blocks supporting multiple
> precision levels.
>
> That's pretty much all there is to it, but as always the devil is in the
> details.
>
One such detail that might need some thought is whether the specific
pipeline configuration exposed by a driver becomes uAPI. In theory I
might be breaking use-cases userspace has if I change my color pipeline,
but it would still discoverable and usable if userspace uses the uAPI in
a truly vendor-neutral way.
Thoughts?
> First, we realized that we need a way to indicate where the scaling operation
> is happening. The contents of the framebuffer attached to the plane might be
> scaled up or down depending on the CRTC_W and CRTC_H properties. Depending on
> the colorspace scaling is applied in, the result will be different, so we need
> a way for the kernel to indicate which hardware blocks are pre-scaling, and
> which ones are post-scaling. We introduce a special "scaling" operation type,
> which is part of the pipeline like other operations but serves an informational
> role only (effectively, the operation cannot be configured by user-space, all
> of its properties are immutable). For example:
>
> Color operation 43
> ├─ "type": immutable enum {Scaling} = Scaling
> └─ "next": immutable color operation ID = 44
>
> [Simon note: an alternative would be to split the color pipeline into two, by
> having two plane properties ("color_pipeline_pre_scale" and
> "color_pipeline_post_scale") instead of a single one. This would be similar to
> the way we want to split pre-blending and post-blending. This could be less
> expressive for drivers, there may be hardware where there are dependencies
> between the pre- and post-scaling pipeline?]
>
I would prefer to avoid splitting the pipeline again. We can't easily
avoid the pre-/post-blending split but for scaling it might be more
straight-forward to add a read-only scaling op. This isn't a strong
preference since I could see either way working out well.
> Then, Alex from NVIDIA described how their hardware works. NVIDIA hardware
> contains some fixed-function blocks which convert from LMS to ICtCp and cannot
> be disabled/bypassed. NVIDIA hardware has been designed for descriptive APIs
> where user-space provides a high-level description of the colorspace
> conversions it needs to perform, and this is at odds with our KMS uAPI
> proposal. To address this issue, we suggest adding a special block type which
> describes a fixed conversion from one colorspace to another and cannot be
> configured by user-space. Then user-space will need to accomodate its pipeline
> for these special blocks. Such fixed hardware blocks need to be well enough
> documented so that they can be implemented via shaders.
>
> We also noted that it should always be possible for user-space to completely
> disable the color pipeline and switch back to bypass/identity without a
> modeset. Some drivers will need to fail atomic commits for some color
> pipelines, in particular for some specific LUT payloads. For instance, AMD
> doesn't support curves which are too steep, and Intel doesn't support curves
> which decrease. This isn't something which routinely happens, but there might
> be more cases where the hardware needs to reject the pipeline. Thus, when
> user-space has a running KMS color pipeline, then hits a case where the
> pipeline cannot keep running (gets rejected by the driver), user-space needs to
> be able to immediately fall back to shaders without any glitch. This doesn't
> seem to be an issue for AMD, Intel and NVIDIA.
>
> This uAPI is extensible: we can add more color operations, and we can add more
> properties for each color operation type. For instance, we might want to add
> support for Intel piece-wise linear (PWL) 1D curves, or might want to advertise
> the effective precision of the LUTs. The uAPI is deliberately somewhat minimal
> to keep the scope of the proposal manageable.
>
> Later on, we plan to re-use the same machinery for post-blending color
> pipelines. There are some more details about post-blending which have been
> separately debated at the hackfest, but we believe it's a viable plan. This
> solution would supersede the existing DEGAMMA_LUT/CTM/GAMMA_LUT properties, so
> we'd like to introduce a client cap to hide the old properties and show the new
> post-blending color pipeline properties.
>
> We envision a future user-space library to translate a high-level descriptive
> color pipeline into low-level prescriptive KMS color pipeline ("libliftoff but
> for color pipelines"). The library could also offer a translation into shaders.
> This should help share more infrastructure between compositors and ease KMS
> offloading. This should also help dealing with the NVIDIA case.
>
> To wrap things up, let's take a real-world example: how would gamescope [2]
> configure the AMD DCN 3.0 hardware for its color pipeline? The gamescope color
> pipeline is described in [3]. The AMD DCN 3.0 hardware is described in [4].
>
> AMD would expose the following objects and properties:
>
> Plane 10
> ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> └─ "color_pipeline": enum {0, 42} = 0
> Color operation 42 (input CSC)
> ├─ "type": enum {Bypass, Matrix} = Matrix
> ├─ "matrix_data": blob
> └─ "next": immutable color operation ID = 43
> Color operation 43
> ├─ "type": enum {Scaling} = Scaling
> └─ "next": immutable color operation ID = 44
> Color operation 44 (DeGamma)
> ├─ "type": enum {Bypass, 1D curve} = 1D curve
> ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> └─ "next": immutable color operation ID = 45
> Color operation 45 (gamut remap)
> ├─ "type": enum {Bypass, Matrix} = Matrix
> ├─ "matrix_data": blob
> └─ "next": immutable color operation ID = 46
> Color operation 46 (shaper LUT RAM)
> ├─ "type": enum {Bypass, 1D curve} = 1D curve
> ├─ "1d_curve_type": enum {LUT} = LUT
> ├─ "lut_size": immutable range = 4096
> ├─ "lut_data": blob
> └─ "next": immutable color operation ID = 47
> Color operation 47 (3D LUT RAM)
> ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> ├─ "lut_size": immutable range = 17
> ├─ "lut_data": blob
> └─ "next": immutable color operation ID = 48
> Color operation 48 (blend gamma)
> ├─ "type": enum {Bypass, 1D curve} = 1D curve
> ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
> ├─ "lut_size": immutable range = 4096
> ├─ "lut_data": blob
> └─ "next": immutable color operation ID = 0
>
> To configure the pipeline for an HDR10 PQ plane (path at the top) and a HDR
> display, gamescope would perform an atomic commit with the following property
> values:
>
> Plane 10
> └─ "color_pipeline" = 42
> Color operation 42 (input CSC)
> └─ "matrix_data" = PQ → scRGB (TF)
> Color operation 44 (DeGamma)
> └─ "type" = Bypass
> Color operation 45 (gamut remap)
> └─ "matrix_data" = scRGB (TF) → PQ
> Color operation 46 (shaper LUT RAM)
> └─ "lut_data" = PQ → Display native
> Color operation 47 (3D LUT RAM)
> └─ "lut_data" = Gamut mapping + tone mapping + night mode
> Color operation 48 (blend gamma)
> └─ "1d_curve_type" = PQ
>
> I hope comparing these properties to the diagrams linked above can help
> understand how the uAPI would be used and give an idea of its viability.
>
> Please feel free to provide feedback! It would be especially useful to have
> someone familiar with Arm SoCs look at this, to confirm that this proposal
> would work there.
>
This is the major gap we have with this proposal, so I hope someone
working on the Arm SoC drivers sees this and can comment.
Again, thanks for typing this up, Simon.
Harry
> Unless there is a show-stopper, we plan to follow up this RFC with
> implementations for AMD, Intel, NVIDIA, gamescope, and IGT.
>
> Many thanks to everybody who contributed to the hackfest, on-site or remotely!
> Let's work together to make this happen!
>
> Simon, on behalf of the hackfest participants
>
> [1]: https://wiki.gnome.org/Hackfests/ShellDisplayNext2023
> [2]: https://github.com/ValveSoftware/gamescope
> [3]: https://github.com/ValveSoftware/gamescope/blob/5af321724c8b8a29cef5ae9e31293fd5d560c4ec/src/docs/Steam%20Deck%20Display%20Pipeline.png
> [4]: https://kernel.org/doc/html/latest/_images/dcn3_cm_drm_current.svg
next prev parent reply other threads:[~2023-05-04 21:11 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-04 15:22 [RFC] Plane color pipeline KMS uAPI Simon Ser
2023-05-04 21:10 ` Harry Wentland [this message]
2023-05-05 11:41 ` Pekka Paalanen
2023-05-05 13:30 ` Joshua Ashton
2023-05-05 14:16 ` Pekka Paalanen
2023-05-05 17:01 ` Joshua Ashton
2023-05-09 11:23 ` Melissa Wen
2023-05-09 11:47 ` Pekka Paalanen
2023-05-09 17:01 ` Melissa Wen
2023-05-11 21:21 ` Simon Ser
2023-05-05 15:28 ` Daniel Vetter
2023-05-05 15:57 ` Sebastian Wick
2023-05-05 19:51 ` Daniel Vetter
2023-05-08 8:24 ` Pekka Paalanen
2023-05-08 9:00 ` Daniel Vetter
2023-05-05 16:06 ` Simon Ser
2023-05-05 19:53 ` Daniel Vetter
2023-05-08 8:58 ` Simon Ser
2023-05-08 9:18 ` Daniel Vetter
2023-05-08 18:10 ` Harry Wentland
[not found] ` <20230508185409.07501f40@n2pa>
2023-05-09 8:17 ` Pekka Paalanen
2023-05-05 20:40 ` Dave Airlie
2023-05-05 22:20 ` Sebastian Wick
2023-05-07 23:14 ` Dave Airlie
2023-05-08 9:37 ` Pekka Paalanen
2023-05-08 10:03 ` Jonas Ådahl
2023-05-09 14:31 ` Harry Wentland
2023-05-09 19:53 ` Dave Airlie
2023-05-09 20:22 ` Simon Ser
2023-05-10 7:59 ` Jonas Ådahl
2023-05-10 8:59 ` Pekka Paalanen
2023-05-11 9:51 ` Karol Herbst
2023-05-11 16:56 ` Joshua Ashton
2023-05-11 18:56 ` Jonas Ådahl
2023-05-11 19:29 ` Simon Ser
2023-05-12 7:24 ` Pekka Paalanen
2023-05-10 8:48 ` Pekka Paalanen
[not found] ` <20230505160435.6e3ffa4a@n2pa>
2023-05-08 8:49 ` Pekka Paalanen
2023-05-09 8:04 ` Pekka Paalanen
[not found] ` <4341dac6-ada1-2a75-1c22-086d96408a85@quicinc.com>
2023-06-09 15:52 ` Christopher Braga
2023-06-09 16:30 ` Simon Ser
2023-06-09 23:11 ` Christopher Braga
2023-06-12 9:21 ` Pekka Paalanen
2023-06-12 16:56 ` Christopher Braga
2023-06-13 8:23 ` Pekka Paalanen
2023-06-13 16:29 ` Christopher Braga
2023-06-14 9:00 ` Pekka Paalanen
2023-06-15 21:44 ` Christopher Braga
2023-06-16 7:59 ` Pekka Paalanen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=67191ae5-460c-53d0-24c2-06fd6d20fcb1@amd.com \
--to=harry.wentland@amd.com \
--cc=aleixpol@kde.org \
--cc=contact@emersion.fr \
--cc=dri-devel@lists.freedesktop.org \
--cc=jadahl@redhat.com \
--cc=joshua@froggi.es \
--cc=mdaenzer@redhat.com \
--cc=mwen@igalia.com \
--cc=pekka.paalanen@collabora.com \
--cc=sebastian.wick@redhat.com \
--cc=uma.shankar@intel.com \
--cc=victoria@system76.com \
--cc=wayland-devel@lists.freedesktop.org \
--cc=xaver.hugl@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).