[ANNOUNCE] Wayland/Weston/Mesa HDR support (proof of concept)

[ANNOUNCE] Wayland/Weston/Mesa HDR support (proof of concept)

[Ville Syrjälä]

Here's a quick proof of concept implementation of HDR support
for Wayland/Weston/Mesa.

I'm not posting this as patches right now because I'm not sure
that would do much good given how rough this is. But here are
all the repos/branches:
git://github.com/vsyrjala/wayland.git hdr_poc
git://github.com/vsyrjala/wayland-protocols.git hdr_poc
git://github.com/vsyrjala/weston.git hdr_poc
git://github.com/vsyrjala/mesa.git hdr_poc
git://github.com/vsyrjala/linux.git hdr_poc

The kernel HDR bits were partially done by Uma Shankar, the rest
I hacked together myself.

As far as Wayland protocol goes I'm adding three new
extensions (should probably just have one with several requests?):
- zwp_colorspace_v1 - Specify the primaries/whitepoint chromacities 
  and transfer function for a surface
- zwp_ycbcr_encoding_v1 - Specify the encoding for YCbCr surfaces
- zwp_hdr_metadata_v1 - Allow the client to pass HDR metadata to 
  the compositor
Note that I've not given any thought to how the compositor might
advertize its capabilities.

I also hacked in a bunch of 10bpc+ YCbCr support to the protocol and
Weston so that I can actually get some HDR video onto the screen.

On the Mesa side I've crudely implementated the following egl/vk
extesions:
- EXT_gl_colorspace_*
- EXT_surface_SMPTE2086_metadata
- EXT_surface_CTA861_3_metadata
  (sidenote: these egl extension don't seem to match CTA-861.3 nicely
   when it comes to the min/max luminance stuff)
- VK_EXT_hdr_metadata

VK_EXT_hdr_metadata I plugged in for anv only, but the implementation
is in the common wayland wsi code. Note that I haven't actually tested
the vulkan stuff at all because I don't talk Vulkan (at least not yet).

Also note that I've not connected up the HDR metadata pipeline 
properly. The client can provide the metadata, but the compositor
doesn't actually pass it on to the display. For the time being the
HDR metadata that gets passed to the display is partially specified
in weston.ini and partially just hardocded (see
libweston/compositor-drm.c).

The Weston implementation involves a bunch of shaders and matrices to
do the ycbcr->rgb, "degamma", csc for each surface, blend it all as
linear RGB into an fp16 fbo, and finally blit that out to the final
framebuffer while applying the ST2084 PQ transfer function in the
process.

The reason for the fp16 fbo is that we store the full 10000 nits of
linear RGB. That needs plenty of precisions in the low end so your
regular 10bpc fb doesn't seem to cut it. And also the display gamma LUT
doesn't have enough input precision for it either. Sadly there's no
fixed function hardware in the GPU to do the ST2084 PQ when blending.
When the output is not HDR I do skip the fp16 fbo step and use the
gamma LUT in the display engine instead. 

Another approach to the precisions problem might be to not store the
entire 10000 nits of linear, and just cut off the super bright stuff
(your display can't show it anyway). But I've not really bothered to
figure out how low in nits we'd have to go here, probably too low.
Maybe blending as sRGB and the doing sRGB->PQ with the gamma LUT might
help a little bit?

Ideally we would bypass this all for a single fullscreen HDR surface
and just pass the PQ encoded data directly through. But I've not
implemented that. In fact I even disable the buffer_age damage stuff
when using the fp16 fbo, so we'll recompose the entire screen every
time. Yeah, I'm lazy.

Another thought that occurred to me was that it shouldn't be too hard
to make Weston do some tone mapping when there's a HDR client and no
HDR screen. To that end I included the ACES colorspaces in my
colorspace list, but I didn't actually look into plugging the ACES tone
mapping curve into the shaders. Might be a fun excercise, even though
the practical applications might be close to nil. Probably better to
not advertize HDR/wide gamuts when we can't actually output the stuff,
and instead let the client do its own tone mapping.

OK, so what can you do with this? I've included a few test clients:
- simple-colorspace
  Just a copy of simple-egl but it uses the egl extension to specify
  the colorspace, and produces ST2084 PQ encoded data when asked
- simple-hdr-video
  Uses ffmpeg to decode video into shm buffers, and sets the 
  colorspace/ycbcr encoding etc. appropriately. Ie. this one can
  actually output HDR video

Here's a weston.ini snippet that gets you outputting HDR:
[core]
gbm-format=xrgb2101010

[output]
name=HDMI-A-2
colorspace=BT.2020
gamma=ST2084
max_sdr_nits=100

Hardware wise you'll need a HDR capable display obviously, and 
you'll need an Intel Geminilake GPU. Older Intel platforms don't
support the HDR infoframe, so the display wouldn't know what to do
with the data you're feeding it.

As for the future, right now I don't really have any solid plans on
continuing to develop this. I might dabble with it a bit more out of
curiosity, but I'm more hoping we can find other people to move this
forward properly.

-- 
Ville Syrjälä
Intel OTC
_______________________________________________
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: [Mesa-dev] [ANNOUNCE] Wayland/Weston/Mesa HDR support (proof of concept)

[Kristian Høgsberg]

On Thu, Dec 21, 2017 at 6:21 AM, Ville Syrjälä
<ville.syrjala@linux.intel.com> wrote:
> Here's a quick proof of concept implementation of HDR support
> for Wayland/Weston/Mesa.
>
> I'm not posting this as patches right now because I'm not sure
> that would do much good given how rough this is. But here are
> all the repos/branches:
> git://github.com/vsyrjala/wayland.git hdr_poc
> git://github.com/vsyrjala/wayland-protocols.git hdr_poc
> git://github.com/vsyrjala/weston.git hdr_poc
> git://github.com/vsyrjala/mesa.git hdr_poc
> git://github.com/vsyrjala/linux.git hdr_poc
>
> The kernel HDR bits were partially done by Uma Shankar, the rest
> I hacked together myself.

Hi Ville,

This looks really interesting, thanks for the heads up.

Kristian

> As far as Wayland protocol goes I'm adding three new
> extensions (should probably just have one with several requests?):
> - zwp_colorspace_v1 - Specify the primaries/whitepoint chromacities
>   and transfer function for a surface
> - zwp_ycbcr_encoding_v1 - Specify the encoding for YCbCr surfaces
> - zwp_hdr_metadata_v1 - Allow the client to pass HDR metadata to
>   the compositor
> Note that I've not given any thought to how the compositor might
> advertize its capabilities.
>
> I also hacked in a bunch of 10bpc+ YCbCr support to the protocol and
> Weston so that I can actually get some HDR video onto the screen.
>
> On the Mesa side I've crudely implementated the following egl/vk
> extesions:
> - EXT_gl_colorspace_*
> - EXT_surface_SMPTE2086_metadata
> - EXT_surface_CTA861_3_metadata
>   (sidenote: these egl extension don't seem to match CTA-861.3 nicely
>    when it comes to the min/max luminance stuff)
> - VK_EXT_hdr_metadata
>
> VK_EXT_hdr_metadata I plugged in for anv only, but the implementation
> is in the common wayland wsi code. Note that I haven't actually tested
> the vulkan stuff at all because I don't talk Vulkan (at least not yet).
>
> Also note that I've not connected up the HDR metadata pipeline
> properly. The client can provide the metadata, but the compositor
> doesn't actually pass it on to the display. For the time being the
> HDR metadata that gets passed to the display is partially specified
> in weston.ini and partially just hardocded (see
> libweston/compositor-drm.c).
>
> The Weston implementation involves a bunch of shaders and matrices to
> do the ycbcr->rgb, "degamma", csc for each surface, blend it all as
> linear RGB into an fp16 fbo, and finally blit that out to the final
> framebuffer while applying the ST2084 PQ transfer function in the
> process.
>
> The reason for the fp16 fbo is that we store the full 10000 nits of
> linear RGB. That needs plenty of precisions in the low end so your
> regular 10bpc fb doesn't seem to cut it. And also the display gamma LUT
> doesn't have enough input precision for it either. Sadly there's no
> fixed function hardware in the GPU to do the ST2084 PQ when blending.
> When the output is not HDR I do skip the fp16 fbo step and use the
> gamma LUT in the display engine instead.
>
> Another approach to the precisions problem might be to not store the
> entire 10000 nits of linear, and just cut off the super bright stuff
> (your display can't show it anyway). But I've not really bothered to
> figure out how low in nits we'd have to go here, probably too low.
> Maybe blending as sRGB and the doing sRGB->PQ with the gamma LUT might
> help a little bit?
>
> Ideally we would bypass this all for a single fullscreen HDR surface
> and just pass the PQ encoded data directly through. But I've not
> implemented that. In fact I even disable the buffer_age damage stuff
> when using the fp16 fbo, so we'll recompose the entire screen every
> time. Yeah, I'm lazy.
>
> Another thought that occurred to me was that it shouldn't be too hard
> to make Weston do some tone mapping when there's a HDR client and no
> HDR screen. To that end I included the ACES colorspaces in my
> colorspace list, but I didn't actually look into plugging the ACES tone
> mapping curve into the shaders. Might be a fun excercise, even though
> the practical applications might be close to nil. Probably better to
> not advertize HDR/wide gamuts when we can't actually output the stuff,
> and instead let the client do its own tone mapping.
>
> OK, so what can you do with this? I've included a few test clients:
> - simple-colorspace
>   Just a copy of simple-egl but it uses the egl extension to specify
>   the colorspace, and produces ST2084 PQ encoded data when asked
> - simple-hdr-video
>   Uses ffmpeg to decode video into shm buffers, and sets the
>   colorspace/ycbcr encoding etc. appropriately. Ie. this one can
>   actually output HDR video
>
> Here's a weston.ini snippet that gets you outputting HDR:
> [core]
> gbm-format=xrgb2101010
>
> [output]
> name=HDMI-A-2
> colorspace=BT.2020
> gamma=ST2084
> max_sdr_nits=100
>
> Hardware wise you'll need a HDR capable display obviously, and
> you'll need an Intel Geminilake GPU. Older Intel platforms don't
> support the HDR infoframe, so the display wouldn't know what to do
> with the data you're feeding it.
>
> As for the future, right now I don't really have any solid plans on
> continuing to develop this. I might dabble with it a bit more out of
> curiosity, but I'm more hoping we can find other people to move this
> forward properly.
>
> --
> Ville Syrjälä
> Intel OTC
> _______________________________________________
> mesa-dev mailing list
> mesa-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel