[TECH TOPIC] New userspace API for display-panel brightness control

[TECH TOPIC] New userspace API for display-panel brightness control

[Hans de Goede]

<resend to both lists, because of confusion of which list to use>

Hi All,

As requested here is a copy of my LPC kernel summit track submission:

Title: New userspace API for display-panel brightness control

The current userspace API for brightness control offered by
/sys/class/backlight devices has various problems:

1. There is no way to map the backlight device to a specific
display-output / panel

2. On x86 there can be multiple firmware + direct-hw-access
methods for controlling the backlight and the kernel may
register multiple backlight-devices based on this which are
all controlling the brightness for the same display-panel.
To make things worse sometimes only one of the registered
backlight devices actually works.

3. Controlling the brightness requires root-rights requiring
desktop-environments to use suid-root helpers for this.

4. The scale of the brightness value is unclear, the API does
not define if "perceived brightness" or electrical power is
being controlled and in practice both are used without userspace
knowing which is which.

5. The API does not define if a brightness value of 0 means off,
or lowest brightness at which the screen is still readable
(in a low lit room), again in practice both variants happen.

This talk will present a proposal for a new userspace API
which intends to address these problems in the form of a
number of new properties for drm/kms properties on the
drm_connector object for the display-panel.

This talk will also focus on how to implement this proposal
which brings several challenges with it:

1. The mess of having multiple interfaces to control a laptop's
internal-panel will have to be sorted out because with the new
API we can no longer just register multiple backlight devices
and let userspace sort things out.

2. In various cases the drm/kms driver driving the panel
does not control the brightness itself, but the brightness
is controlled through some (ACPI) firmware interface such
as the acpi_video or nvidia-wmi-ec-backlight interfaces.

This introduces some challenging probe-ordering issues,
the solution for which is not entirely clear yet, so this
part of the talk will be actively seeking audience input
on this topic.


Comments:
This is a duplicate submission with one which I submitted for
the "LPC Refereed Track" before the "Kernel Summit 2022 CFP" posting.

I recently send a RFC email about this to the relevant mailinglists:
https://lore.kernel.org/dri-devel/0d188965-d809-81b5-74ce-7d30c49fee2d@redhat.com/

As well as another RFC laying out some initial backlight code
refactoring steps. As there is a bunch of technical debt which
needs to be addressed before work on a new uAPI can even begin:
https://lore.kernel.org/dri-devel/98519ba0-7f18-201a-ea34-652f50343158@redhat.com/

I'm working on the refactoring now. I believe the refactoring
is more likely to land in 5.21 then in 5.20. Let alone that
the new uAPI on the kernel side + the mandatory userspace code
consuming the uAPI will be ready before plumbers.

IOW I expect this to still be very much in flux during Plumbers,
so this won't be a presentation presenting only already finished
work.

Regards,

Hans

Re: [TECH TOPIC] New userspace API for display-panel brightness control

[Hans de Goede]

Hi All,

On 6/16/22 22:04, Linus Walleij wrote:
> On Thu, Jun 16, 2022 at 11:33 AM Hans de Goede <hdegoede@redhat.com> wrote:
> 
>> The current userspace API for brightness control offered by
>> /sys/class/backlight devices has various problems:
>>
>> 1. There is no way to map the backlight device to a specific
>> display-output / panel
> 
> For userspace, using just sysfs you mean?

Yes,

> But that makes it sound like userspace needs to understand
> things like backlight-to-panel topology etc.

The linked RFC/proposal for the new API makes brightness a property
on the drm connector object, making the backlight-to-panel topology
mapping a kernel problem.

One of the feature-requests behind this API is the ability to
control the beightness of external monitors over DDC/DI this
will mean (for laptops) multiple brightness controls (one
per supported display) at which point userspace indeed needs
to be aware of the brightness-control <-> panel mapping.

Making this a property on the drm connector object makes this
mapping very explicit.

> If you add the presence of ambient light sensors to this mix
> things get even messier.

Mapping ambient light-sensors to dipslays is admittedly still
an unsolved problem, even with my proposal. But I'm not aware
of any external monitors with the capability to report ambient
light-sensors back to the connected PC (or other display source).

So for ambient light sensors for now userspace can keep assuming
these belong to the device's internal panel, like it currently
is also doing for the /sys/class/backlight devices.

For external monitors with ambient light sensors, if
anything I would expect the entire auto-brightness "stack"
to be inside the external monitor, with the possibility
to turn it on/off and set some parameters (like desired
perceived brightness) through DDC/DI . In which case
the auto setting + parameters can be made parameters on
the drm-connector object just like brightness.

And I guess we could make the ambient-light-level a drm-connector
property too. This will require some kernel glue and is a bit
outside of the scope of the current proposal. But it would
make sense to do things this way and I think this would
be a good/clean userspace API for the ambient light level
reporting which automatically solves the display mapping
problem.

> I would rather make the analogy to the thermal subsystem:
> 
> - Handles multiple thermal sensors
> - Handle linearization of measurements
> - Some accumulate and work to monitor a thermal zone
> - Handle multiple thermal zones
> - Also has cooling devices (such as CPU frequency and fans)
> - Policies are applied in the kernel to handle thermal sensors
>   and cooling devices and control them in an orchestrated
>   manner
> - Userspace can sit back and enjoy the show, but it works
>   out-of-the box. No magic thermal daemon.
> 
> Examples:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/thermal/thermal-zones.yaml
> 
> Wouldn't backlight rather:
> 
> - Handle multiple backlight devices.
> - Handle linearization of backlight intensity
> - Some accumulate and work as a composite backlight
> - Handle multiple composite backlights such as screens
> - Also handle ambient light sensors
> - Policies are applied in the kernel to handle backlight
>   and ambient light sensors together
> - Userspace can sit back and enjoy the show but it works
>   out-of-the box, no magic backlight daemon
> 
> I understand userspace will want to force backlight to user
> preferences, older people need more backlight etc.
> 
> But isn't it more compelling to handle that as a composite
> backlight device than to expose several of them to
> userspace? I imagine one big knob per screen
> 1-100 for userspace, a bool for on/off and a bool to select
> augmentation from ambient light sensors or not, the rest
> the kernel can figure out.
> 
> My point is that this is not just a userspace API, it is
> a policy extension of the backlight subsystem.
> 
> Maybe this is in line with what you're suggesting.
> I guess I just needed to mention ambient light sensors
> here.
> 
> My personal annoyance is to see several diverging
> userspace implementations of policy for using ambient
> light sensors with backlight. It is already annoying,
> Android has something etc.
> 
> I understand that this drives a truck through the old mantra
> to keep policy in userspace, but so does thermal already,
> so I'd just ask myself what makes most sense.

I'm actually a firm believer in keeping policy in
userspace, especially for something like panel brightness
control.

There not only is an ambient-light sensor to take into
account (with multiple possible algorithms to deal with
+ various knobs to tweak the algorithm) but also e.g. dimming
the brightness when the machine is idle (no user input for X
amount of time), when the battery goes below a certain treshold
(battery saver mode) and when enabling the new-fangled builtin
electronic privacy-screens.

And more use-cases which impact the brightness control might
popup in the future but just the above list is IMHO complicated
enough to leave this to userspace.

Thermal is different because letting things overheat is really
really bad, so the kernel really must get involved here. But
in general I do believe keeping policy in userspace is best
(when possible).

E.g. we also don't have any sound volume-control policy
in the kernel and IMHO brightness control is more like
sound volume-control then like thermalzones.

Regards,

Hans

Re: [TECH TOPIC] New userspace API for display-panel brightness control

[Jani Nikula]

On Mon, 20 Jun 2022, Hans de Goede <hdegoede@redhat.com> wrote:
> <resend to both lists, because of confusion of which list to use>
>
> Hi All,
>
> As requested here is a copy of my LPC kernel summit track submission:
>
> Title: New userspace API for display-panel brightness control
>
> The current userspace API for brightness control offered by
> /sys/class/backlight devices has various problems:
>
> 1. There is no way to map the backlight device to a specific
> display-output / panel
>
> 2. On x86 there can be multiple firmware + direct-hw-access
> methods for controlling the backlight and the kernel may
> register multiple backlight-devices based on this which are
> all controlling the brightness for the same display-panel.
> To make things worse sometimes only one of the registered
> backlight devices actually works.
>
> 3. Controlling the brightness requires root-rights requiring
> desktop-environments to use suid-root helpers for this.
>
> 4. The scale of the brightness value is unclear, the API does
> not define if "perceived brightness" or electrical power is
> being controlled and in practice both are used without userspace
> knowing which is which.
>
> 5. The API does not define if a brightness value of 0 means off,
> or lowest brightness at which the screen is still readable
> (in a low lit room), again in practice both variants happen.

6. It's not possible to change both the gamma and the brightness in the
same KMS atomic commit. You'd want to be able to reduce brightness to
conserve power, and counter the effects of that by changing gamma to
reach a visually similar image. And you'd want to have the changes take
effect at the same time instead of reducing brightness at some frame and
change gamma at some other frame. This is pretty much impossible to do
via the sysfs interface.


BR,
Jani.

> This talk will present a proposal for a new userspace API
> which intends to address these problems in the form of a
> number of new properties for drm/kms properties on the
> drm_connector object for the display-panel.
>
> This talk will also focus on how to implement this proposal
> which brings several challenges with it:
>
> 1. The mess of having multiple interfaces to control a laptop's
> internal-panel will have to be sorted out because with the new
> API we can no longer just register multiple backlight devices
> and let userspace sort things out.
>
> 2. In various cases the drm/kms driver driving the panel
> does not control the brightness itself, but the brightness
> is controlled through some (ACPI) firmware interface such
> as the acpi_video or nvidia-wmi-ec-backlight interfaces.
>
> This introduces some challenging probe-ordering issues,
> the solution for which is not entirely clear yet, so this
> part of the talk will be actively seeking audience input
> on this topic.
>
>
> Comments:
> This is a duplicate submission with one which I submitted for
> the "LPC Refereed Track" before the "Kernel Summit 2022 CFP" posting.
>
> I recently send a RFC email about this to the relevant mailinglists:
> https://lore.kernel.org/dri-devel/0d188965-d809-81b5-74ce-7d30c49fee2d@redhat.com/
>
> As well as another RFC laying out some initial backlight code
> refactoring steps. As there is a bunch of technical debt which
> needs to be addressed before work on a new uAPI can even begin:
> https://lore.kernel.org/dri-devel/98519ba0-7f18-201a-ea34-652f50343158@redhat.com/
>
> I'm working on the refactoring now. I believe the refactoring
> is more likely to land in 5.21 then in 5.20. Let alone that
> the new uAPI on the kernel side + the mandatory userspace code
> consuming the uAPI will be ready before plumbers.
>
> IOW I expect this to still be very much in flux during Plumbers,
> so this won't be a presentation presenting only already finished
> work.
>
> Regards,
>
> Hans
>
>

-- 
Jani Nikula, Intel Open Source Graphics Center

Re: [TECH TOPIC] New userspace API for display-panel brightness control

[Hans de Goede]

Hi,

On 6/28/22 15:53, Jani Nikula wrote:
> On Mon, 20 Jun 2022, Hans de Goede <hdegoede@redhat.com> wrote:
>> <resend to both lists, because of confusion of which list to use>
>>
>> Hi All,
>>
>> As requested here is a copy of my LPC kernel summit track submission:
>>
>> Title: New userspace API for display-panel brightness control
>>
>> The current userspace API for brightness control offered by
>> /sys/class/backlight devices has various problems:
>>
>> 1. There is no way to map the backlight device to a specific
>> display-output / panel
>>
>> 2. On x86 there can be multiple firmware + direct-hw-access
>> methods for controlling the backlight and the kernel may
>> register multiple backlight-devices based on this which are
>> all controlling the brightness for the same display-panel.
>> To make things worse sometimes only one of the registered
>> backlight devices actually works.
>>
>> 3. Controlling the brightness requires root-rights requiring
>> desktop-environments to use suid-root helpers for this.
>>
>> 4. The scale of the brightness value is unclear, the API does
>> not define if "perceived brightness" or electrical power is
>> being controlled and in practice both are used without userspace
>> knowing which is which.
>>
>> 5. The API does not define if a brightness value of 0 means off,
>> or lowest brightness at which the screen is still readable
>> (in a low lit room), again in practice both variants happen.
> 
> 6. It's not possible to change both the gamma and the brightness in the
> same KMS atomic commit. You'd want to be able to reduce brightness to
> conserve power, and counter the effects of that by changing gamma to
> reach a visually similar image. And you'd want to have the changes take
> effect at the same time instead of reducing brightness at some frame and
> change gamma at some other frame. This is pretty much impossible to do
> via the sysfs interface.

Ack, that is a good point.

Regards,

Hans



>> This talk will present a proposal for a new userspace API
>> which intends to address these problems in the form of a
>> number of new properties for drm/kms properties on the
>> drm_connector object for the display-panel.
>>
>> This talk will also focus on how to implement this proposal
>> which brings several challenges with it:
>>
>> 1. The mess of having multiple interfaces to control a laptop's
>> internal-panel will have to be sorted out because with the new
>> API we can no longer just register multiple backlight devices
>> and let userspace sort things out.
>>
>> 2. In various cases the drm/kms driver driving the panel
>> does not control the brightness itself, but the brightness
>> is controlled through some (ACPI) firmware interface such
>> as the acpi_video or nvidia-wmi-ec-backlight interfaces.
>>
>> This introduces some challenging probe-ordering issues,
>> the solution for which is not entirely clear yet, so this
>> part of the talk will be actively seeking audience input
>> on this topic.
>>
>>
>> Comments:
>> This is a duplicate submission with one which I submitted for
>> the "LPC Refereed Track" before the "Kernel Summit 2022 CFP" posting.
>>
>> I recently send a RFC email about this to the relevant mailinglists:
>> https://lore.kernel.org/dri-devel/0d188965-d809-81b5-74ce-7d30c49fee2d@redhat.com/
>>
>> As well as another RFC laying out some initial backlight code
>> refactoring steps. As there is a bunch of technical debt which
>> needs to be addressed before work on a new uAPI can even begin:
>> https://lore.kernel.org/dri-devel/98519ba0-7f18-201a-ea34-652f50343158@redhat.com/
>>
>> I'm working on the refactoring now. I believe the refactoring
>> is more likely to land in 5.21 then in 5.20. Let alone that
>> the new uAPI on the kernel side + the mandatory userspace code
>> consuming the uAPI will be ready before plumbers.
>>
>> IOW I expect this to still be very much in flux during Plumbers,
>> so this won't be a presentation presenting only already finished
>> work.
>>
>> Regards,
>>
>> Hans
>>
>>
> 

Re: [TECH TOPIC] New userspace API for display-panel brightness control

[Linus Walleij]

On Thu, Jun 16, 2022 at 11:33 AM Hans de Goede <hdegoede@redhat.com> wrote:

> The current userspace API for brightness control offered by
> /sys/class/backlight devices has various problems:
>
> 1. There is no way to map the backlight device to a specific
> display-output / panel

For userspace, using just sysfs you mean?
But that makes it sound like userspace needs to understand
things like backlight-to-panel topology etc.

If you add the presence of ambient light sensors to this mix
things get even messier.

I would rather make the analogy to the thermal subsystem:

- Handles multiple thermal sensors
- Handle linearization of measurements
- Some accumulate and work to monitor a thermal zone
- Handle multiple thermal zones
- Also has cooling devices (such as CPU frequency and fans)
- Policies are applied in the kernel to handle thermal sensors
  and cooling devices and control them in an orchestrated
  manner
- Userspace can sit back and enjoy the show, but it works
  out-of-the box. No magic thermal daemon.

Examples:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/thermal/thermal-zones.yaml

Wouldn't backlight rather:

- Handle multiple backlight devices.
- Handle linearization of backlight intensity
- Some accumulate and work as a composite backlight
- Handle multiple composite backlights such as screens
- Also handle ambient light sensors
- Policies are applied in the kernel to handle backlight
  and ambient light sensors together
- Userspace can sit back and enjoy the show but it works
  out-of-the box, no magic backlight daemon

I understand userspace will want to force backlight to user
preferences, older people need more backlight etc.

But isn't it more compelling to handle that as a composite
backlight device than to expose several of them to
userspace? I imagine one big knob per screen
1-100 for userspace, a bool for on/off and a bool to select
augmentation from ambient light sensors or not, the rest
the kernel can figure out.

My point is that this is not just a userspace API, it is
a policy extension of the backlight subsystem.

Maybe this is in line with what you're suggesting.
I guess I just needed to mention ambient light sensors
here.

My personal annoyance is to see several diverging
userspace implementations of policy for using ambient
light sensors with backlight. It is already annoying,
Android has something etc.

I understand that this drives a truck through the old mantra
to keep policy in userspace, but so does thermal already,
so I'd just ask myself what makes most sense.

Just my €0.01

Linus Walleij

[TECH TOPIC] New userspace API for display-panel brightness control

[Hans de Goede]

Hi All,

As requested here is a copy of my LPC kernel summit track submission:

Title: New userspace API for display-panel brightness control

The current userspace API for brightness control offered by
/sys/class/backlight devices has various problems:

1. There is no way to map the backlight device to a specific
display-output / panel

2. On x86 there can be multiple firmware + direct-hw-access
methods for controlling the backlight and the kernel may
register multiple backlight-devices based on this which are
all controlling the brightness for the same display-panel.
To make things worse sometimes only one of the registered
backlight devices actually works.

3. Controlling the brightness requires root-rights requiring
desktop-environments to use suid-root helpers for this.

4. The scale of the brightness value is unclear, the API does
not define if "perceived brightness" or electrical power is
being controlled and in practice both are used without userspace
knowing which is which.

5. The API does not define if a brightness value of 0 means off,
or lowest brightness at which the screen is still readable
(in a low lit room), again in practice both variants happen.

This talk will present a proposal for a new userspace API
which intends to address these problems in the form of a
number of new properties for drm/kms properties on the
drm_connector object for the display-panel.

This talk will also focus on how to implement this proposal
which brings several challenges with it:

1. The mess of having multiple interfaces to control a laptop's
internal-panel will have to be sorted out because with the new
API we can no longer just register multiple backlight devices
and let userspace sort things out.

2. In various cases the drm/kms driver driving the panel
does not control the brightness itself, but the brightness
is controlled through some (ACPI) firmware interface such
as the acpi_video or nvidia-wmi-ec-backlight interfaces.

This introduces some challenging probe-ordering issues,
the solution for which is not entirely clear yet, so this
part of the talk will be actively seeking audience input
on this topic.


Comments:
This is a duplicate submission with one which I submitted for
the "LPC Refereed Track" before the "Kernel Summit 2022 CFP" posting.

I recently send a RFC email about this to the relevant mailinglists:
https://lore.kernel.org/dri-devel/0d188965-d809-81b5-74ce-7d30c49fee2d@redhat.com/

As well as another RFC laying out some initial backlight code
refactoring steps. As there is a bunch of technical debt which
needs to be addressed before work on a new uAPI can even begin:
https://lore.kernel.org/dri-devel/98519ba0-7f18-201a-ea34-652f50343158@redhat.com/

I'm working on the refactoring now. I believe the refactoring
is more likely to land in 5.21 then in 5.20. Let alone that
the new uAPI on the kernel side + the mandatory userspace code
consuming the uAPI will be ready before plumbers.

IOW I expect this to still be very much in flux during Plumbers,
so this won't be a presentation presenting only already finished
work.

Regards,

Hans