Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Hans de Goede]

Hi Rafael,

We (Fedora) have been receiving a whole bunch of bug reports about
laptops getting hot/toasty while suspended with kernels >= 5.16.10
and this seems to still happen with 5.17-rc7 too.

The following are all bugzilla.redhat.com bug numbers:

   1750910 - Laptop failed to suspend and completely drained the battery
   2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
   2053957 - Package c-states never go below C2
   2056729 - No lid events when closing lid / laptop does not suspend
   2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
   2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
   2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10

And one of the bugs has also been mirrored at bugzilla.kernel.org by
the reporter:

 bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10

The common denominator here (besides the kernel version) seems to
be that these are all Ice or Tiger Lake systems (I did not do
check this applies 100% to all bugs, but it does see, to be a pattern).

A similar arch-linux report:

https://bbs.archlinux.org/viewtopic.php?id=274292&p=2

Suggest that reverting 
"ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"

which was cherry-picked into 5.16.10 fixes things.

If you want I can create Fedora kernel test-rpms of a recent
5.16.y with just that one commit reverted and ask users to
confirm if that helps. Please let me know if doing that woulkd
be useful ?

Regards,

Hans

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Rafael J. Wysocki]

On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
>
> Hi Rafael,
>
> We (Fedora) have been receiving a whole bunch of bug reports about
> laptops getting hot/toasty while suspended with kernels >= 5.16.10
> and this seems to still happen with 5.17-rc7 too.
>
> The following are all bugzilla.redhat.com bug numbers:
>
>    1750910 - Laptop failed to suspend and completely drained the battery
>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
>    2053957 - Package c-states never go below C2
>    2056729 - No lid events when closing lid / laptop does not suspend
>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>
> And one of the bugs has also been mirrored at bugzilla.kernel.org by
> the reporter:
>
>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>
> The common denominator here (besides the kernel version) seems to
> be that these are all Ice or Tiger Lake systems (I did not do
> check this applies 100% to all bugs, but it does see, to be a pattern).
>
> A similar arch-linux report:
>
> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
>
> Suggest that reverting
> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
>
> which was cherry-picked into 5.16.10 fixes things.

Thanks for letting me know!

> If you want I can create Fedora kernel test-rpms of a recent
> 5.16.y with just that one commit reverted and ask users to
> confirm if that helps. Please let me know if doing that woulkd
> be useful ?

Yes, it would.

However, it follows from the arch-linux report linked above that
5.17-rc is fine, so it would be good to also check if reverting that
commit from 5.17-rc helps.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Hans de Goede]

Hi,

On 3/9/22 14:57, Rafael J. Wysocki wrote:
> On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>
>> Hi Rafael,
>>
>> We (Fedora) have been receiving a whole bunch of bug reports about
>> laptops getting hot/toasty while suspended with kernels >= 5.16.10
>> and this seems to still happen with 5.17-rc7 too.
>>
>> The following are all bugzilla.redhat.com bug numbers:
>>
>>    1750910 - Laptop failed to suspend and completely drained the battery
>>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
>>    2053957 - Package c-states never go below C2
>>    2056729 - No lid events when closing lid / laptop does not suspend
>>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
>>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
>>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>
>> And one of the bugs has also been mirrored at bugzilla.kernel.org by
>> the reporter:
>>
>>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>
>> The common denominator here (besides the kernel version) seems to
>> be that these are all Ice or Tiger Lake systems (I did not do
>> check this applies 100% to all bugs, but it does see, to be a pattern).
>>
>> A similar arch-linux report:
>>
>> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
>>
>> Suggest that reverting
>> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
>>
>> which was cherry-picked into 5.16.10 fixes things.
> 
> Thanks for letting me know!
> 
>> If you want I can create Fedora kernel test-rpms of a recent
>> 5.16.y with just that one commit reverted and ask users to
>> confirm if that helps. Please let me know if doing that woulkd
>> be useful ?
> 
> Yes, it would.
> 
> However, it follows from the arch-linux report linked above that
> 5.17-rc is fine, so it would be good to also check if reverting that
> commit from 5.17-rc helps.

Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
the patch reverted and asked the bug-reporters to test both.

Regards,

Hans

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Rafael J. Wysocki]

On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede <hdegoede@redhat.com> wrote:
>
> Hi,
>
> On 3/9/22 14:57, Rafael J. Wysocki wrote:
> > On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
> >>
> >> Hi Rafael,
> >>
> >> We (Fedora) have been receiving a whole bunch of bug reports about
> >> laptops getting hot/toasty while suspended with kernels >= 5.16.10
> >> and this seems to still happen with 5.17-rc7 too.
> >>
> >> The following are all bugzilla.redhat.com bug numbers:
> >>
> >>    1750910 - Laptop failed to suspend and completely drained the battery
> >>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
> >>    2053957 - Package c-states never go below C2
> >>    2056729 - No lid events when closing lid / laptop does not suspend
> >>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
> >>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
> >>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> >>
> >> And one of the bugs has also been mirrored at bugzilla.kernel.org by
> >> the reporter:
> >>
> >>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> >>
> >> The common denominator here (besides the kernel version) seems to
> >> be that these are all Ice or Tiger Lake systems (I did not do
> >> check this applies 100% to all bugs, but it does see, to be a pattern).
> >>
> >> A similar arch-linux report:
> >>
> >> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
> >>
> >> Suggest that reverting
> >> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
> >>
> >> which was cherry-picked into 5.16.10 fixes things.
> >
> > Thanks for letting me know!
> >
> >> If you want I can create Fedora kernel test-rpms of a recent
> >> 5.16.y with just that one commit reverted and ask users to
> >> confirm if that helps. Please let me know if doing that woulkd
> >> be useful ?
> >
> > Yes, it would.
> >
> > However, it follows from the arch-linux report linked above that
> > 5.17-rc is fine, so it would be good to also check if reverting that
> > commit from 5.17-rc helps.
>
> Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
> the patch reverted and asked the bug-reporters to test both.

Thanks!

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Rafael J. Wysocki]

On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>
> On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede <hdegoede@redhat.com> wrote:
> >
> > Hi,
> >
> > On 3/9/22 14:57, Rafael J. Wysocki wrote:
> > > On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
> > >>
> > >> Hi Rafael,
> > >>
> > >> We (Fedora) have been receiving a whole bunch of bug reports about
> > >> laptops getting hot/toasty while suspended with kernels >= 5.16.10
> > >> and this seems to still happen with 5.17-rc7 too.
> > >>
> > >> The following are all bugzilla.redhat.com bug numbers:
> > >>
> > >>    1750910 - Laptop failed to suspend and completely drained the battery
> > >>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
> > >>    2053957 - Package c-states never go below C2
> > >>    2056729 - No lid events when closing lid / laptop does not suspend
> > >>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
> > >>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
> > >>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> > >>
> > >> And one of the bugs has also been mirrored at bugzilla.kernel.org by
> > >> the reporter:
> > >>
> > >>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> > >>
> > >> The common denominator here (besides the kernel version) seems to
> > >> be that these are all Ice or Tiger Lake systems (I did not do
> > >> check this applies 100% to all bugs, but it does see, to be a pattern).
> > >>
> > >> A similar arch-linux report:
> > >>
> > >> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
> > >>
> > >> Suggest that reverting
> > >> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
> > >>
> > >> which was cherry-picked into 5.16.10 fixes things.
> > >
> > > Thanks for letting me know!
> > >
> > >> If you want I can create Fedora kernel test-rpms of a recent
> > >> 5.16.y with just that one commit reverted and ask users to
> > >> confirm if that helps. Please let me know if doing that woulkd
> > >> be useful ?
> > >
> > > Yes, it would.
> > >
> > > However, it follows from the arch-linux report linked above that
> > > 5.17-rc is fine, so it would be good to also check if reverting that
> > > commit from 5.17-rc helps.
> >
> > Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
> > the patch reverted and asked the bug-reporters to test both.
>
> Thanks!

Also, in the cases where people have not tested 5.17-rc7 without any
reverts, it would be good to ask them to do so.

I have received another report related to this issue where the problem
is not present in 5.17-rc7 (see
https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADyNnTkpMbQ@mail.gmail.com/).

It is likely that the commit in question actually depends on some
other commits that were not backported into 5.16.y.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Hans de Goede]

Hi,

On 3/9/22 19:27, Rafael J. Wysocki wrote:
> On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>
>> On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>>
>>> Hi,
>>>
>>> On 3/9/22 14:57, Rafael J. Wysocki wrote:
>>>> On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>>>>
>>>>> Hi Rafael,
>>>>>
>>>>> We (Fedora) have been receiving a whole bunch of bug reports about
>>>>> laptops getting hot/toasty while suspended with kernels >= 5.16.10
>>>>> and this seems to still happen with 5.17-rc7 too.
>>>>>
>>>>> The following are all bugzilla.redhat.com bug numbers:
>>>>>
>>>>>    1750910 - Laptop failed to suspend and completely drained the battery
>>>>>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
>>>>>    2053957 - Package c-states never go below C2
>>>>>    2056729 - No lid events when closing lid / laptop does not suspend
>>>>>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
>>>>>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
>>>>>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>>>>
>>>>> And one of the bugs has also been mirrored at bugzilla.kernel.org by
>>>>> the reporter:
>>>>>
>>>>>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>>>>
>>>>> The common denominator here (besides the kernel version) seems to
>>>>> be that these are all Ice or Tiger Lake systems (I did not do
>>>>> check this applies 100% to all bugs, but it does see, to be a pattern).
>>>>>
>>>>> A similar arch-linux report:
>>>>>
>>>>> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
>>>>>
>>>>> Suggest that reverting
>>>>> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
>>>>>
>>>>> which was cherry-picked into 5.16.10 fixes things.
>>>>
>>>> Thanks for letting me know!
>>>>
>>>>> If you want I can create Fedora kernel test-rpms of a recent
>>>>> 5.16.y with just that one commit reverted and ask users to
>>>>> confirm if that helps. Please let me know if doing that woulkd
>>>>> be useful ?
>>>>
>>>> Yes, it would.
>>>>
>>>> However, it follows from the arch-linux report linked above that
>>>> 5.17-rc is fine, so it would be good to also check if reverting that
>>>> commit from 5.17-rc helps.
>>>
>>> Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
>>> the patch reverted and asked the bug-reporters to test both.
>>
>> Thanks!
> 
> Also, in the cases where people have not tested 5.17-rc7 without any
> reverts, it would be good to ask them to do so.

Ok, done.

> I have received another report related to this issue where the problem
> is not present in 5.17-rc7 (see
> https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADyNnTkpMbQ@mail.gmail.com/).

The first results from the Fedora test kernel builds are in:

"HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart"
https://bugzilla.redhat.com/show_bug.cgi?id=2059668

5.16.9: good
5.16.10+: bad
5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
5.17-rc7 (plain): good

So this seems to match the arch-linux report and the email report
you linked. There is a problem with the backport in 5.16.10+,
while 5.17-rc7 is fine.

> It is likely that the commit in question actually depends on some
> other commits that were not backported into 5.16.y.
I was thinking the same thing, but I've no idea which commits
that would be.

Regards,

Hans


p.s.

I've also gotten results in for:
"Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Appear from Kernel 5.16.10"
but the results there are mixed. There might be some EC firmware
issue in play there.

I'm getting the feeling that on the x1c9 there really are
2 issues which both only trigger sometimes, making testing this
problematic. One of those 2 issues is likely the same 5.16.10+
issue reported on laptops from other vendors.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Rafael J. Wysocki]

On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede <hdegoede@redhat.com> wrote:
>
> Hi,
>
> On 3/9/22 19:27, Rafael J. Wysocki wrote:
> > On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
> >>
> >> On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede <hdegoede@redhat.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On 3/9/22 14:57, Rafael J. Wysocki wrote:
> >>>> On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
> >>>>>
> >>>>> Hi Rafael,
> >>>>>
> >>>>> We (Fedora) have been receiving a whole bunch of bug reports about
> >>>>> laptops getting hot/toasty while suspended with kernels >= 5.16.10
> >>>>> and this seems to still happen with 5.17-rc7 too.
> >>>>>
> >>>>> The following are all bugzilla.redhat.com bug numbers:
> >>>>>
> >>>>>    1750910 - Laptop failed to suspend and completely drained the battery
> >>>>>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
> >>>>>    2053957 - Package c-states never go below C2
> >>>>>    2056729 - No lid events when closing lid / laptop does not suspend
> >>>>>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
> >>>>>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
> >>>>>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> >>>>>
> >>>>> And one of the bugs has also been mirrored at bugzilla.kernel.org by
> >>>>> the reporter:
> >>>>>
> >>>>>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> >>>>>
> >>>>> The common denominator here (besides the kernel version) seems to
> >>>>> be that these are all Ice or Tiger Lake systems (I did not do
> >>>>> check this applies 100% to all bugs, but it does see, to be a pattern).
> >>>>>
> >>>>> A similar arch-linux report:
> >>>>>
> >>>>> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
> >>>>>
> >>>>> Suggest that reverting
> >>>>> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
> >>>>>
> >>>>> which was cherry-picked into 5.16.10 fixes things.
> >>>>
> >>>> Thanks for letting me know!
> >>>>
> >>>>> If you want I can create Fedora kernel test-rpms of a recent
> >>>>> 5.16.y with just that one commit reverted and ask users to
> >>>>> confirm if that helps. Please let me know if doing that woulkd
> >>>>> be useful ?
> >>>>
> >>>> Yes, it would.
> >>>>
> >>>> However, it follows from the arch-linux report linked above that
> >>>> 5.17-rc is fine, so it would be good to also check if reverting that
> >>>> commit from 5.17-rc helps.
> >>>
> >>> Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
> >>> the patch reverted and asked the bug-reporters to test both.
> >>
> >> Thanks!
> >
> > Also, in the cases where people have not tested 5.17-rc7 without any
> > reverts, it would be good to ask them to do so.
>
> Ok, done.
>
> > I have received another report related to this issue where the problem
> > is not present in 5.17-rc7 (see
> > https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADyNnTkpMbQ@mail.gmail.com/).
>
> The first results from the Fedora test kernel builds are in:
>
> "HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart"
> https://bugzilla.redhat.com/show_bug.cgi?id=2059668
>
> 5.16.9: good
> 5.16.10+: bad
> 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
> 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
> 5.17-rc7 (plain): good
>
> So this seems to match the arch-linux report and the email report
> you linked. There is a problem with the backport in 5.16.10+,
> while 5.17-rc7 is fine.
>
> > It is likely that the commit in question actually depends on some
> > other commits that were not backported into 5.16.y.
> I was thinking the same thing, but I've no idea which commits
> that would be.

I do have an idea, but regardless of this, IMO the least risky way
forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel
wakeup before dispatching EC GPE" which has been backported, because
it carried a Fixes tag and not because it was marked for "stable".

Let me do that.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Hans de Goede]

Hi,

On 3/10/22 11:56, Rafael J. Wysocki wrote:
> On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede <hdegoede@redhat.com> wrote:
>>
>> Hi,
>>
>> On 3/9/22 19:27, Rafael J. Wysocki wrote:
>>> On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>>>
>>>> On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 3/9/22 14:57, Rafael J. Wysocki wrote:
>>>>>> On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>>>>>>
>>>>>>> Hi Rafael,
>>>>>>>
>>>>>>> We (Fedora) have been receiving a whole bunch of bug reports about
>>>>>>> laptops getting hot/toasty while suspended with kernels >= 5.16.10
>>>>>>> and this seems to still happen with 5.17-rc7 too.
>>>>>>>
>>>>>>> The following are all bugzilla.redhat.com bug numbers:
>>>>>>>
>>>>>>>    1750910 - Laptop failed to suspend and completely drained the battery
>>>>>>>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
>>>>>>>    2053957 - Package c-states never go below C2
>>>>>>>    2056729 - No lid events when closing lid / laptop does not suspend
>>>>>>>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
>>>>>>>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
>>>>>>>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>>>>>>
>>>>>>> And one of the bugs has also been mirrored at bugzilla.kernel.org by
>>>>>>> the reporter:
>>>>>>>
>>>>>>>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>>>>>>
>>>>>>> The common denominator here (besides the kernel version) seems to
>>>>>>> be that these are all Ice or Tiger Lake systems (I did not do
>>>>>>> check this applies 100% to all bugs, but it does see, to be a pattern).
>>>>>>>
>>>>>>> A similar arch-linux report:
>>>>>>>
>>>>>>> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
>>>>>>>
>>>>>>> Suggest that reverting
>>>>>>> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
>>>>>>>
>>>>>>> which was cherry-picked into 5.16.10 fixes things.
>>>>>>
>>>>>> Thanks for letting me know!
>>>>>>
>>>>>>> If you want I can create Fedora kernel test-rpms of a recent
>>>>>>> 5.16.y with just that one commit reverted and ask users to
>>>>>>> confirm if that helps. Please let me know if doing that woulkd
>>>>>>> be useful ?
>>>>>>
>>>>>> Yes, it would.
>>>>>>
>>>>>> However, it follows from the arch-linux report linked above that
>>>>>> 5.17-rc is fine, so it would be good to also check if reverting that
>>>>>> commit from 5.17-rc helps.
>>>>>
>>>>> Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
>>>>> the patch reverted and asked the bug-reporters to test both.
>>>>
>>>> Thanks!
>>>
>>> Also, in the cases where people have not tested 5.17-rc7 without any
>>> reverts, it would be good to ask them to do so.
>>
>> Ok, done.
>>
>>> I have received another report related to this issue where the problem
>>> is not present in 5.17-rc7 (see
>>> https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADyNnTkpMbQ@mail.gmail.com/).
>>
>> The first results from the Fedora test kernel builds are in:
>>
>> "HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart"
>> https://bugzilla.redhat.com/show_bug.cgi?id=2059668
>>
>> 5.16.9: good
>> 5.16.10+: bad
>> 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
>> 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
>> 5.17-rc7 (plain): good
>>
>> So this seems to match the arch-linux report and the email report
>> you linked. There is a problem with the backport in 5.16.10+,
>> while 5.17-rc7 is fine.
>>
>>> It is likely that the commit in question actually depends on some
>>> other commits that were not backported into 5.16.y.
>> I was thinking the same thing, but I've no idea which commits
>> that would be.
> 
> I do have an idea, but regardless of this, IMO the least risky way
> forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel
> wakeup before dispatching EC GPE" which has been backported, because
> it carried a Fixes tag and not because it was marked for "stable".
> 
> Let me do that.

Ok, that sounds good, thank you.

Regards,

Hans

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Limonciello, Mario]

+ KH

On 3/10/2022 06:22, Hans de Goede wrote:
> Hi,
> 
> On 3/10/22 11:56, Rafael J. Wysocki wrote:
>> On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede <hdegoede@redhat.com> wrote:
>>>
>>> Hi,
>>>
>>> On 3/9/22 19:27, Rafael J. Wysocki wrote:
>>>> On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>>>>>
>>>>> On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 3/9/22 14:57, Rafael J. Wysocki wrote:
>>>>>>> On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>>>>>>>
>>>>>>>> Hi Rafael,
>>>>>>>>
>>>>>>>> We (Fedora) have been receiving a whole bunch of bug reports about
>>>>>>>> laptops getting hot/toasty while suspended with kernels >= 5.16.10
>>>>>>>> and this seems to still happen with 5.17-rc7 too.
>>>>>>>>
>>>>>>>> The following are all bugzilla.redhat.com bug numbers:
>>>>>>>>
>>>>>>>>     1750910 - Laptop failed to suspend and completely drained the battery
>>>>>>>>     2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
>>>>>>>>     2053957 - Package c-states never go below C2
>>>>>>>>     2056729 - No lid events when closing lid / laptop does not suspend
>>>>>>>>     2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
>>>>>>>>     2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
>>>>>>>>     2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>>>>>>>
>>>>>>>> And one of the bugs has also been mirrored at bugzilla.kernel.org by
>>>>>>>> the reporter:
>>>>>>>>
>>>>>>>>   bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>>>>>>>>
>>>>>>>> The common denominator here (besides the kernel version) seems to
>>>>>>>> be that these are all Ice or Tiger Lake systems (I did not do
>>>>>>>> check this applies 100% to all bugs, but it does see, to be a pattern).
>>>>>>>>
>>>>>>>> A similar arch-linux report:
>>>>>>>>
>>>>>>>> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
>>>>>>>>
>>>>>>>> Suggest that reverting
>>>>>>>> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
>>>>>>>>
>>>>>>>> which was cherry-picked into 5.16.10 fixes things.
>>>>>>>
>>>>>>> Thanks for letting me know!
>>>>>>>
>>>>>>>> If you want I can create Fedora kernel test-rpms of a recent
>>>>>>>> 5.16.y with just that one commit reverted and ask users to
>>>>>>>> confirm if that helps. Please let me know if doing that woulkd
>>>>>>>> be useful ?
>>>>>>>
>>>>>>> Yes, it would.
>>>>>>>
>>>>>>> However, it follows from the arch-linux report linked above that
>>>>>>> 5.17-rc is fine, so it would be good to also check if reverting that
>>>>>>> commit from 5.17-rc helps.
>>>>>>
>>>>>> Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
>>>>>> the patch reverted and asked the bug-reporters to test both.
>>>>>
>>>>> Thanks!
>>>>
>>>> Also, in the cases where people have not tested 5.17-rc7 without any
>>>> reverts, it would be good to ask them to do so.
>>>
>>> Ok, done.
>>>
>>>> I have received another report related to this issue where the problem
>>>> is not present in 5.17-rc7 (see
>>>> https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADyNnTkpMbQ@mail.gmail.com/).
>>>
>>> The first results from the Fedora test kernel builds are in:
>>>
>>> "HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart"
>>> https://bugzilla.redhat.com/show_bug.cgi?id=2059668
>>>
>>> 5.16.9: good
>>> 5.16.10+: bad
>>> 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
>>> 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
>>> 5.17-rc7 (plain): good
>>>
>>> So this seems to match the arch-linux report and the email report
>>> you linked. There is a problem with the backport in 5.16.10+,
>>> while 5.17-rc7 is fine.
>>>
>>>> It is likely that the commit in question actually depends on some
>>>> other commits that were not backported into 5.16.y.
>>> I was thinking the same thing, but I've no idea which commits
>>> that would be.
>>
>> I do have an idea, but regardless of this, IMO the least risky way
>> forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel
>> wakeup before dispatching EC GPE" which has been backported, because
>> it carried a Fixes tag and not because it was marked for "stable".
>>
>> Let me do that.
> 
> Ok, that sounds good, thank you.
> 

Just FWIW this fix that was backported to stable also fixed keyboard 
wakeup from s2idle on a number of HP laptops too.  I know for sure that 
it fixed it on the AMD versions of them, and Kai Heng Feng suspected it 
will also fix it for the Intel versions.  So if there is another commit 
that can be backported from 5.17 to make it safer for the other systems, 
I think we should consider doing that to solve it too.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Rafael J. Wysocki]

On Mon, Mar 14, 2022 at 3:37 PM Limonciello, Mario
<mario.limonciello@amd.com> wrote:
>
> + KH
>
> On 3/10/2022 06:22, Hans de Goede wrote:
> > Hi,
> >
> > On 3/10/22 11:56, Rafael J. Wysocki wrote:
> >> On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede <hdegoede@redhat.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On 3/9/22 19:27, Rafael J. Wysocki wrote:
> >>>> On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
> >>>>>
> >>>>> On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede <hdegoede@redhat.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 3/9/22 14:57, Rafael J. Wysocki wrote:
> >>>>>>> On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede <hdegoede@redhat.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi Rafael,
> >>>>>>>>
> >>>>>>>> We (Fedora) have been receiving a whole bunch of bug reports about
> >>>>>>>> laptops getting hot/toasty while suspended with kernels >= 5.16.10
> >>>>>>>> and this seems to still happen with 5.17-rc7 too.
> >>>>>>>>
> >>>>>>>> The following are all bugzilla.redhat.com bug numbers:
> >>>>>>>>
> >>>>>>>>     1750910 - Laptop failed to suspend and completely drained the battery
> >>>>>>>>     2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
> >>>>>>>>     2053957 - Package c-states never go below C2
> >>>>>>>>     2056729 - No lid events when closing lid / laptop does not suspend
> >>>>>>>>     2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
> >>>>>>>>     2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
> >>>>>>>>     2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> >>>>>>>>
> >>>>>>>> And one of the bugs has also been mirrored at bugzilla.kernel.org by
> >>>>>>>> the reporter:
> >>>>>>>>
> >>>>>>>>   bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> >>>>>>>>
> >>>>>>>> The common denominator here (besides the kernel version) seems to
> >>>>>>>> be that these are all Ice or Tiger Lake systems (I did not do
> >>>>>>>> check this applies 100% to all bugs, but it does see, to be a pattern).
> >>>>>>>>
> >>>>>>>> A similar arch-linux report:
> >>>>>>>>
> >>>>>>>> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
> >>>>>>>>
> >>>>>>>> Suggest that reverting
> >>>>>>>> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
> >>>>>>>>
> >>>>>>>> which was cherry-picked into 5.16.10 fixes things.
> >>>>>>>
> >>>>>>> Thanks for letting me know!
> >>>>>>>
> >>>>>>>> If you want I can create Fedora kernel test-rpms of a recent
> >>>>>>>> 5.16.y with just that one commit reverted and ask users to
> >>>>>>>> confirm if that helps. Please let me know if doing that woulkd
> >>>>>>>> be useful ?
> >>>>>>>
> >>>>>>> Yes, it would.
> >>>>>>>
> >>>>>>> However, it follows from the arch-linux report linked above that
> >>>>>>> 5.17-rc is fine, so it would be good to also check if reverting that
> >>>>>>> commit from 5.17-rc helps.
> >>>>>>
> >>>>>> Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with
> >>>>>> the patch reverted and asked the bug-reporters to test both.
> >>>>>
> >>>>> Thanks!
> >>>>
> >>>> Also, in the cases where people have not tested 5.17-rc7 without any
> >>>> reverts, it would be good to ask them to do so.
> >>>
> >>> Ok, done.
> >>>
> >>>> I have received another report related to this issue where the problem
> >>>> is not present in 5.17-rc7 (see
> >>>> https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADyNnTkpMbQ@mail.gmail.com/).
> >>>
> >>> The first results from the Fedora test kernel builds are in:
> >>>
> >>> "HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart"
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=2059668
> >>>
> >>> 5.16.9: good
> >>> 5.16.10+: bad
> >>> 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
> >>> 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good
> >>> 5.17-rc7 (plain): good
> >>>
> >>> So this seems to match the arch-linux report and the email report
> >>> you linked. There is a problem with the backport in 5.16.10+,
> >>> while 5.17-rc7 is fine.
> >>>
> >>>> It is likely that the commit in question actually depends on some
> >>>> other commits that were not backported into 5.16.y.
> >>> I was thinking the same thing, but I've no idea which commits
> >>> that would be.
> >>
> >> I do have an idea, but regardless of this, IMO the least risky way
> >> forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel
> >> wakeup before dispatching EC GPE" which has been backported, because
> >> it carried a Fixes tag and not because it was marked for "stable".
> >>
> >> Let me do that.
> >
> > Ok, that sounds good, thank you.
> >
>
> Just FWIW this fix that was backported to stable also fixed keyboard
> wakeup from s2idle on a number of HP laptops too.  I know for sure that
> it fixed it on the AMD versions of them, and Kai Heng Feng suspected it
> will also fix it for the Intel versions.  So if there is another commit
> that can be backported from 5.17 to make it safer for the other systems,
> I think we should consider doing that to solve it too.

There is a series of ACPI EC driver commits that are present in
5.17-rc, but have not been included in any "stable" series:

befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop
acpi_ec_delete_query()
c33676aa4824 ACPI: EC: Make the event work state machine visible
c793570d8725 ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event()
eafe7509ab8c ACPI: EC: Rename three functions
a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler()
388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler()
98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into acpi_ec_event_handler()
1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query()
ca8283dcd933 ACPI: EC: Call advance_transaction() from acpi_ec_dispatch_gpe()

It is likely that they prevent the problem exposed by the problematic
commit from occurring, but I'm not sure which ones do that.  Some of
them are clearly cosmetic, but the ordering matters.

RE: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Limonciello, Mario]

[Public]

> > Just FWIW this fix that was backported to stable also fixed keyboard
> > wakeup from s2idle on a number of HP laptops too.  I know for sure that
> > it fixed it on the AMD versions of them, and Kai Heng Feng suspected it
> > will also fix it for the Intel versions.  So if there is another commit
> > that can be backported from 5.17 to make it safer for the other systems,
> > I think we should consider doing that to solve it too.
> 
> There is a series of ACPI EC driver commits that are present in
> 5.17-rc, but have not been included in any "stable" series:
> 
> befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop
> acpi_ec_delete_query()
> c33676aa4824 ACPI: EC: Make the event work state machine visible
> c793570d8725 ACPI: EC: Avoid queuing unnecessary work in
> acpi_ec_submit_event()
> eafe7509ab8c ACPI: EC: Rename three functions
> a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler()
> 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler()
> 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into
> acpi_ec_event_handler()
> 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query()
> ca8283dcd933 ACPI: EC: Call advance_transaction() from
> acpi_ec_dispatch_gpe()
> 
> It is likely that they prevent the problem exposed by the problematic
> commit from occurring, but I'm not sure which ones do that.  Some of
> them are clearly cosmetic, but the ordering matters.

Hans,

Do you think you could get one of the folks who reported this regression to do
a bisect to see which one "fixed" it?  If we get lucky we can come down to
some smaller hunks of code that can come back to stable instead of reverting.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Rafael J. Wysocki]

On Wed, Mar 16, 2022 at 2:38 PM Limonciello, Mario
<Mario.Limonciello@amd.com> wrote:
>
> [Public]
>
> > > Just FWIW this fix that was backported to stable also fixed keyboard
> > > wakeup from s2idle on a number of HP laptops too.  I know for sure that
> > > it fixed it on the AMD versions of them, and Kai Heng Feng suspected it
> > > will also fix it for the Intel versions.  So if there is another commit
> > > that can be backported from 5.17 to make it safer for the other systems,
> > > I think we should consider doing that to solve it too.
> >
> > There is a series of ACPI EC driver commits that are present in
> > 5.17-rc, but have not been included in any "stable" series:
> >
> > befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop
> > acpi_ec_delete_query()
> > c33676aa4824 ACPI: EC: Make the event work state machine visible
> > c793570d8725 ACPI: EC: Avoid queuing unnecessary work in
> > acpi_ec_submit_event()
> > eafe7509ab8c ACPI: EC: Rename three functions
> > a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler()
> > 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler()
> > 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into
> > acpi_ec_event_handler()
> > 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query()
> > ca8283dcd933 ACPI: EC: Call advance_transaction() from
> > acpi_ec_dispatch_gpe()
> >
> > It is likely that they prevent the problem exposed by the problematic
> > commit from occurring, but I'm not sure which ones do that.  Some of
> > them are clearly cosmetic, but the ordering matters.
>
> Hans,
>
> Do you think you could get one of the folks who reported this regression to do
> a bisect to see which one "fixed" it?  If we get lucky we can come down to
> some smaller hunks of code that can come back to stable instead of reverting.

It's been reverted already.

What we can do is to request adding it back along with other commits
needed for it to work as expected.

Also, I think we'll need all of the commits listed up to and including
c793570d8725 ("ACPI: EC: Avoid queuing unnecessary work in
acpi_ec_submit_event()") at least, but that's just a guess.

RE: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Limonciello, Mario]

[Public]



> -----Original Message-----
> From: Rafael J. Wysocki <rafael@kernel.org>
> Sent: Wednesday, March 16, 2022 08:46
> To: Limonciello, Mario <Mario.Limonciello@amd.com>
> Cc: Rafael J. Wysocki <rafael@kernel.org>; Hans de Goede
> <hdegoede@redhat.com>; Linux PM <linux-pm@vger.kernel.org>; Stable
> <stable@vger.kernel.org>; Justin Forbes <jmforbes@linuxtx.org>; Mark
> Pearson <markpearson@lenovo.com>; ACPI Devel Maling List <linux-
> acpi@vger.kernel.org>; Kai-Heng Feng <kai.heng.feng@canonical.com>
> Subject: Re: Many reports of laptops getting hot while suspended with
> kernels >= 5.16.10 || >= 5.17-rc1
> 
> On Wed, Mar 16, 2022 at 2:38 PM Limonciello, Mario
> <Mario.Limonciello@amd.com> wrote:
> >
> > [Public]
> >
> > > > Just FWIW this fix that was backported to stable also fixed keyboard
> > > > wakeup from s2idle on a number of HP laptops too.  I know for sure
> that
> > > > it fixed it on the AMD versions of them, and Kai Heng Feng suspected it
> > > > will also fix it for the Intel versions.  So if there is another commit
> > > > that can be backported from 5.17 to make it safer for the other
> systems,
> > > > I think we should consider doing that to solve it too.
> > >
> > > There is a series of ACPI EC driver commits that are present in
> > > 5.17-rc, but have not been included in any "stable" series:
> > >
> > > befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop
> > > acpi_ec_delete_query()
> > > c33676aa4824 ACPI: EC: Make the event work state machine visible
> > > c793570d8725 ACPI: EC: Avoid queuing unnecessary work in
> > > acpi_ec_submit_event()
> > > eafe7509ab8c ACPI: EC: Rename three functions
> > > a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler()
> > > 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler()
> > > 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into
> > > acpi_ec_event_handler()
> > > 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query()
> > > ca8283dcd933 ACPI: EC: Call advance_transaction() from
> > > acpi_ec_dispatch_gpe()
> > >
> > > It is likely that they prevent the problem exposed by the problematic
> > > commit from occurring, but I'm not sure which ones do that.  Some of
> > > them are clearly cosmetic, but the ordering matters.
> >
> > Hans,
> >
> > Do you think you could get one of the folks who reported this regression to
> do
> > a bisect to see which one "fixed" it?  If we get lucky we can come down to
> > some smaller hunks of code that can come back to stable instead of
> reverting.
> 
> It's been reverted already.
> 
> What we can do is to request adding it back along with other commits
> needed for it to work as expected.

OK thanks, makes sense.

> 
> Also, I think we'll need all of the commits listed up to and including
> c793570d8725 ("ACPI: EC: Avoid queuing unnecessary work in
> acpi_ec_submit_event()") at least, but that's just a guess.

Yeah so we'll for sure need a bisect to confirm and come up with that list.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Hans de Goede]

Hi,

On 3/16/22 14:37, Limonciello, Mario wrote:
> [Public]
> 
>>> Just FWIW this fix that was backported to stable also fixed keyboard
>>> wakeup from s2idle on a number of HP laptops too.  I know for sure that
>>> it fixed it on the AMD versions of them, and Kai Heng Feng suspected it
>>> will also fix it for the Intel versions.  So if there is another commit
>>> that can be backported from 5.17 to make it safer for the other systems,
>>> I think we should consider doing that to solve it too.
>>
>> There is a series of ACPI EC driver commits that are present in
>> 5.17-rc, but have not been included in any "stable" series:
>>
>> befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop
>> acpi_ec_delete_query()
>> c33676aa4824 ACPI: EC: Make the event work state machine visible
>> c793570d8725 ACPI: EC: Avoid queuing unnecessary work in
>> acpi_ec_submit_event()
>> eafe7509ab8c ACPI: EC: Rename three functions
>> a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler()
>> 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler()
>> 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into
>> acpi_ec_event_handler()
>> 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query()
>> ca8283dcd933 ACPI: EC: Call advance_transaction() from
>> acpi_ec_dispatch_gpe()
>>
>> It is likely that they prevent the problem exposed by the problematic
>> commit from occurring, but I'm not sure which ones do that.  Some of
>> them are clearly cosmetic, but the ordering matters.
> 
> Hans,
> 
> Do you think you could get one of the folks who reported this regression to do
> a bisect to see which one "fixed" it?

We already know which commit is causing the regression. As Rafael already
said the question is why things are not broken in 5.17 and that is not
a straight forward bisect. So figuring this out is going to be a lot
of work and I'm not sure of that it is worth it. I certainly don't
have time to help users with debugging this.

> If we get lucky we can come down to
> some smaller hunks of code that can come back to stable instead of reverting.

5.17 is almost done and in a couple of weeks Fedora (and Arch and other
distros tracking the mainline kernel) will move to 5.17 resolving the
wakeup by keyboard issue not working there.

5.16 is not a LTS kernel, so for other distros we would at a minimum
figure out what needs to be backported to make things work with 5.15
making the delta / set of possible patches we need even bigger.
So as already said IMHO this is not worth it, at least assuming that
nothing bad happens when attempting wakeup by keyboard, iow it
just does not work and does not put the laptop in some bad state?

Regards,

Hans

RE: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Limonciello, Mario]

[Public]


> > Hans,
> >
> > Do you think you could get one of the folks who reported this regression to
> do
> > a bisect to see which one "fixed" it?
> 
> We already know which commit is causing the regression. As Rafael already
> said the question is why things are not broken in 5.17 and that is not
> a straight forward bisect. So figuring this out is going to be a lot
> of work and I'm not sure of that it is worth it. I certainly don't
> have time to help users with debugging this.
> 
> > If we get lucky we can come down to
> > some smaller hunks of code that can come back to stable instead of
> reverting.
> 
> 5.17 is almost done and in a couple of weeks Fedora (and Arch and other
> distros tracking the mainline kernel) will move to 5.17 resolving the
> wakeup by keyboard issue not working there.
> 
> 5.16 is not a LTS kernel, so for other distros we would at a minimum
> figure out what needs to be backported to make things work with 5.15
> making the delta / set of possible patches we need even bigger.
> So as already said IMHO this is not worth it, at least assuming that
> nothing bad happens when attempting wakeup by keyboard, iow it
> just does not work and does not put the laptop in some bad state?
> 
> Regards,
> 
> Hans

You're right, no bad state as a result and probably not worth the tradeoff for a bisect effort.
OK, thanks.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker.

On 09.03.22 14:44, Hans de Goede wrote:
> 
> We (Fedora) have been receiving a whole bunch of bug reports about
> laptops getting hot/toasty while suspended with kernels >= 5.16.10
> and this seems to still happen with 5.17-rc7 too.

I was about to sent a similar mail, but then I found this one. Thx for
making my life easier. :-D

But could you do me a big favor and CC the regression mailing list
(regressions@lists.linux.dev) in case similar situations arise in the
future? tia!

> The following are all bugzilla.redhat.com bug numbers:
> 
>    1750910 - Laptop failed to suspend and completely drained the battery
>    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
>    2053957 - Package c-states never go below C2
>    2056729 - No lid events when closing lid / laptop does not suspend
>    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
>    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
>    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> 
> And one of the bugs has also been mirrored at bugzilla.kernel.org by
> the reporter:
> 
>  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10

Here is another, but it's basically linking to reports you already
mentioned:
https://bugzilla.kernel.org/show_bug.cgi?id=215661

> The common denominator here (besides the kernel version) seems to
> be that these are all Ice or Tiger Lake systems (I did not do
> check this applies 100% to all bugs, but it does see, to be a pattern).
> 
> A similar arch-linux report:
> 
> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
> 
> Suggest that reverting 
> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
> 
> which was cherry-picked into 5.16.10 fixes things.

From the thread I gather that it looks like 5.17 is not affected; if
that changes, could anybody please give me a heads up please?

> If you want I can create Fedora kernel test-rpms of a recent
> 5.16.y with just that one commit reverted and ask users to
> confirm if that helps. Please let me know if doing that woulkd
> be useful ?

FWIW: To be sure below issue doesn't fall through the cracks unnoticed,
I'm adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced 4287509b4d21e34dc49266c
#regzbot ignore-activity

If it turns out this isn't a regression, free free to remove it from the
tracking by sending a reply to this thread containing a paragraph like
"#regzbot invalid: reason why this is invalid" (without the quotes).

Reminder for developers: when fixing the issue, please add a 'Link:'
tags pointing to the report (the mail quoted above) using
lore.kernel.org/r/, as explained in
'Documentation/process/submitting-patches.rst' and
'Documentation/process/5.Posting.rst'. Regzbot needs them to
automatically connect reports with fixes, but they are useful in
general, too.

I'm sending this to everyone that got the initial report, to make
everyone aware of the tracking. I also hope that messages like this
motivate people to directly get at least the regression mailing list and
ideally even regzbot involved when dealing with regressions, as messages
like this wouldn't be needed then. And don't worry, if I need to send
other mails regarding this regression only relevant for regzbot I'll
send them to the regressions lists only (with a tag in the subject so
people can filter them away). With a bit of luck no such messages will
be needed anyway.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I'm getting a lot of
reports on my table. I can only look briefly into most of them and lack
knowledge about most of the areas they concern. I thus unfortunately
will sometimes get things wrong or miss something important. I hope
that's not the case here; if you think it is, don't hesitate to tell me
in a public reply, it's in everyone's interest to set the public record
straight.

-- 
Additional information about regzbot:

If you want to know more about regzbot, check out its web-interface, the
getting start guide, and the references documentation:

https://linux-regtracking.leemhuis.info/regzbot/
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md

The last two documents will explain how you can interact with regzbot
yourself if your want to.

Hint for reporters: when reporting a regression it's in your interest to
CC the regression list and tell regzbot about the issue, as that ensures
the regression makes it onto the radar of the Linux kernel's regression
tracker -- that's in your interest, as it ensures your report won't fall
through the cracks unnoticed.

Hint for developers: you normally don't need to care about regzbot once
it's involved. Fix the issue as you normally would, just remember to
include 'Link:' tag in the patch descriptions pointing to all reports
about the issue. This has been expected from developers even before
regzbot showed up for reasons explained in
'Documentation/process/submitting-patches.rst' and
'Documentation/process/5.Posting.rst'.

Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1

[Rafael J. Wysocki]

On Thu, Mar 10, 2022 at 11:02 AM Thorsten Leemhuis
<regressions@leemhuis.info> wrote:
>
> Hi, this is your Linux kernel regression tracker.
>
> On 09.03.22 14:44, Hans de Goede wrote:
> >
> > We (Fedora) have been receiving a whole bunch of bug reports about
> > laptops getting hot/toasty while suspended with kernels >= 5.16.10
> > and this seems to still happen with 5.17-rc7 too.
>
> I was about to sent a similar mail, but then I found this one. Thx for
> making my life easier. :-D
>
> But could you do me a big favor and CC the regression mailing list
> (regressions@lists.linux.dev) in case similar situations arise in the
> future? tia!
>
> > The following are all bugzilla.redhat.com bug numbers:
> >
> >    1750910 - Laptop failed to suspend and completely drained the battery
> >    2050036 - Framework laptop: 5.16.5 breaks s2idle sleep
> >    2053957 - Package c-states never go below C2
> >    2056729 - No lid events when closing lid / laptop does not suspend
> >    2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap
> >    2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge
> >    2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
> >
> > And one of the bugs has also been mirrored at bugzilla.kernel.org by
> > the reporter:
> >
> >  bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
>
> Here is another, but it's basically linking to reports you already
> mentioned:
> https://bugzilla.kernel.org/show_bug.cgi?id=215661
>
> > The common denominator here (besides the kernel version) seems to
> > be that these are all Ice or Tiger Lake systems (I did not do
> > check this applies 100% to all bugs, but it does see, to be a pattern).
> >
> > A similar arch-linux report:
> >
> > https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
> >
> > Suggest that reverting
> > "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
> >
> > which was cherry-picked into 5.16.10 fixes things.
>
> From the thread I gather that it looks like 5.17 is not affected; if

This is most likely correct.

There are at least 3 different sources that have confirmed that.

> that changes, could anybody please give me a heads up please?

Sure.