'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

[Danny van Heumen]

Hi all,

This is my first report directly to linux kernel mailing lists. I'll do my best, but I'll invariably make some mistakes. This bug report is based on "educated" guessing, some insight and intuition, so I will do my best to explain all the curiosities and why I blame `schedutil`. To clarify, I seems behavior of schedutil triggers issues elsewhere.

I write the message not to say "this is the problem, fix it!" but rather, I hope to check with you if you recognize potential things to look for, *if the reports make any sense*.

## short summary

On multiple computers, with kernel builds from distros and also vanilla custom (minimal) config builds by myself, I have had a number of issues that all center around suspend/resume with 'schedutil' scaling governor active. Some issues seem to be explained by other causes/bug-reports, but I report this here because `schedutil` is a common denominator.

Characteristics:
- Kabylake, upon suspend, problems with going into suspend, with "runaway processor" behavior starting to run at full power causing excessive build-up of heat. (intel_pstate=passive) (more details follow ...)
- Haswell laptop, upon suspend, sometimes does not suspend, instead screen off but then returned to active.
- Pinebook Pro: upon resume from suspend (s2idle, I'm fairly certain): problems getting "display panel up in time", "eMMC/SDIO failing to initialize", screen-flickering from excessive refreshing many times the blinking rate. (more details follow ...)

These issues start to happen upon suspend. The issues do not happen when `schedutil` is not involved.

My interpretation: somewhere just before, or just after entering suspend-state/initiating restore, `schedutil` messes up something w.r.t. timings: timings/delays/repeats happen at many times the intended speed.

## `schedutil` before suspend: all good

I have been using `schedutil` for many months. Upon normal operations there never seem to be any issues, or unexpected events. What I describe all happens centered around the suspension-process, and sometimes as consequence afterwards.

## Pinebook Pro issues

- original Debian kernel,
- custom-built kernel with very minimal config: versions 5.15.{38,39,41} 5.17.9
- `schedutil` cpufreq scaling governor
- Debian bullseye (original, no third-party kernels), up-to-date install
- no tlp, manual suspend procedure (either button- or lid-triggered)

The Pinebook Pro does not exhibit issues (AFAICT) upon entering suspend. However, resume will fail often, but not always. I have seen different symptoms exhibited:

1.) 3-line error message regarding analogix_dp_resume about rockchip-module not succeeding (through analogix) to re-initialize the display panel in time. (It shows on that exact display panel, so clearly it worked.)
Those 3 lines of error message, are being refreshed at many times the necessary rate. The excessive refreshing interferes with the ability to switch terminals (TTY1, TTY2, etc.), because I see a "single frame blink" of the terminal and then gets overridden with the 3-line error message again. Same holds for switch to GUI DISPLAY terminal. Display does not have sufficient time to refresh and ends up with variations of extra long DPMS-off state before returning to the error-message-screen.
Sometimes I am able to "interrupt" this excessive run-away behavior by keeping e.g. CTRL+ALT+1 pressed such that it will try to switch TTY at "key-repeat"-speed.

2.) error message regarding eMMC/SDIO issues. Similar to (1.) issue with failing to initialize eMMC and consequently I lose access to my persistent storage.

3.) issue also occurs when entering sleep from prolonged idleness with lid already closed (DPMS off + locked) as starting-point.

[x.] Not sure if relevant at all: "crashes with wifi firmware". A buggy early firmware version for a broadcom wifi chip would occasionally crash. However, in particular around the suspend process there would be issues.
I don't want to attribute this to schedutil, but maybe schedutil's behavior significantly increases the chance of this happening.

NOTE: I tried switching back to 'ondemand' scheduler after first issues had occurred with 'schedutil' active. However, reverting to 'ondemand' did not resolve the outstanding issues at that point.

## Kabylake-laptop

- Distro-kernel (i.e. not vanilla)
- `intel_pstate=passive` kernel parameter
- `schedutil` as cpufreq scaling governor (udev-rule)

1.) Trigger suspend on the laptop. System goes into suspend state as expected. Then (in about 2 seconds) fans start spinning and apparently excessive heat is being produced. (This was reported somewhere already as being an issue with PCH temp being too high, w.r.t. `S0ix` `intel_pch_thermal`.) However, I suspect that high-temperature may be caused through `schedutil`, because the laptop did not run any intensive tasks of any kind. (idling)
Upon resuming, the laptop has lost significant amounts of battery, corresponding with the excessive CPU usage and cooling fans.

## Haswell laptop

1.) Not much to say: enter suspend, find out later that it did not truly enter suspend, but was kept back and is rather active.

## Without `schedutil`

In all cases, when taking schedutil out of the loop, these issues disappear. In case of the Kabylake laptop, I do set intel_pstate=active. All have `ondemand` governor. Suspend/resume works, repeated many times over. Even if some errors are still shown, they no longer pose a problem. The excessive screen blinking behavior is not exhibited, for example.

## Other things I have tried

As mentioned before, I have tried looking for other bugs. However, the issues seem to be too persistent. I have tried removing all external devices, tried using different USB ports, tried different suspend/resume settings in `/etc/systemd/sleep.conf`. Tried using 'tlp' package for additional power-management tricks. Tried switching governors, ... and probably more.

I realize this bug report is far from perfect. I'm afraid I cannot make the report much more exact/clear. I am happy to answer any questions you may have. I will keep an eye out for further peculiarities.

Regards,
Danny

PS: I hope I address this to the right maintainers. I checked here: https://docs.kernel.org/process/maintainers.html#cpu-frequency-scaling-framework

Re: 'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

[Viresh Kumar]

+Peter

On 21-05-22, 22:13, Danny van Heumen wrote:
> Hi all,
> 
> This is my first report directly to linux kernel mailing lists. I'll
> do my best, but I'll invariably make some mistakes.

That's fine. Just two things you need to take care of normally, keep
the line length to 100 columns (as I have done now) and don't do
top-posting on replies. That's all :)

> This bug report
> is based on "educated" guessing, some insight and intuition, so I
> will do my best to explain all the curiosities and why I blame
> `schedutil`. To clarify, I seems behavior of schedutil triggers
> issues elsewhere.
> 
> I write the message not to say "this is the problem, fix it!" but
> rather, I hope to check with you if you recognize potential things
> to look for, *if the reports make any sense*.
> 
> ## short summary
> 
> On multiple computers, with kernel builds from distros and also
> vanilla custom (minimal) config builds by myself, I have had a
> number of issues that all center around suspend/resume with
> 'schedutil' scaling governor active. Some issues seem to be
> explained by other causes/bug-reports, but I report this here
> because `schedutil` is a common denominator.
> 
> Characteristics:
> - Kabylake, upon suspend, problems with going into suspend, with
> "runaway processor" behavior starting to run at full power causing
> excessive build-up of heat. (intel_pstate=passive) (more details
> follow ...)

If this is actually related to cpufreq subsystem, then maybe the CPU
going down was programmed at a higher frequency (because there was
work to do during suspend) while going offline. I am not sure how it
works on Intel, but on ARM we have something like a
"suspend-frequency", which is programmed right before we suspend to
make sure not to consume a lot of power. Maybe we can do something
like this in intel-pstate driver, not sure though.

> - Haswell laptop, upon suspend, sometimes does not suspend, instead
> screen off but then returned to active.
> - Pinebook Pro: upon resume from suspend (s2idle, I'm fairly
> certain): problems getting "display panel up in time", "eMMC/SDIO
> failing to initialize", screen-flickering from excessive refreshing
> many times the blinking rate. (more details follow ...)
> 
> These issues start to happen upon suspend. The issues do not happen
> when `schedutil` is not involved.
> 
> My interpretation: somewhere just before, or just after entering
> suspend-state/initiating restore, `schedutil` messes up something
> w.r.t. timings: timings/delays/repeats happen at many times the
> intended speed.
> 
> ## `schedutil` before suspend: all good
> 
> I have been using `schedutil` for many months. Upon normal
> operations there never seem to be any issues, or unexpected events.
> What I describe all happens centered around the suspension-process,
> and sometimes as consequence afterwards.
> 
> ## Pinebook Pro issues
> 
> - original Debian kernel,
> - custom-built kernel with very minimal config: versions
> 5.15.{38,39,41} 5.17.9
> - `schedutil` cpufreq scaling governor
> - Debian bullseye (original, no third-party kernels), up-to-date
> install
> - no tlp, manual suspend procedure (either button- or lid-triggered)
> 
> The Pinebook Pro does not exhibit issues (AFAICT) upon entering
> suspend. However, resume will fail often, but not always. I have
> seen different symptoms exhibited:
> 
> 1.) 3-line error message regarding analogix_dp_resume about
>   rockchip-module not succeeding (through analogix) to re-initialize
>   the display panel in time. (It shows on that exact display panel,
>   so clearly it worked.)
> Those 3 lines of error message, are being refreshed at many times
> the necessary rate. The excessive refreshing interferes with the
> ability to switch terminals (TTY1, TTY2, etc.), because I see a
> "single frame blink" of the terminal and then gets overridden with
> the 3-line error message again. Same holds for switch to GUI DISPLAY
> terminal. Display does not have sufficient time to refresh and ends
> up with variations of extra long DPMS-off state before returning to
> the error-message-screen.
> Sometimes I am able to "interrupt" this excessive run-away behavior by keeping e.g. CTRL+ALT+1 pressed such that it will try to switch TTY at "key-repeat"-speed.
> 
> 2.) error message regarding eMMC/SDIO issues. Similar to (1.) issue
>   with failing to initialize eMMC and consequently I lose access to
>   my persistent storage.
> 
> 3.) issue also occurs when entering sleep from prolonged idleness
>   with lid already closed (DPMS off + locked) as starting-point.
> 
> [x.] Not sure if relevant at all: "crashes with wifi firmware". A
> buggy early firmware version for a broadcom wifi chip would
> occasionally crash. However, in particular around the suspend
> process there would be issues.
> I don't want to attribute this to schedutil, but maybe schedutil's
> behavior significantly increases the chance of this happening.
> 
> NOTE: I tried switching back to 'ondemand' scheduler after first
> issues had occurred with 'schedutil' active. However, reverting to
> 'ondemand' did not resolve the outstanding issues at that point.
> 
> ## Kabylake-laptop
> 
> - Distro-kernel (i.e. not vanilla)
> - `intel_pstate=passive` kernel parameter
> - `schedutil` as cpufreq scaling governor (udev-rule)
> 
> 1.) Trigger suspend on the laptop. System goes into suspend state as
>   expected. Then (in about 2 seconds) fans start spinning and
>   apparently excessive heat is being produced. (This was reported
>   somewhere already as being an issue with PCH temp being too high,
>   w.r.t. `S0ix` `intel_pch_thermal`.) However, I suspect that
>   high-temperature may be caused through `schedutil`, because the
>   laptop did not run any intensive tasks of any kind. (idling)
> Upon resuming, the laptop has lost significant amounts of battery,
> corresponding with the excessive CPU usage and cooling fans.
> 
> ## Haswell laptop
> 
> 1.) Not much to say: enter suspend, find out later that it did not
>   truly enter suspend, but was kept back and is rather active.
> 
> ## Without `schedutil`
> 
> In all cases, when taking schedutil out of the loop, these issues
> disappear. In case of the Kabylake laptop, I do set
> intel_pstate=active.

Just to clarify here, from what I understand about the active/passive
parts of intel-pstate driver, if you set intel_pstate=active then none
of the cpufreq governor's will be used. This enables the setpolicy()
callback of the driver, which will decide how the frequency changes
later on. The cpufreq governors, ondemand or schedutil, are only in
play if intel_pstate=passive.

> All have `ondemand` governor. Suspend/resume
> works, repeated many times over. Even if some errors are still
> shown, they no longer pose a problem. The excessive screen blinking
> behavior is not exhibited, for example.
> 
> ## Other things I have tried
> 
> As mentioned before, I have tried looking for other bugs. However,
> the issues seem to be too persistent. I have tried removing all
> external devices, tried using different USB ports, tried different
> suspend/resume settings in `/etc/systemd/sleep.conf`. Tried using
> 'tlp' package for additional power-management tricks. Tried
> switching governors, ... and probably more.
> 
> I realize this bug report is far from perfect. I'm afraid I cannot
> make the report much more exact/clear. I am happy to answer any
> questions you may have. I will keep an eye out for further
> peculiarities.

Also I am not yet sure it is related to cpufreq right now.

-- 
viresh

Re: 'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

[Danny van Heumen]

------- Original Message -------
On Wednesday, May 25th, 2022 at 07:28, Viresh Kumar <viresh.kumar@linaro.org> wrote:

> [..]
>
> On 21-05-22, 22:13, Danny van Heumen wrote:
>
> > Hi all,
> >
> > This is my first report directly to linux kernel mailing lists. I'll
> > do my best, but I'll invariably make some mistakes.
>
>
> That's fine. Just two things you need to take care of normally, keep
> the line length to 100 columns (as I have done now) and don't do
> top-posting on replies. That's all :)

Okay. Good to know.

>
> > This bug report
> > is based on "educated" guessing, some insight and intuition, so I
> > will do my best to explain all the curiosities and why I blame
> > `schedutil`. To clarify, I seems behavior of schedutil triggers
> > issues elsewhere.
> >
> > I write the message not to say "this is the problem, fix it!" but
> > rather, I hope to check with you if you recognize potential things
> > to look for, if the reports make any sense.
> >
> > ## short summary
> >
> > On multiple computers, with kernel builds from distros and also
> > vanilla custom (minimal) config builds by myself, I have had a
> > number of issues that all center around suspend/resume with
> > 'schedutil' scaling governor active. Some issues seem to be
> > explained by other causes/bug-reports, but I report this here
> > because `schedutil` is a common denominator.
> >
> > Characteristics:
> > - Kabylake, upon suspend, problems with going into suspend, with
> > "runaway processor" behavior starting to run at full power causing
> > excessive build-up of heat. (intel_pstate=passive) (more details
> > follow ...)
>
>
> If this is actually related to cpufreq subsystem, then maybe the CPU
> going down was programmed at a higher frequency (because there was
> work to do during suspend) while going offline. I am not sure how it
> works on Intel, but on ARM we have something like a
> "suspend-frequency", which is programmed right before we suspend to
> make sure not to consume a lot of power. Maybe we can do something
> like this in intel-pstate driver, not sure though.

To clarify, I am trying to figure out/debug why schedutil *seems to
be* closely involved in a number of issues regarding suspend/resume.

In particular, on the Pinebook Pro, with a custom compiled kernel and
the only change being the 'schedutil' governor, it does not seem right
to have often (not always) suspend-resume issues, compared to 0 so far
with other governors.

> > - Haswell laptop, upon suspend, sometimes does not suspend, instead
> > screen off but then returned to active.
> > - Pinebook Pro: upon resume from suspend (s2idle, I'm fairly
> > certain): problems getting "display panel up in time", "eMMC/SDIO
> > failing to initialize", screen-flickering from excessive refreshing
> > many times the blinking rate. (more details follow ...)
> >
> > These issues start to happen upon suspend. The issues do not happen
> > when `schedutil` is not involved.
> >
> > My interpretation: somewhere just before, or just after entering
> > suspend-state/initiating restore, `schedutil` messes up something
> > w.r.t. timings: timings/delays/repeats happen at many times the
> > intended speed.
> >
> > ## `schedutil` before suspend: all good
> >
> > I have been using `schedutil` for many months. Upon normal
> > operations there never seem to be any issues, or unexpected events.
> > What I describe all happens centered around the suspension-process,
> > and sometimes as consequence afterwards.
> >
> > ## Pinebook Pro issues
> >
> > - original Debian kernel,
> > - custom-built kernel with very minimal config: versions
> > 5.15.{38,39,41} 5.17.9
> > - `schedutil` cpufreq scaling governor
> > - Debian bullseye (original, no third-party kernels), up-to-date
> > install
> > - no tlp, manual suspend procedure (either button- or lid-triggered)
> >
> > The Pinebook Pro does not exhibit issues (AFAICT) upon entering
> > suspend. However, resume will fail often, but not always. I have
> > seen different symptoms exhibited:
> >
> > 1.) 3-line error message regarding analogix_dp_resume about
> > rockchip-module not succeeding (through analogix) to re-initialize
> > the display panel in time. (It shows on that exact display panel,
> > so clearly it worked.)
> > Those 3 lines of error message, are being refreshed at many times
> > the necessary rate. The excessive refreshing interferes with the
> > ability to switch terminals (TTY1, TTY2, etc.), because I see a
> > "single frame blink" of the terminal and then gets overridden with
> > the 3-line error message again. Same holds for switch to GUI DISPLAY
> > terminal. Display does not have sufficient time to refresh and ends
> > up with variations of extra long DPMS-off state before returning to
> > the error-message-screen.
> > Sometimes I am able to "interrupt" this excessive run-away behavior
> > by keeping e.g. CTRL+ALT+1 pressed such that it will try to switch
> > TTY at "key-repeat"-speed.
> >
> > 2.) error message regarding eMMC/SDIO issues. Similar to (1.) issue
> > with failing to initialize eMMC and consequently I lose access to
> > my persistent storage.
> >
> > 3.) issue also occurs when entering sleep from prolonged idleness
> > with lid already closed (DPMS off + locked) as starting-point.
> >
> > [x.] Not sure if relevant at all: "crashes with wifi firmware". A
> > buggy early firmware version for a broadcom wifi chip would
> > occasionally crash. However, in particular around the suspend
> > process there would be issues.
> > I don't want to attribute this to schedutil, but maybe schedutil's
> > behavior significantly increases the chance of this happening.
> >
> > NOTE: I tried switching back to 'ondemand' scheduler after first
> > issues had occurred with 'schedutil' active. However, reverting to
> > 'ondemand' did not resolve the outstanding issues at that point.
> >
> > ## Kabylake-laptop
> >
> > - Distro-kernel (i.e. not vanilla)
> > - `intel_pstate=passive` kernel parameter
> > - `schedutil` as cpufreq scaling governor (udev-rule)
> >
> > 1.) Trigger suspend on the laptop. System goes into suspend state as
> > expected. Then (in about 2 seconds) fans start spinning and
> > apparently excessive heat is being produced. (This was reported
> > somewhere already as being an issue with PCH temp being too high,
> > w.r.t. `S0ix` `intel_pch_thermal`.) However, I suspect that
> > high-temperature may be caused through `schedutil`, because the
> > laptop did not run any intensive tasks of any kind. (idling)
> > Upon resuming, the laptop has lost significant amounts of battery,
> > corresponding with the excessive CPU usage and cooling fans.
> >
> > ## Haswell laptop
> >
> > 1.) Not much to say: enter suspend, find out later that it did not
> > truly enter suspend, but was kept back and is rather active.
> >
> > ## Without `schedutil`
> >
> > In all cases, when taking schedutil out of the loop, these issues
> > disappear. In case of the Kabylake laptop, I do set
> > intel_pstate=active.
>
>
> Just to clarify here, from what I understand about the active/passive
> parts of intel-pstate driver, if you set intel_pstate=active then none
> of the cpufreq governor's will be used. This enables the setpolicy()
> callback of the driver, which will decide how the frequency changes
> later on. The cpufreq governors, ondemand or schedutil, are only in
> play if intel_pstate=passive.

So the fact that 'schedutil' is not available but the other governors
are, is actually meaningless?

Does that mean that setting 'powersave' vs 'performance' is meaningless
too? If not, does that mean that the meaning changes (as in "determined
by") intel_pstate?

>
> > All have `ondemand` governor. Suspend/resume
> > works, repeated many times over. Even if some errors are still
> > shown, they no longer pose a problem. The excessive screen blinking
> > behavior is not exhibited, for example.
> >
> > ## Other things I have tried
> >
> > As mentioned before, I have tried looking for other bugs. However,
> > the issues seem to be too persistent. I have tried removing all
> > external devices, tried using different USB ports, tried different
> > suspend/resume settings in `/etc/systemd/sleep.conf`. Tried using
> > 'tlp' package for additional power-management tricks. Tried
> > switching governors, ... and probably more.
> >
> > I realize this bug report is far from perfect. I'm afraid I cannot
> > make the report much more exact/clear. I am happy to answer any
> > questions you may have. I will keep an eye out for further
> > peculiarities.
>
>
> Also I am not yet sure it is related to cpufreq right now.

I am trying to figure out whether or not some problems are caused by
'schedutil' vs it just being present. I guessed from the MAINTAINERS
index that you are probably the maintainers with knowledge on
schedutil. If not, do you know where I should look then?

To clarify, I interpreted 'cpufreq' as in:
/sys/devices/system/cpu/cpufreq/policy*/scaling_governor

I wonder if I am taking a wrong approach in tackling these issues,
so if you have recommendations, please let me know.

> [..]

Regards,
Danny

Re: 'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

[Viresh Kumar]

On 25-05-22, 13:39, Danny van Heumen wrote:
> On Wednesday, May 25th, 2022 at 07:28, Viresh Kumar <viresh.kumar@linaro.org> wrote:
> > On 21-05-22, 22:13, Danny van Heumen wrote:
> > > In all cases, when taking schedutil out of the loop, these issues
> > > disappear. In case of the Kabylake laptop, I do set
> > > intel_pstate=active.
> >
> >
> > Just to clarify here, from what I understand about the active/passive
> > parts of intel-pstate driver, if you set intel_pstate=active then none
> > of the cpufreq governor's will be used. This enables the setpolicy()
> > callback of the driver, which will decide how the frequency changes
> > later on. The cpufreq governors, ondemand or schedutil, are only in
> > play if intel_pstate=passive.
> 
> So the fact that 'schedutil' is not available but the other governors
> are, is actually meaningless?

Cpufreq core in the kernel has two separate entities: drivers and
governors. Drivers mostly decide how frequency is read or updated on
the hardware and governors decide on the policy, i.e. what frequency
to go to and when.

When intel-pstate is set to "active", the governor algorithms from
cpufreq core are take out, i.e. files like
drivers/cpufreq/cpufreq_ondemand.c or cpufreq_conservative.c or
sched/kernel/cpufreq_schedutil.c. Instead the driver, along with help
from the hardware, decides the next frequency by itself. In this case
we have two policies available (these are still called as governors in
userspace), powersave and performance. These two tell how aggressive
we need to be, but again the governor algorithms are gone.

> Does that mean that setting 'powersave' vs 'performance' is meaningless
> too?

No.

> If not, does that mean that the meaning changes (as in "determined
> by") intel_pstate?

What I wanted to say earlier was that if you want to pin point it to
schedutil (I know you are just trying to find it out yourself as
well), then you must always keep intel_pstate="passive" and then test
between ondemand and schedutil and see if problem happens or not. That
way we can see if it is one of the governors or both, when the problem
happens.

> > Also I am not yet sure it is related to cpufreq right now.
> 
> I am trying to figure out whether or not some problems are caused by
> 'schedutil' vs it just being present. I guessed from the MAINTAINERS
> index that you are probably the maintainers with knowledge on
> schedutil. If not, do you know where I should look then?

That's right. Me, Rafael and Peter look at the schedutil governor
normally. You have the mail to right people.

> To clarify, I interpreted 'cpufreq' as in:
> /sys/devices/system/cpu/cpufreq/policy*/scaling_governor

Right.

> I wonder if I am taking a wrong approach in tackling these issues,
> so if you have recommendations, please let me know.

I would also try intel_pstate=passive with powersave and performance.
The heating issue, if related to cpufreq leaving CPU at higher freq,
should also occur with performance (which keeps CPUs at highest freq).

-- 
viresh

Re: 'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

[Danny van Heumen]

------- Original Message -------
On Thursday, May 26th, 2022 at 05:26, Viresh Kumar <viresh.kumar@linaro.org> wrote:


> On 25-05-22, 13:39, Danny van Heumen wrote:
>
> > On Wednesday, May 25th, 2022 at 07:28, Viresh Kumar viresh.kumar@linaro.org wrote:
> >
> > [..]
> >
> > So the fact that 'schedutil' is not available but the other governors
> > are, is actually meaningless?
>
>
> Cpufreq core in the kernel has two separate entities: drivers and
> governors. Drivers mostly decide how frequency is read or updated on
> the hardware and governors decide on the policy, i.e. what frequency
> to go to and when.

Understood. No surprises here.

> When intel-pstate is set to "active", the governor algorithms from
> cpufreq core are take out, i.e. files like
> drivers/cpufreq/cpufreq_ondemand.c or cpufreq_conservative.c or
> sched/kernel/cpufreq_schedutil.c. Instead the driver, along with help
> from the hardware, decides the next frequency by itself. In this case
> we have two policies available (these are still called as governors in
> userspace), powersave and performance. These two tell how aggressive
> we need to be, but again the governor algorithms are gone.

Right, I see now that the set of governors is different.

> > Does that mean that setting 'powersave' vs 'performance' is meaningless
> > too?
>
>
> No.

This is clear, also by your explanation above.

> > If not, does that mean that the meaning changes (as in "determined
> > by") intel_pstate?
>
>
> What I wanted to say earlier was that if you want to pin point it to
> schedutil (I know you are just trying to find it out yourself as
> well), then you must always keep intel_pstate="passive" and then test
> between ondemand and schedutil and see if problem happens or not. That
> way we can see if it is one of the governors or both, when the problem
> happens.

Right. Thas already was my intention. I have already done more tests. I
notice now that this seems to be a pattern such that if it occurs once,
it keeps on occurring.

I see patterns such as below, where first thermal is before suspend,
second thermal is immediately after waking up. Note that apparently
some intel driver logic prevents full suspend with temp > 50 degrees.
I have very similar behavior to that case, but with temperature confirmed
below 50 degrees.

Below is a quick test of about 3 minutes time.

```console
> sensors pch_skylake-virtual-0; echo 'mem' | sudo tee /sys/power/state; \
sensors pch_skylake-virtual-0

pch_skylake-virtual-0
Adapter: Virtual device
temp1:        +47.5°C

mem

pch_skylake-virtual-0
Adapter: Virtual device
temp1:        +57.5°C
```

```dmesg
intel_pch_thermal 0000:00:14.2: CPU-PCH is cool [47C], continue to suspend
pcieport 0000:00:1c.5: Intel SPT PCH root port ACS workaround enabled
ACPI: EC: interrupt blocked
nvme nvme0: I/O 1015 QID 2 timeout, aborting
nvme nvme0: I/O 22 QID 0 timeout, completion polled
nvme nvme0: Abort status: 0x0
ACPI: EC: interrupt unblocked
pcieport 0000:00:1c.5: Intel SPT PCH root port ACS workaround enabled
pcieport 0000:00:1c.4: Intel SPT PCH root port ACS workaround enabled
pcieport 0000:00:1c.0: Intel SPT PCH root port ACS workaround enabled
pcieport 0000:00:1d.0: Intel SPT PCH root port ACS workaround enabled
usb 1-3: reset full-speed USB device number 2 using xhci_hcd
OOM killer enabled.
Restarting tasks ...
mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound \
0000:00:02.0 (ops i915_hdcp_component_ops [i915])
done.
systemd[1]: systemd-journald.service: Main process exited, code=killed, \
status=6/ABRT
systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
systemd[1]: systemd-journald.service: Consumed 1.844s CPU time.
systemd[1]: systemd-journald.service: Scheduled restart job, restart counter \
is at 1.
systemd[1]: Stopping Flush Journal to Persistent Storage...
systemd[1]: Stopped target Bluetooth.
systemd[1]: Started Load/Save RF Kill Switch Status.
PM: suspend exit
```

This is just to illustrate some tries. I will continue to test, to
figure out if I can either trigger the circumstances without
'schedutil' (which clears my suspicion) or otherwise try to make it
more deterministic, reproducible.

The nvme errors in dmesg are new to me. I have not seen those before.
So I will also keep an eye out for those.


> > > Also I am not yet sure it is related to cpufreq right now.
> >
> > I am trying to figure out whether or not some problems are caused by
> > 'schedutil' vs it just being present. I guessed from the MAINTAINERS
> > index that you are probably the maintainers with knowledge on
> > schedutil. If not, do you know where I should look then?
>
>
> That's right. Me, Rafael and Peter look at the schedutil governor
> normally. You have the mail to right people.
>
> [..]
>
> > I wonder if I am taking a wrong approach in tackling these issues,
> > so if you have recommendations, please let me know.
>
>
> I would also try intel_pstate=passive with powersave and performance.
> The heating issue, if related to cpufreq leaving CPU at higher freq,
> should also occur with performance (which keeps CPUs at highest freq).

I described the approach above. I will try 'performance' and/or
'powersave' too. It does seem like temperature is not the driving
factor here. (As temperature is low enough even intel_pch_thermal.)
Maybe it does make reproduction easier.

>
> --
> viresh

Re: 'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

[Danny van Heumen]

Hi,

------- Original Message -------
On Thursday, May 26th, 2022 at 17:01, Danny van Heumen <danny@dannyvanheumen.nl> wrote:

> [..]
>
> Right. Thas already was my intention. I have already done more tests. I
> notice now that this seems to be a pattern such that if it occurs once,
> it keeps on occurring.
>
> I see patterns such as below, where first thermal is before suspend,
> second thermal is immediately after waking up. Note that apparently
> some intel driver logic prevents full suspend with temp > 50 degrees.
>
> I have very similar behavior to that case, but with temperature confirmed
> below 50 degrees.
>
> Below is a quick test of about 3 minutes time.
>
> ```console
>
> > sensors pch_skylake-virtual-0; echo 'mem' | sudo tee /sys/power/state; \
>
> sensors pch_skylake-virtual-0
>
> pch_skylake-virtual-0
> Adapter: Virtual device
> temp1: +47.5°C
>
> mem
>
> pch_skylake-virtual-0
> Adapter: Virtual device
> temp1: +57.5°C
> dmesg
> intel_pch_thermal 0000:00:14.2: CPU-PCH is cool [47C], continue to suspend
> pcieport 0000:00:1c.5: Intel SPT PCH root port ACS workaround enabled
> ACPI: EC: interrupt blocked
> nvme nvme0: I/O 1015 QID 2 timeout, aborting
> nvme nvme0: I/O 22 QID 0 timeout, completion polled
> nvme nvme0: Abort status: 0x0
> ACPI: EC: interrupt unblocked
> pcieport 0000:00:1c.5: Intel SPT PCH root port ACS workaround enabled
> pcieport 0000:00:1c.4: Intel SPT PCH root port ACS workaround enabled
> pcieport 0000:00:1c.0: Intel SPT PCH root port ACS workaround enabled
> pcieport 0000:00:1d.0: Intel SPT PCH root port ACS workaround enabled
> usb 1-3: reset full-speed USB device number 2 using xhci_hcd
> OOM killer enabled.
> Restarting tasks ...
> mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound \
> 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
> done.
> systemd[1]: systemd-journald.service: Main process exited, code=killed, \
> status=6/ABRT
> systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
> systemd[1]: systemd-journald.service: Consumed 1.844s CPU time.
> systemd[1]: systemd-journald.service: Scheduled restart job, restart counter \
> is at 1.
> systemd[1]: Stopping Flush Journal to Persistent Storage...
> systemd[1]: Stopped target Bluetooth.
> systemd[1]: Started Load/Save RF Kill Switch Status.
> PM: suspend exit
> ```
>
> This is just to illustrate some tries. I will continue to test, to
> figure out if I can either trigger the circumstances without
> 'schedutil' (which clears my suspicion) or otherwise try to make it
> more deterministic, reproducible.
>
> The nvme errors in dmesg are new to me. I have not seen those before.
> So I will also keep an eye out for those.

Today I stumbled into the issue with 'ondemand' governor enabled from
boot. So schedutil does not need to be or have been active for
this issue to emerge. I still have many questions left, but they are
not schedutil related.

The aarch64-based Pinebook Pro issues are not explained yet, but I will
investigate further and possibly follow up later.

> [..]
>
> > --
> > viresh

Re: 'schedutil' (possibly) aberrant behavior surrounding suspend/resume process (timing/delay/run-away CPU issues)

[Viresh Kumar]

On 26-05-22, 23:35, Danny van Heumen wrote:
> Today I stumbled into the issue with 'ondemand' governor enabled from
> boot. So schedutil does not need to be or have been active for
> this issue to emerge. I still have many questions left, but they are
> not schedutil related.
> 
> The aarch64-based Pinebook Pro issues are not explained yet, but I will
> investigate further and possibly follow up later.

Good. I don't think this is cpufreq related at all. Something else is
going crazy. For simplicity you can set governor to performance with
intel_pstate=passive. That will almost remove cpufreq from the
equation.

-- 
viresh