Help me help you debug what seems to be an EC resume issue

Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

I have a 2019-era HP Spectre X360 13". I generally like it, but it's plagued
with "Modern Suspend" (s2idle) instead of a genuine S3, and as such I'm doing
whatever I (and you and Linux' PM) can to keep the power consumption down during
"suspend" (for now, "rfkill block all" before suspending, and removing the
XHC[I] as a wakeup source via "/proc/acpi/wakeup" during boot).

I bought one of those USB-C inline power meters and I can tell when the system
is burning more power than "normal" - on a full-charged battery during s2idle
it draws ~30-50 mA @ 20v, and if I see more than that I know something is amiss,
even though the laptop (and dmesg) think it's fully suspended.

I'm running the tip of Linus' tree as of now (b60fe990c6b07) and it's got
your latest PM/s2idle fixes in it. Before these commits, I used to set
"acpi.ec_no_wakeup=1" because the orientation sensor (at least, and probably
other things) would wake up the laptop (then immediately suspend), which
I'm sure was using up battery while I'm just walking around.

I've turned off "ec_no_wakeup" for testing and the good news is the orientation
sensor doesn't cause the laptop to draw more power when shaking it.

However:

- Randomly, it'll draw 250-300 mA when suspended, vs. the 30-50 mA
- Randomly, if left suspended, nothing other than a hard power off will get
  it back (and I can't be sure, but I think current consumption can be normal
  when it suspends, but this seems to only happen if I've unplugged the
  charger after suspending (so no power meter)).

I have pstore set up to catch BUG_*/oopses and save them away on boots, but
nothing comes thru, so I'm guessing it's just hanging somewhere either on
suspend or resume.

What can I do to diagnose where it's getting hung up? It appears my normal
"ec_no_wakeup=1" case has no such issues, but I wouldn't mind being able to
resume via the lid switch again.

Thanks,

	-Kenny

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley

Re: Help me help you debug what seems to be an EC resume issue

[Rafael J. Wysocki]

On Thu, Sep 19, 2019 at 3:24 AM Kenneth R. Crudup <kenny@panix.com> wrote:
>
>
> I have a 2019-era HP Spectre X360 13". I generally like it, but it's plagued
> with "Modern Suspend" (s2idle) instead of a genuine S3, and as such I'm doing
> whatever I (and you and Linux' PM) can to keep the power consumption down during
> "suspend" (for now, "rfkill block all" before suspending, and removing the
> XHC[I] as a wakeup source via "/proc/acpi/wakeup" during boot).

You may still be able to use S3 on this machine if that's supported by
the platform firmware.

If reading from /sys/power/mem_sleep returns something like "[s2idle] deep" do

# echo deep > /sys/power/mem_sleep

(as root) and then it should change to "s2idle [deep]".  In that
configuration, try to suspend and see if you can wake up etc and if it
works reliably, you may use it going forward (in which case you'll
probably want to add mem_sleep_default=deep to the kernel command line
persistently).

> I bought one of those USB-C inline power meters and I can tell when the system
> is burning more power than "normal" - on a full-charged battery during s2idle
> it draws ~30-50 mA @ 20v, and if I see more than that I know something is amiss,
> even though the laptop (and dmesg) think it's fully suspended.
>
> I'm running the tip of Linus' tree as of now (b60fe990c6b07) and it's got
> your latest PM/s2idle fixes in it. Before these commits, I used to set
> "acpi.ec_no_wakeup=1" because the orientation sensor (at least, and probably
> other things) would wake up the laptop (then immediately suspend), which
> I'm sure was using up battery while I'm just walking around.
>
> I've turned off "ec_no_wakeup" for testing and the good news is the orientation
> sensor doesn't cause the laptop to draw more power when shaking it.
>
> However:
>
> - Randomly, it'll draw 250-300 mA when suspended, vs. the 30-50 mA
> - Randomly, if left suspended, nothing other than a hard power off will get
>   it back (and I can't be sure, but I think current consumption can be normal
>   when it suspends, but this seems to only happen if I've unplugged the
>   charger after suspending (so no power meter)).
>
> I have pstore set up to catch BUG_*/oopses and save them away on boots, but
> nothing comes thru, so I'm guessing it's just hanging somewhere either on
> suspend or resume.
>
> What can I do to diagnose where it's getting hung up? It appears my normal
> "ec_no_wakeup=1" case has no such issues, but I wouldn't mind being able to
> resume via the lid switch again.

I would recommend to try 5.4-rc1 when it's out to see if the problems
above are still there.

Cheers,
Rafael

Re: Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

On Thu, 19 Sep 2019, Rafael J. Wysocki wrote:

> You may still be able to use S3 on this machine if that's supported by
> the platform firmware.

It's not; I tried "deep" two days after getting it.

> I would recommend to try 5.4-rc1 when it's out to see if the problems
> above are still there.

Well, I'm running Linus' master tip right now and pull and run the latest
bleeding-edge kernel daily; does this mean you have further patches coming
down the line later?

	-Kenny

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley

Re: Help me help you debug what seems to be an EC resume issue

[Rafael J. Wysocki]

On Thu, Sep 19, 2019 at 1:46 PM Kenneth R. Crudup <kenny@panix.com> wrote:
>
>
> On Thu, 19 Sep 2019, Rafael J. Wysocki wrote:
>
> > You may still be able to use S3 on this machine if that's supported by
> > the platform firmware.
>
> It's not; I tried "deep" two days after getting it.
>
> > I would recommend to try 5.4-rc1 when it's out to see if the problems
> > above are still there.
>
> Well, I'm running Linus' master tip right now and pull and run the latest
> bleeding-edge kernel daily;

Specifically, the material I had in mind was merged on Tuesday, so I
guess you're running it by now.

If so, you can try writing 1 into
/sys/module/acpi/parameters/sleep_no_lps0 and see if that makes any
difference (without using ec_no_wakeup).

> does this mean you have further patches coming
> down the line later?

There may be a couple, but not related to the EC.

Re: Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

On Thu, 19 Sep 2019, Rafael J. Wysocki wrote:

> If so, you can try writing 1 into
> /sys/module/acpi/parameters/sleep_no_lps0 and see if that makes any
> difference (without using ec_no_wakeup).

I'll try that soon; in a classic case of "it never makes that sound at the
mechanic's shop" I've done some 6+ sleep cycles since E-mailing you (and
with "ec_no_wakeup" off), and all have come back OK. Increased power drain
during random suspend cycles is still an issue, though, but that may be
unrelated.

(I did remove the MEI driver from the kernel, as I was seeing error returns
during probing on the resume end with "no_console_suspend/initcall_debug" on,
but I'd be surprised if that were an issue here.  I have no need for the MEI
at all, but it's been my experience that loading drivers for devices seem to
reduce the power consumption in s2idle even if I don't (directly) use them
(i.e., I see higher s2idle power drain if I don't enable the "skl_uncore" driver).)

	-Kenny

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley

Re: Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

BTW, is there any way to determine which devices are still on (out of d3cold)
at the time the kernel relinquishes control to the BIOS/ACPI as the last thing
to happen at suspend?

... and would it make any difference? IOW, when the handoff above occurs,
can/do/will the BIOS/ACPI re-enable devices we've explicitly shut down?

I'd scribble all over "/proc/acpi/wakeup" turning off everything except the
power button if I thought it would make a difference (and I believe at one
time I actually may have) but I wonder if that means anything once the
machine takes over?

Thanks,

	-Kenny, just trying to get more than a day's battery life on s2idle

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley

Re: Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

> BTW, is there any way to determine which devices are still on (out of d3cold)
> at the time the kernel relinquishes control to the BIOS/ACPI as the last thing
> to happen at suspend?

So I turned on "DEBUG" at the top of ".../include/linux/acpi.h"
I have no idea if I'm reading this correctly, but does this mean that there are
devices that will always(?) stay in d3hot during s2idle?

If this is the case, is there anything I can/should do about it?

----
[  +0.000249] ACPI: \_SB_.PEPD: _DSM function mask: 0x7f
[  +0.007429] ACPI: \_SB_.PEPD: _DSM function 1 eval successful
[  +0.000003] ACPI: \_SB_.PEPD: LPI: constraints list begin:
[  +0.000002] ACPI: \_SB_.PEPD: index:0 Name:\_SB.PR00
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000002] ACPI: \_SB_.PEPD: index:1 Name:\_SB.PR01
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000002] ACPI: \_SB_.PEPD: index:2 Name:\_SB.PR02
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000002] ACPI: \_SB_.PEPD: index:3 Name:\_SB.PR03
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000002] ACPI: \_SB_.PEPD: index:4 Name:\_SB.PR04
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000002] ACPI: \_SB_.PEPD: index:5 Name:\_SB.PR05
[  +0.000001] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000003] ACPI: \_SB_.PEPD: index:6 Name:\_SB.PR06
[  +0.000001] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000002] ACPI: \_SB_.PEPD: index:7 Name:\_SB.PR07
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000003] ACPI: \_SB_.PEPD: index:8 Name:\_SB.PCI0.GFX0
[  +0.000001] ACPI: \_SB_.PEPD: uid:255 min_dstate:D3hot
[  +0.000004] ACPI: \_SB_.PEPD: index:12 Name:\_SB.PCI0.I2C0
[  +0.000001] ACPI: \_SB_.PEPD: uid:255 min_dstate:D3hot
[  +0.000003] ACPI: \_SB_.PEPD: index:13 Name:\_SB.PCI0.I2C1
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D3hot
[  +0.000002] ACPI: \_SB_.PEPD: index:14 Name:\_SB.PCI0.XHC
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D3hot
[  +0.000002] ACPI: \_SB_.PEPD: index:15 Name:\_SB.PCI0.HDAS
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000003] ACPI: \_SB_.PEPD: index:37 Name:\_SB.PCI0.RP13.PXSX
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000002] ACPI: \_SB_.PEPD: index:55 Name:\_SB.PCI0.CNVW
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D3hot
[  +0.000003] ACPI: \_SB_.PEPD: index:57 Name:\_SB.PCI0.HECI
[  +0.000002] ACPI: \_SB_.PEPD: uid:255 min_dstate:D0
[  +0.000001] ACPI: \_SB_.PEPD: LPI: constraints list end
----

	-Kenny

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley

Re: Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

This is after a resume. Maybe this is helpful, too?

----
[  +0.002487] intel_pmc_core INT33A1:00: CPU did not enter SLP_S0!!! (S0ix cnt=0)
[  +0.000005] intel_pmc_core INT33A1:00: SLP_S0_DBG: AUDIO_D3                           State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: OTG_D3                             State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: XHCI_D3                            State: Yes
[  +0.000002] intel_pmc_core INT33A1:00: SLP_S0_DBG: LPIO_D3                            State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: SDX_D3                             State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: SATA_D3                            State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: UFS0_D3                            State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: UFS1_D3                            State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: EMMC_D3                            State: Yes
[  +0.000005] intel_pmc_core INT33A1:00: SLP_S0_DBG: SDIO_PLL_OFF                       State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: USB2_PLL_OFF                       State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: AUDIO_PLL_OFF                      State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: OC_PLL_OFF                         State: Yes
[  +0.000002] intel_pmc_core INT33A1:00: SLP_S0_DBG: MAIN_PLL_OFF                       State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: XOSC_OFF                           State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: LPC_CLKS_GATED                     State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: PCIE_CLKREQS_IDLE                  State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: AUDIO_ROSC_OFF                     State: Yes
[  +0.000002] intel_pmc_core INT33A1:00: SLP_S0_DBG: HPET_XOSC_CLK_REQ                  State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: PMC_ROSC_SLOW_CLK                  State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: AON2_ROSC_GATED                    State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: CLKACKS_DEASSERTED                 State: No
[  +0.000004] intel_pmc_core INT33A1:00: SLP_S0_DBG: MPHY_CORE_GATED                    State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: CSME_GATED                         State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: USB2_SUS_GATED                     State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: DYN_FLEX_IO_IDLE                   State: Yes
[  +0.000002] intel_pmc_core INT33A1:00: SLP_S0_DBG: GBE_NO_LINK                        State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: THERM_SEN_DISABLED                 State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: PCIE_LOW_POWER                     State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: ISH_VNNAON_REQ_ACT                 State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: ISH_VNN_REQ_ACT                    State: No
[  +0.000002] intel_pmc_core INT33A1:00: SLP_S0_DBG: CNV_VNNAON_REQ_ACT                 State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: CNV_VNN_REQ_ACT                    State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: NPK_VNNON_REQ_ACT                  State: No
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: PMSYNC_STATE_IDLE                  State: Yes
[  +0.000001] intel_pmc_core INT33A1:00: SLP_S0_DBG: ALST_GT_THRES                      State: Yes
[  +0.000002] intel_pmc_core INT33A1:00: SLP_S0_DBG: PMC_ARC_PG_READY                   State: No
----

	-Kenny

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley

Re: Help me help you debug what seems to be an EC resume issue

[Rafael J. Wysocki]

On Thu, Sep 19, 2019 at 6:35 PM Kenneth R. Crudup <kenny@panix.com> wrote:
>
>
> > BTW, is there any way to determine which devices are still on (out of d3cold)
> > at the time the kernel relinquishes control to the BIOS/ACPI as the last thing
> > to happen at suspend?

In the suspend-to-idle case the BIOS doesn't get control at all.

I would recommend to read

https://01.org/blogs/qwang59/2018/how-achieve-s0ix-states-linux

at this point if you have not done it yet.

> So I turned on "DEBUG" at the top of ".../include/linux/acpi.h"
> I have no idea if I'm reading this correctly, but does this mean that there are
> devices that will always(?) stay in d3hot during s2idle?
>
> If this is the case, is there anything I can/should do about it?

Not really.  That's just debug output.

Re: Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

On Thu, 19 Sep 2019, Rafael J. Wysocki wrote:

> I would recommend to read
> https://01.org/blogs/qwang59/2018/how-achieve-s0ix-states-linux
> at this point if you have not done it yet.

Yeah, I have ... and I get this after a resume:

> intel_pmc_core INT33A1:00: CPU did not enter SLP_S0!!! (S0ix cnt=0)

... and this even after multiple suspend/resume cycles:

> /sys/kernel/debug/pmc_core/slp_s0_residency_usec:0

I did determine I'm going in to C10 though, but that's a "when powered on"
issue; I'm trying to maximize the time I get in s2idle- it's the only thing
infuriating about this laptop. Hibernate works, and I do that when I know
I won't be using it for a while, but going from a laptop that would give me
nearly a week in S3 to this one where I can lose ~20-35% overnight sucks and
I've been pulling at any straw trying to reduce s2idle power draw.

	-Kenny

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley

Re: Help me help you debug what seems to be an EC resume issue

[Kenneth R. Crudup]

Oh, and:

$  sudo egrep -r . /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/low_power_idle_system_residency_us:0
$

	-Kenny

-- 
Kenneth R. Crudup  Sr. SW Engineer, Scott County Consulting, Silicon Valley