* Re: Low Latency Tolerance preventing Intel Package from entering deep sleep states [not found] ` <1505028180.591737.1589564161284@mail.yahoo.com> @ 2020-05-15 21:41 ` Andy Shevchenko 2020-05-18 10:26 ` Rafael J. Wysocki 0 siblings, 1 reply; 7+ messages in thread From: Andy Shevchenko @ 2020-05-15 21:41 UTC (permalink / raw) To: larsh, Rafael J. Wysocki Cc: ibm-acpi-devel, platform-driver-x86, ACPI Devel Maling List +Cc: ACPI ML and Rafael On Fri, May 15, 2020 at 8:36 PM larsh@apache.org <larsh@apache.org> wrote: > > Hi. I hope this is the right forum to raise this... > > For a while I have noticed that my CPU (i9-9880H in a Lenovo X1 Extreme Gen2) never enters any sleep mode below pc2. > (Confirmed with powertop and /sys/kernel/debug/pmc_core/package_cstate_show) > > Interestingly the CPU *can* reachers deeper C states *after* a resume from sleep (either S0ix or S3, i.e. freeze or mem). > > This article finally pointed me in the right direction: https://01.org/blogs/qwang59/2020/linux-s0ix-troubleshooting > > Somehow SOUTHPORT_A is requesting a max latency of 1 us. > There are no external devices attached. > > This is before a resume: > > $ cat /sys/kernel/debug/pmc_core/ltr_show > SOUTHPORT_A LTR: RAW: 0x88018c01 Non-Snoop(ns): 1024 Snoop(ns): 32768 <------- > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SATA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > GIGABIT_ETHERNET LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > XHCI LTR: RAW: 0x13ff Non-Snoop(ns): 0 Snoop(ns): 0 > Reserved LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > ME LTR: RAW: 0x8000800 Non-Snoop(ns): 0 Snoop(ns): 0 > EVA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 > HD_AUDIO LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > CNV LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > LPSS LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > CAMERA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > ESPI LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SCC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > ISH LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > UFSX2 LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > EMMC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > WIGIG LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > CURRENT_PLATFORM LTR: RAW: 0x40201 Non-Snoop(ns): 0 Snoop(ns): 0 > AGGREGATED_SYSTEM LTR: RAW: 0x7fbfdfe Non-Snoop(ns): 0 Snoop(ns): 0 > > Notice the 1000ns max latency requirement for SOUTHPORT_A. > > Ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore subsequently allows the CPU to reach deep sleep states. > > After a resume it looks like suddenly SOUTHPORT_C is active and with a less tight latency requirement: > > $ cat /sys/kernel/debug/pmc_core/ltr_show > SOUTHPORT_A LTR: RAW: 0x8010c01 Non-Snoop(ns): 0 Snoop(ns): 0 <-------- > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SATA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > GIGABIT_ETHERNET LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > XHCI LTR: RAW: 0x13ff Non-Snoop(ns): 0 Snoop(ns): 0 > Reserved LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > ME LTR: RAW: 0x8000800 Non-Snoop(ns): 0 Snoop(ns): 0 > EVA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_C LTR: RAW: 0x88468846 Non-Snoop(ns): 71680 Snoop(ns): 71680 <--------- > HD_AUDIO LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > CNV LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > LPSS LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > CAMERA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > ESPI LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SCC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > ISH LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > UFSX2 LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > EMMC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > WIGIG LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > CURRENT_PLATFORM LTR: RAW: 0x40201 Non-Snoop(ns): 0 Snoop(ns): 0 > AGGREGATED_SYSTEM LTR: RAW: 0x904824 Non-Snoop(ns): 0 Snoop(ns): 0 > > Does anybody know what's going on or how to debug this further? > > As stated above, I was able to work around this problem by ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore. > There has to be a better way, and I'm sure I'm not the only one running into this. > > Thanks. > > -- Lars -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Low Latency Tolerance preventing Intel Package from entering deep sleep states 2020-05-15 21:41 ` Low Latency Tolerance preventing Intel Package from entering deep sleep states Andy Shevchenko @ 2020-05-18 10:26 ` Rafael J. Wysocki 2020-05-19 16:03 ` David E. Box 0 siblings, 1 reply; 7+ messages in thread From: Rafael J. Wysocki @ 2020-05-18 10:26 UTC (permalink / raw) To: Andy Shevchenko Cc: larsh, ibm-acpi-devel, platform-driver-x86, ACPI Devel Maling List, David Box On Friday, May 15, 2020 11:41:16 PM CEST Andy Shevchenko wrote: > +Cc: ACPI ML and Rafael +Cc: David Box > On Fri, May 15, 2020 at 8:36 PM larsh@apache.org <larsh@apache.org> wrote: > > > > Hi. I hope this is the right forum to raise this... > > > > For a while I have noticed that my CPU (i9-9880H in a Lenovo X1 Extreme Gen2) never enters any sleep mode below pc2. > > (Confirmed with powertop and /sys/kernel/debug/pmc_core/package_cstate_show) > > > > Interestingly the CPU *can* reachers deeper C states *after* a resume from sleep (either S0ix or S3, i.e. freeze or mem). > > > > This article finally pointed me in the right direction: https://01.org/blogs/qwang59/2020/linux-s0ix-troubleshooting > > > > Somehow SOUTHPORT_A is requesting a max latency of 1 us. > > There are no external devices attached. > > > > This is before a resume: > > > > $ cat /sys/kernel/debug/pmc_core/ltr_show > > SOUTHPORT_A LTR: RAW: 0x88018c01 Non-Snoop(ns): 1024 Snoop(ns): 32768 <------- > > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SATA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > GIGABIT_ETHERNET LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > XHCI LTR: RAW: 0x13ff Non-Snoop(ns): 0 Snoop(ns): 0 > > Reserved LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > ME LTR: RAW: 0x8000800 Non-Snoop(ns): 0 Snoop(ns): 0 > > EVA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 > > HD_AUDIO LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > CNV LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > LPSS LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 > > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > CAMERA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > ESPI LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SCC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > ISH LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > UFSX2 LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > EMMC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > WIGIG LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > CURRENT_PLATFORM LTR: RAW: 0x40201 Non-Snoop(ns): 0 Snoop(ns): 0 > > AGGREGATED_SYSTEM LTR: RAW: 0x7fbfdfe Non-Snoop(ns): 0 Snoop(ns): 0 > > > > Notice the 1000ns max latency requirement for SOUTHPORT_A. > > > > Ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore subsequently allows the CPU to reach deep sleep states. > > > > After a resume it looks like suddenly SOUTHPORT_C is active and with a less tight latency requirement: > > > > $ cat /sys/kernel/debug/pmc_core/ltr_show > > SOUTHPORT_A LTR: RAW: 0x8010c01 Non-Snoop(ns): 0 Snoop(ns): 0 <-------- > > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SATA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > GIGABIT_ETHERNET LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > XHCI LTR: RAW: 0x13ff Non-Snoop(ns): 0 Snoop(ns): 0 > > Reserved LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > ME LTR: RAW: 0x8000800 Non-Snoop(ns): 0 Snoop(ns): 0 > > EVA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SOUTHPORT_C LTR: RAW: 0x88468846 Non-Snoop(ns): 71680 Snoop(ns): 71680 <--------- > > HD_AUDIO LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > CNV LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > LPSS LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 > > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > CAMERA LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > ESPI LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > SCC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > ISH LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > UFSX2 LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > EMMC LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > WIGIG LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > CURRENT_PLATFORM LTR: RAW: 0x40201 Non-Snoop(ns): 0 Snoop(ns): 0 > > AGGREGATED_SYSTEM LTR: RAW: 0x904824 Non-Snoop(ns): 0 Snoop(ns): 0 > > > > Does anybody know what's going on or how to debug this further? > > > > As stated above, I was able to work around this problem by ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore. > > There has to be a better way, and I'm sure I'm not the only one running into this. > > > > Thanks. > > > > -- Lars > > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Low Latency Tolerance preventing Intel Package from entering deep sleep states 2020-05-18 10:26 ` Rafael J. Wysocki @ 2020-05-19 16:03 ` David E. Box 2020-05-22 6:14 ` larsh 0 siblings, 1 reply; 7+ messages in thread From: David E. Box @ 2020-05-19 16:03 UTC (permalink / raw) To: Rafael J. Wysocki, Andy Shevchenko Cc: larsh, ibm-acpi-devel, platform-driver-x86, ACPI Devel Maling List > > > Does anybody know what's going on or how to debug this further? > > > As stated above, I was able to work around this problem by > > > ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore. > > > There has to be a better way, and I'm sure I'm not the only one > > > running into this. ltr_show shows the PMC's (Power Management Controller) view of SoC devices and busses. The SOUTHPORTs are the PCIe root ports on your system. When you run lspci they are the PCI bridges. Generally, the bridges are enumerated in the same order as the SOUTHPORTs, so SOUTHPORT_A is your first bridge and the device attached to it (shown in lspci -t) is the device that was blocking deeper PC states according to your debug. Determine what this device is on your system. If the ltr was low it's because that is what the device requested. You should first check that runtime pm is enabled for the device. To do this, check the control file in /sys/bus/pci/devices/<SSSS:BB:DD.F>/power, where SSSS:BB:DD.F is the enumeration of your device as shown in lspci. If it is 'on' then runtime pm is disabled. To enable it echo 'auto' into the file with root privileges. Enabling runtime pm should allow the driver to reduce functionality of the device when idle. This should lead to a larger latency request on the PCI bus which should be reflected in ltr_show. You can see if the device is actually runtime suspended and how much time it's been suspended (or active) by reading the associated files in the power folder. If this doesn't work, then it's possible that your device doesn't support runtime pm. This may be purposely for reliability reasons or the driver may just lack support. Check forums discussing issues with the device and look for possible options in the driver to force pm support (generally this will be centered around enabling ASPM). You can also download powertop to see the package c-state residencies more clearly as percentages of time. powertop also has a tunables tab that will show the status of runtime pm on all devices on the system and allow you to enable them individually. David ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Low Latency Tolerance preventing Intel Package from entering deep sleep states 2020-05-19 16:03 ` David E. Box @ 2020-05-22 6:14 ` larsh 2020-05-22 8:58 ` Andy Shevchenko 0 siblings, 1 reply; 7+ messages in thread From: larsh @ 2020-05-22 6:14 UTC (permalink / raw) To: Rafael J. Wysocki, Andy Shevchenko, David E. Box Cc: ibm-acpi-devel, platform-driver-x86, ACPI Devel Maling List Thanks David! With this I tracked down the SD Card Reader (Genesys Logic, Inc Device 9755) as the culprit. These are standard in many ThinkPads. The curious part is that resume from suspend (S3 or S0iX) also fixes the problem. Looks like the driver is not initializing correctly at boot time. Transcript: $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT SOUTHPORT_A LTR: RAW: 0x88018c01 Non-Snoop(ns): 1024 Snoop(ns): 32768 SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 SOUTHPORT_D LTR: RAW: 0x88aa88aa Non-Snoop(ns): 174080 Snoop(ns): 174080 SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 $ lspci -t -[0000:00]-+-00.0 +-01.0-[01]--+-00.0 | \-00.1 +-02.0 +-04.0 +-08.0 +-12.0 +-14.0 +-14.2 +-15.0 +-16.0 +-1c.0-[53]----00.0 +-1d.0-[02]----00.0 +-1d.6-[52]----00.0 +-1e.0 +-1f.0 +-1f.3 +-1f.4 +-1f.5 \-1f.6 $ lspci | grep 53 53:00.0 SD Host controller: Genesys Logic, Inc Device 9755 $ cat /sys/bus/pci/devices/0000\:53\:00.0/power/control auto $ echo 1 > /sys/bus/pci/devices/0000\:53\:00.0/remove 1 $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT SOUTHPORT_A LTR: RAW: 0x8010c01 Non-Snoop(ns): 0 Snoop(ns): 0 SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 Cheers. -- Lars On Tuesday, May 19, 2020, 9:03:53 AM PDT, David E. Box <david.e.box@linux.intel.com> wrote: > > > Does anybody know what's going on or how to debug this further? > > > As stated above, I was able to work around this problem by > > > ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore. > > > There has to be a better way, and I'm sure I'm not the only one > > > running into this. ltr_show shows the PMC's (Power Management Controller) view of SoC devices and busses. The SOUTHPORTs are the PCIe root ports on your system. When you run lspci they are the PCI bridges. Generally, the bridges are enumerated in the same order as the SOUTHPORTs, so SOUTHPORT_A is your first bridge and the device attached to it (shown in lspci -t) is the device that was blocking deeper PC states according to your debug. Determine what this device is on your system. If the ltr was low it's because that is what the device requested. You should first check that runtime pm is enabled for the device. To do this, check the control file in /sys/bus/pci/devices/<SSSS:BB:DD.F>/power, where SSSS:BB:DD.F is the enumeration of your device as shown in lspci. If it is 'on' then runtime pm is disabled. To enable it echo 'auto' into the file with root privileges. Enabling runtime pm should allow the driver to reduce functionality of the device when idle. This should lead to a larger latency request on the PCI bus which should be reflected in ltr_show. You can see if the device is actually runtime suspended and how much time it's been suspended (or active) by reading the associated files in the power folder. If this doesn't work, then it's possible that your device doesn't support runtime pm. This may be purposely for reliability reasons or the driver may just lack support. Check forums discussing issues with the device and look for possible options in the driver to force pm support (generally this will be centered around enabling ASPM). You can also download powertop to see the package c-state residencies more clearly as percentages of time. powertop also has a tunables tab that will show the status of runtime pm on all devices on the system and allow you to enable them individually. David ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Low Latency Tolerance preventing Intel Package from entering deep sleep states 2020-05-22 6:14 ` larsh @ 2020-05-22 8:58 ` Andy Shevchenko 2020-05-22 16:16 ` larsh 0 siblings, 1 reply; 7+ messages in thread From: Andy Shevchenko @ 2020-05-22 8:58 UTC (permalink / raw) To: larsh, Adrian Hunter Cc: Rafael J. Wysocki, David E. Box, ibm-acpi-devel, platform-driver-x86, ACPI Devel Maling List +Cc: Adrian On Fri, May 22, 2020 at 9:15 AM larsh@apache.org <larsh@apache.org> wrote: > > Thanks David! > > With this I tracked down the SD Card Reader (Genesys Logic, Inc Device 9755) as the culprit. > These are standard in many ThinkPads. > The curious part is that resume from suspend (S3 or S0iX) also fixes the problem. > Looks like the driver is not initializing correctly at boot time. > > Transcript: > > $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT > SOUTHPORT_A LTR: RAW: 0x88018c01 Non-Snoop(ns): 1024 Snoop(ns): 32768 > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_D LTR: RAW: 0x88aa88aa Non-Snoop(ns): 174080 Snoop(ns): 174080 > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > $ lspci -t > -[0000:00]-+-00.0 > +-01.0-[01]--+-00.0 > | \-00.1 > +-02.0 > +-04.0 > +-08.0 > +-12.0 > +-14.0 > +-14.2 > +-15.0 > +-16.0 > +-1c.0-[53]----00.0 > +-1d.0-[02]----00.0 > +-1d.6-[52]----00.0 > +-1e.0 > +-1f.0 > +-1f.3 > +-1f.4 > +-1f.5 > \-1f.6 > > $ lspci | grep 53 > 53:00.0 SD Host controller: Genesys Logic, Inc Device 9755 > > $ cat /sys/bus/pci/devices/0000\:53\:00.0/power/control > auto > > $ echo 1 > /sys/bus/pci/devices/0000\:53\:00.0/remove > 1 > > $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT > SOUTHPORT_A LTR: RAW: 0x8010c01 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > Cheers. > > -- Lars > > > > > > > > > On Tuesday, May 19, 2020, 9:03:53 AM PDT, David E. Box <david.e.box@linux.intel.com> wrote: > > > > > > > > > Does anybody know what's going on or how to debug this further? > > > > As stated above, I was able to work around this problem by > > > > ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore. > > > > There has to be a better way, and I'm sure I'm not the only one > > > > running into this. > > ltr_show shows the PMC's (Power Management Controller) view of SoC > devices and busses. The SOUTHPORTs are the PCIe root ports on your > system. When you run lspci they are the PCI bridges. Generally, the > bridges are enumerated in the same order as the SOUTHPORTs, so > SOUTHPORT_A is your first bridge and the device attached to it (shown > in lspci -t) is the device that was blocking deeper PC states according > to your debug. > > Determine what this device is on your system. If the ltr was low it's > because that is what the device requested. You should first check that > runtime pm is enabled for the device. To do this, check the control > file in /sys/bus/pci/devices/<SSSS:BB:DD.F>/power, where SSSS:BB:DD.F > is the enumeration of your device as shown in lspci. If it is 'on' then > runtime pm is disabled. To enable it echo 'auto' into the file with > root privileges. Enabling runtime pm should allow the driver to reduce > functionality of the device when idle. This should lead to a larger > latency request on the PCI bus which should be reflected in ltr_show. > You can see if the device is actually runtime suspended and how much > time it's been suspended (or active) by reading the associated files in > the power folder. > > If this doesn't work, then it's possible that your device doesn't > support runtime pm. This may be purposely for reliability reasons or > the driver may just lack support. Check forums discussing issues with > the device and look for possible options in the driver to force pm > support (generally this will be centered around enabling ASPM). > > You can also download powertop to see the package c-state residencies > more clearly as percentages of time. powertop also has a tunables tab > that will show the status of runtime pm on all devices on the system > and allow you to enable them individually. > > > David > -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Low Latency Tolerance preventing Intel Package from entering deep sleep states 2020-05-22 8:58 ` Andy Shevchenko @ 2020-05-22 16:16 ` larsh 2020-05-25 5:37 ` Adrian Hunter 0 siblings, 1 reply; 7+ messages in thread From: larsh @ 2020-05-22 16:16 UTC (permalink / raw) To: Adrian Hunter, Andy Shevchenko Cc: Rafael J. Wysocki, David E. Box, ibm-acpi-devel, platform-driver-x86, ACPI Devel Maling List Relevant logs: May 21 23:05:31 host kernel: sdhci-pci 0000:53:00.0: SDHCI controller found [17a0:9755] (rev 0) May 21 23:05:31 host kernel: sdhci-pci 0000:53:00.0: enabling device (0000 -> 0002) May 21 23:05:31 host kernel: mmc0: SDHCI controller on PCI [0000:53:00.0] using ADMA 64-bit On Friday, May 22, 2020, 1:59:08 AM PDT, Andy Shevchenko <andy.shevchenko@gmail.com> wrote: +Cc: Adrian On Fri, May 22, 2020 at 9:15 AM larsh@apache.org <larsh@apache.org> wrote: > > Thanks David! > > With this I tracked down the SD Card Reader (Genesys Logic, Inc Device 9755) as the culprit. > These are standard in many ThinkPads. > The curious part is that resume from suspend (S3 or S0iX) also fixes the problem. > Looks like the driver is not initializing correctly at boot time. > > Transcript: > > $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT > SOUTHPORT_A LTR: RAW: 0x88018c01 Non-Snoop(ns): 1024 Snoop(ns): 32768 > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_D LTR: RAW: 0x88aa88aa Non-Snoop(ns): 174080 Snoop(ns): 174080 > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > $ lspci -t > -[0000:00]-+-00.0 > +-01.0-[01]--+-00.0 > | \-00.1 > +-02.0 > +-04.0 > +-08.0 > +-12.0 > +-14.0 > +-14.2 > +-15.0 > +-16.0 > +-1c.0-[53]----00.0 > +-1d.0-[02]----00.0 > +-1d.6-[52]----00.0 > +-1e.0 > +-1f.0 > +-1f.3 > +-1f.4 > +-1f.5 > \-1f.6 > > $ lspci | grep 53 > 53:00.0 SD Host controller: Genesys Logic, Inc Device 9755 > > $ cat /sys/bus/pci/devices/0000\:53\:00.0/power/control > auto > > $ echo 1 > /sys/bus/pci/devices/0000\:53\:00.0/remove > 1 > > $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT > SOUTHPORT_A LTR: RAW: 0x8010c01 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 > SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 > SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 > > Cheers. > > -- Lars > > > > > > > > > On Tuesday, May 19, 2020, 9:03:53 AM PDT, David E. Box <david.e.box@linux.intel.com> wrote: > > > > > > > > > Does anybody know what's going on or how to debug this further? > > > > As stated above, I was able to work around this problem by > > > > ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore. > > > > There has to be a better way, and I'm sure I'm not the only one > > > > running into this. > > ltr_show shows the PMC's (Power Management Controller) view of SoC > devices and busses. The SOUTHPORTs are the PCIe root ports on your > system. When you run lspci they are the PCI bridges. Generally, the > bridges are enumerated in the same order as the SOUTHPORTs, so > SOUTHPORT_A is your first bridge and the device attached to it (shown > in lspci -t) is the device that was blocking deeper PC states according > to your debug. > > Determine what this device is on your system. If the ltr was low it's > because that is what the device requested. You should first check that > runtime pm is enabled for the device. To do this, check the control > file in /sys/bus/pci/devices/<SSSS:BB:DD.F>/power, where SSSS:BB:DD.F > is the enumeration of your device as shown in lspci. If it is 'on' then > runtime pm is disabled. To enable it echo 'auto' into the file with > root privileges. Enabling runtime pm should allow the driver to reduce > functionality of the device when idle. This should lead to a larger > latency request on the PCI bus which should be reflected in ltr_show. > You can see if the device is actually runtime suspended and how much > time it's been suspended (or active) by reading the associated files in > the power folder. > > If this doesn't work, then it's possible that your device doesn't > support runtime pm. This may be purposely for reliability reasons or > the driver may just lack support. Check forums discussing issues with > the device and look for possible options in the driver to force pm > support (generally this will be centered around enabling ASPM). > > You can also download powertop to see the package c-state residencies > more clearly as percentages of time. powertop also has a tunables tab > that will show the status of runtime pm on all devices on the system > and allow you to enable them individually. > > > David > -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Low Latency Tolerance preventing Intel Package from entering deep sleep states 2020-05-22 16:16 ` larsh @ 2020-05-25 5:37 ` Adrian Hunter 0 siblings, 0 replies; 7+ messages in thread From: Adrian Hunter @ 2020-05-25 5:37 UTC (permalink / raw) To: larsh, Andy Shevchenko Cc: Rafael J. Wysocki, David E. Box, ibm-acpi-devel, platform-driver-x86, ACPI Devel Maling List, Ben Chuang + Ben Chuang <ben.chuang@genesyslogic.com.tw> It does not look like that driver allows runtime pm. On 22/05/20 7:16 pm, larsh@apache.org wrote: > Relevant logs: > > May 21 23:05:31 host kernel: sdhci-pci 0000:53:00.0: SDHCI controller found [17a0:9755] (rev 0) > May 21 23:05:31 host kernel: sdhci-pci 0000:53:00.0: enabling device (0000 -> 0002) > May 21 23:05:31 host kernel: mmc0: SDHCI controller on PCI [0000:53:00.0] using ADMA 64-bit > > > > On Friday, May 22, 2020, 1:59:08 AM PDT, Andy Shevchenko <andy.shevchenko@gmail.com> wrote: > > > > > > +Cc: Adrian > > On Fri, May 22, 2020 at 9:15 AM larsh@apache.org <larsh@apache.org> wrote: >> >> Thanks David! >> >> With this I tracked down the SD Card Reader (Genesys Logic, Inc Device 9755) as the culprit. >> These are standard in many ThinkPads. >> The curious part is that resume from suspend (S3 or S0iX) also fixes the problem. >> Looks like the driver is not initializing correctly at boot time. >> >> Transcript: >> >> $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT >> SOUTHPORT_A LTR: RAW: 0x88018c01 Non-Snoop(ns): 1024 Snoop(ns): 32768 >> SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 >> SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 >> SOUTHPORT_D LTR: RAW: 0x88aa88aa Non-Snoop(ns): 174080 Snoop(ns): 174080 >> SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 >> >> $ lspci -t >> -[0000:00]-+-00.0 >> +-01.0-[01]--+-00.0 >> | \-00.1 >> +-02.0 >> +-04.0 >> +-08.0 >> +-12.0 >> +-14.0 >> +-14.2 >> +-15.0 >> +-16.0 >> +-1c.0-[53]----00.0 >> +-1d.0-[02]----00.0 >> +-1d.6-[52]----00.0 >> +-1e.0 >> +-1f.0 >> +-1f.3 >> +-1f.4 >> +-1f.5 >> \-1f.6 >> >> $ lspci | grep 53 >> 53:00.0 SD Host controller: Genesys Logic, Inc Device 9755 >> >> $ cat /sys/bus/pci/devices/0000\:53\:00.0/power/control >> auto >> >> $ echo 1 > /sys/bus/pci/devices/0000\:53\:00.0/remove >> 1 >> >> $ cat /sys/kernel/debug/pmc_core/ltr_show | grep SOUTHPORT >> SOUTHPORT_A LTR: RAW: 0x8010c01 Non-Snoop(ns): 0 Snoop(ns): 0 >> SOUTHPORT_B LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 >> SOUTHPORT_C LTR: RAW: 0x9f409f4 Non-Snoop(ns): 0 Snoop(ns): 0 >> SOUTHPORT_D LTR: RAW: 0x8c548c54 Non-Snoop(ns): 2752512 Snoop(ns): 2752512 >> SOUTHPORT_E LTR: RAW: 0x0 Non-Snoop(ns): 0 Snoop(ns): 0 >> >> Cheers. >> >> -- Lars >> >> >> >> >> >> >> >> >> On Tuesday, May 19, 2020, 9:03:53 AM PDT, David E. Box <david.e.box@linux.intel.com> wrote: >> >> >> >> >> >>>>> Does anybody know what's going on or how to debug this further? >>>>> As stated above, I was able to work around this problem by >>>>> ignoring SOUTHPORT_A via /sys/kernel/debug/pmc_core/ltr_ignore. >>>>> There has to be a better way, and I'm sure I'm not the only one >>>>> running into this. >> >> ltr_show shows the PMC's (Power Management Controller) view of SoC >> devices and busses. The SOUTHPORTs are the PCIe root ports on your >> system. When you run lspci they are the PCI bridges. Generally, the >> bridges are enumerated in the same order as the SOUTHPORTs, so >> SOUTHPORT_A is your first bridge and the device attached to it (shown >> in lspci -t) is the device that was blocking deeper PC states according >> to your debug. >> >> Determine what this device is on your system. If the ltr was low it's >> because that is what the device requested. You should first check that >> runtime pm is enabled for the device. To do this, check the control >> file in /sys/bus/pci/devices/<SSSS:BB:DD.F>/power, where SSSS:BB:DD.F >> is the enumeration of your device as shown in lspci. If it is 'on' then >> runtime pm is disabled. To enable it echo 'auto' into the file with >> root privileges. Enabling runtime pm should allow the driver to reduce >> functionality of the device when idle. This should lead to a larger >> latency request on the PCI bus which should be reflected in ltr_show. >> You can see if the device is actually runtime suspended and how much >> time it's been suspended (or active) by reading the associated files in >> the power folder. >> >> If this doesn't work, then it's possible that your device doesn't >> support runtime pm. This may be purposely for reliability reasons or >> the driver may just lack support. Check forums discussing issues with >> the device and look for possible options in the driver to force pm >> support (generally this will be centered around enabling ASPM). >> >> You can also download powertop to see the package c-state residencies >> more clearly as percentages of time. powertop also has a tunables tab >> that will show the status of runtime pm on all devices on the system >> and allow you to enable them individually. >> >> >> David > >> > > ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-05-25 5:37 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <1505028180.591737.1589564161284.ref@mail.yahoo.com>
[not found] ` <1505028180.591737.1589564161284@mail.yahoo.com>
2020-05-15 21:41 ` Low Latency Tolerance preventing Intel Package from entering deep sleep states Andy Shevchenko
2020-05-18 10:26 ` Rafael J. Wysocki
2020-05-19 16:03 ` David E. Box
2020-05-22 6:14 ` larsh
2020-05-22 8:58 ` Andy Shevchenko
2020-05-22 16:16 ` larsh
2020-05-25 5:37 ` Adrian Hunter
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).