Re: [Bug 215533] [BISECTED][REGRESSION] UI becomes unresponsive every couple of seconds

Re: [Bug 215533] [BISECTED][REGRESSION] UI becomes unresponsive every couple of seconds

[Bjorn Helgaas]

[+cc Jan, author of 0e8ae5a6ff59, linux-pci]

On Wed, Jan 26, 2022 at 08:18:12AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=215533
> 
> --- Comment #1 from joey.corleone@mail.ru ---
> I accidentally sent the report prematurely. So here come my findings:
> 
> Since 5.16
> (1) my system becomes unresponsive every couple of seconds (micro lags), which
> makes it more or less unusable.
> (2) wrong(?) CPU frequencies are reported. 
> 
> - 5.15 works fine.
> - Starting from some commit in 5.17, it seems (1) is fixed (unsure), but
> definitely not (2).
> 
> I have bisected the kernel between 5.15 and 5.16, and found that the offending
> commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009 ("PCI/portdrv: Do not setup
> up IRQs if there are no users"). Bisection log attached.
> 
> Reverting this commit on linux-git[1] fixes both (1) and (2).
> 
> Important notes:
> - This regression was reported on a DELL XPS 9550 laptop by two users [2], so
> it might be related strictly to that model. 
> - According to user mallocman, the issue can also be fixed by reverting the
> BIOS version of the laptop to v1.12.
> - The issue ONLY occurs when AC is plugged in (and stays there even when I
> unplug it).
> - When booting on battery power, there is no issue at all.
> 
> You can easily observe the regression via: 
> 
> watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
> 
> As soon as I plug in AC, all frequencies go up to values around 3248338 and
> stay there even if I unplug AC. This does not happen at all when booted on
> battery power. 
> 
> Also note: 
> - the laptop's fans are not really affected by the high frequencies.
> - setting the governor to "powersave" has no effect on the frequencies (as
> compared to when on battery power).
> - lowering the maximum frequency manually works, but does not fix (1).
> 
> [1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to
> 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
> [2] https://bbs.archlinux.org/viewtopic.php?id=273330
> 
> -- 
> You may reply to this email to add a comment.
> 
> You are receiving this mail because:
> You are watching the assignee of the bug.

Re: [Bug 215533] [BISECTED][REGRESSION] UI becomes unresponsive every couple of seconds

[Bjorn Helgaas]

[+cc linux-kernel for visibility]

On Wed, Jan 26, 2022 at 06:12:50AM -0600, Bjorn Helgaas wrote:
> On Wed, Jan 26, 2022 at 08:18:12AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=215533
> > 
> > --- Comment #1 from joey.corleone@mail.ru ---
> > I accidentally sent the report prematurely. So here come my findings:
> > 
> > Since 5.16
> > (1) my system becomes unresponsive every couple of seconds (micro lags), which
> > makes it more or less unusable.
> > (2) wrong(?) CPU frequencies are reported. 
> > 
> > - 5.15 works fine.
> > - Starting from some commit in 5.17, it seems (1) is fixed (unsure), but
> > definitely not (2).
> > 
> > I have bisected the kernel between 5.15 and 5.16, and found that the offending
> > commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009 ("PCI/portdrv: Do not setup
> > up IRQs if there are no users"). Bisection log attached.
> > 
> > Reverting this commit on linux-git[1] fixes both (1) and (2).
> > 
> > Important notes:
> > - This regression was reported on a DELL XPS 9550 laptop by two users [2], so
> > it might be related strictly to that model. 
> > - According to user mallocman, the issue can also be fixed by reverting the
> > BIOS version of the laptop to v1.12.
> > - The issue ONLY occurs when AC is plugged in (and stays there even when I
> > unplug it).
> > - When booting on battery power, there is no issue at all.
> > 
> > You can easily observe the regression via: 
> > 
> > watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
> > 
> > As soon as I plug in AC, all frequencies go up to values around 3248338 and
> > stay there even if I unplug AC. This does not happen at all when booted on
> > battery power. 
> > 
> > Also note: 
> > - the laptop's fans are not really affected by the high frequencies.
> > - setting the governor to "powersave" has no effect on the frequencies (as
> > compared to when on battery power).
> > - lowering the maximum frequency manually works, but does not fix (1).
> > 
> > [1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to
> > 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
> > [2] https://bbs.archlinux.org/viewtopic.php?id=273330

I hope we can find a better solution, but since the responsiveness
issue is a significant regression, I queued up a revert of
0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no
users") in case we don't find one.

If/when we get to the bottom of this, I'll replace the revert with the
solution.  0e8ae5a6ff59 appeared in v5.16, so we'll have to make sure
we fix that as well.

Bjorn

Re: [Bug 215533] [BISECTED][REGRESSION] UI becomes unresponsive every couple of seconds

[Jan Kiszka]

On 07.02.22 23:45, Bjorn Helgaas wrote:
> [+cc linux-kernel for visibility]
> 
> On Wed, Jan 26, 2022 at 06:12:50AM -0600, Bjorn Helgaas wrote:
>> On Wed, Jan 26, 2022 at 08:18:12AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
>>> https://bugzilla.kernel.org/show_bug.cgi?id=215533
>>>
>>> --- Comment #1 from joey.corleone@mail.ru ---
>>> I accidentally sent the report prematurely. So here come my findings:
>>>
>>> Since 5.16
>>> (1) my system becomes unresponsive every couple of seconds (micro lags), which
>>> makes it more or less unusable.
>>> (2) wrong(?) CPU frequencies are reported. 
>>>
>>> - 5.15 works fine.
>>> - Starting from some commit in 5.17, it seems (1) is fixed (unsure), but
>>> definitely not (2).
>>>
>>> I have bisected the kernel between 5.15 and 5.16, and found that the offending
>>> commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009 ("PCI/portdrv: Do not setup
>>> up IRQs if there are no users"). Bisection log attached.
>>>
>>> Reverting this commit on linux-git[1] fixes both (1) and (2).
>>>
>>> Important notes:
>>> - This regression was reported on a DELL XPS 9550 laptop by two users [2], so
>>> it might be related strictly to that model. 
>>> - According to user mallocman, the issue can also be fixed by reverting the
>>> BIOS version of the laptop to v1.12.
>>> - The issue ONLY occurs when AC is plugged in (and stays there even when I
>>> unplug it).
>>> - When booting on battery power, there is no issue at all.
>>>
>>> You can easily observe the regression via: 
>>>
>>> watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
>>>
>>> As soon as I plug in AC, all frequencies go up to values around 3248338 and
>>> stay there even if I unplug AC. This does not happen at all when booted on
>>> battery power. 
>>>
>>> Also note: 
>>> - the laptop's fans are not really affected by the high frequencies.
>>> - setting the governor to "powersave" has no effect on the frequencies (as
>>> compared to when on battery power).
>>> - lowering the maximum frequency manually works, but does not fix (1).
>>>
>>> [1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to
>>> 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
>>> [2] https://bbs.archlinux.org/viewtopic.php?id=273330
> 
> I hope we can find a better solution, but since the responsiveness
> issue is a significant regression, I queued up a revert of
> 0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no
> users") in case we don't find one.

Likely best for now.

> 
> If/when we get to the bottom of this, I'll replace the revert with the
> solution.  0e8ae5a6ff59 appeared in v5.16, so we'll have to make sure
> we fix that as well.

If you could give some feedback/hints on the questions I posted last
week on the original patch, that might accelerate understanding the real
issue.

Thanks,
Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [Bug 215533] [BISECTED][REGRESSION] UI becomes unresponsive every couple of seconds

[Bjorn Helgaas]

On Mon, Feb 07, 2022 at 04:45:40PM -0600, Bjorn Helgaas wrote:
> On Wed, Jan 26, 2022 at 06:12:50AM -0600, Bjorn Helgaas wrote:
> > On Wed, Jan 26, 2022 at 08:18:12AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=215533
> > > 
> > > --- Comment #1 from joey.corleone@mail.ru ---
> > > I accidentally sent the report prematurely. So here come my findings:
> > > 
> > > Since 5.16
> > > (1) my system becomes unresponsive every couple of seconds
> > > (micro lags), which makes it more or less unusable.
> > > (2) wrong(?) CPU frequencies are reported. 
> > > 
> > > - 5.15 works fine.
> > > - Starting from some commit in 5.17, it seems (1) is fixed
> > > (unsure), but definitely not (2).
> > > 
> > > I have bisected the kernel between 5.15 and 5.16, and found that
> > > the offending commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009
> > > ("PCI/portdrv: Do not setup up IRQs if there are no users").
> > > Bisection log attached.
> > > 
> > > Reverting this commit on linux-git[1] fixes both (1) and (2).
> > > 
> > > Important notes:
> > > - This regression was reported on a DELL XPS 9550 laptop by two
> > > users [2], so it might be related strictly to that model. 
> > > - According to user mallocman, the issue can also be fixed by
> > > reverting the BIOS version of the laptop to v1.12.
> > > - The issue ONLY occurs when AC is plugged in (and stays there
> > > even when I unplug it).
> > > - When booting on battery power, there is no issue at all.
> > > 
> > > You can easily observe the regression via: 
> > > 
> > > watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
> > > 
> > > As soon as I plug in AC, all frequencies go up to values around
> > > 3248338 and stay there even if I unplug AC. This does not happen
> > > at all when booted on battery power. 
> > > 
> > > Also note: 
> > > - the laptop's fans are not really affected by the high
> > > frequencies.
> > > - setting the governor to "powersave" has no effect on the
> > > frequencies (as compared to when on battery power).
> > > - lowering the maximum frequency manually works, but does not
> > > fix (1).
> > > 
> > > [1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to
> > > 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
> > > [2] https://bbs.archlinux.org/viewtopic.php?id=273330
> 
> I hope we can find a better solution, but since the responsiveness
> issue is a significant regression, I queued up a revert of
> 0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no
> users") in case we don't find one.

Here's the revert I plan to merge for v5.17.  As mentioned at [3], I
think the problem is a BIOS defect that leaves interrupts enabled when
handing off to the kernel.

Bjorn

[3] https://lore.kernel.org/r/20220208185658.GA489586@bhelgaas


commit b139e2632409 ("Revert "PCI/portdrv: Do not setup up IRQs if there are no users"")
Author: Bjorn Helgaas <bhelgaas@google.com>
Date:   Mon Feb 7 16:33:30 2022 -0600

    Revert "PCI/portdrv: Do not setup up IRQs if there are no users"
    
    This reverts commit 0e8ae5a6ff5952253cd7cc0260df838ab4c21009.
    
    0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no users")
    reduced usage of IRQs when we don't think we need them.  But several people
    reported unresponsive systems and bisected it to this commit.
    
    Revert 0e8ae5a6ff59 until we figure out a better solution.
    
    Reported-by: Joey Corleone <joey.corleone@mail.ru>
    Reported-by: Sergiu Deitsch <sergiu.deitsch@gmail.com>
    Reported-by: David Spencer <dspencer577@gmail.com>
    Link: https://bugzilla.kernel.org/show_bug.cgi?id=215533
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Cc: stable@vger.kernel.org	# v5.16+
    Cc: Jan Kiszka <jan.kiszka@siemens.com>

diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index bda630889f95..604feeb84ee4 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -166,6 +166,9 @@ static int pcie_init_service_irqs(struct pci_dev *dev, int *irqs, int mask)
 {
 	int ret, i;
 
+	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
+		irqs[i] = -1;
+
 	/*
 	 * If we support PME but can't use MSI/MSI-X for it, we have to
 	 * fall back to INTx or other interrupts, e.g., a system shared
@@ -314,10 +317,8 @@ static int pcie_device_init(struct pci_dev *pdev, int service, int irq)
  */
 int pcie_port_device_register(struct pci_dev *dev)
 {
-	int status, capabilities, irq_services, i, nr_service;
-	int irqs[PCIE_PORT_DEVICE_MAXSERVICES] = {
-		[0 ... PCIE_PORT_DEVICE_MAXSERVICES-1] = -1
-	};
+	int status, capabilities, i, nr_service;
+	int irqs[PCIE_PORT_DEVICE_MAXSERVICES];
 
 	/* Enable PCI Express port device */
 	status = pci_enable_device(dev);
@@ -330,32 +331,18 @@ int pcie_port_device_register(struct pci_dev *dev)
 		return 0;
 
 	pci_set_master(dev);
-
-	irq_services = 0;
-	if (IS_ENABLED(CONFIG_PCIE_PME))
-		irq_services |= PCIE_PORT_SERVICE_PME;
-	if (IS_ENABLED(CONFIG_PCIEAER))
-		irq_services |= PCIE_PORT_SERVICE_AER;
-	if (IS_ENABLED(CONFIG_HOTPLUG_PCI_PCIE))
-		irq_services |= PCIE_PORT_SERVICE_HP;
-	if (IS_ENABLED(CONFIG_PCIE_DPC))
-		irq_services |= PCIE_PORT_SERVICE_DPC;
-	irq_services &= capabilities;
-
-	if (irq_services) {
-		/*
-		 * Initialize service IRQs. Don't use service devices that
-		 * require interrupts if there is no way to generate them.
-		 * However, some drivers may have a polling mode (e.g.
-		 * pciehp_poll_mode) that can be used in the absence of IRQs.
-		 * Allow them to determine if that is to be used.
-		 */
-		status = pcie_init_service_irqs(dev, irqs, irq_services);
-		if (status) {
-			irq_services &= PCIE_PORT_SERVICE_HP;
-			if (!irq_services)
-				goto error_disable;
-		}
+	/*
+	 * Initialize service irqs. Don't use service devices that
+	 * require interrupts if there is no way to generate them.
+	 * However, some drivers may have a polling mode (e.g. pciehp_poll_mode)
+	 * that can be used in the absence of irqs.  Allow them to determine
+	 * if that is to be used.
+	 */
+	status = pcie_init_service_irqs(dev, irqs, capabilities);
+	if (status) {
+		capabilities &= PCIE_PORT_SERVICE_HP;
+		if (!capabilities)
+			goto error_disable;
 	}
 
 	/* Allocate child services if any */