Re: [PATCH 2/3] PCI: Do not poll for PME if the device is in D3cold

[Mika Westerberg <mika.westerberg@linux.intel.com>]

On Wed, Jun 05, 2019 at 09:05:57PM +0200, Lukas Wunner wrote:
> On Wed, Jun 05, 2019 at 05:58:19PM +0300, Mika Westerberg wrote:
> > PME polling does not take into account that a device that is directly
> > connected to the host bridge may go into D3cold as well. This leads to a
> > situation where the PME poll thread reads from a config space of a
> > device that is in D3cold and gets incorrect information because the
> > config space is not accessible.
> > 
> > Here is an example from Intel Ice Lake system where two PCIe root ports
> > are in D3cold (I've instrumented the kernel to log the PMCSR register
> > contents):
> > 
> >   [   62.971442] pcieport 0000:00:07.1: Check PME status, PMCSR=0xffff
> >   [   62.971504] pcieport 0000:00:07.0: Check PME status, PMCSR=0xffff
> > 
> > Since 0xffff is interpreted so that PME is pending, the root ports will
> > be runtime resumed. This repeats over and over again essentially
> > blocking all runtime power management.
> > 
> > Prevent this from happening by checking whether the device is in D3cold
> > before its PME status is read.
> 
> There's more broken here.  The below patch fixes a PME polling race
> and should also fix the issue you're witnessing, could you verify that?

It fixes the issue but I needed to tune it a bit ->

> The patch has been rotting on my development branch for several months,
> I just didn't get around to posting it, my apologies.

Better late than never :)

> -- >8 --
> Subject: [PATCH] PCI / PM: Fix race on PME polling
> 
> Since commit df17e62e5bff ("PCI: Add support for polling PME state on
> suspended legacy PCI devices"), the work item pci_pme_list_scan() polls
> the PME status flag of devices and wakes them up if the bit is set.
> 
> The function performs a check whether a device's upstream bridge is in
> D0 for otherwise the device is inaccessible, rendering PME polling
> impossible.  However the check is racy because it is performed before
> polling the device.  If the upstream bridge runtime suspends to D3hot
> after pci_pme_list_scan() checks its power state and before it invokes
> pci_pme_wakeup(), the latter will read the PMCSR as "all ones" and
> mistake it for a set PME status flag.  I am seeing this race play out as
> a Thunderbolt controller going to D3cold and occasionally immediately
> going to D0 again because PM polling was performed at just the wrong
> time.
> 
> Avoid by checking for an "all ones" PMCSR in pci_check_pme_status().
> 
> Fixes: 58ff463396ad ("PCI PM: Add function for checking PME status of devices")
> Signed-off-by: Lukas Wunner <lukas@wunner.de>
> Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> Cc: stable@vger.kernel.org # v2.6.34+
> ---
>  drivers/pci/pci.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index b98a564..2e05348 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -1753,6 +1753,8 @@ bool pci_check_pme_status(struct pci_dev *dev)
>  	pci_read_config_word(dev, pmcsr_pos, &pmcsr);
>  	if (!(pmcsr & PCI_PM_CTRL_PME_STATUS))
>  		return false;
> +	if (pmcsr == ~0)

<- Here I needed to do

	if (pmcsr == (u16)~0)

I think it is because pmcsr is u16 so we end up comparing:

	0xffff == 0xffffffff

> +		return false;
>  
>  	/* Clear PME status. */
>  	pmcsr |= PCI_PM_CTRL_PME_STATUS;
> -- 
> 2.20.1