Re: [PATCH] PCI: increase D3 delay for AMD Ryzen5/7 XHCI controllers

From: Bjorn Helgaas <helgaas@kernel.org>
To: Mika Westerberg <mika.westerberg@linux.intel.com>
Cc: Daniel Drake <drake@endlessm.com>,
	Linux PCI <linux-pci@vger.kernel.org>,
	"Wysocki, Rafael J" <rafael.j.wysocki@intel.com>,
	Linux Upstreaming Team <linux@endlessm.com>,
	Linux PM <linux-pm@vger.kernel.org>,
	Linux USB Mailing List <linux-usb@vger.kernel.org>
Subject: Re: [PATCH] PCI: increase D3 delay for AMD Ryzen5/7 XHCI controllers
Date: Wed, 23 Oct 2019 17:40:03 -0500	[thread overview]
Message-ID: <20191023224003.GA31692@google.com> (raw)
In-Reply-To: <20191022093349.GC2819@lahna.fi.intel.com>

On Tue, Oct 22, 2019 at 12:33:49PM +0300, Mika Westerberg wrote:
> On Tue, Oct 22, 2019 at 10:40:00AM +0800, Daniel Drake wrote:
> > On Mon, Oct 21, 2019 at 7:33 PM Mika Westerberg
> > <mika.westerberg@linux.intel.com> wrote:
> > > Just to be sure, did you try the patch or just looked at it? Because
> > > what the patch does is that it does the delay when the downstream/root
> > > port is resumed, not the xHCI itself.
> > 
> > I tried it, it didn't fix the problem.
> 
> :(
> 
> It may very well be that this particular xHCI controller needs more than
> that 10ms from D3hot -> D0 transition. Again the PCIe spec says the 10ms
> is the minimum time system software needs to delay but it does not say
> what would be the maximum time the function absolutely must be properly
> in D0.

Hmm, PCIe r5.0, sec 2.3.1, says devices are permitted to return
Configuration Request Retry Status (CRS) after a "reset initiated in
response to a D3hot to D0uninitialized" transition.

I think that applies to this device because your lspci [1] shows:

	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

so No_Soft_Reset is clear, which means the D3hot to D0 transition goes
to D0uninitialized.

pci_raw_set_power_state() *does* delay, generally for 10ms:

  pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, pmcsr);
  if (state == PCI_D3hot || dev->current_state == PCI_D3hot)
    pci_dev_d3_sleep(dev);
  pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);

but there's no mention of CRS, so I think that config read is liable
to fail with CRS if the device isn't ready, and we won't notice.

I think we need something like the patch below.  We already do
basically the same thing in pci_pm_reset().

[1] https://gist.github.com/dsd/bd9370b35defdf43680b81ecb34381d5

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index e7982af9a5d8..e8702388830f 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -883,9 +883,10 @@ static int pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state)
 	 * Mandatory power management transition delays; see PCI PM 1.1
 	 * 5.6.1 table 18
 	 */
-	if (state == PCI_D3hot || dev->current_state == PCI_D3hot)
+	if (state == PCI_D3hot || dev->current_state == PCI_D3hot) {
 		pci_dev_d3_sleep(dev);
-	else if (state == PCI_D2 || dev->current_state == PCI_D2)
+		pci_dev_wait(dev, "D3 transition", PCIE_RESET_READY_POLL_MS);
+	} else if (state == PCI_D2 || dev->current_state == PCI_D2)
 		udelay(PCI_PM_D2_DELAY);
 
 	pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
