Re: Possible PCI Regression Linux 5.3-rc1

From: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
To: Logan Gunthorpe <logang@deltatee.com>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	Mika Westerberg <mika.westerberg@linux.intel.com>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>
Subject: Re: Possible PCI Regression Linux 5.3-rc1
Date: Sun, 4 Aug 2019 08:47:43 +0000	[thread overview]
Message-ID: <SL2P216MB018719A03F048FFA0F745FED80DB0@SL2P216MB0187.KORP216.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <4f5afb8e-9013-980f-0553-c687d17ed8d5@deltatee.com>

On Thu, Jul 25, 2019 at 09:50:48AM -0600, Logan Gunthorpe wrote:
> 
> 
> On 2019-07-25 7:18 a.m., Nicholas Johnson wrote:
> > On Wed, Jul 24, 2019 at 08:38:14AM -0500, Bjorn Helgaas wrote:
> >> On Wed, Jul 24, 2019 at 12:54:00PM +0000, Nicholas Johnson wrote:
> >>> Hi all,
> >>>
> >>> I was just rebasing my patches for linux 5.3-rc1 and noticed a possible 
> >>> regression that shows on both of my machines. It is also reproducible 
> >>> with the unmodified Ubuntu mainline kernel, downloadable at [1].
> >>>
> >>> Running the lspci command takes 1-3 seconds with 5.3-rc1 (rather than an 
> >>> imperceivable amount of time). Booting with pci.dyndbg does not reveal 
> >>> why.
> >>>
> >>> $ uname -r
> >>> 5.3.0-050300rc1-generic
> >>> $ time lspci -vt 1>/dev/null
> >>>
> >>> real	0m2.321s
> >>> user	0m0.026s
> >>> sys	0m0.000s
> >>>
> >>> If none of you are aware of this or what is causing it, I will submit a 
> >>> bug report to Bugzilla.
> >>
> >> I wasn't aware of this; thanks for reporting it!  I wasn't able to
> >> reproduce this in qemu.  Can you play with "strace -r lspci -vt" and
> >> the like?  Maybe try "lspci -n" to see if it's related to looking up
> >> the names?
> > 
> > For a second you had me doubting myself - it could have been a Ubuntu 
> > thing. But no, I just reproduced it on Arch Linux, and double checked 
> > that it was not doing it on 5.2. Also, the problem occurs even without 
> > the PCI kernel parameters which I usually pass.
> 
> Ok, can you bisect to find the commit that causes this issue?

I have done a partial bisect and then found the culprit commit by visual 
inspection. I would have done the full bisect, but I am using highly
underpowered i7-7700K so each round requires 20-30 minutes of compiling.

Reversing the following commit solves the issue:

commit c2bf1fc212f7e6f25ace1af8f0b3ac061ea48ba5
Author: Mika Westerberg <mika.westerberg@linux.intel.com>
PCI: Add missing link delays required by the PCIe spec

Mika, care to weigh in (assuming you are back from four weeks leave)? 
Clearly this creates delays in "lspci -vt" in some Thunderbolt systems, 
but not all - otherwise you would have caught it. You mentioned Ice Lake 
in the commit log so perhaps it works fine on Ice Lake.

Thanks,
Nicholas

> 
> Thanks,
> 
> Logan