* Re: PCI IRQ assignment broken from 4.9 onwards (swizzle?)
[not found] <CAEt4U6V22Abudp1TRJ2SGkYH=2o+RFWa8nrGbKDoLFXcZJV7qA@mail.gmail.com>
@ 2020-10-07 16:41 ` Bjorn Helgaas
2020-10-08 8:39 ` Billy Araujo
0 siblings, 1 reply; 3+ messages in thread
From: Bjorn Helgaas @ 2020-10-07 16:41 UTC (permalink / raw)
To: Billy Araujo; +Cc: linux-pci, Kishon Vijay Abraham I, linux-omap
[+cc Kishon, linux-omap (maybe this is dra7xx-related?)]
On Wed, Oct 07, 2020 at 10:56:33AM +0100, Billy Araujo wrote:
> Hi,
>
> I have been testing a TI AM57xx board and a NXP iMX8 board with a GPIB
> PCIe card.
>
> TI board (Phytec): https://www.phytec.com/product/phycore-am57x/
> NXP board (Variscite):
> https://www.variscite.com/product/system-on-module-som/cortex-a53-krait/var-som-mx8m-mini-nxp-i-mx8m-mini/
>
> The GPIB PCIe card has a Texas Instruments XIO2000(A)/XIO2200A PCI
> Express-to-PCI Bridge.
>
> Issue:
> I have noticed is that on Linux kernel 4.9, the Linux PCI driver
> assigns correctly an IRQ number:
>
> Linux am5728-phycore-rdk 4.9.41-ga962b18-BSP-Yocto-TISDK-AM57xx-PD18.1.0
> 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> Subsystem: National Instruments PCIe-GPIB
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 470
>
> On a newer kernel (this case 4.19), PCI driver doesn't assign an IRQ number.
>
> Linux am57xx-phycore-kit 4.19.79-g35d36cd54d #1 SMP PREEMPT Wed Sep 30
> 14:04:18 UTC 2020 armv7l GNU/Linux
> 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> Subsystem: National Instruments PCIe-GPIB
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 0
>
> Same issue happened on the NXP board, so it seems Linux related. I
> have tested kernels 4.14, 4.19 and 5.4.3.
>
> The IRQ is important to get the legacy interrupts working.
>
> Looking at the code there has been some refactoring of how PCI assigns
> IRQ number when there is a chain of bridges. I am not too familiar
> with how the code works but I wonder if this has affected how the PCI
> assignment works.
>
> Looking in setup-irq.c:
>
> /* If this device is not on the primary bus, we need to figure out
> which interrupt pin it will come in on. We know which slot it
> will come in on 'cos that slot is where the bridge is. Each
> time the interrupt line passes through a PCI-PCI bridge we must
> apply the swizzle function. */
>
> Line 44: if (hbrg->swizzle_irq)
>
> From my understanding, this "if" didn't exist in Linux kernel 4.9. If
> swizzle function isn't assigned in the newer kernels it just stays as
> 0.
>
> This might be completely unrelated as I said I have no understanding
> how this code is supposed to work.
>
> What I ask is if anyone has experienced any issues similar to this in
> these more recent kernel versions.
Sorry for the issue, and thanks very much for the report. Is it
possible to test a current kernel, e.g., v5.8 or v5.9-rc8?
My guess is this is related to the PCI controller driver; would that
be pci-dra7xx.c?
> Debug output with the issue:
>
> root@am57xx-phycore-kit:~# uname -a
> Linux am57xx-phycore-kit 4.19.79-g35d36cd54d #1 SMP PREEMPT Wed Sep 30
> 14:04:18 UTC 2020 armv7l GNU/Linux
>
> root@am57xx-phycore-kit:~# lspci -vv
> 00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II
> SOC (rev 01) (prog-if 00 [Normal decode])
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 180
> Region 0: Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: 20200000-202fffff [size=1M]
> Prefetchable memory behind bridge: None
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort+ <SERR- <PERR-
> BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: [40] Power Management version 3
> Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA
> PME(D0+,D1+,D2-,D3hot+,D3cold-)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> Address: 00000000ae15b000 Data: 0000
> Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
> DevCap: MaxPayload 256 bytes, PhantFunc 0
> ExtTag- RBE+
> DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> AuxPwr- TransPend-
> LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1,
> Exit Latency L0s <512ns, L1 <64us
> ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
> LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
> TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna+ CRSVisible-
> RootCap: CRSVisible-
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported ARIFwd-
> AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> DevCtl2: Completion Timeout: 50us to 50ms,
> TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> AtomicOpsCtl: ReqEn- EgressBlck-
> LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete-, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
> Capabilities: [100 v2] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr-
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr+
> AERCap: First Error Pointer: 00, ECRCGenCap+
> ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> HeaderLog: 00000000 00000000 00000000 00000000
> RootCmd: CERptEn+ NFERptEn+ FERptEn+
> RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
> FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
> ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
> Kernel driver in use: pcieport
>
> 01:00.0 PCI bridge: Texas Instruments XIO2000(A)/XIO2200A PCI
> Express-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: 20200000-202fffff [size=1M]
> Prefetchable memory behind bridge: None
> Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B+
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: [50] Power Management version 2
> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> Bridge: PM- B3+
> Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+
> Address: 0000000000000000 Data: 0000
> Capabilities: [80] Subsystem: Device 0000:0000
> Capabilities: [90] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
> DevCap: MaxPayload 512 bytes, PhantFunc 0
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE-
> SlotPowerLimit 0.000W
> DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq-
> AuxPwr- TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> Exit Latency L0s <1us, L1 <16us
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
> TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> Capabilities: [100 v1] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr-
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr-
> AERCap: First Error Pointer: 00, ECRCGenCap+
> ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> HeaderLog: 00000000 00000000 00000000 00000000
>
> 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> Subsystem: National Instruments PCIe-GPIB
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 0
> Region 0: Memory at 20204000 (32-bit, non-prefetchable)
> [disabled] [size=2K]
> Region 1: Memory at 20200000 (32-bit, non-prefetchable)
> [disabled] [size=16K]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: PCI IRQ assignment broken from 4.9 onwards (swizzle?)
2020-10-07 16:41 ` PCI IRQ assignment broken from 4.9 onwards (swizzle?) Bjorn Helgaas
@ 2020-10-08 8:39 ` Billy Araujo
2020-10-08 15:38 ` Bjorn Helgaas
0 siblings, 1 reply; 3+ messages in thread
From: Billy Araujo @ 2020-10-08 8:39 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: linux-pci, Kishon Vijay Abraham I, linux-omap
Hi Bjorn,
Thanks for your answer. Yes, that would be my next step.
I am also hoping the latest kernel doesn't have this issue because
that would mean I would be stuck with version 4.9.
Also I could discover from which patch this stopped working but that
would be very time consuming. If I was to add some debug prints where
should I start? Would this be done in setup-irq.c?
In other words, is "pci_assign_irq" function responsible for all irq
assignments?
Regards,
Billy.
On Wed, Oct 7, 2020 at 5:41 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Kishon, linux-omap (maybe this is dra7xx-related?)]
>
> On Wed, Oct 07, 2020 at 10:56:33AM +0100, Billy Araujo wrote:
> > Hi,
> >
> > I have been testing a TI AM57xx board and a NXP iMX8 board with a GPIB
> > PCIe card.
> >
> > TI board (Phytec): https://www.phytec.com/product/phycore-am57x/
> > NXP board (Variscite):
> > https://www.variscite.com/product/system-on-module-som/cortex-a53-krait/var-som-mx8m-mini-nxp-i-mx8m-mini/
> >
> > The GPIB PCIe card has a Texas Instruments XIO2000(A)/XIO2200A PCI
> > Express-to-PCI Bridge.
> >
> > Issue:
> > I have noticed is that on Linux kernel 4.9, the Linux PCI driver
> > assigns correctly an IRQ number:
> >
> > Linux am5728-phycore-rdk 4.9.41-ga962b18-BSP-Yocto-TISDK-AM57xx-PD18.1.0
> > 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> > Subsystem: National Instruments PCIe-GPIB
> > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > Interrupt: pin A routed to IRQ 470
> >
> > On a newer kernel (this case 4.19), PCI driver doesn't assign an IRQ number.
> >
> > Linux am57xx-phycore-kit 4.19.79-g35d36cd54d #1 SMP PREEMPT Wed Sep 30
> > 14:04:18 UTC 2020 armv7l GNU/Linux
> > 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> > Subsystem: National Instruments PCIe-GPIB
> > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > Interrupt: pin A routed to IRQ 0
> >
> > Same issue happened on the NXP board, so it seems Linux related. I
> > have tested kernels 4.14, 4.19 and 5.4.3.
> >
> > The IRQ is important to get the legacy interrupts working.
> >
> > Looking at the code there has been some refactoring of how PCI assigns
> > IRQ number when there is a chain of bridges. I am not too familiar
> > with how the code works but I wonder if this has affected how the PCI
> > assignment works.
> >
> > Looking in setup-irq.c:
> >
> > /* If this device is not on the primary bus, we need to figure out
> > which interrupt pin it will come in on. We know which slot it
> > will come in on 'cos that slot is where the bridge is. Each
> > time the interrupt line passes through a PCI-PCI bridge we must
> > apply the swizzle function. */
> >
> > Line 44: if (hbrg->swizzle_irq)
> >
> > From my understanding, this "if" didn't exist in Linux kernel 4.9. If
> > swizzle function isn't assigned in the newer kernels it just stays as
> > 0.
> >
> > This might be completely unrelated as I said I have no understanding
> > how this code is supposed to work.
> >
> > What I ask is if anyone has experienced any issues similar to this in
> > these more recent kernel versions.
>
> Sorry for the issue, and thanks very much for the report. Is it
> possible to test a current kernel, e.g., v5.8 or v5.9-rc8?
>
> My guess is this is related to the PCI controller driver; would that
> be pci-dra7xx.c?
>
> > Debug output with the issue:
> >
> > root@am57xx-phycore-kit:~# uname -a
> > Linux am57xx-phycore-kit 4.19.79-g35d36cd54d #1 SMP PREEMPT Wed Sep 30
> > 14:04:18 UTC 2020 armv7l GNU/Linux
> >
> > root@am57xx-phycore-kit:~# lspci -vv
> > 00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II
> > SOC (rev 01) (prog-if 00 [Normal decode])
> > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > ParErr+ Stepping- SERR+ FastB2B- DisINTx+
> > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- >SERR- <PERR- INTx-
> > Latency: 0, Cache Line Size: 64 bytes
> > Interrupt: pin A routed to IRQ 180
> > Region 0: Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
> > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > I/O behind bridge: None
> > Memory behind bridge: 20200000-202fffff [size=1M]
> > Prefetchable memory behind bridge: None
> > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort+ <SERR- <PERR-
> > BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
> > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> > Capabilities: [40] Power Management version 3
> > Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA
> > PME(D0+,D1+,D2-,D3hot+,D3cold-)
> > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > Address: 00000000ae15b000 Data: 0000
> > Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
> > DevCap: MaxPayload 256 bytes, PhantFunc 0
> > ExtTag- RBE+
> > DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
> > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> > MaxPayload 128 bytes, MaxReadReq 512 bytes
> > DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> > AuxPwr- TransPend-
> > LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1,
> > Exit Latency L0s <512ns, L1 <64us
> > ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
> > LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
> > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> > LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
> > TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
> > RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> > PMEIntEna+ CRSVisible-
> > RootCap: CRSVisible-
> > RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> > LTR-, OBFF Not Supported ARIFwd-
> > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> > DevCtl2: Completion Timeout: 50us to 50ms,
> > TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> > AtomicOpsCtl: ReqEn- EgressBlck-
> > LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> > Transmit Margin: Normal Operating Range,
> > EnterModifiedCompliance- ComplianceSOS-
> > Compliance De-emphasis: -6dB
> > LnkSta2: Current De-emphasis Level: -3.5dB,
> > EqualizationComplete-, EqualizationPhase1-
> > EqualizationPhase2-, EqualizationPhase3-,
> > LinkEqualizationRequest-
> > Capabilities: [100 v2] Advanced Error Reporting
> > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr-
> > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr+
> > AERCap: First Error Pointer: 00, ECRCGenCap+
> > ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> > HeaderLog: 00000000 00000000 00000000 00000000
> > RootCmd: CERptEn+ NFERptEn+ FERptEn+
> > RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
> > FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
> > ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
> > Kernel driver in use: pcieport
> >
> > 01:00.0 PCI bridge: Texas Instruments XIO2000(A)/XIO2200A PCI
> > Express-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
> > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- >SERR- <PERR- INTx-
> > Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
> > I/O behind bridge: None
> > Memory behind bridge: 20200000-202fffff [size=1M]
> > Prefetchable memory behind bridge: None
> > Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium
> > >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> > BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B+
> > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> > Capabilities: [50] Power Management version 2
> > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> > PME(D0-,D1-,D2-,D3hot-,D3cold-)
> > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> > Bridge: PM- B3+
> > Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+
> > Address: 0000000000000000 Data: 0000
> > Capabilities: [80] Subsystem: Device 0000:0000
> > Capabilities: [90] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
> > DevCap: MaxPayload 512 bytes, PhantFunc 0
> > ExtTag- AttnBtn- AttnInd- PwrInd- RBE-
> > SlotPowerLimit 0.000W
> > DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
> > MaxPayload 128 bytes, MaxReadReq 512 bytes
> > DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq-
> > AuxPwr- TransPend-
> > LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> > Exit Latency L0s <1us, L1 <16us
> > ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> > LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
> > TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> > Capabilities: [100 v1] Advanced Error Reporting
> > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr-
> > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr-
> > AERCap: First Error Pointer: 00, ECRCGenCap+
> > ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> > HeaderLog: 00000000 00000000 00000000 00000000
> >
> > 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> > Subsystem: National Instruments PCIe-GPIB
> > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > Interrupt: pin A routed to IRQ 0
> > Region 0: Memory at 20204000 (32-bit, non-prefetchable)
> > [disabled] [size=2K]
> > Region 1: Memory at 20200000 (32-bit, non-prefetchable)
> > [disabled] [size=16K]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: PCI IRQ assignment broken from 4.9 onwards (swizzle?)
2020-10-08 8:39 ` Billy Araujo
@ 2020-10-08 15:38 ` Bjorn Helgaas
0 siblings, 0 replies; 3+ messages in thread
From: Bjorn Helgaas @ 2020-10-08 15:38 UTC (permalink / raw)
To: Billy Araujo; +Cc: linux-pci, Kishon Vijay Abraham I, linux-omap
On Thu, Oct 08, 2020 at 09:39:23AM +0100, Billy Araujo wrote:
> Hi Bjorn,
>
> Thanks for your answer. Yes, that would be my next step.
> I am also hoping the latest kernel doesn't have this issue because
> that would mean I would be stuck with version 4.9.
> Also I could discover from which patch this stopped working but that
> would be very time consuming. If I was to add some debug prints where
> should I start? Would this be done in setup-irq.c?
> In other words, is "pci_assign_irq" function responsible for all irq
> assignments?
Start by testing the latest kernel. Don't bother trying to debug it
from first principles until you know whether somebody has already
fixed it.
> On Wed, Oct 7, 2020 at 5:41 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > [+cc Kishon, linux-omap (maybe this is dra7xx-related?)]
> >
> > On Wed, Oct 07, 2020 at 10:56:33AM +0100, Billy Araujo wrote:
> > > Hi,
> > >
> > > I have been testing a TI AM57xx board and a NXP iMX8 board with a GPIB
> > > PCIe card.
> > >
> > > TI board (Phytec): https://www.phytec.com/product/phycore-am57x/
> > > NXP board (Variscite):
> > > https://www.variscite.com/product/system-on-module-som/cortex-a53-krait/var-som-mx8m-mini-nxp-i-mx8m-mini/
> > >
> > > The GPIB PCIe card has a Texas Instruments XIO2000(A)/XIO2200A PCI
> > > Express-to-PCI Bridge.
> > >
> > > Issue:
> > > I have noticed is that on Linux kernel 4.9, the Linux PCI driver
> > > assigns correctly an IRQ number:
> > >
> > > Linux am5728-phycore-rdk 4.9.41-ga962b18-BSP-Yocto-TISDK-AM57xx-PD18.1.0
> > > 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> > > Subsystem: National Instruments PCIe-GPIB
> > > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > > Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> > > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > Interrupt: pin A routed to IRQ 470
> > >
> > > On a newer kernel (this case 4.19), PCI driver doesn't assign an IRQ number.
> > >
> > > Linux am57xx-phycore-kit 4.19.79-g35d36cd54d #1 SMP PREEMPT Wed Sep 30
> > > 14:04:18 UTC 2020 armv7l GNU/Linux
> > > 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> > > Subsystem: National Instruments PCIe-GPIB
> > > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > > Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> > > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > Interrupt: pin A routed to IRQ 0
> > >
> > > Same issue happened on the NXP board, so it seems Linux related. I
> > > have tested kernels 4.14, 4.19 and 5.4.3.
> > >
> > > The IRQ is important to get the legacy interrupts working.
> > >
> > > Looking at the code there has been some refactoring of how PCI assigns
> > > IRQ number when there is a chain of bridges. I am not too familiar
> > > with how the code works but I wonder if this has affected how the PCI
> > > assignment works.
> > >
> > > Looking in setup-irq.c:
> > >
> > > /* If this device is not on the primary bus, we need to figure out
> > > which interrupt pin it will come in on. We know which slot it
> > > will come in on 'cos that slot is where the bridge is. Each
> > > time the interrupt line passes through a PCI-PCI bridge we must
> > > apply the swizzle function. */
> > >
> > > Line 44: if (hbrg->swizzle_irq)
> > >
> > > From my understanding, this "if" didn't exist in Linux kernel 4.9. If
> > > swizzle function isn't assigned in the newer kernels it just stays as
> > > 0.
> > >
> > > This might be completely unrelated as I said I have no understanding
> > > how this code is supposed to work.
> > >
> > > What I ask is if anyone has experienced any issues similar to this in
> > > these more recent kernel versions.
> >
> > Sorry for the issue, and thanks very much for the report. Is it
> > possible to test a current kernel, e.g., v5.8 or v5.9-rc8?
> >
> > My guess is this is related to the PCI controller driver; would that
> > be pci-dra7xx.c?
> >
> > > Debug output with the issue:
> > >
> > > root@am57xx-phycore-kit:~# uname -a
> > > Linux am57xx-phycore-kit 4.19.79-g35d36cd54d #1 SMP PREEMPT Wed Sep 30
> > > 14:04:18 UTC 2020 armv7l GNU/Linux
> > >
> > > root@am57xx-phycore-kit:~# lspci -vv
> > > 00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II
> > > SOC (rev 01) (prog-if 00 [Normal decode])
> > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > > ParErr+ Stepping- SERR+ FastB2B- DisINTx+
> > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > > <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > Latency: 0, Cache Line Size: 64 bytes
> > > Interrupt: pin A routed to IRQ 180
> > > Region 0: Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
> > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > I/O behind bridge: None
> > > Memory behind bridge: 20200000-202fffff [size=1M]
> > > Prefetchable memory behind bridge: None
> > > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > > <TAbort- <MAbort+ <SERR- <PERR-
> > > BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
> > > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> > > Capabilities: [40] Power Management version 3
> > > Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA
> > > PME(D0+,D1+,D2-,D3hot+,D3cold-)
> > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > > Address: 00000000ae15b000 Data: 0000
> > > Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
> > > DevCap: MaxPayload 256 bytes, PhantFunc 0
> > > ExtTag- RBE+
> > > DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
> > > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> > > MaxPayload 128 bytes, MaxReadReq 512 bytes
> > > DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> > > AuxPwr- TransPend-
> > > LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1,
> > > Exit Latency L0s <512ns, L1 <64us
> > > ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
> > > LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
> > > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> > > LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
> > > TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
> > > RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> > > PMEIntEna+ CRSVisible-
> > > RootCap: CRSVisible-
> > > RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> > > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> > > LTR-, OBFF Not Supported ARIFwd-
> > > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> > > DevCtl2: Completion Timeout: 50us to 50ms,
> > > TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> > > AtomicOpsCtl: ReqEn- EgressBlck-
> > > LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> > > Transmit Margin: Normal Operating Range,
> > > EnterModifiedCompliance- ComplianceSOS-
> > > Compliance De-emphasis: -6dB
> > > LnkSta2: Current De-emphasis Level: -3.5dB,
> > > EqualizationComplete-, EqualizationPhase1-
> > > EqualizationPhase2-, EqualizationPhase3-,
> > > LinkEqualizationRequest-
> > > Capabilities: [100 v2] Advanced Error Reporting
> > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
> > > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> > > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > > AdvNonFatalErr-
> > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > > AdvNonFatalErr+
> > > AERCap: First Error Pointer: 00, ECRCGenCap+
> > > ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> > > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> > > HeaderLog: 00000000 00000000 00000000 00000000
> > > RootCmd: CERptEn+ NFERptEn+ FERptEn+
> > > RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
> > > FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
> > > ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
> > > Kernel driver in use: pcieport
> > >
> > > 01:00.0 PCI bridge: Texas Instruments XIO2000(A)/XIO2200A PCI
> > > Express-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
> > > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > > <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
> > > I/O behind bridge: None
> > > Memory behind bridge: 20200000-202fffff [size=1M]
> > > Prefetchable memory behind bridge: None
> > > Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium
> > > >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> > > BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B+
> > > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> > > Capabilities: [50] Power Management version 2
> > > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> > > PME(D0-,D1-,D2-,D3hot-,D3cold-)
> > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> > > Bridge: PM- B3+
> > > Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+
> > > Address: 0000000000000000 Data: 0000
> > > Capabilities: [80] Subsystem: Device 0000:0000
> > > Capabilities: [90] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
> > > DevCap: MaxPayload 512 bytes, PhantFunc 0
> > > ExtTag- AttnBtn- AttnInd- PwrInd- RBE-
> > > SlotPowerLimit 0.000W
> > > DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> > > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
> > > MaxPayload 128 bytes, MaxReadReq 512 bytes
> > > DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq-
> > > AuxPwr- TransPend-
> > > LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> > > Exit Latency L0s <1us, L1 <16us
> > > ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> > > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> > > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> > > LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
> > > TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> > > Capabilities: [100 v1] Advanced Error Reporting
> > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt-
> > > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> > > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > > AdvNonFatalErr-
> > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > > AdvNonFatalErr-
> > > AERCap: First Error Pointer: 00, ECRCGenCap+
> > > ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> > > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> > > HeaderLog: 00000000 00000000 00000000 00000000
> > >
> > > 02:00.0 Communication controller: National Instruments PCIe-GPIB (rev 02)
> > > Subsystem: National Instruments PCIe-GPIB
> > > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> > > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> > > Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> > > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > Interrupt: pin A routed to IRQ 0
> > > Region 0: Memory at 20204000 (32-bit, non-prefetchable)
> > > [disabled] [size=2K]
> > > Region 1: Memory at 20200000 (32-bit, non-prefetchable)
> > > [disabled] [size=16K]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-10-08 15:38 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAEt4U6V22Abudp1TRJ2SGkYH=2o+RFWa8nrGbKDoLFXcZJV7qA@mail.gmail.com>
2020-10-07 16:41 ` PCI IRQ assignment broken from 4.9 onwards (swizzle?) Bjorn Helgaas
2020-10-08 8:39 ` Billy Araujo
2020-10-08 15:38 ` Bjorn Helgaas
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).