* Re: VPD access Blocked by commit 0d5370d1d85251e5893ab7c90a429464de2e140b [not found] ` <CAErSpo5qy6WuUe9cz1vTBBnc5P_uZaPzc-Yqbag2eBBxzi+ENg@mail.gmail.com> @ 2019-05-21 20:11 ` Bjorn Helgaas 2019-05-30 19:33 ` [EXT] " Himanshu Madhani 0 siblings, 1 reply; 5+ messages in thread From: Bjorn Helgaas @ 2019-05-21 20:11 UTC (permalink / raw) To: Himanshu Madhani Cc: Andrew Vasquez, Girish Basrur, Giridhar Malavali, Myron Stowe, Linux Kernel Mailing List, Quinn Tran, Linux PCI [fix linux-pci, remove ethan.zhao (bounces)] From: Bjorn Helgaas <bhelgaas@google.com> Date: Tue, May 21, 2019 at 3:02 PM To: Himanshu Madhani Cc: ethan.zhao@oracle.com, Andrew Vasquez, Girish Basrur, Giridhar Malavali, Myron Stowe, <linux-pci@kernel.org>, Linux Kernel Mailing List, Quinn Tran > [+cc Myron, Quinn, linux-pci, linux-kernel] > > From: Himanshu Madhani <hmadhani@marvell.com> > Date: Fri, May 17, 2019 at 5:21 PM > To: ethan.zhao@oracle.com, bhelgaas@google.com > Cc: Andrew Vasquez, Girish Basrur, Giridhar Malavali > > > Hi Ethan, > > > > Our OEM partners reported to us that VPD access with latest distros were returning I/O error for them. They indicated this to be issue only with newer kernels. > > > > One of the distro vendor pointed out patch posted by you to be reason for IO error trying to VPD. The patch looks like blocks access to VPD by blacklisting ISP. > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0d5370d1d85251e5893ab7c90a429464de2e140b > > > > I setup PCIe analyzer to reproduce this in our lab to root cause it and discovered that after reverting the patch. I am able to get VPD data okay with upstream 5.1.0 and I used RHEL8. > > > > I also used "lspci" and "cat" to dump out VPD data and do not see any issue. > > > > # lspci -vvv -s 03:00.0 > > 03:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) > > Subsystem: QLogic Corp. QLE2742 Dual Port 32Gb Fibre Channel to PCIe Adapter > > Physical Slot: 15 > > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > > Latency: 0, Cache Line Size: 64 bytes > > Interrupt: pin A routed to IRQ 67 > > NUMA node: 0 > > Region 0: Memory at fbe05000 (64-bit, prefetchable) [size=4K] > > Region 2: Memory at fbe02000 (64-bit, prefetchable) [size=8K] > > Region 4: Memory at fbd00000 (64-bit, prefetchable) [size=1M] > > Expansion ROM at fb540000 [disabled] [size=256K] > > Capabilities: [44] Power Management version 3 > > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) > > Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- > > Capabilities: [4c] Express (v2) Endpoint, MSI 00 > > DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <4us, L1 <1us > > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W > > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ > > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- > > MaxPayload 256 bytes, MaxReadReq 4096 bytes > > DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- > > LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <2us > > ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ > > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ > > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > > LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- > > DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported > > AtomicOpsCap: 32bit- 64bit- 128bitCAS- > > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled > > AtomicOpsCtl: ReqEn- > > LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- > > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > > Compliance De-emphasis: -6dB > > LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ > > EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- > > Capabilities: [88] Vital Product Data > > Product Name: QLogic 32Gb 2-port FC to PCIe Gen3 x8 Adapter > > Read-only fields: > > [PN] Part number: QLE2742 > > [SN] Serial number: RFD1706R22611 > > [EC] Engineering changes: BK3210408-05 04 > > [V9] Vendor specific: 010189 > > [RV] Reserved: checksum good, 0 byte(s) reserved > > End > > Capabilities: [90] MSI-X: Enable+ Count=16 Masked- > > Vector table: BAR=2 offset=00000000 > > PBA: BAR=2 offset=00001000 > > Capabilities: [9c] Vendor Specific Information: Len=0c <?> > > Capabilities: [100 v1] Advanced Error Reporting > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ > > AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- > > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- > > HeaderLog: 00000000 00000000 00000000 00000000 > > Capabilities: [154 v1] Alternative Routing-ID Interpretation (ARI) > > ARICap: MFVC- ACS-, Next Function: 1 > > ARICtl: MFVC- ACS-, Function Group: 0 > > Capabilities: [1c0 v1] #19 > > Capabilities: [1f4 v1] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?> > > Kernel driver in use: qla2xxx > > Kernel modules: qla2xxx > > > > # cat /sys/bus/pci/devices/0000\:03\:00.0/vpd > > RFD1706R22611ECBK3210408-05 04V9010189RV�x > > > > Can you share some more insight into where you encountered issue? I am in process of reverting this patch from upstream kernel but wanted to reach out and find out if you still have setup to provide more context. > > 0d5370d1d852 ("PCI: Prevent VPD access for QLogic ISP2722") prevented > a panic while reading VPD, so we can't simply revert it. > > Since you don't see a panic while reading VPD from that device, it's > possible that a QLogic firmware change fixed the VPD format so Linux > no longer reads the area that caused the problem. Or possibly your > system doesn't handle the config read error the same way Ethan's HP > DL380 does. Unfortunately we don't have an actual PCIe analyzer trace > from Ethan's system, so we don't know exactly what happened on PCIe. > > I suggest that you capture the entire VPD area and hexdump it, e.g., > with "xxd", and look at its structure. pci_vpd_size() parses it and > computes the valid size based on a PCI_VPD_STIN_END tag, and > pci_vpd_read() should not read past that size. > > And you *do* have an analyzer trace. If new QLogic firmware fixed the > VPD format, the trace should show that Linux read only the valid part > of VPD, and there should be no errors in the trace. Then it might > just be a question of tweaking the quirk so it allows VPD reads if the > firmware is new enough. > > But if the trace does show config reads with errors, then it might be > that your system just tolerates the errors while the DL380 did not. > Then we'd have to figure out exactly what the error was and how to > deal with it so things work on both your system and Ethan's. > > Bjorn ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [EXT] VPD access Blocked by commit 0d5370d1d85251e5893ab7c90a429464de2e140b 2019-05-21 20:11 ` VPD access Blocked by commit 0d5370d1d85251e5893ab7c90a429464de2e140b Bjorn Helgaas @ 2019-05-30 19:33 ` Himanshu Madhani 2019-05-30 20:58 ` Bjorn Helgaas 0 siblings, 1 reply; 5+ messages in thread From: Himanshu Madhani @ 2019-05-30 19:33 UTC (permalink / raw) To: Bjorn Helgaas Cc: Andrew Vasquez, Girish Basrur, Giridhar Malavali, Myron Stowe, Linux Kernel Mailing List, Quinn Tran, Linux PCI Hi Bjorn, Thank you for responding to my question. > On May 21, 2019, at 1:11 PM, Bjorn Helgaas <bhelgaas@google.com> wrote: > > External Email > > ---------------------------------------------------------------------- > [fix linux-pci, remove ethan.zhao (bounces)] > > From: Bjorn Helgaas <bhelgaas@google.com> > Date: Tue, May 21, 2019 at 3:02 PM > To: Himanshu Madhani > Cc: ethan.zhao@oracle.com, Andrew Vasquez, Girish Basrur, Giridhar > Malavali, Myron Stowe, <linux-pci@kernel.org>, Linux Kernel Mailing > List, Quinn Tran > >> [+cc Myron, Quinn, linux-pci, linux-kernel] >> >> From: Himanshu Madhani <hmadhani@marvell.com> >> Date: Fri, May 17, 2019 at 5:21 PM >> To: ethan.zhao@oracle.com, bhelgaas@google.com >> Cc: Andrew Vasquez, Girish Basrur, Giridhar Malavali >> >>> Hi Ethan, >>> >>> Our OEM partners reported to us that VPD access with latest distros were returning I/O error for them. They indicated this to be issue only with newer kernels. >>> >>> One of the distro vendor pointed out patch posted by you to be reason for IO error trying to VPD. The patch looks like blocks access to VPD by blacklisting ISP. >>> >>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0d5370d1d85251e5893ab7c90a429464de2e140b >>> >>> I setup PCIe analyzer to reproduce this in our lab to root cause it and discovered that after reverting the patch. I am able to get VPD data okay with upstream 5.1.0 and I used RHEL8. >>> >>> I also used "lspci" and "cat" to dump out VPD data and do not see any issue. >>> >>> # lspci -vvv -s 03:00.0 >>> 03:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) >>> Subsystem: QLogic Corp. QLE2742 Dual Port 32Gb Fibre Channel to PCIe Adapter >>> Physical Slot: 15 >>> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- >>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- >>> Latency: 0, Cache Line Size: 64 bytes >>> Interrupt: pin A routed to IRQ 67 >>> NUMA node: 0 >>> Region 0: Memory at fbe05000 (64-bit, prefetchable) [size=4K] >>> Region 2: Memory at fbe02000 (64-bit, prefetchable) [size=8K] >>> Region 4: Memory at fbd00000 (64-bit, prefetchable) [size=1M] >>> Expansion ROM at fb540000 [disabled] [size=256K] >>> Capabilities: [44] Power Management version 3 >>> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) >>> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- >>> Capabilities: [4c] Express (v2) Endpoint, MSI 00 >>> DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <4us, L1 <1us >>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W >>> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ >>> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- >>> MaxPayload 256 bytes, MaxReadReq 4096 bytes >>> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- >>> LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <2us >>> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ >>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ >>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- >>> LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- >>> DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported >>> AtomicOpsCap: 32bit- 64bit- 128bitCAS- >>> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled >>> AtomicOpsCtl: ReqEn- >>> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- >>> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- >>> Compliance De-emphasis: -6dB >>> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ >>> EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- >>> Capabilities: [88] Vital Product Data >>> Product Name: QLogic 32Gb 2-port FC to PCIe Gen3 x8 Adapter >>> Read-only fields: >>> [PN] Part number: QLE2742 >>> [SN] Serial number: RFD1706R22611 >>> [EC] Engineering changes: BK3210408-05 04 >>> [V9] Vendor specific: 010189 >>> [RV] Reserved: checksum good, 0 byte(s) reserved >>> End >>> Capabilities: [90] MSI-X: Enable+ Count=16 Masked- >>> Vector table: BAR=2 offset=00000000 >>> PBA: BAR=2 offset=00001000 >>> Capabilities: [9c] Vendor Specific Information: Len=0c <?> >>> Capabilities: [100 v1] Advanced Error Reporting >>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >>> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- >>> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- >>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ >>> AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- >>> MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- >>> HeaderLog: 00000000 00000000 00000000 00000000 >>> Capabilities: [154 v1] Alternative Routing-ID Interpretation (ARI) >>> ARICap: MFVC- ACS-, Next Function: 1 >>> ARICtl: MFVC- ACS-, Function Group: 0 >>> Capabilities: [1c0 v1] #19 >>> Capabilities: [1f4 v1] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?> >>> Kernel driver in use: qla2xxx >>> Kernel modules: qla2xxx >>> >>> # cat /sys/bus/pci/devices/0000\:03\:00.0/vpd >>> RFD1706R22611ECBK3210408-05 04V9010189RV�x >>> >>> Can you share some more insight into where you encountered issue? I am in process of reverting this patch from upstream kernel but wanted to reach out and find out if you still have setup to provide more context. >> >> 0d5370d1d852 ("PCI: Prevent VPD access for QLogic ISP2722") prevented >> a panic while reading VPD, so we can't simply revert it. >> >> Since you don't see a panic while reading VPD from that device, it's >> possible that a QLogic firmware change fixed the VPD format so Linux >> no longer reads the area that caused the problem. Or possibly your >> system doesn't handle the config read error the same way Ethan's HP >> DL380 does. Unfortunately we don't have an actual PCIe analyzer trace >> from Ethan's system, so we don't know exactly what happened on PCIe. >> >> I suggest that you capture the entire VPD area and hexdump it, e.g., >> with "xxd", and look at its structure. pci_vpd_size() parses it and >> computes the valid size based on a PCI_VPD_STIN_END tag, and >> pci_vpd_read() should not read past that size. >> >> And you *do* have an analyzer trace. If new QLogic firmware fixed the >> VPD format, the trace should show that Linux read only the valid part >> of VPD, and there should be no errors in the trace. Then it might >> just be a question of tweaking the quirk so it allows VPD reads if the >> firmware is new enough. >> >> But if the trace does show config reads with errors, then it might be >> that your system just tolerates the errors while the DL380 did not. >> Then we'd have to figure out exactly what the error was and how to >> deal with it so things work on both your system and Ethan's. >> >> Bjorn I have verified after reverting patch on HP DL380 Gen10 hardware by capturing PCIe trace Here’s snippet from dmidecode output System Information: Manufacturer: HPE Product Name: ProLiant DL380 Gen10 Family: ProLiant We are able to successfully read VPD config data using lspci and cat command # lspci -d1077: 13:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) 13:00.1 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) [root@R8 ~]# lspci -vvv -s 13:00.0 13:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) Subsystem: QLogic Corp. QLE2742 Dual Port 32Gb Fibre Channel to PCIe Adapter Physical Slot: 3 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 28 NUMA node: 0 Region 0: Memory at de605000 (64-bit, prefetchable) [size=4K] Region 2: Memory at de602000 (64-bit, prefetchable) [size=8K] Region 4: Memory at de500000 (64-bit, prefetchable) [size=1M] [virtual] Expansion ROM at de800000 [disabled] [size=256K] Capabilities: [44] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [4c] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <4us, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 4096 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <2us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [88] Vital Product Data Product Name: QLogic 32Gb 2-port FC to PCIe Gen3 x8 Adapter Read-only fields: [PN] Part number: QLE2742 [SN] Serial number: AFD1533Y02999 [EC] Engineering changes: BK3210407-05 03 [V9] Vendor specific: 010189 [RV] Reserved: checksum good, 0 byte(s) reserved End Capabilities: [90] MSI-X: Enable+ Count=16 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00001000 Capabilities: [9c] Vendor Specific Information: Len=0c <?> Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [154 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 1 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [1c0 v1] #19 Capabilities: [1f4 v1] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?> Kernel driver in use: qla2xxx Kernel modules: qla2xxx We also verified this same configuration on a SuperMicro X10SRA-F server (which i had sent in earlier email)’ and were able to verify that the VPD read was good and there were no errors on PCIe trace. Given this information, Please consider reverting the patch until we further debug the issue and resolve as it is affecting general availability of our adapter. Thanks, Himanshu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [EXT] VPD access Blocked by commit 0d5370d1d85251e5893ab7c90a429464de2e140b 2019-05-30 19:33 ` [EXT] " Himanshu Madhani @ 2019-05-30 20:58 ` Bjorn Helgaas 2019-06-03 21:30 ` Himanshu Madhani 0 siblings, 1 reply; 5+ messages in thread From: Bjorn Helgaas @ 2019-05-30 20:58 UTC (permalink / raw) To: Himanshu Madhani Cc: Andrew Vasquez, Girish Basrur, Giridhar Malavali, Myron Stowe, Linux Kernel Mailing List, Quinn Tran, Linux PCI On Thu, May 30, 2019 at 07:33:01PM +0000, Himanshu Madhani wrote: > We are able to successfully read VPD config data using lspci and cat > command Yes, you mentioned that in the very first email. I was hoping you would include the actual data, e.g., "cat vpd | xxd". That would help us figure out why you don't see the panic any more. I suspect either: - new QLogic firmware fixed the structure of the VPD data so Linux no longer attempts to read past the end of the implemented region, or, - we still read past the end of the implemented VPD region, but the device doesn't report an error or the platform deals with the error without causing a panic. > We also verified this same configuration on a SuperMicro X10SRA-F > server (which i had sent in earlier email)’ and were able to verify > that the VPD read was good and there were no errors on PCIe trace. Since you saw no PCIe errors here, this suggests that new firmware has changed the format of the VPD data. > Given this information, Please consider reverting the patch until we > further debug the issue and resolve as it is affecting general > availability of our adapter. 1) The way Linux works is that you would post a patch that does the revert you'd like to see done. 2) It's unlikely that a simple revert of 0d5370d1d852 ("PCI: Prevent VPD access for QLogic ISP2722") is the right answer because that would make Ethan's machine panic again. It's possible that a QLogic firmware update would avoid the panic, but we can't simply revert the patch and force users to do that update. If a QLogic firmware update indeed fixed the VPD format, I suggest that you ask the folks responsible for the firmware to identify the specific version where that was fixed and how the OS can figure that out. Then you could make a new quirk specific to this device that allows VPD reads if the adapter has new enough firmware. If it finds older firmware, it could even print a message suggesting that users could update the firmware if they need to read VPD data. Bjorn ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [EXT] VPD access Blocked by commit 0d5370d1d85251e5893ab7c90a429464de2e140b 2019-05-30 20:58 ` Bjorn Helgaas @ 2019-06-03 21:30 ` Himanshu Madhani 2019-06-03 22:40 ` Bjorn Helgaas 0 siblings, 1 reply; 5+ messages in thread From: Himanshu Madhani @ 2019-06-03 21:30 UTC (permalink / raw) To: Bjorn Helgaas Cc: Andrew Vasquez, Girish Basrur, Giridhar Malavali, Myron Stowe, Linux Kernel Mailing List, Quinn Tran, Linux PCI Hi Bjorn, > On May 30, 2019, at 1:58 PM, Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Thu, May 30, 2019 at 07:33:01PM +0000, Himanshu Madhani wrote: > >> We are able to successfully read VPD config data using lspci and cat >> command > > Yes, you mentioned that in the very first email. I was hoping you > would include the actual data, e.g., "cat vpd | xxd". That would help > us figure out why you don't see the panic any more. I suspect either: > Missed the request for xxd output. I got access back today for the system and captured it for you # cat /sys/class/pci_bus/0000\:13/device/0000\:13\:00.0/vpd | xxd 00000000: 822d 0051 4c6f 6769 6320 3332 4762 2032 .-.QLogic 32Gb 2 00000010: 2d70 6f72 7420 4643 2074 6f20 5043 4965 -port FC to PCIe 00000020: 2047 656e 3320 7838 2041 6461 7074 6572 Gen3 x8 Adapter 00000030: 9039 0050 4e07 514c 4532 3734 3253 4e0d .9.PN.QLE2742SN. 00000040: 4146 4431 3533 3359 3032 3939 3945 430f AFD1533Y02999EC. 00000050: 424b 3332 3130 3430 372d 3035 2030 3356 BK3210407-05 03V 00000060: 3906 3031 3031 3839 5256 01a0 78 9.010189RV..x PCIe trace also confirmed there are no READ errors. (if you need i can attach .pex file for review) > - new QLogic firmware fixed the structure of the VPD data so Linux > no longer attempts to read past the end of the implemented region, > or, > > - we still read past the end of the implemented VPD region, but the > device doesn't report an error or the platform deals with the > error without causing a panic. > >> We also verified this same configuration on a SuperMicro X10SRA-F >> server (which i had sent in earlier email)’ and were able to verify >> that the VPD read was good and there were no errors on PCIe trace. > > Since you saw no PCIe errors here, this suggests that new firmware has > changed the format of the VPD data. > >> Given this information, Please consider reverting the patch until we >> further debug the issue and resolve as it is affecting general >> availability of our adapter. > > 1) The way Linux works is that you would post a patch that does the > revert you'd like to see done. > Correct. I was trying get your buy-in before i send out patch. > 2) It's unlikely that a simple revert of 0d5370d1d852 ("PCI: Prevent > VPD access for QLogic ISP2722") is the right answer because that would > make Ethan's machine panic again. It's possible that a QLogic > firmware update would avoid the panic, but we can't simply revert the > patch and force users to do that update. > I did reached out to Oracle to help locate original card where Ethan had issue and i learned that he is no longer with Oracle. > If a QLogic firmware update indeed fixed the VPD format, I suggest > that you ask the folks responsible for the firmware to identify the > specific version where that was fixed and how the OS can figure that > out. > Still waiting on this data. > Then you could make a new quirk specific to this device that allows > VPD reads if the adapter has new enough firmware. If it finds older > firmware, it could even print a message suggesting that users could > update the firmware if they need to read VPD data. > > Bjorn Since major OEMs are having issues using adapter to extract VPD data, We would like to get them relief first and then approach this issue with more detailed fix if needed. Thanks, Himanshu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [EXT] VPD access Blocked by commit 0d5370d1d85251e5893ab7c90a429464de2e140b 2019-06-03 21:30 ` Himanshu Madhani @ 2019-06-03 22:40 ` Bjorn Helgaas 0 siblings, 0 replies; 5+ messages in thread From: Bjorn Helgaas @ 2019-06-03 22:40 UTC (permalink / raw) To: Himanshu Madhani Cc: Andrew Vasquez, Girish Basrur, Giridhar Malavali, Myron Stowe, Linux Kernel Mailing List, Quinn Tran, Linux PCI On Mon, Jun 03, 2019 at 09:30:50PM +0000, Himanshu Madhani wrote: > On May 30, 2019, at 1:58 PM, Bjorn Helgaas <helgaas@kernel.org> wrote: >> On Thu, May 30, 2019 at 07:33:01PM +0000, Himanshu Madhani wrote: >>> We are able to successfully read VPD config data using lspci and cat >>> command > Missed the request for xxd output. I got access back today for the system > and captured it for you > > # cat /sys/class/pci_bus/0000\:13/device/0000\:13\:00.0/vpd | xxd > 00000000: 822d 0051 4c6f 6769 6320 3332 4762 2032 .-.QLogic 32Gb 2 > 00000010: 2d70 6f72 7420 4643 2074 6f20 5043 4965 -port FC to PCIe > 00000020: 2047 656e 3320 7838 2041 6461 7074 6572 Gen3 x8 Adapter > 00000030: 9039 0050 4e07 514c 4532 3734 3253 4e0d .9.PN.QLE2742SN. > 00000040: 4146 4431 3533 3359 3032 3939 3945 430f AFD1533Y02999EC. > 00000050: 424b 3332 3130 3430 372d 3035 2030 3356 BK3210407-05 03V > 00000060: 3906 3031 3031 3839 5256 01a0 78 9.010189RV..x > > PCIe trace also confirmed there are no READ errors. > (if you need i can attach .pex file for review) Thank you! It would be really excellent to have a report at https://bugzilla.kernel.org with these details (hex VPD dump, .pex file, QLogic firmware version info) attached. Your patch commit log could then include the bugzilla URL. If we had this sort of information for Ethan's original patch, we would have a good start at making a smarter quirk. But we don't, so I think all we can assume at this point is that all QLogic firmware older than your current version is broken and we shouldn't try reading VPD. >> If a QLogic firmware update indeed fixed the VPD format, I suggest >> that you ask the folks responsible for the firmware to identify the >> specific version where that was fixed and how the OS can figure that >> out. > Still waiting on this data. Don't hold your breath :) > Since major OEMs are having issues using adapter to extract VPD data, We > would like to get them relief first and then approach this issue with more > detailed fix if needed. I don't think it's a good idea to simply revert 0d5370d1d852. That would mean any users that have the same QLogic firmware version Ethan had would start seeing panics. But I think you could certainly make a quirk that allows VPD access for the firmware version you have on your card (or newer), leaving the original "no VPD at all" behavior for older versions. Bjorn ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-06-03 22:40 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <B5B745A3-96B4-46ED-8F3F-D3636A96057F@marvell.com>
[not found] ` <CAErSpo5qy6WuUe9cz1vTBBnc5P_uZaPzc-Yqbag2eBBxzi+ENg@mail.gmail.com>
2019-05-21 20:11 ` VPD access Blocked by commit 0d5370d1d85251e5893ab7c90a429464de2e140b Bjorn Helgaas
2019-05-30 19:33 ` [EXT] " Himanshu Madhani
2019-05-30 20:58 ` Bjorn Helgaas
2019-06-03 21:30 ` Himanshu Madhani
2019-06-03 22:40 ` Bjorn Helgaas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).