From: Ben Greear <greearb@candelatech.com>
To: intel-wired-lan@osuosl.org
Subject: [Intel-wired-lan] Problem when igb is forced to 10-HD on both sides.
Date: Tue, 2 Apr 2019 11:12:57 -0700 [thread overview]
Message-ID: <4e4b56a7-8695-ca9e-018c-4c54d8ba9da1@candelatech.com> (raw)
Hello,
We found a problem with igb when forcing the negotiation rates.
If I leave one side to 1GB AUTO, then I can force the other side to any
supported combination and it appears to work fine.
But, if I set both sides to 10-HD, then link will not be established. I
added a bit of debugging to the kernel and I see this in the logs.
Our user was also setting MTU to 3800, but it turns out that is not
needed to reproduce the issue.
[360212.156670] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[360212.156672] igb: Copyright (c) 2007-2014 Intel Corporation.
[360212.216114] igb 0000:01:00.0: added PHC on eth0
[360212.216116] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[360212.216118] igb 0000:01:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:30:18:01:64:77
[360212.216200] igb 0000:01:00.0: eth0: PBA No: 106300-000
[360212.216202] igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[360212.271608] igb 0000:01:00.1: added PHC on eth1
[360212.271610] igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection
[360212.271611] igb 0000:01:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:30:18:01:64:78
[360212.271694] igb 0000:01:00.1: eth1: PBA No: 106300-000
[360212.271695] igb 0000:01:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[360212.326533] igb 0000:01:00.2: added PHC on eth2
[360212.326535] igb 0000:01:00.2: Intel(R) Gigabit Ethernet Network Connection
[360212.326537] igb 0000:01:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:30:18:01:64:79
[360212.326620] igb 0000:01:00.2: eth2: PBA No: 106300-000
[360212.326621] igb 0000:01:00.2: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[360212.438974] igb 0000:01:00.3: added PHC on eth3
[360212.438977] igb 0000:01:00.3: Intel(R) Gigabit Ethernet Network Connection
[360212.438979] igb 0000:01:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:30:18:01:64:7a
[360212.439070] igb 0000:01:00.3: eth3: PBA No: 106300-000
[360212.439076] igb 0000:01:00.3: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[360212.482565] igb 0000:02:00.0: added PHC on eth4
[360212.482566] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[360212.482568] igb 0000:02:00.0: eth4: (PCIe:2.5Gb/s:Width x1) 00:30:18:01:64:7b
[360212.482569] igb 0000:02:00.0: eth4: PBA No: FFFFFF-0FF
[360212.482570] igb 0000:02:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[360215.943458] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360216.276567] igb 0000:01:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360216.493576] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360255.427240] igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360275.927145] igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360388.503634] igb 0000:01:00.2: Set Speed: 10 dplx: 0 autoneg: 0 forced-speed-duplex: 1
[360389.299798] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360391.609996] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[360391.609998] igb 0000:01:00.2: EEE Disabled: unsupported at half duplex. Re-enable using ethtool when at full duplex.
[360391.610233] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[360391.610234] igb 0000:01:00.3: EEE Disabled: unsupported at half duplex. Re-enable using ethtool when at full duplex.
[360421.400233] igb 0000:01:00.2: Set Speed: 10 dplx: 1 autoneg: 0 forced-speed-duplex: 2
[360421.513446] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360423.815595] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[360423.815928] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[360465.832992] igb 0000:01:00.2: Set Speed: 100 dplx: 0 autoneg: 0 forced-speed-duplex: 4
[360465.948361] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360468.272516] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[360468.318388] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[360486.514016] igb 0000:01:00.2: Set Speed: 100 dplx: 1 autoneg: 0 forced-speed-duplex: 8
[360486.539733] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360488.707926] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[360488.753727] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[360503.658416] igb 0000:01:00.2: Set Speed: 1000 dplx: 1 autoneg: 1 forced-speed-duplex: 8
[360503.684089] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360506.572410] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[360507.120348] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[360543.873779] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360546.701102] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[360547.204119] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[360564.547193] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360567.404614] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360567.973547] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360597.366482] igb 0000:01:00.2: changing MTU from 1500 to 3800
[360598.098996] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360601.028214] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360601.581234] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360611.266742] igb 0000:01:00.3: changing MTU from 1500 to 3800
[360611.591268] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Down
[360614.051481] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360614.605478] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360632.902730] igb 0000:01:00.2: Set Speed: 10 dplx: 0 autoneg: 0 forced-speed-duplex: 1
[360633.276708] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360635.584020] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[360635.584253] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[360695.743292] igb 0000:01:00.3: Set Speed: 10 dplx: 0 autoneg: 0 forced-speed-duplex: 1
[360695.863858] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Down
[360614.051481] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360614.605478] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[360632.902730] igb 0000:01:00.2: Set Speed: 10 dplx: 0 autoneg: 0 forced-speed-duplex: 1
[360633.276708] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Down
[360635.584020] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[360635.584253] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[360695.743292] igb 0000:01:00.3: Set Speed: 10 dplx: 0 autoneg: 0 forced-speed-duplex: 1
[360695.863858] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Down
[361049.119412] igb 0000:01:00.2: changing MTU from 3800 to 1500
[361064.275721] igb 0000:01:00.3: changing MTU from 3800 to 1500
[361106.100172] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[361106.101351] igb 0000:01:00.3 eth3: igb: eth3 NIC Link is Up 10 Mbps Half Duplex, Flow Control: None
[361120.462094] igb 0000:01:00.3: Set Speed: 10 dplx: 0 autoneg: 0 forced-speed-duplex: 1
[361120.578506] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Down
Kernel is 4.20.17+ local hacks (no significant local patches applied to igb though)
lspci output:
01:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin C routed to IRQ 18
Region 0: Memory at df720000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at e020 [size=32]
Region 3: Memory at df784000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number 00-30-18-ff-ff-01-64-77
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 02
VF offset: 128, stride: 4, Device ID: 1520
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 000000008b080000 (64-bit, prefetchable)
Region 3: Memory@000000008b0a0000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1a0 v1] Transaction Processing Hints
Device specific mode supported
Steering table in TPH capability structure
Capabilities: [1d0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: igb
Kernel modules: igb
01:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin D routed to IRQ 19
Region 0: Memory at df700000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at e000 [size=32]
Region 3: Memory at df780000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number 00-30-18-ff-ff-01-64-77
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 03
VF offset: 128, stride: 4, Device ID: 1520
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 000000008b0c0000 (64-bit, prefetchable)
Region 3: Memory at 000000008b0e0000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1a0 v1] Transaction Processing Hints
Device specific mode supported
Steering table in TPH capability structure
Capabilities: [1d0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: igb
Kernel modules: igb
I will be happy to try patches or provide other debugging. The problem is fully reproducible.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
next reply other threads:[~2019-04-02 18:12 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-02 18:12 Ben Greear [this message]
2019-04-02 18:36 ` [Intel-wired-lan] Problem when igb is forced to 10-HD on both sides Fujinaka, Todd
2019-04-02 18:50 ` Ben Greear
2019-04-02 19:03 ` Ben Greear
2019-04-02 19:20 ` Ben Greear
2019-04-02 21:19 ` Fujinaka, Todd
2019-04-02 21:29 ` Ben Greear
2019-04-02 21:33 ` Fujinaka, Todd
2019-04-02 21:41 ` Fujinaka, Todd
2019-04-03 1:51 ` Brown, Aaron F
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4e4b56a7-8695-ca9e-018c-4c54d8ba9da1@candelatech.com \
--to=greearb@candelatech.com \
--cc=intel-wired-lan@osuosl.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.