* e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) @ 2012-04-19 23:27 Ben Greear 2012-04-20 2:39 ` Tom Herbert 2012-04-20 6:46 ` Dave, Tushar N 0 siblings, 2 replies; 17+ messages in thread From: Ben Greear @ 2012-04-19 23:27 UTC (permalink / raw) To: netdev, e1000-devel list, therbert Test case: Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce) reset peer interface ----> tx queue timeout Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, re-enabling all cpus for irq balancing Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------ Apr 19 16:12:55 localhost kernel: WARNING: at /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 dev_watchdog+0xf4/0x154() Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw 8021q garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801 i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: nf_nat] Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted 3.2.0-rc2+ #36 Apr 19 16:12:55 localhost kernel: Call Trace: Apr 19 16:12:55 localhost kernel: <IRQ> [<ffffffff81042902>] warn_slowpath_common+0x80/0x98 Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>] warn_slowpath_fmt+0x41/0x43 Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>] dev_watchdog+0xf4/0x154 Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>] run_timer_softirq+0x16f/0x201 Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? netif_tx_unlock+0x57/0x57 Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>] __do_softirq+0x86/0x12f Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ? hrtimer_interrupt+0x12b/0x1bd Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>] call_softirq+0x1c/0x30 Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] smp_apic_timer_interrupt+0x85/0x93 Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] apic_timer_interrupt+0x6e/0x80 Apr 19 16:12:55 localhost kernel: <EOI> [<ffffffff81010b8c>] ? mwait_idle+0x6e/0x8c Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ? mwait_idle+0x61/0x8c Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>] start_secondary+0x194/0x199 Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses: 1 Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating dump directories Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04-19-16:12:57-898-0' creation detected Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt Apr 19 16:12:57 localhost abrtd: Can't open file '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or directory Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 Apr 19 16:12:57 localhost abrtd: Deleting dump directory oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), sending dbus signal Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, re-enabling all cpus for irq balancing Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194 Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready .... lspci: 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 74 Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32] [virtual] Expansion ROM at d8d00000 [disabled] [size=128K] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000feeff00c Data: 41a3 Capabilities: [e0] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, Latency L0 <4us, L1 <64us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e Kernel driver in use: e1000e Kernel modules: e1000e 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c Author: Tom Herbert <therbert@google.com> Date: Mon Nov 28 16:33:16 2011 +0000 e1000e: Support for byte queue limits Changes to e1000e to use byte queue limits. Signed-off-by: Tom Herbert <therbert@google.com> Acked-by: Eric Dumazet <eric.dumazet@gmail.com> Signed-off-by: David S. Miller <davem@davemloft.net> :040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe 51914e221547b95a989b5c7e9b037c9370fd734e M drivers Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-19 23:27 e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) Ben Greear @ 2012-04-20 2:39 ` Tom Herbert 2012-04-20 6:44 ` Ying Cai 2012-04-20 19:00 ` Ben Greear 2012-04-20 6:46 ` Dave, Tushar N 1 sibling, 2 replies; 17+ messages in thread From: Tom Herbert @ 2012-04-20 2:39 UTC (permalink / raw) To: Ben Greear; +Cc: netdev, e1000-devel list Thanks, will try to reproduce. Tom On Thu, Apr 19, 2012 at 4:27 PM, Ben Greear <greearb@candelatech.com> wrote: > Test case: > > Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic > (moderate speeds of traffic has issues as well, maybe not as easy to > reproduce) > reset peer interface > ----> tx queue timeout > > > Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down > Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter > Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down > Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full > Duplex, Flow Control: Rx/Tx > Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link > becomes ready > Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full > Duplex, Flow Control: Rx/Tx > Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link > becomes ready > Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, > re-enabling all cpus for irq balancing > Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------ > Apr 19 16:12:55 localhost kernel: WARNING: at > /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 > dev_watchdog+0xf4/0x154() > Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU > Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit > queue 0 timed out > Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw 8021q > garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid > coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma > i2c_i801 > i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy > radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: > nf_nat] > Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted > 3.2.0-rc2+ #36 > Apr 19 16:12:55 localhost kernel: Call Trace: > Apr 19 16:12:55 localhost kernel: <IRQ> [<ffffffff81042902>] > warn_slowpath_common+0x80/0x98 > Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>] > warn_slowpath_fmt+0x41/0x43 > Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>] > dev_watchdog+0xf4/0x154 > Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>] > run_timer_softirq+0x16f/0x201 > Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? > netif_tx_unlock+0x57/0x57 > Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>] > __do_softirq+0x86/0x12f > Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ? > hrtimer_interrupt+0x12b/0x1bd > Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>] > call_softirq+0x1c/0x30 > Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e > Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb > Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] > smp_apic_timer_interrupt+0x85/0x93 > Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] > apic_timer_interrupt+0x6e/0x80 > Apr 19 16:12:55 localhost kernel: <EOI> [<ffffffff81010b8c>] ? > mwait_idle+0x6e/0x8c > Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ? > mwait_idle+0x61/0x8c > Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe > Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>] > start_secondary+0x194/0x199 > Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- > Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter > Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses: > 1 > Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating dump > directories > Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04-19-16:12:57-898-0' > creation detected > Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt > Apr 19 16:12:57 localhost abrtd: Can't open file > '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or > directory > Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: > /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 > Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of > /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 > Apr 19 16:12:57 localhost abrtd: Deleting dump directory > oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), > sending dbus signal > Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full > Duplex, Flow Control: Rx/Tx > Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link > becomes ready > Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, > re-enabling all cpus for irq balancing > Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter > Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194 > Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full > Duplex, Flow Control: Rx/Tx > Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link > becomes ready > .... > > lspci: > > 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet > Controller (rev 06) > Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ > Stepping- SERR+ FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 32 bytes > Interrupt: pin A routed to IRQ 74 > Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K] > Region 2: I/O ports at 3000 [size=32] > [virtual] Expansion ROM at d8d00000 [disabled] [size=128K] > Capabilities: [c8] Power Management version 2 > Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > PME(D0+,D1-,D2-,D3hot+,D3cold-) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- > Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Address: 00000000feeff00c Data: 41a3 > Capabilities: [e0] Express (v1) Endpoint, MSI 00 > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s > <512ns, L1 <64us > ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ > Unsupported+ > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 4096 bytes > DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- > TransPend- > LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, > Latency L0 <4us, L1 <64us > ClockPM- Surprise- LLActRep- BwNot- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- > CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ > DLActive- BWMgmt- ABWMgmt- > Capabilities: [100 v1] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- > ChkEn- > Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e > Kernel driver in use: e1000e > Kernel modules: e1000e > > > 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit > commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c > Author: Tom Herbert <therbert@google.com> > Date: Mon Nov 28 16:33:16 2011 +0000 > > e1000e: Support for byte queue limits > > Changes to e1000e to use byte queue limits. > > Signed-off-by: Tom Herbert <therbert@google.com> > Acked-by: Eric Dumazet <eric.dumazet@gmail.com> > Signed-off-by: David S. Miller <davem@davemloft.net> > > :040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe > 51914e221547b95a989b5c7e9b037c9370fd734e M drivers > > > Thanks, > Ben > > -- > Ben Greear <greearb@candelatech.com> > Candela Technologies Inc http://www.candelatech.com > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 2:39 ` Tom Herbert @ 2012-04-20 6:44 ` Ying Cai 2012-04-20 19:00 ` Ben Greear 1 sibling, 0 replies; 17+ messages in thread From: Ying Cai @ 2012-04-20 6:44 UTC (permalink / raw) To: Tom Herbert; +Cc: e1000-devel list, netdev [-- Attachment #1.1: Type: text/plain, Size: 12102 bytes --] No, we have seen it in our test. I just checked all our Ilium machines' /var/log/messages back to earlier to month. We got numerous "tg3 transmit timed out" on Ilium GLAG machines only, no e1000e transmit queue timed out. The "tg3 transmit queue timed out" does not seem to be caused by BQL, since I can see it happen even in 420 kernel! Apr 13 18:36:08 lpb77 kernel: [ 51.684212] ------------[ cut here ]------------ Apr 13 18:36:08 lpb77 kernel: [ 51.684223] WARNING: at net/sched/sch_generic.c:263 dev_watchdog+0x1c8/0x1f0() Apr 13 18:36:08 lpb77 kernel: [ 51.684226] Hardware name: Greencreek,ESB2 Apr 13 18:36:08 lpb77 kernel: [ 51.684228] NETDEV WATCHDOG: eth4 (tg3): transmit queue 0 timed out Apr 13 18:36:08 lpb77 kernel: [ 51.684230] Modules linked in: bonding sata_mv i2c_imch pca954x i2c_virtual uhci_hcd msr cpuid i2c_dev i2c_i801 i2c_core i2c_debug tg3 e1000e ipv6 genrtc Apr 13 18:36:08 lpb77 kernel: [ 51.684246] Pid: 0, comm: swapper Not tainted 2.6.34-smp-420.16 #1 Apr 13 18:36:08 lpb77 kernel: [ 51.684248] Call Trace: Apr 13 18:36:08 lpb77 kernel: [ 51.684251] <IRQ> [<ffffffff811dc939>] warn_slowpath_common+0x7c/0xce Apr 13 18:36:08 lpb77 kernel: [ 51.684261] [<ffffffff811dc9e2>] warn_slowpath_fmt+0x41/0x43 Apr 13 18:36:08 lpb77 kernel: [ 51.684265] [<ffffffff811494d8>] dev_watchdog+0x1c8/0x1f0 Apr 13 18:36:08 lpb77 kernel: [ 51.684270] [<ffffffff811f14a5>] ? sched_clock_local+0x1c/0x82 Apr 13 18:36:08 lpb77 kernel: [ 51.684275] [<ffffffff8101826c>] run_timer_softirq+0x34c/0x360 Apr 13 18:36:08 lpb77 kernel: [ 51.684278] [<ffffffff81149310>] ? dev_watchdog+0x0/0x1f0 Apr 13 18:36:08 lpb77 kernel: [ 51.684282] [<ffffffff8101780e>] __do_softirq+0x39e/0x490 Apr 13 18:36:08 lpb77 kernel: [ 51.684316] [<ffffffff81020360>] ? tick_program_event+0x60/0x120 Apr 13 18:36:08 lpb77 kernel: [ 51.684319] [<ffffffff811f14a5>] ? sched_clock_local+0x1c/0x82 Apr 13 18:36:08 lpb77 kernel: [ 51.684323] [<ffffffff8107770c>] call_softirq+0x1c/0x30 Apr 13 18:36:08 lpb77 kernel: [ 51.684327] [<ffffffff810009a2>] do_softirq+0x42/0x80 Apr 13 18:36:08 lpb77 kernel: [ 51.684329] [<ffffffff81017159>] irq_exit+0x49/0xa0 Apr 13 18:36:08 lpb77 kernel: [ 51.684334] [<ffffffff814ac184>] smp_apic_timer_interrupt+0x84/0xf4 Apr 13 18:36:08 lpb77 kernel: [ 51.684338] [<ffffffff810771d3>] apic_timer_interrupt+0x13/0x20 Apr 13 18:36:08 lpb77 kernel: [ 51.684339] <EOI> [<ffffffff811b5709>] ? mwait_idle+0x7a/0x81 Apr 13 18:36:08 lpb77 kernel: [ 51.684355] [<ffffffff811b56bb>] ? mwait_idle+0x2c/0x81 Apr 13 18:36:08 lpb77 kernel: [ 51.684358] [<ffffffff810005c2>] cpu_idle+0x92/0x140 Apr 13 18:36:08 lpb77 kernel: [ 51.684363] [<ffffffff81b80a53>] start_secondary+0x1c3/0x1c7 Apr 13 18:36:08 lpb77 kernel: [ 51.684373] ---[ end trace fc8464e86676a1b7 ]--- On Thu, Apr 19, 2012 at 7:39 PM, Tom Herbert <therbert@google.com> wrote: > Thanks, will try to reproduce. > > Tom > > On Thu, Apr 19, 2012 at 4:27 PM, Ben Greear <greearb@candelatech.com> > wrote: > > Test case: > > > > Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic > > (moderate speeds of traffic has issues as well, maybe not as easy to > > reproduce) > > reset peer interface > > ----> tx queue timeout > > > > > > Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down > > Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset > adapter > > Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down > > Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps > Full > > Duplex, Flow Control: Rx/Tx > > Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link > > becomes ready > > Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps > Full > > Duplex, Flow Control: Rx/Tx > > Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link > > becomes ready > > Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, > > re-enabling all cpus for irq balancing > > Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------ > > Apr 19 16:12:55 localhost kernel: WARNING: at > > /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 > > dev_watchdog+0xf4/0x154() > > Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU > > Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): > transmit > > queue 0 timed out > > Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw > 8021q > > garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen > iscsi_tcp > > libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf > hwmon_vid > > coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma > > i2c_i801 > > i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy > > radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: > > nf_nat] > > Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted > > 3.2.0-rc2+ #36 > > Apr 19 16:12:55 localhost kernel: Call Trace: > > Apr 19 16:12:55 localhost kernel: <IRQ> [<ffffffff81042902>] > > warn_slowpath_common+0x80/0x98 > > Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>] > > warn_slowpath_fmt+0x41/0x43 > > Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>] > > dev_watchdog+0xf4/0x154 > > Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>] > > run_timer_softirq+0x16f/0x201 > > Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? > > netif_tx_unlock+0x57/0x57 > > Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>] > > __do_softirq+0x86/0x12f > > Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ? > > hrtimer_interrupt+0x12b/0x1bd > > Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>] > > call_softirq+0x1c/0x30 > > Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] > do_softirq+0x41/0x7e > > Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb > > Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] > > smp_apic_timer_interrupt+0x85/0x93 > > Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] > > apic_timer_interrupt+0x6e/0x80 > > Apr 19 16:12:55 localhost kernel: <EOI> [<ffffffff81010b8c>] ? > > mwait_idle+0x6e/0x8c > > Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ? > > mwait_idle+0x61/0x8c > > Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe > > Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>] > > start_secondary+0x194/0x199 > > Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- > > Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset > adapter > > Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found > oopses: > > 1 > > Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating > dump > > directories > > Apr 19 16:12:57 localhost abrtd: Directory > 'oops-2012-04-19-16:12:57-898-0' > > creation detected > > Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to > Abrt > > Apr 19 16:12:57 localhost abrtd: Can't open file > > '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or > > directory > > Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: > > /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 > > Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of > > /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 > > Apr 19 16:12:57 localhost abrtd: Deleting dump directory > > oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), > > sending dbus signal > > Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps > Full > > Duplex, Flow Control: Rx/Tx > > Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link > > becomes ready > > Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, > > re-enabling all cpus for irq balancing > > Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset > adapter > > Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194 > > Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps > Full > > Duplex, Flow Control: Rx/Tx > > Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link > > becomes ready > > .... > > > > lspci: > > > > 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet > > Controller (rev 06) > > Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- > ParErr+ > > Stepping- SERR+ FastB2B- DisINTx+ > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > > <TAbort- <MAbort- >SERR- <PERR- INTx- > > Latency: 0, Cache Line Size: 32 bytes > > Interrupt: pin A routed to IRQ 74 > > Region 0: Memory at d8300000 (32-bit, non-prefetchable) > [size=128K] > > Region 2: I/O ports at 3000 [size=32] > > [virtual] Expansion ROM at d8d00000 [disabled] [size=128K] > > Capabilities: [c8] Power Management version 2 > > Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > > PME(D0+,D1-,D2-,D3hot+,D3cold-) > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- > > Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > > Address: 00000000feeff00c Data: 41a3 > > Capabilities: [e0] Express (v1) Endpoint, MSI 00 > > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s > > <512ns, L1 <64us > > ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- > > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ > > Unsupported+ > > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ > > MaxPayload 128 bytes, MaxReadReq 4096 bytes > > DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- > > TransPend- > > LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, > > Latency L0 <4us, L1 <64us > > ClockPM- Surprise- LLActRep- BwNot- > > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- > > CommClk- > > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > > LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ > > DLActive- BWMgmt- ABWMgmt- > > Capabilities: [100 v1] Advanced Error Reporting > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > > NonFatalErr- > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > > NonFatalErr- > > AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- > > ChkEn- > > Capabilities: [140 v1] Device Serial Number > 00-e0-ed-ff-ff-0c-11-6e > > Kernel driver in use: e1000e > > Kernel modules: e1000e > > > > > > 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit > > commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c > > Author: Tom Herbert <therbert@google.com> > > Date: Mon Nov 28 16:33:16 2011 +0000 > > > > e1000e: Support for byte queue limits > > > > Changes to e1000e to use byte queue limits. > > > > Signed-off-by: Tom Herbert <therbert@google.com> > > Acked-by: Eric Dumazet <eric.dumazet@gmail.com> > > Signed-off-by: David S. Miller <davem@davemloft.net> > > > > :040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe > > 51914e221547b95a989b5c7e9b037c9370fd734e M drivers > > > > > > Thanks, > > Ben > > > > -- > > Ben Greear <greearb@candelatech.com> > > Candela Technologies Inc http://www.candelatech.com > > > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > [-- Attachment #2: Type: text/plain, Size: 271 bytes --] ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 [-- Attachment #3: Type: text/plain, Size: 257 bytes --] _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 2:39 ` Tom Herbert 2012-04-20 6:44 ` Ying Cai @ 2012-04-20 19:00 ` Ben Greear 2012-04-20 19:05 ` Tom Herbert 1 sibling, 1 reply; 17+ messages in thread From: Ben Greear @ 2012-04-20 19:00 UTC (permalink / raw) To: Tom Herbert; +Cc: netdev, e1000-devel list, Eric Dumazet On 04/19/2012 07:39 PM, Tom Herbert wrote: > Thanks, will try to reproduce. I am seeing something similar with the 'igb' driver, though this NIC also involves a side-driver that does bypass. When I enable/disable bypass, the links bounce (as expected), and igb reports the same timeout that I was seeing with e1000e. If I revert the igb BQL patch, then it works fine (kernel 3.3.2+). I did not do an exhaustive bisect, nor have tested non-bypass igb hardware, but since reverting the patch fixes the problem.... Maybe there is some fundamental issue with BQL when a NIC resets itself (in this case, due to remote port doing a reset) Maybe we are not properly accounting pkts cleared from the xmit queue on reset or something of that nature? I'm happy to test patches of someone has suggestions... Thanks, Ben > > Tom > > On Thu, Apr 19, 2012 at 4:27 PM, Ben Greear<greearb@candelatech.com> wrote: >> Test case: >> >> Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic >> (moderate speeds of traffic has issues as well, maybe not as easy to >> reproduce) >> reset peer interface >> ----> tx queue timeout >> >> >> Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down >> Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter >> Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down >> Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full >> Duplex, Flow Control: Rx/Tx >> Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >> becomes ready >> Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full >> Duplex, Flow Control: Rx/Tx >> Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link >> becomes ready >> Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, >> re-enabling all cpus for irq balancing >> Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------ >> Apr 19 16:12:55 localhost kernel: WARNING: at >> /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 >> dev_watchdog+0xf4/0x154() >> Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU >> Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit >> queue 0 timed out >> Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw 8021q >> garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid >> coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma >> i2c_i801 >> i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy >> radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: >> nf_nat] >> Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted >> 3.2.0-rc2+ #36 >> Apr 19 16:12:55 localhost kernel: Call Trace: >> Apr 19 16:12:55 localhost kernel:<IRQ> [<ffffffff81042902>] >> warn_slowpath_common+0x80/0x98 >> Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>] >> warn_slowpath_fmt+0x41/0x43 >> Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>] >> dev_watchdog+0xf4/0x154 >> Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>] >> run_timer_softirq+0x16f/0x201 >> Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? >> netif_tx_unlock+0x57/0x57 >> Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>] >> __do_softirq+0x86/0x12f >> Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ? >> hrtimer_interrupt+0x12b/0x1bd >> Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>] >> call_softirq+0x1c/0x30 >> Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e >> Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb >> Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] >> smp_apic_timer_interrupt+0x85/0x93 >> Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] >> apic_timer_interrupt+0x6e/0x80 >> Apr 19 16:12:55 localhost kernel:<EOI> [<ffffffff81010b8c>] ? >> mwait_idle+0x6e/0x8c >> Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ? >> mwait_idle+0x61/0x8c >> Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe >> Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>] >> start_secondary+0x194/0x199 >> Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- >> Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter >> Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses: >> 1 >> Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating dump >> directories >> Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04-19-16:12:57-898-0' >> creation detected >> Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt >> Apr 19 16:12:57 localhost abrtd: Can't open file >> '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or >> directory >> Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: >> /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >> Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of >> /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >> Apr 19 16:12:57 localhost abrtd: Deleting dump directory >> oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), >> sending dbus signal >> Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full >> Duplex, Flow Control: Rx/Tx >> Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >> becomes ready >> Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, >> re-enabling all cpus for irq balancing >> Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter >> Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194 >> Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full >> Duplex, Flow Control: Rx/Tx >> Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >> becomes ready >> .... >> >> lspci: >> >> 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet >> Controller (rev 06) >> Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter >> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ >> Stepping- SERR+ FastB2B- DisINTx+ >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast>TAbort- >> <TAbort-<MAbort->SERR-<PERR- INTx- >> Latency: 0, Cache Line Size: 32 bytes >> Interrupt: pin A routed to IRQ 74 >> Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K] >> Region 2: I/O ports at 3000 [size=32] >> [virtual] Expansion ROM at d8d00000 [disabled] [size=128K] >> Capabilities: [c8] Power Management version 2 >> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA >> PME(D0+,D1-,D2-,D3hot+,D3cold-) >> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- >> Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ >> Address: 00000000feeff00c Data: 41a3 >> Capabilities: [e0] Express (v1) Endpoint, MSI 00 >> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s >> <512ns, L1<64us >> ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- >> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ >> Unsupported+ >> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ >> MaxPayload 128 bytes, MaxReadReq 4096 bytes >> DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- >> TransPend- >> LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, >> Latency L0<4us, L1<64us >> ClockPM- Surprise- LLActRep- BwNot- >> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- >> CommClk- >> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- >> LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ >> DLActive- BWMgmt- ABWMgmt- >> Capabilities: [100 v1] Advanced Error Reporting >> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >> RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- >> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- >> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- >> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- >> ChkEn- >> Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e >> Kernel driver in use: e1000e >> Kernel modules: e1000e >> >> >> 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit >> commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c >> Author: Tom Herbert<therbert@google.com> >> Date: Mon Nov 28 16:33:16 2011 +0000 >> >> e1000e: Support for byte queue limits >> >> Changes to e1000e to use byte queue limits. >> >> Signed-off-by: Tom Herbert<therbert@google.com> >> Acked-by: Eric Dumazet<eric.dumazet@gmail.com> >> Signed-off-by: David S. Miller<davem@davemloft.net> >> >> :040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe >> 51914e221547b95a989b5c7e9b037c9370fd734e M drivers >> >> >> Thanks, >> Ben >> >> -- >> Ben Greear<greearb@candelatech.com> >> Candela Technologies Inc http://www.candelatech.com >> -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 19:00 ` Ben Greear @ 2012-04-20 19:05 ` Tom Herbert 2012-04-20 19:13 ` Ben Greear 0 siblings, 1 reply; 17+ messages in thread From: Tom Herbert @ 2012-04-20 19:05 UTC (permalink / raw) To: Ben Greear; +Cc: e1000-devel list, netdev > I am seeing something similar with the 'igb' driver, though this > NIC also involves a side-driver that does bypass. When I enable/disable > bypass, the links bounce (as expected), and igb reports the same > timeout that I was seeing with e1000e. > Hi Ben, Are the circumstances the same between igb and e1000e for the issue? That is in both cases are links being bounced or reset? Thanks, Tom > If I revert the igb BQL patch, then it works fine (kernel 3.3.2+). > > I did not do an exhaustive bisect, nor have tested non-bypass > igb hardware, but since reverting the patch fixes the problem.... > > Maybe there is some fundamental issue with BQL when a NIC > resets itself (in this case, due to remote port doing > a reset) Maybe we are not properly accounting pkts cleared > from the xmit queue on reset or something of that nature? > > I'm happy to test patches of someone has suggestions... > > Thanks, > Ben > > >> >> Tom >> >> On Thu, Apr 19, 2012 at 4:27 PM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> Test case: >>> >>> Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic >>> (moderate speeds of traffic has issues as well, maybe not as easy to >>> reproduce) >>> reset peer interface >>> ----> tx queue timeout >>> >>> >>> Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down >>> Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset >>> adapter >>> Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down >>> Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps >>> Full >>> Duplex, Flow Control: Rx/Tx >>> Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >>> becomes ready >>> Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps >>> Full >>> Duplex, Flow Control: Rx/Tx >>> Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link >>> becomes ready >>> Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, >>> re-enabling all cpus for irq balancing >>> Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------ >>> Apr 19 16:12:55 localhost kernel: WARNING: at >>> /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 >>> dev_watchdog+0xf4/0x154() >>> Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU >>> Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): >>> transmit >>> queue 0 timed out >>> Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw >>> 8021q >>> garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen >>> iscsi_tcp >>> libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf >>> hwmon_vid >>> coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma >>> i2c_i801 >>> i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy >>> radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: >>> nf_nat] >>> Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted >>> 3.2.0-rc2+ #36 >>> Apr 19 16:12:55 localhost kernel: Call Trace: >>> Apr 19 16:12:55 localhost kernel:<IRQ> [<ffffffff81042902>] >>> warn_slowpath_common+0x80/0x98 >>> Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>] >>> warn_slowpath_fmt+0x41/0x43 >>> Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>] >>> dev_watchdog+0xf4/0x154 >>> Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>] >>> run_timer_softirq+0x16f/0x201 >>> Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? >>> netif_tx_unlock+0x57/0x57 >>> Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>] >>> __do_softirq+0x86/0x12f >>> Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ? >>> hrtimer_interrupt+0x12b/0x1bd >>> Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>] >>> call_softirq+0x1c/0x30 >>> Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] >>> do_softirq+0x41/0x7e >>> Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb >>> Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] >>> smp_apic_timer_interrupt+0x85/0x93 >>> Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] >>> apic_timer_interrupt+0x6e/0x80 >>> Apr 19 16:12:55 localhost kernel:<EOI> [<ffffffff81010b8c>] ? >>> mwait_idle+0x6e/0x8c >>> Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ? >>> mwait_idle+0x61/0x8c >>> Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe >>> Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>] >>> start_secondary+0x194/0x199 >>> Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- >>> Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset >>> adapter >>> Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found >>> oopses: >>> 1 >>> Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating >>> dump >>> directories >>> Apr 19 16:12:57 localhost abrtd: Directory >>> 'oops-2012-04-19-16:12:57-898-0' >>> creation detected >>> Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to >>> Abrt >>> Apr 19 16:12:57 localhost abrtd: Can't open file >>> '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or >>> directory >>> Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: >>> /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >>> Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of >>> /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >>> Apr 19 16:12:57 localhost abrtd: Deleting dump directory >>> oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), >>> sending dbus signal >>> Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps >>> Full >>> Duplex, Flow Control: Rx/Tx >>> Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >>> becomes ready >>> Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, >>> re-enabling all cpus for irq balancing >>> Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset >>> adapter >>> Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194 >>> Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps >>> Full >>> Duplex, Flow Control: Rx/Tx >>> Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >>> becomes ready >>> .... >>> >>> lspci: >>> >>> 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet >>> Controller (rev 06) >>> Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter >>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- >>> ParErr+ >>> Stepping- SERR+ FastB2B- DisINTx+ >>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast>TAbort- >>> <TAbort-<MAbort->SERR-<PERR- INTx- >>> Latency: 0, Cache Line Size: 32 bytes >>> Interrupt: pin A routed to IRQ 74 >>> Region 0: Memory at d8300000 (32-bit, non-prefetchable) >>> [size=128K] >>> Region 2: I/O ports at 3000 [size=32] >>> [virtual] Expansion ROM at d8d00000 [disabled] [size=128K] >>> Capabilities: [c8] Power Management version 2 >>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA >>> PME(D0+,D1-,D2-,D3hot+,D3cold-) >>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- >>> Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ >>> Address: 00000000feeff00c Data: 41a3 >>> Capabilities: [e0] Express (v1) Endpoint, MSI 00 >>> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s >>> <512ns, L1<64us >>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- >>> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ >>> Unsupported+ >>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ >>> MaxPayload 128 bytes, MaxReadReq 4096 bytes >>> DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- >>> TransPend- >>> LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, >>> Latency L0<4us, L1<64us >>> ClockPM- Surprise- LLActRep- BwNot- >>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- >>> CommClk- >>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- >>> LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ >>> DLActive- BWMgmt- ABWMgmt- >>> Capabilities: [100 v1] Advanced Error Reporting >>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >>> RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- >>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >>> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >>> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- >>> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- >>> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >>> NonFatalErr- >>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >>> NonFatalErr- >>> AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- >>> ChkEn- >>> Capabilities: [140 v1] Device Serial Number >>> 00-e0-ed-ff-ff-0c-11-6e >>> Kernel driver in use: e1000e >>> Kernel modules: e1000e >>> >>> >>> 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit >>> commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c >>> Author: Tom Herbert<therbert@google.com> >>> Date: Mon Nov 28 16:33:16 2011 +0000 >>> >>> e1000e: Support for byte queue limits >>> >>> Changes to e1000e to use byte queue limits. >>> >>> Signed-off-by: Tom Herbert<therbert@google.com> >>> Acked-by: Eric Dumazet<eric.dumazet@gmail.com> >>> Signed-off-by: David S. Miller<davem@davemloft.net> >>> >>> :040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe >>> 51914e221547b95a989b5c7e9b037c9370fd734e M drivers >>> >>> >>> Thanks, >>> Ben >>> >>> -- >>> Ben Greear<greearb@candelatech.com> >>> Candela Technologies Inc http://www.candelatech.com >>> > > > -- > Ben Greear <greearb@candelatech.com> > Candela Technologies Inc http://www.candelatech.com > ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 19:05 ` Tom Herbert @ 2012-04-20 19:13 ` Ben Greear 2012-04-20 19:44 ` John Fastabend 0 siblings, 1 reply; 17+ messages in thread From: Ben Greear @ 2012-04-20 19:13 UTC (permalink / raw) To: Tom Herbert; +Cc: e1000-devel list, netdev On 04/20/2012 12:05 PM, Tom Herbert wrote: >> I am seeing something similar with the 'igb' driver, though this >> NIC also involves a side-driver that does bypass. When I enable/disable >> bypass, the links bounce (as expected), and igb reports the same >> timeout that I was seeing with e1000e. >> > > Hi Ben, > > Are the circumstances the same between igb and e1000e for the issue? > That is in both cases are links being bounced or reset? Yes, peer interface is being bounced in both cases. The igb NIC is in a different machine, running Fedora 14 instead of F16 for the e1000e. I have the bypass NIC cabled to itself..2 ports sending/receiving traffic, and 2 doing bypass (in hardware, or bridging in software). When I twiddle to/from hardware/software bridging on the two bypass ports, link is re-negotiated on the traffic-generating ports, and the traffic generating ports have the tx queue timeout issue. If you have an igb or e1000e system that can generate traffic while resetting the peer interface, I think you should easily see the problem. If you DON'T see the problem, please let me know and I will try to provide a more concise test case using open-source traffic generators (we're using our own traffic generator, but at least on the e1000e bisect, I was using standard upstream kernels and pure user-space API to generate traffic). Thanks, Ben > > Thanks, > Tom > >> If I revert the igb BQL patch, then it works fine (kernel 3.3.2+). >> >> I did not do an exhaustive bisect, nor have tested non-bypass >> igb hardware, but since reverting the patch fixes the problem.... >> >> Maybe there is some fundamental issue with BQL when a NIC >> resets itself (in this case, due to remote port doing >> a reset) Maybe we are not properly accounting pkts cleared >> from the xmit queue on reset or something of that nature? >> >> I'm happy to test patches of someone has suggestions... >> >> Thanks, >> Ben >> >> >>> >>> Tom >>> >>> On Thu, Apr 19, 2012 at 4:27 PM, Ben Greear<greearb@candelatech.com> >>> wrote: >>>> >>>> Test case: >>>> >>>> Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic >>>> (moderate speeds of traffic has issues as well, maybe not as easy to >>>> reproduce) >>>> reset peer interface >>>> ----> tx queue timeout >>>> >>>> >>>> Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down >>>> Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset >>>> adapter >>>> Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down >>>> Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps >>>> Full >>>> Duplex, Flow Control: Rx/Tx >>>> Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >>>> becomes ready >>>> Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps >>>> Full >>>> Duplex, Flow Control: Rx/Tx >>>> Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link >>>> becomes ready >>>> Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, >>>> re-enabling all cpus for irq balancing >>>> Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------ >>>> Apr 19 16:12:55 localhost kernel: WARNING: at >>>> /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 >>>> dev_watchdog+0xf4/0x154() >>>> Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU >>>> Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): >>>> transmit >>>> queue 0 timed out >>>> Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw >>>> 8021q >>>> garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen >>>> iscsi_tcp >>>> libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf >>>> hwmon_vid >>>> coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma >>>> i2c_i801 >>>> i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy >>>> radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: >>>> nf_nat] >>>> Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted >>>> 3.2.0-rc2+ #36 >>>> Apr 19 16:12:55 localhost kernel: Call Trace: >>>> Apr 19 16:12:55 localhost kernel:<IRQ> [<ffffffff81042902>] >>>> warn_slowpath_common+0x80/0x98 >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>] >>>> warn_slowpath_fmt+0x41/0x43 >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>] >>>> dev_watchdog+0xf4/0x154 >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>] >>>> run_timer_softirq+0x16f/0x201 >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? >>>> netif_tx_unlock+0x57/0x57 >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>] >>>> __do_softirq+0x86/0x12f >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ? >>>> hrtimer_interrupt+0x12b/0x1bd >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>] >>>> call_softirq+0x1c/0x30 >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] >>>> do_softirq+0x41/0x7e >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] >>>> smp_apic_timer_interrupt+0x85/0x93 >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] >>>> apic_timer_interrupt+0x6e/0x80 >>>> Apr 19 16:12:55 localhost kernel:<EOI> [<ffffffff81010b8c>] ? >>>> mwait_idle+0x6e/0x8c >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ? >>>> mwait_idle+0x61/0x8c >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe >>>> Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>] >>>> start_secondary+0x194/0x199 >>>> Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- >>>> Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset >>>> adapter >>>> Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found >>>> oopses: >>>> 1 >>>> Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating >>>> dump >>>> directories >>>> Apr 19 16:12:57 localhost abrtd: Directory >>>> 'oops-2012-04-19-16:12:57-898-0' >>>> creation detected >>>> Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to >>>> Abrt >>>> Apr 19 16:12:57 localhost abrtd: Can't open file >>>> '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or >>>> directory >>>> Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: >>>> /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >>>> Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of >>>> /var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >>>> Apr 19 16:12:57 localhost abrtd: Deleting dump directory >>>> oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), >>>> sending dbus signal >>>> Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps >>>> Full >>>> Duplex, Flow Control: Rx/Tx >>>> Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >>>> becomes ready >>>> Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, >>>> re-enabling all cpus for irq balancing >>>> Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset >>>> adapter >>>> Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194 >>>> Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps >>>> Full >>>> Duplex, Flow Control: Rx/Tx >>>> Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link >>>> becomes ready >>>> .... >>>> >>>> lspci: >>>> >>>> 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet >>>> Controller (rev 06) >>>> Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter >>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- >>>> ParErr+ >>>> Stepping- SERR+ FastB2B- DisINTx+ >>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast>TAbort- >>>> <TAbort-<MAbort->SERR-<PERR- INTx- >>>> Latency: 0, Cache Line Size: 32 bytes >>>> Interrupt: pin A routed to IRQ 74 >>>> Region 0: Memory at d8300000 (32-bit, non-prefetchable) >>>> [size=128K] >>>> Region 2: I/O ports at 3000 [size=32] >>>> [virtual] Expansion ROM at d8d00000 [disabled] [size=128K] >>>> Capabilities: [c8] Power Management version 2 >>>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA >>>> PME(D0+,D1-,D2-,D3hot+,D3cold-) >>>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- >>>> Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ >>>> Address: 00000000feeff00c Data: 41a3 >>>> Capabilities: [e0] Express (v1) Endpoint, MSI 00 >>>> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s >>>> <512ns, L1<64us >>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- >>>> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ >>>> Unsupported+ >>>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ >>>> MaxPayload 128 bytes, MaxReadReq 4096 bytes >>>> DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- >>>> TransPend- >>>> LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, >>>> Latency L0<4us, L1<64us >>>> ClockPM- Surprise- LLActRep- BwNot- >>>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- >>>> CommClk- >>>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- >>>> LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ >>>> DLActive- BWMgmt- ABWMgmt- >>>> Capabilities: [100 v1] Advanced Error Reporting >>>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >>>> RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- >>>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >>>> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >>>> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- >>>> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- >>>> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >>>> NonFatalErr- >>>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >>>> NonFatalErr- >>>> AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- >>>> ChkEn- >>>> Capabilities: [140 v1] Device Serial Number >>>> 00-e0-ed-ff-ff-0c-11-6e >>>> Kernel driver in use: e1000e >>>> Kernel modules: e1000e >>>> >>>> >>>> 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit >>>> commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c >>>> Author: Tom Herbert<therbert@google.com> >>>> Date: Mon Nov 28 16:33:16 2011 +0000 >>>> >>>> e1000e: Support for byte queue limits >>>> >>>> Changes to e1000e to use byte queue limits. >>>> >>>> Signed-off-by: Tom Herbert<therbert@google.com> >>>> Acked-by: Eric Dumazet<eric.dumazet@gmail.com> >>>> Signed-off-by: David S. Miller<davem@davemloft.net> >>>> >>>> :040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe >>>> 51914e221547b95a989b5c7e9b037c9370fd734e M drivers >>>> >>>> >>>> Thanks, >>>> Ben >>>> >>>> -- >>>> Ben Greear<greearb@candelatech.com> >>>> Candela Technologies Inc http://www.candelatech.com >>>> >> >> >> -- >> Ben Greear<greearb@candelatech.com> >> Candela Technologies Inc http://www.candelatech.com >> > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 19:13 ` Ben Greear @ 2012-04-20 19:44 ` John Fastabend 2012-04-20 21:21 ` Tom Herbert 0 siblings, 1 reply; 17+ messages in thread From: John Fastabend @ 2012-04-20 19:44 UTC (permalink / raw) To: Tom Herbert; +Cc: Ben Greear, netdev, e1000-devel list, Eric Dumazet On 4/20/2012 12:13 PM, Ben Greear wrote: > On 04/20/2012 12:05 PM, Tom Herbert wrote: >>> I am seeing something similar with the 'igb' driver, though this >>> NIC also involves a side-driver that does bypass. When I enable/disable >>> bypass, the links bounce (as expected), and igb reports the same >>> timeout that I was seeing with e1000e. >>> >> [...] >>>>> >>>>> 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit >>>>> commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c >>>>> Author: Tom Herbert<therbert@google.com> >>>>> Date: Mon Nov 28 16:33:16 2011 +0000 >>>>> >>>>> e1000e: Support for byte queue limits >>>>> >>>>> Changes to e1000e to use byte queue limits. >>>>> >>>>> Signed-off-by: Tom Herbert<therbert@google.com> >>>>> Acked-by: Eric Dumazet<eric.dumazet@gmail.com> >>>>> Signed-off-by: David S. Miller<davem@davemloft.net> >>>>> >>>>> :040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe >>>>> 51914e221547b95a989b5c7e9b037c9370fd734e M drivers >>>>> >>>>> >>>>> Thanks, >>>>> Ben >>>>> Tom, did you see these two patches? Maybe this is resolved by the second patch. We needed these to fixup ixgbe and igb (i didn't test e1000e) looks like we might want to push these at stable. I don't believe they are in 3.3. commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 Author: Alexander Duyck <alexander.h.duyck@intel.com> Date: Tue Feb 7 02:29:06 2012 +0000 net: Add memory barriers to prevent possible race in byte queue limits This change adds a memory barrier to the byte queue limit code to address a possible race as has been seen in the past with the netif_stop_queue/netif_wake_queue logic. Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com> Tested-by: Stephen Ko <stephen.s.ko@intel.com> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 commit 5c4903549c05bbb373479e0ce2992573c120654a Author: Alexander Duyck <alexander.h.duyck@intel.com> Date: Tue Feb 7 02:29:01 2012 +0000 net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state We are seeing dev_watchdog hangs on several drivers. I suspect this is due to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link change, and then not being cleared by netdev_tx_reset_queue. This change corrects that. In addition we were seeing dev_watchdog hangs on igb after running the ethtool tests. We found this to be due to the fact that the ethtool test runs the same logic as ndo_start_xmit, but we were never clearing the XOFF flag since the loopback test in ethtool does not do byte queue accounting. Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com> Tested-by: Stephen Ko <stephen.s.ko@intel.com> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 19:44 ` John Fastabend @ 2012-04-20 21:21 ` Tom Herbert 2012-04-20 21:24 ` Ben Greear 2012-04-20 21:56 ` Ben Greear 0 siblings, 2 replies; 17+ messages in thread From: Tom Herbert @ 2012-04-20 21:21 UTC (permalink / raw) To: John Fastabend; +Cc: Ben Greear, netdev, e1000-devel list, Eric Dumazet Thanks John for pointers to those. Ben, are you running a kernel with these patches? Tom > > Tom, did you see these two patches? Maybe this is resolved by > the second patch. > > We needed these to fixup ixgbe and igb (i didn't test e1000e) > looks like we might want to push these at stable. I don't > believe they are in 3.3. > > commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 > Author: Alexander Duyck <alexander.h.duyck@intel.com> > Date: Tue Feb 7 02:29:06 2012 +0000 > > net: Add memory barriers to prevent possible race in byte queue limits > > This change adds a memory barrier to the byte queue limit code to address a > possible race as has been seen in the past with the > netif_stop_queue/netif_wake_queue logic. > > Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com> > Tested-by: Stephen Ko <stephen.s.ko@intel.com> > Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com> > > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 > > > commit 5c4903549c05bbb373479e0ce2992573c120654a > Author: Alexander Duyck <alexander.h.duyck@intel.com> > Date: Tue Feb 7 02:29:01 2012 +0000 > > net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state > > We are seeing dev_watchdog hangs on several drivers. I suspect this is due > to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link > change, and then not being cleared by netdev_tx_reset_queue. This change > corrects that. > > In addition we were seeing dev_watchdog hangs on igb after running the > ethtool tests. We found this to be due to the fact that the ethtool test > runs the same logic as ndo_start_xmit, but we were never clearing the XOFF > flag since the loopback test in ethtool does not do byte queue accounting. > > Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com> > Tested-by: Stephen Ko <stephen.s.ko@intel.com> > Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com> > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 21:21 ` Tom Herbert @ 2012-04-20 21:24 ` Ben Greear 2012-04-20 21:56 ` Ben Greear 1 sibling, 0 replies; 17+ messages in thread From: Ben Greear @ 2012-04-20 21:24 UTC (permalink / raw) To: Tom Herbert; +Cc: John Fastabend, netdev, e1000-devel list On 04/20/2012 02:21 PM, Tom Herbert wrote: > Thanks John for pointers to those. Ben, are you running a kernel with > these patches? I don't think so. I'll add them and re-test. Thanks, Ben > > Tom > >> >> Tom, did you see these two patches? Maybe this is resolved by >> the second patch. >> >> We needed these to fixup ixgbe and igb (i didn't test e1000e) >> looks like we might want to push these at stable. I don't >> believe they are in 3.3. >> >> commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 >> Author: Alexander Duyck<alexander.h.duyck@intel.com> >> Date: Tue Feb 7 02:29:06 2012 +0000 >> >> net: Add memory barriers to prevent possible race in byte queue limits >> >> This change adds a memory barrier to the byte queue limit code to address a >> possible race as has been seen in the past with the >> netif_stop_queue/netif_wake_queue logic. >> >> Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com> >> Tested-by: Stephen Ko<stephen.s.ko@intel.com> >> Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com> >> >> >> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 >> >> >> commit 5c4903549c05bbb373479e0ce2992573c120654a >> Author: Alexander Duyck<alexander.h.duyck@intel.com> >> Date: Tue Feb 7 02:29:01 2012 +0000 >> >> net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state >> >> We are seeing dev_watchdog hangs on several drivers. I suspect this is due >> to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link >> change, and then not being cleared by netdev_tx_reset_queue. This change >> corrects that. >> >> In addition we were seeing dev_watchdog hangs on igb after running the >> ethtool tests. We found this to be due to the fact that the ethtool test >> runs the same logic as ndo_start_xmit, but we were never clearing the XOFF >> flag since the loopback test in ethtool does not do byte queue accounting. >> >> Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com> >> Tested-by: Stephen Ko<stephen.s.ko@intel.com> >> Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com> >> >> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a >> >> -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 21:21 ` Tom Herbert 2012-04-20 21:24 ` Ben Greear @ 2012-04-20 21:56 ` Ben Greear 2012-05-01 21:10 ` [E1000-devel] " Ben Greear 1 sibling, 1 reply; 17+ messages in thread From: Ben Greear @ 2012-04-20 21:56 UTC (permalink / raw) To: Tom Herbert; +Cc: John Fastabend, netdev, e1000-devel list On 04/20/2012 02:21 PM, Tom Herbert wrote: > Thanks John for pointers to those. Ben, are you running a kernel with > these patches? I just tested this on my e1000e and igb machine. With these patches, I can no longer reproduce the problem. So, please make sure those are queued up for 3.3 stable! Thanks, Ben > > Tom > >> >> Tom, did you see these two patches? Maybe this is resolved by >> the second patch. >> >> We needed these to fixup ixgbe and igb (i didn't test e1000e) >> looks like we might want to push these at stable. I don't >> believe they are in 3.3. >> >> commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 >> Author: Alexander Duyck<alexander.h.duyck@intel.com> >> Date: Tue Feb 7 02:29:06 2012 +0000 >> >> net: Add memory barriers to prevent possible race in byte queue limits >> >> This change adds a memory barrier to the byte queue limit code to address a >> possible race as has been seen in the past with the >> netif_stop_queue/netif_wake_queue logic. >> >> Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com> >> Tested-by: Stephen Ko<stephen.s.ko@intel.com> >> Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com> >> >> >> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 >> >> >> commit 5c4903549c05bbb373479e0ce2992573c120654a >> Author: Alexander Duyck<alexander.h.duyck@intel.com> >> Date: Tue Feb 7 02:29:01 2012 +0000 >> >> net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state >> >> We are seeing dev_watchdog hangs on several drivers. I suspect this is due >> to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link >> change, and then not being cleared by netdev_tx_reset_queue. This change >> corrects that. >> >> In addition we were seeing dev_watchdog hangs on igb after running the >> ethtool tests. We found this to be due to the fact that the ethtool test >> runs the same logic as ndo_start_xmit, but we were never clearing the XOFF >> flag since the loopback test in ethtool does not do byte queue accounting. >> >> Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com> >> Tested-by: Stephen Ko<stephen.s.ko@intel.com> >> Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com> >> >> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a >> >> -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [E1000-devel] e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-20 21:56 ` Ben Greear @ 2012-05-01 21:10 ` Ben Greear 2012-05-01 21:49 ` David Miller 0 siblings, 1 reply; 17+ messages in thread From: Ben Greear @ 2012-05-01 21:10 UTC (permalink / raw) To: Tom Herbert, David Miller; +Cc: John Fastabend, netdev, e1000-devel list On 04/20/2012 02:56 PM, Ben Greear wrote: > On 04/20/2012 02:21 PM, Tom Herbert wrote: >> Thanks John for pointers to those. Ben, are you running a kernel with >> these patches? > > I just tested this on my e1000e and igb machine. With these patches, > I can no longer reproduce the problem. > > So, please make sure those are queued up for 3.3 stable! Dave: I think these patches below should go to 3.3 stable. They are not queued for stable yet as far as I can tell. Thanks, Ben > > Thanks, > Ben > >> >> Tom >> >>> >>> Tom, did you see these two patches? Maybe this is resolved by >>> the second patch. >>> >>> We needed these to fixup ixgbe and igb (i didn't test e1000e) >>> looks like we might want to push these at stable. I don't >>> believe they are in 3.3. >>> >>> commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 >>> Author: Alexander Duyck<alexander.h.duyck@intel.com> >>> Date: Tue Feb 7 02:29:06 2012 +0000 >>> >>> net: Add memory barriers to prevent possible race in byte queue limits >>> >>> This change adds a memory barrier to the byte queue limit code to address a >>> possible race as has been seen in the past with the >>> netif_stop_queue/netif_wake_queue logic. >>> >>> Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com> >>> Tested-by: Stephen Ko<stephen.s.ko@intel.com> >>> Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com> >>> >>> >>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635 >>> >>> >>> commit 5c4903549c05bbb373479e0ce2992573c120654a >>> Author: Alexander Duyck<alexander.h.duyck@intel.com> >>> Date: Tue Feb 7 02:29:01 2012 +0000 >>> >>> net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state >>> >>> We are seeing dev_watchdog hangs on several drivers. I suspect this is due >>> to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link >>> change, and then not being cleared by netdev_tx_reset_queue. This change >>> corrects that. >>> >>> In addition we were seeing dev_watchdog hangs on igb after running the >>> ethtool tests. We found this to be due to the fact that the ethtool test >>> runs the same logic as ndo_start_xmit, but we were never clearing the XOFF >>> flag since the loopback test in ethtool does not do byte queue accounting. >>> >>> Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com> >>> Tested-by: Stephen Ko<stephen.s.ko@intel.com> >>> Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com> >>> >>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a >>> >>> > > -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-05-01 21:10 ` [E1000-devel] " Ben Greear @ 2012-05-01 21:49 ` David Miller 2012-05-01 22:08 ` Ben Greear 0 siblings, 1 reply; 17+ messages in thread From: David Miller @ 2012-05-01 21:49 UTC (permalink / raw) To: greearb; +Cc: john.r.fastabend, netdev, e1000-devel, therbert From: Ben Greear <greearb@candelatech.com> Date: Tue, 01 May 2012 14:10:43 -0700 > On 04/20/2012 02:56 PM, Ben Greear wrote: >> On 04/20/2012 02:21 PM, Tom Herbert wrote: >>> Thanks John for pointers to those. Ben, are you running a kernel with >>> these patches? >> >> I just tested this on my e1000e and igb machine. With these patches, >> I can no longer reproduce the problem. >> >> So, please make sure those are queued up for 3.3 stable! > > Dave: I think these patches below should go to 3.3 stable. > > They are not queued for stable yet as far as I can tell. I let the Intel developers handle -stable submissions for their drivers. ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-05-01 21:49 ` David Miller @ 2012-05-01 22:08 ` Ben Greear 2012-05-01 22:42 ` [E1000-devel] " Jeff Kirsher 0 siblings, 1 reply; 17+ messages in thread From: Ben Greear @ 2012-05-01 22:08 UTC (permalink / raw) To: David Miller; +Cc: john.r.fastabend, netdev, e1000-devel, therbert On 05/01/2012 02:49 PM, David Miller wrote: > From: Ben Greear<greearb@candelatech.com> > Date: Tue, 01 May 2012 14:10:43 -0700 > >> On 04/20/2012 02:56 PM, Ben Greear wrote: >>> On 04/20/2012 02:21 PM, Tom Herbert wrote: >>>> Thanks John for pointers to those. Ben, are you running a kernel with >>>> these patches? >>> >>> I just tested this on my e1000e and igb machine. With these patches, >>> I can no longer reproduce the problem. >>> >>> So, please make sure those are queued up for 3.3 stable! >> >> Dave: I think these patches below should go to 3.3 stable. >> >> They are not queued for stable yet as far as I can tell. > > I let the Intel developers handle -stable submissions for their > drivers. I don't think this is specific to their drivers. But, as long as _someone_ is pushing it to stable, it is fine by me. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [E1000-devel] e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-05-01 22:08 ` Ben Greear @ 2012-05-01 22:42 ` Jeff Kirsher 2012-05-01 22:46 ` David Miller 0 siblings, 1 reply; 17+ messages in thread From: Jeff Kirsher @ 2012-05-01 22:42 UTC (permalink / raw) To: Ben Greear; +Cc: David Miller, john.r.fastabend, netdev, e1000-devel, therbert [-- Attachment #1: Type: text/plain, Size: 1059 bytes --] On Tue, 2012-05-01 at 15:08 -0700, Ben Greear wrote: > On 05/01/2012 02:49 PM, David Miller wrote: > > From: Ben Greear<greearb@candelatech.com> > > Date: Tue, 01 May 2012 14:10:43 -0700 > > > >> On 04/20/2012 02:56 PM, Ben Greear wrote: > >>> On 04/20/2012 02:21 PM, Tom Herbert wrote: > >>>> Thanks John for pointers to those. Ben, are you running a kernel > with > >>>> these patches? > >>> > >>> I just tested this on my e1000e and igb machine. With these > patches, > >>> I can no longer reproduce the problem. > >>> > >>> So, please make sure those are queued up for 3.3 stable! > >> > >> Dave: I think these patches below should go to 3.3 stable. > >> > >> They are not queued for stable yet as far as I can tell. > > > > I let the Intel developers handle -stable submissions for their > > drivers. > > I don't think this is specific to their drivers. > > But, as long as _someone_ is pushing it to stable, > it is fine by me. > > Thanks, > Ben I will take care of the stable submission's Ben. Cheers, Jeff [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-05-01 22:42 ` [E1000-devel] " Jeff Kirsher @ 2012-05-01 22:46 ` David Miller 2012-05-01 22:52 ` [E1000-devel] " Jeff Kirsher 0 siblings, 1 reply; 17+ messages in thread From: David Miller @ 2012-05-01 22:46 UTC (permalink / raw) To: jeffrey.t.kirsher; +Cc: john.r.fastabend, netdev, therbert, e1000-devel From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> Date: Tue, 01 May 2012 15:42:28 -0700 > I will take care of the stable submission's Ben. Actually, Jeff, hold off on that. I misread Ben's email and didn't see that these were not driver-specific changes. Therefore, I'll take care of the -stable submissions and you don't need to worry about it. Thanks everyone. ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [E1000-devel] e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-05-01 22:46 ` David Miller @ 2012-05-01 22:52 ` Jeff Kirsher 0 siblings, 0 replies; 17+ messages in thread From: Jeff Kirsher @ 2012-05-01 22:52 UTC (permalink / raw) To: David Miller; +Cc: greearb, john.r.fastabend, netdev, e1000-devel, therbert [-- Attachment #1: Type: text/plain, Size: 490 bytes --] On Tue, 2012-05-01 at 18:46 -0400, David Miller wrote: > From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> > Date: Tue, 01 May 2012 15:42:28 -0700 > > > I will take care of the stable submission's Ben. > > Actually, Jeff, hold off on that. > > I misread Ben's email and didn't see that these were not > driver-specific changes. > > Therefore, I'll take care of the -stable submissions and you don't > need to worry about it. > > Thanks everyone. Ok, sounds good Dave. [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) 2012-04-19 23:27 e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) Ben Greear 2012-04-20 2:39 ` Tom Herbert @ 2012-04-20 6:46 ` Dave, Tushar N 1 sibling, 0 replies; 17+ messages in thread From: Dave, Tushar N @ 2012-04-20 6:46 UTC (permalink / raw) To: Ben Greear, netdev, e1000-devel list, therbert I had done some work on this and to me it looks like this can only happen if driver does not report bytes_compl and pkts_compl stats correctly. I will experiment more tomorrow. -Tushar >-----Original Message----- >From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] >On Behalf Of Ben Greear >Sent: Thursday, April 19, 2012 4:27 PM >To: netdev; e1000-devel list; therbert@google.com >Subject: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for >e1000e) > >Test case: > >Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate >speeds of traffic has issues as well, maybe not as easy to reproduce) >reset peer interface >----> tx queue timeout > > >Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down Apr 19 >16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19 >16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down Apr 19 16:12:50 >localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow >Control: Rx/Tx Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): >eth2: link becomes ready Apr 19 16:12:50 localhost kernel: e1000e: eth3 >NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:12:50 >localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready Apr 19 >16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, re- >enabling all cpus for irq balancing Apr 19 16:12:55 localhost kernel: ---- >--------[ cut here ]------------ Apr 19 16:12:55 localhost kernel: >WARNING: at /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 >dev_watchdog+0xf4/0x154() Apr 19 16:12:55 localhost kernel: Hardware name: >X7DBU Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): >transmit queue 0 timed out Apr 19 16:12:55 localhost kernel: Modules >linked in: xt_CT iptable_raw 8021q garp stp llc veth ppdev parport_pc lp >parport fuse macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi >scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid coretemp iTCO_wdt >microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801 i5000_edac >dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm >drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: nf_nat] Apr >19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted 3.2.0- >rc2+ #36 Apr 19 16:12:55 localhost kernel: Call Trace: >Apr 19 16:12:55 localhost kernel: <IRQ> [<ffffffff81042902>] >warn_slowpath_common+0x80/0x98 Apr 19 16:12:55 localhost kernel: >[<ffffffff810429ae>] warn_slowpath_fmt+0x41/0x43 Apr 19 16:12:55 localhost >kernel: [<ffffffff8139f8a3>] dev_watchdog+0xf4/0x154 Apr 19 16:12:55 >localhost kernel: [<ffffffff8104d371>] run_timer_softirq+0x16f/0x201 Apr >19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? >netif_tx_unlock+0x57/0x57 Apr 19 16:12:55 localhost kernel: >[<ffffffff81047e47>] __do_softirq+0x86/0x12f Apr 19 16:12:55 localhost >kernel: [<ffffffff8105d54e>] ? hrtimer_interrupt+0x12b/0x1bd Apr 19 >16:12:55 localhost kernel: [<ffffffff8144296c>] call_softirq+0x1c/0x30 Apr >19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e >Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb >Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] >smp_apic_timer_interrupt+0x85/0x93 >Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] >apic_timer_interrupt+0x6e/0x80 Apr 19 16:12:55 localhost kernel: <EOI> >[<ffffffff81010b8c>] ? mwait_idle+0x6e/0x8c Apr 19 16:12:55 localhost >kernel: [<ffffffff81010b7f>] ? mwait_idle+0x61/0x8c Apr 19 16:12:55 >localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe Apr 19 16:12:55 >localhost kernel: [<ffffffff81435477>] start_secondary+0x194/0x199 Apr 19 >16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- Apr 19 >16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19 >16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses: 1 >Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating >dump directories Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04- >19-16:12:57-898-0' creation detected Apr 19 16:12:57 localhost abrt-dump- >oops: Reported 1 kernel oopses to Abrt Apr 19 16:12:57 localhost abrtd: >Can't open file '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No >such file or directory Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: >/var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of >/var/spool/abrt/oops-2012-04-19-15:02:13-862-0 >Apr 19 16:12:57 localhost abrtd: Deleting dump directory oops-2012-04-19- >16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), sending dbus >signal Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 >Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:12:58 localhost kernel: >ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready Apr 19 16:13:03 >localhost /usr/sbin/irqbalance: Load average increasing, re-enabling all >cpus for irq balancing Apr 19 16:13:04 localhost kernel: e1000e >0000:08:00.0: eth2: Reset adapter Apr 19 16:13:05 localhost chronyd[1003]: >Selected source 108.59.2.194 Apr 19 16:13:07 localhost kernel: e1000e: >eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 >16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes >ready .... > >lspci: > >08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet >Controller (rev 06) > Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ >Stepping- SERR+ FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- ><TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 32 bytes > Interrupt: pin A routed to IRQ 74 > Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K] > Region 2: I/O ports at 3000 [size=32] > [virtual] Expansion ROM at d8d00000 [disabled] [size=128K] > Capabilities: [c8] Power Management version 2 > Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2- >,D3hot+,D3cold-) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- > Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Address: 00000000feeff00c Data: 41a3 > Capabilities: [e0] Express (v1) Endpoint, MSI 00 > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s ><512ns, L1 <64us > ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ >Unsupported+ > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 4096 bytes > DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- >TransPend- > LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, >Latency L0 <4us, L1 <64us > ClockPM- Surprise- LLActRep- BwNot- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- >CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ >DLActive- BWMgmt- ABWMgmt- > Capabilities: [100 v1] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- >UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- > AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- >ChkEn- > Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e > Kernel driver in use: e1000e > Kernel modules: e1000e > > >3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit >3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c >Author: Tom Herbert <therbert@google.com> >Date: Mon Nov 28 16:33:16 2011 +0000 > > e1000e: Support for byte queue limits > > Changes to e1000e to use byte queue limits. > > Signed-off-by: Tom Herbert <therbert@google.com> > Acked-by: Eric Dumazet <eric.dumazet@gmail.com> > Signed-off-by: David S. Miller <davem@davemloft.net> > >:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe >51914e221547b95a989b5c7e9b037c9370fd734e M drivers > > >Thanks, >Ben > >-- >Ben Greear <greearb@candelatech.com> >Candela Technologies Inc http://www.candelatech.com > >-- >To unsubscribe from this list: send the line "unsubscribe netdev" in the >body of a message to majordomo@vger.kernel.org More majordomo info at >http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2012-05-01 22:52 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-04-19 23:27 e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e) Ben Greear 2012-04-20 2:39 ` Tom Herbert 2012-04-20 6:44 ` Ying Cai 2012-04-20 19:00 ` Ben Greear 2012-04-20 19:05 ` Tom Herbert 2012-04-20 19:13 ` Ben Greear 2012-04-20 19:44 ` John Fastabend 2012-04-20 21:21 ` Tom Herbert 2012-04-20 21:24 ` Ben Greear 2012-04-20 21:56 ` Ben Greear 2012-05-01 21:10 ` [E1000-devel] " Ben Greear 2012-05-01 21:49 ` David Miller 2012-05-01 22:08 ` Ben Greear 2012-05-01 22:42 ` [E1000-devel] " Jeff Kirsher 2012-05-01 22:46 ` David Miller 2012-05-01 22:52 ` [E1000-devel] " Jeff Kirsher 2012-04-20 6:46 ` Dave, Tushar N
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.