* e1000e hardware unit hangs
@ 2018-01-23 23:46 Ben Greear
2018-01-24 16:11 ` [Intel-wired-lan] " Alexander Duyck
0 siblings, 1 reply; 13+ messages in thread
From: Ben Greear @ 2018-01-23 23:46 UTC (permalink / raw)
To: netdev
Hello,
Anyone have any more suggestions for making e1000e work better? This is from a 4.9.65+ kernel,
with these additional e1000e patches applied:
e1000e: Fix error path in link detection
e1000e: Fix wrong comment related to link detection
e1000e: Fix return value test
e1000e: Separate signaling for link check/link up
e1000e: Avoid receiver overrun interrupt bursts
Test case is simply to run 30000 tcp connections each trying to send 56Kbps of bi-directional
data between a pair of e1000e interfaces :)
No OOM related issues are seen on this kernel...similar test on 4.13 showed some OOM
issues, but I have not debugged that yet...
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4294737199, wd-timeout: 5000
jiffies: 4294745088 tx-queues: 1
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4294737200, wd-timeout: 5000
jiffies: 4294745088 tx-queues: 1
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut here ]------------
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 7 PID: 0 at /home/greearb/git/linux-4.9.dev.y/net/sched/sch_generic.c:322 dev_watchdog+0x267/0x270
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in: nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 cfg80211 bnep bluetooth
macvlan wanlink(O) pktgen fuse corete...sunrpc ipmi_d
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: CPU: 7 PID: 0 Comm: swapper/7 Tainted: G O 4.9.65+ #21
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Hardware name: Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3df0 ffffffff8142d791 0000000000000000 0000000000000000
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3e30 ffffffff8110f266 000001422fdc3e08 0000000000000000
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: 0000000000001388 00000000fffc7d30 ffff880417d0c000 00000000fffc9c00
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8142d791>] dump_stack+0x63/0x82
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f266>] __warn+0xc6/0xe0
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f338>] warn_slowpath_null+0x18/0x20
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da497>] dev_watchdog+0x267/0x270
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ? qdisc_rcu_free+0x40/0x40
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117bf70>] call_timer_fn+0x30/0x150
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ? qdisc_rcu_free+0x40/0x40
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117c350>] run_timer_softirq+0x1f0/0x450
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051021>] ? lapic_next_deadline+0x21/0x30
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8118a54d>] ? clockevents_program_event+0x7d/0x120
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115101>] __do_softirq+0xc1/0x2c0
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115461>] irq_exit+0xb1/0xc0
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051c9d>] smp_apic_timer_interrupt+0x3d/0x50
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81895842>] apic_timer_interrupt+0x82/0x90
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <EOI>
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81726e46>] ? cpuidle_enter_state+0x126/0x300
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81727042>] cpuidle_enter+0x12/0x20
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff811521ce>] call_cpuidle+0x1e/0x40
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8115240a>] cpu_startup_entry+0x13a/0x220
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8104fbd9>] start_secondary+0x149/0x170
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace 69e31de175b59d4f ]---
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Detected Hardware Unit Hang:
TDH <a8>
TDT <f3>...
Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout: 5000
jiffies: 4294759424 tx-queues: 1
Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout: 5000
jiffies: 4294759424 tx-queues: 1
Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4294766123, wd-timeout: 5000
jiffies: 4294771200 tx-queues: 1
Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4294766125, wd-timeout: 5000
jiffies: 4294771200 tx-queues: 1
Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Detected Hardware Unit Hang:
TDH <c8>
TDT <f5>...
Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: e1000e hardware unit hangs
2018-01-23 23:46 e1000e hardware unit hangs Ben Greear
@ 2018-01-24 16:11 ` Alexander Duyck
0 siblings, 0 replies; 13+ messages in thread
From: Alexander Duyck @ 2018-01-24 16:11 UTC (permalink / raw)
To: Ben Greear, intel-wired-lan, e1000-devel; +Cc: netdev, Neftin, Sasha
On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
> Hello,
>
> Anyone have any more suggestions for making e1000e work better? This is
> from a 4.9.65+ kernel,
> with these additional e1000e patches applied:
>
> e1000e: Fix error path in link detection
> e1000e: Fix wrong comment related to link detection
> e1000e: Fix return value test
> e1000e: Separate signaling for link check/link up
> e1000e: Avoid receiver overrun interrupt bursts
Most of these patches shouldn't address anything that would trigger Tx
hangs. They are mostly related to just link detection.
> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
> of bi-directional
> data between a pair of e1000e interfaces :)
>
> No OOM related issues are seen on this kernel...similar test on 4.13 showed
> some OOM
> issues, but I have not debugged that yet...
Really a question like this probably belongs on e1000-devel or
intel-wired-lan so I have added those lists and the e1000e maintainer
to the thread.
It would be useful if you could provide more information about the
device itself such as the ID and the kind of test you are running.
Keep in mind the e1000e driver supports a pretty broad swath of
devices so we need to narrow things down a bit.
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
> (e1000e): transmit queue 0 timed out, trans_start: 4294737199, wd-timeout:
> 5000 jiffies: 4294745088 tx-queues: 1
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
> (e1000e): transmit queue 0 timed out, trans_start: 4294737200, wd-timeout:
> 5000 jiffies: 4294745088 tx-queues: 1
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut here
> ]------------
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 7 PID: 0
> at /home/greearb/git/linux-4.9.dev.y/net/sched/sch_generic.c:322
> dev_watchdog+0x267/0x270
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 cfg80211 bnep
> bluetooth macvlan wanlink(O) pktgen fuse corete...sunrpc ipmi_d
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: CPU: 7 PID: 0 Comm:
> swapper/7 Tainted: G O 4.9.65+ #21
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3df0
> ffffffff8142d791 0000000000000000 0000000000000000
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3e30
> ffffffff8110f266 000001422fdc3e08 0000000000000000
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: 0000000000001388
> 00000000fffc7d30 ffff880417d0c000 00000000fffc9c00
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8142d791>]
> dump_stack+0x63/0x82
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f266>]
> __warn+0xc6/0xe0
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f338>]
> warn_slowpath_null+0x18/0x20
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da497>]
> dev_watchdog+0x267/0x270
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
> qdisc_rcu_free+0x40/0x40
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117bf70>]
> call_timer_fn+0x30/0x150
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
> qdisc_rcu_free+0x40/0x40
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117c350>]
> run_timer_softirq+0x1f0/0x450
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051021>] ?
> lapic_next_deadline+0x21/0x30
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8118a54d>] ?
> clockevents_program_event+0x7d/0x120
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115101>]
> __do_softirq+0xc1/0x2c0
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115461>]
> irq_exit+0xb1/0xc0
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051c9d>]
> smp_apic_timer_interrupt+0x3d/0x50
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81895842>]
> apic_timer_interrupt+0x82/0x90
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <EOI>
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81726e46>] ?
> cpuidle_enter_state+0x126/0x300
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81727042>]
> cpuidle_enter+0x12/0x20
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff811521ce>]
> call_cpuidle+0x1e/0x40
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8115240a>]
> cpu_startup_entry+0x13a/0x220
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8104fbd9>]
> start_secondary+0x149/0x170
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
> 69e31de175b59d4f ]---
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Reset adapter unexpectedly
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
> eth3: Reset adapter unexpectedly
> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Detected Hardware Unit Hang:
> TDH
> <a8>
> TDT
> <f3>...
> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
> 5000 jiffies: 4294759424 tx-queues: 1
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
> 5000 jiffies: 4294759424 tx-queues: 1
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
> eth3: Reset adapter unexpectedly
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Reset adapter unexpectedly
> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
> (e1000e): transmit queue 0 timed out, trans_start: 4294766123, wd-timeout:
> 5000 jiffies: 4294771200 tx-queues: 1
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
> (e1000e): transmit queue 0 timed out, trans_start: 4294766125, wd-timeout:
> 5000 jiffies: 4294771200 tx-queues: 1
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Reset adapter unexpectedly
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
> eth3: Reset adapter unexpectedly
> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Detected Hardware Unit Hang:
> TDH
> <c8>
> TDT
> <f5>...
> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>
>
> Thanks,
> Ben
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc http://www.candelatech.com
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Intel-wired-lan] e1000e hardware unit hangs
@ 2018-01-24 16:11 ` Alexander Duyck
0 siblings, 0 replies; 13+ messages in thread
From: Alexander Duyck @ 2018-01-24 16:11 UTC (permalink / raw)
To: intel-wired-lan
On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
> Hello,
>
> Anyone have any more suggestions for making e1000e work better? This is
> from a 4.9.65+ kernel,
> with these additional e1000e patches applied:
>
> e1000e: Fix error path in link detection
> e1000e: Fix wrong comment related to link detection
> e1000e: Fix return value test
> e1000e: Separate signaling for link check/link up
> e1000e: Avoid receiver overrun interrupt bursts
Most of these patches shouldn't address anything that would trigger Tx
hangs. They are mostly related to just link detection.
> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
> of bi-directional
> data between a pair of e1000e interfaces :)
>
> No OOM related issues are seen on this kernel...similar test on 4.13 showed
> some OOM
> issues, but I have not debugged that yet...
Really a question like this probably belongs on e1000-devel or
intel-wired-lan so I have added those lists and the e1000e maintainer
to the thread.
It would be useful if you could provide more information about the
device itself such as the ID and the kind of test you are running.
Keep in mind the e1000e driver supports a pretty broad swath of
devices so we need to narrow things down a bit.
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
> (e1000e): transmit queue 0 timed out, trans_start: 4294737199, wd-timeout:
> 5000 jiffies: 4294745088 tx-queues: 1
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
> (e1000e): transmit queue 0 timed out, trans_start: 4294737200, wd-timeout:
> 5000 jiffies: 4294745088 tx-queues: 1
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut here
> ]------------
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 7 PID: 0
> at /home/greearb/git/linux-4.9.dev.y/net/sched/sch_generic.c:322
> dev_watchdog+0x267/0x270
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 cfg80211 bnep
> bluetooth macvlan wanlink(O) pktgen fuse corete...sunrpc ipmi_d
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: CPU: 7 PID: 0 Comm:
> swapper/7 Tainted: G O 4.9.65+ #21
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3df0
> ffffffff8142d791 0000000000000000 0000000000000000
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3e30
> ffffffff8110f266 000001422fdc3e08 0000000000000000
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: 0000000000001388
> 00000000fffc7d30 ffff880417d0c000 00000000fffc9c00
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8142d791>]
> dump_stack+0x63/0x82
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f266>]
> __warn+0xc6/0xe0
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f338>]
> warn_slowpath_null+0x18/0x20
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da497>]
> dev_watchdog+0x267/0x270
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
> qdisc_rcu_free+0x40/0x40
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117bf70>]
> call_timer_fn+0x30/0x150
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
> qdisc_rcu_free+0x40/0x40
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117c350>]
> run_timer_softirq+0x1f0/0x450
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051021>] ?
> lapic_next_deadline+0x21/0x30
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8118a54d>] ?
> clockevents_program_event+0x7d/0x120
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115101>]
> __do_softirq+0xc1/0x2c0
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115461>]
> irq_exit+0xb1/0xc0
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051c9d>]
> smp_apic_timer_interrupt+0x3d/0x50
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81895842>]
> apic_timer_interrupt+0x82/0x90
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <EOI>
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81726e46>] ?
> cpuidle_enter_state+0x126/0x300
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81727042>]
> cpuidle_enter+0x12/0x20
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff811521ce>]
> call_cpuidle+0x1e/0x40
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8115240a>]
> cpu_startup_entry+0x13a/0x220
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8104fbd9>]
> start_secondary+0x149/0x170
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
> 69e31de175b59d4f ]---
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Reset adapter unexpectedly
> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
> eth3: Reset adapter unexpectedly
> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Detected Hardware Unit Hang:
> TDH
> <a8>
> TDT
> <f3>...
> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
> 5000 jiffies: 4294759424 tx-queues: 1
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
> 5000 jiffies: 4294759424 tx-queues: 1
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
> eth3: Reset adapter unexpectedly
> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Reset adapter unexpectedly
> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
> (e1000e): transmit queue 0 timed out, trans_start: 4294766123, wd-timeout:
> 5000 jiffies: 4294771200 tx-queues: 1
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
> (e1000e): transmit queue 0 timed out, trans_start: 4294766125, wd-timeout:
> 5000 jiffies: 4294771200 tx-queues: 1
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Reset adapter unexpectedly
> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
> eth3: Reset adapter unexpectedly
> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
> eth2: Detected Hardware Unit Hang:
> TDH
> <c8>
> TDT
> <f5>...
> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>
>
> Thanks,
> Ben
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc http://www.candelatech.com
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: e1000e hardware unit hangs
2018-01-24 16:11 ` [Intel-wired-lan] " Alexander Duyck
@ 2018-01-24 16:34 ` Neftin, Sasha
-1 siblings, 0 replies; 13+ messages in thread
From: Neftin, Sasha @ 2018-01-24 16:34 UTC (permalink / raw)
To: Alexander Duyck, Ben Greear, intel-wired-lan, e1000-devel; +Cc: netdev
On 1/24/2018 18:11, Alexander Duyck wrote:
> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Hello,
>>
>> Anyone have any more suggestions for making e1000e work better? This is
>> from a 4.9.65+ kernel,
>> with these additional e1000e patches applied:
>>
>> e1000e: Fix error path in link detection
>> e1000e: Fix wrong comment related to link detection
>> e1000e: Fix return value test
>> e1000e: Separate signaling for link check/link up
>> e1000e: Avoid receiver overrun interrupt bursts
>
> Most of these patches shouldn't address anything that would trigger Tx
> hangs. They are mostly related to just link detection.
>
>> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
>> of bi-directional
>> data between a pair of e1000e interfaces :)
>>
>> No OOM related issues are seen on this kernel...similar test on 4.13 showed
>> some OOM
>> issues, but I have not debugged that yet...
>
> Really a question like this probably belongs on e1000-devel or
> intel-wired-lan so I have added those lists and the e1000e maintainer
> to the thread.
>
> It would be useful if you could provide more information about the
> device itself such as the ID and the kind of test you are running.
> Keep in mind the e1000e driver supports a pretty broad swath of
> devices so we need to narrow things down a bit.
>
please, also re-check if your kernel include:
e1000e: fix buffer overrun while the I219 is processing DMA transactions
e1000e: fix the use of magic numbers for buffer overrun issue
where you take fresh version of kernel?
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
>> (e1000e): transmit queue 0 timed out, trans_start: 4294737199, wd-timeout:
>> 5000 jiffies: 4294745088 tx-queues: 1
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
>> (e1000e): transmit queue 0 timed out, trans_start: 4294737200, wd-timeout:
>> 5000 jiffies: 4294745088 tx-queues: 1
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut here
>> ]------------
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 7 PID: 0
>> at /home/greearb/git/linux-4.9.dev.y/net/sched/sch_generic.c:322
>> dev_watchdog+0x267/0x270
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
>> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 cfg80211 bnep
>> bluetooth macvlan wanlink(O) pktgen fuse corete...sunrpc ipmi_d
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: CPU: 7 PID: 0 Comm:
>> swapper/7 Tainted: G O 4.9.65+ #21
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
>> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3df0
>> ffffffff8142d791 0000000000000000 0000000000000000
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3e30
>> ffffffff8110f266 000001422fdc3e08 0000000000000000
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: 0000000000001388
>> 00000000fffc7d30 ffff880417d0c000 00000000fffc9c00
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8142d791>]
>> dump_stack+0x63/0x82
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f266>]
>> __warn+0xc6/0xe0
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f338>]
>> warn_slowpath_null+0x18/0x20
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da497>]
>> dev_watchdog+0x267/0x270
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
>> qdisc_rcu_free+0x40/0x40
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117bf70>]
>> call_timer_fn+0x30/0x150
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
>> qdisc_rcu_free+0x40/0x40
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117c350>]
>> run_timer_softirq+0x1f0/0x450
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051021>] ?
>> lapic_next_deadline+0x21/0x30
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8118a54d>] ?
>> clockevents_program_event+0x7d/0x120
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115101>]
>> __do_softirq+0xc1/0x2c0
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115461>]
>> irq_exit+0xb1/0xc0
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051c9d>]
>> smp_apic_timer_interrupt+0x3d/0x50
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81895842>]
>> apic_timer_interrupt+0x82/0x90
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <EOI>
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81726e46>] ?
>> cpuidle_enter_state+0x126/0x300
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81727042>]
>> cpuidle_enter+0x12/0x20
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff811521ce>]
>> call_cpuidle+0x1e/0x40
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8115240a>]
>> cpu_startup_entry+0x13a/0x220
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8104fbd9>]
>> start_secondary+0x149/0x170
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
>> 69e31de175b59d4f ]---
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Reset adapter unexpectedly
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
>> eth3: Reset adapter unexpectedly
>> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Detected Hardware Unit Hang:
>> TDH
>> <a8>
>> TDT
>> <f3>...
>> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
>> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
>> 5000 jiffies: 4294759424 tx-queues: 1
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
>> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
>> 5000 jiffies: 4294759424 tx-queues: 1
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
>> eth3: Reset adapter unexpectedly
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Reset adapter unexpectedly
>> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
>> (e1000e): transmit queue 0 timed out, trans_start: 4294766123, wd-timeout:
>> 5000 jiffies: 4294771200 tx-queues: 1
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
>> (e1000e): transmit queue 0 timed out, trans_start: 4294766125, wd-timeout:
>> 5000 jiffies: 4294771200 tx-queues: 1
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Reset adapter unexpectedly
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
>> eth3: Reset adapter unexpectedly
>> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Detected Hardware Unit Hang:
>> TDH
>> <c8>
>> TDT
>> <f5>...
>> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>
>>
>> Thanks,
>> Ben
>>
>> --
>> Ben Greear <greearb@candelatech.com>
>> Candela Technologies Inc http://www.candelatech.com
>>
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Intel-wired-lan] e1000e hardware unit hangs
@ 2018-01-24 16:34 ` Neftin, Sasha
0 siblings, 0 replies; 13+ messages in thread
From: Neftin, Sasha @ 2018-01-24 16:34 UTC (permalink / raw)
To: intel-wired-lan
On 1/24/2018 18:11, Alexander Duyck wrote:
> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Hello,
>>
>> Anyone have any more suggestions for making e1000e work better? This is
>> from a 4.9.65+ kernel,
>> with these additional e1000e patches applied:
>>
>> e1000e: Fix error path in link detection
>> e1000e: Fix wrong comment related to link detection
>> e1000e: Fix return value test
>> e1000e: Separate signaling for link check/link up
>> e1000e: Avoid receiver overrun interrupt bursts
>
> Most of these patches shouldn't address anything that would trigger Tx
> hangs. They are mostly related to just link detection.
>
>> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
>> of bi-directional
>> data between a pair of e1000e interfaces :)
>>
>> No OOM related issues are seen on this kernel...similar test on 4.13 showed
>> some OOM
>> issues, but I have not debugged that yet...
>
> Really a question like this probably belongs on e1000-devel or
> intel-wired-lan so I have added those lists and the e1000e maintainer
> to the thread.
>
> It would be useful if you could provide more information about the
> device itself such as the ID and the kind of test you are running.
> Keep in mind the e1000e driver supports a pretty broad swath of
> devices so we need to narrow things down a bit.
>
please, also re-check if your kernel include:
e1000e: fix buffer overrun while the I219 is processing DMA transactions
e1000e: fix the use of magic numbers for buffer overrun issue
where you take fresh version of kernel?
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
>> (e1000e): transmit queue 0 timed out, trans_start: 4294737199, wd-timeout:
>> 5000 jiffies: 4294745088 tx-queues: 1
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
>> (e1000e): transmit queue 0 timed out, trans_start: 4294737200, wd-timeout:
>> 5000 jiffies: 4294745088 tx-queues: 1
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut here
>> ]------------
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 7 PID: 0
>> at /home/greearb/git/linux-4.9.dev.y/net/sched/sch_generic.c:322
>> dev_watchdog+0x267/0x270
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
>> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 cfg80211 bnep
>> bluetooth macvlan wanlink(O) pktgen fuse corete...sunrpc ipmi_d
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: CPU: 7 PID: 0 Comm:
>> swapper/7 Tainted: G O 4.9.65+ #21
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
>> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3df0
>> ffffffff8142d791 0000000000000000 0000000000000000
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ffff88042fdc3e30
>> ffffffff8110f266 000001422fdc3e08 0000000000000000
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: 0000000000001388
>> 00000000fffc7d30 ffff880417d0c000 00000000fffc9c00
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8142d791>]
>> dump_stack+0x63/0x82
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f266>]
>> __warn+0xc6/0xe0
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8110f338>]
>> warn_slowpath_null+0x18/0x20
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da497>]
>> dev_watchdog+0x267/0x270
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
>> qdisc_rcu_free+0x40/0x40
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117bf70>]
>> call_timer_fn+0x30/0x150
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff817da230>] ?
>> qdisc_rcu_free+0x40/0x40
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8117c350>]
>> run_timer_softirq+0x1f0/0x450
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051021>] ?
>> lapic_next_deadline+0x21/0x30
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8118a54d>] ?
>> clockevents_program_event+0x7d/0x120
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115101>]
>> __do_softirq+0xc1/0x2c0
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81115461>]
>> irq_exit+0xb1/0xc0
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81051c9d>]
>> smp_apic_timer_interrupt+0x3d/0x50
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81895842>]
>> apic_timer_interrupt+0x82/0x90
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: <EOI>
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81726e46>] ?
>> cpuidle_enter_state+0x126/0x300
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff81727042>]
>> cpuidle_enter+0x12/0x20
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff811521ce>]
>> call_cpuidle+0x1e/0x40
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8115240a>]
>> cpu_startup_entry+0x13a/0x220
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: [<ffffffff8104fbd9>]
>> start_secondary+0x149/0x170
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
>> 69e31de175b59d4f ]---
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Reset adapter unexpectedly
>> Jan 23 15:38:59 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
>> eth3: Reset adapter unexpectedly
>> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Detected Hardware Unit Hang:
>> TDH
>> <a8>
>> TDT
>> <f3>...
>> Jan 23 15:39:02 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
>> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
>> 5000 jiffies: 4294759424 tx-queues: 1
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
>> (e1000e): transmit queue 0 timed out, trans_start: 4294748730, wd-timeout:
>> 5000 jiffies: 4294759424 tx-queues: 1
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
>> eth3: Reset adapter unexpectedly
>> Jan 23 15:39:13 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Reset adapter unexpectedly
>> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2
>> (e1000e): transmit queue 0 timed out, trans_start: 4294766123, wd-timeout:
>> 5000 jiffies: 4294771200 tx-queues: 1
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
>> (e1000e): transmit queue 0 timed out, trans_start: 4294766125, wd-timeout:
>> 5000 jiffies: 4294771200 tx-queues: 1
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Reset adapter unexpectedly
>> Jan 23 15:39:25 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
>> eth3: Reset adapter unexpectedly
>> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0
>> eth2: Detected Hardware Unit Hang:
>> TDH
>> <c8>
>> TDT
>> <f5>...
>> Jan 23 15:39:28 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>
>>
>> Thanks,
>> Ben
>>
>> --
>> Ben Greear <greearb@candelatech.com>
>> Candela Technologies Inc http://www.candelatech.com
>>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: e1000e hardware unit hangs
2018-01-24 16:34 ` [Intel-wired-lan] " Neftin, Sasha
@ 2018-01-24 18:31 ` Ben Greear
-1 siblings, 0 replies; 13+ messages in thread
From: Ben Greear @ 2018-01-24 18:31 UTC (permalink / raw)
To: Neftin, Sasha, Alexander Duyck, intel-wired-lan, e1000-devel; +Cc: netdev
On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
> On 1/24/2018 18:11, Alexander Duyck wrote:
>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>>> Hello,
>>>
>>> Anyone have any more suggestions for making e1000e work better? This is
>>> from a 4.9.65+ kernel,
>>> with these additional e1000e patches applied:
>>>
>>> e1000e: Fix error path in link detection
>>> e1000e: Fix wrong comment related to link detection
>>> e1000e: Fix return value test
>>> e1000e: Separate signaling for link check/link up
>>> e1000e: Avoid receiver overrun interrupt bursts
>>
>> Most of these patches shouldn't address anything that would trigger Tx
>> hangs. They are mostly related to just link detection.
>>
>>> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
>>> of bi-directional
>>> data between a pair of e1000e interfaces :)
>>>
>>> No OOM related issues are seen on this kernel...similar test on 4.13 showed
>>> some OOM
>>> issues, but I have not debugged that yet...
>>
>> Really a question like this probably belongs on e1000-devel or
>> intel-wired-lan so I have added those lists and the e1000e maintainer
>> to the thread.
>>
>> It would be useful if you could provide more information about the
>> device itself such as the ID and the kind of test you are running.
>> Keep in mind the e1000e driver supports a pretty broad swath of
>> devices so we need to narrow things down a bit.
>>
> please, also re-check if your kernel include:
> e1000e: fix buffer overrun while the I219 is processing DMA transactions
> e1000e: fix the use of magic numbers for buffer overrun issue
> where you take fresh version of kernel?
Hello,
I tried adding those two patches, but I still see this splat shortly after starting
my test. The kernel I am using is here:
https://github.com/greearb/linux-ct-4.13
I've seen similar issues at least back to the 4.0 kernel, including stock kernels and my
own kernels with additional patches.
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499, wd-timeout: 5000
jiffies: 4295304192 tx-queues: 1
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut here ]------------
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0 PID: 0 at /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
dev_watchdog+0x228/0x250
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in: nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c cfg80211 macvlan
wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.13.16+ #22
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name: Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task: ffffffff81e104c0 task.stack: ffffffff81e00000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP: 0010:dev_watchdog+0x228/0x250
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP: 0018:ffff88042fc03e50 EFLAGS: 00010282
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX: 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX: ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP: ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10: 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13: 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2: 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_timer_fn+0x30/0x160
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: run_timer_softirq+0x1f0/0x450
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? lapic_next_deadline+0x21/0x30
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? clockevents_program_event+0x78/0xf0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: __do_softirq+0xc1/0x2c0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: irq_exit+0xb1/0xc0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: smp_apic_timer_interrupt+0x38/0x50
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: apic_timer_interrupt+0x89/0x90
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP: 0010:cpuidle_enter_state+0x12b/0x310
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP: 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX: 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX: 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP: ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10: 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13: ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: </IRQ>
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? cpuidle_enter_state+0x119/0x310
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpuidle_enter+0x12/0x20
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_cpuidle+0x1e/0x40
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: do_idle+0x17f/0x1d0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpu_startup_entry+0x5f/0x70
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: rest_init+0xc9/0xd0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: start_kernel+0x483/0x490
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? early_idt_handler_array+0x120/0x120
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: x86_64_start_reservations+0x2a/0x2c
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: x86_64_start_kernel+0x13c/0x14b
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: secondary_startup_64+0x9f/0x9f
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48 c7 c7 98 23 d4 81 51
41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace 04264863cdced748 ]---
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Down
Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
....
Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337, wd-timeout: 5000
jiffies: 4295767040 tx-queues: 1
Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Detected Hardware Unit Hang:
TDH <43>
TDT <90>...
Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403, wd-timeout: 5000
jiffies: 4295789056 tx-queues: 1
Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883, wd-timeout: 5000
jiffies: 4295809024 tx-queues: 1
Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Detected Hardware Unit Hang:
TDH <10>
TDT <5d>...
Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457, wd-timeout: 5000
jiffies: 4295833088 tx-queues: 1
Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678, wd-timeout: 5000
jiffies: 4295847424 tx-queues: 1
Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Detected Hardware Unit Hang:
TDH <8>
TDT <55>...
Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528, wd-timeout: 5000
jiffies: 4295882240 tx-queues: 1
Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Down
Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
.....
[root@lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
driver: e1000e
version: 3.2.6-k
firmware-version: 2.1-2
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
[root@lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Subsystem: Super Micro Computer Inc Device 0000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 18
Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at b000 [size=32]
Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <128ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
Kernel driver in use: e1000e
Kernel modules: e1000e
My test is a (custom) traffic generator that is setting up 30k tcp connections
between two e1000e ports and sending traffic as fast as possible.
I'd be happy to help you set up this exact tool on your system(s),
but we have seen similar issues with e1000e in other high-speed tests, so I don't think it
is specific to this particular test. Maybe this test makes it easier to reproduce
however.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Intel-wired-lan] e1000e hardware unit hangs
@ 2018-01-24 18:31 ` Ben Greear
0 siblings, 0 replies; 13+ messages in thread
From: Ben Greear @ 2018-01-24 18:31 UTC (permalink / raw)
To: intel-wired-lan
On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
> On 1/24/2018 18:11, Alexander Duyck wrote:
>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>>> Hello,
>>>
>>> Anyone have any more suggestions for making e1000e work better? This is
>>> from a 4.9.65+ kernel,
>>> with these additional e1000e patches applied:
>>>
>>> e1000e: Fix error path in link detection
>>> e1000e: Fix wrong comment related to link detection
>>> e1000e: Fix return value test
>>> e1000e: Separate signaling for link check/link up
>>> e1000e: Avoid receiver overrun interrupt bursts
>>
>> Most of these patches shouldn't address anything that would trigger Tx
>> hangs. They are mostly related to just link detection.
>>
>>> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
>>> of bi-directional
>>> data between a pair of e1000e interfaces :)
>>>
>>> No OOM related issues are seen on this kernel...similar test on 4.13 showed
>>> some OOM
>>> issues, but I have not debugged that yet...
>>
>> Really a question like this probably belongs on e1000-devel or
>> intel-wired-lan so I have added those lists and the e1000e maintainer
>> to the thread.
>>
>> It would be useful if you could provide more information about the
>> device itself such as the ID and the kind of test you are running.
>> Keep in mind the e1000e driver supports a pretty broad swath of
>> devices so we need to narrow things down a bit.
>>
> please, also re-check if your kernel include:
> e1000e: fix buffer overrun while the I219 is processing DMA transactions
> e1000e: fix the use of magic numbers for buffer overrun issue
> where you take fresh version of kernel?
Hello,
I tried adding those two patches, but I still see this splat shortly after starting
my test. The kernel I am using is here:
https://github.com/greearb/linux-ct-4.13
I've seen similar issues at least back to the 4.0 kernel, including stock kernels and my
own kernels with additional patches.
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499, wd-timeout: 5000
jiffies: 4295304192 tx-queues: 1
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut here ]------------
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0 PID: 0 at /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
dev_watchdog+0x228/0x250
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in: nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c cfg80211 macvlan
wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.13.16+ #22
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name: Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task: ffffffff81e104c0 task.stack: ffffffff81e00000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP: 0010:dev_watchdog+0x228/0x250
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP: 0018:ffff88042fc03e50 EFLAGS: 00010282
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX: 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX: ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP: ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10: 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13: 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2: 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_timer_fn+0x30/0x160
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: run_timer_softirq+0x1f0/0x450
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? lapic_next_deadline+0x21/0x30
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? clockevents_program_event+0x78/0xf0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: __do_softirq+0xc1/0x2c0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: irq_exit+0xb1/0xc0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: smp_apic_timer_interrupt+0x38/0x50
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: apic_timer_interrupt+0x89/0x90
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP: 0010:cpuidle_enter_state+0x12b/0x310
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP: 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX: 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX: 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP: ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10: 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13: ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: </IRQ>
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? cpuidle_enter_state+0x119/0x310
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpuidle_enter+0x12/0x20
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_cpuidle+0x1e/0x40
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: do_idle+0x17f/0x1d0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpu_startup_entry+0x5f/0x70
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: rest_init+0xc9/0xd0
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: start_kernel+0x483/0x490
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? early_idt_handler_array+0x120/0x120
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: x86_64_start_reservations+0x2a/0x2c
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: x86_64_start_kernel+0x13c/0x14b
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: secondary_startup_64+0x9f/0x9f
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48 c7 c7 98 23 d4 81 51
41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace 04264863cdced748 ]---
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Down
Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
....
Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337, wd-timeout: 5000
jiffies: 4295767040 tx-queues: 1
Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Detected Hardware Unit Hang:
TDH <43>
TDT <90>...
Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403, wd-timeout: 5000
jiffies: 4295789056 tx-queues: 1
Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883, wd-timeout: 5000
jiffies: 4295809024 tx-queues: 1
Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Detected Hardware Unit Hang:
TDH <10>
TDT <5d>...
Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457, wd-timeout: 5000
jiffies: 4295833088 tx-queues: 1
Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678, wd-timeout: 5000
jiffies: 4295847424 tx-queues: 1
Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:06:00.0 eth2: Reset adapter unexpectedly
Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Detected Hardware Unit Hang:
TDH <8>
TDT <55>...
Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528, wd-timeout: 5000
jiffies: 4295882240 tx-queues: 1
Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Down
Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
.....
[root at lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
driver: e1000e
version: 3.2.6-k
firmware-version: 2.1-2
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
[root at lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Subsystem: Super Micro Computer Inc Device 0000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 18
Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at b000 [size=32]
Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <128ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
Kernel driver in use: e1000e
Kernel modules: e1000e
My test is a (custom) traffic generator that is setting up 30k tcp connections
between two e1000e ports and sending traffic as fast as possible.
I'd be happy to help you set up this exact tool on your system(s),
but we have seen similar issues with e1000e in other high-speed tests, so I don't think it
is specific to this particular test. Maybe this test makes it easier to reproduce
however.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: e1000e hardware unit hangs
2018-01-24 18:31 ` [Intel-wired-lan] " Ben Greear
@ 2018-01-24 18:38 ` Denys Fedoryshchenko
-1 siblings, 0 replies; 13+ messages in thread
From: Denys Fedoryshchenko @ 2018-01-24 18:38 UTC (permalink / raw)
To: Ben Greear
Cc: Neftin, Sasha, Alexander Duyck, intel-wired-lan, e1000-devel,
netdev, netdev-owner
On 2018-01-24 20:31, Ben Greear wrote:
> On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
>> On 1/24/2018 18:11, Alexander Duyck wrote:
>>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com>
>>> wrote:
>>>> Hello,
>>>>
>>>> Anyone have any more suggestions for making e1000e work better?
>>>> This is
>>>> from a 4.9.65+ kernel,
>>>> with these additional e1000e patches applied:
>>>>
>>>> e1000e: Fix error path in link detection
>>>> e1000e: Fix wrong comment related to link detection
>>>> e1000e: Fix return value test
>>>> e1000e: Separate signaling for link check/link up
>>>> e1000e: Avoid receiver overrun interrupt bursts
>>>
>>> Most of these patches shouldn't address anything that would trigger
>>> Tx
>>> hangs. They are mostly related to just link detection.
>>>
>>>> Test case is simply to run 30000 tcp connections each trying to send
>>>> 56Kbps
>>>> of bi-directional
>>>> data between a pair of e1000e interfaces :)
>>>>
>>>> No OOM related issues are seen on this kernel...similar test on 4.13
>>>> showed
>>>> some OOM
>>>> issues, but I have not debugged that yet...
>>>
>>> Really a question like this probably belongs on e1000-devel or
>>> intel-wired-lan so I have added those lists and the e1000e maintainer
>>> to the thread.
>>>
>>> It would be useful if you could provide more information about the
>>> device itself such as the ID and the kind of test you are running.
>>> Keep in mind the e1000e driver supports a pretty broad swath of
>>> devices so we need to narrow things down a bit.
>>>
>> please, also re-check if your kernel include:
>> e1000e: fix buffer overrun while the I219 is processing DMA
>> transactions
>> e1000e: fix the use of magic numbers for buffer overrun issue
>> where you take fresh version of kernel?
>
> Hello,
>
> I tried adding those two patches, but I still see this splat shortly
> after starting
> my test. The kernel I am using is here:
>
> https://github.com/greearb/linux-ct-4.13
>
> I've seen similar issues at least back to the 4.0 kernel, including
> stock kernels and my
> own kernels with additional patches.
>
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499,
> wd-timeout: 5000 jiffies: 4295304192 tx-queues: 1
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut
> here ]------------
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0
> PID: 0 at
> /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
> dev_watchdog+0x228/0x250
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c
> cfg80211 macvlan wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0
> Comm: swapper/0 Tainted: G O 4.13.16+ #22
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task:
> ffffffff81e104c0 task.stack: ffffffff81e00000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
> 0010:dev_watchdog+0x228/0x250
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
> 0018:ffff88042fc03e50 EFLAGS: 00010282
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
> 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
> ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
> ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
> 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
> 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS:
> 0000000000000000(0000) GS:ffff88042fc00000(0000)
> knlGS:0000000000000000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS: 0010 DS: 0000
> ES: 0000 CR0: 0000000080050033
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2:
> 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> qdisc_rcu_free+0x40/0x40
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> call_timer_fn+0x30/0x160
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> qdisc_rcu_free+0x40/0x40
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> run_timer_softirq+0x1f0/0x450
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> lapic_next_deadline+0x21/0x30
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> clockevents_program_event+0x78/0xf0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> __do_softirq+0xc1/0x2c0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: irq_exit+0xb1/0xc0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> smp_apic_timer_interrupt+0x38/0x50
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> apic_timer_interrupt+0x89/0x90
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
> 0010:cpuidle_enter_state+0x12b/0x310
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
> 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
> 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
> 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
> ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
> 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
> ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: </IRQ>
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> cpuidle_enter_state+0x119/0x310
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> cpuidle_enter+0x12/0x20
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> call_cpuidle+0x1e/0x40
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> do_idle+0x17f/0x1d0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> cpu_startup_entry+0x5f/0x70
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> rest_init+0xc9/0xd0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> start_kernel+0x483/0x490
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> early_idt_handler_array+0x120/0x120
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> x86_64_start_reservations+0x2a/0x2c
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> x86_64_start_kernel+0x13c/0x14b
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> secondary_startup_64+0x9f/0x9f
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89
> 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48
> c7 c7 98 23 d4 81 51 41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
> 04264863cdced748 ]---
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Reset adapter unexpectedly
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Down
> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>
> ....
>
>
> Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337,
> wd-timeout: 5000 jiffies: 4295767040 tx-queues: 1
> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Detected Hardware Unit Hang:
> TDH
> <43>
> TDT
> <90>...
> Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403,
> wd-timeout: 5000 jiffies: 4295789056 tx-queues: 1
> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883,
> wd-timeout: 5000 jiffies: 4295809024 tx-queues: 1
> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Reset adapter unexpectedly
> Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
> TDH
> <10>
> TDT
> <5d>...
> Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457,
> wd-timeout: 5000 jiffies: 4295833088 tx-queues: 1
> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678,
> wd-timeout: 5000 jiffies: 4295847424 tx-queues: 1
> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Reset adapter unexpectedly
> Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
> TDH
> <8>
> TDT
> <55>...
> Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528,
> wd-timeout: 5000 jiffies: 4295882240 tx-queues: 1
> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Down
> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>
> .....
>
>
> [root@lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
> driver: e1000e
> version: 3.2.6-k
> firmware-version: 2.1-2
> bus-info: 0000:06:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> [root@lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
> 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
> Connection
> Subsystem: Super Micro Computer Inc Device 0000
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 18
> Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
> Region 2: I/O ports at b000 [size=32]
> Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: [c8] Power Management version 2
> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
> Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
> Address: 0000000000000000 Data: 0000
> Capabilities: [e0] Express (v1) Endpoint, MSI 00
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1
> <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency
> L0s <128ns, L1 <64us
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
> BWMgmt- ABWMgmt-
> Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
> Vector table: BAR=3 offset=00000000
> PBA: BAR=3 offset=00002000
> Capabilities: [100 v1] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
> MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
> MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
> MalfTLP+ ECRC- UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
> Kernel driver in use: e1000e
> Kernel modules: e1000e
>
>
> My test is a (custom) traffic generator that is setting up 30k tcp
> connections
> between two e1000e ports and sending traffic as fast as possible.
> I'd be happy to help you set up this exact tool on your system(s),
> but we have seen similar issues with e1000e in other high-speed tests,
> so I don't think it
> is specific to this particular test. Maybe this test makes it easier
> to reproduce
> however.
Silly suggestion:
Maybe worth to try disabling TSO?
ethtool -K eth2 tso off
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Intel-wired-lan] e1000e hardware unit hangs
@ 2018-01-24 18:38 ` Denys Fedoryshchenko
0 siblings, 0 replies; 13+ messages in thread
From: Denys Fedoryshchenko @ 2018-01-24 18:38 UTC (permalink / raw)
To: intel-wired-lan
On 2018-01-24 20:31, Ben Greear wrote:
> On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
>> On 1/24/2018 18:11, Alexander Duyck wrote:
>>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com>
>>> wrote:
>>>> Hello,
>>>>
>>>> Anyone have any more suggestions for making e1000e work better?
>>>> This is
>>>> from a 4.9.65+ kernel,
>>>> with these additional e1000e patches applied:
>>>>
>>>> e1000e: Fix error path in link detection
>>>> e1000e: Fix wrong comment related to link detection
>>>> e1000e: Fix return value test
>>>> e1000e: Separate signaling for link check/link up
>>>> e1000e: Avoid receiver overrun interrupt bursts
>>>
>>> Most of these patches shouldn't address anything that would trigger
>>> Tx
>>> hangs. They are mostly related to just link detection.
>>>
>>>> Test case is simply to run 30000 tcp connections each trying to send
>>>> 56Kbps
>>>> of bi-directional
>>>> data between a pair of e1000e interfaces :)
>>>>
>>>> No OOM related issues are seen on this kernel...similar test on 4.13
>>>> showed
>>>> some OOM
>>>> issues, but I have not debugged that yet...
>>>
>>> Really a question like this probably belongs on e1000-devel or
>>> intel-wired-lan so I have added those lists and the e1000e maintainer
>>> to the thread.
>>>
>>> It would be useful if you could provide more information about the
>>> device itself such as the ID and the kind of test you are running.
>>> Keep in mind the e1000e driver supports a pretty broad swath of
>>> devices so we need to narrow things down a bit.
>>>
>> please, also re-check if your kernel include:
>> e1000e: fix buffer overrun while the I219 is processing DMA
>> transactions
>> e1000e: fix the use of magic numbers for buffer overrun issue
>> where you take fresh version of kernel?
>
> Hello,
>
> I tried adding those two patches, but I still see this splat shortly
> after starting
> my test. The kernel I am using is here:
>
> https://github.com/greearb/linux-ct-4.13
>
> I've seen similar issues at least back to the 4.0 kernel, including
> stock kernels and my
> own kernels with additional patches.
>
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499,
> wd-timeout: 5000 jiffies: 4295304192 tx-queues: 1
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut
> here ]------------
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0
> PID: 0 at
> /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
> dev_watchdog+0x228/0x250
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c
> cfg80211 macvlan wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0
> Comm: swapper/0 Tainted: G O 4.13.16+ #22
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task:
> ffffffff81e104c0 task.stack: ffffffff81e00000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
> 0010:dev_watchdog+0x228/0x250
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
> 0018:ffff88042fc03e50 EFLAGS: 00010282
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
> 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
> ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
> ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
> 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
> 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS:
> 0000000000000000(0000) GS:ffff88042fc00000(0000)
> knlGS:0000000000000000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS: 0010 DS: 0000
> ES: 0000 CR0: 0000000080050033
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2:
> 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> qdisc_rcu_free+0x40/0x40
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> call_timer_fn+0x30/0x160
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> qdisc_rcu_free+0x40/0x40
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> run_timer_softirq+0x1f0/0x450
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> lapic_next_deadline+0x21/0x30
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> clockevents_program_event+0x78/0xf0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> __do_softirq+0xc1/0x2c0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: irq_exit+0xb1/0xc0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> smp_apic_timer_interrupt+0x38/0x50
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> apic_timer_interrupt+0x89/0x90
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
> 0010:cpuidle_enter_state+0x12b/0x310
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
> 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
> 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
> 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
> ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
> 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
> ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: </IRQ>
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> cpuidle_enter_state+0x119/0x310
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> cpuidle_enter+0x12/0x20
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> call_cpuidle+0x1e/0x40
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> do_idle+0x17f/0x1d0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> cpu_startup_entry+0x5f/0x70
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> rest_init+0xc9/0xd0
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> start_kernel+0x483/0x490
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
> early_idt_handler_array+0x120/0x120
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> x86_64_start_reservations+0x2a/0x2c
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> x86_64_start_kernel+0x13c/0x14b
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
> secondary_startup_64+0x9f/0x9f
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89
> 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48
> c7 c7 98 23 d4 81 51 41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
> 04264863cdced748 ]---
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Reset adapter unexpectedly
> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Down
> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>
> ....
>
>
> Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337,
> wd-timeout: 5000 jiffies: 4295767040 tx-queues: 1
> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Detected Hardware Unit Hang:
> TDH
> <43>
> TDT
> <90>...
> Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403,
> wd-timeout: 5000 jiffies: 4295789056 tx-queues: 1
> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883,
> wd-timeout: 5000 jiffies: 4295809024 tx-queues: 1
> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Reset adapter unexpectedly
> Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
> TDH
> <10>
> TDT
> <5d>...
> Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457,
> wd-timeout: 5000 jiffies: 4295833088 tx-queues: 1
> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678,
> wd-timeout: 5000 jiffies: 4295847424 tx-queues: 1
> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:06:00.0 eth2: Reset adapter unexpectedly
> Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
> TDH
> <8>
> TDT
> <55>...
> Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528,
> wd-timeout: 5000 jiffies: 4295882240 tx-queues: 1
> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e
> 0000:07:00.0 eth3: Reset adapter unexpectedly
> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Down
> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>
> .....
>
>
> [root at lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
> driver: e1000e
> version: 3.2.6-k
> firmware-version: 2.1-2
> bus-info: 0000:06:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> [root at lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
> 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
> Connection
> Subsystem: Super Micro Computer Inc Device 0000
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 18
> Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
> Region 2: I/O ports at b000 [size=32]
> Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: [c8] Power Management version 2
> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
> Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
> Address: 0000000000000000 Data: 0000
> Capabilities: [e0] Express (v1) Endpoint, MSI 00
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1
> <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency
> L0s <128ns, L1 <64us
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
> BWMgmt- ABWMgmt-
> Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
> Vector table: BAR=3 offset=00000000
> PBA: BAR=3 offset=00002000
> Capabilities: [100 v1] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
> MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
> MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
> MalfTLP+ ECRC- UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
> Kernel driver in use: e1000e
> Kernel modules: e1000e
>
>
> My test is a (custom) traffic generator that is setting up 30k tcp
> connections
> between two e1000e ports and sending traffic as fast as possible.
> I'd be happy to help you set up this exact tool on your system(s),
> but we have seen similar issues with e1000e in other high-speed tests,
> so I don't think it
> is specific to this particular test. Maybe this test makes it easier
> to reproduce
> however.
Silly suggestion:
Maybe worth to try disabling TSO?
ethtool -K eth2 tso off
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: e1000e hardware unit hangs
2018-01-24 18:38 ` [Intel-wired-lan] " Denys Fedoryshchenko
@ 2018-01-24 18:41 ` Ben Greear
-1 siblings, 0 replies; 13+ messages in thread
From: Ben Greear @ 2018-01-24 18:41 UTC (permalink / raw)
To: Denys Fedoryshchenko
Cc: Neftin, Sasha, Alexander Duyck, intel-wired-lan, e1000-devel,
netdev, netdev-owner
On 01/24/2018 10:38 AM, Denys Fedoryshchenko wrote:
> On 2018-01-24 20:31, Ben Greear wrote:
>> On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
>>> On 1/24/2018 18:11, Alexander Duyck wrote:
>>>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>>>>> Hello,
>>>>>
>>>>> Anyone have any more suggestions for making e1000e work better? This is
>>>>> from a 4.9.65+ kernel,
>>>>> with these additional e1000e patches applied:
>>>>>
>>>>> e1000e: Fix error path in link detection
>>>>> e1000e: Fix wrong comment related to link detection
>>>>> e1000e: Fix return value test
>>>>> e1000e: Separate signaling for link check/link up
>>>>> e1000e: Avoid receiver overrun interrupt bursts
>>>>
>>>> Most of these patches shouldn't address anything that would trigger Tx
>>>> hangs. They are mostly related to just link detection.
>>>>
>>>>> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
>>>>> of bi-directional
>>>>> data between a pair of e1000e interfaces :)
>>>>>
>>>>> No OOM related issues are seen on this kernel...similar test on 4.13 showed
>>>>> some OOM
>>>>> issues, but I have not debugged that yet...
>>>>
>>>> Really a question like this probably belongs on e1000-devel or
>>>> intel-wired-lan so I have added those lists and the e1000e maintainer
>>>> to the thread.
>>>>
>>>> It would be useful if you could provide more information about the
>>>> device itself such as the ID and the kind of test you are running.
>>>> Keep in mind the e1000e driver supports a pretty broad swath of
>>>> devices so we need to narrow things down a bit.
>>>>
>>> please, also re-check if your kernel include:
>>> e1000e: fix buffer overrun while the I219 is processing DMA transactions
>>> e1000e: fix the use of magic numbers for buffer overrun issue
>>> where you take fresh version of kernel?
>>
>> Hello,
>>
>> I tried adding those two patches, but I still see this splat shortly
>> after starting
>> my test. The kernel I am using is here:
>>
>> https://github.com/greearb/linux-ct-4.13
>>
>> I've seen similar issues at least back to the 4.0 kernel, including
>> stock kernels and my
>> own kernels with additional patches.
>>
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499,
>> wd-timeout: 5000 jiffies: 4295304192 tx-queues: 1
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut
>> here ]------------
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0
>> PID: 0 at
>> /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
>> dev_watchdog+0x228/0x250
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
>> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c
>> cfg80211 macvlan wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0
>> Comm: swapper/0 Tainted: G O 4.13.16+ #22
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
>> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task:
>> ffffffff81e104c0 task.stack: ffffffff81e00000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>> 0010:dev_watchdog+0x228/0x250
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>> 0018:ffff88042fc03e50 EFLAGS: 00010282
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>> 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>> ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>> ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>> 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>> 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS:
>> 0000000000000000(0000) GS:ffff88042fc00000(0000)
>> knlGS:0000000000000000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS: 0010 DS: 0000
>> ES: 0000 CR0: 0000000080050033
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2:
>> 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_timer_fn+0x30/0x160
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> run_timer_softirq+0x1f0/0x450
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> lapic_next_deadline+0x21/0x30
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> clockevents_program_event+0x78/0xf0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: __do_softirq+0xc1/0x2c0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: irq_exit+0xb1/0xc0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> smp_apic_timer_interrupt+0x38/0x50
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> apic_timer_interrupt+0x89/0x90
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>> 0010:cpuidle_enter_state+0x12b/0x310
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>> 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>> 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>> 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>> ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>> 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>> ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: </IRQ>
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> cpuidle_enter_state+0x119/0x310
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpuidle_enter+0x12/0x20
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_cpuidle+0x1e/0x40
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: do_idle+0x17f/0x1d0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpu_startup_entry+0x5f/0x70
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: rest_init+0xc9/0xd0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: start_kernel+0x483/0x490
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> early_idt_handler_array+0x120/0x120
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> x86_64_start_reservations+0x2a/0x2c
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> x86_64_start_kernel+0x13c/0x14b
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> secondary_startup_64+0x9f/0x9f
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89
>> 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48
>> c7 c7 98 23 d4 81 51 41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
>> 04264863cdced748 ]---
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Down
>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>
>> ....
>>
>>
>> Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337,
>> wd-timeout: 5000 jiffies: 4295767040 tx-queues: 1
>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Detected Hardware Unit Hang:
>> TDH <43>
>> TDT
>> <90>...
>> Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403,
>> wd-timeout: 5000 jiffies: 4295789056 tx-queues: 1
>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883,
>> wd-timeout: 5000 jiffies: 4295809024 tx-queues: 1
>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>> Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>> TDH <10>
>> TDT
>> <5d>...
>> Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457,
>> wd-timeout: 5000 jiffies: 4295833088 tx-queues: 1
>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678,
>> wd-timeout: 5000 jiffies: 4295847424 tx-queues: 1
>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>> Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>> TDH <8>
>> TDT
>> <55>...
>> Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528,
>> wd-timeout: 5000 jiffies: 4295882240 tx-queues: 1
>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Down
>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>
>> .....
>>
>>
>> [root@lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
>> driver: e1000e
>> version: 3.2.6-k
>> firmware-version: 2.1-2
>> bus-info: 0000:06:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> [root@lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
>> 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
>> Subsystem: Super Micro Computer Inc Device 0000
>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR+ FastB2B- DisINTx+
>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>> Latency: 0, Cache Line Size: 64 bytes
>> Interrupt: pin A routed to IRQ 18
>> Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
>> Region 2: I/O ports at b000 [size=32]
>> Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
>> Capabilities: [c8] Power Management version 2
>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
>> Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>> Address: 0000000000000000 Data: 0000
>> Capabilities: [e0] Express (v1) Endpoint, MSI 00
>> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
>> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
>> MaxPayload 128 bytes, MaxReadReq 512 bytes
>> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency
>> L0s <128ns, L1 <64us
>> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
>> BWMgmt- ABWMgmt-
>> Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>> Vector table: BAR=3 offset=00000000
>> PBA: BAR=3 offset=00002000
>> Capabilities: [100 v1] Advanced Error Reporting
>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
>> MalfTLP- ECRC- UnsupReq- ACSViol-
>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
>> MalfTLP- ECRC- UnsupReq- ACSViol-
>> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
>> MalfTLP+ ECRC- UnsupReq- ACSViol-
>> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>> Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
>> Kernel driver in use: e1000e
>> Kernel modules: e1000e
>>
>>
>> My test is a (custom) traffic generator that is setting up 30k tcp connections
>> between two e1000e ports and sending traffic as fast as possible.
>> I'd be happy to help you set up this exact tool on your system(s),
>> but we have seen similar issues with e1000e in other high-speed tests,
>> so I don't think it
>> is specific to this particular test. Maybe this test makes it easier
>> to reproduce
>> however.
>
> Silly suggestion:
> Maybe worth to try disabling TSO?
> ethtool -K eth2 tso off
I tried that just now...and the problem did not change.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Intel-wired-lan] e1000e hardware unit hangs
@ 2018-01-24 18:41 ` Ben Greear
0 siblings, 0 replies; 13+ messages in thread
From: Ben Greear @ 2018-01-24 18:41 UTC (permalink / raw)
To: intel-wired-lan
On 01/24/2018 10:38 AM, Denys Fedoryshchenko wrote:
> On 2018-01-24 20:31, Ben Greear wrote:
>> On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
>>> On 1/24/2018 18:11, Alexander Duyck wrote:
>>>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>>>>> Hello,
>>>>>
>>>>> Anyone have any more suggestions for making e1000e work better? This is
>>>>> from a 4.9.65+ kernel,
>>>>> with these additional e1000e patches applied:
>>>>>
>>>>> e1000e: Fix error path in link detection
>>>>> e1000e: Fix wrong comment related to link detection
>>>>> e1000e: Fix return value test
>>>>> e1000e: Separate signaling for link check/link up
>>>>> e1000e: Avoid receiver overrun interrupt bursts
>>>>
>>>> Most of these patches shouldn't address anything that would trigger Tx
>>>> hangs. They are mostly related to just link detection.
>>>>
>>>>> Test case is simply to run 30000 tcp connections each trying to send 56Kbps
>>>>> of bi-directional
>>>>> data between a pair of e1000e interfaces :)
>>>>>
>>>>> No OOM related issues are seen on this kernel...similar test on 4.13 showed
>>>>> some OOM
>>>>> issues, but I have not debugged that yet...
>>>>
>>>> Really a question like this probably belongs on e1000-devel or
>>>> intel-wired-lan so I have added those lists and the e1000e maintainer
>>>> to the thread.
>>>>
>>>> It would be useful if you could provide more information about the
>>>> device itself such as the ID and the kind of test you are running.
>>>> Keep in mind the e1000e driver supports a pretty broad swath of
>>>> devices so we need to narrow things down a bit.
>>>>
>>> please, also re-check if your kernel include:
>>> e1000e: fix buffer overrun while the I219 is processing DMA transactions
>>> e1000e: fix the use of magic numbers for buffer overrun issue
>>> where you take fresh version of kernel?
>>
>> Hello,
>>
>> I tried adding those two patches, but I still see this splat shortly
>> after starting
>> my test. The kernel I am using is here:
>>
>> https://github.com/greearb/linux-ct-4.13
>>
>> I've seen similar issues at least back to the 4.0 kernel, including
>> stock kernels and my
>> own kernels with additional patches.
>>
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499,
>> wd-timeout: 5000 jiffies: 4295304192 tx-queues: 1
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut
>> here ]------------
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0
>> PID: 0 at
>> /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
>> dev_watchdog+0x228/0x250
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
>> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c
>> cfg80211 macvlan wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0
>> Comm: swapper/0 Tainted: G O 4.13.16+ #22
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
>> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task:
>> ffffffff81e104c0 task.stack: ffffffff81e00000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>> 0010:dev_watchdog+0x228/0x250
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>> 0018:ffff88042fc03e50 EFLAGS: 00010282
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>> 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>> ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>> ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>> 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>> 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS:
>> 0000000000000000(0000) GS:ffff88042fc00000(0000)
>> knlGS:0000000000000000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS: 0010 DS: 0000
>> ES: 0000 CR0: 0000000080050033
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2:
>> 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_timer_fn+0x30/0x160
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ? qdisc_rcu_free+0x40/0x40
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> run_timer_softirq+0x1f0/0x450
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> lapic_next_deadline+0x21/0x30
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> clockevents_program_event+0x78/0xf0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: __do_softirq+0xc1/0x2c0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: irq_exit+0xb1/0xc0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> smp_apic_timer_interrupt+0x38/0x50
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> apic_timer_interrupt+0x89/0x90
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>> 0010:cpuidle_enter_state+0x12b/0x310
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>> 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>> 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>> 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>> ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>> 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>> ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: </IRQ>
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> cpuidle_enter_state+0x119/0x310
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpuidle_enter+0x12/0x20
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: call_cpuidle+0x1e/0x40
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: do_idle+0x17f/0x1d0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: cpu_startup_entry+0x5f/0x70
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: rest_init+0xc9/0xd0
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: start_kernel+0x483/0x490
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>> early_idt_handler_array+0x120/0x120
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> x86_64_start_reservations+0x2a/0x2c
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> x86_64_start_kernel+0x13c/0x14b
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>> secondary_startup_64+0x9f/0x9f
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89
>> 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48
>> c7 c7 98 23 d4 81 51 41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
>> 04264863cdced748 ]---
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Down
>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>
>> ....
>>
>>
>> Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337,
>> wd-timeout: 5000 jiffies: 4295767040 tx-queues: 1
>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Detected Hardware Unit Hang:
>> TDH <43>
>> TDT
>> <90>...
>> Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403,
>> wd-timeout: 5000 jiffies: 4295789056 tx-queues: 1
>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883,
>> wd-timeout: 5000 jiffies: 4295809024 tx-queues: 1
>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>> Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>> TDH <10>
>> TDT
>> <5d>...
>> Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457,
>> wd-timeout: 5000 jiffies: 4295833088 tx-queues: 1
>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678,
>> wd-timeout: 5000 jiffies: 4295847424 tx-queues: 1
>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>> Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>> TDH <8>
>> TDT
>> <55>...
>> Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528,
>> wd-timeout: 5000 jiffies: 4295882240 tx-queues: 1
>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Down
>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>
>> .....
>>
>>
>> [root at lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
>> driver: e1000e
>> version: 3.2.6-k
>> firmware-version: 2.1-2
>> bus-info: 0000:06:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> [root at lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
>> 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
>> Subsystem: Super Micro Computer Inc Device 0000
>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR+ FastB2B- DisINTx+
>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>> Latency: 0, Cache Line Size: 64 bytes
>> Interrupt: pin A routed to IRQ 18
>> Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
>> Region 2: I/O ports at b000 [size=32]
>> Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
>> Capabilities: [c8] Power Management version 2
>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
>> Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>> Address: 0000000000000000 Data: 0000
>> Capabilities: [e0] Express (v1) Endpoint, MSI 00
>> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
>> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
>> MaxPayload 128 bytes, MaxReadReq 512 bytes
>> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency
>> L0s <128ns, L1 <64us
>> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
>> BWMgmt- ABWMgmt-
>> Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>> Vector table: BAR=3 offset=00000000
>> PBA: BAR=3 offset=00002000
>> Capabilities: [100 v1] Advanced Error Reporting
>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
>> MalfTLP- ECRC- UnsupReq- ACSViol-
>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
>> MalfTLP- ECRC- UnsupReq- ACSViol-
>> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
>> MalfTLP+ ECRC- UnsupReq- ACSViol-
>> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>> Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
>> Kernel driver in use: e1000e
>> Kernel modules: e1000e
>>
>>
>> My test is a (custom) traffic generator that is setting up 30k tcp connections
>> between two e1000e ports and sending traffic as fast as possible.
>> I'd be happy to help you set up this exact tool on your system(s),
>> but we have seen similar issues with e1000e in other high-speed tests,
>> so I don't think it
>> is specific to this particular test. Maybe this test makes it easier
>> to reproduce
>> however.
>
> Silly suggestion:
> Maybe worth to try disabling TSO?
> ethtool -K eth2 tso off
I tried that just now...and the problem did not change.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: e1000e hardware unit hangs
2018-01-24 18:41 ` [Intel-wired-lan] " Ben Greear
@ 2018-01-25 8:29 ` Neftin, Sasha
-1 siblings, 0 replies; 13+ messages in thread
From: Neftin, Sasha @ 2018-01-25 8:29 UTC (permalink / raw)
To: Ben Greear, Denys Fedoryshchenko
Cc: Alexander Duyck, intel-wired-lan, e1000-devel, netdev, netdev-owner
On 1/24/2018 20:41, Ben Greear wrote:
> On 01/24/2018 10:38 AM, Denys Fedoryshchenko wrote:
>> On 2018-01-24 20:31, Ben Greear wrote:
>>> On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
>>>> On 1/24/2018 18:11, Alexander Duyck wrote:
>>>>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear
>>>>> <greearb@candelatech.com> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> Anyone have any more suggestions for making e1000e work better?
>>>>>> This is
>>>>>> from a 4.9.65+ kernel,
>>>>>> with these additional e1000e patches applied:
>>>>>>
>>>>>> e1000e: Fix error path in link detection
>>>>>> e1000e: Fix wrong comment related to link detection
>>>>>> e1000e: Fix return value test
>>>>>> e1000e: Separate signaling for link check/link up
>>>>>> e1000e: Avoid receiver overrun interrupt bursts
>>>>>
>>>>> Most of these patches shouldn't address anything that would trigger Tx
>>>>> hangs. They are mostly related to just link detection.
>>>>>
>>>>>> Test case is simply to run 30000 tcp connections each trying to
>>>>>> send 56Kbps
>>>>>> of bi-directional
>>>>>> data between a pair of e1000e interfaces :)
>>>>>>
>>>>>> No OOM related issues are seen on this kernel...similar test on
>>>>>> 4.13 showed
>>>>>> some OOM
>>>>>> issues, but I have not debugged that yet...
>>>>>
>>>>> Really a question like this probably belongs on e1000-devel or
>>>>> intel-wired-lan so I have added those lists and the e1000e maintainer
>>>>> to the thread.
>>>>>
>>>>> It would be useful if you could provide more information about the
>>>>> device itself such as the ID and the kind of test you are running.
>>>>> Keep in mind the e1000e driver supports a pretty broad swath of
>>>>> devices so we need to narrow things down a bit.
>>>>>
>>>> please, also re-check if your kernel include:
>>>> e1000e: fix buffer overrun while the I219 is processing DMA
>>>> transactions
>>>> e1000e: fix the use of magic numbers for buffer overrun issue
>>>> where you take fresh version of kernel?
>>>
>>> Hello,
>>>
>>> I tried adding those two patches, but I still see this splat shortly
>>> after starting
>>> my test. The kernel I am using is here:
>>>
>>> https://github.com/greearb/linux-ct-4.13
>>>
>>> I've seen similar issues at least back to the 4.0 kernel, including
>>> stock kernels and my
>>> own kernels with additional patches.
>>>
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499,
>>> wd-timeout: 5000 jiffies: 4295304192 tx-queues: 1
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut
>>> here ]------------
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0
>>> PID: 0 at
>>> /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
>>> dev_watchdog+0x228/0x250
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
>>> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c
>>> cfg80211 macvlan wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0
>>> Comm: swapper/0 Tainted: G O 4.13.16+ #22
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
>>> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task:
>>> ffffffff81e104c0 task.stack: ffffffff81e00000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>>> 0010:dev_watchdog+0x228/0x250
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>>> 0018:ffff88042fc03e50 EFLAGS: 00010282
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>>> 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>>> ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>>> ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>>> 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>>> 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS:
>>> 0000000000000000(0000) GS:ffff88042fc00000(0000)
>>> knlGS:0000000000000000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS: 0010 DS: 0000
>>> ES: 0000 CR0: 0000000080050033
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2:
>>> 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: <IRQ>
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>>> qdisc_rcu_free+0x40/0x40
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> call_timer_fn+0x30/0x160
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>>> qdisc_rcu_free+0x40/0x40
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> run_timer_softirq+0x1f0/0x450
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>>> lapic_next_deadline+0x21/0x30
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>>> clockevents_program_event+0x78/0xf0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> __do_softirq+0xc1/0x2c0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: irq_exit+0xb1/0xc0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> smp_apic_timer_interrupt+0x38/0x50
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> apic_timer_interrupt+0x89/0x90
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>>> 0010:cpuidle_enter_state+0x12b/0x310
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>>> 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>>> 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>>> 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>>> ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>>> 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>>> ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: </IRQ>
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>>> cpuidle_enter_state+0x119/0x310
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> cpuidle_enter+0x12/0x20
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> call_cpuidle+0x1e/0x40
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: do_idle+0x17f/0x1d0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> cpu_startup_entry+0x5f/0x70
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: rest_init+0xc9/0xd0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> start_kernel+0x483/0x490
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ?
>>> early_idt_handler_array+0x120/0x120
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> x86_64_start_reservations+0x2a/0x2c
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> x86_64_start_kernel+0x13c/0x14b
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> secondary_startup_64+0x9f/0x9f
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89
>>> 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48
>>> c7 c7 98 23 d4 81 51 41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
>>> 04264863cdced748 ]---
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Down
>>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>>
>>> ....
>>>
>>>
>>> Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337,
>>> wd-timeout: 5000 jiffies: 4295767040 tx-queues: 1
>>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Detected Hardware Unit Hang:
>>>
>>> TDH <43>
>>> TDT
>>> <90>...
>>> Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403,
>>> wd-timeout: 5000 jiffies: 4295789056 tx-queues: 1
>>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883,
>>> wd-timeout: 5000 jiffies: 4295809024 tx-queues: 1
>>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>>> Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>>>
>>> TDH <10>
>>> TDT
>>> <5d>...
>>> Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457,
>>> wd-timeout: 5000 jiffies: 4295833088 tx-queues: 1
>>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678,
>>> wd-timeout: 5000 jiffies: 4295847424 tx-queues: 1
>>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>>> Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>>>
>>> TDH <8>
>>> TDT
>>> <55>...
>>> Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528,
>>> wd-timeout: 5000 jiffies: 4295882240 tx-queues: 1
>>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Down
>>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>>
>>> .....
>>>
>>>
>>> [root@lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
>>> driver: e1000e
>>> version: 3.2.6-k
>>> firmware-version: 2.1-2
>>> bus-info: 0000:06:00.0
>>> supports-statistics: yes
>>> supports-test: yes
>>> supports-eeprom-access: yes
>>> supports-register-dump: yes
>>> supports-priv-flags: no
>>>
>>> [root@lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
>>> 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
>>> Connection
>>> Subsystem: Super Micro Computer Inc Device 0000
>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR+ FastB2B- DisINTx+
>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>> Latency: 0, Cache Line Size: 64 bytes
>>> Interrupt: pin A routed to IRQ 18
>>> Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
>>> Region 2: I/O ports at b000 [size=32]
>>> Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
>>> Capabilities: [c8] Power Management version 2
>>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
>>> Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>>> Address: 0000000000000000 Data: 0000
>>> Capabilities: [e0] Express (v1) Endpoint, MSI 00
>>> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <512ns, L1 <64us
>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>>> DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
>>> Unsupported+
>>> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>> MaxPayload 128 bytes, MaxReadReq 512 bytes
>>> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+
>>> TransPend-
>>> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>> Exit Latency
>>> L0s <128ns, L1 <64us
>>> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
>>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
>>> DLActive-
>>> BWMgmt- ABWMgmt-
>>> Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>>> Vector table: BAR=3 offset=00000000
>>> PBA: BAR=3 offset=00002000
>>> Capabilities: [100 v1] Advanced Error Reporting
>>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
>>> RxOF-
>>> MalfTLP- ECRC- UnsupReq- ACSViol-
>>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
>>> RxOF-
>>> MalfTLP- ECRC- UnsupReq- ACSViol-
>>> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
>>> RxOF+
>>> MalfTLP+ ECRC- UnsupReq- ACSViol-
>>> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>>> NonFatalErr-
>>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>>> NonFatalErr+
>>> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap-
>>> ChkEn-
>>> Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
>>> Kernel driver in use: e1000e
>>> Kernel modules: e1000e
>>>
>>>
>>> My test is a (custom) traffic generator that is setting up 30k tcp
>>> connections
>>> between two e1000e ports and sending traffic as fast as possible.
>>> I'd be happy to help you set up this exact tool on your system(s),
>>> but we have seen similar issues with e1000e in other high-speed tests,
>>> so I don't think it
>>> is specific to this particular test. Maybe this test makes it easier
>>> to reproduce
>>> however.
>>
>> Silly suggestion:
>> Maybe worth to try disabling TSO?
>> ethtool -K eth2 tso off
>
>
> I tried that just now...and the problem did not change.
>
> Thanks,
> Ben
>
>
>
82574L is pretty old HW - I am not sure we still support it. Is more
older kernel version also hit on this problem? Can you try latest Linus
kernel version? Anyway, I suggest fill ticket on source forge
(https://sourceforge.net/projects/e1000/files/?source=navbar),attach
dmesg, lspci and all relevant information.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Intel-wired-lan] e1000e hardware unit hangs
@ 2018-01-25 8:29 ` Neftin, Sasha
0 siblings, 0 replies; 13+ messages in thread
From: Neftin, Sasha @ 2018-01-25 8:29 UTC (permalink / raw)
To: intel-wired-lan
On 1/24/2018 20:41, Ben Greear wrote:
> On 01/24/2018 10:38 AM, Denys Fedoryshchenko wrote:
>> On 2018-01-24 20:31, Ben Greear wrote:
>>> On 01/24/2018 08:34 AM, Neftin, Sasha wrote:
>>>> On 1/24/2018 18:11, Alexander Duyck wrote:
>>>>> On Tue, Jan 23, 2018 at 3:46 PM, Ben Greear
>>>>> <greearb@candelatech.com> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> Anyone have any more suggestions for making e1000e work better?
>>>>>> This is
>>>>>> from a 4.9.65+ kernel,
>>>>>> with these additional e1000e patches applied:
>>>>>>
>>>>>> e1000e: Fix error path in link detection
>>>>>> e1000e: Fix wrong comment related to link detection
>>>>>> e1000e: Fix return value test
>>>>>> e1000e: Separate signaling for link check/link up
>>>>>> e1000e: Avoid receiver overrun interrupt bursts
>>>>>
>>>>> Most of these patches shouldn't address anything that would trigger Tx
>>>>> hangs. They are mostly related to just link detection.
>>>>>
>>>>>> Test case is simply to run 30000 tcp connections each trying to
>>>>>> send 56Kbps
>>>>>> of bi-directional
>>>>>> data between a pair of e1000e interfaces :)
>>>>>>
>>>>>> No OOM related issues are seen on this kernel...similar test on
>>>>>> 4.13 showed
>>>>>> some OOM
>>>>>> issues, but I have not debugged that yet...
>>>>>
>>>>> Really a question like this probably belongs on e1000-devel or
>>>>> intel-wired-lan so I have added those lists and the e1000e maintainer
>>>>> to the thread.
>>>>>
>>>>> It would be useful if you could provide more information about the
>>>>> device itself such as the ID and the kind of test you are running.
>>>>> Keep in mind the e1000e driver supports a pretty broad swath of
>>>>> devices so we need to narrow things down a bit.
>>>>>
>>>> please, also re-check if your kernel include:
>>>> e1000e: fix buffer overrun while the I219 is processing DMA
>>>> transactions
>>>> e1000e: fix the use of magic numbers for buffer overrun issue
>>>> where you take fresh version of kernel?
>>>
>>> Hello,
>>>
>>> I tried adding those two patches, but I still see this splat shortly
>>> after starting
>>> my test.? The kernel I am using is here:
>>>
>>> https://github.com/greearb/linux-ct-4.13
>>>
>>> I've seen similar issues at least back to the 4.0 kernel, including
>>> stock kernels and my
>>> own kernels with additional patches.
>>>
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295298499,
>>> wd-timeout: 5000 jiffies: 4295304192 tx-queues: 1
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ------------[ cut
>>> here ]------------
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: WARNING: CPU: 0
>>> PID: 0 at
>>> /home/greearb/git/linux-4.13.dev.y/net/sched/sch_generic.c:322
>>> dev_watchdog+0x228/0x250
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Modules linked in:
>>> nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 libcrc32c
>>> cfg80211 macvlan wanlink(O) pktgen bnep bluetooth f...ss tpm_tis ip
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CPU: 0 PID: 0
>>> Comm: swapper/0 Tainted: G?????????? O??? 4.13.16+ #22
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Hardware name:
>>> Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: task:
>>> ffffffff81e104c0 task.stack: ffffffff81e00000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>>> 0010:dev_watchdog+0x228/0x250
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>>> 0018:ffff88042fc03e50 EFLAGS: 00010282
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>>> 0000000000000086 RBX: 0000000000000000 RCX: 0000000000000000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>>> ffff88042fc15b40 RSI: ffff88042fc0dbf8 RDI: ffff88042fc0dbf8
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>>> ffff88042fc03e98 R08: 0000000000000001 R09: 00000000000003c4
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>>> 0000000000000000 R11: 00000000000003c4 R12: 0000000000001388
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>>> 0000000100050dc3 R14: ffff880417670000 R15: 0000000100052400
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: FS:
>>> 0000000000000000(0000) GS:ffff88042fc00000(0000)
>>> knlGS:0000000000000000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CS:? 0010 DS: 0000
>>> ES: 0000 CR0: 0000000080050033
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: CR2:
>>> 0000000001d14000 CR3: 0000000001e09000 CR4: 00000000001406f0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Call Trace:
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? <IRQ>
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? ?
>>> qdisc_rcu_free+0x40/0x40
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> call_timer_fn+0x30/0x160
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? ?
>>> qdisc_rcu_free+0x40/0x40
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> run_timer_softirq+0x1f0/0x450
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? ?
>>> lapic_next_deadline+0x21/0x30
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? ?
>>> clockevents_program_event+0x78/0xf0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> __do_softirq+0xc1/0x2c0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? irq_exit+0xb1/0xc0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> smp_apic_timer_interrupt+0x38/0x50
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> apic_timer_interrupt+0x89/0x90
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RIP:
>>> 0010:cpuidle_enter_state+0x12b/0x310
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RSP:
>>> 0018:ffffffff81e03de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RAX:
>>> 0000000000000000 RBX: 0000000000000003 RCX: 000000000000001f
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RDX:
>>> 0000000000000000 RSI: 00000000238e2b4c RDI: 0000000000000000
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: RBP:
>>> ffffffff81e03e20 R08: 00000000000000af R09: 0000000000000018
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R10:
>>> 00000000000000af R11: 0000000000000f27 R12: 0000000000000003
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: R13:
>>> ffff88042fc24918 R14: ffffffff81eae658 R15: 00000093fd9af742
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? </IRQ>
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? ?
>>> cpuidle_enter_state+0x119/0x310
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> cpuidle_enter+0x12/0x20
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> call_cpuidle+0x1e/0x40
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? do_idle+0x17f/0x1d0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> cpu_startup_entry+0x5f/0x70
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? rest_init+0xc9/0xd0
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> start_kernel+0x483/0x490
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:? ?
>>> early_idt_handler_array+0x120/0x120
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> x86_64_start_reservations+0x2a/0x2c
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> x86_64_start_kernel+0x13c/0x14b
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel:
>>> secondary_startup_64+0x9f/0x9f
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: Code: 04 00 00 89
>>> 4d cc e8 b8 88 fd ff 8b 4d cc 45 89 e1 4d 89 e8 48 89 c2 4c 89 f6 48
>>> c7 c7 98 23 d4 81 51 41 57 89 d9 e8 44 48 94 ff <0f>... 63 8e 60 04
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: ---[ end trace
>>> 04264863cdced748 ]---
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>>> Jan 24 10:19:42 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Down
>>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:19:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>>
>>> ....
>>>
>>>
>>> Jan 24 10:27:05 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295760337,
>>> wd-timeout: 5000 jiffies: 4295767040 tx-queues: 1
>>> Jan 24 10:27:24 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:27:27 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Detected Hardware Unit Hang:
>>>
>>> TDH????????????????? <43>
>>> ????????????????????????????????????????????????????? TDT
>>> ??? <90>...
>>> Jan 24 10:27:29 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295782403,
>>> wd-timeout: 5000 jiffies: 4295789056 tx-queues: 1
>>> Jan 24 10:27:46 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:27:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295802883,
>>> wd-timeout: 5000 jiffies: 4295809024 tx-queues: 1
>>> Jan 24 10:28:06 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>>> Jan 24 10:28:10 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>>>
>>> TDH????????????????? <10>
>>> ????????????????????????????????????????????????????? TDT
>>> ??? <5d>...
>>> Jan 24 10:28:11 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295827457,
>>> wd-timeout: 5000 jiffies: 4295833088 tx-queues: 1
>>> Jan 24 10:28:30 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:28:35 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth2 (e1000e): transmit queue 0 timed out, trans_start: 4295841678,
>>> wd-timeout: 5000 jiffies: 4295847424 tx-queues: 1
>>> Jan 24 10:28:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:06:00.0 eth2: Reset adapter unexpectedly
>>> Jan 24 10:28:48 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Detected Hardware Unit Hang:
>>>
>>> TDH????????????????? <8>
>>> ????????????????????????????????????????????????????? TDT
>>> ??? <55>...
>>> Jan 24 10:28:49 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG:
>>> eth3 (e1000e): transmit queue 0 timed out, trans_start: 4295874528,
>>> wd-timeout: 5000 jiffies: 4295882240 tx-queues: 1
>>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e
>>> 0000:07:00.0 eth3: Reset adapter unexpectedly
>>> Jan 24 10:29:20 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Down
>>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth2 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>> Jan 24 10:29:26 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC
>>> Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>>>
>>> .....
>>>
>>>
>>> [root at lf1003-e3v2-13100124-f20x64 ~]# ethtool -i eth2
>>> driver: e1000e
>>> version: 3.2.6-k
>>> firmware-version: 2.1-2
>>> bus-info: 0000:06:00.0
>>> supports-statistics: yes
>>> supports-test: yes
>>> supports-eeprom-access: yes
>>> supports-register-dump: yes
>>> supports-priv-flags: no
>>>
>>> [root at lf1003-e3v2-13100124-f20x64 ~]# lspci -vvv -s 0000:06:00.0
>>> 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
>>> Connection
>>> ??? Subsystem: Super Micro Computer Inc Device 0000
>>> ??? Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR+ FastB2B- DisINTx+
>>> ??? Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>> ??? Latency: 0, Cache Line Size: 64 bytes
>>> ??? Interrupt: pin A routed to IRQ 18
>>> ??? Region 0: Memory at df600000 (32-bit, non-prefetchable) [size=128K]
>>> ??? Region 2: I/O ports at b000 [size=32]
>>> ??? Region 3: Memory at df620000 (32-bit, non-prefetchable) [size=16K]
>>> ??? Capabilities: [c8] Power Management version 2
>>> ??????? Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>>> ??????? Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
>>> ??? Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>>> ??????? Address: 0000000000000000? Data: 0000
>>> ??? Capabilities: [e0] Express (v1) Endpoint, MSI 00
>>> ??????? DevCap:??? MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <512ns, L1 <64us
>>> ??????????? ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>>> ??????? DevCtl:??? Report errors: Correctable+ Non-Fatal+ Fatal+
>>> Unsupported+
>>> ??????????? RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>> ??????????? MaxPayload 128 bytes, MaxReadReq 512 bytes
>>> ??????? DevSta:??? CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+
>>> TransPend-
>>> ??????? LnkCap:??? Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>> Exit Latency
>>> L0s <128ns, L1 <64us
>>> ??????????? ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>>> ??????? LnkCtl:??? ASPM Disabled; RCB 64 bytes Disabled- CommClk+
>>> ??????????? ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>> ??????? LnkSta:??? Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
>>> DLActive-
>>> BWMgmt- ABWMgmt-
>>> ??? Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>>> ??????? Vector table: BAR=3 offset=00000000
>>> ??????? PBA: BAR=3 offset=00002000
>>> ??? Capabilities: [100 v1] Advanced Error Reporting
>>> ??????? UESta:??? DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
>>> RxOF-
>>> MalfTLP- ECRC- UnsupReq- ACSViol-
>>> ??????? UEMsk:??? DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
>>> RxOF-
>>> MalfTLP- ECRC- UnsupReq- ACSViol-
>>> ??????? UESvrt:??? DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
>>> RxOF+
>>> MalfTLP+ ECRC- UnsupReq- ACSViol-
>>> ??????? CESta:??? RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>>> NonFatalErr-
>>> ??????? CEMsk:??? RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>>> NonFatalErr+
>>> ??????? AERCap:??? First Error Pointer: 00, GenCap- CGenEn- ChkCap-
>>> ChkEn-
>>> ??? Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d2-06-aa
>>> ??? Kernel driver in use: e1000e
>>> ??? Kernel modules: e1000e
>>>
>>>
>>> My test is a (custom) traffic generator that is setting up 30k tcp
>>> connections
>>> between two e1000e ports and sending traffic as fast as possible.
>>> I'd be happy to help you set up this exact tool on your system(s),
>>> but we have seen similar issues with e1000e in other high-speed tests,
>>> so I don't think it
>>> is specific to this particular test.? Maybe this test makes it easier
>>> to reproduce
>>> however.
>>
>> Silly suggestion:
>> Maybe worth to try disabling TSO?
>> ethtool -K eth2 tso off
>
>
> I tried that just now...and the problem did not change.
>
> Thanks,
> Ben
>
>
>
82574L is pretty old HW - I am not sure we still support it. Is more
older kernel version also hit on this problem? Can you try latest Linus
kernel version? Anyway, I suggest fill ticket on source forge
(https://sourceforge.net/projects/e1000/files/?source=navbar),attach
dmesg, lspci and all relevant information.
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2018-01-25 8:29 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-23 23:46 e1000e hardware unit hangs Ben Greear
2018-01-24 16:11 ` Alexander Duyck
2018-01-24 16:11 ` [Intel-wired-lan] " Alexander Duyck
2018-01-24 16:34 ` Neftin, Sasha
2018-01-24 16:34 ` [Intel-wired-lan] " Neftin, Sasha
2018-01-24 18:31 ` Ben Greear
2018-01-24 18:31 ` [Intel-wired-lan] " Ben Greear
2018-01-24 18:38 ` Denys Fedoryshchenko
2018-01-24 18:38 ` [Intel-wired-lan] " Denys Fedoryshchenko
2018-01-24 18:41 ` Ben Greear
2018-01-24 18:41 ` [Intel-wired-lan] " Ben Greear
2018-01-25 8:29 ` Neftin, Sasha
2018-01-25 8:29 ` [Intel-wired-lan] " Neftin, Sasha
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.