* bnx2 cards intermittantly going offline @ 2011-01-18 10:54 Mills, Tony 2011-01-18 17:55 ` Michael Chan 2011-11-15 17:41 ` Ken 0 siblings, 2 replies; 8+ messages in thread From: Mills, Tony @ 2011-01-18 10:54 UTC (permalink / raw) To: netdev Hi, I was running Debian lenny 64bit with a 2.6.24 kernel which seemed to have a rather old version of the bnx2 driver, I have been getting reports that there have been issues with connectivity, this seems to happen randomly across many different servers in different data centres. Further investigation showed that the interfaces become completely unresponsive for periods of time, whereby machines are unable to ping the host with the problem and the server with the problem is unable to ping out, our tcp application which is time critical will kick off connections. The network cards are Broadcom NetXtreme II BCM5708 Gigabit Ethernet cards on Dell 2950's plugged into Cisco 3750E's. Reading various posts indicated that I might be experiencing a problem that may have already been solved so attempted to build the drivers from the Broadcom website into the 2.4.24 kernel without success, eventually compiling against a 2.3.32 kernel worked great. I have installed this on 4 machines in different data centres and followed some of the other posts I have found, in an attempt to fix the issues, however none of the things i have tried appear to be affective :- 1. Was seeing rx_fw_discards so upped rx ring buffer to both 1020 and 4080, this stopped the rx_fw_discards but not the "unresponsiveness". 2. Have enabled flow control on one of the machines, this still has the unresponsive behaviour and the port on the switch shows 0 pause frames received. 3. Upgraded the kernel and driver to latest available. Last night i setup a machine to monitor overnight and at 3:52 this morning it became unresponsive. The switch was setup to "flowcontrol desired", and the machine had the following settings:- ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4080 RX Mini: 0 RX Jumbo: 16320 TX: 255 Current hardware settings: RX: 4080 RX Mini: 0 RX Jumbo: 0 TX: 255 ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: on TX: on The output from ethtool -S eth0 NIC statistics: rx_bytes: 65832403312 rx_error_bytes: 0 tx_bytes: 141615699363 tx_error_bytes: 0 rx_ucast_packets: 565468011 rx_mcast_packets: 3 rx_bcast_packets: 193008 tx_ucast_packets: 768277404 tx_mcast_packets: 8 tx_bcast_packets: 657 tx_mac_errors: 0 tx_carrier_errors: 0 rx_crc_errors: 0 rx_align_errors: 0 tx_single_collisions: 0 tx_multi_collisions: 0 tx_deferred: 0 tx_excess_collisions: 0 tx_late_collisions: 0 tx_total_collisions: 0 rx_fragments: 0 rx_jabbers: 0 rx_undersize_packets: 0 rx_oversize_packets: 0 rx_64_byte_packets: 398958533 rx_65_to_127_byte_packets: 125222178 rx_128_to_255_byte_packets: 16962519 rx_256_to_511_byte_packets: 6100929 rx_512_to_1023_byte_packets: 2314593 rx_1024_to_1522_byte_packets: 16102270 rx_1523_to_9022_byte_packets: 0 tx_64_byte_packets: 331974057 tx_65_to_127_byte_packets: 239480821 tx_128_to_255_byte_packets: 78102231 tx_256_to_511_byte_packets: 33163946 tx_512_to_1023_byte_packets: 57321357 tx_1024_to_1522_byte_packets: 28235657 tx_1523_to_9022_byte_packets: 0 rx_xon_frames: 0 rx_xoff_frames: 0 tx_xon_frames: 0 tx_xoff_frames: 0 rx_mac_ctrl_frames: 0 rx_filtered_packets: 43417 rx_ftq_discards: 0 rx_discards: 0 rx_fw_discards: 0 The switch port showed no pause frames show interfaces gigabitEthernet X/X/X flowcontrol Port Send FlowControl Receive FlowControl RxPause TxPause admin oper admin oper --------- -------- -------- -------- -------- ------- ------- GiX/X/X Unsupp. Unsupp. desired on 0 0 (The switch is unable to send flow control packets but can receive). Would someone be able to point me at anything else that may help identify/fix the issue. Best Regards Tony Mills -- IMPORTANT NOTICE The sender does not guarantee that this message, including any attachment, is secure or virus free. Also, it is confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, do not disclose or copy it or its contents. Please telephone or email the sender and delete the message entirely from your system. Jagex Limited is a company registered in England & Wales with company number 03982706 and a registered office at St John's Innovation Centre, Cowley Road, Cambridge, CB4 0WS, UK. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: bnx2 cards intermittantly going offline 2011-01-18 10:54 bnx2 cards intermittantly going offline Mills, Tony @ 2011-01-18 17:55 ` Michael Chan 2011-01-26 12:44 ` Mills, Tony 2012-09-13 13:51 ` Marc A. Donges 2011-11-15 17:41 ` Ken 1 sibling, 2 replies; 8+ messages in thread From: Michael Chan @ 2011-01-18 17:55 UTC (permalink / raw) To: Mills, Tony; +Cc: netdev On Tue, 2011-01-18 at 02:54 -0800, Mills, Tony wrote: > Last night i setup a machine to monitor overnight and at 3:52 this > morning it became unresponsive. > When it becomes unresponsive, please send some packets to the NIC (such as ping) and monitor statistics with ethtool -S. See if the packets are being received or discarded. Also, run tcpdump on the machine to see if the packets are properly received by the stack. Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: bnx2 cards intermittantly going offline 2011-01-18 17:55 ` Michael Chan @ 2011-01-26 12:44 ` Mills, Tony 2012-09-13 13:51 ` Marc A. Donges 1 sibling, 0 replies; 8+ messages in thread From: Mills, Tony @ 2011-01-26 12:44 UTC (permalink / raw) To: Michael Chan; +Cc: netdev Hi, Thanks for your response. I have done some further investigation and found that we have had a massive amount of interrupts occurring on our Broadcom cards, i have set the affinity for irq 36 and 48 to run on the first two vcpu's on the box and the java processes to run on the others. I have also replaced the debian bnx2 driver with the latest from the Broadcom website and made the rx ring buffer to 4080, this has stopped a multiplexed server running at 1.6 cycles per second from missing cycles due to interrupts on the interface and allowing much better processing time, and the ring buffer up at 4080 stops the rx_fw_discards i was seeing periodically, (even upping that to 1020 or 2040 did not sort the issue but the maximum setting from the Broadcom driver does. I am now monitoring the system to see if the card ever becomes unresponsive. But i do have a question. If i setup the smp_affinity with a mask of cpu 0 and 1 (the first two on the box) for the APIC-fasteoi irq's for the Ethernet devices, it appears that the kernel does not balance and will continue to use the same cpu to do the interrupts even though there is processing power on the other one. Is this a known issue or am i doing something wrong? Can i add it's on an dell r610 with a 12 core intel Xeon X5680, this shows up as 24 vcpu's. Best Regards Tony Mills -----Original Message----- From: Michael Chan [mailto:mchan@broadcom.com] Sent: 18 January 2011 17:56 To: Mills, Tony Cc: netdev@vger.kernel.org Subject: Re: bnx2 cards intermittantly going offline On Tue, 2011-01-18 at 02:54 -0800, Mills, Tony wrote: > Last night i setup a machine to monitor overnight and at 3:52 this > morning it became unresponsive. > When it becomes unresponsive, please send some packets to the NIC (such as ping) and monitor statistics with ethtool -S. See if the packets are being received or discarded. Also, run tcpdump on the machine to see if the packets are properly received by the stack. Thanks. -- IMPORTANT NOTICE The sender does not guarantee that this message, including any attachment, is secure or virus free. Also, it is confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, do not disclose or copy it or its contents. Please telephone or email the sender and delete the message entirely from your system. Jagex Limited is a company registered in England & Wales with company number 03982706 and a registered office at St John's Innovation Centre, Cowley Road, Cambridge, CB4 0WS, UK. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: bnx2 cards intermittantly going offline 2011-01-18 17:55 ` Michael Chan 2011-01-26 12:44 ` Mills, Tony @ 2012-09-13 13:51 ` Marc A. Donges 2012-09-13 15:45 ` Sven Ulland 1 sibling, 1 reply; 8+ messages in thread From: Marc A. Donges @ 2012-09-13 13:51 UTC (permalink / raw) To: netdev; +Cc: Michael Chan [This is a reply to a somewhat older thread] "Michael Chan" wrote: > On Tue, 2011-01-18 at 02:54 -0800, Mills, Tony wrote: >> Last night i setup a machine to monitor overnight and at 3:52 this >> morning it became unresponsive. >> > > When it becomes unresponsive, please send some packets to the NIC (such > as ping) and monitor statistics with ethtool -S. See if the packets are > being received or discarded. Also, run tcpdump on the machine to see if > the packets are properly received by the stack. Thanks. Hi Michael, hi netdev, I appear to be having the same problem as Tony (or at least a problem matching his description). The machine uses the BCM5709 chipset: 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 04:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) It is running Debian stable (with the Debian stable firmware-bnx2 package). After 55 days of operation the machine (A) suddenly was no longer reachable via network. Strangely, a second machine (B) that should take over the IP addresses (keepalived) did not take over. Only after shutting the switchport to which A is attached did B take over. Logging in to the machine via serial, I noticed that it did not receive any packets via the network interface (after unshutting the switchport), only traffic sent by the host A was visible in tcpdump, no traffic that was sent to it (there should have been at least ARP traffic). In order to verify this, I dumped traffic on another host in the broadcast domain and indeed, the traffic sent out by A is seen on the network, it just doesn't receive any that is sent to it. This explains the lack of failover of keepalived, because A still considers itself master and is able to announce that to the network, while it cannot see the packets from its partner B (that wants to take over because of its, meanwhile, higher priority). No neighbors see the machine in their ARP tables any more. I think the number of packets that are sent to the host are reflected in the interface variable rx_ftq_discards: It increases by about 10 per second while idle, and by about 80 per second when I send floodpings to the machine. Here you see a dump of the interface statistics spaced ten seconds apart, while floodpinging the host: A:~# ethtool -S eth0; sleep 10; echo ---; ethtool -S eth0 NIC statistics: rx_bytes: 35498373071360 rx_error_bytes: 0 tx_bytes: 35475382869262 tx_error_bytes: 0 rx_ucast_packets: 45479514105 rx_mcast_packets: 9800399 rx_bcast_packets: 4901866 tx_ucast_packets: 45364190447 tx_mcast_packets: 7285029 tx_bcast_packets: 3111 tx_mac_errors: 0 tx_carrier_errors: 0 rx_crc_errors: 0 rx_align_errors: 0 tx_single_collisions: 0 tx_multi_collisions: 0 tx_deferred: 0 tx_excess_collisions: 0 tx_late_collisions: 0 tx_total_collisions: 0 rx_fragments: 0 rx_jabbers: 0 rx_undersize_packets: 0 rx_oversize_packets: 0 rx_64_byte_packets: 3465587589 rx_65_to_127_byte_packets: 422897833 rx_128_to_255_byte_packets: 3996306350 rx_256_to_511_byte_packets: 1500221686 rx_512_to_1023_byte_packets: 1351649898 rx_1024_to_1522_byte_packets: 397814646 rx_1523_to_9022_byte_packets: 0 tx_64_byte_packets: 3451623430 tx_65_to_127_byte_packets: 366024709 tx_128_to_255_byte_packets: 3954496418 tx_256_to_511_byte_packets: 1499757422 tx_512_to_1023_byte_packets: 1351506958 tx_1024_to_1522_byte_packets: 388331444 tx_1523_to_9022_byte_packets: 0 rx_xon_frames: 0 rx_xoff_frames: 0 tx_xon_frames: 81 tx_xoff_frames: 81 rx_mac_ctrl_frames: 0 rx_filtered_packets: 26701433 rx_ftq_discards: 1796839 rx_discards: 369 rx_fw_discards: 0 --- NIC statistics: rx_bytes: 35498373162770 rx_error_bytes: 0 tx_bytes: 35475382869262 tx_error_bytes: 0 rx_ucast_packets: 45479514920 rx_mcast_packets: 9800483 rx_bcast_packets: 4901876 tx_ucast_packets: 45364190447 tx_mcast_packets: 7285029 tx_bcast_packets: 3111 tx_mac_errors: 0 tx_carrier_errors: 0 rx_crc_errors: 0 rx_align_errors: 0 tx_single_collisions: 0 tx_multi_collisions: 0 tx_deferred: 0 tx_excess_collisions: 0 tx_late_collisions: 0 tx_total_collisions: 0 rx_fragments: 0 rx_jabbers: 0 rx_undersize_packets: 0 rx_oversize_packets: 0 rx_64_byte_packets: 3465587625 rx_65_to_127_byte_packets: 422898706 rx_128_to_255_byte_packets: 3996306350 rx_256_to_511_byte_packets: 1500221686 rx_512_to_1023_byte_packets: 1351649898 rx_1024_to_1522_byte_packets: 397814646 rx_1523_to_9022_byte_packets: 0 tx_64_byte_packets: 3451623430 tx_65_to_127_byte_packets: 366024709 tx_128_to_255_byte_packets: 3954496418 tx_256_to_511_byte_packets: 1499757422 tx_512_to_1023_byte_packets: 1351506958 tx_1024_to_1522_byte_packets: 388331444 tx_1523_to_9022_byte_packets: 0 rx_xon_frames: 0 rx_xoff_frames: 0 tx_xon_frames: 81 tx_xoff_frames: 81 rx_mac_ctrl_frames: 0 rx_filtered_packets: 26701433 rx_ftq_discards: 1797748 rx_discards: 369 rx_fw_discards: 0 The number of interrupts for the NIC is no longer increasing on host A. It is increasing on the otherwise identical and now active host B. A:~# cat /proc/interrupts | fgrep eth0; sleep 10; echo ---; cat /proc/interrupts | fgrep eth0 74: 7353715 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-0 75: 150160682 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-1 76: 261739096 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-2 77: 3118389637 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-3 78: 3538415303 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-4 79: 3437432016 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-5 80: 4130864322 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-6 81: 3844677189 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-7 --- 74: 7353715 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-0 75: 150160682 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-1 76: 261739096 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-2 77: 3118389637 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-3 78: 3538415303 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-4 79: 3437432016 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-5 80: 4130864322 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-6 81: 3844677189 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-7 B:~# cat /proc/interrupts | fgrep eth0; sleep 10; echo ---; cat /proc/interrupts | fgrep eth0 74: 8496700 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-0 75: 2605649299 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-1 76: 2278350057 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-2 77: 2119009356 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-3 78: 2004958460 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-4 79: 2005171437 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-5 80: 2318332903 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-6 81: 2087470150 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-7 --- 74: 8496713 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-0 75: 2605688265 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-1 76: 2278397958 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-2 77: 2119043500 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-3 78: 2005000430 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-4 79: 2005205617 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-5 80: 2318373260 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-6 81: 2087518969 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-7 There are no (significant) interface errors on the switchport of machine A (Cisco 6500): Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 3354643 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 73000 bits/sec, 90 packets/sec 139005756894 packets input, 106028470724434 bytes, 0 no buffer Received 41673355 broadcasts (41644823 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 139565849434 packets output, 106109148647056 bytes, 0 underruns 0 output errors, 0 collisions, 3 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out For reference, switchport of machine B: Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 561319 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 168420000 bits/sec, 27846 packets/sec 5 minute output rate 168547000 bits/sec, 27951 packets/sec 12477681177 packets input, 9891434829664 bytes, 0 no buffer Received 4452361 broadcasts (4434737 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 12725512555 packets output, 9944380037353 bytes, 0 underruns 0 output errors, 0 collisions, 2 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out This error occured about five hours ago, the interface did not recover. We have five pairs of basically identical machines performing the same task (each pair for one site). The error has not occured with any other one, but this site is the busiest: eth0 Link encap:Ethernet HWaddr 3c:d9:2b:ef:f6:3c inet addr:172.16.100.23 Bcast:172.16.100.63 Mask:255.255.255.192 inet6 addr: fe80::3ed9:2bff:feef:f63c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:45494315484 errors:1896322 dropped:1896322 overruns:0 frame:1896322 TX packets:45371478602 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:35498383041926 (32.2 TiB) TX bytes:35475382870222 (32.2 TiB) Interrupt:30 Memory:f4000000-f4012800 The host performs NAT, input and output interface being eth0, therefore the RX and TX counters are similar. I would appreciate any suggestions for diagnosing this further. Kind regards Marc ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: bnx2 cards intermittantly going offline 2012-09-13 13:51 ` Marc A. Donges @ 2012-09-13 15:45 ` Sven Ulland 2012-09-13 20:30 ` Michael Chan 2012-09-16 3:47 ` Ben Hutchings 0 siblings, 2 replies; 8+ messages in thread From: Sven Ulland @ 2012-09-13 15:45 UTC (permalink / raw) To: netdev; +Cc: Marc A. Donges, Michael Chan On 09/13/2012 03:51 PM, Marc A. Donges wrote: > After 55 days of operation the machine (A) suddenly was no longer > reachable via network. Strangely, a second machine (B) that should > take over the IP addresses (keepalived) did not take over. Only > after shutting the switchport to which A is attached did B take > over. Hi. We've had the same symptom with our BCM5709S [14e4:163a] on Debian. Like you, we were on stable's 2.6.32-41squeeze2. Google led us to many similar issues [1,2,3]. They concluded with the fix being in mainline commit c441b8d2 [4]: "bnx2: Fix lost MSI-X problem on 5709 NICs". Broadcom: Can you publish a tool that decodes ethtool -d dumps to make debugging easier, or do you deem it no longer necessary with the the register dump commits in 555069da? Now, Debian's 2.6.32-41squeeze2 is based on longterm release 2.6.32.54 [5]. That version includes commit 0b7817ed [6], which is a backport of the already mentioned mainline commit c441b8d2. So we tried digging further and applying some seemingly relevant commits [7,8] to our 2.6.32, but without any change in behaviour. Our temporary fix was to run 'ethtool -t ethX' to reset the device every time it locked up. This dragged on with various builds, until we ended up on mainline 2.6.38 where we no longer saw any symptoms. I don't know in which kernel version it was fixed, but we ended up on that one, sort of by chance. Unfortunately, it had severe issues with kswapd memory compaction causing CPU soft lockups [9], so we went straight to squeeze-backports' 3.2.23-1~bpo60+2. We've been happy since then. > We have five pairs of basically identical machines performing the > same task (each pair for one site). The error has not occured with > any other one, but this site is the busiest: We also saw the issue only at a site with generally higher load compared to other sites. I'd love to know exactly which commit fixed the issue, but it's fairly tricky to reproduce the issue, and the bisect count is fairly high (it need not be a specific fix for bnx2). sven [1]: bnx2 driver crashes under random circumstances https://bugzilla.redhat.com/show_bug.cgi?id=520888 [2]: Access denied. Come on, Red Hat! https://bugzilla.redhat.com/show_bug.cgi?id=511368 [3]: NIC doesn't register packets [rhel-5.5.z] https://bugzilla.redhat.com/show_bug.cgi?id=587799 [4]: bnx2: Fix lost MSI-X problem on 5709 NICs. http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=object;h=c441b8d2cb2194b05550a558d6d95d8944e56a84 [5]: Debian Changelog linux-2.6 (2.6.32-45) http://packages.debian.org/changelogs/pool/main/l/linux-2.6/linux-2.6_2.6.32-45/changelog#version2.6.32-41 [6]: bnx2: Fix lost MSI-X problem on 5709 NICs. http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=0b7817edda5e44e5fa769645bd1220f5e7b0beb5 [7]: bnx2: reset_task is crashing the kernel. Fixing it. http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4529819c45161e4a119134f56ef504e69420bc98 [8]: bnx2: fixing a timout error due not refreshing TX timers correctly http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=e6bf95ffa8d6f8f4b7ee33ea01490d95b0bbeb6e [9]: [PATCH] remove compaction from kswapd http://thread.gmane.org/gmane.linux.kernel.mm/58962 https://lkml.org/lkml/2011/3/25/664 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: bnx2 cards intermittantly going offline 2012-09-13 15:45 ` Sven Ulland @ 2012-09-13 20:30 ` Michael Chan 2012-09-16 3:47 ` Ben Hutchings 1 sibling, 0 replies; 8+ messages in thread From: Michael Chan @ 2012-09-13 20:30 UTC (permalink / raw) To: Sven Ulland; +Cc: netdev, Marc A. Donges On Thu, 2012-09-13 at 17:45 +0200, Sven Ulland wrote: > On 09/13/2012 03:51 PM, Marc A. Donges wrote: > > After 55 days of operation the machine (A) suddenly was no longer > > reachable via network. Strangely, a second machine (B) that should > > take over the IP addresses (keepalived) did not take over. Only > > after shutting the switchport to which A is attached did B take > > over. The rx_ftq_discards problem is a firmware problem. FTQ discards mean that the firmware is no longer running and the packets are dropped at the FTQ. This is likely fixed in: commit 22fa159d37efbfe781bbb99279efe83f58b87d29 Author: Michael Chan <mchan@broadcom.com> Date: Mon Oct 11 16:12:00 2010 -0700 bnx2: Update firmware to 6.0.x. > > Hi. We've had the same symptom with our BCM5709S [14e4:163a] on > Debian. Like you, we were on stable's 2.6.32-41squeeze2. Google led us > to many similar issues [1,2,3]. They concluded with the fix being in > mainline commit c441b8d2 [4]: "bnx2: Fix lost MSI-X problem on 5709 > NICs". This is a different problem and will not result in FTQ discards. > > Broadcom: Can you publish a tool that decodes ethtool -d dumps to make > debugging easier, or do you deem it no longer necessary with the the > register dump commits in 555069da? The register dump during tx timeout is now quite comprehensive. > > Now, Debian's 2.6.32-41squeeze2 is based on longterm release 2.6.32.54 > [5]. That version includes commit 0b7817ed [6], which is a backport of > the already mentioned mainline commit c441b8d2. > > So we tried digging further and applying some seemingly relevant > commits [7,8] to our 2.6.32, but without any change in behaviour. Our > temporary fix was to run 'ethtool -t ethX' to reset the device every > time it locked up. > > This dragged on with various builds, until we ended up on mainline > 2.6.38 where we no longer saw any symptoms. I don't know in which > kernel version it was fixed, but we ended up on that one, sort of by > chance. Unfortunately, it had severe issues with kswapd memory > compaction causing CPU soft lockups [9], so we went straight to > squeeze-backports' 3.2.23-1~bpo60+2. We've been happy since then. > > > We have five pairs of basically identical machines performing the > > same task (each pair for one site). The error has not occured with > > any other one, but this site is the busiest: > > We also saw the issue only at a site with generally higher load > compared to other sites. > > I'd love to know exactly which commit fixed the issue, but it's fairly > tricky to reproduce the issue, and the bisect count is fairly high (it > need not be a specific fix for bnx2). If you see the same FTQ discards, please try that firmware commit mentioned above. Thanks. > > sven > > > [1]: bnx2 driver crashes under random circumstances > https://bugzilla.redhat.com/show_bug.cgi?id=520888 > > [2]: Access denied. Come on, Red Hat! > https://bugzilla.redhat.com/show_bug.cgi?id=511368 > > [3]: NIC doesn't register packets [rhel-5.5.z] > https://bugzilla.redhat.com/show_bug.cgi?id=587799 > > [4]: bnx2: Fix lost MSI-X problem on 5709 NICs. > http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=object;h=c441b8d2cb2194b05550a558d6d95d8944e56a84 > > [5]: Debian Changelog linux-2.6 (2.6.32-45) > http://packages.debian.org/changelogs/pool/main/l/linux-2.6/linux-2.6_2.6.32-45/changelog#version2.6.32-41 > > [6]: bnx2: Fix lost MSI-X problem on 5709 NICs. > http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=0b7817edda5e44e5fa769645bd1220f5e7b0beb5 > > [7]: bnx2: reset_task is crashing the kernel. Fixing it. > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4529819c45161e4a119134f56ef504e69420bc98 > > [8]: bnx2: fixing a timout error due not refreshing TX timers correctly > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=e6bf95ffa8d6f8f4b7ee33ea01490d95b0bbeb6e > > [9]: [PATCH] remove compaction from kswapd > http://thread.gmane.org/gmane.linux.kernel.mm/58962 > https://lkml.org/lkml/2011/3/25/664 > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: bnx2 cards intermittantly going offline 2012-09-13 15:45 ` Sven Ulland 2012-09-13 20:30 ` Michael Chan @ 2012-09-16 3:47 ` Ben Hutchings 1 sibling, 0 replies; 8+ messages in thread From: Ben Hutchings @ 2012-09-16 3:47 UTC (permalink / raw) To: Sven Ulland; +Cc: netdev, Marc A. Donges, Michael Chan [-- Attachment #1: Type: text/plain, Size: 2520 bytes --] On Thu, 2012-09-13 at 17:45 +0200, Sven Ulland wrote: > On 09/13/2012 03:51 PM, Marc A. Donges wrote: > > After 55 days of operation the machine (A) suddenly was no longer > > reachable via network. Strangely, a second machine (B) that should > > take over the IP addresses (keepalived) did not take over. Only > > after shutting the switchport to which A is attached did B take > > over. > > Hi. We've had the same symptom with our BCM5709S [14e4:163a] on > Debian. Like you, we were on stable's 2.6.32-41squeeze2. Google led us > to many similar issues [1,2,3]. They concluded with the fix being in > mainline commit c441b8d2 [4]: "bnx2: Fix lost MSI-X problem on 5709 > NICs". > > Broadcom: Can you publish a tool that decodes ethtool -d dumps to make > debugging easier, or do you deem it no longer necessary with the the > register dump commits in 555069da? This tool should be ethtool itself (it includes dump decoders for many drivers). > Now, Debian's 2.6.32-41squeeze2 is based on longterm release 2.6.32.54 > [5]. That version includes commit 0b7817ed [6], which is a backport of > the already mentioned mainline commit c441b8d2. > > So we tried digging further and applying some seemingly relevant > commits [7,8] to our 2.6.32, but without any change in behaviour. Our > temporary fix was to run 'ethtool -t ethX' to reset the device every > time it locked up. > > This dragged on with various builds, until we ended up on mainline > 2.6.38 where we no longer saw any symptoms. I don't know in which > kernel version it was fixed, but we ended up on that one, sort of by > chance. Unfortunately, it had severe issues with kswapd memory > compaction causing CPU soft lockups [9], so we went straight to > squeeze-backports' 3.2.23-1~bpo60+2. We've been happy since then. > > > We have five pairs of basically identical machines performing the > > same task (each pair for one site). The error has not occured with > > any other one, but this site is the busiest: > > We also saw the issue only at a site with generally higher load > compared to other sites. > > I'd love to know exactly which commit fixed the issue, but it's fairly > tricky to reproduce the issue, and the bisect count is fairly high (it > need not be a specific fix for bnx2). I don't see any changes to the driver itself that look relevant. Perhaps this was a firmware bug? Ben. -- Ben Hutchings Experience is what causes a person to make new mistakes instead of old ones. [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: bnx2 cards intermittantly going offline 2011-01-18 10:54 bnx2 cards intermittantly going offline Mills, Tony 2011-01-18 17:55 ` Michael Chan @ 2011-11-15 17:41 ` Ken 1 sibling, 0 replies; 8+ messages in thread From: Ken @ 2011-11-15 17:41 UTC (permalink / raw) To: netdev +1 with identical L2 components and symptoms. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-09-16 3:47 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-01-18 10:54 bnx2 cards intermittantly going offline Mills, Tony 2011-01-18 17:55 ` Michael Chan 2011-01-26 12:44 ` Mills, Tony 2012-09-13 13:51 ` Marc A. Donges 2012-09-13 15:45 ` Sven Ulland 2012-09-13 20:30 ` Michael Chan 2012-09-16 3:47 ` Ben Hutchings 2011-11-15 17:41 ` Ken
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.