TC BPF Program Crashing With Bnx2x Drivers

* TC BPF Program Crashing With Bnx2x Drivers
@ 2020-04-08 17:15 Christian Deacon
  2020-04-08 22:58 ` Daniel Borkmann
  0 siblings, 1 reply; 8+ messages in thread
From: Christian Deacon @ 2020-04-08 17:15 UTC (permalink / raw)
  To: bpf

Hey everyone,

I apologize if this is the wrong mailing list. I wasn't able to find any 
mailing list for TC BPF specifically. If it is the wrong mailing list, 
any guidance to the correct mailing list would be appreciated.

I made a TC BPF program which has source code available on GitHub here:

https://github.com/gamemann/IPIPDirect-TC

I installed this program on several machines of mine running Ubuntu 
18.04 and Debian 10. I've used kernels from 5.4 to 5.6.2. The network 
drivers used on these machines include Virtio, e1000e, and e1000. Things 
have been running stable for 3 - 4 days now without any issues.

Another individual is trying to get this to work on their machine with 
Debian 10 and they've tried a backport kernel of 5.4.0 along with a 
manually built 5.6.2 kernel. I've built the 5.6.2 kernel for them, 
tested it on my machine, and everything worked fine. The only difference 
is this machine is using bnx2x network drivers. Here is the output from 
`lshw -class network`:

```

*-network:0
    description: Ethernet interface
    product: NetXtreme II BCM57810 10 Gigabit Ethernet
    vendor: Broadcom Limited
    physical id: 0
    bus info: pci@0000:04:00.0
    logical name: eth0
    version: 11
    serial:  xx:xx:xx:xx:xx:xx
    size: 10Gbit/s
    width: 64 bits
    clock: 33MHz
    capabilities: pm vpd msix pciexpress bus_master cap_list rom 
ethernet physical fibre autonegotiation
    configuration: autonegotiation=on broadcast=yes driver=bnx2x 
driverversion=1.713.36-0 storm 7.13.11.0 duplex=full firmware=mbi 
7.14.79 bc 7.13.75 ip=xxx.xxx.xxx.xxx latency=0 link=yes multicast=yes 
port=fibre speed=10Gbit/s
    resources: irq:117 memory:f6000000-f67fffff memory:f5800000-f5ffffff 
memory:f57f0000-f57fffff memory:f1000000-f107ffff
*-network:1 DISABLED
    description: Ethernet interface
    product: NetXtreme II BCM57810 10 Gigabit Ethernet
    vendor: Broadcom Limited
    physical id: 0.1
    bus info: pci@0000:04:00.1
    logical name: eth1
    version: 11
    serial: xx:xx:xx:xx:xx:xx
    width: 64 bits
    clock: 33MHz
    capabilities: pm vpd msix pciexpress bus_master cap_list rom 
ethernet physical fibre autonegotiation
    configuration: autonegotiation=on broadcast=yes driver=bnx2x 
driverversion=1.713.36-0 storm 7.13.11.0 firmware=mbi 7.14.79 bc 7.13.75 
latency=0 link=no multicast=yes port=fibre
    resources: irq:128 memory:f4800000-f4ffffff memory:f4000000-f47fffff 
memory:f3ff0000-f3ffffff memory:f1080000-f10fffff
```

When the program is loaded on their machine, it is working properly for 
20 - 30 seconds. However, it eventually crashes the network driver and 
network access to the machine is completely lost. I found the following 
in the `kern.log` file (it is very long, I apologize):

```

Apr  8 17:53:48 serverhostname kernel: [ 1150.253584] bnx2x: 
[bnx2x_stats_update:1232(eth0)]storm stats were not updated for 3 times
Apr  8 17:53:48 serverhostname kernel: [ 1150.253698] bnx2x: 
[bnx2x_stats_update:1233(eth0)]driver assert
Apr  8 17:53:48 serverhostname kernel: [ 1150.253768] bnx2x: 
[bnx2x_panic_dump:917(eth0)]begin crash dump -----------------
Apr  8 17:53:48 serverhostname kernel: [ 1150.253854] bnx2x: 
[bnx2x_panic_dump:927(eth0)]def_idx(0x46f)  def_att_idx(0x8) 
attn_state(0x0)  spq_prod_idx(0x88) next_stats_cnt(0x457)
Apr  8 17:53:48 serverhostname kernel: [ 1150.253994] bnx2x: 
[bnx2x_panic_dump:932(eth0)]DSB: attn bits(0x0)  ack(0x10) id(0x0)  idx(0x8)
Apr  8 17:53:48 serverhostname kernel: [ 1150.254094] bnx2x: 
[bnx2x_panic_dump:933(eth0)]     def (0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x473 
0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0)  igu_sb_id(0x0) igu_seg_id(0x1) 
pf_id(0x0)  vnic_id(0x0)  vf_id(0xff)  vf_valid (0x0) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.254323] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp0: rx_bd_prod(0x102b) rx_bd_cons(0xe64)  
rx_comp_prod(0x1677)  rx_comp_cons(0x14ab) *rx_cons_sb(0x14ac)
Apr  8 17:53:48 serverhostname kernel: [ 1150.254476] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0x8d48)
Apr  8 17:53:48 serverhostname kernel: [ 1150.254580] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp0: tx_pkt_prod(0x8756) 
tx_pkt_cons(0x8735)  tx_bd_prod(0xc86f)  tx_bd_cons(0xc82b) 
*tx_cons_sb(0x8735)
Apr  8 17:53:48 serverhostname kernel: [ 1150.254735] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp0: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.254874] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp0: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255015] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0x8d48 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255019] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0x14ac 0x0 0x0 0x0 0x8735 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0)  
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255285] SM[0] __flags 
(0x0) igu_sb_id (0x2)  igu_seg_id(0x0) time_to_expire (0x43ddd960) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255288] SM[1] __flags 
(0x0) igu_sb_id (0x2)  igu_seg_id(0x0) time_to_expire (0x43aa48a0) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255290] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255291] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255293] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255295] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255296] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255298] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255300] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255302] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255308] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp1: rx_bd_prod(0x1267) rx_bd_cons(0xa0)  
rx_comp_prod(0x18ba)  rx_comp_cons(0x16ee) *rx_cons_sb(0x16ee)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255461] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0xf62)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255566] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp1: tx_pkt_prod(0x952) tx_pkt_cons(0x946)  
tx_bd_prod(0x4d0c)  tx_bd_cons(0x4cf2) *tx_cons_sb(0x946)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255726] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp1: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.255863] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp1: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256002] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0xf62 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256006] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0x16ee 0x0 0x0 0x0 0x946 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0) 
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256276] SM[0] __flags 
(0x0) igu_sb_id (0x3)  igu_seg_id(0x0) time_to_expire (0x43ddd31b) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256279] SM[1] __flags 
(0x0) igu_sb_id (0x3)  igu_seg_id(0x0) time_to_expire (0x43aa5b45) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256280] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256282] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256284] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256285] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256287] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256288] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256290] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256291] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256296] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp2: rx_bd_prod(0xc0b) rx_bd_cons(0xa44)  
rx_comp_prod(0x124b)  rx_comp_cons(0x107e) *rx_cons_sb(0x107e)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256446] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0x1b0b)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256554] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp2: tx_pkt_prod(0x19f8) 
tx_pkt_cons(0x19ef)  tx_bd_prod(0xc3cc)  tx_bd_cons(0xc3b9) 
*tx_cons_sb(0x19ef)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256709] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp2: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256842] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp2: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256979] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0x1b0b 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.256982] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0x107e 0x0 0x0 0x0 0x19ef 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0)  
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257263] SM[0] __flags 
(0x0) igu_sb_id (0x4)  igu_seg_id(0x0) time_to_expire (0x43dddc4c) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257265] SM[1] __flags 
(0x0) igu_sb_id (0x4)  igu_seg_id(0x0) time_to_expire (0x43aa538f) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257267] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257269] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257271] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257272] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257274] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257275] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257277] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257278] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.257283] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp3: rx_bd_prod(0x4c7c) rx_bd_cons(0xab5)  
rx_comp_prod(0x5074)  rx_comp_cons(0x4ea8) *rx_cons_sb(0x4ea8)
Apr  8 17:53:48 serverhostname kernel: [ 1150.266351] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0xd3ec)
Apr  8 17:53:48 serverhostname kernel: [ 1150.270245] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp3: tx_pkt_prod(0x93b8) 
tx_pkt_cons(0x93a3)  tx_bd_prod(0x268b)  tx_bd_cons(0x265b) 
*tx_cons_sb(0x93a3)
Apr  8 17:53:48 serverhostname kernel: [ 1150.277423] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp3: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.282796] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp3: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.291180] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0xd3ee 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.291184] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0x4eaf 0x0 0x0 0x0 0x93a3 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0)  
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300220] SM[0] __flags 
(0x0) igu_sb_id (0x5)  igu_seg_id(0x0) time_to_expire (0x43de684c) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300227] SM[1] __flags 
(0x0) igu_sb_id (0x5)  igu_seg_id(0x0) time_to_expire (0x43aa5299) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300229] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300231] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300232] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300233] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300235] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300236] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300238] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300239] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.300245] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp4: rx_bd_prod(0xb458) rx_bd_cons(0x291)  
rx_comp_prod(0xb98d)  rx_comp_cons(0xb7c1) *rx_cons_sb(0xb7c1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.308272] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0x6dfb)
Apr  8 17:53:48 serverhostname kernel: [ 1150.311107] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp4: tx_pkt_prod(0xc32a) 
tx_pkt_cons(0xc317)  tx_bd_prod(0x5727)  tx_bd_cons(0x56ff) 
*tx_cons_sb(0xc317)
Apr  8 17:53:48 serverhostname kernel: [ 1150.318786] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp4: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.327491] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp4: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.336062] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0x6e02 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.336066] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0xb7c9 0x0 0x0 0x0 0xc317 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0)  
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345048] SM[0] __flags 
(0x0) igu_sb_id (0x6)  igu_seg_id(0x0) time_to_expire (0x43df1a5a) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345052] SM[1] __flags 
(0x0) igu_sb_id (0x6)  igu_seg_id(0x0) time_to_expire (0x43aa4993) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345054] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345055] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345057] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345059] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345061] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345062] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345064] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345066] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.345071] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp5: rx_bd_prod(0x4bfb) rx_bd_cons(0xa36)  
rx_comp_prod(0x4ff3)  rx_comp_cons(0x4e27) *rx_cons_sb(0x4e27)
Apr  8 17:53:48 serverhostname kernel: [ 1150.352421] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0x2a26)
Apr  8 17:53:48 serverhostname kernel: [ 1150.355184] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp5: tx_pkt_prod(0xe5da) 
tx_pkt_cons(0xe5c8)  tx_bd_prod(0x1041)  tx_bd_cons(0x1019) 
*tx_cons_sb(0xe5c8)
Apr  8 17:53:48 serverhostname kernel: [ 1150.362160] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp5: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.371350] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp5: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.379219] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0x2a2b 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.379223] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0x4e2d 0x0 0x0 0x0 0xe5c8 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0)  
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387666] SM[0] __flags 
(0x0) igu_sb_id (0x7)  igu_seg_id(0x0) time_to_expire (0x43dfc253) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387668] SM[1] __flags 
(0x0) igu_sb_id (0x7)  igu_seg_id(0x0) time_to_expire (0x43aa4ff2) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387669] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387670] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387671] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387673] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387674] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387675] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387676] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387677] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.387681] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp6: rx_bd_prod(0x1e90) rx_bd_cons(0xcc9)  
rx_comp_prod(0x2e2d)  rx_comp_cons(0x2c61) *rx_cons_sb(0x2c61)
Apr  8 17:53:48 serverhostname kernel: [ 1150.396441] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0xb60)
Apr  8 17:53:48 serverhostname kernel: [ 1150.399248] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp6: tx_pkt_prod(0xf788) 
tx_pkt_cons(0xf770)  tx_bd_prod(0x2535)  tx_bd_cons(0x2504) 
*tx_cons_sb(0xf770)
Apr  8 17:53:48 serverhostname kernel: [ 1150.405697] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp6: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.414921] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp6: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.422686] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0xb73 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.422690] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0x2c81 0x0 0x0 0x0 0xf770 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0)  
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431523] SM[0] __flags 
(0x0) igu_sb_id (0x8)  igu_seg_id(0x0) time_to_expire (0x43e075b8) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431525] SM[1] __flags 
(0x0) igu_sb_id (0x8)  igu_seg_id(0x0) time_to_expire (0x43aa4229) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431527] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431528] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431529] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431530] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431531] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431532] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431533] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431534] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.431539] bnx2x: 
[bnx2x_panic_dump:984(eth0)]fp7: rx_bd_prod(0x6f3d) rx_bd_cons(0xd76)  
rx_comp_prod(0x7094)  rx_comp_cons(0x6ec8) *rx_cons_sb(0x6ec8)
Apr  8 17:53:48 serverhostname kernel: [ 1150.439505] bnx2x: 
[bnx2x_panic_dump:987(eth0)]     rx_sge_prod(0x0) last_max_sge(0x0)  
fp_hc_idx(0x685f)
Apr  8 17:53:48 serverhostname kernel: [ 1150.443056] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp7: tx_pkt_prod(0xc7) tx_pkt_cons(0xb7)  
tx_bd_prod(0x1412)  tx_bd_cons(0x13f0) *tx_cons_sb(0xb7)
Apr  8 17:53:48 serverhostname kernel: [ 1150.448817] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp7: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.455960] bnx2x: 
[bnx2x_panic_dump:1004(eth0)]fp7: tx_pkt_prod(0x0) tx_pkt_cons(0x0)  
tx_bd_prod(0x0)  tx_bd_cons(0x0) *tx_cons_sb(0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.458546] bnx2x: 
[bnx2x_panic_dump:1015(eth0)]     run indexes (0x6860 0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.458548] bnx2x: 
[bnx2x_panic_dump:1021(eth0)]     indexes (0x0 0x6eca 0x0 0x0 0x0 0xb7 
0x0 0x0)pf_id(0x0)  vf_id(0xff)  vf_valid(0x0) vnic_id(0x0) 
same_igu_sb_1b(0x1) state(0x1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462410] SM[0] __flags 
(0x0) igu_sb_id (0x9)  igu_seg_id(0x0) time_to_expire (0x43e0f737) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462413] SM[1] __flags 
(0x0) igu_sb_id (0x9)  igu_seg_id(0x0) time_to_expire (0x43aa51f7) 
timer_value(0xff)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462413] INDEX[0] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462414] INDEX[1] flags 
(0x2) timeout (0x6)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462414] INDEX[2] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462415] INDEX[3] flags 
(0x0) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462415] INDEX[4] flags 
(0x1) timeout (0x0)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462416] INDEX[5] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462416] INDEX[6] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462417] INDEX[7] flags 
(0x3) timeout (0xc)
Apr  8 17:53:48 serverhostname kernel: [ 1150.462422] bnx2x 0000:04:00.0 
eth0: bc 7.13.75
Apr  8 17:53:48 serverhostname kernel: [ 1150.464103] begin fw dump 
(mark 0x3c6748)
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] 0x0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] En Vaux
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] M lic key
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] [0.0]lldp strt dis
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] [1.0]lldp strt dis
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] pf_num 0: d89d 
67723b38
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] WC loaded OK
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] 
link_init[0.0]:NO-LFA(1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] init_phy[0.0]: done
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] 
link_init[1.0]:NO-LFA(1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] init_phy[1.0]: done
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] evnt[0.0] 0x0->0x1000
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] evnt[0.0] 0x1000->0x0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] link_update[0.0]: 
(rc 0) link 1 speed 0x2710
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] clp exit
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] p0: DCC OV 0x7d1
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] p0: DCC OV 0x7d1
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] [0:0] Resetting link
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] 
link_init[0.0]:NO-LFA(1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] init_phy[0.0]: done
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] [1:0] Resetting link
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] clp exit
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] p0: DCC OV 0x7d1
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] p0: DCC OV 0x7d1
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] 
link_init[1.0]:NO-LFA(1)
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] init_phy[1.0]: done
Apr  8 17:53:48 serverhostname kernel: [ 1150.466164] [0:0] Resetting link
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] 7d2 data=63b ibft=0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] f0: 
UNLOAD_REQ_WOL_DIS 0x1
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] f0:drv 
msg=20010000,rsp=20100001
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] f0: UNLOAD_DONE 0x2 2
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] Vaux 1
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] f0:drv 
msg=21000000,rsp=21100002
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] f1: 
UNLOAD_REQ_WOL_DIS 0x1
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] f1:drv 
msg=20010000,rsp=20100001
Apr  8 17:53:48 serverhostname kernel: [ 1150.466328] f1: UNLOAD_DONE 0x2 2
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] Vaux 0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] f1:drv 
msg=21000000,rsp=21100002
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] f0: LOAD_REQ 0x3
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] 0.0:PMF->f0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] f0:drv 
msg=10000000,rsp=10130003
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] f0: LOAD_DONE 0x4
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] Ap▒Af0:drv 
msg=11000000,rsp=11100004
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] evnt[0.0] 0x0->0x1000
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] evnt[0.0] 0x1000->0x0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] f0: LNK_CHANGED 0x5
Apr  8 17:53:48 serverhostname kernel: [ 1150.466365] f0:drv 
msg=1000000,rsp=1100005
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] ver_query_tlv p0: 
cap_grp1 0x7
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] f0: dcc status 
0x301 config 0x640a0102
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] f0: DCC_OK
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] f0:drv 
msg=30000000,rsp=30100006
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] ocbb_etoc 1 ffffffff 0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] Num ports 2, 
Num_ether_func 1, fcoe 0 iscsi 2
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] Func instance err 
1, fcoe 0 iscsi 2 ports 2 pf_num 8
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] ibft_host_addr is 
not initialized
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] ibft_host_addr is 
not initialized
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] OCBB size: 
ETOC=7d2 data=5ac ibft=0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] ocbb_etoc ffffffff 
ffffffff 0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466408] Num ports 2, 
Num_ether_func 1, fcoe 0 iscsi 2
Apr  8 17:53:48 serverhostname kernel: [ 1150.466476] Func instance err 
1, fcoe 0 iscsi 2 ports 2 pf_num 8
Apr  8 17:53:48 serverhostname kernel: [ 1150.466476] ibft_host_addr is 
not initialized
Apr  8 17:53:48 serverhostname kernel: [ 1150.466476] ibft_host_addr is 
not initialized
Apr  8 17:53:48 serverhostname kernel: [ 1150.466476] OCBB size: 
ETOC=7d2 data=5ac ibft=0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466503] 0x0
Apr  8 17:53:48 serverhostname kernel: [ 1150.466503] En Vaux
Apr  8 17:53:48 serverhostname kernel: [ 1150.466503] M lic key
Apr  8 17:53:48 serverhostname kernel: [ 1150.466503] [0.0]lldp
Apr  8 17:53:48 serverhostname kernel: [ 1150.466504] end of fw dump
Apr  8 17:53:48 serverhostname kernel: [ 1150.468695] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x0 = 0x00000000 
0x00000000 0x00000001 0x00020027
Apr  8 17:53:48 serverhostname kernel: [ 1150.470147] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x1 = 0x00021bbb 
0x00000008 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.471374] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x2 = 0x00026e41 
0x00000000 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.472559] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x3 = 0x0002762e 
0x00000000 0x04000008 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.473738] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x4 = 0x00027db6 
0x00000000 0x04000010 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.474874] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x5 = 0x000285b0 
0x00000000 0x01000001 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.476006] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x6 = 0x00028d52 
0x00000000 0x04000009 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.477112] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x7 = 0x00029568 
0x00000000 0x04000011 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.478350] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x8 = 0x00029ce7 
0x00000000 0x01000002 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.480084] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x9 = 0x0002a4e9 
0x00000000 0x0400000a 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.481806] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0xa = 0x0002ac77 
0x00000000 0x04000012 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.482830] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0xb = 0x0002b475 
0x00000000 0x01000003 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.483823] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0xc = 0x0002bc12 
0x00000000 0x0400000b 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.484791] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0xd = 0x0002c405 
0x00000000 0x04000013 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.485747] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0xe = 0x0002cbba 
0x00000000 0x01000004 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.486667] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0xf = 0x0002d3a7 
0x00000000 0x0400000c 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.487559] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x10 = 0x0002db4c 
0x00000000 0x04000014 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.488432] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x11 = 0x0002e342 
0x00000000 0x01000005 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.489307] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x12 = 0x0002eae5 
0x00000000 0x0400000d 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.490173] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x13 = 0x0002f2d4 
0x00000000 0x04000015 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.490997] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x14 = 0x0002fa88 
0x00000000 0x01000006 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.491794] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x15 = 0x0003027c 
0x00000000 0x0400000e 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.492578] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x16 = 0x00030a1e 
0x00000000 0x04000016 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.493542] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x17 = 0x00031212 
0x00000000 0x01000007 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.494769] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x18 = 0x000319bd 
0x00000000 0x0400000f 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.495967] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x19 = 0x000321ad 
0x00000000 0x04000017 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.496798] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x1a = 0x00032959 
0x00000000 0x0c000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.497456] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x1b = 0x000329d1 
0x00000000 0x09000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.498106] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x1c = 0x00042320 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.498713] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x1d = 0x00042326 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.499319] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x1e = 0x000423cb 
0x00000008 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.499896] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x1f = 0x000437b2 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.500447] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x20 = 0x000437b8 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.500975] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x21 = 0x0004f98e 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.501477] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x22 = 0x00054523 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.501971] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x23 = 0x0005901a 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.502434] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x24 = 0x0005da81 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.502869] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x25 = 0x0006255f 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.503277] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x26 = 0x0006715c 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.503672] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x27 = 0x0006bb1b 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.504028] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x28 = 0x000704c9 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.504358] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x29 = 0x00074f54 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.504684] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x2a = 0x00079973 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.504985] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x2b = 0x0007e3bf 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.505257] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x2c = 0x00082d78 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.505519] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x2d = 0x00087928 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.505789] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x2e = 0x000b6597 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.506041] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x2f = 0x000b659d 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.506267] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x30 = 0x000ba2c7 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.506489] bnx2x: 
[bnx2x_mc_assert:736(eth0)]XSTORM_ASSERT_INDEX 0x31 = 0x0029ff1f 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.506712] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x0 = 0x00000000 
0x00000000 0x00000001 0x00020027
Apr  8 17:53:48 serverhostname kernel: [ 1150.506944] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x1 = 0x00021bbf 
0x00000008 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.507179] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x2 = 0x00026e45 
0x00000000 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.507411] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x3 = 0x00027631 
0x00000000 0x04000008 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.507645] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x4 = 0x00027db9 
0x00000000 0x04000010 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.507878] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x5 = 0x000285b3 
0x00000000 0x01000001 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.508186] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x6 = 0x00028d55 
0x00000000 0x04000009 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.508541] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x7 = 0x0002956b 
0x00000000 0x04000011 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.508926] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x8 = 0x00029ceb 
0x00000000 0x01000002 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.509304] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x9 = 0x0002a4ec 
0x00000000 0x0400000a 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.509699] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0xa = 0x0002ac7a 
0x00000000 0x04000012 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.510073] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0xb = 0x0002b479 
0x00000000 0x01000003 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.510473] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0xc = 0x0002bc15 
0x00000000 0x0400000b 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.510854] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0xd = 0x0002c408 
0x00000000 0x04000013 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.511224] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0xe = 0x0002cbbd 
0x00000000 0x01000004 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.511578] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0xf = 0x0002d3aa 
0x00000000 0x0400000c 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.511903] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x10 = 0x0002db50 
0x00000000 0x04000014 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.512295] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x11 = 0x0002e345 
0x00000000 0x01000005 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.512691] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x12 = 0x0002eae8 
0x00000000 0x0400000d 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.513083] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x13 = 0x0002f2d7 
0x00000000 0x04000015 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.513490] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x14 = 0x0002fa8c 
0x00000000 0x01000006 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.513908] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x15 = 0x0003027f 
0x00000000 0x0400000e 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.514296] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x16 = 0x00030a21 
0x00000000 0x04000016 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.514551] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x17 = 0x00031216 
0x00000000 0x01000007 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.514784] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x18 = 0x000319c0 
0x00000000 0x0400000f 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.515016] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x19 = 0x000321b0 
0x00000000 0x04000017 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.515248] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x1a = 0x0003295a 
0x00000000 0x0c000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.515481] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x1b = 0x000329d4 
0x00000000 0x09000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.515713] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x1c = 0x00042323 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.515944] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x1d = 0x00042329 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.516173] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x1e = 0x000423cb 
0x00000008 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.516402] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x1f = 0x000437b5 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.516631] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x20 = 0x000437bb 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.516860] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x21 = 0x0004f991 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.517088] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x22 = 0x00054526 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.517318] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x23 = 0x0005901d 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.517546] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x24 = 0x0005da84 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.517789] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x25 = 0x00062562 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.518018] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x26 = 0x0006715f 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.518261] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x27 = 0x0006bb1e 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.518490] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x28 = 0x000704cc 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.518720] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x29 = 0x00074f57 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.518949] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x2a = 0x00079976 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.519178] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x2b = 0x0007e3c2 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.519407] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x2c = 0x00082d7b 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.519637] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x2d = 0x0008792b 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.519864] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x2e = 0x000b659b 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.520091] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x2f = 0x000b65a1 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.520309] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x30 = 0x000ba2ca 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.520524] bnx2x: 
[bnx2x_mc_assert:736(eth0)]TSTORM_ASSERT_INDEX 0x31 = 0x0029ff22 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.520752] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x0 = 0x00000000 
0x00000000 0x00000001 0x00020027
Apr  8 17:53:48 serverhostname kernel: [ 1150.520976] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x1 = 0x00021bc2 
0x00000008 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.521202] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x2 = 0x00026e47 
0x00000000 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.521427] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x3 = 0x00027632 
0x00000000 0x04000008 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.521665] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x4 = 0x00027dba 
0x00000000 0x04000010 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.521893] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x5 = 0x000285b5 
0x00000000 0x01000001 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.522118] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x6 = 0x00028d55 
0x00000000 0x04000009 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.522343] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x7 = 0x0002956c 
0x00000000 0x04000011 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.522570] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x8 = 0x00029cec 
0x00000000 0x01000002 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.522794] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x9 = 0x0002a4ec 
0x00000000 0x0400000a 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.523048] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0xa = 0x0002ac7a 
0x00000000 0x04000012 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.523399] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0xb = 0x0002b47b 
0x00000000 0x01000003 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.523752] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0xc = 0x0002bc15 
0x00000000 0x0400000b 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.524123] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0xd = 0x0002c409 
0x00000000 0x04000013 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.524482] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0xe = 0x0002cbbf 
0x00000000 0x01000004 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.524835] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0xf = 0x0002d3aa 
0x00000000 0x0400000c 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.525198] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x10 = 0x0002db50 
0x00000000 0x04000014 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.525578] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x11 = 0x0002e347 
0x00000000 0x01000005 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.525950] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x12 = 0x0002eae8 
0x00000000 0x0400000d 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.526320] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x13 = 0x0002f2d7 
0x00000000 0x04000015 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.526697] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x14 = 0x0002fa8e 
0x00000000 0x01000006 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.527056] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x15 = 0x00030280 
0x00000000 0x0400000e 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.527357] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x16 = 0x00030a21 
0x00000000 0x04000016 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.527584] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x17 = 0x00031218 
0x00000000 0x01000007 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.527810] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x18 = 0x000319c1 
0x00000000 0x0400000f 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.528035] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x19 = 0x000321b0 
0x00000000 0x04000017 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.528266] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x1a = 0x0003295c 
0x00000000 0x0c000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.528502] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x1b = 0x000329d7 
0x00000000 0x09000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.528729] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x1c = 0x00042326 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.528954] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x1d = 0x0004232c 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.529179] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x1e = 0x000423cd 
0x00000008 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.529404] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x1f = 0x000437b8 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.529648] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x20 = 0x000437be 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.529877] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x21 = 0x0004f994 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.530105] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x22 = 0x00054529 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.530331] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x23 = 0x00059020 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.530558] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x24 = 0x0005da87 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.530786] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x25 = 0x00062565 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.531012] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x26 = 0x00067161 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.531241] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x27 = 0x0006bb21 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.531467] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x28 = 0x000704cf 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.531694] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x29 = 0x00074f5a 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.531921] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x2a = 0x00079978 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.532149] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x2b = 0x0007e3c5 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.532378] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x2c = 0x00082d7e 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.532606] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x2d = 0x0008792e 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.532844] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x2e = 0x000b659e 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.533070] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x2f = 0x000b65a4 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.533287] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x30 = 0x000ba2cd 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.533502] bnx2x: 
[bnx2x_mc_assert:736(eth0)]CSTORM_ASSERT_INDEX 0x31 = 0x0029ff26 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.533726] bnx2x: 
[bnx2x_mc_assert:720(eth0)]USTORM_ASSERT_LIST_INDEX 0x62
Apr  8 17:53:48 serverhostname kernel: [ 1150.533951] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x0 = 0x00021bc1 
0x00000008 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.534193] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x1 = 0x00026e47 
0x00000000 0x01000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.534434] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x2 = 0x00027632 
0x00000000 0x04000008 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.534675] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x3 = 0x00027dba 
0x00000000 0x04000010 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.534917] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x4 = 0x000285b5 
0x00000000 0x01000001 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.535159] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x5 = 0x00028d56 
0x00000000 0x04000009 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.535402] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x6 = 0x0002956c 
0x00000000 0x04000011 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.535645] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x7 = 0x00029ced 
0x00000000 0x01000002 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.535899] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x8 = 0x0002a4ec 
0x00000000 0x0400000a 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.536142] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x9 = 0x0002ac7b 
0x00000000 0x04000012 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.536395] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0xa = 0x0002b47b 
0x00000000 0x01000003 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.536638] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0xb = 0x0002bc15 
0x00000000 0x0400000b 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.536893] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0xc = 0x0002c409 
0x00000000 0x04000013 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.537128] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0xd = 0x0002cbbf 
0x00000000 0x01000004 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.537362] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0xe = 0x0002d3aa 
0x00000000 0x0400000c 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.537612] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0xf = 0x0002db50 
0x00000000 0x04000014 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.537848] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x10 = 0x0002e347 
0x00000000 0x01000005 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.538159] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x11 = 0x0002eae8 
0x00000000 0x0400000d 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.538527] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x12 = 0x0002f2d7 
0x00000000 0x04000015 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.538907] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x13 = 0x0002fa8e 
0x00000000 0x01000006 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.539312] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x14 = 0x00030280 
0x00000000 0x0400000e 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.539702] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x15 = 0x00030a21 
0x00000000 0x04000016 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.540102] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x16 = 0x00031218 
0x00000000 0x01000007 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.540498] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x17 = 0x000319c1 
0x00000000 0x0400000f 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.540888] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x18 = 0x000321b0 
0x00000000 0x04000017 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.541286] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x19 = 0x0003295c 
0x00000000 0x0c000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.541688] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x1a = 0x000329d7 
0x00000000 0x09000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.542022] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x1b = 0x00042326 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.542266] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x1c = 0x0004232b 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.542509] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x1d = 0x000423cd 
0x00000008 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.542751] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x1e = 0x000437b8 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.543008] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x1f = 0x000437be 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.543263] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x20 = 0x0004f994 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.543506] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x21 = 0x00054529 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.543751] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x22 = 0x00059020 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.543996] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x23 = 0x0005da86 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.544240] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x24 = 0x00062564 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.544483] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x25 = 0x00067161 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.544727] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x26 = 0x0006bb21 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.544971] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x27 = 0x000704cf 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.545216] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x28 = 0x00074f5a 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.545459] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x29 = 0x00079978 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.545718] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x2a = 0x0007e3c5 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.545964] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x2b = 0x00082d7d 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.546206] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x2c = 0x0008792e 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.546451] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x2d = 0x000b659e 
0x00000000 0x0b000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.546693] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x2e = 0x000b65a4 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.546920] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x2f = 0x000ba2cd 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.547135] bnx2x: 
[bnx2x_mc_assert:736(eth0)]USTORM_ASSERT_INDEX 0x30 = 0x0029ff26 
0x00000000 0x0a000000 0x000200ac
Apr  8 17:53:48 serverhostname kernel: [ 1150.547362] bnx2x: 
[bnx2x_mc_assert:750(eth0)]Chip Revision: everest3, FW Version: 7_13_11
Apr  8 17:53:48 serverhostname kernel: [ 1150.547585] bnx2x: 
[bnx2x_panic_dump:1180(eth0)]end crash dump -----------------
Apr  8 17:53:52 serverhostname kernel: [ 1154.185533] ------------[ cut 
here ]------------
Apr  8 17:53:52 serverhostname kernel: [ 1154.185539] NETDEV WATCHDOG: 
eth0 (bnx2x): transmit queue 0 timed out
Apr  8 17:53:52 serverhostname kernel: [ 1154.185571] WARNING: CPU: 25 
PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x253/0x260
Apr  8 17:53:52 serverhostname kernel: [ 1154.185573] Modules linked in: 
cls_bpf algif_hash af_alg sch_ingress ts_bm xt_tcpudp ts_kmp xt_string 
nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 
nf_defrag_ipv4 ipip tunnel4 ip_tunnel nft_counter nf_tables nfnetlink 
intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal 
intel_powerclamp ipmi_ssif coretemp kvm_intel kvm irqbypass 
cpufreq_powersave cpufreq_conservative crct10dif_pclmul 
cpufreq_userspace crc32_pclmul ghash_clmulni_intel aesni_intel mgag200 
crypto_simd drm_vram_helper cryptd ttm glue_helper drm_kms_helper 
intel_cstate intel_uncore drm intel_rapl_perf joydev evdev ipmi_si sg 
ioatdma serio_raw i2c_algo_bit ipmi_devintf pcspkr iTCO_wdt hpwdt 
iTCO_vendor_support hpilo watchdog dca ipmi_msghandler acpi_power_meter 
button binfmt_misc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 
hid_generic usbhid hid sd_mod bnx2x uhci_hcd hpsa ehci_pci ehci_hcd 
scsi_transport_sas psmouse usbcore scsi_mod ptp pps_core mdio libcrc32c 
lpc_ich crc32c_generic mfd_core
Apr  8 17:53:52 serverhostname kernel: [ 1154.185635]  usb_common 
crc32c_intel video
Apr  8 17:53:52 serverhostname kernel: [ 1154.185642] CPU: 25 PID: 0 
Comm: swapper/25 Not tainted 5.4.0-0.bpo.4-amd64 #1 Debian 5.4.19-1~bpo10+1
Apr  8 17:53:52 serverhostname kernel: [ 1154.185643] Hardware name: HP 
ProLiant BL460c Gen8, BIOS I31 05/21/2018
Apr  8 17:53:52 serverhostname kernel: [ 1154.185648] RIP: 
0010:dev_watchdog+0x253/0x260
Apr  8 17:53:52 serverhostname kernel: [ 1154.185651] Code: 48 85 c0 75 
e4 eb 9d 4c 89 ef c6 05 ec 3e a8 00 01 e8 e1 e4 fa ff 89 d9 4c 89 ee 48 
c7 c7 c8 0e 13 8f 48 89 c2 e8 d6 86 a0 ff <0f> 0b e9 7c ff ff ff 66 0f 
1f 44 00 00 0f 1f 44 00 00 41 57 41 56
Apr  8 17:53:52 serverhostname kernel: [ 1154.185653] RSP: 
0018:ffff9e2f4380ce88 EFLAGS: 00010282
Apr  8 17:53:52 serverhostname kernel: [ 1154.185656] RAX: 
0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
Apr  8 17:53:52 serverhostname kernel: [ 1154.185657] RDX: 
0000000000000007 RSI: 0000000000000086 RDI: ffff91a92f7d7680
Apr  8 17:53:52 serverhostname kernel: [ 1154.185659] RBP: 
ffff91a92bbf845c R08: 000000000000064e R09: 0000000000000004
Apr  8 17:53:52 serverhostname kernel: [ 1154.185660] R10: 
0000000000000000 R11: 0000000000000001 R12: ffff91a92bbcf100
Apr  8 17:53:52 serverhostname kernel: [ 1154.185662] R13: 
ffff91a92bbf8000 R14: ffff91a92bbf8480 R15: 000000000000005b
Apr  8 17:53:52 serverhostname kernel: [ 1154.185665] FS: 
0000000000000000(0000) GS:ffff91a92f7c0000(0000) knlGS:0000000000000000
Apr  8 17:53:52 serverhostname kernel: [ 1154.185666] CS:  0010 DS: 0000 
ES: 0000 CR0: 0000000080050033
Apr  8 17:53:52 serverhostname kernel: [ 1154.185668] CR2: 
00000000ced16240 CR3: 0000000417a0a003 CR4: 00000000001606e0
Apr  8 17:53:52 serverhostname kernel: [ 1154.185669] Call Trace:
Apr  8 17:53:52 serverhostname kernel: [ 1154.185674]  <IRQ>
Apr  8 17:53:52 serverhostname kernel: [ 1154.185681]  ? 
pfifo_fast_enqueue+0x140/0x140
Apr  8 17:53:52 serverhostname kernel: [ 1154.185688] 
call_timer_fn+0x2d/0x130
Apr  8 17:53:52 serverhostname kernel: [ 1154.185691] 
run_timer_softirq+0x1a6/0x430
Apr  8 17:53:52 serverhostname kernel: [ 1154.185695]  ? 
__hrtimer_run_queues+0x130/0x280
Apr  8 17:53:52 serverhostname kernel: [ 1154.185700]  ? 
recalibrate_cpu_khz+0x10/0x10
Apr  8 17:53:52 serverhostname kernel: [ 1154.185703]  ? ktime_get+0x36/0xa0
Apr  8 17:53:52 serverhostname kernel: [ 1154.185709] 
__do_softirq+0xdf/0x2e5
Apr  8 17:53:52 serverhostname kernel: [ 1154.185716] irq_exit+0xa3/0xb0
Apr  8 17:53:52 serverhostname kernel: [ 1154.185721] 
smp_apic_timer_interrupt+0x74/0x130
Apr  8 17:53:52 serverhostname kernel: [ 1154.185725] 
apic_timer_interrupt+0xf/0x20
Apr  8 17:53:52 serverhostname kernel: [ 1154.185727] </IRQ>
Apr  8 17:53:52 serverhostname kernel: [ 1154.185734] RIP: 
0010:cpuidle_enter_state+0xbc/0x450
Apr  8 17:53:52 serverhostname kernel: [ 1154.185736] Code: e8 b9 86 ad 
ff 80 7c 24 13 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 67 03 00 00 
31 ff e8 2b b1 b3 ff fb 66 0f 1f 44 00 00 <45> 85 e4 0f 88 92 02 00 00 
49 63 cc 4c 8b 3c 24 4c 2b 7c 24 08 48
Apr  8 17:53:52 serverhostname kernel: [ 1154.185738] RSP: 
0018:ffff9e2f4327fe78 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Apr  8 17:53:52 serverhostname kernel: [ 1154.185740] RAX: 
ffff91a92f7ea6c0 RBX: ffffffff8f2b7a40 RCX: 000000000000001f
Apr  8 17:53:52 serverhostname kernel: [ 1154.185742] RDX: 
0000010cbad136e6 RSI: 000000003333348b RDI: 0000000000000000
Apr  8 17:53:52 serverhostname kernel: [ 1154.185743] RBP: 
ffffbe2b3eff8200 R08: 0000000000000002 R09: 0000000000029f40
Apr  8 17:53:52 serverhostname kernel: [ 1154.185745] R10: 
0059ffb91faccf15 R11: ffff91a92f7e9580 R12: 0000000000000004
Apr  8 17:53:52 serverhostname kernel: [ 1154.185746] R13: 
ffffffff8f2b7bd8 R14: 0000000000000004 R15: 0000000000000000
Apr  8 17:53:52 serverhostname kernel: [ 1154.185752]  ? 
cpuidle_enter_state+0x97/0x450
Apr  8 17:53:52 serverhostname kernel: [ 1154.185756] 
cpuidle_enter+0x29/0x40
Apr  8 17:53:52 serverhostname kernel: [ 1154.185761] do_idle+0x228/0x270
Apr  8 17:53:52 serverhostname kernel: [ 1154.185765] 
cpu_startup_entry+0x19/0x20
Apr  8 17:53:52 serverhostname kernel: [ 1154.185769] 
start_secondary+0x15f/0x1b0
Apr  8 17:53:52 serverhostname kernel: [ 1154.185774] 
secondary_startup_64+0xa4/0xb0
Apr  8 17:53:52 serverhostname kernel: [ 1154.185778] ---[ end trace 
43b1706a27024516 ]---
Apr  8 17:53:52 serverhostname kernel: [ 1154.185828] bnx2x: 
[bnx2x_sp_rtnl_task:10330(eth0)]Indicating link is down due to Tx-timeout
Apr  8 17:53:54 serverhostname kernel: [ 1156.071098] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[0]: 
txdata->tx_pkt_prod(34646) != txdata->tx_pkt_cons(34613)
Apr  8 17:53:56 serverhostname kernel: [ 1157.961524] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[1]: 
txdata->tx_pkt_prod(2386) != txdata->tx_pkt_cons(2374)
Apr  8 17:53:58 serverhostname kernel: [ 1159.858694] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[2]: 
txdata->tx_pkt_prod(6648) != txdata->tx_pkt_cons(6639)
Apr  8 17:54:00 serverhostname kernel: [ 1161.772522] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[3]: 
txdata->tx_pkt_prod(37816) != txdata->tx_pkt_cons(37795)
Apr  8 17:54:02 serverhostname kernel: [ 1163.683181] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[4]: 
txdata->tx_pkt_prod(49962) != txdata->tx_pkt_cons(49943)
Apr  8 17:54:02 serverhostname kernel: [ 1164.459899] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.457470] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.461819] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.469797] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.469832] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.470011] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.482640] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.482683] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.484735] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:03 serverhostname kernel: [ 1165.487322] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:04 serverhostname kernel: [ 1165.573405] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[5]: 
txdata->tx_pkt_prod(58842) != txdata->tx_pkt_cons(58824)
Apr  8 17:54:05 serverhostname kernel: [ 1167.483398] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[6]: 
txdata->tx_pkt_prod(63368) != txdata->tx_pkt_cons(63344)
Apr  8 17:54:07 serverhostname kernel: [ 1169.411378] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[7]: 
txdata->tx_pkt_prod(199) != txdata->tx_pkt_cons(183)
Apr  8 17:54:09 serverhostname kernel: [ 1171.316654] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[0]: 
txdata->tx_pkt_prod(34646) != txdata->tx_pkt_cons(34613)
Apr  8 17:54:09 serverhostname kernel: [ 1171.476484] net_ratelimit: 11 
callbacks suppressed
Apr  8 17:54:09 serverhostname kernel: [ 1171.476487] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:09 serverhostname kernel: [ 1171.481018] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:11 serverhostname kernel: [ 1173.229309] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[1]: 
txdata->tx_pkt_prod(2386) != txdata->tx_pkt_cons(2374)
Apr  8 17:54:12 serverhostname kernel: [ 1174.479480] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:12 serverhostname kernel: [ 1174.483879] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:13 serverhostname kernel: [ 1175.136214] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[2]: 
txdata->tx_pkt_prod(6648) != txdata->tx_pkt_cons(6639)
Apr  8 17:54:15 serverhostname kernel: [ 1177.036533] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[3]: 
txdata->tx_pkt_prod(37816) != txdata->tx_pkt_cons(37795)
Apr  8 17:54:15 serverhostname kernel: [ 1177.483111] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:15 serverhostname kernel: [ 1177.487417] TCP: 
request_sock_TCP: Possible SYN flooding on port 27015. Sending cookies.  
Check SNMP counters.
Apr  8 17:54:17 serverhostname kernel: [ 1178.971188] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[4]: 
txdata->tx_pkt_prod(49962) != txdata->tx_pkt_cons(49943)
Apr  8 17:54:19 serverhostname kernel: [ 1180.859460] bnx2x: 
[bnx2x_clean_tx_queue:1204(eth0)]timeout waiting for queue[5]: 
txdata->tx_pkt_prod(58842) != txdata->tx_pkt_cons(58824)

```

A reboot is required to get connectivity back to the machine. I've tried 
looking around for any bug reports on this and found a couple, but I 
don't believe they were related to crashing with TC BPF programs and 
these were in earlier kernels (apparently it should be resolved in the 
kernels we've tried). I'm also unable to pinpoint the issue itself in 
the logs above.

The individual has tried firmware upgrades for the network drivers, but 
no luck. We are compiling the TC BPF program from source on the machine 
via the following commands:

```

git clone https://github.com/gamemann/IPIPDirect-TC.git

cd IPIPDirect-TC/

git submodule update --init

make

make install

```

The program is loaded via:

```

IPIPDirect_Loader eth0

```

Has anybody else ran into this issues with BPF programs using bnx2x 
network drivers before? I'm unsure if this is something with my 
program's code or a bug with the network drivers. I also tried changing 
'TC_ACT_SHOT' to 'TC_ACT_OK' in my TC BPF program's code, but the 
network drivers still crash after 20 - 30 seconds.

Any help is highly appreciated and thank you for your time!

^ permalink raw reply	[flat|nested] 8+ messages in thread