RE: bnx2x - odd behaviour

[Sudarsana Reddy Kalluru <skalluru@marvell.com>]

Hi Ian,
    Thanks for your info. Mfw team already analyzed the "nig timer" related logs but can't infer anything. From the boot-code version, the device look to be from the older generation of Broadcom nics. Besides the elink-logs/register-dump, could you also share the lspci output (lspci -vvv).

Thanks,
Sudarsana
> -----Original Message-----
> From: Ian Kumlien <ian.kumlien@gmail.com>
> Sent: Wednesday, April 17, 2019 6:51 PM
> To: Sudarsana Reddy Kalluru <skalluru@marvell.com>
> Cc: Linux Kernel Network Developers <netdev@vger.kernel.org>; Ariel Elior
> <aelior@marvell.com>; Ameen Rahman <arahman@marvell.com>
> Subject: Re: bnx2x - odd behaviour
> 
> On Wed, Apr 17, 2019 at 3:05 PM Sudarsana Reddy Kalluru
> <skalluru@marvell.com> wrote:
> >
> > > -----Original Message-----
> > > From: Ian Kumlien <ian.kumlien@gmail.com>
> > > Sent: Wednesday, April 17, 2019 4:32 PM
> > > To: Sudarsana Reddy Kalluru <skalluru@marvell.com>
> > > Cc: Linux Kernel Network Developers <netdev@vger.kernel.org>; Ariel
> > > Elior <aelior@marvell.com>; Ameen Rahman <arahman@marvell.com>
> > > Subject: Re: bnx2x - odd behaviour
> > >
> > > On Wed, Apr 17, 2019 at 9:58 AM Sudarsana Reddy Kalluru
> > > <skalluru@marvell.com> wrote:
> > > >
> > > > +Ameen
> > > >
> > > > Ian,
> > > >     We couldn't find the root-cause from the logs/register-dump.
> > > > Could you please load the driver with link-debugs enabled, i.e.,
> > > > modprobe
> > > bnx2x debug=0x4 or 'ethtool -s <interface> msglvl 0x4'. And collect
> > > the complete kernel logs and the register-dump(collected before
> > > performing ifconfig-down). Please also provide the output of "ethtool -i
> <interface>".
> > >
> > > I'll try, this is a production system...
> > >
> > > Could it be related to the gro changes for UDP that was done in 5.x?
> > >
> > Thanks for your help. I'm not sure if this is related to gro, link related code
> is handled by different component [management firmware (mfw)]. May be
> the complete logs/register-dump provide some additional pointers. There
> were some fixes in the newer version of mfw, getting the mfw version on the
> chip would help (ethtool -i <interface> provides mfw/boot-code version).
> 
> ethtool -i enp2s0f0
> driver: bnx2x
> version: 1.712.30-0 storm 7.13.1.0
> firmware-version: bc 6.2.28 phy baa0.105
> expansion-rom-version:
> bus-info: 0000:02:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
> 
> What we can see in the logs (not with the linkdebug enabled) is:
> apr 12 06:22:35 localhost kernel: bnx2x 0000:02:00.0 enp2s0f0: NIC Link is
> Down apr 12 06:22:35 localhost kernel: bond0: link status down for active
> interface enp2s0f0, disabling it in 1000 ms apr 12 06:22:35 localhost kernel:
> bnx2x 0000:02:00.0 enp2s0f0: NIC Link is Up, 10000 Mbps full duplex, Flow
> control: ON - transmit apr 12 06:22:35 localhost kernel: bond0: link status up
> again after
> 400 ms for interface enp2s0f0
> apr 12 06:22:36 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:36 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:37 localhost kernel: bnx2x:
> [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (1) apr 12 06:22:37
> localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:37 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:38 localhost kernel: bnx2x:
> [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (2) apr 12 06:22:38
> localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:38 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:39 localhost kernel: bnx2x:
> [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (3) apr 12 06:22:39
> localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:39 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:40 localhost kernel: bnx2x:
> [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (4) apr 12 06:22:40
> localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:40 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:41 localhost kernel: bnx2x:
> [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (5) apr 12 06:22:41
> localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:41 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:42 localhost kernel: bnx2x:
> [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (6) apr 12 06:22:42
> localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:42 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:43 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> 0x04000000 (masked)
> apr 12 06:22:43 localhost kernel: bnx2x:
> [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC time-out 0x08004384 apr
> 12 06:22:44 localhost kernel: bnx2x:
> [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (7) ... and so it
> begins =)
> 
> > > > Thanks,
> > > > Sudarsana
> > > > > -----Original Message-----
> > > > > From: Ian Kumlien <ian.kumlien@gmail.com>
> > > > > Sent: Friday, April 12, 2019 4:39 PM
> > > > > To: Sudarsana Reddy Kalluru <skalluru@marvell.com>
> > > > > Cc: Linux Kernel Network Developers <netdev@vger.kernel.org>;
> > > > > Ariel Elior <aelior@marvell.com>
> > > > > Subject: Re: bnx2x - odd behaviour
> > > > >
> > > > > On Fri, Apr 12, 2019 at 12:53 PM Sudarsana Reddy Kalluru
> > > > > <skalluru@marvell.com> wrote:
> > > > > >
> > > > > > Hi Ian,
> > > > > >    Thanks for your info/help. There's not much info in the
> > > > > > logs (e.g., FW
> > > > > traces, calltraces). Will contact our firmware team on the
> > > > > register-dump analysis and provide you the update.
> > > > >
> > > > > Thank you =)
> > > > >
> > > > > > Thanks,
> > > > > > Sudarsana
> > > > > > > -----Original Message-----
> > > > > > > From: Ian Kumlien <ian.kumlien@gmail.com>
> > > > > > > Sent: Friday, April 12, 2019 2:44 PM
> > > > > > > To: Sudarsana Reddy Kalluru <skalluru@marvell.com>
> > > > > > > Cc: Linux Kernel Network Developers
> > > > > > > <netdev@vger.kernel.org>; Ariel Elior <aelior@marvell.com>
> > > > > > > Subject: Re: bnx2x - odd behaviour
> > > > > > >
> > > > > > > Finally!
> > > > > > >
> > > > > > > Just had a machine with the same issue!
> > > > > > >
> > > > > > > On Thu, Apr 11, 2019 at 10:56 AM Ian Kumlien
> > > > > > > <ian.kumlien@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > On Thu, Apr 4, 2019 at 4:27 PM Sudarsana Reddy Kalluru
> > > > > > > > <skalluru@marvell.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >    We are not aware of this issue. Please collect the
> > > > > > > > > register dump i.e.,
> > > > > > > "ethtool -d <interface>" output when this issue happens
> > > > > > > (before performing
> > > > > > > link-flap) and share it for the analysis.
> > > > > > >
> > > > > > > Sent the dump separately :)