Re: ixgbe - only presenting one out of four interfaces on 5.9

Re: ixgbe - only presenting one out of four interfaces on 5.9

[Ian Kumlien]

Adding netdev, someone might have a clue of what to look at...

On Mon, Oct 12, 2020 at 9:20 PM Ian Kumlien <ian.kumlien@gmail.com> wrote:
>
> Hi,
>
> I was really surprised when i rebooted my firewall and there was
> network issues, I was even more surprised when
> only one of the four ports of my ixbe (x553) nic was available when booted.
>
> You can actually see it dmesg... And i tried some basic looking at
> changes to see if it was obvious.... but..
>
> anyway, on v5.8.14:
> dmesg |grep ixgbe
> [    1.355454] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
> version 5.1.0-k
> [    1.355455] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
> [    1.711629] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    1.838170] ixgbe 0000:06:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    1.838173] ixgbe 0000:06:00.0: 0c:c4:7a:fa:3d:4a
> [    1.882060] ixgbe 0000:06:00.0: Intel(R) 10 Gigabit Network Connection
> [    1.882196] libphy: ixgbe-mdio: probed
> [    2.234835] ixgbe 0000:06:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.361374] ixgbe 0000:06:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.361377] ixgbe 0000:06:00.1: 0c:c4:7a:fa:3d:4b
> [    2.405281] ixgbe 0000:06:00.1: Intel(R) 10 Gigabit Network Connection
> [    2.757541] ixgbe 0000:07:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.884104] ixgbe 0000:07:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.884107] ixgbe 0000:07:00.0: 0c:c4:7a:fa:3d:4c
> [    2.928022] ixgbe 0000:07:00.0: Intel(R) 10 Gigabit Network Connection
> [    3.280728] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    3.407274] ixgbe 0000:07:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    3.407276] ixgbe 0000:07:00.1: 0c:c4:7a:fa:3d:4d
> [    3.451155] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
> [    4.648725] ixgbe 0000:07:00.0 eno3: renamed from eth2
> [    4.659827] ixgbe 0000:06:00.0 eno1: renamed from eth0
> [    4.674272] ixgbe 0000:07:00.1 eno4: renamed from eth3
> [    4.685838] ixgbe 0000:06:00.1 eno2: renamed from eth1
> [    6.070294] ixgbe 0000:06:00.0: registered PHC device on eno1
> [    7.178307] ixgbe 0000:07:00.0: registered PHC device on eno3
> [    7.421199] ixgbe 0000:07:00.1: registered PHC device on eno4
> [    7.662712] ixgbe 0000:06:00.1: registered PHC device on eno2
> [    9.587640] ixgbe 0000:06:00.0 eno1: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
> [   10.605594] ixgbe 0000:07:00.1 eno4: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
> [   10.754615] ixgbe 0000:07:00.0 eno3: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
> [   10.768586] ixgbe 0000:06:00.1 eno2: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
>
> on 5.9:
> dmesg |grep ixbge
> [    1.570400] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
> [    1.570401] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
> [    1.928030] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.054554] ixgbe 0000:06:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.054556] ixgbe 0000:06:00.0: 0c:c4:7a:fa:3d:4a
> [    2.098404] ixgbe 0000:06:00.0: Intel(R) 10 Gigabit Network Connection
> [    2.098541] libphy: ixgbe-mdio: probed
> [    2.453350] ixgbe 0000:06:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.579938] ixgbe 0000:06:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.579941] ixgbe 0000:06:00.1: 0c:c4:7a:fa:3d:4b
> [    2.623783] ixgbe 0000:06:00.1: Intel(R) 10 Gigabit Network Connection
> [    2.987339] ixgbe 0000:07:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    3.113864] ixgbe 0000:07:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    3.113867] ixgbe 0000:07:00.0: 0c:c4:7a:fa:3d:4c
> [    3.157635] ixgbe 0000:07:00.0: Intel(R) 10 Gigabit Network Connection
> [    3.529503] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    3.656041] ixgbe 0000:07:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    3.656043] ixgbe 0000:07:00.1: 0c:c4:7a:fa:3d:4d
> [    3.699856] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
> [    4.646247] ixgbe 0000:06:00.0 eno1: renamed from eth0
> [    6.183074] ixgbe 0000:06:00.0: registered PHC device on eno1
> [    9.570382] ixgbe 0000:06:00.0 eno1: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
>
> lspci....
> 06:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
> 06:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
> 07:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
> 07:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
>
> ethtool -i eno1 and eno2
> driver: ixgbe
> version: 5.1.0-k
> firmware-version: 0x80000877
> expansion-rom-version:
> bus-info: 0000:06:00.0 (and 0000:06:00.1)
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
>
> while eno3 and eno4 gives:
> driver: ixgbe
> version: 5.1.0-k
> firmware-version: 0x8000087c
> expansion-rom-version:
> bus-info: 0000:07:00.0 (and 0000:07:00.1)
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> Any ideas?

Re: ixgbe - only presenting one out of four interfaces on 5.9

[Jakub Kicinski]

On Fri, 16 Oct 2020 11:35:15 +0200 Ian Kumlien wrote:
> Adding netdev, someone might have a clue of what to look at...
> 
> On Mon, Oct 12, 2020 at 9:20 PM Ian Kumlien <ian.kumlien@gmail.com> wrote:
> >
> > Hi,
> >
> > I was really surprised when i rebooted my firewall and there was
> > network issues, I was even more surprised when
> > only one of the four ports of my ixbe (x553) nic was available when booted.

Just to be sure you mean that the 3 devices are not present in ip link
show?

> > You can actually see it dmesg... And i tried some basic looking at
> > changes to see if it was obvious.... but..

Showing a full dmesg may be helpful, but looking at what you posted it
seems like the driver gets past the point where netdev gets registered,
so the only thing that could fail after that point AFIACT is
mdiobus_register(). Could be some breakage in MDIO.

Any chance you could fire up perf, bpftrace and install a kretprobe to
see what mdiobus_register() returns? You can rebind the device to the
driver through sysfs.

Re: ixgbe - only presenting one out of four interfaces on 5.9

[Ian Kumlien]

On Fri, Oct 16, 2020 at 9:21 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 16 Oct 2020 11:35:15 +0200 Ian Kumlien wrote:
> > Adding netdev, someone might have a clue of what to look at...
> >
> > On Mon, Oct 12, 2020 at 9:20 PM Ian Kumlien <ian.kumlien@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I was really surprised when i rebooted my firewall and there was
> > > network issues, I was even more surprised when
> > > only one of the four ports of my ixbe (x553) nic was available when booted.
>
> Just to be sure you mean that the 3 devices are not present in ip link
> show?

or ifconfig or /proc etc etc, so yes

> > > You can actually see it dmesg... And i tried some basic looking at
> > > changes to see if it was obvious.... but..
>
> Showing a full dmesg may be helpful, but looking at what you posted it
> seems like the driver gets past the point where netdev gets registered,
> so the only thing that could fail after that point AFIACT is
> mdiobus_register(). Could be some breakage in MDIO.

Humm... interesting, will have a look at it

> Any chance you could fire up perf, bpftrace and install a kretprobe to
> see what mdiobus_register() returns? You can rebind the device to the
> driver through sysfs.

Do you need a difference between the kernels?

Re: ixgbe - only presenting one out of four interfaces on 5.9

[Jakub Kicinski]

On Sat, 17 Oct 2020 00:39:11 +0200 Ian Kumlien wrote:
> On Fri, Oct 16, 2020 at 9:21 PM Jakub Kicinski <kuba@kernel.org> wrote:
> > > > You can actually see it dmesg... And i tried some basic looking at
> > > > changes to see if it was obvious.... but..  
> >
> > Showing a full dmesg may be helpful, but looking at what you posted it
> > seems like the driver gets past the point where netdev gets registered,
> > so the only thing that could fail after that point AFIACT is
> > mdiobus_register(). Could be some breakage in MDIO.  
> 
> Humm... interesting, will have a look at it
> 
> > Any chance you could fire up perf, bpftrace and install a kretprobe to
> > see what mdiobus_register() returns? You can rebind the device to the
> > driver through sysfs.  
> 
> Do you need a difference between the kernels?

Nope, we can safely assume that it return 0 on kernels where things
work.

Looking at the changes in this area now - it's gotta be:

commit 09ef193fef7e ("net: ethernet: ixgbe: check the return value of ixgbe_mii_bus_init()")

This should make things work again for you:

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
index f77fa3e4fdd1..ac5bfc2b5f81 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
@@ -922,7 +922,7 @@ s32 ixgbe_mii_bus_init(struct ixgbe_hw *hw)
        case IXGBE_DEV_ID_X550EM_A_1G_T:
        case IXGBE_DEV_ID_X550EM_A_1G_T_L:
                if (!ixgbe_x550em_a_has_mii(hw))
-                       return -ENODEV;
+                       return 0;
                bus->read = &ixgbe_x550em_a_mii_bus_read;
                bus->write = &ixgbe_x550em_a_mii_bus_write;
                break;