Re: ixgbe - only presenting one out of four interfaces on 5.9

netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* Re: ixgbe - only presenting one out of four interfaces on 5.9
       [not found] <CAA85sZv=13q8NXbjdf7+R=wu0Q5=Vj9covZ24e9Ew2DCd7S==A@mail.gmail.com>
@ 2020-10-16  9:35 ` Ian Kumlien
  2020-10-16 19:21   ` Jakub Kicinski
  0 siblings, 1 reply; 4+ messages in thread
From: Ian Kumlien @ 2020-10-16  9:35 UTC (permalink / raw)
  To: jeffrey.t.kirsher, intel-wired-lan, Linux Kernel Network Developers

Adding netdev, someone might have a clue of what to look at...

On Mon, Oct 12, 2020 at 9:20 PM Ian Kumlien <ian.kumlien@gmail.com> wrote:
>
> Hi,
>
> I was really surprised when i rebooted my firewall and there was
> network issues, I was even more surprised when
> only one of the four ports of my ixbe (x553) nic was available when booted.
>
> You can actually see it dmesg... And i tried some basic looking at
> changes to see if it was obvious.... but..
>
> anyway, on v5.8.14:
> dmesg |grep ixgbe
> [    1.355454] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
> version 5.1.0-k
> [    1.355455] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
> [    1.711629] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    1.838170] ixgbe 0000:06:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    1.838173] ixgbe 0000:06:00.0: 0c:c4:7a:fa:3d:4a
> [    1.882060] ixgbe 0000:06:00.0: Intel(R) 10 Gigabit Network Connection
> [    1.882196] libphy: ixgbe-mdio: probed
> [    2.234835] ixgbe 0000:06:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.361374] ixgbe 0000:06:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.361377] ixgbe 0000:06:00.1: 0c:c4:7a:fa:3d:4b
> [    2.405281] ixgbe 0000:06:00.1: Intel(R) 10 Gigabit Network Connection
> [    2.757541] ixgbe 0000:07:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.884104] ixgbe 0000:07:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.884107] ixgbe 0000:07:00.0: 0c:c4:7a:fa:3d:4c
> [    2.928022] ixgbe 0000:07:00.0: Intel(R) 10 Gigabit Network Connection
> [    3.280728] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    3.407274] ixgbe 0000:07:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    3.407276] ixgbe 0000:07:00.1: 0c:c4:7a:fa:3d:4d
> [    3.451155] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
> [    4.648725] ixgbe 0000:07:00.0 eno3: renamed from eth2
> [    4.659827] ixgbe 0000:06:00.0 eno1: renamed from eth0
> [    4.674272] ixgbe 0000:07:00.1 eno4: renamed from eth3
> [    4.685838] ixgbe 0000:06:00.1 eno2: renamed from eth1
> [    6.070294] ixgbe 0000:06:00.0: registered PHC device on eno1
> [    7.178307] ixgbe 0000:07:00.0: registered PHC device on eno3
> [    7.421199] ixgbe 0000:07:00.1: registered PHC device on eno4
> [    7.662712] ixgbe 0000:06:00.1: registered PHC device on eno2
> [    9.587640] ixgbe 0000:06:00.0 eno1: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
> [   10.605594] ixgbe 0000:07:00.1 eno4: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
> [   10.754615] ixgbe 0000:07:00.0 eno3: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
> [   10.768586] ixgbe 0000:06:00.1 eno2: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
>
> on 5.9:
> dmesg |grep ixbge
> [    1.570400] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
> [    1.570401] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
> [    1.928030] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.054554] ixgbe 0000:06:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.054556] ixgbe 0000:06:00.0: 0c:c4:7a:fa:3d:4a
> [    2.098404] ixgbe 0000:06:00.0: Intel(R) 10 Gigabit Network Connection
> [    2.098541] libphy: ixgbe-mdio: probed
> [    2.453350] ixgbe 0000:06:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    2.579938] ixgbe 0000:06:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    2.579941] ixgbe 0000:06:00.1: 0c:c4:7a:fa:3d:4b
> [    2.623783] ixgbe 0000:06:00.1: Intel(R) 10 Gigabit Network Connection
> [    2.987339] ixgbe 0000:07:00.0: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    3.113864] ixgbe 0000:07:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
> [    3.113867] ixgbe 0000:07:00.0: 0c:c4:7a:fa:3d:4c
> [    3.157635] ixgbe 0000:07:00.0: Intel(R) 10 Gigabit Network Connection
> [    3.529503] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count
> = 12, Tx Queue count = 12 XDP Queue count = 0
> [    3.656041] ixgbe 0000:07:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
> [    3.656043] ixgbe 0000:07:00.1: 0c:c4:7a:fa:3d:4d
> [    3.699856] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
> [    4.646247] ixgbe 0000:06:00.0 eno1: renamed from eth0
> [    6.183074] ixgbe 0000:06:00.0: registered PHC device on eno1
> [    9.570382] ixgbe 0000:06:00.0 eno1: NIC Link is Up 1 Gbps, Flow
> Control: RX/TX
>
> lspci....
> 06:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
> 06:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
> 07:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
> 07:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> X553 1GbE (rev 11)
>
> ethtool -i eno1 and eno2
> driver: ixgbe
> version: 5.1.0-k
> firmware-version: 0x80000877
> expansion-rom-version:
> bus-info: 0000:06:00.0 (and 0000:06:00.1)
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
>
> while eno3 and eno4 gives:
> driver: ixgbe
> version: 5.1.0-k
> firmware-version: 0x8000087c
> expansion-rom-version:
> bus-info: 0000:07:00.0 (and 0000:07:00.1)
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> Any ideas?

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: ixgbe - only presenting one out of four interfaces on 5.9
  2020-10-16  9:35 ` ixgbe - only presenting one out of four interfaces on 5.9 Ian Kumlien
@ 2020-10-16 19:21   ` Jakub Kicinski
  2020-10-16 22:39     ` Ian Kumlien
  0 siblings, 1 reply; 4+ messages in thread
From: Jakub Kicinski @ 2020-10-16 19:21 UTC (permalink / raw)
  To: Ian Kumlien
  Cc: jeffrey.t.kirsher, intel-wired-lan, Linux Kernel Network Developers

On Fri, 16 Oct 2020 11:35:15 +0200 Ian Kumlien wrote:
> Adding netdev, someone might have a clue of what to look at...
> 
> On Mon, Oct 12, 2020 at 9:20 PM Ian Kumlien <ian.kumlien@gmail.com> wrote:
> >
> > Hi,
> >
> > I was really surprised when i rebooted my firewall and there was
> > network issues, I was even more surprised when
> > only one of the four ports of my ixbe (x553) nic was available when booted.

Just to be sure you mean that the 3 devices are not present in ip link
show?

> > You can actually see it dmesg... And i tried some basic looking at
> > changes to see if it was obvious.... but..

Showing a full dmesg may be helpful, but looking at what you posted it
seems like the driver gets past the point where netdev gets registered,
so the only thing that could fail after that point AFIACT is
mdiobus_register(). Could be some breakage in MDIO.

Any chance you could fire up perf, bpftrace and install a kretprobe to
see what mdiobus_register() returns? You can rebind the device to the
driver through sysfs.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: ixgbe - only presenting one out of four interfaces on 5.9
  2020-10-16 19:21   ` Jakub Kicinski
@ 2020-10-16 22:39     ` Ian Kumlien
  2020-10-16 23:05       ` Jakub Kicinski
  0 siblings, 1 reply; 4+ messages in thread
From: Ian Kumlien @ 2020-10-16 22:39 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: jeffrey.t.kirsher, intel-wired-lan, Linux Kernel Network Developers

On Fri, Oct 16, 2020 at 9:21 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 16 Oct 2020 11:35:15 +0200 Ian Kumlien wrote:
> > Adding netdev, someone might have a clue of what to look at...
> >
> > On Mon, Oct 12, 2020 at 9:20 PM Ian Kumlien <ian.kumlien@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I was really surprised when i rebooted my firewall and there was
> > > network issues, I was even more surprised when
> > > only one of the four ports of my ixbe (x553) nic was available when booted.
>
> Just to be sure you mean that the 3 devices are not present in ip link
> show?

or ifconfig or /proc etc etc, so yes

> > > You can actually see it dmesg... And i tried some basic looking at
> > > changes to see if it was obvious.... but..
>
> Showing a full dmesg may be helpful, but looking at what you posted it
> seems like the driver gets past the point where netdev gets registered,
> so the only thing that could fail after that point AFIACT is
> mdiobus_register(). Could be some breakage in MDIO.

Humm... interesting, will have a look at it

> Any chance you could fire up perf, bpftrace and install a kretprobe to
> see what mdiobus_register() returns? You can rebind the device to the
> driver through sysfs.

Do you need a difference between the kernels?

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: ixgbe - only presenting one out of four interfaces on 5.9
  2020-10-16 22:39     ` Ian Kumlien
@ 2020-10-16 23:05       ` Jakub Kicinski
  0 siblings, 0 replies; 4+ messages in thread
From: Jakub Kicinski @ 2020-10-16 23:05 UTC (permalink / raw)
  To: Ian Kumlien
  Cc: jeffrey.t.kirsher, intel-wired-lan, Linux Kernel Network Developers

On Sat, 17 Oct 2020 00:39:11 +0200 Ian Kumlien wrote:
> On Fri, Oct 16, 2020 at 9:21 PM Jakub Kicinski <kuba@kernel.org> wrote:
> > > > You can actually see it dmesg... And i tried some basic looking at
> > > > changes to see if it was obvious.... but..  
> >
> > Showing a full dmesg may be helpful, but looking at what you posted it
> > seems like the driver gets past the point where netdev gets registered,
> > so the only thing that could fail after that point AFIACT is
> > mdiobus_register(). Could be some breakage in MDIO.  
> 
> Humm... interesting, will have a look at it
> 
> > Any chance you could fire up perf, bpftrace and install a kretprobe to
> > see what mdiobus_register() returns? You can rebind the device to the
> > driver through sysfs.  
> 
> Do you need a difference between the kernels?

Nope, we can safely assume that it return 0 on kernels where things
work.

Looking at the changes in this area now - it's gotta be:

commit 09ef193fef7e ("net: ethernet: ixgbe: check the return value of ixgbe_mii_bus_init()")

This should make things work again for you:

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
index f77fa3e4fdd1..ac5bfc2b5f81 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
@@ -922,7 +922,7 @@ s32 ixgbe_mii_bus_init(struct ixgbe_hw *hw)
        case IXGBE_DEV_ID_X550EM_A_1G_T:
        case IXGBE_DEV_ID_X550EM_A_1G_T_L:
                if (!ixgbe_x550em_a_has_mii(hw))
-                       return -ENODEV;
+                       return 0;
                bus->read = &ixgbe_x550em_a_mii_bus_read;
                bus->write = &ixgbe_x550em_a_mii_bus_write;
                break;

^ permalink raw reply related	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2020-10-16 23:05 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <CAA85sZv=13q8NXbjdf7+R=wu0Q5=Vj9covZ24e9Ew2DCd7S==A@mail.gmail.com>
2020-10-16  9:35 ` ixgbe - only presenting one out of four interfaces on 5.9 Ian Kumlien
2020-10-16 19:21   ` Jakub Kicinski
2020-10-16 22:39     ` Ian Kumlien
2020-10-16 23:05       ` Jakub Kicinski

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).