Linux-Modules Archive on lore.kernel.org
 help / Atom feed
* Diagnosing network module for missing link establishment (cxgb3, Chelsio T320)
@ 2018-06-28  9:33 U.Mutlu
  2018-06-28 16:07 ` Lucas De Marchi
  0 siblings, 1 reply; 2+ messages in thread
From: U.Mutlu @ 2018-06-28  9:33 UTC (permalink / raw)
  To: linux-modules

Hi,

I have got a pair of used old dual port 10GbE NICs (Chelsio T320
10GBASE-R RNIC (rev 3) PCI Express x4 MSI-X) with 2 modular
transceivers on board the 2 NICs (ie. these can be taken off
of the card for replacement etc.).

The problem is that the cards don't establish a link;
the green LEDs go off after a few seconds after loading
the driver named cxgb3, and there is no indication in the syslog
about any error.

The behavior is the same whether the transceivers are present
on the card or not; the drivers always load successfully.

In the kernel sources the driver is located under
drivers/net/ethernet/chelsio/cxgb3
(I haven't recompiled it; just using the stock kernel from Debian repo).

Below, the interfaces are eth4 and eth5.

There was just once a link, but it never happens again, and
such log entries about link are not happening since then:

Jun 27 13:06:09 c6-local vmunix: [  504.108102] cxgb3 0000:01:00.0 eth4: link 
up, 10Gbps, full-duplex
Jun 27 13:17:30 c6-local vmunix: [ 1185.450256] cxgb3 0000:01:00.0 eth4: link down

How can I diagnose and pinpoint what the reason is for not establishing a link?


The following is from a later reboot:

# dmesg | grep -i "Chelsio\|cxgb3\|eth[0-9]"
[    0.979800] cxgb3: Chelsio T3 Network Driver - version 1.1.5-ko
[    0.984162] r8169 0000:02:00.0 eth0: RTL8168evl/8111evl at 
0xffffc90000002000, 74:d4:35:92:72:1b, XID 0c900800 IRQ 43
[    0.984163] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx 
checksumming: ko]
[    1.319780] cxgb3 0000:01:00.0: irq 55 for MSI/MSI-X
[    1.319785] cxgb3 0000:01:00.0: irq 56 for MSI/MSI-X
[    1.319788] cxgb3 0000:01:00.0: irq 57 for MSI/MSI-X
[    1.319791] cxgb3 0000:01:00.0: irq 58 for MSI/MSI-X
[    1.319794] cxgb3 0000:01:00.0: irq 59 for MSI/MSI-X
[    1.319797] cxgb3 0000:01:00.0: irq 60 for MSI/MSI-X
[    1.319799] cxgb3 0000:01:00.0: irq 61 for MSI/MSI-X
[    1.319803] cxgb3 0000:01:00.0: irq 62 for MSI/MSI-X
[    1.319805] cxgb3 0000:01:00.0: irq 63 for MSI/MSI-X
[    1.319828] cxgb3 0000:01:00.0: Port 0 using 4 queue sets.
[    1.319868] cxgb3 0000:01:00.0: Port 1 using 4 queue sets.
[    1.319909] cxgb3 0000:01:00.0 eth1: Chelsio T320 10GBASE-R RNIC (rev 3) 
PCI Express x8 MSI-X
[    1.319949] cxgb3: eth1: 128MB CM, 256MB PMTX, 256MB PMRX, S/N: PT37080022
[    1.319986] cxgb3 0000:01:00.0 eth2: Chelsio T320 10GBASE-R RNIC (rev 3) 
PCI Express x8 MSI-X
[    5.217529] systemd-udevd[384]: renamed network interface eth0 to eth3
[    5.342653] systemd-udevd[387]: renamed network interface eth2 to eth4
[    5.358663] systemd-udevd[383]: renamed network interface eth1 to eth5
[   11.563555] r8169 0000:02:00.0 eth3: link down
[   11.563668] r8169 0000:02:00.0 eth3: link down
[   13.917486] r8169 0000:02:00.0 eth3: link up


# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode 
DEFAULT group default
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP 
mode DEFAULT group default qlen 1000
     link/ether 74:d4:35:92:72:1b brd ff:ff:ff:ff:ff:ff
3: eth5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 00:07:43:05:8b:16 brd ff:ff:ff:ff:ff:ff
4: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 00:07:43:05:8b:17 brd ff:ff:ff:ff:ff:ff


# ethtool eth4
Settings for eth4:
     Supported ports: [ AUI FIBRE ]
     Supported link modes:   10000baseT/Full
     Supported pause frame use: No
     Supports auto-negotiation: No
     Advertised link modes:  Not reported
     Advertised pause frame use: No
     Advertised auto-negotiation: No
     Speed: Unknown!
     Duplex: Unknown! (255)
     Port: FIBRE
     PHYAD: 1
     Transceiver: external
     Auto-negotiation: off
     Supports Wake-on: d
     Wake-on: d
     Current message level: 0x000000ff (255)
                    drv probe link timer ifdown ifup rx_err tx_err
     Link detected: no


# ifconfig
...
eth4      Link encap:Ethernet  HWaddr 00:07:43:05:8b:17
           inet addr:192.168.50.4  Bcast:192.168.50.255  Mask:255.255.255.0
           UP BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
           Interrupt:18 Memory:fe811000-fe811fff

eth5      Link encap:Ethernet  HWaddr 00:07:43:05:8b:16
           inet addr:192.168.60.5  Bcast:192.168.60.255  Mask:255.255.255.0
           UP BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
           Interrupt:18 Memory:fe811000-fe811fff

(I also tried same network 192.168.50 for both, no difference in outcome)


Btw, can a link be established between the 2 transceiver ports on the same 
card? I think this should be possible, right?


Kernel: 3.16.0-4-amd64 (Debian v8 stock kernel)

-- 
Thx


^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: Diagnosing network module for missing link establishment (cxgb3, Chelsio T320)
  2018-06-28  9:33 Diagnosing network module for missing link establishment (cxgb3, Chelsio T320) U.Mutlu
@ 2018-06-28 16:07 ` Lucas De Marchi
  0 siblings, 0 replies; 2+ messages in thread
From: Lucas De Marchi @ 2018-06-28 16:07 UTC (permalink / raw)
  To: um; +Cc: linux-modules

Hi,

This mailing list is specific for the module loading tools and related
stuff. You may have a better success on getting a response if you send
this to the linux-netdev mailing list or lkml.

Lucas De Marchi

On Thu, Jun 28, 2018 at 2:42 AM U.Mutlu <um@mutluit.com> wrote:
>
> Hi,
>
> I have got a pair of used old dual port 10GbE NICs (Chelsio T320
> 10GBASE-R RNIC (rev 3) PCI Express x4 MSI-X) with 2 modular
> transceivers on board the 2 NICs (ie. these can be taken off
> of the card for replacement etc.).
>
> The problem is that the cards don't establish a link;
> the green LEDs go off after a few seconds after loading
> the driver named cxgb3, and there is no indication in the syslog
> about any error.
>
> The behavior is the same whether the transceivers are present
> on the card or not; the drivers always load successfully.
>
> In the kernel sources the driver is located under
> drivers/net/ethernet/chelsio/cxgb3
> (I haven't recompiled it; just using the stock kernel from Debian repo).
>
> Below, the interfaces are eth4 and eth5.
>
> There was just once a link, but it never happens again, and
> such log entries about link are not happening since then:
>
> Jun 27 13:06:09 c6-local vmunix: [  504.108102] cxgb3 0000:01:00.0 eth4: link
> up, 10Gbps, full-duplex
> Jun 27 13:17:30 c6-local vmunix: [ 1185.450256] cxgb3 0000:01:00.0 eth4: link down
>
> How can I diagnose and pinpoint what the reason is for not establishing a link?
>
>
> The following is from a later reboot:
>
> # dmesg | grep -i "Chelsio\|cxgb3\|eth[0-9]"
> [    0.979800] cxgb3: Chelsio T3 Network Driver - version 1.1.5-ko
> [    0.984162] r8169 0000:02:00.0 eth0: RTL8168evl/8111evl at
> 0xffffc90000002000, 74:d4:35:92:72:1b, XID 0c900800 IRQ 43
> [    0.984163] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx
> checksumming: ko]
> [    1.319780] cxgb3 0000:01:00.0: irq 55 for MSI/MSI-X
> [    1.319785] cxgb3 0000:01:00.0: irq 56 for MSI/MSI-X
> [    1.319788] cxgb3 0000:01:00.0: irq 57 for MSI/MSI-X
> [    1.319791] cxgb3 0000:01:00.0: irq 58 for MSI/MSI-X
> [    1.319794] cxgb3 0000:01:00.0: irq 59 for MSI/MSI-X
> [    1.319797] cxgb3 0000:01:00.0: irq 60 for MSI/MSI-X
> [    1.319799] cxgb3 0000:01:00.0: irq 61 for MSI/MSI-X
> [    1.319803] cxgb3 0000:01:00.0: irq 62 for MSI/MSI-X
> [    1.319805] cxgb3 0000:01:00.0: irq 63 for MSI/MSI-X
> [    1.319828] cxgb3 0000:01:00.0: Port 0 using 4 queue sets.
> [    1.319868] cxgb3 0000:01:00.0: Port 1 using 4 queue sets.
> [    1.319909] cxgb3 0000:01:00.0 eth1: Chelsio T320 10GBASE-R RNIC (rev 3)
> PCI Express x8 MSI-X
> [    1.319949] cxgb3: eth1: 128MB CM, 256MB PMTX, 256MB PMRX, S/N: PT37080022
> [    1.319986] cxgb3 0000:01:00.0 eth2: Chelsio T320 10GBASE-R RNIC (rev 3)
> PCI Express x8 MSI-X
> [    5.217529] systemd-udevd[384]: renamed network interface eth0 to eth3
> [    5.342653] systemd-udevd[387]: renamed network interface eth2 to eth4
> [    5.358663] systemd-udevd[383]: renamed network interface eth1 to eth5
> [   11.563555] r8169 0000:02:00.0 eth3: link down
> [   11.563668] r8169 0000:02:00.0 eth3: link down
> [   13.917486] r8169 0000:02:00.0 eth3: link up
>
>
> # ip link show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
> DEFAULT group default
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
> mode DEFAULT group default qlen 1000
>      link/ether 74:d4:35:92:72:1b brd ff:ff:ff:ff:ff:ff
> 3: eth5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode
> DEFAULT group default qlen 1000
>      link/ether 00:07:43:05:8b:16 brd ff:ff:ff:ff:ff:ff
> 4: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode
> DEFAULT group default qlen 1000
>      link/ether 00:07:43:05:8b:17 brd ff:ff:ff:ff:ff:ff
>
>
> # ethtool eth4
> Settings for eth4:
>      Supported ports: [ AUI FIBRE ]
>      Supported link modes:   10000baseT/Full
>      Supported pause frame use: No
>      Supports auto-negotiation: No
>      Advertised link modes:  Not reported
>      Advertised pause frame use: No
>      Advertised auto-negotiation: No
>      Speed: Unknown!
>      Duplex: Unknown! (255)
>      Port: FIBRE
>      PHYAD: 1
>      Transceiver: external
>      Auto-negotiation: off
>      Supports Wake-on: d
>      Wake-on: d
>      Current message level: 0x000000ff (255)
>                     drv probe link timer ifdown ifup rx_err tx_err
>      Link detected: no
>
>
> # ifconfig
> ...
> eth4      Link encap:Ethernet  HWaddr 00:07:43:05:8b:17
>            inet addr:192.168.50.4  Bcast:192.168.50.255  Mask:255.255.255.0
>            UP BROADCAST MULTICAST  MTU:1500  Metric:1
>            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>            Interrupt:18 Memory:fe811000-fe811fff
>
> eth5      Link encap:Ethernet  HWaddr 00:07:43:05:8b:16
>            inet addr:192.168.60.5  Bcast:192.168.60.255  Mask:255.255.255.0
>            UP BROADCAST MULTICAST  MTU:1500  Metric:1
>            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>            Interrupt:18 Memory:fe811000-fe811fff
>
> (I also tried same network 192.168.50 for both, no difference in outcome)
>
>
> Btw, can a link be established between the 2 transceiver ports on the same
> card? I think this should be possible, right?
>
>
> Kernel: 3.16.0-4-amd64 (Debian v8 stock kernel)
>
> --
> Thx
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-modules" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Lucas De Marchi

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, back to index

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-28  9:33 Diagnosing network module for missing link establishment (cxgb3, Chelsio T320) U.Mutlu
2018-06-28 16:07 ` Lucas De Marchi

Linux-Modules Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-modules/0 linux-modules/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-modules linux-modules/ https://lore.kernel.org/linux-modules \
		linux-modules@vger.kernel.org linux-modules@archiver.kernel.org
	public-inbox-index linux-modules


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-modules


AGPL code for this site: git clone https://public-inbox.org/ public-inbox