* bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
@ 2020-07-18 14:27 Chris Healy
2020-07-18 14:42 ` Marek Behun
0 siblings, 1 reply; 11+ messages in thread
From: Chris Healy @ 2020-07-18 14:27 UTC (permalink / raw)
To: netdev
I've been trying to get the fiber interface of the vf610-zii-dev-rev-c
board working with net-next to no avail. This platform utilizes a
Cotsworks SFF attached to port 9 of a Marvell 88E6390X.
I have fiber link up on port 9 and am able to send packets from the
management CPU of the switch through the switch and out port 9 through
the fiber interface to a fiber link partner successfully. I'm also
able to send packets from that fiber link partner back (to the point
of the switches SERDES) and am seeing the fiber ports SERDES RX
counters increment for each packet transmitted by the link partner.
The switches port 9 MAC is not showing any RX counters incrementing
though and I do not receive the frames at the management CPU.
Because the SERDES RX counters are incrementing while the MAC RX
counters are not incrementing, it seems to me that the issue is
between the MAC and SERDES. This is odd to me given that TX works
fine.
In support of debugging this issue, I've applied an ethtool patch
which allows decoding the 88E6390X SERDES registers from userspace.
Looking at the register dump, nothing obvious sticks out to me though.
I'm not sure what the right next steps are and would appreciate any
theories on what to try/test to root cause this issue.
Below is an ethtool register dump and an ethtool statistics dump from
when the link is up and I have done some attempts at pinging a remote
host over fiber.
root@(none):~$ ethtool -d sff2
88E6390X Switch Port Registers
------------------------------
00: Port Status 0xce49
Transmit Pause Enable bit 1
Receive Pause Enable bit 1
802.3 PHY Detected 0
Link Status Up
Duplex Full
Speed 1000 Mbps
Duplex Fixed 0
EEE Enabled 1
Transmitter Paused 0
Flow Control 0
Config Mode 0x9
01: Physical Control 0x203e
RGMII Receive Timing Control Default
RGMII Transmit Timing Control Default
Force Speed 1
Alternate Speed Mode Normal
MII PHY Mode MAC
EEE force value 0
Force EEE 0
Link's Forced value Up
Force Link 1
Duplex's Forced value Full
Force Duplex 1
Force Speed 1000 Mbps
02: Flow Control 0x0100
03: Switch Identifier 0x0a11
04: Port Control 0x0433
Source Address Filtering controls Disabled
Egress Mode Unmodified
Ingress & Egress Header Mode 0
IGMP and MLD Snooping 1
Frame Mode Normal
VLAN Tunnel 0
TagIfBoth 0
Initial Priority assignment Tag & IP Priority
Egress Flooding mode No unknown DA
Port State Forwarding
05: Port Control 1 0x0000
Message Port 0
LAG Port 0
VTU Page 0
LAG ID 0
FID[11:4] 0x000
06: Port Base VLAN Map (Header) 0x0400
FID[3:0] 0x000
Force Mapping 0
VLANTable 10
07: Default VLAN ID & Priority 0x0000
Default Priority 0x0
Force to use Default VID 0
Default VLAN Identifier 0
08: Port Control 2 0x0080
Force good FCS in the frame 0
Allow bad FCS 0
Jumbo Mode 1522
802.1QMode Disabled
Discard Tagged Frames 0
Discard Untagged Frames 0
Map using DA hits 1
ARP Mirror enable 0
Egress Monitor Source Port 0
Ingress Monitor Source Port 0
Allow VID of Zero 0
Default Queue Priority 0x0
09: Egress Rate Control 0x0001
10: Egress Rate Control 2 0x0000
11: Port Association Vector 0x0200
12: Port ATU Control 0x0000
13: Override 0x0000
14: Policy Control 0x0000
15: Port Ether Type 0x9100
16: Reserved 0x0000
17: Reserved 0x0000
18: Reserved 0x0000
19: Reserved 0x0000
20: Reserved 0x0000
21: Reserved 0x0000
22: LED Control 0x0033
23: IP Priority Mapping Table 0x0000
24: IEEE Priority Mapping Table 0x3e07
25: Port Control 3 0x0000
26: Reserved 0x0000
27: Queue Counters 0x8000
28: Queue Control 0x0000
29: Reserved 0x0000
30: Cut Through Control 0x0000
31: Debug Counters 0x0000
88E6390X Switch Port SERDES Registers
-------------------------------------
f000: Global Clock Configuration 1 0x0000
f001: Global Clock Configuration 2 0x0002
f002: Port Operational Configuration 0x0003
f00a: FIFO and CRC Int Enable 0x0000
f00b: FIFO and CRC Int Status 0x0000
f00c: PPM FIFO Control 1 0x0000
f00d: PPM FIFO Control 2 0x0000
f00e: PPM FIFO Status 0x0000
f010: Packet Generation Control 1 0x0501
f011: Packet Generation Control 2 0x0000
f012: Initial Payload 0-1/Packet Generation 0x0000
f013: Initial Payload 2-3/Packet Generation 0x0000
f016: Packet Generation Length 0x0000
f017: Packet Generation Burst Sequence 0x0000
f018: Packet Generation IPG 0x0002
f019: Packet Gen_Chkr Clock Control 0x0000
f01a: Transmit Powerdown Delay 0x0032
f01b: Transmit Packet Counter [15:0] 0x0000
f01c: Transmit Packet Counter [31:16] 0x0000
f01d: Transmit Packet Counter [47:32] 0x0000
f01e: Transmit Byte Counter [15:0] 0x0000
f01f: Transmit Byte Counter [31:16] 0x0000
f020: Transmit Byte Counter [47:32] 0x0000
f021: Receive Packet Counter [15:0] 0x0000
f022: Receive Packet Counter [31:16] 0x0000
f023: Receive Packet Counter [47:32] 0x0000
f024: Receive Byte Count [15:0] 0x0000
f025: Receive Byte Count [31:16] 0x0000
f026: Receive Byte Count [47:32] 0x0000
f027: Receive Packet Error Count [15:0] 0x0000
f028: Receive Packet Error Count [31:16] 0x0000
f029: Receive Packet Error Count [47:32] 0x0000
f030: PRBS Control 0x0200
f031: PRBS Symbol Tx Counter [15:0] 0x0000
f032: PRBS Symbol Tx Counter [31:16] 0x0000
f033: PRBS Symbol Tx Counter [47:32] 0x0000
f034: PRBS Symbol Rx Counter [15:0] 0x0000
f035: PRBS Symbol Rx Counter [31:16] 0x0000
f036: PRBS Symbol RX Counter [47:32] 0x0000
f037: PRBS Error Count [15:0] 0x0000
f038: PRBS Error Count [31:16] 0x0000
f039: PRBS Error Count [47:32] 0x0000
2000: 1000BASE-X/SGMII Control Register 0x1140
Reset 0
Loopback 0
SGMII Speed 1000 Mbps
Autoneg Enable 1
Power down 0
Isolate 0
Restart Autonet 0
Duplex Full
2001: 1000BASE-X/SGMII Status Register 0x016d
Autoneg Complete 1
Remote Fault 0
Link Status Up
2002: PHY Identifier 0x0141
2003: PHY Identifier 0x0c00
2004: SGMII (Media side) Auto-Negotiation Ad 0x00a0
Link Status Down
Duplex Half
SGMII Speed 10 Mbps
Transmit Pause 0
Receive Pause 0
Fibre/Copper Fibre
EEE mode 0
Clock stopped during LPI 1
2005: SGMII (Media side) Link Partner Abilit 0x40a0
Acknowledge 1
2006: 1000BASE-X Auto-Negotiation Expansion 0x0007
2007: 1000BASE-X Next Page Transmit Register 0x2801
2008: 1000BASE-X Link Partner Next Page Regi 0x0000
200f: Extended Status Register 0x8000
a000: 1000BASE-X Timer Mode Select Register 0x2000
a001: 1000BASE-X Interrupt Enable Register 0x0600
Speed Changed 0
Duplex Changed 0
Page Received 0
Autoneg Complete 0
Link Up->Link Down 1
Link Down->Link Up 1
Symbol Error 0
False Carrier 0
a002: 1000BASE-X Interrupt Status Register 0x0000
Speed Changed 0
Duplex Changed 0
Page Received 0
Autoneg Complete 0
Link Up->Link Down 0
Link Down->Link Up 0
Symbol Error 0
False Carrier 0
a003: 1000BASE-X PHY Specific Status 0xac2c
Speed 1000 Mbps
Duplex Full
Page Received 0
Speed/Duplex Resolved 1
Link Up
Sync 1
Energy Detect 0
Transmit Pause 0
Receive Pause 0
1000: 10GBASE-X4 PCS Control 1 0x2040
1001: 10GBASE-X4 PCS Status 1 0x0082
1002: PCS Device Identifier 1 0x0141
1003: PCS Device Identifier 2 0x0c00
1004: PCS Speed Ability 0x0001
1005: PCS Devices In Package 1 0x009a
1006: PCS Devices In Package 2 0x4000
1007: Reserved 0x0001
1008: 10GBASE-X4 PCS Status 2 0x8402
100e: PCS Package Identifier 1 0x0141
100f: PCS Package Identifier 2 0x0c00
1014: PCS EEE Capability Register 0x0000
1018: 10GBase-X Lane Status 0x0c01
1019: 10GBase-X Test Control 0x0000
9000: 10GBase-X Control 0x0001
9001: 10GBase-X Interrupt Enable 1 0x0000
9002: 10GBase-X Interrupt Enable 2 0x0000
9003: 10GBase-X Interrupt Status 1 0x0000
9004: 10GBase-X Interrupt Status 2 0x00e1
9006: 10GBase-X Real Time Status 0x0011
9010: 10GBase-X Random Sequence Control 0x0000
9011: 10GBase-X Jitter Packet Transmit Count 0x0000
9012: 10GBase-X Jitter Packet Transmit Count 0x0000
9013: 10GBase-X Jitter Packet Received Count 0x0000
9014: 10GBase-X Jitter Packet Received Count 0x0000
9015: 10GBase-X Jitter Packet Error Counter 0x0000
9016: 10GBase-X Jitter Packet Error Counter 0x0000
1020: 10GBASE-R PCS Status 1 0x0000
1021: 10GBASE-R PCS Status 2 0x0000
1022: 10GBASE-R PCS Test Pattern Seed A 0 0x0000
1023: 10GBASE-R PCS Test Pattern Seed A 1 0x0000
1024: 10GBASE-R PCS Test Pattern Seed A 2 0x0000
1025: 10GBASE-R PCS Test Pattern Seed A 3 0x0000
1026: 10GBASE-R PCS Test Pattern Seed B 0 0x0000
1027: 10GBASE-R PCS Test Pattern Seed B 1 0x0000
1028: 10GBASE-R PCS Test Pattern Seed B 2 0x0000
1029: 10GBASE-R PCS Test Pattern Seed B 3 0x0000
102a: 10GBASE-R PCS Test Pattern Control 0x0000
102b: 10GBASE-R PCS Test Error Counter 0x0000
root@(none):~$ ethtool -S sff2
NIC statistics:
tx_packets: 3
tx_bytes: 126
rx_packets: 0
rx_bytes: 0
in_good_octets: 0
in_bad_octets: 0
in_unicast: 0
in_broadcasts: 0
in_multicasts: 0
in_pause: 0
in_undersize: 0
in_fragments: 0
in_oversize: 0
in_jabber: 0
in_rx_error: 0
in_fcs_error: 0
out_octets: 192
out_unicast: 0
out_broadcasts: 3
out_multicasts: 0
out_pause: 0
excessive: 0
collisions: 0
deferred: 0
single: 0
multiple: 0
out_fcs_error: 0
late: 0
hist_64bytes: 3
hist_65_127bytes: 0
hist_128_255bytes: 0
hist_256_511bytes: 0
hist_512_1023bytes: 0
hist_1024_max_bytes: 0
in_discards: 0
in_filtered: 0
in_accepted: 0
in_bad_accepted: 0
in_good_avb_class_a: 0
in_good_avb_class_b: 0
in_bad_avb_class_a: 0
in_bad_avb_class_b: 0
tcam_counter_0: 0
tcam_counter_1: 0
tcam_counter_2: 0
tcam_counter_3: 0
in_da_unknown: 0
in_management: 0
out_queue_0: 3
out_queue_1: 0
out_queue_2: 0
out_queue_3: 0
out_queue_4: 0
out_queue_5: 0
out_queue_6: 0
out_queue_7: 0
out_cut_through: 0
out_octets_a: 0
out_octets_b: 0
out_management: 3
serdes_rx_pkts: 6
serdes_rx_bytes: 384
serdes_rx_pkts_error: 0
atu_member_violation: 0
atu_miss_violation: 0
atu_full_violation: 0
vtu_member_violation: 0
vtu_miss_violation: 0
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2020-07-18 14:27 bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c Chris Healy
@ 2020-07-18 14:42 ` Marek Behun
2020-07-18 14:49 ` Chris Healy
0 siblings, 1 reply; 11+ messages in thread
From: Marek Behun @ 2020-07-18 14:42 UTC (permalink / raw)
To: Chris Healy; +Cc: netdev
Hmm, nothing sticks out in the register dump.
I encountered a similar problem 2 years ago on Topaz SERDES port when
the cmode was set to 2500BASE-X but the speed register was set to speed
incompatible with 2500BASE-X (I don't remember what, though). This
issue was solved by a patch I sent to netdev.
Are you sure that your board isn't broken? Maybe the SerDes traces on
RX path are damaged...
Marek
On Sat, 18 Jul 2020 07:27:33 -0700
Chris Healy <cphealy@gmail.com> wrote:
> I've been trying to get the fiber interface of the vf610-zii-dev-rev-c
> board working with net-next to no avail. This platform utilizes a
> Cotsworks SFF attached to port 9 of a Marvell 88E6390X.
>
> I have fiber link up on port 9 and am able to send packets from the
> management CPU of the switch through the switch and out port 9 through
> the fiber interface to a fiber link partner successfully. I'm also
> able to send packets from that fiber link partner back (to the point
> of the switches SERDES) and am seeing the fiber ports SERDES RX
> counters increment for each packet transmitted by the link partner.
> The switches port 9 MAC is not showing any RX counters incrementing
> though and I do not receive the frames at the management CPU.
>
> Because the SERDES RX counters are incrementing while the MAC RX
> counters are not incrementing, it seems to me that the issue is
> between the MAC and SERDES. This is odd to me given that TX works
> fine.
>
> In support of debugging this issue, I've applied an ethtool patch
> which allows decoding the 88E6390X SERDES registers from userspace.
> Looking at the register dump, nothing obvious sticks out to me though.
>
> I'm not sure what the right next steps are and would appreciate any
> theories on what to try/test to root cause this issue.
>
> Below is an ethtool register dump and an ethtool statistics dump from
> when the link is up and I have done some attempts at pinging a remote
> host over fiber.
>
> root@(none):~$ ethtool -d sff2
> 88E6390X Switch Port Registers
> ------------------------------
> 00: Port Status 0xce49
> Transmit Pause Enable bit 1
> Receive Pause Enable bit 1
> 802.3 PHY Detected 0
> Link Status Up
> Duplex Full
> Speed 1000 Mbps
> Duplex Fixed 0
> EEE Enabled 1
> Transmitter Paused 0
> Flow Control 0
> Config Mode 0x9
> 01: Physical Control 0x203e
> RGMII Receive Timing Control Default
> RGMII Transmit Timing Control Default
> Force Speed 1
> Alternate Speed Mode Normal
> MII PHY Mode MAC
> EEE force value 0
> Force EEE 0
> Link's Forced value Up
> Force Link 1
> Duplex's Forced value Full
> Force Duplex 1
> Force Speed 1000 Mbps
> 02: Flow Control 0x0100
> 03: Switch Identifier 0x0a11
> 04: Port Control 0x0433
> Source Address Filtering controls Disabled
> Egress Mode Unmodified
> Ingress & Egress Header Mode 0
> IGMP and MLD Snooping 1
> Frame Mode Normal
> VLAN Tunnel 0
> TagIfBoth 0
> Initial Priority assignment Tag & IP Priority
> Egress Flooding mode No unknown DA
> Port State Forwarding
> 05: Port Control 1 0x0000
> Message Port 0
> LAG Port 0
> VTU Page 0
> LAG ID 0
> FID[11:4] 0x000
> 06: Port Base VLAN Map (Header) 0x0400
> FID[3:0] 0x000
> Force Mapping 0
> VLANTable 10
> 07: Default VLAN ID & Priority 0x0000
> Default Priority 0x0
> Force to use Default VID 0
> Default VLAN Identifier 0
> 08: Port Control 2 0x0080
> Force good FCS in the frame 0
> Allow bad FCS 0
> Jumbo Mode 1522
> 802.1QMode Disabled
> Discard Tagged Frames 0
> Discard Untagged Frames 0
> Map using DA hits 1
> ARP Mirror enable 0
> Egress Monitor Source Port 0
> Ingress Monitor Source Port 0
> Allow VID of Zero 0
> Default Queue Priority 0x0
> 09: Egress Rate Control 0x0001
> 10: Egress Rate Control 2 0x0000
> 11: Port Association Vector 0x0200
> 12: Port ATU Control 0x0000
> 13: Override 0x0000
> 14: Policy Control 0x0000
> 15: Port Ether Type 0x9100
> 16: Reserved 0x0000
> 17: Reserved 0x0000
> 18: Reserved 0x0000
> 19: Reserved 0x0000
> 20: Reserved 0x0000
> 21: Reserved 0x0000
> 22: LED Control 0x0033
> 23: IP Priority Mapping Table 0x0000
> 24: IEEE Priority Mapping Table 0x3e07
> 25: Port Control 3 0x0000
> 26: Reserved 0x0000
> 27: Queue Counters 0x8000
> 28: Queue Control 0x0000
> 29: Reserved 0x0000
> 30: Cut Through Control 0x0000
> 31: Debug Counters 0x0000
>
> 88E6390X Switch Port SERDES Registers
> -------------------------------------
> f000: Global Clock Configuration 1 0x0000
> f001: Global Clock Configuration 2 0x0002
> f002: Port Operational Configuration 0x0003
> f00a: FIFO and CRC Int Enable 0x0000
> f00b: FIFO and CRC Int Status 0x0000
> f00c: PPM FIFO Control 1 0x0000
> f00d: PPM FIFO Control 2 0x0000
> f00e: PPM FIFO Status 0x0000
> f010: Packet Generation Control 1 0x0501
> f011: Packet Generation Control 2 0x0000
> f012: Initial Payload 0-1/Packet Generation 0x0000
> f013: Initial Payload 2-3/Packet Generation 0x0000
> f016: Packet Generation Length 0x0000
> f017: Packet Generation Burst Sequence 0x0000
> f018: Packet Generation IPG 0x0002
> f019: Packet Gen_Chkr Clock Control 0x0000
> f01a: Transmit Powerdown Delay 0x0032
> f01b: Transmit Packet Counter [15:0] 0x0000
> f01c: Transmit Packet Counter [31:16] 0x0000
> f01d: Transmit Packet Counter [47:32] 0x0000
> f01e: Transmit Byte Counter [15:0] 0x0000
> f01f: Transmit Byte Counter [31:16] 0x0000
> f020: Transmit Byte Counter [47:32] 0x0000
> f021: Receive Packet Counter [15:0] 0x0000
> f022: Receive Packet Counter [31:16] 0x0000
> f023: Receive Packet Counter [47:32] 0x0000
> f024: Receive Byte Count [15:0] 0x0000
> f025: Receive Byte Count [31:16] 0x0000
> f026: Receive Byte Count [47:32] 0x0000
> f027: Receive Packet Error Count [15:0] 0x0000
> f028: Receive Packet Error Count [31:16] 0x0000
> f029: Receive Packet Error Count [47:32] 0x0000
> f030: PRBS Control 0x0200
> f031: PRBS Symbol Tx Counter [15:0] 0x0000
> f032: PRBS Symbol Tx Counter [31:16] 0x0000
> f033: PRBS Symbol Tx Counter [47:32] 0x0000
> f034: PRBS Symbol Rx Counter [15:0] 0x0000
> f035: PRBS Symbol Rx Counter [31:16] 0x0000
> f036: PRBS Symbol RX Counter [47:32] 0x0000
> f037: PRBS Error Count [15:0] 0x0000
> f038: PRBS Error Count [31:16] 0x0000
> f039: PRBS Error Count [47:32] 0x0000
> 2000: 1000BASE-X/SGMII Control Register 0x1140
> Reset 0
> Loopback 0
> SGMII Speed 1000 Mbps
> Autoneg Enable 1
> Power down 0
> Isolate 0
> Restart Autonet 0
> Duplex Full
> 2001: 1000BASE-X/SGMII Status Register 0x016d
> Autoneg Complete 1
> Remote Fault 0
> Link Status Up
> 2002: PHY Identifier 0x0141
> 2003: PHY Identifier 0x0c00
> 2004: SGMII (Media side) Auto-Negotiation Ad 0x00a0
> Link Status Down
> Duplex Half
> SGMII Speed 10 Mbps
> Transmit Pause 0
> Receive Pause 0
> Fibre/Copper Fibre
> EEE mode 0
> Clock stopped during LPI 1
> 2005: SGMII (Media side) Link Partner Abilit 0x40a0
> Acknowledge 1
> 2006: 1000BASE-X Auto-Negotiation Expansion 0x0007
> 2007: 1000BASE-X Next Page Transmit Register 0x2801
> 2008: 1000BASE-X Link Partner Next Page Regi 0x0000
> 200f: Extended Status Register 0x8000
> a000: 1000BASE-X Timer Mode Select Register 0x2000
> a001: 1000BASE-X Interrupt Enable Register 0x0600
> Speed Changed 0
> Duplex Changed 0
> Page Received 0
> Autoneg Complete 0
> Link Up->Link Down 1
> Link Down->Link Up 1
> Symbol Error 0
> False Carrier 0
> a002: 1000BASE-X Interrupt Status Register 0x0000
> Speed Changed 0
> Duplex Changed 0
> Page Received 0
> Autoneg Complete 0
> Link Up->Link Down 0
> Link Down->Link Up 0
> Symbol Error 0
> False Carrier 0
> a003: 1000BASE-X PHY Specific Status 0xac2c
> Speed 1000 Mbps
> Duplex Full
> Page Received 0
> Speed/Duplex Resolved 1
> Link Up
> Sync 1
> Energy Detect 0
> Transmit Pause 0
> Receive Pause 0
> 1000: 10GBASE-X4 PCS Control 1 0x2040
> 1001: 10GBASE-X4 PCS Status 1 0x0082
> 1002: PCS Device Identifier 1 0x0141
> 1003: PCS Device Identifier 2 0x0c00
> 1004: PCS Speed Ability 0x0001
> 1005: PCS Devices In Package 1 0x009a
> 1006: PCS Devices In Package 2 0x4000
> 1007: Reserved 0x0001
> 1008: 10GBASE-X4 PCS Status 2 0x8402
> 100e: PCS Package Identifier 1 0x0141
> 100f: PCS Package Identifier 2 0x0c00
> 1014: PCS EEE Capability Register 0x0000
> 1018: 10GBase-X Lane Status 0x0c01
> 1019: 10GBase-X Test Control 0x0000
> 9000: 10GBase-X Control 0x0001
> 9001: 10GBase-X Interrupt Enable 1 0x0000
> 9002: 10GBase-X Interrupt Enable 2 0x0000
> 9003: 10GBase-X Interrupt Status 1 0x0000
> 9004: 10GBase-X Interrupt Status 2 0x00e1
> 9006: 10GBase-X Real Time Status 0x0011
> 9010: 10GBase-X Random Sequence Control 0x0000
> 9011: 10GBase-X Jitter Packet Transmit Count 0x0000
> 9012: 10GBase-X Jitter Packet Transmit Count 0x0000
> 9013: 10GBase-X Jitter Packet Received Count 0x0000
> 9014: 10GBase-X Jitter Packet Received Count 0x0000
> 9015: 10GBase-X Jitter Packet Error Counter 0x0000
> 9016: 10GBase-X Jitter Packet Error Counter 0x0000
> 1020: 10GBASE-R PCS Status 1 0x0000
> 1021: 10GBASE-R PCS Status 2 0x0000
> 1022: 10GBASE-R PCS Test Pattern Seed A 0 0x0000
> 1023: 10GBASE-R PCS Test Pattern Seed A 1 0x0000
> 1024: 10GBASE-R PCS Test Pattern Seed A 2 0x0000
> 1025: 10GBASE-R PCS Test Pattern Seed A 3 0x0000
> 1026: 10GBASE-R PCS Test Pattern Seed B 0 0x0000
> 1027: 10GBASE-R PCS Test Pattern Seed B 1 0x0000
> 1028: 10GBASE-R PCS Test Pattern Seed B 2 0x0000
> 1029: 10GBASE-R PCS Test Pattern Seed B 3 0x0000
> 102a: 10GBASE-R PCS Test Pattern Control 0x0000
> 102b: 10GBASE-R PCS Test Error Counter 0x0000
>
> root@(none):~$ ethtool -S sff2
> NIC statistics:
> tx_packets: 3
> tx_bytes: 126
> rx_packets: 0
> rx_bytes: 0
> in_good_octets: 0
> in_bad_octets: 0
> in_unicast: 0
> in_broadcasts: 0
> in_multicasts: 0
> in_pause: 0
> in_undersize: 0
> in_fragments: 0
> in_oversize: 0
> in_jabber: 0
> in_rx_error: 0
> in_fcs_error: 0
> out_octets: 192
> out_unicast: 0
> out_broadcasts: 3
> out_multicasts: 0
> out_pause: 0
> excessive: 0
> collisions: 0
> deferred: 0
> single: 0
> multiple: 0
> out_fcs_error: 0
> late: 0
> hist_64bytes: 3
> hist_65_127bytes: 0
> hist_128_255bytes: 0
> hist_256_511bytes: 0
> hist_512_1023bytes: 0
> hist_1024_max_bytes: 0
> in_discards: 0
> in_filtered: 0
> in_accepted: 0
> in_bad_accepted: 0
> in_good_avb_class_a: 0
> in_good_avb_class_b: 0
> in_bad_avb_class_a: 0
> in_bad_avb_class_b: 0
> tcam_counter_0: 0
> tcam_counter_1: 0
> tcam_counter_2: 0
> tcam_counter_3: 0
> in_da_unknown: 0
> in_management: 0
> out_queue_0: 3
> out_queue_1: 0
> out_queue_2: 0
> out_queue_3: 0
> out_queue_4: 0
> out_queue_5: 0
> out_queue_6: 0
> out_queue_7: 0
> out_cut_through: 0
> out_octets_a: 0
> out_octets_b: 0
> out_management: 3
> serdes_rx_pkts: 6
> serdes_rx_bytes: 384
> serdes_rx_pkts_error: 0
> atu_member_violation: 0
> atu_miss_violation: 0
> atu_full_violation: 0
> vtu_member_violation: 0
> vtu_miss_violation: 0
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2020-07-18 14:42 ` Marek Behun
@ 2020-07-18 14:49 ` Chris Healy
2020-07-18 15:05 ` Andrew Lunn
0 siblings, 1 reply; 11+ messages in thread
From: Chris Healy @ 2020-07-18 14:49 UTC (permalink / raw)
To: Marek Behun; +Cc: netdev
On Sat, Jul 18, 2020 at 7:42 AM Marek Behun <marek.behun@nic.cz> wrote:
>
> Hmm, nothing sticks out in the register dump.
>
> I encountered a similar problem 2 years ago on Topaz SERDES port when
> the cmode was set to 2500BASE-X but the speed register was set to speed
> incompatible with 2500BASE-X (I don't remember what, though). This
> issue was solved by a patch I sent to netdev.
>
> Are you sure that your board isn't broken? Maybe the SerDes traces on
> RX path are damaged...
In my case, both the SERDES and the MAC are inside the switch so I
don't think it's likely that the SERDES traces are broken in there.
If you are referring to the traces between the SERDES and the fiber
module, that doesn't feel likely either as the SERDES appears to be
reporting successfully received frames:
From "ethtool -S" after sending 6 packets to the unit:
serdes_rx_pkts: 6
serdes_rx_bytes: 384
serdes_rx_pkts_error: 0
If the traces were broken between the fiber module and the SERDES, I
should not see these counters incrementing.
>
> Marek
>
> On Sat, 18 Jul 2020 07:27:33 -0700
> Chris Healy <cphealy@gmail.com> wrote:
>
> > I've been trying to get the fiber interface of the vf610-zii-dev-rev-c
> > board working with net-next to no avail. This platform utilizes a
> > Cotsworks SFF attached to port 9 of a Marvell 88E6390X.
> >
> > I have fiber link up on port 9 and am able to send packets from the
> > management CPU of the switch through the switch and out port 9 through
> > the fiber interface to a fiber link partner successfully. I'm also
> > able to send packets from that fiber link partner back (to the point
> > of the switches SERDES) and am seeing the fiber ports SERDES RX
> > counters increment for each packet transmitted by the link partner.
> > The switches port 9 MAC is not showing any RX counters incrementing
> > though and I do not receive the frames at the management CPU.
> >
> > Because the SERDES RX counters are incrementing while the MAC RX
> > counters are not incrementing, it seems to me that the issue is
> > between the MAC and SERDES. This is odd to me given that TX works
> > fine.
> >
> > In support of debugging this issue, I've applied an ethtool patch
> > which allows decoding the 88E6390X SERDES registers from userspace.
> > Looking at the register dump, nothing obvious sticks out to me though.
> >
> > I'm not sure what the right next steps are and would appreciate any
> > theories on what to try/test to root cause this issue.
> >
> > Below is an ethtool register dump and an ethtool statistics dump from
> > when the link is up and I have done some attempts at pinging a remote
> > host over fiber.
> >
> > root@(none):~$ ethtool -d sff2
> > 88E6390X Switch Port Registers
> > ------------------------------
> > 00: Port Status 0xce49
> > Transmit Pause Enable bit 1
> > Receive Pause Enable bit 1
> > 802.3 PHY Detected 0
> > Link Status Up
> > Duplex Full
> > Speed 1000 Mbps
> > Duplex Fixed 0
> > EEE Enabled 1
> > Transmitter Paused 0
> > Flow Control 0
> > Config Mode 0x9
> > 01: Physical Control 0x203e
> > RGMII Receive Timing Control Default
> > RGMII Transmit Timing Control Default
> > Force Speed 1
> > Alternate Speed Mode Normal
> > MII PHY Mode MAC
> > EEE force value 0
> > Force EEE 0
> > Link's Forced value Up
> > Force Link 1
> > Duplex's Forced value Full
> > Force Duplex 1
> > Force Speed 1000 Mbps
> > 02: Flow Control 0x0100
> > 03: Switch Identifier 0x0a11
> > 04: Port Control 0x0433
> > Source Address Filtering controls Disabled
> > Egress Mode Unmodified
> > Ingress & Egress Header Mode 0
> > IGMP and MLD Snooping 1
> > Frame Mode Normal
> > VLAN Tunnel 0
> > TagIfBoth 0
> > Initial Priority assignment Tag & IP Priority
> > Egress Flooding mode No unknown DA
> > Port State Forwarding
> > 05: Port Control 1 0x0000
> > Message Port 0
> > LAG Port 0
> > VTU Page 0
> > LAG ID 0
> > FID[11:4] 0x000
> > 06: Port Base VLAN Map (Header) 0x0400
> > FID[3:0] 0x000
> > Force Mapping 0
> > VLANTable 10
> > 07: Default VLAN ID & Priority 0x0000
> > Default Priority 0x0
> > Force to use Default VID 0
> > Default VLAN Identifier 0
> > 08: Port Control 2 0x0080
> > Force good FCS in the frame 0
> > Allow bad FCS 0
> > Jumbo Mode 1522
> > 802.1QMode Disabled
> > Discard Tagged Frames 0
> > Discard Untagged Frames 0
> > Map using DA hits 1
> > ARP Mirror enable 0
> > Egress Monitor Source Port 0
> > Ingress Monitor Source Port 0
> > Allow VID of Zero 0
> > Default Queue Priority 0x0
> > 09: Egress Rate Control 0x0001
> > 10: Egress Rate Control 2 0x0000
> > 11: Port Association Vector 0x0200
> > 12: Port ATU Control 0x0000
> > 13: Override 0x0000
> > 14: Policy Control 0x0000
> > 15: Port Ether Type 0x9100
> > 16: Reserved 0x0000
> > 17: Reserved 0x0000
> > 18: Reserved 0x0000
> > 19: Reserved 0x0000
> > 20: Reserved 0x0000
> > 21: Reserved 0x0000
> > 22: LED Control 0x0033
> > 23: IP Priority Mapping Table 0x0000
> > 24: IEEE Priority Mapping Table 0x3e07
> > 25: Port Control 3 0x0000
> > 26: Reserved 0x0000
> > 27: Queue Counters 0x8000
> > 28: Queue Control 0x0000
> > 29: Reserved 0x0000
> > 30: Cut Through Control 0x0000
> > 31: Debug Counters 0x0000
> >
> > 88E6390X Switch Port SERDES Registers
> > -------------------------------------
> > f000: Global Clock Configuration 1 0x0000
> > f001: Global Clock Configuration 2 0x0002
> > f002: Port Operational Configuration 0x0003
> > f00a: FIFO and CRC Int Enable 0x0000
> > f00b: FIFO and CRC Int Status 0x0000
> > f00c: PPM FIFO Control 1 0x0000
> > f00d: PPM FIFO Control 2 0x0000
> > f00e: PPM FIFO Status 0x0000
> > f010: Packet Generation Control 1 0x0501
> > f011: Packet Generation Control 2 0x0000
> > f012: Initial Payload 0-1/Packet Generation 0x0000
> > f013: Initial Payload 2-3/Packet Generation 0x0000
> > f016: Packet Generation Length 0x0000
> > f017: Packet Generation Burst Sequence 0x0000
> > f018: Packet Generation IPG 0x0002
> > f019: Packet Gen_Chkr Clock Control 0x0000
> > f01a: Transmit Powerdown Delay 0x0032
> > f01b: Transmit Packet Counter [15:0] 0x0000
> > f01c: Transmit Packet Counter [31:16] 0x0000
> > f01d: Transmit Packet Counter [47:32] 0x0000
> > f01e: Transmit Byte Counter [15:0] 0x0000
> > f01f: Transmit Byte Counter [31:16] 0x0000
> > f020: Transmit Byte Counter [47:32] 0x0000
> > f021: Receive Packet Counter [15:0] 0x0000
> > f022: Receive Packet Counter [31:16] 0x0000
> > f023: Receive Packet Counter [47:32] 0x0000
> > f024: Receive Byte Count [15:0] 0x0000
> > f025: Receive Byte Count [31:16] 0x0000
> > f026: Receive Byte Count [47:32] 0x0000
> > f027: Receive Packet Error Count [15:0] 0x0000
> > f028: Receive Packet Error Count [31:16] 0x0000
> > f029: Receive Packet Error Count [47:32] 0x0000
> > f030: PRBS Control 0x0200
> > f031: PRBS Symbol Tx Counter [15:0] 0x0000
> > f032: PRBS Symbol Tx Counter [31:16] 0x0000
> > f033: PRBS Symbol Tx Counter [47:32] 0x0000
> > f034: PRBS Symbol Rx Counter [15:0] 0x0000
> > f035: PRBS Symbol Rx Counter [31:16] 0x0000
> > f036: PRBS Symbol RX Counter [47:32] 0x0000
> > f037: PRBS Error Count [15:0] 0x0000
> > f038: PRBS Error Count [31:16] 0x0000
> > f039: PRBS Error Count [47:32] 0x0000
> > 2000: 1000BASE-X/SGMII Control Register 0x1140
> > Reset 0
> > Loopback 0
> > SGMII Speed 1000 Mbps
> > Autoneg Enable 1
> > Power down 0
> > Isolate 0
> > Restart Autonet 0
> > Duplex Full
> > 2001: 1000BASE-X/SGMII Status Register 0x016d
> > Autoneg Complete 1
> > Remote Fault 0
> > Link Status Up
> > 2002: PHY Identifier 0x0141
> > 2003: PHY Identifier 0x0c00
> > 2004: SGMII (Media side) Auto-Negotiation Ad 0x00a0
> > Link Status Down
> > Duplex Half
> > SGMII Speed 10 Mbps
> > Transmit Pause 0
> > Receive Pause 0
> > Fibre/Copper Fibre
> > EEE mode 0
> > Clock stopped during LPI 1
> > 2005: SGMII (Media side) Link Partner Abilit 0x40a0
> > Acknowledge 1
> > 2006: 1000BASE-X Auto-Negotiation Expansion 0x0007
> > 2007: 1000BASE-X Next Page Transmit Register 0x2801
> > 2008: 1000BASE-X Link Partner Next Page Regi 0x0000
> > 200f: Extended Status Register 0x8000
> > a000: 1000BASE-X Timer Mode Select Register 0x2000
> > a001: 1000BASE-X Interrupt Enable Register 0x0600
> > Speed Changed 0
> > Duplex Changed 0
> > Page Received 0
> > Autoneg Complete 0
> > Link Up->Link Down 1
> > Link Down->Link Up 1
> > Symbol Error 0
> > False Carrier 0
> > a002: 1000BASE-X Interrupt Status Register 0x0000
> > Speed Changed 0
> > Duplex Changed 0
> > Page Received 0
> > Autoneg Complete 0
> > Link Up->Link Down 0
> > Link Down->Link Up 0
> > Symbol Error 0
> > False Carrier 0
> > a003: 1000BASE-X PHY Specific Status 0xac2c
> > Speed 1000 Mbps
> > Duplex Full
> > Page Received 0
> > Speed/Duplex Resolved 1
> > Link Up
> > Sync 1
> > Energy Detect 0
> > Transmit Pause 0
> > Receive Pause 0
> > 1000: 10GBASE-X4 PCS Control 1 0x2040
> > 1001: 10GBASE-X4 PCS Status 1 0x0082
> > 1002: PCS Device Identifier 1 0x0141
> > 1003: PCS Device Identifier 2 0x0c00
> > 1004: PCS Speed Ability 0x0001
> > 1005: PCS Devices In Package 1 0x009a
> > 1006: PCS Devices In Package 2 0x4000
> > 1007: Reserved 0x0001
> > 1008: 10GBASE-X4 PCS Status 2 0x8402
> > 100e: PCS Package Identifier 1 0x0141
> > 100f: PCS Package Identifier 2 0x0c00
> > 1014: PCS EEE Capability Register 0x0000
> > 1018: 10GBase-X Lane Status 0x0c01
> > 1019: 10GBase-X Test Control 0x0000
> > 9000: 10GBase-X Control 0x0001
> > 9001: 10GBase-X Interrupt Enable 1 0x0000
> > 9002: 10GBase-X Interrupt Enable 2 0x0000
> > 9003: 10GBase-X Interrupt Status 1 0x0000
> > 9004: 10GBase-X Interrupt Status 2 0x00e1
> > 9006: 10GBase-X Real Time Status 0x0011
> > 9010: 10GBase-X Random Sequence Control 0x0000
> > 9011: 10GBase-X Jitter Packet Transmit Count 0x0000
> > 9012: 10GBase-X Jitter Packet Transmit Count 0x0000
> > 9013: 10GBase-X Jitter Packet Received Count 0x0000
> > 9014: 10GBase-X Jitter Packet Received Count 0x0000
> > 9015: 10GBase-X Jitter Packet Error Counter 0x0000
> > 9016: 10GBase-X Jitter Packet Error Counter 0x0000
> > 1020: 10GBASE-R PCS Status 1 0x0000
> > 1021: 10GBASE-R PCS Status 2 0x0000
> > 1022: 10GBASE-R PCS Test Pattern Seed A 0 0x0000
> > 1023: 10GBASE-R PCS Test Pattern Seed A 1 0x0000
> > 1024: 10GBASE-R PCS Test Pattern Seed A 2 0x0000
> > 1025: 10GBASE-R PCS Test Pattern Seed A 3 0x0000
> > 1026: 10GBASE-R PCS Test Pattern Seed B 0 0x0000
> > 1027: 10GBASE-R PCS Test Pattern Seed B 1 0x0000
> > 1028: 10GBASE-R PCS Test Pattern Seed B 2 0x0000
> > 1029: 10GBASE-R PCS Test Pattern Seed B 3 0x0000
> > 102a: 10GBASE-R PCS Test Pattern Control 0x0000
> > 102b: 10GBASE-R PCS Test Error Counter 0x0000
> >
> > root@(none):~$ ethtool -S sff2
> > NIC statistics:
> > tx_packets: 3
> > tx_bytes: 126
> > rx_packets: 0
> > rx_bytes: 0
> > in_good_octets: 0
> > in_bad_octets: 0
> > in_unicast: 0
> > in_broadcasts: 0
> > in_multicasts: 0
> > in_pause: 0
> > in_undersize: 0
> > in_fragments: 0
> > in_oversize: 0
> > in_jabber: 0
> > in_rx_error: 0
> > in_fcs_error: 0
> > out_octets: 192
> > out_unicast: 0
> > out_broadcasts: 3
> > out_multicasts: 0
> > out_pause: 0
> > excessive: 0
> > collisions: 0
> > deferred: 0
> > single: 0
> > multiple: 0
> > out_fcs_error: 0
> > late: 0
> > hist_64bytes: 3
> > hist_65_127bytes: 0
> > hist_128_255bytes: 0
> > hist_256_511bytes: 0
> > hist_512_1023bytes: 0
> > hist_1024_max_bytes: 0
> > in_discards: 0
> > in_filtered: 0
> > in_accepted: 0
> > in_bad_accepted: 0
> > in_good_avb_class_a: 0
> > in_good_avb_class_b: 0
> > in_bad_avb_class_a: 0
> > in_bad_avb_class_b: 0
> > tcam_counter_0: 0
> > tcam_counter_1: 0
> > tcam_counter_2: 0
> > tcam_counter_3: 0
> > in_da_unknown: 0
> > in_management: 0
> > out_queue_0: 3
> > out_queue_1: 0
> > out_queue_2: 0
> > out_queue_3: 0
> > out_queue_4: 0
> > out_queue_5: 0
> > out_queue_6: 0
> > out_queue_7: 0
> > out_cut_through: 0
> > out_octets_a: 0
> > out_octets_b: 0
> > out_management: 3
> > serdes_rx_pkts: 6
> > serdes_rx_bytes: 384
> > serdes_rx_pkts_error: 0
> > atu_member_violation: 0
> > atu_miss_violation: 0
> > atu_full_violation: 0
> > vtu_member_violation: 0
> > vtu_miss_violation: 0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2020-07-18 14:49 ` Chris Healy
@ 2020-07-18 15:05 ` Andrew Lunn
2020-07-18 15:22 ` Marek Behun
0 siblings, 1 reply; 11+ messages in thread
From: Andrew Lunn @ 2020-07-18 15:05 UTC (permalink / raw)
To: Chris Healy; +Cc: Marek Behun, netdev
On Sat, Jul 18, 2020 at 07:49:26AM -0700, Chris Healy wrote:
> On Sat, Jul 18, 2020 at 7:42 AM Marek Behun <marek.behun@nic.cz> wrote:
> >
> > Hmm, nothing sticks out in the register dump.
> >
> > I encountered a similar problem 2 years ago on Topaz SERDES port when
> > the cmode was set to 2500BASE-X but the speed register was set to speed
> > incompatible with 2500BASE-X (I don't remember what, though). This
> > issue was solved by a patch I sent to netdev.
> >
> > Are you sure that your board isn't broken? Maybe the SerDes traces on
> > RX path are damaged...
>
> In my case, both the SERDES and the MAC are inside the switch so I
> don't think it's likely that the SERDES traces are broken in there.
> If you are referring to the traces between the SERDES and the fiber
> module, that doesn't feel likely either as the SERDES appears to be
> reporting successfully received frames:
>
> >From "ethtool -S" after sending 6 packets to the unit:
> serdes_rx_pkts: 6
> serdes_rx_bytes: 384
> serdes_rx_pkts_error: 0
>
> If the traces were broken between the fiber module and the SERDES, I
> should not see these counters incrementing.
Plus it is reproducible on multiple boards, of different designs.
This is somehow specific to the 6390X ports 9 and 10.
Andrew
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2020-07-18 15:05 ` Andrew Lunn
@ 2020-07-18 15:22 ` Marek Behun
2020-07-19 21:43 ` Chris Healy
0 siblings, 1 reply; 11+ messages in thread
From: Marek Behun @ 2020-07-18 15:22 UTC (permalink / raw)
To: Andrew Lunn; +Cc: Chris Healy, netdev
On Sat, 18 Jul 2020 17:05:14 +0200
Andrew Lunn <andrew@lunn.ch> wrote:
> > If the traces were broken between the fiber module and the SERDES, I
> > should not see these counters incrementing.
>
> Plus it is reproducible on multiple boards, of different designs.
>
> This is somehow specific to the 6390X ports 9 and 10.
>
> Andrew
Hmm.
What about the errata setup?
It says:
/* The 6390 copper ports have an errata which require poking magic
* values into undocumented hidden registers and then performing a
* software reset.
*/
But then the port_hidden_write function is called for every port in the
function mv88e6390_setup_errata, not just for copper ports. Maybe Chris
should try to not write this hidden register for SerDes ports.
Marek
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2020-07-18 15:22 ` Marek Behun
@ 2020-07-19 21:43 ` Chris Healy
2020-07-19 21:52 ` Marek Behun
2021-01-18 17:31 ` Tobias Waldekranz
0 siblings, 2 replies; 11+ messages in thread
From: Chris Healy @ 2020-07-19 21:43 UTC (permalink / raw)
To: Marek Behun; +Cc: Andrew Lunn, netdev
On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote:
>
> On Sat, 18 Jul 2020 17:05:14 +0200
> Andrew Lunn <andrew@lunn.ch> wrote:
>
> > > If the traces were broken between the fiber module and the SERDES, I
> > > should not see these counters incrementing.
> >
> > Plus it is reproducible on multiple boards, of different designs.
> >
> > This is somehow specific to the 6390X ports 9 and 10.
> >
> > Andrew
>
> Hmm.
>
> What about the errata setup?
> It says:
> /* The 6390 copper ports have an errata which require poking magic
> * values into undocumented hidden registers and then performing a
> * software reset.
> */
> But then the port_hidden_write function is called for every port in the
> function mv88e6390_setup_errata, not just for copper ports. Maybe Chris
> should try to not write this hidden register for SerDes ports.
I just disabled the mv88e6390_setup_errata all together and this did
not result in any different behaviour on this broken fiber port.
>
> Marek
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2020-07-19 21:43 ` Chris Healy
@ 2020-07-19 21:52 ` Marek Behun
2021-01-18 17:31 ` Tobias Waldekranz
1 sibling, 0 replies; 11+ messages in thread
From: Marek Behun @ 2020-07-19 21:52 UTC (permalink / raw)
To: Chris Healy; +Cc: Andrew Lunn, netdev
On Sun, 19 Jul 2020 14:43:49 -0700
Chris Healy <cphealy@gmail.com> wrote:
> > Hmm.
> >
> > What about the errata setup?
> > It says:
> > /* The 6390 copper ports have an errata which require poking magic
> > * values into undocumented hidden registers and then performing a
> > * software reset.
> > */
> > But then the port_hidden_write function is called for every port in the
> > function mv88e6390_setup_errata, not just for copper ports. Maybe Chris
> > should try to not write this hidden register for SerDes ports.
>
> I just disabled the mv88e6390_setup_errata all together and this did
> not result in any different behaviour on this broken fiber port.
:-( In that case I really have no idea what could be the problem.
Another thing you could try is resoldering resistors on the board so
that the switches configure themselves into NOCPU mode and the port you
are talking about configures itself into the cmode you need
(was it 1000base-x or sgmii?). Disable DSA, write yourself sysfs API
via which you can read/write switch registers by yourself. Then you can
chech if the problem on the RX path occurs. If no, read all the
registers and compare their values with the ones the mv88e6xxx driver
configures. If yes, then we know that the problem is there even if the
switch is NOCPU mode and you can inform Marvell about the bug.
Marek
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2020-07-19 21:43 ` Chris Healy
2020-07-19 21:52 ` Marek Behun
@ 2021-01-18 17:31 ` Tobias Waldekranz
2021-01-18 17:41 ` Andrew Lunn
1 sibling, 1 reply; 11+ messages in thread
From: Tobias Waldekranz @ 2021-01-18 17:31 UTC (permalink / raw)
To: Chris Healy, Marek Behun; +Cc: Andrew Lunn, netdev
On Sun, Jul 19, 2020 at 14:43, Chris Healy <cphealy@gmail.com> wrote:
> On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote:
>>
>> On Sat, 18 Jul 2020 17:05:14 +0200
>> Andrew Lunn <andrew@lunn.ch> wrote:
>>
>> > > If the traces were broken between the fiber module and the SERDES, I
>> > > should not see these counters incrementing.
>> >
>> > Plus it is reproducible on multiple boards, of different designs.
>> >
>> > This is somehow specific to the 6390X ports 9 and 10.
>> >
>> > Andrew
>>
>> Hmm.
>>
>> What about the errata setup?
>> It says:
>> /* The 6390 copper ports have an errata which require poking magic
>> * values into undocumented hidden registers and then performing a
>> * software reset.
>> */
>> But then the port_hidden_write function is called for every port in the
>> function mv88e6390_setup_errata, not just for copper ports. Maybe Chris
>> should try to not write this hidden register for SerDes ports.
>
> I just disabled the mv88e6390_setup_errata all together and this did
> not result in any different behaviour on this broken fiber port.
Hi Chris,
Did you manage to track this down?
I am seeing the exact same issue. I have tried both a 1000base-x SFP and
a copper 1000base-T and get the same result on both - transmit is fine
but rx only works up to the SERDES, no rx MAC counters are moving.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2021-01-18 17:31 ` Tobias Waldekranz
@ 2021-01-18 17:41 ` Andrew Lunn
2021-01-18 17:47 ` Tobias Waldekranz
0 siblings, 1 reply; 11+ messages in thread
From: Andrew Lunn @ 2021-01-18 17:41 UTC (permalink / raw)
To: Tobias Waldekranz; +Cc: Chris Healy, Marek Behun, netdev
On Mon, Jan 18, 2021 at 06:31:19PM +0100, Tobias Waldekranz wrote:
> On Sun, Jul 19, 2020 at 14:43, Chris Healy <cphealy@gmail.com> wrote:
> > On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote:
> >>
> >> On Sat, 18 Jul 2020 17:05:14 +0200
> >> Andrew Lunn <andrew@lunn.ch> wrote:
> >>
> >> > > If the traces were broken between the fiber module and the SERDES, I
> >> > > should not see these counters incrementing.
> >> >
> >> > Plus it is reproducible on multiple boards, of different designs.
> >> >
> >> > This is somehow specific to the 6390X ports 9 and 10.
> >> >
> >> > Andrew
> >>
> >> Hmm.
> >>
> >> What about the errata setup?
> >> It says:
> >> /* The 6390 copper ports have an errata which require poking magic
> >> * values into undocumented hidden registers and then performing a
> >> * software reset.
> >> */
> >> But then the port_hidden_write function is called for every port in the
> >> function mv88e6390_setup_errata, not just for copper ports. Maybe Chris
> >> should try to not write this hidden register for SerDes ports.
> >
> > I just disabled the mv88e6390_setup_errata all together and this did
> > not result in any different behaviour on this broken fiber port.
>
> Hi Chris,
>
> Did you manage to track this down?
>
> I am seeing the exact same issue. I have tried both a 1000base-x SFP and
> a copper 1000base-T and get the same result on both - transmit is fine
> but rx only works up to the SERDES, no rx MAC counters are moving.
Hi Tobias
We never tracked this down. I spent many hours bashing my head against
this. I could not bisect it, which did not help.
FYI: Chris has moved onto a new job, and is unlikely to be involved
with Marvell switches any more.
Andrew
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2021-01-18 17:41 ` Andrew Lunn
@ 2021-01-18 17:47 ` Tobias Waldekranz
2021-01-18 19:26 ` Andrew Lunn
0 siblings, 1 reply; 11+ messages in thread
From: Tobias Waldekranz @ 2021-01-18 17:47 UTC (permalink / raw)
To: Andrew Lunn; +Cc: Chris Healy, Marek Behun, netdev
On Mon, Jan 18, 2021 at 18:41, Andrew Lunn <andrew@lunn.ch> wrote:
> On Mon, Jan 18, 2021 at 06:31:19PM +0100, Tobias Waldekranz wrote:
>> On Sun, Jul 19, 2020 at 14:43, Chris Healy <cphealy@gmail.com> wrote:
>> > On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote:
>> >>
>> >> On Sat, 18 Jul 2020 17:05:14 +0200
>> >> Andrew Lunn <andrew@lunn.ch> wrote:
>> >>
>> >> > > If the traces were broken between the fiber module and the SERDES, I
>> >> > > should not see these counters incrementing.
>> >> >
>> >> > Plus it is reproducible on multiple boards, of different designs.
>> >> >
>> >> > This is somehow specific to the 6390X ports 9 and 10.
>> >> >
>> >> > Andrew
>> >>
>> >> Hmm.
>> >>
>> >> What about the errata setup?
>> >> It says:
>> >> /* The 6390 copper ports have an errata which require poking magic
>> >> * values into undocumented hidden registers and then performing a
>> >> * software reset.
>> >> */
>> >> But then the port_hidden_write function is called for every port in the
>> >> function mv88e6390_setup_errata, not just for copper ports. Maybe Chris
>> >> should try to not write this hidden register for SerDes ports.
>> >
>> > I just disabled the mv88e6390_setup_errata all together and this did
>> > not result in any different behaviour on this broken fiber port.
>>
>> Hi Chris,
>>
>> Did you manage to track this down?
>>
>> I am seeing the exact same issue. I have tried both a 1000base-x SFP and
>> a copper 1000base-T and get the same result on both - transmit is fine
>> but rx only works up to the SERDES, no rx MAC counters are moving.
>
> Hi Tobias
>
> We never tracked this down. I spent many hours bashing my head against
> this. I could not bisect it, which did not help.
Well that is disheartening :) "I could not bisect it", does that mean
that it did work at some point but your CPU platform was not supported
far enough back, or has it never worked?
> FYI: Chris has moved onto a new job, and is unlikely to be involved
> with Marvell switches any more.
Good to know, thanks.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c
2021-01-18 17:47 ` Tobias Waldekranz
@ 2021-01-18 19:26 ` Andrew Lunn
0 siblings, 0 replies; 11+ messages in thread
From: Andrew Lunn @ 2021-01-18 19:26 UTC (permalink / raw)
To: Tobias Waldekranz; +Cc: Chris Healy, Marek Behun, netdev
> >> I am seeing the exact same issue. I have tried both a 1000base-x SFP and
> >> a copper 1000base-T and get the same result on both - transmit is fine
> >> but rx only works up to the SERDES, no rx MAC counters are moving.
> >
> > Hi Tobias
> >
> > We never tracked this down. I spent many hours bashing my head against
> > this. I could not bisect it, which did not help.
>
> Well that is disheartening :) "I could not bisect it", does that mean
> that it did work at some point but your CPU platform was not supported
> far enough back, or has it never worked?
As far as i remember, Ports 9 and 10 did work once. I suspect it could
of been early on, before we had much support for the SERDES, it relied
on the strapping being correct, and the switch powering up in the
right mode.
We also do a dance with the cmode of ports 9 and 10, dropping them
down to slower speeds making the SERDES available for the other ports
and then changing the cmode again if the port is supposed to use a
higher speed and needs multiple SERDESes.
The board which had trouble and i was debugging on only has limited
support, not going back too far. I would probably need to reproduce
the issue on different hardware to have more scope for going backwards
and trying to find a setup where ports 9 or 10 did work, as a basis
for a bisect.
Andrew
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-01-18 19:46 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-18 14:27 bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c Chris Healy
2020-07-18 14:42 ` Marek Behun
2020-07-18 14:49 ` Chris Healy
2020-07-18 15:05 ` Andrew Lunn
2020-07-18 15:22 ` Marek Behun
2020-07-19 21:43 ` Chris Healy
2020-07-19 21:52 ` Marek Behun
2021-01-18 17:31 ` Tobias Waldekranz
2021-01-18 17:41 ` Andrew Lunn
2021-01-18 17:47 ` Tobias Waldekranz
2021-01-18 19:26 ` Andrew Lunn
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).