* bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c @ 2020-07-18 14:27 Chris Healy 2020-07-18 14:42 ` Marek Behun 0 siblings, 1 reply; 11+ messages in thread From: Chris Healy @ 2020-07-18 14:27 UTC (permalink / raw) To: netdev I've been trying to get the fiber interface of the vf610-zii-dev-rev-c board working with net-next to no avail. This platform utilizes a Cotsworks SFF attached to port 9 of a Marvell 88E6390X. I have fiber link up on port 9 and am able to send packets from the management CPU of the switch through the switch and out port 9 through the fiber interface to a fiber link partner successfully. I'm also able to send packets from that fiber link partner back (to the point of the switches SERDES) and am seeing the fiber ports SERDES RX counters increment for each packet transmitted by the link partner. The switches port 9 MAC is not showing any RX counters incrementing though and I do not receive the frames at the management CPU. Because the SERDES RX counters are incrementing while the MAC RX counters are not incrementing, it seems to me that the issue is between the MAC and SERDES. This is odd to me given that TX works fine. In support of debugging this issue, I've applied an ethtool patch which allows decoding the 88E6390X SERDES registers from userspace. Looking at the register dump, nothing obvious sticks out to me though. I'm not sure what the right next steps are and would appreciate any theories on what to try/test to root cause this issue. Below is an ethtool register dump and an ethtool statistics dump from when the link is up and I have done some attempts at pinging a remote host over fiber. root@(none):~$ ethtool -d sff2 88E6390X Switch Port Registers ------------------------------ 00: Port Status 0xce49 Transmit Pause Enable bit 1 Receive Pause Enable bit 1 802.3 PHY Detected 0 Link Status Up Duplex Full Speed 1000 Mbps Duplex Fixed 0 EEE Enabled 1 Transmitter Paused 0 Flow Control 0 Config Mode 0x9 01: Physical Control 0x203e RGMII Receive Timing Control Default RGMII Transmit Timing Control Default Force Speed 1 Alternate Speed Mode Normal MII PHY Mode MAC EEE force value 0 Force EEE 0 Link's Forced value Up Force Link 1 Duplex's Forced value Full Force Duplex 1 Force Speed 1000 Mbps 02: Flow Control 0x0100 03: Switch Identifier 0x0a11 04: Port Control 0x0433 Source Address Filtering controls Disabled Egress Mode Unmodified Ingress & Egress Header Mode 0 IGMP and MLD Snooping 1 Frame Mode Normal VLAN Tunnel 0 TagIfBoth 0 Initial Priority assignment Tag & IP Priority Egress Flooding mode No unknown DA Port State Forwarding 05: Port Control 1 0x0000 Message Port 0 LAG Port 0 VTU Page 0 LAG ID 0 FID[11:4] 0x000 06: Port Base VLAN Map (Header) 0x0400 FID[3:0] 0x000 Force Mapping 0 VLANTable 10 07: Default VLAN ID & Priority 0x0000 Default Priority 0x0 Force to use Default VID 0 Default VLAN Identifier 0 08: Port Control 2 0x0080 Force good FCS in the frame 0 Allow bad FCS 0 Jumbo Mode 1522 802.1QMode Disabled Discard Tagged Frames 0 Discard Untagged Frames 0 Map using DA hits 1 ARP Mirror enable 0 Egress Monitor Source Port 0 Ingress Monitor Source Port 0 Allow VID of Zero 0 Default Queue Priority 0x0 09: Egress Rate Control 0x0001 10: Egress Rate Control 2 0x0000 11: Port Association Vector 0x0200 12: Port ATU Control 0x0000 13: Override 0x0000 14: Policy Control 0x0000 15: Port Ether Type 0x9100 16: Reserved 0x0000 17: Reserved 0x0000 18: Reserved 0x0000 19: Reserved 0x0000 20: Reserved 0x0000 21: Reserved 0x0000 22: LED Control 0x0033 23: IP Priority Mapping Table 0x0000 24: IEEE Priority Mapping Table 0x3e07 25: Port Control 3 0x0000 26: Reserved 0x0000 27: Queue Counters 0x8000 28: Queue Control 0x0000 29: Reserved 0x0000 30: Cut Through Control 0x0000 31: Debug Counters 0x0000 88E6390X Switch Port SERDES Registers ------------------------------------- f000: Global Clock Configuration 1 0x0000 f001: Global Clock Configuration 2 0x0002 f002: Port Operational Configuration 0x0003 f00a: FIFO and CRC Int Enable 0x0000 f00b: FIFO and CRC Int Status 0x0000 f00c: PPM FIFO Control 1 0x0000 f00d: PPM FIFO Control 2 0x0000 f00e: PPM FIFO Status 0x0000 f010: Packet Generation Control 1 0x0501 f011: Packet Generation Control 2 0x0000 f012: Initial Payload 0-1/Packet Generation 0x0000 f013: Initial Payload 2-3/Packet Generation 0x0000 f016: Packet Generation Length 0x0000 f017: Packet Generation Burst Sequence 0x0000 f018: Packet Generation IPG 0x0002 f019: Packet Gen_Chkr Clock Control 0x0000 f01a: Transmit Powerdown Delay 0x0032 f01b: Transmit Packet Counter [15:0] 0x0000 f01c: Transmit Packet Counter [31:16] 0x0000 f01d: Transmit Packet Counter [47:32] 0x0000 f01e: Transmit Byte Counter [15:0] 0x0000 f01f: Transmit Byte Counter [31:16] 0x0000 f020: Transmit Byte Counter [47:32] 0x0000 f021: Receive Packet Counter [15:0] 0x0000 f022: Receive Packet Counter [31:16] 0x0000 f023: Receive Packet Counter [47:32] 0x0000 f024: Receive Byte Count [15:0] 0x0000 f025: Receive Byte Count [31:16] 0x0000 f026: Receive Byte Count [47:32] 0x0000 f027: Receive Packet Error Count [15:0] 0x0000 f028: Receive Packet Error Count [31:16] 0x0000 f029: Receive Packet Error Count [47:32] 0x0000 f030: PRBS Control 0x0200 f031: PRBS Symbol Tx Counter [15:0] 0x0000 f032: PRBS Symbol Tx Counter [31:16] 0x0000 f033: PRBS Symbol Tx Counter [47:32] 0x0000 f034: PRBS Symbol Rx Counter [15:0] 0x0000 f035: PRBS Symbol Rx Counter [31:16] 0x0000 f036: PRBS Symbol RX Counter [47:32] 0x0000 f037: PRBS Error Count [15:0] 0x0000 f038: PRBS Error Count [31:16] 0x0000 f039: PRBS Error Count [47:32] 0x0000 2000: 1000BASE-X/SGMII Control Register 0x1140 Reset 0 Loopback 0 SGMII Speed 1000 Mbps Autoneg Enable 1 Power down 0 Isolate 0 Restart Autonet 0 Duplex Full 2001: 1000BASE-X/SGMII Status Register 0x016d Autoneg Complete 1 Remote Fault 0 Link Status Up 2002: PHY Identifier 0x0141 2003: PHY Identifier 0x0c00 2004: SGMII (Media side) Auto-Negotiation Ad 0x00a0 Link Status Down Duplex Half SGMII Speed 10 Mbps Transmit Pause 0 Receive Pause 0 Fibre/Copper Fibre EEE mode 0 Clock stopped during LPI 1 2005: SGMII (Media side) Link Partner Abilit 0x40a0 Acknowledge 1 2006: 1000BASE-X Auto-Negotiation Expansion 0x0007 2007: 1000BASE-X Next Page Transmit Register 0x2801 2008: 1000BASE-X Link Partner Next Page Regi 0x0000 200f: Extended Status Register 0x8000 a000: 1000BASE-X Timer Mode Select Register 0x2000 a001: 1000BASE-X Interrupt Enable Register 0x0600 Speed Changed 0 Duplex Changed 0 Page Received 0 Autoneg Complete 0 Link Up->Link Down 1 Link Down->Link Up 1 Symbol Error 0 False Carrier 0 a002: 1000BASE-X Interrupt Status Register 0x0000 Speed Changed 0 Duplex Changed 0 Page Received 0 Autoneg Complete 0 Link Up->Link Down 0 Link Down->Link Up 0 Symbol Error 0 False Carrier 0 a003: 1000BASE-X PHY Specific Status 0xac2c Speed 1000 Mbps Duplex Full Page Received 0 Speed/Duplex Resolved 1 Link Up Sync 1 Energy Detect 0 Transmit Pause 0 Receive Pause 0 1000: 10GBASE-X4 PCS Control 1 0x2040 1001: 10GBASE-X4 PCS Status 1 0x0082 1002: PCS Device Identifier 1 0x0141 1003: PCS Device Identifier 2 0x0c00 1004: PCS Speed Ability 0x0001 1005: PCS Devices In Package 1 0x009a 1006: PCS Devices In Package 2 0x4000 1007: Reserved 0x0001 1008: 10GBASE-X4 PCS Status 2 0x8402 100e: PCS Package Identifier 1 0x0141 100f: PCS Package Identifier 2 0x0c00 1014: PCS EEE Capability Register 0x0000 1018: 10GBase-X Lane Status 0x0c01 1019: 10GBase-X Test Control 0x0000 9000: 10GBase-X Control 0x0001 9001: 10GBase-X Interrupt Enable 1 0x0000 9002: 10GBase-X Interrupt Enable 2 0x0000 9003: 10GBase-X Interrupt Status 1 0x0000 9004: 10GBase-X Interrupt Status 2 0x00e1 9006: 10GBase-X Real Time Status 0x0011 9010: 10GBase-X Random Sequence Control 0x0000 9011: 10GBase-X Jitter Packet Transmit Count 0x0000 9012: 10GBase-X Jitter Packet Transmit Count 0x0000 9013: 10GBase-X Jitter Packet Received Count 0x0000 9014: 10GBase-X Jitter Packet Received Count 0x0000 9015: 10GBase-X Jitter Packet Error Counter 0x0000 9016: 10GBase-X Jitter Packet Error Counter 0x0000 1020: 10GBASE-R PCS Status 1 0x0000 1021: 10GBASE-R PCS Status 2 0x0000 1022: 10GBASE-R PCS Test Pattern Seed A 0 0x0000 1023: 10GBASE-R PCS Test Pattern Seed A 1 0x0000 1024: 10GBASE-R PCS Test Pattern Seed A 2 0x0000 1025: 10GBASE-R PCS Test Pattern Seed A 3 0x0000 1026: 10GBASE-R PCS Test Pattern Seed B 0 0x0000 1027: 10GBASE-R PCS Test Pattern Seed B 1 0x0000 1028: 10GBASE-R PCS Test Pattern Seed B 2 0x0000 1029: 10GBASE-R PCS Test Pattern Seed B 3 0x0000 102a: 10GBASE-R PCS Test Pattern Control 0x0000 102b: 10GBASE-R PCS Test Error Counter 0x0000 root@(none):~$ ethtool -S sff2 NIC statistics: tx_packets: 3 tx_bytes: 126 rx_packets: 0 rx_bytes: 0 in_good_octets: 0 in_bad_octets: 0 in_unicast: 0 in_broadcasts: 0 in_multicasts: 0 in_pause: 0 in_undersize: 0 in_fragments: 0 in_oversize: 0 in_jabber: 0 in_rx_error: 0 in_fcs_error: 0 out_octets: 192 out_unicast: 0 out_broadcasts: 3 out_multicasts: 0 out_pause: 0 excessive: 0 collisions: 0 deferred: 0 single: 0 multiple: 0 out_fcs_error: 0 late: 0 hist_64bytes: 3 hist_65_127bytes: 0 hist_128_255bytes: 0 hist_256_511bytes: 0 hist_512_1023bytes: 0 hist_1024_max_bytes: 0 in_discards: 0 in_filtered: 0 in_accepted: 0 in_bad_accepted: 0 in_good_avb_class_a: 0 in_good_avb_class_b: 0 in_bad_avb_class_a: 0 in_bad_avb_class_b: 0 tcam_counter_0: 0 tcam_counter_1: 0 tcam_counter_2: 0 tcam_counter_3: 0 in_da_unknown: 0 in_management: 0 out_queue_0: 3 out_queue_1: 0 out_queue_2: 0 out_queue_3: 0 out_queue_4: 0 out_queue_5: 0 out_queue_6: 0 out_queue_7: 0 out_cut_through: 0 out_octets_a: 0 out_octets_b: 0 out_management: 3 serdes_rx_pkts: 6 serdes_rx_bytes: 384 serdes_rx_pkts_error: 0 atu_member_violation: 0 atu_miss_violation: 0 atu_full_violation: 0 vtu_member_violation: 0 vtu_miss_violation: 0 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2020-07-18 14:27 bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c Chris Healy @ 2020-07-18 14:42 ` Marek Behun 2020-07-18 14:49 ` Chris Healy 0 siblings, 1 reply; 11+ messages in thread From: Marek Behun @ 2020-07-18 14:42 UTC (permalink / raw) To: Chris Healy; +Cc: netdev Hmm, nothing sticks out in the register dump. I encountered a similar problem 2 years ago on Topaz SERDES port when the cmode was set to 2500BASE-X but the speed register was set to speed incompatible with 2500BASE-X (I don't remember what, though). This issue was solved by a patch I sent to netdev. Are you sure that your board isn't broken? Maybe the SerDes traces on RX path are damaged... Marek On Sat, 18 Jul 2020 07:27:33 -0700 Chris Healy <cphealy@gmail.com> wrote: > I've been trying to get the fiber interface of the vf610-zii-dev-rev-c > board working with net-next to no avail. This platform utilizes a > Cotsworks SFF attached to port 9 of a Marvell 88E6390X. > > I have fiber link up on port 9 and am able to send packets from the > management CPU of the switch through the switch and out port 9 through > the fiber interface to a fiber link partner successfully. I'm also > able to send packets from that fiber link partner back (to the point > of the switches SERDES) and am seeing the fiber ports SERDES RX > counters increment for each packet transmitted by the link partner. > The switches port 9 MAC is not showing any RX counters incrementing > though and I do not receive the frames at the management CPU. > > Because the SERDES RX counters are incrementing while the MAC RX > counters are not incrementing, it seems to me that the issue is > between the MAC and SERDES. This is odd to me given that TX works > fine. > > In support of debugging this issue, I've applied an ethtool patch > which allows decoding the 88E6390X SERDES registers from userspace. > Looking at the register dump, nothing obvious sticks out to me though. > > I'm not sure what the right next steps are and would appreciate any > theories on what to try/test to root cause this issue. > > Below is an ethtool register dump and an ethtool statistics dump from > when the link is up and I have done some attempts at pinging a remote > host over fiber. > > root@(none):~$ ethtool -d sff2 > 88E6390X Switch Port Registers > ------------------------------ > 00: Port Status 0xce49 > Transmit Pause Enable bit 1 > Receive Pause Enable bit 1 > 802.3 PHY Detected 0 > Link Status Up > Duplex Full > Speed 1000 Mbps > Duplex Fixed 0 > EEE Enabled 1 > Transmitter Paused 0 > Flow Control 0 > Config Mode 0x9 > 01: Physical Control 0x203e > RGMII Receive Timing Control Default > RGMII Transmit Timing Control Default > Force Speed 1 > Alternate Speed Mode Normal > MII PHY Mode MAC > EEE force value 0 > Force EEE 0 > Link's Forced value Up > Force Link 1 > Duplex's Forced value Full > Force Duplex 1 > Force Speed 1000 Mbps > 02: Flow Control 0x0100 > 03: Switch Identifier 0x0a11 > 04: Port Control 0x0433 > Source Address Filtering controls Disabled > Egress Mode Unmodified > Ingress & Egress Header Mode 0 > IGMP and MLD Snooping 1 > Frame Mode Normal > VLAN Tunnel 0 > TagIfBoth 0 > Initial Priority assignment Tag & IP Priority > Egress Flooding mode No unknown DA > Port State Forwarding > 05: Port Control 1 0x0000 > Message Port 0 > LAG Port 0 > VTU Page 0 > LAG ID 0 > FID[11:4] 0x000 > 06: Port Base VLAN Map (Header) 0x0400 > FID[3:0] 0x000 > Force Mapping 0 > VLANTable 10 > 07: Default VLAN ID & Priority 0x0000 > Default Priority 0x0 > Force to use Default VID 0 > Default VLAN Identifier 0 > 08: Port Control 2 0x0080 > Force good FCS in the frame 0 > Allow bad FCS 0 > Jumbo Mode 1522 > 802.1QMode Disabled > Discard Tagged Frames 0 > Discard Untagged Frames 0 > Map using DA hits 1 > ARP Mirror enable 0 > Egress Monitor Source Port 0 > Ingress Monitor Source Port 0 > Allow VID of Zero 0 > Default Queue Priority 0x0 > 09: Egress Rate Control 0x0001 > 10: Egress Rate Control 2 0x0000 > 11: Port Association Vector 0x0200 > 12: Port ATU Control 0x0000 > 13: Override 0x0000 > 14: Policy Control 0x0000 > 15: Port Ether Type 0x9100 > 16: Reserved 0x0000 > 17: Reserved 0x0000 > 18: Reserved 0x0000 > 19: Reserved 0x0000 > 20: Reserved 0x0000 > 21: Reserved 0x0000 > 22: LED Control 0x0033 > 23: IP Priority Mapping Table 0x0000 > 24: IEEE Priority Mapping Table 0x3e07 > 25: Port Control 3 0x0000 > 26: Reserved 0x0000 > 27: Queue Counters 0x8000 > 28: Queue Control 0x0000 > 29: Reserved 0x0000 > 30: Cut Through Control 0x0000 > 31: Debug Counters 0x0000 > > 88E6390X Switch Port SERDES Registers > ------------------------------------- > f000: Global Clock Configuration 1 0x0000 > f001: Global Clock Configuration 2 0x0002 > f002: Port Operational Configuration 0x0003 > f00a: FIFO and CRC Int Enable 0x0000 > f00b: FIFO and CRC Int Status 0x0000 > f00c: PPM FIFO Control 1 0x0000 > f00d: PPM FIFO Control 2 0x0000 > f00e: PPM FIFO Status 0x0000 > f010: Packet Generation Control 1 0x0501 > f011: Packet Generation Control 2 0x0000 > f012: Initial Payload 0-1/Packet Generation 0x0000 > f013: Initial Payload 2-3/Packet Generation 0x0000 > f016: Packet Generation Length 0x0000 > f017: Packet Generation Burst Sequence 0x0000 > f018: Packet Generation IPG 0x0002 > f019: Packet Gen_Chkr Clock Control 0x0000 > f01a: Transmit Powerdown Delay 0x0032 > f01b: Transmit Packet Counter [15:0] 0x0000 > f01c: Transmit Packet Counter [31:16] 0x0000 > f01d: Transmit Packet Counter [47:32] 0x0000 > f01e: Transmit Byte Counter [15:0] 0x0000 > f01f: Transmit Byte Counter [31:16] 0x0000 > f020: Transmit Byte Counter [47:32] 0x0000 > f021: Receive Packet Counter [15:0] 0x0000 > f022: Receive Packet Counter [31:16] 0x0000 > f023: Receive Packet Counter [47:32] 0x0000 > f024: Receive Byte Count [15:0] 0x0000 > f025: Receive Byte Count [31:16] 0x0000 > f026: Receive Byte Count [47:32] 0x0000 > f027: Receive Packet Error Count [15:0] 0x0000 > f028: Receive Packet Error Count [31:16] 0x0000 > f029: Receive Packet Error Count [47:32] 0x0000 > f030: PRBS Control 0x0200 > f031: PRBS Symbol Tx Counter [15:0] 0x0000 > f032: PRBS Symbol Tx Counter [31:16] 0x0000 > f033: PRBS Symbol Tx Counter [47:32] 0x0000 > f034: PRBS Symbol Rx Counter [15:0] 0x0000 > f035: PRBS Symbol Rx Counter [31:16] 0x0000 > f036: PRBS Symbol RX Counter [47:32] 0x0000 > f037: PRBS Error Count [15:0] 0x0000 > f038: PRBS Error Count [31:16] 0x0000 > f039: PRBS Error Count [47:32] 0x0000 > 2000: 1000BASE-X/SGMII Control Register 0x1140 > Reset 0 > Loopback 0 > SGMII Speed 1000 Mbps > Autoneg Enable 1 > Power down 0 > Isolate 0 > Restart Autonet 0 > Duplex Full > 2001: 1000BASE-X/SGMII Status Register 0x016d > Autoneg Complete 1 > Remote Fault 0 > Link Status Up > 2002: PHY Identifier 0x0141 > 2003: PHY Identifier 0x0c00 > 2004: SGMII (Media side) Auto-Negotiation Ad 0x00a0 > Link Status Down > Duplex Half > SGMII Speed 10 Mbps > Transmit Pause 0 > Receive Pause 0 > Fibre/Copper Fibre > EEE mode 0 > Clock stopped during LPI 1 > 2005: SGMII (Media side) Link Partner Abilit 0x40a0 > Acknowledge 1 > 2006: 1000BASE-X Auto-Negotiation Expansion 0x0007 > 2007: 1000BASE-X Next Page Transmit Register 0x2801 > 2008: 1000BASE-X Link Partner Next Page Regi 0x0000 > 200f: Extended Status Register 0x8000 > a000: 1000BASE-X Timer Mode Select Register 0x2000 > a001: 1000BASE-X Interrupt Enable Register 0x0600 > Speed Changed 0 > Duplex Changed 0 > Page Received 0 > Autoneg Complete 0 > Link Up->Link Down 1 > Link Down->Link Up 1 > Symbol Error 0 > False Carrier 0 > a002: 1000BASE-X Interrupt Status Register 0x0000 > Speed Changed 0 > Duplex Changed 0 > Page Received 0 > Autoneg Complete 0 > Link Up->Link Down 0 > Link Down->Link Up 0 > Symbol Error 0 > False Carrier 0 > a003: 1000BASE-X PHY Specific Status 0xac2c > Speed 1000 Mbps > Duplex Full > Page Received 0 > Speed/Duplex Resolved 1 > Link Up > Sync 1 > Energy Detect 0 > Transmit Pause 0 > Receive Pause 0 > 1000: 10GBASE-X4 PCS Control 1 0x2040 > 1001: 10GBASE-X4 PCS Status 1 0x0082 > 1002: PCS Device Identifier 1 0x0141 > 1003: PCS Device Identifier 2 0x0c00 > 1004: PCS Speed Ability 0x0001 > 1005: PCS Devices In Package 1 0x009a > 1006: PCS Devices In Package 2 0x4000 > 1007: Reserved 0x0001 > 1008: 10GBASE-X4 PCS Status 2 0x8402 > 100e: PCS Package Identifier 1 0x0141 > 100f: PCS Package Identifier 2 0x0c00 > 1014: PCS EEE Capability Register 0x0000 > 1018: 10GBase-X Lane Status 0x0c01 > 1019: 10GBase-X Test Control 0x0000 > 9000: 10GBase-X Control 0x0001 > 9001: 10GBase-X Interrupt Enable 1 0x0000 > 9002: 10GBase-X Interrupt Enable 2 0x0000 > 9003: 10GBase-X Interrupt Status 1 0x0000 > 9004: 10GBase-X Interrupt Status 2 0x00e1 > 9006: 10GBase-X Real Time Status 0x0011 > 9010: 10GBase-X Random Sequence Control 0x0000 > 9011: 10GBase-X Jitter Packet Transmit Count 0x0000 > 9012: 10GBase-X Jitter Packet Transmit Count 0x0000 > 9013: 10GBase-X Jitter Packet Received Count 0x0000 > 9014: 10GBase-X Jitter Packet Received Count 0x0000 > 9015: 10GBase-X Jitter Packet Error Counter 0x0000 > 9016: 10GBase-X Jitter Packet Error Counter 0x0000 > 1020: 10GBASE-R PCS Status 1 0x0000 > 1021: 10GBASE-R PCS Status 2 0x0000 > 1022: 10GBASE-R PCS Test Pattern Seed A 0 0x0000 > 1023: 10GBASE-R PCS Test Pattern Seed A 1 0x0000 > 1024: 10GBASE-R PCS Test Pattern Seed A 2 0x0000 > 1025: 10GBASE-R PCS Test Pattern Seed A 3 0x0000 > 1026: 10GBASE-R PCS Test Pattern Seed B 0 0x0000 > 1027: 10GBASE-R PCS Test Pattern Seed B 1 0x0000 > 1028: 10GBASE-R PCS Test Pattern Seed B 2 0x0000 > 1029: 10GBASE-R PCS Test Pattern Seed B 3 0x0000 > 102a: 10GBASE-R PCS Test Pattern Control 0x0000 > 102b: 10GBASE-R PCS Test Error Counter 0x0000 > > root@(none):~$ ethtool -S sff2 > NIC statistics: > tx_packets: 3 > tx_bytes: 126 > rx_packets: 0 > rx_bytes: 0 > in_good_octets: 0 > in_bad_octets: 0 > in_unicast: 0 > in_broadcasts: 0 > in_multicasts: 0 > in_pause: 0 > in_undersize: 0 > in_fragments: 0 > in_oversize: 0 > in_jabber: 0 > in_rx_error: 0 > in_fcs_error: 0 > out_octets: 192 > out_unicast: 0 > out_broadcasts: 3 > out_multicasts: 0 > out_pause: 0 > excessive: 0 > collisions: 0 > deferred: 0 > single: 0 > multiple: 0 > out_fcs_error: 0 > late: 0 > hist_64bytes: 3 > hist_65_127bytes: 0 > hist_128_255bytes: 0 > hist_256_511bytes: 0 > hist_512_1023bytes: 0 > hist_1024_max_bytes: 0 > in_discards: 0 > in_filtered: 0 > in_accepted: 0 > in_bad_accepted: 0 > in_good_avb_class_a: 0 > in_good_avb_class_b: 0 > in_bad_avb_class_a: 0 > in_bad_avb_class_b: 0 > tcam_counter_0: 0 > tcam_counter_1: 0 > tcam_counter_2: 0 > tcam_counter_3: 0 > in_da_unknown: 0 > in_management: 0 > out_queue_0: 3 > out_queue_1: 0 > out_queue_2: 0 > out_queue_3: 0 > out_queue_4: 0 > out_queue_5: 0 > out_queue_6: 0 > out_queue_7: 0 > out_cut_through: 0 > out_octets_a: 0 > out_octets_b: 0 > out_management: 3 > serdes_rx_pkts: 6 > serdes_rx_bytes: 384 > serdes_rx_pkts_error: 0 > atu_member_violation: 0 > atu_miss_violation: 0 > atu_full_violation: 0 > vtu_member_violation: 0 > vtu_miss_violation: 0 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2020-07-18 14:42 ` Marek Behun @ 2020-07-18 14:49 ` Chris Healy 2020-07-18 15:05 ` Andrew Lunn 0 siblings, 1 reply; 11+ messages in thread From: Chris Healy @ 2020-07-18 14:49 UTC (permalink / raw) To: Marek Behun; +Cc: netdev On Sat, Jul 18, 2020 at 7:42 AM Marek Behun <marek.behun@nic.cz> wrote: > > Hmm, nothing sticks out in the register dump. > > I encountered a similar problem 2 years ago on Topaz SERDES port when > the cmode was set to 2500BASE-X but the speed register was set to speed > incompatible with 2500BASE-X (I don't remember what, though). This > issue was solved by a patch I sent to netdev. > > Are you sure that your board isn't broken? Maybe the SerDes traces on > RX path are damaged... In my case, both the SERDES and the MAC are inside the switch so I don't think it's likely that the SERDES traces are broken in there. If you are referring to the traces between the SERDES and the fiber module, that doesn't feel likely either as the SERDES appears to be reporting successfully received frames: From "ethtool -S" after sending 6 packets to the unit: serdes_rx_pkts: 6 serdes_rx_bytes: 384 serdes_rx_pkts_error: 0 If the traces were broken between the fiber module and the SERDES, I should not see these counters incrementing. > > Marek > > On Sat, 18 Jul 2020 07:27:33 -0700 > Chris Healy <cphealy@gmail.com> wrote: > > > I've been trying to get the fiber interface of the vf610-zii-dev-rev-c > > board working with net-next to no avail. This platform utilizes a > > Cotsworks SFF attached to port 9 of a Marvell 88E6390X. > > > > I have fiber link up on port 9 and am able to send packets from the > > management CPU of the switch through the switch and out port 9 through > > the fiber interface to a fiber link partner successfully. I'm also > > able to send packets from that fiber link partner back (to the point > > of the switches SERDES) and am seeing the fiber ports SERDES RX > > counters increment for each packet transmitted by the link partner. > > The switches port 9 MAC is not showing any RX counters incrementing > > though and I do not receive the frames at the management CPU. > > > > Because the SERDES RX counters are incrementing while the MAC RX > > counters are not incrementing, it seems to me that the issue is > > between the MAC and SERDES. This is odd to me given that TX works > > fine. > > > > In support of debugging this issue, I've applied an ethtool patch > > which allows decoding the 88E6390X SERDES registers from userspace. > > Looking at the register dump, nothing obvious sticks out to me though. > > > > I'm not sure what the right next steps are and would appreciate any > > theories on what to try/test to root cause this issue. > > > > Below is an ethtool register dump and an ethtool statistics dump from > > when the link is up and I have done some attempts at pinging a remote > > host over fiber. > > > > root@(none):~$ ethtool -d sff2 > > 88E6390X Switch Port Registers > > ------------------------------ > > 00: Port Status 0xce49 > > Transmit Pause Enable bit 1 > > Receive Pause Enable bit 1 > > 802.3 PHY Detected 0 > > Link Status Up > > Duplex Full > > Speed 1000 Mbps > > Duplex Fixed 0 > > EEE Enabled 1 > > Transmitter Paused 0 > > Flow Control 0 > > Config Mode 0x9 > > 01: Physical Control 0x203e > > RGMII Receive Timing Control Default > > RGMII Transmit Timing Control Default > > Force Speed 1 > > Alternate Speed Mode Normal > > MII PHY Mode MAC > > EEE force value 0 > > Force EEE 0 > > Link's Forced value Up > > Force Link 1 > > Duplex's Forced value Full > > Force Duplex 1 > > Force Speed 1000 Mbps > > 02: Flow Control 0x0100 > > 03: Switch Identifier 0x0a11 > > 04: Port Control 0x0433 > > Source Address Filtering controls Disabled > > Egress Mode Unmodified > > Ingress & Egress Header Mode 0 > > IGMP and MLD Snooping 1 > > Frame Mode Normal > > VLAN Tunnel 0 > > TagIfBoth 0 > > Initial Priority assignment Tag & IP Priority > > Egress Flooding mode No unknown DA > > Port State Forwarding > > 05: Port Control 1 0x0000 > > Message Port 0 > > LAG Port 0 > > VTU Page 0 > > LAG ID 0 > > FID[11:4] 0x000 > > 06: Port Base VLAN Map (Header) 0x0400 > > FID[3:0] 0x000 > > Force Mapping 0 > > VLANTable 10 > > 07: Default VLAN ID & Priority 0x0000 > > Default Priority 0x0 > > Force to use Default VID 0 > > Default VLAN Identifier 0 > > 08: Port Control 2 0x0080 > > Force good FCS in the frame 0 > > Allow bad FCS 0 > > Jumbo Mode 1522 > > 802.1QMode Disabled > > Discard Tagged Frames 0 > > Discard Untagged Frames 0 > > Map using DA hits 1 > > ARP Mirror enable 0 > > Egress Monitor Source Port 0 > > Ingress Monitor Source Port 0 > > Allow VID of Zero 0 > > Default Queue Priority 0x0 > > 09: Egress Rate Control 0x0001 > > 10: Egress Rate Control 2 0x0000 > > 11: Port Association Vector 0x0200 > > 12: Port ATU Control 0x0000 > > 13: Override 0x0000 > > 14: Policy Control 0x0000 > > 15: Port Ether Type 0x9100 > > 16: Reserved 0x0000 > > 17: Reserved 0x0000 > > 18: Reserved 0x0000 > > 19: Reserved 0x0000 > > 20: Reserved 0x0000 > > 21: Reserved 0x0000 > > 22: LED Control 0x0033 > > 23: IP Priority Mapping Table 0x0000 > > 24: IEEE Priority Mapping Table 0x3e07 > > 25: Port Control 3 0x0000 > > 26: Reserved 0x0000 > > 27: Queue Counters 0x8000 > > 28: Queue Control 0x0000 > > 29: Reserved 0x0000 > > 30: Cut Through Control 0x0000 > > 31: Debug Counters 0x0000 > > > > 88E6390X Switch Port SERDES Registers > > ------------------------------------- > > f000: Global Clock Configuration 1 0x0000 > > f001: Global Clock Configuration 2 0x0002 > > f002: Port Operational Configuration 0x0003 > > f00a: FIFO and CRC Int Enable 0x0000 > > f00b: FIFO and CRC Int Status 0x0000 > > f00c: PPM FIFO Control 1 0x0000 > > f00d: PPM FIFO Control 2 0x0000 > > f00e: PPM FIFO Status 0x0000 > > f010: Packet Generation Control 1 0x0501 > > f011: Packet Generation Control 2 0x0000 > > f012: Initial Payload 0-1/Packet Generation 0x0000 > > f013: Initial Payload 2-3/Packet Generation 0x0000 > > f016: Packet Generation Length 0x0000 > > f017: Packet Generation Burst Sequence 0x0000 > > f018: Packet Generation IPG 0x0002 > > f019: Packet Gen_Chkr Clock Control 0x0000 > > f01a: Transmit Powerdown Delay 0x0032 > > f01b: Transmit Packet Counter [15:0] 0x0000 > > f01c: Transmit Packet Counter [31:16] 0x0000 > > f01d: Transmit Packet Counter [47:32] 0x0000 > > f01e: Transmit Byte Counter [15:0] 0x0000 > > f01f: Transmit Byte Counter [31:16] 0x0000 > > f020: Transmit Byte Counter [47:32] 0x0000 > > f021: Receive Packet Counter [15:0] 0x0000 > > f022: Receive Packet Counter [31:16] 0x0000 > > f023: Receive Packet Counter [47:32] 0x0000 > > f024: Receive Byte Count [15:0] 0x0000 > > f025: Receive Byte Count [31:16] 0x0000 > > f026: Receive Byte Count [47:32] 0x0000 > > f027: Receive Packet Error Count [15:0] 0x0000 > > f028: Receive Packet Error Count [31:16] 0x0000 > > f029: Receive Packet Error Count [47:32] 0x0000 > > f030: PRBS Control 0x0200 > > f031: PRBS Symbol Tx Counter [15:0] 0x0000 > > f032: PRBS Symbol Tx Counter [31:16] 0x0000 > > f033: PRBS Symbol Tx Counter [47:32] 0x0000 > > f034: PRBS Symbol Rx Counter [15:0] 0x0000 > > f035: PRBS Symbol Rx Counter [31:16] 0x0000 > > f036: PRBS Symbol RX Counter [47:32] 0x0000 > > f037: PRBS Error Count [15:0] 0x0000 > > f038: PRBS Error Count [31:16] 0x0000 > > f039: PRBS Error Count [47:32] 0x0000 > > 2000: 1000BASE-X/SGMII Control Register 0x1140 > > Reset 0 > > Loopback 0 > > SGMII Speed 1000 Mbps > > Autoneg Enable 1 > > Power down 0 > > Isolate 0 > > Restart Autonet 0 > > Duplex Full > > 2001: 1000BASE-X/SGMII Status Register 0x016d > > Autoneg Complete 1 > > Remote Fault 0 > > Link Status Up > > 2002: PHY Identifier 0x0141 > > 2003: PHY Identifier 0x0c00 > > 2004: SGMII (Media side) Auto-Negotiation Ad 0x00a0 > > Link Status Down > > Duplex Half > > SGMII Speed 10 Mbps > > Transmit Pause 0 > > Receive Pause 0 > > Fibre/Copper Fibre > > EEE mode 0 > > Clock stopped during LPI 1 > > 2005: SGMII (Media side) Link Partner Abilit 0x40a0 > > Acknowledge 1 > > 2006: 1000BASE-X Auto-Negotiation Expansion 0x0007 > > 2007: 1000BASE-X Next Page Transmit Register 0x2801 > > 2008: 1000BASE-X Link Partner Next Page Regi 0x0000 > > 200f: Extended Status Register 0x8000 > > a000: 1000BASE-X Timer Mode Select Register 0x2000 > > a001: 1000BASE-X Interrupt Enable Register 0x0600 > > Speed Changed 0 > > Duplex Changed 0 > > Page Received 0 > > Autoneg Complete 0 > > Link Up->Link Down 1 > > Link Down->Link Up 1 > > Symbol Error 0 > > False Carrier 0 > > a002: 1000BASE-X Interrupt Status Register 0x0000 > > Speed Changed 0 > > Duplex Changed 0 > > Page Received 0 > > Autoneg Complete 0 > > Link Up->Link Down 0 > > Link Down->Link Up 0 > > Symbol Error 0 > > False Carrier 0 > > a003: 1000BASE-X PHY Specific Status 0xac2c > > Speed 1000 Mbps > > Duplex Full > > Page Received 0 > > Speed/Duplex Resolved 1 > > Link Up > > Sync 1 > > Energy Detect 0 > > Transmit Pause 0 > > Receive Pause 0 > > 1000: 10GBASE-X4 PCS Control 1 0x2040 > > 1001: 10GBASE-X4 PCS Status 1 0x0082 > > 1002: PCS Device Identifier 1 0x0141 > > 1003: PCS Device Identifier 2 0x0c00 > > 1004: PCS Speed Ability 0x0001 > > 1005: PCS Devices In Package 1 0x009a > > 1006: PCS Devices In Package 2 0x4000 > > 1007: Reserved 0x0001 > > 1008: 10GBASE-X4 PCS Status 2 0x8402 > > 100e: PCS Package Identifier 1 0x0141 > > 100f: PCS Package Identifier 2 0x0c00 > > 1014: PCS EEE Capability Register 0x0000 > > 1018: 10GBase-X Lane Status 0x0c01 > > 1019: 10GBase-X Test Control 0x0000 > > 9000: 10GBase-X Control 0x0001 > > 9001: 10GBase-X Interrupt Enable 1 0x0000 > > 9002: 10GBase-X Interrupt Enable 2 0x0000 > > 9003: 10GBase-X Interrupt Status 1 0x0000 > > 9004: 10GBase-X Interrupt Status 2 0x00e1 > > 9006: 10GBase-X Real Time Status 0x0011 > > 9010: 10GBase-X Random Sequence Control 0x0000 > > 9011: 10GBase-X Jitter Packet Transmit Count 0x0000 > > 9012: 10GBase-X Jitter Packet Transmit Count 0x0000 > > 9013: 10GBase-X Jitter Packet Received Count 0x0000 > > 9014: 10GBase-X Jitter Packet Received Count 0x0000 > > 9015: 10GBase-X Jitter Packet Error Counter 0x0000 > > 9016: 10GBase-X Jitter Packet Error Counter 0x0000 > > 1020: 10GBASE-R PCS Status 1 0x0000 > > 1021: 10GBASE-R PCS Status 2 0x0000 > > 1022: 10GBASE-R PCS Test Pattern Seed A 0 0x0000 > > 1023: 10GBASE-R PCS Test Pattern Seed A 1 0x0000 > > 1024: 10GBASE-R PCS Test Pattern Seed A 2 0x0000 > > 1025: 10GBASE-R PCS Test Pattern Seed A 3 0x0000 > > 1026: 10GBASE-R PCS Test Pattern Seed B 0 0x0000 > > 1027: 10GBASE-R PCS Test Pattern Seed B 1 0x0000 > > 1028: 10GBASE-R PCS Test Pattern Seed B 2 0x0000 > > 1029: 10GBASE-R PCS Test Pattern Seed B 3 0x0000 > > 102a: 10GBASE-R PCS Test Pattern Control 0x0000 > > 102b: 10GBASE-R PCS Test Error Counter 0x0000 > > > > root@(none):~$ ethtool -S sff2 > > NIC statistics: > > tx_packets: 3 > > tx_bytes: 126 > > rx_packets: 0 > > rx_bytes: 0 > > in_good_octets: 0 > > in_bad_octets: 0 > > in_unicast: 0 > > in_broadcasts: 0 > > in_multicasts: 0 > > in_pause: 0 > > in_undersize: 0 > > in_fragments: 0 > > in_oversize: 0 > > in_jabber: 0 > > in_rx_error: 0 > > in_fcs_error: 0 > > out_octets: 192 > > out_unicast: 0 > > out_broadcasts: 3 > > out_multicasts: 0 > > out_pause: 0 > > excessive: 0 > > collisions: 0 > > deferred: 0 > > single: 0 > > multiple: 0 > > out_fcs_error: 0 > > late: 0 > > hist_64bytes: 3 > > hist_65_127bytes: 0 > > hist_128_255bytes: 0 > > hist_256_511bytes: 0 > > hist_512_1023bytes: 0 > > hist_1024_max_bytes: 0 > > in_discards: 0 > > in_filtered: 0 > > in_accepted: 0 > > in_bad_accepted: 0 > > in_good_avb_class_a: 0 > > in_good_avb_class_b: 0 > > in_bad_avb_class_a: 0 > > in_bad_avb_class_b: 0 > > tcam_counter_0: 0 > > tcam_counter_1: 0 > > tcam_counter_2: 0 > > tcam_counter_3: 0 > > in_da_unknown: 0 > > in_management: 0 > > out_queue_0: 3 > > out_queue_1: 0 > > out_queue_2: 0 > > out_queue_3: 0 > > out_queue_4: 0 > > out_queue_5: 0 > > out_queue_6: 0 > > out_queue_7: 0 > > out_cut_through: 0 > > out_octets_a: 0 > > out_octets_b: 0 > > out_management: 3 > > serdes_rx_pkts: 6 > > serdes_rx_bytes: 384 > > serdes_rx_pkts_error: 0 > > atu_member_violation: 0 > > atu_miss_violation: 0 > > atu_full_violation: 0 > > vtu_member_violation: 0 > > vtu_miss_violation: 0 > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2020-07-18 14:49 ` Chris Healy @ 2020-07-18 15:05 ` Andrew Lunn 2020-07-18 15:22 ` Marek Behun 0 siblings, 1 reply; 11+ messages in thread From: Andrew Lunn @ 2020-07-18 15:05 UTC (permalink / raw) To: Chris Healy; +Cc: Marek Behun, netdev On Sat, Jul 18, 2020 at 07:49:26AM -0700, Chris Healy wrote: > On Sat, Jul 18, 2020 at 7:42 AM Marek Behun <marek.behun@nic.cz> wrote: > > > > Hmm, nothing sticks out in the register dump. > > > > I encountered a similar problem 2 years ago on Topaz SERDES port when > > the cmode was set to 2500BASE-X but the speed register was set to speed > > incompatible with 2500BASE-X (I don't remember what, though). This > > issue was solved by a patch I sent to netdev. > > > > Are you sure that your board isn't broken? Maybe the SerDes traces on > > RX path are damaged... > > In my case, both the SERDES and the MAC are inside the switch so I > don't think it's likely that the SERDES traces are broken in there. > If you are referring to the traces between the SERDES and the fiber > module, that doesn't feel likely either as the SERDES appears to be > reporting successfully received frames: > > >From "ethtool -S" after sending 6 packets to the unit: > serdes_rx_pkts: 6 > serdes_rx_bytes: 384 > serdes_rx_pkts_error: 0 > > If the traces were broken between the fiber module and the SERDES, I > should not see these counters incrementing. Plus it is reproducible on multiple boards, of different designs. This is somehow specific to the 6390X ports 9 and 10. Andrew ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2020-07-18 15:05 ` Andrew Lunn @ 2020-07-18 15:22 ` Marek Behun 2020-07-19 21:43 ` Chris Healy 0 siblings, 1 reply; 11+ messages in thread From: Marek Behun @ 2020-07-18 15:22 UTC (permalink / raw) To: Andrew Lunn; +Cc: Chris Healy, netdev On Sat, 18 Jul 2020 17:05:14 +0200 Andrew Lunn <andrew@lunn.ch> wrote: > > If the traces were broken between the fiber module and the SERDES, I > > should not see these counters incrementing. > > Plus it is reproducible on multiple boards, of different designs. > > This is somehow specific to the 6390X ports 9 and 10. > > Andrew Hmm. What about the errata setup? It says: /* The 6390 copper ports have an errata which require poking magic * values into undocumented hidden registers and then performing a * software reset. */ But then the port_hidden_write function is called for every port in the function mv88e6390_setup_errata, not just for copper ports. Maybe Chris should try to not write this hidden register for SerDes ports. Marek ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2020-07-18 15:22 ` Marek Behun @ 2020-07-19 21:43 ` Chris Healy 2020-07-19 21:52 ` Marek Behun 2021-01-18 17:31 ` Tobias Waldekranz 0 siblings, 2 replies; 11+ messages in thread From: Chris Healy @ 2020-07-19 21:43 UTC (permalink / raw) To: Marek Behun; +Cc: Andrew Lunn, netdev On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote: > > On Sat, 18 Jul 2020 17:05:14 +0200 > Andrew Lunn <andrew@lunn.ch> wrote: > > > > If the traces were broken between the fiber module and the SERDES, I > > > should not see these counters incrementing. > > > > Plus it is reproducible on multiple boards, of different designs. > > > > This is somehow specific to the 6390X ports 9 and 10. > > > > Andrew > > Hmm. > > What about the errata setup? > It says: > /* The 6390 copper ports have an errata which require poking magic > * values into undocumented hidden registers and then performing a > * software reset. > */ > But then the port_hidden_write function is called for every port in the > function mv88e6390_setup_errata, not just for copper ports. Maybe Chris > should try to not write this hidden register for SerDes ports. I just disabled the mv88e6390_setup_errata all together and this did not result in any different behaviour on this broken fiber port. > > Marek ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2020-07-19 21:43 ` Chris Healy @ 2020-07-19 21:52 ` Marek Behun 2021-01-18 17:31 ` Tobias Waldekranz 1 sibling, 0 replies; 11+ messages in thread From: Marek Behun @ 2020-07-19 21:52 UTC (permalink / raw) To: Chris Healy; +Cc: Andrew Lunn, netdev On Sun, 19 Jul 2020 14:43:49 -0700 Chris Healy <cphealy@gmail.com> wrote: > > Hmm. > > > > What about the errata setup? > > It says: > > /* The 6390 copper ports have an errata which require poking magic > > * values into undocumented hidden registers and then performing a > > * software reset. > > */ > > But then the port_hidden_write function is called for every port in the > > function mv88e6390_setup_errata, not just for copper ports. Maybe Chris > > should try to not write this hidden register for SerDes ports. > > I just disabled the mv88e6390_setup_errata all together and this did > not result in any different behaviour on this broken fiber port. :-( In that case I really have no idea what could be the problem. Another thing you could try is resoldering resistors on the board so that the switches configure themselves into NOCPU mode and the port you are talking about configures itself into the cmode you need (was it 1000base-x or sgmii?). Disable DSA, write yourself sysfs API via which you can read/write switch registers by yourself. Then you can chech if the problem on the RX path occurs. If no, read all the registers and compare their values with the ones the mv88e6xxx driver configures. If yes, then we know that the problem is there even if the switch is NOCPU mode and you can inform Marvell about the bug. Marek ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2020-07-19 21:43 ` Chris Healy 2020-07-19 21:52 ` Marek Behun @ 2021-01-18 17:31 ` Tobias Waldekranz 2021-01-18 17:41 ` Andrew Lunn 1 sibling, 1 reply; 11+ messages in thread From: Tobias Waldekranz @ 2021-01-18 17:31 UTC (permalink / raw) To: Chris Healy, Marek Behun; +Cc: Andrew Lunn, netdev On Sun, Jul 19, 2020 at 14:43, Chris Healy <cphealy@gmail.com> wrote: > On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote: >> >> On Sat, 18 Jul 2020 17:05:14 +0200 >> Andrew Lunn <andrew@lunn.ch> wrote: >> >> > > If the traces were broken between the fiber module and the SERDES, I >> > > should not see these counters incrementing. >> > >> > Plus it is reproducible on multiple boards, of different designs. >> > >> > This is somehow specific to the 6390X ports 9 and 10. >> > >> > Andrew >> >> Hmm. >> >> What about the errata setup? >> It says: >> /* The 6390 copper ports have an errata which require poking magic >> * values into undocumented hidden registers and then performing a >> * software reset. >> */ >> But then the port_hidden_write function is called for every port in the >> function mv88e6390_setup_errata, not just for copper ports. Maybe Chris >> should try to not write this hidden register for SerDes ports. > > I just disabled the mv88e6390_setup_errata all together and this did > not result in any different behaviour on this broken fiber port. Hi Chris, Did you manage to track this down? I am seeing the exact same issue. I have tried both a 1000base-x SFP and a copper 1000base-T and get the same result on both - transmit is fine but rx only works up to the SERDES, no rx MAC counters are moving. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2021-01-18 17:31 ` Tobias Waldekranz @ 2021-01-18 17:41 ` Andrew Lunn 2021-01-18 17:47 ` Tobias Waldekranz 0 siblings, 1 reply; 11+ messages in thread From: Andrew Lunn @ 2021-01-18 17:41 UTC (permalink / raw) To: Tobias Waldekranz; +Cc: Chris Healy, Marek Behun, netdev On Mon, Jan 18, 2021 at 06:31:19PM +0100, Tobias Waldekranz wrote: > On Sun, Jul 19, 2020 at 14:43, Chris Healy <cphealy@gmail.com> wrote: > > On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote: > >> > >> On Sat, 18 Jul 2020 17:05:14 +0200 > >> Andrew Lunn <andrew@lunn.ch> wrote: > >> > >> > > If the traces were broken between the fiber module and the SERDES, I > >> > > should not see these counters incrementing. > >> > > >> > Plus it is reproducible on multiple boards, of different designs. > >> > > >> > This is somehow specific to the 6390X ports 9 and 10. > >> > > >> > Andrew > >> > >> Hmm. > >> > >> What about the errata setup? > >> It says: > >> /* The 6390 copper ports have an errata which require poking magic > >> * values into undocumented hidden registers and then performing a > >> * software reset. > >> */ > >> But then the port_hidden_write function is called for every port in the > >> function mv88e6390_setup_errata, not just for copper ports. Maybe Chris > >> should try to not write this hidden register for SerDes ports. > > > > I just disabled the mv88e6390_setup_errata all together and this did > > not result in any different behaviour on this broken fiber port. > > Hi Chris, > > Did you manage to track this down? > > I am seeing the exact same issue. I have tried both a 1000base-x SFP and > a copper 1000base-T and get the same result on both - transmit is fine > but rx only works up to the SERDES, no rx MAC counters are moving. Hi Tobias We never tracked this down. I spent many hours bashing my head against this. I could not bisect it, which did not help. FYI: Chris has moved onto a new job, and is unlikely to be involved with Marvell switches any more. Andrew ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2021-01-18 17:41 ` Andrew Lunn @ 2021-01-18 17:47 ` Tobias Waldekranz 2021-01-18 19:26 ` Andrew Lunn 0 siblings, 1 reply; 11+ messages in thread From: Tobias Waldekranz @ 2021-01-18 17:47 UTC (permalink / raw) To: Andrew Lunn; +Cc: Chris Healy, Marek Behun, netdev On Mon, Jan 18, 2021 at 18:41, Andrew Lunn <andrew@lunn.ch> wrote: > On Mon, Jan 18, 2021 at 06:31:19PM +0100, Tobias Waldekranz wrote: >> On Sun, Jul 19, 2020 at 14:43, Chris Healy <cphealy@gmail.com> wrote: >> > On Sat, Jul 18, 2020 at 8:22 AM Marek Behun <marek.behun@nic.cz> wrote: >> >> >> >> On Sat, 18 Jul 2020 17:05:14 +0200 >> >> Andrew Lunn <andrew@lunn.ch> wrote: >> >> >> >> > > If the traces were broken between the fiber module and the SERDES, I >> >> > > should not see these counters incrementing. >> >> > >> >> > Plus it is reproducible on multiple boards, of different designs. >> >> > >> >> > This is somehow specific to the 6390X ports 9 and 10. >> >> > >> >> > Andrew >> >> >> >> Hmm. >> >> >> >> What about the errata setup? >> >> It says: >> >> /* The 6390 copper ports have an errata which require poking magic >> >> * values into undocumented hidden registers and then performing a >> >> * software reset. >> >> */ >> >> But then the port_hidden_write function is called for every port in the >> >> function mv88e6390_setup_errata, not just for copper ports. Maybe Chris >> >> should try to not write this hidden register for SerDes ports. >> > >> > I just disabled the mv88e6390_setup_errata all together and this did >> > not result in any different behaviour on this broken fiber port. >> >> Hi Chris, >> >> Did you manage to track this down? >> >> I am seeing the exact same issue. I have tried both a 1000base-x SFP and >> a copper 1000base-T and get the same result on both - transmit is fine >> but rx only works up to the SERDES, no rx MAC counters are moving. > > Hi Tobias > > We never tracked this down. I spent many hours bashing my head against > this. I could not bisect it, which did not help. Well that is disheartening :) "I could not bisect it", does that mean that it did work at some point but your CPU platform was not supported far enough back, or has it never worked? > FYI: Chris has moved onto a new job, and is unlikely to be involved > with Marvell switches any more. Good to know, thanks. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c 2021-01-18 17:47 ` Tobias Waldekranz @ 2021-01-18 19:26 ` Andrew Lunn 0 siblings, 0 replies; 11+ messages in thread From: Andrew Lunn @ 2021-01-18 19:26 UTC (permalink / raw) To: Tobias Waldekranz; +Cc: Chris Healy, Marek Behun, netdev > >> I am seeing the exact same issue. I have tried both a 1000base-x SFP and > >> a copper 1000base-T and get the same result on both - transmit is fine > >> but rx only works up to the SERDES, no rx MAC counters are moving. > > > > Hi Tobias > > > > We never tracked this down. I spent many hours bashing my head against > > this. I could not bisect it, which did not help. > > Well that is disheartening :) "I could not bisect it", does that mean > that it did work at some point but your CPU platform was not supported > far enough back, or has it never worked? As far as i remember, Ports 9 and 10 did work once. I suspect it could of been early on, before we had much support for the SERDES, it relied on the strapping being correct, and the switch powering up in the right mode. We also do a dance with the cmode of ports 9 and 10, dropping them down to slower speeds making the SERDES available for the other ports and then changing the cmode again if the port is supposed to use a higher speed and needs multiple SERDESes. The board which had trouble and i was debugging on only has limited support, not going back too far. I would probably need to reproduce the issue on different hardware to have more scope for going backwards and trying to find a setup where ports 9 or 10 did work, as a basis for a bisect. Andrew ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-01-18 19:46 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-07-18 14:27 bug: net: dsa: mv88e6xxx: serdes Unable to communicate on fiber with vf610-zii-dev-rev-c Chris Healy 2020-07-18 14:42 ` Marek Behun 2020-07-18 14:49 ` Chris Healy 2020-07-18 15:05 ` Andrew Lunn 2020-07-18 15:22 ` Marek Behun 2020-07-19 21:43 ` Chris Healy 2020-07-19 21:52 ` Marek Behun 2021-01-18 17:31 ` Tobias Waldekranz 2021-01-18 17:41 ` Andrew Lunn 2021-01-18 17:47 ` Tobias Waldekranz 2021-01-18 19:26 ` Andrew Lunn
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).