[PATCH net-next v4 0/3] at803x fiber/SFP support

[PATCH net-next v4 0/3] at803x fiber/SFP support

[Robert Hancock]

Add support for 1000Base-X fiber modes to the at803x PHY driver, as
well as support for connecting a downstream SFP cage.

Changes since v3:
-Renamed some constants with OHM suffix for clarity

Changes since v2:
-fixed tabs/spaces issue in one patch

Changes since v1:
-moved page selection to config_init so it is handled properly
after suspend/resume
-added explicit check for empty sfp_support bitmask

Robert Hancock (3):
  net: phy: at803x: move page selection fix to config_init
  net: phy: at803x: add fiber support
  net: phy: at803x: Support downstream SFP cage

 drivers/net/phy/at803x.c | 146 +++++++++++++++++++++++++++++++++------
 1 file changed, 126 insertions(+), 20 deletions(-)

-- 
2.31.1

[PATCH net-next v4 1/3] net: phy: at803x: move page selection fix to config_init

[Robert Hancock]

The fix to select the copper page on AR8031 was being done in the probe
function rather than config_init, so it would not be redone after resume
from suspend. Move this to config_init so it is always redone when
needed.

Fixes: c329e5afb42f ("net: phy: at803x: select correct page on config init")
Signed-off-by: Robert Hancock <robert.hancock@calian.com>
---
 drivers/net/phy/at803x.c | 40 ++++++++++++++++------------------------
 1 file changed, 16 insertions(+), 24 deletions(-)

diff --git a/drivers/net/phy/at803x.c b/drivers/net/phy/at803x.c
index 5b6c0d120e09..052b2eb9f101 100644
--- a/drivers/net/phy/at803x.c
+++ b/drivers/net/phy/at803x.c
@@ -784,25 +784,7 @@ static int at803x_probe(struct phy_device *phydev)
 			return ret;
 	}
 
-	/* Some bootloaders leave the fiber page selected.
-	 * Switch to the copper page, as otherwise we read
-	 * the PHY capabilities from the fiber side.
-	 */
-	if (phydev->drv->phy_id == ATH8031_PHY_ID) {
-		phy_lock_mdio_bus(phydev);
-		ret = at803x_write_page(phydev, AT803X_PAGE_COPPER);
-		phy_unlock_mdio_bus(phydev);
-		if (ret)
-			goto err;
-	}
-
 	return 0;
-
-err:
-	if (priv->vddio)
-		regulator_disable(priv->vddio);
-
-	return ret;
 }
 
 static void at803x_remove(struct phy_device *phydev)
@@ -912,6 +894,22 @@ static int at803x_config_init(struct phy_device *phydev)
 {
 	int ret;
 
+	if (phydev->drv->phy_id == ATH8031_PHY_ID) {
+		/* Some bootloaders leave the fiber page selected.
+		 * Switch to the copper page, as otherwise we read
+		 * the PHY capabilities from the fiber side.
+		 */
+		phy_lock_mdio_bus(phydev);
+		ret = at803x_write_page(phydev, AT803X_PAGE_COPPER);
+		phy_unlock_mdio_bus(phydev);
+		if (ret)
+			return ret;
+
+		ret = at8031_pll_config(phydev);
+		if (ret < 0)
+			return ret;
+	}
+
 	/* The RX and TX delay default is:
 	 *   after HW reset: RX delay enabled and TX delay disabled
 	 *   after SW reset: RX delay enabled, while TX delay retains the
@@ -941,12 +939,6 @@ static int at803x_config_init(struct phy_device *phydev)
 	if (ret < 0)
 		return ret;
 
-	if (phydev->drv->phy_id == ATH8031_PHY_ID) {
-		ret = at8031_pll_config(phydev);
-		if (ret < 0)
-			return ret;
-	}
-
 	/* Ar803x extended next page bit is enabled by default. Cisco
 	 * multigig switches read this bit and attempt to negotiate 10Gbps
 	 * rates even if the next page bit is disabled. This is incorrect
-- 
2.31.1

[PATCH net-next v4 2/3] net: phy: at803x: add fiber support

[Robert Hancock]

Previously this driver always forced the copper page to be selected,
however for AR8031 in 100Base-FX or 1000Base-X modes, the fiber page
needs to be selected. Set the appropriate mode based on the hardware
mode_cfg strap selection.

Enable the appropriate interrupt bits to detect fiber-side link up
or down events.

Update config_aneg and read_status methods to use the appropriate
Clause 37 calls when fiber mode is in use.

Signed-off-by: Robert Hancock <robert.hancock@calian.com>
---
 drivers/net/phy/at803x.c | 76 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 67 insertions(+), 9 deletions(-)

diff --git a/drivers/net/phy/at803x.c b/drivers/net/phy/at803x.c
index 052b2eb9f101..3f3d4c164df4 100644
--- a/drivers/net/phy/at803x.c
+++ b/drivers/net/phy/at803x.c
@@ -51,6 +51,8 @@
 #define AT803X_INTR_ENABLE_PAGE_RECEIVED	BIT(12)
 #define AT803X_INTR_ENABLE_LINK_FAIL		BIT(11)
 #define AT803X_INTR_ENABLE_LINK_SUCCESS		BIT(10)
+#define AT803X_INTR_ENABLE_LINK_FAIL_BX		BIT(8)
+#define AT803X_INTR_ENABLE_LINK_SUCCESS_BX	BIT(7)
 #define AT803X_INTR_ENABLE_WIRESPEED_DOWNGRADE	BIT(5)
 #define AT803X_INTR_ENABLE_POLARITY_CHANGED	BIT(1)
 #define AT803X_INTR_ENABLE_WOL			BIT(0)
@@ -85,7 +87,17 @@
 #define AT803X_DEBUG_DATA			0x1E
 
 #define AT803X_MODE_CFG_MASK			0x0F
-#define AT803X_MODE_CFG_SGMII			0x01
+#define AT803X_MODE_CFG_BASET_RGMII		0x00
+#define AT803X_MODE_CFG_BASET_SGMII		0x01
+#define AT803X_MODE_CFG_BX1000_RGMII_50OHM	0x02
+#define AT803X_MODE_CFG_BX1000_RGMII_75OHM	0x03
+#define AT803X_MODE_CFG_BX1000_CONV_50OHM	0x04
+#define AT803X_MODE_CFG_BX1000_CONV_75OHM	0x05
+#define AT803X_MODE_CFG_FX100_RGMII_50OHM	0x06
+#define AT803X_MODE_CFG_FX100_CONV_50OHM	0x07
+#define AT803X_MODE_CFG_RGMII_AUTO_MDET		0x0B
+#define AT803X_MODE_CFG_FX100_RGMII_75OHM	0x0E
+#define AT803X_MODE_CFG_FX100_CONV_75OHM	0x0F
 
 #define AT803X_PSSR				0x11	/*PHY-Specific Status Register*/
 #define AT803X_PSSR_MR_AN_COMPLETE		0x0200
@@ -283,6 +295,8 @@ struct at803x_priv {
 	u16 clk_25m_mask;
 	u8 smarteee_lpi_tw_1g;
 	u8 smarteee_lpi_tw_100m;
+	bool is_fiber;
+	bool is_1000basex;
 	struct regulator_dev *vddio_rdev;
 	struct regulator_dev *vddh_rdev;
 	struct regulator *vddio;
@@ -784,7 +798,33 @@ static int at803x_probe(struct phy_device *phydev)
 			return ret;
 	}
 
+	if (phydev->drv->phy_id == ATH8031_PHY_ID) {
+		int ccr = phy_read(phydev, AT803X_REG_CHIP_CONFIG);
+		int mode_cfg;
+
+		if (ccr < 0)
+			goto err;
+		mode_cfg = ccr & AT803X_MODE_CFG_MASK;
+
+		switch (mode_cfg) {
+		case AT803X_MODE_CFG_BX1000_RGMII_50OHM:
+		case AT803X_MODE_CFG_BX1000_RGMII_75OHM:
+			priv->is_1000basex = true;
+			fallthrough;
+		case AT803X_MODE_CFG_FX100_RGMII_50OHM:
+		case AT803X_MODE_CFG_FX100_RGMII_75OHM:
+			priv->is_fiber = true;
+			break;
+		}
+	}
+
 	return 0;
+
+err:
+	if (priv->vddio)
+		regulator_disable(priv->vddio);
+
+	return ret;
 }
 
 static void at803x_remove(struct phy_device *phydev)
@@ -797,6 +837,7 @@ static void at803x_remove(struct phy_device *phydev)
 
 static int at803x_get_features(struct phy_device *phydev)
 {
+	struct at803x_priv *priv = phydev->priv;
 	int err;
 
 	err = genphy_read_abilities(phydev);
@@ -823,12 +864,13 @@ static int at803x_get_features(struct phy_device *phydev)
 	 * As a result of that, ESTATUS_1000_XFULL is set
 	 * to 1 even when operating in copper TP mode.
 	 *
-	 * Remove this mode from the supported link modes,
-	 * as this driver currently only supports copper
-	 * operation.
+	 * Remove this mode from the supported link modes
+	 * when not operating in 1000BaseX mode.
 	 */
-	linkmode_clear_bit(ETHTOOL_LINK_MODE_1000baseX_Full_BIT,
-			   phydev->supported);
+	if (!priv->is_1000basex)
+		linkmode_clear_bit(ETHTOOL_LINK_MODE_1000baseX_Full_BIT,
+				   phydev->supported);
+
 	return 0;
 }
 
@@ -892,15 +934,18 @@ static int at8031_pll_config(struct phy_device *phydev)
 
 static int at803x_config_init(struct phy_device *phydev)
 {
+	struct at803x_priv *priv = phydev->priv;
 	int ret;
 
 	if (phydev->drv->phy_id == ATH8031_PHY_ID) {
 		/* Some bootloaders leave the fiber page selected.
-		 * Switch to the copper page, as otherwise we read
-		 * the PHY capabilities from the fiber side.
+		 * Switch to the appropriate page (fiber or copper), as otherwise we
+		 * read the PHY capabilities from the wrong page.
 		 */
 		phy_lock_mdio_bus(phydev);
-		ret = at803x_write_page(phydev, AT803X_PAGE_COPPER);
+		ret = at803x_write_page(phydev,
+					priv->is_fiber ? AT803X_PAGE_FIBER :
+							 AT803X_PAGE_COPPER);
 		phy_unlock_mdio_bus(phydev);
 		if (ret)
 			return ret;
@@ -959,6 +1004,7 @@ static int at803x_ack_interrupt(struct phy_device *phydev)
 
 static int at803x_config_intr(struct phy_device *phydev)
 {
+	struct at803x_priv *priv = phydev->priv;
 	int err;
 	int value;
 
@@ -975,6 +1021,10 @@ static int at803x_config_intr(struct phy_device *phydev)
 		value |= AT803X_INTR_ENABLE_DUPLEX_CHANGED;
 		value |= AT803X_INTR_ENABLE_LINK_FAIL;
 		value |= AT803X_INTR_ENABLE_LINK_SUCCESS;
+		if (priv->is_fiber) {
+			value |= AT803X_INTR_ENABLE_LINK_FAIL_BX;
+			value |= AT803X_INTR_ENABLE_LINK_SUCCESS_BX;
+		}
 
 		err = phy_write(phydev, AT803X_INTR_ENABLE, value);
 	} else {
@@ -1107,8 +1157,12 @@ static int at803x_read_specific_status(struct phy_device *phydev)
 
 static int at803x_read_status(struct phy_device *phydev)
 {
+	struct at803x_priv *priv = phydev->priv;
 	int err, old_link = phydev->link;
 
+	if (priv->is_1000basex)
+		return genphy_c37_read_status(phydev);
+
 	/* Update the link, but return if there was an error */
 	err = genphy_update_link(phydev);
 	if (err)
@@ -1162,6 +1216,7 @@ static int at803x_config_mdix(struct phy_device *phydev, u8 ctrl)
 
 static int at803x_config_aneg(struct phy_device *phydev)
 {
+	struct at803x_priv *priv = phydev->priv;
 	int ret;
 
 	ret = at803x_config_mdix(phydev, phydev->mdix_ctrl);
@@ -1178,6 +1233,9 @@ static int at803x_config_aneg(struct phy_device *phydev)
 			return ret;
 	}
 
+	if (priv->is_1000basex)
+		return genphy_c37_config_aneg(phydev);
+
 	/* Do not restart auto-negotiation by setting ret to 0 defautly,
 	 * when calling __genphy_config_aneg later.
 	 */
-- 
2.31.1

[PATCH net-next v4 3/3] net: phy: at803x: Support downstream SFP cage

[Robert Hancock]

Add support for downstream SFP cages for AR8031 and AR8033. This is
primarily intended for fiber modules or direct-attach cables, however
copper modules which work in 1000Base-X mode may also function. Such
modules are allowed with a warning.

Signed-off-by: Robert Hancock <robert.hancock@calian.com>
---
 drivers/net/phy/at803x.c | 56 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/drivers/net/phy/at803x.c b/drivers/net/phy/at803x.c
index 3f3d4c164df4..f504fe536fe4 100644
--- a/drivers/net/phy/at803x.c
+++ b/drivers/net/phy/at803x.c
@@ -19,6 +19,8 @@
 #include <linux/regulator/of_regulator.h>
 #include <linux/regulator/driver.h>
 #include <linux/regulator/consumer.h>
+#include <linux/phylink.h>
+#include <linux/sfp.h>
 #include <dt-bindings/net/qca-ar803x.h>
 
 #define AT803X_SPECIFIC_FUNCTION_CONTROL	0x10
@@ -664,6 +666,55 @@ static int at8031_register_regulators(struct phy_device *phydev)
 	return 0;
 }
 
+static int at803x_sfp_insert(void *upstream, const struct sfp_eeprom_id *id)
+{
+	struct phy_device *phydev = upstream;
+	__ETHTOOL_DECLARE_LINK_MODE_MASK(phy_support);
+	__ETHTOOL_DECLARE_LINK_MODE_MASK(sfp_support);
+	phy_interface_t iface;
+
+	linkmode_zero(phy_support);
+	phylink_set(phy_support, 1000baseX_Full);
+	phylink_set(phy_support, 1000baseT_Full);
+	phylink_set(phy_support, Autoneg);
+	phylink_set(phy_support, Pause);
+	phylink_set(phy_support, Asym_Pause);
+
+	linkmode_zero(sfp_support);
+	sfp_parse_support(phydev->sfp_bus, id, sfp_support);
+	/* Some modules support 10G modes as well as others we support.
+	 * Mask out non-supported modes so the correct interface is picked.
+	 */
+	linkmode_and(sfp_support, phy_support, sfp_support);
+
+	if (linkmode_empty(sfp_support)) {
+		dev_err(&phydev->mdio.dev, "incompatible SFP module inserted\n");
+		return -EINVAL;
+	}
+
+	iface = sfp_select_interface(phydev->sfp_bus, sfp_support);
+
+	/* Only 1000Base-X is supported by AR8031/8033 as the downstream SerDes
+	 * interface for use with SFP modules.
+	 * However, some copper modules detected as having a preferred SGMII
+	 * interface do default to and function in 1000Base-X mode, so just
+	 * print a warning and allow such modules, as they may have some chance
+	 * of working.
+	 */
+	if (iface == PHY_INTERFACE_MODE_SGMII)
+		dev_warn(&phydev->mdio.dev, "module may not function if 1000Base-X not supported\n");
+	else if (iface != PHY_INTERFACE_MODE_1000BASEX)
+		return -EINVAL;
+
+	return 0;
+}
+
+static const struct sfp_upstream_ops at803x_sfp_ops = {
+	.attach = phy_sfp_attach,
+	.detach = phy_sfp_detach,
+	.module_insert = at803x_sfp_insert,
+};
+
 static int at803x_parse_dt(struct phy_device *phydev)
 {
 	struct device_node *node = phydev->mdio.dev.of_node;
@@ -771,6 +822,11 @@ static int at803x_parse_dt(struct phy_device *phydev)
 			phydev_err(phydev, "failed to get VDDIO regulator\n");
 			return PTR_ERR(priv->vddio);
 		}
+
+		/* Only AR8031/8033 support 1000Base-X for SFP modules */
+		ret = phy_sfp_probe(phydev, &at803x_sfp_ops);
+		if (ret < 0)
+			return ret;
 	}
 
 	return 0;
-- 
2.31.1

Re: [PATCH net-next v4 0/3] at803x fiber/SFP support

[Andrey Jr. Melnikov]

On Tue, Jan 25, 2022 at 10:54:07AM -0600, Robert Hancock wrote:
> Add support for 1000Base-X fiber modes to the at803x PHY driver, as
> well as support for connecting a downstream SFP cage.
> 
> Changes since v3:
> -Renamed some constants with OHM suffix for clarity
> 
> Changes since v2:
> -fixed tabs/spaces issue in one patch
> 
> Changes since v1:
> -moved page selection to config_init so it is handled properly
> after suspend/resume
> -added explicit check for empty sfp_support bitmask
> 
> Robert Hancock (3):
>   net: phy: at803x: move page selection fix to config_init
>   net: phy: at803x: add fiber support
>   net: phy: at803x: Support downstream SFP cage
> 
Backported this series to 5.15.16 and tested on Ubiquiti EdgeRouter X SFP
hardware. Optical SFP modules working without problems, but cooper SFP with
Marvell 88E1111 not work - link is up but no packets sent/recieved via
interface.

# dmesg | grep -iE '(dsa|eth|mdio|sfp)'
[    0.000000] MIPS: machine is Ubiquiti EdgeRouter X SFP
[    1.349162] gpio-7 (sfp_i2c_clk_gate): hogged as output/high
[    4.685535] libphy: Fixed MDIO Bus: probed
[    4.732207] libphy: mdio: probed
[    4.739151] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module
[    4.768955] mtk_soc_eth 1e100000.ethernet dsa: mediatek frame engine at 0xbe100000, irq 20
[    4.852254] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module
[    4.973393] mt7530 mdio-bus:00 eth0 (uninitialized): PHY [mt7530-0:00] driver [Generic PHY] (irq=POLL)
[    4.993676] mt7530 mdio-bus:00 eth1 (uninitialized): PHY [mt7530-0:01] driver [Generic PHY] (irq=POLL)
[    5.014054] mt7530 mdio-bus:00 eth2 (uninitialized): PHY [mt7530-0:02] driver [Generic PHY] (irq=POLL)
[    5.034405] mt7530 mdio-bus:00 eth3 (uninitialized): PHY [mt7530-0:03] driver [Generic PHY] (irq=POLL)
[    5.054704] mt7530 mdio-bus:00 eth4 (uninitialized): PHY [mt7530-0:04] driver [Generic PHY] (irq=POLL)
[    5.131850] mt7530 mdio-bus:00 eth5 (uninitialized): PHY [mdio-bus:07] driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)
[    5.155828] mt7530 mdio-bus:00: configuring for fixed/rgmii link mode
[    5.172290] DSA: tree 0 setup
[    5.179302] mt7530 mdio-bus:00: Link is Up - 1Gbps/Full - flow control off
[    9.103291] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii link mode
[    9.119230] device dsa entered promiscuous mode
[    9.128654] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full - flow control rx/tx
[    9.131563] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode
[    9.158926] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready
[   13.306590] device dsa left promiscuous mode
[   14.587953] libphy: SFP I2C Bus: probed
[   14.751420] sfp sfp_eth5: Host maximum power 1.0W
[   14.821296] sfp sfp_eth5: No tx_disable pin: SFP modules will always be emitting.
[   15.691261] sfp sfp_eth5: module Gateray          GR-S1-RJ         rev A    sn X201604162293    dc 160422  
[   15.710686] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not function if 1000Base-X not supported
[   36.440072] mtk_soc_eth 1e100000.ethernet dsa: Link is Down
[   36.461374] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii link mode
[   36.477856] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full - flow control rx/tx
[   36.495142] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready
[   36.508246] device dsa entered promiscuous mode
[   36.517976] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode
[   36.532794] br-lan: port 1(eth1) entered blocking state
[   36.543322] br-lan: port 1(eth1) entered disabled state
[   36.555768] device eth1 entered promiscuous mode
[   36.592332] mt7530 mdio-bus:00 eth2: configuring for phy/gmii link mode
[   36.607113] br-lan: port 2(eth2) entered blocking state
[   36.617651] br-lan: port 2(eth2) entered disabled state
[   36.630268] device eth2 entered promiscuous mode
[   36.655233] mt7530 mdio-bus:00 eth3: configuring for phy/gmii link mode
[   36.669957] br-lan: port 3(eth3) entered blocking state
[   36.680454] br-lan: port 3(eth3) entered disabled state
[   36.693351] device eth3 entered promiscuous mode
[   36.718596] mt7530 mdio-bus:00 eth4: configuring for phy/gmii link mode
[   36.733435] br-lan: port 4(eth4) entered blocking state
[   36.743977] br-lan: port 4(eth4) entered disabled state
[   36.756966] device eth4 entered promiscuous mode
[   36.779983] mt7530 mdio-bus:00 eth5: configuring for phy/rgmii-rxid link mode
[   36.795073] mt7530 mdio-bus:00 eth5: No phy led trigger registered for speed(-1)
[   36.810051] br-lan: port 5(eth5) entered blocking state
[   36.810277] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow control off

Link is up - but what mode is selected ?? No packets received from
interface, remote side report 1000/Full speed negotiated, TX counters
increased.

[   36.820606] br-lan: port 5(eth5) entered disabled state
[   36.849515] device eth5 entered promiscuous mode
[   36.862142] br-lan: port 5(eth5) entered blocking state
[   36.872673] br-lan: port 5(eth5) entered forwarding state
[   37.260595] mt7530 mdio-bus:00 eth0: configuring for phy/gmii link mode
[   40.362029] mt7530 mdio-bus:00 eth0: Link is Up - 1Gbps/Full - flow control off
[   40.376656] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  670.201487] mt7530 mdio-bus:00 eth5: Link is Down
[  670.211423] br-lan: port 5(eth5) entered disabled state
[  671.241339] mt7530 mdio-bus:00 eth5: No phy led trigger registered for speed(-1)
[  671.256789] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow control off
[  671.272343] br-lan: port 5(eth5) entered blocking state
[  671.282821] br-lan: port 5(eth5) entered forwarding state
[  671.471210] sfp sfp_eth5: module removed

Now, I'm removed copper SFP, and install optilcal SFP:

[  672.281451] mt7530 mdio-bus:00 eth5: Link is Down
[  672.290943] br-lan: port 5(eth5) entered disabled state
[  688.921868] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Full - flow control off
[  688.936512] br-lan: port 5(eth5) entered blocking state
[  688.946944] br-lan: port 5(eth5) entered forwarding state
[  689.591192] sfp sfp_eth5: module Gateray          GR-S1-X3120L-D   rev A    sn X201602220961    dc 160229  

Link is up, 1000/Full, packets traverse in both directions. Now, insert back
copper SFP:

[  731.561446] mt7530 mdio-bus:00 eth5: Link is Down
[  731.570947] br-lan: port 5(eth5) entered disabled state
[  733.841609] sfp sfp_eth5: module removed
[  751.321576] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow control off

Whoa, link up with 1Gbps speed.

[  751.336733] br-lan: port 5(eth5) entered blocking state
[  751.347167] br-lan: port 5(eth5) entered forwarding state
[  751.961193] sfp sfp_eth5: module Gateray          GR-S1-RJ         rev A    sn X201604162293    dc 160422  
[  751.980667] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not function if 1000Base-X not supported
[  753.401483] mt7530 mdio-bus:00 eth5: Link is Down
[  753.410984] br-lan: port 5(eth5) entered disabled state
[  754.441627] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow control off
[  754.457144] br-lan: port 5(eth5) entered blocking state
[  754.467619] br-lan: port 5(eth5) entered forwarding state

Now packets traverse in both directions for copper module too.

Can someone explain - why copper module not work from boot? And how controll 88E1111
inside SFP ?

Re: [PATCH net-next v4 0/3] at803x fiber/SFP support

[Russell King (Oracle)]

On Wed, Jan 26, 2022 at 04:56:40PM +0300, Andrey Jr. Melnikov wrote:
> On Tue, Jan 25, 2022 at 10:54:07AM -0600, Robert Hancock wrote:
> > Add support for 1000Base-X fiber modes to the at803x PHY driver, as
> > well as support for connecting a downstream SFP cage.
> > 
> > Changes since v3:
> > -Renamed some constants with OHM suffix for clarity
> > 
> > Changes since v2:
> > -fixed tabs/spaces issue in one patch
> > 
> > Changes since v1:
> > -moved page selection to config_init so it is handled properly
> > after suspend/resume
> > -added explicit check for empty sfp_support bitmask
> > 
> > Robert Hancock (3):
> >   net: phy: at803x: move page selection fix to config_init
> >   net: phy: at803x: add fiber support
> >   net: phy: at803x: Support downstream SFP cage
> > 
> Backported this series to 5.15.16 and tested on Ubiquiti EdgeRouter X SFP
> hardware. Optical SFP modules working without problems, but cooper SFP with
> Marvell 88E1111 not work - link is up but no packets sent/recieved via
> interface.

Could be that the 88E1111 is in SGMII mode, and with the 803x in
1000base-x mode, they just don't want to talk... and having a link up
event with an optical SFP (in 1000base-x mode) changes the state in the
803x so it somehow works.

SGMII and 1000base-x are similar enough that it can appear to work at
gigabit speeds if everything is just right.

> Can someone explain - why copper module not work from boot? And how controll 88E1111
> inside SFP ?

The Linux networking layer only permits one PHY per network interface,
so in the case of a SFP with a PHY connected to another PHY, the only
PHY we can expose is the PHY closest to the network interface. There
is no way e.g. via ethtool to be able to direct the PHY specific
ethtool calls to a specific PHY.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH net-next v4 0/3] at803x fiber/SFP support

[Robert Hancock]

On Wed, 2022-01-26 at 16:56 +0300, Andrey Jr. Melnikov wrote:
> On Tue, Jan 25, 2022 at 10:54:07AM -0600, Robert Hancock wrote:
> > Add support for 1000Base-X fiber modes to the at803x PHY driver, as
> > well as support for connecting a downstream SFP cage.
> > 
> > Changes since v3:
> > -Renamed some constants with OHM suffix for clarity
> > 
> > Changes since v2:
> > -fixed tabs/spaces issue in one patch
> > 
> > Changes since v1:
> > -moved page selection to config_init so it is handled properly
> > after suspend/resume
> > -added explicit check for empty sfp_support bitmask
> > 
> > Robert Hancock (3):
> >   net: phy: at803x: move page selection fix to config_init
> >   net: phy: at803x: add fiber support
> >   net: phy: at803x: Support downstream SFP cage
> > 
> Backported this series to 5.15.16 and tested on Ubiquiti EdgeRouter X SFP
> hardware. Optical SFP modules working without problems, but cooper SFP with
> Marvell 88E1111 not work - link is up but no packets sent/recieved via
> interface.

Interesting, apparently we weren't the only ones weird enough to use an AR8031
for an SFP module..

> 
> # dmesg | grep -iE '(dsa|eth|mdio|sfp)'
> [    0.000000] MIPS: machine is Ubiquiti EdgeRouter X SFP
> [    1.349162] gpio-7 (sfp_i2c_clk_gate): hogged as output/high
> [    4.685535] libphy: Fixed MDIO Bus: probed
> [    4.732207] libphy: mdio: probed
> [    4.739151] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module
> [    4.768955] mtk_soc_eth 1e100000.ethernet dsa: mediatek frame engine at
> 0xbe100000, irq 20
> [    4.852254] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module
> [    4.973393] mt7530 mdio-bus:00 eth0 (uninitialized): PHY [mt7530-0:00]
> driver [Generic PHY] (irq=POLL)
> [    4.993676] mt7530 mdio-bus:00 eth1 (uninitialized): PHY [mt7530-0:01]
> driver [Generic PHY] (irq=POLL)
> [    5.014054] mt7530 mdio-bus:00 eth2 (uninitialized): PHY [mt7530-0:02]
> driver [Generic PHY] (irq=POLL)
> [    5.034405] mt7530 mdio-bus:00 eth3 (uninitialized): PHY [mt7530-0:03]
> driver [Generic PHY] (irq=POLL)
> [    5.054704] mt7530 mdio-bus:00 eth4 (uninitialized): PHY [mt7530-0:04]
> driver [Generic PHY] (irq=POLL)
> [    5.131850] mt7530 mdio-bus:00 eth5 (uninitialized): PHY [mdio-bus:07]
> driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)
> [    5.155828] mt7530 mdio-bus:00: configuring for fixed/rgmii link mode
> [    5.172290] DSA: tree 0 setup
> [    5.179302] mt7530 mdio-bus:00: Link is Up - 1Gbps/Full - flow control off
> [    9.103291] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii
> link mode
> [    9.119230] device dsa entered promiscuous mode
> [    9.128654] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full -
> flow control rx/tx
> [    9.131563] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode
> [    9.158926] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready
> [   13.306590] device dsa left promiscuous mode
> [   14.587953] libphy: SFP I2C Bus: probed
> [   14.751420] sfp sfp_eth5: Host maximum power 1.0W
> [   14.821296] sfp sfp_eth5: No tx_disable pin: SFP modules will always be
> emitting.
> [   15.691261] sfp sfp_eth5: module Gateray          GR-S1-RJ         rev
> A    sn X201604162293    dc 160422  
> [   15.710686] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not
> function if 1000Base-X not supported
> [   36.440072] mtk_soc_eth 1e100000.ethernet dsa: Link is Down
> [   36.461374] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii
> link mode
> [   36.477856] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full -
> flow control rx/tx
> [   36.495142] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready
> [   36.508246] device dsa entered promiscuous mode
> [   36.517976] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode
> [   36.532794] br-lan: port 1(eth1) entered blocking state
> [   36.543322] br-lan: port 1(eth1) entered disabled state
> [   36.555768] device eth1 entered promiscuous mode
> [   36.592332] mt7530 mdio-bus:00 eth2: configuring for phy/gmii link mode
> [   36.607113] br-lan: port 2(eth2) entered blocking state
> [   36.617651] br-lan: port 2(eth2) entered disabled state
> [   36.630268] device eth2 entered promiscuous mode
> [   36.655233] mt7530 mdio-bus:00 eth3: configuring for phy/gmii link mode
> [   36.669957] br-lan: port 3(eth3) entered blocking state
> [   36.680454] br-lan: port 3(eth3) entered disabled state
> [   36.693351] device eth3 entered promiscuous mode
> [   36.718596] mt7530 mdio-bus:00 eth4: configuring for phy/gmii link mode
> [   36.733435] br-lan: port 4(eth4) entered blocking state
> [   36.743977] br-lan: port 4(eth4) entered disabled state
> [   36.756966] device eth4 entered promiscuous mode
> [   36.779983] mt7530 mdio-bus:00 eth5: configuring for phy/rgmii-rxid link
> mode
> [   36.795073] mt7530 mdio-bus:00 eth5: No phy led trigger registered for
> speed(-1)
> [   36.810051] br-lan: port 5(eth5) entered blocking state
> [   36.810277] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow
> control off
> 
> Link is up - but what mode is selected ?? No packets received from
> interface, remote side report 1000/Full speed negotiated, TX counters
> increased.

I think you're running into the inherent limitation of this setup (which the
"module may not function if 1000Base-X not supported" message is trying to tell
you) - the AR8031 only supports 1000Base-X on this interface, not SGMII. Many
(perhaps most) copper SFP modules use or default to SGMII mode because it
allows the link to work at 100 or 10 Mbps speeds as well as 1 Gbps.

The difference between 1000Base-X and SGMII, at least when at 1000 Mbps speeds,
is mostly in the auto-negotiation, so we've found that in many cases you can
get it to work by using ethtool to disable auto-negotiation and forcing 1000
Mbps full duplex mode on the interface. It seems that (at least from what we
have seen) the SFP module side will often decide to give up on auto-negotiation 
when it sees a link and just run at the proper rate - that doesn't generally
affect the separate negotiation that happens on the copper link to whatever is
connected on the other end. Obviously if the link partner connected on the
copper side doesn't support 1 Gbps it won't work.

I'm not entirely sure why it worked after the module was unplugged and plugged
back in however. But my guess it's related to this issue somehow.

> 
> [   36.820606] br-lan: port 5(eth5) entered disabled state
> [   36.849515] device eth5 entered promiscuous mode
> [   36.862142] br-lan: port 5(eth5) entered blocking state
> [   36.872673] br-lan: port 5(eth5) entered forwarding state
> [   37.260595] mt7530 mdio-bus:00 eth0: configuring for phy/gmii link mode
> [   40.362029] mt7530 mdio-bus:00 eth0: Link is Up - 1Gbps/Full - flow
> control off
> [   40.376656] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [  670.201487] mt7530 mdio-bus:00 eth5: Link is Down
> [  670.211423] br-lan: port 5(eth5) entered disabled state
> [  671.241339] mt7530 mdio-bus:00 eth5: No phy led trigger registered for
> speed(-1)
> [  671.256789] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow
> control off
> [  671.272343] br-lan: port 5(eth5) entered blocking state
> [  671.282821] br-lan: port 5(eth5) entered forwarding state
> [  671.471210] sfp sfp_eth5: module removed
> 
> Now, I'm removed copper SFP, and install optilcal SFP:
> 
> [  672.281451] mt7530 mdio-bus:00 eth5: Link is Down
> [  672.290943] br-lan: port 5(eth5) entered disabled state
> [  688.921868] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Full - flow
> control off
> [  688.936512] br-lan: port 5(eth5) entered blocking state
> [  688.946944] br-lan: port 5(eth5) entered forwarding state
> [  689.591192] sfp sfp_eth5: module Gateray          GR-S1-X3120L-D   rev
> A    sn X201602220961    dc 160229  
> 
> Link is up, 1000/Full, packets traverse in both directions. Now, insert back
> copper SFP:
> 
> [  731.561446] mt7530 mdio-bus:00 eth5: Link is Down
> [  731.570947] br-lan: port 5(eth5) entered disabled state
> [  733.841609] sfp sfp_eth5: module removed
> [  751.321576] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow
> control off
> 
> Whoa, link up with 1Gbps speed.
> 
> [  751.336733] br-lan: port 5(eth5) entered blocking state
> [  751.347167] br-lan: port 5(eth5) entered forwarding state
> [  751.961193] sfp sfp_eth5: module Gateray          GR-S1-RJ         rev
> A    sn X201604162293    dc 160422  
> [  751.980667] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not
> function if 1000Base-X not supported
> [  753.401483] mt7530 mdio-bus:00 eth5: Link is Down
> [  753.410984] br-lan: port 5(eth5) entered disabled state
> [  754.441627] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow
> control off
> [  754.457144] br-lan: port 5(eth5) entered blocking state
> [  754.467619] br-lan: port 5(eth5) entered forwarding state
> 
> Now packets traverse in both directions for copper module too.
> 
> Can someone explain - why copper module not work from boot? And how controll
> 88E1111
> inside SFP ?

That's the other issue with this setup with copper modules - you end up with
the AR8031 PHY device, which is basically acting as an RGMII etc. to 1000Base-X 
converter, and the PHY device inside the module which is the one talking to the
outside world. As Russell mentioned, this sort of dual-PHY setup isn't handled
well in Linux right now - in this AR8031 setup, the kernel talks to the SFP
module enough to identify it, but the driver for the module PHY is not probed.
Even if it was, it can't really be hooked up to the network interface because
the network layer only supports binding one PHY to a device, so there's not
really a way to control it.

There's support for MAC drivers which have an integrated or semi-integrated PCS
layer which handles 1000Base-X or SGMII - some of which look like a PHY in that
they support standard PHY registers and/or have an MDIO interface - but that
essentially is handled internally to phylink and the MAC driver and the "PHY"-
ness isn't really exposed to the rest of the kernel. But in this case, the MAC
driver has no real knowledge of the downstream SFP contraption that's been
attached on its PHY interface, and the only PHY the kernel associates with the
interface is the AR8031.

In the AR8031 case however, this isn't really a big issue because the inherent
limitation of only supporting 1000Base-X is more significant than the
limitations the kernel side is imposing..

-- 
Robert Hancock
Senior Hardware Designer, Calian Advanced Technologies
www.calian.com

Re: [PATCH net-next v4 0/3] at803x fiber/SFP support

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net-next.git (master)
by David S. Miller <davem@davemloft.net>:

On Tue, 25 Jan 2022 10:54:07 -0600 you wrote:
> Add support for 1000Base-X fiber modes to the at803x PHY driver, as
> well as support for connecting a downstream SFP cage.
> 
> Changes since v3:
> -Renamed some constants with OHM suffix for clarity
> 
> Changes since v2:
> -fixed tabs/spaces issue in one patch
> 
> [...]

Here is the summary with links:
  - [net-next,v4,1/3] net: phy: at803x: move page selection fix to config_init
    https://git.kernel.org/netdev/net-next/c/4f3a00c7f5b2
  - [net-next,v4,2/3] net: phy: at803x: add fiber support
    https://git.kernel.org/netdev/net-next/c/3265f4218878
  - [net-next,v4,3/3] net: phy: at803x: Support downstream SFP cage
    https://git.kernel.org/netdev/net-next/c/dc4d5fcc5d36

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html