mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Marek Behún]

Hello,

it seems that inband AN is broken on Amethyst when on 2500base-x mode.

Even SERDES scripts for Amethyst from Marvell has autonegotiation
disabled for 2500base-x mode. For all the other supported Serdes modes
autonegotiation is enabled in these scripts.

The current implementation in mv88e6390_serdes_pcs_config() enables
autonegotiation if phylink_autoneg_inband(mode) is true:

  if (phylink_autoneg_inband(mode))
    bmcr = val | BMCR_ANENABLE;
  else
    bmcr = val & ~BMCR_ANENABLE;

But for PHY_INTERFACE_MODE_2500BASEX this is broken on Amethyst. The
2500base-x mode seems to work only with autoneg disabled.

The result is that when I connect via a passive SFP cable Amethyst
serdes port with a Peridot serdes port, they will not link. If I
disable autonegotiation on both sides, they will link, though.

What is strange is that if I don't use Peridot, but connect the SFP
directly to Serdes on Armada 3720, where the mvneta driver also enables
autonegotiation for 2500base-x mode, they will link even if Amethyst
does not enable 2500base-x.

To summarize:
	Amethyst  <->	Peridot
	AN -		AN -		works
	AN -		AN +		does not work

	Amethyst  <->	Armada 3720 serdes
	AN -		AN +		works

(It is possible that Marvell may find some workaround by touch some
 undocumented registers, to solve this. I will try to open a bug
 report.)

Should we just print an error in the serdes_pcs_config method if inband
autonegotiation is being requested?

phylink's code currently allows connecting SFPs in non MLO_AN_INBAND
mode only for when there is Broadcom BCM84881 PHY inside the SFP (by
method phylink_phy_no_inband() in phylink.c).

I wonder whether we can somehow in a sane way implement code to inform
phylink from the mv88e6xxx driver that inband is not supported for the
specific mode. Maybe the .mac_config/.pcs_config method could return an
error indicating this? Or the mv88e6xxx driver can just print an error
that the mode is not supported, and try to ask the user to disable AN?
That would need implementing this in ethtool for SFP, though.

What do you guys think?

Marek

Re: mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Marek Behun]

It seems this problem can manifest somehow with Peridot as well, at
least when in combination with 88X3310 PHY.

I will do some more experiments and try to come up with a solution.

Marek

Re: mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Russell King - ARM Linux admin]

On Wed, Jan 13, 2021 at 01:18:23AM +0100, Marek Behún wrote:
> Hello,
> 
> it seems that inband AN is broken on Amethyst when on 2500base-x mode.
> 
> Even SERDES scripts for Amethyst from Marvell has autonegotiation
> disabled for 2500base-x mode. For all the other supported Serdes modes
> autonegotiation is enabled in these scripts.
> 
> The current implementation in mv88e6390_serdes_pcs_config() enables
> autonegotiation if phylink_autoneg_inband(mode) is true:
> 
>   if (phylink_autoneg_inband(mode))
>     bmcr = val | BMCR_ANENABLE;
>   else
>     bmcr = val & ~BMCR_ANENABLE;
> 
> But for PHY_INTERFACE_MODE_2500BASEX this is broken on Amethyst. The
> 2500base-x mode seems to work only with autoneg disabled.
> 
> The result is that when I connect via a passive SFP cable Amethyst
> serdes port with a Peridot serdes port, they will not link. If I
> disable autonegotiation on both sides, they will link, though.
> 
> What is strange is that if I don't use Peridot, but connect the SFP
> directly to Serdes on Armada 3720, where the mvneta driver also enables
> autonegotiation for 2500base-x mode, they will link even if Amethyst
> does not enable 2500base-x.
> 
> To summarize:
> 	Amethyst  <->	Peridot
> 	AN -		AN -		works
> 	AN -		AN +		does not work
> 
> 	Amethyst  <->	Armada 3720 serdes
> 	AN -		AN +		works
> 
> (It is possible that Marvell may find some workaround by touch some
>  undocumented registers, to solve this. I will try to open a bug
>  report.)
> 
> Should we just print an error in the serdes_pcs_config method if inband
> autonegotiation is being requested?
> 
> phylink's code currently allows connecting SFPs in non MLO_AN_INBAND
> mode only for when there is Broadcom BCM84881 PHY inside the SFP (by
> method phylink_phy_no_inband() in phylink.c).
> 
> I wonder whether we can somehow in a sane way implement code to inform
> phylink from the mv88e6xxx driver that inband is not supported for the
> specific mode. Maybe the .mac_config/.pcs_config method could return an
> error indicating this? Or the mv88e6xxx driver can just print an error
> that the mode is not supported, and try to ask the user to disable AN?
> That would need implementing this in ethtool for SFP, though.
> 
> What do you guys think?

It's regrettable that some have decided not to have working in-band
AN for 2500base-X, since it prevents the automatic use of pause modes -
which is essentially the only use of in-band negotiation with
1000base-X since many Gigabit MACs do not support half duplex.

I don't know whether the 88x3310 on the host side uses AN or not for
2500base-X - that detail isn't mentioned in the datasheet, and I don't
have a setup that I could test that with.

I had assumed that most people would implement 2500base-X as merely an
upclocked 1000base-X, as mvneta does - unfortunately, there is /no/
published standard for 2500base-X, so we're left guessing what this
should be from the implementations available at the time.

As you have found, whether AN is supported at 2500base-X appears to be
pretty random.

However, phylib has no way to report whether a PHY wants to use in-band
AN to the MAC. Hence the hack in the phylink code for BCM84881. We
really need a better way for phylib to communicate the capabilities of
the PHY to its user (things like which interface modes are supported,
which interface modes may be dynamically switched between, and whether
they can use in-band AN, and whether in-band AN bypass is supported).
Also similar properties for MAC/PCS drivers, then phylink can work out
what should be done.

That still leaves an issue for SFP modules - when they're capable of
supporting 2500base-X, we have no knowledge of the remote side. This is
where knowing whether the MAC/PCS supports in-band AN bypass or not
matters - if it doesn't then disabling in-band AN for 2500base-X may
make sense, otherwise having in-and AN enabled but with bypass enabled
would probably be sensible.

Right now, we just don't have the information to make the correct
decisions.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Marek Behún]

On Wed, 13 Jan 2021 10:28:49 +0000
Russell King - ARM Linux admin <linux@armlinux.org.uk> wrote:

> I don't know whether the 88x3310 on the host side uses AN or not for
> 2500base-X - that detail isn't mentioned in the datasheet, and I don't
> have a setup that I could test that with.

It does not. I have been poking the registers on 88x3310, and also on
Peridot and Amethyst SERDES PHYs when cmode on the switch is set to
2500base-x, and by default the inband AN is disabled.

Enabling it on the PHY in Amethyst does not work at all. On Peridot
your code in mv88e6xxx enables it, but when 88x3310 is connected to
Peridot SERDES, it does not work until the AN is disabled on Peridot.

Enabling the AN on 2500base-x mode on 88x3310 does not seem to work.

I will do some more tests by poking the registers and report this.

Marek

Re: mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Marek Behún]

OK, so I did some tests with Peridot and 88X3310:

On 88E6390 switch, when CMODE in the port register for a port capable of
SerDes is set to 1000base-x, the switch self-configures the SerDes PHY
to inband AN:
  register 4.2000 has value 0x1940

but when CMODE is set to 2500base-x, the switch self-configure the PHY
without inband AN:
  register 4.2000 has value 0x0940

Also the 88X3310 PHY, when configured with MACTYPE=4
 (that is the mode that switches host interface between
  10gbase-r/5gbase-r/2500base-x/sgmii, depending on copper speed)
and when copper speed is 2500, the PHY self-configures without inband
AN:
  register 4.2000 has value 0x0140

It seems to me that on these Marvell devices, they consider 2500base-x
not capable of inband AN and disable it by default.

Moreover when the PHY has disabled inband AN and the Peridot switch has
it enabled (by software), they won't link. I tried enabling the inband
AN on the PHY, but it does not seem to work. Peridot can only
communicate with the PHY in 2500base-x with inband AN disabled.

This means that the commit
  a5a6858b793ff ("net: dsa: mv88e6xxx: extend phylink to Serdes PHYs")
causes a regression, since the code started enabling inband AN on
2500base-x mode on the mv88e6390 family, and they stopped working with
the PHY.

Russell, could we, for now, just edit the code so that when
  mv88e6390_serdes_pcs_config
is being configured with inband mode in 2500base-x, the inband mode
won't be enabled and the function will print a warning?
This could come with a Fixes tag so that it is backported to stable.

Afterwards we can work on refactoring the phylink code so that either
the driver can inform phylink whether 2500base-x inband AN is supported,
or maybe we can determine from some documentation or whatnot whether
inband AN is supported on 2500base-x at all.

Marek

Re: mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Russell King - ARM Linux admin]

On Wed, Jan 13, 2021 at 09:08:39PM +0100, Marek Behún wrote:
> OK, so I did some tests with Peridot and 88X3310:
> 
> On 88E6390 switch, when CMODE in the port register for a port capable of
> SerDes is set to 1000base-x, the switch self-configures the SerDes PHY
> to inband AN:
>   register 4.2000 has value 0x1940
> 
> but when CMODE is set to 2500base-x, the switch self-configure the PHY
> without inband AN:
>   register 4.2000 has value 0x0940
> 
> Also the 88X3310 PHY, when configured with MACTYPE=4
>  (that is the mode that switches host interface between
>   10gbase-r/5gbase-r/2500base-x/sgmii, depending on copper speed)
> and when copper speed is 2500, the PHY self-configures without inband
> AN:
>   register 4.2000 has value 0x0140
> 
> It seems to me that on these Marvell devices, they consider 2500base-x
> not capable of inband AN and disable it by default.
> 
> Moreover when the PHY has disabled inband AN and the Peridot switch has
> it enabled (by software), they won't link. I tried enabling the inband
> AN on the PHY, but it does not seem to work. Peridot can only
> communicate with the PHY in 2500base-x with inband AN disabled.
> 
> This means that the commit
>   a5a6858b793ff ("net: dsa: mv88e6xxx: extend phylink to Serdes PHYs")
> causes a regression, since the code started enabling inband AN on
> 2500base-x mode on the mv88e6390 family, and they stopped working with
> the PHY.
> 
> Russell, could we, for now, just edit the code so that when
>   mv88e6390_serdes_pcs_config
> is being configured with inband mode in 2500base-x, the inband mode
> won't be enabled and the function will print a warning?
> This could come with a Fixes tag so that it is backported to stable.

I don't see any other easy option, so yes, please do that.

> Afterwards we can work on refactoring the phylink code so that either
> the driver can inform phylink whether 2500base-x inband AN is supported,
> or maybe we can determine from some documentation or whatnot whether
> inband AN is supported on 2500base-x at all.

I suspect there is no definitive documentation on exactly what
2500base-x actually is. I suspect it may just be easier to turn off AN
for 2500base-x everywhere, so at least all Linux systems are compatible
irrespective of the hardware.

Yes, it means losing pause negotiation, and people will have to
manually set pause on each end.

One thing that I don't know is whether the GPON SFP ONT modules that
use 2500base-x will still function with AN disabled - although I have
the modules, it appeared that they both needed a connection to the ONU
to switch from 1000base-x to 2500base-x on the host side - and as I
don't have an ONU I can test with, I have no way to check their
behaviour.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Marek Behún]

On Wed, 13 Jan 2021 21:21:25 +0000
Russell King - ARM Linux admin <linux@armlinux.org.uk> wrote:

> > Russell, could we, for now, just edit the code so that when
> >   mv88e6390_serdes_pcs_config
> > is being configured with inband mode in 2500base-x, the inband mode
> > won't be enabled and the function will print a warning?
> > This could come with a Fixes tag so that it is backported to stable.  
> 
> I don't see any other easy option, so yes, please do that.

Will do.

> > Afterwards we can work on refactoring the phylink code so that either
> > the driver can inform phylink whether 2500base-x inband AN is supported,
> > or maybe we can determine from some documentation or whatnot whether
> > inband AN is supported on 2500base-x at all.  
> 
> I suspect there is no definitive documentation on exactly what
> 2500base-x actually is. I suspect it may just be easier to turn off AN
> for 2500base-x everywhere, so at least all Linux systems are compatible
> irrespective of the hardware.
> 
> Yes, it means losing pause negotiation, and people will have to
> manually set pause on each end.

This would need a little refactoring of the phylink_sfp_config
function, but I don't think it will be that hard.

> One thing that I don't know is whether the GPON SFP ONT modules that
> use 2500base-x will still function with AN disabled - although I have
> the modules, it appeared that they both needed a connection to the ONU
> to switch from 1000base-x to 2500base-x on the host side - and as I
> don't have an ONU I can test with, I have no way to check their
> behaviour.

We have an ISP here in Prague who is willing to lend us some GPON
ONU devices to get GPON SFP modules work on Turris. I will ask him if he
has one capable of 2.5g and try to borrow it.

Marek

Re: mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?

[Russell King - ARM Linux admin]

On Wed, Jan 13, 2021 at 10:37:29PM +0100, Marek Behún wrote:
> On Wed, 13 Jan 2021 21:21:25 +0000
> Russell King - ARM Linux admin <linux@armlinux.org.uk> wrote:
> > One thing that I don't know is whether the GPON SFP ONT modules that
> > use 2500base-x will still function with AN disabled - although I have
> > the modules, it appeared that they both needed a connection to the ONU
> > to switch from 1000base-x to 2500base-x on the host side - and as I
> > don't have an ONU I can test with, I have no way to check their
> > behaviour.
> 
> We have an ISP here in Prague who is willing to lend us some GPON
> ONU devices to get GPON SFP modules work on Turris. I will ask him if he
> has one capable of 2.5g and try to borrow it.

I know the Huawei MA5671A and the Nokia 3FE46541AA are - see the quirks
table in sfp-bus.c ! They come from an ISP on the American continent
who really really wanted these modules to link at 2.5Gbps.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!