mvpp2 switch from gmac to xlg requires ifdown/ifup

mvpp2 switch from gmac to xlg requires ifdown/ifup

[Marek Behún]

Hello Russell (and possibly others),

I discovered that with mvpp2 when switching from gmac (sgmii or
2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
change, the link won't come up unless I do
  ifconfig ethX down
  ifconfig ethX up

Can be reproduced on MacchiatoBIN:
1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
2. bring the interfaces up
3. the PHYs should link in 10gbase-t, links on MACs will go up in
   10gbase-r
4. use ethtool on one of the interfaces to advertise modes only up to
   2500base-t
5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
   go up in 2500base-x
6. use ethtool on the same interface as in step 4 to advertise all
   supported modes

7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
   !!!
8. execute
     ifconfig ethX down ; ifconfig ethX up
   on both interfaces. After this, the MACs will successfully link in
   10gbase-r with the PHYs

It seems that the mvpp2 driver code needs to make additional stuff when
chaning mode from gmac to xlg. I haven't yet been able to find out
what, though.

BTW I discovered this because I am working on adding support for
5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
side.
The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
	sgmii		to	5gbase-r
	sgmii		to	10gbase-r
	2500base-x	to	5gbase-r
	2500base-x	to	10gbase-r
but also when switching from xlg to different xlg:
	5gbase-r	to	10gbase-r
	10gbase-r	to	5gbase-r

Did someone notice this bug? I see that Russell made some changes in
the phylink pcs API that touched mvpp2 (the .mac_config method got
split into .mac_prepare, .mac_config and .mac_finish, and also some
other changes). I haven't tried yet if the switch from gmac to xlg
worked at some time in the past. But if it did, maybe these changes
could be the cause?

Marek

Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

[Russell King (Oracle)]

On Fri, Jul 23, 2021 at 03:52:02AM +0200, Marek Behún wrote:
> Hello Russell (and possibly others),
> 
> I discovered that with mvpp2 when switching from gmac (sgmii or
> 2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
> change, the link won't come up unless I do
>   ifconfig ethX down
>   ifconfig ethX up
> 
> Can be reproduced on MacchiatoBIN:
> 1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
> 2. bring the interfaces up
> 3. the PHYs should link in 10gbase-t, links on MACs will go up in
>    10gbase-r
> 4. use ethtool on one of the interfaces to advertise modes only up to
>    2500base-t
> 5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
>    go up in 2500base-x
> 6. use ethtool on the same interface as in step 4 to advertise all
>    supported modes
> 
> 7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
>    !!!
> 8. execute
>      ifconfig ethX down ; ifconfig ethX up
>    on both interfaces. After this, the MACs will successfully link in
>    10gbase-r with the PHYs
> 
> It seems that the mvpp2 driver code needs to make additional stuff when
> chaning mode from gmac to xlg. I haven't yet been able to find out
> what, though.
> 
> BTW I discovered this because I am working on adding support for
> 5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
> side.
> The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
> 	sgmii		to	5gbase-r
> 	sgmii		to	10gbase-r
> 	2500base-x	to	5gbase-r
> 	2500base-x	to	10gbase-r
> but also when switching from xlg to different xlg:
> 	5gbase-r	to	10gbase-r
> 	10gbase-r	to	5gbase-r
> 
> Did someone notice this bug? I see that Russell made some changes in
> the phylink pcs API that touched mvpp2 (the .mac_config method got
> split into .mac_prepare, .mac_config and .mac_finish, and also some
> other changes). I haven't tried yet if the switch from gmac to xlg
> worked at some time in the past. But if it did, maybe these changes
> could be the cause?

What are the PHY leds doing when you encounter this bug?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

[Marek Behún]

On Fri, 23 Jul 2021 09:05:38 +0100
"Russell King (Oracle)" <linux@armlinux.org.uk> wrote:

> On Fri, Jul 23, 2021 at 03:52:02AM +0200, Marek Behún wrote:
> > Hello Russell (and possibly others),
> > 
> > I discovered that with mvpp2 when switching from gmac (sgmii or
> > 2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
> > change, the link won't come up unless I do
> >   ifconfig ethX down
> >   ifconfig ethX up
> > 
> > Can be reproduced on MacchiatoBIN:
> > 1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
> > 2. bring the interfaces up
> > 3. the PHYs should link in 10gbase-t, links on MACs will go up in
> >    10gbase-r
> > 4. use ethtool on one of the interfaces to advertise modes only up to
> >    2500base-t
> > 5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
> >    go up in 2500base-x
> > 6. use ethtool on the same interface as in step 4 to advertise all
> >    supported modes
> > 
> > 7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
> >    !!!
> > 8. execute
> >      ifconfig ethX down ; ifconfig ethX up
> >    on both interfaces. After this, the MACs will successfully link in
> >    10gbase-r with the PHYs
> > 
> > It seems that the mvpp2 driver code needs to make additional stuff when
> > chaning mode from gmac to xlg. I haven't yet been able to find out
> > what, though.
> > 
> > BTW I discovered this because I am working on adding support for
> > 5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
> > side.
> > The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
> > 	sgmii		to	5gbase-r
> > 	sgmii		to	10gbase-r
> > 	2500base-x	to	5gbase-r
> > 	2500base-x	to	10gbase-r
> > but also when switching from xlg to different xlg:
> > 	5gbase-r	to	10gbase-r
> > 	10gbase-r	to	5gbase-r
> > 
> > Did someone notice this bug? I see that Russell made some changes in
> > the phylink pcs API that touched mvpp2 (the .mac_config method got
> > split into .mac_prepare, .mac_config and .mac_finish, and also some
> > other changes). I haven't tried yet if the switch from gmac to xlg
> > worked at some time in the past. But if it did, maybe these changes
> > could be the cause?  
> 
> What are the PHY leds doing when you encounter this bug?
> 

Table summary:

			PHY0/eth0	PHY1/eth1
			green	yellow	green	yellow
after boot		ON	OFF	ON	OFF
eth0 up			ON	OFF	ON	OFF
eth1 up			blink	ON	blink	ON
eth0 adv -10g -5g	blink	OFF	blink	OFF
eth0 adv +5g *	 	OFF	OFF	OFF	OFF
eth0 down		ON	OFF	ON	OFF
eth0 up			blink**	OFF	blink** OFF
eth1 down		ON	OFF	ON	OFF
eth1 up			blink	OFF	blink	OFF

 (*  PHYs are linked now, but MACs are not)
 (** blinks only for a while after link, pings do not work,
     read my opinion below)
 (The last 5 lines basically the same happens if I set it to advertise
  10g instead of 5g, but in case of 10g the yellow LED is ON when the
  PHYs are linked.)

In words:

After boot, the green LED is ON on both PHYs.

Bringing both interfaces up changes nothing.

Plugging cable so that they link (at 10gbase-t) bring the yellow LEDs
ON, and the green LEDs OFF, but the green LEDs blinks on activity.
(For example when pinging eth0's ipv6 address via eth1.)

Disabling advertisement of 10gbase-t and 5gbase-t on eth0 makes the
PHYs link at 2.5gbase-t. Both LEDs are OFF, but green blinks on
activity.

Enabling advertisement of 5gbase-t makes the PHYs link, but the MACs
do not link with the PHYs, and there is no blinking on activity, and
pings do not work.

Taking one interface (eth0) down and up makes the PHYs link (we are
still at 5gbase-t), and the green LEDs blink for a few times because of
activity on both PHYs. But the pings do not work. I think this is
because the eth0's PHY sent some neighbour discovery packets, and the
eth1's PHY received them. But pings do not work because those packets
don't go from eth1's PHY to eth1's MAC.

Taking the second interface (eth1) down and up makes the PHYs again
link at 5gbase-t, and the green LEDs start blinking on activity. Pings
work.

Marek

Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

[Russell King (Oracle)]

On Mon, Jul 26, 2021 at 05:52:23PM +0200, Marek Behún wrote:
> On Fri, 23 Jul 2021 09:05:38 +0100
> "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> 
> > On Fri, Jul 23, 2021 at 03:52:02AM +0200, Marek Behún wrote:
> > > Hello Russell (and possibly others),
> > > 
> > > I discovered that with mvpp2 when switching from gmac (sgmii or
> > > 2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
> > > change, the link won't come up unless I do
> > >   ifconfig ethX down
> > >   ifconfig ethX up
> > > 
> > > Can be reproduced on MacchiatoBIN:
> > > 1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
> > > 2. bring the interfaces up
> > > 3. the PHYs should link in 10gbase-t, links on MACs will go up in
> > >    10gbase-r
> > > 4. use ethtool on one of the interfaces to advertise modes only up to
> > >    2500base-t
> > > 5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
> > >    go up in 2500base-x
> > > 6. use ethtool on the same interface as in step 4 to advertise all
> > >    supported modes
> > > 
> > > 7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
> > >    !!!
> > > 8. execute
> > >      ifconfig ethX down ; ifconfig ethX up
> > >    on both interfaces. After this, the MACs will successfully link in
> > >    10gbase-r with the PHYs
> > > 
> > > It seems that the mvpp2 driver code needs to make additional stuff when
> > > chaning mode from gmac to xlg. I haven't yet been able to find out
> > > what, though.
> > > 
> > > BTW I discovered this because I am working on adding support for
> > > 5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
> > > side.
> > > The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
> > > 	sgmii		to	5gbase-r
> > > 	sgmii		to	10gbase-r
> > > 	2500base-x	to	5gbase-r
> > > 	2500base-x	to	10gbase-r
> > > but also when switching from xlg to different xlg:
> > > 	5gbase-r	to	10gbase-r
> > > 	10gbase-r	to	5gbase-r
> > > 
> > > Did someone notice this bug? I see that Russell made some changes in
> > > the phylink pcs API that touched mvpp2 (the .mac_config method got
> > > split into .mac_prepare, .mac_config and .mac_finish, and also some
> > > other changes). I haven't tried yet if the switch from gmac to xlg
> > > worked at some time in the past. But if it did, maybe these changes
> > > could be the cause?  
> > 
> > What are the PHY leds doing when you encounter this bug?
> > 
> 
> Table summary:
> 
> 			PHY0/eth0	PHY1/eth1
> 			green	yellow	green	yellow
> after boot		ON	OFF	ON	OFF
> eth0 up			ON	OFF	ON	OFF
> eth1 up			blink	ON	blink	ON
> eth0 adv -10g -5g	blink	OFF	blink	OFF
> eth0 adv +5g *	 	OFF	OFF	OFF	OFF
> eth0 down		ON	OFF	ON	OFF
> eth0 up			blink**	OFF	blink** OFF
> eth1 down		ON	OFF	ON	OFF
> eth1 up			blink	OFF	blink	OFF
> 
>  (*  PHYs are linked now, but MACs are not)
>  (** blinks only for a while after link, pings do not work,
>      read my opinion below)
>  (The last 5 lines basically the same happens if I set it to advertise
>   10g instead of 5g, but in case of 10g the yellow LED is ON when the
>   PHYs are linked.)
> 
> In words:
> 
> After boot, the green LED is ON on both PHYs.
> 
> Bringing both interfaces up changes nothing.
> 
> Plugging cable so that they link (at 10gbase-t) bring the yellow LEDs
> ON, and the green LEDs OFF, but the green LEDs blinks on activity.
> (For example when pinging eth0's ipv6 address via eth1.)
> 
> Disabling advertisement of 10gbase-t and 5gbase-t on eth0 makes the
> PHYs link at 2.5gbase-t. Both LEDs are OFF, but green blinks on
> activity.
> 
> Enabling advertisement of 5gbase-t makes the PHYs link, but the MACs
> do not link with the PHYs, and there is no blinking on activity, and
> pings do not work.
> 
> Taking one interface (eth0) down and up makes the PHYs link (we are
> still at 5gbase-t), and the green LEDs blink for a few times because of
> activity on both PHYs. But the pings do not work. I think this is
> because the eth0's PHY sent some neighbour discovery packets, and the
> eth1's PHY received them. But pings do not work because those packets
> don't go from eth1's PHY to eth1's MAC.
> 
> Taking the second interface (eth1) down and up makes the PHYs again
> link at 5gbase-t, and the green LEDs start blinking on activity. Pings
> work.

Hmm, the reason I asked was because I know the 88x3310 with older
firmware can get in a bit of a muddle when switching from SGMII back
to 10G mode. When it does, one of the link LEDs (green I think) blinks
rapidly while it is muddled, and iirc taking the interface down and
back up fixes it. There is no workaround for this other than upgrading
the PHY firmware.

If they only blink momentarily after establishing link, then that is
likely just the IPv6 router solicitation packets.

Hope that helps.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

[Marek Behún]

On Mon, 26 Jul 2021 17:43:25 +0100
"Russell King (Oracle)" <linux@armlinux.org.uk> wrote:

> On Mon, Jul 26, 2021 at 05:52:23PM +0200, Marek Behún wrote:
> > On Fri, 23 Jul 2021 09:05:38 +0100
> > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> >   
> > > On Fri, Jul 23, 2021 at 03:52:02AM +0200, Marek Behún wrote:  
> > > > Hello Russell (and possibly others),
> > > > 
> > > > I discovered that with mvpp2 when switching from gmac (sgmii or
> > > > 2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
> > > > change, the link won't come up unless I do
> > > >   ifconfig ethX down
> > > >   ifconfig ethX up
> > > > 
> > > > Can be reproduced on MacchiatoBIN:
> > > > 1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
> > > > 2. bring the interfaces up
> > > > 3. the PHYs should link in 10gbase-t, links on MACs will go up in
> > > >    10gbase-r
> > > > 4. use ethtool on one of the interfaces to advertise modes only up to
> > > >    2500base-t
> > > > 5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
> > > >    go up in 2500base-x
> > > > 6. use ethtool on the same interface as in step 4 to advertise all
> > > >    supported modes
> > > > 
> > > > 7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
> > > >    !!!
> > > > 8. execute
> > > >      ifconfig ethX down ; ifconfig ethX up
> > > >    on both interfaces. After this, the MACs will successfully link in
> > > >    10gbase-r with the PHYs
> > > > 
> > > > It seems that the mvpp2 driver code needs to make additional stuff when
> > > > chaning mode from gmac to xlg. I haven't yet been able to find out
> > > > what, though.
> > > > 
> > > > BTW I discovered this because I am working on adding support for
> > > > 5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
> > > > side.
> > > > The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
> > > > 	sgmii		to	5gbase-r
> > > > 	sgmii		to	10gbase-r
> > > > 	2500base-x	to	5gbase-r
> > > > 	2500base-x	to	10gbase-r
> > > > but also when switching from xlg to different xlg:
> > > > 	5gbase-r	to	10gbase-r
> > > > 	10gbase-r	to	5gbase-r
> > > > 
> > > > Did someone notice this bug? I see that Russell made some changes in
> > > > the phylink pcs API that touched mvpp2 (the .mac_config method got
> > > > split into .mac_prepare, .mac_config and .mac_finish, and also some
> > > > other changes). I haven't tried yet if the switch from gmac to xlg
> > > > worked at some time in the past. But if it did, maybe these changes
> > > > could be the cause?    
> > > 
> > > What are the PHY leds doing when you encounter this bug?
> > >   
> > 
> > Table summary:
> > 
> > 			PHY0/eth0	PHY1/eth1
> > 			green	yellow	green	yellow
> > after boot		ON	OFF	ON	OFF
> > eth0 up			ON	OFF	ON	OFF
> > eth1 up			blink	ON	blink	ON
> > eth0 adv -10g -5g	blink	OFF	blink	OFF
> > eth0 adv +5g *	 	OFF	OFF	OFF	OFF
> > eth0 down		ON	OFF	ON	OFF
> > eth0 up			blink**	OFF	blink** OFF
> > eth1 down		ON	OFF	ON	OFF
> > eth1 up			blink	OFF	blink	OFF
> > 
> >  (*  PHYs are linked now, but MACs are not)
> >  (** blinks only for a while after link, pings do not work,
> >      read my opinion below)
> >  (The last 5 lines basically the same happens if I set it to advertise
> >   10g instead of 5g, but in case of 10g the yellow LED is ON when the
> >   PHYs are linked.)
> > 
> > In words:
> > 
> > After boot, the green LED is ON on both PHYs.
> > 
> > Bringing both interfaces up changes nothing.
> > 
> > Plugging cable so that they link (at 10gbase-t) bring the yellow LEDs
> > ON, and the green LEDs OFF, but the green LEDs blinks on activity.
> > (For example when pinging eth0's ipv6 address via eth1.)
> > 
> > Disabling advertisement of 10gbase-t and 5gbase-t on eth0 makes the
> > PHYs link at 2.5gbase-t. Both LEDs are OFF, but green blinks on
> > activity.
> > 
> > Enabling advertisement of 5gbase-t makes the PHYs link, but the MACs
> > do not link with the PHYs, and there is no blinking on activity, and
> > pings do not work.
> > 
> > Taking one interface (eth0) down and up makes the PHYs link (we are
> > still at 5gbase-t), and the green LEDs blink for a few times because of
> > activity on both PHYs. But the pings do not work. I think this is
> > because the eth0's PHY sent some neighbour discovery packets, and the
> > eth1's PHY received them. But pings do not work because those packets
> > don't go from eth1's PHY to eth1's MAC.
> > 
> > Taking the second interface (eth1) down and up makes the PHYs again
> > link at 5gbase-t, and the green LEDs start blinking on activity. Pings
> > work.  
> 
> Hmm, the reason I asked was because I know the 88x3310 with older
> firmware can get in a bit of a muddle when switching from SGMII back
> to 10G mode. When it does, one of the link LEDs (green I think) blinks
> rapidly while it is muddled, and iirc taking the interface down and
> back up fixes it. There is no workaround for this other than upgrading
> the PHY firmware.
> 
> If they only blink momentarily after establishing link, then that is
> likely just the IPv6 router solicitation packets.
> 
> Hope that helps.
> 

Thanks, I will try different firmwares and let you know. I actually
currently have two different firmware version on the PHYs...

Marek