Setting link down or up in software

Setting link down or up in software

[Mason]

Hello,

I'm wondering what are the semantics of calling

	ip link set dev eth0 down

I was expecting that to somehow instruct the device's ethernet driver
to shut everything down, have the PHY tell the peer that it's going
away, maybe even put the PHY in some low-power mode, etc.

But it doesn't seem to be doing any of that on my HW.

So what exactly is it supposed to do?


And on top of that, I am seeing random occurrences of

	nb8800 26000.ethernet eth0: Link is Down

Sometimes it is printed immediately.
Sometimes it is printed as soon as I run "ip link set dev eth0 up" (?!)
Sometimes it is not printed at all.

I find this erratic behavior very confusing.

Is it the symptom of some deeper bug?

Regards.

Re: Setting link down or up in software

[Mason]

On 12/01/2017 14:05, Mason wrote:

> I'm wondering what are the semantics of calling
> 
> 	ip link set dev eth0 down
> 
> I was expecting that to somehow instruct the device's ethernet driver
> to shut everything down, have the PHY tell the peer that it's going
> away, maybe even put the PHY in some low-power mode, etc.
> 
> But it doesn't seem to be doing any of that on my HW.
> 
> So what exactly is it supposed to do?
> 
> 
> And on top of that, I am seeing random occurrences of
> 
> 	nb8800 26000.ethernet eth0: Link is Down
> 
> Sometimes it is printed immediately.
> Sometimes it is printed as soon as I run "ip link set dev eth0 up" (?!)
> Sometimes it is not printed at all.
> 
> I find this erratic behavior very confusing.
> 
> Is it the symptom of some deeper bug?

Here's an example of "Link is Down" printed when I set link up:

At [   62.750220] I run ip link set dev eth0 down
Then leave the system idle for 10 minutes.
At [  646.263041] I run ip link set dev eth0 up
At [  647.364079] it prints "Link is Down"
At [  649.417434] it prints "Link is Up - 1Gbps/Full - flow control rx/tx"

I think whether I set up the PHY to use interrupts or polling
does have an influence on the weirdness I observe.

AFAICT, changing the interface flags is done in dev_change_flags
which calls __dev_change_flags and __dev_notify_flags

Is one of these supposed to call the device driver through a
callback at some point?

How/when is the phy_state_machine notified of the change in
interface flags?

Regards.

Re: Setting link down or up in software

[Andrew Lunn]

> Here's an example of "Link is Down" printed when I set link up:
> 
> At [   62.750220] I run ip link set dev eth0 down
> Then leave the system idle for 10 minutes.
> At [  646.263041] I run ip link set dev eth0 up
> At [  647.364079] it prints "Link is Down"
> At [  649.417434] it prints "Link is Up - 1Gbps/Full - flow control rx/tx"

Purely a guess, but when you up the interface, it starts auto
negotiation. That often involves resetting the PHY. If the PHY has
already once completed autoneg, e.g. because of the boot loader, it
will be initially UP. The reset will put it DOWN, and then once
autoneg is complete, it will be Up again.

Pure guess. Go read the code and see if i'm write.

     Andrew

Re: Setting link down or up in software

[Mason]

On 12/01/2017 16:28, Andrew Lunn wrote:

> Mason wrote:
> 
>> Here's an example of "Link is Down" printed when I set link up:
>>
>> At [   62.750220] I run ip link set dev eth0 down
>> Then leave the system idle for 10 minutes.
>> At [  646.263041] I run ip link set dev eth0 up
>> At [  647.364079] it prints "Link is Down"
>> At [  649.417434] it prints "Link is Up - 1Gbps/Full - flow control rx/tx"
> 
> Purely a guess, but when you up the interface, it starts auto
> negotiation. That often involves resetting the PHY. If the PHY has
> already once completed autoneg, e.g. because of the boot loader, it
> will be initially UP. The reset will put it DOWN, and then once
> autoneg is complete, it will be Up again.
> 
> Pure guess. Go read the code and see if i'm write.

Thanks for giving me some food for thought, although the net framework
is far from easy to navigate. (So I'm not sure "go read the code" will
take me anywhere in the short term.)

Whatever the reason for the symptoms I'm seeing, some kind of race
condition must be involved, because it occurs randomly.

Regards.

Re: Setting link down or up in software

[Andrew Lunn]

> Whatever the reason for the symptoms I'm seeing, some kind of race
> condition must be involved, because it occurs randomly.

The PHY is polled once a second for its status. So it would depend on
who fast autoneg happens if you see the down.

	Andrew

Re: Setting link down or up in software

[Dan Williams]

On Thu, 2017-01-12 at 16:28 +0100, Andrew Lunn wrote:
> > Here's an example of "Link is Down" printed when I set link up:
> > 
> > At [   62.750220] I run ip link set dev eth0 down
> > Then leave the system idle for 10 minutes.
> > At [  646.263041] I run ip link set dev eth0 up
> > At [  647.364079] it prints "Link is Down"
> > At [  649.417434] it prints "Link is Up - 1Gbps/Full - flow control
> > rx/tx"
> 
> Purely a guess, but when you up the interface, it starts auto
> negotiation. That often involves resetting the PHY. If the PHY has
> already once completed autoneg, e.g. because of the boot loader, it
> will be initially UP. The reset will put it DOWN, and then once
> autoneg is complete, it will be Up again.
> 
> Pure guess. Go read the code and see if i'm write.

Historically, !IFF_UP meant the phy could be powered down or go into
some kind of low-power mode where link negotiation and communication
was unavailable.  Some drivers implemented that, some keep the PHY up
because who knows why, or because they just never got around to real
power management.

Dan

Re: Setting link down or up in software

[Zefir Kurtisi]

On 01/12/2017 04:16 PM, Mason wrote:
> On 12/01/2017 14:05, Mason wrote:
> 
>> I'm wondering what are the semantics of calling
>>
>> 	ip link set dev eth0 down
>>
>> I was expecting that to somehow instruct the device's ethernet driver
>> to shut everything down, have the PHY tell the peer that it's going
>> away, maybe even put the PHY in some low-power mode, etc.
>>
>> But it doesn't seem to be doing any of that on my HW.
>>
>> So what exactly is it supposed to do?
>>
>>
>> And on top of that, I am seeing random occurrences of
>>
>> 	nb8800 26000.ethernet eth0: Link is Down
>>
>> Sometimes it is printed immediately.
>> Sometimes it is printed as soon as I run "ip link set dev eth0 up" (?!)
>> Sometimes it is not printed at all.
>>
>> I find this erratic behavior very confusing.
>>
>> Is it the symptom of some deeper bug?
> 
> Here's an example of "Link is Down" printed when I set link up:
> 
> At [   62.750220] I run ip link set dev eth0 down
> Then leave the system idle for 10 minutes.
> At [  646.263041] I run ip link set dev eth0 up
> At [  647.364079] it prints "Link is Down"
> At [  649.417434] it prints "Link is Up - 1Gbps/Full - flow control rx/tx"
> 
> I think whether I set up the PHY to use interrupts or polling
> does have an influence on the weirdness I observe.
> 
> AFAICT, changing the interface flags is done in dev_change_flags
> which calls __dev_change_flags and __dev_notify_flags
> 
> Is one of these supposed to call the device driver through a
> callback at some point?
> 
> How/when is the phy_state_machine notified of the change in
> interface flags?
> 
> Regards.
> 
Hm, reminds me of something at my side that I recently fixed with [1]. For me it
was pulling the cable got randomly unnoticed at PHY layer - but might be related.

Do you by chance have some component that polls the link states over the ethtool
interface very often (like once per second)? At my side it was a snmpd agent that
pro-actively updated the interface states every second and with that 'stole' the
link change information from the phy link state machine. What you need to have to
run in such a failing situation is:
1) an ETH driver that updates link status in ethtool GSET path (e.g. dsa does)
2) some component that continuously polls states via ethtool GSET


Cheers,
Zefir


[1] https://patchwork.ozlabs.org/patch/711839/

Re: Setting link down or up in software

[Mason]

On 13/01/2017 10:20, Zefir Kurtisi wrote:
> On 01/12/2017 04:16 PM, Mason wrote:
>> On 12/01/2017 14:05, Mason wrote:
>>
>>> I'm wondering what are the semantics of calling
>>>
>>> 	ip link set dev eth0 down
>>>
>>> I was expecting that to somehow instruct the device's ethernet driver
>>> to shut everything down, have the PHY tell the peer that it's going
>>> away, maybe even put the PHY in some low-power mode, etc.
>>>
>>> But it doesn't seem to be doing any of that on my HW.
>>>
>>> So what exactly is it supposed to do?
>>>
>>>
>>> And on top of that, I am seeing random occurrences of
>>>
>>> 	nb8800 26000.ethernet eth0: Link is Down
>>>
>>> Sometimes it is printed immediately.
>>> Sometimes it is printed as soon as I run "ip link set dev eth0 up" (?!)
>>> Sometimes it is not printed at all.
>>>
>>> I find this erratic behavior very confusing.
>>>
>>> Is it the symptom of some deeper bug?
>>
>> Here's an example of "Link is Down" printed when I set link up:
>>
>> At [   62.750220] I run ip link set dev eth0 down
>> Then leave the system idle for 10 minutes.
>> At [  646.263041] I run ip link set dev eth0 up
>> At [  647.364079] it prints "Link is Down"
>> At [  649.417434] it prints "Link is Up - 1Gbps/Full - flow control rx/tx"
>>
>> I think whether I set up the PHY to use interrupts or polling
>> does have an influence on the weirdness I observe.
>>
>> AFAICT, changing the interface flags is done in dev_change_flags
>> which calls __dev_change_flags and __dev_notify_flags
>>
>> Is one of these supposed to call the device driver through a
>> callback at some point?
>>
>> How/when is the phy_state_machine notified of the change in
>> interface flags?
>>
>> Regards.
>>
> Hm, reminds me of something at my side that I recently fixed with [1]. For me it
> was pulling the cable got randomly unnoticed at PHY layer - but might be related.
> 
> Do you by chance have some component that polls the link states over the ethtool
> interface very often (like once per second)? At my side it was a snmpd agent that
> pro-actively updated the interface states every second and with that 'stole' the
> link change information from the phy link state machine. What you need to have to
> run in such a failing situation is:
> 1) an ETH driver that updates link status in ethtool GSET path (e.g. dsa does)
> 2) some component that continuously polls states via ethtool GSET
> 
> 
> Cheers,
> Zefir
> 
> 
> [1] https://patchwork.ozlabs.org/patch/711839/

Hello Zefir,

Thanks for the insightful comment.

This is a minimal buildroot system, with no frills, and not much running.
There definitely is no SNMP daemon running, but I can't be 100% sure that
busybox isn't polling the link state once in a while. (It's unlikely.)

I'm surprised that there are still bugs lurking in the phy state machine,
I expected this to be a "solved problem", but I suppose power management
has broken many assumptions that were once safe...

By the way, I did come across code paths where phy->state was read or
written without taking the lock. Isn't that never supposed to happen?

Regards.

Re: Setting link down or up in software

[Zefir Kurtisi]

On 01/13/2017 04:17 PM, Mason wrote:
> On 13/01/2017 10:20, Zefir Kurtisi wrote:
>> On 01/12/2017 04:16 PM, Mason wrote:
>>> On 12/01/2017 14:05, Mason wrote:
>>>
>>>> I'm wondering what are the semantics of calling
>>>>
>>>> 	ip link set dev eth0 down
>>>>
>>>> I was expecting that to somehow instruct the device's ethernet driver
>>>> to shut everything down, have the PHY tell the peer that it's going
>>>> away, maybe even put the PHY in some low-power mode, etc.
>>>>
>>>> But it doesn't seem to be doing any of that on my HW.
>>>>
>>>> So what exactly is it supposed to do?
>>>>
>>>>
>>>> And on top of that, I am seeing random occurrences of
>>>>
>>>> 	nb8800 26000.ethernet eth0: Link is Down
>>>>
>>>> Sometimes it is printed immediately.
>>>> Sometimes it is printed as soon as I run "ip link set dev eth0 up" (?!)
>>>> Sometimes it is not printed at all.
>>>>
>>>> I find this erratic behavior very confusing.
>>>>
>>>> Is it the symptom of some deeper bug?
>>>
>>> Here's an example of "Link is Down" printed when I set link up:
>>>
>>> At [   62.750220] I run ip link set dev eth0 down
>>> Then leave the system idle for 10 minutes.
>>> At [  646.263041] I run ip link set dev eth0 up
>>> At [  647.364079] it prints "Link is Down"
>>> At [  649.417434] it prints "Link is Up - 1Gbps/Full - flow control rx/tx"
>>>
>>> I think whether I set up the PHY to use interrupts or polling
>>> does have an influence on the weirdness I observe.
>>>
>>> AFAICT, changing the interface flags is done in dev_change_flags
>>> which calls __dev_change_flags and __dev_notify_flags
>>>
>>> Is one of these supposed to call the device driver through a
>>> callback at some point?
>>>
>>> How/when is the phy_state_machine notified of the change in
>>> interface flags?
>>>
>>> Regards.
>>>
>> Hm, reminds me of something at my side that I recently fixed with [1]. For me it
>> was pulling the cable got randomly unnoticed at PHY layer - but might be related.
>>
>> Do you by chance have some component that polls the link states over the ethtool
>> interface very often (like once per second)? At my side it was a snmpd agent that
>> pro-actively updated the interface states every second and with that 'stole' the
>> link change information from the phy link state machine. What you need to have to
>> run in such a failing situation is:
>> 1) an ETH driver that updates link status in ethtool GSET path (e.g. dsa does)
>> 2) some component that continuously polls states via ethtool GSET
>>
>>
>> Cheers,
>> Zefir
>>
>>
>> [1] https://patchwork.ozlabs.org/patch/711839/
> 
> Hello Zefir,
> 
> Thanks for the insightful comment.
> 
> This is a minimal buildroot system, with no frills, and not much running.
> There definitely is no SNMP daemon running, but I can't be 100% sure that
> busybox isn't polling the link state once in a while. (It's unlikely.)
> 
> I'm surprised that there are still bugs lurking in the phy state machine,
> I expected this to be a "solved problem", but I suppose power management
> has broken many assumptions that were once safe...
> 
> By the way, I did come across code paths where phy->state was read or
> written without taking the lock. Isn't that never supposed to happen?
> 
> Regards.
> 

The problem is that phydev->read_status() is not meant to be called from outside
the pyh state machine. Some ETH drivers are doing this in the ethtool GSET path,
either by a wrong assumption that they need to have the link status updated
instantly, or because other drivers are are based on copy-pasting others. I myself
did not find defined rules what ETH drivers are allowed to call and what they must
not, so as a result there is a handful of ETH drivers that are doing it wrong.

Florian is going to push the developers to fix the drivers with time.

As for your specific problem: since I fought myself with the PHY/ETH subsystems
over the past months, I might remember something relevant to your issue. Could you
give some more info on your setup (PHY driver, opmode (SGMII, RGMII, etc.), ETH).


Cheers,
Zefir

Re: Setting link down or up in software

[Mason]

On 13/01/2017 17:28, Zefir Kurtisi wrote:

> As for your specific problem: since I fought myself with the PHY/ETH subsystems
> over the past months, I might remember something relevant to your issue. Could you
> give some more info on your setup (PHY driver, opmode (SGMII, RGMII, etc.), ETH).

Hello Zefir,

My boards are using these drivers:

http://lxr.free-electrons.com/source/drivers/net/ethernet/aurora/nb8800.c
http://lxr.free-electrons.com/source/drivers/net/phy/at803x.c

The relevant device tree nodes are:

		eth0: ethernet@26000 {
			compatible = "sigma,smp8734-ethernet";
			reg = <0x26000 0x800>;
			interrupts = <38 IRQ_TYPE_LEVEL_HIGH>;
			clocks = <&clkgen SYS_CLK>;
		};

&eth0 {
	phy-connection-type = "rgmii";
	phy-handle = <&eth0_phy>;
	#address-cells = <1>;
	#size-cells = <0>;

	/* Atheros AR8035 */
	eth0_phy: ethernet-phy@4 {
		compatible = "ethernet-phy-id004d.d072",
			     "ethernet-phy-ieee802.3-c22";
		interrupts = <37 IRQ_TYPE_EDGE_RISING>;
		reg = <4>;
	};
};

If I comment the PHY "interrupts" property, then the PHY framework
falls back to polling.

Am I forgetting important information?

Regards.

Re: Setting link down or up in software

[Zefir Kurtisi]

On 01/13/2017 06:35 PM, Mason wrote:
> On 13/01/2017 17:28, Zefir Kurtisi wrote:
> 
>> As for your specific problem: since I fought myself with the PHY/ETH subsystems
>> over the past months, I might remember something relevant to your issue. Could you
>> give some more info on your setup (PHY driver, opmode (SGMII, RGMII, etc.), ETH).
> 
> Hello Zefir,
> 
> My boards are using these drivers:
> 
> http://lxr.free-electrons.com/source/drivers/net/ethernet/aurora/nb8800.c
> http://lxr.free-electrons.com/source/drivers/net/phy/at803x.c
> 
> The relevant device tree nodes are:
> 
> 		eth0: ethernet@26000 {
> 			compatible = "sigma,smp8734-ethernet";
> 			reg = <0x26000 0x800>;
> 			interrupts = <38 IRQ_TYPE_LEVEL_HIGH>;
> 			clocks = <&clkgen SYS_CLK>;
> 		};
> 
> &eth0 {
> 	phy-connection-type = "rgmii";
> 	phy-handle = <&eth0_phy>;
> 	#address-cells = <1>;
> 	#size-cells = <0>;
> 
> 	/* Atheros AR8035 */
> 	eth0_phy: ethernet-phy@4 {
> 		compatible = "ethernet-phy-id004d.d072",
> 			     "ethernet-phy-ieee802.3-c22";
> 		interrupts = <37 IRQ_TYPE_EDGE_RISING>;
> 		reg = <4>;
> 	};
> };
> 
> If I comment the PHY "interrupts" property, then the PHY framework
> falls back to polling.
> 
> Am I forgetting important information?
> 
> Regards.
> 
Hi,

in our system we attach the at8031 over SGMII to the gianfar (Freescale eTSEC) and
to fibre optics transceivers, which operate in fixed speeds. Getting this setup to
work reliably was challenging for various reasons, maybe worth to note

1) fixed SGMII speed not working: link is up on both ends, but no data is passed
2) known issue with SGMII link not completing autonegotiation correctly, see [1]
3) once autoneg is started or chip is reset, MII_CTRL1000 can not be written to
until autoneg is completed => breaks phy state machine when the driver loads with
unplugged cable and tries to set fixed speed

Unless you are using fixed speed links in your setup, none of those should affect
it. My experience with at8031 attached to RGMII is that it is genphy compliant,
therefore I would a) disable interrupts, and b) prevent loading at803x.ko and try
the genphy instead. Yours is an at8035, results may vary.


Cheers,
Zefir


[1] https://www.spinics.net/lists/netdev/msg400804.html

Re: Setting link down or up in software

[Mason]

On 18/01/2017 11:29, Zefir Kurtisi wrote:
> On 01/13/2017 06:35 PM, Mason wrote:
>> On 13/01/2017 17:28, Zefir Kurtisi wrote:
>>
>>> As for your specific problem: since I fought myself with the PHY/ETH subsystems
>>> over the past months, I might remember something relevant to your issue. Could you
>>> give some more info on your setup (PHY driver, opmode (SGMII, RGMII, etc.), ETH).
>>
>> Hello Zefir,
>>
>> My boards are using these drivers:
>>
>> http://lxr.free-electrons.com/source/drivers/net/ethernet/aurora/nb8800.c
>> http://lxr.free-electrons.com/source/drivers/net/phy/at803x.c
>>
>> The relevant device tree nodes are:
>>
>> 		eth0: ethernet@26000 {
>> 			compatible = "sigma,smp8734-ethernet";
>> 			reg = <0x26000 0x800>;
>> 			interrupts = <38 IRQ_TYPE_LEVEL_HIGH>;
>> 			clocks = <&clkgen SYS_CLK>;
>> 		};
>>
>> &eth0 {
>> 	phy-connection-type = "rgmii";
>> 	phy-handle = <&eth0_phy>;
>> 	#address-cells = <1>;
>> 	#size-cells = <0>;
>>
>> 	/* Atheros AR8035 */
>> 	eth0_phy: ethernet-phy@4 {
>> 		compatible = "ethernet-phy-id004d.d072",
>> 			     "ethernet-phy-ieee802.3-c22";
>> 		interrupts = <37 IRQ_TYPE_EDGE_RISING>;
>> 		reg = <4>;
>> 	};
>> };
>>
>> If I comment the PHY "interrupts" property, then the PHY framework
>> falls back to polling.
>>
>> Am I forgetting important information?
> 
> In our system we attach the at8031 over SGMII to the gianfar (Freescale eTSEC) and
> to fibre optics transceivers, which operate in fixed speeds. Getting this setup to
> work reliably was challenging for various reasons, maybe worth to note
> 
> 1) fixed SGMII speed not working: link is up on both ends, but no data is passed
> 2) known issue with SGMII link not completing autonegotiation correctly, see [1]
> 3) once autoneg is started or chip is reset, MII_CTRL1000 can not be written to
> until autoneg is completed => breaks phy state machine when the driver loads with
> unplugged cable and tries to set fixed speed
> 
> Unless you are using fixed speed links in your setup, none of those should affect
> it. My experience with at8031 attached to RGMII is that it is genphy compliant,
> therefore I would a) disable interrupts, and b) prevent loading at803x.ko and try
> the genphy instead. Yours is an at8035, results may vary.
> 
> [1] https://www.spinics.net/lists/netdev/msg400804.html

Hello Zefir,

Thanks for having a look at my issue.

I must confess that I'm not up to speed on the different *MIIs
(MII, RMII, GMII, RGMII, SGMII, etc)

One thing I find confusing... If the 8035 could work with the generic
PHY driver, what's the point of the at803x.ko driver?

Regards.

Re: Setting link down or up in software

[Måns Rullgård]

Mason <slash.tmp@free.fr> writes:

> One thing I find confusing... If the 8035 could work with the generic
> PHY driver, what's the point of the at803x.ko driver?

Almost all Ethernet PHYs support the basic operation defined the by the
standard.  Many have additional non-standardised features that require a
specific driver.  Interrupts, for example.

-- 
Måns Rullgård

Re: Setting link down or up in software

[Florian Fainelli]

On 01/19/2017 09:56 AM, Måns Rullgård wrote:
> Mason <slash.tmp@free.fr> writes:
> 
>> One thing I find confusing... If the 8035 could work with the generic
>> PHY driver, what's the point of the at803x.ko driver?
> 
> Almost all Ethernet PHYs support the basic operation defined the by the
> standard.  Many have additional non-standardised features that require a
> specific driver.  Interrupts, for example.

Or special initial configuration wrt. (RG)MII modes, and runtime
configuration (EEE, link speeds, workarounds), obtaining useful counters
etc.

The generic PHY driver only deals with the standard MII registers (and
just a subset of it actually) while the PHY driver allows you to deal
with all the hardware specifics, the whole point behind having both
specialization and abstraction.
-- 
Florian