Problem: net: mvneta: auto-negotiation with disconnected network cable

Problem: net: mvneta: auto-negotiation with disconnected network cable

[Maxime Morin]

Hi all,

I work on an embedded platform based on the Marvell Armada 88F6707, that is connected to a Marvell Alaska 88E1512 ethernet transceiver. A defect has appeared recently, and it turns out to be a regression on the network part. There is a complete lost of the network when following these steps:
    1) boot the board with the network cable disconnected
    2) run the following commands (or equivalent):
        ip link set eth0 up
        ip addr add 10.0.0.80/24 dev eth0
        ethtool -s eth0 autoneg on #this is the command that really breaks the network
    3) plug the network cable
        => there is no network, and no way to have it back except by rebooting
If I do not launch the "ethtool" command, when I plug the network cable it works, so it really seems to be related to the auto-negotiation set to "on" when the network cable has never been connected.
I did a "git bisect" to find when the regression was introduced, because it previously worked with kernel 4.4, but not with the recent ones. The commit that made appear the issue is this one: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c0744fc1dd5b
If I remove on mvneta.c the part that was added on this commit on the function mvneta_ethtool_set_link_ksettings (mvneta_ethtool_set_settings at that time), I do not have the issue, but I can't call that a fix...
So, could it be a driver issue, or maybe a wrong configuration somewhere? If you need additional information to reproduce the problem please ask me, I will be as responsive as possible.

Best regards,
Maxime MORIN

Re: Problem: net: mvneta: auto-negotiation with disconnected network cable

[Stas Sergeev]

22.03.2017 16:23, Maxime Morin пишет:
> Hi all,
>
> I work on an embedded platform based on the Marvell Armada 88F6707, that is connected to a Marvell Alaska 88E1512 ethernet transceiver. A defect has appeared recently, and it turns out to be a regression on the network part. There is a complete lost of the network when following these steps:
>      1) boot the board with the network cable disconnected
>      2) run the following commands (or equivalent):
>          ip link set eth0 up
>          ip addr add 10.0.0.80/24 dev eth0
>          ethtool -s eth0 autoneg on #this is the command that really breaks the network
Why do you call it a regression, if previously
this command did nothing at all?

>      3) plug the network cable
>          => there is no network, and no way to have it back except by rebooting
> If I do not launch the "ethtool" command, when I plug the network cable it works, so it really seems to be related to the auto-negotiation set to "on" when the network cable has never been connected.
So if you do that with the cable plugged it, there
is no breakage?
When you do "ethtool -s eth0 autoneg off" it doesn't
revive?

> I did a "git bisect" to find when the regression was introduced, because it previously worked with kernel 4.4, but not with the recent ones. The commit that made appear the issue is this one: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c0744fc1dd5b
> If I remove on mvneta.c the part that was added on this commit on the function mvneta_ethtool_set_link_ksettings (mvneta_ethtool_set_settings at that time), I do not have the issue, but I can't call that a fix...
> So, could it be a driver issue, or maybe a wrong configuration somewhere? If you need additional information to reproduce the problem please ask me, I will be as responsive as possible.
It seems mvneta_set_autoneg() does some non-symmetric
things. It clears
MVNETA_GMAC_FORCE_LINK_PASS |
MVNETA_GMAC_FORCE_LINK_DOWN |
MVNETA_GMAC_AN_FLOW_CTRL_EN
when enabling autoneg, and does not restore these flags
when disabling it. Try to make it to set or to restore these
flags and see if that makes "ethtool -s eth0 autoneg off" to
get the network back alive.

Re: Problem: net: mvneta: auto-negotiation with disconnected network cable

[Stas Sergeev]

22.03.2017 16:23, Maxime Morin пишет:
> Hi all,
>
> I work on an embedded platform based on the Marvell Armada 88F6707, that is connected to a Marvell Alaska 88E1512 ethernet transceiver. A defect has appeared recently, and it turns out to be a regression on the network part. There is a complete lost of the network when following these steps:
>      1) boot the board with the network cable disconnected
>      2) run the following commands (or equivalent):
>          ip link set eth0 up
>          ip addr add 10.0.0.80/24 dev eth0
>          ethtool -s eth0 autoneg on #this is the command that really breaks the network
In fact, the problem is that this control is
counter-intuitive. It could instead be

ethtool autoneg [on|off|mdio|inband]

with "on" implying "mdio". In that case you would
never notice the change. Unfortunately there seems
to be no way of passing the autoneg mode, so I fused
"on" to "inband"... So, unless I am mistaken, this clearly
needs a rework. But I have really forgot all the details.

Re: Problem: net: mvneta: auto-negotiation with disconnected network cable

[Maxime Morin]

Hi,
Thank you very much for your help and your reactivity! See my answer bellow:

> 22.03.2017 16:23, Maxime Morin пишет:
> > Hi all,
> >
> > I work on an embedded platform based on the Marvell Armada 88F6707, that is connected to a Marvell Alaska 88E1512 ethernet transceiver. A defect has appeared recently, and it turns out to be a regression on the network part. There is a complete lost of the network when following these steps:
> >      1) boot the board with the network cable disconnected
> >      2) run the following commands (or equivalent):
> >          ip link set eth0 up
> >          ip addr add 10.0.0.80/24 dev eth0
> >          ethtool -s eth0 autoneg on #this is the command that really breaks the network
> Why do you call it a regression, if previously
> this command did nothing at all?

I called that a regression because we still pass through the function phy_ethtool_sset(), which I though was also doing something about the auto-negotiation. But apparently not.

> 
> >      3) plug the network cable
> >          => there is no network, and no way to have it back except by rebooting
> > If I do not launch the "ethtool" command, when I plug the network cable it works, so it really seems to be related to the auto-negotiation set to "on" when the network cable has never been > connected.
> So if you do that with the cable plugged it, there
> is no breakage?
> When you do "ethtool -s eth0 autoneg off" it doesn't
> revive?

Unfortunately no, it does not. I tried many things with ethtool, but it never gets back.

> > I did a "git bisect" to find when the regression was introduced, because it previously worked with kernel 4.4, but not with the recent ones. The commit that made appear the issue is this one: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c0744fc1dd5b
> > If I remove on mvneta.c the part that was added on this commit on the function mvneta_ethtool_set_link_ksettings (mvneta_ethtool_set_settings at that time), I do not have the issue, but I can't call that a fix...
> > So, could it be a driver issue, or maybe a wrong configuration somewhere? If you need additional information to reproduce the problem please ask me, I will be as responsive as possible.
> It seems mvneta_set_autoneg() does some non-symmetric
> things. It clears
> MVNETA_GMAC_FORCE_LINK_PASS |
> MVNETA_GMAC_FORCE_LINK_DOWN |
> MVNETA_GMAC_AN_FLOW_CTRL_EN
> when enabling autoneg, and does not restore these flags
> when disabling it. Try to make it to set or to restore these
> flags and see if that makes "ethtool -s eth0 autoneg off" to
> get the network back alive> .

As you suggested, I just set these three flags when the function is called with "enable" set to 0. And it works!
Actually, I did not even have to set autoneg to off. When the module is probed, the default param are applied (mvneta_defaults_set()), and mvneta_set_autoneg() is called with "enable" to 0, and it seems to fix everything. I tested it then, by setting autoneg to off/on, booting with or without the cable plugged, and I failed to break it. It seems to be fixed. Should I submit a patch? (would be the first time...)

Again, thanks a lot for your help.

Re: Problem: net: mvneta: auto-negotiation with disconnected network cable

[Stas Sergeev]

23.03.2017 14:30, Maxime Morin пишет:
> Hi,
> Thank you very much for your help and your reactivity! See my answer bellow:
>
>> 22.03.2017 16:23, Maxime Morin пишет:
>>> Hi all,
>>>
>>> I work on an embedded platform based on the Marvell Armada 88F6707, that is connected to a Marvell Alaska 88E1512 ethernet transceiver. A defect has appeared recently, and it turns out to be a regression on the network part. There is a complete lost of the network when following these steps:
>>>       1) boot the board with the network cable disconnected
>>>       2) run the following commands (or equivalent):
>>>           ip link set eth0 up
>>>           ip addr add 10.0.0.80/24 dev eth0
>>>           ethtool -s eth0 autoneg on #this is the command that really breaks the network
>> Why do you call it a regression, if previously
>> this command did nothing at all?
> I called that a regression because we still pass through the function phy_ethtool_sset(), which I though was also doing something about the auto-negotiation. But apparently not.
It does, but with mdio.
But in the MAC driver we have another autoneg
now, which is a great confusion. mvneta_set_autoneg()
could have been named mvneta_set_inband_autoneg(),
which would already help a lot. If you prepare the patch-set,
you can include such renaming

>>> I did a "git bisect" to find when the regression was introduced, because it previously worked with kernel 4.4, but not with the recent ones. The commit that made appear the issue is this one: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c0744fc1dd5b
>>> If I remove on mvneta.c the part that was added on this commit on the function mvneta_ethtool_set_link_ksettings (mvneta_ethtool_set_settings at that time), I do not have the issue, but I can't call that a fix...
>>> So, could it be a driver issue, or maybe a wrong configuration somewhere? If you need additional information to reproduce the problem please ask me, I will be as responsive as possible.
>> It seems mvneta_set_autoneg() does some non-symmetric
>> things. It clears
>> MVNETA_GMAC_FORCE_LINK_PASS |
>> MVNETA_GMAC_FORCE_LINK_DOWN |
>> MVNETA_GMAC_AN_FLOW_CTRL_EN
>> when enabling autoneg, and does not restore these flags
>> when disabling it. Try to make it to set or to restore these
>> flags and see if that makes "ethtool -s eth0 autoneg off" to
>> get the network back alive> .
> As you suggested, I just set these three flags when the function is called with "enable" set to 0. And it works!
Am I right that it works only when you set autoneg
to "off", while "on" still does not give you any link detect?
So this is a partial fix, right?

> Actually, I did not even have to set autoneg to off. When the module is probed, the default param are applied (mvneta_defaults_set()), and mvneta_set_autoneg() is called with "enable" to 0, and it seems to fix everything.
It turns off SGMII-specific autoned and starts using
mdio autoneg instead... a great confusion.

> I tested it then, by setting autoneg to off/on, booting with or without the cable plugged, and I failed to break it. It seems to be fixed. Should I submit a patch? (would be the first time...)
After looking up my other patches to this driver, I
can see that

MVNETA_GMAC_FORCE_LINK_PASS |
MVNETA_GMAC_FORCE_LINK_DOWN

are left cleared willingly. I suspect that MVNETA_GMAC_AN_FLOW_CTRL_EN
breaks things for you. Could you please try with setting
back only this flag?

Re: Problem: net: mvneta: auto-negotiation with disconnected network cable

[Maxime Morin]

>From: Stas Sergeev <stsp@list.ru>
>
>23.03.2017 14:30, Maxime Morin пишет:
>> Hi,
>> Thank you very much for your help and your reactivity! See my answer bellow:
>>
>>> 22.03.2017 16:23, Maxime Morin пишет:
>>>> Hi all,
>>>>
>>>> I work on an embedded platform based on the Marvell Armada 88F6707, that is connected to a Marvell Alaska 88E1512 ethernet transceiver. A defect has appeared recently, and it turns out to be a regression on the network part. There is a complete lost of the network when following these steps:
>>>>       1) boot the board with the network cable disconnected
>>>>       2) run the following commands (or equivalent):
>>>>           ip link set eth0 up
>>>>           ip addr add 10.0.0.80/24 dev eth0
>>>>           ethtool -s eth0 autoneg on #this is the command that really breaks the network
>>> Why do you call it a regression, if previously
>>> this command did nothing at all?
>> I called that a regression because we still pass through the function phy_ethtool_sset(), which I though was also doing something about the auto-negotiation. But apparently not.
>It does, but with mdio.
>But in the MAC driver we have another autoneg
>now, which is a great confusion. mvneta_set_autoneg()
>could have been named mvneta_set_inband_autoneg(),
>which would already help a lot. If you prepare the patch-set,
>you can include such renaming
>
>>>> I did a "git bisect" to find when the regression was introduced, because it previously worked with kernel 4.4, but not with the recent ones. The commit that made appear the issue is this one: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c0744fc1dd5b
>>>> If I remove on mvneta.c the part that was added on this commit on the function mvneta_ethtool_set_link_ksettings (mvneta_ethtool_set_settings at that time), I do not have the issue, but I can't call that a fix...
>>>> So, could it be a driver issue, or maybe a wrong configuration somewhere? If you need additional information to reproduce the problem please ask me, I will be as responsive as possible.
>>> It seems mvneta_set_autoneg() does some non-symmetric
>>> things. It clears
>>> MVNETA_GMAC_FORCE_LINK_PASS |
>>> MVNETA_GMAC_FORCE_LINK_DOWN |
>>> MVNETA_GMAC_AN_FLOW_CTRL_EN
>>> when enabling autoneg, and does not restore these flags
>>> when disabling it. Try to make it to set or to restore these
>>> flags and see if that makes "ethtool -s eth0 autoneg off" to
>>> get the network back alive> .
>> As you suggested, I just set these three flags when the function is called with "enable" set to 0. And it works!
>Am I right that it works only when you set autoneg
>to "off", while "on" still does not give you any link detect?
>So this is a partial fix, right?
No. It actually works for both modes in this case.

>> Actually, I did not even have to set autoneg to off. When the module is probed, the default param are applied (mvneta_defaults_set()), and mvneta_set_autoneg() is called with "enable" to 0, and it seems to fix everything.
>It turns off SGMII-specific autoned and starts using
>mdio autoneg instead... a great confusion.
>
>> I tested it then, by setting autoneg to off/on, booting with or without the cable plugged, and I failed to break it. It seems to be fixed. Should I submit a patch? (would be the first time...)
>After looking up my other patches to this driver, I
>can see that
>
>MVNETA_GMAC_FORCE_LINK_PASS |
>MVNETA_GMAC_FORCE_LINK_DOWN
>
>are left cleared willingly. I suspect that MVNETA_GMAC_AN_FLOW_CTRL_EN
>breaks things for you. Could you please try with setting
>back only this flag?
I tested it, but not only this case. There are three flags, so 8 possibilities (and already 2 tested), so I tested all of them one by one.
And it shows that:
- when MVNETA_GMAC_FORCE_LINK_PASS is not set,
no matter the values of MVNETA_GMAC_FORCE_LINK_{PASS,DOWN}, the issue is reproduced
- when MVNETA_GMAC_FORCE_LINK_PASS is set,
no matter the values of MVNETA_GMAC_FORCE_LINK_{PASS,DOWN}, the network works (autoneg set off/on, booting with cable plugged/unplugged, etc)

Re: Problem: net: mvneta: auto-negotiation with disconnected network cable

[Stas Sergeev]

24.03.2017 17:39, Maxime Morin пишет:
>>>>> I did a "git bisect" to find when the regression was introduced, because it previously worked with kernel 4.4, but not with the recent ones. The commit that made appear the issue is this one: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c0744fc1dd5b
>>>>> If I remove on mvneta.c the part that was added on this commit on the function mvneta_ethtool_set_link_ksettings (mvneta_ethtool_set_settings at that time), I do not have the issue, but I can't call that a fix...
>>>>> So, could it be a driver issue, or maybe a wrong configuration somewhere? If you need additional information to reproduce the problem please ask me, I will be as responsive as possible.
>>>> It seems mvneta_set_autoneg() does some non-symmetric
>>>> things. It clears
>>>> MVNETA_GMAC_FORCE_LINK_PASS |
>>>> MVNETA_GMAC_FORCE_LINK_DOWN |
>>>> MVNETA_GMAC_AN_FLOW_CTRL_EN
>>>> when enabling autoneg, and does not restore these flags
>>>> when disabling it. Try to make it to set or to restore these
>>>> flags and see if that makes "ethtool -s eth0 autoneg off" to
>>>> get the network back alive> .
>>> As you suggested, I just set these three flags when the function is called with "enable" set to 0. And it works!
>> Am I right that it works only when you set autoneg
>> to "off", while "on" still does not give you any link detect?
>> So this is a partial fix, right?
> No. It actually works for both modes in this case.
You did something unexpected, didn't you?
The idea was to only change the "!enable" case in
mvneta_set_autoneg() to be more symmetric with
the "enable" case, like restoring the flags. You
likely changed both cases. Could you please change
only the "!enable" case and see what flags needs
to be set there for "off" to work? This should leave
"on" case still disfunctional for you, but one step at
a time.

>>> Actually, I did not even have to set autoneg to off. When the module is probed, the default param are applied (mvneta_defaults_set()), and mvneta_set_autoneg() is called with "enable" to 0, and it seems to fix everything.
>> It turns off SGMII-specific autoned and starts using
>> mdio autoneg instead... a great confusion.
>>
>>> I tested it then, by setting autoneg to off/on, booting with or without the cable plugged, and I failed to break it. It seems to be fixed. Should I submit a patch? (would be the first time...)
>> After looking up my other patches to this driver, I
>> can see that
>>
>> MVNETA_GMAC_FORCE_LINK_PASS |
>> MVNETA_GMAC_FORCE_LINK_DOWN
>>
>> are left cleared willingly. I suspect that MVNETA_GMAC_AN_FLOW_CTRL_EN
>> breaks things for you. Could you please try with setting
>> back only this flag?
> I tested it, but not only this case. There are three flags, so 8 possibilities (and already 2 tested), so I tested all of them one by one.
> And it shows that:
> - when MVNETA_GMAC_FORCE_LINK_PASS is not set,
> no matter the values of MVNETA_GMAC_FORCE_LINK_{PASS,DOWN}, the issue is reproduced
There are 2 problems, one with "on" and one with "off".
Since it is unclear which one do you mention, lets first
concentrate on the "off" case.