Meson8b network not stable at gigabit

Meson8b network not stable at gigabit

[Aymeric]

Hello all,

I've just subscribe on this mailing list, so forgive me if I'm not
following some rules.

Anyway, I've an ODROID-C1 board (which mean meson8b/S805) and I've some
network stability issue with both kernel 5.0.x and 5.1.x (as of today
I'm using 5.1.9, compiled myself - config file used is available here [¹]).

The network card is detecting the link as down and up every few seconds
making it unusable.

The kernel from Hardkernel (3.10.something) is working fine.

See a sample of the log (dmesg) here [²].

The only way I found to have the network stable is to use ethtool to
force the link at 100M.

Are you aware of it? Is there anything I can do?

Also, I didn't try with kernel 5.2-rc4 yet, maybe I've made a mistake in
my .config?

Thanks in advance!

Aymeric.

[¹]: https://gitlab.com/mulx/kernel-odroidc1/blob/master/config
[²]:
https://paste.aplu.fr/?b5eb6df48a9c95b6#sqHk8xhWGwRfagWNpL+u7mIsPGWVWFn2d7xBqika8Kc=

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Meson8b network not stable at gigabit

[Martin Blumenstingl]

Hello Aymeric,

On Sat, Jun 15, 2019 at 12:22 AM Aymeric <mulx@aplu.fr> wrote:
>
> Hello all,
>
> I've just subscribe on this mailing list, so forgive me if I'm not
> following some rules.
>
> Anyway, I've an ODROID-C1 board (which mean meson8b/S805) and I've some
> network stability issue with both kernel 5.0.x and 5.1.x (as of today
> I'm using 5.1.9, compiled myself - config file used is available here [¹]).
>
> The network card is detecting the link as down and up every few seconds
> making it unusable.
that is bad :(
how do you trigger this problem?

I just tried it with 5.2-rc4 by just booting the board, bringing up
the eth0 interface and downloading ~50MiB
I cannot reproduce your link up / down issue:
# uptime
11:34:17 up 11 min,  1 user,  load average: 1.70, 2.39, 1.51
# dmesg | tail -n1
[   43.062185] meson8b-dwmac c9410000.ethernet eth0: Link is Up -
1Gbps/Full - flow control off

> The kernel from Hardkernel (3.10.something) is working fine.
the 3.10 kernel uses polling for the PHY status while the mainline
kernel uses an interrupt.
that means 3.10 will only look for a link change every x milliseconds
(I believe it's 500 or 1000 by default) while mainline will
"instantly" detect any link change

can you please:
1. boot mainline Linux
2. bring eth0 up
3. run "cat /proc/interrupts | grep stmmac"
4. wait 1 minute, do not disconnect the network cable etc.
5. run "cat /proc/interrupts | grep stmmac" again

in my case the "count" (second column from the cat output above) is 1
after step 3 and still 1 after step 5
anything else suggests that the Ethernet PHY sees a link change (then
we have to find out why)

> See a sample of the log (dmesg) here [²].
>
> The only way I found to have the network stable is to use ethtool to
> force the link at 100M.
did you rule out the usual suspects (different switch port, try with
another cable)?

> Are you aware of it? Is there anything I can do?
>
> Also, I didn't try with kernel 5.2-rc4 yet, maybe I've made a mistake in
> my .config?
your .config looks fine: meson8b-dwmac (and thus the stmmac Ethernet
controller driver) are detected as well as the RTL8211F PHY


Martin

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Meson8b network not stable at gigabit

[Aymeric]

Le 15/06/2019 à 13:34, Martin Blumenstingl a écrit :
> Hello Aymeric,
Hello Martin,
>
> On Sat, Jun 15, 2019 at 12:22 AM Aymeric <mulx@aplu.fr> wrote:
>> Hello all,
>>
>> I've just subscribe on this mailing list, so forgive me if I'm not
>> following some rules.
>>
>> Anyway, I've an ODROID-C1 board (which mean meson8b/S805) and I've some
>> network stability issue with both kernel 5.0.x and 5.1.x (as of today
>> I'm using 5.1.9, compiled myself - config file used is available here [¹]).
>>
>> The network card is detecting the link as down and up every few seconds
>> making it unusable.
> that is bad :(
> how do you trigger this problem?

I just boot the board with the network attached, I'm not doing anything
special.
I know a friend of mine, with the same board, we bought them together,
has the same issue than me.


>
>> The kernel from Hardkernel (3.10.something) is working fine.
> the 3.10 kernel uses polling for the PHY status while the mainline
> kernel uses an interrupt.
> that means 3.10 will only look for a link change every x milliseconds
> (I believe it's 500 or 1000 by default) while mainline will
> "instantly" detect any link change

Oh, okay, didn't know.


> can you please:
> 1. boot mainline Linux
> 2. bring eth0 up
> 3. run "cat /proc/interrupts | grep stmmac"
> 4. wait 1 minute, do not disconnect the network cable etc.
> 5. run "cat /proc/interrupts | grep stmmac" again

Just after boot, when I was able to login to the console,

$ uname -a
Linux odroid 5.2.0-rc4 #11 SMP PREEMPT Sat Jun 15 19:27:42 CEST 2019
armv7l GNU/Linux
$ cat /proc/interrupts | grep stmmac
 40:         12          0          0          0  meson-gpio-irqchip  17
Level     stmmac-0:00


And here is with one minute delay:

$ (date -R ;  cat /proc/interrupts | grep stmmac ; sleep 1m ; date -R ;
cat /proc/interrupts | grep stmmac )

Sun, 16 Jun 2019 17:22:00 +0200
 40:         32          0          0          0  meson-gpio-irqchip  17
Level     stmmac-0:00
Sun, 16 Jun 2019 17:23:00 +0200
 40:         47          0          0          0  meson-gpio-irqchip  17
Level     stmmac-0:00

> did you rule out the usual suspects (different switch port, try with
> another cable)?

I've tried indeed already try all other port of the switch (it's my
internet box) and an other network cable, with the same result.
But interestingly I've performed one more test today, by connecting the
odroid to an other switch (a D-LINK 8p gigabit) and there whatever cable
I use, the link became stable. This is strange as I've other PCs
directly connected to the internet box and working fine in gigabit…. and
the d-link and the internet box are in gigabit too.

The result from /proc/interrupts above are when I'm connected to my
operator box, from the d-link the value doesn't change any more.

According to the "about page" of the internet box, this is a Sagemcom
(Livebox 3).

So it look like there a incompatibility between the Realtek RTL8211F and
the "Sagemcom" but not with a "stupid" d-link switch….


-- 
Aymeric

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Meson8b network not stable at gigabit

[Martin Blumenstingl]

Hello Aymeric,

On Sun, Jun 16, 2019 at 6:05 PM Aymeric <mulx@aplu.fr> wrote:
[...]
> > can you please:
> > 1. boot mainline Linux
> > 2. bring eth0 up
> > 3. run "cat /proc/interrupts | grep stmmac"
> > 4. wait 1 minute, do not disconnect the network cable etc.
> > 5. run "cat /proc/interrupts | grep stmmac" again
>
> Just after boot, when I was able to login to the console,
>
> $ uname -a
> Linux odroid 5.2.0-rc4 #11 SMP PREEMPT Sat Jun 15 19:27:42 CEST 2019
> armv7l GNU/Linux
> $ cat /proc/interrupts | grep stmmac
>  40:         12          0          0          0  meson-gpio-irqchip  17
> Level     stmmac-0:00
>
>
> And here is with one minute delay:
>
> $ (date -R ;  cat /proc/interrupts | grep stmmac ; sleep 1m ; date -R ;
> cat /proc/interrupts | grep stmmac )
>
> Sun, 16 Jun 2019 17:22:00 +0200
>  40:         32          0          0          0  meson-gpio-irqchip  17
> Level     stmmac-0:00
> Sun, 16 Jun 2019 17:23:00 +0200
>  40:         47          0          0          0  meson-gpio-irqchip  17
> Level     stmmac-0:00
my interpretation of this is: the PHY is detecting a link change and
that's why the kernel behaves accordingly

> > did you rule out the usual suspects (different switch port, try with
> > another cable)?
>
> I've tried indeed already try all other port of the switch (it's my
> internet box) and an other network cable, with the same result.
> But interestingly I've performed one more test today, by connecting the
> odroid to an other switch (a D-LINK 8p gigabit) and there whatever cable
> I use, the link became stable. This is strange as I've other PCs
> directly connected to the internet box and working fine in gigabit…. and
> the d-link and the internet box are in gigabit too.
that's interesting indeed and may explain why I'm not seeing this issue

> The result from /proc/interrupts above are when I'm connected to my
> operator box, from the d-link the value doesn't change any more.
>
> According to the "about page" of the internet box, this is a Sagemcom
> (Livebox 3).
>
> So it look like there a incompatibility between the Realtek RTL8211F and
> the "Sagemcom" but not with a "stupid" d-link switch….
can you try to disable EEE (energy efficient Ethernet) for Gbit links?
you need to rebuild the .dtb (typically built together with the
kernel) for that:
- open arch/arm/boot/dts/meson8b-odroidc1.dts
- go to the "eth_phy" node (for example: [0])
- below the "reg = <0>;" line add: eee-broken-1000t;
- rebuild the kernel to get an updated .dtb
- boot the kernel with that .dtb

3.10 kernel has EEE disabled for 100Mbit and Gbit links but in the
past we have only seen issues with Gbit links in reality
we removed that eee-broken-1000t property with v5.0 in [1]
it seemed that fixing the interrupt polarity also fixes all issues
that we previously worked around by disabling EEE.
maybe you found a case where we still need that interrupt


Martin


[0] https://github.com/torvalds/linux/blob/09ee951617d9af8a86d228b2ed34035076ab6001/arch/arm/boot/dts/meson8b-odroidc1.dts#L236
[1] https://github.com/torvalds/linux/commit/e35e26b26e955c53e61c154ba26b9bb15da6b858

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Meson8b network not stable at gigabit

[Aymeric]

Hello Martin,

Le 16/06/2019 à 18:30, Martin Blumenstingl a écrit :
> On Sun, Jun 16, 2019 at 6:05 PM Aymeric <mulx@aplu.fr> wrote:
> [...]
>> The result from /proc/interrupts above are when I'm connected to my
>> operator box, from the d-link the value doesn't change any more.
>>
>> According to the "about page" of the internet box, this is a Sagemcom
>> (Livebox 3).
>>
>> So it look like there a incompatibility between the Realtek RTL8211F and
>> the "Sagemcom" but not with a "stupid" d-link switch….
> can you try to disable EEE (energy efficient Ethernet) for Gbit links?
> you need to rebuild the .dtb (typically built together with the
> kernel) for that:
> - open arch/arm/boot/dts/meson8b-odroidc1.dts
> - go to the "eth_phy" node (for example: [0])
> - below the "reg = <0>;" line add: eee-broken-1000t;
> - rebuild the kernel to get an updated .dtb
> - boot the kernel with that .dtb

Test done on 5.2.0-rc4, that didn't change anything (I've double
checked, this is the new .dtb I'm using).

-- 
Aymeric

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Meson8b network not stable at gigabit

[Martin Blumenstingl]

Hi Aymeric,

On Sun, Jun 16, 2019 at 8:57 PM Aymeric <mulx@aplu.fr> wrote:
>
> Hello Martin,
>
> Le 16/06/2019 à 18:30, Martin Blumenstingl a écrit :
> > On Sun, Jun 16, 2019 at 6:05 PM Aymeric <mulx@aplu.fr> wrote:
> > [...]
> >> The result from /proc/interrupts above are when I'm connected to my
> >> operator box, from the d-link the value doesn't change any more.
> >>
> >> According to the "about page" of the internet box, this is a Sagemcom
> >> (Livebox 3).
> >>
> >> So it look like there a incompatibility between the Realtek RTL8211F and
> >> the "Sagemcom" but not with a "stupid" d-link switch….
> > can you try to disable EEE (energy efficient Ethernet) for Gbit links?
> > you need to rebuild the .dtb (typically built together with the
> > kernel) for that:
> > - open arch/arm/boot/dts/meson8b-odroidc1.dts
> > - go to the "eth_phy" node (for example: [0])
> > - below the "reg = <0>;" line add: eee-broken-1000t;
> > - rebuild the kernel to get an updated .dtb
> > - boot the kernel with that .dtb
>
> Test done on 5.2.0-rc4, that didn't change anything (I've double
> checked, this is the new .dtb I'm using).
too bad, that was the only "smart" idea I had.

did see this issue with kernel versions earlier than 5.0 as well?
then you can try to git bisect the cause

you can also send a similar mail to the netdev mailing list (please
keep the Amogic list at least Cc'ed) and ask for help there
the Amlogic SoCs use a DesignWare MAC (Ethernet controller, the driver
is called stmmac) with a Relatek RTL8211F Ethernet PHY.
there's little Amlogic specific registers involved: they mostly
control the PHY interface (enabling RMII or RGMII) and the clocks
so it's very likely that someone on the netdev list has an idea how to
debug this because a large part of the Ethernet setup is not Amlogic
SoC specific


Martin

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Meson8b network not stable at gigabit

[Aymeric]

Hi Martin,

On 2019-06-17 21:36, Martin Blumenstingl wrote:
> Hi Aymeric,
> 
> too bad, that was the only "smart" idea I had.

No problem, that was more than me. :)

> 
> did see this issue with kernel versions earlier than 5.0 as well?
> then you can try to git bisect the cause

I've made a switch from 3.10 to 5.0.4 directly (and the issue was 
already there, I just didn't had time to report/investigate), so I don't 
know is the issue was present in any earlier version.

> 
> you can also send a similar mail to the netdev mailing list (please
> keep the Amogic list at least Cc'ed) and ask for help there
> the Amlogic SoCs use a DesignWare MAC (Ethernet controller, the driver
> is called stmmac) with a Relatek RTL8211F Ethernet PHY.
> there's little Amlogic specific registers involved: they mostly
> control the PHY interface (enabling RMII or RGMII) and the clocks
> so it's very likely that someone on the netdev list has an idea how to
> debug this because a large part of the Ethernet setup is not Amlogic
> SoC specific
> 

Okay, I'll do it by the end of the week!

BR,

Aymeric.

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic