[RFC] Permanent Fix for RK33 gmac 1500 mtu bug

[RFC] Permanent Fix for RK33 gmac 1500 mtu bug

[Peter Geis]

Good Morning,

So I've continued work on fixing the rk3328/rk3399 gmac mtu tx coe offload bug.
I've found two fixes that maintain full performance and work consistently.

First, is ayufan's tx coe patch [0], which takes the bugged_jumbo
concept introduced in [1] and applies it to 1498 and above, vice 1500
and above.

The only downside is this disables tx coe for full size packets, which
has a negligible performance impact in my testing.

The other option I've found that reliably works is bringing the mtu
down to 1498.
This allows tx coe to remain enabled, but with the downside of total
loss of jumbo frames.
The reduction in size has a negligible performance impact in my testing.

I have also discovered that the rockchip implementation of the stmmac
driver does not process flags such as max-frame-size.

A third option, which I haven't explored because I don't know enough
about how it works, is possibly tuning the axi settings, via the
snps,axi-config and snps,mtl-tx-config handles.
I don't know if this is feasible, but since tuning the dma settings
affects the rk3328 I have hope.
I do know that my current fix for the rk3328 does not provide full
performance and does not work at all on the rk3399.

Thoughts?

[0] https://github.com/ayufan-rock64/linux-kernel/commit/8a41c672dd77e48b06c1b2dec3aa9db4bad30b49#diff-c897c0b53bd633240f4b12c4d29a5ff1
[1] https://github.com/torvalds/linux/commit/ebbb293f8b3021ae2009fcb7cb3b8a52fb5fd06a

Re: [RFC] Permanent Fix for RK33 gmac 1500 mtu bug

[Peter Geis]

On Thu, Dec 12, 2019 at 8:21 AM Peter Geis <pgwipeout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Good Morning,
>
> So I've continued work on fixing the rk3328/rk3399 gmac mtu tx coe offload bug.
> I've found two fixes that maintain full performance and work consistently.
>
> First, is ayufan's tx coe patch [0], which takes the bugged_jumbo
> concept introduced in [1] and applies it to 1498 and above, vice 1500
> and above.
>
> The only downside is this disables tx coe for full size packets, which
> has a negligible performance impact in my testing.
>
> The other option I've found that reliably works is bringing the mtu
> down to 1498.
> This allows tx coe to remain enabled, but with the downside of total
> loss of jumbo frames.
> The reduction in size has a negligible performance impact in my testing.

Shortly after sending this I discovered that setting the mtu lower is
not sufficient in some corner cases.
I managed to make offload break even at a 1496 mtu by `apt install
kubuntu-desktop` over ssh on a gigabit internet connection.

After porting ayufan's patch the issue went away.
So unless we can fix this by axi tuning, his patch seems to be the
most viable option.

>
> I have also discovered that the rockchip implementation of the stmmac
> driver does not process flags such as max-frame-size.
>
> A third option, which I haven't explored because I don't know enough
> about how it works, is possibly tuning the axi settings, via the
> snps,axi-config and snps,mtl-tx-config handles.
> I don't know if this is feasible, but since tuning the dma settings
> affects the rk3328 I have hope.
> I do know that my current fix for the rk3328 does not provide full
> performance and does not work at all on the rk3399.
>
> Thoughts?
>
> [0] https://github.com/ayufan-rock64/linux-kernel/commit/8a41c672dd77e48b06c1b2dec3aa9db4bad30b49#diff-c897c0b53bd633240f4b12c4d29a5ff1
> [1] https://github.com/torvalds/linux/commit/ebbb293f8b3021ae2009fcb7cb3b8a52fb5fd06a

Re: [RFC] Permanent Fix for RK33 gmac 1500 mtu bug

[Hugh Cole-Baker]

Hi Peter,

> On 13 Dec 2019, at 12:48, Peter Geis <pgwipeout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> On Thu, Dec 12, 2019 at 8:21 AM Peter Geis <pgwipeout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> 
>> Good Morning,
>> 
>> So I've continued work on fixing the rk3328/rk3399 gmac mtu tx coe offload bug.
>> I've found two fixes that maintain full performance and work consistently.
>> 
>> First, is ayufan's tx coe patch [0], which takes the bugged_jumbo
>> concept introduced in [1] and applies it to 1498 and above, vice 1500
>> and above.
>> 
>> The only downside is this disables tx coe for full size packets, which
>> has a negligible performance impact in my testing.
>> 
>> The other option I've found that reliably works is bringing the mtu
>> down to 1498.
>> This allows tx coe to remain enabled, but with the downside of total
>> loss of jumbo frames.
>> The reduction in size has a negligible performance impact in my testing.
> 
> Shortly after sending this I discovered that setting the mtu lower is
> not sufficient in some corner cases.
> I managed to make offload break even at a 1496 mtu by `apt install
> kubuntu-desktop` over ssh on a gigabit internet connection.
> 
> After porting ayufan's patch the issue went away.
> So unless we can fix this by axi tuning, his patch seems to be the
> most viable option.
> 

Have you tried suggestions from Jose in https://lkml.org/lkml/2019/4/1/1382?
I've added "snps,no-pbl-x8;" and "snps,txpbl = <0x20>;" to the
gmac in the DT, on rk3399-rockpro64. This seems to fix the slow performance
on IPv6 specifically that I was seeing.

I haven't done exhaustive testing beyond a few runs of iperf3, which seem
to show OK performance for a gigabit link on home networking hardware.
874 Mbits/sec for IPv4, 856 Mbits/sec for IPv6, with MTU 1500.

>> 
>> 
>> I have also discovered that the rockchip implementation of the stmmac
>> driver does not process flags such as max-frame-size.
>> 
>> A third option, which I haven't explored because I don't know enough
>> about how it works, is possibly tuning the axi settings, via the
>> snps,axi-config and snps,mtl-tx-config handles.
>> I don't know if this is feasible, but since tuning the dma settings
>> affects the rk3328 I have hope.
>> I do know that my current fix for the rk3328 does not provide full
>> performance and does not work at all on the rk3399.
>> 
>> Thoughts?
>> 
>> [0] https://github.com/ayufan-rock64/linux-kernel/commit/8a41c672dd77e48b06c1b2dec3aa9db4bad30b49#diff-c897c0b53bd633240f4b12c4d29a5ff1
>> [1] https://github.com/torvalds/linux/commit/ebbb293f8b3021ae2009fcb7cb3b8a52fb5fd06a
> 
> _______________________________________________
> Linux-rockchip mailing list
> Linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] Permanent Fix for RK33 gmac 1500 mtu bug

[Peter Geis]

On Sat, Dec 14, 2019 at 9:30 AM Hugh Cole-Baker <sigmaris-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi Peter,
>
> > On 13 Dec 2019, at 12:48, Peter Geis <pgwipeout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > On Thu, Dec 12, 2019 at 8:21 AM Peter Geis <pgwipeout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>
> >> Good Morning,
> >>
> >> So I've continued work on fixing the rk3328/rk3399 gmac mtu tx coe offload bug.
> >> I've found two fixes that maintain full performance and work consistently.
> >>
> >> First, is ayufan's tx coe patch [0], which takes the bugged_jumbo
> >> concept introduced in [1] and applies it to 1498 and above, vice 1500
> >> and above.
> >>
> >> The only downside is this disables tx coe for full size packets, which
> >> has a negligible performance impact in my testing.
> >>
> >> The other option I've found that reliably works is bringing the mtu
> >> down to 1498.
> >> This allows tx coe to remain enabled, but with the downside of total
> >> loss of jumbo frames.
> >> The reduction in size has a negligible performance impact in my testing.
> >
> > Shortly after sending this I discovered that setting the mtu lower is
> > not sufficient in some corner cases.
> > I managed to make offload break even at a 1496 mtu by `apt install
> > kubuntu-desktop` over ssh on a gigabit internet connection.
> >
> > After porting ayufan's patch the issue went away.
> > So unless we can fix this by axi tuning, his patch seems to be the
> > most viable option.
> >
>
> Have you tried suggestions from Jose in https://lkml.org/lkml/2019/4/1/1382?
> I've added "snps,no-pbl-x8;" and "snps,txpbl = <0x20>;" to the
> gmac in the DT, on rk3399-rockpro64. This seems to fix the slow performance
> on IPv6 specifically that I was seeing.
>
> I haven't done exhaustive testing beyond a few runs of iperf3, which seem
> to show OK performance for a gigabit link on home networking hardware.
> 874 Mbits/sec for IPv4, 856 Mbits/sec for IPv6, with MTU 1500.

I'll test this out as well.
Unfortunately I didn't catch that reducing the mtu size would break
under heavy load because I was using iperf3 to test.
It wasn't until I did a massive apt upgrade over ssh did it trigger.

I don't experience any difference between ipv4 and ipv6, both work equally well.
With ayufan's selective coe offload disable patch, I'm hitting about
960Mbits/sec on iperf3.

For clarification sake, this is on 5.4.
>
> >>
> >>
> >> I have also discovered that the rockchip implementation of the stmmac
> >> driver does not process flags such as max-frame-size.
> >>
> >> A third option, which I haven't explored because I don't know enough
> >> about how it works, is possibly tuning the axi settings, via the
> >> snps,axi-config and snps,mtl-tx-config handles.
> >> I don't know if this is feasible, but since tuning the dma settings
> >> affects the rk3328 I have hope.
> >> I do know that my current fix for the rk3328 does not provide full
> >> performance and does not work at all on the rk3399.
> >>
> >> Thoughts?
> >>
> >> [0] https://github.com/ayufan-rock64/linux-kernel/commit/8a41c672dd77e48b06c1b2dec3aa9db4bad30b49#diff-c897c0b53bd633240f4b12c4d29a5ff1
> >> [1] https://github.com/torvalds/linux/commit/ebbb293f8b3021ae2009fcb7cb3b8a52fb5fd06a
> >
> > _______________________________________________
> > Linux-rockchip mailing list
> > Linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> > http://lists.infradead.org/mailman/listinfo/linux-rockchip
>

Re: [RFC] Permanent Fix for RK33 gmac 1500 mtu bug

[Peter Geis]

<snip>

Good Afternoon,

There has been significant code dumped in to 5.4.x by Jose Abreu
<Jose.Abreu-HKixBCOQz3hWk0Htik3J/w@public.gmane.org>.
Today I clean built 5.4.10 and it seems at least on the rk3399 our tx
offload issue is resolved.

I'm currently load testing, but if a few others would be willing to
test as well it would be greatly appreciated.

Happy New Year!

Re: [RFC] Permanent Fix for RK33 gmac 1500 mtu bug

[Peter Geis]

On Fri, Jan 10, 2020 at 2:36 PM Peter Geis <pgwipeout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> <snip>
>
> Good Afternoon,
>
> There has been significant code dumped in to 5.4.x by Jose Abreu
> <Jose.Abreu-HKixBCOQz3hWk0Htik3J/w@public.gmane.org>.
> Today I clean built 5.4.10 and it seems at least on the rk3399 our tx
> offload issue is resolved.

Just a note, you'll need to cherry pick the following commit,
otherwise ethernet is broken on 5.4.10
commit 42ed6e2fa9f167ce8197a41ebcf8974c4d90f9d4 (HEAD -> rk3399-rockpro64)
Author: Jose Abreu <Jose.Abreu-HKixBCOQz3hWk0Htik3J/w@public.gmane.org>
Date:   Tue Jan 7 13:35:42 2020 +0100

    net: stmmac: Fixed link does not need MDIO Bus

>
> I'm currently load testing, but if a few others would be willing to
> test as well it would be greatly appreciated.
>
> Happy New Year!