RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Martin Blumenstingl]

Hello,

Amlogic's 12nm SoC generation requires some RGMII timing calibration
within the Ethernet controller glue registers.
This calibration is only needed for the RGMII modes, not for the
(internal) RMII PHY.
With "incorrect" calibration settings Ethernet speeds up to 100Mbit/s
will still work fine, but no data is flowing on 1Gbit/s connections
(similar to when RX or TX delay settings are incorrect).

A high-level description of this calibration (the full code can be
seen in [0] and [1]):
- there are sixteen possible calibration values: [0..15]
- switch the Ethernet PHY to loopback mode
- for each of the sixteen possible calibration values repeat the
following steps five times:
-- write the value to the calibration register
-- construct an Ethernet loopback test frame with protocol 0x0808
("Frame Relay ARP")
-- add 256 bytes of arbitrary data
-- use the MAC address of the controller as source and destination
-- send out this data packet
-- receive this data packet
-- compare the contents and remember if the data is valid or corrupted
- disable loopback mode on the Ethernet PHY
- find the best calibration value by getting the center point of the
"longest streak"
- write this value to the calibration register

My question is: how do I integrate this into the dwmac-meson8b (stmmac
based) driver?
I already found some interesting and relevant bits:
- stmmac_selftests.c uses phy_loopback() and also constructs data
which is sent-out in loopback mode
- there's a serdes_powerup callback in struct plat_stmmacenet_data
which is called after register_netdev()
- I'm not sure if there's any other Ethernet driver doing some similar
calibration (and therefore a way to avoid some code-duplication)


Any recommendations/suggestions/ideas/hints are welcome!
Thank you and best regards,
Martin


[0] https://github.com/khadas/u-boot/blob/4752efbb90b7d048a81760c67f8c826f14baf41c/drivers/net/designware.c#L707
[1] https://github.com/khadas/linux/blob/khadas-vims-4.9.y/drivers/net/ethernet/stmicro/stmmac/dwmac-meson.c#L466

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Vladimir Oltean]

Hi Martin,

On Fri, Sep 25, 2020 at 11:47:18PM +0200, Martin Blumenstingl wrote:
> Hello,
>
> Amlogic's 12nm SoC generation requires some RGMII timing calibration
> within the Ethernet controller glue registers.
> This calibration is only needed for the RGMII modes, not for the
> (internal) RMII PHY.
> With "incorrect" calibration settings Ethernet speeds up to 100Mbit/s
> will still work fine, but no data is flowing on 1Gbit/s connections
> (similar to when RX or TX delay settings are incorrect).
>
> A high-level description of this calibration (the full code can be
> seen in [0] and [1]):
> - there are sixteen possible calibration values: [0..15]
> - switch the Ethernet PHY to loopback mode
> - for each of the sixteen possible calibration values repeat the
> following steps five times:
> -- write the value to the calibration register
> -- construct an Ethernet loopback test frame with protocol 0x0808
> ("Frame Relay ARP")
> -- add 256 bytes of arbitrary data
> -- use the MAC address of the controller as source and destination
> -- send out this data packet
> -- receive this data packet
> -- compare the contents and remember if the data is valid or corrupted
> - disable loopback mode on the Ethernet PHY
> - find the best calibration value by getting the center point of the
> "longest streak"
> - write this value to the calibration register
>
> My question is: how do I integrate this into the dwmac-meson8b (stmmac
> based) driver?
> I already found some interesting and relevant bits:
> - stmmac_selftests.c uses phy_loopback() and also constructs data
> which is sent-out in loopback mode
> - there's a serdes_powerup callback in struct plat_stmmacenet_data
> which is called after register_netdev()
> - I'm not sure if there's any other Ethernet driver doing some similar
> calibration (and therefore a way to avoid some code-duplication)
>
>
> Any recommendations/suggestions/ideas/hints are welcome!
> Thank you and best regards,
> Martin
>
>
> [0] https://github.com/khadas/u-boot/blob/4752efbb90b7d048a81760c67f8c826f14baf41c/drivers/net/designware.c#L707
> [1] https://github.com/khadas/linux/blob/khadas-vims-4.9.y/drivers/net/ethernet/stmicro/stmmac/dwmac-meson.c#L466

Florian attempted something like this before, for the PHY side of things:
https://patchwork.ozlabs.org/project/netdev/patch/20191015224953.24199-3-f.fainelli@gmail.com/

There are quite some assumptions to be made if the code is to be made
generic, such as the fact that the controller should not drop frames
with bad FCS in hardware. Or if it does, the code should be aware of
that and check that counter.

Thanks,
-Vladimir

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Andrew Lunn]

On Fri, Sep 25, 2020 at 11:47:18PM +0200, Martin Blumenstingl wrote:
> Hello,
> 
> Amlogic's 12nm SoC generation requires some RGMII timing calibration
> within the Ethernet controller glue registers.
> This calibration is only needed for the RGMII modes, not for the
> (internal) RMII PHY.
> With "incorrect" calibration settings Ethernet speeds up to 100Mbit/s
> will still work fine, but no data is flowing on 1Gbit/s connections
> (similar to when RX or TX delay settings are incorrect).

Hi Martin

Is this trying to detect the correct RGMII interface mode:
        PHY_INTERFACE_MODE_RGMII,
        PHY_INTERFACE_MODE_RGMII_ID,
        PHY_INTERFACE_MODE_RGMII_RXID,
        PHY_INTERFACE_MODE_RGMII_TXID,

In general, we recommend the MAC does not insert any delay, we leave
it up to the PHY. In DT, you then set the correct phy-mode value,
which gets passed to the PHY when the MAC calls the connect function.

Is there any documentation as to what the calibration values mean?  I
would just hard code it to whatever means 0uS delay, and be done. The
only time the MAC needs to add delays is when the PHY is not capable
of doing it, and generally, they all are.

      Andrew

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Martin Blumenstingl]

Hi Vladimir,

On Sat, Sep 26, 2020 at 12:03 AM Vladimir Oltean <olteanv@gmail.com> wrote:
[...]
> > Any recommendations/suggestions/ideas/hints are welcome!
> > Thank you and best regards,
> > Martin
> >
> >
> > [0] https://github.com/khadas/u-boot/blob/4752efbb90b7d048a81760c67f8c826f14baf41c/drivers/net/designware.c#L707
> > [1] https://github.com/khadas/linux/blob/khadas-vims-4.9.y/drivers/net/ethernet/stmicro/stmmac/dwmac-meson.c#L466
>
> Florian attempted something like this before, for the PHY side of things:
> https://patchwork.ozlabs.org/project/netdev/patch/20191015224953.24199-3-f.fainelli@gmail.com/
thank you for this hint!

> There are quite some assumptions to be made if the code is to be made
> generic, such as the fact that the controller should not drop frames
> with bad FCS in hardware. Or if it does, the code should be aware of
> that and check that counter.
I do not need the auto-detection of the phy-mode nor any RX/TX delay
(these are fixed values)
however, from that patch-set I would need most of
phy_rgmii_probe_interface() (and all of the helpers it's using)
also I'm wondering if the "protocol" 0x0808 is recommended over ETH_P_EDSA


Best regards,
Martin

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Martin Blumenstingl]

Hi Andrew,

On Sat, Sep 26, 2020 at 12:14 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> On Fri, Sep 25, 2020 at 11:47:18PM +0200, Martin Blumenstingl wrote:
> > Hello,
> >
> > Amlogic's 12nm SoC generation requires some RGMII timing calibration
> > within the Ethernet controller glue registers.
> > This calibration is only needed for the RGMII modes, not for the
> > (internal) RMII PHY.
> > With "incorrect" calibration settings Ethernet speeds up to 100Mbit/s
> > will still work fine, but no data is flowing on 1Gbit/s connections
> > (similar to when RX or TX delay settings are incorrect).
>
> Hi Martin
>
> Is this trying to detect the correct RGMII interface mode:
>         PHY_INTERFACE_MODE_RGMII,
>         PHY_INTERFACE_MODE_RGMII_ID,
>         PHY_INTERFACE_MODE_RGMII_RXID,
>         PHY_INTERFACE_MODE_RGMII_TXID,
>
> In general, we recommend the MAC does not insert any delay, we leave
> it up to the PHY. In DT, you then set the correct phy-mode value,
> which gets passed to the PHY when the MAC calls the connect function.
yes and no.
The reference code I linked tries to detect the RGMII interface mode.
However, for each board we know the phy-mode as well as the RX and TX
delay - so I'm not trying to port the RGMII interface detection part
to the mainline driver.

on X96 Air (which I'm using for testing) Amlogic configures phy-mode
"rgmii" with a 2ns TX delay provided by the MAC and 0ns RX delay
anywhere (so I'm assuming that the board adds the 2ns RX delay)
I am aware that the recommendation is to let the PHY generate the delay.
For now I'm trying to get the same configuration working which is used
by Amlogic's vendor kernel and u-boot.

> Is there any documentation as to what the calibration values mean?  I
> would just hard code it to whatever means 0uS delay, and be done. The
> only time the MAC needs to add delays is when the PHY is not capable
> of doing it, and generally, they all are.
This calibration is not the RGMII RX or TX delay - we have other
registers for that and already know how to program these.

This new calibration only exists on 12nm SoCs so I assume (but have no
proof) that they need to solve some challenge that comes with the
advanced node (previous SoCs were manufactured using a 28nm process).
In the old days the vendors put calibration data into the eFuse.
However I think for mass-production of cheap boards this is not nice
for the manufacturers (because they need this eFuse programming step).
So I think Amlogic added a calibration circuit to handle tolerances
within the SoC manufacturing as well as the "environment" of the SoC
(there are some TI SoCs where the MMC controller's clock calibration
depends on the SoC temperature for example. For these Amlogic SoCs I
don't know the factors that influence this calibration though - one
guess however is cost-cutting).

All of that said: I don't have any scope that's fast enough to see the
clock-skew on such high-speed signals so I cannot tell for sure what
problem they are solving.
What I can say is that u-boot programs calibration value 0xf (the
maximum value) on my X96 Air board. With this I cannot get Ethernet
working - regardless of how I change the RX or TX delays.
If I leave everything as-is (2ns TX delay generated by the MAC, 0ns RX
delay, ...) and change the calibration value to 0x0 or 0x3 (the latter
is set by the vendor kernel) then Ethernet starts working.


Best regards,
Martin

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Vladimir Oltean]

On Sat, Sep 26, 2020 at 12:15:59AM +0200, Martin Blumenstingl wrote:
> I do not need the auto-detection of the phy-mode nor any RX/TX delay
> (these are fixed values)
> however, from that patch-set I would need most of
> phy_rgmii_probe_interface() (and all of the helpers it's using)

So if it's not clock skews and it probably isn't equalization either,
since to my knowledge RGMII MACs won't have because they are parallel
and relatively low-speed interfaces, then we need to know what exactly
it is that you calibrate.

As you know, in a serial interface you are likely to find a BIST
function implemented in the SERDES, this would basically offload to
hardware the task of sending and decoding test patterns such as PRBS-11.
With RGMII, you are less likely to see a BIST in hardware, hence the
manual injection of packets that you need to do from software. Whatever
solution you end up choosing, it would be nice if it created a nicely
structured UAPI that could be extended in the future for other types of
electrical interface selftests.

> also I'm wondering if the "protocol" 0x0808 is recommended over ETH_P_EDSA

It probably doesn't make any difference.

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Andrew Lunn]

> The reference code I linked tries to detect the RGMII interface mode.
> However, for each board we know the phy-mode as well as the RX and TX
> delay - so I'm not trying to port the RGMII interface detection part
> to the mainline driver.
> 
> on X96 Air (which I'm using for testing) Amlogic configures phy-mode
> "rgmii" with a 2ns TX delay provided by the MAC and 0ns RX delay
> anywhere (so I'm assuming that the board adds the 2ns RX delay)

Hi Martin

It would be unusual to have an asymmetric design in the PCB. So i
would try to prove that assumption. It could be the PHY driver is
broken, and although it is configured to use RGMII, it is actually
inserting a delay on RX. Also check if the PHY has any strapping.

> I am aware that the recommendation is to let the PHY generate the delay.
> For now I'm trying to get the same configuration working which is used
> by Amlogic's vendor kernel and u-boot.
> 
> > Is there any documentation as to what the calibration values mean?  I
> > would just hard code it to whatever means 0uS delay, and be done. The
> > only time the MAC needs to add delays is when the PHY is not capable
> > of doing it, and generally, they all are.

> This calibration is not the RGMII RX or TX delay - we have other
> registers for that and already know how to program these.

O.K. so maybe this is just fine tuning. Some PHYs also allow this.

> What I can say is that u-boot programs calibration value 0xf (the
> maximum value) on my X96 Air board. With this I cannot get Ethernet
> working - regardless of how I change the RX or TX delays.
> If I leave everything as-is (2ns TX delay generated by the MAC, 0ns RX
> delay, ...) and change the calibration value to 0x0 or 0x3 (the latter
> is set by the vendor kernel) then Ethernet starts working.

So there is just one calibration value? So it assumes the calibration
is symmetric for both RX and TX.

What PHY is it using?

https://dpaste.com/2WJF9EN suggests it is a RTL8211F.

This device does have stripping to set the default delay. Can you
check if there are pull ups on pins 24 and 25?

What i find interesting is in the driver is:

        ret = phy_modify_paged_changed(phydev, 0xd08, 0x11, RTL8211F_TX_DELAY,
                                       val_txdly);

        ret = phy_modify_paged_changed(phydev, 0xd08, 0x15, RTL8211F_RX_DELAY,
                                       val_rxdly);

Different registers, 0x11 vs 0x15. In the datasheets i found with
google, none describe any of these bits, but at least register 0x15 is
mentioned, where as register 0x11 is not.

Git blame shows you added this! Are you sure about this? It seems odd
they are in different registers.

     Andrew

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Martin Blumenstingl]

Hi Andrew,

On Sat, Sep 26, 2020 at 2:41 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > The reference code I linked tries to detect the RGMII interface mode.
> > However, for each board we know the phy-mode as well as the RX and TX
> > delay - so I'm not trying to port the RGMII interface detection part
> > to the mainline driver.
> >
> > on X96 Air (which I'm using for testing) Amlogic configures phy-mode
> > "rgmii" with a 2ns TX delay provided by the MAC and 0ns RX delay
> > anywhere (so I'm assuming that the board adds the 2ns RX delay)
>
> Hi Martin
>
> It would be unusual to have an asymmetric design in the PCB. So i
> would try to prove that assumption. It could be the PHY driver is
> broken, and although it is configured to use RGMII, it is actually
> inserting a delay on RX. Also check if the PHY has any strapping.
I checked this again for the vendor u-boot (where Ethernet is NOT
working) as well as the Android kernel which this board was shipped
with (where Ethernet is working)
- in u-boot the MAC side adds a 2ns TX delay and the PHY side adds a
2ns RX delay
- in Linux the MAC side adds a 2ns TX delay and the RX delay is turned
off (for both, MAC and PHY)

> > I am aware that the recommendation is to let the PHY generate the delay.
> > For now I'm trying to get the same configuration working which is used
> > by Amlogic's vendor kernel and u-boot.
> >
> > > Is there any documentation as to what the calibration values mean?  I
> > > would just hard code it to whatever means 0uS delay, and be done. The
> > > only time the MAC needs to add delays is when the PHY is not capable
> > > of doing it, and generally, they all are.
>
> > This calibration is not the RGMII RX or TX delay - we have other
> > registers for that and already know how to program these.
>
> O.K. so maybe this is just fine tuning. Some PHYs also allow this.
>
> > What I can say is that u-boot programs calibration value 0xf (the
> > maximum value) on my X96 Air board. With this I cannot get Ethernet
> > working - regardless of how I change the RX or TX delays.
> > If I leave everything as-is (2ns TX delay generated by the MAC, 0ns RX
> > delay, ...) and change the calibration value to 0x0 or 0x3 (the latter
> > is set by the vendor kernel) then Ethernet starts working.
>
> So there is just one calibration value? So it assumes the calibration
> is symmetric for both RX and TX.
yes, there's only one calibration value
the reference code is calculating the calibration setting for four
configuration variants:
- 2ns TX delay on the MAC side, no RX or TX delay on the PHY side,
RGMII RX_CLK not inverted
- 2ns TX delay on the MAC side, no RX or TX delay on the PHY side,
RGMII RX_CLK inverted
- 2ns TX delay on the MAC side, 2ns RX delay on the PHY side, RGMII
RX_CLK not inverted
- 2ns TX delay on the MAC side, 2ns RX delay on the PHY side, RGMII
RX_CLK inverted

now that I'm writing this, could it be a calibration of the RX_CLK
signal? The TX delay is fixed in all cases but the RX delay and RX_CLK
signal inversion are variable.

> What PHY is it using?
>
> https://dpaste.com/2WJF9EN suggests it is a RTL8211F.
indeed, it's a RTL8211F

> This device does have stripping to set the default delay. Can you
> check if there are pull ups on pins 24 and 25?
I checked it in software (see above)
to let you sanity-check this:
in vendor u-boot I read: reg 0x11=0x9 and reg 0x15=0x19
in the Android kernel shipped with the board I have: reg 0x11=0x9 and
reg 0x15=0x11

note: I haven't found a way to "fix" Ethernet in u-boot so far. I
cannot get the exact u-boot copy that's used on my board (since
there's no vendor to contact).
so I'm careful with interpreting what I'm seeing in u-boot

if you really want me to I can check the pull-ups but I prefer not to
since I managed to short and break tiny solder joints in the past -
and I only have one of these boards for testing

> What i find interesting is in the driver is:
>
>         ret = phy_modify_paged_changed(phydev, 0xd08, 0x11, RTL8211F_TX_DELAY,
>                                        val_txdly);
>
>         ret = phy_modify_paged_changed(phydev, 0xd08, 0x15, RTL8211F_RX_DELAY,
>                                        val_rxdly);
>
> Different registers, 0x11 vs 0x15. In the datasheets i found with
> google, none describe any of these bits, but at least register 0x15 is
> mentioned, where as register 0x11 is not.
>
> Git blame shows you added this! Are you sure about this? It seems odd
> they are in different registers.
I asked Realtek about it back then and they were kind enough to reply
- so I got this information from Realtek
double-checking with my old mails (I have to type what's shown in the
screenshot as I'm allowed to share the info but not the screenshot
that I received):
Function Name: Manually Enable TXDLY
Function Description: Enable TXDLY by registers (default by hardware
pin configuration)
RTL8211F series: Page: 0xd08, Reg 17, bit[8] = 1

Function Name: Manually Enable RXDLY
Function Description: Enable RXDLY by registers (default by hardware
pin configuration)
RTL8211F series: Page: 0xd08, Reg 21, bit[3] = 1

In the meantime Amlogic's "hacked" PHY driver is also using these registers: [0]
So I assume that I'm doing the right thing in the Realtek PHY driver


Best regards,
Martin


[0] https://github.com/khadas/linux/blob/d140398907da5a3f4f7dba2acd336e2ef469bac2/drivers/net/phy/realtek.c#L174

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Andrew Lunn]

> I checked this again for the vendor u-boot (where Ethernet is NOT
> working) as well as the Android kernel which this board was shipped
> with (where Ethernet is working)
> - in u-boot the MAC side adds a 2ns TX delay and the PHY side adds a
> 2ns RX delay

So that suggest there is nothing on the PCB. It is all down to MAC and
PHY adding delays.

> yes, there's only one calibration value
> the reference code is calculating the calibration setting for four
> configuration variants:
> - 2ns TX delay on the MAC side, no RX or TX delay on the PHY side, RGMII RX_CLK not inverted
> - 2ns TX delay on the MAC side, no RX or TX delay on the PHY side, RGMII RX_CLK inverted
> - 2ns TX delay on the MAC side, 2ns RX delay on the PHY side, RGMII RX_CLK not inverted
> - 2ns TX delay on the MAC side, 2ns RX delay on the PHY side, RGMII RX_CLK inverted
> 
> now that I'm writing this, could it be a calibration of the RX_CLK
> signal?

Yes, seems like it. Which of these four does it end up using? I'm
guessing the 3rd?

So i would forget about configuration clock inversion. Hard code it to
whatever works. It is not something you see other MAC/PHY combinations
allow to configure.

I think you said a value of 0x2 works. I wonder if that corresponds to
something slightly larger than 0ns if option 3 is being used?

> In the meantime Amlogic's "hacked" PHY driver is also using these registers: [0]
> So I assume that I'm doing the right thing in the Realtek PHY driver

Thanks for confirming this. No need to check for the resistors.

       Andrew

Re: RGMII timing calibration (on 12nm Amlogic SoCs) - integration into dwmac-meson8b

[Martin Blumenstingl]

Hi Andrew,

On Sat, Sep 26, 2020 at 4:45 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > I checked this again for the vendor u-boot (where Ethernet is NOT
> > working) as well as the Android kernel which this board was shipped
> > with (where Ethernet is working)
> > - in u-boot the MAC side adds a 2ns TX delay and the PHY side adds a
> > 2ns RX delay
>
> So that suggest there is nothing on the PCB. It is all down to MAC and
> PHY adding delays.
u-boot with it's 2ns RX delay is the non-working case
only if I manually turn off the 2ns RX delay generated by the PHY in
u-boot (phyreg w 0x1f 0xd08; phyreg w 0x11 0x9; phyreg w 0x15 0x11;
phyreg w 0x1f 0x0; phyreg w 0x0 0x9200) I can get ping/tftpboot to
work

the Android kernel disables the 2ns RX delay on the PHY side (and as
far as I can tell does NOT enable it on the MAC side). with that
Ethernet is working

> > yes, there's only one calibration value
> > the reference code is calculating the calibration setting for four
> > configuration variants:
> > - 2ns TX delay on the MAC side, no RX or TX delay on the PHY side, RGMII RX_CLK not inverted
> > - 2ns TX delay on the MAC side, no RX or TX delay on the PHY side, RGMII RX_CLK inverted
> > - 2ns TX delay on the MAC side, 2ns RX delay on the PHY side, RGMII RX_CLK not inverted
> > - 2ns TX delay on the MAC side, 2ns RX delay on the PHY side, RGMII RX_CLK inverted
> >
> > now that I'm writing this, could it be a calibration of the RX_CLK
> > signal?
>
> Yes, seems like it. Which of these four does it end up using? I'm
> guessing the 3rd?
I need to double-check but if I remember correctly was close between
the first and last one (and I think the first case won)

> So i would forget about configuration clock inversion. Hard code it to
> whatever works. It is not something you see other MAC/PHY combinations
> allow to configure.
we have inversion hard-coded to "off". I'm not planning to take this
into consideration unless there's a good reason to do so

> I think you said a value of 0x2 works. I wonder if that corresponds to
> something slightly larger than 0ns if option 3 is being used?
I have tested 0x0, 0x3, 0x4 and 0xf
the first three values are working, but 0xf isn't.

I'll try to reach someone at Amlogic to clarify the meaning of these
new register bits.
I guess Florian's patch is a good starting point for what I need -
thanks again for the suggestion Vladimir.


Thank you all!
Best regards,
Martin