Waiting for the PHY to complete auto-negotiation

Waiting for the PHY to complete auto-negotiation

[Mason]

Hello,

I've been trying to wrap my head around Ethernet auto-negotiation,
vs actual / real packets seen at the MAC layer. I found the relevant
Wikipedia article to be fairly informative:

    https://en.wikipedia.org/wiki/Autonegotiation

The reason I care is that my Ethernet HW does not allow changing the
flow control setting once the MAC has started (more specifically, once
RX DMA has been enabled).

In nb8800_open(), the code currently works in this order:

	nb8800_start_rx(dev);
	phy_start(phydev);

The first line enables the MAC (and DMA).
The second enables the PHY and starts auto-negotiation.

This is a problem: I would like for PHY auto-negotiation to be
/complete/ before I enable the MAC.

What is the recommended way to wait for the PHY?

AFAICT, the PHY layer calls back into the eth driver through the
adjust_link() callback registered through of_phy_connect().
It seems like this might be a good place to enable the MAC?
(When some other conditions are true.)

Regards.

Re: Waiting for the PHY to complete auto-negotiation

[Andrew Lunn]

On Wed, Dec 06, 2017 at 05:39:00PM +0100, Mason wrote:
> Hello,
> 
> I've been trying to wrap my head around Ethernet auto-negotiation,
> vs actual / real packets seen at the MAC layer. I found the relevant
> Wikipedia article to be fairly informative:
> 
>     https://en.wikipedia.org/wiki/Autonegotiation
> 
> The reason I care is that my Ethernet HW does not allow changing the
> flow control setting once the MAC has started (more specifically, once
> RX DMA has been enabled).
> 
> In nb8800_open(), the code currently works in this order:
> 
> 	nb8800_start_rx(dev);
> 	phy_start(phydev);
> 
> The first line enables the MAC (and DMA).
> The second enables the PHY and starts auto-negotiation.
> 
> This is a problem: I would like for PHY auto-negotiation to be
> /complete/ before I enable the MAC.
> 
> What is the recommended way to wait for the PHY?

> AFAICT, the PHY layer calls back into the eth driver through the
> adjust_link() callback registered through of_phy_connect().
> It seems like this might be a good place to enable the MAC?

That probably works, but you might have a few corner cases to handle.
I'm not sure changes at the PHY always transition through down. So you
could for example get a callback saying the link is up, 1Gbps, then a
second call saying it has dropped to 100Mbps, if your
cables/connectors are bad.

If your hardware has problems, it might be safest to stop everything
in the callback, make configuration changes, and they start everything
back up. A link negotiation change is not something you expect to
happen often. So making it slow but robust is O.K.

       Andrew

Re: Waiting for the PHY to complete auto-negotiation

[Mason]

On 06/12/2017 17:59, Andrew Lunn wrote:

> On Wed, Dec 06, 2017 at 05:39:00PM +0100, Mason wrote:
>
>> I've been trying to wrap my head around Ethernet auto-negotiation,
>> vs actual / real packets seen at the MAC layer. I found the relevant
>> Wikipedia article to be fairly informative:
>>
>>     https://en.wikipedia.org/wiki/Autonegotiation
>>
>> The reason I care is that my Ethernet HW does not allow changing the
>> flow control setting once the MAC has started (more specifically, once
>> RX DMA has been enabled).
>>
>> In nb8800_open(), the code currently works in this order:
>>
>> 	nb8800_start_rx(dev);
>> 	phy_start(phydev);
>>
>> The first line enables the MAC (and DMA).
>> The second enables the PHY and starts auto-negotiation.
>>
>> This is a problem: I would like for PHY auto-negotiation to be
>> /complete/ before I enable the MAC.
>>
>> What is the recommended way to wait for the PHY?
> 
>> AFAICT, the PHY layer calls back into the eth driver through the
>> adjust_link() callback registered through of_phy_connect().
>> It seems like this might be a good place to enable the MAC?
> 
> That probably works, but you might have a few corner cases to handle.
> I'm not sure changes at the PHY always transition through down. So you
> could for example get a callback saying the link is up, 1Gbps, then a
> second call saying it has dropped to 100Mbps, if your
> cables/connectors are bad.
> 
> If your hardware has problems, it might be safest to stop everything
> in the callback, make configuration changes, and they start everything
> back up. A link negotiation change is not something you expect to
> happen often. So making it slow but robust is O.K.

What you've described is, in fact, the existing implementation! ;-)

nb8800_pause_config() checks for netif_running() and, when true,
tries to disable everything, make the change, then re-enable
everything.

The problem with this is the following mind-boggling quirk of
the hardware: once RX DMA is enabled, there is no supported
way to disable it! Thus, I'm trying to find a clean way to set
the control flow parameter BEFORE enabling the MAC.

Regards.

Re: Waiting for the PHY to complete auto-negotiation

[Andrew Lunn]

> The problem with this is the following mind-boggling quirk of
> the hardware: once RX DMA is enabled, there is no supported
> way to disable it! Thus, I'm trying to find a clean way to set
> the control flow parameter BEFORE enabling the MAC.

There is no solution. I can pull the cable out, and plug it into a
different computer. The first could use flow control, the second not.

If you cannot disable RX DMA, you should probably disable unloading of
the module. Also kexec.

    Andrew

Re: Waiting for the PHY to complete auto-negotiation

[Mason]

On 06/12/2017 19:26, Andrew Lunn wrote:

> On 06/12/2017 19:03, Mason wrote:
>
>> The problem with this is the following mind-boggling quirk of
>> the hardware: once RX DMA is enabled, there is no supported
>> way to disable it! Thus, I'm trying to find a clean way to set
>> the control flow parameter BEFORE enabling the MAC.
> 
> There is no solution. I can pull the cable out, and plug it into a
> different computer. The first could use flow control, the second not.
> 
> If you cannot disable RX DMA, you should probably disable unloading of
> the module. Also kexec.

It is not possible to /selectively/ disable RX DMA, but after a reset,
DMA is disabled. So the solution I propose is this:

When we detect link down, we put the chip in reset, so that we will
repeat initialization when the link comes back up.

Mans has been reluctant to adopt this approach, but this is what
I implemented in my forked version. I'm trying to push my changes
upstream, to minimize the differences.

Regards.

Re: Waiting for the PHY to complete auto-negotiation

[Andrew Lunn]

> When we detect link down, we put the chip in reset, so that we will
> repeat initialization when the link comes back up.

By chip, you mean the Ethernet controller? Not the whole SoC?

   Andrew

Re: Waiting for the PHY to complete auto-negotiation

[Mason]

On 06/12/2017 20:07, Andrew Lunn wrote:

> By chip, you mean the Ethernet controller? Not the whole SoC?

Doh! Yes. Let me rephrase.

When we detect link down, we put the ethernet HW block in reset,
and repeat initialization when the link comes back up.

Hmmm, however, at the moment, I only reset on an administrative
(user-requested) link down, i.e. through ndo_stop. I would probably
have to handle cable unplug/replug events as well.

Or just consider the quirk to make flow control too complicated
to implement correctly...

Regards.

Re: Waiting for the PHY to complete auto-negotiation

[Florian Fainelli]

On 12/06/2017 11:25 AM, Mason wrote:
> On 06/12/2017 20:07, Andrew Lunn wrote:
> 
>> By chip, you mean the Ethernet controller? Not the whole SoC?
> 
> Doh! Yes. Let me rephrase.
> 
> When we detect link down, we put the ethernet HW block in reset,
> and repeat initialization when the link comes back up.
> 
> Hmmm, however, at the moment, I only reset on an administrative
> (user-requested) link down, i.e. through ndo_stop. I would probably
> have to handle cable unplug/replug events as well.
> 
> Or just consider the quirk to make flow control too complicated
> to implement correctly...

I suppose your procedure is fine, but don't you have a better way to
resolve that by trying to place a special RX DMA ring entry that allows
your RX DMA not to be entirely stopped, but intentionally looped through
a buffer that you control? As long as you can stop the Ethernet MAC RX,
working with such a limitation is probably fine, but this really sounds
like a huge pain in the butt and a major HW flaw.
-- 
Florian

Re: Waiting for the PHY to complete auto-negotiation

[Mason]

On 07/12/2017 00:00, Florian Fainelli wrote:

> On 12/06/2017 11:25 AM, Mason wrote:
>
>> When we detect link down, we put the ethernet HW block in reset,
>> and repeat initialization when the link comes back up.
>>
>> Hmmm, however, at the moment, I only reset on an administrative
>> (user-requested) link down, i.e. through ndo_stop. I would probably
>> have to handle cable unplug/replug events as well.
>>
>> Or just consider the quirk to make flow control too complicated
>> to implement correctly...
> 
> I suppose your procedure is fine, but don't you have a better way to
> resolve that by trying to place a special RX DMA ring entry that allows
> your RX DMA not to be entirely stopped, but intentionally looped through
> a buffer that you control? As long as you can stop the Ethernet MAC RX,
> working with such a limitation is probably fine, but this really sounds
> like a huge pain in the butt and a major HW flaw.

Could you elaborate a bit on your suggestion?
(Special ring entry, looped through a buffer under my control)
Is this a typical thing to do to stop DMA?

Currently the driver tries to stop DMA in nb8800_dma_stop(),
which does the following:

http://elixir.free-electrons.com/linux/latest/source/drivers/net/ethernet/aurora/nb8800.c#L881

1) poll until TX finishes (I assume the system no longer accepts new
   frames to send at this point)
2) set the EOC (end of chain) bit on all descriptors (could there be
   a problem if we receive a frame at that moment? Don't we need some
   kind of lock?)
3) disable address filtering (need to check what this does)
4) enable loop-back mode
5) send up to 5 "fake" packets in order to hit an EOC descriptor

The reason I'm trying to move away from this method is that it doesn't
work on our new SoC; and when pressed, the HW dev said it had never been
supported. (Also I find it somewhat hackish, but that's a matter of taste.)

Regards.

Re: Waiting for the PHY to complete auto-negotiation

[Florian Fainelli]

On 12/07/2017 08:17 AM, Mason wrote:
> On 07/12/2017 00:00, Florian Fainelli wrote:
> 
>> On 12/06/2017 11:25 AM, Mason wrote:
>>
>>> When we detect link down, we put the ethernet HW block in reset,
>>> and repeat initialization when the link comes back up.
>>>
>>> Hmmm, however, at the moment, I only reset on an administrative
>>> (user-requested) link down, i.e. through ndo_stop. I would probably
>>> have to handle cable unplug/replug events as well.
>>>
>>> Or just consider the quirk to make flow control too complicated
>>> to implement correctly...
>>
>> I suppose your procedure is fine, but don't you have a better way to
>> resolve that by trying to place a special RX DMA ring entry that allows
>> your RX DMA not to be entirely stopped, but intentionally looped through
>> a buffer that you control? As long as you can stop the Ethernet MAC RX,
>> working with such a limitation is probably fine, but this really sounds
>> like a huge pain in the butt and a major HW flaw.
> 
> Could you elaborate a bit on your suggestion?
> (Special ring entry, looped through a buffer under my control)
> Is this a typical thing to do to stop DMA?

What I was thinking is something along the lines of what step 5 does
already, without requiring the need to loop back and sending 5 packets,
just placing an EOC descriptor, and forcing a reduction of the RX ring
size such that you hit that descriptor right away, and have DMA be
contained within that single entry in order to safely stop it. Question
is of course, how to take it out of that state considering that usually
the RX DMA produces status bits (unlike TX where the CPU produces those
bits).

> 
> Currently the driver tries to stop DMA in nb8800_dma_stop(),
> which does the following:
> 
> http://elixir.free-electrons.com/linux/latest/source/drivers/net/ethernet/aurora/nb8800.c#L881
> 
> 1) poll until TX finishes (I assume the system no longer accepts new
>    frames to send at this point)
> 2) set the EOC (end of chain) bit on all descriptors (could there be
>    a problem if we receive a frame at that moment? Don't we need some
>    kind of lock?)

If you have disabled the RX interrupt and quiesced NAPI, this should be
safe.

> 3) disable address filtering (need to check what this does)
> 4) enable loop-back mode
> 5) send up to 5 "fake" packets in order to hit an EOC descriptor

That's quite unusual.

> 
> The reason I'm trying to move away from this method is that it doesn't
> work on our new SoC; and when pressed, the HW dev said it had never been
> supported. (Also I find it somewhat hackish, but that's a matter of taste.)

Having any HW state machine requiring X number of clock cycles to
guarantee a full transition to a stopped state is not unusual, however,
the fact that you need to send 5 packets to guarantee an EOC descriptor
is hit is completely unusual. Ideally there is a single bit that tells
the DMA engine: you are enabled, do your thing, or you are now disabled,
and you must stop all accesses to DRAM *now*.

So what would be the correct way to quiesce that controller according to
your HW folks?
-- 
Florian

Re: Waiting for the PHY to complete auto-negotiation

[Mason]

+ Mans

On 09/12/2017 19:49, Florian Fainelli wrote:

> Having any HW state machine requiring X number of clock cycles to
> guarantee a full transition to a stopped state is not unusual, however,
> the fact that you need to send 5 packets to guarantee an EOC descriptor
> is hit is completely unusual. Ideally there is a single bit that tells
> the DMA engine: you are enabled, do your thing, or you are now disabled,
> and you must stop all accesses to DRAM *now*.
> 
> So what would be the correct way to quiesce that controller according to
> your HW folks?

He (it's a single person) offered to run some RTL-level simulations,
but then moved on to more important tasks. At some point he wrote:

> If you reset the DMA enable in the middle of a DMA, the hardware
> state machine doesn't return to the IDLE state if it has more
> descriptors to process. It should be noted that the RX DMA has been
> designed by our IP vendor with no intention of stopping the DMA in
> the middle of its operation.


While the documentation for the IP states:

> Receive DMA Channel Disabling
> 
> When the entire receive frame has been read from the Receive FIFO and
> sent over the AMBA bus, the DMA operation ends, and the Receive DMA
> Channel is automatically disabled.  To do this, hardware resets the
> Enable bit in the Receive Channel Control Register to "0" after the
> last data has been read from the Receive FIFO and sent over the AMBA
> bus.
> 
> When operating in descriptor mode, upon completion of a receive frame
> DMA operation, if the descriptor chain has not ended when a receive
> frame DMA operation completes, the next receive frame DMA operation
> begins.  The last descriptor in a descriptor chain is indicated by
> having its End Of Chain- EOC, flag set to "1".  If this EOC flag is
> "0", to begin the next receive frame DMA operation, the next
> descriptor is automatically retrieved and used to configure the
> Receive DMA Channel.  The Receive DMA Channel is then automatically
> re-enabled and the next receive frame DMA operation begins.
> 
> In descriptor mode, an AMBA bus error can occur when reading receive
> descriptor data.  If this happens, receive descriptor processing ends
> and the Receive DMA Channel is turned off.  The Descriptor Error bit
> in the Receive Status Register is set to "1".


I don't see how sending "fake" packets through the loop back would be
considered "resetting the DMA enable in the middle of a DMA".
(I'm afraid the HW dev didn't grasp what the driver is doing.)

I suppose I should test forcefully setting the enable bit to 0 in
the driver, and see if hell breaks loose.

Regards.

Re: Waiting for the PHY to complete auto-negotiation

[Måns Rullgård]

Mason <slash.tmp@free.fr> writes:

> + Mans
>
> On 09/12/2017 19:49, Florian Fainelli wrote:
>
>> Having any HW state machine requiring X number of clock cycles to
>> guarantee a full transition to a stopped state is not unusual, however,
>> the fact that you need to send 5 packets to guarantee an EOC descriptor
>> is hit is completely unusual. Ideally there is a single bit that tells
>> the DMA engine: you are enabled, do your thing, or you are now disabled,
>> and you must stop all accesses to DRAM *now*.

That's how sane hardware works.  This hardware is not sane.

>> So what would be the correct way to quiesce that controller according to
>> your HW folks?
>
> He (it's a single person) offered to run some RTL-level simulations,
> but then moved on to more important tasks. At some point he wrote:
>
>> If you reset the DMA enable in the middle of a DMA, the hardware
>> state machine doesn't return to the IDLE state if it has more
>> descriptors to process. It should be noted that the RX DMA has been
>> designed by our IP vendor with no intention of stopping the DMA in
>> the middle of its operation.
>
> While the documentation for the IP states:
>
>> Receive DMA Channel Disabling
>> 
>> When the entire receive frame has been read from the Receive FIFO and
>> sent over the AMBA bus, the DMA operation ends, and the Receive DMA
>> Channel is automatically disabled.  To do this, hardware resets the
>> Enable bit in the Receive Channel Control Register to "0" after the
>> last data has been read from the Receive FIFO and sent over the AMBA
>> bus.
>> 
>> When operating in descriptor mode, upon completion of a receive frame
>> DMA operation, if the descriptor chain has not ended when a receive
>> frame DMA operation completes, the next receive frame DMA operation
>> begins.  The last descriptor in a descriptor chain is indicated by
>> having its End Of Chain- EOC, flag set to "1".  If this EOC flag is
>> "0", to begin the next receive frame DMA operation, the next
>> descriptor is automatically retrieved and used to configure the
>> Receive DMA Channel.  The Receive DMA Channel is then automatically
>> re-enabled and the next receive frame DMA operation begins.
>> 
>> In descriptor mode, an AMBA bus error can occur when reading receive
>> descriptor data.  If this happens, receive descriptor processing ends
>> and the Receive DMA Channel is turned off.  The Descriptor Error bit
>> in the Receive Status Register is set to "1".
>
> I don't see how sending "fake" packets through the loop back would be
> considered "resetting the DMA enable in the middle of a DMA".
> (I'm afraid the HW dev didn't grasp what the driver is doing.)
>
> I suppose I should test forcefully setting the enable bit to 0 in
> the driver, and see if hell breaks loose.

You can't.  When the enable bit is 1, writes to that register are
ignored.  It goes back to 0 automatically when the hw runs out of
descriptors.

-- 
Måns Rullgård

Re: Waiting for the PHY to complete auto-negotiation

[Mason]

On 11/12/2017 15:36, Måns Rullgård wrote:

> Mason writes:
> 
>> I suppose I should test forcefully setting the enable bit to 0 in
>> the driver, and see if hell breaks loose.
> 
> You can't.  When the enable bit is 1, writes to that register are
> ignored.  It goes back to 0 automatically when the hw runs out of
> descriptors.

I don't think they are ignored, because toggling the control flow
bit actually breaks RX.

Regards.

Re: Waiting for the PHY to complete auto-negotiation

[Måns Rullgård]

Mason <slash.tmp@free.fr> writes:

> On 11/12/2017 15:36, Måns Rullgård wrote:
>
>> Mason writes:
>> 
>>> I suppose I should test forcefully setting the enable bit to 0 in
>>> the driver, and see if hell breaks loose.
>> 
>> You can't.  When the enable bit is 1, writes to that register are
>> ignored.  It goes back to 0 automatically when the hw runs out of
>> descriptors.
>
> I don't think they are ignored, because toggling the control flow
> bit actually breaks RX.

Oh, then it's even worse than the docs suggest.

-- 
Måns Rullgård