Fwd: net: fec: rx descriptor ring out of order

Fwd: net: fec: rx descriptor ring out of order

[Kegl Rohit]

Hello!

We are using a imx6q platform.
The fec interface is used to receive a continuous stream of custom /
raw ethernet packets. The packet size is fixed ~132 bytes and they get
sent every 250µs.

While testing I observed spontaneous packet delays from time to time.
After digging down deeper I think that the fec peripheral does not
update the rx descriptor status correctly.
I modified the queue_rx function which is called by the NAPI poll
function. "no packet N" is printed when the queue_rx function doesn't
process any descriptor.
Therefore the variable N counts continuous calls without ready
descriptors. When the current descriptor is ready&processed and moved
to the next entry, then N is cleared again.
Additionally an error is printed if the current descriptor is empty
but the next one is already ready. In case this error happens the
current descriptor and the next 11 ones are dumped.
"C"  ... current
"E"  ... empty

[   57.436478 <    0.020005>] no packet 1!
[   57.460850 <    0.024372>] no packet 1!
[   57.461107 <    0.000257>] ring error, current empty but next is not empty
[   57.461118 <    0.000011>] RX ahead
[   57.461135 <    0.000017>] 129 C E 0x8840 0x2c743a40  132
[   57.461146 <    0.000011>] 130     0x0840 0x2c744180  132
[   57.461158 <    0.000012>] 131   E 0x8840 0x2c7448c0  132
[   57.461170 <    0.000012>] 132   E 0x8840 0x2c745000  132
[   57.461181 <    0.000011>] 133   E 0x8840 0x2c745740  132
[   57.461192 <    0.000011>] 134   E 0x8840 0x2c745e80  132
[   57.461204 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
[   57.461215 <    0.000011>] 136   E 0x8840 0x2c746d00  132
[   57.461227 <    0.000012>] 137   E 0x8840 0x2c747440  132
[   57.461239 <    0.000012>] 138   E 0x8840 0x2c748040  132
[   57.461250 <    0.000011>] 139   E 0x8840 0x2c748780  132
[   57.461262 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
[   57.461477 <    0.000008>] no packet 2!
[   57.461506 <    0.000029>] ring error, current empty but next is not empty
[   57.461537 <    0.000031>] RX ahead
[   57.461550 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
[   57.461563 <    0.000013>] 130     0x0840 0x2c744180  132
[   57.461577 <    0.000014>] 131     0x0840 0x2c7448c0  132
[   57.461589 <    0.000012>] 132     0x0840 0x2c745000  132
[   57.461601 <    0.000012>] 133   E 0x8840 0x2c745740  132
[   57.461613 <    0.000012>] 134   E 0x8840 0x2c745e80  132
[   57.461624 <    0.000011>] 135   E 0x8880 0x2c7465c0  114
[   57.461635 <    0.000011>] 136   E 0x8840 0x2c746d00  132
[   57.461645 <    0.000010>] 137   E 0x8840 0x2c747440  132
[   57.461657 <    0.000012>] 138   E 0x8840 0x2c748040  132
[   57.461668 <    0.000011>] 139   E 0x8840 0x2c748780  132
[   57.461680 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
[   57.461894 <    0.000009>] no packet 3!
[   57.461926 <    0.000032>] ring error, current empty but next is not empty
[   57.461935 <    0.000009>] RX ahead
[   57.461947 <    0.000012>] 129 C E 0x8840 0x2c743a40  132
[   57.461959 <    0.000012>] 130     0x0840 0x2c744180  132
[   57.461970 <    0.000011>] 131     0x0840 0x2c7448c0  132
[   57.461982 <    0.000012>] 132     0x0840 0x2c745000  132
[   57.461993 <    0.000011>] 133     0x0840 0x2c745740  132
[   57.462005 <    0.000012>] 134   E 0x8840 0x2c745e80  132
[   57.462017 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
[   57.462028 <    0.000011>] 136   E 0x8840 0x2c746d00  132
[   57.462039 <    0.000011>] 137   E 0x8840 0x2c747440  132
[   57.462051 <    0.000012>] 138   E 0x8840 0x2c748040  132
[   57.462062 <    0.000011>] 139   E 0x8840 0x2c748780  132
[   57.462075 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
[   57.462289 <    0.000009>] no packet 4!
[   57.462316 <    0.000027>] ring error, current empty but next is not empty
[   57.462326 <    0.000010>] RX ahead
[   57.462339 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
[   57.462351 <    0.000012>] 130     0x0840 0x2c744180  132
[   57.462362 <    0.000011>] 131     0x0840 0x2c7448c0  132
[   57.462373 <    0.000011>] 132     0x0840 0x2c745000  132
[   57.462384 <    0.000011>] 133     0x0840 0x2c745740  132
[   57.462397 <    0.000013>] 134     0x0840 0x2c745e80  132
[   57.462408 <    0.000011>] 135     0x0840 0x2c7465c0  132
[   57.462421 <    0.000013>] 136   E 0x8840 0x2c746d00  132
[   57.462431 <    0.000010>] 137   E 0x8840 0x2c747440  132
[   57.462443 <    0.000012>] 138   E 0x8840 0x2c748040  132
[   57.462454 <    0.000011>] 139   E 0x8840 0x2c748780  132
[   57.462467 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
[   57.462697 <    0.000009>] no packet 5!
[   57.462730 <    0.000033>] ring error, current empty but next is not empty
[   57.462739 <    0.000009>] RX ahead
[   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
[   57.462763 <    0.000011>] 130     0x0840 0x2c744180  132
[   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132
[   57.462787 <    0.000012>] 132     0x0840 0x2c745000  132
[   57.462799 <    0.000012>] 133     0x0840 0x2c745740  132
[   57.462809 <    0.000010>] 134     0x0840 0x2c745e80  132
[   57.462820 <    0.000011>] 135     0x0840 0x2c7465c0  132
[   57.462830 <    0.000010>] 136     0x0840 0x2c746d00  132
[   57.462842 <    0.000012>] 137     0x0840 0x2c747440  132
[   57.462853 <    0.000011>] 138   E 0x8840 0x2c748040  132
[   57.462864 <    0.000011>] 139   E 0x8840 0x2c748780  132
[   57.462877 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
[   57.463093 <    0.000009>] no packet 6!
[   57.463120 <    0.000027>] RX ahead
[   57.463133 <    0.000013>] 129 C   0x0840 0x2c743a40  132
[   57.463144 <    0.000011>] 130     0x0840 0x2c744180  132
[   57.463155 <    0.000011>] 131     0x0840 0x2c7448c0  132
[   57.463166 <    0.000011>] 132     0x0840 0x2c745000  132
[   57.463179 <    0.000013>] 133     0x0840 0x2c745740  132
[   57.463190 <    0.000011>] 134     0x0840 0x2c745e80  132
[   57.463201 <    0.000011>] 135     0x0840 0x2c7465c0  132
[   57.463213 <    0.000012>] 136     0x0840 0x2c746d00  132
[   57.463224 <    0.000011>] 137     0x0840 0x2c747440  132
[   57.463235 <    0.000011>] 138     0x0840 0x2c748040  132
[   57.463245 <    0.000010>] 139   E 0x8840 0x2c748780  132
[   57.463256 <    0.000011>] 140   E 0x8840 0x2c748ec0  132
[   57.463695 <    0.000244>] rx 12

As you can see, the described error is catched and the ring is dumped.
9 descriptors got ready before the current descriptor is ready.
After that the current descriptor got ready and 12 packets were
processed at once.
I could also observe cases where the ring (512 entries) got full
before the current descriptor was cleared.
And also cases where the current and next descriptor were not ready.
[   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
[   57.462763 <    0.000011>] 130    E 0x0840 0x2c744180  132
[   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132

I am suspecting the errata:

ERR005783 ENET: ENET Status FIFO may overflow due to consecutive short frames
Description:
When the MAC receives shorter frames (size 64 bytes) at a rate
exceeding the average line-rate
burst traffic of 400 Mbps the DMA is able to absorb, the receiver
might drop incoming frames
before a Pause frame is issued.
Projected Impact:
No malfunction will result aside from the frame drops.
Workarounds:
The application might want to implement some flow control to ensure
the line-rate burst traffic is
below 400 Mbps if it only uses consecutive small frames with minimal
(96 bit times) or short
Inter-frame gap (IFG) time following large frames at such a high rate.
The limit does not exist for
frames of size larger than 800 bytes.
Proposed Solution:
No fix scheduled
Linux BSP Status:
Workaround possible but not implemented in the BSP, impacting
functionality as described above.

Is the "ENET Status FIFO" some internal hardware FIFO or is it the
descriptor ring.
What would be the workaround when a "Workaround is possible"?

I could only think of skipping/dropping the descriptor when the
current is still busy but the next one is ready.
But it is not easily possible because the "stuck" descriptor gets
ready after a huge delay.

Is this issue known already? Any suggestions?


Thanks in advance

Re: Fwd: net: fec: rx descriptor ring out of order

[Eric Dumazet]

On 11/11/20 3:27 PM, Kegl Rohit wrote:
> Hello!
> 
> We are using a imx6q platform.
> The fec interface is used to receive a continuous stream of custom /
> raw ethernet packets. The packet size is fixed ~132 bytes and they get
> sent every 250µs.
> 
> While testing I observed spontaneous packet delays from time to time.
> After digging down deeper I think that the fec peripheral does not
> update the rx descriptor status correctly.
> I modified the queue_rx function which is called by the NAPI poll
> function. "no packet N" is printed when the queue_rx function doesn't
> process any descriptor.
> Therefore the variable N counts continuous calls without ready
> descriptors. When the current descriptor is ready&processed and moved
> to the next entry, then N is cleared again.
> Additionally an error is printed if the current descriptor is empty
> but the next one is already ready. In case this error happens the
> current descriptor and the next 11 ones are dumped.
> "C"  ... current
> "E"  ... empty
> 
> [   57.436478 <    0.020005>] no packet 1!
> [   57.460850 <    0.024372>] no packet 1!
> [   57.461107 <    0.000257>] ring error, current empty but next is not empty
> [   57.461118 <    0.000011>] RX ahead
> [   57.461135 <    0.000017>] 129 C E 0x8840 0x2c743a40  132
> [   57.461146 <    0.000011>] 130     0x0840 0x2c744180  132
> [   57.461158 <    0.000012>] 131   E 0x8840 0x2c7448c0  132
> [   57.461170 <    0.000012>] 132   E 0x8840 0x2c745000  132
> [   57.461181 <    0.000011>] 133   E 0x8840 0x2c745740  132
> [   57.461192 <    0.000011>] 134   E 0x8840 0x2c745e80  132
> [   57.461204 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
> [   57.461215 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> [   57.461227 <    0.000012>] 137   E 0x8840 0x2c747440  132
> [   57.461239 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.461250 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.461262 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
> [   57.461477 <    0.000008>] no packet 2!
> [   57.461506 <    0.000029>] ring error, current empty but next is not empty
> [   57.461537 <    0.000031>] RX ahead
> [   57.461550 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.461563 <    0.000013>] 130     0x0840 0x2c744180  132
> [   57.461577 <    0.000014>] 131     0x0840 0x2c7448c0  132
> [   57.461589 <    0.000012>] 132     0x0840 0x2c745000  132
> [   57.461601 <    0.000012>] 133   E 0x8840 0x2c745740  132
> [   57.461613 <    0.000012>] 134   E 0x8840 0x2c745e80  132
> [   57.461624 <    0.000011>] 135   E 0x8880 0x2c7465c0  114
> [   57.461635 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> [   57.461645 <    0.000010>] 137   E 0x8840 0x2c747440  132
> [   57.461657 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.461668 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.461680 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
> [   57.461894 <    0.000009>] no packet 3!
> [   57.461926 <    0.000032>] ring error, current empty but next is not empty
> [   57.461935 <    0.000009>] RX ahead
> [   57.461947 <    0.000012>] 129 C E 0x8840 0x2c743a40  132
> [   57.461959 <    0.000012>] 130     0x0840 0x2c744180  132
> [   57.461970 <    0.000011>] 131     0x0840 0x2c7448c0  132
> [   57.461982 <    0.000012>] 132     0x0840 0x2c745000  132
> [   57.461993 <    0.000011>] 133     0x0840 0x2c745740  132
> [   57.462005 <    0.000012>] 134   E 0x8840 0x2c745e80  132
> [   57.462017 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
> [   57.462028 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> [   57.462039 <    0.000011>] 137   E 0x8840 0x2c747440  132
> [   57.462051 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.462062 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.462075 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> [   57.462289 <    0.000009>] no packet 4!
> [   57.462316 <    0.000027>] ring error, current empty but next is not empty
> [   57.462326 <    0.000010>] RX ahead
> [   57.462339 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.462351 <    0.000012>] 130     0x0840 0x2c744180  132
> [   57.462362 <    0.000011>] 131     0x0840 0x2c7448c0  132
> [   57.462373 <    0.000011>] 132     0x0840 0x2c745000  132
> [   57.462384 <    0.000011>] 133     0x0840 0x2c745740  132
> [   57.462397 <    0.000013>] 134     0x0840 0x2c745e80  132
> [   57.462408 <    0.000011>] 135     0x0840 0x2c7465c0  132
> [   57.462421 <    0.000013>] 136   E 0x8840 0x2c746d00  132
> [   57.462431 <    0.000010>] 137   E 0x8840 0x2c747440  132
> [   57.462443 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.462454 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.462467 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> [   57.462697 <    0.000009>] no packet 5!
> [   57.462730 <    0.000033>] ring error, current empty but next is not empty
> [   57.462739 <    0.000009>] RX ahead
> [   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.462763 <    0.000011>] 130     0x0840 0x2c744180  132
> [   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132
> [   57.462787 <    0.000012>] 132     0x0840 0x2c745000  132
> [   57.462799 <    0.000012>] 133     0x0840 0x2c745740  132
> [   57.462809 <    0.000010>] 134     0x0840 0x2c745e80  132
> [   57.462820 <    0.000011>] 135     0x0840 0x2c7465c0  132
> [   57.462830 <    0.000010>] 136     0x0840 0x2c746d00  132
> [   57.462842 <    0.000012>] 137     0x0840 0x2c747440  132
> [   57.462853 <    0.000011>] 138   E 0x8840 0x2c748040  132
> [   57.462864 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.462877 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> [   57.463093 <    0.000009>] no packet 6!
> [   57.463120 <    0.000027>] RX ahead
> [   57.463133 <    0.000013>] 129 C   0x0840 0x2c743a40  132
> [   57.463144 <    0.000011>] 130     0x0840 0x2c744180  132
> [   57.463155 <    0.000011>] 131     0x0840 0x2c7448c0  132
> [   57.463166 <    0.000011>] 132     0x0840 0x2c745000  132
> [   57.463179 <    0.000013>] 133     0x0840 0x2c745740  132
> [   57.463190 <    0.000011>] 134     0x0840 0x2c745e80  132
> [   57.463201 <    0.000011>] 135     0x0840 0x2c7465c0  132
> [   57.463213 <    0.000012>] 136     0x0840 0x2c746d00  132
> [   57.463224 <    0.000011>] 137     0x0840 0x2c747440  132
> [   57.463235 <    0.000011>] 138     0x0840 0x2c748040  132
> [   57.463245 <    0.000010>] 139   E 0x8840 0x2c748780  132
> [   57.463256 <    0.000011>] 140   E 0x8840 0x2c748ec0  132
> [   57.463695 <    0.000244>] rx 12
> 
> As you can see, the described error is catched and the ring is dumped.
> 9 descriptors got ready before the current descriptor is ready.
> After that the current descriptor got ready and 12 packets were
> processed at once.
> I could also observe cases where the ring (512 entries) got full
> before the current descriptor was cleared.
> And also cases where the current and next descriptor were not ready.
> [   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.462763 <    0.000011>] 130    E 0x0840 0x2c744180  132
> [   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132
> 
> I am suspecting the errata:
> 
> ERR005783 ENET: ENET Status FIFO may overflow due to consecutive short frames
> Description:
> When the MAC receives shorter frames (size 64 bytes) at a rate
> exceeding the average line-rate
> burst traffic of 400 Mbps the DMA is able to absorb, the receiver
> might drop incoming frames
> before a Pause frame is issued.
> Projected Impact:
> No malfunction will result aside from the frame drops.
> Workarounds:
> The application might want to implement some flow control to ensure
> the line-rate burst traffic is
> below 400 Mbps if it only uses consecutive small frames with minimal
> (96 bit times) or short
> Inter-frame gap (IFG) time following large frames at such a high rate.
> The limit does not exist for
> frames of size larger than 800 bytes.
> Proposed Solution:
> No fix scheduled
> Linux BSP Status:
> Workaround possible but not implemented in the BSP, impacting
> functionality as described above.
> 
> Is the "ENET Status FIFO" some internal hardware FIFO or is it the
> descriptor ring.
> What would be the workaround when a "Workaround is possible"?
> 
> I could only think of skipping/dropping the descriptor when the
> current is still busy but the next one is ready.
> But it is not easily possible because the "stuck" descriptor gets
> ready after a huge delay.
> 
> Is this issue known already? Any suggestions?
> 
> 
> Thanks in advance
> 

It could help if you could provide your patch, or at least the kernel version and file (driver)
you are looking at :)

RE: Fwd: net: fec: rx descriptor ring out of order

[David Laight]

> On 11/11/20 3:27 PM, Kegl Rohit wrote:
> > Hello!
> >
> > We are using a imx6q platform.
> > The fec interface is used to receive a continuous stream of custom /
> > raw ethernet packets. The packet size is fixed ~132 bytes and they get
> > sent every 250µs.
> >
> > While testing I observed spontaneous packet delays from time to time.
> > After digging down deeper I think that the fec peripheral does not
> > update the rx descriptor status correctly.
> > I modified the queue_rx function which is called by the NAPI poll
> > function. "no packet N" is printed when the queue_rx function doesn't
> > process any descriptor.
> > Therefore the variable N counts continuous calls without ready
> > descriptors. When the current descriptor is ready&processed and moved
> > to the next entry, then N is cleared again.
> > Additionally an error is printed if the current descriptor is empty
> > but the next one is already ready. In case this error happens the
> > current descriptor and the next 11 ones are dumped.
> > "C"  ... current
> > "E"  ... empty
> >
> > [   57.436478 <    0.020005>] no packet 1!
> > [   57.460850 <    0.024372>] no packet 1!
> > [   57.461107 <    0.000257>] ring error, current empty but next is not empty
> > [   57.461118 <    0.000011>] RX ahead
> > [   57.461135 <    0.000017>] 129 C E 0x8840 0x2c743a40  132
> > [   57.461146 <    0.000011>] 130     0x0840 0x2c744180  132
> > [   57.461158 <    0.000012>] 131   E 0x8840 0x2c7448c0  132

What are the addresses of the ring entries?
I bet there is something wrong with the cache coherency and/or
flushing.

So the MAC hardware has done the write but (somewhere) it
isn't visible to the cpu for ages.

I've seen a 'fec' ethernet block in a freescale DSP.
IIRC it is a fairly simple block - won't be doing out-of-order writes.

The imx6q seems to be arm based.
I'm guessing that means it doesn't do cache coherency for ethernet dma
accesses.
That (more or less) means the rings need to be mapped uncached.
Any attempt to just flush/invalidate the cache lines is doomed.

...
> > I am suspecting the errata:
> >
> > ERR005783 ENET: ENET Status FIFO may overflow due to consecutive short frames
> > Description:
> > When the MAC receives shorter frames (size 64 bytes) at a rate
> > exceeding the average line-rate
> > burst traffic of 400 Mbps the DMA is able to absorb, the receiver
> > might drop incoming frames
> > before a Pause frame is issued.
> > Projected Impact:
> > No malfunction will result aside from the frame drops.
> > Workarounds:
> > The application might want to implement some flow control to ensure
> > the line-rate burst traffic is
> > below 400 Mbps if it only uses consecutive small frames with minimal
> > (96 bit times) or short
> > Inter-frame gap (IFG) time following large frames at such a high rate.
> > The limit does not exist for
> > frames of size larger than 800 bytes.
> > Proposed Solution:
> > No fix scheduled
> > Linux BSP Status:
> > Workaround possible but not implemented in the BSP, impacting
> > functionality as described above.
> >
> > Is the "ENET Status FIFO" some internal hardware FIFO or is it the
> > descriptor ring.
> > What would be the workaround when a "Workaround is possible"?

I don't think that is applicable.
It looks like it just drops frames under high load.

I've no idea what a 'Linux BSP' might be.
That term is usually used for the (often broken) board support
for things like Vx(no-longer)Works.

> > I could only think of skipping/dropping the descriptor when the
> > current is still busy but the next one is ready.
> > But it is not easily possible because the "stuck" descriptor gets
> > ready after a huge delay.

I bet the descriptor is at the end of a cache line which finally
gets re-read.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Fwd: net: fec: rx descriptor ring out of order

[Fabio Estevam]

Hi David,

On Wed, Nov 11, 2020 at 2:52 PM David Laight <David.Laight@aculab.com> wrote:

> I've seen a 'fec' ethernet block in a freescale DSP.
> IIRC it is a fairly simple block - won't be doing out-of-order writes.
>
> The imx6q seems to be arm based.

This is correct.

> I'm guessing that means it doesn't do cache coherency for ethernet dma
> accesses.
> That (more or less) means the rings need to be mapped uncached.
> Any attempt to just flush/invalidate the cache lines is doomed.

This is the driver: drivers/net/ethernet/freescale/fec_main.c

Re: Fwd: net: fec: rx descriptor ring out of order

[Kegl Rohit]

On Wed, Nov 11, 2020 at 6:52 PM David Laight <David.Laight@aculab.com> wrote:
>
> > On 11/11/20 3:27 PM, Kegl Rohit wrote:
> > > Hello!
> > >
> > > We are using a imx6q platform.
> > > The fec interface is used to receive a continuous stream of custom /
> > > raw ethernet packets. The packet size is fixed ~132 bytes and they get
> > > sent every 250µs.
> > >
> > > While testing I observed spontaneous packet delays from time to time.
> > > After digging down deeper I think that the fec peripheral does not
> > > update the rx descriptor status correctly.
> > > I modified the queue_rx function which is called by the NAPI poll
> > > function. "no packet N" is printed when the queue_rx function doesn't
> > > process any descriptor.
> > > Therefore the variable N counts continuous calls without ready
> > > descriptors. When the current descriptor is ready&processed and moved
> > > to the next entry, then N is cleared again.
> > > Additionally an error is printed if the current descriptor is empty
> > > but the next one is already ready. In case this error happens the
> > > current descriptor and the next 11 ones are dumped.
> > > "C"  ... current
> > > "E"  ... empty
> > >
> > > [   57.436478 <    0.020005>] no packet 1!
> > > [   57.460850 <    0.024372>] no packet 1!
> > > [   57.461107 <    0.000257>] ring error, current empty but next is not empty
> > > [   57.461118 <    0.000011>] RX ahead
> > > [   57.461135 <    0.000017>] 129 C E 0x8840 0x2c743a40  132
> > > [   57.461146 <    0.000011>] 130     0x0840 0x2c744180  132
> > > [   57.461158 <    0.000012>] 131   E 0x8840 0x2c7448c0  132
>
> What are the addresses of the ring entries?
> I bet there is something wrong with the cache coherency and/or
> flushing.

The ring descriptors are allocated via dma_alloc_coherent().  I will
extend the dump with their addresses.
The current output shows the dma_map_single() skb data buffer.
I tried calling  flush_cache_all() before reading the descriptors
status => no change.
Are there any flush options to try?


> So the MAC hardware has done the write but (somewhere) it
> isn't visible to the cpu for ages.
It looks like that. After an error occured i will also read the skb
data (dma_sync_single() before) to check if the new data is already
there.
So I can prove that the status is wrong, because the data could be
already there.


> I've seen a 'fec' ethernet block in a freescale DSP.
> IIRC it is a fairly simple block - won't be doing out-of-order writes.
>
> The imx6q seems to be arm based.
> I'm guessing that means it doesn't do cache coherency for ethernet dma
> accesses.
> That (more or less) means the rings need to be mapped uncached.
> Any attempt to just flush/invalidate the cache lines is doomed.
The descriptors are allocated using dma_alloc_coherent(). So flushes
should not be needed? Synchronizing is done via barriers e.g. wmb()
before resetting the descriptor status.
The skb data itself is mapped using the DMA API.

> ...
> > > I am suspecting the errata:
> > >
> > > ERR005783 ENET: ENET Status FIFO may overflow due to consecutive short frames
> > > Description:
> > > When the MAC receives shorter frames (size 64 bytes) at a rate
> > > exceeding the average line-rate
> > > burst traffic of 400 Mbps the DMA is able to absorb, the receiver
> > > might drop incoming frames
> > > before a Pause frame is issued.
> > > Projected Impact:
> > > No malfunction will result aside from the frame drops.
> > > Workarounds:
> > > The application might want to implement some flow control to ensure
> > > the line-rate burst traffic is
> > > below 400 Mbps if it only uses consecutive small frames with minimal
> > > (96 bit times) or short
> > > Inter-frame gap (IFG) time following large frames at such a high rate.
> > > The limit does not exist for
> > > frames of size larger than 800 bytes.
> > > Proposed Solution:
> > > No fix scheduled
> > > Linux BSP Status:
> > > Workaround possible but not implemented in the BSP, impacting
> > > functionality as described above.
> > >
> > > Is the "ENET Status FIFO" some internal hardware FIFO or is it the
> > > descriptor ring.
> > > What would be the workaround when a "Workaround is possible"?
>
> I don't think that is applicable.
> It looks like it just drops frames under high load.
Hm ok.

> I've no idea what a 'Linux BSP' might be.
> That term is usually used for the (often broken) board support
> for things like Vx(no-longer)Works.
Hm ok.

> > > I could only think of skipping/dropping the descriptor when the
> > > current is still busy but the next one is ready.
> > > But it is not easily possible because the "stuck" descriptor gets
> > > ready after a huge delay.
>
> I bet the descriptor is at the end of a cache line which finally
> gets re-read.

Would have cache_flush_all() solved this problem?

Re: Fwd: net: fec: rx descriptor ring out of order

[Kegl Rohit]

> What are the addresses of the ring entries?
> I bet there is something wrong with the cache coherency and/or
> flushing.
>
> So the MAC hardware has done the write but (somewhere) it
> isn't visible to the cpu for ages.

CMA memory is disabled in our kernel config.
So the descriptors allocated with dma_alloc_coherent() won't be CMA memory.
Could this cause a different caching/flushing behaviour?

> I've seen a 'fec' ethernet block in a freescale DSP.
> IIRC it is a fairly simple block - won't be doing out-of-order writes.
>
> The imx6q seems to be arm based.
> I'm guessing that means it doesn't do cache coherency for ethernet dma
> accesses.
> That (more or less) means the rings need to be mapped uncached.
> Any attempt to just flush/invalidate the cache lines is doomed.
>
> ...

> > > I could only think of skipping/dropping the descriptor when the
> > > current is still busy but the next one is ready.
> > > But it is not easily possible because the "stuck" descriptor gets
> > > ready after a huge delay.
>
> I bet the descriptor is at the end of a cache line which finally
> gets re-read.
I stumbled across FEC ethernet issues [Was: PL310 errata workarounds]
https://www.spinics.net/lists/arm-kernel/thrd312.html#315574.
Changes to the PL310 cache driver (used in imx6q) were made, to also
fix fec issues.
This PL310 cleanup/fixes are not contained in the 3.10.108 kernel.
So maybe i have to look also there.

Re: Fwd: net: fec: rx descriptor ring out of order

[Kegl Rohit]

On Fri, Nov 13, 2020 at 8:33 AM Kegl Rohit <keglrohit@gmail.com> wrote:
>
> > What are the addresses of the ring entries?
> > I bet there is something wrong with the cache coherency and/or
> > flushing.
> >
> > So the MAC hardware has done the write but (somewhere) it
> > isn't visible to the cpu for ages.
>
> CMA memory is disabled in our kernel config.
> So the descriptors allocated with dma_alloc_coherent() won't be CMA memory.
> Could this cause a different caching/flushing behaviour?

Yes, after tests I think it is caused by the disabled CMA.

@Andy
I could find this mail and the attached "i.MX6 dma memory bufferable
issue.pptx" in the archive
https://marc.info/?l=linux-netdev&m=140135147823760
Was this issue solved in some kernel versions later on?
Is CMA still necessary with a 5.4 Kernel?

RE: [EXT] Re: Fwd: net: fec: rx descriptor ring out of order

[Andy Duan]

From: Kegl Rohit <keglrohit@gmail.com> Sent: Friday, November 13, 2020 8:21 PM
> On Fri, Nov 13, 2020 at 8:33 AM Kegl Rohit <keglrohit@gmail.com> wrote:
> >
> > > What are the addresses of the ring entries?
> > > I bet there is something wrong with the cache coherency and/or
> > > flushing.
> > >
> > > So the MAC hardware has done the write but (somewhere) it isn't
> > > visible to the cpu for ages.
> >
> > CMA memory is disabled in our kernel config.
> > So the descriptors allocated with dma_alloc_coherent() won't be CMA memory.
> > Could this cause a different caching/flushing behaviour?
> 
> Yes, after tests I think it is caused by the disabled CMA.
> 
> @Andy
> I could find this mail and the attached "i.MX6 dma memory bufferable
> issue.pptx" in the archive
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmarc.info
> %2F%3Fl%3Dlinux-netdev%26m%3D140135147823760&amp;data=04%7C01
> %7Cfugang.duan%40nxp.com%7C121e73ec66684a125e2a08d887cea578%7C
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637408668924362983
> %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=e7Cm24Ay1Ay52UKtzT
> BiX9KlhuublndP30vnwxAaugM%3D&amp;reserved=0
> Was this issue solved in some kernel versions later on?
> Is CMA still necessary with a 5.4 Kernel?

Yes, CMA is required. Otherwise it requires one patch for L2 cache.

Re: [EXT] Re: Fwd: net: fec: rx descriptor ring out of order

[Kegl Rohit]

On Sat, Nov 14, 2020 at 2:58 AM Andy Duan <fugang.duan@nxp.com> wrote:
>
> From: Kegl Rohit <keglrohit@gmail.com> Sent: Friday, November 13, 2020 8:21 PM
> > On Fri, Nov 13, 2020 at 8:33 AM Kegl Rohit <keglrohit@gmail.com> wrote:
> > >
> > > > What are the addresses of the ring entries?
> > > > I bet there is something wrong with the cache coherency and/or
> > > > flushing.
> > > >
> > > > So the MAC hardware has done the write but (somewhere) it isn't
> > > > visible to the cpu for ages.
> > >
> > > CMA memory is disabled in our kernel config.
> > > So the descriptors allocated with dma_alloc_coherent() won't be CMA memory.
> > > Could this cause a different caching/flushing behaviour?
> >
> > Yes, after tests I think it is caused by the disabled CMA.
> >
> > @Andy
> > I could find this mail and the attached "i.MX6 dma memory bufferable
> > issue.pptx" in the archive
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmarc.info
> > %2F%3Fl%3Dlinux-netdev%26m%3D140135147823760&amp;data=04%7C01
> > %7Cfugang.duan%40nxp.com%7C121e73ec66684a125e2a08d887cea578%7C
> > 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637408668924362983
> > %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> > BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=e7Cm24Ay1Ay52UKtzT
> > BiX9KlhuublndP30vnwxAaugM%3D&amp;reserved=0
> > Was this issue solved in some kernel versions later on?
> > Is CMA still necessary with a 5.4 Kernel?
>
> Yes, CMA is required. Otherwise it requires one patch for L2 cache.

Where can I find the patch / is the patch already mainline?
Is it some development patch or already well tested?
Or would you recommend enabling CMA instead?
Are other components affected apart from the already mentioned
peripherals (ENET, Audio, USB) in the attachment?

RE: [EXT] Re: Fwd: net: fec: rx descriptor ring out of order

[Andy Duan]

From: Kegl Rohit <keglrohit@gmail.com> Sent: Sunday, November 15, 2020 1:37 AM
> On Sat, Nov 14, 2020 at 2:58 AM Andy Duan <fugang.duan@nxp.com> wrote:
> >
> > From: Kegl Rohit <keglrohit@gmail.com> Sent: Friday, November 13, 2020
> > 8:21 PM
> > > On Fri, Nov 13, 2020 at 8:33 AM Kegl Rohit <keglrohit@gmail.com> wrote:
> > > >
> > > > > What are the addresses of the ring entries?
> > > > > I bet there is something wrong with the cache coherency and/or
> > > > > flushing.
> > > > >
> > > > > So the MAC hardware has done the write but (somewhere) it isn't
> > > > > visible to the cpu for ages.
> > > >
> > > > CMA memory is disabled in our kernel config.
> > > > So the descriptors allocated with dma_alloc_coherent() won't be CMA
> memory.
> > > > Could this cause a different caching/flushing behaviour?
> > >
> > > Yes, after tests I think it is caused by the disabled CMA.
> > >
> > > @Andy
> > > I could find this mail and the attached "i.MX6 dma memory bufferable
> > > issue.pptx" in the archive
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fma
> > > rc.info
> > > %2F%3Fl%3Dlinux-netdev%26m%3D140135147823760&amp;data=04%7C
> 01
> > > %7Cfugang.duan%40nxp.com%7C121e73ec66684a125e2a08d887cea578%
> 7C
> > >
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637408668924362983
> > > %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> LCJ
> > >
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=e7Cm24Ay1Ay52UKtzT
> > > BiX9KlhuublndP30vnwxAaugM%3D&amp;reserved=0
> > > Was this issue solved in some kernel versions later on?
> > > Is CMA still necessary with a 5.4 Kernel?
> >
> > Yes, CMA is required. Otherwise it requires one patch for L2 cache.
> 
> Where can I find the patch / is the patch already mainline?
No, the patch is not in mainline. CMA can fix the issue.

The original patch is: set shared override bit in PL310 AUX_CTRL register

> Is it some development patch or already well tested?
> Or would you recommend enabling CMA instead?
> Are other components affected apart from the already mentioned peripherals
> (ENET, Audio, USB) in the attachment?
Yes,  recommend CMA that can fix the cache issue for all components.

Re: [EXT] Re: Fwd: net: fec: rx descriptor ring out of order

[Kegl Rohit]

On Sun, Nov 15, 2020 at 8:48 AM Andy Duan <fugang.duan@nxp.com> wrote:
>
> From: Kegl Rohit <keglrohit@gmail.com> Sent: Sunday, November 15, 2020 1:37 AM
> > On Sat, Nov 14, 2020 at 2:58 AM Andy Duan <fugang.duan@nxp.com> wrote:
> > >
> > > From: Kegl Rohit <keglrohit@gmail.com> Sent: Friday, November 13, 2020
> > > 8:21 PM
> > > > On Fri, Nov 13, 2020 at 8:33 AM Kegl Rohit <keglrohit@gmail.com> wrote:
> > > > >
> > > > > > What are the addresses of the ring entries?
> > > > > > I bet there is something wrong with the cache coherency and/or
> > > > > > flushing.
> > > > > >
> > > > > > So the MAC hardware has done the write but (somewhere) it isn't
> > > > > > visible to the cpu for ages.
> > > > >
> > > > > CMA memory is disabled in our kernel config.
> > > > > So the descriptors allocated with dma_alloc_coherent() won't be CMA
> > memory.
> > > > > Could this cause a different caching/flushing behaviour?
> > > >
> > > > Yes, after tests I think it is caused by the disabled CMA.
> > > >
> > > > @Andy
> > > > I could find this mail and the attached "i.MX6 dma memory bufferable
> > > > issue.pptx" in the archive
> > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fma
> > > > rc.info
> > > > %2F%3Fl%3Dlinux-netdev%26m%3D140135147823760&amp;data=04%7C
> > 01
> > > > %7Cfugang.duan%40nxp.com%7C121e73ec66684a125e2a08d887cea578%
> > 7C
> > > >
> > 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637408668924362983
> > > > %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> > LCJ
> > > >
> > BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=e7Cm24Ay1Ay52UKtzT
> > > > BiX9KlhuublndP30vnwxAaugM%3D&amp;reserved=0
> > > > Was this issue solved in some kernel versions later on?
> > > > Is CMA still necessary with a 5.4 Kernel?
> > >
> > > Yes, CMA is required. Otherwise it requires one patch for L2 cache.
> >
> > Where can I find the patch / is the patch already mainline?
> No, the patch is not in mainline. CMA can fix the issue.
>
> The original patch is: set shared override bit in PL310 AUX_CTRL register
>
> > Is it some development patch or already well tested?
> > Or would you recommend enabling CMA instead?
> > Are other components affected apart from the already mentioned peripherals
> > (ENET, Audio, USB) in the attachment?
> Yes,  recommend CMA that can fix the cache issue for all components.

Ok, thanks. One more Question: The kernel's default CMA size is 16MB,
is this enough for a headless system without usage of the IPU?

Re: net: fec: rx descriptor ring out of order

[Fabio Estevam]

On Wed, Nov 11, 2020 at 11:27 AM Kegl Rohit <keglrohit@gmail.com> wrote:
>
> Hello!
>
> We are using a imx6q platform.
> The fec interface is used to receive a continuous stream of custom /
> raw ethernet packets. The packet size is fixed ~132 bytes and they get
> sent every 250µs.
>
> While testing I observed spontaneous packet delays from time to time.
> After digging down deeper I think that the fec peripheral does not
> update the rx descriptor status correctly.

What is the kernel version that you are using?

Re: net: fec: rx descriptor ring out of order

[Kegl Rohit]

On Wed, Nov 11, 2020 at 11:18 PM Fabio Estevam <festevam@gmail.com> wrote:
>
> On Wed, Nov 11, 2020 at 11:27 AM Kegl Rohit <keglrohit@gmail.com> wrote:
> >
> > Hello!
> >
> > We are using a imx6q platform.
> > The fec interface is used to receive a continuous stream of custom /
> > raw ethernet packets. The packet size is fixed ~132 bytes and they get
> > sent every 250µs.
> >
> > While testing I observed spontaneous packet delays from time to time.
> > After digging down deeper I think that the fec peripheral does not
> > update the rx descriptor status correctly.
>
> What is the kernel version that you are using?

Sadly stuck at 3.10.108.
https://github.com/gregkh/linux/blob/v3.10.108/drivers/net/ethernet/freescale/fec_main.c
The rx queue status handling did not change much compared to 5.x. Only
the NAPI handling / clearing IRQs was changed more than once.
I also backported the newer NAPI handling style / clearing irqs not in
the irq handler but in napi_poll() => same issue.
The issue is pretty rare => To reproduce i have to reboot the system
every 3 min. Sometimes after 1~2min on the first, sometimes on the
~10th reboot it will happen.

Re: net: fec: rx descriptor ring out of order

[Eric Dumazet]

On 11/12/20 7:52 AM, Kegl Rohit wrote:
> On Wed, Nov 11, 2020 at 11:18 PM Fabio Estevam <festevam@gmail.com> wrote:
>>
>> On Wed, Nov 11, 2020 at 11:27 AM Kegl Rohit <keglrohit@gmail.com> wrote:
>>>
>>> Hello!
>>>
>>> We are using a imx6q platform.
>>> The fec interface is used to receive a continuous stream of custom /
>>> raw ethernet packets. The packet size is fixed ~132 bytes and they get
>>> sent every 250µs.
>>>
>>> While testing I observed spontaneous packet delays from time to time.
>>> After digging down deeper I think that the fec peripheral does not
>>> update the rx descriptor status correctly.
>>
>> What is the kernel version that you are using?
> 
> Sadly stuck at 3.10.108.
> https://github.com/gregkh/linux/blob/v3.10.108/drivers/net/ethernet/freescale/fec_main.c
> The rx queue status handling did not change much compared to 5.x. Only
> the NAPI handling / clearing IRQs was changed more than once.
> I also backported the newer NAPI handling style / clearing irqs not in
> the irq handler but in napi_poll() => same issue.
> The issue is pretty rare => To reproduce i have to reboot the system
> every 3 min. Sometimes after 1~2min on the first, sometimes on the
> ~10th reboot it will happen.
> 

Is seems some rmb() & wmb() are missing.

Honestly 3.10 is way too old for us spending much time on this issue.

I would try this :

diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
index fbd0d7df67d8dec64f712602cb0c17e3cb585e2b..99767728f2b501813f9d4a833fa3146caea50ed6 100644
--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -344,6 +344,7 @@ fec_enet_start_xmit(struct sk_buff *skb, struct net_device *ndev)
         */
        status |= (BD_ENET_TX_READY | BD_ENET_TX_INTR
                        | BD_ENET_TX_LAST | BD_ENET_TX_TC);
+       wmb();
        bdp->cbd_sc = status;
 
        if (fep->bufdesc_ex) {
@@ -810,10 +811,12 @@ fec_enet_rx(struct net_device *ndev, int budget)
         */
        bdp = fep->cur_rx;
 
-       while (!((status = bdp->cbd_sc) & BD_ENET_RX_EMPTY)) {
-
-               if (pkt_received >= budget)
+       while (pkt_received < budget) {
+               status = bdp->cbd_sc;
+               rmb();
+               if (status & BD_ENET_RX_EMPTY)
                        break;
+
                pkt_received++;
 
                /* Since we have allocated space to hold a complete frame,
@@ -918,7 +921,6 @@ rx_processing_done:
 
                /* Mark the buffer empty */
                status |= BD_ENET_RX_EMPTY;
-               bdp->cbd_sc = status;
 
                if (fep->bufdesc_ex) {
                        struct bufdesc_ex *ebdp = (struct bufdesc_ex *)bdp;
@@ -927,6 +929,9 @@ rx_processing_done:
                        ebdp->cbd_prot = 0;
                        ebdp->cbd_bdu = 0;
                }
+               /* This needs to be the final write, otherwise NIC could catch garbage values in surrounding fields */
+               wmb();
+               bdp->cbd_sc = status;
 
                /* Update BD pointer to next entry */
                if (status & BD_ENET_RX_WRAP)

RE: net: fec: rx descriptor ring out of order

[David Laight]

From: Eric Dumazet
> Sent: 12 November 2020 10:42
> 
> On 11/12/20 7:52 AM, Kegl Rohit wrote:
> > On Wed, Nov 11, 2020 at 11:18 PM Fabio Estevam <festevam@gmail.com> wrote:
> >>
> >> On Wed, Nov 11, 2020 at 11:27 AM Kegl Rohit <keglrohit@gmail.com> wrote:
> >>>
> >>> Hello!
> >>>
> >>> We are using a imx6q platform.
> >>> The fec interface is used to receive a continuous stream of custom /
> >>> raw ethernet packets. The packet size is fixed ~132 bytes and they get
> >>> sent every 250µs.
> >>>
> >>> While testing I observed spontaneous packet delays from time to time.
> >>> After digging down deeper I think that the fec peripheral does not
> >>> update the rx descriptor status correctly.
> >>
> >> What is the kernel version that you are using?
> >
> > Sadly stuck at 3.10.108.

If you build a newer kernel it should work with your
existing userspace.

> > https://github.com/gregkh/linux/blob/v3.10.108/drivers/net/ethernet/freescale/fec_main.c
> > The rx queue status handling did not change much compared to 5.x. Only
> > the NAPI handling / clearing IRQs was changed more than once.
> > I also backported the newer NAPI handling style / clearing irqs not in
> > the irq handler but in napi_poll() => same issue.
> > The issue is pretty rare => To reproduce i have to reboot the system
> > every 3 min. Sometimes after 1~2min on the first, sometimes on the
> > ~10th reboot it will happen.
> >
> 
> Is seems some rmb() & wmb() are missing.

They are unlikely to make any difference since the 'bad'
rx status persists between calls to the receive function.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: net: fec: rx descriptor ring out of order

[Kegl Rohit]

On Thu, Nov 12, 2020 at 12:10 PM David Laight <David.Laight@aculab.com> wrote:
>
> From: Eric Dumazet
> > Sent: 12 November 2020 10:42
> >
> > On 11/12/20 7:52 AM, Kegl Rohit wrote:
> > > On Wed, Nov 11, 2020 at 11:18 PM Fabio Estevam <festevam@gmail.com> wrote:
> > >>
> > >> On Wed, Nov 11, 2020 at 11:27 AM Kegl Rohit <keglrohit@gmail.com> wrote:
> > >>>
> > >>> Hello!
> > >>>
> > >>> We are using a imx6q platform.
> > >>> The fec interface is used to receive a continuous stream of custom /
> > >>> raw ethernet packets. The packet size is fixed ~132 bytes and they get
> > >>> sent every 250µs.
> > >>>
> > >>> While testing I observed spontaneous packet delays from time to time.
> > >>> After digging down deeper I think that the fec peripheral does not
> > >>> update the rx descriptor status correctly.
> > >>
> > >> What is the kernel version that you are using?
> > >
> > > Sadly stuck at 3.10.108.
>
> If you build a newer kernel it should work with your
> existing userspace.
Not so easily possible because there are custom drivers and some
kernel modifications in the mix.
I have a dirty ported system with a 5.4 kernel ready. I will also try it there.
But I am afraid the error will not happen but still exist.


> > > https://github.com/gregkh/linux/blob/v3.10.108/drivers/net/ethernet/freescale/fec_main.c
> > > The rx queue status handling did not change much compared to 5.x. Only
> > > the NAPI handling / clearing IRQs was changed more than once.
> > > I also backported the newer NAPI handling style / clearing irqs not in
> > > the irq handler but in napi_poll() => same issue.
> > > The issue is pretty rare => To reproduce i have to reboot the system
> > > every 3 min. Sometimes after 1~2min on the first, sometimes on the
> > > ~10th reboot it will happen.
> > >
> >
> > Is seems some rmb() & wmb() are missing.
>
> They are unlikely to make any difference since the 'bad'
> rx status persists between calls to the receive function.

Our kernel already has some patches like the wmb() for the rx path and
the rmb() for the tx path applied.
I tried the rmb() at the rx path, because this is not in master
https://github.com/gregkh/linux/blob/master/drivers/net/ethernet/freescale/fec_main.c#L1434.
=> Still the same issue, no change

I extended the debugging:
descriptor index, current, empty, desc.status, desc.buffer (mapped
skb->data), desc.length
[  137.758009 <    0.000015>] 409 0xa09d5320 C E 0x8840 0x2c6f0780    0

I also reset the desc.length field to 0 after the packet was received
and before the descriptor was set to empty again.
So I could observe that the length is also not set like the status.

Because i know the content and size of my rx packets, i used
dma_sync_single(mapped skb->data) to get the data even if the status
is empty.
Each packet contains a counter, so i verified that the data is already
there and not lost.
Only the descriptor status and length is not updated.

[  137.757966 <    0.000021>] cnt: 2341          .... counter of
current ("empty") packet; index 409  in example
[  137.757984 <    0.000018>] nxcnt: 2342      .... counter of next
not empty packet; index 410 in example
=> content is there but status is not. As next step i will also check
if all bytes are correct, not only the two counter bytes.

[   40.888181 <    0.000344>] --- start test application ---
[  137.757945 <   96.869764>] ring error, next is ready
[  137.757966 <    0.000021>] cnt: 2341
[  137.757984 <    0.000018>] nxcnt: 2342
[  137.757994 <    0.000010>] RX ahead
[  137.758009 <    0.000015>] 409 0xa09d5320 C E 0x8840 0x2c6f0780    0
[  137.758024 <    0.000015>] 410 0xa09d5340     0x0840 0x2c6f0ec0  132
[  137.758038 <    0.000014>] 411 0xa09d5360   E 0x8840 0x2c6f1600    0
[  137.758051 <    0.000013>] 412 0xa09d5380   E 0x8840 0x2c6f1d40    0
[  137.758064 <    0.000013>] 413 0xa09d53a0   E 0x8840 0x2c6f2480    0
[  137.758076 <    0.000012>] 414 0xa09d53c0   E 0x8840 0x2c6f2bc0    0
[  137.758089 <    0.000013>] 415 0xa09d53e0   E 0x8840 0x2c6f3300    0
[  137.758102 <    0.000013>] 416 0xa09d5400   E 0x8840 0x2c6f3a40    0
[  137.758115 <    0.000013>] 417 0xa09d5420   E 0x8840 0x2c6f4180    0
[  137.758127 <    0.000012>] 418 0xa09d5440   E 0x8840 0x2c6f48c0    0
[  137.758140 <    0.000013>] 419 0xa09d5460   E 0x8840 0x2c6f5000    0
[  137.758152 <    0.000012>] 420 0xa09d5480   E 0x8840 0x2c6f5740    0
[  137.758165 <    0.000013>] 421 0xa09d54a0   E 0x8840 0x2c6f5e80    0
[  137.758414 <    0.000025>] ring error, next is ready
[  137.758426 <    0.000012>] cnt: 2341
[  137.758439 <    0.000013>] nxcnt: 2342
[  137.758448 <    0.000009>] RX ahead
[  137.758485 <    0.000037>] 409 0xa09d5320 C E 0x8840 0x2c6f0780    0
[  137.758500 <    0.000015>] 410 0xa09d5340     0x0840 0x2c6f0ec0  132
[  137.758515 <    0.000015>] 411 0xa09d5360     0x0840 0x2c6f1600  132
[  137.758529 <    0.000014>] 412 0xa09d5380     0x0840 0x2c6f1d40  132
[  137.758542 <    0.000013>] 413 0xa09d53a0   E 0x8840 0x2c6f2480    0
[  137.758556 <    0.000014>] 414 0xa09d53c0   E 0x8840 0x2c6f2bc0    0
[  137.758569 <    0.000013>] 415 0xa09d53e0   E 0x8840 0x2c6f3300    0
[  137.758582 <    0.000013>] 416 0xa09d5400   E 0x8840 0x2c6f3a40    0
[  137.758596 <    0.000014>] 417 0xa09d5420   E 0x8840 0x2c6f4180    0
[  137.758609 <    0.000013>] 418 0xa09d5440   E 0x8840 0x2c6f48c0    0
[  137.758622 <    0.000013>] 419 0xa09d5460   E 0x8840 0x2c6f5000    0
[  137.758905 <    0.000031>] ring error, next is ready
[  137.758917 <    0.000012>] cnt: 2341
[  137.758930 <    0.000013>] nxcnt: 2342
[  137.758938 <    0.000008>] RX ahead
[  137.758951 <    0.000013>] 409 0xa09d5320 C E 0x8840 0x2c6f0780    0
[  137.758965 <    0.000014>] 410 0xa09d5340     0x0840 0x2c6f0ec0  132
[  137.758978 <    0.000013>] 411 0xa09d5360     0x0840 0x2c6f1600  132
[  137.758991 <    0.000013>] 412 0xa09d5380     0x0840 0x2c6f1d40  132
[  137.759005 <    0.000014>] 413 0xa09d53a0     0x0840 0x2c6f2480  132
[  137.759018 <    0.000013>] 414 0xa09d53c0     0x0840 0x2c6f2bc0  132
[  137.759031 <    0.000013>] 415 0xa09d53e0   E 0x8840 0x2c6f3300    0
[  137.759044 <    0.000013>] 416 0xa09d5400   E 0x8840 0x2c6f3a40    0
[  137.759057 <    0.000013>] 417 0xa09d5420   E 0x8840 0x2c6f4180    0
[  137.759071 <    0.000014>] 418 0xa09d5440   E 0x8840 0x2c6f48c0    0
[  137.759084 <    0.000013>] 419 0xa09d5460   E 0x8840 0x2c6f5000    0

Re: net: fec: rx descriptor ring out of order

[Eric Dumazet]

On 11/12/20 12:56 PM, Kegl Rohit wrote:

> Our kernel already has some patches like the wmb() for the rx path and
> the rmb() for the tx path applied.

Well, please do not claim you use 3.10.108 then. :/

Really, there is no point for us trying to guess if one of your local change
went wrong.

I will stop trying to help right there.

Re: net: fec: rx descriptor ring out of order

[Fabio Estevam]

On Thu, Nov 12, 2020 at 8:56 AM Kegl Rohit <keglrohit@gmail.com> wrote:

> Not so easily possible because there are custom drivers and some
> kernel modifications in the mix.

imx6 is a well supported SoC in mainline.
You should try a mainline kernel, otherwise we cannot help you.

RE: [EXT] Fwd: net: fec: rx descriptor ring out of order

[Andy Duan]

From: Kegl Rohit <keglrohit@gmail.com> Sent: Wednesday, November 11, 2020 10:27 PM
> Hello!
> 
> We are using a imx6q platform.
> The fec interface is used to receive a continuous stream of custom / raw
> ethernet packets. The packet size is fixed ~132 bytes and they get sent every
> 250µs.
> 
> While testing I observed spontaneous packet delays from time to time.
> After digging down deeper I think that the fec peripheral does not update the rx
> descriptor status correctly.
> I modified the queue_rx function which is called by the NAPI poll function. "no
> packet N" is printed when the queue_rx function doesn't process any descriptor.
> Therefore the variable N counts continuous calls without ready descriptors.
> When the current descriptor is ready&processed and moved to the next entry,
> then N is cleared again.
> Additionally an error is printed if the current descriptor is empty but the next
> one is already ready. In case this error happens the current descriptor and the
> next 11 ones are dumped.
> "C"  ... current
> "E"  ... empty
> 
> [   57.436478 <    0.020005>] no packet 1!
> [   57.460850 <    0.024372>] no packet 1!
> [   57.461107 <    0.000257>] ring error, current empty but next is not
> empty
> [   57.461118 <    0.000011>] RX ahead
> [   57.461135 <    0.000017>] 129 C E 0x8840 0x2c743a40  132
> [   57.461146 <    0.000011>] 130     0x0840 0x2c744180  132
> [   57.461158 <    0.000012>] 131   E 0x8840 0x2c7448c0  132
> [   57.461170 <    0.000012>] 132   E 0x8840 0x2c745000  132
> [   57.461181 <    0.000011>] 133   E 0x8840 0x2c745740  132
> [   57.461192 <    0.000011>] 134   E 0x8840 0x2c745e80  132
> [   57.461204 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
> [   57.461215 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> [   57.461227 <    0.000012>] 137   E 0x8840 0x2c747440  132
> [   57.461239 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.461250 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.461262 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
> [   57.461477 <    0.000008>] no packet 2!
> [   57.461506 <    0.000029>] ring error, current empty but next is not
> empty
> [   57.461537 <    0.000031>] RX ahead
> [   57.461550 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.461563 <    0.000013>] 130     0x0840 0x2c744180  132
> [   57.461577 <    0.000014>] 131     0x0840 0x2c7448c0  132
> [   57.461589 <    0.000012>] 132     0x0840 0x2c745000  132
> [   57.461601 <    0.000012>] 133   E 0x8840 0x2c745740  132
> [   57.461613 <    0.000012>] 134   E 0x8840 0x2c745e80  132
> [   57.461624 <    0.000011>] 135   E 0x8880 0x2c7465c0  114
> [   57.461635 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> [   57.461645 <    0.000010>] 137   E 0x8840 0x2c747440  132
> [   57.461657 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.461668 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.461680 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
> [   57.461894 <    0.000009>] no packet 3!
> [   57.461926 <    0.000032>] ring error, current empty but next is not
> empty
> [   57.461935 <    0.000009>] RX ahead
> [   57.461947 <    0.000012>] 129 C E 0x8840 0x2c743a40  132
> [   57.461959 <    0.000012>] 130     0x0840 0x2c744180  132
> [   57.461970 <    0.000011>] 131     0x0840 0x2c7448c0  132
> [   57.461982 <    0.000012>] 132     0x0840 0x2c745000  132
> [   57.461993 <    0.000011>] 133     0x0840 0x2c745740  132
> [   57.462005 <    0.000012>] 134   E 0x8840 0x2c745e80  132
> [   57.462017 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
> [   57.462028 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> [   57.462039 <    0.000011>] 137   E 0x8840 0x2c747440  132
> [   57.462051 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.462062 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.462075 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> [   57.462289 <    0.000009>] no packet 4!
> [   57.462316 <    0.000027>] ring error, current empty but next is not
> empty
> [   57.462326 <    0.000010>] RX ahead
> [   57.462339 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.462351 <    0.000012>] 130     0x0840 0x2c744180  132
> [   57.462362 <    0.000011>] 131     0x0840 0x2c7448c0  132
> [   57.462373 <    0.000011>] 132     0x0840 0x2c745000  132
> [   57.462384 <    0.000011>] 133     0x0840 0x2c745740  132
> [   57.462397 <    0.000013>] 134     0x0840 0x2c745e80  132
> [   57.462408 <    0.000011>] 135     0x0840 0x2c7465c0  132
> [   57.462421 <    0.000013>] 136   E 0x8840 0x2c746d00  132
> [   57.462431 <    0.000010>] 137   E 0x8840 0x2c747440  132
> [   57.462443 <    0.000012>] 138   E 0x8840 0x2c748040  132
> [   57.462454 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.462467 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> [   57.462697 <    0.000009>] no packet 5!
> [   57.462730 <    0.000033>] ring error, current empty but next is not
> empty
> [   57.462739 <    0.000009>] RX ahead
> [   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.462763 <    0.000011>] 130     0x0840 0x2c744180  132
> [   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132
> [   57.462787 <    0.000012>] 132     0x0840 0x2c745000  132
> [   57.462799 <    0.000012>] 133     0x0840 0x2c745740  132
> [   57.462809 <    0.000010>] 134     0x0840 0x2c745e80  132
> [   57.462820 <    0.000011>] 135     0x0840 0x2c7465c0  132
> [   57.462830 <    0.000010>] 136     0x0840 0x2c746d00  132
> [   57.462842 <    0.000012>] 137     0x0840 0x2c747440  132
> [   57.462853 <    0.000011>] 138   E 0x8840 0x2c748040  132
> [   57.462864 <    0.000011>] 139   E 0x8840 0x2c748780  132
> [   57.462877 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> [   57.463093 <    0.000009>] no packet 6!
> [   57.463120 <    0.000027>] RX ahead
> [   57.463133 <    0.000013>] 129 C   0x0840 0x2c743a40  132
> [   57.463144 <    0.000011>] 130     0x0840 0x2c744180  132
> [   57.463155 <    0.000011>] 131     0x0840 0x2c7448c0  132
> [   57.463166 <    0.000011>] 132     0x0840 0x2c745000  132
> [   57.463179 <    0.000013>] 133     0x0840 0x2c745740  132
> [   57.463190 <    0.000011>] 134     0x0840 0x2c745e80  132
> [   57.463201 <    0.000011>] 135     0x0840 0x2c7465c0  132
> [   57.463213 <    0.000012>] 136     0x0840 0x2c746d00  132
> [   57.463224 <    0.000011>] 137     0x0840 0x2c747440  132
> [   57.463235 <    0.000011>] 138     0x0840 0x2c748040  132
> [   57.463245 <    0.000010>] 139   E 0x8840 0x2c748780  132
> [   57.463256 <    0.000011>] 140   E 0x8840 0x2c748ec0  132
> [   57.463695 <    0.000244>] rx 12
> 
> As you can see, the described error is catched and the ring is dumped.
> 9 descriptors got ready before the current descriptor is ready.
> After that the current descriptor got ready and 12 packets were processed at
> once.
> I could also observe cases where the ring (512 entries) got full before the
> current descriptor was cleared.
> And also cases where the current and next descriptor were not ready.
> [   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> [   57.462763 <    0.000011>] 130    E 0x0840 0x2c744180  132
> [   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132
> 
> I am suspecting the errata:
> 
> ERR005783 ENET: ENET Status FIFO may overflow due to consecutive short
> frames
> Description:
> When the MAC receives shorter frames (size 64 bytes) at a rate exceeding the
> average line-rate burst traffic of 400 Mbps the DMA is able to absorb, the
> receiver might drop incoming frames before a Pause frame is issued.
> Projected Impact:
> No malfunction will result aside from the frame drops.
> Workarounds:
> The application might want to implement some flow control to ensure the
> line-rate burst traffic is below 400 Mbps if it only uses consecutive small frames
> with minimal
> (96 bit times) or short
> Inter-frame gap (IFG) time following large frames at such a high rate.
> The limit does not exist for
> frames of size larger than 800 bytes.
> Proposed Solution:
> No fix scheduled
> Linux BSP Status:
> Workaround possible but not implemented in the BSP, impacting functionality as
> described above.
> 
> Is the "ENET Status FIFO" some internal hardware FIFO or is it the descriptor
> ring.
> What would be the workaround when a "Workaround is possible"?
> 
> I could only think of skipping/dropping the descriptor when the current is still
> busy but the next one is ready.
> But it is not easily possible because the "stuck" descriptor gets ready after a
> huge delay.
> 
> Is this issue known already? Any suggestions?
> 

We don't see the issue.

Yes, the IP has the errata on i.MX6Q,  so the workaround is to enable HW flow control.
Keep HW flow control is enabled on your networking connection to avoid FIFO overrun happen.

Regards,
Andy 
> 
> Thanks in advance

Re: [EXT] Fwd: net: fec: rx descriptor ring out of order

[Kegl Rohit]

On Thu, Nov 12, 2020 at 2:29 AM Andy Duan <fugang.duan@nxp.com> wrote:
>
> From: Kegl Rohit <keglrohit@gmail.com> Sent: Wednesday, November 11, 2020 10:27 PM
> > Hello!
> >
> > We are using a imx6q platform.
> > The fec interface is used to receive a continuous stream of custom / raw
> > ethernet packets. The packet size is fixed ~132 bytes and they get sent every
> > 250µs.
> >
> > While testing I observed spontaneous packet delays from time to time.
> > After digging down deeper I think that the fec peripheral does not update the rx
> > descriptor status correctly.
> > I modified the queue_rx function which is called by the NAPI poll function. "no
> > packet N" is printed when the queue_rx function doesn't process any descriptor.
> > Therefore the variable N counts continuous calls without ready descriptors.
> > When the current descriptor is ready&processed and moved to the next entry,
> > then N is cleared again.
> > Additionally an error is printed if the current descriptor is empty but the next
> > one is already ready. In case this error happens the current descriptor and the
> > next 11 ones are dumped.
> > "C"  ... current
> > "E"  ... empty
> >
> > [   57.436478 <    0.020005>] no packet 1!
> > [   57.460850 <    0.024372>] no packet 1!
> > [   57.461107 <    0.000257>] ring error, current empty but next is not
> > empty
> > [   57.461118 <    0.000011>] RX ahead
> > [   57.461135 <    0.000017>] 129 C E 0x8840 0x2c743a40  132
> > [   57.461146 <    0.000011>] 130     0x0840 0x2c744180  132
> > [   57.461158 <    0.000012>] 131   E 0x8840 0x2c7448c0  132
> > [   57.461170 <    0.000012>] 132   E 0x8840 0x2c745000  132
> > [   57.461181 <    0.000011>] 133   E 0x8840 0x2c745740  132
> > [   57.461192 <    0.000011>] 134   E 0x8840 0x2c745e80  132
> > [   57.461204 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
> > [   57.461215 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> > [   57.461227 <    0.000012>] 137   E 0x8840 0x2c747440  132
> > [   57.461239 <    0.000012>] 138   E 0x8840 0x2c748040  132
> > [   57.461250 <    0.000011>] 139   E 0x8840 0x2c748780  132
> > [   57.461262 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
> > [   57.461477 <    0.000008>] no packet 2!
> > [   57.461506 <    0.000029>] ring error, current empty but next is not
> > empty
> > [   57.461537 <    0.000031>] RX ahead
> > [   57.461550 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> > [   57.461563 <    0.000013>] 130     0x0840 0x2c744180  132
> > [   57.461577 <    0.000014>] 131     0x0840 0x2c7448c0  132
> > [   57.461589 <    0.000012>] 132     0x0840 0x2c745000  132
> > [   57.461601 <    0.000012>] 133   E 0x8840 0x2c745740  132
> > [   57.461613 <    0.000012>] 134   E 0x8840 0x2c745e80  132
> > [   57.461624 <    0.000011>] 135   E 0x8880 0x2c7465c0  114
> > [   57.461635 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> > [   57.461645 <    0.000010>] 137   E 0x8840 0x2c747440  132
> > [   57.461657 <    0.000012>] 138   E 0x8840 0x2c748040  132
> > [   57.461668 <    0.000011>] 139   E 0x8840 0x2c748780  132
> > [   57.461680 <    0.000012>] 140   E 0x8840 0x2c748ec0  132
> > [   57.461894 <    0.000009>] no packet 3!
> > [   57.461926 <    0.000032>] ring error, current empty but next is not
> > empty
> > [   57.461935 <    0.000009>] RX ahead
> > [   57.461947 <    0.000012>] 129 C E 0x8840 0x2c743a40  132
> > [   57.461959 <    0.000012>] 130     0x0840 0x2c744180  132
> > [   57.461970 <    0.000011>] 131     0x0840 0x2c7448c0  132
> > [   57.461982 <    0.000012>] 132     0x0840 0x2c745000  132
> > [   57.461993 <    0.000011>] 133     0x0840 0x2c745740  132
> > [   57.462005 <    0.000012>] 134   E 0x8840 0x2c745e80  132
> > [   57.462017 <    0.000012>] 135   E 0x8880 0x2c7465c0  114
> > [   57.462028 <    0.000011>] 136   E 0x8840 0x2c746d00  132
> > [   57.462039 <    0.000011>] 137   E 0x8840 0x2c747440  132
> > [   57.462051 <    0.000012>] 138   E 0x8840 0x2c748040  132
> > [   57.462062 <    0.000011>] 139   E 0x8840 0x2c748780  132
> > [   57.462075 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> > [   57.462289 <    0.000009>] no packet 4!
> > [   57.462316 <    0.000027>] ring error, current empty but next is not
> > empty
> > [   57.462326 <    0.000010>] RX ahead
> > [   57.462339 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> > [   57.462351 <    0.000012>] 130     0x0840 0x2c744180  132
> > [   57.462362 <    0.000011>] 131     0x0840 0x2c7448c0  132
> > [   57.462373 <    0.000011>] 132     0x0840 0x2c745000  132
> > [   57.462384 <    0.000011>] 133     0x0840 0x2c745740  132
> > [   57.462397 <    0.000013>] 134     0x0840 0x2c745e80  132
> > [   57.462408 <    0.000011>] 135     0x0840 0x2c7465c0  132
> > [   57.462421 <    0.000013>] 136   E 0x8840 0x2c746d00  132
> > [   57.462431 <    0.000010>] 137   E 0x8840 0x2c747440  132
> > [   57.462443 <    0.000012>] 138   E 0x8840 0x2c748040  132
> > [   57.462454 <    0.000011>] 139   E 0x8840 0x2c748780  132
> > [   57.462467 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> > [   57.462697 <    0.000009>] no packet 5!
> > [   57.462730 <    0.000033>] ring error, current empty but next is not
> > empty
> > [   57.462739 <    0.000009>] RX ahead
> > [   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> > [   57.462763 <    0.000011>] 130     0x0840 0x2c744180  132
> > [   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132
> > [   57.462787 <    0.000012>] 132     0x0840 0x2c745000  132
> > [   57.462799 <    0.000012>] 133     0x0840 0x2c745740  132
> > [   57.462809 <    0.000010>] 134     0x0840 0x2c745e80  132
> > [   57.462820 <    0.000011>] 135     0x0840 0x2c7465c0  132
> > [   57.462830 <    0.000010>] 136     0x0840 0x2c746d00  132
> > [   57.462842 <    0.000012>] 137     0x0840 0x2c747440  132
> > [   57.462853 <    0.000011>] 138   E 0x8840 0x2c748040  132
> > [   57.462864 <    0.000011>] 139   E 0x8840 0x2c748780  132
> > [   57.462877 <    0.000013>] 140   E 0x8840 0x2c748ec0  132
> > [   57.463093 <    0.000009>] no packet 6!
> > [   57.463120 <    0.000027>] RX ahead
> > [   57.463133 <    0.000013>] 129 C   0x0840 0x2c743a40  132
> > [   57.463144 <    0.000011>] 130     0x0840 0x2c744180  132
> > [   57.463155 <    0.000011>] 131     0x0840 0x2c7448c0  132
> > [   57.463166 <    0.000011>] 132     0x0840 0x2c745000  132
> > [   57.463179 <    0.000013>] 133     0x0840 0x2c745740  132
> > [   57.463190 <    0.000011>] 134     0x0840 0x2c745e80  132
> > [   57.463201 <    0.000011>] 135     0x0840 0x2c7465c0  132
> > [   57.463213 <    0.000012>] 136     0x0840 0x2c746d00  132
> > [   57.463224 <    0.000011>] 137     0x0840 0x2c747440  132
> > [   57.463235 <    0.000011>] 138     0x0840 0x2c748040  132
> > [   57.463245 <    0.000010>] 139   E 0x8840 0x2c748780  132
> > [   57.463256 <    0.000011>] 140   E 0x8840 0x2c748ec0  132
> > [   57.463695 <    0.000244>] rx 12
> >
> > As you can see, the described error is catched and the ring is dumped.
> > 9 descriptors got ready before the current descriptor is ready.
> > After that the current descriptor got ready and 12 packets were processed at
> > once.
> > I could also observe cases where the ring (512 entries) got full before the
> > current descriptor was cleared.
> > And also cases where the current and next descriptor were not ready.
> > [   57.462752 <    0.000013>] 129 C E 0x8840 0x2c743a40  132
> > [   57.462763 <    0.000011>] 130    E 0x0840 0x2c744180  132
> > [   57.462775 <    0.000012>] 131     0x0840 0x2c7448c0  132
> >
> > I am suspecting the errata:
> >
> > ERR005783 ENET: ENET Status FIFO may overflow due to consecutive short
> > frames
> > Description:
> > When the MAC receives shorter frames (size 64 bytes) at a rate exceeding the
> > average line-rate burst traffic of 400 Mbps the DMA is able to absorb, the
> > receiver might drop incoming frames before a Pause frame is issued.
> > Projected Impact:
> > No malfunction will result aside from the frame drops.
> > Workarounds:
> > The application might want to implement some flow control to ensure the
> > line-rate burst traffic is below 400 Mbps if it only uses consecutive small frames
> > with minimal
> > (96 bit times) or short
> > Inter-frame gap (IFG) time following large frames at such a high rate.
> > The limit does not exist for
> > frames of size larger than 800 bytes.
> > Proposed Solution:
> > No fix scheduled
> > Linux BSP Status:
> > Workaround possible but not implemented in the BSP, impacting functionality as
> > described above.
> >
> > Is the "ENET Status FIFO" some internal hardware FIFO or is it the descriptor
> > ring.
> > What would be the workaround when a "Workaround is possible"?
> >
> > I could only think of skipping/dropping the descriptor when the current is still
> > busy but the next one is ready.
> > But it is not easily possible because the "stuck" descriptor gets ready after a
> > huge delay.
> >
> > Is this issue known already? Any suggestions?
> >
>
> We don't see the issue.
>
> Yes, the IP has the errata on i.MX6Q,  so the workaround is to enable HW flow control.
> Keep HW flow control is enabled on your networking connection to avoid FIFO overrun happen.
>
> Regards,
> Andy
> >
> > Thanks in advance

Ok, after rereading the errata I don't think that they are the problem.

ERR004512 ENET: 1 Gb Ethernet MAC (ENET) system limitation.

Here flow control should be the solution.
We are using a 100MBit/s full duplex link and the generated test
stream is only 4MBit/s, so this issue should not apply.


ERR005783 ENET: ENET Status FIFO may overflow due to consecutive short frames.
When the MAC receives shorter frames (size 64 bytes) at a rate
exceeding the average line-rate
burst traffic of 400 Mbps the DMA is able to absorb, the receiver
might drop incoming frames
before a Pause frame is issued.

In this case the hardware flow control workaround will not help and a
flow control has to be done at software protocol level.