ep93xx_eth stopps receiving packages

ep93xx_eth stopps receiving packages

[Stefan Agner]

Hello,

I'm using Linux 2.6.32.9 on a technologic systems TS-7250 SBC board, with
the ep93xx_eth driver for networking. On three identical, but independent
systems I noted that the system is unreachable after a while. On a serial
terminal I noted that only the TX counter counts onward, RX stays where it is,
no matter if i try to ping from or to the system. Wireshark tells me exactly
that too: I see helpless ARP requests which gets answered, but no ICMP. The
system doesnt receive the ARP requests, and just sends another one.
With a simple program which sends small packages in a fast pace I can
reproduce the problem after several seconds (additional CPU load seem to
provoke the problem even more). Remove and replug the network cable doesn't
solve the problem, but ifup/down does. I don't see any messages in dmesg,
memory is still available.

Is it a network driver problem or even a network stack problem?

It looks to me like a race condition in the network driver, which can be
triggered by short packets.

What can I do to track the problem further down?

Thanks for hints,
Stefan

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: ep93xx_eth stopps receiving packages

[Lennert Buytenhek]

On Mon, Apr 19, 2010 at 05:38:13PM +0200, Stefan Agner wrote:

> I'm using Linux 2.6.32.9 on a technologic systems TS-7250 SBC board, with
> the ep93xx_eth driver for networking. On three identical, but independent
> systems I noted that the system is unreachable after a while. On a serial
> terminal I noted that only the TX counter counts onward, RX stays where it 
> is,
> no matter if i try to ping from or to the system. Wireshark tells me exactly
> that too: I see helpless ARP requests which gets answered, but no ICMP. The
> system doesnt receive the ARP requests, and just sends another one.

(So does the board or does it not respond to ARP requests for its IP?)


> With a simple program which sends small packages in a fast pace I can
> reproduce the problem after several seconds (additional CPU load seem to
> provoke the problem even more). Remove and replug the network cable doesn't
> solve the problem, but ifup/down does. I don't see any messages in dmesg,
> memory is still available.

Do you see interrupts increasing in /proc/interrupts when this happens?

Re: ep93xx_eth stopps receiving packages

[Stefan Agner]

Quoting Lennert Buytenhek <buytenh@wantstofly.org>:

> On Mon, Apr 19, 2010 at 05:38:13PM +0200, Stefan Agner wrote:
>
>> I'm using Linux 2.6.32.9 on a technologic systems TS-7250 SBC board, with
>> the ep93xx_eth driver for networking. On three identical, but independent
>> systems I noted that the system is unreachable after a while. On a serial
>> terminal I noted that only the TX counter counts onward, RX stays where it
>> is,
>> no matter if i try to ping from or to the system. Wireshark tells me exactly
>> that too: I see helpless ARP requests which gets answered, but no ICMP. The
>> system doesnt receive the ARP requests, and just sends another one.
>
> (So does the board or does it not respond to ARP requests for its IP?)
The board does not responds to ARP requests for its IP...

>> With a simple program which sends small packages in a fast pace I can
>> reproduce the problem after several seconds (additional CPU load seem to
>> provoke the problem even more). Remove and replug the network cable doesn't
>> solve the problem, but ifup/down does. I don't see any messages in dmesg,
>> memory is still available.
>
> Do you see interrupts increasing in /proc/interrupts when this happens?
No, interrupt doesn't increase anymore when it happens...

I debugged the problem myself inbetween, I just took not the time to format
and send the patch, sorry! There is a bug when interrupts gets disabled for
a longer period, each a frame arrives when one is just been processed.
ep93xx_rx gets called twice then, and the second time marks to many buffers
as relased. I corrected this error by releasing the correct number of buffers
when ep93xx_poll ends... Patch follows!

Stefan


-- 
Stefan Agner

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

[PATCH] ep93xx_eth stopps receiving packets

[Stefan Agner]

Receiving small packet(s) in a fast pace leads to not receiving any
packets at all after some time.

After ethernet packet(s) arrived the receive descriptor is incremented
by the number of frames processed. If another packet arrives while
processing, this is processed in another call of ep93xx_rx. This
second call leads that too many receive descriptors getting released.

This fix increments, even in these case, the right number of processed
receive descriptors.

Signed-off-by: Stefan Agner <stefan@agner.ch>
---
  drivers/net/arm/ep93xx_eth.c |   10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/net/arm/ep93xx_eth.c b/drivers/net/arm/ep93xx_eth.c
index 6995169..cd6cd3e 100644
--- a/drivers/net/arm/ep93xx_eth.c
+++ b/drivers/net/arm/ep93xx_eth.c
@@ -311,11 +311,6 @@ err:
  		processed++;
  	}

-	if (processed) {
-		wrw(ep, REG_RXDENQ, processed);
-		wrw(ep, REG_RXSTSENQ, processed);
-	}
-
  	return processed;
  }

@@ -350,6 +345,11 @@ poll_some_more:
  			goto poll_some_more;
  	}

+	if (rx) {
+                wrw(ep, REG_RXDENQ, rx);
+                wrw(ep, REG_RXSTSENQ, rx);
+        }
+
  	return rx;
  }

-- 
1.7.0


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: [PATCH] ep93xx_eth stopps receiving packets

[Lennert Buytenhek]

On Mon, May 03, 2010 at 01:42:44PM +0200, Stefan Agner wrote:

> Receiving small packet(s) in a fast pace leads to not receiving any
> packets at all after some time.
> 
> After ethernet packet(s) arrived the receive descriptor is incremented
> by the number of frames processed. If another packet arrives while
> processing, this is processed in another call of ep93xx_rx. This
> second call leads that too many receive descriptors getting released.
> 
> This fix increments, even in these case, the right number of processed
> receive descriptors.
> 
> Signed-off-by: Stefan Agner <stefan@agner.ch>

I haven't opened my ep93xx docs for a while, but if this works for you:

Acked-by: Lennert Buytenhek <buytenh@wantstofly.org>

Thanks!

Re: [PATCH] ep93xx_eth stopps receiving packets

[David Miller]

From: Lennert Buytenhek <buytenh@wantstofly.org>
Date: Tue, 4 May 2010 03:46:07 +0200

> On Mon, May 03, 2010 at 01:42:44PM +0200, Stefan Agner wrote:
> 
>> Receiving small packet(s) in a fast pace leads to not receiving any
>> packets at all after some time.
>> 
>> After ethernet packet(s) arrived the receive descriptor is incremented
>> by the number of frames processed. If another packet arrives while
>> processing, this is processed in another call of ep93xx_rx. This
>> second call leads that too many receive descriptors getting released.
>> 
>> This fix increments, even in these case, the right number of processed
>> receive descriptors.
>> 
>> Signed-off-by: Stefan Agner <stefan@agner.ch>
> 
> I haven't opened my ep93xx docs for a while, but if this works for you:
> 
> Acked-by: Lennert Buytenhek <buytenh@wantstofly.org>

I had to apply this by hand because Stefan's email client corrupted the
spacing et al. in the patch.  Stefan please post a usable patch next
time.