More details on why received UDP packets are treated as errors?

More details on why received UDP packets are treated as errors?

[Ben Greear]

We have a test case on 3.9.9+ (local patches applied) where sending from
VETH interface, through peer VETH bridged (with our own emulator bridge module)
to physical interface, which is then looped to another physical interface (B).
The VETH and the wired B interface are sending UDP traffic to each other.
Routing rules should be configured such that this all works appropriately.

Replacing our bridging module with a user-space bridge has same behaviour.

This setup works on the 3.7.y kernel, but we only get one-way traffic
(B to VETH) on 3.9.9+.

I sniffed the B port, and traffic appears to be sent and received
properly (ie, no checksum errors, etc).  But, our user-space app
shows no received UDP frames on B, and netstat -s gives the
output below.

Is there any way to get more details about what these 'packet receive errors'
are caused by using normal-ish tools?

Udp:
     6827 packets received
     0 packets to unknown port received.
     6172 packet receive errors
     12964 packets sent
     0 receive buffer errors
     0 send buffer errors

Tcp connections also fail to work, and I see these TCP stats.  I'm sure
not all of this is related to the problem at hand, but likely some of it
is.

Tcp:
     81 active connections openings
     11 passive connection openings
     66 failed connection attempts
     0 connection resets received
     17 connections established
     204643 segments received
     229686 segments send out
     465 segments retransmited
     322 bad segments received.
     70 resets sent


Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: More details on why received UDP packets are treated as errors?

[Eric Dumazet]

On Mon, 2013-07-29 at 11:21 -0700, Ben Greear wrote:
> We have a test case on 3.9.9+ (local patches applied) where sending from
> VETH interface, through peer VETH bridged (with our own emulator bridge module)
> to physical interface, which is then looped to another physical interface (B).
> The VETH and the wired B interface are sending UDP traffic to each other.
> Routing rules should be configured such that this all works appropriately.
> 
> Replacing our bridging module with a user-space bridge has same behaviour.
> 
> This setup works on the 3.7.y kernel, but we only get one-way traffic
> (B to VETH) on 3.9.9+.
> 
> I sniffed the B port, and traffic appears to be sent and received
> properly (ie, no checksum errors, etc).  But, our user-space app
> shows no received UDP frames on B, and netstat -s gives the
> output below.
> 
> Is there any way to get more details about what these 'packet receive errors'
> are caused by using normal-ish tools?

You could try dropwatch for this kind of obscure drops

https://fedorahosted.org/dropwatch/

Re: More details on why received UDP packets are treated as errors?

[Benjamin Poirier]

On 2013/07/29 13:21, Eric Dumazet wrote:
> On Mon, 2013-07-29 at 11:21 -0700, Ben Greear wrote:
> > We have a test case on 3.9.9+ (local patches applied) where sending from
> > VETH interface, through peer VETH bridged (with our own emulator bridge module)
> > to physical interface, which is then looped to another physical interface (B).
> > The VETH and the wired B interface are sending UDP traffic to each other.
> > Routing rules should be configured such that this all works appropriately.
> > 
> > Replacing our bridging module with a user-space bridge has same behaviour.
> > 
> > This setup works on the 3.7.y kernel, but we only get one-way traffic
> > (B to VETH) on 3.9.9+.
> > 
> > I sniffed the B port, and traffic appears to be sent and received
> > properly (ie, no checksum errors, etc).  But, our user-space app
> > shows no received UDP frames on B, and netstat -s gives the
> > output below.
> > 
> > Is there any way to get more details about what these 'packet receive errors'
> > are caused by using normal-ish tools?
> 
> You could try dropwatch for this kind of obscure drops
> 
> https://fedorahosted.org/dropwatch/

If the drop happens in __udp_queue_rcv_skb() you can also get some info from
the udp_fail_queue_rcv_skb tracepoint. See "296f7ea udp: add tracepoints for
queueing skb to rcvbuf (v3.1-rc1)".

Re: More details on why received UDP packets are treated as errors?

[Ben Greear]

On 07/29/2013 01:21 PM, Eric Dumazet wrote:
> On Mon, 2013-07-29 at 11:21 -0700, Ben Greear wrote:
>> We have a test case on 3.9.9+ (local patches applied) where sending from
>> VETH interface, through peer VETH bridged (with our own emulator bridge module)
>> to physical interface, which is then looped to another physical interface (B).
>> The VETH and the wired B interface are sending UDP traffic to each other.
>> Routing rules should be configured such that this all works appropriately.
>>
>> Replacing our bridging module with a user-space bridge has same behaviour.
>>
>> This setup works on the 3.7.y kernel, but we only get one-way traffic
>> (B to VETH) on 3.9.9+.
>>
>> I sniffed the B port, and traffic appears to be sent and received
>> properly (ie, no checksum errors, etc).  But, our user-space app
>> shows no received UDP frames on B, and netstat -s gives the
>> output below.
>>
>> Is there any way to get more details about what these 'packet receive errors'
>> are caused by using normal-ish tools?
>
> You could try dropwatch for this kind of obscure drops
>
> https://fedorahosted.org/dropwatch/

I had a chance to crank this up today.

dropwatch is pretty sweet...turns out it's dropping due to bad UDP checksums,
due to the VETH bug/issue where it claims to calculate HW checksum but appearantly does
not actually do so (see previous email in another thread).

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com