UDP packets arriving on wrong sockets

UDP packets arriving on wrong sockets

[Andrew Cann]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

Hi, 

I posted this on stackoverflow yesterday but I'm reposting it here since it got
no response. Original post: https://stackoverflow.com/questions/51630337/udp-packets-arriving-on-wrong-sockets-on-linux

I have two UDP sockets bound to the same address and connected to addresses A
and B. I have two more UDP sockets bound to A and B and not connected.

This is what my /proc/net/udp looks like (trimmed for readability):

      sl  local_address rem_address
     3937: 0100007F:DD9C 0300007F:9910
     3937: 0100007F:DD9C 0200007F:907D
    16962: 0200007F:907D 00000000:0000
    19157: 0300007F:9910 00000000:0000

According to connect(2): "If the socket sockfd is of type SOCK_DGRAM, then addr
is the address to which datagrams are sent by default, *and the only address
from which datagrams are received*."

For some reason, my connected sockets are receiving packets that were destined
for each other. eg: The UDP socket connected to A sends a message to A, A then
sends a reply back. The UDP socket connected to B sends a message to B, B then
sends a reply back. But the reply from A arrives at the socket connected to B
and the reply from B arrives at the socket connected to A.

Why on earth would this be happening? Note that it happens randomly - sometimes
the replies arrive at the correct sockets and sometimes they don't. Is there
any way to prevent this or any situation under which connect() is supposed to
not work?

Any help explaining this would be hugely appreciated :)

 - Andrew


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: UDP packets arriving on wrong sockets

[Eric Dumazet]

On 08/02/2018 02:05 AM, Andrew Cann wrote:
> Hi, 
> 
> I posted this on stackoverflow yesterday but I'm reposting it here since it got
> no response. Original post: https://stackoverflow.com/questions/51630337/udp-packets-arriving-on-wrong-sockets-on-linux
> 
> I have two UDP sockets bound to the same address and connected to addresses A
> and B. I have two more UDP sockets bound to A and B and not connected.
> 
> This is what my /proc/net/udp looks like (trimmed for readability):
> 
>       sl  local_address rem_address
>      3937: 0100007F:DD9C 0300007F:9910
>      3937: 0100007F:DD9C 0200007F:907D
>     16962: 0200007F:907D 00000000:0000
>     19157: 0300007F:9910 00000000:0000
> 
> According to connect(2): "If the socket sockfd is of type SOCK_DGRAM, then addr
> is the address to which datagrams are sent by default, *and the only address
> from which datagrams are received*."
> 
> For some reason, my connected sockets are receiving packets that were destined
> for each other. eg: The UDP socket connected to A sends a message to A, A then
> sends a reply back. The UDP socket connected to B sends a message to B, B then
> sends a reply back. But the reply from A arrives at the socket connected to B
> and the reply from B arrives at the socket connected to A.
> 
> Why on earth would this be happening? Note that it happens randomly - sometimes
> the replies arrive at the correct sockets and sometimes they don't. Is there
> any way to prevent this or any situation under which connect() is supposed to
> not work?
> 
> Any help explaining this would be hugely appreciated :)

Hi Andrew

Well, you should first give much more details, as there are thousands of different UDP stacks out there.

Documentation/admin-guide/reporting-bugs.rst

...
[4.1.] Kernel version (from /proc/version): 
...

Ideally you could give us a C reproducer, so that we can run it ourselves and fix the kernel bug if there is one.

This C reproducer could be part of an official patch, adding a test in tools/testing/selftests/net

Thanks !

Re: UDP packets arriving on wrong sockets

[Eric Dumazet]

On 08/02/2018 06:20 AM, Eric Dumazet wrote:
> 
> Ideally you could give us a C reproducer, so that we can run it ourselves and fix the kernel bug if there is one.
> 
> This C reproducer could be part of an official patch, adding a test in tools/testing/selftests/net

Alternatively a test in Python would be accepted ;)

Re: UDP packets arriving on wrong sockets

[Willem de Bruijn]

On Thu, Aug 2, 2018 at 9:21 AM Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
>
>
> On 08/02/2018 02:05 AM, Andrew Cann wrote:
> > Hi,
> >
> > I posted this on stackoverflow yesterday but I'm reposting it here since it got
> > no response. Original post: https://stackoverflow.com/questions/51630337/udp-packets-arriving-on-wrong-sockets-on-linux
> >
> > I have two UDP sockets bound to the same address and connected to addresses A
> > and B. I have two more UDP sockets bound to A and B and not connected.
> >
> > This is what my /proc/net/udp looks like (trimmed for readability):
> >
> >       sl  local_address rem_address
> >      3937: 0100007F:DD9C 0300007F:9910
> >      3937: 0100007F:DD9C 0200007F:907D

You have two sockets bound to the same address and port? Is this using
SO_REUSEPORT?

Re: UDP packets arriving on wrong sockets

[Eric Dumazet]

On 08/02/2018 08:21 AM, Willem de Bruijn wrote:
> On Thu, Aug 2, 2018 at 9:21 AM Eric Dumazet <eric.dumazet@gmail.com> wrote:
>>
>>
>>
>> On 08/02/2018 02:05 AM, Andrew Cann wrote:
>>> Hi,
>>>
>>> I posted this on stackoverflow yesterday but I'm reposting it here since it got
>>> no response. Original post: https://stackoverflow.com/questions/51630337/udp-packets-arriving-on-wrong-sockets-on-linux
>>>
>>> I have two UDP sockets bound to the same address and connected to addresses A
>>> and B. I have two more UDP sockets bound to A and B and not connected.
>>>
>>> This is what my /proc/net/udp looks like (trimmed for readability):
>>>
>>>       sl  local_address rem_address
>>>      3937: 0100007F:DD9C 0300007F:9910
>>>      3937: 0100007F:DD9C 0200007F:907D
> 
> You have two sockets bound to the same address and port? Is this using
> SO_REUSEPORT?


Yeah, time to add listen() and accept() support for UDP :)

Re: UDP packets arriving on wrong sockets

[Andrew Cann]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Thu, Aug 02, 2018 at 11:21:41AM -0400, Willem de Bruijn wrote:
> You have two sockets bound to the same address and port? Is this using
> SO_REUSEPORT?

Yes, this is using SO_REUSEPORT.

My colleague wrote a python reproducer for this here:
https://gist.github.com/povilasb/53f1c802dbc2aca36a0ffa5b4cb95536

If you run server.py, then client.py, you should see packets arriving at
opposite sockets about half the time. My kernel version is NixOS 4.14.51, he
tested on two different machines - a debian 4.9.0 and fedora 28 4.17.2. We can
reproduce this on all kernels we tested.

Thanks in advance for any help you can give :)

 - Andrew


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: UDP packets arriving on wrong sockets

[Willem de Bruijn]

On Fri, Aug 3, 2018 at 12:20 AM Andrew Cann <shum@canndrew.org> wrote:
>
> On Thu, Aug 02, 2018 at 11:21:41AM -0400, Willem de Bruijn wrote:
> > You have two sockets bound to the same address and port? Is this using
> > SO_REUSEPORT?
>
> Yes, this is using SO_REUSEPORT.

Then this is working as intended.

Without SO_REUSEPORT it would not be possible to bind two sockets to
the same address and port. See documentation, e.g., at
https://lwn.net/Articles/542629/

Re: UDP packets arriving on wrong sockets

[Andrew Cann]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

On Fri, Aug 03, 2018 at 10:20:06AM -0400, Willem de Bruijn wrote:
> On Fri, Aug 3, 2018 at 12:20 AM Andrew Cann <shum@canndrew.org> wrote:
> >
> > On Thu, Aug 02, 2018 at 11:21:41AM -0400, Willem de Bruijn wrote:
> > > You have two sockets bound to the same address and port? Is this using
> > > SO_REUSEPORT?
> >
> > Yes, this is using SO_REUSEPORT.
> 
> Then this is working as intended.
> 
> Without SO_REUSEPORT it would not be possible to bind two sockets to
> the same address and port. See documentation, e.g., at
> https://lwn.net/Articles/542629/

The man page for connect clearly states that a connected UDP socket should only
receive datagrams from the address that it is connected to. This isn't the
behaviour I'm seeing. That's the issue.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]