Fw: [Bug 199995] New: Ramdomly sent TCP Reset from Kernel with bonding mode "brodcast"

Fw: [Bug 199995] New: Ramdomly sent TCP Reset from Kernel with bonding mode "brodcast"

[Stephen Hemminger]

Begin forwarded message:

Date: Fri, 08 Jun 2018 16:06:40 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 199995] New: Ramdomly sent TCP Reset from Kernel with bonding mode "brodcast"


https://bugzilla.kernel.org/show_bug.cgi?id=199995

            Bug ID: 199995
           Summary: Ramdomly sent TCP Reset from Kernel with bonding mode
                    "brodcast"
           Product: Networking
           Version: 2.5
    Kernel Version: since 4.15.0
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
          Assignee: stephen@networkplumber.org
          Reporter: l.bendel@portunity.de
        Regression: No

Created attachment 276401
  --> https://bugzilla.kernel.org/attachment.cgi?id=276401&action=edit  
TCP Dump

Hi,

after a dist upgrade from Ubuntu 17.10 (Kernel 4.13.x) to Ubuntu 18.04 (Kernel
4.15.0) I suffer from ramdomly generated TCP RST packets sent (presumably) by
the Kernel 
on a bonding device that uses bonding mode "brodcast" with 2 physical NICs.

With tcpdump/whireshark I can see that the kernel randomly sends TCP-RST
packets after the SYN/ACK/ACK packet is received (see attached PCAP).
This only happens if the kernel receives the initial SYN packet on both
physical NICs (and therefore seeing it twice), before the connection is
established by sending SYN/ACK.
It's not happening in 100% of all cases and only, if the system can use two or
more CPU cores/threads. With only one CPU available to the system, this
behaviour is not reproducable.


I can reproduce this on multiple physical servers with 2 bonded Intel NICs
connected over 2 seperate Switches and with virtual machines on a KVM Host
using 2 dedicated host bridges.
This also happens with a fresh installed Ubuntu 18.04 and Fedora 28 (kernel
4.16), so I decided to compile and boot with Kernel 4.17.0 on ubuntu, getting
the same result.
Only disabling/blocking the second network connection or reducing the amount of
CPU cores of the VM to one core solves the problem, so I think this could be a
race condition on systems with more than one CPU core and thread.

For my tests I used a very basic Ubuntu 18.04 (x86-64) running xinetd tcp-echo
service (port 7/TCP).
On the client I used the netcat-traditional packet with the following command:

  while true; do echo $(date) | nc.traditional -q 1 ECHO-SERVER 7; sleep 0.1 ;
done


This gives the following output:

---------------------------------------
Fr 8. Jun 09:12:43 UTC 2018
Fr 8. Jun 09:12:43 UTC 2018
Fr 8. Jun 09:12:43 UTC 2018
Fr 8. Jun 09:12:43 UTC 2018
Fr 8. Jun 09:12:43 UTC 2018
Fr 8. Jun 09:12:43 UTC 2018
Fr 8. Jun 09:12:43 UTC 2018
Fr 8. Jun 09:12:43 UTC 2018
(UNKNOWN) [192.168.86.101] 7 (echo) : Connection reset by peer
(UNKNOWN) [192.168.86.101] 7 (echo) : Connection reset by peer
(UNKNOWN) [192.168.86.101] 7 (echo) : Connection reset by peer
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
Fr 8. Jun 09:12:44 UTC 2018
(UNKNOWN) [192.168.86.101] 7 (echo) : Connection reset by peer
(UNKNOWN) [192.168.86.101] 7 (echo) : Connection reset by peer
Fr 8. Jun 09:12:44 UTC 2018
---------------------------------------

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: Fw: [Bug 199995] New: Ramdomly sent TCP Reset from Kernel with bonding mode "brodcast"

[Michal Kubecek]

On Fri, Jun 08, 2018 at 09:59:54AM -0700, Stephen Hemminger wrote:
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=199995
> 
>             Bug ID: 199995
>            Summary: Ramdomly sent TCP Reset from Kernel with bonding mode
>                     "brodcast"
> 
> after a dist upgrade from Ubuntu 17.10 (Kernel 4.13.x) to Ubuntu 18.04 (Kernel
> 4.15.0) I suffer from ramdomly generated TCP RST packets sent (presumably) by
> the Kernel 
> on a bonding device that uses bonding mode "brodcast" with 2 physical NICs.
> 
> With tcpdump/whireshark I can see that the kernel randomly sends TCP-RST
> packets after the SYN/ACK/ACK packet is received (see attached PCAP).
> This only happens if the kernel receives the initial SYN packet on both
> physical NICs (and therefore seeing it twice), before the connection is
> established by sending SYN/ACK.
> It's not happening in 100% of all cases and only, if the system can use two or
> more CPU cores/threads. With only one CPU available to the system, this
> behaviour is not reproducable.

I have seen similar report earlier from one of our customers running
SLE12 SP2 (kernel 4.4). The problem is that if duplicated SYN packet is
received on both slaves, these two copies can be processed by the
lockless listener simultaneously on different CPUs and each can reply by
SYNACK with different sequence number which results in a reset.

I tried to think of a way to prevent this race without losing the
performance gain of lockless listener but couldn't come with anything.
Eventually, I managed to persuade the customer that this setup (where
each packet is received twice under normal circumstances) is not what
broadcast mode was designed for (based on the description in
Documentation/networking/bonding.txt).

However, the lockless listener was introduced in 4.4 so it's not clear
why reporter started encountering this after an upgrade from 4.13 to
4.15.

Michal Kubecek

Re: Fw: [Bug 199995] New: Ramdomly sent TCP Reset from Kernel with bonding mode "brodcast"

[Eric Dumazet]

On 06/08/2018 02:04 PM, Michal Kubecek wrote:
> On Fri, Jun 08, 2018 at 09:59:54AM -0700, Stephen Hemminger wrote:
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=199995
>>
>>             Bug ID: 199995
>>            Summary: Ramdomly sent TCP Reset from Kernel with bonding mode
>>                     "brodcast"
>>
>> after a dist upgrade from Ubuntu 17.10 (Kernel 4.13.x) to Ubuntu 18.04 (Kernel
>> 4.15.0) I suffer from ramdomly generated TCP RST packets sent (presumably) by
>> the Kernel 
>> on a bonding device that uses bonding mode "brodcast" with 2 physical NICs.
>>
>> With tcpdump/whireshark I can see that the kernel randomly sends TCP-RST
>> packets after the SYN/ACK/ACK packet is received (see attached PCAP).
>> This only happens if the kernel receives the initial SYN packet on both
>> physical NICs (and therefore seeing it twice), before the connection is
>> established by sending SYN/ACK.
>> It's not happening in 100% of all cases and only, if the system can use two or
>> more CPU cores/threads. With only one CPU available to the system, this
>> behaviour is not reproducable.
> 
> I have seen similar report earlier from one of our customers running
> SLE12 SP2 (kernel 4.4). The problem is that if duplicated SYN packet is
> received on both slaves, these two copies can be processed by the
> lockless listener simultaneously on different CPUs and each can reply by
> SYNACK with different sequence number which results in a reset.
> 
> I tried to think of a way to prevent this race without losing the
> performance gain of lockless listener but couldn't come with anything.
> Eventually, I managed to persuade the customer that this setup (where
> each packet is received twice under normal circumstances) is not what
> broadcast mode was designed for (based on the description in
> Documentation/networking/bonding.txt).
> 
> However, the lockless listener was introduced in 4.4 so it's not clear
> why reporter started encountering this after an upgrade from 4.13 to
> 4.15.

Yes, I do not buy this at all.

If two identical SYN are received by two cpus, we should create one SYN_RECV and send
two SYNACK.

But it is a bit hard to test this :/

I will take a look, thanks.

Re: Fw: [Bug 199995] New: Ramdomly sent TCP Reset from Kernel with bonding mode "brodcast"

[Eric Dumazet]

On 06/08/2018 02:38 PM, Eric Dumazet wrote:
> 
> 
> On 06/08/2018 02:04 PM, Michal Kubecek wrote:

>>
>> However, the lockless listener was introduced in 4.4 so it's not clear
>> why reporter started encountering this after an upgrade from 4.13 to
>> 4.15.
> 
> Yes, I do not buy this at all.
> 
> If two identical SYN are received by two cpus, we should create one SYN_RECV and send
> two SYNACK.
> 
> But it is a bit hard to test this :/
> 
> I will take a look, thanks.


Oh well, this is not done as I thought, this needs a fix, I will work on this.

reqsk_queue_hash_req() calls inet_ehash_insert() without making sure that the same 4-tuple
is not already there.

Do not worry, we will keep the listener lockless :)