Re: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection

From: Jakub Kicinski <kuba@kernel.org>
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: netdev@vger.kernel.org, Jay Vosburgh <j.vosburgh@gmail.com>,
	Jiri Pirko <jiri@resnulli.us>, Jarod Wilson <jarod@redhat.com>
Subject: Re: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection
Date: Wed, 21 Oct 2020 11:37:29 -0700	[thread overview]
Message-ID: <20201021113729.4d4eeffa@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> (raw)
In-Reply-To: <20201020075429.291c34bb@hermes.local>

CC some bonding folks

On Tue, 20 Oct 2020 07:54:29 -0700 Stephen Hemminger wrote:
> Begin forwarded message:
> 
> Date: Tue, 20 Oct 2020 10:42:34 +0000
> From: bugzilla-daemon@bugzilla.kernel.org
> To: stephen@networkplumber.org
> Subject: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection
> 
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=209767
> 
>             Bug ID: 209767
>            Summary: Bonding 802.3ad layer2+3 transmits on both slaves
>                     within single connection
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 5.8.11-1.el8.elrepo.x86_64 and
>                     3.10.0-1127.19.1.el7.x86_64
>           Hardware: All
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>           Assignee: stephen@networkplumber.org
>           Reporter: onno.zweers@surf.nl
>         Regression: No
> 
> Dear people,
> 
> I'm seeing bonding behavior I don't understand and neither do several network
> experts I've consulted.
> 
> We have two servers, both with two 25 Gbit interfaces in a bond (802.3ad with
> layer2+3 hashing). We tuned the systems according to
> https://fasterdata.es.net/host-tuning/linux/. I run `iperf3 --server` on server
> 1 and connect to it with `iperf3 --client server1` from server 2. We notice
> that sometimes the connection is good (24.7 Gbit/s, no retransmits) and
> sometimes there are many retransmits (sometimes as many as >30,000 in a 10
> second run) and then the bandwidth may drop to 15 Gbit/s or even lower. The
> servers are idle except for the iperf3 runs. When we bring down one slave on
> server 1, the result is always perfect; no retransmits and good throughput.
> 
> We have captured traffic with tcpdump on server 1 at the slave level (I'll try
> to add the pcap files). To our surprise, we see that the data channel ACK
> packets are sometimes sent over one slave and sometimes over the other. We
> think this causes packet misordering in the network switches, and thus
> retransmits and loss of bandwidth.
> 
> Our understanding of layer2+3 hashing is that for a single connection, all
> traffic should go over the same slave. Therefore, we don't understand why
> server 1 sends ACK packets out over both slaves.
> 
> I've read the documentation at
> https://www.kernel.org/doc/Documentation/networking/bonding.txt but I couldn't
> find the observed behaviour explained there.
> 
> We have tested several Centos 7 and Centos 8 kernels, including recent elrepo
> kernels, but all show this behaviour. We have tried teaming instead of bonding
> but it has the same behaviour. We have tried other hashing algorithms like
> layer3+4 but they seem to have the same issue. It occurs with both IPv4 and
> IPv6.
> 
> Is this behaviour to be expected? If yes, is it documented anywhere? Will it
> degrade throughput in real life traffic (with multiple concurrent data
> streams)?
> If the behaviour is not expected, are we doing something wrong, or might it be
> a bug?
> 
> Thanks,
> Onno
>