Fw: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection

Fw: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection

[Stephen Hemminger]

Begin forwarded message:

Date: Tue, 20 Oct 2020 10:42:34 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection


https://bugzilla.kernel.org/show_bug.cgi?id=209767

            Bug ID: 209767
           Summary: Bonding 802.3ad layer2+3 transmits on both slaves
                    within single connection
           Product: Networking
           Version: 2.5
    Kernel Version: 5.8.11-1.el8.elrepo.x86_64 and
                    3.10.0-1127.19.1.el7.x86_64
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
          Assignee: stephen@networkplumber.org
          Reporter: onno.zweers@surf.nl
        Regression: No

Dear people,

I'm seeing bonding behavior I don't understand and neither do several network
experts I've consulted.

We have two servers, both with two 25 Gbit interfaces in a bond (802.3ad with
layer2+3 hashing). We tuned the systems according to
https://fasterdata.es.net/host-tuning/linux/. I run `iperf3 --server` on server
1 and connect to it with `iperf3 --client server1` from server 2. We notice
that sometimes the connection is good (24.7 Gbit/s, no retransmits) and
sometimes there are many retransmits (sometimes as many as >30,000 in a 10
second run) and then the bandwidth may drop to 15 Gbit/s or even lower. The
servers are idle except for the iperf3 runs. When we bring down one slave on
server 1, the result is always perfect; no retransmits and good throughput.

We have captured traffic with tcpdump on server 1 at the slave level (I'll try
to add the pcap files). To our surprise, we see that the data channel ACK
packets are sometimes sent over one slave and sometimes over the other. We
think this causes packet misordering in the network switches, and thus
retransmits and loss of bandwidth.

Our understanding of layer2+3 hashing is that for a single connection, all
traffic should go over the same slave. Therefore, we don't understand why
server 1 sends ACK packets out over both slaves.

I've read the documentation at
https://www.kernel.org/doc/Documentation/networking/bonding.txt but I couldn't
find the observed behaviour explained there.

We have tested several Centos 7 and Centos 8 kernels, including recent elrepo
kernels, but all show this behaviour. We have tried teaming instead of bonding
but it has the same behaviour. We have tried other hashing algorithms like
layer3+4 but they seem to have the same issue. It occurs with both IPv4 and
IPv6.

Is this behaviour to be expected? If yes, is it documented anywhere? Will it
degrade throughput in real life traffic (with multiple concurrent data
streams)?
If the behaviour is not expected, are we doing something wrong, or might it be
a bug?

Thanks,
Onno

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection

[Jakub Kicinski]

CC some bonding folks

On Tue, 20 Oct 2020 07:54:29 -0700 Stephen Hemminger wrote:
> Begin forwarded message:
> 
> Date: Tue, 20 Oct 2020 10:42:34 +0000
> From: bugzilla-daemon@bugzilla.kernel.org
> To: stephen@networkplumber.org
> Subject: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection
> 
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=209767
> 
>             Bug ID: 209767
>            Summary: Bonding 802.3ad layer2+3 transmits on both slaves
>                     within single connection
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 5.8.11-1.el8.elrepo.x86_64 and
>                     3.10.0-1127.19.1.el7.x86_64
>           Hardware: All
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>           Assignee: stephen@networkplumber.org
>           Reporter: onno.zweers@surf.nl
>         Regression: No
> 
> Dear people,
> 
> I'm seeing bonding behavior I don't understand and neither do several network
> experts I've consulted.
> 
> We have two servers, both with two 25 Gbit interfaces in a bond (802.3ad with
> layer2+3 hashing). We tuned the systems according to
> https://fasterdata.es.net/host-tuning/linux/. I run `iperf3 --server` on server
> 1 and connect to it with `iperf3 --client server1` from server 2. We notice
> that sometimes the connection is good (24.7 Gbit/s, no retransmits) and
> sometimes there are many retransmits (sometimes as many as >30,000 in a 10
> second run) and then the bandwidth may drop to 15 Gbit/s or even lower. The
> servers are idle except for the iperf3 runs. When we bring down one slave on
> server 1, the result is always perfect; no retransmits and good throughput.
> 
> We have captured traffic with tcpdump on server 1 at the slave level (I'll try
> to add the pcap files). To our surprise, we see that the data channel ACK
> packets are sometimes sent over one slave and sometimes over the other. We
> think this causes packet misordering in the network switches, and thus
> retransmits and loss of bandwidth.
> 
> Our understanding of layer2+3 hashing is that for a single connection, all
> traffic should go over the same slave. Therefore, we don't understand why
> server 1 sends ACK packets out over both slaves.
> 
> I've read the documentation at
> https://www.kernel.org/doc/Documentation/networking/bonding.txt but I couldn't
> find the observed behaviour explained there.
> 
> We have tested several Centos 7 and Centos 8 kernels, including recent elrepo
> kernels, but all show this behaviour. We have tried teaming instead of bonding
> but it has the same behaviour. We have tried other hashing algorithms like
> layer3+4 but they seem to have the same issue. It occurs with both IPv4 and
> IPv6.
> 
> Is this behaviour to be expected? If yes, is it documented anywhere? Will it
> degrade throughput in real life traffic (with multiple concurrent data
> streams)?
> If the behaviour is not expected, are we doing something wrong, or might it be
> a bug?
> 
> Thanks,
> Onno
>