Re: [PATCH v4 net-next 00/19] inet: frags: bring rhashtables to IP defrag

From: Eric Dumazet <edumazet@google.com>
To: moshe@mellanox.com
Cc: Tariq Toukan <tariqt@mellanox.com>,
	Eric Dumazet <eric.dumazet@gmail.com>,
	aring@mojatatu.com, David Miller <davem@davemloft.net>,
	netdev <netdev@vger.kernel.org>, Florian Westphal <fw@strlen.de>,
	Herbert Xu <herbert@gondor.apana.org.au>,
	Thomas Graf <tgraf@suug.ch>,
	Jesper Dangaard Brouer <brouer@redhat.com>,
	Alexander Aring <alex.aring@gmail.com>,
	Stefan Schmidt <stefan@osg.samsung.com>,
	Kirill Tkhai <ktkhai@virtuozzo.com>,
	Eran Ben Elisha <eranbe@mellanox.com>
Subject: Re: [PATCH v4 net-next 00/19] inet: frags: bring rhashtables to IP defrag
Date: Thu, 31 May 2018 10:05:10 -0400	[thread overview]
Message-ID: <CANn89iJh4_urk440UAWYxr0RvVLjBJ+d4B-hDovSJ+d_0oWfMw@mail.gmail.com> (raw)
In-Reply-To: <0bb2faec-ba8d-51d7-0c1a-12520b9168e8@mellanox.com>

On Thu, May 31, 2018 at 8:18 AM Moshe Shemesh <moshe@mellanox.com> wrote:
>
>
>
> I do see big improvement after changing the 3 parameters as Eric suggested:
> /proc/sys/net/ipv6/ip6frag_time  set to 2
> /proc/sys/net/ipv6/ip6frag_low_thresh set to 104857600
> /proc/sys/net/ipv6/ip6frag_high_thresh set to 78643200
>
>
> [root@reg-l-vrt-67100-104 linux-stable]#  netperf -H
> fe80::7efe:90ff:fed5:bb48%ens9,inet6 -t udp_stream --
> MIGRATED UDP STREAM TEST from ::0 (::) port 0 AF_INET6 to
> fe80::7efe:90ff:fed5:bb48%ens9 () port 0 AF_INET6
> Socket  Message  Elapsed      Messages
> Size    Size     Time         Okay Errors   Throughput
> bytes   bytes    secs            #      #   10^6bits/sec
>
> 212992   65507   10.00      156387      0    8194.60
> 212992           10.00       76901           4029.57
>
> #kernel
> Ip6InReceives                   7107999            0.0
> Ip6InDelivers                   114126             0.0
> Ip6OutRequests                  47                 0.0
> Ip6ReasmTimeout                 5115               0.0
> Ip6ReasmReqds                   7107987            0.0
> Ip6ReasmOKs                     114114             0.0
> Ip6ReasmFails                   1714146            0.0
> ...
> Udp6InDatagrams                 112486             0.0
> Udp6InErrors                    1629               0.0
> Udp6RcvbufErrors                1629               0.0
> ...
>
> While before these parameters settings I got:
> [root@reg-l-vrt-67100-104 ~]# netperf -H
> fe80::e61d:2dff:feca:c7c3%ens9,inet6 -t udp_stream --
> MIGRATED UDP STREAM TEST from ::0 (::) port 0 AF_INET6 to
> fe80::e61d:2dff:feca:c7c3%ens9 () port 0 AF_INET6
> Socket  Message  Elapsed      Messages
> Size    Size     Time         Okay Errors   Throughput
> bytes   bytes    secs            #      #   10^6bits/sec
>
> 212992   65507   10.00      145419      0    7620.35
> 212992           10.00         285             14.93
>
> #kernel
> Ip6InReceives                   6665965            0.0
> Ip6InDelivers                   300                0.0
> Ip6OutRequests                  9                  0.0
> Ip6ReasmReqds                   6665950            0.0
> Ip6ReasmOKs                     285                0.0
> Ip6ReasmFails                   6650890            0.0
> ...
> Udp6InDatagrams                 286                0.0
>
>
> however, before the patchset, I got much better results:
> [root@reg-l-vrt-67100-104 linux-stable]#  netperf -H
> fe80::7efe:90ff:fed5:bb48%ens9,inet6 -t udp_stream --
> MIGRATED UDP STREAM TEST from ::0 (::) port 0 AF_INET6 to
> fe80::7efe:90ff:fed5:bb48%ens9 () port 0 AF_INET6
> Socket  Message  Elapsed      Messages
> Size    Size     Time         Okay Errors   Throughput
> bytes   bytes    secs            #      #   10^6bits/sec
>
> 212992   65507   10.00      158935      0    8328.32
> 212992           10.00      144652           7579.88
>
>
> #kernel
> Ip6InReceives                   7088903            0.0
> Ip6InDelivers                   154117             0.0
> Ip6OutRequests                  9                  0.0
> Ip6ReasmReqds                   7088889            0.0
> Ip6ReasmOKs                     154103             0.0
> ...
> Udp6InDatagrams                 144653             0.0
> Udp6InErrors                    9451               0.0
> Udp6RcvbufErrors                9451               0.0
>
>

Hi Moshe

Your environment seems to be very lossy.

Frags and packets losses do not mix well, there is nothing really
magic we can do with that,
unless we can reserve GB of memory for frags.

Which is exactly was the reason for my patches in the first place.

Thanks.