All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Toke Høiland-Jørgensen" <toke@redhat.com>
To: "Toshiaki Makita" <makita.toshiaki@lab.ntt.co.jp>,
	"Felix Fietkau" <nbd@nbd.name>,
	"Rafał Miłecki" <zajec5@gmail.com>
Cc: Toshiaki Makita <toshiaki.makita1@gmail.com>,
	netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
	Stefano Brivio <sbrivio@redhat.com>,
	Sabrina Dubroca <sd@queasysnail.net>,
	David Ahern <dsahern@gmail.com>, Jo-Philipp Wich <jo@mein.io>,
	Koen Vandeputte <koen.vandeputte@ncentric.com>
Subject: Re: NAT performance regression caused by vlan GRO support
Date: Fri, 05 Apr 2019 12:18:23 +0200	[thread overview]
Message-ID: <87mul421sg.fsf@toke.dk> (raw)
In-Reply-To: <67d634cd-cf16-df21-7b8a-5d865d95e4e6@lab.ntt.co.jp>

Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp> writes:

> On 2019/04/05 16:14, Felix Fietkau wrote:
>> On 2019-04-05 09:11, Rafał Miłecki wrote:
>>> On 05.04.2019 07:48, Rafał Miłecki wrote:
>>>> On 05.04.2019 06:26, Toshiaki Makita wrote:
>>>>> My test results:
>>>>>
>>>>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying
>>>>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13.
>>>>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive().
>>>>> Measured TCP throughput by netperf.
>>>>>
>>>>> GRO on : 17 Gbps
>>>>> GRO off:  5 Gbps
>>>>>
>>>>> So I failed to reproduce your problem.
>>>>
>>>> :( Thanks for trying & checking that!
>>>>
>>>>
>>>>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar
>>>>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on
>>>>> your machine?
>>>>
>>>> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s)
>>>> root@OpenWrt:/# mpstat -P ALL 10 3
>>>> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU)
>>>>
>>>> 16:33:40     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 16:33:50     all    0.00    0.00    0.00    0.00    0.00   58.79    0.00    0.00   41.21
>>>> 16:33:50       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00
>>>> 16:33:50       1    0.00    0.00    0.00    0.00    0.00   17.58    0.00    0.00   82.42
>>>>
>>>> 16:33:50     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 16:34:00     all    0.00    0.00    0.05    0.00    0.00   59.44    0.00    0.00   40.51
>>>> 16:34:00       0    0.00    0.00    0.10    0.00    0.00   99.90    0.00    0.00    0.00
>>>> 16:34:00       1    0.00    0.00    0.00    0.00    0.00   18.98    0.00    0.00   81.02
>>>>
>>>> 16:34:00     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 16:34:10     all    0.00    0.00    0.00    0.00    0.00   59.59    0.00    0.00   40.41
>>>> 16:34:10       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00
>>>> 16:34:10       1    0.00    0.00    0.00    0.00    0.00   19.18    0.00    0.00   80.82
>>>>
>>>> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> Average:     all    0.00    0.00    0.02    0.00    0.00   59.27    0.00    0.00   40.71
>>>> Average:       0    0.00    0.00    0.03    0.00    0.00   99.97    0.00    0.00    0.00
>>>> Average:       1    0.00    0.00    0.00    0.00    0.00   18.58    0.00    0.00   81.42
>>>>
>>>>
>>>> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s)
>>>> root@OpenWrt:/# mpstat -P ALL 10 3
>>>> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU)
>>>>
>>>> 16:34:39     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 16:34:49     all    0.00    0.00    0.05    0.00    0.00   86.91    0.00    0.00   13.04
>>>> 16:34:49       0    0.00    0.00    0.10    0.00    0.00   78.22    0.00    0.00   21.68
>>>> 16:34:49       1    0.00    0.00    0.00    0.00    0.00   95.60    0.00    0.00    4.40
>>>>
>>>> 16:34:49     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 16:34:59     all    0.00    0.00    0.10    0.00    0.00   87.06    0.00    0.00   12.84
>>>> 16:34:59       0    0.00    0.00    0.20    0.00    0.00   79.72    0.00    0.00   20.08
>>>> 16:34:59       1    0.00    0.00    0.00    0.00    0.00   94.41    0.00    0.00    5.59
>>>>
>>>> 16:34:59     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 16:35:09     all    0.00    0.00    0.05    0.00    0.00   85.71    0.00    0.00   14.24
>>>> 16:35:09       0    0.00    0.00    0.10    0.00    0.00   79.42    0.00    0.00   20.48
>>>> 16:35:09       1    0.00    0.00    0.00    0.00    0.00   92.01    0.00    0.00    7.99
>>>>
>>>> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> Average:     all    0.00    0.00    0.07    0.00    0.00   86.56    0.00    0.00   13.37
>>>> Average:       0    0.00    0.00    0.13    0.00    0.00   79.12    0.00    0.00   20.75
>>>> Average:       1    0.00    0.00    0.00    0.00    0.00   94.01    0.00    0.00    5.99
>>>>
>>>>
>>>> 3) System idle (no iperf)
>>>> root@OpenWrt:/# mpstat -P ALL 10 1
>>>> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU)
>>>>
>>>> 16:35:31     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 16:35:41     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>>>> 16:35:41       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>>>> 16:35:41       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>>>>
>>>> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> Average:     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>>>> Average:       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>>>> Average:       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>>>>
>>>>
>>>>> If CPU is 100%, perf may help us analyze your problem. If it's
>>>>> available, try running below while testing:
>>>>> # perf record -a -g -- sleep 5
>>>>>
>>>>> And then run this after testing:
>>>>> # perf report --no-child
>>>>
>>>> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now.
>>>
>>> I guess its GRO + csum_partial() to be blamed for this performance drop.
>>>
>>> Maybe csum_partial() is very fast on your powerful machine and few extra calls
>>> don't make a difference? I can imagine it affecting much slower home router with
>>> ARM cores.
>> Most high performance Ethernet devices implement hardware checksum
>> offload, which completely gets rid of this overhead.
>> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is
>> why you're getting such crappy performance.
>
> Hmm... now I disabled rx checksum and tried the test again, and indeed I
> see csum_partial from GRO path. But I also see csum_partial even without
> GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete.
> Probably Rafał disabled nf_conntrack_checksum sysctl knob?
>
> But anyway even with disabling rx csum offload my machine has better
> performance with GRO.

But you're also running at way higher speeds, where the benefit of GRO
is higher.

> I'm sure in some cases GRO should be disabled, but I guess it's
> difficult to determine whether we should disable GRO or not
> automatically when csum offload is not available.

As a first approximation, maybe just:

if (!has_hardware_cksum_offload(netdev) && link_rate(netdev) <= 1Gbps)
  disable_gro();

We used 1Gbps as the threshold for when to split GRO packets by default
in sck_cake as well...

-Toke

  parent reply	other threads:[~2019-04-05 10:18 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-04 12:57 NAT performance regression caused by vlan GRO support Rafał Miłecki
2019-04-04 15:17 ` Toshiaki Makita
2019-04-04 20:22   ` Rafał Miłecki
2019-04-05  4:26     ` Toshiaki Makita
2019-04-05  5:48       ` Rafał Miłecki
2019-04-05  7:11         ` Rafał Miłecki
2019-04-05  7:14           ` Felix Fietkau
2019-04-05  7:58             ` Toshiaki Makita
2019-04-05  8:12               ` Rafał Miłecki
2019-04-05  8:24                 ` Rafał Miłecki
2019-04-05 10:18               ` Toke Høiland-Jørgensen [this message]
2019-04-05 10:51                 ` Florian Westphal
2019-04-05 11:00                   ` Eric Dumazet
2019-04-07 11:53 ` Rafał Miłecki
2019-04-07 11:54   ` Rafał Miłecki
2019-04-08 13:31     ` David Laight

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87mul421sg.fsf@toke.dk \
    --to=toke@redhat.com \
    --cc=davem@davemloft.net \
    --cc=dsahern@gmail.com \
    --cc=jo@mein.io \
    --cc=koen.vandeputte@ncentric.com \
    --cc=makita.toshiaki@lab.ntt.co.jp \
    --cc=nbd@nbd.name \
    --cc=netdev@vger.kernel.org \
    --cc=sbrivio@redhat.com \
    --cc=sd@queasysnail.net \
    --cc=toshiaki.makita1@gmail.com \
    --cc=zajec5@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.