All of lore.kernel.org
 help / color / mirror / Atom feed
From: Felix Fietkau <nbd@nbd.name>
To: "Rafał Miłecki" <zajec5@gmail.com>,
	"Toshiaki Makita" <makita.toshiaki@lab.ntt.co.jp>
Cc: Toshiaki Makita <toshiaki.makita1@gmail.com>,
	netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
	Stefano Brivio <sbrivio@redhat.com>,
	Sabrina Dubroca <sd@queasysnail.net>,
	David Ahern <dsahern@gmail.com>, Jo-Philipp Wich <jo@mein.io>,
	Koen Vandeputte <koen.vandeputte@ncentric.com>
Subject: Re: NAT performance regression caused by vlan GRO support
Date: Fri, 5 Apr 2019 09:14:11 +0200	[thread overview]
Message-ID: <45b6fe37-ba1a-91c2-1d4a-2d045793babd@nbd.name> (raw)
In-Reply-To: <daab8b76-edec-d6af-0ef4-91dd4ae2f8e8@gmail.com>

On 2019-04-05 09:11, Rafał Miłecki wrote:
> On 05.04.2019 07:48, Rafał Miłecki wrote:
>> On 05.04.2019 06:26, Toshiaki Makita wrote:
>>> My test results:
>>>
>>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying
>>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13.
>>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive().
>>> Measured TCP throughput by netperf.
>>>
>>> GRO on : 17 Gbps
>>> GRO off:  5 Gbps
>>>
>>> So I failed to reproduce your problem.
>> 
>> :( Thanks for trying & checking that!
>> 
>> 
>>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar
>>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on
>>> your machine?
>> 
>> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s)
>> root@OpenWrt:/# mpstat -P ALL 10 3
>> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU)
>> 
>> 16:33:40     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> 16:33:50     all    0.00    0.00    0.00    0.00    0.00   58.79    0.00    0.00   41.21
>> 16:33:50       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00
>> 16:33:50       1    0.00    0.00    0.00    0.00    0.00   17.58    0.00    0.00   82.42
>> 
>> 16:33:50     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> 16:34:00     all    0.00    0.00    0.05    0.00    0.00   59.44    0.00    0.00   40.51
>> 16:34:00       0    0.00    0.00    0.10    0.00    0.00   99.90    0.00    0.00    0.00
>> 16:34:00       1    0.00    0.00    0.00    0.00    0.00   18.98    0.00    0.00   81.02
>> 
>> 16:34:00     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> 16:34:10     all    0.00    0.00    0.00    0.00    0.00   59.59    0.00    0.00   40.41
>> 16:34:10       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00
>> 16:34:10       1    0.00    0.00    0.00    0.00    0.00   19.18    0.00    0.00   80.82
>> 
>> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> Average:     all    0.00    0.00    0.02    0.00    0.00   59.27    0.00    0.00   40.71
>> Average:       0    0.00    0.00    0.03    0.00    0.00   99.97    0.00    0.00    0.00
>> Average:       1    0.00    0.00    0.00    0.00    0.00   18.58    0.00    0.00   81.42
>> 
>> 
>> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s)
>> root@OpenWrt:/# mpstat -P ALL 10 3
>> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU)
>> 
>> 16:34:39     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> 16:34:49     all    0.00    0.00    0.05    0.00    0.00   86.91    0.00    0.00   13.04
>> 16:34:49       0    0.00    0.00    0.10    0.00    0.00   78.22    0.00    0.00   21.68
>> 16:34:49       1    0.00    0.00    0.00    0.00    0.00   95.60    0.00    0.00    4.40
>> 
>> 16:34:49     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> 16:34:59     all    0.00    0.00    0.10    0.00    0.00   87.06    0.00    0.00   12.84
>> 16:34:59       0    0.00    0.00    0.20    0.00    0.00   79.72    0.00    0.00   20.08
>> 16:34:59       1    0.00    0.00    0.00    0.00    0.00   94.41    0.00    0.00    5.59
>> 
>> 16:34:59     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> 16:35:09     all    0.00    0.00    0.05    0.00    0.00   85.71    0.00    0.00   14.24
>> 16:35:09       0    0.00    0.00    0.10    0.00    0.00   79.42    0.00    0.00   20.48
>> 16:35:09       1    0.00    0.00    0.00    0.00    0.00   92.01    0.00    0.00    7.99
>> 
>> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> Average:     all    0.00    0.00    0.07    0.00    0.00   86.56    0.00    0.00   13.37
>> Average:       0    0.00    0.00    0.13    0.00    0.00   79.12    0.00    0.00   20.75
>> Average:       1    0.00    0.00    0.00    0.00    0.00   94.01    0.00    0.00    5.99
>> 
>> 
>> 3) System idle (no iperf)
>> root@OpenWrt:/# mpstat -P ALL 10 1
>> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU)
>> 
>> 16:35:31     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> 16:35:41     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>> 16:35:41       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>> 16:35:41       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>> 
>> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>> Average:     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>> Average:       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>> Average:       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
>> 
>> 
>>> If CPU is 100%, perf may help us analyze your problem. If it's
>>> available, try running below while testing:
>>> # perf record -a -g -- sleep 5
>>>
>>> And then run this after testing:
>>> # perf report --no-child
>> 
>> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now.
> 
> I guess its GRO + csum_partial() to be blamed for this performance drop.
> 
> Maybe csum_partial() is very fast on your powerful machine and few extra calls
> don't make a difference? I can imagine it affecting much slower home router with
> ARM cores.
Most high performance Ethernet devices implement hardware checksum
offload, which completely gets rid of this overhead.
Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is
why you're getting such crappy performance.

- Felix

  reply	other threads:[~2019-04-05  7:16 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-04 12:57 NAT performance regression caused by vlan GRO support Rafał Miłecki
2019-04-04 15:17 ` Toshiaki Makita
2019-04-04 20:22   ` Rafał Miłecki
2019-04-05  4:26     ` Toshiaki Makita
2019-04-05  5:48       ` Rafał Miłecki
2019-04-05  7:11         ` Rafał Miłecki
2019-04-05  7:14           ` Felix Fietkau [this message]
2019-04-05  7:58             ` Toshiaki Makita
2019-04-05  8:12               ` Rafał Miłecki
2019-04-05  8:24                 ` Rafał Miłecki
2019-04-05 10:18               ` Toke Høiland-Jørgensen
2019-04-05 10:51                 ` Florian Westphal
2019-04-05 11:00                   ` Eric Dumazet
2019-04-07 11:53 ` Rafał Miłecki
2019-04-07 11:54   ` Rafał Miłecki
2019-04-08 13:31     ` David Laight

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=45b6fe37-ba1a-91c2-1d4a-2d045793babd@nbd.name \
    --to=nbd@nbd.name \
    --cc=davem@davemloft.net \
    --cc=dsahern@gmail.com \
    --cc=jo@mein.io \
    --cc=koen.vandeputte@ncentric.com \
    --cc=makita.toshiaki@lab.ntt.co.jp \
    --cc=netdev@vger.kernel.org \
    --cc=sbrivio@redhat.com \
    --cc=sd@queasysnail.net \
    --cc=toshiaki.makita1@gmail.com \
    --cc=zajec5@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.