From: Alexander Duyck <alexander.duyck@gmail.com>
To: William Kucharski <william.kucharski@oracle.com>
Cc: LKML <linux-kernel@vger.kernel.org>,
Netdev <netdev@vger.kernel.org>,
intel-wired-lan <intel-wired-lan@lists.osuosl.org>,
Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
"Duyck, Alexander H" <alexander.h.duyck@intel.com>
Subject: Re: Regression: Approximate 34% performance hit in receive throughput over ixgbe seen due to build_skb patch
Date: Tue, 22 May 2018 11:23:58 -0700 [thread overview]
Message-ID: <CAKgT0Ud2qriAkWcs56WEkw2vhYzBbWvzZOoY24ndsqPKaWn=Sg@mail.gmail.com> (raw)
In-Reply-To: <A940030A-CDBF-41CD-816F-0BBC76E7C4F4@oracle.com>
On Tue, May 22, 2018 at 11:00 AM, William Kucharski
<william.kucharski@oracle.com> wrote:
> A performance hit of approximately 34% in receive numbers for some packet sizes is
> seen when testing traffic over ixgbe links using the network test netperf.
>
> Starting with the top of tree commit 7addb3e4ad3db6a95a953c59884921b5883dcdec,
> a git bisect narrowed the issue down to:
>
> commit 6f429223b31c550b835b4f066ac034d0cf0cc71e
>
> ixgbe: Add support for build_skb
>
> This patch adds build_skb support to the Rx path. There are several
> advantages to this change.
>
> 1. It avoids the memcpy and skb->head allocation for small packets which
> improves performance by about 5% in my tests.
> 2. It avoids the memcpy, skb->head allocation, and eth_get_headlen
> for larger packets improving performance by about 10% in my tests.
> 3. For VXLAN packets it allows the full header to be in skb->data which
> improves the performance by as much as 30% in some of my tests.
>
> Netperf was sourced from:
>
> https://hewlettpackard.github.io/netperf/
>
> Two machines were directly connected via ixgbe links.
>
> The process "netserver" was started on 10.196.11.8, and running this test:
>
> # netperf -l 60 -H 10.196.11.8 -i 10,2 -I 99,10 -t UDP_STREAM -- -m 64 -s 32768 -S 32768
Okay, so I can already see what the most likely issue is. The
build_skb code is more CPU efficient, but it will consume more memory
in the process since it is avoiding the memcpy and is instead using a
full 2K block of memory for a small frame. I'm suspecting any
performance issue you are seeing may be due to a slow interrupt rate
causing us to either exhaust available Tx memory, or overrun the
available Rx memory.
There end up being multiple ways to address this.
1. Use a larger value for your "-s/-S" values to allow for more memory
to be handled in the queues.
2. Update the interrupt moderation code for the driver. You can either
manually decrease the per-interrupt delay via "ethtool -C" or just
update the adaptive ITR code, see commit b4ded8327fea ("ixgbe: Update
adaptive ITR algorithm").
3. There should be a private flag that can be updated via "ethtool
--set-priv-flags" called "legacy-rx" that you can enable that will
roll back to the original that did the copy-break type approach for
small packets and the headers of the frame.
Thanks.
- Alex
next prev parent reply other threads:[~2018-05-22 18:24 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-05-22 18:00 Regression: Approximate 34% performance hit in receive throughput over ixgbe seen due to build_skb patch William Kucharski
2018-05-22 18:23 ` Alexander Duyck [this message]
2018-05-22 19:29 ` William Kucharski
2018-05-22 20:03 ` Alexander Duyck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAKgT0Ud2qriAkWcs56WEkw2vhYzBbWvzZOoY24ndsqPKaWn=Sg@mail.gmail.com' \
--to=alexander.duyck@gmail.com \
--cc=alexander.h.duyck@intel.com \
--cc=intel-wired-lan@lists.osuosl.org \
--cc=jeffrey.t.kirsher@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=william.kucharski@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).