From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Tom Barbette <barbette@kth.se>
Cc: "Saeed Mahameed" <saeedm@mellanox.com>,
"toke@redhat.com" <toke@redhat.com>,
"xdp-newbies@vger.kernel.org" <xdp-newbies@vger.kernel.org>,
"Leon Romanovsky" <leonro@mellanox.com>,
"Tariq Toukan" <tariqt@mellanox.com>,
brouer@redhat.com, "Björn Töpel" <bjorn.topel@intel.com>,
"Karlsson, Magnus" <magnus.karlsson@intel.com>,
"Jakub Kicinski" <jakub.kicinski@netronome.com>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: Bad XDP performance with mlx5
Date: Tue, 4 Jun 2019 11:15:54 +0200 [thread overview]
Message-ID: <20190604111554.749ddd87@carbon> (raw)
In-Reply-To: <9f116335-0fad-079b-4070-89f24af4ab55@kth.se>
On Tue, 4 Jun 2019 09:28:22 +0200
Tom Barbette <barbette@kth.se> wrote:
> Thanks Jesper for looking into this!
>
> I don't think I will be of much help further on this matter. My take
> out would be: as a first-time user looking into XDP after watching a
> dozen of XDP talks, I would have expected XDP default settings to be
> identical to SKB, so I don't have to watch out for a set of
> per-driver parameter checklist to avoid increasing my CPU consumption
> by 15% when inserting "a super efficient and light BPF program". But
> I understand it's not that easy...
The gap should not be this large, but as I demonstrated it was primarily
because you hit an unfortunate interaction with TCP and how the mlx5
driver does page-caching (p.s. we are working on removing this driver
local recycle-cache).
When loading an XDP/eBPF-prog then the driver change the underlying RX
memory model, which waste memory to gain packets-per-sec speed, but TCP
sees this memory waste and gives us a penalty.
It is important to understand, that XDP is not optimized for TCP. XDP
is designed and optimized for L2-L3 handling of packets (TCP is L4).
Before XDP these L2-L3 use-cases were "slow", because the kernel
netstack assumes a L4/socket use-case (full SKB), when less was really
needed.
This is actually another good example of why XDP programs per RX-queue,
will be useful (notice: which is not implemented upstream, yet...).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
prev parent reply other threads:[~2019-06-04 9:16 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <d7968b89-7218-1e76-86bf-c452b2f8d0c2@kth.se>
[not found] ` <20190529191602.71eb6c87@carbon>
[not found] ` <0836bd30-828a-9126-5d99-1d35b931e3ab@kth.se>
[not found] ` <20190530094053.364b1147@carbon>
[not found] ` <d695d08a-9ee1-0228-2cbb-4b2538a1d2f8@kth.se>
[not found] ` <2218141a-7026-1cb8-c594-37e38eef7b15@kth.se>
2019-05-31 16:18 ` Bad XDP performance with mlx5 Jesper Dangaard Brouer
2019-05-31 18:00 ` David Miller
2019-05-31 18:06 ` Saeed Mahameed
2019-05-31 21:57 ` Jesper Dangaard Brouer
[not found] ` <9f116335-0fad-079b-4070-89f24af4ab55@kth.se>
2019-06-04 9:15 ` Jesper Dangaard Brouer [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190604111554.749ddd87@carbon \
--to=brouer@redhat.com \
--cc=barbette@kth.se \
--cc=bjorn.topel@intel.com \
--cc=jakub.kicinski@netronome.com \
--cc=leonro@mellanox.com \
--cc=magnus.karlsson@intel.com \
--cc=netdev@vger.kernel.org \
--cc=saeedm@mellanox.com \
--cc=tariqt@mellanox.com \
--cc=toke@redhat.com \
--cc=xdp-newbies@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).