From: Zoltan Kiss <zoltan.kiss@citrix.com>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: <wei.liu2@citrix.com>, <xen-devel@lists.xenproject.org>,
<netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
<jonathan.davies@citrix.com>,
Marcus Granado <Marcus.Granado@eu.citrix.com>
Subject: Re: [PATCH net-next v7 0/9] xen-netback: TX grant mapping with SKBTX_DEV_ZEROCOPY instead of copy
Date: Thu, 13 Mar 2014 18:23:04 +0000 [thread overview]
Message-ID: <5321F788.2010402@citrix.com> (raw)
In-Reply-To: <1394705313.25873.13.camel@kazak.uk.xensource.com>
On 13/03/14 10:08, Ian Campbell wrote:
> On Thu, 2014-03-06 at 21:48 +0000, Zoltan Kiss wrote:
>> >A long known problem of the upstream netback implementation that on the TX
>> >path (from guest to Dom0) it copies the whole packet from guest memory into
>> >Dom0. That simply became a bottleneck with 10Gb NICs, and generally it's a
>> >huge perfomance penalty. The classic kernel version of netback used grant
>> >mapping, and to get notified when the page can be unmapped, it used page
>> >destructors. Unfortunately that destructor is not an upstreamable solution.
>> >Ian Campbell's skb fragment destructor patch series [1] tried to solve this
>> >problem, however it seems to be very invasive on the network stack's code,
>> >and therefore haven't progressed very well.
>> >This patch series use SKBTX_DEV_ZEROCOPY flags to tell the stack it needs to
>> >know when the skb is freed up. That is the way KVM solved the same problem,
>> >and based on my initial tests it can do the same for us. Avoiding the extra
>> >copy boosted up TX throughput from 6.8 Gbps to 7.9 (I used a slower AMD
>> >Interlagos box, both Dom0 and guest on upstream kernel, on the same NUMA node,
>> >running iperf 2.0.5, and the remote end was a bare metal box on the same 10Gb
>> >switch)
> Do you have any other numbers? e.g. for a modern Intel or AMD system? A
> slower box is likely to make the difference between copy and map larger,
> whereas modern Intel for example is supposed to be very good at copying.
Performance team made a lot of measurements, I've added Marcus to
comment on that.
With the latest version and tip net-next kernel I could see even ~9.3
Gbps peak throughput on the same AMD box, which is the practical maximum
for 10G cards. However with older guests I couldn't reach that. A lot
depends on netfront and TCP stack, e.g. the tcp_limit_output_bytes
sysctl can cause an artificial cap.
Perf team now has 40 Gbps NICs I guess, it would be interesting to see
how does this perform there.
I just checked the intrahost guest-to-guest throughput with 2 upstream
kernel, I could get out 5.6-5.8 Gbps at most.
>
>> >Based on my investigations the packet get only copied if it is delivered to
>> >Dom0 IP stack through deliver_skb, which is due to this [2] patch. This affects
>> >DomU->Dom0 IP traffic and when Dom0 does routing/NAT for the guest. That's a bit
>> >unfortunate, but luckily it doesn't cause a major regression for this usecase.
> Numbers?
I've checked that back in November:
https://lkml.org/lkml/2013/11/5/288
Originally it was 5.4 vs with my patch it was 5.2. I've checked DomU to
Dom0 iperf again, about the same still with my series.
Zoli
prev parent reply other threads:[~2014-03-13 18:23 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-06 21:48 [PATCH net-next v7 0/9] xen-netback: TX grant mapping with SKBTX_DEV_ZEROCOPY instead of copy Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 1/9] xen-netback: Use skb->cb for pending_idx Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 2/9] xen-netback: Minor refactoring of netback code Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 3/9] xen-netback: Handle foreign mapped pages on the guest RX path Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 4/9] xen-netback: Introduce TX grant mapping Zoltan Kiss
2014-03-13 10:17 ` Ian Campbell
2014-03-13 12:34 ` Zoltan Kiss
2014-03-13 10:33 ` Ian Campbell
2014-03-13 10:56 ` [Xen-devel] " David Vrabel
2014-03-13 11:02 ` Ian Campbell
2014-03-13 11:09 ` David Vrabel
2014-03-13 11:13 ` Wei Liu
2014-03-13 13:17 ` Zoltan Kiss
2014-03-13 13:56 ` Ian Campbell
2014-03-13 17:43 ` Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 5/9] xen-netback: Remove old TX grant copy definitons and fix indentations Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 6/9] xen-netback: Add stat counters for zerocopy Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 7/9] xen-netback: Handle guests with too many frags Zoltan Kiss
2014-03-06 21:48 ` [PATCH net-next v7 8/9] xen-netback: Timeout packets in RX path Zoltan Kiss
2014-03-13 10:39 ` Ian Campbell
2014-03-06 21:48 ` [PATCH net-next v7 9/9] xen-netback: Aggregate TX unmap operations Zoltan Kiss
2014-03-19 21:16 ` Zoltan Kiss
2014-03-20 9:53 ` Paul Durrant
2014-03-20 10:48 ` Wei Liu
2014-03-20 11:14 ` Paul Durrant
2014-03-20 12:38 ` Wei Liu
2014-03-20 16:11 ` Zoltan Kiss
2014-03-07 21:05 ` [PATCH net-next v7 0/9] xen-netback: TX grant mapping with SKBTX_DEV_ZEROCOPY instead of copy David Miller
2014-03-08 14:37 ` Zoltan Kiss
2014-03-08 23:57 ` David Miller
2014-03-10 10:15 ` Wei Liu
2014-03-12 15:40 ` Ian Campbell
2014-03-12 18:49 ` Zoltan Kiss
2014-03-13 10:43 ` Ian Campbell
2014-03-13 10:08 ` Ian Campbell
2014-03-13 18:23 ` Zoltan Kiss [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5321F788.2010402@citrix.com \
--to=zoltan.kiss@citrix.com \
--cc=Ian.Campbell@citrix.com \
--cc=Marcus.Granado@eu.citrix.com \
--cc=jonathan.davies@citrix.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=wei.liu2@citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).