From mboxrd@z Thu Jan 1 00:00:00 1970 From: William Tu Subject: Re: [PATCH v2 0/5] Introducing ixgbe AF_XDP ZC support Date: Tue, 2 Oct 2018 11:23:54 -0700 Message-ID: References: <20181002080034.11754-1-bjorn.topel@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Cc: jeffrey.t.kirsher@intel.com, intel-wired-lan@lists.osuosl.org, =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= , "Karlsson, Magnus" , Magnus Karlsson , Alexei Starovoitov , Daniel Borkmann , Linux Kernel Network Developers , Jesper Dangaard Brouer , Test , Jakub Kicinski To: =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= Return-path: Received: from mail-qt1-f194.google.com ([209.85.160.194]:36323 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725951AbeJCBJN (ORCPT ); Tue, 2 Oct 2018 21:09:13 -0400 Received: by mail-qt1-f194.google.com with SMTP id u34-v6so3062293qth.3 for ; Tue, 02 Oct 2018 11:24:31 -0700 (PDT) In-Reply-To: <20181002080034.11754-1-bjorn.topel@gmail.com> Sender: netdev-owner@vger.kernel.org List-ID: On Tue, Oct 2, 2018 at 1:01 AM Bj=C3=B6rn T=C3=B6pel wrote: > > From: Bj=C3=B6rn T=C3=B6pel > > Jeff: Please remove the v1 patches from your dev-queue! > > This patch set introduces zero-copy AF_XDP support for Intel's ixgbe > driver. > > The ixgbe zero-copy code is located in its own file ixgbe_xsk.[ch], > analogous to the i40e ZC support. Again, as in i40e, code paths have > been copied from the XDP path to the zero-copy path. Going forward we > will try to generalize more code between the AF_XDP ZC drivers, and > also reduce the heavy C&P. > > We have run some benchmarks on a dual socket system with two Broadwell > E5 2660 @ 2.0 GHz with hyperthreading turned off. Each socket has 14 > cores which gives a total of 28, but only two cores are used in these > experiments. One for TR/RX and one for the user space application. The > memory is DDR4 @ 2133 MT/s (1067 MHz) and the size of each DIMM is > 8192MB and with 8 of those DIMMs in the system we have 64 GB of total > memory. The compiler used is GCC 7.3.0. The NIC is Intel > 82599ES/X520-2 10Gbit/s using the ixgbe driver. > > Below are the results in Mpps of the 82599ES/X520-2 NIC benchmark runs > for 64B and 1500B packets, generated by a commercial packet generator > HW blasting packets at full 10Gbit/s line rate. The results are with > retpoline and all other spectre and meltdown fixes. > > AF_XDP performance 64B packets: > Benchmark XDP_DRV with zerocopy > rxdrop 14.7 > txpush 14.6 > l2fwd 11.1 > > AF_XDP performance 1500B packets: > Benchmark XDP_DRV with zerocopy > rxdrop 0.8 > l2fwd 0.8 > > XDP performance on our system as a base line. > > 64B packets: > XDP stats CPU Mpps issue-pps > XDP-RX CPU 16 14.7 0 > > 1500B packets: > XDP stats CPU Mpps issue-pps > XDP-RX CPU 16 0.8 0 > > The structure of the patch set is as follows: > > Patch 1: Introduce Rx/Tx ring enable/disable functionality > Patch 2: Preparatory patche to ixgbe driver code for RX > Patch 3: ixgbe zero-copy support for RX > Patch 4: Preparatory patch to ixgbe driver code for TX > Patch 5: ixgbe zero-copy support for TX > > Changes since v1: > > * Removed redundant AF_XDP precondition checks, pointed out by > Jakub. Now, the preconditions are only checked at XDP enable time. > * Fixed a crash in the egress path, due to incorrect usage of > ixgbe_ring queue_index member. In v2 a ring_idx back reference is > introduced, and used in favor of queue_index. William reported the > crash, and helped me smoke out the issue. Kudos! Thanks! I tested this series and no more crash. The number is pretty good (*without* spectre and meltdown fixes) model name : Intel(R) Xeon(R) CPU E5-2440 v2 @ 1.90GHz, total 16 cores/ AF_XDP performance 64B packets: Benchmark XDP_DRV with zerocopy rxdrop 20 txpush 18 l2fwd 20 Regards, William From mboxrd@z Thu Jan 1 00:00:00 1970 From: William Tu Date: Tue, 2 Oct 2018 11:23:54 -0700 Subject: [Intel-wired-lan] [PATCH v2 0/5] Introducing ixgbe AF_XDP ZC support In-Reply-To: <20181002080034.11754-1-bjorn.topel@gmail.com> References: <20181002080034.11754-1-bjorn.topel@gmail.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Tue, Oct 2, 2018 at 1:01 AM Bj?rn T?pel wrote: > > From: Bj?rn T?pel > > Jeff: Please remove the v1 patches from your dev-queue! > > This patch set introduces zero-copy AF_XDP support for Intel's ixgbe > driver. > > The ixgbe zero-copy code is located in its own file ixgbe_xsk.[ch], > analogous to the i40e ZC support. Again, as in i40e, code paths have > been copied from the XDP path to the zero-copy path. Going forward we > will try to generalize more code between the AF_XDP ZC drivers, and > also reduce the heavy C&P. > > We have run some benchmarks on a dual socket system with two Broadwell > E5 2660 @ 2.0 GHz with hyperthreading turned off. Each socket has 14 > cores which gives a total of 28, but only two cores are used in these > experiments. One for TR/RX and one for the user space application. The > memory is DDR4 @ 2133 MT/s (1067 MHz) and the size of each DIMM is > 8192MB and with 8 of those DIMMs in the system we have 64 GB of total > memory. The compiler used is GCC 7.3.0. The NIC is Intel > 82599ES/X520-2 10Gbit/s using the ixgbe driver. > > Below are the results in Mpps of the 82599ES/X520-2 NIC benchmark runs > for 64B and 1500B packets, generated by a commercial packet generator > HW blasting packets at full 10Gbit/s line rate. The results are with > retpoline and all other spectre and meltdown fixes. > > AF_XDP performance 64B packets: > Benchmark XDP_DRV with zerocopy > rxdrop 14.7 > txpush 14.6 > l2fwd 11.1 > > AF_XDP performance 1500B packets: > Benchmark XDP_DRV with zerocopy > rxdrop 0.8 > l2fwd 0.8 > > XDP performance on our system as a base line. > > 64B packets: > XDP stats CPU Mpps issue-pps > XDP-RX CPU 16 14.7 0 > > 1500B packets: > XDP stats CPU Mpps issue-pps > XDP-RX CPU 16 0.8 0 > > The structure of the patch set is as follows: > > Patch 1: Introduce Rx/Tx ring enable/disable functionality > Patch 2: Preparatory patche to ixgbe driver code for RX > Patch 3: ixgbe zero-copy support for RX > Patch 4: Preparatory patch to ixgbe driver code for TX > Patch 5: ixgbe zero-copy support for TX > > Changes since v1: > > * Removed redundant AF_XDP precondition checks, pointed out by > Jakub. Now, the preconditions are only checked at XDP enable time. > * Fixed a crash in the egress path, due to incorrect usage of > ixgbe_ring queue_index member. In v2 a ring_idx back reference is > introduced, and used in favor of queue_index. William reported the > crash, and helped me smoke out the issue. Kudos! Thanks! I tested this series and no more crash. The number is pretty good (*without* spectre and meltdown fixes) model name : Intel(R) Xeon(R) CPU E5-2440 v2 @ 1.90GHz, total 16 cores/ AF_XDP performance 64B packets: Benchmark XDP_DRV with zerocopy rxdrop 20 txpush 18 l2fwd 20 Regards, William