Re: [PATCH 0/5] Support TCP/IPv4, VxLAN and GRE GSO in DPDK

["Kavanagh, Mark B" <mark.b.kavanagh@intel.com>]

>From: Ananyev, Konstantin
>Sent: Wednesday, August 30, 2017 11:49 AM
>To: Hu, Jiayu <jiayu.hu@intel.com>
>Cc: dev@dpdk.org; Kavanagh, Mark B <mark.b.kavanagh@intel.com>; Tan, Jianfeng
><jianfeng.tan@intel.com>
>Subject: RE: [PATCH 0/5] Support TCP/IPv4, VxLAN and GRE GSO in DPDK
>
>
>
>> -----Original Message-----
>> From: Hu, Jiayu
>> Sent: Wednesday, August 30, 2017 8:37 AM
>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>
>> Cc: dev@dpdk.org; Kavanagh, Mark B <mark.b.kavanagh@intel.com>; Tan,
>Jianfeng <jianfeng.tan@intel.com>
>> Subject: Re: [PATCH 0/5] Support TCP/IPv4, VxLAN and GRE GSO in DPDK
>>
>> Hi Konstantin,
>>
>> Thanks for your suggestions. Feedbacks are inline.
>>
>> Thanks,
>> Jiayu
>>
>> On Wed, Aug 30, 2017 at 09:37:42AM +0800, Ananyev, Konstantin wrote:
>> >
>> > Hi Jiayu,
>> > Few questions/comments from me below in in next few mails.
>> > Thanks
>> > Konstantin
>> >
>> > >
>> > > Generic Segmentation Offload (GSO) is a SW technique to split large
>> > > packets into small ones. Akin to TSO, GSO enables applications to
>> > > operate on large packets, thus reducing per-packet processing overhead.
>> > >
>> > > To enable more flexibility to applications, DPDK GSO is implemented
>> > > as a standalone library. Applications explicitly use the GSO library
>> > > to segment packets. This patch adds GSO support to DPDK for specific
>> > > packet types: specifically, TCP/IPv4, VxLAN, and GRE.
>> > >
>> > > The first patch introduces the GSO API framework. The second patch
>> > > adds GSO support for TCP/IPv4 packets (containing an optional VLAN
>> > > tag). The third patch adds GSO support for VxLAN packets that contain
>> > > outer IPv4, and inner TCP/IPv4 headers (plus optional inner and/or
>> > > outer VLAN tags). The fourth patch adds GSO support for GRE packets
>> > > that contain outer IPv4, and inner TCP/IPv4 headers (with optional
>> > > outer VLAN tag). The last patch in the series enables TCP/IPv4, VxLAN,
>> > > and GRE GSO in testpmd's checksum forwarding engine.
>> > >
>> > > The performance of TCP/IPv4 GSO on a 10Gbps link is demonstrated using
>> > > iperf. Setup for the test is described as follows:
>> > >
>> > > a. Connect 2 x 10Gbps physical ports (P0, P1), together physically.
>> > > b. Launch testpmd with P0 and a vhost-user port, and use csum
>> > >    forwarding engine.
>> > > c. Select IP and TCP HW checksum calculation for P0; select TCP HW
>> > >    checksum calculation for vhost-user port.
>> > > d. Launch a VM with csum and tso offloading enabled.
>> > > e. Run iperf-client on virtio-net port in the VM to send TCP packets.
>> >
>> > Not sure I understand the setup correctly:
>> > So testpmd forwards packets between P0 and vhost-user port, right?
>>
>> Yes.
>>
>> > And who uses P1? iperf-server over linux kernel?
>>
>> P1 is possessed by linux kernel.
>>
>> > Also is P1 on another box or not?
>>
>> P0 and P1 are in the same machine and are connected physically.
>>
>> >
>> > >
>> > > With GSO enabled for P0 in testpmd, observed iperf throughput is ~9Gbps.
>> >
>> > Ok, and if GSO is disabled what is the throughput?
>> > Another stupid question: if P0 is physical 10G (ixgbe?) we can just enable
>a TSO on it, right?
>> > If so, what would be the TSO numbers here?
>>
>> Here are more detailed experiment information:
>>
>> test1: only enable GSO for p0, GSO size is 1518, use two iperf-clients (i.e.
>"-P 2")
>> test2: only enable TSO for p0, TSO size is 1518, use two iperf-clients
>> test3: disable TSO and GSO, use two iperf-clients
>>
>> test1 performance: 8.6Gpbs
>> test2 throughput: 9.5Gbps
>> test3 throughput: 3Mbps
>
>Ok thanks for detailed explanation.
>I' d suggest you put it into next version cover letter.

Thanks Konstantin - will do.

>
>>
>> >
>> > In fact, could you probably explain a bit more, what supposed to be a main
>usage model for that library?
>>
>> The GSO library is just a SW segmentation method, which can be used by
>applications, like OVS.
>> Currently, most of NICs supports to segment TCP and UDP packets, but not for
>all NICs. So current
>> OVS doesn't enable TSO, as a result of lacking a SW segmentation fallback.
>Besides, the protocol
>> types in HW segmentation are limited. So it's necessary to provide a SW
>segmentation solution.
>>
>> With the GSO library, OVS and other applications are able to receive large
>packets from VMs and
>> process these large packets, instead of standard ones (i.e. 1518B). So the
>per-packet overhead is
>> reduced, since the number of packets needed processing is much fewer.
>
>Ok, just for my curiosity what is the size of the packets coming from VM?
>Konstantin

In the case of TSO (and as a corollary, GSO), I guess that the packet size is bounded to ~64k. In OvS, that packet is dequeued using the rte_vhost_dequeue_burst API, and stored in an mbuf chain. The data capacity of mbufs in OvS is user-defined, up to a limit of 9728B.

Thanks,
Mark

>
>
>>
>> > Is that to perform segmentation on (virtual) devices that doesn't support
>HW TSO or ...?
>>
>> When launch qemu with enabling TSO or GSO, the virtual device doesn't really
>do segmentation.
>> It directly sends large packets. Therefore, testpmd can receive large
>packets from the VM and
>> then perform GSO. The GSO/TSO behavior of virtual devices is different from
>physical NICs.
>>
>> > Again would it be for a termination point (packets were just formed and
>filled) by the caller,
>> > or is that for box in the middle which just forwards packets between
>nodes?
>> > If the later one, then we'll probably already have most of our packets
>segmented properly, no?
>> >
>> > > The experimental data of VxLAN and GRE will be shown later.
>> > >
>> > > Jiayu Hu (3):
>> > >   lib: add Generic Segmentation Offload API framework
>> > >   gso/lib: add TCP/IPv4 GSO support
>> > >   app/testpmd: enable TCP/IPv4, VxLAN and GRE GSO
>> > >
>> > > Mark Kavanagh (2):
>> > >   lib/gso: add VxLAN GSO support
>> > >   lib/gso: add GRE GSO support
>> > >
>> > >  app/test-pmd/cmdline.c                  | 121 +++++++++
>> > >  app/test-pmd/config.c                   |  25 ++
>> > >  app/test-pmd/csumonly.c                 |  68 ++++-
>> > >  app/test-pmd/testpmd.c                  |   9 +
>> > >  app/test-pmd/testpmd.h                  |  10 +
>> > >  config/common_base                      |   5 +
>> > >  lib/Makefile                            |   2 +
>> > >  lib/librte_eal/common/include/rte_log.h |   1 +
>> > >  lib/librte_gso/Makefile                 |  52 ++++
>> > >  lib/librte_gso/gso_common.c             | 431
>++++++++++++++++++++++++++++++++
>> > >  lib/librte_gso/gso_common.h             | 180 +++++++++++++
>> > >  lib/librte_gso/gso_tcp.c                |  82 ++++++
>> > >  lib/librte_gso/gso_tcp.h                |  73 ++++++
>> > >  lib/librte_gso/gso_tunnel.c             |  62 +++++
>> > >  lib/librte_gso/gso_tunnel.h             |  46 ++++
>> > >  lib/librte_gso/rte_gso.c                | 100 ++++++++
>> > >  lib/librte_gso/rte_gso.h                | 122 +++++++++
>> > >  lib/librte_gso/rte_gso_version.map      |   7 +
>> > >  mk/rte.app.mk                           |   1 +
>> > >  19 files changed, 1392 insertions(+), 5 deletions(-)
>> > >  create mode 100644 lib/librte_gso/Makefile
>> > >  create mode 100644 lib/librte_gso/gso_common.c
>> > >  create mode 100644 lib/librte_gso/gso_common.h
>> > >  create mode 100644 lib/librte_gso/gso_tcp.c
>> > >  create mode 100644 lib/librte_gso/gso_tcp.h
>> > >  create mode 100644 lib/librte_gso/gso_tunnel.c
>> > >  create mode 100644 lib/librte_gso/gso_tunnel.h
>> > >  create mode 100644 lib/librte_gso/rte_gso.c
>> > >  create mode 100644 lib/librte_gso/rte_gso.h
>> > >  create mode 100644 lib/librte_gso/rte_gso_version.map
>> > >
>> > > --
>> > > 2.7.4