[PATCH net-next 0/3] openvswitch: Add STT support.

[PATCH net-next 0/3] openvswitch: Add STT support.

[Pravin B Shelar]

Following patch series adds support for Stateless Transport
Tunneling protocol.
STT uses TCP segmentation offload available in most of NIC. On
packet xmit STT driver appends STT header along with TCP header
to the packet. For GSO packet GSO parameters are set according
to tunnel configuration and packet is handed over to networking
stack. This allows use of segmentation offload available in NICs

Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
NIC with 1500 byte MTU with two TCP streams.

The protocol is documented at
http://www.ietf.org/archive/id/draft-davie-stt-06.txt

I will send out OVS userspace patch on ovs-dev mailing list.

Pravin B Shelar (3):
  skbuff: Add skb_list_linearize()
  net: Add STT tunneling protocol.
  openvswitch: Add support for STT tunneling.

 include/linux/skbuff.h           |    2 +
 include/net/stt.h                |   55 ++
 include/uapi/linux/openvswitch.h |    1 +
 net/core/skbuff.c                |   35 +
 net/ipv4/Kconfig                 |   11 +
 net/ipv4/Makefile                |    1 +
 net/ipv4/stt.c                   | 1386 ++++++++++++++++++++++++++++++++++++++
 net/openvswitch/Kconfig          |   10 +
 net/openvswitch/Makefile         |    1 +
 net/openvswitch/vport-stt.c      |  214 ++++++
 10 files changed, 1716 insertions(+)
 create mode 100644 include/net/stt.h
 create mode 100644 net/ipv4/stt.c
 create mode 100644 net/openvswitch/vport-stt.c

-- 
1.9.1

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
> Following patch series adds support for Stateless Transport
> Tunneling protocol.
> STT uses TCP segmentation offload available in most of NIC. On
> packet xmit STT driver appends STT header along with TCP header
> to the packet. For GSO packet GSO parameters are set according
> to tunnel configuration and packet is handed over to networking
> stack. This allows use of segmentation offload available in NICs
>
> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
> NIC with 1500 byte MTU with two TCP streams.
>
Having packets marked TCP which really aren't TCP is a rather scary
prospect to deploy in a real data center (TCP is kind of an important
protocol ;-) ). Can you give some more motivation on this, more data
that shows what the benefits are and how this compares to equivalent
encapsulation protocols that implement GRO and GSO.

Thanks,
Tom

> The protocol is documented at
> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
>
> I will send out OVS userspace patch on ovs-dev mailing list.
>
> Pravin B Shelar (3):
>   skbuff: Add skb_list_linearize()
>   net: Add STT tunneling protocol.
>   openvswitch: Add support for STT tunneling.
>
>  include/linux/skbuff.h           |    2 +
>  include/net/stt.h                |   55 ++
>  include/uapi/linux/openvswitch.h |    1 +
>  net/core/skbuff.c                |   35 +
>  net/ipv4/Kconfig                 |   11 +
>  net/ipv4/Makefile                |    1 +
>  net/ipv4/stt.c                   | 1386 ++++++++++++++++++++++++++++++++++++++
>  net/openvswitch/Kconfig          |   10 +
>  net/openvswitch/Makefile         |    1 +
>  net/openvswitch/vport-stt.c      |  214 ++++++
>  10 files changed, 1716 insertions(+)
>  create mode 100644 include/net/stt.h
>  create mode 100644 net/ipv4/stt.c
>  create mode 100644 net/openvswitch/vport-stt.c
>
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Pravin Shelar]

On Tue, Jan 20, 2015 at 3:06 PM, Tom Herbert <therbert@google.com> wrote:
> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>> Following patch series adds support for Stateless Transport
>> Tunneling protocol.
>> STT uses TCP segmentation offload available in most of NIC. On
>> packet xmit STT driver appends STT header along with TCP header
>> to the packet. For GSO packet GSO parameters are set according
>> to tunnel configuration and packet is handed over to networking
>> stack. This allows use of segmentation offload available in NICs
>>
>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>> NIC with 1500 byte MTU with two TCP streams.
>>
> Having packets marked TCP which really aren't TCP is a rather scary
> prospect to deploy in a real data center (TCP is kind of an important
> protocol ;-) ). Can you give some more motivation on this, more data
> that shows what the benefits are and how this compares to equivalent
> encapsulation protocols that implement GRO and GSO.
>
There are multi-year deployments of STT, So it is already in real data center.
Biggest advantage is STT does not need new NIC with tunnel offload.
Any NIC that supports TCP offload can be used to achieve better
performance.

Following are numbers you asked for.
Setup: net-next branch on server and client.
netperf: TCP unidirectional tests with 5 streams. Numbers are averaged
over 3 runs of 50 sec.

VXLAN:
CPU
  Client: 1.6
  Server: 14.2
Throughput: 5.6 Gbit/s

VXLAN with rcsum:
CPU
  Client: 0.89
  Server: 12.4
Throughput: 5.8 Gbit/s

STT:
CPU
  Client: 1.28
  Server: 4.0
Throughput: 9.5 Gbit/s

Thanks,
Pravin.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Wed, Jan 21, 2015 at 1:08 AM, Pravin Shelar <pshelar@nicira.com> wrote:
> On Tue, Jan 20, 2015 at 3:06 PM, Tom Herbert <therbert@google.com> wrote:
>> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>>> Following patch series adds support for Stateless Transport
>>> Tunneling protocol.
>>> STT uses TCP segmentation offload available in most of NIC. On
>>> packet xmit STT driver appends STT header along with TCP header
>>> to the packet. For GSO packet GSO parameters are set according
>>> to tunnel configuration and packet is handed over to networking
>>> stack. This allows use of segmentation offload available in NICs
>>>
>>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>>> NIC with 1500 byte MTU with two TCP streams.
>>>
>> Having packets marked TCP which really aren't TCP is a rather scary
>> prospect to deploy in a real data center (TCP is kind of an important
>> protocol ;-) ). Can you give some more motivation on this, more data
>> that shows what the benefits are and how this compares to equivalent
>> encapsulation protocols that implement GRO and GSO.
>>
> There are multi-year deployments of STT, So it is already in real data center.
> Biggest advantage is STT does not need new NIC with tunnel offload.
> Any NIC that supports TCP offload can be used to achieve better
> performance.
>
> Following are numbers you asked for.
> Setup: net-next branch on server and client.
> netperf: TCP unidirectional tests with 5 streams. Numbers are averaged
> over 3 runs of 50 sec.
>
Please provides more details on your configuration so that others
might be able to reproduce your results. Also, it would be quite
helpful if you could implement STT as a normal network interface like
VXLAN does so that we can isolate performance of the protocol. For
instance I have no problem getting line rate with VXLAN using 5
streams with or without RCO in my testing. I assume you tested with
OVS and maybe VMs which may have a significant impact beyond the
protocol changes.

Another thing to consider in your analysis is the performance with
flows using small packets. STT should demonstrate better performance
with bulk flows since LSO and LRO are better performing relative to
GSO and GRO. But for flows with small packets, I don't see how there
could be any performance advantage. We already have ways to leverage
simple UDP checksum offload with UDP encapsulations, seems like STT
might just represent unnecessary header overhead in those cases.

> VXLAN:
> CPU
>   Client: 1.6
>   Server: 14.2
> Throughput: 5.6 Gbit/s
>
> VXLAN with rcsum:
> CPU
>   Client: 0.89
>   Server: 12.4
> Throughput: 5.8 Gbit/s
>
> STT:
> CPU
>   Client: 1.28
>   Server: 4.0
> Throughput: 9.5 Gbit/s
>
9.5Gbps? Rounding error or is this 40Gbps or larger than 1500 byte MTU?

Thanks,
Tom

> Thanks,
> Pravin.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Pravin Shelar]

On Wed, Jan 21, 2015 at 8:51 AM, Tom Herbert <therbert@google.com> wrote:
> On Wed, Jan 21, 2015 at 1:08 AM, Pravin Shelar <pshelar@nicira.com> wrote:
>> On Tue, Jan 20, 2015 at 3:06 PM, Tom Herbert <therbert@google.com> wrote:
>>> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>>>> Following patch series adds support for Stateless Transport
>>>> Tunneling protocol.
>>>> STT uses TCP segmentation offload available in most of NIC. On
>>>> packet xmit STT driver appends STT header along with TCP header
>>>> to the packet. For GSO packet GSO parameters are set according
>>>> to tunnel configuration and packet is handed over to networking
>>>> stack. This allows use of segmentation offload available in NICs
>>>>
>>>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>>>> NIC with 1500 byte MTU with two TCP streams.
>>>>
>>> Having packets marked TCP which really aren't TCP is a rather scary
>>> prospect to deploy in a real data center (TCP is kind of an important
>>> protocol ;-) ). Can you give some more motivation on this, more data
>>> that shows what the benefits are and how this compares to equivalent
>>> encapsulation protocols that implement GRO and GSO.
>>>
>> There are multi-year deployments of STT, So it is already in real data center.
>> Biggest advantage is STT does not need new NIC with tunnel offload.
>> Any NIC that supports TCP offload can be used to achieve better
>> performance.
>>
>> Following are numbers you asked for.
>> Setup: net-next branch on server and client.
>> netperf: TCP unidirectional tests with 5 streams. Numbers are averaged
>> over 3 runs of 50 sec.
>>
> Please provides more details on your configuration so that others
> might be able to reproduce your results. Also, it would be quite
> helpful if you could implement STT as a normal network interface like
> VXLAN does so that we can isolate performance of the protocol. For
> instance I have no problem getting line rate with VXLAN using 5
> streams with or without RCO in my testing. I assume you tested with
> OVS and maybe VMs which may have a significant impact beyond the
> protocol changes.
>
I used bare metal intel servers. All VXLAN tests were done using linux
kernel device without any VMs. All STT tests are done using OVS bridge
and STT port.

> Another thing to consider in your analysis is the performance with
> flows using small packets. STT should demonstrate better performance
> with bulk flows since LSO and LRO are better performing relative to
> GSO and GRO. But for flows with small packets, I don't see how there
> could be any performance advantage. We already have ways to leverage
> simple UDP checksum offload with UDP encapsulations, seems like STT
> might just represent unnecessary header overhead in those cases.
>
All tunneling protocol has performance issue with small packet, I do
not see how is it related to STT patch. STT also make use of checksum
offload, so there should not be much overhead.

>> VXLAN:
>> CPU
>>   Client: 1.6
>>   Server: 14.2
>> Throughput: 5.6 Gbit/s
>>
>> VXLAN with rcsum:
>> CPU
>>   Client: 0.89
>>   Server: 12.4
>> Throughput: 5.8 Gbit/s
>>
>> STT:
>> CPU
>>   Client: 1.28
>>   Server: 4.0
>> Throughput: 9.5 Gbit/s
>>
> 9.5Gbps? Rounding error or is this 40Gbps or larger than 1500 byte MTU?
>
Nope, its same as VXLAN setup, 10Gbps NIC with 1500MTU.

Thanks.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

> I used bare metal intel servers. All VXLAN tests were done using linux
> kernel device without any VMs. All STT tests are done using OVS bridge
> and STT port.
>
So right off the bat you're running the baseline differently than the
target. Anyway, I cannot replicate your numbers for VXLAN, I see much
better performance and this with pretty old servers and dumb NICs. I
suspect you might not have GSO/GRO properly enabled, but instead of
trying to debug your setup, I'd rather restate my request that you
provide a network interface to STT so we can do our own fair
comparison.

>> Another thing to consider in your analysis is the performance with
>> flows using small packets. STT should demonstrate better performance
>> with bulk flows since LSO and LRO are better performing relative to
>> GSO and GRO. But for flows with small packets, I don't see how there
>> could be any performance advantage. We already have ways to leverage
>> simple UDP checksum offload with UDP encapsulations, seems like STT
>> might just represent unnecessary header overhead in those cases.
>>
> All tunneling protocol has performance issue with small packet, I do
> not see how is it related to STT patch. STT also make use of checksum
> offload, so there should not be much overhead.
>

Given the pervasiveness of these patches and the fact that this is
"modifying" the definition of TCP protocol we see on the wire, I would
like to see more effort into analyzing performance and effects of this
encapsulation. I assume that STT is intended to be used for small
packets so it seems entirely reasonable to ask what exactly the
performance effects are for that case. It's really not that hard to
run some performance numbers with comparisons (TCP_RR, TCP_STREAM,
TCP_CRR, with IPv6, etc.) and report them in the patch set
description. You can look at FOU, GUE, vxlan rco, and the checksum
patches I did for some examples.

>>> VXLAN:
>>> CPU
>>>   Client: 1.6
>>>   Server: 14.2
>>> Throughput: 5.6 Gbit/s
>>>
>>> VXLAN with rcsum:
>>> CPU
>>>   Client: 0.89
>>>   Server: 12.4
>>> Throughput: 5.8 Gbit/s
>>>
>>> STT:
>>> CPU
>>>   Client: 1.28
>>>   Server: 4.0
>>> Throughput: 9.5 Gbit/s
>>>
>> 9.5Gbps? Rounding error or is this 40Gbps or larger than 1500 byte MTU?
>>
> Nope, its same as VXLAN setup, 10Gbps NIC with 1500MTU.
>
That would exceed that theoretical maximum for TCP over 10Gbps
Ethernet. How are you measuring throughput? How many bytes of protocol
headers are in STT case?

Thanks,
Tom

> Thanks.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Eric Dumazet]

On Wed, 2015-01-21 at 11:45 -0800, Tom Herbert wrote:
> > I used bare metal intel servers. All VXLAN tests were done using linux
> > kernel device without any VMs. All STT tests are done using OVS bridge
> > and STT port.
> >
> So right off the bat you're running the baseline differently than the
> target. Anyway, I cannot replicate your numbers for VXLAN, I see much
> better performance and this with pretty old servers and dumb NICs. I
> suspect you might not have GSO/GRO properly enabled, but instead of
> trying to debug your setup, I'd rather restate my request that you
> provide a network interface to STT so we can do our own fair
> comparison.

I have not read specs nor the patch, but I would expect that with a TCP
stream, you simply can coalesce as many inner segments as you want.

So you could build a single TSO packet containing 400 'messages' of 160
bytes or so.

Something that a datagram tunneling cannot really do, assuming you want
full offloading support.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Jesse Gross]

On Wed, Jan 21, 2015 at 11:45 AM, Tom Herbert <therbert@google.com> wrote:
>> I used bare metal intel servers. All VXLAN tests were done using linux
>> kernel device without any VMs. All STT tests are done using OVS bridge
>> and STT port.
>>
> So right off the bat you're running the baseline differently than the
> target. Anyway, I cannot replicate your numbers for VXLAN, I see much
> better performance and this with pretty old servers and dumb NICs. I
> suspect you might not have GSO/GRO properly enabled, but instead of
> trying to debug your setup, I'd rather restate my request that you
> provide a network interface to STT so we can do our own fair
> comparison.

If I had to guess, I suspect the difference is that UDP RSS wasn't
enabled, since it doesn't come that way out of the box. Regardless,
you can clearly see a significant difference in single core
performance and CPU consumption.

STT has been fairly well known in network virtualization circles for
the past few years and has some large deployments, so the reported
performance is not a fluke. I remember Pankaj from Microsoft also
mentioning to you that they weren't able to get performance to
reasonable level without TSO. Totally different environment obviously
but same reasoning.

>>>> VXLAN:
>>>> CPU
>>>>   Client: 1.6
>>>>   Server: 14.2
>>>> Throughput: 5.6 Gbit/s
>>>>
>>>> VXLAN with rcsum:
>>>> CPU
>>>>   Client: 0.89
>>>>   Server: 12.4
>>>> Throughput: 5.8 Gbit/s
>>>>
>>>> STT:
>>>> CPU
>>>>   Client: 1.28
>>>>   Server: 4.0
>>>> Throughput: 9.5 Gbit/s
>>>>
>>> 9.5Gbps? Rounding error or is this 40Gbps or larger than 1500 byte MTU?
>>>
>> Nope, its same as VXLAN setup, 10Gbps NIC with 1500MTU.
>>
> That would exceed that theoretical maximum for TCP over 10Gbps
> Ethernet. How are you measuring throughput? How many bytes of protocol
> headers are in STT case?

For large packet cases, STT actually has less header overhead compared
to the unencapsulated traffic stream. This is because for a group of
STT packets generated by a TSO burst from the guest there is only a
single copy of the inner header. Even though TCP headers are used for
encapsulation, there are no options - as opposed to the inner headers,
which typically contain timestamps. Over the course of the ~45 packets
that could be generated from a maximum sized transmission, this
results in negative encapsulation overhead.

I would recommend you take a look at the draft if you haven't already:
http://tools.ietf.org/html/draft-davie-stt-06

It is currently in the final stages of the RFC publication process.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Wed, Jan 21, 2015 at 12:35 PM, Jesse Gross <jesse@nicira.com> wrote:
> On Wed, Jan 21, 2015 at 11:45 AM, Tom Herbert <therbert@google.com> wrote:
>>> I used bare metal intel servers. All VXLAN tests were done using linux
>>> kernel device without any VMs. All STT tests are done using OVS bridge
>>> and STT port.
>>>
>> So right off the bat you're running the baseline differently than the
>> target. Anyway, I cannot replicate your numbers for VXLAN, I see much
>> better performance and this with pretty old servers and dumb NICs. I
>> suspect you might not have GSO/GRO properly enabled, but instead of
>> trying to debug your setup, I'd rather restate my request that you
>> provide a network interface to STT so we can do our own fair
>> comparison.
>
> If I had to guess, I suspect the difference is that UDP RSS wasn't
> enabled, since it doesn't come that way out of the box. Regardless,
> you can clearly see a significant difference in single core
> performance and CPU consumption.
>
I'm not going to try to draw conclusions from data which is obviously
biased and incomplete. If you want to move forward on this, then just
provide network interface for STT so we can independently run our own
comparisons against other encapsulations like we've been doing all
along.

> STT has been fairly well known in network virtualization circles for
> the past few years and has some large deployments, so the reported
> performance is not a fluke. I remember Pankaj from Microsoft also
> mentioning to you that they weren't able to get performance to
> reasonable level without TSO. Totally different environment obviously
> but same reasoning.
>
>>>>> VXLAN:
>>>>> CPU
>>>>>   Client: 1.6
>>>>>   Server: 14.2
>>>>> Throughput: 5.6 Gbit/s
>>>>>
>>>>> VXLAN with rcsum:
>>>>> CPU
>>>>>   Client: 0.89
>>>>>   Server: 12.4
>>>>> Throughput: 5.8 Gbit/s
>>>>>
>>>>> STT:
>>>>> CPU
>>>>>   Client: 1.28
>>>>>   Server: 4.0
>>>>> Throughput: 9.5 Gbit/s
>>>>>
>>>> 9.5Gbps? Rounding error or is this 40Gbps or larger than 1500 byte MTU?
>>>>
>>> Nope, its same as VXLAN setup, 10Gbps NIC with 1500MTU.
>>>
>> That would exceed that theoretical maximum for TCP over 10Gbps
>> Ethernet. How are you measuring throughput? How many bytes of protocol
>> headers are in STT case?
>
> For large packet cases, STT actually has less header overhead compared
> to the unencapsulated traffic stream. This is because for a group of
> STT packets generated by a TSO burst from the guest there is only a
> single copy of the inner header. Even though TCP headers are used for
> encapsulation, there are no options - as opposed to the inner headers,
> which typically contain timestamps. Over the course of the ~45 packets
> that could be generated from a maximum sized transmission, this
> results in negative encapsulation overhead.
>
> I would recommend you take a look at the draft if you haven't already:
> http://tools.ietf.org/html/draft-davie-stt-06
>
> It is currently in the final stages of the RFC publication process.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Jesse Gross]

On Wed, Jan 21, 2015 at 1:54 PM, Tom Herbert <therbert@google.com> wrote:
> On Wed, Jan 21, 2015 at 12:35 PM, Jesse Gross <jesse@nicira.com> wrote:
>> On Wed, Jan 21, 2015 at 11:45 AM, Tom Herbert <therbert@google.com> wrote:
>>>> I used bare metal intel servers. All VXLAN tests were done using linux
>>>> kernel device without any VMs. All STT tests are done using OVS bridge
>>>> and STT port.
>>>>
>>> So right off the bat you're running the baseline differently than the
>>> target. Anyway, I cannot replicate your numbers for VXLAN, I see much
>>> better performance and this with pretty old servers and dumb NICs. I
>>> suspect you might not have GSO/GRO properly enabled, but instead of
>>> trying to debug your setup, I'd rather restate my request that you
>>> provide a network interface to STT so we can do our own fair
>>> comparison.
>>
>> If I had to guess, I suspect the difference is that UDP RSS wasn't
>> enabled, since it doesn't come that way out of the box. Regardless,
>> you can clearly see a significant difference in single core
>> performance and CPU consumption.
>>
> I'm not going to try to draw conclusions from data which is obviously
> biased and incomplete. If you want to move forward on this, then just
> provide network interface for STT so we can independently run our own
> comparisons against other encapsulations like we've been doing all
> along.

You have the source code, so you are totally free to run whatever
tests you like to draw your own conclusions. Personally, I find a more
than doubling of performance in the environments that I have seen
compelling. Your mileage may vary.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Vincent JARDIN]

Jesse, Tom,

On 21/01/2015 23:14, Jesse Gross wrote:
>> I'm not going to try to draw conclusions from data which is obviously
>> >biased and incomplete. If you want to move forward on this, then just
>> >provide network interface for STT so we can independently run our own
>> >comparisons against other encapsulations like we've been doing all
>> >along.
> You have the source code, so you are totally free to run whatever
> tests you like to draw your own conclusions. Personally, I find a more
> than doubling of performance in the environments that I have seen
> compelling. Your mileage may vary.

+1 for STT in the kernel:
   - whatever the performances, it is needed because it happened to be 
used. If the patch can be optimized, someone will do and provide the 
related patches. The patch from Pravin is ok but...

   - ...I agree with Tom, a netdevice is a must have to ack't this 
patch. Such feature should not be added into openvswitch without its 
counter-part netdevice.

thank you,
   Vincent

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Wed, Jan 21, 2015 at 3:46 PM, Vincent JARDIN
<vincent.jardin@6wind.com> wrote:
> Jesse, Tom,
>
> On 21/01/2015 23:14, Jesse Gross wrote:
>>>
>>> I'm not going to try to draw conclusions from data which is obviously
>>> >biased and incomplete. If you want to move forward on this, then just
>>> >provide network interface for STT so we can independently run our own
>>> >comparisons against other encapsulations like we've been doing all
>>> >along.
>>
>> You have the source code, so you are totally free to run whatever
>> tests you like to draw your own conclusions. Personally, I find a more
>> than doubling of performance in the environments that I have seen
>> compelling. Your mileage may vary.
>
>
> +1 for STT in the kernel:
>   - whatever the performances, it is needed because it happened to be used.

Vincent, it's not that simple. This is not just another case of an
encapsulation protocol that we can easily ignore and filter because it
hides behind a UDP port or even a new protocol number. This is adding
a new definition to *the* most critical protocol on the planet. I
don't see how this anyone can claim this doesn't violate a whole bunch
of long standing RFCs and break a whole bunch of baked in assumptions
we make about TCP (IP protocol number 6).

For example, TCP MUST have congestion avoidance as per RFC2581:

"The slow start and congestion avoidance algorithms MUST be used by a
TCP sender to control the amount of outstanding data being injected
into the network."

But from STT draft:

"STT segments are transmitted as IP datagrams using the TCP protocol
number (6)."-- that makes this a TCP sender.

"STT does not provide any sort of congestion control."-- that puts STT
in clear violation of RFC2581.

Congestion avoidance for TCP is not just a nice to have, it's not
optional, it's not just best effort like UDP is. There is no provision
for reusing the TCP protocol number and claiming to be exempt from
standards compliance because it's now a different protocol. We build
whole data centers around these principles, and in fact the very
operation of the Internet depends on them. For instance, if we allow
this into the kernel and this becomes available on billions of
devices, what assurances do we have that people won't start abusing
this because they find congestion avoidance annoying and this a
convenient way to bypass it?

STT is undoubtedly a creative and unique solution I'll give you that,
but the potential repercussions of allowing this to be widely deployed
are profound. IMO this needs to be fully explored before it can ever
be allowed in the kernel. If there has already been discussion on this
please forward a pointer (I didn't find anything in the IETF mailing
list archives other than the draft posting announcements), but at the
minimum these patches warrant a lot of scrutiny.

Thanks,
Tom

> If the patch can be optimized, someone will do and provide the related
> patches. The patch from Pravin is ok but...
>
>   - ...I agree with Tom, a netdevice is a must have to ack't this patch.
> Such feature should not be added into openvswitch without its counter-part
> netdevice.
>
> thank you,
>   Vincent

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Vincent JARDIN]

On 22/01/2015 17:24, Tom Herbert wrote:
> STT is undoubtedly a creative and unique solution I'll give you that,
> but the potential repercussions of allowing this to be widely deployed
> are profound. IMO this needs to be fully explored before it can ever
> be allowed in the kernel. If there has already been discussion on this
> please forward a pointer (I didn't find anything in the IETF mailing
> list archives other than the draft posting announcements), but at the
> minimum these patches warrant a lot of scrutiny.

I share this concern of biased use of TCP, all the critics will be 
valid. But anyone can hack TCP (so peer to peer software does or CDN 
software does). So, I prefer the let the freedom to the sysadmin to 
enable/disable it for their networks. Not having this feature into the 
kernel prevent sysadmin from doing it. To be safe, it can be an 
experimental feature of the Linux kernel.

Same: LRO/GRO is is bad features: it breaks many times networking (most 
IP forwarders must disable it), but it helps for servers. Same for STT 
in fact, it has its narrow set of use-cases which are valid.

Best regards,
   Vincent

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[David Miller]

From: Vincent JARDIN <vincent.jardin@6wind.com>
Date: Thu, 22 Jan 2015 18:51:55 +0100

> Same: LRO/GRO is is bad features: it breaks many times networking
> (most IP forwarders must disable it), but it helps for servers. Same
> for STT in fact, it has its narrow set of use-cases which are valid.

False, GRO helps for forwarders too, because it decreases the number
of IP route lookups, one of the most expensive operations during
formwarding.

Please do not spread misinformation, and especially do not try to show
that GRO is in _ANY_ way something like STT.  That's completely wrong.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[David Miller]

From: Tom Herbert <therbert@google.com>
Date: Thu, 22 Jan 2015 08:24:15 -0800

> STT is undoubtedly a creative and unique solution I'll give you that,
> but the potential repercussions of allowing this to be widely deployed
> are profound. IMO this needs to be fully explored before it can ever
> be allowed in the kernel. If there has already been discussion on this
> please forward a pointer (I didn't find anything in the IETF mailing
> list archives other than the draft posting announcements), but at the
> minimum these patches warrant a lot of scrutiny.

+1

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

> I would recommend you take a look at the draft if you haven't already:
> http://tools.ietf.org/html/draft-davie-stt-06
>
> It is currently in the final stages of the RFC publication process.

Sorry, but this statement is completely wrong and misleading.
According to datatracker this draft has been expired since October,
there's been no discussion on it in IETF, and this has not gone to
IESG. You cannot say this is an IETF standard nor that it is about to
be published as one.  See
https://datatracker.ietf.org/doc/draft-davie-stt/ and please read the
Internet Standards process in RFC2026.

I suggest that you update the draft and post it on both nvo3 and tsvwg
so there can be some real discussion on the implications of
repurposing an IP protocol number and breaking TCP protocol standards.

Tom

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Jesse Gross]

On Mon, Feb 2, 2015 at 8:23 AM, Tom Herbert <therbert@google.com> wrote:
>> I would recommend you take a look at the draft if you haven't already:
>> http://tools.ietf.org/html/draft-davie-stt-06
>>
>> It is currently in the final stages of the RFC publication process.
>
> Sorry, but this statement is completely wrong and misleading.
> According to datatracker this draft has been expired since October,
> there's been no discussion on it in IETF, and this has not gone to
> IESG. You cannot say this is an IETF standard nor that it is about to
> be published as one.  See
> https://datatracker.ietf.org/doc/draft-davie-stt/ and please read the
> Internet Standards process in RFC2026.
>
> I suggest that you update the draft and post it on both nvo3 and tsvwg
> so there can be some real discussion on the implications of
> repurposing an IP protocol number and breaking TCP protocol standards.

Seriously, Tom?

It's currently in the ISE queue to be published as an RFC, which you
can see in the history in the datatracker link. I didn't say that it
was being published as a standard, I said RFC. This is the same
process and status that VXLAN has.. It also was presented in nvo3
before you started coming.

Please get your facts straight before making accusations.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Mon, Feb 2, 2015 at 12:39 PM, Jesse Gross <jesse@nicira.com> wrote:
> On Mon, Feb 2, 2015 at 8:23 AM, Tom Herbert <therbert@google.com> wrote:
>>> I would recommend you take a look at the draft if you haven't already:
>>> http://tools.ietf.org/html/draft-davie-stt-06
>>>
>>> It is currently in the final stages of the RFC publication process.
>>
>> Sorry, but this statement is completely wrong and misleading.
>> According to datatracker this draft has been expired since October,
>> there's been no discussion on it in IETF, and this has not gone to
>> IESG. You cannot say this is an IETF standard nor that it is about to
>> be published as one.  See
>> https://datatracker.ietf.org/doc/draft-davie-stt/ and please read the
>> Internet Standards process in RFC2026.
>>
>> I suggest that you update the draft and post it on both nvo3 and tsvwg
>> so there can be some real discussion on the implications of
>> repurposing an IP protocol number and breaking TCP protocol standards.
>
> Seriously, Tom?
>
> It's currently in the ISE queue to be published as an RFC, which you
> can see in the history in the datatracker link. I didn't say that it
> was being published as a standard, I said RFC. This is the same
> process and status that VXLAN has.. It also was presented in nvo3
> before you started coming.
>
> Please get your facts straight before making accusations.

The draft has not come up before the IESG, I have confirmed that with the ADs.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
> Following patch series adds support for Stateless Transport
> Tunneling protocol.
> STT uses TCP segmentation offload available in most of NIC. On
> packet xmit STT driver appends STT header along with TCP header
> to the packet. For GSO packet GSO parameters are set according
> to tunnel configuration and packet is handed over to networking
> stack. This allows use of segmentation offload available in NICs
>
> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
> NIC with 1500 byte MTU with two TCP streams.
>
The reason you're able to get 9.4 Gbit/s with an L2 encapsulation
using STT is that it has less protocol overhead per packet when doing
segmentation compared to VXLAN (without segmentation STT packets will
have more overhead than VXLAN).

A VXLAN packet with TCP/IP has headers
IP|UDP|VXLAN|Ethernet|IP|TCP+options. Assuming TCP is stuffed with
options, this is 20+8+8+16+20+40=112 bytes, or 7.4% MTU. Each STT
segment created in GSO, other than the first, has just IP|TCP headers
which is 20+20=40 bytes or 2.6% MTU. So this explains throughput
differences between VXLAN and STT.

With some clever coding, this same effect can be achieved using a UDP
encapsulation protocol that allows options. Suppose we create a new
option that we'll call Remote Segmentation Offload (RSO). The option
contains sequence number, fragment number that is coming from TCP
header in STT. We'll give this option eight bytes, and we'll also need
Remote Checksum Offload (therefore UDP checksum enabled). The headers
for each segment (except first) in GUE would then be something like
IP|UDP|GUE+options, with overhead  20+8+4+16 bytes=48 bytes, or 3.2%
MTU.

Like STT, RSO requires reassembly. Most of this can probably be done
in GRO. Processing is also a little cheaper since we don't need to
walk down as many protocol layers.

Conceptually, in the presence of packet loss we can recover any
segments as long as the first one in the chain that contains the full
set of headers isn't lost. If the first one is lost, then everything
in that block is lost since there's not enough context for reassembly
(I suspect this is also true in STT).

Tom

> The protocol is documented at
> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
>
> I will send out OVS userspace patch on ovs-dev mailing list.
>
> Pravin B Shelar (3):
>   skbuff: Add skb_list_linearize()
>   net: Add STT tunneling protocol.
>   openvswitch: Add support for STT tunneling.
>
>  include/linux/skbuff.h           |    2 +
>  include/net/stt.h                |   55 ++
>  include/uapi/linux/openvswitch.h |    1 +
>  net/core/skbuff.c                |   35 +
>  net/ipv4/Kconfig                 |   11 +
>  net/ipv4/Makefile                |    1 +
>  net/ipv4/stt.c                   | 1386 ++++++++++++++++++++++++++++++++++++++
>  net/openvswitch/Kconfig          |   10 +
>  net/openvswitch/Makefile         |    1 +
>  net/openvswitch/vport-stt.c      |  214 ++++++
>  10 files changed, 1716 insertions(+)
>  create mode 100644 include/net/stt.h
>  create mode 100644 net/ipv4/stt.c
>  create mode 100644 net/openvswitch/vport-stt.c
>
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Jesse Gross]

On Fri, Jan 23, 2015 at 8:58 AM, Tom Herbert <therbert@google.com> wrote:
> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>> Following patch series adds support for Stateless Transport
>> Tunneling protocol.
>> STT uses TCP segmentation offload available in most of NIC. On
>> packet xmit STT driver appends STT header along with TCP header
>> to the packet. For GSO packet GSO parameters are set according
>> to tunnel configuration and packet is handed over to networking
>> stack. This allows use of segmentation offload available in NICs
>>
>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>> NIC with 1500 byte MTU with two TCP streams.
>>
> The reason you're able to get 9.4 Gbit/s with an L2 encapsulation
> using STT is that it has less protocol overhead per packet when doing
> segmentation compared to VXLAN (without segmentation STT packets will
> have more overhead than VXLAN).
>
> A VXLAN packet with TCP/IP has headers
> IP|UDP|VXLAN|Ethernet|IP|TCP+options. Assuming TCP is stuffed with
> options, this is 20+8+8+16+20+40=112 bytes, or 7.4% MTU. Each STT
> segment created in GSO, other than the first, has just IP|TCP headers
> which is 20+20=40 bytes or 2.6% MTU. So this explains throughput
> differences between VXLAN and STT.

Tom, what performance do you see with a single stream of VXLAN running
on net-next with default configuration? The difference in numbers
being posted here is greater than a few percent caused by protocol
overheard.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Fri, Jan 23, 2015 at 9:38 AM, Jesse Gross <jesse@nicira.com> wrote:
> On Fri, Jan 23, 2015 at 8:58 AM, Tom Herbert <therbert@google.com> wrote:
>> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>>> Following patch series adds support for Stateless Transport
>>> Tunneling protocol.
>>> STT uses TCP segmentation offload available in most of NIC. On
>>> packet xmit STT driver appends STT header along with TCP header
>>> to the packet. For GSO packet GSO parameters are set according
>>> to tunnel configuration and packet is handed over to networking
>>> stack. This allows use of segmentation offload available in NICs
>>>
>>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>>> NIC with 1500 byte MTU with two TCP streams.
>>>
>> The reason you're able to get 9.4 Gbit/s with an L2 encapsulation
>> using STT is that it has less protocol overhead per packet when doing
>> segmentation compared to VXLAN (without segmentation STT packets will
>> have more overhead than VXLAN).
>>
>> A VXLAN packet with TCP/IP has headers
>> IP|UDP|VXLAN|Ethernet|IP|TCP+options. Assuming TCP is stuffed with
>> options, this is 20+8+8+16+20+40=112 bytes, or 7.4% MTU. Each STT
>> segment created in GSO, other than the first, has just IP|TCP headers
>> which is 20+20=40 bytes or 2.6% MTU. So this explains throughput
>> differences between VXLAN and STT.
>
> Tom, what performance do you see with a single stream of VXLAN running
> on net-next with default configuration? The difference in numbers
> being posted here is greater than a few percent caused by protocol
> overheard.

Please look at the data I posted with the VXLAN RCO patches.

TCP/IP 10Gbps with 1500 byte MTU, 14 bytes for Ethernet header, 15
bytes for preamble, FCS, IPG, and no TCP options gives 9.54Gbps
maximum throughput. About what you see in STT minus overhead in first
segment.

TCP/IPv4 on VXLAN with TCP options gives max throughout 9.09Gbps,
TCP/IPv6 over VXLAN then gives 8.96Gbps. Pretty much what we see give
or take a few bytes for TCP options and rounding error.

Tom

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Jesse Gross]

On Fri, Jan 23, 2015 at 10:25 AM, Tom Herbert <therbert@google.com> wrote:
> On Fri, Jan 23, 2015 at 9:38 AM, Jesse Gross <jesse@nicira.com> wrote:
>> On Fri, Jan 23, 2015 at 8:58 AM, Tom Herbert <therbert@google.com> wrote:
>>> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>>>> Following patch series adds support for Stateless Transport
>>>> Tunneling protocol.
>>>> STT uses TCP segmentation offload available in most of NIC. On
>>>> packet xmit STT driver appends STT header along with TCP header
>>>> to the packet. For GSO packet GSO parameters are set according
>>>> to tunnel configuration and packet is handed over to networking
>>>> stack. This allows use of segmentation offload available in NICs
>>>>
>>>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>>>> NIC with 1500 byte MTU with two TCP streams.
>>>>
>>> The reason you're able to get 9.4 Gbit/s with an L2 encapsulation
>>> using STT is that it has less protocol overhead per packet when doing
>>> segmentation compared to VXLAN (without segmentation STT packets will
>>> have more overhead than VXLAN).
>>>
>>> A VXLAN packet with TCP/IP has headers
>>> IP|UDP|VXLAN|Ethernet|IP|TCP+options. Assuming TCP is stuffed with
>>> options, this is 20+8+8+16+20+40=112 bytes, or 7.4% MTU. Each STT
>>> segment created in GSO, other than the first, has just IP|TCP headers
>>> which is 20+20=40 bytes or 2.6% MTU. So this explains throughput
>>> differences between VXLAN and STT.
>>
>> Tom, what performance do you see with a single stream of VXLAN running
>> on net-next with default configuration? The difference in numbers
>> being posted here is greater than a few percent caused by protocol
>> overheard.
>
> Please look at the data I posted with the VXLAN RCO patches.

The data you posted uses 200 streams, so I assume that you are using
multiple CPUs. It's not surprising that you would be able to consume a
10G link in that case. STT can do this with a single stream and less
than 1 core (or alternately handle higher throughput). Claiming that
since both can hit 10G they are same is not accurate.

Discussing performance like this seems a little silly given that the
code is available. Pravin posted some numbers that he got, if you want
to dispute them then why don't you just try running it?

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

On Fri, Jan 23, 2015 at 12:20 PM, Jesse Gross <jesse@nicira.com> wrote:
> On Fri, Jan 23, 2015 at 10:25 AM, Tom Herbert <therbert@google.com> wrote:
>> On Fri, Jan 23, 2015 at 9:38 AM, Jesse Gross <jesse@nicira.com> wrote:
>>> On Fri, Jan 23, 2015 at 8:58 AM, Tom Herbert <therbert@google.com> wrote:
>>>> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>>>>> Following patch series adds support for Stateless Transport
>>>>> Tunneling protocol.
>>>>> STT uses TCP segmentation offload available in most of NIC. On
>>>>> packet xmit STT driver appends STT header along with TCP header
>>>>> to the packet. For GSO packet GSO parameters are set according
>>>>> to tunnel configuration and packet is handed over to networking
>>>>> stack. This allows use of segmentation offload available in NICs
>>>>>
>>>>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>>>>> NIC with 1500 byte MTU with two TCP streams.
>>>>>
>>>> The reason you're able to get 9.4 Gbit/s with an L2 encapsulation
>>>> using STT is that it has less protocol overhead per packet when doing
>>>> segmentation compared to VXLAN (without segmentation STT packets will
>>>> have more overhead than VXLAN).
>>>>
>>>> A VXLAN packet with TCP/IP has headers
>>>> IP|UDP|VXLAN|Ethernet|IP|TCP+options. Assuming TCP is stuffed with
>>>> options, this is 20+8+8+16+20+40=112 bytes, or 7.4% MTU. Each STT
>>>> segment created in GSO, other than the first, has just IP|TCP headers
>>>> which is 20+20=40 bytes or 2.6% MTU. So this explains throughput
>>>> differences between VXLAN and STT.
>>>
>>> Tom, what performance do you see with a single stream of VXLAN running
>>> on net-next with default configuration? The difference in numbers
>>> being posted here is greater than a few percent caused by protocol
>>> overheard.
>>
>> Please look at the data I posted with the VXLAN RCO patches.
>
> The data you posted uses 200 streams, so I assume that you are using
> multiple CPUs. It's not surprising that you would be able to consume a
> 10G link in that case. STT can do this with a single stream and less
> than 1 core (or alternately handle higher throughput). Claiming that
> since both can hit 10G they are same is not accurate.
>
> Discussing performance like this seems a little silly given that the
> code is available. Pravin posted some numbers that he got, if you want
> to dispute them then why don't you just try running it?

Because you haven't provided network interface like I already
requested twice, and I really don't have time or motivation to do
development on your patches or figure out how to do this with OVS. If
you want me to test your patches re-spin them with a network interface
included.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Pravin Shelar]

On Fri, Jan 23, 2015 at 12:57 PM, Tom Herbert <therbert@google.com> wrote:
> On Fri, Jan 23, 2015 at 12:20 PM, Jesse Gross <jesse@nicira.com> wrote:
>> On Fri, Jan 23, 2015 at 10:25 AM, Tom Herbert <therbert@google.com> wrote:
>>> On Fri, Jan 23, 2015 at 9:38 AM, Jesse Gross <jesse@nicira.com> wrote:
>>>> On Fri, Jan 23, 2015 at 8:58 AM, Tom Herbert <therbert@google.com> wrote:
>>>>> On Tue, Jan 20, 2015 at 12:25 PM, Pravin B Shelar <pshelar@nicira.com> wrote:
>>>>>> Following patch series adds support for Stateless Transport
>>>>>> Tunneling protocol.
>>>>>> STT uses TCP segmentation offload available in most of NIC. On
>>>>>> packet xmit STT driver appends STT header along with TCP header
>>>>>> to the packet. For GSO packet GSO parameters are set according
>>>>>> to tunnel configuration and packet is handed over to networking
>>>>>> stack. This allows use of segmentation offload available in NICs
>>>>>>
>>>>>> Netperf unidirectional test gives ~9.4 Gbits/s performance on 10Gbit
>>>>>> NIC with 1500 byte MTU with two TCP streams.
>>>>>>
>>>>> The reason you're able to get 9.4 Gbit/s with an L2 encapsulation
>>>>> using STT is that it has less protocol overhead per packet when doing
>>>>> segmentation compared to VXLAN (without segmentation STT packets will
>>>>> have more overhead than VXLAN).
>>>>>
>>>>> A VXLAN packet with TCP/IP has headers
>>>>> IP|UDP|VXLAN|Ethernet|IP|TCP+options. Assuming TCP is stuffed with
>>>>> options, this is 20+8+8+16+20+40=112 bytes, or 7.4% MTU. Each STT
>>>>> segment created in GSO, other than the first, has just IP|TCP headers
>>>>> which is 20+20=40 bytes or 2.6% MTU. So this explains throughput
>>>>> differences between VXLAN and STT.
>>>>
>>>> Tom, what performance do you see with a single stream of VXLAN running
>>>> on net-next with default configuration? The difference in numbers
>>>> being posted here is greater than a few percent caused by protocol
>>>> overheard.
>>>
>>> Please look at the data I posted with the VXLAN RCO patches.
>>
>> The data you posted uses 200 streams, so I assume that you are using
>> multiple CPUs. It's not surprising that you would be able to consume a
>> 10G link in that case. STT can do this with a single stream and less
>> than 1 core (or alternately handle higher throughput). Claiming that
>> since both can hit 10G they are same is not accurate.
>>
>> Discussing performance like this seems a little silly given that the
>> code is available. Pravin posted some numbers that he got, if you want
>> to dispute them then why don't you just try running it?
>
> Because you haven't provided network interface like I already
> requested twice, and I really don't have time or motivation to do
> development on your patches or figure out how to do this with OVS. If
> you want me to test your patches re-spin them with a network interface
> included.

I am woking in it. I will have patches by next week.

Re: [PATCH net-next 0/3] openvswitch: Add STT support.

[Tom Herbert]

Here is my data running STT and comparing against VXLAN, GUE, and
native (no encapsulation). This is using bnx2x.

TCP_STREAM IPv4 1 connection
  STT
    2.33% TX CPU utilization
    3.63% RX CPU utilization
    8790 Mbps
  GRE/GUE (RCO, GSO, GRO)
    5.55% TX CPU utilization
    6.67% RX CPU utilization
    9106 Mbps
  VXLAN (RCO, GSO, GRO)
    5.07% TX CPU utilization
    5.95% RX CPU utilization
    9061 Mbps
  Native (no ecapsulation)
    2.48% TX CPU utilization
    3.96% RX CPU utilization
    9411 Mbps

TCP_STREAM IPv4 200 connections
  STT
    8.36% TX CPU utilization
    9.29% RX CPU utilization
    9389 Mbps
  GUE/GRE (RCO, GSO, GRO)
    9.98% TX CPU utilization
    13.69% RX CPU utilization
    9132 Mbps
  VXLAN (RCO, GSO, GRO)
    11.79% TX CPU utilization
    13.88% RX CPU utilization
    9093 Mbps
  Native (no ecapsulation)
    6.05% TX CPU utilization
    7.27% RX CPU utilization
    9378 Mbps

TCP_RR IPv4 200 connections
  STT
    41.91% CPU utilization
    345/406/594 90/95/99% latencies
    562560 tps
  GUE/GRE (RCO, GSO, GRO)
    93.03% CPU utilization
    160/255/467 90/95/99% latencies
    1.1401e+06 tps
  VXLAN (RCO, GSO, GRO)
    93.73% CPU utilization
    157/249/448 90/95/99% latencies
    1.15655e+06 tps
  Native (no ecapsulation)
    87.46% CPU utilization
    120/178/292 90/95/99% latencies
    1.51747e+06 tps