A question on the design of OVS GRE tunnel

A question on the design of OVS GRE tunnel

[Cong Wang]

Hi, Jesse, Pravin

I have a question on the design of OVS GRE tunnel. Why OVS GRE tunnel
doesn't register a netdev? I understand it is enough for GRE to function
without registering a netdev, just a GRE vport is sufficient and
probably even simpler.

However, I noticed there is some problem with such design:

I saw very bad performance with the _default_ setup with OVS GRE. After
digging it a little bit, clearly the cause is that OVS GRE tunnel adds
an outer IP header and a GRE header for every packet that passed to it,
which could result in a packet whose length is larger than the MTU of
the uplink, therefore after the packet goes through OVS, it has to be
fragmented by IP before going to the wire.

Of course we can workaround this problem by:

1) lower down the MTU of the first net device to reserve some room for
GRE header

2) pass vnet_hdr=on to KVM guests so that packets going out there are
still GSO packets even on hosts (I never try this, just analysis it by
reading code)

Do we have to live with this? Can we solve this problem at OVS layer? So
that no matter how large the packets are we can avoid IP fragmentation?
One solution in my mind is registering a netdev for OVS GRE tunnel too,
so that we could probably reuse GRO cells, thus packets can be merged
before OVS processing them, and it will be segmented again before going
to the wire. But I could easily miss something here. ;)

What do you think? Any better idea to fix this?

Thanks!

Re: A question on the design of OVS GRE tunnel

[Pravin Shelar]

On Mon, Jul 8, 2013 at 2:51 AM, Cong Wang <amwang@redhat.com> wrote:
> Hi, Jesse, Pravin
>
> I have a question on the design of OVS GRE tunnel. Why OVS GRE tunnel
> doesn't register a netdev? I understand it is enough for GRE to function
> without registering a netdev, just a GRE vport is sufficient and
> probably even simpler.
>
kernel-gre device has gre-parameters/state associated with it and
ovs-gre-vport is completely stateless. ovs-gre state is in user-space
which make kernel module alot simpler. Therefore I doubt it will be
easy or simpler to use netdev at this point.

> However, I noticed there is some problem with such design:
>
> I saw very bad performance with the _default_ setup with OVS GRE. After
> digging it a little bit, clearly the cause is that OVS GRE tunnel adds
> an outer IP header and a GRE header for every packet that passed to it,
> which could result in a packet whose length is larger than the MTU of
> the uplink, therefore after the packet goes through OVS, it has to be
> fragmented by IP before going to the wire.
>
I do not understand what do you mean, gre packets greater than MTU
must be fragmented before sent on wire and it is done by GRE-GSO code.

> Of course we can workaround this problem by:
>
> 1) lower down the MTU of the first net device to reserve some room for
> GRE header
>
> 2) pass vnet_hdr=on to KVM guests so that packets going out there are
> still GSO packets even on hosts (I never try this, just analysis it by
> reading code)
>
> Do we have to live with this? Can we solve this problem at OVS layer? So
> that no matter how large the packets are we can avoid IP fragmentation?
> One solution in my mind is registering a netdev for OVS GRE tunnel too,
> so that we could probably reuse GRO cells, thus packets can be merged
> before OVS processing them, and it will be segmented again before going
> to the wire. But I could easily miss something here. ;)
>
> What do you think? Any better idea to fix this?
>
> Thanks!
>

Re: A question on the design of OVS GRE tunnel

[Cong Wang]

On Mon, 2013-07-08 at 09:28 -0700, Pravin Shelar wrote:
> On Mon, Jul 8, 2013 at 2:51 AM, Cong Wang <amwang@redhat.com> wrote:
> > Hi, Jesse, Pravin
> >
> > I have a question on the design of OVS GRE tunnel. Why OVS GRE tunnel
> > doesn't register a netdev? I understand it is enough for GRE to function
> > without registering a netdev, just a GRE vport is sufficient and
> > probably even simpler.
> >
> kernel-gre device has gre-parameters/state associated with it and
> ovs-gre-vport is completely stateless. ovs-gre state is in user-space
> which make kernel module alot simpler. Therefore I doubt it will be
> easy or simpler to use netdev at this point.

Understood, from users' point of view, it is simpler. At least no one is
able to assign any IP address to it.

> 
> > However, I noticed there is some problem with such design:
> >
> > I saw very bad performance with the _default_ setup with OVS GRE. After
> > digging it a little bit, clearly the cause is that OVS GRE tunnel adds
> > an outer IP header and a GRE header for every packet that passed to it,
> > which could result in a packet whose length is larger than the MTU of
> > the uplink, therefore after the packet goes through OVS, it has to be
> > fragmented by IP before going to the wire.
> >
> I do not understand what do you mean, gre packets greater than MTU
> must be fragmented before sent on wire and it is done by GRE-GSO code.
> 

Well, I said fragment, not segment. This is exactly why performance is
so bad.

In my _default_ setup, every net device on the path has MTU=1500,
therefore, the packets coming out of a KVM guest can have length=1500,
after they go through OVS GRE tunnel, their length becomes 1538 because
of the added GRE header and IP header.

After that, since the packets are not GSO (unless you pass vnet_hdr=on
to KVM guest), the packets with length=1538 will be _fragmented_ by IP
layer, since the dest uplink has MTU=1500 too. This is why I proposed to
reuse GRO cell to merge the packets, which requires a netdev...

This is the problem.

Re: A question on the design of OVS GRE tunnel

[Jesse Gross]

On Mon, Jul 8, 2013 at 7:41 PM, Cong Wang <amwang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> On Mon, 2013-07-08 at 09:28 -0700, Pravin Shelar wrote:
>> On Mon, Jul 8, 2013 at 2:51 AM, Cong Wang <amwang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> > However, I noticed there is some problem with such design:
>> >
>> > I saw very bad performance with the _default_ setup with OVS GRE. After
>> > digging it a little bit, clearly the cause is that OVS GRE tunnel adds
>> > an outer IP header and a GRE header for every packet that passed to it,
>> > which could result in a packet whose length is larger than the MTU of
>> > the uplink, therefore after the packet goes through OVS, it has to be
>> > fragmented by IP before going to the wire.
>> >
>> I do not understand what do you mean, gre packets greater than MTU
>> must be fragmented before sent on wire and it is done by GRE-GSO code.
>>
>
> Well, I said fragment, not segment. This is exactly why performance is
> so bad.
>
> In my _default_ setup, every net device on the path has MTU=1500,
> therefore, the packets coming out of a KVM guest can have length=1500,
> after they go through OVS GRE tunnel, their length becomes 1538 because
> of the added GRE header and IP header.
>
> After that, since the packets are not GSO (unless you pass vnet_hdr=on
> to KVM guest), the packets with length=1538 will be _fragmented_ by IP
> layer, since the dest uplink has MTU=1500 too. This is why I proposed to
> reuse GRO cell to merge the packets, which requires a netdev...

Large packets coming from a modern KVM guest will use TSO because this
is a huge performance win regardless of whether any tunneling is used.
It doesn't make any sense for the guest IP stack to take a stream of
packets, split them apart, merge them in the hypervisor stack, and
split them again before transmission. Any packets potentially worth
merging will almost certainly have originated as a single buffer in
the guest, so we should keep them together all the way from the guest
to the GSO/TSO layer.

The real problem is that the requested MSS size is not correct. In the
"best" situation we would first segment the packet to the requested
size, add the tunnel headers, and then fragment. However, it looks to
me like the original size is being carried all the way to the GSO
code, which will then generate packets that are greater than the MTU.
Both of these can likely be improved upon by either convincing the
guest to automatically use a lower MSS or adjusting it ourselves.
X-CudaMail-Whitelist-To: dev-yBygre7rU0TnMu66kgdUjQ@public.gmane.org

Re: A question on the design of OVS GRE tunnel

[Cong Wang]

On Mon, 2013-07-08 at 23:26 -0700, Jesse Gross wrote:
> On Mon, Jul 8, 2013 at 7:41 PM, Cong Wang <amwang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > On Mon, 2013-07-08 at 09:28 -0700, Pravin Shelar wrote:
> >> On Mon, Jul 8, 2013 at 2:51 AM, Cong Wang <amwang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >> > However, I noticed there is some problem with such design:
> >> >
> >> > I saw very bad performance with the _default_ setup with OVS GRE. After
> >> > digging it a little bit, clearly the cause is that OVS GRE tunnel adds
> >> > an outer IP header and a GRE header for every packet that passed to it,
> >> > which could result in a packet whose length is larger than the MTU of
> >> > the uplink, therefore after the packet goes through OVS, it has to be
> >> > fragmented by IP before going to the wire.
> >> >
> >> I do not understand what do you mean, gre packets greater than MTU
> >> must be fragmented before sent on wire and it is done by GRE-GSO code.
> >>
> >
> > Well, I said fragment, not segment. This is exactly why performance is
> > so bad.
> >
> > In my _default_ setup, every net device on the path has MTU=1500,
> > therefore, the packets coming out of a KVM guest can have length=1500,
> > after they go through OVS GRE tunnel, their length becomes 1538 because
> > of the added GRE header and IP header.
> >
> > After that, since the packets are not GSO (unless you pass vnet_hdr=on
> > to KVM guest), the packets with length=1538 will be _fragmented_ by IP
> > layer, since the dest uplink has MTU=1500 too. This is why I proposed to
> > reuse GRO cell to merge the packets, which requires a netdev...
> 
> Large packets coming from a modern KVM guest will use TSO because this
> is a huge performance win regardless of whether any tunneling is used.
> It doesn't make any sense for the guest IP stack to take a stream of
> packets, split them apart, merge them in the hypervisor stack, and
> split them again before transmission. Any packets potentially worth
> merging will almost certainly have originated as a single buffer in
> the guest, so we should keep them together all the way from the guest
> to the GSO/TSO layer.
> 
> The real problem is that the requested MSS size is not correct. In the
> "best" situation we would first segment the packet to the requested
> size, add the tunnel headers, and then fragment. However, it looks to
> me like the original size is being carried all the way to the GSO
> code, which will then generate packets that are greater than the MTU.
> Both of these can likely be improved upon by either convincing the
> guest to automatically use a lower MSS or adjusting it ourselves.

Yeah, unfortunately this is not easy to discover, people need some
knowledge and some time to find the problem and "fix" it. This is why I
think we should find a way to fix it if possible, or at least document
it.

Thanks!