GRO: very bad routing performance with 3.14.4 for GRE-packets

GRO: very bad routing performance with 3.14.4 for GRE-packets

[Wolfgang Walter]

Hello,

I upgraded a router from 3.10.46 to 3.14.4 and I see a dramatic perfomance 
loss for GRE-pakets if (and only if) GRO is enabled on the incoming or 
outgoing interface (drops to 1/50th to 1/100th).

Other traffic is just fine.

The router itself is not endpoint of the GRE-tunnel.

The router has connection tracking enabled and does NAT (but most of the 
traffic is not NATed nor are GRE-packets).

The network card is an Intel Corporation I350 Gigabit (both interfaces).

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: GRO: very bad routing performance with 3.14.4 for GRE-packets

[Eric Dumazet]

On Tue, 2014-07-08 at 22:35 +0200, Wolfgang Walter wrote:
> Hello,
> 
> I upgraded a router from 3.10.46 to 3.14.4 and I see a dramatic perfomance 
> loss for GRE-pakets if (and only if) GRO is enabled on the incoming or 
> outgoing interface (drops to 1/50th to 1/100th).
> 
> Other traffic is just fine.
> 
> The router itself is not endpoint of the GRE-tunnel.
> 
> The router has connection tracking enabled and does NAT (but most of the 
> traffic is not NATed nor are GRE-packets).
> 
> The network card is an Intel Corporation I350 Gigabit (both interfaces).
> 
> Regards,

GRO is a win if the consumer of the packets understands GSO packets.

In this case, Intel driver is not able to send GSO packets with GRE
encapsulation, so they need to be segmented before being sent.

That's extra work and not worth it, as you noticed.

One way to get good performance would be to implement a software TSO in
the Intel driver for the GRE encapsulated packets.

You might revert bf5a755f5e9186406bbf50f4087100af5bd68e40, or submit a
patch to make the GRE handling in GRO an optional feature.

Re: GRO: very bad routing performance with 3.14.4 for GRE-packets

[Jerry Chu]

On Tue, Jul 8, 2014 at 3:44 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Tue, 2014-07-08 at 22:35 +0200, Wolfgang Walter wrote:
>> Hello,
>>
>> I upgraded a router from 3.10.46 to 3.14.4 and I see a dramatic perfomance
>> loss for GRE-pakets if (and only if) GRO is enabled on the incoming or
>> outgoing interface (drops to 1/50th to 1/100th).
>>
>> Other traffic is just fine.
>>
>> The router itself is not endpoint of the GRE-tunnel.
>>
>> The router has connection tracking enabled and does NAT (but most of the
>> traffic is not NATed nor are GRE-packets).
>>
>> The network card is an Intel Corporation I350 Gigabit (both interfaces).
>>
>> Regards,
>
> GRO is a win if the consumer of the packets understands GSO packets.
>
> In this case, Intel driver is not able to send GSO packets with GRE
> encapsulation, so they need to be segmented before being sent.
>
> That's extra work and not worth it, as you noticed.
>
> One way to get good performance would be to implement a software TSO in
> the Intel driver for the GRE encapsulated packets.
>
> You might revert bf5a755f5e9186406bbf50f4087100af5bd68e40, or submit a
> patch to make the GRE handling in GRO an optional feature.
>

There is also a bug to the forwarding path introduced by my GRE-GRO
patch that I discovered a while back and submitted a fix to it. Please
see http://patchwork.ozlabs.org/patch/324953/ and let me know if that
addresses the problem you've seen. Sorry I have not got the time to
follow through with the patch.

Re: GRO: very bad routing performance with 3.14.4 for GRE-packets

[Wolfgang Walter]

Am Dienstag, 8. Juli 2014, 17:19:33 schrieb Jerry Chu:
> On Tue, Jul 8, 2014 at 3:44 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > On Tue, 2014-07-08 at 22:35 +0200, Wolfgang Walter wrote:
> >> Hello,
> >> 
> >> I upgraded a router from 3.10.46 to 3.14.4 and I see a dramatic
> >> perfomance
> >> loss for GRE-pakets if (and only if) GRO is enabled on the incoming or
> >> outgoing interface (drops to 1/50th to 1/100th).
> >> 
> >> Other traffic is just fine.
> >> 
> >> The router itself is not endpoint of the GRE-tunnel.
> >> 
> >> The router has connection tracking enabled and does NAT (but most of the
> >> traffic is not NATed nor are GRE-packets).
> >> 
> >> The network card is an Intel Corporation I350 Gigabit (both interfaces).
> >> 
> >> Regards,
> > 
> > GRO is a win if the consumer of the packets understands GSO packets.
> > 
> > In this case, Intel driver is not able to send GSO packets with GRE
> > encapsulation, so they need to be segmented before being sent.
> > 
> > That's extra work and not worth it, as you noticed.
> > 
> > One way to get good performance would be to implement a software TSO in
> > the Intel driver for the GRE encapsulated packets.
> > 
> > You might revert bf5a755f5e9186406bbf50f4087100af5bd68e40, or submit a
> > patch to make the GRE handling in GRO an optional feature.
> 
> There is also a bug to the forwarding path introduced by my GRE-GRO
> patch that I discovered a while back and submitted a fix to it. Please
> see http://patchwork.ozlabs.org/patch/324953/ and let me know if that
> addresses the problem you've seen. Sorry I have not got the time to
> follow through with the patch.

Yes, this patch fixes the problem.

Is there any chance this will make it into 3.14, soon?

I would recommend to revert bf5a755f5e9186406bbf50f4087100af5bd68e40 in 3.14 
for now and reapply it when this fix is available as this is a real 
regression.

Thanks for your help.

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: GRO: very bad routing performance with 3.14.4 for GRE-packets

[Jerry Chu]

On Thu, Jul 10, 2014 at 7:34 AM, Wolfgang Walter <linux@stwm.de> wrote:
> Am Dienstag, 8. Juli 2014, 17:19:33 schrieb Jerry Chu:
>> On Tue, Jul 8, 2014 at 3:44 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> > On Tue, 2014-07-08 at 22:35 +0200, Wolfgang Walter wrote:
>> >> Hello,
>> >>
>> >> I upgraded a router from 3.10.46 to 3.14.4 and I see a dramatic
>> >> perfomance
>> >> loss for GRE-pakets if (and only if) GRO is enabled on the incoming or
>> >> outgoing interface (drops to 1/50th to 1/100th).
>> >>
>> >> Other traffic is just fine.
>> >>
>> >> The router itself is not endpoint of the GRE-tunnel.
>> >>
>> >> The router has connection tracking enabled and does NAT (but most of the
>> >> traffic is not NATed nor are GRE-packets).
>> >>
>> >> The network card is an Intel Corporation I350 Gigabit (both interfaces).
>> >>
>> >> Regards,
>> >
>> > GRO is a win if the consumer of the packets understands GSO packets.
>> >
>> > In this case, Intel driver is not able to send GSO packets with GRE
>> > encapsulation, so they need to be segmented before being sent.
>> >
>> > That's extra work and not worth it, as you noticed.
>> >
>> > One way to get good performance would be to implement a software TSO in
>> > the Intel driver for the GRE encapsulated packets.
>> >
>> > You might revert bf5a755f5e9186406bbf50f4087100af5bd68e40, or submit a
>> > patch to make the GRE handling in GRO an optional feature.
>>
>> There is also a bug to the forwarding path introduced by my GRE-GRO
>> patch that I discovered a while back and submitted a fix to it. Please
>> see http://patchwork.ozlabs.org/patch/324953/ and let me know if that
>> addresses the problem you've seen. Sorry I have not got the time to
>> follow through with the patch.
>
> Yes, this patch fixes the problem.
>
> Is there any chance this will make it into 3.14, soon?

Will try to resubmit the patch soon.

>
> I would recommend to revert bf5a755f5e9186406bbf50f4087100af5bd68e40 in 3.14
> for now and reapply it when this fix is available as this is a real
> regression.
>
> Thanks for your help.
>
> Regards,
> --
> Wolfgang Walter
> Studentenwerk München
> Anstalt des öffentlichen Rechts