TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Kirby]

Hello!

We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards
(partno(BCM95721) rev 4201) with VLAN tags in use, to think that
everything looked great, but in fact...

LVS was receiving magically-merged TCP packets which it tried to forward
on to the real server, only to get annoyed at itself for trying to
forward a packet bigger than the device MTU:

IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081>
IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098>
IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556

This caused packet loss for any merged frames, which caused abysmal
performance for uploads via the LVS server.  Local performance to or
from the box is still fine, because the stack doesn't care, only the
forwarding part of LVS is running into the problem.

Furthermore, disabling _everything_ reported by ethtool -k doesn't seem
to change the result, even if I down/up the interface after, and even if
I try on every single interface including the VLANned ones.  This seems
to be another bug.  Reverting to 2.6.35 makes it all work again.

Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653

So how should this be fixed?  Should LVS be taught to fragment, or must
we disable the merging in this case?  It seems like it would work well if
the sending side could do the same offload in reverse, but I'm not sure
if that would be possible.

Simon-

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Eric Dumazet]

Le vendredi 03 décembre 2010 à 02:34 -0800, Simon Kirby a écrit :
> Hello!
> 
> We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards
> (partno(BCM95721) rev 4201) with VLAN tags in use, to think that
> everything looked great, but in fact...
> 
> LVS was receiving magically-merged TCP packets which it tried to forward
> on to the real server, only to get annoyed at itself for trying to
> forward a packet bigger than the device MTU:
> 
> IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081>
> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098>
> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
> 

Hi Simon

This is a tcpdump on A ?
Could you take it also on B ?

tcpdump displays large buffers, but they should be split (of course)
when sent on wire.

> This caused packet loss for any merged frames, which caused abysmal
> performance for uploads via the LVS server.  Local performance to or
> from the box is still fine, because the stack doesn't care, only the
> forwarding part of LVS is running into the problem.
> 
> Furthermore, disabling _everything_ reported by ethtool -k doesn't seem
> to change the result, even if I down/up the interface after, and even if
> I try on every single interface including the VLANned ones.  This seems
> to be another bug.  Reverting to 2.6.35 makes it all work again.
> 
> Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653
> 
> So how should this be fixed?  Should LVS be taught to fragment, or must
> we disable the merging in this case?  It seems like it would work well if
> the sending side could do the same offload in reverse, but I'm not sure
> if that would be possible.
> 
> Simon-


I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Horman]

On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:
> Le vendredi 03 décembre 2010 à 02:34 -0800, Simon Kirby a écrit :
> > Hello!
> > 
> > We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards
> > (partno(BCM95721) rev 4201) with VLAN tags in use, to think that
> > everything looked great, but in fact...
> > 
> > LVS was receiving magically-merged TCP packets which it tried to forward
> > on to the real server, only to get annoyed at itself for trying to
> > forward a packet bigger than the device MTU:
> > 
> > IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081>
> > IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098>
> > IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
> > 
> 
> Hi Simon
> 
> This is a tcpdump on A ?
> Could you take it also on B ?
> 
> tcpdump displays large buffers, but they should be split (of course)
> when sent on wire.
> 
> > This caused packet loss for any merged frames, which caused abysmal
> > performance for uploads via the LVS server.  Local performance to or
> > from the box is still fine, because the stack doesn't care, only the
> > forwarding part of LVS is running into the problem.
> > 
> > Furthermore, disabling _everything_ reported by ethtool -k doesn't seem
> > to change the result, even if I down/up the interface after, and even if
> > I try on every single interface including the VLANned ones.  This seems
> > to be another bug.  Reverting to 2.6.35 makes it all work again.
> > 
> > Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653
> > 
> > So how should this be fixed?  Should LVS be taught to fragment, or must
> > we disable the merging in this case?  It seems like it would work well if
> > the sending side could do the same offload in reverse, but I'm not sure
> > if that would be possible.
> > 
> > Simon-
> 
> 
> I believe Simon Horman has some patches for GRO and LVS.
> 
> Please send the results of "ethtool -k eth0" on all your nics / vlans ?
> 
> For TSO, I am not sure why and where it could matter...

There is a patch to teach LVS how to cope with GRO in nf-next-2.6
and I expect it to be included in 2.6.38. The patch is "ipvs: allow
transmit of GRO aggregated skbs" and perhaps it should be considered
for 2.6.37 and stable. In general the work around is to disable GRO.

The patch does not resolve the incompatibility of LVS with LRO.
The work around there is to disable LRO. I'm not entirely sure
how to teach LVS to disable LRO automatically, or if its desirable.

Simon, you mention that you disabled everything with ethtool, but the
tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is
active. So perhaps as you speculate that is a bug

I will prepare a backport of the "ipvs: allow transmit of GRO aggregated
skbs" patch to v2.6.36 and post it shortly.  Testing to see if that
resolves the problem that you are seeing would probably be a good start.

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Horman]

[ CCed lvs-devel, Julian Anastasov and Herbert Xu ]

On Fri, Dec 03, 2010 at 09:36:19PM +0900, Simon Horman wrote:
> On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:
> > Le vendredi 03 décembre 2010 à 02:34 -0800, Simon Kirby a écrit :
> > > Hello!
> > > 
> > > We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards
> > > (partno(BCM95721) rev 4201) with VLAN tags in use, to think that
> > > everything looked great, but in fact...
> > > 
> > > LVS was receiving magically-merged TCP packets which it tried to forward
> > > on to the real server, only to get annoyed at itself for trying to
> > > forward a packet bigger than the device MTU:
> > > 
> > > IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081>
> > > IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098>
> > > IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
> > > 
> > 
> > Hi Simon
> > 
> > This is a tcpdump on A ?
> > Could you take it also on B ?
> > 
> > tcpdump displays large buffers, but they should be split (of course)
> > when sent on wire.
> > 
> > > This caused packet loss for any merged frames, which caused abysmal
> > > performance for uploads via the LVS server.  Local performance to or
> > > from the box is still fine, because the stack doesn't care, only the
> > > forwarding part of LVS is running into the problem.
> > > 
> > > Furthermore, disabling _everything_ reported by ethtool -k doesn't seem
> > > to change the result, even if I down/up the interface after, and even if
> > > I try on every single interface including the VLANned ones.  This seems
> > > to be another bug.  Reverting to 2.6.35 makes it all work again.
> > > 
> > > Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653
> > > 
> > > So how should this be fixed?  Should LVS be taught to fragment, or must
> > > we disable the merging in this case?  It seems like it would work well if
> > > the sending side could do the same offload in reverse, but I'm not sure
> > > if that would be possible.
> > > 
> > > Simon-
> > 
> > 
> > I believe Simon Horman has some patches for GRO and LVS.
> > 
> > Please send the results of "ethtool -k eth0" on all your nics / vlans ?
> > 
> > For TSO, I am not sure why and where it could matter...
> 
> There is a patch to teach LVS how to cope with GRO in nf-next-2.6
> and I expect it to be included in 2.6.38. The patch is "ipvs: allow
> transmit of GRO aggregated skbs" and perhaps it should be considered
> for 2.6.37 and stable. In general the work around is to disable GRO.
> 
> The patch does not resolve the incompatibility of LVS with LRO.
> The work around there is to disable LRO. I'm not entirely sure
> how to teach LVS to disable LRO automatically, or if its desirable.
> 
> Simon, you mention that you disabled everything with ethtool, but the
> tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is
> active. So perhaps as you speculate that is a bug
> 
> I will prepare a backport of the "ipvs: allow transmit of GRO aggregated
> skbs" patch to v2.6.36 and post it shortly.  Testing to see if that
> resolves the problem that you are seeing would probably be a good start.

Here is the patch for v2.6.36.

From: Simon Horman <horms@verge.net.au>

ipvs: allow transmit of GRO aggregated skbs

Attempt at allowing LVS to transmit skbs of greater than MTU length that
have been aggregated by GRO and can thus be deaggregated by GSO.

Cc: Julian Anastasov <ja@ssi.bg>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Simon Horman <horms@verge.net.au>

---
 net/netfilter/ipvs/ip_vs_xmit.c |   25 +++++++++++++++----------
 1 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index 49df6be..577f502 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -247,7 +247,8 @@ ip_vs_bypass_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF))) {
+	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF)) &&
+	    !skb_is_gso(skb)) {
 		ip_rt_put(rt);
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -311,7 +312,7 @@ ip_vs_bypass_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if (skb->len > mtu) {
+	if (skb->len > mtu && !skb_is_gso(skb)) {
 		dst_release(&rt->dst);
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -408,7 +409,8 @@ ip_vs_nat_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF))) {
+	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF)) &&
+	    !skb_is_gso(skb)) {
 		ip_rt_put(rt);
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		IP_VS_DBG_RL_PKT(0, pp, skb, 0, "ip_vs_nat_xmit(): frag needed for");
@@ -486,7 +488,7 @@ ip_vs_nat_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if (skb->len > mtu) {
+	if (skb->len > mtu && !skb_is_gso(skb)) {
 		dst_release(&rt->dst);
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		IP_VS_DBG_RL_PKT(0, pp, skb, 0,
@@ -597,8 +599,8 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	df |= (old_iph->frag_off & htons(IP_DF));
 
-	if ((old_iph->frag_off & htons(IP_DF))
-	    && mtu < ntohs(old_iph->tot_len)) {
+	if ((old_iph->frag_off & htons(IP_DF) &&
+	    mtu < ntohs(old_iph->tot_len) && !skb_is_gso(skb))) {
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		ip_rt_put(rt);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -707,7 +709,8 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 	if (skb_dst(skb))
 		skb_dst(skb)->ops->update_pmtu(skb_dst(skb), mtu);
 
-	if (mtu < ntohs(old_iph->payload_len) + sizeof(struct ipv6hdr)) {
+	if (mtu < ntohs(old_iph->payload_len) + sizeof(struct ipv6hdr) &&
+	    !skb_is_gso(skb)) {
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		dst_release(&rt->dst);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -796,7 +799,8 @@ ip_vs_dr_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((iph->frag_off & htons(IP_DF)) && skb->len > mtu) {
+	if ((iph->frag_off & htons(IP_DF)) && skb->len > mtu &&
+	    !skb_is_gso(skb)) {
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		ip_rt_put(rt);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -924,7 +928,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((skb->len > mtu) && (ip_hdr(skb)->frag_off & htons(IP_DF))) {
+	if ((skb->len > mtu) && (ip_hdr(skb)->frag_off & htons(IP_DF)) &&
+	    !skb_is_gso(skb)) {
 		ip_rt_put(rt);
 		icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED, htonl(mtu));
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -999,7 +1004,7 @@ ip_vs_icmp_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if (skb->len > mtu) {
+	if (skb->len > mtu && !skb_is_gso(skb)) {
 		dst_release(&rt->dst);
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
-- 
1.7.2.3

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Ben Hutchings]

On Fri, 2010-12-03 at 21:36 +0900, Simon Horman wrote:
> On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:
[...]
> > I believe Simon Horman has some patches for GRO and LVS.
> > 
> > Please send the results of "ethtool -k eth0" on all your nics / vlans ?
> > 
> > For TSO, I am not sure why and where it could matter...
> 
> There is a patch to teach LVS how to cope with GRO in nf-next-2.6
> and I expect it to be included in 2.6.38. The patch is "ipvs: allow
> transmit of GRO aggregated skbs" and perhaps it should be considered
> for 2.6.37 and stable. In general the work around is to disable GRO.
>
> The patch does not resolve the incompatibility of LVS with LRO.
> The work around there is to disable LRO. I'm not entirely sure
> how to teach LVS to disable LRO automatically, or if its desirable.
[...]

I think it is desirable, and we already do it for bridging and
forwarding - we call dev_disable_lro() whenever a device is added to a
bridge or has forwarding enabled.  For IPVS it looks like you would have
to resolve the server IP address to a device first.  Though if the
administrator assigns the server address to a different interface later
then the failure mode will be very hard to understand.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Horman]

On Fri, Dec 03, 2010 at 01:27:19PM +0000, Ben Hutchings wrote:
> On Fri, 2010-12-03 at 21:36 +0900, Simon Horman wrote:
> > On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:
> [...]
> > > I believe Simon Horman has some patches for GRO and LVS.
> > > 
> > > Please send the results of "ethtool -k eth0" on all your nics / vlans ?
> > > 
> > > For TSO, I am not sure why and where it could matter...
> > 
> > There is a patch to teach LVS how to cope with GRO in nf-next-2.6
> > and I expect it to be included in 2.6.38. The patch is "ipvs: allow
> > transmit of GRO aggregated skbs" and perhaps it should be considered
> > for 2.6.37 and stable. In general the work around is to disable GRO.
> >
> > The patch does not resolve the incompatibility of LVS with LRO.
> > The work around there is to disable LRO. I'm not entirely sure
> > how to teach LVS to disable LRO automatically, or if its desirable.
> [...]
> 
> I think it is desirable, and we already do it for bridging and
> forwarding - we call dev_disable_lro() whenever a device is added to a
> bridge or has forwarding enabled.  For IPVS it looks like you would have
> to resolve the server IP address to a device first.  Though if the
> administrator assigns the server address to a different interface later
> then the failure mode will be very hard to understand.

Thanks, I'll try and make it so.

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Kirby]

On Fri, Dec 03, 2010 at 09:36:19PM +0900, Simon Horman wrote:

> On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:
> > Le vendredi 03 d??cembre 2010 ?? 02:34 -0800, Simon Kirby a ??crit :
> > > Hello!
> > > 
> > > We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards
> > > (partno(BCM95721) rev 4201) with VLAN tags in use, to think that
> > > everything looked great, but in fact...
> > > 
> > > LVS was receiving magically-merged TCP packets which it tried to forward
> > > on to the real server, only to get annoyed at itself for trying to
> > > forward a packet bigger than the device MTU:
> > > 
> > > IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081>
> > > IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098>
> > > IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
> > > 
> > 
> > Hi Simon
> > 
> > This is a tcpdump on A ?
> > Could you take it also on B ?
> > 
> > tcpdump displays large buffers, but they should be split (of course)
> > when sent on wire.

I assume you don't need this anymore since the problem was figured out?
Otherwise, let me know.

> > > This caused packet loss for any merged frames, which caused abysmal
> > > performance for uploads via the LVS server.  Local performance to or
> > > from the box is still fine, because the stack doesn't care, only the
> > > forwarding part of LVS is running into the problem.
> > > 
> > > Furthermore, disabling _everything_ reported by ethtool -k doesn't seem
> > > to change the result, even if I down/up the interface after, and even if
> > > I try on every single interface including the VLANned ones.  This seems
> > > to be another bug.  Reverting to 2.6.35 makes it all work again.
> > > 
> > > Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653
> > > 
> > > So how should this be fixed?  Should LVS be taught to fragment, or must
> > > we disable the merging in this case?  It seems like it would work well if
> > > the sending side could do the same offload in reverse, but I'm not sure
> > > if that would be possible.
> > > 
> > > Simon-
> > 
> > 
> > I believe Simon Horman has some patches for GRO and LVS.
> > 
> > Please send the results of "ethtool -k eth0" on all your nics / vlans ?
> > 
> > For TSO, I am not sure why and where it could matter...
> 
> There is a patch to teach LVS how to cope with GRO in nf-next-2.6
> and I expect it to be included in 2.6.38. The patch is "ipvs: allow
> transmit of GRO aggregated skbs" and perhaps it should be considered
> for 2.6.37 and stable. In general the work around is to disable GRO.
> 
> The patch does not resolve the incompatibility of LVS with LRO.
> The work around there is to disable LRO. I'm not entirely sure
> how to teach LVS to disable LRO automatically, or if its desirable.

Yeah, things will still break if the interfaces have mismatched GRO
support, etc.

It would be nice if it could be all smart about it, but I don't see how
this could be done sanely, as it doesn't know if the destination
interface could support GRO transmission until it has gone through the
LVS processing.  Would software GRO be faster on transmission be faster
than no GRO at all?

> Simon, you mention that you disabled everything with ethtool, but the
> tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is
> active. So perhaps as you speculate that is a bug

Right.  So, on 2.6.35, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off

# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off

On 2.6.36, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off

# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off

And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off,
it says "off", but I still see merged frames, as verified with an nc <
/dev/zero and tcpdump -i any -n 'length > 1500'.

> I will prepare a backport of the "ipvs: allow transmit of GRO aggregated
> skbs" patch to v2.6.36 and post it shortly.  Testing to see if that
> resolves the problem that you are seeing would probably be a good start.

I can test this in our case, and it should specifically work on the
servers we were seeing the problem on, but it still needs some
complicated logic to be safe for all cases (like when LVS decides to
route out an interface without GRO).  Anyway, thanks!

Simon-

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Eric Dumazet]

Le mercredi 08 décembre 2010 à 18:35 -0800, Simon Kirby a écrit :

> Right.  So, on 2.6.35, ethtool shows:
> 
> # ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp segmentation offload: on
> udp fragmentation offload: off
> generic segmentation offload: on
> large receive offload: off
> 
> # ethtool -k eth1.39
> Offload parameters for eth1.39:
> rx-checksumming: on
> tx-checksumming: off
> scatter-gather: off
> tcp segmentation offload: off
> udp fragmentation offload: off
> generic segmentation offload: off
> large receive offload: off
> 
> On 2.6.36, ethtool shows:
> 
> # ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp segmentation offload: on
> udp fragmentation offload: off
> generic segmentation offload: on
> large receive offload: off
> 
> # ethtool -k eth1.39
> Offload parameters for eth1.39:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp segmentation offload: on
> udp fragmentation offload: off
> generic segmentation offload: on
> large receive offload: off
> 
> And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off,
> it says "off", but I still see merged frames, as verified with an nc <
> /dev/zero and tcpdump -i any -n 'length > 1500'.
> 


Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)

> > I will prepare a backport of the "ipvs: allow transmit of GRO aggregated
> > skbs" patch to v2.6.36 and post it shortly.  Testing to see if that
> > resolves the problem that you are seeing would probably be a good start.
> 
> I can test this in our case, and it should specifically work on the
> servers we were seeing the problem on, but it still needs some
> complicated logic to be safe for all cases (like when LVS decides to
> route out an interface without GRO).  Anyway, thanks!
> 
> Simon-

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Kirby]

On Thu, Dec 09, 2010 at 03:51:59AM +0100, Eric Dumazet wrote:

> Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit :
> 
> > Right.  So, on 2.6.35, ethtool shows:
> > 
> > # ethtool -k eth1
> > Offload parameters for eth1:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp segmentation offload: on
> > udp fragmentation offload: off
> > generic segmentation offload: on
> > large receive offload: off
> > 
> > # ethtool -k eth1.39
> > Offload parameters for eth1.39:
> > rx-checksumming: on
> > tx-checksumming: off
> > scatter-gather: off
> > tcp segmentation offload: off
> > udp fragmentation offload: off
> > generic segmentation offload: off
> > large receive offload: off
> > 
> > On 2.6.36, ethtool shows:
> > 
> > # ethtool -k eth1
> > Offload parameters for eth1:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp segmentation offload: on
> > udp fragmentation offload: off
> > generic segmentation offload: on
> > large receive offload: off
> > 
> > # ethtool -k eth1.39
> > Offload parameters for eth1.39:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp segmentation offload: on
> > udp fragmentation offload: off
> > generic segmentation offload: on
> > large receive offload: off
> > 
> > And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off,
> > it says "off", but I still see merged frames, as verified with an nc <
> > /dev/zero and tcpdump -i any -n 'length > 1500'.
> 
> Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)

and turning it off makes no difference, either.  Even if I down/up the
interface after.  I still see merged packets on receive, even if I try to
use ethtool to turn off everything that is "on".  It's as if the new code
to enable it on vlans works, but the disable from ethtool does not.

In other words, I cannot find a way to disable it (and thus make LVS
work) on 2.6.36 or 2.6.37-rc6.

One interesting note is that if I destroy and recreate the VLAN interface,
it comes back with gso enabled:

# ethtool -K eth1 gso off
# ethtool -K eth1.39 gso off
# ethtool -K eth1 tso off
# ethtool -K eth1.39 tso off
# ifdown eth1.39
Removed VLAN -:eth1.39:-
# ifconfig eth1 down
# ifup eth1.39
Added VLAN with VID == 39 to IF -:eth1:-
# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -K eth1.39 gso off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off
# ifdown eth1.39; ifup eth1.39
Removed VLAN -:eth1.39:-
Added VLAN with VID == 39 to IF -:eth1:-
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off

So, the bug seems to related using it on VLANs.

Simon-

> > > I will prepare a backport of the "ipvs: allow transmit of GRO aggregated
> > > skbs" patch to v2.6.36 and post it shortly.  Testing to see if that
> > > resolves the problem that you are seeing would probably be a good start.
> > 
> > I can test this in our case, and it should specifically work on the
> > servers we were seeing the problem on, but it still needs some
> > complicated logic to be safe for all cases (like when LVS decides to
> > route out an interface without GRO).  Anyway, thanks!
> > 
> > Simon-
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Horman]

On Wed, Dec 22, 2010 at 03:39:48PM -0800, Simon Kirby wrote:
> On Thu, Dec 09, 2010 at 03:51:59AM +0100, Eric Dumazet wrote:
> 
> > Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit :
> > 
> > > Right.  So, on 2.6.35, ethtool shows:
> > > 
> > > # ethtool -k eth1
> > > Offload parameters for eth1:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > scatter-gather: on
> > > tcp segmentation offload: on
> > > udp fragmentation offload: off
> > > generic segmentation offload: on
> > > large receive offload: off
> > > 
> > > # ethtool -k eth1.39
> > > Offload parameters for eth1.39:
> > > rx-checksumming: on
> > > tx-checksumming: off
> > > scatter-gather: off
> > > tcp segmentation offload: off
> > > udp fragmentation offload: off
> > > generic segmentation offload: off
> > > large receive offload: off
> > > 
> > > On 2.6.36, ethtool shows:
> > > 
> > > # ethtool -k eth1
> > > Offload parameters for eth1:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > scatter-gather: on
> > > tcp segmentation offload: on
> > > udp fragmentation offload: off
> > > generic segmentation offload: on
> > > large receive offload: off
> > > 
> > > # ethtool -k eth1.39
> > > Offload parameters for eth1.39:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > scatter-gather: on
> > > tcp segmentation offload: on
> > > udp fragmentation offload: off
> > > generic segmentation offload: on
> > > large receive offload: off
> > > 
> > > And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off,
> > > it says "off", but I still see merged frames, as verified with an nc <
> > > /dev/zero and tcpdump -i any -n 'length > 1500'.
> > 
> > Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)
> 
> and turning it off makes no difference, either.  Even if I down/up the
> interface after.  I still see merged packets on receive, even if I try to
> use ethtool to turn off everything that is "on".  It's as if the new code
> to enable it on vlans works, but the disable from ethtool does not.
> 
> In other words, I cannot find a way to disable it (and thus make LVS
> work) on 2.6.36 or 2.6.37-rc6.

Hi Simon,

thanks for prodding me to respond to this post offline and sorry for not
responding earlier.

Firstly, I think that this is a receive-side problem so I don't believe
that GSO (generic segmentation offload) or other transmit-side options are
likely to have any affect.

My understanding is that on the receive-side there are two options which
when enabled can result in the behaviour that you describe.

* LRO (large receive offload)

  You have this disabled, and assuming it really is disabled it
  shouldn't be causing a problem.

* GRO (generic receive offload)

  This does not seem to be in the output of your ethtool commands at all.
  So I wonder if your ethtool is too old to support this option?

  In any case, I was able to reproduce the problem that you describe (or at
  least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
  the problem did not manifest when I disabled GRO on eth1.1.

N.B: I'm not sure that I have any hardware whose driver supports LRO
     so my tests were made without LRO support.

> One interesting note is that if I destroy and recreate the VLAN interface,
> it comes back with gso enabled:

I guess that relate to how the default is set.
I don't imagine that it is the cause of the problem that you are seeing.

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Kirby]

On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote:

> Hi Simon,
> 
> thanks for prodding me to respond to this post offline and sorry for not
> responding earlier.
> 
> Firstly, I think that this is a receive-side problem so I don't believe
> that GSO (generic segmentation offload) or other transmit-side options are
> likely to have any affect.
> 
> My understanding is that on the receive-side there are two options which
> when enabled can result in the behaviour that you describe.
> 
> * LRO (large receive offload)
> 
>   You have this disabled, and assuming it really is disabled it
>   shouldn't be causing a problem.
> 
> * GRO (generic receive offload)
> 
>   This does not seem to be in the output of your ethtool commands at all.
>   So I wonder if your ethtool is too old to support this option?

So, this was the case.  Our ethtool (lenny) was too old to see the GRO
option, only GSO.  Disabling GRO on eth1.39 has no effect, but disabling
it on eth1 caused it to stop receiving the merged frames, fixing the LVS
packet loss (due to no sending GSO support from LVS/IPVS).

Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? 

>   In any case, I was able to reproduce the problem that you describe (or at
>   least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
>   the problem did not manifest when I disabled GRO on eth1.1.

It worked for you to do ethtool -K eth1.1 gro off, then?  For me on
2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even
though packets arrive on eth1.39.

Also, strangely, 2.6.35.4's default state (with no received merged frames)
has GRO on for eth1 but off for eth1.39:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: off

# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off

If I set 2.6.37 to have all of the same options, I still see GRO frames
on 2.6.37 (tg3), which is weird.

Cheers,

Simon-

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Horman]

On Wed, Jan 26, 2011 at 04:48:05PM -0800, Simon Kirby wrote:
> On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote:
> 
> > Hi Simon,
> > 
> > thanks for prodding me to respond to this post offline and sorry for not
> > responding earlier.
> > 
> > Firstly, I think that this is a receive-side problem so I don't believe
> > that GSO (generic segmentation offload) or other transmit-side options are
> > likely to have any affect.
> > 
> > My understanding is that on the receive-side there are two options which
> > when enabled can result in the behaviour that you describe.
> > 
> > * LRO (large receive offload)
> > 
> >   You have this disabled, and assuming it really is disabled it
> >   shouldn't be causing a problem.
> > 
> > * GRO (generic receive offload)
> > 
> >   This does not seem to be in the output of your ethtool commands at all.
> >   So I wonder if your ethtool is too old to support this option?
> 
> So, this was the case.  Our ethtool (lenny) was too old to see the GRO
> option, only GSO.  Disabling GRO on eth1.39 has no effect, but disabling
> it on eth1 caused it to stop receiving the merged frames, fixing the LVS
> packet loss (due to no sending GSO support from LVS/IPVS).
> 
> Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? 

The patch for IPVS GRO support has been merged and should appear in 2.6.39.
This is somewhat later than I previously anticipated due to a merge mix-up
on my part.

> >   In any case, I was able to reproduce the problem that you describe (or at
> >   least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
> >   the problem did not manifest when I disabled GRO on eth1.1.
> 
> It worked for you to do ethtool -K eth1.1 gro off, then?  For me on
> 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even
> though packets arrive on eth1.39.

I will recheck my results, but in general I think it is a bit
of an open question as to how ethtool settings should be propagated
between related devices.

> Also, strangely, 2.6.35.4's default state (with no received merged frames)
> has GRO on for eth1 but off for eth1.39:
> 
> # ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> 
> # ethtool -k eth1.39
> Offload parameters for eth1.39:
> rx-checksumming: on
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> 
> If I set 2.6.37 to have all of the same options, I still see GRO frames
> on 2.6.37 (tg3), which is weird.

Yes that is a weird.

There has been quite a lot of work on VLANs recently and
I suspect that the behaviour that you are observing with 2.6.37
is a regression that occurred during that work. It would
be good to fix things to restore the 2.6.35 behaviour.

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Eric Dumazet]

Le mercredi 26 janvier 2011 à 16:48 -0800, Simon Kirby a écrit :
> On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote:
> 
> > Hi Simon,
> > 
> > thanks for prodding me to respond to this post offline and sorry for not
> > responding earlier.
> > 
> > Firstly, I think that this is a receive-side problem so I don't believe
> > that GSO (generic segmentation offload) or other transmit-side options are
> > likely to have any affect.
> > 
> > My understanding is that on the receive-side there are two options which
> > when enabled can result in the behaviour that you describe.
> > 
> > * LRO (large receive offload)
> > 
> >   You have this disabled, and assuming it really is disabled it
> >   shouldn't be causing a problem.
> > 
> > * GRO (generic receive offload)
> > 
> >   This does not seem to be in the output of your ethtool commands at all.
> >   So I wonder if your ethtool is too old to support this option?
> 
> So, this was the case.  Our ethtool (lenny) was too old to see the GRO
> option, only GSO.  Disabling GRO on eth1.39 has no effect, but disabling
> it on eth1 caused it to stop receiving the merged frames, fixing the LVS
> packet loss (due to no sending GSO support from LVS/IPVS).
> 
> Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? 
> 
> >   In any case, I was able to reproduce the problem that you describe (or at
> >   least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
> >   the problem did not manifest when I disabled GRO on eth1.1.
> 
> It worked for you to do ethtool -K eth1.1 gro off, then?  For me on
> 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even
> though packets arrive on eth1.39.
> 
> Also, strangely, 2.6.35.4's default state (with no received merged frames)
> has GRO on for eth1 but off for eth1.39:
> 
> # ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> 
> # ethtool -k eth1.39
> Offload parameters for eth1.39:
> rx-checksumming: on
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> 
> If I set 2.6.37 to have all of the same options, I still see GRO frames
> on 2.6.37 (tg3), which is weird.
> 

Weird maybe, but GRO check/handling is done in dev_gro_receive(), on
eth1 receive path.

Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO
machine state) before being delivered to eth1.39

It would be useless/expensive to add another GRO layer on eth1.39

We might not report GRO state on vlan/bonding (or reflect real device
GRO state)

Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

[Simon Horman]

On Thu, Jan 27, 2011 at 08:42:05AM +0100, Eric Dumazet wrote:
> Le mercredi 26 janvier 2011 à 16:48 -0800, Simon Kirby a écrit :
> > On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote:
> > 
> > > Hi Simon,
> > > 
> > > thanks for prodding me to respond to this post offline and sorry for not
> > > responding earlier.
> > > 
> > > Firstly, I think that this is a receive-side problem so I don't believe
> > > that GSO (generic segmentation offload) or other transmit-side options are
> > > likely to have any affect.
> > > 
> > > My understanding is that on the receive-side there are two options which
> > > when enabled can result in the behaviour that you describe.
> > > 
> > > * LRO (large receive offload)
> > > 
> > >   You have this disabled, and assuming it really is disabled it
> > >   shouldn't be causing a problem.
> > > 
> > > * GRO (generic receive offload)
> > > 
> > >   This does not seem to be in the output of your ethtool commands at all.
> > >   So I wonder if your ethtool is too old to support this option?
> > 
> > So, this was the case.  Our ethtool (lenny) was too old to see the GRO
> > option, only GSO.  Disabling GRO on eth1.39 has no effect, but disabling
> > it on eth1 caused it to stop receiving the merged frames, fixing the LVS
> > packet loss (due to no sending GSO support from LVS/IPVS).
> > 
> > Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? 
> > 
> > >   In any case, I was able to reproduce the problem that you describe (or at
> > >   least something very similar) using 2.6.36 with GRO enabled on eth1.1 and
> > >   the problem did not manifest when I disabled GRO on eth1.1.
> > 
> > It worked for you to do ethtool -K eth1.1 gro off, then?  For me on
> > 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even
> > though packets arrive on eth1.39.
> > 
> > Also, strangely, 2.6.35.4's default state (with no received merged frames)
> > has GRO on for eth1 but off for eth1.39:
> > 
> > # ethtool -k eth1
> > Offload parameters for eth1:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp-segmentation-offload: on
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off
> > ntuple-filters: off
> > receive-hashing: off
> > 
> > # ethtool -k eth1.39
> > Offload parameters for eth1.39:
> > rx-checksumming: on
> > tx-checksumming: off
> > scatter-gather: off
> > tcp-segmentation-offload: off
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: off
> > generic-receive-offload: off
> > large-receive-offload: off
> > ntuple-filters: off
> > receive-hashing: off
> > 
> > If I set 2.6.37 to have all of the same options, I still see GRO frames
> > on 2.6.37 (tg3), which is weird.
> > 
> 
> Weird maybe, but GRO check/handling is done in dev_gro_receive(), on
> eth1 receive path.
> 
> Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO
> machine state) before being delivered to eth1.39
> 
> It would be useless/expensive to add another GRO layer on eth1.39
> 
> We might not report GRO state on vlan/bonding (or reflect real device
> GRO state)

That makes sense. So to clarify, this is the expected behaviour?