* TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 @ 2010-12-03 10:34 Simon Kirby 2010-12-03 11:29 ` Eric Dumazet 0 siblings, 1 reply; 14+ messages in thread From: Simon Kirby @ 2010-12-03 10:34 UTC (permalink / raw) To: Eric Dumazet, netdev Hello! We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact... LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU: IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556 This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem. Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again. Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653 So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible. Simon- ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-03 10:34 TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 Simon Kirby @ 2010-12-03 11:29 ` Eric Dumazet 2010-12-03 12:36 ` Simon Horman 0 siblings, 1 reply; 14+ messages in thread From: Eric Dumazet @ 2010-12-03 11:29 UTC (permalink / raw) To: Simon Kirby; +Cc: netdev, Simon Horman Le vendredi 03 décembre 2010 à 02:34 -0800, Simon Kirby a écrit : > Hello! > > We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards > (partno(BCM95721) rev 4201) with VLAN tags in use, to think that > everything looked great, but in fact... > > LVS was receiving magically-merged TCP packets which it tried to forward > on to the real server, only to get annoyed at itself for trying to > forward a packet bigger than the device MTU: > > IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> > IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> > IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556 > Hi Simon This is a tcpdump on A ? Could you take it also on B ? tcpdump displays large buffers, but they should be split (of course) when sent on wire. > This caused packet loss for any merged frames, which caused abysmal > performance for uploads via the LVS server. Local performance to or > from the box is still fine, because the stack doesn't care, only the > forwarding part of LVS is running into the problem. > > Furthermore, disabling _everything_ reported by ethtool -k doesn't seem > to change the result, even if I down/up the interface after, and even if > I try on every single interface including the VLANned ones. This seems > to be another bug. Reverting to 2.6.35 makes it all work again. > > Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653 > > So how should this be fixed? Should LVS be taught to fragment, or must > we disable the merging in this case? It seems like it would work well if > the sending side could do the same offload in reverse, but I'm not sure > if that would be possible. > > Simon- I believe Simon Horman has some patches for GRO and LVS. Please send the results of "ethtool -k eth0" on all your nics / vlans ? For TSO, I am not sure why and where it could matter... ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-03 11:29 ` Eric Dumazet @ 2010-12-03 12:36 ` Simon Horman 2010-12-03 12:42 ` Simon Horman ` (2 more replies) 0 siblings, 3 replies; 14+ messages in thread From: Simon Horman @ 2010-12-03 12:36 UTC (permalink / raw) To: Eric Dumazet; +Cc: Simon Kirby, netdev On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote: > Le vendredi 03 décembre 2010 à 02:34 -0800, Simon Kirby a écrit : > > Hello! > > > > We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards > > (partno(BCM95721) rev 4201) with VLAN tags in use, to think that > > everything looked great, but in fact... > > > > LVS was receiving magically-merged TCP packets which it tried to forward > > on to the real server, only to get annoyed at itself for trying to > > forward a packet bigger than the device MTU: > > > > IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> > > IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> > > IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556 > > > > Hi Simon > > This is a tcpdump on A ? > Could you take it also on B ? > > tcpdump displays large buffers, but they should be split (of course) > when sent on wire. > > > This caused packet loss for any merged frames, which caused abysmal > > performance for uploads via the LVS server. Local performance to or > > from the box is still fine, because the stack doesn't care, only the > > forwarding part of LVS is running into the problem. > > > > Furthermore, disabling _everything_ reported by ethtool -k doesn't seem > > to change the result, even if I down/up the interface after, and even if > > I try on every single interface including the VLANned ones. This seems > > to be another bug. Reverting to 2.6.35 makes it all work again. > > > > Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653 > > > > So how should this be fixed? Should LVS be taught to fragment, or must > > we disable the merging in this case? It seems like it would work well if > > the sending side could do the same offload in reverse, but I'm not sure > > if that would be possible. > > > > Simon- > > > I believe Simon Horman has some patches for GRO and LVS. > > Please send the results of "ethtool -k eth0" on all your nics / vlans ? > > For TSO, I am not sure why and where it could matter... There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO. The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable. Simon, you mention that you disabled everything with ethtool, but the tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is active. So perhaps as you speculate that is a bug I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-03 12:36 ` Simon Horman @ 2010-12-03 12:42 ` Simon Horman 2010-12-03 13:27 ` Ben Hutchings 2010-12-09 2:35 ` Simon Kirby 2 siblings, 0 replies; 14+ messages in thread From: Simon Horman @ 2010-12-03 12:42 UTC (permalink / raw) To: Eric Dumazet; +Cc: Simon Kirby, netdev, lvs-devel, Julian Anastasov, Herbert Xu [ CCed lvs-devel, Julian Anastasov and Herbert Xu ] On Fri, Dec 03, 2010 at 09:36:19PM +0900, Simon Horman wrote: > On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote: > > Le vendredi 03 décembre 2010 à 02:34 -0800, Simon Kirby a écrit : > > > Hello! > > > > > > We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards > > > (partno(BCM95721) rev 4201) with VLAN tags in use, to think that > > > everything looked great, but in fact... > > > > > > LVS was receiving magically-merged TCP packets which it tried to forward > > > on to the real server, only to get annoyed at itself for trying to > > > forward a packet bigger than the device MTU: > > > > > > IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> > > > IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> > > > IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556 > > > > > > > Hi Simon > > > > This is a tcpdump on A ? > > Could you take it also on B ? > > > > tcpdump displays large buffers, but they should be split (of course) > > when sent on wire. > > > > > This caused packet loss for any merged frames, which caused abysmal > > > performance for uploads via the LVS server. Local performance to or > > > from the box is still fine, because the stack doesn't care, only the > > > forwarding part of LVS is running into the problem. > > > > > > Furthermore, disabling _everything_ reported by ethtool -k doesn't seem > > > to change the result, even if I down/up the interface after, and even if > > > I try on every single interface including the VLANned ones. This seems > > > to be another bug. Reverting to 2.6.35 makes it all work again. > > > > > > Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653 > > > > > > So how should this be fixed? Should LVS be taught to fragment, or must > > > we disable the merging in this case? It seems like it would work well if > > > the sending side could do the same offload in reverse, but I'm not sure > > > if that would be possible. > > > > > > Simon- > > > > > > I believe Simon Horman has some patches for GRO and LVS. > > > > Please send the results of "ethtool -k eth0" on all your nics / vlans ? > > > > For TSO, I am not sure why and where it could matter... > > There is a patch to teach LVS how to cope with GRO in nf-next-2.6 > and I expect it to be included in 2.6.38. The patch is "ipvs: allow > transmit of GRO aggregated skbs" and perhaps it should be considered > for 2.6.37 and stable. In general the work around is to disable GRO. > > The patch does not resolve the incompatibility of LVS with LRO. > The work around there is to disable LRO. I'm not entirely sure > how to teach LVS to disable LRO automatically, or if its desirable. > > Simon, you mention that you disabled everything with ethtool, but the > tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is > active. So perhaps as you speculate that is a bug > > I will prepare a backport of the "ipvs: allow transmit of GRO aggregated > skbs" patch to v2.6.36 and post it shortly. Testing to see if that > resolves the problem that you are seeing would probably be a good start. Here is the patch for v2.6.36. From: Simon Horman <horms@verge.net.au> ipvs: allow transmit of GRO aggregated skbs Attempt at allowing LVS to transmit skbs of greater than MTU length that have been aggregated by GRO and can thus be deaggregated by GSO. Cc: Julian Anastasov <ja@ssi.bg> Cc: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Simon Horman <horms@verge.net.au> --- net/netfilter/ipvs/ip_vs_xmit.c | 25 +++++++++++++++---------- 1 files changed, 15 insertions(+), 10 deletions(-) diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c index 49df6be..577f502 100644 --- a/net/netfilter/ipvs/ip_vs_xmit.c +++ b/net/netfilter/ipvs/ip_vs_xmit.c @@ -247,7 +247,8 @@ ip_vs_bypass_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, /* MTU checking */ mtu = dst_mtu(&rt->dst); - if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF))) { + if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF)) && + !skb_is_gso(skb)) { ip_rt_put(rt); icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu)); IP_VS_DBG_RL("%s(): frag needed\n", __func__); @@ -311,7 +312,7 @@ ip_vs_bypass_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, /* MTU checking */ mtu = dst_mtu(&rt->dst); - if (skb->len > mtu) { + if (skb->len > mtu && !skb_is_gso(skb)) { dst_release(&rt->dst); icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu); IP_VS_DBG_RL("%s(): frag needed\n", __func__); @@ -408,7 +409,8 @@ ip_vs_nat_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, /* MTU checking */ mtu = dst_mtu(&rt->dst); - if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF))) { + if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF)) && + !skb_is_gso(skb)) { ip_rt_put(rt); icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu)); IP_VS_DBG_RL_PKT(0, pp, skb, 0, "ip_vs_nat_xmit(): frag needed for"); @@ -486,7 +488,7 @@ ip_vs_nat_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, /* MTU checking */ mtu = dst_mtu(&rt->dst); - if (skb->len > mtu) { + if (skb->len > mtu && !skb_is_gso(skb)) { dst_release(&rt->dst); icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu); IP_VS_DBG_RL_PKT(0, pp, skb, 0, @@ -597,8 +599,8 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, df |= (old_iph->frag_off & htons(IP_DF)); - if ((old_iph->frag_off & htons(IP_DF)) - && mtu < ntohs(old_iph->tot_len)) { + if ((old_iph->frag_off & htons(IP_DF) && + mtu < ntohs(old_iph->tot_len) && !skb_is_gso(skb))) { icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu)); ip_rt_put(rt); IP_VS_DBG_RL("%s(): frag needed\n", __func__); @@ -707,7 +709,8 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, if (skb_dst(skb)) skb_dst(skb)->ops->update_pmtu(skb_dst(skb), mtu); - if (mtu < ntohs(old_iph->payload_len) + sizeof(struct ipv6hdr)) { + if (mtu < ntohs(old_iph->payload_len) + sizeof(struct ipv6hdr) && + !skb_is_gso(skb)) { icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu); dst_release(&rt->dst); IP_VS_DBG_RL("%s(): frag needed\n", __func__); @@ -796,7 +799,8 @@ ip_vs_dr_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, /* MTU checking */ mtu = dst_mtu(&rt->dst); - if ((iph->frag_off & htons(IP_DF)) && skb->len > mtu) { + if ((iph->frag_off & htons(IP_DF)) && skb->len > mtu && + !skb_is_gso(skb)) { icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu)); ip_rt_put(rt); IP_VS_DBG_RL("%s(): frag needed\n", __func__); @@ -924,7 +928,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, /* MTU checking */ mtu = dst_mtu(&rt->dst); - if ((skb->len > mtu) && (ip_hdr(skb)->frag_off & htons(IP_DF))) { + if ((skb->len > mtu) && (ip_hdr(skb)->frag_off & htons(IP_DF)) && + !skb_is_gso(skb)) { ip_rt_put(rt); icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED, htonl(mtu)); IP_VS_DBG_RL("%s(): frag needed\n", __func__); @@ -999,7 +1004,7 @@ ip_vs_icmp_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, /* MTU checking */ mtu = dst_mtu(&rt->dst); - if (skb->len > mtu) { + if (skb->len > mtu && !skb_is_gso(skb)) { dst_release(&rt->dst); icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu); IP_VS_DBG_RL("%s(): frag needed\n", __func__); -- 1.7.2.3 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-03 12:36 ` Simon Horman 2010-12-03 12:42 ` Simon Horman @ 2010-12-03 13:27 ` Ben Hutchings 2010-12-03 13:39 ` Simon Horman 2010-12-09 2:35 ` Simon Kirby 2 siblings, 1 reply; 14+ messages in thread From: Ben Hutchings @ 2010-12-03 13:27 UTC (permalink / raw) To: Simon Horman; +Cc: Eric Dumazet, Simon Kirby, netdev On Fri, 2010-12-03 at 21:36 +0900, Simon Horman wrote: > On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote: [...] > > I believe Simon Horman has some patches for GRO and LVS. > > > > Please send the results of "ethtool -k eth0" on all your nics / vlans ? > > > > For TSO, I am not sure why and where it could matter... > > There is a patch to teach LVS how to cope with GRO in nf-next-2.6 > and I expect it to be included in 2.6.38. The patch is "ipvs: allow > transmit of GRO aggregated skbs" and perhaps it should be considered > for 2.6.37 and stable. In general the work around is to disable GRO. > > The patch does not resolve the incompatibility of LVS with LRO. > The work around there is to disable LRO. I'm not entirely sure > how to teach LVS to disable LRO automatically, or if its desirable. [...] I think it is desirable, and we already do it for bridging and forwarding - we call dev_disable_lro() whenever a device is added to a bridge or has forwarding enabled. For IPVS it looks like you would have to resolve the server IP address to a device first. Though if the administrator assigns the server address to a different interface later then the failure mode will be very hard to understand. Ben. -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-03 13:27 ` Ben Hutchings @ 2010-12-03 13:39 ` Simon Horman 0 siblings, 0 replies; 14+ messages in thread From: Simon Horman @ 2010-12-03 13:39 UTC (permalink / raw) To: Ben Hutchings; +Cc: Eric Dumazet, Simon Kirby, netdev On Fri, Dec 03, 2010 at 01:27:19PM +0000, Ben Hutchings wrote: > On Fri, 2010-12-03 at 21:36 +0900, Simon Horman wrote: > > On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote: > [...] > > > I believe Simon Horman has some patches for GRO and LVS. > > > > > > Please send the results of "ethtool -k eth0" on all your nics / vlans ? > > > > > > For TSO, I am not sure why and where it could matter... > > > > There is a patch to teach LVS how to cope with GRO in nf-next-2.6 > > and I expect it to be included in 2.6.38. The patch is "ipvs: allow > > transmit of GRO aggregated skbs" and perhaps it should be considered > > for 2.6.37 and stable. In general the work around is to disable GRO. > > > > The patch does not resolve the incompatibility of LVS with LRO. > > The work around there is to disable LRO. I'm not entirely sure > > how to teach LVS to disable LRO automatically, or if its desirable. > [...] > > I think it is desirable, and we already do it for bridging and > forwarding - we call dev_disable_lro() whenever a device is added to a > bridge or has forwarding enabled. For IPVS it looks like you would have > to resolve the server IP address to a device first. Though if the > administrator assigns the server address to a different interface later > then the failure mode will be very hard to understand. Thanks, I'll try and make it so. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-03 12:36 ` Simon Horman 2010-12-03 12:42 ` Simon Horman 2010-12-03 13:27 ` Ben Hutchings @ 2010-12-09 2:35 ` Simon Kirby 2010-12-09 2:51 ` Eric Dumazet 2 siblings, 1 reply; 14+ messages in thread From: Simon Kirby @ 2010-12-09 2:35 UTC (permalink / raw) To: Simon Horman; +Cc: Eric Dumazet, netdev On Fri, Dec 03, 2010 at 09:36:19PM +0900, Simon Horman wrote: > On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote: > > Le vendredi 03 d??cembre 2010 ?? 02:34 -0800, Simon Kirby a ??crit : > > > Hello! > > > > > > We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards > > > (partno(BCM95721) rev 4201) with VLAN tags in use, to think that > > > everything looked great, but in fact... > > > > > > LVS was receiving magically-merged TCP packets which it tried to forward > > > on to the real server, only to get annoyed at itself for trying to > > > forward a packet bigger than the device MTU: > > > > > > IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> > > > IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> > > > IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556 > > > > > > > Hi Simon > > > > This is a tcpdump on A ? > > Could you take it also on B ? > > > > tcpdump displays large buffers, but they should be split (of course) > > when sent on wire. I assume you don't need this anymore since the problem was figured out? Otherwise, let me know. > > > This caused packet loss for any merged frames, which caused abysmal > > > performance for uploads via the LVS server. Local performance to or > > > from the box is still fine, because the stack doesn't care, only the > > > forwarding part of LVS is running into the problem. > > > > > > Furthermore, disabling _everything_ reported by ethtool -k doesn't seem > > > to change the result, even if I down/up the interface after, and even if > > > I try on every single interface including the VLANned ones. This seems > > > to be another bug. Reverting to 2.6.35 makes it all work again. > > > > > > Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653 > > > > > > So how should this be fixed? Should LVS be taught to fragment, or must > > > we disable the merging in this case? It seems like it would work well if > > > the sending side could do the same offload in reverse, but I'm not sure > > > if that would be possible. > > > > > > Simon- > > > > > > I believe Simon Horman has some patches for GRO and LVS. > > > > Please send the results of "ethtool -k eth0" on all your nics / vlans ? > > > > For TSO, I am not sure why and where it could matter... > > There is a patch to teach LVS how to cope with GRO in nf-next-2.6 > and I expect it to be included in 2.6.38. The patch is "ipvs: allow > transmit of GRO aggregated skbs" and perhaps it should be considered > for 2.6.37 and stable. In general the work around is to disable GRO. > > The patch does not resolve the incompatibility of LVS with LRO. > The work around there is to disable LRO. I'm not entirely sure > how to teach LVS to disable LRO automatically, or if its desirable. Yeah, things will still break if the interfaces have mismatched GRO support, etc. It would be nice if it could be all smart about it, but I don't see how this could be done sanely, as it doesn't know if the destination interface could support GRO transmission until it has gone through the LVS processing. Would software GRO be faster on transmission be faster than no GRO at all? > Simon, you mention that you disabled everything with ethtool, but the > tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is > active. So perhaps as you speculate that is a bug Right. So, on 2.6.35, ethtool shows: # ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: off On 2.6.36, ethtool shows: # ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc < /dev/zero and tcpdump -i any -n 'length > 1500'. > I will prepare a backport of the "ipvs: allow transmit of GRO aggregated > skbs" patch to v2.6.36 and post it shortly. Testing to see if that > resolves the problem that you are seeing would probably be a good start. I can test this in our case, and it should specifically work on the servers we were seeing the problem on, but it still needs some complicated logic to be safe for all cases (like when LVS decides to route out an interface without GRO). Anyway, thanks! Simon- ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-09 2:35 ` Simon Kirby @ 2010-12-09 2:51 ` Eric Dumazet 2010-12-22 23:39 ` Simon Kirby 0 siblings, 1 reply; 14+ messages in thread From: Eric Dumazet @ 2010-12-09 2:51 UTC (permalink / raw) To: Simon Kirby; +Cc: Simon Horman, netdev Le mercredi 08 décembre 2010 à 18:35 -0800, Simon Kirby a écrit : > Right. So, on 2.6.35, ethtool shows: > > # ethtool -k eth1 > Offload parameters for eth1: > rx-checksumming: on > tx-checksumming: on > scatter-gather: on > tcp segmentation offload: on > udp fragmentation offload: off > generic segmentation offload: on > large receive offload: off > > # ethtool -k eth1.39 > Offload parameters for eth1.39: > rx-checksumming: on > tx-checksumming: off > scatter-gather: off > tcp segmentation offload: off > udp fragmentation offload: off > generic segmentation offload: off > large receive offload: off > > On 2.6.36, ethtool shows: > > # ethtool -k eth1 > Offload parameters for eth1: > rx-checksumming: on > tx-checksumming: on > scatter-gather: on > tcp segmentation offload: on > udp fragmentation offload: off > generic segmentation offload: on > large receive offload: off > > # ethtool -k eth1.39 > Offload parameters for eth1.39: > rx-checksumming: on > tx-checksumming: on > scatter-gather: on > tcp segmentation offload: on > udp fragmentation offload: off > generic segmentation offload: on > large receive offload: off > > And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, > it says "off", but I still see merged frames, as verified with an nc < > /dev/zero and tcpdump -i any -n 'length > 1500'. > Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan) > > I will prepare a backport of the "ipvs: allow transmit of GRO aggregated > > skbs" patch to v2.6.36 and post it shortly. Testing to see if that > > resolves the problem that you are seeing would probably be a good start. > > I can test this in our case, and it should specifically work on the > servers we were seeing the problem on, but it still needs some > complicated logic to be safe for all cases (like when LVS decides to > route out an interface without GRO). Anyway, thanks! > > Simon- ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-09 2:51 ` Eric Dumazet @ 2010-12-22 23:39 ` Simon Kirby 2011-01-13 6:34 ` Simon Horman 0 siblings, 1 reply; 14+ messages in thread From: Simon Kirby @ 2010-12-22 23:39 UTC (permalink / raw) To: Eric Dumazet; +Cc: Simon Horman, netdev On Thu, Dec 09, 2010 at 03:51:59AM +0100, Eric Dumazet wrote: > Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit : > > > Right. So, on 2.6.35, ethtool shows: > > > > # ethtool -k eth1 > > Offload parameters for eth1: > > rx-checksumming: on > > tx-checksumming: on > > scatter-gather: on > > tcp segmentation offload: on > > udp fragmentation offload: off > > generic segmentation offload: on > > large receive offload: off > > > > # ethtool -k eth1.39 > > Offload parameters for eth1.39: > > rx-checksumming: on > > tx-checksumming: off > > scatter-gather: off > > tcp segmentation offload: off > > udp fragmentation offload: off > > generic segmentation offload: off > > large receive offload: off > > > > On 2.6.36, ethtool shows: > > > > # ethtool -k eth1 > > Offload parameters for eth1: > > rx-checksumming: on > > tx-checksumming: on > > scatter-gather: on > > tcp segmentation offload: on > > udp fragmentation offload: off > > generic segmentation offload: on > > large receive offload: off > > > > # ethtool -k eth1.39 > > Offload parameters for eth1.39: > > rx-checksumming: on > > tx-checksumming: on > > scatter-gather: on > > tcp segmentation offload: on > > udp fragmentation offload: off > > generic segmentation offload: on > > large receive offload: off > > > > And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, > > it says "off", but I still see merged frames, as verified with an nc < > > /dev/zero and tcpdump -i any -n 'length > 1500'. > > Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan) and turning it off makes no difference, either. Even if I down/up the interface after. I still see merged packets on receive, even if I try to use ethtool to turn off everything that is "on". It's as if the new code to enable it on vlans works, but the disable from ethtool does not. In other words, I cannot find a way to disable it (and thus make LVS work) on 2.6.36 or 2.6.37-rc6. One interesting note is that if I destroy and recreate the VLAN interface, it comes back with gso enabled: # ethtool -K eth1 gso off # ethtool -K eth1.39 gso off # ethtool -K eth1 tso off # ethtool -K eth1.39 tso off # ifdown eth1.39 Removed VLAN -:eth1.39:- # ifconfig eth1 down # ifup eth1.39 Added VLAN with VID == 39 to IF -:eth1:- # ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: off # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: on large receive offload: off # ethtool -K eth1.39 gso off # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: off # ifdown eth1.39; ifup eth1.39 Removed VLAN -:eth1.39:- Added VLAN with VID == 39 to IF -:eth1:- # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: on large receive offload: off So, the bug seems to related using it on VLANs. Simon- > > > I will prepare a backport of the "ipvs: allow transmit of GRO aggregated > > > skbs" patch to v2.6.36 and post it shortly. Testing to see if that > > > resolves the problem that you are seeing would probably be a good start. > > > > I can test this in our case, and it should specifically work on the > > servers we were seeing the problem on, but it still needs some > > complicated logic to be safe for all cases (like when LVS decides to > > route out an interface without GRO). Anyway, thanks! > > > > Simon- > > > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2010-12-22 23:39 ` Simon Kirby @ 2011-01-13 6:34 ` Simon Horman 2011-01-27 0:48 ` Simon Kirby 0 siblings, 1 reply; 14+ messages in thread From: Simon Horman @ 2011-01-13 6:34 UTC (permalink / raw) To: Simon Kirby; +Cc: Eric Dumazet, netdev On Wed, Dec 22, 2010 at 03:39:48PM -0800, Simon Kirby wrote: > On Thu, Dec 09, 2010 at 03:51:59AM +0100, Eric Dumazet wrote: > > > Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit : > > > > > Right. So, on 2.6.35, ethtool shows: > > > > > > # ethtool -k eth1 > > > Offload parameters for eth1: > > > rx-checksumming: on > > > tx-checksumming: on > > > scatter-gather: on > > > tcp segmentation offload: on > > > udp fragmentation offload: off > > > generic segmentation offload: on > > > large receive offload: off > > > > > > # ethtool -k eth1.39 > > > Offload parameters for eth1.39: > > > rx-checksumming: on > > > tx-checksumming: off > > > scatter-gather: off > > > tcp segmentation offload: off > > > udp fragmentation offload: off > > > generic segmentation offload: off > > > large receive offload: off > > > > > > On 2.6.36, ethtool shows: > > > > > > # ethtool -k eth1 > > > Offload parameters for eth1: > > > rx-checksumming: on > > > tx-checksumming: on > > > scatter-gather: on > > > tcp segmentation offload: on > > > udp fragmentation offload: off > > > generic segmentation offload: on > > > large receive offload: off > > > > > > # ethtool -k eth1.39 > > > Offload parameters for eth1.39: > > > rx-checksumming: on > > > tx-checksumming: on > > > scatter-gather: on > > > tcp segmentation offload: on > > > udp fragmentation offload: off > > > generic segmentation offload: on > > > large receive offload: off > > > > > > And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, > > > it says "off", but I still see merged frames, as verified with an nc < > > > /dev/zero and tcpdump -i any -n 'length > 1500'. > > > > Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan) > > and turning it off makes no difference, either. Even if I down/up the > interface after. I still see merged packets on receive, even if I try to > use ethtool to turn off everything that is "on". It's as if the new code > to enable it on vlans works, but the disable from ethtool does not. > > In other words, I cannot find a way to disable it (and thus make LVS > work) on 2.6.36 or 2.6.37-rc6. Hi Simon, thanks for prodding me to respond to this post offline and sorry for not responding earlier. Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect. My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe. * LRO (large receive offload) You have this disabled, and assuming it really is disabled it shouldn't be causing a problem. * GRO (generic receive offload) This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option? In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1. N.B: I'm not sure that I have any hardware whose driver supports LRO so my tests were made without LRO support. > One interesting note is that if I destroy and recreate the VLAN interface, > it comes back with gso enabled: I guess that relate to how the default is set. I don't imagine that it is the cause of the problem that you are seeing. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2011-01-13 6:34 ` Simon Horman @ 2011-01-27 0:48 ` Simon Kirby 2011-01-27 1:36 ` Simon Horman 2011-01-27 7:42 ` Eric Dumazet 0 siblings, 2 replies; 14+ messages in thread From: Simon Kirby @ 2011-01-27 0:48 UTC (permalink / raw) To: Simon Horman; +Cc: Eric Dumazet, netdev On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote: > Hi Simon, > > thanks for prodding me to respond to this post offline and sorry for not > responding earlier. > > Firstly, I think that this is a receive-side problem so I don't believe > that GSO (generic segmentation offload) or other transmit-side options are > likely to have any affect. > > My understanding is that on the receive-side there are two options which > when enabled can result in the behaviour that you describe. > > * LRO (large receive offload) > > You have this disabled, and assuming it really is disabled it > shouldn't be causing a problem. > > * GRO (generic receive offload) > > This does not seem to be in the output of your ethtool commands at all. > So I wonder if your ethtool is too old to support this option? So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS). Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? > In any case, I was able to reproduce the problem that you describe (or at > least something very similar) using 2.6.36 with GRO enabled on eth1.1 and > the problem did not manifest when I disabled GRO on eth1.1. It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39. Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39: # ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off ntuple-filters: off receive-hashing: off # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: off If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird. Cheers, Simon- ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2011-01-27 0:48 ` Simon Kirby @ 2011-01-27 1:36 ` Simon Horman 2011-01-27 7:42 ` Eric Dumazet 1 sibling, 0 replies; 14+ messages in thread From: Simon Horman @ 2011-01-27 1:36 UTC (permalink / raw) To: Simon Kirby; +Cc: Eric Dumazet, netdev On Wed, Jan 26, 2011 at 04:48:05PM -0800, Simon Kirby wrote: > On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote: > > > Hi Simon, > > > > thanks for prodding me to respond to this post offline and sorry for not > > responding earlier. > > > > Firstly, I think that this is a receive-side problem so I don't believe > > that GSO (generic segmentation offload) or other transmit-side options are > > likely to have any affect. > > > > My understanding is that on the receive-side there are two options which > > when enabled can result in the behaviour that you describe. > > > > * LRO (large receive offload) > > > > You have this disabled, and assuming it really is disabled it > > shouldn't be causing a problem. > > > > * GRO (generic receive offload) > > > > This does not seem to be in the output of your ethtool commands at all. > > So I wonder if your ethtool is too old to support this option? > > So, this was the case. Our ethtool (lenny) was too old to see the GRO > option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling > it on eth1 caused it to stop receiving the merged frames, fixing the LVS > packet loss (due to no sending GSO support from LVS/IPVS). > > Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? The patch for IPVS GRO support has been merged and should appear in 2.6.39. This is somewhat later than I previously anticipated due to a merge mix-up on my part. > > In any case, I was able to reproduce the problem that you describe (or at > > least something very similar) using 2.6.36 with GRO enabled on eth1.1 and > > the problem did not manifest when I disabled GRO on eth1.1. > > It worked for you to do ethtool -K eth1.1 gro off, then? For me on > 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even > though packets arrive on eth1.39. I will recheck my results, but in general I think it is a bit of an open question as to how ethtool settings should be propagated between related devices. > Also, strangely, 2.6.35.4's default state (with no received merged frames) > has GRO on for eth1 but off for eth1.39: > > # ethtool -k eth1 > Offload parameters for eth1: > rx-checksumming: on > tx-checksumming: on > scatter-gather: on > tcp-segmentation-offload: on > udp-fragmentation-offload: off > generic-segmentation-offload: on > generic-receive-offload: on > large-receive-offload: off > ntuple-filters: off > receive-hashing: off > > # ethtool -k eth1.39 > Offload parameters for eth1.39: > rx-checksumming: on > tx-checksumming: off > scatter-gather: off > tcp-segmentation-offload: off > udp-fragmentation-offload: off > generic-segmentation-offload: off > generic-receive-offload: off > large-receive-offload: off > ntuple-filters: off > receive-hashing: off > > If I set 2.6.37 to have all of the same options, I still see GRO frames > on 2.6.37 (tg3), which is weird. Yes that is a weird. There has been quite a lot of work on VLANs recently and I suspect that the behaviour that you are observing with 2.6.37 is a regression that occurred during that work. It would be good to fix things to restore the 2.6.35 behaviour. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2011-01-27 0:48 ` Simon Kirby 2011-01-27 1:36 ` Simon Horman @ 2011-01-27 7:42 ` Eric Dumazet 2011-01-27 14:42 ` Simon Horman 1 sibling, 1 reply; 14+ messages in thread From: Eric Dumazet @ 2011-01-27 7:42 UTC (permalink / raw) To: Simon Kirby; +Cc: Simon Horman, netdev Le mercredi 26 janvier 2011 à 16:48 -0800, Simon Kirby a écrit : > On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote: > > > Hi Simon, > > > > thanks for prodding me to respond to this post offline and sorry for not > > responding earlier. > > > > Firstly, I think that this is a receive-side problem so I don't believe > > that GSO (generic segmentation offload) or other transmit-side options are > > likely to have any affect. > > > > My understanding is that on the receive-side there are two options which > > when enabled can result in the behaviour that you describe. > > > > * LRO (large receive offload) > > > > You have this disabled, and assuming it really is disabled it > > shouldn't be causing a problem. > > > > * GRO (generic receive offload) > > > > This does not seem to be in the output of your ethtool commands at all. > > So I wonder if your ethtool is too old to support this option? > > So, this was the case. Our ethtool (lenny) was too old to see the GRO > option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling > it on eth1 caused it to stop receiving the merged frames, fixing the LVS > packet loss (due to no sending GSO support from LVS/IPVS). > > Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? > > > In any case, I was able to reproduce the problem that you describe (or at > > least something very similar) using 2.6.36 with GRO enabled on eth1.1 and > > the problem did not manifest when I disabled GRO on eth1.1. > > It worked for you to do ethtool -K eth1.1 gro off, then? For me on > 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even > though packets arrive on eth1.39. > > Also, strangely, 2.6.35.4's default state (with no received merged frames) > has GRO on for eth1 but off for eth1.39: > > # ethtool -k eth1 > Offload parameters for eth1: > rx-checksumming: on > tx-checksumming: on > scatter-gather: on > tcp-segmentation-offload: on > udp-fragmentation-offload: off > generic-segmentation-offload: on > generic-receive-offload: on > large-receive-offload: off > ntuple-filters: off > receive-hashing: off > > # ethtool -k eth1.39 > Offload parameters for eth1.39: > rx-checksumming: on > tx-checksumming: off > scatter-gather: off > tcp-segmentation-offload: off > udp-fragmentation-offload: off > generic-segmentation-offload: off > generic-receive-offload: off > large-receive-offload: off > ntuple-filters: off > receive-hashing: off > > If I set 2.6.37 to have all of the same options, I still see GRO frames > on 2.6.37 (tg3), which is weird. > Weird maybe, but GRO check/handling is done in dev_gro_receive(), on eth1 receive path. Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO machine state) before being delivered to eth1.39 It would be useless/expensive to add another GRO layer on eth1.39 We might not report GRO state on vlan/bonding (or reflect real device GRO state) ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 2011-01-27 7:42 ` Eric Dumazet @ 2011-01-27 14:42 ` Simon Horman 0 siblings, 0 replies; 14+ messages in thread From: Simon Horman @ 2011-01-27 14:42 UTC (permalink / raw) To: Eric Dumazet; +Cc: Simon Kirby, netdev On Thu, Jan 27, 2011 at 08:42:05AM +0100, Eric Dumazet wrote: > Le mercredi 26 janvier 2011 à 16:48 -0800, Simon Kirby a écrit : > > On Thu, Jan 13, 2011 at 03:34:22PM +0900, Simon Horman wrote: > > > > > Hi Simon, > > > > > > thanks for prodding me to respond to this post offline and sorry for not > > > responding earlier. > > > > > > Firstly, I think that this is a receive-side problem so I don't believe > > > that GSO (generic segmentation offload) or other transmit-side options are > > > likely to have any affect. > > > > > > My understanding is that on the receive-side there are two options which > > > when enabled can result in the behaviour that you describe. > > > > > > * LRO (large receive offload) > > > > > > You have this disabled, and assuming it really is disabled it > > > shouldn't be causing a problem. > > > > > > * GRO (generic receive offload) > > > > > > This does not seem to be in the output of your ethtool commands at all. > > > So I wonder if your ethtool is too old to support this option? > > > > So, this was the case. Our ethtool (lenny) was too old to see the GRO > > option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling > > it on eth1 caused it to stop receiving the merged frames, fixing the LVS > > packet loss (due to no sending GSO support from LVS/IPVS). > > > > Speaking of this, did your patch for LVS/IPVS GSO support go anywhere? > > > > > In any case, I was able to reproduce the problem that you describe (or at > > > least something very similar) using 2.6.36 with GRO enabled on eth1.1 and > > > the problem did not manifest when I disabled GRO on eth1.1. > > > > It worked for you to do ethtool -K eth1.1 gro off, then? For me on > > 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even > > though packets arrive on eth1.39. > > > > Also, strangely, 2.6.35.4's default state (with no received merged frames) > > has GRO on for eth1 but off for eth1.39: > > > > # ethtool -k eth1 > > Offload parameters for eth1: > > rx-checksumming: on > > tx-checksumming: on > > scatter-gather: on > > tcp-segmentation-offload: on > > udp-fragmentation-offload: off > > generic-segmentation-offload: on > > generic-receive-offload: on > > large-receive-offload: off > > ntuple-filters: off > > receive-hashing: off > > > > # ethtool -k eth1.39 > > Offload parameters for eth1.39: > > rx-checksumming: on > > tx-checksumming: off > > scatter-gather: off > > tcp-segmentation-offload: off > > udp-fragmentation-offload: off > > generic-segmentation-offload: off > > generic-receive-offload: off > > large-receive-offload: off > > ntuple-filters: off > > receive-hashing: off > > > > If I set 2.6.37 to have all of the same options, I still see GRO frames > > on 2.6.37 (tg3), which is weird. > > > > Weird maybe, but GRO check/handling is done in dev_gro_receive(), on > eth1 receive path. > > Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO > machine state) before being delivered to eth1.39 > > It would be useless/expensive to add another GRO layer on eth1.39 > > We might not report GRO state on vlan/bonding (or reflect real device > GRO state) That makes sense. So to clarify, this is the expected behaviour? ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2011-01-27 14:42 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-12-03 10:34 TSO/GRO/LRO/somethingO breaks LVS on 2.6.36 Simon Kirby 2010-12-03 11:29 ` Eric Dumazet 2010-12-03 12:36 ` Simon Horman 2010-12-03 12:42 ` Simon Horman 2010-12-03 13:27 ` Ben Hutchings 2010-12-03 13:39 ` Simon Horman 2010-12-09 2:35 ` Simon Kirby 2010-12-09 2:51 ` Eric Dumazet 2010-12-22 23:39 ` Simon Kirby 2011-01-13 6:34 ` Simon Horman 2011-01-27 0:48 ` Simon Kirby 2011-01-27 1:36 ` Simon Horman 2011-01-27 7:42 ` Eric Dumazet 2011-01-27 14:42 ` Simon Horman
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.