IPv6 path MTU discovery broken

IPv6 path MTU discovery broken

[Steinar H. Gunderson]

Hi,

PMTU discovery over IPv6 has been flaky for me for a while, but at some point
between 3.10 and 3.11, it broke for me completely. Just checked with
3.12.0-rc2 and the problem is still there.

First, a look at my routing table, which is slightly unusual due to tunnels
and BGP being involved:

  pannekake:~> sudo ip -6 route                 
  2001:500:72::/48 via 2001:67c:29f4::1 dev eth0  proto zebra  metric 1024 
  2001:67c:a4::/48 via fe80::230:48ff:fe55:5743 dev eth0  proto zebra  metric 100 
  2001:67c:29f4::/64 dev eth0  proto kernel  metric 256 
  2001:67c:29f4:1::/64 via fe80::230:48ff:fe55:5743 dev eth0  proto zebra  metric 100 
  2001:67c:29f4:1000::/64 via fe80::230:48ff:fe55:5743 dev eth0  proto zebra  metric 100 
  2001:67c:29f4:1001::/64 via fe80::230:48ff:fe55:5743 dev eth0  proto zebra  metric 100 
  2001:67c:29f4:1003::/64 via fe80::230:48ff:fe55:5743 dev eth0  proto zebra  metric 100 
  2001:67c:29f4:1005::/64 via fe80::230:48ff:fe55:5743 dev eth0  proto zebra  metric 100 
  2001:67c:29f4:1007::/64 via fe80::230:48ff:fe55:5743 dev eth0  proto zebra  metric 100 
  2001:67c:29f4::/48 via 2001:67c:29f4::1 dev eth0  proto zebra  metric 1024 
  2001:700::/32 via 2001:67c:29f4::1 dev eth0  proto zebra  metric 1024 
  2a02:2368::/32 via 2001:67c:29f4::1 dev eth0  proto zebra  metric 1024 
  fe80::c30b:9a61 dev k_sessesveits  proto kernel  metric 256 
  fe80::c30b:9a61 dev k_wikene  proto kernel  metric 256 
  fe80::c30b:9a61 dev k_trygve  proto kernel  metric 256 
  fe80::c30b:9a61 dev k_magne  proto kernel  metric 256 
  fe80::c30b:9a61 dev k_berge  proto kernel  metric 256 
  fe80::c30b:9a61 dev k_molven  proto kernel  metric 256 
  fe80::/64 dev eth0  proto kernel  metric 256 
  fe80::/64 dev k_sessesveits  proto kernel  metric 256 
  fe80::/64 dev k_wikene  proto kernel  metric 256 
  fe80::/64 dev k_trygve  proto kernel  metric 256 
  fe80::/64 dev k_magne  proto kernel  metric 256 
  fe80::/64 dev k_berge  proto kernel  metric 256 
  fe80::/64 dev k_molven  proto kernel  metric 256 
  default via 2001:67c:29f4::1 dev eth0  metric 1024 

Note in particular that 2001:67c:a4::/48 goes to a link-local address. The
tcpdump is classic; when I try to do something big over my SSH session,
like an ls, it starts failing:

22:02:13.597303 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [P.], seq 5281:5521, ack 2802, win 149, options [nop,nop,TS val 1526325 ecr 45546957], length 240
22:02:13.597331 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [P.], seq 5521:5713, ack 2802, win 149, options [nop,nop,TS val 1526325 ecr 45546957], length 192
22:02:13.597353 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [P.], seq 5713:5921, ack 2802, win 149, options [nop,nop,TS val 1526325 ecr 45546957], length 208
22:02:13.597372 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [P.], seq 5921:6049, ack 2802, win 149, options [nop,nop,TS val 1526325 ecr 45546957], length 128
22:02:13.638445 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 3313, win 173, options [nop,nop,TS val 45546972 ecr 1526306], length 0
22:02:13.638468 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 6049:7477, ack 2802, win 149, options [nop,nop,TS val 1526366 ecr 45546972], length 1428
22:02:13.638475 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 7477:8905, ack 2802, win 149, options [nop,nop,TS val 1526366 ecr 45546972], length 1428
22:02:13.654519 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 3585, win 188, options [nop,nop,TS val 45546977 ecr 1526325], length 0
22:02:13.654538 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 8905:10333, ack 2802, win 149, options [nop,nop,TS val 1526382 ecr 45546977], length 1428
22:02:13.654545 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 10333:11761, ack 2802, win 149, options [nop,nop,TS val 1526382 ecr 45546977], length 1428
22:02:13.661389 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 4097, win 203, options [nop,nop,TS val 45546977 ecr 1526325], length 0
22:02:13.661408 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 11761:13189, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546977], length 1428
22:02:13.661415 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 13189:14617, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546977], length 1428
22:02:13.661420 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 4705, win 218, options [nop,nop,TS val 45546977 ecr 1526325], length 0
22:02:13.661431 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 14617:16045, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546977], length 1428
22:02:13.661436 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 16045:17473, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546977], length 1428
22:02:13.661441 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 5009, win 233, options [nop,nop,TS val 45546978 ecr 1526325], length 0
22:02:13.661449 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 17473:18901, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661454 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 18901:20329, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661458 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 5281, win 248, options [nop,nop,TS val 45546978 ecr 1526325], length 0
22:02:13.661464 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 20329:21757, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661468 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 21757:23185, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661472 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 5521, win 263, options [nop,nop,TS val 45546978 ecr 1526325], length 0
22:02:13.661478 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 23185:24613, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661483 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 24613:26041, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661486 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 5713, win 278, options [nop,nop,TS val 45546978 ecr 1526325], length 0
22:02:13.661493 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 26041:27469, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661496 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 27469:28897, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546978], length 1428
22:02:13.661500 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 5921, win 293, options [nop,nop,TS val 45546979 ecr 1526325], length 0
22:02:13.661506 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 28897:30325, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546979], length 1428
22:02:13.661510 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 30325:31753, ack 2802, win 149, options [nop,nop,TS val 1526389 ecr 45546979], length 1428
22:02:13.662006 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240
22:02:13.667419 IP6 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943 > 2001:67c:29f4::50.22: Flags [.], ack 6049, win 307, options [nop,nop,TS val 45546979 ecr 1526325], length 0
22:02:13.667437 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 31753:33181, ack 2802, win 149, options [nop,nop,TS val 1526395 ecr 45546979], length 1428
22:02:13.667444 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 33181:34609, ack 2802, win 149, options [nop,nop,TS val 1526395 ecr 45546979], length 1428
22:02:13.667724 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240
22:02:13.667743 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240
22:02:13.794924 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 34609:36037, ack 2802, win 149, options [nop,nop,TS val 1526523 ecr 45546979], length 1428
22:02:13.795182 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240
22:02:14.059981 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 6049:7477, ack 2802, win 149, options [nop,nop,TS val 1526788 ecr 45546979], length 1428
22:02:14.060308 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240
22:02:14.589978 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 6049:7477, ack 2802, win 149, options [nop,nop,TS val 1527318 ecr 45546979], length 1428
22:02:14.590185 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240
22:02:15.647967 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 6049:7477, ack 2802, win 149, options [nop,nop,TS val 1528376 ecr 45546979], length 1428
22:02:15.648223 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240
22:02:17.768006 IP6 2001:67c:29f4::50.22 > 2001:67c:a4:1:5c5f:5194:3a7e:4878.48943: Flags [.], seq 6049:7477, ack 2802, win 149, options [nop,nop,TS val 1530496 ecr 45546979], length 1428
22:02:17.768307 IP6 2001:67c:29f4::31 > 2001:67c:29f4::50: ICMP6, packet too big, mtu 1468, length 1240

So the “packet too big” packets really look like they're being ignored.
However, they _do_ reach the kernel somehow, since Icmp6InPktTooBigs
seems to increase.

Could this be related somehow to the packets coming from 2001:67c:29f4::31,
while the default route is to a link-local address? (An RPF issue?) This used
to work (although it was often flaky for me) in 3.10 and before. I can't
easily bisect, though, as I don't boot this machine too often.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Hannes Frederic Sowa]

Hello!

On Fri, Sep 27, 2013 at 10:14:20PM +0200, Steinar H. Gunderson wrote:
> So the “packet too big” packets really look like they're being ignored.
> However, they _do_ reach the kernel somehow, since Icmp6InPktTooBigs
> seems to increase.
> 
> Could this be related somehow to the packets coming from 2001:67c:29f4::31,
> while the default route is to a link-local address? (An RPF issue?) This used
> to work (although it was often flaky for me) in 3.10 and before. I can't
> easily bisect, though, as I don't boot this machine too often.

This looks like a bug and should definitely get fixed. There should be
no RPF issue. May I have a look at your /proc/net/ipv6_route?

Thanks,

  Hannes

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

On Sat, Sep 28, 2013 at 10:33:18PM +0200, Hannes Frederic Sowa wrote:
>> Could this be related somehow to the packets coming from 2001:67c:29f4::31,
>> while the default route is to a link-local address? (An RPF issue?) This used
>> to work (although it was often flaky for me) in 3.10 and before. I can't
>> easily bisect, though, as I don't boot this machine too often.
> This looks like a bug and should definitely get fixed. There should be
> no RPF issue. May I have a look at your /proc/net/ipv6_route?

Hi,

I removed all the “weird” routes, and confirmed it fixed the problem.
However, upon adding them back again, the problem was still gone
(despite flushing the route cache).

This means that the issue has gone back to being intermittent, which is of
course the worst kind of bug to trace down. :-) I'll dump
/proc/net/ipv6_route and send you once I see the bug manifest itself again,
OK?

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Hannes Frederic Sowa]

On Sat, Sep 28, 2013 at 10:51:31PM +0200, Steinar H. Gunderson wrote:
> On Sat, Sep 28, 2013 at 10:33:18PM +0200, Hannes Frederic Sowa wrote:
> >> Could this be related somehow to the packets coming from 2001:67c:29f4::31,
> >> while the default route is to a link-local address? (An RPF issue?) This used
> >> to work (although it was often flaky for me) in 3.10 and before. I can't
> >> easily bisect, though, as I don't boot this machine too often.
> > This looks like a bug and should definitely get fixed. There should be
> > no RPF issue. May I have a look at your /proc/net/ipv6_route?
> 
> Hi,
> 
> I removed all the “weird” routes, and confirmed it fixed the problem.
> However, upon adding them back again, the problem was still gone
> (despite flushing the route cache).
> 
> This means that the issue has gone back to being intermittent, which is of
> course the worst kind of bug to trace down. :-) I'll dump
> /proc/net/ipv6_route and send you once I see the bug manifest itself again,
> OK?

Yes, that would be very helpful.

Also, you can try to churn up your bgp connection a bit so that the fib
serial numbers get incremented a lot (drop and install new routes). When
tcp_ipv6 processes the icmp errors it will drop the in-socket cached
routing entry then and will reinstall a relookuped one.  This is my only
suspect currently. If that would help to reproduce the problem the suspects
would be the changes in the next-hop selection. Sorry, no other idea
currently.

Thanks,

  Hannes

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

On Sat, Sep 28, 2013 at 10:33:18PM +0200, Hannes Frederic Sowa wrote:
>> So the “packet too big” packets really look like they're being ignored.
>> However, they _do_ reach the kernel somehow, since Icmp6InPktTooBigs
>> seems to increase.
>> 
>> Could this be related somehow to the packets coming from 2001:67c:29f4::31,
>> while the default route is to a link-local address? (An RPF issue?) This used
>> to work (although it was often flaky for me) in 3.10 and before. I can't
>> easily bisect, though, as I don't boot this machine too often.
> This looks like a bug and should definitely get fixed. There should be
> no RPF issue. May I have a look at your /proc/net/ipv6_route?

It started again, so now I could capture what you asked for:

pannekake:~> cat /proc/net/ipv6_route 
20010500007200000000000000000000 30 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000400 00000000 00000000 00000003     eth0
2001067c00a400037c4d9ae8ab73230f 80 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000000 00000001 00000137 01000023     eth0
2001067c00a400000000000000000000 30 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000064 00000000 00000000 00000003     eth0
2001067c29f400000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00003dc0 01000001     eth0
2001067c29f400000000000000000031 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00047359 01000001     eth0
2001067c29f400000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000001 00000001     eth0
2001067c29f400010000000000000000 40 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000064 00000000 00000000 00000003     eth0
2001067c29f410000000000000000000 40 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000064 00000000 00000000 00000003     eth0
2001067c29f410010000000000000000 40 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000064 00000000 00000000 00000003     eth0
2001067c29f410030000000000000000 40 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000064 00000000 00000000 00000003     eth0
2001067c29f410050000000000000000 40 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000064 00000000 00000000 00000003     eth0
2001067c29f410070000000000000000 40 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000064 00000000 00000000 00000003     eth0
2001067c29f400000000000000000000 30 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000400 00000000 00000000 00000003     eth0
20010700000000000000000000000000 20 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000400 00000000 00000000 00000003     eth0
2001070800402001a822bafffec42428 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000001 0000ab13 01000003     eth0
20010840000010000001000000000001 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000001 0001114c 01000003     eth0
26200000105f0004be305bfffed17063 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000003 0000002f 01000003     eth0
2a01023842d7aa006667669799990042 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000001 000081ef 01000003     eth0
2a022368000000000000000000000000 20 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000400 00000000 00000000 00000003     eth0
fe80000000000000000000002ae38049 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000001 01000001  k_magne
fe80000000000000000000002cb7a217 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000004 01000001 k_molvenfinnoy
fe8000000000000000000000315e851e 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000001 01000001 k_trygve
fe8000000000000000000000419b407a 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000001 01000001 k_molven
fe80000000000000000000005ccc2ed1 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000001 01000001 k_sessesveits
fe800000000000000000000077707d72 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000001 01000001  k_berge
fe8000000000000000000000c2faf88c 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000001 01000001 k_wikene
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_sessesveits
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_trygve
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_wikene
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_molven
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  k_magne
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  k_berge
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_molvenfinnoy
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001     eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_sessesveits
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_trygve
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_wikene
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_molven
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  k_magne
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  k_berge
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_molvenfinnoy
00000000000000000000000000000000 00 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000400 00000000 00000000 00000003     eth0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0009a4db 00200200       lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0000a62c 80200001       lo
2001067c29f400000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001       lo
2001067c29f400000000000000000050 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 0000000a 1a692b94 80200001       lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001       lo
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 0000602f 80200001       lo
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 00003c59 80200001       lo
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 0000530e 80200001       lo
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 00005bbb 80200001       lo
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 00003fad 80200001       lo
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 00002953 80200001       lo
fe8000000000000000000000c30b9a61 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001       lo
fe80000000000000022590fffe006b52 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0000541c 80200001       lo
ff020000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000002 01000001     eth0
ff02000000000000000000000000000d 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000003a 01000001     eth0
ff02000000000000000000000000000d 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000000b 01000001 k_sessesveits
ff02000000000000000000000000000d 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000000b 01000001 k_molven
ff020000000000000000000000010002 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000004 01000001     eth0
ff0200000000000000000001ff000020 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001     eth0
ff0200000000000000000001ff000029 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000000a 01000001     eth0
ff0200000000000000000001ff000030 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000004 01000001     eth0
ff0200000000000000000001ff000031 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001     eth0
ff0200000000000000000001ff000032 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000045 01000001     eth0
ff0200000000000000000001ff000057 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001     eth0
ff0200000000000000000001ff003004 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001     eth0
ff0200000000000000000001ff003005 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000002 01000001     eth0
ff0200000000000000000001ff4793da 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000003 01000001     eth0
ff0200000000000000000001ff5945c2 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000003 01000001     eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001     eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_sessesveits
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_trygve
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_wikene
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_molven
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  k_magne
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  k_berge
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 k_molvenfinnoy
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0009a4db 00200200       lo

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Hannes Frederic Sowa]

On Sun, Oct 06, 2013 at 02:06:12PM +0200, Steinar H. Gunderson wrote:
> On Sat, Sep 28, 2013 at 10:33:18PM +0200, Hannes Frederic Sowa wrote:
> >> So the “packet too big” packets really look like they're being ignored.
> >> However, they _do_ reach the kernel somehow, since Icmp6InPktTooBigs
> >> seems to increase.
> >> 
> >> Could this be related somehow to the packets coming from 2001:67c:29f4::31,
> >> while the default route is to a link-local address? (An RPF issue?) This used
> >> to work (although it was often flaky for me) in 3.10 and before. I can't
> >> easily bisect, though, as I don't boot this machine too often.
> > This looks like a bug and should definitely get fixed. There should be
> > no RPF issue. May I have a look at your /proc/net/ipv6_route?
> 
> It started again, so now I could capture what you asked for:

Thanks!

Do you know which remote IP address is part in the failing communication? Same
as above?

Greetings,

  Hannes

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

On Sun, Oct 06, 2013 at 02:44:03PM +0200, Hannes Frederic Sowa wrote:
> Do you know which remote IP address is part in the failing communication? Same
> as above?

It's in 2001:67c:a4:3::/64 or 2001:67c:a4:1::/64. Unsure which, because now
I rebooted the client for a different reason (giving it a different privacy
address and possibly subnet), and now it's unreproducible again...

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Hannes Frederic Sowa]

On Sun, Oct 06, 2013 at 02:06:12PM +0200, Steinar H. Gunderson wrote:
> On Sat, Sep 28, 2013 at 10:33:18PM +0200, Hannes Frederic Sowa wrote:
> >> So the “packet too big” packets really look like they're being ignored.
> >> However, they _do_ reach the kernel somehow, since Icmp6InPktTooBigs
> >> seems to increase.
> >> 
> >> Could this be related somehow to the packets coming from 2001:67c:29f4::31,
> >> while the default route is to a link-local address? (An RPF issue?) This used
> >> to work (although it was often flaky for me) in 3.10 and before. I can't
> >> easily bisect, though, as I don't boot this machine too often.
> > This looks like a bug and should definitely get fixed. There should be
> > no RPF issue. May I have a look at your /proc/net/ipv6_route?
> 
> It started again, so now I could capture what you asked for:
> 
> pannekake:~> cat /proc/net/ipv6_route 
> 2001067c00a400037c4d9ae8ab73230f 80 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000000 00000001 00000137 01000023     eth0

This one does look like the most probable route which could have the problem.
It has a RTF_MODIFIED flag indicating it received a pmtu update.

Did you take the snapshot while the tcp connection was hanging? We normally
take 2 references to the rt6_info while the tcp connection is running, this
oddly only has one (but got used a lot). But doing a judgement on the
reference count is imprecise.

If you write that this got worse in recent kernels I suspect commit

commit ca4c3fc24e293719fe7410c4e63da9b6bc633b83
Author: fan.du <fan.du@windriver.com>
Date:   Tue Jul 30 08:33:53 2013 +0800

    net: split rt_genid for ipv4 and ipv6


The commit itself is fine, we may have a problem in our dst check logic
or do not bump rt6_genid at some point? If this is the case I might have
an idea how to reproduce the problem.

Greetings,

  Hannes

Re: IPv6 path MTU discovery broken

[Hannes Frederic Sowa]

On Mon, Oct 07, 2013 at 03:52:52AM +0200, Hannes Frederic Sowa wrote:
> On Sun, Oct 06, 2013 at 02:06:12PM +0200, Steinar H. Gunderson wrote:
> > On Sat, Sep 28, 2013 at 10:33:18PM +0200, Hannes Frederic Sowa wrote:
> > >> So the “packet too big” packets really look like they're being ignored.
> > >> However, they _do_ reach the kernel somehow, since Icmp6InPktTooBigs
> > >> seems to increase.
> > >> 
> > >> Could this be related somehow to the packets coming from 2001:67c:29f4::31,
> > >> while the default route is to a link-local address? (An RPF issue?) This used
> > >> to work (although it was often flaky for me) in 3.10 and before. I can't
> > >> easily bisect, though, as I don't boot this machine too often.
> > > This looks like a bug and should definitely get fixed. There should be
> > > no RPF issue. May I have a look at your /proc/net/ipv6_route?
> > 
> > It started again, so now I could capture what you asked for:
> > 
> > pannekake:~> cat /proc/net/ipv6_route 
> > 2001067c00a400037c4d9ae8ab73230f 80 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000000 00000001 00000137 01000023     eth0
> 
> This one does look like the most probable route which could have the problem.
> It has a RTF_MODIFIED flag indicating it received a pmtu update.
> 
> Did you take the snapshot while the tcp connection was hanging? We normally
> take 2 references to the rt6_info while the tcp connection is running, this
> oddly only has one (but got used a lot). But doing a judgement on the
> reference count is imprecise.
> 
> If you write that this got worse in recent kernels I suspect commit
> 
> commit ca4c3fc24e293719fe7410c4e63da9b6bc633b83
> Author: fan.du <fan.du@windriver.com>
> Date:   Tue Jul 30 08:33:53 2013 +0800
> 
>     net: split rt_genid for ipv4 and ipv6
> 

Hm, this actually went in in 3.12, so a bit too late for things to start
failing in 3.11.

> The commit itself is fine, we may have a problem in our dst check logic
> or do not bump rt6_genid at some point? If this is the case I might have
> an idea how to reproduce the problem.

Seems fine.

Could you try to check (with e.g. nstat) if any of the following counters
change if the icmp messages hit the host?

TcpExtOutOfWindowIcmps
Icmp6InErrors
TcpExtTCPMinTTLDrop
TcpExtListenDrops

Thanks,

  Hannes

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

On Mon, Oct 07, 2013 at 03:52:52AM +0200, Hannes Frederic Sowa wrote:
>> 2001067c00a400037c4d9ae8ab73230f 80 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000000 00000001 00000137 01000023     eth0
> Did you take the snapshot while the tcp connection was hanging? We normally
> take 2 references to the rt6_info while the tcp connection is running, this
> oddly only has one (but got used a lot). But doing a judgement on the
> reference count is imprecise.

This was while the problem was present; I had broken the TCP connection,
though.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

On Mon, Oct 07, 2013 at 05:09:10AM +0200, Hannes Frederic Sowa wrote:
> Could you try to check (with e.g. nstat) if any of the following counters
> change if the icmp messages hit the host?
> 
> TcpExtOutOfWindowIcmps
> Icmp6InErrors
> TcpExtTCPMinTTLDrop
> TcpExtListenDrops

Icmp6InErrors is 0, so that's not it. How do I find the Tcp* counters?
None of them show up in nstat, although other Tcp* counters do.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

On Mon, Oct 07, 2013 at 03:52:52AM +0200, Hannes Frederic Sowa wrote:
>> pannekake:~> cat /proc/net/ipv6_route 
>> 2001067c00a400037c4d9ae8ab73230f 80 00000000000000000000000000000000 00 fe80000000000000023048fffe555743 00000000 00000001 00000137 01000023     eth0
> This one does look like the most probable route which could have the problem.
> It has a RTF_MODIFIED flag indicating it received a pmtu update.

I got the idea to correlate auth.log with the email I sent:

Oct  6 14:01:12 pannekake sshd[13211]: Accepted keyboard-interactive/pam for
sesse from 2001:67c:a4:3:7c4d:9ae8:ab73:230f port 60116 ssh2

So that's definitely the right address.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Hannes Frederic Sowa]

On Mon, Oct 07, 2013 at 10:32:28AM +0200, Steinar H. Gunderson wrote:
> On Mon, Oct 07, 2013 at 05:09:10AM +0200, Hannes Frederic Sowa wrote:
> > Could you try to check (with e.g. nstat) if any of the following counters
> > change if the icmp messages hit the host?
> > 
> > TcpExtOutOfWindowIcmps
> > Icmp6InErrors
> > TcpExtTCPMinTTLDrop
> > TcpExtListenDrops
> 
> Icmp6InErrors is 0, so that's not it. How do I find the Tcp* counters?
> None of them show up in nstat, although other Tcp* counters do.

Strange, I have them in nstat:

$ nstat -z | egrep  '(TcpExtOutOfWindowIcmps|Icmp6InErrors|TcpExtTCPMinTTLDrop|TcpExtListenDrops)'
Icmp6InErrors                   0                  0.0
TcpExtOutOfWindowIcmps          0                  0.0
TcpExtListenDrops               0                  0.0
TcpExtTCPMinTTLDrop             0                  0.0

Otherwise you can manually extract them from /proc/net/netstat.

Greetings,

  Hannes

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

On Mon, Oct 07, 2013 at 04:32:06PM +0200, Hannes Frederic Sowa wrote:
> Strange, I have them in nstat:
> 
> $ nstat -z | egrep  '(TcpExtOutOfWindowIcmps|Icmp6InErrors|TcpExtTCPMinTTLDrop|TcpExtListenDrops)'

-z was the trick. (I've never used nstat before.)

pannekake:~> nstat -z | egrep '(TcpExtOutOfWindowIcmps|Icmp6InErrors|TcpExtTCPMinTTLDrop|TcpExtListenDrops)'              
Icmp6InErrors                   0                  0.0
TcpExtOutOfWindowIcmps          4                  0.0
TcpExtListenDrops               0                  0.0
TcpExtTCPMinTTLDrop             0                  0.0

Next time it triggers, I'll see if TcpExtOutOfWindowIcmps increases, then.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Eric Dumazet]

On Mon, 2013-10-07 at 16:34 +0200, Steinar H. Gunderson wrote:
> On Mon, Oct 07, 2013 at 04:32:06PM +0200, Hannes Frederic Sowa wrote:
> > Strange, I have them in nstat:
> > 
> > $ nstat -z | egrep  '(TcpExtOutOfWindowIcmps|Icmp6InErrors|TcpExtTCPMinTTLDrop|TcpExtListenDrops)'
> 
> -z was the trick. (I've never used nstat before.)

Best might be to use -a  (absolute counters).

Re: IPv6 path MTU discovery broken

[Steinar H. Gunderson]

[resending with proper quoting and cc]

On Mon, Oct 07, 2013 at 05:09:10AM +0200, Hannes Frederic Sowa wrote:
> Could you try to check (with e.g. nstat) if any of the following counters
> change if the icmp messages hit the host?
> 
> TcpExtOutOfWindowIcmps
> Icmp6InErrors
> TcpExtTCPMinTTLDrop
> TcpExtListenDrops

Hi,

I am now seeing the same problem against a 3.9.0 machine (that I've never had
problems with before). I looked through the four nstat counters, and none of
them are increasing. Here's /proc/net/ipv6_route from the server while it's
going on; the client is 2001:67c:a4:1:71f9:c84b:8b2a:c65b (second line) and
the server is 2001:67c:29f4::2007.

20010500002e00000000000000000001 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
2001067c00a4000171f9c84b8b2ac65b 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000003 000005fb 01000023  eth0.10
2001067c2804100100000000c21f2797 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
2001067c29f400000000000000000021 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 00000723 01000001  eth0.10
2001067c29f400000000000000000022 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000003c 01000001  eth0.10
2001067c29f400000000000000000029 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 0000000c 025f8b9d 01000001  eth0.10
2001067c29f400000000000000000031 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 000000c5 01000001  eth0.10
2001067c29f400000000000000000052 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000040 01000001  eth0.10
2001067c29f400000000000000000056 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00001a37 01000001  eth0.10
2001067c29f400000000000000000065 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000041 01000001  eth0.10
2001067c29f400000000000000000070 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 000073fb 01000001  eth0.10
2001067c29f400000000000000000072 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000002 01000001  eth0.10
2001067c29f400000000000000000086 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000244b 01000001  eth0.10
2001067c29f400000000000000002010 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000020 01000001  eth0.10
2001067c29f400000000000000002012 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 000001d6 01000001  eth0.10
2001067c29f400000000000000003001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000005f 01000001  eth0.10
2001067c29f400000000000000005001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001  eth0.10
2001067c29f400000000000000005015 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000016 01000001  eth0.10
2001067c29f400000000000000005201 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0000509f 01000001  eth0.10
2001067c29f400000000000000005205 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00039f0b 01000001  eth0.10
2001067c29f400000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000001 00000001  eth0.10
200107000300146281ec7e498c3e8d5e 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000002 01000003  eth0.10
200141d0006000000000000000000012 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
20014600000400110000000000000005 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000003 01000003  eth0.10
20014600000410120000000000000005 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000002 01000003  eth0.10
20014600000410120000000000000006 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
20014600000410120000000000000007 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000002 01000003  eth0.10
20014801000d00000000000000530001 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000002 01000003  eth0.10
2406da00ff0000000000000036d3b443 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000002 01000003  eth0.10
2406da00ff0000000000000036e313b8 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000002 01000003  eth0.10
26003c03000000000000000000000003 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
2607f8b040020c010000000000000154 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
2a001450400b0c010000000000000154 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
2a00145040130c00000000000000005f 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000002 01000003  eth0.10
2a0103b0000100fa000000000000000f 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
2a020fe0000100020001000000010110 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000003 01000003  eth0.10
2a028070000002010082021200620039 80 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000000 00000000 00000001 01000003  eth0.10
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001     eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  eth0.10
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001   eth0.4
00000000000000000000000000000000 00 00000000000000000000000000000000 00 2001067c29f400000000000000000001 00000400 00000000 00000000 00000003  eth0.10
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0a896d3b 00200200       lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000002 3348a5ad 80200001       lo
2001067c29f400000000000000000028 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 01afbbc1 80200001       lo
2001067c29f400000000000000000030 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000020 29f1bc63 80200001       lo
2001067c29f400000000000000000049 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 03174149 80200001       lo
2001067c29f400000000000000002001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 000b20f8 80200001       lo
2001067c29f400000000000000002002 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0003c38f 80200001       lo
2001067c29f400000000000000002003 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 000bf060 80200001       lo
2001067c29f400000000000000002004 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0000e383 80200001       lo
2001067c29f400000000000000002005 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00058cae 80200001       lo
2001067c29f400000000000000002006 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 000082f7 80200001       lo
2001067c29f400000000000000002007 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000003 001a353e 80200001       lo
fe80000000000000022590fffec352c8 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001       lo
fe80000000000000022590fffec352c8 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0020538d 80200001       lo
fe80000000000000022590fffec352c8 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001       lo
ff020000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000003 01000001  eth0.10
ff020000000000000000000000000002 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001  eth0.10
ff02000000000000000000000000000d 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000003 01000001  eth0.10
ff020000000000000000000000010003 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 0000000e 01000001  eth0.10
ff0200000000000000000001ff000030 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000002 01000001  eth0.10
ff0200000000000000000001ff000031 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000002 01000001  eth0.10
ff0200000000000000000001ff003005 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000003 01000001  eth0.10
ff0200000000000000000001ff006001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001  eth0.10
ff0200000000000000000001ff2c2de5 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001  eth0.10
ff0200000000000000000001ff4793da 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001  eth0.10
ff0200000000000000000001ff555743 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000001 01000001  eth0.10
ff0200000000000000000001ff5945c2 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000007 01000001  eth0.10
ff0200000000000000000001ffdae370 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000000 00000002 01000001  eth0.10
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001     eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001  eth0.10
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001   eth0.4
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0a896d3b 00200200       lo

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: IPv6 path MTU discovery broken

[Eric Dumazet]

On Sun, 2013-10-13 at 12:40 +0200, Steinar H. Gunderson wrote:
> [resending with proper quoting and cc]
> 
> On Mon, Oct 07, 2013 at 05:09:10AM +0200, Hannes Frederic Sowa wrote:
> > Could you try to check (with e.g. nstat) if any of the following counters
> > change if the icmp messages hit the host?
> > 
> > TcpExtOutOfWindowIcmps
> > Icmp6InErrors
> > TcpExtTCPMinTTLDrop
> > TcpExtListenDrops
> 
> Hi,
> 
> I am now seeing the same problem against a 3.9.0 machine (that I've never had
> problems with before). I looked through the four nstat counters, and none of
> them are increasing. Here's /proc/net/ipv6_route from the server while it's
> going on; the client is 2001:67c:a4:1:71f9:c84b:8b2a:c65b (second line) and
> the server is 2001:67c:29f4::2007.

I am not sure, but the skb_is_gso() checks seem too lazy.

Why don't we check that gso_size+headers is below mtu, I dont know.

Re: IPv6 path MTU discovery broken

[Hannes Frederic Sowa]

On Sun, Oct 13, 2013 at 09:51:03AM -0700, Eric Dumazet wrote:
> On Sun, 2013-10-13 at 12:40 +0200, Steinar H. Gunderson wrote:
> > [resending with proper quoting and cc]
> > 
> > On Mon, Oct 07, 2013 at 05:09:10AM +0200, Hannes Frederic Sowa wrote:
> > > Could you try to check (with e.g. nstat) if any of the following counters
> > > change if the icmp messages hit the host?
> > > 
> > > TcpExtOutOfWindowIcmps
> > > Icmp6InErrors
> > > TcpExtTCPMinTTLDrop
> > > TcpExtListenDrops
> > 
> > Hi,
> > 
> > I am now seeing the same problem against a 3.9.0 machine (that I've never had
> > problems with before). I looked through the four nstat counters, and none of
> > them are increasing. Here's /proc/net/ipv6_route from the server while it's
> > going on; the client is 2001:67c:a4:1:71f9:c84b:8b2a:c65b (second line) and
> > the server is 2001:67c:29f4::2007.
> 
> I am not sure, but the skb_is_gso() checks seem too lazy.
> 
> Why don't we check that gso_size+headers is below mtu, I dont know.

Was this intended for the UFO gso thread? ;)