tcp timestamp issues with google servers

tcp timestamp issues with google servers

[Miklos Szeredi]

Sometimes connection to google.com, gmail.com and other google servers
doesn't work or takes ages to connect.  When this hits it hits all
google servers at the same time and it's persistent.  It never happens
to anything other than google.  Rebooting helps.  Rarely it goes away
spontaneously.

Apparently google is sometimes replying with an invalid TSecr timestamp
value (smaller than the one sent in the last packet) and this confuses
the Linux TCP stack which either discards the packet or sends a Reset.

Network dump attached.

I found only a couple of references to this issue:

http://gotchas.livejournal.com/3028.html

http://groups.google.com/group/comp.os.linux.networking/browse_thread/thread/29f56feded11b42a

Turning tcp timestamps fixes the issue:

  sysctl -w net.ipv4.tcp_timestamps=0

Not sure why this happens only to me and a very few others.

It appears to be an issue with google TCP stack (is it a modified
stack?) but I thought about issues in my network switch (restarting it
doesn't help) or something in the ISP, but those look unlikely.

Any ideas?

Thanks,
Miklos



  1   0.000000 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35355050 TSER=0 WS=5
  2   0.002730 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=0 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184565067 TSER=35325344 WS=6
  3   0.002776 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [RST] Seq=1 Win=0 Len=0
  4   1.001408 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35356052 TSER=0 WS=5
  5   1.004136 74.125.232.226 -> 192.168.28.100 TCP [TCP Previous segment lost] http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184566068 TSER=35325344 WS=6
  6   1.411915 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184566476 TSER=35325344 WS=6
  7   2.011568 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184567076 TSER=35325344 WS=6
  8   3.005400 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35358056 TSER=0 WS=5
  9   3.007972 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184568072 TSER=35325344 WS=6
 10   3.212862 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184568277 TSER=35325344 WS=6
 11   5.612449 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184570677 TSER=35325344 WS=6
 12   7.013405 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35362064 TSER=0 WS=5
 13   7.016627 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184572080 TSER=35325344 WS=6
 14  10.412642 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184575477 TSER=35325344 WS=6
 15  15.029547 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35370080 TSER=0 WS=5
 16  15.032931 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=15638919 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184580097 TSER=35325344 WS=6
 17  31.061400 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35386112 TSER=0 WS=5
 18  31.064538 74.125.232.226 -> 192.168.28.100 TCP [TCP Previous segment lost] http > 51303 [SYN, ACK] Seq=485350292 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184596129 TSER=35325344 WS=6
 19  31.416339 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=485350292 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184596480 TSER=35325344 WS=6
 20  32.015998 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=485350292 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184597081 TSER=35325344 WS=6
 21  33.216276 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=485350292 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184598281 TSER=35325344 WS=6
 22  35.616879 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=485350292 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184600681 TSER=35325344 WS=6
 23  40.417065 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=485350292 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184605482 TSER=35325344 WS=6

Re: tcp timestamp issues with google servers

[Eric Dumazet]

On Thu, 2012-05-17 at 11:39 +0200, Miklos Szeredi wrote:
> Sometimes connection to google.com, gmail.com and other google servers
> doesn't work or takes ages to connect.  When this hits it hits all
> google servers at the same time and it's persistent.  It never happens
> to anything other than google.  Rebooting helps.  Rarely it goes away
> spontaneously.
> 
> Apparently google is sometimes replying with an invalid TSecr timestamp
> value (smaller than the one sent in the last packet) and this confuses
> the Linux TCP stack which either discards the packet or sends a Reset.
> 
> Network dump attached.
> 
> I found only a couple of references to this issue:
> 
> http://gotchas.livejournal.com/3028.html
> 
> http://groups.google.com/group/comp.os.linux.networking/browse_thread/thread/29f56feded11b42a
> 
> Turning tcp timestamps fixes the issue:
> 
>   sysctl -w net.ipv4.tcp_timestamps=0
> 
> Not sure why this happens only to me and a very few others.
> 
> It appears to be an issue with google TCP stack (is it a modified
> stack?) but I thought about issues in my network switch (restarting it
> doesn't help) or something in the ISP, but those look unlikely.
> 
> Any ideas?
> 
> Thanks,
> Miklos
> 
> 
> 
>   1   0.000000 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35355050 TSER=0 WS=5
>   2   0.002730 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=0 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184565067 TSER=35325344 WS=6


Do you really have 2730 usec RTT between you and this (Google ?)
server ?

Are you sure this is not a broken middle box ?

Re: tcp timestamp issues with google servers

[Miklos Szeredi]

Eric Dumazet <eric.dumazet@gmail.com> writes:

> On Thu, 2012-05-17 at 11:39 +0200, Miklos Szeredi wrote:
>> Sometimes connection to google.com, gmail.com and other google servers
>> doesn't work or takes ages to connect.  When this hits it hits all
>> google servers at the same time and it's persistent.  It never happens
>> to anything other than google.  Rebooting helps.  Rarely it goes away
>> spontaneously.
>> 
>> Apparently google is sometimes replying with an invalid TSecr timestamp
>> value (smaller than the one sent in the last packet) and this confuses
>> the Linux TCP stack which either discards the packet or sends a Reset.
>> 
>> Network dump attached.
>> 
>> I found only a couple of references to this issue:
>> 
>> http://gotchas.livejournal.com/3028.html
>> 
>> http://groups.google.com/group/comp.os.linux.networking/browse_thread/thread/29f56feded11b42a
>> 
>> Turning tcp timestamps fixes the issue:
>> 
>>   sysctl -w net.ipv4.tcp_timestamps=0
>> 
>> Not sure why this happens only to me and a very few others.
>> 
>> It appears to be an issue with google TCP stack (is it a modified
>> stack?) but I thought about issues in my network switch (restarting it
>> doesn't help) or something in the ISP, but those look unlikely.
>> 
>> Any ideas?
>> 
>> Thanks,
>> Miklos
>> 
>> 
>> 
>>   1   0.000000 192.168.28.100 -> 74.125.232.226 TCP 51303 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSV=35355050 TSER=0 WS=5
>>   2   0.002730 74.125.232.226 -> 192.168.28.100 TCP http > 51303 [SYN, ACK] Seq=0 Ack=1 Win=14180 Len=0 MSS=1430 SACK_PERM=1 TSV=1184565067 TSER=35325344 WS=6
>
>
> Do you really have 2730 usec RTT between you and this (Google ?)
> server ?

So it appears.  The IP address is certainly registered to Google.

Thanks,
Miklos

Re: tcp timestamp issues with google servers

[Eric Dumazet]

On Tue, 2012-05-22 at 17:25 +0200, Miklos Szeredi wrote:

> So it appears.  The IP address is certainly registered to Google.

Good, but you could have a middlebox doing transparent proxying.

The SYNACK could be send by this box.

Re: tcp timestamp issues with google servers

[Miklos Szeredi]

Eric Dumazet <eric.dumazet@gmail.com> writes:

> On Tue, 2012-05-22 at 17:25 +0200, Miklos Szeredi wrote:
>
>> So it appears.  The IP address is certainly registered to Google.
>
> Good, but you could have a middlebox doing transparent proxying.
>
> The SYNACK could be send by this box.

Okay.  Is there a way to find out whether there is a middlebox or not?

Thanks,
Miklos

Re: tcp timestamp issues with google servers

[Vijay Subramanian]

> Okay.  Is there a way to find out whether there is a middlebox or not?
>

Miklos,
Maybe tcptraceroute[1] can help you figure this out.

Hope this helps.
Vijay

[1] http://michael.toren.net/code/tcptraceroute/

Re: tcp timestamp issues with google servers

[Eric Dumazet]

On Tue, 2012-05-22 at 09:38 -0700, Vijay Subramanian wrote:
> > Okay.  Is there a way to find out whether there is a middlebox or not?
> >
> 
> Miklos,
> Maybe tcptraceroute[1] can help you figure this out.
> 
> Hope this helps.
> Vijay
> 
> [1] http://michael.toren.net/code/tcptraceroute/


The transparent proxy can intercept TCP connections to port 80/443, and
let ICMP being NATed by the box.

So its better to check of the delay between SYN and SYNACK is roughly
independent of the HTTP server.

If you have very large range of delays, you can conclude its not a
transparent proxy.

Re: tcp timestamp issues with google servers

[Rick Jones]

On 05/22/2012 08:54 AM, Miklos Szeredi wrote:
> Eric Dumazet<eric.dumazet@gmail.com>  writes:
>
>> On Tue, 2012-05-22 at 17:25 +0200, Miklos Szeredi wrote:
>>
>>> So it appears.  The IP address is certainly registered to Google.
>>
>> Good, but you could have a middlebox doing transparent proxying.
>>
>> The SYNACK could be send by this box.
>
> Okay.  Is there a way to find out whether there is a middlebox or not?

The source IP in the trace was a 192.168 IP - is it possible/desirable 
to reproduce the problem without the device doing NAT in the path?

What is your "public" IP address?  Given that, and the IP address to 
which you are connecting, it should be possible to validate the RTT you 
are seeing.  If the geographic/topological location of the destination 
Google IP address is far enough from your public source IP that would 
show whether  the RTT you are seeing is even physically possible and so 
could suggest there is a middlebox (other than your NAT), though it 
couldn't show there was not a middlebox.


rick jones

Re: tcp timestamp issues with google servers

[Vijay Subramanian]

>> Maybe tcptraceroute[1] can help you figure this out.
>>
>> [1] http://michael.toren.net/code/tcptraceroute/
>
>
> The transparent proxy can intercept TCP connections to port 80/443, and
> let ICMP being NATed by the box.

Just to be clear..tcptraceroute uses TCP SYN packets to trace the
route instead of using ICMP packets used by vanilla traceroute
precisely because
of the issue you raised.
The idea is that if the connection is getting terminated at a
middlebox, the trace will end there. Otherwise, the trace route will
end
at destination (google in this case). This avoids the problems of ICMP
and TCP flows being treated differently by the middlebox.
Is this approach workable?

Thanks,
Vijay

Re: tcp timestamp issues with google servers

[Eric Dumazet]

On Tue, 2012-05-22 at 10:38 -0700, Vijay Subramanian wrote:
> >> Maybe tcptraceroute[1] can help you figure this out.
> >>
> >> [1] http://michael.toren.net/code/tcptraceroute/
> >
> >
> > The transparent proxy can intercept TCP connections to port 80/443, and
> > let ICMP being NATed by the box.
> 
> Just to be clear..tcptraceroute uses TCP SYN packets to trace the
> route instead of using ICMP packets used by vanilla traceroute
> precisely because
> of the issue you raised.
> The idea is that if the connection is getting terminated at a
> middlebox, the trace will end there. Otherwise, the trace route will
> end
> at destination (google in this case). This avoids the problems of ICMP
> and TCP flows being treated differently by the middlebox.
> Is this approach workable?

Yes probably, thanks for detailed description (I indeed thought it was
traceroute)