icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Scott Hall]

I am fairly new to iptables and have recently been forced to launch a 
system that wasn't fully tested.  Now my users are having troubles 
getting to a couple of websites.  Everything seems to work fine, so far, 
other then a couple of web sites.   We have 9 phones and 8 computers all 
sharing this pipe without any other complaints so far.

I am using this setup to share a single T1 pipe for both voice (via IP 
telephone) and data.  That is the reason for the 500 MTU across the t1 
link.  The smaller MTU helps with the priority queueing on the voice 
traffic. 

Here is my setup.  

-- Customer network (10.1.4.0)--
                   |
                   |
      |   multihomed - 10.1.4.1/24  -- xxx.xxx.xx7.13/29        |
  -- |Customer side router| -- Fedora core 
1kernel-2.4.22-1.2115.nptl/iptables 1.2.8
      |   10.255.0.14/29    |
                   |
                   |
          (T1/ 500 MTU)
                   |
                   |
     |   10.255.0.13/29  |
<--| My side router -->|  -- Redhat 9 kernel-2.4.20-8smp/iptables 1.2.7a
     |   SNATed xxx.xxx.xx6.21/26   |


The problem happens when users on the customer side public or private 
(PUBLIC_IP or 10.1.4.0) network try to connect to a couple of different 
websites.  Here is the info from the tcpdump on the 'My side router' 
public interface.

 >>>>> Not sure why I don't see the original request <<<<<<<<
 >>>> Request for http connection from customer private network to 
Fasttrack2.machinerytrader.com <<<<<<<

16:48:57.275929 Fasttrack2.machinerytrader.com.http > 
--PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 
65161 (DF)
16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 
10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0]
16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain:  25222+ 
PTR? 17.164.70.63.in-addr.arpa. (43)  (DF)

 >>>>>> still don't see the original request go out. Not sure why <<<<<<<
 >>>>>> Request for connection from customer public network to 
Fasttrack2.machinerytrader.com


16:59:25.962167 Fasttrack2.machinerytrader.com.http > 
--cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF)
16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 
--cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0]

 >>>> I see this traffic next but the customer side doesn't receive the 
packets <<<<<<

16:59:33.605043 --cust-publicIP--.29695 > 
Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)
16:59:38.142184 --cust-publicIP--.29714 > 
Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 
65280 <mss 1360,nop,nop,sackOK> (DF)
16:59:38.174050 Fasttrack2.machinerytrader.com.http > 
--cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 
65535 <mss 1380,nop,nop,sackOK> (DF)
16:59:38.178358 --cust-publicIP--.29714 > 
Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)
16:59:38.181116 --cust-publicIP--.29714 > 
Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF)
16:59:38.216940 Fasttrack2.machinerytrader.com.http > 
--cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)
16:59:38.220596 Fasttrack2.machinerytrader.com.http > 
--cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)


I have done a fair amount of searching on this and come up with 
nothing.  Any help will be greatly appreciated.  I have read about 
several people with similar issues that tried changing the MTU to no 
avail.  Based on the QoS stress testing we did do before putting the 
server in production, we need to leave the MTU set to 500 for best service.

Thanks to anyone who can help

--Scott

Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Chris Brenton]

GReets Scott,

On Tue, 2004-01-06 at 02:07, Scott Hall wrote:
>
> I am using this setup to share a single T1 pipe for both voice (via IP 
> telephone) and data.  That is the reason for the 500 MTU across the t1 
> link. 

You may find you are going to have this problem a bit more often. See
below.

>  >>>>> Not sure why I don't see the original request <<<<<<<<

What do you mean by "original request"? DNS query? TCP three packet
handshake? Both appear to be missing from your traces. The DNS info
could have still been in cache however.

> 16:48:57.275929 Fasttrack2.machinerytrader.com.http > 
> --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 
> 65161 (DF)

ACK packet with absolute sequence numbers tells me we missed the initial
handshake somehow. From the window size it looks like we are well into
the data session.

> 16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 
> 10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0]

Two problems here:
1) A lot of networks, and even some ISP's for that matter, are now
dropping all ICMP due to some of the Window's worms running around.
Don't tell me that its a brain dead idea, you are preaching to the
choir, just letting you know I see people doing it in the wild.

Also, even if the sending system does receive this packet, I'm guessing
it will pitch it. You are telling the remote Web server that the
connection to 10.1.4.50 needs to have its traffic fragmented. I'm
guessing the Web server sees the connection as originating from a public
address. (the NAT address you are using). Since the payload IP does not
match, it will disregard the packet as an unsolicited ICMP error.

> 16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain:  25222+ 
> PTR? 17.164.70.63.in-addr.arpa. (43)  (DF)

Weird its doing a reverse lookup, but not a problem.

> 16:59:25.962167 Fasttrack2.machinerytrader.com.http > 
> --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF)

OK, completely new session well that's a few packets into the data
transfer.

> 16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 
> --cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0]

There goes our ICMP error with a private address in the payload again.
:)

>  >>>> I see this traffic next but the customer side doesn't receive the 
> packets <<<<<<
> 
> 16:59:33.605043 --cust-publicIP--.29695 > 
> Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)

OK, something is weird. If the frag info did not get though we should
have seen the Web site attempt some retransmissions. Also, we're looking
at absolute sequence numbers. I think we again didn't get to see some of
the packets in the session. Are you stopping and restarting your packet
capture?

> 16:59:38.142184 --cust-publicIP--.29714 > 
> Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 
> 65280 <mss 1360,nop,nop,sackOK> (DF)
> 16:59:38.174050 Fasttrack2.machinerytrader.com.http > 
> --cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 
> 65535 <mss 1380,nop,nop,sackOK> (DF)
> 16:59:38.178358 --cust-publicIP--.29714 > 
> Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)

WAHOO! A three packet handshake. This time we got in from the beginning!
:)

> 16:59:38.181116 --cust-publicIP--.29714 > 
> Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF)
> 16:59:38.216940 Fasttrack2.machinerytrader.com.http > 
> --cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)

OK, up to here we are good because your MTU did not get exceeded. 

> 16:59:38.220596 Fasttrack2.machinerytrader.com.http > 
> --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)

And again, we hit an MTU that exceeds you max, so this is were the
problems start. I'm guessing after this we had an outbound ICMP frag
needed, followed by a couple of inbound retransmissions of the above
packet, but it would have been cool to see it in order to confirm.

HTH,
C

Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Scott Hall]

[-- Attachment #1: Type: text/html, Size: 6467 bytes --]

Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Scott Hall]

So the one question that this whole issue raises in my mind is, Isn't 
there anyway to handle the (DF) packets differently?  These packets 
aren't important as far as QoS they just need to be delivered.  I ask 
becuase we have two cisco routers and 6 Adtran routers that handle this 
same scenario quietly.  Does anyone have any idea how they handle the 
"don't frag" packets?  I wonder if there is someway to ignore the DF bit 
and frag the packets anyway. 

Thanks for any suggestions you can offer.

--Scott

Chris Brenton wrote:

>GReets Scott,
>
>On Tue, 2004-01-06 at 02:07, Scott Hall wrote:
>  
>
>>I am using this setup to share a single T1 pipe for both voice (via IP 
>>telephone) and data.  That is the reason for the 500 MTU across the t1 
>>link. 
>>    
>>
>
>You may find you are going to have this problem a bit more often. See
>below.
>
>  
>
>> >>>>> Not sure why I don't see the original request <<<<<<<<
>>    
>>
>
>What do you mean by "original request"? DNS query? TCP three packet
>handshake? Both appear to be missing from your traces. The DNS info
>could have still been in cache however.
>
>  
>
>>16:48:57.275929 Fasttrack2.machinerytrader.com.http > 
>>--PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 
>>65161 (DF)
>>    
>>
>
>ACK packet with absolute sequence numbers tells me we missed the initial
>handshake somehow. From the window size it looks like we are well into
>the data session.
>
>  
>
>>16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 
>>10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0]
>>    
>>
>
>Two problems here:
>1) A lot of networks, and even some ISP's for that matter, are now
>dropping all ICMP due to some of the Window's worms running around.
>Don't tell me that its a brain dead idea, you are preaching to the
>choir, just letting you know I see people doing it in the wild.
>
>Also, even if the sending system does receive this packet, I'm guessing
>it will pitch it. You are telling the remote Web server that the
>connection to 10.1.4.50 needs to have its traffic fragmented. I'm
>guessing the Web server sees the connection as originating from a public
>address. (the NAT address you are using). Since the payload IP does not
>match, it will disregard the packet as an unsolicited ICMP error.
>
>  
>
>>16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain:  25222+ 
>>PTR? 17.164.70.63.in-addr.arpa. (43)  (DF)
>>    
>>
>
>Weird its doing a reverse lookup, but not a problem.
>
>  
>
>>16:59:25.962167 Fasttrack2.machinerytrader.com.http > 
>>--cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF)
>>    
>>
>
>OK, completely new session well that's a few packets into the data
>transfer.
>
>  
>
>>16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 
>>--cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0]
>>    
>>
>
>There goes our ICMP error with a private address in the payload again.
>:)
>
>  
>
>> >>>> I see this traffic next but the customer side doesn't receive the 
>>packets <<<<<<
>>
>>16:59:33.605043 --cust-publicIP--.29695 > 
>>Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)
>>    
>>
>
>OK, something is weird. If the frag info did not get though we should
>have seen the Web site attempt some retransmissions. Also, we're looking
>at absolute sequence numbers. I think we again didn't get to see some of
>the packets in the session. Are you stopping and restarting your packet
>capture?
>
>  
>
>>16:59:38.142184 --cust-publicIP--.29714 > 
>>Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 
>>65280 <mss 1360,nop,nop,sackOK> (DF)
>>16:59:38.174050 Fasttrack2.machinerytrader.com.http > 
>>--cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 
>>65535 <mss 1380,nop,nop,sackOK> (DF)
>>16:59:38.178358 --cust-publicIP--.29714 > 
>>Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)
>>    
>>
>
>WAHOO! A three packet handshake. This time we got in from the beginning!
>:)
>
>  
>
>>16:59:38.181116 --cust-publicIP--.29714 > 
>>Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF)
>>16:59:38.216940 Fasttrack2.machinerytrader.com.http > 
>>--cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)
>>    
>>
>
>OK, up to here we are good because your MTU did not get exceeded. 
>
>  
>
>>16:59:38.220596 Fasttrack2.machinerytrader.com.http > 
>>--cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)
>>    
>>
>
>And again, we hit an MTU that exceeds you max, so this is were the
>problems start. I'm guessing after this we had an outbound ICMP frag
>needed, followed by a couple of inbound retransmissions of the above
>packet, but it would have been cool to see it in order to confirm.
>
>HTH,
>C
>
>
>
>  
>

Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Chris Brenton]

On Tue, 2004-01-13 at 03:02, Scott Hall wrote:
> So the one question that this whole issue raises in my mind is, Isn't 
> there anyway to handle the (DF) packets differently?

Absolutely. Config the stacks on both ends of the connection to _not_
set DF. This will cause the router at the MTU border to frag the packets
and will not require an ICMP error packet.

> I ask 
> becuase we have two cisco routers and 6 Adtran routers that handle this 
> same scenario quietly. 

I'm guessing if you check the decodes from those packets you will see
the public rather than the private IP embedded in the payload. I think
this is what is killing you. This is an old Netfilter bug that I
*thought* was fixed ages ago.

HTH,
C

Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Scott Hall]

Thank for the response Chris,

The problem that I see with that solution is that most of the sites, 
which are many by this point, that I have had problems with aren't under 
my control.  Including aol.com, I can just see me trying to convince AOL 
to reconfigure their servers to not set the DF :).  Are there anyother 
work arounds that you can propose? 

thnx,
  --scott

Chris Brenton wrote:

>On Tue, 2004-01-13 at 03:02, Scott Hall wrote:
>  
>
>>So the one question that this whole issue raises in my mind is, Isn't 
>>there anyway to handle the (DF) packets differently?
>>    
>>
>
>Absolutely. Config the stacks on both ends of the connection to _not_
>set DF. This will cause the router at the MTU border to frag the packets
>and will not require an ICMP error packet.
>
>  
>
>>I ask 
>>becuase we have two cisco routers and 6 Adtran routers that handle this 
>>same scenario quietly. 
>>    
>>
>
>I'm guessing if you check the decodes from those packets you will see
>the public rather than the private IP embedded in the payload. I think
>this is what is killing you. This is an old Netfilter bug that I
>*thought* was fixed ages ago.
>
>HTH,
>C
>
>
>
>  
>

Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Chris Brenton]

On Tue, 2004-01-13 at 11:12, Scott Hall wrote:
>
> Thank for the response Chris,

Glad to!

> Are there anyother work arounds that you can propose? 

Do a:
iptables -V

I'm guessing you are running an older version that is not patched for
this problem (1.2.6a or prior). Here is the original advisory:
http://www.linuxsecurity.com/advisories/other_advisory-2063.html

HTH,
C

Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Scott Hall]

Here is the iptables -V from both routers.  the 1.2.7a is the my side 
router and the router that the original tcpdump traffic I posted came from.

[root@gandalf root]# iptables -V
iptables v1.2.7a (My side)

[root@indinj root]# iptables -V
iptables v1.2.8 (customer side)

--Scott


Chris Brenton wrote:

>On Tue, 2004-01-13 at 11:12, Scott Hall wrote:
>  
>
>>Thank for the response Chris,
>>    
>>
>
>Glad to!
>
>  
>
>>Are there anyother work arounds that you can propose? 
>>    
>>
>
>Do a:
>iptables -V
>
>I'm guessing you are running an older version that is not patched for
>this problem (1.2.6a or prior). Here is the original advisory:
>http://www.linuxsecurity.com/advisories/other_advisory-2063.html
>
>HTH,
>C
>
>
>
>  
>

RE: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

[Mark Weaver]

> Do a:
> iptables -V
>
> I'm guessing you are running an older version that is not patched for
> this problem (1.2.6a or prior). Here is the original advisory:
> http://www.linuxsecurity.com/advisories/other_advisory-2063.html
>
That's not enough: you need a patched (or later) kernel as well as the bug
actually existed in the netfilter module.  I can't remember OTOMH which
kernel release this went into, although it was much later than the mentioned
version because the kernel team rejected the original fix (for some good
reasons).  I know 2.4.23+ don't have this problem.