* icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] @ 2004-01-06 7:07 Scott Hall 2004-01-06 11:23 ` Chris Brenton 0 siblings, 1 reply; 9+ messages in thread From: Scott Hall @ 2004-01-06 7:07 UTC (permalink / raw) To: netfilter I am fairly new to iptables and have recently been forced to launch a system that wasn't fully tested. Now my users are having troubles getting to a couple of websites. Everything seems to work fine, so far, other then a couple of web sites. We have 9 phones and 8 computers all sharing this pipe without any other complaints so far. I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link. The smaller MTU helps with the priority queueing on the voice traffic. Here is my setup. -- Customer network (10.1.4.0)-- | | | multihomed - 10.1.4.1/24 -- xxx.xxx.xx7.13/29 | -- |Customer side router| -- Fedora core 1kernel-2.4.22-1.2115.nptl/iptables 1.2.8 | 10.255.0.14/29 | | | (T1/ 500 MTU) | | | 10.255.0.13/29 | <--| My side router -->| -- Redhat 9 kernel-2.4.20-8smp/iptables 1.2.7a | SNATed xxx.xxx.xx6.21/26 | The problem happens when users on the customer side public or private (PUBLIC_IP or 10.1.4.0) network try to connect to a couple of different websites. Here is the info from the tcpdump on the 'My side router' public interface. >>>>> Not sure why I don't see the original request <<<<<<<< >>>> Request for http connection from customer private network to Fasttrack2.machinerytrader.com <<<<<<< 16:48:57.275929 Fasttrack2.machinerytrader.com.http > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 65161 (DF) 16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0] 16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ PTR? 17.164.70.63.in-addr.arpa. (43) (DF) >>>>>> still don't see the original request go out. Not sure why <<<<<<< >>>>>> Request for connection from customer public network to Fasttrack2.machinerytrader.com 16:59:25.962167 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF) 16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: --cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0] >>>> I see this traffic next but the customer side doesn't receive the packets <<<<<< 16:59:33.605043 --cust-publicIP--.29695 > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF) 16:59:38.142184 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 65280 <mss 1360,nop,nop,sackOK> (DF) 16:59:38.174050 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 65535 <mss 1380,nop,nop,sackOK> (DF) 16:59:38.178358 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF) 16:59:38.181116 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF) 16:59:38.216940 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF) 16:59:38.220596 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF) I have done a fair amount of searching on this and come up with nothing. Any help will be greatly appreciated. I have read about several people with similar issues that tried changing the MTU to no avail. Based on the QoS stress testing we did do before putting the server in production, we need to leave the MTU set to 500 for best service. Thanks to anyone who can help --Scott ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-06 7:07 icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] Scott Hall @ 2004-01-06 11:23 ` Chris Brenton 2004-01-06 15:48 ` Scott Hall 2004-01-13 8:02 ` Scott Hall 0 siblings, 2 replies; 9+ messages in thread From: Chris Brenton @ 2004-01-06 11:23 UTC (permalink / raw) To: Scott Hall; +Cc: netfilter GReets Scott, On Tue, 2004-01-06 at 02:07, Scott Hall wrote: > > I am using this setup to share a single T1 pipe for both voice (via IP > telephone) and data. That is the reason for the 500 MTU across the t1 > link. You may find you are going to have this problem a bit more often. See below. > >>>>> Not sure why I don't see the original request <<<<<<<< What do you mean by "original request"? DNS query? TCP three packet handshake? Both appear to be missing from your traces. The DNS info could have still been in cache however. > 16:48:57.275929 Fasttrack2.machinerytrader.com.http > > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win > 65161 (DF) ACK packet with absolute sequence numbers tells me we missed the initial handshake somehow. From the window size it looks like we are well into the data session. > 16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: > 10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0] Two problems here: 1) A lot of networks, and even some ISP's for that matter, are now dropping all ICMP due to some of the Window's worms running around. Don't tell me that its a brain dead idea, you are preaching to the choir, just letting you know I see people doing it in the wild. Also, even if the sending system does receive this packet, I'm guessing it will pitch it. You are telling the remote Web server that the connection to 10.1.4.50 needs to have its traffic fragmented. I'm guessing the Web server sees the connection as originating from a public address. (the NAT address you are using). Since the payload IP does not match, it will disregard the packet as an unsolicited ICMP error. > 16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ > PTR? 17.164.70.63.in-addr.arpa. (43) (DF) Weird its doing a reverse lookup, but not a problem. > 16:59:25.962167 Fasttrack2.machinerytrader.com.http > > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF) OK, completely new session well that's a few packets into the data transfer. > 16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: > --cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0] There goes our ICMP error with a private address in the payload again. :) > >>>> I see this traffic next but the customer side doesn't receive the > packets <<<<<< > > 16:59:33.605043 --cust-publicIP--.29695 > > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF) OK, something is weird. If the frag info did not get though we should have seen the Web site attempt some retransmissions. Also, we're looking at absolute sequence numbers. I think we again didn't get to see some of the packets in the session. Are you stopping and restarting your packet capture? > 16:59:38.142184 --cust-publicIP--.29714 > > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win > 65280 <mss 1360,nop,nop,sackOK> (DF) > 16:59:38.174050 Fasttrack2.machinerytrader.com.http > > --cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win > 65535 <mss 1380,nop,nop,sackOK> (DF) > 16:59:38.178358 --cust-publicIP--.29714 > > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF) WAHOO! A three packet handshake. This time we got in from the beginning! :) > 16:59:38.181116 --cust-publicIP--.29714 > > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF) > 16:59:38.216940 Fasttrack2.machinerytrader.com.http > > --cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF) OK, up to here we are good because your MTU did not get exceeded. > 16:59:38.220596 Fasttrack2.machinerytrader.com.http > > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF) And again, we hit an MTU that exceeds you max, so this is were the problems start. I'm guessing after this we had an outbound ICMP frag needed, followed by a couple of inbound retransmissions of the above packet, but it would have been cool to see it in order to confirm. HTH, C ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-06 11:23 ` Chris Brenton @ 2004-01-06 15:48 ` Scott Hall 2004-01-13 8:02 ` Scott Hall 1 sibling, 0 replies; 9+ messages in thread From: Scott Hall @ 2004-01-06 15:48 UTC (permalink / raw) To: Chris Brenton; +Cc: netfilter [-- Attachment #1: Type: text/html, Size: 6467 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-06 11:23 ` Chris Brenton 2004-01-06 15:48 ` Scott Hall @ 2004-01-13 8:02 ` Scott Hall 2004-01-13 15:51 ` Chris Brenton 1 sibling, 1 reply; 9+ messages in thread From: Scott Hall @ 2004-01-13 8:02 UTC (permalink / raw) To: Chris Brenton; +Cc: netfilter So the one question that this whole issue raises in my mind is, Isn't there anyway to handle the (DF) packets differently? These packets aren't important as far as QoS they just need to be delivered. I ask becuase we have two cisco routers and 6 Adtran routers that handle this same scenario quietly. Does anyone have any idea how they handle the "don't frag" packets? I wonder if there is someway to ignore the DF bit and frag the packets anyway. Thanks for any suggestions you can offer. --Scott Chris Brenton wrote: >GReets Scott, > >On Tue, 2004-01-06 at 02:07, Scott Hall wrote: > > >>I am using this setup to share a single T1 pipe for both voice (via IP >>telephone) and data. That is the reason for the 500 MTU across the t1 >>link. >> >> > >You may find you are going to have this problem a bit more often. See >below. > > > >> >>>>> Not sure why I don't see the original request <<<<<<<< >> >> > >What do you mean by "original request"? DNS query? TCP three packet >handshake? Both appear to be missing from your traces. The DNS info >could have still been in cache however. > > > >>16:48:57.275929 Fasttrack2.machinerytrader.com.http > >>--PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win >>65161 (DF) >> >> > >ACK packet with absolute sequence numbers tells me we missed the initial >handshake somehow. From the window size it looks like we are well into >the data session. > > > >>16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: >>10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0] >> >> > >Two problems here: >1) A lot of networks, and even some ISP's for that matter, are now >dropping all ICMP due to some of the Window's worms running around. >Don't tell me that its a brain dead idea, you are preaching to the >choir, just letting you know I see people doing it in the wild. > >Also, even if the sending system does receive this packet, I'm guessing >it will pitch it. You are telling the remote Web server that the >connection to 10.1.4.50 needs to have its traffic fragmented. I'm >guessing the Web server sees the connection as originating from a public >address. (the NAT address you are using). Since the payload IP does not >match, it will disregard the packet as an unsolicited ICMP error. > > > >>16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ >>PTR? 17.164.70.63.in-addr.arpa. (43) (DF) >> >> > >Weird its doing a reverse lookup, but not a problem. > > > >>16:59:25.962167 Fasttrack2.machinerytrader.com.http > >>--cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF) >> >> > >OK, completely new session well that's a few packets into the data >transfer. > > > >>16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: >>--cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0] >> >> > >There goes our ICMP error with a private address in the payload again. >:) > > > >> >>>> I see this traffic next but the customer side doesn't receive the >>packets <<<<<< >> >>16:59:33.605043 --cust-publicIP--.29695 > >>Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF) >> >> > >OK, something is weird. If the frag info did not get though we should >have seen the Web site attempt some retransmissions. Also, we're looking >at absolute sequence numbers. I think we again didn't get to see some of >the packets in the session. Are you stopping and restarting your packet >capture? > > > >>16:59:38.142184 --cust-publicIP--.29714 > >>Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win >>65280 <mss 1360,nop,nop,sackOK> (DF) >>16:59:38.174050 Fasttrack2.machinerytrader.com.http > >>--cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win >>65535 <mss 1380,nop,nop,sackOK> (DF) >>16:59:38.178358 --cust-publicIP--.29714 > >>Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF) >> >> > >WAHOO! A three packet handshake. This time we got in from the beginning! >:) > > > >>16:59:38.181116 --cust-publicIP--.29714 > >>Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF) >>16:59:38.216940 Fasttrack2.machinerytrader.com.http > >>--cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF) >> >> > >OK, up to here we are good because your MTU did not get exceeded. > > > >>16:59:38.220596 Fasttrack2.machinerytrader.com.http > >>--cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF) >> >> > >And again, we hit an MTU that exceeds you max, so this is were the >problems start. I'm guessing after this we had an outbound ICMP frag >needed, followed by a couple of inbound retransmissions of the above >packet, but it would have been cool to see it in order to confirm. > >HTH, >C > > > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-13 8:02 ` Scott Hall @ 2004-01-13 15:51 ` Chris Brenton 2004-01-13 16:12 ` Scott Hall 0 siblings, 1 reply; 9+ messages in thread From: Chris Brenton @ 2004-01-13 15:51 UTC (permalink / raw) To: Scott Hall; +Cc: netfilter On Tue, 2004-01-13 at 03:02, Scott Hall wrote: > So the one question that this whole issue raises in my mind is, Isn't > there anyway to handle the (DF) packets differently? Absolutely. Config the stacks on both ends of the connection to _not_ set DF. This will cause the router at the MTU border to frag the packets and will not require an ICMP error packet. > I ask > becuase we have two cisco routers and 6 Adtran routers that handle this > same scenario quietly. I'm guessing if you check the decodes from those packets you will see the public rather than the private IP embedded in the payload. I think this is what is killing you. This is an old Netfilter bug that I *thought* was fixed ages ago. HTH, C ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-13 15:51 ` Chris Brenton @ 2004-01-13 16:12 ` Scott Hall 2004-01-13 16:38 ` Chris Brenton 0 siblings, 1 reply; 9+ messages in thread From: Scott Hall @ 2004-01-13 16:12 UTC (permalink / raw) To: Chris Brenton; +Cc: netfilter Thank for the response Chris, The problem that I see with that solution is that most of the sites, which are many by this point, that I have had problems with aren't under my control. Including aol.com, I can just see me trying to convince AOL to reconfigure their servers to not set the DF :). Are there anyother work arounds that you can propose? thnx, --scott Chris Brenton wrote: >On Tue, 2004-01-13 at 03:02, Scott Hall wrote: > > >>So the one question that this whole issue raises in my mind is, Isn't >>there anyway to handle the (DF) packets differently? >> >> > >Absolutely. Config the stacks on both ends of the connection to _not_ >set DF. This will cause the router at the MTU border to frag the packets >and will not require an ICMP error packet. > > > >>I ask >>becuase we have two cisco routers and 6 Adtran routers that handle this >>same scenario quietly. >> >> > >I'm guessing if you check the decodes from those packets you will see >the public rather than the private IP embedded in the payload. I think >this is what is killing you. This is an old Netfilter bug that I >*thought* was fixed ages ago. > >HTH, >C > > > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-13 16:12 ` Scott Hall @ 2004-01-13 16:38 ` Chris Brenton 2004-01-13 17:52 ` Scott Hall 2004-01-14 18:11 ` Mark Weaver 0 siblings, 2 replies; 9+ messages in thread From: Chris Brenton @ 2004-01-13 16:38 UTC (permalink / raw) To: Scott Hall; +Cc: netfilter On Tue, 2004-01-13 at 11:12, Scott Hall wrote: > > Thank for the response Chris, Glad to! > Are there anyother work arounds that you can propose? Do a: iptables -V I'm guessing you are running an older version that is not patched for this problem (1.2.6a or prior). Here is the original advisory: http://www.linuxsecurity.com/advisories/other_advisory-2063.html HTH, C ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-13 16:38 ` Chris Brenton @ 2004-01-13 17:52 ` Scott Hall 2004-01-14 18:11 ` Mark Weaver 1 sibling, 0 replies; 9+ messages in thread From: Scott Hall @ 2004-01-13 17:52 UTC (permalink / raw) To: Chris Brenton; +Cc: netfilter Here is the iptables -V from both routers. the 1.2.7a is the my side router and the router that the original tcpdump traffic I posted came from. [root@gandalf root]# iptables -V iptables v1.2.7a (My side) [root@indinj root]# iptables -V iptables v1.2.8 (customer side) --Scott Chris Brenton wrote: >On Tue, 2004-01-13 at 11:12, Scott Hall wrote: > > >>Thank for the response Chris, >> >> > >Glad to! > > > >>Are there anyother work arounds that you can propose? >> >> > >Do a: >iptables -V > >I'm guessing you are running an older version that is not patched for >this problem (1.2.6a or prior). Here is the original advisory: >http://www.linuxsecurity.com/advisories/other_advisory-2063.html > >HTH, >C > > > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] 2004-01-13 16:38 ` Chris Brenton 2004-01-13 17:52 ` Scott Hall @ 2004-01-14 18:11 ` Mark Weaver 1 sibling, 0 replies; 9+ messages in thread From: Mark Weaver @ 2004-01-14 18:11 UTC (permalink / raw) To: netfilter > Do a: > iptables -V > > I'm guessing you are running an older version that is not patched for > this problem (1.2.6a or prior). Here is the original advisory: > http://www.linuxsecurity.com/advisories/other_advisory-2063.html > That's not enough: you need a patched (or later) kernel as well as the bug actually existed in the netfilter module. I can't remember OTOMH which kernel release this went into, although it was much later than the mentioned version because the kernel team rejected the original fix (for some good reasons). I know 2.4.23+ don't have this problem. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2004-01-14 18:11 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2004-01-06 7:07 icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0] Scott Hall 2004-01-06 11:23 ` Chris Brenton 2004-01-06 15:48 ` Scott Hall 2004-01-13 8:02 ` Scott Hall 2004-01-13 15:51 ` Chris Brenton 2004-01-13 16:12 ` Scott Hall 2004-01-13 16:38 ` Chris Brenton 2004-01-13 17:52 ` Scott Hall 2004-01-14 18:11 ` Mark Weaver
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.