NAT and MTU issues

NAT and MTU issues

[Nigel Metheringham]

I have a somewhat complex setup where I am using a box running both
source and destination NAT as a demarcation point between 2 disparately
addressed networks [specifically this allows a customer to access a
service on our network, even through other customers that we connect to
may have networks with otherlapping address spaces - not always
"private" space]

So from the customer's point of view, they connect to 192.168.50.119
port 1500.
Our kit wants to see a connection from 10.0.1.0/24 to 10.0.2.2 port
2502.

In theory this works fine... but theres a wrinkle.  Our box is remote
from our data centre and connects to it using an IPSec link using
FreeSWAN.  The ipsec0 interface has an MTU on it of 1450 - this prevents
us fragmenting 1500 byte packets when they hit the ipsec engine (which
expands the packets up).  A 1450 MTU has historically worked well for
us.

What appears to be happening is that everything works while packets are
short, however when long packets come in they bounce off the lower MTU
interface, and the returned ICMP packet is not getting back to the
originator in a sane form.   So the connection freezes.

Having looked closer at this I find there is an ICMP dest unreach packet
emitted from my box back to the originator.  However inside the packet
the SNAT has been undone, but the DNAT is still in place.

Any ideas how I can fix this...?
This is all on a 2.4.21 kernel.

NAT rules I have in place are:-
  [root@t003 admin]# /sbin/iptables -t nat -n -L 
  Chain PREROUTING (policy ACCEPT)
  target     prot opt source         destination
  DNAT       tcp  --  0.0.0.0/0      192.168.50.119  tcp dpt:1500 to:10.0.2.2:2502
 
  Chain POSTROUTING (policy ACCEPT)
  target     prot opt source         destination
  SNAT       tcp  --  0.0.0.0/0      10.0.2.2        tcp dpt:2502 to:10.0.1.239-10.0.1.254

The ICMP packet I get back looks like this:-
 Frame 229 (590 bytes on wire, 590 bytes captured)
    Arrival Time: Sep 19, 2003 15:21:31.728906000
    Time delta from previous packet: 0.000019000 seconds
    Time relative to first packet: 70.396568000 seconds
    Frame Number: 229
    Packet Length: 590 bytes
    Capture Length: 590 bytes
 Ethernet II, Src: 00:02:a5:da:5f:7b, Dst: 00:10:db:ff:20:70
    Destination: 00:10:db:ff:20:70 (00:10:db:ff:20:70)
    Source: 00:02:a5:da:5f:7b (00:02:a5:da:5f:7b)
    Type: IP (0x0800)
 Internet Protocol, Src Addr: 192.168.50.119 (192.168.50.119), Dst Addr: 172.16.28.5 (172.16.28.5)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 576
    Identification: 0x8d00 (36096)
    Flags: 0x00
        .0.. = Don't fragment: Not set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: ICMP (0x01)
    Header checksum: 0x2fc8 (correct)
    Source: 192.168.50.119 (192.168.50.119)
    Destination: 172.16.28.5 (172.16.28.5)
 Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 4 (Fragmentation needed)
    Checksum: 0x8479 (correct)
    MTU of next hop: 1450
    Internet Protocol, Src Addr: 172.16.28.5 (172.16.28.5), Dst Addr: 10.0.2.2 (10.0.2.2)
        Version: 4
        Header length: 20 bytes
        Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
            0000 00.. = Differentiated Services Codepoint: Default (0x00)
            .... ..0. = ECN-Capable Transport (ECT): 0
            .... ...0 = ECN-CE: 0
        Total Length: 1500
        Identification: 0x7749 (30537)
        Flags: 0x04
            .1.. = Don't fragment: Set
            ..0. = More fragments: Not set
        Fragment offset: 0
        Time to live: 62
        Protocol: TCP (0x06)
        Header checksum: 0xcda0 (correct)
        Source: 172.16.28.5 (172.16.28.5)
        Destination: 10.0.2.2 (10.0.2.2)
    Transmission Control Protocol, Src Port: 50794 (50794), Dst Port: 2502 (2502), Seq: 3025715234, Ack: 3268150508
        Source port: 50794 (50794)
        Destination port: 2502 (2502)
        Sequence number: 3025715234
        Acknowledgement number: 3268150508
        Header length: 32 bytes
        Flags: 0x0010 (ACK)
            0... .... = Congestion Window Reduced (CWR): Not set
            .0.. .... = ECN-Echo: Not set
            ..0. .... = Urgent: Not set
            ...1 .... = Acknowledgment: Set
            .... 0... = Push: Not set
            .... .0.. = Reset: Not set
            .... ..0. = Syn: Not set
            .... ...0 = Fin: Not set
        Window size: 6432
        Checksum: 0xb25c (incorrect, should be 0x4b3b)
        Options: (12 bytes)
            NOP
            NOP
            Time stamp: tsval 19647219, tsecr 1709539721
    Data (496 bytes)


Cheers
	Nigel.

-- 
[ Nigel Metheringham           Nigel.Metheringham@InTechnology.co.uk ]
[ - Comments in this message are my own and not ITO opinion/policy - ]

________________________________________________________________________
This message has been checked for all known viruses by the 
CitC Virus Scanning Service powered by SkyLabs. For further information visit
http://www.citc.it

___

n00b question..... How to get details on active connections

[Paul Gibson]

Hello,

	I know this is a n00b question but how can I get details of active
connections, eg what inside address/pc is connected to what outside address
???


TIA

Re: n00b question..... How to get details on active connections

[Nox]

On Sat, 2003-09-20 at 05:15, Paul Gibson wrote:
> Hello,
> 
> 	I know this is a n00b question but how can I get details of active
> connections, eg what inside address/pc is connected to what outside address
> ???

Do you mean connections through the Firewall?
Something like /proc/net/ip_conntrack will tell you the active
connections. (might be /proc/net/ipv4/ip_conntrack on your box)

If you mean just the box your on, something like a netstat will do that

Hope that helps

Nox
GenMicro Systems

Re: NAT and MTU issues

[Martin Josefsson]

On Fri, 2003-09-19 at 17:28, Nigel Metheringham wrote:

> Having looked closer at this I find there is an ICMP dest unreach packet
> emitted from my box back to the originator.  However inside the packet
> the SNAT has been undone, but the DNAT is still in place.
> 
> Any ideas how I can fix this...?
> This is all on a 2.4.21 kernel.

Gah, I hoped we had fixed all these problems. Getting all the
corner-cases right isn't as easy as one thinks when we perform multiple
translations.
I've looked at the code responsible for the rewriting of the inner
ipheader and it looks ok to me.

Is the NAT-rules on the machine that has the tunnel? If they are that
might explain a thing or two since the code looks correct for the case
where the packets pass through and another machine down the pipe sends
the icmp message back.

-- 
/Martin

Re: NAT and MTU issues

[Nigel Metheringham]

On Sat, 2003-09-20 at 20:04, Martin Josefsson wrote:
> Gah, I hoped we had fixed all these problems. Getting all the
> corner-cases right isn't as easy as one thinks when we perform multiple
> translations.

:-)

> Is the NAT-rules on the machine that has the tunnel? If they are that
> might explain a thing or two since the code looks correct for the case
> where the packets pass through and another machine down the pipe sends
> the icmp message back.

Yes - all of this is on one machine.
One interface has the effective  listening port on it, another interface
of the same box has the ipsec0 interface layered on top.

	Nigel.
-- 
[ Nigel Metheringham           Nigel.Metheringham@InTechnology.co.uk ]
[ - Comments in this message are my own and not ITO opinion/policy - ]

Re: NAT and MTU issues

[Martin Josefsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 752 bytes --]

On Mon, 22 Sep 2003, Nigel Metheringham wrote:

> > Is the NAT-rules on the machine that has the tunnel? If they are that
> > might explain a thing or two since the code looks correct for the case
> > where the packets pass through and another machine down the pipe sends
> > the icmp message back.
>
> Yes - all of this is on one machine.
> One interface has the effective  listening port on it, another interface
> of the same box has the ipsec0 interface layered on top.

Could you please apply the attached patch and reproduce it again?
It's just a small patch that enables a little debugging for this.

The debugmessages comes out through the normal kernellog, run 'dmesg' and
see what it says. It's the lines beginning with "icmp_reply:"

/Martin

[-- Attachment #2: Type: TEXT/PLAIN, Size: 950 bytes --]

--- linux-2.4.21/net/ipv4/netfilter/ip_nat_core.c	2003-06-14 16:46:09.000000000 +0200
+++ linux-2.4.21.test/net/ipv4/netfilter/ip_nat_core.c	2003-09-20 20:59:10.000000000 +0200
@@ -913,7 +913,7 @@
 		   where we would normally apply a dst manip, we apply
 		   a src, and vice versa. */
 		if (info->manips[i].hooknum == hooknum) {
-			DEBUGP("icmp_reply: inner %s -> %u.%u.%u.%u %u\n",
+			printk("icmp_reply: inner %s -> %u.%u.%u.%u %u\n",
 			       info->manips[i].maniptype == IP_NAT_MANIP_SRC
 			       ? "DST" : "SRC",
 			       NIPQUAD(info->manips[i].manip.ip),
@@ -928,7 +928,7 @@
 
 			/* Use mapping to map outer packet: 0 give no
                            per-proto mapping */
-			DEBUGP("icmp_reply: outer %s -> %u.%u.%u.%u\n",
+			printk("icmp_reply: outer %s -> %u.%u.%u.%u\n",
 			       info->manips[i].maniptype == IP_NAT_MANIP_SRC
 			       ? "SRC" : "DST",
 			       NIPQUAD(info->manips[i].manip.ip));

Re: NAT and MTU issues

[Nigel Metheringham]

On Mon, 2003-09-22 at 13:00, Martin Josefsson wrote:
> Could you please apply the attached patch and reproduce it again?
> It's just a small patch that enables a little debugging for this.

Very odd - I am seeing ICMPs generated:-
# /usr/sbin/tcpdump -n -i eth0 icmp
tcpdump: listening on eth0
15:26:12.146557 192.168.50.119 > 172.16.28.33: icmp: 10.0.2.2
unreachable - need to frag (mtu 1450) [tos 0xc0]

but no extra chatter in dmesg despite ensuring dmesg -n is turned up. 
Checking the module object file shows the extra log messages in there,
so its not me doing something completely silly.

Putting a 
  iptables -t mangle -A FORWARD -p tcp --syn -j TCPMSS \
	--clamp-mss-to-pmtu

in appears to fix things for me.

	Nigel.
-- 
[ Nigel Metheringham           Nigel.Metheringham@InTechnology.co.uk ]
[ - Comments in this message are my own and not ITO opinion/policy - ]

Re: NAT and MTU issues

[Martin Josefsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1162 bytes --]

On Mon, 22 Sep 2003, Nigel Metheringham wrote:

> On Mon, 2003-09-22 at 13:00, Martin Josefsson wrote:
> > Could you please apply the attached patch and reproduce it again?
> > It's just a small patch that enables a little debugging for this.
>
> Very odd - I am seeing ICMPs generated:-
> # /usr/sbin/tcpdump -n -i eth0 icmp
> tcpdump: listening on eth0
> 15:26:12.146557 192.168.50.119 > 172.16.28.33: icmp: 10.0.2.2
> unreachable - need to frag (mtu 1450) [tos 0xc0]
>
> but no extra chatter in dmesg despite ensuring dmesg -n is turned up.
> Checking the module object file shows the extra log messages in there,
> so its not me doing something completely silly.

Ok, no wonder that the ipaddress isn't rewritten to the correct one, the
rewriting is never called. Now lets find out if the loop is run at all.
I forgot to enable that debug-statement in the previous patch, sorry.
A new patch is applied, reverse the old one and apply this and retest.

> Putting a
>   iptables -t mangle -A FORWARD -p tcp --syn -j TCPMSS \
> 	--clamp-mss-to-pmtu
>
> in appears to fix things for me.

For tcp... it avoids the problem, but we still need to fix this :)

/Martin

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1272 bytes --]

--- linux-2.4.21/net/ipv4/netfilter/ip_nat_core.c	2003-06-14 16:46:09.000000000 +0200
+++ linux-2.4.21.test/net/ipv4/netfilter/ip_nat_core.c	2003-09-22 16:59:01.000000000 +0200
@@ -901,7 +901,7 @@
 
 	READ_LOCK(&ip_nat_lock);
 	for (i = 0; i < info->num_manips; i++) {
-		DEBUGP("icmp_reply: manip %u dir %s hook %u\n",
+		printk("icmp_reply: manip %u dir %s hook %u\n",
 		       i, info->manips[i].direction == IP_CT_DIR_ORIGINAL ?
 		       "ORIG" : "REPLY", info->manips[i].hooknum);
 
@@ -913,7 +913,7 @@
 		   where we would normally apply a dst manip, we apply
 		   a src, and vice versa. */
 		if (info->manips[i].hooknum == hooknum) {
-			DEBUGP("icmp_reply: inner %s -> %u.%u.%u.%u %u\n",
+			printk("icmp_reply: inner %s -> %u.%u.%u.%u %u\n",
 			       info->manips[i].maniptype == IP_NAT_MANIP_SRC
 			       ? "DST" : "SRC",
 			       NIPQUAD(info->manips[i].manip.ip),
@@ -928,7 +928,7 @@
 
 			/* Use mapping to map outer packet: 0 give no
                            per-proto mapping */
-			DEBUGP("icmp_reply: outer %s -> %u.%u.%u.%u\n",
+			printk("icmp_reply: outer %s -> %u.%u.%u.%u\n",
 			       info->manips[i].maniptype == IP_NAT_MANIP_SRC
 			       ? "SRC" : "DST",
 			       NIPQUAD(info->manips[i].manip.ip));

Re: NAT and MTU issues

[Nigel Metheringham]

On Mon, 2003-09-22 at 15:52, Nigel Metheringham wrote:
> On Mon, 2003-09-22 at 13:00, Martin Josefsson wrote:
> > Could you please apply the attached patch and reproduce it again?
> > It's just a small patch that enables a little debugging for this.
> 
> Very odd - I am seeing ICMPs generated:-
> # /usr/sbin/tcpdump -n -i eth0 icmp
> tcpdump: listening on eth0
> 15:26:12.146557 192.168.50.119 > 172.16.28.33: icmp: 10.0.2.2
> unreachable - need to frag (mtu 1450) [tos 0xc0]
> 
> but no extra chatter in dmesg despite ensuring dmesg -n is turned up. 
> Checking the module object file shows the extra log messages in there,
> so its not me doing something completely silly.
> 
> Putting a 
>   iptables -t mangle -A FORWARD -p tcp --syn -j TCPMSS \
> 	--clamp-mss-to-pmtu
> 
> in appears to fix things for me.

Took a closer look.
If I put that mangle rule in then:-
      * I see no ICMP packets on the wire between the originating box
        and the linux g/w (tested in 2 places to make sure I don't have
        any packet sniffing/netfilter interactions).  Previously I saw
        ICMP need frag packets as quoted above
      * those icmp_reply log messages appear to fire on each and every
        packet

   icmp_reply: outer SRC -> 192.168.50.119
   icmp_reply: inner DST -> 192.168.50.119 1500

	Nigel.

-- 
[ Nigel Metheringham           Nigel.Metheringham@InTechnology.co.uk ]
[ - Comments in this message are my own and not ITO opinion/policy - ]

Re: NAT and MTU issues

[Martin Josefsson]

On Mon, 22 Sep 2003, Nigel Metheringham wrote:

> Took a closer look.
> If I put that mangle rule in then:-
>       * I see no ICMP packets on the wire between the originating box
>         and the linux g/w (tested in 2 places to make sure I don't have
>         any packet sniffing/netfilter interactions).  Previously I saw
>         ICMP need frag packets as quoted above
>       * those icmp_reply log messages appear to fire on each and every
>         packet
>
>    icmp_reply: outer SRC -> 192.168.50.119
>    icmp_reply: inner DST -> 192.168.50.119 1500

Uhm, let me see if I got this right...

If you add that mangle rule you don't see any icmp packets on the wire but
you see the icmp_reply messages?

Or do you see those messages when you don't have the rule and thus see the
icmp packets?

/Martin

Re: NAT and MTU issues

[Nigel Metheringham]

On Mon, 2003-09-22 at 16:41, Martin Josefsson wrote:
> On Mon, 22 Sep 2003, Nigel Metheringham wrote:
> 
> > Took a closer look.
> > If I put that mangle rule in then:-
> >       * I see no ICMP packets on the wire between the originating box
> >         and the linux g/w (tested in 2 places to make sure I don't have
> >         any packet sniffing/netfilter interactions).  Previously I saw
> >         ICMP need frag packets as quoted above
> >       * those icmp_reply log messages appear to fire on each and every
> >         packet
> >
> >    icmp_reply: outer SRC -> 192.168.50.119
> >    icmp_reply: inner DST -> 192.168.50.119 1500
> 
> Uhm, let me see if I got this right...
> 
> If you add that mangle rule you don't see any icmp packets on the wire but
> you see the icmp_reply messages?

yup.

With no mangle rule I get broken ICMP frag-needed messages on the wire
and your debug messages do not trigger.

With the mangle rule I see no ICMP on the wire but the debug messages
trigger frequently (ie probably once per packet).

I'm confused too!

	Nigel.

-- 
[ Nigel Metheringham           Nigel.Metheringham@InTechnology.co.uk ]
[ - Comments in this message are my own and not ITO opinion/policy - ]

Re: NAT and MTU issues

[Martin Josefsson]

On Mon, 2003-09-22 at 17:46, Nigel Metheringham wrote:

> > If you add that mangle rule you don't see any icmp packets on the wire but
> > you see the icmp_reply messages?
> 
> yup.
> 
> With no mangle rule I get broken ICMP frag-needed messages on the wire
> and your debug messages do not trigger.
> 
> With the mangle rule I see no ICMP on the wire but the debug messages
> trigger frequently (ie probably once per packet).
> 
> I'm confused too!

I think I'll set up something that reminds of your situation here to be
able to more clearly see what's going on, this sounds very weird...

-- 
/Martin

Re: NAT and MTU issues

[Nigel Metheringham]

I've just redone the tests using the ip_nat_core.c with additional
debug, with similar results to last time.  So I then tried a 2.4.22
kernel just to make sure there were no other changes I've missed -
kernel is now a vanilla 2.4.22 plus super-freeswan patches for ipsec and
a few others that don't touch networking at all.

Using just the normal NAT rules, the extra debugging statements are not
triggered at all, and ICMPs (unreach/need-frag) are sent to the
connection originator with the contained packet having the DNAT still in
place (ie the packet source addresses are correct, the destination
addresses are the DNATed versions).

With the MSS clamping mangle rule in place, the debugging lines log for
many many packets (probably all), but no ICMP packets are seen
externally.
        icmp_reply_translation: translating error ce75b800 hook 3 dir REPLY
        icmp_reply: manip 0 dir ORIG hook 0
        icmp_reply: manip 1 dir REPLY hook 4
        icmp_reply: manip 2 dir ORIG hook 4
        icmp_reply: manip 3 dir REPLY hook 0
        icmp_reply_translation: translating error ce75b800 hook 4 dir REPLY
        icmp_reply: manip 0 dir ORIG hook 0
        icmp_reply: manip 1 dir REPLY hook 4
        icmp_reply: inner DST -> 192.168.50.119 1500
        icmp_reply: outer SRC -> 192.168.50.119
        icmp_reply: manip 2 dir ORIG hook 4
        icmp_reply: manip 3 dir REPLY hook 0
        icmp_reply_translation: translating error ce4af1a0 hook 3 dir REPLY
Thats a complete cycle of the logged data (you'll see I set a couple
more of the lines to log as well as the ones in your patch).

	Nigel.
-- 
[ Nigel Metheringham           Nigel.Metheringham@InTechnology.co.uk ]
[ - Comments in this message are my own and not ITO opinion/policy - ]