SNAT with ipsec => return packets not de-natted

SNAT with ipsec => return packets not de-natted

[Jari Laurila]

Hi,

I'm trying to do SNAT with ipsec tunnel mode connection, but can't get
it working.
I'm trying to accomplish the following:

1. Local server li sends packet with its internal ip to remote server re.
2. Local vpn gateway lg receives packet and SNATs it to external ip le.
3. lg sends packet through vpn tunnel between lg and rg
5. re responds through vpn tunnel between rg and lg
6. lg de-nats packet (le=>li) and sends packet to li

My setup currently fails at point 6. (Packet doesn't get de-natted)
Am I missing something?  I understood that Patrick McHardy added necessary hooks
to kernel few years ago, so this should work.

I'm using kernel 2.6.30.5 and iptables 1.4.5.

I attached some packet dumps and iptables output below.

le.le.le.le == local server external ip
li.li.li.li == local server internal ip
lg.lg.lg.lg == local vpn gw ip
re.re.re.re == remote server ip
rg.rg.rg.rg == remote vpn gw ip

# iptables -t nat -vnL | grep le.le.le.le
    0     0 DNAT       all  --  *      *       re.re.re.re
le.le.le.le       to:li.li.li.li
    6   288 SNAT       all  --  *      *       li.li.li.li
re.re.re.re       to:le.le.le.le

# Connection attemp seen from internal interface	
# tcpdump -ni int0.1 'host re.re.re.re'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on int0.1, link-type EN10MB (Ethernet), capture size 96 bytes
19:53:06.925200 IP li.li.li.li.2921 > re.re.re.re.21: S
3229210479:3229210479(0) win 64240 <mss 1460,nop,nop,sackOK>
19:53:09.838539 IP li.li.li.li.2921 > re.re.re.re.21: S
3229210479:3229210479(0) win 64240 <mss 1460,nop,nop,sackOK>
19:53:15.873102 IP li.li.li.li.2921 > re.re.re.re.21: S
3229210479:3229210479(0) win 64240 <mss 1460,nop,nop,sackOK>

# Connection attempt seen from external interface
# tcpdump -ni ext1 'host rg.rg.rg.rg or host re.re.re.re'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ext1, link-type EN10MB (Ethernet), capture size 96 bytes

## Encrypted syn packet from local server to remote server
19:53:06.925295 IP lg.lg.lg.lg > rg.rg.rg.rg:
ESP(spi=0xb4e85134,seq=0x3), length 84
## Encrypted synack packet from remote server to local server
19:53:06.943724 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0x8), length 76
## Decrypted synack packet from remote server goes to external interface
## because de-natting does not work (le.le.le.le should be translated
back to li.li.li.li)
19:53:06.943724 IP re.re.re.re.21 > le.le.le.le.2921: S
400270854:400270854(0) ack 3229210480 win 65535 <mss 1400>
19:53:09.507623 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0x9), length 76
19:53:09.507623 IP re.re.re.re.21 > le.le.le.le.2921: S
400270854:400270854(0) ack 3229210480 win 65535 <mss 1400>

## Local server tries again
19:53:09.838590 IP lg.lg.lg.lg > rg.rg.rg.rg:
ESP(spi=0xb4e85134,seq=0x4), length 84
19:53:09.844910 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0xa), length 76
19:53:09.844910 IP re.re.re.re.21 > le.le.le.le.2921: S
400270854:400270854(0) ack 3229210480 win 65535 <mss 1400>
19:53:15.526342 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0xb), length 76
19:53:15.526342 IP re.re.re.re.21 > le.le.le.le.2921: S
400270854:400270854(0) ack 3229210480 win 65535 <mss 1400>

## ...And again
19:53:15.873146 IP lg.lg.lg.lg > rg.rg.rg.rg:
ESP(spi=0xb4e85134,seq=0x5), length 84
19:53:15.880125 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0xc), length 76
19:53:15.880125 IP re.re.re.re.21 > le.le.le.le.2921: S
400270854:400270854(0) ack 3229210480 win 65535 <mss 1400>
19:53:27.746678 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0xd), length 76
19:53:27.746678 IP re.re.re.re.21 > le.le.le.le.2921: S
400270854:400270854(0) ack 3229210480 win 65535 <mss 1400>
19:53:51.926765 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0xe), length 76
19:53:51.926765 IP re.re.re.re.21 > le.le.le.le.2921: S
400270854:400270854(0) ack 3229210480 win 65535 <mss 1460>
## Remote server gives up
19:54:31.256860 IP rg.rg.rg.rg > lg.lg.lg.lg:
ESP(spi=0x06525201,seq=0xf), length 76
19:54:31.256860 IP re.re.re.re.21 > le.le.le.le.2921: R 1:1(0) ack 1 win 65535


# iptables -t nat -vnL | grep le.le.le.le
    0     0 DNAT       all  --  *      *       re.re.re.re
le.le.le.le       to:li.li.li.li
    7   336 SNAT       all  --  *      *       li.li.li.li
re.re.re.re       to:le.le.le.le
	
SNAT rule counter has increased by one so the connection to re has
been source natted correctly
I even tried to add explicit rule to nat connections back to li, but
the rule doesn't seem to match at all

Re: SNAT with ipsec => return packets not de-natted

[Jari Laurila]

Don't anyone have any clues for the problem I sent to the list on sunday?

I find it really strange that decrypted packets coming from ipsec
tunnel with destination address xx.xx.xx.42 are sent through interface
ext1 even though ip -s route get xx.xx.xx.42 says that packet should
go through interface ext0b. Ipsec tunnel itself is going through
inteface ext1 but shouldn't packets get routed after they come from
tunnel? I even tried to look at kernel code to figure out why this
happens but I don't know enough about kernel and my c skills are a bit
lacking, so I couldn't find the cause.

Re: SNAT with ipsec => return packets not de-natted

[Jari Laurila]

On Tue, Nov 3, 2009 at 8:54 AM, Jari Laurila <jari.laurila@gmail.com> wrote:
> Don't anyone have any clues for the problem I sent to the list on sunday?
>
> I find it really strange that decrypted packets coming from ipsec
> tunnel with destination address xx.xx.xx.42 are sent through interface
> ext1 even though ip -s route get xx.xx.xx.42 says that packet should
> go through interface ext0b. Ipsec tunnel itself is going through
> inteface ext1 but shouldn't packets get routed after they come from
> tunnel? I even tried to look at kernel code to figure out why this
> happens but I don't know enough about kernel and my c skills are a bit
> lacking, so I couldn't find the cause.
>

Update Netfilter sees packet at mangle table in PREROUTING chain (I
added LOG rule), but nat table does not see the packet.

I also have fwd policy defined for the connection in question:

src srcip.srcip.srcip.secip/32 dst dstip.dstip.dstip.42/32
        dir fwd priority 0
        tmpl src gwip.gwip.gwip.gwip dst remgw.remgw.remgw.remgw
                proto esp reqid 0 mode tunnel

Re: SNAT with ipsec => return packets not de-natted

[Patrick McHardy]

Jari Laurila wrote:
> On Tue, Nov 3, 2009 at 8:54 AM, Jari Laurila <jari.laurila@gmail.com> wrote:
>> Don't anyone have any clues for the problem I sent to the list on sunday?
>>
>> I find it really strange that decrypted packets coming from ipsec
>> tunnel with destination address xx.xx.xx.42 are sent through interface
>> ext1 even though ip -s route get xx.xx.xx.42 says that packet should
>> go through interface ext0b. Ipsec tunnel itself is going through
>> inteface ext1 but shouldn't packets get routed after they come from
>> tunnel? I even tried to look at kernel code to figure out why this
>> happens but I don't know enough about kernel and my c skills are a bit
>> lacking, so I couldn't find the cause.
>>
> 
> Update Netfilter sees packet at mangle table in PREROUTING chain (I
> added LOG rule), but nat table does not see the packet.
> 
> I also have fwd policy defined for the connection in question:
> 
> src srcip.srcip.srcip.secip/32 dst dstip.dstip.dstip.42/32
>         dir fwd priority 0
>         tmpl src gwip.gwip.gwip.gwip dst remgw.remgw.remgw.remgw
>                 proto esp reqid 0 mode tunnel

Try adding a TRACE rule to see how the packet traverses the netfilter
hooks.

Re: SNAT with ipsec => return packets not de-natted

[Jari Laurila]

On Wed, Nov 4, 2009 at 2:27 PM, Patrick McHardy <kaber@trash.net> wrote:
> Jari Laurila wrote:
>> On Tue, Nov 3, 2009 at 8:54 AM, Jari Laurila <jari.laurila@gmail.com> wrote:
>>> Don't anyone have any clues for the problem I sent to the list on sunday?
>>>
>>> I find it really strange that decrypted packets coming from ipsec
>>> tunnel with destination address xx.xx.xx.42 are sent through interface
>>> ext1 even though ip -s route get xx.xx.xx.42 says that packet should
>>> go through interface ext0b. Ipsec tunnel itself is going through
>>> inteface ext1 but shouldn't packets get routed after they come from
>>> tunnel? I even tried to look at kernel code to figure out why this
>>> happens but I don't know enough about kernel and my c skills are a bit
>>> lacking, so I couldn't find the cause.
>>>
>>
>> Update Netfilter sees packet at mangle table in PREROUTING chain (I
>> added LOG rule), but nat table does not see the packet.
>>
>> I also have fwd policy defined for the connection in question:
>>
>> src srcip.srcip.srcip.secip/32 dst dstip.dstip.dstip.42/32
>>         dir fwd priority 0
>>         tmpl src gwip.gwip.gwip.gwip dst remgw.remgw.remgw.remgw
>>                 proto esp reqid 0 mode tunnel
>
> Try adding a TRACE rule to see how the packet traverses the netfilter
> hooks.
>
I did that. Here are the results:

kernel: TRACE: raw:PREROUTING:policy:2 IN=ext1 OUT= MAC=xxx
SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
RES=0x00 ACK RST URGP=0
kernel: TRACE: mangle:PREROUTING:rule:1 IN=ext1 OUT= MAC=xxx
SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
RES=0x00 ACK RST URGP=0
kernel: TRACE: mangle:PREROUTING:rule:12 IN=ext1 OUT= MAC=xxx
SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
RES=0x00 ACK RST URGP=0
kernel: TRACE: mangle:sanitychk_acl:return:38 IN=ext1 OUT= MAC=xxx
SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
RES=0x00 ACK RST URGP=0
kernel: TRACE: mangle:PREROUTING:policy:13 IN=ext1 OUT= MAC=xxx
SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
RES=0x00 ACK RST URGP=0


Mangle chains policy is accept:
# iptables -t mangle -vnL PREROUTING
Chain PREROUTING (policy ACCEPT 46M packets, 21G bytes)
...
Other rules
...

Re: SNAT with ipsec => return packets not de-natted

[Jari Laurila]

On Thu, Nov 5, 2009 at 8:44 AM, Jari Laurila <jari.laurila@gmail.com> wrote:
> On Wed, Nov 4, 2009 at 2:27 PM, Patrick McHardy <kaber@trash.net> wrote:
>> Jari Laurila wrote:
>>> On Tue, Nov 3, 2009 at 8:54 AM, Jari Laurila <jari.laurila@gmail.com> wrote:
>>>> Don't anyone have any clues for the problem I sent to the list on sunday?
>>>>
>>>> I find it really strange that decrypted packets coming from ipsec
>>>> tunnel with destination address xx.xx.xx.42 are sent through interface
>>>> ext1 even though ip -s route get xx.xx.xx.42 says that packet should
>>>> go through interface ext0b. Ipsec tunnel itself is going through
>>>> inteface ext1 but shouldn't packets get routed after they come from
>>>> tunnel? I even tried to look at kernel code to figure out why this
>>>> happens but I don't know enough about kernel and my c skills are a bit
>>>> lacking, so I couldn't find the cause.
>>>>
>>>
>>> Update Netfilter sees packet at mangle table in PREROUTING chain (I
>>> added LOG rule), but nat table does not see the packet.
>>>
>>> I also have fwd policy defined for the connection in question:
>>>
>>> src srcip.srcip.srcip.secip/32 dst dstip.dstip.dstip.42/32
>>>         dir fwd priority 0
>>>         tmpl src gwip.gwip.gwip.gwip dst remgw.remgw.remgw.remgw
>>>                 proto esp reqid 0 mode tunnel
>>
>> Try adding a TRACE rule to see how the packet traverses the netfilter
>> hooks.
>>
> I did that. Here are the results:
>
> kernel: TRACE: raw:PREROUTING:policy:2 IN=ext1 OUT= MAC=xxx
> SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
> PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
> RES=0x00 ACK RST URGP=0
> kernel: TRACE: mangle:PREROUTING:rule:1 IN=ext1 OUT= MAC=xxx
> SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
> PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
> RES=0x00 ACK RST URGP=0
> kernel: TRACE: mangle:PREROUTING:rule:12 IN=ext1 OUT= MAC=xxx
> SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
> PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
> RES=0x00 ACK RST URGP=0
> kernel: TRACE: mangle:sanitychk_acl:return:38 IN=ext1 OUT= MAC=xxx
> SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
> PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
> RES=0x00 ACK RST URGP=0
> kernel: TRACE: mangle:PREROUTING:policy:13 IN=ext1 OUT= MAC=xxx
> SRC=remote_server DST=xxx.42 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=43368
> PROTO=TCP SPT=21 DPT=1261 SEQ=4137897325 ACK=3664330034 WINDOW=65535
> RES=0x00 ACK RST URGP=0
>
>
> Mangle chains policy is accept:
> # iptables -t mangle -vnL PREROUTING
> Chain PREROUTING (policy ACCEPT 46M packets, 21G bytes)
> ...
> Other rules
> ...
>
Gah.. I feel really stupid right now :|

My ipsec-tools.conf had typo I didn't notice until I read the confs
hundredth time:

I had lines
spdadd xx.xx.xx.42/32 aa.aa.aa.aa/32 any -P out ipsec
esp/tunnel/yy.yy.yy.yy-bb.bb.bb.bb/require;
spdadd aa.aa.aa.aa/32 xx.xx.xx.42/32 any -P in ipsec
esp/tunnel/yy.yy.yy.yy-bb.bb.bb.bb/require;

and the correct lines were
spdadd xx.xx.xx.42/32 aa.aa.aa.aa/32 any -P out ipsec
esp/tunnel/yy.yy.yy.yy-bb.bb.bb.bb/require;
spdadd aa.aa.aa.aa/32 xx.xx.xx.42/32 any -P in ipsec
esp/tunnel/bb.bb.bb.bb-yy.yy.yy.yy/require;

Still I think kernel should have better logging in situations like
these. I found absolutely nothing from logs when i investigated why my
setup failed...

Thanks anyway.. I hope I didn't waste too much of your time :)