Outbound SNAT on non-local connections

Outbound SNAT on non-local connections

[Ryan]

Hello,
I have been trying to set up a linux box as a L3 DSR backend and in
order for this to work I need to rewrite the source IP on a outbound
reply to a inbound connection.

So what is happening is I have three machines, client, load balancer,
and server.

The client sends a TCP (in my testing i'm using ICMP though) to the load
balancer. The load balancer then does DNAT to rewrite the destination
address to the server.

So now the server has a connection coming in from the load balancer
however it has the source IP of the client.

The server will respond directly to the client (which is what I want)
however because the client see's the connection coming from a different
IP it will ignore it.

I think the simplest solution here is to have SNAT on the server that
rewrites the source address to the load balancer IP however this does
not seem to work.

If i put log rules for every chain in both the filter and nat tables
(and have no other rules) and send a single ICMP request from the client
to the load balancer I see this on the server:

Mar 14 21:15:35 vagrant-ubuntu-precise-64 kernel: [16634.197968] NAT
PREROUTING IN=eth1 OUT= MAC=08:00:27:d1:0d:09:08:00:27:94:ad:1b:08:00
SRC=192.168.54.20 DST=192.168.54.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=17427 DF PROTO=ICMP TYPE=8 CODE=0 ID=3724 SEQ=1 
Mar 14 21:15:35 vagrant-ubuntu-precise-64 kernel: [16634.199879] INPUT
IN=eth1 OUT= MAC=08:00:27:d1:0d:09:08:00:27:94:ad:1b:08:00
SRC=192.168.54.20 DST=192.168.54.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=17427 DF PROTO=ICMP TYPE=8 CODE=0 ID=3724 SEQ=1 
Mar 14 21:15:35 vagrant-ubuntu-precise-64 kernel: [16634.201220] NAT
INPUT IN=eth1 OUT= MAC=08:00:27:d1:0d:09:08:00:27:94:ad:1b:08:00
SRC=192.168.54.20 DST=192.168.54.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=17427 DF PROTO=ICMP TYPE=8 CODE=0 ID=3724 SEQ=1 
Mar 14 21:15:35 vagrant-ubuntu-precise-64 kernel: [16634.201891] OUTPUT
IN= OUT=eth1 SRC=192.168.54.10 DST=192.168.54.20 LEN=84 TOS=0x00
PREC=0x00 TTL=64 ID=29751 PROTO=ICMP TYPE=0 CODE=0 ID=3724 SEQ=1 

It seems to hit the PREROUTING chain and never touches POSTROUTING.

If I do a connmark -L after the ping I will see the ICMP connection show
up there:

icmp     1 28 src=192.168.54.20 dst=192.168.54.10 type=8 code=0 id=3729
src=192.168.54.10 dst=192.168.54.20 type=0 code=0 id=3729 mark=0 use=1

If I understand correctly the nat table won't be consulted if a
connection exists in conntrack already. However because I can't have
SNAT in PREROUTING I can't really do what I want.

Am I doing something wrong or is am I understanding this behavor
correctly? If so is there a way to work around this issue?


Thanks!
Ryan G.

Re: Outbound SNAT on non-local connections

[Ryan]

So just to follow up on my email I did find a solution to my issue by
using the RAWSNAT from the xtables-addons package. This target works in
either raw or rawpost however I also needed to use connection tracking
which doesn't work in raw.  Connection tracking does seem to work in
rawpost but I do not know why as there is no real documentation on
rawpost and it's a xtables specific table.


Thanks!
Ryan G.

Re: Outbound SNAT on non-local connections

[Pascal Hambourg]

Hello,

Ryan a écrit :
> 
> The client sends a TCP (in my testing i'm using ICMP though) to the load
> balancer. The load balancer then does DNAT to rewrite the destination
> address to the server.
> 
> So now the server has a connection coming in from the load balancer
> however it has the source IP of the client.
> 
> The server will respond directly to the client (which is what I want)

Why ?

> however because the client see's the connection coming from a different
> IP it will ignore it.

As expected.

> I think the simplest solution here is to have SNAT on the server that
> rewrites the source address to the load balancer IP however this does
> not seem to work.

Indeed.

> If I understand correctly the nat table won't be consulted if a
> connection exists in conntrack already.

Correct.

> However because I can't have
> SNAT in PREROUTING I can't really do what I want.

I do not see how SNAT in PREROUTING would help in any way. Recent
kernels can do SNAT in INPUT, however it wouldn't help either.

> Am I doing something wrong

Yes, this :

"The server will respond directly to the client (which is what I want)"

Stateful NAT (DNAT, SNAT) needs connection tracking and requires
symmetric routing. If you break symmetric routing, you break stateful NAT.

> If so is there a way to work around this issue?

As you found out, you can use stateless NAT on the server (RAWSNAT) and
also on the load balancer (RAWDNAT, no need to waste resources for
stateful NAT). Or you can just make the routing symmetric and route the
reply traffic through the load balancer.

Re: Outbound SNAT on non-local connections

[Ryan]

> > 
> > The server will respond directly to the client (which is what I want)
> 
> Why ?
Because the connection to the server from the LB has the source address
of the client. 

 
> I do not see how SNAT in PREROUTING would help in any way. Recent
> kernels can do SNAT in INPUT, however it wouldn't help either.

You're right that doesn't make sense, forget what I was thinking
there...

I think we experimented with putting the LB's IP on lo and DNAT'ing to
it on inbound but
didn't seem to work either.

> 
> > Am I doing something wrong
> 
> Yes, this :
> 
> "The server will respond directly to the client (which is what I want)"
> 
> Stateful NAT (DNAT, SNAT) needs connection tracking and requires
> symmetric routing. If you break symmetric routing, you break stateful
> NAT.
> 
> > If so is there a way to work around this issue?
> 
> As you found out, you can use stateless NAT on the server (RAWSNAT) and
> also on the load balancer (RAWDNAT, no need to waste resources for
> stateful NAT). Or you can just make the routing symmetric and route the
> reply traffic through the load balancer.

Yeah I think that makes sense, just not obvious without understanding
how
iptables works. This is working for us now using those targets in
xtables
however unfortunately it looks like RAWDNAT and RAWSNAT has been removed
from the latest version of xtables. The other options are iproute
(deprecated) and 
tc, which I didn't look into much for this but has always been a pain to
use in the past.