POSTROUTING SNAT only reply packets

POSTROUTING SNAT only reply packets

[GMail Isaac Gonzalez]

Hi,

I need to modify the reply packets of one web server to allow the 
connections between a webserver and client using a load balancer.

The client connections goes to a load balancer, the load balancer 
forwards the connection to a one web server changing the destination ip, 
the web server anwser the client with it's own ip address without 
passing again for the load balancer. In order to stablish the 
connection, the client needs to receive the web server answer with the 
correct ip address (in this case, the load balancer VIP address), in 
other case it receives ACK that it doesn't know about it and the 
connections is not ESTABLISHED.

I've doing some testing and seems that iptables only do SNAT on NEW 
connections, and I need to change the ip address of replied packets. 
Anybody know some workaround? If anobody do not know some workaround can 
you confirm that it's not posible to do this with iptables?

I've tried the next ip tables rules and only work when I do NEW 
connections from the web server.

-A POSTROUTING -o br0 -s WE_SERVER_ADDR -p tcp -m tcp --sport 80 --dport 
1024:65535 -j SNAT --to-source LOAD_BALANCER_ADDR

Thanks in advance.

Isaac González

Re: POSTROUTING SNAT only reply packets

[Gáspár Lajos]

Hi,

2011-01-17 11:16 keltezéssel, GMail Isaac Gonzalez írta:
> Hi,
>

> I've doing some testing and seems that iptables only do SNAT on NEW 
> connections, and I need to change the ip address of replied packets. 
> Anybody know some workaround? If anobody do not know some workaround 
> can you confirm that it's not posible to do this with iptables?

read again the NAT part in the manual:
man iptables

nat table:
               nat:
                   This table is consulted when a packet that creates a 
new connection is encountered.  It consists of three built-ins: 
PREROUTING (for altering packets as soon as they come in), OUTPUT (for  
altering  locally-gener-
                   ated packets before routing), and POSTROUTING (for 
altering packets as they are about to go out).

DNAT target:

    DNAT
        This  target  is  only valid in the nat table, in the PREROUTING 
and OUTPUT chains, and user-defined chains which are only called from 
those chains.  It specifies that the destination address of the packet 
should be modified
        (and all future packets in this connection will also be 
mangled), and rules should cease being examined.  It takes one type of 
option:

SNAT target:

    SNAT
        This target is only valid in the nat table, in the POSTROUTING 
chain.  It specifies that the source address of the packet should be 
modified (and all future packets in this connection will also be 
mangled), and rules  should
        cease being examined.  It takes one type of option:


> I've tried the next ip tables rules and only work when I do NEW 
> connections from the web server.
>
> -A POSTROUTING -o br0 -s WE_SERVER_ADDR -p tcp -m tcp --sport 80 
> --dport 1024:65535 -j SNAT --to-source LOAD_BALANCER_ADDR
>
> Thanks in advance.
>
> Isaac González
>

You should do all of the NAT-ing ON THE LOAD BALANCER:

iptables -t nat -A PREROUTING -j DNAT -p tcp --dport 80 --to-destination 
WEBSERVER1 (some load balancing options here)
iptables -t nat -A PREROUTING -j DNAT -p tcp --dport 80 --to-destination 
WEBSERVER2 (some load balancing options here)

iptables -t nat -A POSTROUTING -j SNAT -p tcp --dport 80 -d WEBSERVER1 
--to-source BALANCER_IP_ON_WEBSERVER1_NET
iptables -t nat -A POSTROUTING -j SNAT -p tcp --dport 80 -d WEBSERVER1 
--to-source BALANCER_IP_ON_WEBSERVER2_NET

But some other rules may be in effect....

Swifty

Re: POSTROUTING SNAT only reply packets

[GMail Isaac Gonzalez]

Hi,

I know that doing the NAT in the firewall will do the trick, but the 
problem is that the "firewall and webserver" and the load balancer are 
in differents networks, then the webserver replies only goes through the 
firewall, and not though the load balancer. On the other hand the 
loadbalancer isn't a Linux box, then I can't not modify anything about 
packets, moreover I can't do any kind of routing.

VIP 10.0.0.10 |---LB---| LAN 80.67.12.X <<---ROUTERS---> FW 72.10.10.1 
--> WS 72.10.10.10


HTTP REQUEST:
CUSTOMER IP 25.0.0.222 --> VIP 10.0.0.10
VIP 10.0.0.10 --> WS 72.10.10.10
WS 72.10.10.10 --> CUSTOMER 25.0.0.222

The CUSTOMER sees ACK that does'n not correspond with the original 
request (10.0.0.10) then the connection is not established.

I need a FW rule that change to source address of the webservers replies.

WS 72.10.10.10 --> VIP 10.0.0.10

Thanks in advance.


El 17/01/11 11:38, Gáspár Lajos escribió:
> Hi,
>
> 2011-01-17 11:16 keltezéssel, GMail Isaac Gonzalez írta:
>> Hi,
>>
>
>> I've doing some testing and seems that iptables only do SNAT on NEW 
>> connections, and I need to change the ip address of replied packets. 
>> Anybody know some workaround? If anobody do not know some workaround 
>> can you confirm that it's not posible to do this with iptables?
>
> read again the NAT part in the manual:
> man iptables
>
> nat table:
>               nat:
>                   This table is consulted when a packet that creates a 
> new connection is encountered.  It consists of three built-ins: 
> PREROUTING (for altering packets as soon as they come in), OUTPUT 
> (for  altering  locally-gener-
>                   ated packets before routing), and POSTROUTING (for 
> altering packets as they are about to go out).
>
> DNAT target:
>
>    DNAT
>        This  target  is  only valid in the nat table, in the 
> PREROUTING and OUTPUT chains, and user-defined chains which are only 
> called from those chains.  It specifies that the destination address 
> of the packet should be modified
>        (and all future packets in this connection will also be 
> mangled), and rules should cease being examined.  It takes one type of 
> option:
>
> SNAT target:
>
>    SNAT
>        This target is only valid in the nat table, in the POSTROUTING 
> chain.  It specifies that the source address of the packet should be 
> modified (and all future packets in this connection will also be 
> mangled), and rules  should
>        cease being examined.  It takes one type of option:
>
>
>> I've tried the next ip tables rules and only work when I do NEW 
>> connections from the web server.
>>
>> -A POSTROUTING -o br0 -s WE_SERVER_ADDR -p tcp -m tcp --sport 80 
>> --dport 1024:65535 -j SNAT --to-source LOAD_BALANCER_ADDR
>>
>> Thanks in advance.
>>
>> Isaac González
>>
>
> You should do all of the NAT-ing ON THE LOAD BALANCER:
>
> iptables -t nat -A PREROUTING -j DNAT -p tcp --dport 80 
> --to-destination WEBSERVER1 (some load balancing options here)
> iptables -t nat -A PREROUTING -j DNAT -p tcp --dport 80 
> --to-destination WEBSERVER2 (some load balancing options here)
>
> iptables -t nat -A POSTROUTING -j SNAT -p tcp --dport 80 -d WEBSERVER1 
> --to-source BALANCER_IP_ON_WEBSERVER1_NET
> iptables -t nat -A POSTROUTING -j SNAT -p tcp --dport 80 -d WEBSERVER1 
> --to-source BALANCER_IP_ON_WEBSERVER2_NET
>
> But some other rules may be in effect....
>
> Swifty
>

Re: POSTROUTING SNAT only reply packets

[Giles Coochey]

[-- Attachment #1: Type: text/plain, Size: 2625 bytes --]

On 17/01/2011 11:38, Gáspár Lajos wrote:
> Hi,
>
> 2011-01-17 11:16 keltezéssel, GMail Isaac Gonzalez írta:
>> Hi,
>>
>
>> I've doing some testing and seems that iptables only do SNAT on NEW 
>> connections, and I need to change the ip address of replied packets. 
>> Anybody know some workaround? If anobody do not know some workaround 
>> can you confirm that it's not posible to do this with iptables?
>
> read again the NAT part in the manual:
> man iptables
>
> nat table:
>               nat:
>                   This table is consulted when a packet that creates a 
> new connection is encountered.  It consists of three built-ins: 
> PREROUTING (for altering packets as soon as they come in), OUTPUT 
> (for  altering  locally-gener-
>                   ated packets before routing), and POSTROUTING (for 
> altering packets as they are about to go out).
>
> DNAT target:
>
>    DNAT
>        This  target  is  only valid in the nat table, in the 
> PREROUTING and OUTPUT chains, and user-defined chains which are only 
> called from those chains.  It specifies that the destination address 
> of the packet should be modified
>        (and all future packets in this connection will also be 
> mangled), and rules should cease being examined.  It takes one type of 
> option:
>
> SNAT target:
>
>    SNAT
>        This target is only valid in the nat table, in the POSTROUTING 
> chain.  It specifies that the source address of the packet should be 
> modified (and all future packets in this connection will also be 
> mangled), and rules  should
>        cease being examined.  It takes one type of option:
>
>
>> I've tried the next ip tables rules and only work when I do NEW 
>> connections from the web server.
>>
>> -A POSTROUTING -o br0 -s WE_SERVER_ADDR -p tcp -m tcp --sport 80 
>> --dport 1024:65535 -j SNAT --to-source LOAD_BALANCER_ADDR
>>
>> Thanks in advance.
>>
>> Isaac González
>>
>
> You should do all of the NAT-ing ON THE LOAD BALANCER:
>

I have to agree - if you are doing NAT you want to avoid any type of 
asymmetric routing - especially you NEED to make sure that the device 
that is doing the NAT (be it for load balancing or other reasons) 
receives the return packets. You cannot keep a TCP connection going if 
there is not some sort of state information being shared between the 
devices otherwise.

-- 
Best Regards,

Giles Coochey
NetSecSpec Ltd
NL T-Systems Mobile: +31 681 265 086
NL Mobile: +31 626 508 131
Gib Mobile: +350 5401 6693
Email/MSN/Live Messenger: giles@coochey.net
Skype: gilescoochey




[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5137 bytes --]

Re: POSTROUTING SNAT only reply packets

[GMail Isaac Gonzalez]

Ok,

Thanks for the replies, I was doing some tests to asure the connection 
issues. It's compulsory that the load balancer receive the return packets.


El 17/01/11 11:55, Giles Coochey escribió:
> On 17/01/2011 11:38, Gáspár Lajos wrote:
>> Hi,
>>
>> 2011-01-17 11:16 keltezéssel, GMail Isaac Gonzalez írta:
>>> Hi,
>>>
>>
>>> I've doing some testing and seems that iptables only do SNAT on NEW 
>>> connections, and I need to change the ip address of replied packets. 
>>> Anybody know some workaround? If anobody do not know some workaround 
>>> can you confirm that it's not posible to do this with iptables?
>>
>> read again the NAT part in the manual:
>> man iptables
>>
>> nat table:
>>               nat:
>>                   This table is consulted when a packet that creates 
>> a new connection is encountered.  It consists of three built-ins: 
>> PREROUTING (for altering packets as soon as they come in), OUTPUT 
>> (for  altering  locally-gener-
>>                   ated packets before routing), and POSTROUTING (for 
>> altering packets as they are about to go out).
>>
>> DNAT target:
>>
>>    DNAT
>>        This  target  is  only valid in the nat table, in the 
>> PREROUTING and OUTPUT chains, and user-defined chains which are only 
>> called from those chains.  It specifies that the destination address 
>> of the packet should be modified
>>        (and all future packets in this connection will also be 
>> mangled), and rules should cease being examined.  It takes one type 
>> of option:
>>
>> SNAT target:
>>
>>    SNAT
>>        This target is only valid in the nat table, in the POSTROUTING 
>> chain.  It specifies that the source address of the packet should be 
>> modified (and all future packets in this connection will also be 
>> mangled), and rules  should
>>        cease being examined.  It takes one type of option:
>>
>>
>>> I've tried the next ip tables rules and only work when I do NEW 
>>> connections from the web server.
>>>
>>> -A POSTROUTING -o br0 -s WE_SERVER_ADDR -p tcp -m tcp --sport 80 
>>> --dport 1024:65535 -j SNAT --to-source LOAD_BALANCER_ADDR
>>>
>>> Thanks in advance.
>>>
>>> Isaac González
>>>
>>
>> You should do all of the NAT-ing ON THE LOAD BALANCER:
>>
>
> I have to agree - if you are doing NAT you want to avoid any type of 
> asymmetric routing - especially you NEED to make sure that the device 
> that is doing the NAT (be it for load balancing or other reasons) 
> receives the return packets. You cannot keep a TCP connection going if 
> there is not some sort of state information being shared between the 
> devices otherwise.
>

Re: POSTROUTING SNAT only reply packets

[Amos Jeffries]

On 17/01/11 23:51, GMail Isaac Gonzalez wrote:
> Hi,
>
> I know that doing the NAT in the firewall will do the trick, but the
> problem is that the "firewall and webserver" and the load balancer are
> in differents networks, then the webserver replies only goes through the
> firewall, and not though the load balancer. On the other hand the
> loadbalancer isn't a Linux box, then I can't not modify anything about
> packets, moreover I can't do any kind of routing.


Slow down, read again Gáspár response. Think particularly carefully 
about what the SNAT lines are doing there.

I'm fairly sure the LB will be capable of it somehow. If not, you have a 
nice heater to sit your feet on in winter.

As a giant hack you could also do SNAT on the device(s) receiving 
packets from the LB such that the reply packets get routed back through 
the LB.

>
> El 17/01/11 11:38, Gáspár Lajos escribió:
>> Hi,
>>
>> You should do all of the NAT-ing ON THE LOAD BALANCER:
>>
>> iptables -t nat -A PREROUTING -j DNAT -p tcp --dport 80
>> --to-destination WEBSERVER1 (some load balancing options here)
>> iptables -t nat -A PREROUTING -j DNAT -p tcp --dport 80
>> --to-destination WEBSERVER2 (some load balancing options here)
>>
>> iptables -t nat -A POSTROUTING -j SNAT -p tcp --dport 80 -d WEBSERVER1
>> --to-source BALANCER_IP_ON_WEBSERVER1_NET
>> iptables -t nat -A POSTROUTING -j SNAT -p tcp --dport 80 -d WEBSERVER1
>> --to-source BALANCER_IP_ON_WEBSERVER2_NET
>>
>> But some other rules may be in effect....
>>
>> Swifty
>>

AYJ

Re: POSTROUTING SNAT only reply packets

[Jan Engelhardt]

On Monday 2011-01-17 11:55, Giles Coochey wrote:
>>
>> You should do all of the NAT-ing ON THE LOAD BALANCER:
>>
>
>I have to agree - if you are doing NAT you want to avoid any type of asymmetric
>routing - especially you NEED to make sure that the device that is doing the
>NAT (be it for load balancing or other reasons) receives the return packets.

Not strictly. You could utilize a second device whose CTs are synchronized
with the LB to apply the reverse transform, using conntrackd.
Sort of like

digraph { internet -> lb; lb -> web; web -> unnat; unnat -> internet; };

but it only looks feasible to me if your LB is already computationally 
crowded.

Re: POSTROUTING SNAT only reply packets

[Giles Coochey]

[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]

On 17/01/2011 12:36, Jan Engelhardt wrote:
> On Monday 2011-01-17 11:55, Giles Coochey wrote:
>>> You should do all of the NAT-ing ON THE LOAD BALANCER:
>>>
>> I have to agree - if you are doing NAT you want to avoid any type of asymmetric
>> routing - especially you NEED to make sure that the device that is doing the
>> NAT (be it for load balancing or other reasons) receives the return packets.
> Not strictly. You could utilize a second device whose CTs are synchronized
> with the LB to apply the reverse transform, using conntrackd.
> Sort of like
>
> digraph { internet ->  lb; lb ->  web; web ->  unnat; unnat ->  internet; };
>
> but it only looks feasible to me if your LB is already computationally
> crowded.
> --
It also requires the loadbalancer to be using netfilter as well.

If it's a hardware load balancer with proprietary methods then you will 
need symmetric routing through it, unless it supports some form of TCP 
state sharing.

-- 
Best Regards,

Giles Coochey
NetSecSpec Ltd
NL T-Systems Mobile: +31 681 265 086
NL Mobile: +31 626 508 131
Gib Mobile: +350 5401 6693
Email/MSN/Live Messenger: giles@coochey.net
Skype: gilescoochey




[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5137 bytes --]

Re: POSTROUTING SNAT only reply packets

[GMail Isaac Gonzalez]

El 17/01/11 12:41, Giles Coochey escribió:
> On 17/01/2011 12:36, Jan Engelhardt wrote:
>> On Monday 2011-01-17 11:55, Giles Coochey wrote:
>>>> You should do all of the NAT-ing ON THE LOAD BALANCER:
>>>>
>>> I have to agree - if you are doing NAT you want to avoid any type of 
>>> asymmetric
>>> routing - especially you NEED to make sure that the device that is 
>>> doing the
>>> NAT (be it for load balancing or other reasons) receives the return 
>>> packets.
>> Not strictly. You could utilize a second device whose CTs are 
>> synchronized
>> with the LB to apply the reverse transform, using conntrackd.
>> Sort of like
>>
>> digraph { internet ->  lb; lb ->  web; web ->  unnat; unnat ->  
>> internet; };
>>
>> but it only looks feasible to me if your LB is already computationally
>> crowded.
>> -- 
> It also requires the loadbalancer to be using netfilter as well.
>
> If it's a hardware load balancer with proprietary methods then you 
> will need symmetric routing through it, unless it supports some form 
> of TCP state sharing.
>
It works with propietary methods. The real solution if do symmetrical 
routing, all the replies must pass through the LB.  I can't use 
conntrackd because I can't install anything in the LB.

Re: POSTROUTING SNAT only reply packets

[Jan Engelhardt]

On Monday 2011-01-17 12:57, GMail Isaac Gonzalez wrote:
>
>>>>NAT (be it for load balancing or other reasons) receives the
>>>>return packets.
>>>
>>>Not strictly. You could utilize a second device whose CTs are
>>>synchronized with the LB to apply the reverse transform, using
>>>conntrackd.
>>
>>It also requires the loadbalancer to be using netfilter as well.
>
>It works with propietary methods. The real solution if do
>symmetrical routing, all the replies must pass through the LB. I
>can't use conntrackd because I can't install anything in the LB.

Too bad. That heater statement from earlier does have value ;-)
("Did you know?..." That Linux also has a load balancer included.)

Re: POSTROUTING SNAT only reply packets

[Michael Vallaly]

Have you tried the RAWNAT modules from xtables-addons? 

<snip>
iptables -A OUTPUT -s $ORIG_IP -j RAWSNAT --to-source $NEW_IP
</snip>

I've used RAWSNAT in the past, with fairly good success.

-Mike

On Mon, 17 Jan 2011 11:16:48 +0100
GMail Isaac Gonzalez <isaak.gonzalez@gmail.com> wrote:

> Hi,
> 
> I need to modify the reply packets of one web server to allow the 
> connections between a webserver and client using a load balancer.
> 
> The client connections goes to a load balancer, the load balancer 
> forwards the connection to a one web server changing the destination ip, 
> the web server anwser the client with it's own ip address without 
> passing again for the load balancer. In order to stablish the 
> connection, the client needs to receive the web server answer with the 
> correct ip address (in this case, the load balancer VIP address), in 
> other case it receives ACK that it doesn't know about it and the 
> connections is not ESTABLISHED.
> 
> I've doing some testing and seems that iptables only do SNAT on NEW 
> connections, and I need to change the ip address of replied packets. 
> Anybody know some workaround? If anobody do not know some workaround can 
> you confirm that it's not posible to do this with iptables?
> 
> I've tried the next ip tables rules and only work when I do NEW 
> connections from the web server.
> 
> -A POSTROUTING -o br0 -s WE_SERVER_ADDR -p tcp -m tcp --sport 80 --dport 
> 1024:65535 -j SNAT --to-source LOAD_BALANCER_ADDR
> 
> Thanks in advance.
> 
> Isaac González
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Michael Vallaly <mvallaly@nolatency.com>

POSTROUTING SNAT only reply packets

[Isaac González]

Hi,

I need to modify the reply packets of one web server to allow the 
connections between a webserver and client using a load balancer.

The client connections goes to a load balancer, the load balancer 
forwards the connection to a one web server changing the destination ip, 
the web server anwser the client with it's own ip address without 
passing again for the load balancer. In order to stablish the 
connection, the client needs to receive the web server answer with the 
correct ip address (in this case, the load balancer VIP address), in 
other case it receives ACK that it doesn't know about it and the 
connections is not ESTABLISHED.

I've doing some testing and seems that iptables only do SNAT on NEW 
connections, and I need to change the ip address of replied packets. 
Anybody know some workaround? If anobody do not know some workaround can 
you confirm that it's not posible to do this with iptables?

I've tried the next ip tables rules and only work when I do NEW 
connections from the web server.

-A POSTROUTING -o br0 -s WE_SERVER_ADDR -p tcp -m tcp --sport 80 --dport 
1024:65535 -j SNAT --to-source LOAD_BALANCER_ADDR

Thanks in advance.

Isaac González