Re: Multi ISP router/firewall ...

Re: Multi ISP router/firewall ...

[Pierre JUHEN]

Suggested reading : http://www.ssi.bg/~ja/nano.txt

It explains how the routing is done.

Using the ROUTE patch, you force the routing outside the 
routing table.

When a packet comes back on an established connection,
the kernel lookup in the cache, with the input interface 
as part of the matching.

Since in the cache, you have the other interface (ISP1)
for this connection, it doesn't route the packet.

You should use something else than the ROUTE target, for example
what is explained in nano.txt.



> Message du 27/03/07 14:54
> De : "Ray Leach" <spoons@rchq.co.za>
> A : "Netfilter" <netfilter@lists.netfilter.org>
> Copie à : 
> Objet : Multi ISP router/firewall ...
> 
> Hi All
> 
> I've been trying to configure a router/firewall with 5 zones (2 of which
> are ISP links) unsuccessfully.
> 
> So far I've managed to patch the kernel with the ROUTE target, and I've
> got rules as follows:
> 
> FORWARD chain filter table:
> ACCEPT     0    --  ethxx0 eth4    10.0.0.0/24    0.0.0.0/0           
> ACCEPT     0    --  eth4   ethxx0  0.0.0.0/0      10.0.0.0/24
> 
> FORWARD chain mangle table:
> ROUTE      tcp  --  *      *       10.0.0.67    !192.168.1.0/24      tcp
> dpts:80:90 ROUTE oif:eth4 continue
> 
> 
> POSTROUTING chain nat table:
> SNAT       0    --  *      eth4    0.0.0.0/0    0.0.0.0/0  to:10.1.0.2
> 
> 
> ethxx0 is my internal LAN
> eth4 is my 2nd ISP connection
> 10.0.0.67 is my workstation
> 10.1.0.2 is the ip bound to eth4
> 
> The other ISP link is working, we host around 12 websites on servers in
> our DMZ, we have outgoing web browsing, email, MSN, etc. using various
> nat rules for that interface, they all work.
> 
> I can see using tcpdump that the packets are leaving my workstation,
> being mangles to change the route, going out the 2nd ISP link, coming
> back into the 2nd ISP link, but then they disappear.
> 
> My forwarding rules as shown above do not show any traffic.
> 
> This has me stumped.
> 
> As I've stated, I have nat (both SNAT and DNAT) working on the other ISP
> interface on the same machine.
> 
> Where should I start looking?
> 
> Regards
> 
> Ray
> -- 
> Raymond Leach
> RCHQ Hobbies (http://www.rchq.co.za/)
> (T)+27-82-575-6975 (F)+27-86-652-2773
> 
> 
> 
>

Re: Multi ISP router/firewall ...

[Ray Leach]

On Tue, 2007-03-27 at 17:32 +0200, Pierre JUHEN wrote:
> Suggested reading : http://www.ssi.bg/~ja/nano.txt
> 
> It explains how the routing is done.
> 
> Using the ROUTE patch, you force the routing outside the 
> routing table.
> 
> When a packet comes back on an established connection,
> the kernel lookup in the cache, with the input interface 
> as part of the matching.
> 
> Since in the cache, you have the other interface (ISP1)
> for this connection, it doesn't route the packet.
> 
> You should use something else than the ROUTE target, for example
> what is explained in nano.txt.
> 
> 

Hi

I tried both methods - iptables using the ROUTE target as well as using
iptables to mark the packets, then using iproute2 to lookup and route
using a table with an ip fwmark rule.

In both cases, the traffic is routed out and return traffic comes back
in the correct interface, but it does not get NATed backed to the
client.

iptables -A FORWARD -i eth0 -p tcp --dport 80 -s 10.0.0.3 -j ACCEPT
iptables -A FORWARD -i eth4 -p tcp --sport 80 -d 10.0.0.3 -j ACCEPT

iptables -A FORWARD -t mangle -p tcp --dport 80 -s 10.0.0.3 -j MARK
--set-mark 0x4

iptables -A POSTROUTING -t nat -o eth4 -p tcp --dport 80 -s 10.0.0.3 -j
SNAT --to 10.1.0.2

ip rule del fwmark 4 table 4 priority 32000
ip route flush table 4
ip route add table 4 default via 10.1.0.1
ip rule add fwmark 4 table 4 priority 32000
ip route flush cache


What am I doing wrong?

Looking in /proc/net/ip_conntrack I can find an entry for http traffic
from machine at ip 10.0.0.3 created by the SNAT rule above. When the
traffic returns back in eth4 it seems to disappear on the firewall ...

Regards

Ray

> 
> > Message du 27/03/07 14:54
> > De : "Ray Leach" <spoons@rchq.co.za>
> > A : "Netfilter" <netfilter@lists.netfilter.org>
> > Copie à : 
> > Objet : Multi ISP router/firewall ...
> > 
> > Hi All
> > 
> > I've been trying to configure a router/firewall with 5 zones (2 of which
> > are ISP links) unsuccessfully.
> > 
> > So far I've managed to patch the kernel with the ROUTE target, and I've
> > got rules as follows:
> > 
> > FORWARD chain filter table:
> > ACCEPT     0    --  ethxx0 eth4    10.0.0.0/24    0.0.0.0/0           
> > ACCEPT     0    --  eth4   ethxx0  0.0.0.0/0      10.0.0.0/24
> > 
> > FORWARD chain mangle table:
> > ROUTE      tcp  --  *      *       10.0.0.67    !192.168.1.0/24      tcp
> > dpts:80:90 ROUTE oif:eth4 continue
> > 
> > 
> > POSTROUTING chain nat table:
> > SNAT       0    --  *      eth4    0.0.0.0/0    0.0.0.0/0  to:10.1.0.2
> > 
> > 
> > ethxx0 is my internal LAN
> > eth4 is my 2nd ISP connection
> > 10.0.0.67 is my workstation
> > 10.1.0.2 is the ip bound to eth4
> > 
> > The other ISP link is working, we host around 12 websites on servers in
> > our DMZ, we have outgoing web browsing, email, MSN, etc. using various
> > nat rules for that interface, they all work.
> > 
> > I can see using tcpdump that the packets are leaving my workstation,
> > being mangles to change the route, going out the 2nd ISP link, coming
> > back into the 2nd ISP link, but then they disappear.
> > 
> > My forwarding rules as shown above do not show any traffic.
> > 
> > This has me stumped.
> > 
> > As I've stated, I have nat (both SNAT and DNAT) working on the other ISP
> > interface on the same machine.
> > 
> > Where should I start looking?
> > 
> > Regards
> > 
> > Ray
> > -- 
> > Raymond Leach
> > RCHQ Hobbies (http://www.rchq.co.za/)
> > (T)+27-82-575-6975 (F)+27-86-652-2773
> > 
> > 
> > 
> >
> 
> 
> 
-- 
Raymond Leach
RCHQ Hobbies (http://www.rchq.co.za/)
(T)+27-82-575-6975 (F)+27-86-652-2773

Re: Multi ISP router/firewall ...

[Jan Engelhardt]

On Mar 28 2007 07:51, Ray Leach wrote:
>
>I tried both methods - iptables using the ROUTE target as well as using
>iptables to mark the packets, then using iproute2 to lookup and route
>using a table with an ip fwmark rule.
>
>In both cases, the traffic is routed out and return traffic comes back
>in the correct interface, but it does not get NATed backed to the
>client.
>
>iptables -A FORWARD -i eth0 -p tcp --dport 80 -s 10.0.0.3 -j ACCEPT
>iptables -A FORWARD -i eth4 -p tcp --sport 80 -d 10.0.0.3 -j ACCEPT
>
>iptables -A FORWARD -t mangle -p tcp --dport 80 -s 10.0.0.3 -j MARK
>--set-mark 0x4

The routing decision is done before the FORWARDing chain is entered.
Try moving the MARK to INPUT.

>iptables -A POSTROUTING -t nat -o eth4 -p tcp --dport 80 -s 10.0.0.3 -j
>SNAT --to 10.1.0.2
>
>ip rule del fwmark 4 table 4 priority 32000
>ip route flush table 4
>ip route add table 4 default via 10.1.0.1
>ip rule add fwmark 4 table 4 priority 32000
>ip route flush cache
>
>
>What am I doing wrong?
>
>Looking in /proc/net/ip_conntrack I can find an entry for http traffic
>from machine at ip 10.0.0.3 created by the SNAT rule above. When the
>traffic returns back in eth4 it seems to disappear on the firewall ...

Jan
-- 

Re: Multi ISP router/firewall ...

[Ray Leach]

On Wed, 2007-03-28 at 08:18 +0200, Jan Engelhardt wrote:
> On Mar 28 2007 07:51, Ray Leach wrote:
> >
> >I tried both methods - iptables using the ROUTE target as well as using
> >iptables to mark the packets, then using iproute2 to lookup and route
> >using a table with an ip fwmark rule.
> >
> >In both cases, the traffic is routed out and return traffic comes back
> >in the correct interface, but it does not get NATed backed to the
> >client.
> >
> >iptables -A FORWARD -i eth0 -p tcp --dport 80 -s 10.0.0.3 -j ACCEPT
> >iptables -A FORWARD -i eth4 -p tcp --sport 80 -d 10.0.0.3 -j ACCEPT
> >
> >iptables -A FORWARD -t mangle -p tcp --dport 80 -s 10.0.0.3 -j MARK
> >--set-mark 0x4
> 
> The routing decision is done before the FORWARDing chain is entered.
> Try moving the MARK to INPUT.
> 

The source traffic is not from the firewall machine, but another machine
on the local LAN. The mark is being set properly and the traffic is
routed out the eth4 interface correctly, so the mark logic is working.


The setup is something like this:



                                |    eth6|(196.7.34.98)<---->ISP1
|PROXY|(10.0.0.3)<--->(10.0.0.2)|FIREWALL|
                                |    eth4|(10.1.0.2)<--->(10.1.0.1)ISP2


ip route show table main
196.7.34.96/28 dev eth6  proto kernel  scope link  src 196.7.34.98
10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.2
10.1.0.0/24 dev eth4  proto kernel  scope link  src 10.1.0.2

ip route show table 4
default via 10.1.0.1 dev eth4

ip rule
0:      from all lookup local 
32000:  from all fwmark 0x4 lookup 4 
32766:  from all lookup main 
32767:  from all lookup default


tcpdump -n -i eth4 shows my traffic exiting after being SNATed to
10.1.0.2 and reply traffic re-entering .

tcpdump -n -i eth0 src or dst 10.0.0.3 and port 80 shows my traffic
exiting from the source (10.0.0.3), but nothing ever returns.

I have checked to make sure nothing is dropped, and also noticed that
the 2 forwarding rules above only show traffic out from eth0, the return
rule does not show any traffic. This is why I think the SNAT is not
working correctly when the traffic comes back in.

My last option is to do the nat using iproute2 instead of iptables.
My question is, why is SNAT working on the other 4 interfaces on this
firewall, but not on this one?


> >iptables -A POSTROUTING -t nat -o eth4 -p tcp --dport 80 -s 10.0.0.3 -j
> >SNAT --to 10.1.0.2
> >
> >ip rule del fwmark 4 table 4 priority 32000
> >ip route flush table 4
> >ip route add table 4 default via 10.1.0.1
> >ip rule add fwmark 4 table 4 priority 32000
> >ip route flush cache
> >
> >
> >What am I doing wrong?
> >
> >Looking in /proc/net/ip_conntrack I can find an entry for http traffic
> >from machine at ip 10.0.0.3 created by the SNAT rule above. When the
> >traffic returns back in eth4 it seems to disappear on the firewall ...
> 
> Jan
-- 
Raymond Leach
RCHQ Hobbies (http://www.rchq.co.za/)
(T)+27-82-575-6975 (F)+27-86-652-2773

Re: Multi ISP router/firewall ...

[Raymond Leach]

Hi All

Managed to sort this out.

The problem was that reverse path filtering had to be enabled on the
eth4 interface. All my other interfaces have reverse path filtering
disabled, so what's the difference with this one?

Regards

Ray

On Wed, 2007-03-28 at 08:58 +0200, Ray Leach wrote:
> On Wed, 2007-03-28 at 08:18 +0200, Jan Engelhardt wrote:
> > On Mar 28 2007 07:51, Ray Leach wrote:
> > >
> > >I tried both methods - iptables using the ROUTE target as well as using
> > >iptables to mark the packets, then using iproute2 to lookup and route
> > >using a table with an ip fwmark rule.
> > >
> > >In both cases, the traffic is routed out and return traffic comes back
> > >in the correct interface, but it does not get NATed backed to the
> > >client.
> > >
> > >iptables -A FORWARD -i eth0 -p tcp --dport 80 -s 10.0.0.3 -j ACCEPT
> > >iptables -A FORWARD -i eth4 -p tcp --sport 80 -d 10.0.0.3 -j ACCEPT
> > >
> > >iptables -A FORWARD -t mangle -p tcp --dport 80 -s 10.0.0.3 -j MARK
> > >--set-mark 0x4
> > 
> > The routing decision is done before the FORWARDing chain is entered.
> > Try moving the MARK to INPUT.
> > 
> 
> The source traffic is not from the firewall machine, but another machine
> on the local LAN. The mark is being set properly and the traffic is
> routed out the eth4 interface correctly, so the mark logic is working.
> 
> 
> The setup is something like this:
> 
> 
> 
>                                 |    eth6|(196.7.34.98)<---->ISP1
> |PROXY|(10.0.0.3)<--->(10.0.0.2)|FIREWALL|
>                                 |    eth4|(10.1.0.2)<--->(10.1.0.1)ISP2
> 
> 
> ip route show table main
> 196.7.34.96/28 dev eth6  proto kernel  scope link  src 196.7.34.98
> 10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.2
> 10.1.0.0/24 dev eth4  proto kernel  scope link  src 10.1.0.2
> 
> ip route show table 4
> default via 10.1.0.1 dev eth4
> 
> ip rule
> 0:      from all lookup local 
> 32000:  from all fwmark 0x4 lookup 4 
> 32766:  from all lookup main 
> 32767:  from all lookup default
> 
> 
> tcpdump -n -i eth4 shows my traffic exiting after being SNATed to
> 10.1.0.2 and reply traffic re-entering .
> 
> tcpdump -n -i eth0 src or dst 10.0.0.3 and port 80 shows my traffic
> exiting from the source (10.0.0.3), but nothing ever returns.
> 
> I have checked to make sure nothing is dropped, and also noticed that
> the 2 forwarding rules above only show traffic out from eth0, the return
> rule does not show any traffic. This is why I think the SNAT is not
> working correctly when the traffic comes back in.
> 
> My last option is to do the nat using iproute2 instead of iptables.
> My question is, why is SNAT working on the other 4 interfaces on this
> firewall, but not on this one?
> 
> 
> > >iptables -A POSTROUTING -t nat -o eth4 -p tcp --dport 80 -s 10.0.0.3 -j
> > >SNAT --to 10.1.0.2
> > >
> > >ip rule del fwmark 4 table 4 priority 32000
> > >ip route flush table 4
> > >ip route add table 4 default via 10.1.0.1
> > >ip rule add fwmark 4 table 4 priority 32000
> > >ip route flush cache
> > >
> > >
> > >What am I doing wrong?
> > >
> > >Looking in /proc/net/ip_conntrack I can find an entry for http traffic
> > >from machine at ip 10.0.0.3 created by the SNAT rule above. When the
> > >traffic returns back in eth4 it seems to disappear on the firewall ...
> > 
> > Jan
-- 
Raymond Leach
Knowledge Factory (http://www.knowledgefactory.co.za)
(Tel)+27-11-445-8100  (Fax)+27-11-445-8101



------------------------------------------------------------------------------------------
This e-mail was checked by the e-Sweeper Service.
For more information visit our website, Clearswift Corporation e-Sweeper :
http://www.mimesweeper.com/products/esweeper/
------------------------------------------------------------------------------------------

Re: Multi ISP router/firewall ...

[Ray Leach]

Hi All

Managed to sort this out.

The problem was that reverse path filtering had to be enabled on the
eth4 interface. All my other interfaces have reverse path filtering
disabled, so what's the difference with this one?

Regards

Ray

On Wed, 2007-03-28 at 08:58 +0200, Ray Leach wrote:
> On Wed, 2007-03-28 at 08:18 +0200, Jan Engelhardt wrote:
> > On Mar 28 2007 07:51, Ray Leach wrote:
> > >
> > >I tried both methods - iptables using the ROUTE target as well as using
> > >iptables to mark the packets, then using iproute2 to lookup and route
> > >using a table with an ip fwmark rule.
> > >
> > >In both cases, the traffic is routed out and return traffic comes back
> > >in the correct interface, but it does not get NATed backed to the
> > >client.
> > >
> > >iptables -A FORWARD -i eth0 -p tcp --dport 80 -s 10.0.0.3 -j ACCEPT
> > >iptables -A FORWARD -i eth4 -p tcp --sport 80 -d 10.0.0.3 -j ACCEPT
> > >
> > >iptables -A FORWARD -t mangle -p tcp --dport 80 -s 10.0.0.3 -j MARK
> > >--set-mark 0x4
> > 
> > The routing decision is done before the FORWARDing chain is entered.
> > Try moving the MARK to INPUT.
> > 
> 
> The source traffic is not from the firewall machine, but another machine
> on the local LAN. The mark is being set properly and the traffic is
> routed out the eth4 interface correctly, so the mark logic is working.
> 
> 
> The setup is something like this:
> 
> 
> 
>                                 |    eth6|(196.7.34.98)<---->ISP1
> |PROXY|(10.0.0.3)<--->(10.0.0.2)|FIREWALL|
>                                 |    eth4|(10.1.0.2)<--->(10.1.0.1)ISP2
> 
> 
> ip route show table main
> 196.7.34.96/28 dev eth6  proto kernel  scope link  src 196.7.34.98
> 10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.2
> 10.1.0.0/24 dev eth4  proto kernel  scope link  src 10.1.0.2
> 
> ip route show table 4
> default via 10.1.0.1 dev eth4
> 
> ip rule
> 0:      from all lookup local 
> 32000:  from all fwmark 0x4 lookup 4 
> 32766:  from all lookup main 
> 32767:  from all lookup default
> 
> 
> tcpdump -n -i eth4 shows my traffic exiting after being SNATed to
> 10.1.0.2 and reply traffic re-entering .
> 
> tcpdump -n -i eth0 src or dst 10.0.0.3 and port 80 shows my traffic
> exiting from the source (10.0.0.3), but nothing ever returns.
> 
> I have checked to make sure nothing is dropped, and also noticed that
> the 2 forwarding rules above only show traffic out from eth0, the return
> rule does not show any traffic. This is why I think the SNAT is not
> working correctly when the traffic comes back in.
> 
> My last option is to do the nat using iproute2 instead of iptables.
> My question is, why is SNAT working on the other 4 interfaces on this
> firewall, but not on this one?
> 
> 
> > >iptables -A POSTROUTING -t nat -o eth4 -p tcp --dport 80 -s 10.0.0.3 -j
> > >SNAT --to 10.1.0.2
> > >
> > >ip rule del fwmark 4 table 4 priority 32000
> > >ip route flush table 4
> > >ip route add table 4 default via 10.1.0.1
> > >ip rule add fwmark 4 table 4 priority 32000
> > >ip route flush cache
> > >
> > >
> > >What am I doing wrong?
> > >
> > >Looking in /proc/net/ip_conntrack I can find an entry for http traffic
> > >from machine at ip 10.0.0.3 created by the SNAT rule above. When the
> > >traffic returns back in eth4 it seems to disappear on the firewall ...
> > 
> > Jan
-- 
Raymond Leach
Knowledge Factory (http://www.knowledgefactory.co.za)
(Tel)+27-11-445-8100  (Fax)+27-11-445-8101
-- 
Raymond Leach
RCHQ Hobbies (http://www.rchq.co.za/)
(T)+27-82-575-6975 (F)+27-86-652-2773

Re: Multi ISP router/firewall ...

[Jan Engelhardt]

On Mar 28 2007 14:21, Ray Leach wrote:
>
>The problem was that reverse path filtering had to be enabled on the
>eth4 interface. All my other interfaces have reverse path filtering
>disabled, so what's the difference with this one?

I suspect something, hence please try

>> > >iptables -A FORWARD -i eth0 -p tcp --dport 80 -s 10.0.0.3 -j ACCEPT
>> > >iptables -A FORWARD -i eth4 -p tcp --sport 80 -d 10.0.0.3 -j ACCEPT
>> > >
>> > >iptables -A FORWARD -t mangle -p tcp --dport 80 -s 10.0.0.3 -j MARK
>> > >--set-mark 0x4

iptables -t mangle -A PREROUTING/FORWARD -i eth4 -p tcp --sport 80 -d 10.0.0.3
-j MARK --set-mark 0x4

with rp_filter=1 again.

I believe the reverse route stuff does not get it, because return packets are
not marked. Or, I'd also suggest

iptables -t mangle -A FORWARD -m conntrack --ctstate NEW
-s 10.0.0.3 -p tcp --dport 80 -j CONNMARK --set-mark 4;
iptables -t mangle -A FORWARD -j CONNMARK --restore-mark;

(Or use nat's PREROUTING for the initial CONNMARK - many ways of doing things)


Jan
-- 

Multi ISP router/firewall ...

[Ray Leach]

Hi All

I've been trying to configure a router/firewall with 5 zones (2 of which
are ISP links) unsuccessfully.

So far I've managed to patch the kernel with the ROUTE target, and I've
got rules as follows:

FORWARD chain filter table:
ACCEPT     0    --  ethxx0 eth4    10.0.0.0/24    0.0.0.0/0           
ACCEPT     0    --  eth4   ethxx0  0.0.0.0/0      10.0.0.0/24

FORWARD chain mangle table:
ROUTE      tcp  --  *      *       10.0.0.67    !192.168.1.0/24      tcp
dpts:80:90 ROUTE oif:eth4 continue


POSTROUTING chain nat table:
SNAT       0    --  *      eth4    0.0.0.0/0    0.0.0.0/0  to:10.1.0.2


ethxx0 is my internal LAN
eth4 is my 2nd ISP connection
10.0.0.67 is my workstation
10.1.0.2 is the ip bound to eth4

The other ISP link is working, we host around 12 websites on servers in
our DMZ, we have outgoing web browsing, email, MSN, etc. using various
nat rules for that interface, they all work.

I can see using tcpdump that the packets are leaving my workstation,
being mangles to change the route, going out the 2nd ISP link, coming
back into the 2nd ISP link, but then they disappear.

My forwarding rules as shown above do not show any traffic.

This has me stumped.

As I've stated, I have nat (both SNAT and DNAT) working on the other ISP
interface on the same machine.

Where should I start looking?

Regards

Ray
-- 
Raymond Leach
RCHQ Hobbies (http://www.rchq.co.za/)
(T)+27-82-575-6975 (F)+27-86-652-2773