[LARTC] fwmark routing of locally generated packets

[LARTC] fwmark routing of locally generated packets

[Thomas Themel]

[-- Attachment #1: Type: text/plain, Size: 3970 bytes --]

Hi,

I'm currently trying to get a Linux machine to route all traffic coming
from a certain UID over a dedicated PPP interface. After going throught
the available documentation and experimenting a bit, I settled for the
following attempt:


# 62.46.87.104 - local PPP address
# 195.4.91.104 - PPP peer
ip route add 195.3.91.104 dev ppp0 src 62.46.87.104 table special
# local for DNS etc
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table special
ip route add default via 195.3.91.104 src 62.46.87.104 table special
ip rule add fwmark 3 lookup special
iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK --set-mark 3
ip route flush cache

This seems to work in a way. It correctly sends the packets generated by
that user out the ppp0 interface, but they don't get the correct source
address:

| sophokles:~# sh -x description.txt 
| + ip route flush table aonc
| + ip route add 195.3.91.103 dev ppp0 src 62.46.86.137 table aonc
| + ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc
| + ip route add default via 195.3.91.103 src 62.46.86.137 table aonc
| + ip rule add fwmark 3 lookup aonc
| + iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK
| --set-mark 3
| + ip route flush cache
| sophokles:~# tcpdump -ni ppp0 port 22  & 
| [1] 841
| sophokles:~# tcpdump: listening on ppp0
| 
| sophokles:~# nc iwoars.net 22
| SSH-1.99-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3
| 
| sophokles:~# su - freenet
| freenet@sophokles:~$ nc iwoars.net 22
| 01:25:17.044883 192.168.1.1.32848 > 217.160.110.113.22: SWE
| 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2056533
| 0,nop,wscale 0> (DF)
| 01:25:20.043828 192.168.1.1.32848 > 217.160.110.113.22: SWE
| 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2059533
| 0,nop,wscale 0> (DF)
| 01:25:26.042584 192.168.1.1.32848 > 217.160.110.113.22: SWE
| 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2065533
| 0,nop,wscale 0> (DF)
| 
| freenet@sophokles:~$ 

I've read on this list that owner-based policy routing is impossible
because the routing decision is made before the packet traverses the
OUTPUT chain. However, if this is true, then I don't understand how the
packet can go out via the correct interface unless there are separate
route lookups to determine the source address and outgoing interface.

Could someone who knows please elaborate?

I have also tried, unsuccessfully, to just mangle the source address
using an iptables SNAT rule, but even though that produces correct
network traffic, it seems to break something internally that keeps the
TCP handshake from completing:

| sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 --to-source
| 62.46.86.137
| sophokles:~# su - freenet
| freenet@sophokles:~$ nc iwoars.net 22
| 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE
| 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000
| 0,nop,wscale 0> (DF)
| 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S
| 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
| 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF)
| 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE
| 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000
| 0,nop,wscale 0> (DF)
| 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S
| 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
| 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF)
| 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S
| 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
| 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF)
| [...]

ciao,
-- 
[*Thomas  Themel*]   Since you obviously haven't lurked enough,
[extended contact]   we subtly tell lusers to piss off via the words "piss
[info provided in]   off, luser".  
[*message header*]      - David P. Murphy in the monastery

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [LARTC] fwmark routing of locally generated packets

[Thomas Themel]

[-- Attachment #1: Type: text/plain, Size: 2975 bytes --]

Hi,
christopher cuse (ccuse@manchotnetworks.net) wrote on 2003-10-28:
> i had the same issue ... try adding to your iptables:
> 
> iptables --append POSTROUTING --table nat --match mark \ 
>          --mark 0x3 --jump SNAT --to-source 62.46.87.104
> 
> ip route flush cache

I already tried that (though with a different match):

> > I have also tried, unsuccessfully, to just mangle the source address
> > using an iptables SNAT rule, but even though that produces correct
> > network traffic, it seems to break something internally that keeps the
> > TCP handshake from completing:
> > 
> > | sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 --to-source
> > | 62.46.86.137
> > | sophokles:~# su - freenet
> > | freenet@sophokles:~$ nc iwoars.net 22
> > | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE
> > | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000
> > | 0,nop,wscale 0> (DF)
> > | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S
> > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
> > | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF)
> > | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE
> > | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000
> > | 0,nop,wscale 0> (DF)
> > | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S
> > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
> > | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF)
> > | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S
> > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
> > | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF)

So, the packets go out with the correct source address, the SYN-ACK
comes back from the target host, but the local IP stack then doesn't
send the last ACK back because it doesn't feel that these packets 
belong to a local socket.

That is quite understandable, since doing a netstat reveals that the
socket believes it's using 192.168.1.1 as its source address (netstat
output doesn't match tcpdump output from above, of course, because the
tcpdump is from yesterday, but I did the same things).

| freenet@sophokles:~$ netstat -ant | grep SYN_SENT
| tcp        0      1 192.168.1.1:32915       217.160.110.113:22 SYN_SENT   

Now, with the ugly hacks mounting, I've considered something like 

iptables -A INPUT -i ppp0 -j DNAT --to-destination 192.168.1.1 

but of course that doesn't work.

At the core of the problem is still how the socket ever got to be have the
local address of 192.168.1.1. I'm pretty sure netcat doesn't bind to a
specific local address, so it must be something in iproute2 behaving
odd.

ciao,
-- 
[*Thomas  Themel*]      Frankly, many of you on this list really need to 
[extended contact]      be doused with gasoline and then lit.
[info provided in]      
[*message header*]      - Tim May on cypherpunks

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[LARTC] fwmark routing of locally generated packets

[Brad Barnett]

Hi Thomas,

We have the same problem. ;)  You're right, it doesn't make any sense.

Can anyone elaborate??  My setup is virtually identical to what Thomas
has.  However, I'm using IPMASQ on my outgoing connection, so I am able to
see that the problem exists even without using SNAT. 

Something is borked...  

> Thomas Themel  themel@iwoars.net
> Tue, 28 Oct 2003 01:32:09 +0100
> 
> Hi,
> 
> I'm currently trying to get a Linux machine to route all traffic coming
> from a certain UID over a dedicated PPP interface. After going throught
> the available documentation and experimenting a bit, I settled for the
> following attempt:
> 
> 
> # 62.46.87.104 - local PPP address
> # 195.4.91.104 - PPP peer
> ip route add 195.3.91.104 dev ppp0 src 62.46.87.104 table special
> # local for DNS etc
> ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table special
> ip route add default via 195.3.91.104 src 62.46.87.104 table special
> ip rule add fwmark 3 lookup special
> iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK
> --set-mar= k 3
> ip route flush cache
> 
> This seems to work in a way. It correctly sends the packets generated by
> that user out the ppp0 interface, but they don't get the correct source
> address:
> 
> | sophokles:~# sh -x description.txt 
> | + ip route flush table aon> | + ip route add 195.3.91.103 dev ppp0 src 62.46.86.137 table aonc
> | + ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc
> | + ip route add default via 195.3.91.103 src 62.46.86.137 table aonc
> | + ip rule add fwmark 3 lookup aonc
> | + iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK
> | --set-mark 3
> | + ip route flush cache
> | sophokles:~# tcpdump -ni ppp0 port 22  & 
> | [1] 841
> | sophokles:~# tcpdump: listening on ppp0
> | 
> | sophokles:~# nc iwoars.net 22
> | SSH-1.99-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3
> | 
> | sophokles:~# su - freenet
> | freenet@sophokles:~$ nc iwoars.net 22
> | 01:25:17.044883 192.168.1.1.32848 > 217.160.110.113.22: SWE
> | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2056533
> | 0,nop,wscale 0> (DF)
> | 01:25:20.043828 192.168.1.1.32848 > 217.160.110.113.22: SWE
> | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2059533
> | 0,nop,wscale 0> (DF)
> | 01:25:26.042584 192.168.1.1.32848 > 217.160.110.113.22: SWE
> | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2065533
> | 0,nop,wscale 0> (DF)
> | 
> | freenet@sophokles:~$ 
> 
> I've read on this list that owner-based policy routing is impossible
> because the routing decision is made before the packet traverses the
> OUTPUT chain. However, if this is true, then I don't understand how the
> packet can go out via the correct interface unless there are separate
> route lookups to determine the source address and outgoing interface.
> 
> Could someone who knows please elaborate?
> 
> I have also tried, unsuccessfully, to just mangle the source address
> using an iptables SNAT rule, but even though that produces correct
> network traffic, it seems to break something internally that keeps the
> TCP handshake from completing:
> 
> | sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0
> --to-source| 62.46.86.137
> | sophokles:~# su - freenet
> | freenet@sophokles:~$ nc iwoars.net 22
> | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE
> | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000
> | 0,nop,wscale 0> (DF)
> | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S
> | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
> | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF)
> | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE
> | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000
> | 0,nop,wscale 0> (DF)
> | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S
> | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
> | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF)
> | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S
> | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss
> | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF)
> | [...]

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] fwmark routing of locally generated packets

[Thomas Themel]

Hi,
Brad Barnett (bb@L8R.net) wrote on 2003-11-01:
> We have the same problem. ;)  You're right, it doesn't make any sense.

Yeah, that's what I feel. Unfortunately, my message to linux-netdev went
unnoticed, and I don't really know of any reference against which to
compare whether the current behaviour is intended or not.

> Can anyone elaborate??  My setup is virtually identical to what Thomas
> has.  However, I'm using IPMASQ on my outgoing connection, so I am able to
> see that the problem exists even without using SNAT. 

This is what I do currently, and it seems to work. This is from an ip-up
script for my PPP link:

ip route flush table aonc
ip route add $GWIP dev $PPP_IFACE src $PPP_LOCAL table aonc
# local for DNS etc
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc
# internet via PPP lnk
ip route add default via $PPP_REMOTE src $PPP_LOCAL table aonc
ip rule add fwmark 3 lookup aonc

# owner tagging
iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK --set-mark 3
# SNAT for outgoing packets
iptables -A POSTROUTING -t nat -o $PPP_IFACE --match mark --mark 0x03 -j SNAT --to-source $PPP_LOCAL
# DNAT for incoming packets
iptables -t nat -A PREROUTING -i $PPP_IFACE  -d $PPP_LOCAL -j DNAT --to-destination 192.168.1.1
ip route flush cache

However, this is feels like a very bad hack - the final DNAT makes the
interface unusable for other purposes, and the interaction between the
sockets' addresses and the packets' addresses are too complicated for my
taste. It's okay for my home network, but I wouldn't want to run this
productively. 

ciao,
-- 
[*Thomas  Themel*] Our national shield displays a lion with two tails. It's
[extended contact] rumoured that it symbolizes the nature of our governments;
[info provided in] that there actually were two lions, but one is now stuck
[*message header*] deep in the ass of the other one.  - Thomas Shaddack
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] fwmark routing of locally generated packets

[Thomas Themel]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

Hi,
Thomas Themel (themel@iwoars.net) wrote on 2003-11-01:
> # SNAT for outgoing packets
> iptables -A POSTROUTING -t nat -o $PPP_IFACE --match mark --mark 0x03 -j SNAT --to-source $PPP_LOCAL

I've been able to do away with the DNAT rule now.

> # DNAT for incoming packets
> iptables -t nat -A PREROUTING -i $PPP_IFACE  -d $PPP_LOCAL -j DNAT --to-destination 192.168.1.1

I couldn't get it to work with just the SNAT rule originally (see
original post, the SACKs would be ignored), but I've finally figured out
why: I had enabled rp_filter on that machine.  

Quite obviously, enabling rp_filter in combination with policy routing
is a bad idea.

echo 0 > /proc/sys/net/ipv4/conf/$PPP_IFACE/rp_filter 

fixed it for me.

I still think I shouldn't need the netfilter SNAT rule, but even
enabling NAT with the routing rule (ip rule add fwmark 3 table aonc nat
$PPP_REMOTE) doesn't seem to set the source address of the outgoing
packets correctly.

ciao,
-- 
[*Thomas  Themel*] "If we're not supposed to eat animals, how come 
[extended contact]  they're made of meat?"
[info provided in]      - Treat Carnivores Ethically,
[*message header*]              in the fuckedcompany.com forums

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]