still can't route using fwmark

still can't route using fwmark

[Lloyd Standish]

Hello,

I am still having trouble doing load balancing over ppp0 and ppp1 for a Debian Etch PC (kernel 2.6.20.3, iptables 1.3.8).  I used patch-o-matic to update my kernel sources and recompiled the kernel, and compiled and installed iptables 1.3.8.  Also all the netfilter-related kernel configuration options I need are set, I believe.

My basic problem is that no matter how packets and connections are marked with the MARK and CONNMARK targets, only the interface with a "default" entry in the main routing table is used.   In this case, that is ppp0.  It is set as the default route using "ip route add default dev ppp0."   After that, ONLY ppp0 is used for outgoing/incoming packets, even though packets/connections carry fwmark 2, which I have associated with a user-defined routing table whose default route specifies ppp1.

For example, my /etc/iproute2/rt_tables looks like this:

debiandesk:/home/lloyd/data/loadbal# cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
251     rt_link1
252     rt_link2

If $gw0 is the ppp0 gateway, and $gw1 is the ppp1 gateway, I understand the following commands set up custom tables rt_link1 and rt_link2 to use ppp0 and ppp1, respectively:

ip route flush table rt_link1 2>/dev/null
ip route add $gw0 dev ppp0 table rt_link1
ip route add default via $gw0 dev ppp0 table rt_link1

ip route flush table rt_link2 2>/dev/null
ip route add $gw1 dev ppp1 table rt_link2
ip route add default via $gw1 dev ppp1 table rt_link2

(Actually, I think only a default route for each table should be enough, and that it could be set for example as follows: ip route add table rt_link1 default dev ppp0)

I also added the following rules to link the fwmark to tables rt_link1 and rt_link2 (Is this correct?)

ip rule add fwmark 1 table rt_link1
ip rule add fwmark 2 table rt_link2

I did this to made the rules take effect:
ip route flush cache

10.60.255.254 is the gateway IP for both ppp0 and ppp1.  The local IP address for ppp0 is 10.60.10.106, and for ppp1 it is 10.60.0.249.  The following commands verify that the settings are all correct:

debiandesk:/home/lloyd/data/loadbal# ip route show
10.60.255.254 dev ppp0  proto kernel  scope link  src 10.60.10.106
10.60.255.254 dev ppp1  proto kernel  scope link  src 10.60.0.249
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default dev ppp0  scope link
debiandesk:/home/lloyd/data/loadbal# ip rule show
0:      from all lookup 255
32764:  from all fwmark 0x2 lookup rt_link2
32765:  from all fwmark 0x1 lookup rt_link1
32766:  from all lookup main
32767:  from all lookup default

(I am also doing SNAT to set the destination IP to that of the proper interface.)

And yet, it appears that packets marked with fwmark 2 do NOT leave through ppp1, as I expect:

debiandesk:/home/lloyd/data/loadbal# cat /proc/net/ip_conntrack
tcp      6 431978 ESTABLISHED src=10.60.10.106 dst=94.229.68.32 sport=54788 dport=5223 packets=181 bytes=19054 src=94.229.68.32 dst=10.60.10.106 sport=5223 dport=54788 packets=176 bytes=21161 [ASSURED] mark=2 secmark=0 use=1
<snip>

And:
debiandesk:/home/lloyd/data/loadbal# netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500 0         0      0      0      0     3036      0      0      0 BMRU
lo    16436 0       708      0      0      0      708      0      0      0 LRU
ppp0   1500 0      3466      0      0      0     3605      0      0      0 MOPRU
ppp1   1500 0        60      0      0      0        3      0      0      0 MOPRU

(Most packets, regardless of fwmark, are routed through ppp0, the default route.  But what about the 60 RX-OK figure for ppp1?)

Please, can someone help me find the error here?  Is some sort of configuration setting necessary to make fwmark kick in?  I do not find a kernel config option to turn FWMARK on.  Nor do I find a "multiple routing tables" setting for IPv4.  Here are my netfilter-related kernel config settings:

# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK_ENABLED=m
CONFIG_NF_CONNTRACK_SUPPORT=y
# CONFIG_IP_NF_CONNTRACK_SUPPORT is not set
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CT_PROTO_GRE=m
# CONFIG_NF_CT_PROTO_SCTP is not set
# CONFIG_NF_CONNTRACK_AMANDA is not set
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
# CONFIG_NETFILTER_XT_MATCH_MAC is not set
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
# CONFIG_NETFILTER_XT_MATCH_HASHLIMIT is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
# CONFIG_NF_NAT_AMANDA is not set
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

Here are the relevant modules loaded:

Module                  Size  Used by
ipt_MASQUERADE          3584  1
ipt_LOG                 6912  5
xt_CONNMARK             3328  3
xt_MARK                 2560  2
iptable_mangle          3072  1
iptable_filter          3200  0
xt_state                2688  1
iptable_nat             7684  1
ip_tables              12616  3 iptable_mangle,iptable_filter,iptable_nat
nf_nat                 18604  2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4      17932  6 iptable_nat
xt_mark                 2048  1
nf_conntrack_ftp        9984  0
xt_statistic            2560  1
xt_connmark             2560  0
nf_conntrack           60248  8 ipt_MASQUERADE,xt_CONNMARK,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4,nf_conntrack_ftp,xt_connmark
nfnetlink               6936  3 nf_nat,nf_conntrack_ipv4,nf_conntrack
x_tables               15236  10 ipt_MASQUERADE,ipt_LOG,xt_CONNMARK,xt_MARK,xt_state,iptable_nat,ip_tables,xt_mark,xt_statistic,xt_connmark

Re: still can't route using fwmark

[Thomas Jacob]

On Fri, Apr 17, 2009 at 10:40:49PM -0600, Lloyd Standish wrote:
> Hello,
> 
> I am still having trouble doing load balancing over ppp0 and ppp1 for a Debian Etch PC (kernel 2.6.20.3, iptables 1.3.8).  I used patch-o-matic to update my kernel sources and recompiled the kernel, and compiled and installed iptables 1.3.8.  Also all the netfilter-related kernel configuration options I need are set, I believe.

Why the strange versions? There are neither the ones that comes with Etch nor
are they anything current? Anyway, Etch's stock kernel and iptables (2.6.18/v1.3.6 based)
are fine for what you want to do. 

You need the iptables (conn)marking stuff, and advanced router + policy/fw routing,
and that's all part of stock packages.

/boot/config-2.6.18-6-686

CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
# CONFIG_IP_ROUTE_MULTIPATH_CACHED is not set
CONFIG_IP_ROUTE_VERBOSE=y

dpkg -L iptables | grep -i mark
 /lib/iptables/libipt_connmark.so
 /lib/iptables/libipt_mark.so
 /lib/iptables/libipt_CONNMARK.so
 /lib/iptables/libipt_MARK.so
 /lib/iptables/libipt_SECMARK.so
 /lib/iptables/libipt_CONNSECMARK.so
 /lib/iptables/libip6t_connmark.so
 /lib/iptables/libip6t_mark.so
 /lib/iptables/libip6t_CONNMARK.so
 /lib/iptables/libip6t_MARK.so
 /lib/iptables/libip6t_SECMARK.so
 /lib/iptables/libip6t_CONNSECMARK.so

Here's what I used some time ago to do something similar (either with Etch
or maybe even Sarge), sending outgoing packets back to the router the
corresponding incoming packets of a connection came from, for a single machine.


#!/bin/sh

iptables=iptables


$iptables -t mangle -A INPUT -m mac --mac <ROUTERMAC1> -j CONNMARK --set-mark 1
$iptables -t mangle -A INPUT -m mac --mac <ROUTERMAC2> -j CONNMARK --set-mark 2


$iptables -t mangle -N TONET 
$iptables -t mangle -F TONET 
$iptables -t mangle -A OUTPUT -o <DEVICE> -d ! <LOCAL_NET> -j TONET

$iptables -t mangle -A TONET -m connmark ! --mark 0 -j CONNMARK --restore-mark


The route/rule stuff was more or less what you've got. If you want to
forward packets for other hosts, this need to go into PREROUTING of
course.

    Thomas

Re: still can't route using fwmark

[Lloyd Standish]

Thanks for the reply, Thomas.  I've got iptables 1.3.8 because I downloaded it (and the patch-o-matic stuff) quite a while back when it was the most current version.  I'm using it because my kernel is not the stock Etch kernel (it is in fact a Ubuntu kernel, necessary to get a module that Debian removed over licensing issues).  I recompiled my kernel and installed itables 1.3.8 out of desperation, since I have studied this load-balancing  stuff pretty thoroughly and it seems what I have ought to work.

My kernel 2.6.20.3 does not have a CONFIG_IP_ROUTE_FWMARK configuration setting!  Has this been absorbed into another configuration setting?  How can I be sure fwmark is working?  Judging by my setup, which is very simple, packets and connections are being marked, but routing is not affected by fwmark.

I have the kernel config options IP_NF_CONNTRACK_SUPPORT and XFRM_SUB_POLICY both set OFF.  Might this be a problem?

--
Lloyd

On Sat, 18 Apr 2009 02:23:15 -0600, Thomas Jacob <jacob@internet24.de> wrote:

> On Fri, Apr 17, 2009 at 10:40:49PM -0600, Lloyd Standish wrote:
>> Hello,
>>
>> I am still having trouble doing load balancing over ppp0 and ppp1 for a Debian Etch PC (kernel 2.6.20.3, iptables 1.3.8).  I used patch-o-matic to update my kernel sources and recompiled the kernel, and compiled and installed iptables 1.3.8.  Also all the netfilter-related kernel configuration options I need are set, I believe.
>
> Why the strange versions? There are neither the ones that comes with Etch nor
> are they anything current? Anyway, Etch's stock kernel and iptables (2.6.18/v1.3.6 based)
> are fine for what you want to do.
>
> You need the iptables (conn)marking stuff, and advanced router + policy/fw routing,
> and that's all part of stock packages.
>
> /boot/config-2.6.18-6-686
>
> CONFIG_IP_ADVANCED_ROUTER=y
> CONFIG_ASK_IP_FIB_HASH=y
> # CONFIG_IP_FIB_TRIE is not set
> CONFIG_IP_FIB_HASH=y
> CONFIG_IP_MULTIPLE_TABLES=y
> CONFIG_IP_ROUTE_FWMARK=y
> CONFIG_IP_ROUTE_MULTIPATH=y
> # CONFIG_IP_ROUTE_MULTIPATH_CACHED is not set
> CONFIG_IP_ROUTE_VERBOSE=y
>
> dpkg -L iptables | grep -i mark
>  /lib/iptables/libipt_connmark.so
>  /lib/iptables/libipt_mark.so
>  /lib/iptables/libipt_CONNMARK.so
>  /lib/iptables/libipt_MARK.so
>  /lib/iptables/libipt_SECMARK.so
>  /lib/iptables/libipt_CONNSECMARK.so
>  /lib/iptables/libip6t_connmark.so
>  /lib/iptables/libip6t_mark.so
>  /lib/iptables/libip6t_CONNMARK.so
>  /lib/iptables/libip6t_MARK.so
>  /lib/iptables/libip6t_SECMARK.so
>  /lib/iptables/libip6t_CONNSECMARK.so
>
> Here's what I used some time ago to do something similar (either with Etch
> or maybe even Sarge), sending outgoing packets back to the router the
> corresponding incoming packets of a connection came from, for a single machine.
>
>
> #!/bin/sh
>
> iptables=iptables
>
>
> $iptables -t mangle -A INPUT -m mac --mac <ROUTERMAC1> -j CONNMARK --set-mark 1
> $iptables -t mangle -A INPUT -m mac --mac <ROUTERMAC2> -j CONNMARK --set-mark 2
>
>
> $iptables -t mangle -N TONET
> $iptables -t mangle -F TONET
> $iptables -t mangle -A OUTPUT -o <DEVICE> -d ! <LOCAL_NET> -j TONET
>
> $iptables -t mangle -A TONET -m connmark ! --mark 0 -j CONNMARK --restore-mark
>
>
> The route/rule stuff was more or less what you've got. If you want to
> forward packets for other hosts, this need to go into PREROUTING of
> course.
>
>     Thomas
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> !DSPAM:49e992ac62591343511188!
>
> 

Re: still can't route using fwmark

[Thomas Jacob]

On Sat, Apr 18, 2009 at 11:12:36AM -0600, Lloyd Standish wrote:
> Thanks for the reply, Thomas.  I've got iptables 1.3.8 because I downloaded it (and the patch-o-matic stuff) quite a while back when it was the most current version.  I'm using it because my kernel is not the stock Etch kernel (it is in fact a Ubuntu kernel, necessary to get a module that Debian removed over licensing issues).  I recompiled my kernel and installed itables 1.3.8 out of desperation, since I have studied this load-balancing  stuff pretty thoroughly and it seems what I have ought to work.

> My kernel 2.6.20.3 does not have a CONFIG_IP_ROUTE_FWMARK configuration setting!  Has this been absorbed into another configuration setting?

Ok, I've never used 2.6.20, but 2.6.27 definitely has all the required features,
but the *_ROUTE_FWMARK seems indeed to be gone, advanced router and policy
routing should be enough though.

>How can I be sure fwmark is working?  Judging by my setup, which is very simple, packets and connections are being marked, but routing is not affected by fwmark.

You can check the routing cache using "ip route show cache", and see what routing
decisions were taken.

> I have the kernel config options IP_NF_CONNTRACK_SUPPORT and XFRM_SUB_POLICY both set OFF.  Might this be a problem?

XFRM_SUB_POLICY is IPsec related, and IP_NF_CONNTRACK_SUPPORT doesn't exist anymore
in 2.6.27, but you've got /proc/net/ip_conntrack so connection tracking support
is clearly there.

Maybe you could post your connmark related iptables rules as well.

Also you could try to remove those two routes in the default table, which I
gather from your description are routes for $gw0=$gw1=10.60.255.254 (btw, the
second will normally never be used anyway), I remember having problems before
when routes matched after the policy route tables in the main table.

Why are you using the same IP (10.60.255.254) on different links? Possibly you could
try different IPs for that. I've haven't used PPP links for policy routing so far,
could be that they behave differently in some significant way wrt. to all this,
compared to broadcast networks.

Re: still can't route using fwmark

[Lloyd Standish]

On Sat, 18 Apr 2009 12:48:09 -0600, Thomas Jacob <jacob@internet24.de> wrote:

>>How can I be sure fwmark is working?  Judging by my setup, which is very simple, packets and connections are being marked, but routing is not affected by fwmark.
>
> You can check the routing cache using "ip route show cache", and see what routing
> decisions were taken.

(Thanks), I see this:
debiandesk:/home/lloyd/data/loadbal# ip route show cache
72.249.38.123 from 10.60.27.199 dev ppp0
    cache  mtu 1500 advmss 1460 hoplimit 64
local 10.60.27.199 from 72.249.38.123 dev lo  src 10.60.27.199
    cache <local,src-direct>  iif ppp0
local 10.60.27.199 from 72.249.38.123 dev lo  src 10.60.27.199
    cache <local,src-direct>  iif ppp0
broadcast 192.168.1.255 dev eth0  src 192.168.1.1
    cache <local,brd>  mtu 1500 advmss 1460 hoplimit 64
209.40.204.55 from 10.60.27.199 dev ppp0
    cache  mtu 1500 advmss 1460 hoplimit 64
local 10.60.27.199 from 209.40.204.55 dev lo  src 10.60.27.199
    cache <local,src-direct>  iif ppp0
local 10.60.27.199 from 209.40.204.55 dev lo  src 10.60.27.199
    cache <local,src-direct>  iif ppp0

Should there be signs here of fwmark-related decisions?
 
> Maybe you could post your connmark related iptables rules as well.

I am logging the CONNMARK stuff now.  The logs show the connection marking taking place.  I just added a few comments.  Here are the rules, taken from my shell script:

# define CONNMARK1
iptables -t mangle -N CONNMARK1
iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
iptables -t mangle -A CONNMARK1 -j LOG --log-prefix 'iptables-mark1: ' --log-level info

# define CONNMARK1
iptables -t mangle -N CONNMARK2
iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
iptables -t mangle -A CONNMARK2 -j LOG --log-prefix 'iptables-mark2: ' --log-level info

# define RESTOREMARK
iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t mangle -A RESTOREMARK -j LOG --log-prefix 'restore-mark: ' --log-level info

# define SNAT1
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j LOG --log-prefix "SNAT $src0: " --log-level info
iptables -t nat -A SNAT1 -j SNAT --to-source $src0

# define SNAT2
iptables -t nat -N SNAT2
iptables -t nat -A SNAT2 -j LOG --log-prefix "SNAT $src1: " --log-level info
iptables -t nat -A SNAT2 -j SNAT --to-source $src1

# restore the fwmark on packets that belong to an existing connection
iptables -A PREROUTING -i eth0  -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK
# if there is a mark, can quit
iptables -A PREROUTING -t mangle -m mark ! --mark 0 -j RETURN

# mark all packets with fwmark 1
iptables -A PREROUTING -t mangle -j CONNMARK1

# mark every other packet with fwmark 2
iptables -A PREROUTING -t mangle -m statistic --mode nth --every 2 --packet 0 -j CONNMARK2

# fix source IPs to match interface IP
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2

# masquerade eth0
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE


>
> Also you could try to remove those two routes in the default table, which I
> gather from your description are routes for $gw0=$gw1=10.60.255.254 (btw, the
> second will normally never be used anyway), I remember having problems before
> when routes matched after the policy route tables in the main table.

Do you mean the routes in the rt_link1 and rt_link2 tables?  I saw in a couple of articles on load balancing the suggestion that only the default route is necessary in each of the user-defined tables.  Of course, the only reason I use these tables is to be able to route through a different interface.

> Why are you using the same IP (10.60.255.254) on different links? Possibly you could
> try different IPs for that. I've haven't used PPP links for policy routing so far,
> could be that they behave differently in some significant way wrt. to all this,
> compared to broadcast networks.

10.60.255.254 is the gateway for each of the links ppp0 and ppp1.  I cannot change that, and I don't see why the gateway should have to be different.

I ought to point out that I am connecting to my ISP using GPRS (ppp tunneled over GSM telephone connection).  The ISP assigns a private IP to me, and the gateway IP is also private.   The ISP is natting to a public IP, and that public IP changes constantly!  I assume this is part of some load-balancing that my ISP is doing!

--
Lloyd

> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> !DSPAM:49ea206b166987818312239!
>
> 

Re: still can't route using fwmark

[Thomas Jacob]

On Sat, Apr 18, 2009 at 01:34:42PM -0600, Lloyd Standish wrote:
> On Sat, 18 Apr 2009 12:48:09 -0600, Thomas Jacob <jacob@internet24.de> wrote:
> 
> >>How can I be sure fwmark is working?  Judging by my setup, which is very simple, packets and connections are being marked, but routing is not affected by fwmark.
> >
> > You can check the routing cache using "ip route show cache", and see what routing
> > decisions were taken.

<route cache output>

> Should there be signs here of fwmark-related decisions?

Don't think so, but you can see where packets are directed
for certain destinations

> > Maybe you could post your connmark related iptables rules as well.
> 
> I am logging the CONNMARK stuff now.  The logs show the connection marking taking place.  Here are the rules from my shell script, with a few comments I just now added.
> 
> # define CONNMARK1
> iptables -t mangle -N CONNMARK1
> iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
> iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
> iptables -t mangle -A CONNMARK1 -j LOG --log-prefix 'iptables-mark1: ' --log-level info
> 
> # define CONNMARK1
> iptables -t mangle -N CONNMARK2
> iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
> iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
> iptables -t mangle -A CONNMARK2 -j LOG --log-prefix 'iptables-mark2: ' --log-level info
> 
> # define RESTOREMARK
> iptables -t mangle -N RESTOREMARK
> iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
> iptables -t mangle -A RESTOREMARK -j LOG --log-prefix 'restore-mark: ' --log-level info
> 
> # define SNAT1
> iptables -t nat -N SNAT1
> iptables -t nat -A SNAT1 -j LOG --log-prefix "SNAT $src0: " --log-level info
> iptables -t nat -A SNAT1 -j SNAT --to-source $src0
> 
> # define SNAT2
> iptables -t nat -N SNAT2
> iptables -t nat -A SNAT2 -j LOG --log-prefix "SNAT $src1: " --log-level info
> iptables -t nat -A SNAT2 -j SNAT --to-source $src1
> 
> # restore the fwmark on packets that belong to an existing connection
> iptables -A PREROUTING -i eth0  -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK
> # if there is a mark, can quit
> iptables -A PREROUTING -t mangle -m mark ! --mark 0 -j RETURN
> 
> # mark all packets with fwmark 1 (A)
> iptables -A PREROUTING -t mangle -j CONNMARK1
> 
> # mark every other packet with fwmark 2 (B)
> iptables -A PREROUTING -t mangle -m statistic --mode nth --every 2 --packet 0 -j CONNMARK2
> 
> # fix source IPs to match interface IP
> iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
> iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
> 
> # masquerade eth0
> iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Are you forwarding packets via this box, or do you want to loadbalance
packets from the local machine? In the latter case the PREROUTING
stuff needs to go into INPUT/OUTPUT.

Otherwise the rest looks fine to me, maybe making sure that you only
do the initial mark assignments for NEW packets might be a good
idea, but that most likely isn't a real problem.

> > Also you could try to remove those two routes in the default table, which I
> > gather from your description are routes for $gw0=$gw1=10.60.255.254 (btw, the
> > second will normally never be used anyway), I remember having problems before
> > when routes matched after the policy route tables in the main table.
> 
> Do you mean the routes in the rt_link1 and rt_link2 tables?  

no, the ones in the "main" routing table (that you displayed using "ip route show [table main]).


>I saw in a couple of articles on load balancing the suggestion that only the default route is necessary in each of the user-defined tables.  Of course, the only reason I use these tables is to be able to route through a different interface.

Yes, if you have a way to find your gateway (which is sort of implicit
when both gateway are on the same ethernet link with mask /24 for instance), but
in your case you seem to need host routes to designate where your default
gateways are (the first route in rt_link1/2), and the two routes in main
are the same as the first route in rt_link1 and 2 combined, right?

But maybe one does not need gateways for ppp since there should be no one else
on that link anyway (as you suggested with your "default dev pppX" routes).

Don't know sorry.

> > Why are you using the same IP (10.60.255.254) on different links? Possibly you could
> > try different IPs for that. I've haven't used PPP links for policy routing so far,
> > could be that they behave differently in some significant way wrt. to all this,
> > compared to broadcast networks.
> 
> 10.60.255.254 is the gateway for each of the links ppp0 and ppp1.  I cannot change that, and I don't see why the gateway should have to be different.

Well, if you have a route that says use gateway 10.60.255.254, then the kernel needs
to determine where that is by looking at the rest of the route tables, and if
you have several destinations, this could maybe be confusing...

> I ought to point out that I am connecting to my ISP using GPRS (ppp tunneled over GSM telephone connection).  The ISP assigns a private IP to me, and the gateway IP is also private.   The ISP is natting to a public IP, and that public IP changes constantly!  I assume this is part of some load-balancing that my ISP is doing!

There aren't enough IPv4 addresses around to give each mobile internet
device a real one, also changing IPs like that fcks with lot of
applications that mobile service providers really would not
like you to use (Skype, BT, etc ;)

I am running out of ideas, sorry, maybe someone else on the list
can chime in, I've only ever done policy routing with public IPs and
broadcast networks, and there it always worked without problems
on stock Debian Etch, CentOS 5.X or custom built kernels.

Re: still can't route using fwmark

[Lloyd Standish]

On Sat, 18 Apr 2009 14:58:02 -0600, Thomas Jacob <jacob@internet24.de> wrote:


> Are you forwarding packets via this box, or do you want to loadbalance
> packets from the local machine? In the latter case the PREROUTING
> stuff needs to go into INPUT/OUTPUT.

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.

Might this point you made be my problem?  I will try removing the network 192.168.1.0 and just use lo, to see what happens.

>> > Also you could try to remove those two routes in the default table, which I
>> > gather from your description are routes for $gw0=$gw1=10.60.255.254 (btw, the
>> > second will normally never be used anyway), I remember having problems before
>> > when routes matched after the policy route tables in the main table.
>>
>> Do you mean the routes in the rt_link1 and rt_link2 tables?
>
> no, the ones in the "main" routing table (that you displayed using "ip route show [table main]).
>

When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.


> Yes, if you have a way to find your gateway (which is sort of implicit
> when both gateway are on the same ethernet link with mask /24 for instance), but
> in your case you seem to need host routes to designate where your default
> gateways are (the first route in rt_link1/2), 

>and the two routes in main
> are the same as the first route in rt_link1 and 2 combined, right?
YES

The gateways are assigned at the time the ppp0 and ppp1 links are established.  I get it by:
debiandesk:/home/lloyd/data/loadbal# ip route show dev ppp1
10.60.255.254  proto kernel  scope link  src 10.60.9.178

So, I can use this shell script line to get the gateway for ppp1 into variable gw1:
gw1=`ip route show dev ppp1 | head -n1 | cut --delimiter=" " --fields=1`

The only experience I have with iptables is simple firewall stuff for my Internet-connected server.  My grasp of routing is weak, and this trouble is good experience in an area I would like to become expert in.  

>
> But maybe one does not need gateways for ppp since there should be no one else
> on that link anyway (as you suggested with your "default dev pppX" routes).

I don't think I understand your comment.   I need a gateway IP to forward Internet queries to... 
>
>
> I am running out of ideas, sorry, maybe someone else on the list
> can chime in, I've only ever done policy routing with public IPs and
> broadcast networks, and there it always worked without problems
> on stock Debian Etch, CentOS 5.X or custom built kernels.

I really appreciate your advice.  I'm a networking novice so my ideas are not rooted in experience, but I think it should not matter that the gateway IP is a private IP.

Re: still can't route using fwmark

[Lloyd Standish]

On Sat, 18 Apr 2009 14:58:02 -0600, Thomas Jacob <jacob@internet24.de> wrote:

> Are you forwarding packets via this box, or do you want to loadbalance
> packets from the local machine? In the latter case the PREROUTING
> stuff needs to go into INPUT/OUTPUT.

Thomas, I would like this to load balance all traffic going out of the local machine (which is gateway for the home LAN), including locally-generated packets.  I was under the impression that PREROUTING happens before everything else and affects all packets, both locally-generated ones and forwarded ones.  My conntrack tables show that locally-generated packets are getting the fwmark.   Could you explain why PREROUTING isn't the right place for the marking rules?

Re: still can't route using fwmark

[Thomas Jacob]

On Sat, Apr 18, 2009 at 03:49:29PM -0600, Lloyd Standish wrote:
> On Sat, 18 Apr 2009 14:58:02 -0600, Thomas Jacob <jacob@internet24.de> wrote:
> 
> 
> > Are you forwarding packets via this box, or do you want to loadbalance
> > packets from the local machine? In the latter case the PREROUTING
> > stuff needs to go into INPUT/OUTPUT.
> 
> Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.

Then your current setup in PREROUTING is what you want to go for, just keep
in mind that this does not give you load balancing for connections originating
from your router box, just the ones from your LAN.

C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png

I'm not sure why you need NAT on your eth0 though then, what are you
trying to achieve with this? But that should not be the cause
of the load balancing failure.

> When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

That's right, but if all your /proc/net/ip_conntrack entries contain mark values
then there really must be something wrong with the fw mark <-> route interaction.

My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for
instance, then maybe more people have a comparable setup to look at.

> The only experience I have with iptables is simple firewall stuff for my Internet-connected server.  My grasp of routing is weak, and this trouble is good experience in an area I would like to become expert in.  

Do you know about LARTC? The best way to get started IMO: http://lartc.org/
> >
> > But maybe one does not need gateways for ppp since there should be no one else
> > on that link anyway (as you suggested with your "default dev pppX" routes).
> 
> I don't think I understand your comment.   I need a gateway IP to forward Internet queries to... 

Your router needs to know where to send packets it doesn't have routes for, sure. In an ethernet
you need to specify a certain nexthop gateway machine, because there are more
than two nodes on the local link, but I am not sure that is actually necessary for
ppp connections, even though it is usually done that way (as in your setup) since
all packets sent do the ppp link are definitely intended for the other node
on the link.

> I really appreciate your advice.  I'm a networking novice so my ideas are not rooted in experience, but I think it should not matter that the gateway IP is a private IP.

Using private IPs isn't a problem, having several routes to the same
destination might possibly be (even if it shouldn't be as they are in different
routing tables). It's most certainly a problem in a single routing table, as only
the first route will ever get used though.

But as you lose connectivity when you remove the two routes
from the "main" table, you definitely have another problem regardless
of this.

   Thomas

Re: still can't route using fwmark

[Lloyd Standish]

On Sun, 19 Apr 2009 03:00:17 -0600, Thomas Jacob <jacob@internet24.de> wrote:

>>Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.
>
> Then your current setup in PREROUTING is what you want to go for, just keep
> in mind that this does not give you load balancing for connections originating
> from your router box, just the ones from your LAN.

I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)

However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

>
> C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png
>
> I'm not sure why you need NAT on your eth0 though then, what are you
> trying to achieve with this? But that should not be the cause
> of the load balancing failure.

I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.

The lines:
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

>
>> When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.
>
> That's right, but if all your /proc/net/ip_conntrack entries contain mark values
> then there really must be something wrong with the fw mark <-> route interaction.
>
> My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for
> instance, then maybe more people have a comparable setup to look at.
That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.

My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)
>
> Do you know about LARTC? The best way to get started IMO: http://lartc.org/

Yes, I downloaded the tutorial a couple of days ago, thanks!

--
Lloyd

Re: still can't route using fwmark

[Javier Gálvez Guerrero]

Hi,

I've been trying something similar to Lloyd's configuration for a long
time with no success. I want to reach a host (server) from my laptop
with two WiFi interfaces attached to different subnetworks/routers;
the server is in the public network, both routers have public IPs (the
same domain than the server) while setting up two different private
networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface
in my laptop are attached to each one of them.

The problem is that I can't manage to dynamically configure which
interface to be chosen to redirect my traffic. In my case, I want the
packets with destination port 8554 to be routed to one or other
interface as I want (so load one network or the other), so I tried to
change MARK iptables rules accordingly with no success, just getting
what Lloyd had: packets are routed through the default route in the
main ip routing table, while if I have not this route configured I
lose network access, despite having the corresponding tables and rules
properly configured. What I ended figuring out is that using the TOS
target it worked properly, but this is not what I'd like to preserve
as TOS changes the type of service parameter and could affect network
management beyond my laptop; I need to use MARK to manage my outgoing
and incoming traffic.

This is the script I'm currently using (stripped):

    ip route flush table 1
    ip route flush table 2
    ip rule del prio 1
    ip rule del prio 1
    iptables -F OUTPUT -t mangle
    iptables -F POSTROUTING -t nat
    iptables -F PREROUTING -t nat

    ip route add default via 192.168.0.1 dev ra0 ---> I need a
configure a route in the main table! Why?
    ip route add table 1 192.168.0.0/24 dev ra0
    ip route add table 1 default via 192.168.0.1 dev ra0
    ip route add table 2 192.168.1.0/24 dev ra1
    ip route add table 2 default via 192.168.1.1 dev ra1

    ip rule add from all fwmark 1 table 1 prio 1
    ip rule add from all fwmark 2 table 2 prio 1

    iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
--to-source 192.168.0.2
    iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1
    iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1


So, when I want to change the marking rules:

	iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
--to-source 192.168.1.2
	iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2
	iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2


Anyway, I'm not able to get this working, but only when the selected
IP matches the default IP in the main routing table, so... it doesn't
work. Changing MARKing rules with TOS works.

So, I can't dynamically change the used interface for the selected packets.


I opened a bug (#589) in the Netfilter bugzilla system and Jan
Engelhardt has kindly tried to help me but I haven't been able to
solve it yet.

By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.


Any help would be much appreciated, as I'm not sure if this is really
a bug or just a problem with iptables rules understanding or misuse.


Regards,
Javi

2009/4/20 Lloyd Standish <lloyd@crnatural.net>
>
> On Sun, 19 Apr 2009 03:00:17 -0600, Thomas Jacob <jacob@internet24.de> wrote:
>
> >>Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.
> >
> > Then your current setup in PREROUTING is what you want to go for, just keep
> > in mind that this does not give you load balancing for connections originating
> > from your router box, just the ones from your LAN.
>
> I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)
>
> However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)
>
> >
> > C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png
> >
> > I'm not sure why you need NAT on your eth0 though then, what are you
> > trying to achieve with this? But that should not be the cause
> > of the load balancing failure.
>
> I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.
>
> The lines:
> iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
> iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
> should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).
>
> >
> >> When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.
> >
> > That's right, but if all your /proc/net/ip_conntrack entries contain mark values
> > then there really must be something wrong with the fw mark <-> route interaction.
> >
> > My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for
> > instance, then maybe more people have a comparable setup to look at.
> That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.
>
> My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)
> >
> > Do you know about LARTC? The best way to get started IMO: http://lartc.org/
>
> Yes, I downloaded the tutorial a couple of days ago, thanks!
>
> --
> Lloyd
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: still can't route using fwmark

[Thomas Jacob]

On Sun, 2009-04-19 at 23:56 -0600, Lloyd Standish wrote:

> I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, 
> since all my testing up until now has been with connections (to the Internet) originating on the router box.  
> (I haven't even tried connecting from the LAN.)
>
> However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, 
> put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables? 
>  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

Take a look at the packet traversal graph, outgoing packets from local
processes do not pass thru PREROUTING, but incoming packets do, that's
where your markings come from. But in your case, I'm sure you want to
select a link when the outgoing connection is first established and then
stay with that link. Selecting a different link with the second packet
doesn't work with NATed connections, right?

So you if you want to load balance local packets correctly as well,
you need to put some rules into OUTPUT. Possibly that's really your
basic problem here, but I don't have time to think about that at the
moment.

> I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, 
> but I mistakenly specified "-o eth0" instead of Internet connected interface.

OK

> The lines:
> iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
> iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
> should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of 
> outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

You definitely need the masquerading, else your local LAN IPs will show
up at your providers door, and they won't know what do with them. 

> That's good advice, although I can't use kernel 2.6.27 I'm afraid.  
> At some point after 2.6.21 the code for a USB serial driver changed.  
> I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  

Figured something like ;)

> (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  
> GPRS is my only Internet option in my remote area of Costa Rica.

Nice one. The address in your whois record is kinda cool as well ...

> My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need. 
>  (It is a drag to be tied to an old kernel version due to hardware dependency.)

I don't really believe that 2.6.20.3 misses something basic like that,
which is nevertheless contained in an older kernel. Unless maybe the
patched Debian kernel has it (Debian's 2.6.18 contains lots of
extra security and bug patches from later kernels). And the iptables
user space is most likely irrelevant anyway, if you can get the rules
you want to load already.

Re: still can't route using fwmark

[Thomas Jacob]

I haven't read thru all this, and you also haven't posted
your TOS variant, but a few observations:

You should use CONNMARK like Lloyd does, MARK does not
affect the connection, just a single packet, and you have to
reconstruct your MARK values for each packet.

Your TOS rules most likely change the packet TOS field, and that
is sort of "sticky" if all the intermediate hosts play ball,
according to the standards, i.e. does do something like CONNMARK
does.

Also be aware of the different paths thru the various
chains taken by packets from/to the local host and
by forwarded packets.

A small ascii diagram for your setup would be very helpful.

On Mon, 2009-04-20 at 10:48 +0200, Javier Gálvez Guerrero wrote:
> Hi,
> 
> I've been trying something similar to Lloyd's configuration for a long
> time with no success. I want to reach a host (server) from my laptop
> with two WiFi interfaces attached to different subnetworks/routers;
> the server is in the public network, both routers have public IPs (the
> same domain than the server) while setting up two different private
> networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface
> in my laptop are attached to each one of them.
> 
> The problem is that I can't manage to dynamically configure which
> interface to be chosen to redirect my traffic. In my case, I want the
> packets with destination port 8554 to be routed to one or other
> interface as I want (so load one network or the other), so I tried to
> change MARK iptables rules accordingly with no success, just getting
> what Lloyd had: packets are routed through the default route in the
> main ip routing table, while if I have not this route configured I
> lose network access, despite having the corresponding tables and rules
> properly configured. What I ended figuring out is that using the TOS
> target it worked properly, but this is not what I'd like to preserve
> as TOS changes the type of service parameter and could affect network
> management beyond my laptop; I need to use MARK to manage my outgoing
> and incoming traffic.
> 
> This is the script I'm currently using (stripped):
> 
>     ip route flush table 1
>     ip route flush table 2
>     ip rule del prio 1
>     ip rule del prio 1
>     iptables -F OUTPUT -t mangle
>     iptables -F POSTROUTING -t nat
>     iptables -F PREROUTING -t nat
> 
>     ip route add default via 192.168.0.1 dev ra0 ---> I need a
> configure a route in the main table! Why?
>     ip route add table 1 192.168.0.0/24 dev ra0
>     ip route add table 1 default via 192.168.0.1 dev ra0
>     ip route add table 2 192.168.1.0/24 dev ra1
>     ip route add table 2 default via 192.168.1.1 dev ra1
> 
>     ip rule add from all fwmark 1 table 1 prio 1
>     ip rule add from all fwmark 2 table 2 prio 1
> 
>     iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
> --to-source 192.168.0.2
>     iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1
>     iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1
> 
> 
> So, when I want to change the marking rules:
> 
> 	iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
> --to-source 192.168.1.2
> 	iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2
> 	iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2
> 
> 
> Anyway, I'm not able to get this working, but only when the selected
> IP matches the default IP in the main routing table, so... it doesn't
> work. Changing MARKing rules with TOS works.
> 
> So, I can't dynamically change the used interface for the selected packets.
> 
> 
> I opened a bug (#589) in the Netfilter bugzilla system and Jan
> Engelhardt has kindly tried to help me but I haven't been able to
> solve it yet.
> 
> By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.
> 
> 
> Any help would be much appreciated, as I'm not sure if this is really
> a bug or just a problem with iptables rules understanding or misuse.
> 
> 
> Regards,
> Javi
> 
> 2009/4/20 Lloyd Standish <lloyd@crnatural.net>
> >
> > On Sun, 19 Apr 2009 03:00:17 -0600, Thomas Jacob <jacob@internet24.de> wrote:
> >
> > >>Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.
> > >
> > > Then your current setup in PREROUTING is what you want to go for, just keep
> > > in mind that this does not give you load balancing for connections originating
> > > from your router box, just the ones from your LAN.
> >
> > I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)
> >
> > However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)
> >
> > >
> > > C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png
> > >
> > > I'm not sure why you need NAT on your eth0 though then, what are you
> > > trying to achieve with this? But that should not be the cause
> > > of the load balancing failure.
> >
> > I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.
> >
> > The lines:
> > iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
> > iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
> > should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).
> >
> > >
> > >> When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.
> > >
> > > That's right, but if all your /proc/net/ip_conntrack entries contain mark values
> > > then there really must be something wrong with the fw mark <-> route interaction.
> > >
> > > My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for
> > > instance, then maybe more people have a comparable setup to look at.
> > That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.
> >
> > My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)
> > >
> > > Do you know about LARTC? The best way to get started IMO: http://lartc.org/
> >
> > Yes, I downloaded the tutorial a couple of days ago, thanks!
> >
> > --
> > Lloyd
> > --
> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: still can't route using fwmark

[Brian Austin - Standard Universal]

Thomas Jacob wrote:
>
> So you if you want to load balance local packets correctly as well,
> you need to put some rules into OUTPUT. Possibly that's really your
> basic problem here, but I don't have time to think about that at the
> moment.
>   
its spooky, that was *exactly* my problem when I visited load balance 
via conntracking.

you need to consider each path and rule for each.. and use the syslog 
and conntrack table
to diagnose what is going on.

b
 

Re: still can't route using fwmark

[Javier Gálvez Guerrero]

Thanks for your answer, Thomas.

Once I explained to Jan what I had to implement, he pointed me to the
item 5 in http://dev.medozas.de/NF-Cookbook.txt, so I've also used
CONNTRACK with no success:

"I'll try to explain what I need to implement. I'm developing a daemon
application that manages WiFi handovers between two private networks
(192.168.0.0/24 and 192.168.1.0/24) using two different WiFi NICs.
When the link RSSI becomes low enough, the application triggers the
handover in order to keep a seamless video streaming service, so first
I must attach to the new network with the inactive NIC, ask for the
media content through it (the video session configuration is done
through the server port 8554) and, once I'm receiving the content
through the new link (the media content is actually sent from and to
ports differents from 8554), detach de old one.

In order to do this in a seamless way I need to play with ip route, ip
rule and iptables. Marking traffic with iptables is not enough as it
is done in POSTROUTING, so the kernel previously decides which IP the
packets will have according to the default route in the main ip
routing table. Using MARK I send the packet through the interface I
want in each specific moment, while with SNAT I change the source IP
to the one attached to the used interface (thus changing the IP given
by the default routing table in case the new interface is not the one
configured as the default one). The received (answer) packets should
come through the same interface with the corresponding IP but they
just follow the route configured in the default ip routing table (then
not matching the connection established according to my custom setup),
so I need to setup more MARKing and ip rules so the local application
receives and delivers packets accordingly. This is where MARK fails
and TOS success, but I don't want to mess TOS network configuration
and use a more proper packet mangling with MARK as the only device
that should care about the handover is the client device."



Anyway, it didn't work: as soon as I change the default route in the
main table the old connection is broken. Instead, connections should
be established and maintained through the links pointed in the
respective tables (not the main one), according with the marking rules
(that's what I need, to have two connections established with
different interfaces and to different networks and dynamically chose
one of both to send and receive the video stream).

Time for ASCII art (quite "simple"):

                               ,''''''''''''''`.
                               |  Server |
                               L_______J  147.83.47.178
                                 _/__
                              ,-'       `-.
                            ,'               `.
                           |                   '.
                           |     Network  |
                          /\                   /
                         /  \                /.
                       ,'    `._        _,'  `.
                      /         `----'          `.
                 |---'--------Y           ....:.............
                 |              |          |                   |
                 |Router A|           |    Router B |
                 '------------            `.................'
         192.168.0.0/24             192.168.1.0/24


                                       ,'''''''''''''''''''|
                         ra0        |               |  ra1
                                      |     Client |
                  192.168.0.2  L________|  192.168.1.2



The rules with TOS are exactly the same just changing the marking
parameters by the TOS ones:

ip rule add from all tos 0x10 table 1 prio 1
ip rule add from all tos 0x04 table 2 prio 1

iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source
192.168.0.2
iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x10
iptables -A PREROUTING -t mangle -s 147.83.47.178 -j TOS --set-tos 0x10

iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
--to-source 192.168.1.2
iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x04
iptables -R PREROUTING 1 -t mangle -s 147.83.47.178 -j TOS --set-tos 0x04


This works but, as said before, I don't what this TOS stuff to be used.


So... any idea why the previous "item 5" didn't work for my
environment? What should I try?


Thank you for your help,
Javi


El día 20 de abril de 2009 13:44, Thomas Jacob <jacob@internet24.de> escribió:
> I haven't read thru all this, and you also haven't posted
> your TOS variant, but a few observations:
>
> You should use CONNMARK like Lloyd does, MARK does not
> affect the connection, just a single packet, and you have to
> reconstruct your MARK values for each packet.
>
> Your TOS rules most likely change the packet TOS field, and that
> is sort of "sticky" if all the intermediate hosts play ball,
> according to the standards, i.e. does do something like CONNMARK
> does.
>
> Also be aware of the different paths thru the various
> chains taken by packets from/to the local host and
> by forwarded packets.
>
> A small ascii diagram for your setup would be very helpful.
>
> On Mon, 2009-04-20 at 10:48 +0200, Javier Gálvez Guerrero wrote:
>> Hi,
>>
>> I've been trying something similar to Lloyd's configuration for a long
>> time with no success. I want to reach a host (server) from my laptop
>> with two WiFi interfaces attached to different subnetworks/routers;
>> the server is in the public network, both routers have public IPs (the
>> same domain than the server) while setting up two different private
>> networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface
>> in my laptop are attached to each one of them.
>>
>> The problem is that I can't manage to dynamically configure which
>> interface to be chosen to redirect my traffic. In my case, I want the
>> packets with destination port 8554 to be routed to one or other
>> interface as I want (so load one network or the other), so I tried to
>> change MARK iptables rules accordingly with no success, just getting
>> what Lloyd had: packets are routed through the default route in the
>> main ip routing table, while if I have not this route configured I
>> lose network access, despite having the corresponding tables and rules
>> properly configured. What I ended figuring out is that using the TOS
>> target it worked properly, but this is not what I'd like to preserve
>> as TOS changes the type of service parameter and could affect network
>> management beyond my laptop; I need to use MARK to manage my outgoing
>> and incoming traffic.
>>
>> This is the script I'm currently using (stripped):
>>
>>     ip route flush table 1
>>     ip route flush table 2
>>     ip rule del prio 1
>>     ip rule del prio 1
>>     iptables -F OUTPUT -t mangle
>>     iptables -F POSTROUTING -t nat
>>     iptables -F PREROUTING -t nat
>>
>>     ip route add default via 192.168.0.1 dev ra0 ---> I need a
>> configure a route in the main table! Why?
>>     ip route add table 1 192.168.0.0/24 dev ra0
>>     ip route add table 1 default via 192.168.0.1 dev ra0
>>     ip route add table 2 192.168.1.0/24 dev ra1
>>     ip route add table 2 default via 192.168.1.1 dev ra1
>>
>>     ip rule add from all fwmark 1 table 1 prio 1
>>     ip rule add from all fwmark 2 table 2 prio 1
>>
>>     iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
>> --to-source 192.168.0.2
>>     iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1
>>     iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1
>>
>>
>> So, when I want to change the marking rules:
>>
>>       iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
>> --to-source 192.168.1.2
>>       iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2
>>       iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2
>>
>>
>> Anyway, I'm not able to get this working, but only when the selected
>> IP matches the default IP in the main routing table, so... it doesn't
>> work. Changing MARKing rules with TOS works.
>>
>> So, I can't dynamically change the used interface for the selected packets.
>>
>>
>> I opened a bug (#589) in the Netfilter bugzilla system and Jan
>> Engelhardt has kindly tried to help me but I haven't been able to
>> solve it yet.
>>
>> By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.
>>
>>
>> Any help would be much appreciated, as I'm not sure if this is really
>> a bug or just a problem with iptables rules understanding or misuse.
>>
>>
>> Regards,
>> Javi
>>
>> 2009/4/20 Lloyd Standish <lloyd@crnatural.net>
>> >
>> > On Sun, 19 Apr 2009 03:00:17 -0600, Thomas Jacob <jacob@internet24.de> wrote:
>> >
>> > >>Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.
>> > >
>> > > Then your current setup in PREROUTING is what you want to go for, just keep
>> > > in mind that this does not give you load balancing for connections originating
>> > > from your router box, just the ones from your LAN.
>> >
>> > I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)
>> >
>> > However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)
>> >
>> > >
>> > > C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png
>> > >
>> > > I'm not sure why you need NAT on your eth0 though then, what are you
>> > > trying to achieve with this? But that should not be the cause
>> > > of the load balancing failure.
>> >
>> > I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.
>> >
>> > The lines:
>> > iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
>> > iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
>> > should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).
>> >
>> > >
>> > >> When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.
>> > >
>> > > That's right, but if all your /proc/net/ip_conntrack entries contain mark values
>> > > then there really must be something wrong with the fw mark <-> route interaction.
>> > >
>> > > My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for
>> > > instance, then maybe more people have a comparable setup to look at.
>> > That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.
>> >
>> > My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)
>> > >
>> > > Do you know about LARTC? The best way to get started IMO: http://lartc.org/
>> >
>> > Yes, I downloaded the tutorial a couple of days ago, thanks!
>> >
>> > --
>> > Lloyd
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: still can't route using fwmark

[Thomas Jacob]

Nice diagram ;)

Well, it is as thought, your TOS rules set TOS fields in the packets
and those are usually bounced back to you from the reply direction,
so effectively you are using something like CONNMARK already.

Jan's item 5 description is definitely a good starting point.

Maybe you could try this, I am assuming that you really
are talking only about connections originating from client
here:


    iptables -F OUTPUT -t mangle
    iptables -F POSTROUTING -t nat
    iptables -F PREROUTING -t nat


    iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
--to-source 192.168.0.2

    iptables -t mangle -A INPUT   -j CONNMARK --restore-mark

    iptables -t mangle -A OUTPUT -m state --state NEW \
        -p tcp --dport 8554 -d 147.83.47.178 -j CONNMARK --set-mark 1
    iptables -t mangle -A OUTPUT   -j CONNMARK --restore-mark


If you want to load balance connections from the local net you need
to use PREROUTING like Jan does. See the chain traversal diagrams I
posted in the reply to Lloyd, understanding which packets pass thru
which chains is really important for stuff like this.

On Mon, 2009-04-20 at 15:08 +0200, Javier Gálvez Guerrero wrote:
> Thanks for your answer, Thomas.
> 
> Once I explained to Jan what I had to implement, he pointed me to the
> item 5 in http://dev.medozas.de/NF-Cookbook.txt, so I've also used
> CONNTRACK with no success:
> 
> "I'll try to explain what I need to implement. I'm developing a daemon
> application that manages WiFi handovers between two private networks
> (192.168.0.0/24 and 192.168.1.0/24) using two different WiFi NICs.
> When the link RSSI becomes low enough, the application triggers the
> handover in order to keep a seamless video streaming service, so first
> I must attach to the new network with the inactive NIC, ask for the
> media content through it (the video session configuration is done
> through the server port 8554) and, once I'm receiving the content
> through the new link (the media content is actually sent from and to
> ports differents from 8554), detach de old one.
> 
> In order to do this in a seamless way I need to play with ip route, ip
> rule and iptables. Marking traffic with iptables is not enough as it
> is done in POSTROUTING, so the kernel previously decides which IP the
> packets will have according to the default route in the main ip
> routing table. Using MARK I send the packet through the interface I
> want in each specific moment, while with SNAT I change the source IP
> to the one attached to the used interface (thus changing the IP given
> by the default routing table in case the new interface is not the one
> configured as the default one). The received (answer) packets should
> come through the same interface with the corresponding IP but they
> just follow the route configured in the default ip routing table (then
> not matching the connection established according to my custom setup),
> so I need to setup more MARKing and ip rules so the local application
> receives and delivers packets accordingly. This is where MARK fails
> and TOS success, but I don't want to mess TOS network configuration
> and use a more proper packet mangling with MARK as the only device
> that should care about the handover is the client device."
> 
> 
> 
> Anyway, it didn't work: as soon as I change the default route in the
> main table the old connection is broken. Instead, connections should
> be established and maintained through the links pointed in the
> respective tables (not the main one), according with the marking rules
> (that's what I need, to have two connections established with
> different interfaces and to different networks and dynamically chose
> one of both to send and receive the video stream).
> 
> Time for ASCII art (quite "simple"):
> 
>                                ,''''''''''''''`.
>                                |  Server |
>                                L_______J  147.83.47.178
>                                  _/__
>                               ,-'       `-.
>                             ,'               `.
>                            |                   '.
>                            |     Network  |
>                           /\                   /
>                          /  \                /.
>                        ,'    `._        _,'  `.
>                       /         `----'          `.
>                  |---'--------Y           ....:.............
>                  |              |          |                   |
>                  |Router A|           |    Router B |
>                  '------------            `.................'
>          192.168.0.0/24             192.168.1.0/24
> 
> 
>                                        ,'''''''''''''''''''|
>                          ra0        |               |  ra1
>                                       |     Client |
>                   192.168.0.2  L________|  192.168.1.2
> 
> 
> 
> The rules with TOS are exactly the same just changing the marking
> parameters by the TOS ones:
> 
> ip rule add from all tos 0x10 table 1 prio 1
> ip rule add from all tos 0x04 table 2 prio 1
> 
> iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source
> 192.168.0.2
> iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x10
> iptables -A PREROUTING -t mangle -s 147.83.47.178 -j TOS --set-tos 0x10
> 
> iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
> --to-source 192.168.1.2
> iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x04
> iptables -R PREROUTING 1 -t mangle -s 147.83.47.178 -j TOS --set-tos 0x04
> 
> 
> This works but, as said before, I don't what this TOS stuff to be used.
> 
> 
> So... any idea why the previous "item 5" didn't work for my
> environment? What should I try?
> 
> 
> Thank you for your help,
> Javi
> 
> 
> El día 20 de abril de 2009 13:44, Thomas Jacob <jacob@internet24.de> escribió:
> > I haven't read thru all this, and you also haven't posted
> > your TOS variant, but a few observations:
> >
> > You should use CONNMARK like Lloyd does, MARK does not
> > affect the connection, just a single packet, and you have to
> > reconstruct your MARK values for each packet.
> >
> > Your TOS rules most likely change the packet TOS field, and that
> > is sort of "sticky" if all the intermediate hosts play ball,
> > according to the standards, i.e. does do something like CONNMARK
> > does.
> >
> > Also be aware of the different paths thru the various
> > chains taken by packets from/to the local host and
> > by forwarded packets.
> >
> > A small ascii diagram for your setup would be very helpful.
> >
> > On Mon, 2009-04-20 at 10:48 +0200, Javier Gálvez Guerrero wrote:
> >> Hi,
> >>
> >> I've been trying something similar to Lloyd's configuration for a long
> >> time with no success. I want to reach a host (server) from my laptop
> >> with two WiFi interfaces attached to different subnetworks/routers;
> >> the server is in the public network, both routers have public IPs (the
> >> same domain than the server) while setting up two different private
> >> networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface
> >> in my laptop are attached to each one of them.
> >>
> >> The problem is that I can't manage to dynamically configure which
> >> interface to be chosen to redirect my traffic. In my case, I want the
> >> packets with destination port 8554 to be routed to one or other
> >> interface as I want (so load one network or the other), so I tried to
> >> change MARK iptables rules accordingly with no success, just getting
> >> what Lloyd had: packets are routed through the default route in the
> >> main ip routing table, while if I have not this route configured I
> >> lose network access, despite having the corresponding tables and rules
> >> properly configured. What I ended figuring out is that using the TOS
> >> target it worked properly, but this is not what I'd like to preserve
> >> as TOS changes the type of service parameter and could affect network
> >> management beyond my laptop; I need to use MARK to manage my outgoing
> >> and incoming traffic.
> >>
> >> This is the script I'm currently using (stripped):
> >>
> >>     ip route flush table 1
> >>     ip route flush table 2
> >>     ip rule del prio 1
> >>     ip rule del prio 1
> >>     iptables -F OUTPUT -t mangle
> >>     iptables -F POSTROUTING -t nat
> >>     iptables -F PREROUTING -t nat
> >>
> >>     ip route add default via 192.168.0.1 dev ra0 ---> I need a
> >> configure a route in the main table! Why?
> >>     ip route add table 1 192.168.0.0/24 dev ra0
> >>     ip route add table 1 default via 192.168.0.1 dev ra0
> >>     ip route add table 2 192.168.1.0/24 dev ra1
> >>     ip route add table 2 default via 192.168.1.1 dev ra1
> >>
> >>     ip rule add from all fwmark 1 table 1 prio 1
> >>     ip rule add from all fwmark 2 table 2 prio 1
> >>
> >>     iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
> >> --to-source 192.168.0.2
> >>     iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1
> >>     iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1
> >>
> >>
> >> So, when I want to change the marking rules:
> >>
> >>       iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
> >> --to-source 192.168.1.2
> >>       iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2
> >>       iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2
> >>
> >>
> >> Anyway, I'm not able to get this working, but only when the selected
> >> IP matches the default IP in the main routing table, so... it doesn't
> >> work. Changing MARKing rules with TOS works.
> >>
> >> So, I can't dynamically change the used interface for the selected packets.
> >>
> >>
> >> I opened a bug (#589) in the Netfilter bugzilla system and Jan
> >> Engelhardt has kindly tried to help me but I haven't been able to
> >> solve it yet.
> >>
> >> By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.
> >>
> >>
> >> Any help would be much appreciated, as I'm not sure if this is really
> >> a bug or just a problem with iptables rules understanding or misuse.
> >>
> >>
> >> Regards,
> >> Javi
> >>
> >> 2009/4/20 Lloyd Standish <lloyd@crnatural.net>
> >> >
> >> > On Sun, 19 Apr 2009 03:00:17 -0600, Thomas Jacob <jacob@internet24.de> wrote:
> >> >
> >> > >>Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.
> >> > >
> >> > > Then your current setup in PREROUTING is what you want to go for, just keep
> >> > > in mind that this does not give you load balancing for connections originating
> >> > > from your router box, just the ones from your LAN.
> >> >
> >> > I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)
> >> >
> >> > However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)
> >> >
> >> > >
> >> > > C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png
> >> > >
> >> > > I'm not sure why you need NAT on your eth0 though then, what are you
> >> > > trying to achieve with this? But that should not be the cause
> >> > > of the load balancing failure.
> >> >
> >> > I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.
> >> >
> >> > The lines:
> >> > iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
> >> > iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
> >> > should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).
> >> >
> >> > >
> >> > >> When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.
> >> > >
> >> > > That's right, but if all your /proc/net/ip_conntrack entries contain mark values
> >> > > then there really must be something wrong with the fw mark <-> route interaction.
> >> > >
> >> > > My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for
> >> > > instance, then maybe more people have a comparable setup to look at.
> >> > That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.
> >> >
> >> > My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)
> >> > >
> >> > > Do you know about LARTC? The best way to get started IMO: http://lartc.org/
> >> >
> >> > Yes, I downloaded the tutorial a couple of days ago, thanks!
> >> >
> >> > --
> >> > Lloyd
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >

Re: still can't route using fwmark

[Javier Gálvez Guerrero]

Thanks for your suggestions.

As said before, customizing Jan's proposal didn't work:

#!/bin/sh

# Adapted from http://dev.medozas.de/NF-Cookbook.txt

-t mangle -N preferra0
-t mangle -A preferra0 -j CONNMARK --set-mark 10

-t mangle -N preferra1
-t mangle -A preferra1 -j CONNMARK --set-mark 11

# For connections coming from the Internet
-t mangle -N prefin
-t mangle -A prefin -i ra0 -g preferra0
-t mangle -A prefin -i ra1 -g preferra1

# For connections initiated from the LAN
-t mangle -N prefout
-t mangle -A prefout -o ra0 -g preferra0
-t mangle -A prefout -o ra1 -g preferra1

-t mangle -A PREROUTING -m connmark --mark 0 -m conntrack --ctstate
NEW -j prefin
-t mangle -A PREROUTING -j CONNMARK --restore-mark

-t mangle -A FORWARD -m connmark --mark 0 -m conntrack --ctstate NEW -j prefout
-t mangle -A OUTPUT -m connmark --mark 0 -m conntrack --ctstate NEW -j prefout

ip rule add fwmark 10 table 10
ip rule add fwmark 11 table 11
ip route add default via 192.168.0.1 table 10
ip route add default via 192.168.1.1 table 11


Using your rules I get this setup:

$ sudo iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
CONNMARK   tcp  --  anywhere             anywhere            state NEW
tcp spt:rtsp CONNMARK set 0x1

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
CONNMARK   all  --  anywhere             anywhere            CONNMARK restore

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
CONNMARK   tcp  --  anywhere             jet.local           state NEW
tcp dpt:rtsp CONNMARK set 0x1
CONNMARK   all  --  anywhere             anywhere            CONNMARK restore

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination


$ sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT       tcp  --  anywhere             anywhere            tcp
dpt:rtsp to:192.168.1.2

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

$ sudo ip rule
0:	from all lookup local
32764:	from all fwmark 0x2 lookup 2
32765:	from all fwmark 0x1 lookup 1
32766:	from all lookup main
32767:	from all lookup default

$ sudo ip route show table 1
default via 192.168.1.1 dev ra1

$ sudo ip route show table 2
default via 192.168.0.1 dev ra0

$ sudo ip route
192.168.1.0/24 dev ra1  proto kernel  scope link  src 192.168.1.2
192.168.0.0/24 dev ra0  proto kernel  scope link  src 192.168.0.2
default via 192.168.0.1 dev ra0


With this environment I get the same results. I send the first TCP
packet (SYN, dport 8554) through the interface ra1 (OK) with the IP
bound to this interface (SNAT OK) and I get the (SYN,ACK) to the same
IP and through the same interface (OK!), but my application does not
send the final acknowledgement to the TCP connection establishment
(ACK), so the RTSP messages are not sent and the client retries over
and over again the TCP session establishment. This happens when the
default route in the main routing table is not the marked one (in this
case, 192.168.0.1 - ra0 is the one in the default table, while I mark
the connection to be established through 192.168.1.1 - ra1).

I don't really need to balance the load, just to decide which
interface configure the streaming session through and start it while
preserving the previous streaming sessions; this is: changing the
interface through which I ask and receive video streaming while not
messing up old connections with other videos. I can't understand how
this can be so difficult to configure. I must be missing something in
my rules...


Any help would be much appreciated. Thanks,
Javi


El día 20 de abril de 2009 15:37, Thomas Jacob <jacob@internet24.de> escribió:
> Nice diagram ;)
>
> Well, it is as thought, your TOS rules set TOS fields in the packets
> and those are usually bounced back to you from the reply direction,
> so effectively you are using something like CONNMARK already.
>
> Jan's item 5 description is definitely a good starting point.
>
> Maybe you could try this, I am assuming that you really
> are talking only about connections originating from client
> here:
>
>
>    iptables -F OUTPUT -t mangle
>    iptables -F POSTROUTING -t nat
>    iptables -F PREROUTING -t nat
>
>
>    iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
> --to-source 192.168.0.2
>
>    iptables -t mangle -A INPUT   -j CONNMARK --restore-mark
>
>    iptables -t mangle -A OUTPUT -m state --state NEW \
>        -p tcp --dport 8554 -d 147.83.47.178 -j CONNMARK --set-mark 1
>    iptables -t mangle -A OUTPUT   -j CONNMARK --restore-mark
>
>
> If you want to load balance connections from the local net you need
> to use PREROUTING like Jan does. See the chain traversal diagrams I
> posted in the reply to Lloyd, understanding which packets pass thru
> which chains is really important for stuff like this.
>
> On Mon, 2009-04-20 at 15:08 +0200, Javier Gálvez Guerrero wrote:
>> Thanks for your answer, Thomas.
>>
>> Once I explained to Jan what I had to implement, he pointed me to the
>> item 5 in http://dev.medozas.de/NF-Cookbook.txt, so I've also used
>> CONNTRACK with no success:
>>
>> "I'll try to explain what I need to implement. I'm developing a daemon
>> application that manages WiFi handovers between two private networks
>> (192.168.0.0/24 and 192.168.1.0/24) using two different WiFi NICs.
>> When the link RSSI becomes low enough, the application triggers the
>> handover in order to keep a seamless video streaming service, so first
>> I must attach to the new network with the inactive NIC, ask for the
>> media content through it (the video session configuration is done
>> through the server port 8554) and, once I'm receiving the content
>> through the new link (the media content is actually sent from and to
>> ports differents from 8554), detach de old one.
>>
>> In order to do this in a seamless way I need to play with ip route, ip
>> rule and iptables. Marking traffic with iptables is not enough as it
>> is done in POSTROUTING, so the kernel previously decides which IP the
>> packets will have according to the default route in the main ip
>> routing table. Using MARK I send the packet through the interface I
>> want in each specific moment, while with SNAT I change the source IP
>> to the one attached to the used interface (thus changing the IP given
>> by the default routing table in case the new interface is not the one
>> configured as the default one). The received (answer) packets should
>> come through the same interface with the corresponding IP but they
>> just follow the route configured in the default ip routing table (then
>> not matching the connection established according to my custom setup),
>> so I need to setup more MARKing and ip rules so the local application
>> receives and delivers packets accordingly. This is where MARK fails
>> and TOS success, but I don't want to mess TOS network configuration
>> and use a more proper packet mangling with MARK as the only device
>> that should care about the handover is the client device."
>>
>>
>>
>> Anyway, it didn't work: as soon as I change the default route in the
>> main table the old connection is broken. Instead, connections should
>> be established and maintained through the links pointed in the
>> respective tables (not the main one), according with the marking rules
>> (that's what I need, to have two connections established with
>> different interfaces and to different networks and dynamically chose
>> one of both to send and receive the video stream).
>>
>> Time for ASCII art (quite "simple"):
>>
>>                                ,''''''''''''''`.
>>                                |  Server |
>>                                L_______J  147.83.47.178
>>                                  _/__
>>                               ,-'       `-.
>>                             ,'               `.
>>                            |                   '.
>>                            |     Network  |
>>                           /\                   /
>>                          /  \                /.
>>                        ,'    `._        _,'  `.
>>                       /         `----'          `.
>>                  |---'--------Y           ....:.............
>>                  |              |          |                   |
>>                  |Router A|           |    Router B |
>>                  '------------            `.................'
>>          192.168.0.0/24             192.168.1.0/24
>>
>>
>>                                        ,'''''''''''''''''''|
>>                          ra0        |               |  ra1
>>                                       |     Client |
>>                   192.168.0.2  L________|  192.168.1.2
>>
>>
>>
>> The rules with TOS are exactly the same just changing the marking
>> parameters by the TOS ones:
>>
>> ip rule add from all tos 0x10 table 1 prio 1
>> ip rule add from all tos 0x04 table 2 prio 1
>>
>> iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source
>> 192.168.0.2
>> iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x10
>> iptables -A PREROUTING -t mangle -s 147.83.47.178 -j TOS --set-tos 0x10
>>
>> iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
>> --to-source 192.168.1.2
>> iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x04
>> iptables -R PREROUTING 1 -t mangle -s 147.83.47.178 -j TOS --set-tos 0x04
>>
>>
>> This works but, as said before, I don't what this TOS stuff to be used.
>>
>>
>> So... any idea why the previous "item 5" didn't work for my
>> environment? What should I try?
>>
>>
>> Thank you for your help,
>> Javi
>>
>>
>> El día 20 de abril de 2009 13:44, Thomas Jacob <jacob@internet24.de> escribió:
>> > I haven't read thru all this, and you also haven't posted
>> > your TOS variant, but a few observations:
>> >
>> > You should use CONNMARK like Lloyd does, MARK does not
>> > affect the connection, just a single packet, and you have to
>> > reconstruct your MARK values for each packet.
>> >
>> > Your TOS rules most likely change the packet TOS field, and that
>> > is sort of "sticky" if all the intermediate hosts play ball,
>> > according to the standards, i.e. does do something like CONNMARK
>> > does.
>> >
>> > Also be aware of the different paths thru the various
>> > chains taken by packets from/to the local host and
>> > by forwarded packets.
>> >
>> > A small ascii diagram for your setup would be very helpful.
>> >
>> > On Mon, 2009-04-20 at 10:48 +0200, Javier Gálvez Guerrero wrote:
>> >> Hi,
>> >>
>> >> I've been trying something similar to Lloyd's configuration for a long
>> >> time with no success. I want to reach a host (server) from my laptop
>> >> with two WiFi interfaces attached to different subnetworks/routers;
>> >> the server is in the public network, both routers have public IPs (the
>> >> same domain than the server) while setting up two different private
>> >> networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface
>> >> in my laptop are attached to each one of them.
>> >>
>> >> The problem is that I can't manage to dynamically configure which
>> >> interface to be chosen to redirect my traffic. In my case, I want the
>> >> packets with destination port 8554 to be routed to one or other
>> >> interface as I want (so load one network or the other), so I tried to
>> >> change MARK iptables rules accordingly with no success, just getting
>> >> what Lloyd had: packets are routed through the default route in the
>> >> main ip routing table, while if I have not this route configured I
>> >> lose network access, despite having the corresponding tables and rules
>> >> properly configured. What I ended figuring out is that using the TOS
>> >> target it worked properly, but this is not what I'd like to preserve
>> >> as TOS changes the type of service parameter and could affect network
>> >> management beyond my laptop; I need to use MARK to manage my outgoing
>> >> and incoming traffic.
>> >>
>> >> This is the script I'm currently using (stripped):
>> >>
>> >>     ip route flush table 1
>> >>     ip route flush table 2
>> >>     ip rule del prio 1
>> >>     ip rule del prio 1
>> >>     iptables -F OUTPUT -t mangle
>> >>     iptables -F POSTROUTING -t nat
>> >>     iptables -F PREROUTING -t nat
>> >>
>> >>     ip route add default via 192.168.0.1 dev ra0 ---> I need a
>> >> configure a route in the main table! Why?
>> >>     ip route add table 1 192.168.0.0/24 dev ra0
>> >>     ip route add table 1 default via 192.168.0.1 dev ra0
>> >>     ip route add table 2 192.168.1.0/24 dev ra1
>> >>     ip route add table 2 default via 192.168.1.1 dev ra1
>> >>
>> >>     ip rule add from all fwmark 1 table 1 prio 1
>> >>     ip rule add from all fwmark 2 table 2 prio 1
>> >>
>> >>     iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
>> >> --to-source 192.168.0.2
>> >>     iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1
>> >>     iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1
>> >>
>> >>
>> >> So, when I want to change the marking rules:
>> >>
>> >>       iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
>> >> --to-source 192.168.1.2
>> >>       iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2
>> >>       iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2
>> >>
>> >>
>> >> Anyway, I'm not able to get this working, but only when the selected
>> >> IP matches the default IP in the main routing table, so... it doesn't
>> >> work. Changing MARKing rules with TOS works.
>> >>
>> >> So, I can't dynamically change the used interface for the selected packets.
>> >>
>> >>
>> >> I opened a bug (#589) in the Netfilter bugzilla system and Jan
>> >> Engelhardt has kindly tried to help me but I haven't been able to
>> >> solve it yet.
>> >>
>> >> By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.
>> >>
>> >>
>> >> Any help would be much appreciated, as I'm not sure if this is really
>> >> a bug or just a problem with iptables rules understanding or misuse.
>> >>
>> >>
>> >> Regards,
>> >> Javi
>> >>
>> >> 2009/4/20 Lloyd Standish <lloyd@crnatural.net>
>> >> >
>> >> > On Sun, 19 Apr 2009 03:00:17 -0600, Thomas Jacob <jacob@internet24.de> wrote:
>> >> >
>> >> > >>Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.
>> >> > >
>> >> > > Then your current setup in PREROUTING is what you want to go for, just keep
>> >> > > in mind that this does not give you load balancing for connections originating
>> >> > > from your router box, just the ones from your LAN.
>> >> >
>> >> > I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)
>> >> >
>> >> > However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)
>> >> >
>> >> > >
>> >> > > C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png
>> >> > >
>> >> > > I'm not sure why you need NAT on your eth0 though then, what are you
>> >> > > trying to achieve with this? But that should not be the cause
>> >> > > of the load balancing failure.
>> >> >
>> >> > I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.
>> >> >
>> >> > The lines:
>> >> > iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
>> >> > iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
>> >> > should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).
>> >> >
>> >> > >
>> >> > >> When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.
>> >> > >
>> >> > > That's right, but if all your /proc/net/ip_conntrack entries contain mark values
>> >> > > then there really must be something wrong with the fw mark <-> route interaction.
>> >> > >
>> >> > > My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for
>> >> > > instance, then maybe more people have a comparable setup to look at.
>> >> > That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.
>> >> >
>> >> > My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)
>> >> > >
>> >> > > Do you know about LARTC? The best way to get started IMO: http://lartc.org/
>> >> >
>> >> > Yes, I downloaded the tutorial a couple of days ago, thanks!
>> >> >
>> >> > --
>> >> > Lloyd
>> >> > --
>> >> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> >> > the body of a message to majordomo@vger.kernel.org
>> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>> >
>
>

Re: still can't route using fwmark

[Lloyd Standish]

On Mon, 20 Apr 2009 05:09:12 -0600, Thomas Jacob <jacob@internet24.de> wrote:

> Take a look at the packet traversal graph, outgoing packets from local
> processes do not pass thru PREROUTING, but incoming packets do, that's
> where your markings come from. But in your case, I'm sure you want to
> select a link when the outgoing connection is first established and then
> stay with that link. Selecting a different link with the second packet
> doesn't work with NATed connections, right?
> 
> So you if you want to load balance local packets correctly as well,
> you need to put some rules into OUTPUT. Possibly that's really your
> basic problem here, but I don't have time to think about that at the
> moment.

Thomas, somehow I had the erroneous idea that ALL packets, even those originating at the local machine, go through PREROUTING.  I finally understand now why I need the OUTPUT statements.  As soon as I added those, the load balancing started to work!

There's still one aspect that is not working.  Connections to the Internet initiated from the LAN, which go though PREROUTING (but not through OUTPUT) do not get a reply.  I suspect there is something wrong with the SNAT rules intended to masquerade LAN-connected machines onto the 2 interfaces, but I still have not had time to examine the logs to try to see what is going on.

Here's a copy of my (stripped) iptables script, now load balancing over ppp0 and ppp1 for local process only.

src0=`ip route show dev ppp0 | head -n1 | cut --delimiter=" " --fields=10`
src1=`ip route show dev ppp1 | head -n1 | cut --delimiter=" " --fields=10`
gw0=`ip route show dev ppp0 | head -n1 | cut --delimiter=" " --fields=1`
gw1=`ip route show dev ppp1 | head -n1 | cut --delimiter=" " --fields=1`

iptables -t mangle -N CONNMARK1
iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
iptables -t mangle -A CONNMARK1 -j LOG --log-prefix 'iptables-mark1: ' --log-level info

iptables -t mangle -N CONNMARK2
iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
iptables -t mangle -A CONNMARK2 -j LOG --log-prefix 'iptables-mark2: ' --log-level info

iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t mangle -A RESTOREMARK -j LOG --log-prefix 'restore-mark: ' --log-level info

iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j LOG --log-prefix "SNAT $src0: " --log-level info
iptables -t nat -A SNAT1 -j SNAT --to-source $src0

iptables -t nat -N SNAT2
iptables -t nat -A SNAT2 -j LOG --log-prefix "SNAT $src1: " --log-level info
iptables -t nat -A SNAT2 -j SNAT --to-source $src1
#iptables -t nat -A SNAT2 -j LOG --log-prefix "$src1: " --log-level info

# restore the fwmark on packets that belong to an existing connection
# this prerouting stuff would only be for connections initiated on the LAN

iptables -A PREROUTING -i eth0  -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK
iptables -A PREROUTING -i eth0 -t mangle -m mark ! --mark 0 -j RETURN
iptables -A PREROUTING -t mangle -j CONNMARK1
iptables -A PREROUTING -t mangle -m statistic --mode nth --every 2 --packet 0 -j CONNMARK2

# for local process (this has to be a new connection)
iptables -A OUTPUT -t mangle -j CONNMARK1
iptables -A OUTPUT -t mangle -m statistic --mode nth --every 2 --packet 0 -j CONNMARK2

# fox source ip address on packets to match used interface
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
#iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

if ! cat /etc/iproute2/rt_tables | grep -q '^251'
then
	echo '251     rt_link1' >> /etc/iproute2/rt_tables
fi
if ! cat /etc/iproute2/rt_tables | grep -q '^252'
then
	echo '252     rt_link2' >> /etc/iproute2/rt_tables
fi

ip route flush table rt_link1 2>/dev/null
ip route add $gw0 dev ppp0 table rt_link1
ip route add default via $gw0 dev ppp0 table rt_link1

ip route flush table rt_link2 2>/dev/null
ip route add $gw1 dev ppp1 table rt_link2
ip route add default via $gw1 dev ppp1 table rt_link2

if ! ip rule show | grep -q 'rt_link2'
then
	ip rule add fwmark 1 table rt_link1
	ip rule add fwmark 2 table rt_link2
fi
ip route flush cache

Once this is working for LAN I want to try load balancing according the byte count of each interface, if that is possible.

Re: still can't route using fwmark

[Thomas Jacob]

On Mon, Apr 20, 2009 at 05:15:21PM +0200, Javier Gálvez Guerrero wrote:
> $ sudo iptables -L -t mangle
> Chain PREROUTING (policy ACCEPT)
> target     prot opt source               destination
> CONNMARK   tcp  --  anywhere             anywhere            state NEW
> tcp spt:rtsp CONNMARK set 0x1

If you are forwarding packets via this host you need the CONNMARK restore
here as well, then you can also drop the CONNMARK restore from the INPUT chain,
PREROUTING is also traversed for packets destined for the local host.

> With this environment I get the same results. I send the first TCP
> packet (SYN, dport 8554) through the interface ra1 (OK) with the IP
> bound to this interface (SNAT OK) and I get the (SYN,ACK) to the same
> IP and through the same interface (OK!), but my application does not
> send the final acknowledgement to the TCP connection establishment
> (ACK), so the RTSP messages are not sent and the client retries over
> and over again the TCP session establishment. 

Your application does not send the ACK in the 3 way handshake, the
client kernel does. Somehow it doesn't receive the SYN,ACK or
the ACK does not reach the point where you're tcpdumping packets. 
Try to sniff as  close to your client app as possible. Could be
a NAT issue. Or maybe rp_filter or something else is breaking
it for you, you could try to enable /proc/sys/net/ipv4/conf/*/log_martians
to see any issues.

> messing up old connections with other videos. I can't understand how
> this can be so difficult to configure. I must be missing something in
> my rules...

Find another general purpose OS where you can do this AT ALL without additional
products. Then we'll talk about what is difficult or not ;=)

Re: still can't route using fwmark

[Thomas Jacob]

On Mon, Apr 20, 2009 at 09:38:13AM -0600, Lloyd Standish wrote:
> Thomas, somehow I had the erroneous idea that ALL packets, even those originating at the local machine, go through PREROUTING.  I finally understand now why I need the OUTPUT statements.  As soon as I added those, the load balancing started to work!

Great

> # restore the fwmark on packets that belong to an existing connection
> # this prerouting stuff would only be for connections initiated on the LAN
> 
> iptables -A PREROUTING -i eth0  -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK

This is too restrictive, drop the -i eth0, your forwarded packets from the internet
will not arrive thru eth0 but still need a CONNMARK -> MARK restore.

> iptables -A PREROUTING -i eth0 -t mangle -m mark ! --mark 0 -j RETURN
> iptables -A PREROUTING -t mangle -j CONNMARK1
> iptables -A PREROUTING -t mangle -m statistic --mode nth --every 2 --packet 0 -j CONNMARK2

Otherwise you might assign a different mark again here, and that doesn't work
with NATed connections. Possibly you should also use -m state --state NEW to prevent
the marking stuff taking place on any other packets than the initial ones.

> Once this is working for LAN I want to try load balancing according the byte count of each interface, if that is possible.

Hmm, I don't see how to do this with out using some kind of script
that constantly checks the load of your links and then modifies iptables
rules accordingly. Don't forget that you are loadbalancing connections, not
individual packets. But maybe you can cook something up the recent match or
something, if you do, please keep the list posted ;)

   Thomas

Re: still can't route using fwmark

[Lloyd Standish]

On Mon, 20 Apr 2009 13:26:01 -0600, Thomas Jacob <jacob@internet24.de> wrote:

>>
>> iptables -A PREROUTING -i eth0  -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK
>
> This is too restrictive, drop the -i eth0, your forwarded packets from the internet
> will not arrive thru eth0 but still need a CONNMARK -> MARK restore.

Thanks for your help.

I tightened up the policies and added rules to pass any packets, for new connections or otherwise, to or from eth0 (the LAN), and drop NEW connection packets except those from the local machine and from LAN. :

iptables -t filter -P INPUT DROP
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD DROP

iptables -t mangle -A INPUT -i eth0 -j ACCEPT
iptables -t mangle -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t mangle -A FORWARD -i eth0 -j ACCEPT
iptables -t mangle -A FORWARD -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT

However, there is a mistake here.  I am finding that I cannot access local files on the router box from a LAN connected machine (nor can I connect from the router box to Internet) unless I set the policy for INPUT to be ACCEPT.

--
Lloyd

Re: still can't route using fwmark

[Thomas Jacob]

On Tue, 2009-04-21 at 13:54 -0600, Lloyd Standish wrote:
> On Mon, 20 Apr 2009 13:26:01 -0600, Thomas Jacob <jacob@internet24.de> wrote:
> 
> >>
> >> iptables -A PREROUTING -i eth0  -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK
> >
> > This is too restrictive, drop the -i eth0, your forwarded packets from the internet
> > will not arrive thru eth0 but still need a CONNMARK -> MARK restore.
> 
> Thanks for your help.
> 
> I tightened up the policies and added rules to pass any packets, for new connections or otherwise, to or from eth0 (the LAN), and drop NEW connection packets except those from the local machine and from LAN. :
> 
> iptables -t filter -P INPUT DROP
> iptables -t filter -P OUTPUT ACCEPT
> iptables -t filter -P FORWARD DROP
> 
> iptables -t mangle -A INPUT -i eth0 -j ACCEPT
> iptables -t mangle -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
> iptables -t mangle -A FORWARD -i eth0 -j ACCEPT
> iptables -t mangle -A FORWARD -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT

Do you have the ESTABLISHED stuff also in Output? Otherwise you will
block direct access to that box (see the packet traversal graph
again)....

Re: still can't route using fwmark

[Javier Gálvez Guerrero]

I've switched to Debian Lenny 5.01 (kernel 2.6.26-2, iptables 1.4.2)
and Jan's suggestion has worked out-of-the-box: the old streams are
stable when changing the default route in the main table =).

Anyway, I would like to know if you have any clue about what could be
happening in Ubuntu 8.10 (kernel 2.6.27-11, iptables 1.4.0), since it
would be very helpful in case I need to develop some iptables-related
stuff in the future.


Thank you so much for your help,
Javi

El día 20 de abril de 2009 20:59, Thomas Jacob <jacob@internet24.de> escribió:
> On Mon, Apr 20, 2009 at 05:15:21PM +0200, Javier Gálvez Guerrero wrote:
>> $ sudo iptables -L -t mangle
>> Chain PREROUTING (policy ACCEPT)
>> target     prot opt source               destination
>> CONNMARK   tcp  --  anywhere             anywhere            state NEW
>> tcp spt:rtsp CONNMARK set 0x1
>
> If you are forwarding packets via this host you need the CONNMARK restore
> here as well, then you can also drop the CONNMARK restore from the INPUT chain,
> PREROUTING is also traversed for packets destined for the local host.
>
>> With this environment I get the same results. I send the first TCP
>> packet (SYN, dport 8554) through the interface ra1 (OK) with the IP
>> bound to this interface (SNAT OK) and I get the (SYN,ACK) to the same
>> IP and through the same interface (OK!), but my application does not
>> send the final acknowledgement to the TCP connection establishment
>> (ACK), so the RTSP messages are not sent and the client retries over
>> and over again the TCP session establishment.
>
> Your application does not send the ACK in the 3 way handshake, the
> client kernel does. Somehow it doesn't receive the SYN,ACK or
> the ACK does not reach the point where you're tcpdumping packets.
> Try to sniff as  close to your client app as possible. Could be
> a NAT issue. Or maybe rp_filter or something else is breaking
> it for you, you could try to enable /proc/sys/net/ipv4/conf/*/log_martians
> to see any issues.
>
>> messing up old connections with other videos. I can't understand how
>> this can be so difficult to configure. I must be missing something in
>> my rules...
>
> Find another general purpose OS where you can do this AT ALL without additional
> products. Then we'll talk about what is difficult or not ;=)
>
>

Re: still can't route using fwmark

[Thomas Jacob]

On Wed, 2009-04-22 at 11:53 +0200, Javier Gálvez Guerrero wrote:
> I've switched to Debian Lenny 5.01 (kernel 2.6.26-2, iptables 1.4.2)
> and Jan's suggestion has worked out-of-the-box: the old streams are
> stable when changing the default route in the main table =).
> 
> Anyway, I would like to know if you have any clue about what could be
> happening in Ubuntu 8.10 (kernel 2.6.27-11, iptables 1.4.0), since it
> would be very helpful in case I need to develop some iptables-related
> stuff in the future.

I am not running Ubuntu 8.10 anywhere, but I am employing self-compiled
2.6.27.X and 1.4.X versions on a Lenny base, and there I haven't had any
problems with policy routing (in a lab environment). But then again, I
didn't have any with 2.6.18 and 1.3.X either ;) 

In generally all this stuff is pretty mature and doesn't simply stop
working when you switch to the next generation. Also there should be a
number of people out there that do something similar with stock Lenny
and 8.10 packages, and they'd be sure to complain, maybe you should take
a look at the Debian/Ubuntu bug tracker .....

Re: still can't route using fwmark

[Lloyd Standish]

On Wed, 22 Apr 2009 03:35:33 -0600, Thomas Jacob <jacob@internet24.de> wrote:

>>
>> I tightened up the policies and added rules to pass any packets, for new connections or otherwise, to or from eth0 (the LAN), and drop NEW connection packets except those from the local machine and from LAN. :
>>
>> iptables -t filter -P INPUT DROP
>> iptables -t filter -P OUTPUT ACCEPT
>> iptables -t filter -P FORWARD DROP
>>
>> iptables -t mangle -A INPUT -i eth0 -j ACCEPT
>> iptables -t mangle -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
>> iptables -t mangle -A FORWARD -i eth0 -j ACCEPT
>> iptables -t mangle -A FORWARD -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
>
> Do you have the ESTABLISHED stuff also in Output? Otherwise you will
> block direct access to that box (see the packet traversal graph
> again)....

I have the OUTPUT policy set to ACCEPT (I want to accept any packets sent by my local box).   According to my understanding, any packet that does not match an OUTPUT rule would therefore be accepted.  It should not be necessary to have any OUTPUT rule, is that right?

My setup is working for LAN-connected machines access only when I set the INPUT policy to ACCEPT, which is too loose for my taste, even though I am behind my ISP's nat/firewall.  Therefore, it seems that the problem must be in my INPUT chain rules.

I got load-balancing for LAN-connected machines (which are natting to both ppp0 or ppp1) working.  I fixed it to configure any single interface (no load-balancing), or both ppp0 and ppp1 for load-balancing, if both interfaces are up.  I put the script in /etc/ppp/ip-up.d and /etc/ppp/ip-down.d, which allows this to auto-configure the interfaces whenever an interface comes up or goes down.  It works very well.

When there is but a single interface up, then, all the load balancing stuff (MARK, CONNTRACK, etc.) is skipped, and the iptables commands become very simple (just a simple firewall with nat for the LAN).   Only the following commands are used when only one interface is up:

src0=`ip route show dev ppp0 2>/dev/null | head -n1 | cut --delimiter=" " --fields=10`
src1=`ip route show dev ppp1 2>/dev/null | head -n1 | cut --delimiter=" " --fields=10`
gw0=`ip route show dev ppp0 2>/dev/null | head -n1 | cut --delimiter=" " --fields=1`
gw1=`ip route show dev ppp1 2>/dev/null | head -n1 | cut --delimiter=" " --fields=1`

# flush all iptables entries
iptables -t filter -F
iptables -t filter -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t filter -P INPUT DROP
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD DROP

iptables -t mangle -A INPUT -i lo -j ACCEPT
iptables -t mangle -A INPUT -i eth0 -j ACCEPT
iptables -t mangle -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t mangle -A FORWARD -i eth0 -j ACCEPT
iptables -t mangle -A FORWARD -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t mangle -A OUTPUT -o eth0 -j ACCEPT
iptables -t mangle -A OUTPUT -o lo -j ACCEPT

if [ "$src0" == "" -o "$src1" == "" ]; then
#	echo a device is not online, setting default gateway and quitting
	ip route del default dev ppp0 2>/dev/null
	ip route del default dev ppp1 2>/dev/null
	if [ "$src0" != "" ]; then
		ip route add default dev ppp0
		iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
	fi
	if [ "$src1" != "" ]; then
		ip route add default dev ppp1
		iptables -t nat -A POSTROUTING -o ppp1 -j MASQUERADE
	fi
	exit
fi


I have studied the iptables flow diagram, it seems to me that my INPUT rules above should allow all traffic to/from a LAN-connected (eth0) machine.  However, I am missing something, since I have to set the INPUT policy to ACCEPT to get connectivity from the LAN.  And I have to set the FORWARD policy to ACCEPT to allow my load-balancing stuff to work!

(When this is all fixed I'll post an article about it on my "Linux Notes" site at http://www.voluntary-simplicity.org/linux.)