Iproute2 and fwmark usage

Iproute2 and fwmark usage

[Thhoep]

hi,

i have problems with the usage if netfilter marks in connection with
advanced routing using the iproute2 package. standing alone and using its
(lousy) inbuilt filter thingy iproute works normal, but trying to use
netfilter marks for routing decisions some pretty odd things happen. see for
yourself.


Routing tables (for this case uninteresting tables left out):

 router:~# ip route show table test
 217.5.98.133 dev ppp4  scope link  src 80.144.178.206
 192.168.0.0/24 dev eth4  scope link
 default via 217.5.98.133 dev ppp4

 router:~# ip route show table offline
 192.168.0.0/24 dev eth4  scope link
 unreachable default

 router:~# ip route show table main
 217.5.98.133 dev ppp0  proto kernel  scope link  src 80.144.190.200
 217.5.98.133 dev ppp1  proto kernel  scope link  src 80.144.188.142
 217.5.98.133 dev ppp2  proto kernel  scope link  src 80.144.179.55
 217.5.98.133 dev ppp5  proto kernel  scope link  src 80.144.184.186
 217.5.98.133 dev ppp4  proto kernel  scope link  src 80.144.178.206
 217.5.98.133 dev ppp3  proto kernel  scope link  src 80.144.188.180
 192.168.0.0/24 dev eth4  scope link
 default via 217.5.98.133 dev ppp0

Routing policies (for this case uninteresting rules left out):

 router:~# ip rule
 0:      from all lookup local
 1000:    from all fwmark        1 lookup test
 2000:   from 192.168.0.0/24 lookup offline
 32766:  from all lookup main
 32767:  from all lookup default

Netfilter configuration (for this case uninteresting rules left out,
uninteresting chain counters zeroed):

 - filter chains empty, policies = ACCEPT

 router:~# iptables -t nat -L -n -x -v
 Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination
     0   0 MASQUERADE  all  --  *      ppp4    0.0.0.0/0    0.0.0.0/0

 Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination


 router:~# iptables -t mangle -L -n -x -v
 Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination
      0    0 MARK       all  --  eth4   *       192.168.0.22
141.24.12.2/0          MARK set 0x1

 Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

the test:

 i ran a ping on my internal host 192.168.0.22 to a known external host
141.24.12.2.

 iptables shows, that the requests get marked correctly. for testing reasons
added just counting rules also show, that replies are getting in but dont
reach the FORWARD chain. so i assume they get lost while doing the routing.

  router:~# iptables -t mangle -L -n -x -v -Z
  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         2      120            all  --  ppp4   *       141.24.12.2
0.0.0.0/0
         2      120 MARK       all  --  eth4   *       192.168.0.22
141.24.12.2        MARK set 0x1

  Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         0        0            all  --  ppp4   *       141.24.12.2
0.0.0.0/0

  Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         0        0            all  --  *      eth4    141.24.12.2
0.0.0.0/0

 tcpdump verifies, that indeed replies are coming in and shows, that no icmp
error messages are sent out, what should be the case if no route back could
be found due to incorrect routing.

  router:~# tcpdump -n -i ppp4
  tcpdump: listening on ppp4
  15:06:59.512539 80.144.178.206 > 141.24.12.2: icmp: echo request
  15:06:59.579680 141.24.12.2 > 80.144.178.206: icmp: echo reply (DF)

 changing from "fwmark" usage to the inbuilt filter of iproute the routing
works absolutely fine.

  router:~# ip rule del fwmark 1
  router:~# ip rule add from 192.168.0.22 table test

  <doing some pings from the internal host>

  router:~# iptables -t mangle -L -n -x -v -Z
  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         4      240            all  --  ppp4   *       141.24.12.2
0.0.0.0/0
         4      240 MARK       all  --  eth4   *       192.168.0.22
141.24.12.2        MARK set 0x1

  Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         4      240            all  --  ppp4   *       141.24.12.2
0.0.0.0/0

  Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         4      500            all  --  *      eth4    141.24.12.2
0.0.0.0/0

router:~# iptables
iptables v1.2.6a: no command specified
Try `iptables -h' or 'iptables --help' for more information.
router:~# ip -V
ip utility, iproute2-ss010824
router:~# cat /proc/version
Linux version 2.4.22 (root@dslrouter) (gcc version 2.95.4 20011002 (Debian
prerelease)) #1 SMP Wed Nov 12 23:08:31 CET 2003


i am grateful for any help. thanks.

thomas hoeppler

Re: Iproute2 and fwmark usage

[Thhoep]

> I'm sorry. I tried to read and understand your problem several times but
> I'm lost. Can you explain a bit more what the problem is?

oh, of course. in short terms:

if using the inbuilt "ip rule add from .." directive of iproute for advanced
routing everything is working fine. but if i try to use netfilter marks to
do the exact same thing using "ip rule add fwmark ..." outgoing packets find
their way through my routing stuff and out of the net, but incoming replies
just get lost between PREROUTING and FORWARD, probably during the routing
process, for just no reason and without any icmp error messages or
something.

Re: Iproute2 and fwmark usage

[Ramin Dousti]

On Fri, Jan 09, 2004 at 12:05:30AM +0100, Thhoep wrote:

> hi,
> 
> i have problems with the usage if netfilter marks in connection with
> advanced routing using the iproute2 package. standing alone and using its
> (lousy) inbuilt filter thingy iproute works normal, but trying to use
> netfilter marks for routing decisions some pretty odd things happen. see for
> yourself.

I'm sorry. I tried to read and understand your problem several times but
I'm lost. Can you explain a bit more what the problem is?

Ramin

Re: Iproute2 and fwmark usage

[Thhoep]

> OK. You mean when doing MARK in mangle for ping a -> b, you see both
> ping and pong on ppp4 but the pongs get dropped somewhere between
> PREROUTING and FORWARD. When you remove the MARK from netfilter then
> all works fine??

nonono, the marking isnt the problem. but first:

for marking i use:
"iptables -t mangle -A PREROUTING -i eth4 -s 192.168.0.22 -d 141.24.12.2 -j
MARK --mark-with 1"

for selecting a routing table upon that marking i use:
"ip rule add fwmark 1 table test"

while table test has a local network route, a host route to the ppp peer and
a default route to the internet over the ppp peer as gateway.

the main table is the same but using another ppp interface.

all the other tables only differ only in the way out, but all include the
route to the one and only local net.


back to the problem:
the marking doesnt seem to be the problem. if i let the marking stay and
just remove the upon routing rule and replace it with:
"ip rule add from 192.168.0.22 table test"
then everything works absolutely fine. so i think its something with the
iproute trying to use the firewall mark what drops the packet, because it
seems to disappear while beeing routed.

Re: Iproute2 and fwmark usage

[Ramin Dousti]

On Fri, Jan 09, 2004 at 05:17:36PM +0100, Thhoep wrote:

> if using the inbuilt "ip rule add from .." directive of iproute for advanced
> routing everything is working fine. but if i try to use netfilter marks to
> do the exact same thing using "ip rule add fwmark ..." outgoing packets find
> their way through my routing stuff and out of the net, but incoming replies
> just get lost between PREROUTING and FORWARD, probably during the routing
> process, for just no reason and without any icmp error messages or
> something.

OK. You mean when doing MARK in mangle for ping a -> b, you see both
ping and pong on ppp4 but the pongs get dropped somewhere between
PREROUTING and FORWARD. When you remove the MARK from netfilter then
all works fine??

Instead of "iptables -L..." can you send the exact commands you enter
from scratch for both netfilter and iproute2 instructions? "iptables -L..."
wraps the lines and is hard to read for me and also doesn't show you all the
actions taken (or at least I have difficulty interpretting them).

Just make sure that you start from scratch and a clean setup.

Ramin

Re: Iproute2 and fwmark usage

[Thhoep]

> > for marking i use:
> > "iptables -t mangle -A PREROUTING -i eth4 -s 192.168.0.22 -d
141.24.12.2 -j
> > MARK --mark-with 1"
> >
> > for selecting a routing table upon that marking i use:
> > "ip rule add fwmark 1 table test"
> >
> OK. Are these two instructions the only instructions active on the box?


besides of the marking there is a masquerading active, as mentioned in
previous mails:
"iptables -t nat -A POSTROUTING -o ppp4 -j MASQUERADE"

filter table is empty.

there exist other routing rules, but they are all rules from other hosts out
into the world. no rules backward, for internet-lan-direction i use the main
table.


> Both:
> "ip rule add fwmark 1 table test"
> and
> "ip rule add from 192.168.0.22 table test"
>
> are being used for the ping packets and seem to be equivalent. Your
problem
> seems to be the loss of the pong packets, right?

right

> >  tcpdump verifies, that indeed replies are coming in and shows, that no
icmp
> > error messages are sent out, what should be the case if no route back
could
> > be found due to incorrect routing.
>
> This part is not completely correct. The ICMP messages would go out
through
> ppp0, right? and you're not capturing that.
>
> Do a simultanious tcpdump on all the ppp interfaces and let us know the
result.

ok, i did monitor all interfaces using
"tcpdump -n -i ppp0 icmp and host 141.24.12.2"
and so on. as expected nowhere a pong went out. i think any pong leaving the
box would have showed up in the FORWARD counting rule in the first test.

> Although, with the output of "ip rule" it looks to me that table main must
be
> taken for the return traffic and you have a route to 192.168.0.0/24 there.

you are right. and _all_ routing tables contain a route to the local net, so
a packet should always find its way from the outside in. i think even if the
masquerading would be broken, the packet wouldnt get lost. it would show up
somewhere OR an icmp error would be sent to its sender. so is there a bug in
the routing code? and why am i then the only one having this problem?

Re: Iproute2 and fwmark usage

[Ramin Dousti]

> for marking i use:
> "iptables -t mangle -A PREROUTING -i eth4 -s 192.168.0.22 -d 141.24.12.2 -j
> MARK --mark-with 1"
> 
> for selecting a routing table upon that marking i use:
> "ip rule add fwmark 1 table test"
> 
OK. Are these two instructions the only instructions active on the box?

> while table test has a local network route, a host route to the ppp peer and
> a default route to the internet over the ppp peer as gateway.
> 
> the main table is the same but using another ppp interface.
> 
> all the other tables only differ only in the way out, but all include the
> route to the one and only local net.
> 
> 
> back to the problem:
> the marking doesnt seem to be the problem. if i let the marking stay and
> just remove the upon routing rule and replace it with:
> "ip rule add from 192.168.0.22 table test"
> then everything works absolutely fine. so i think its something with the
> iproute trying to use the firewall mark what drops the packet, because it
> seems to disappear while beeing routed.
> 

Both:
"ip rule add fwmark 1 table test"
and
"ip rule add from 192.168.0.22 table test"

are being used for the ping packets and seem to be equivalent. Your problem
seems to be the loss of the pong packets, right?

>  tcpdump verifies, that indeed replies are coming in and shows, that no icmp
> error messages are sent out, what should be the case if no route back could
> be found due to incorrect routing.

This part is not completely correct. The ICMP messages would go out through
ppp0, right? and you're not capturing that.

Do a simultanious tcpdump on all the ppp interfaces and let us know the result.
Although, with the output of "ip rule" it looks to me that table main must be
taken for the return traffic and you have a route to 192.168.0.0/24 there.

Ramin

Re: Iproute2 and fwmark usage

[Thhoep]

> Except if it's being dropped by conntrack... And remember, tcpdump sees
stuff
> before the stack does its thing to the packets.

is there a logical reason (besides some bug in the code) why it should drop
something? i thought (but aint sure) that the "demasquerading" happens right
in the prerouting, before the routing process, so that no routing rule in
the world could disturb it?

> An ICMP is not being sent for another ICMP. But I guess an ICMP for ping
> can be sent, though.

.. you made me think with that.. ;) but i've just tested it with http
requests. no icmp errors either.

> You got me there. Try to reduce the complexity of your setup (just bring
> up one ppp, remove the unnecessary tables and so forth) and retest again.
> If the test passes then add little by little to your setup. I'm sure
you'll
> find/locate the problem that way.

i just did it. killed all other ppp connections, there was just eth4 (the
local lan interface), ppp0 and a bunch of non-ip ethernet interfaces. but
the result was the same, and no mis-guided packages in either direction.

i've really no clue what to do now.

i will try to reproduce the configuration on another server here.

Re: Iproute2 and fwmark usage

[Ramin Dousti]

On Fri, Jan 09, 2004 at 08:06:33PM +0100, Thhoep wrote:

> besides of the marking there is a masquerading active, as mentioned in
> previous mails:
> "iptables -t nat -A POSTROUTING -o ppp4 -j MASQUERADE"
> 
> filter table is empty.
> 
> ok, i did monitor all interfaces using
> "tcpdump -n -i ppp0 icmp and host 141.24.12.2"
> and so on. as expected nowhere a pong went out. i think any pong leaving the
> box would have showed up in the FORWARD counting rule in the first test.

Yes. But one other possibility is if you receive the return traffic through
another interface (which should not happen because you're MASQ'ing) then
the de-MASQ does not take place. But again, it doesn't have anything to do
with the MARK and rule...

> you are right. and _all_ routing tables contain a route to the local net, so
> a packet should always find its way from the outside in.

Except if it's being dropped by conntrack... And remember, tcpdump sees stuff
before the stack does its thing to the packets.


> i think even if the
> masquerading would be broken, the packet wouldnt get lost. it would show up
> somewhere OR an icmp error would be sent to its sender.

An ICMP is not being sent for another ICMP. But I guess an ICMP for ping
can be sent, though.


> so is there a bug in
> the routing code? and why am i then the only one having this problem?

You got me there. Try to reduce the complexity of your setup (just bring
up one ppp, remove the unnecessary tables and so forth) and retest again.
If the test passes then add little by little to your setup. I'm sure you'll
find/locate the problem that way.

Ramin

Re: Iproute2 and fwmark usage

[Ramin Dousti]

On Fri, Jan 09, 2004 at 09:09:14PM +0100, Thhoep wrote:

> is there a logical reason (besides some bug in the code) why it should drop
> something?

I can't think of any logical reason. And to be honest, I don't think it is
due to a bug.

> i thought (but aint sure) that the "demasquerading" happens right
> in the prerouting, before the routing process, so that no routing rule in
> the world could disturb it?

You're right.

> i just did it. killed all other ppp connections, there was just eth4 (the
> local lan interface), ppp0 and a bunch of non-ip ethernet interfaces. but
> the result was the same, and no mis-guided packages in either direction.
> 
> i've really no clue what to do now.

Send the exact rules (netfilter/ip) from scratch for this minimal setup.
I'll try to have an exact setup here...

Ramin

> 
> i will try to reproduce the configuration on another server here.

Re: Iproute2 and fwmark usage

[Thhoep]

i just tried over an hour to bring a tunnel up on one of my other servers,
so i can simulate routing behaviour with it, but just wasted my time with
trying to fix a broken ppp installation on it. dont have the patience and
gotten way too tired to do that right now. so i give you the configuration
of my testing setup and if you would like to reproduce it i would appreciate
it.

iptables -t mangle -A PREROUTING -i eth4 -s 192.168.0.22 -j MARK --set-mark
0

ip route add 192.168.0.0/24 dev eth4 table test

ip rule add fwmark 1 table test
OR
ip rule add from 192.168.0.22 table test

(these ones done in the ppp ip-up script)

ip route add <remote ip> dev <ppp iface> src <local ip>
ip route add default via <remote ip> dev <ppp iface>

(dont use a default route in the main table to the same ppp iface, so you
feel a difference when the routing rules apply or not)


thats all i think.

Re: Iproute2 and fwmark usage

[Thhoep]

> i just tried over an hour to bring a tunnel up on one of my other servers,
> so i can simulate routing behaviour with it, but just wasted my time with
> trying to fix a broken ppp installation on it. dont have the patience and
> gotten way too tired to do that right now. so i give you the configuration
> of my testing setup and if you would like to reproduce it i would
appreciate
> it.
>
> iptables -t mangle -A PREROUTING -i eth4 -s 192.168.0.22 -j
MARK --set-mark 1
>

damn, forgot the masquerading rule here:
iptables -t nat -A POSTROUTING -o ppp4 -j MASQUERADE


> ip route add 192.168.0.0/24 dev eth4 table test
>
> ip rule add fwmark 1 table test
> OR
> ip rule add from 192.168.0.22 table test
>
> (these ones done in the ppp ip-up script)
>
> ip route add <remote ip> dev <ppp iface> src <local ip>
> ip route add default via <remote ip> dev <ppp iface>
>
> (dont use a default route in the main table to the same ppp iface, so you
> feel a difference when the routing rules apply or not)
>
>
> thats all i think.
>
>

Re: Iproute2 and fwmark usage

[Thhoep]

in my opinion something is completely wrong with my 2.4.23 kernel. i just
used another server of mine to reproduce the same situation. for this
purpose i compiled a completely new 2.4.23 kernel with advanced routing and
netfilter and whatever support and installed fresh versions of the iptables
and iproute2 package with the debian system (using 'stable'). because the
host only has 1 ethernet interface i used a ppp-over-ssh tunnel to simulate
routing behaviour.

in just plain routing mode everything was fine, packets has been routed from
the lan through the ppp tunnel and back.

then i added a firewall rule
"iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE"

is anything wrong with it??

well do make it short: it changed nothing. packets still get un-masqueraded
out of ppp0 with local lan ips. i really dont understand that. kernel new,
iptables new, iproute new. no modules, all network stuff statically built
in. and its the 2nd server which shows strange behaviour with this
configuration. cant believe its a problem with me.

i'm just downloading the 2.4.24 kernelsource and will try with that kernel.

Re: Iproute2 and fwmark usage

[Antony Stone]

On Sunday 11 January 2004 8:57 pm, Thhoep wrote:

> in my opinion something is completely wrong with my 2.4.23 kernel. i just
> used another server of mine to reproduce the same situation. for this
> purpose i compiled a completely new 2.4.23 kernel with advanced routing and
> netfilter and whatever support and installed fresh versions of the iptables
> and iproute2 package with the debian system (using 'stable').

What version of iptables is that?   Is it as up to date as the netfilter 
version you've got in your shiny new kernel?

> because the host only has 1 ethernet interface i used a ppp-over-ssh tunnel
> to simulate routing behaviour.

This sounds a little bit dodgy to me....

> then i added a firewall rule
> "iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE"
>
> is anything wrong with it??

No, looks perfectly reasonable to me.

> well do make it short: it changed nothing. packets still get un-masqueraded
> out of ppp0 with local lan ips.

Any chance you can put in another ethernet card and try a simpler routing 
setup?

Antony.

-- 
You can spend the whole of your life trying to be popular,
but at the end of the day the size of the crowd at your funeral
will be largely dictated by the weather.

 - Frank Skinner

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Iproute2 and fwmark usage

[Thhoep]

> What version of iptables is that?   Is it as up to date as the netfilter
> version you've got in your shiny new kernel?

it is v1.2.6a
i'm not sure which kernels it accepts, theres written
"This version requires kernel >= 2.4.4
This version recommends kernel >= 2.4.18"

> This sounds a little bit dodgy to me....

everything's fine with my tunneling. thats the only thing i'm sure of. and
never the less the underlaying protocol of ppp is of no interest for the
routing/firewalling.

concerning the issue with MASQUERADE in my previous mail, maybe it has
something to do with the "MASQUERADE oops" written in the changelog of
kernel v2.4.24

as 2.4.24 was not working on my router (maybe because of the outdated
iptables, but wasnt in the mood for analysis)  i just step back to 2.4.22
and retry with that. i think fresh versions of iptables and iproute2 cant
hurt either.

Re: Iproute2 and fwmark usage

[Thhoep]

ok, just solved my problem. was just a misunderstanding of mine. i thought
"stable" means "proved and working". didn't know that i had to patch and
bugfix stable kernels till they work in normal order.

found the necassary bugfixes for all my problems as a patch-o-matic on the
netfilter main page.

never the less thanks for helping.

Re: Iproute2 and fwmark usage

[Antony Stone]

On Sunday 11 January 2004 11:12 pm, Thhoep wrote:

> ok, just solved my problem. was just a misunderstanding of mine. i thought
> "stable" means "proved and working". didn't know that i had to patch and
> bugfix stable kernels till they work in normal order.

Normally, you don't.   However if you mix versions of the kernelspace system 
and the userspace utilities, results are not guaranteed.

The general rule is that if you update one you should update the other to 
match.

> found the necassary bugfixes for all my problems as a patch-o-matic on the
> netfilter main page.
>
> never the less thanks for helping.

Glad you've got things working okay now.

Antony.

-- 
Behind the counter a boy with a shaven head stared vacantly into space,
a dozen spikes of microsoft protruding from the socket behind his ear.

 - William Gibson, Neuromancer (1984)

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Iproute2 and fwmark usage

[Thhoep]

> ok, just solved my problem.

sorry, i was mistaking. i found some bugfixes, thats right, but they didnt
fix this bug BUT brought several other bugs into my kernel.

i still have the problem that usage of firewall marks as routing key doesnt
work.
kernel:     2.4.20-22  (all tried)
iptables:   1.2.9
iproute:    ss010824

in addition with kernels 2.4.23 and 2.4.24 routing table selection doesnt
seem to work at all for me.


if you have ANY hints, i would be really grateful. maybe you can name me a
configuration of kernel, kernelpatches, iptables and iproute with which
advanced routing, firewalling and QoS are working fine. to be more precise,
i need a full functional netfilter with the usual matching rules and
targets, esp. marks and masquerading, advanced routing that allows routing
selection upon netfilter marks and qos with the htb queueing disciplin.

thank you.

iproute2 and fwmark usage

[Thhoep]

hi,

i have problems with the usage if netfilter marks in connection with
advanced routing using the iproute2 package. standing alone and using its
(lousy) inbuilt filter thingy iproute works normal, but trying to use
netfilter marks for routing decisions some pretty odd things happen. see for
yourself.


Routing tables (for this case uninteresting tables left out):

 router:~# ip route show table test
 217.5.98.133 dev ppp4  scope link  src 80.144.178.206
 192.168.0.0/24 dev eth4  scope link
 default via 217.5.98.133 dev ppp4

 router:~# ip route show table offline
 192.168.0.0/24 dev eth4  scope link
 unreachable default

 router:~# ip route show table main
 217.5.98.133 dev ppp0  proto kernel  scope link  src 80.144.190.200
 217.5.98.133 dev ppp1  proto kernel  scope link  src 80.144.188.142
 217.5.98.133 dev ppp2  proto kernel  scope link  src 80.144.179.55
 217.5.98.133 dev ppp5  proto kernel  scope link  src 80.144.184.186
 217.5.98.133 dev ppp4  proto kernel  scope link  src 80.144.178.206
 217.5.98.133 dev ppp3  proto kernel  scope link  src 80.144.188.180
 192.168.0.0/24 dev eth4  scope link
 default via 217.5.98.133 dev ppp0

Routing policies (for this case uninteresting rules left out):

 router:~# ip rule
 0:      from all lookup local
 1000:    from all fwmark        1 lookup test
 2000:   from 192.168.0.0/24 lookup offline
 32766:  from all lookup main
 32767:  from all lookup default

Netfilter configuration (for this case uninteresting rules left out,
uninteresting chain counters zeroed):

 - filter chains empty, policies = ACCEPT

 router:~# iptables -t nat -L -n -x -v
 Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination
     0   0 MASQUERADE  all  --  *      ppp4    0.0.0.0/0
0.0.0.0/0

 Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination


 router:~# iptables -t mangle -L -n -x -v
 Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination
      0    0 MARK       all  --  eth4   *       192.168.0.22
141.24.12.2/0          MARK set 0x1

 Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

 Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts      bytes target     prot opt in     out     source
destination

the test:

 i ran a ping on my internal host 192.168.0.22 to a known external host
141.24.12.2.

 iptables shows, that the requests get marked correctly. for testing reasons
added just counting rules also show, that replies are getting in but dont
reach the FORWARD chain. so i assume they get lost while doing the routing.

  router:~# iptables -t mangle -L -n -x -v -Z
  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         2      120            all  --  ppp4   *       141.24.12.2
0.0.0.0/0
         2      120 MARK       all  --  eth4   *       192.168.0.22
141.24.12.2        MARK set 0x1

  Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         0        0            all  --  ppp4   *       141.24.12.2
0.0.0.0/0

  Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         0        0            all  --  *      eth4    141.24.12.2
0.0.0.0/0

 tcpdump verifies, that indeed replies are coming in and shows, that no icmp
error messages are sent out, what should be the case if no route back could
be found due to incorrect routing.

  router:~# tcpdump -n -i ppp4
  tcpdump: listening on ppp4
  15:06:59.512539 80.144.178.206 > 141.24.12.2: icmp: echo request
  15:06:59.579680 141.24.12.2 > 80.144.178.206: icmp: echo reply (DF)

 changing from "fwmark" usage to the inbuilt filter of iproute the routing
works absolutely fine.

  router:~# ip rule del fwmark 1
  router:~# ip rule add from 192.168.0.22 table test

  <doing some pings from the internal host>

  router:~# iptables -t mangle -L -n -x -v -Z
  Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         4      240            all  --  ppp4   *       141.24.12.2
0.0.0.0/0
         4      240 MARK       all  --  eth4   *       192.168.0.22
141.24.12.2        MARK set 0x1

  Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         4      240            all  --  ppp4   *       141.24.12.2
0.0.0.0/0

  Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination

  Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
      pkts      bytes target     prot opt in     out     source
destination
         4      500            all  --  *      eth4    141.24.12.2
0.0.0.0/0

router:~# iptables
iptables v1.2.6a: no command specified
Try `iptables -h' or 'iptables --help' for more information.
router:~# ip -V
ip utility, iproute2-ss010824
router:~# cat /proc/version
Linux version 2.4.22 (root@dslrouter) (gcc version 2.95.4 20011002 (Debian
prerelease)) #1 SMP Wed Nov 12 23:08:31 CET 2003


i am grateful for any help. thanks.

thomas hoeppler