Policy routing (fwmark-based) and local traffic...

Policy routing (fwmark-based) and local traffic...

[Marco Gaiarin]

I was trying to setup some policy routing (using iptables fwmark) for
local traffic; i've just in place rules for forward traffic (in
PREROUTING chain, mangle table) and works as expected.
EG:

 ulysses:~# ip rule show
 0:	from all lookup local 
 32762:	from all fwmark 0x30/0xf0 lookup FIBRA 
 32763:	from 192.168.178.2 lookup FIBRA 
 32764:	from all fwmark 0x20/0xf0 lookup ADSL2 
 32765:	from 194.243.231.54 lookup ADSL2 
 32766:	from all lookup main 
 32767:	from all lookup default 

and

 ulysses:~# iptables -t mangle -vnL PREROUTING
 Chain PREROUTING (policy ACCEPT 278K packets, 175M bytes)
  pkts bytes target     prot opt in     out     source               destination         
     0     0 MARK       tcp  --  *      *       0.0.0.0/0            173.194.79.109       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
     0     0 MARK       tcp  --  *      *       0.0.0.0/0            173.194.79.108       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
  1488  311K MARK       tcp  --  *      *       0.0.0.0/0            173.194.76.109       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
   143 99494 MARK       tcp  --  *      *       0.0.0.0/0            173.194.76.108       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
 [...]


I've tried to setup the same thing for local generated traffic but...
i've discovered that the 'routing decision' happen BEFORE iptables
tables (so, simply, fwmarks get ignored).


It is true or i'm missing something? Thanks.

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

Re: Policy routing (fwmark-based) and local traffic...

[Florian Westphal]

Marco Gaiarin <gaio@sv.lnf.it> wrote:
> I was trying to setup some policy routing (using iptables fwmark) for
> local traffic; i've just in place rules for forward traffic (in
> PREROUTING chain, mangle table) and works as expected.
> EG:
> 
>  ulysses:~# ip rule show
>  0:	from all lookup local 
>  32762:	from all fwmark 0x30/0xf0 lookup FIBRA 
>  32763:	from 192.168.178.2 lookup FIBRA 
>  32764:	from all fwmark 0x20/0xf0 lookup ADSL2 
>  32765:	from 194.243.231.54 lookup ADSL2 
>  32766:	from all lookup main 
>  32767:	from all lookup default 
> 
> and
> 
>  ulysses:~# iptables -t mangle -vnL PREROUTING
>  Chain PREROUTING (policy ACCEPT 278K packets, 175M bytes)
>   pkts bytes target     prot opt in     out     source               destination         
>      0     0 MARK       tcp  --  *      *       0.0.0.0/0            173.194.79.109       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
>      0     0 MARK       tcp  --  *      *       0.0.0.0/0            173.194.79.108       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
>   1488  311K MARK       tcp  --  *      *       0.0.0.0/0            173.194.76.109       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
>    143 99494 MARK       tcp  --  *      *       0.0.0.0/0            173.194.76.108       multiport dports 25,465,587,993,995 mark match 0x0/0xf0 MARK xset 0x20/0xf0
>  [...]
> 
> 
> I've tried to setup the same thing for local generated traffic but...
> i've discovered that the 'routing decision' happen BEFORE iptables
> tables (so, simply, fwmarks get ignored).
> 
> It is true or i'm missing something? Thanks.

Yes, however, mangle OUTPUT will re-route in case of fwmark change.

(same for nftables 'route' basechain type)

Re: Policy routing (fwmark-based) and local traffic...

[Marco Gaiarin]

Mandi! Florian Westphal
  In chel di` si favelave...

> Yes, however, mangle OUTPUT will re-route in case of fwmark change.

A-HA! Seemed a bit too strange to me... i've tried with:

	iptables -t mangle -I OUTPUT -d $gip -p tcp -m multiport --dports ${TCP_PORTS} -m mark --mark 0/$MASK -j MARK --set-mark $MARK/$MASK

and does not work; you say me that i've to use:

	iptables -t mangle -I OUTPUT -d $gip -p tcp -m multiport --dports ${TCP_PORTS} -j MARK --set-mark $MARK/$MASK

(eg: ignore eventually 'implicit' mark set by the routing decision) to
make it work?

're-route' happen automatically?


Thanks.

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

Re: Policy routing (fwmark-based) and local traffic...

[Florian Westphal]

Marco Gaiarin <gaio@sv.lnf.it> wrote:
> > Yes, however, mangle OUTPUT will re-route in case of fwmark change.
> 
> A-HA! Seemed a bit too strange to me... i've tried with:
> 
> 	iptables -t mangle -I OUTPUT -d $gip -p tcp -m multiport --dports ${TCP_PORTS} -m mark --mark 0/$MASK -j MARK --set-mark $MARK/$MASK

Works for me.

> and does not work; you say me that i've to use:
> 
> 	iptables -t mangle -I OUTPUT -d $gip -p tcp -m multiport --dports ${TCP_PORTS} -j MARK --set-mark $MARK/$MASK
> 
> (eg: ignore eventually 'implicit' mark set by the routing decision) to
> make it work?

Whats an implicit mark set by the routign decision?

> 're-route' happen automatically?

Yes, if mark or other attibutes (e.g. dscp) changed.

Re: Policy routing (fwmark-based) and local traffic...

[Marco Gaiarin]

Mandi! Florian Westphal
  In chel di` si favelave...

> > 	iptables -t mangle -I OUTPUT -d $gip -p tcp -m multiport --dports ${TCP_PORTS} -m mark --mark 0/$MASK -j MARK --set-mark $MARK/$MASK
> Works for me.

Ah. For me no. Could be the kernel/iptables version? Currently:

 ulysses:~# uname -a
 Linux ulysses 3.2.0-6-686-pae #1 SMP Debian 3.2.102-1 i686 GNU/Linux
 ulysses:~# dpkg -l | grep iptables
 ii  iptables                             1.4.14-3.1                              i386         administration tools for packet filtering and NAT



> > 	iptables -t mangle -I OUTPUT -d $gip -p tcp -m multiport --dports ${TCP_PORTS} -j MARK --set-mark $MARK/$MASK
> Whats an implicit mark set by the routign decision?

I was only supposing... sorry...

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

Re: Policy routing (fwmark-based) and local traffic...

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

On 7/30/19 6:32 AM, Marco Gaiarin wrote:
> I was trying to setup some policy routing (using iptables fwmark) 
> for local traffic; i've just in place rules for forward traffic 
> (in PREROUTING chain, mangle table) and works as expected.

Please provide a high level overview of why you are wanting to use PBR. 
What is the purpose behind the policies?  It looks like you might be 
wanting to cause traffic to / from specific IPs to use different routes 
from the system default.

If that is the simple case, I'd be tempted to see if the l3-mdev can do 
what you are wanting to do.  I.e. put the interface(s) associated with 
FIBRA into one l3-mdev, put the interface(s) associated with ADSL2 into 
another l3-mdev, and leave the rest of the interface(s) alone.

L3-mdev will cause any traffic from the associated interfaces to use the 
associated routing tables.  It might mean that you don't need to worry 
about this yourself.

I'm also curious to have an idea of what's in your FIBRA, ADSL2, main, 
and default routing tables.

> It is true or i'm missing something? Thanks.

Maybe.  I can't tell if l3-mdev would be a good fit for your use case or 
not.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Policy routing (fwmark-based) and local traffic...

[Marco Gaiarin]

Mandi! Grant Taylor
  In chel di` si favelave...

> Please provide a high level overview of why you are wanting to use PBR. What

Ok. I manage some gateway that handle two or more connection to ISP; i
balance traffic between routes, and sometimes via policy routing i
'bind' some traffic to some interfaces (forward traffic, so PREROUTING
in mangle).

This time i needed to bind local generated traffic, because one of the
two line use dynamic IP and the service i use need static IP (they have
some ACLs).

Initially i've simply applied the same policy i was used to, but does
not work (clearly, PREROUTING is for forwarded traffic, not or local
generated one! Stupid me! ;).
Then i've tried with OUTPUT, but with the same result.

Now i'm using explicit routing, and clearly works. But i was curious,
so i've posted here.


> If that is the simple case, I'd be tempted to see if the l3-mdev can do what
> you are wanting to do.

Do you mean this?
	https://netdevconf.org/1.2/papers/ahern-what-is-l3mdev-paper.pdf

I've no kernel 4.4, but i'm using load balancing with:
	https://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html

and i suppose is a similar concept...

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

Re: Policy routing (fwmark-based) and local traffic...

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 1617 bytes --]

On 7/31/19 3:10 AM, Marco Gaiarin wrote:
> Ok. I manage some gateway that handle two or more connection to ISP; 
> i balance traffic between routes, and sometimes via policy routing i 
> 'bind' some traffic to some interfaces (forward traffic, so PREROUTING 
> in mangle).

Thank you for the high level.  I figured that was the case, but I wanted 
to ask to be sure.

> This time i needed to bind local generated traffic, because one of 
> the two line use dynamic IP and the service i use need static IP 
> (they have some ACLs).

That makes perfect sense.

I do wonder if a static route via the ISP connection with the static IP 
might suffice.  (Which it sounds like you might now be doing that.)

> Initially i've simply applied the same policy i was used to, but 
> does not work (clearly, PREROUTING is for forwarded traffic, not or 
> local generated one! Stupid me! ;).  Then i've tried with OUTPUT, 
> but with the same result.
> 
> Now i'm using explicit routing, and clearly works. But i was curious, 
> so i've posted here.

*nod*

> Do you mean this?
> 	https://netdevconf.org/1.2/papers/ahern-what-is-l3mdev-paper.pdf

Yes.

> I've no kernel 4.4, but i'm using load balancing with:
> 	https://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html
> 
> and i suppose is a similar concept...

Yes.

I think l3mdev takes that concept and extends it such that you don't 
need to worry about rules / fwmarks / etc.  Instead, the l3mdev plumbing 
takes care of choosing what routing table is used automatically.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]