* fail in the connmark load-balancing
@ 2012-02-11 20:19 Usuário do Sistema
2012-02-12 22:10 ` Andrew Beverley
0 siblings, 1 reply; 11+ messages in thread
From: Usuário do Sistema @ 2012-02-11 20:19 UTC (permalink / raw)
To: Mail List - Netfilter
Hello,
I've just deployed the load balance in the my firewall iptables
1.4.3.1 as How to below:
http://www.sysresccd.org/Sysresccd-Networking-EN-Iptables-and-netfilter-load-balancing-using-connmark
some doubts has grown in my mind....the main is:
I need add follow line to occur the load balance ?? because without it
my load balance itsn't work!! and when I put equalize line I have some
problems with forward packages between my two gateway interfaces there
is some sites I'm unable access because occur the load balance even
mark and track packages as how to above.
ip route add default scope global equalize nexthop via x.y.t.z1 weight
2 nexthop via x.y.t.z2 weight 2
so...I wish remove the "equalize" and keep the load balance as how to.
it's possible ??
thanks
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-11 20:19 fail in the connmark load-balancing Usuário do Sistema
@ 2012-02-12 22:10 ` Andrew Beverley
2012-02-13 11:19 ` Usuário do Sistema
0 siblings, 1 reply; 11+ messages in thread
From: Andrew Beverley @ 2012-02-12 22:10 UTC (permalink / raw)
To: Usuário do Sistema; +Cc: Mail List - Netfilter
On Sat, 2012-02-11 at 18:19 -0200, Usuário do Sistema wrote:
> Hello,
>
> I've just deployed the load balance in the my firewall iptables
> 1.4.3.1 as How to below:
>
> http://www.sysresccd.org/Sysresccd-Networking-EN-Iptables-and-netfilter-load-balancing-using-connmark
> I need add follow line to occur the load balance ??
[...]
> ip route add default scope global equalize nexthop via x.y.t.z1 weight
> 2 nexthop via x.y.t.z2 weight 2
No, you don't need that line when doing load sharing with the method
described above. That will break the sharing per-connection, which is
obviously what you are trying to achieve.
If it's not working, there must be another problem. Please show the
output of "ip rule show", "ip ro" and "ip ro show table <table>" for
each of your tables where <table> is the name of the tables.
Andy
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-12 22:10 ` Andrew Beverley
@ 2012-02-13 11:19 ` Usuário do Sistema
2012-02-13 22:03 ` Usuário do Sistema
0 siblings, 1 reply; 11+ messages in thread
From: Usuário do Sistema @ 2012-02-13 11:19 UTC (permalink / raw)
To: Andrew Beverley; +Cc: Mail List - Netfilter
Hi, current I'm unable access my firewall but follow my main part of
configuration script. does the main table routing has have the two
Gateways ?? because my main table has only one as follow.
[root@mtjve ~]# ip ro
192.168.215.0/24 dev eth1 proto kernel scope link src 192.168.215.1
192.168.217.0/24 dev eth2 proto kernel scope link src 192.168.217.254
192.168.216.0/24 dev eth0 proto kernel scope link src 192.168.216.254
default via 192.168.216.1 dev eth0
My script:
iptables -t mangle -N CONNMARK1
iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
iptables -t mangle -N CONNMARK2
iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
Wan Interfaces:
eth2:192.168.217.254
eth0:192.168.216.254
Lan Interface:
eth1:192.168.215.1
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j SNAT --to-source 192.168.217.254
iptables -t nat -N SNAT2
iptables -t nat -A SNAT2 -j SNAT --to-source 192.168.216.254
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state NEW -m statistic --mode nth --every 2 --packet 0 -j CONNMARK1
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state NEW -m statistic --mode nth --every 2 --packet 1 -j CONNMARK2
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state ESTABLISHED,RELATED -j RESTOREMARK
iptables -t nat -A POSTROUTING -o eth2 -j SNAT1
iptables -t nat -A POSTROUTING -o eth0 -j SNAT2
ip rule del fwmark 2
ip rule del fwmark 1
ip route add default via 192.168.217.1 table 1
ip route add default via 192.168.216.1 table 2
ip rule add fwmark 1 table 1
ip rule add fwmark 2 table 2
ip route flush cache
thanks!
Em 12 de fevereiro de 2012 20:10, Andrew Beverley <andy@andybev.com> escreveu:
> On Sat, 2012-02-11 at 18:19 -0200, Usuário do Sistema wrote:
>> Hello,
>>
>> I've just deployed the load balance in the my firewall iptables
>> 1.4.3.1 as How to below:
>>
>> http://www.sysresccd.org/Sysresccd-Networking-EN-Iptables-and-netfilter-load-balancing-using-connmark
>
>> I need add follow line to occur the load balance ??
>
> [...]
>
>> ip route add default scope global equalize nexthop via x.y.t.z1 weight
>> 2 nexthop via x.y.t.z2 weight 2
>
> No, you don't need that line when doing load sharing with the method
> described above. That will break the sharing per-connection, which is
> obviously what you are trying to achieve.
>
> If it's not working, there must be another problem. Please show the
> output of "ip rule show", "ip ro" and "ip ro show table <table>" for
> each of your tables where <table> is the name of the tables.
>
> Andy
>
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-13 11:19 ` Usuário do Sistema
@ 2012-02-13 22:03 ` Usuário do Sistema
2012-02-25 10:53 ` Andrew Beverley
0 siblings, 1 reply; 11+ messages in thread
From: Usuário do Sistema @ 2012-02-13 22:03 UTC (permalink / raw)
To: Andrew Beverley; +Cc: Mail List - Netfilter
Andrew, I have done new test but my load-balance itsn't work!
follow what you asked me in the last message:
[root@mtjve sbin]# ip ro
192.168.215.0/24 dev eth1 proto kernel scope link src 192.168.215.1
192.168.217.0/24 dev eth2 proto kernel scope link src 192.168.217.254
192.168.216.0/24 dev eth0 proto kernel scope link src 192.168.216.254
default via 192.168.216.1 dev eth0
[root@mtjve sbin]# ip rule show
0: from all lookup 255
32763: from all fwmark 0x2 lookup gvttelecom
32764: from all fwmark 0x1 lookup oitelecom
32765: from 192.168.217.254 lookup oitelecom
32766: from all lookup main
32767: from all lookup default
[root@mtjve sbin]# ip ro show table oitelecom
192.168.217.0 via 192.168.217.1 dev eth2
192.168.217.254 via 192.168.217.1 dev eth2
default via 192.168.217.1 dev eth2
[root@mtjve sbin]# ip ro show table gvttelecom
192.168.216.0 via 192.168.216.1 dev eth0
default via 192.168.216.1 dev eth0
I think here is the issue! there are two PREROUTING rule that they
should do the load-balance but I wonder when the socond rule (
CONNMARK2 ) coming use it ?? I think never because data flows always
going to into at the first rule ( CONNMARK1 ). so...the second rule
never is using! today I have done a test download from three diferente
sites and the load-balance it wasn't work.
eth1: is my lan interface
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state NEW -m statistic --mode nth --every 2 --packet 0 -j CONNMARK1
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state NEW -m statistic --mode nth --every 2 --packet 1 -j CONNMARK2
any tips are welcome!
thanks
Em 13 de fevereiro de 2012 09:19, Usuário do Sistema
<maiconlp@ig.com.br> escreveu:
> Hi, current I'm unable access my firewall but follow my main part of
> configuration script. does the main table routing has have the two
> Gateways ?? because my main table has only one as follow.
>
>
> [root@mtjve ~]# ip ro
> 192.168.215.0/24 dev eth1 proto kernel scope link src 192.168.215.1
> 192.168.217.0/24 dev eth2 proto kernel scope link src 192.168.217.254
> 192.168.216.0/24 dev eth0 proto kernel scope link src 192.168.216.254
> default via 192.168.216.1 dev eth0
>
>
> My script:
>
> iptables -t mangle -N CONNMARK1
> iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
> iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
>
> iptables -t mangle -N CONNMARK2
> iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
> iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
>
> Wan Interfaces:
> eth2:192.168.217.254
> eth0:192.168.216.254
>
> Lan Interface:
> eth1:192.168.215.1
>
> iptables -t nat -N SNAT1
> iptables -t nat -A SNAT1 -j SNAT --to-source 192.168.217.254
>
> iptables -t nat -N SNAT2
> iptables -t nat -A SNAT2 -j SNAT --to-source 192.168.216.254
>
>
> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
> --state NEW -m statistic --mode nth --every 2 --packet 0 -j CONNMARK1
>
> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
> --state NEW -m statistic --mode nth --every 2 --packet 1 -j CONNMARK2
>
> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
> --state ESTABLISHED,RELATED -j RESTOREMARK
>
> iptables -t nat -A POSTROUTING -o eth2 -j SNAT1
>
> iptables -t nat -A POSTROUTING -o eth0 -j SNAT2
>
>
> ip rule del fwmark 2
> ip rule del fwmark 1
>
> ip route add default via 192.168.217.1 table 1
> ip route add default via 192.168.216.1 table 2
>
> ip rule add fwmark 1 table 1
> ip rule add fwmark 2 table 2
>
> ip route flush cache
>
>
>
> thanks!
>
>
>
>
> Em 12 de fevereiro de 2012 20:10, Andrew Beverley <andy@andybev.com> escreveu:
>> On Sat, 2012-02-11 at 18:19 -0200, Usuário do Sistema wrote:
>>> Hello,
>>>
>>> I've just deployed the load balance in the my firewall iptables
>>> 1.4.3.1 as How to below:
>>>
>>> http://www.sysresccd.org/Sysresccd-Networking-EN-Iptables-and-netfilter-load-balancing-using-connmark
>>
>>> I need add follow line to occur the load balance ??
>>
>> [...]
>>
>>> ip route add default scope global equalize nexthop via x.y.t.z1 weight
>>> 2 nexthop via x.y.t.z2 weight 2
>>
>> No, you don't need that line when doing load sharing with the method
>> described above. That will break the sharing per-connection, which is
>> obviously what you are trying to achieve.
>>
>> If it's not working, there must be another problem. Please show the
>> output of "ip rule show", "ip ro" and "ip ro show table <table>" for
>> each of your tables where <table> is the name of the tables.
>>
>> Andy
>>
>>
>>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-13 22:03 ` Usuário do Sistema
@ 2012-02-25 10:53 ` Andrew Beverley
2012-02-27 16:40 ` Usuário do Sistema
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Andrew Beverley @ 2012-02-25 10:53 UTC (permalink / raw)
To: Usuário do Sistema; +Cc: Mail List - Netfilter
> >> If it's not working, there must be another problem. Please show the
> >> output of "ip rule show", "ip ro" and "ip ro show table <table>" for
> >> each of your tables where <table> is the name of the tables.
>
> [root@mtjve sbin]# ip ro
> 192.168.215.0/24 dev eth1 proto kernel scope link src 192.168.215.1
> 192.168.217.0/24 dev eth2 proto kernel scope link src 192.168.217.254
> 192.168.216.0/24 dev eth0 proto kernel scope link src 192.168.216.254
> default via 192.168.216.1 dev eth0
>
>
> [root@mtjve sbin]# ip rule show
> 0: from all lookup 255
> 32763: from all fwmark 0x2 lookup gvttelecom
> 32764: from all fwmark 0x1 lookup oitelecom
> 32765: from 192.168.217.254 lookup oitelecom
> 32766: from all lookup main
> 32767: from all lookup default
>
> [root@mtjve sbin]# ip ro show table oitelecom
> 192.168.217.0 via 192.168.217.1 dev eth2
> 192.168.217.254 via 192.168.217.1 dev eth2
> default via 192.168.217.1 dev eth2
>
> [root@mtjve sbin]# ip ro show table gvttelecom
> 192.168.216.0 via 192.168.216.1 dev eth0
> default via 192.168.216.1 dev eth0
>
>
> I think here is the issue! there are two PREROUTING rule that they
> should do the load-balance but I wonder when the socond rule (
> CONNMARK2 ) coming use it ?? I think never because data flows always
> going to into at the first rule ( CONNMARK1 ). so...the second rule
> never is using! today I have done a test download from three diferente
> sites and the load-balance it wasn't work.
>
>
> eth1: is my lan interface
>
> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
> --state NEW -m statistic --mode nth --every 2 --packet 0 -j CONNMARK1
> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
> --state NEW -m statistic --mode nth --every 2 --packet 1 -j CONNMARK2
I can't see any restoration of the marks. You are marking the connection
when it is NEW, but "ip rule fwmark" will match on the packet's mark, so
you need to copy the connection mark to the packet mark with "-j
CONNMARK --restore-mark".
Andy
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-25 10:53 ` Andrew Beverley
@ 2012-02-27 16:40 ` Usuário do Sistema
2012-02-27 17:07 ` Usuário do Sistema
2012-02-27 18:15 ` Usuário do Sistema
2 siblings, 0 replies; 11+ messages in thread
From: Usuário do Sistema @ 2012-02-27 16:40 UTC (permalink / raw)
To: Andrew Beverley; +Cc: Mail List - Netfilter
Thank you Andy, really I have figure out this issue.....go ahead
I forgotten include my CONNMARK RESTORE rule at the last message....
so follow my whole script.
there are two PREROUTING rule for new connections that they should do
the load-balance but I wonder when the socond rule CONNMARK2 coming
use it ?? I think never because data flows always going to into at the
first rule ( CONNMARK1 ). so the second rule never is using! I think
the statistc module could do that rotate at the use rules...but
doesn't work!
eth1: LAN Interface
eth0: WAN1
eth2: WAN2
#!/bin/bash
# flush all iptables entries
iptables -t filter -F
iptables -t filter -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t filter -P INPUT ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT
iptables -t mangle -N CONNMARK1
iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
iptables -t mangle -N CONNMARK2
iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j SNAT --to-source 192.168.217.254
iptables -t nat -N SNAT2
iptables -t nat -A SNAT2 -j SNAT --to-source 192.168.216.254
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state ESTABLISHED,RELATED -j RESTOREMARK
iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
statistic --mode nth --every 2 --packet 0 -j CONNMARK1
iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
statistic --mode nth --every 2 --packet 1 -j CONNMARK2
iptables -t nat -A POSTROUTING -o eth2 -j SNAT1
iptables -t nat -A POSTROUTING -o eth0 -j SNAT2
ip route add 192.168.217.0 via 192.168.217.1 table oitelecom
ip route add 192.168.216.0 via 192.168.216.1 table gvttelecom
ip route add default via 192.168.217.1 table oitelecom
ip route add default via 192.168.216.1 table gvttelecom
ip rule del from 192.168.217.254 table oitelecom
ip rule add from 192.168.217.254 table oitelecom
ip rule del fwmark 1 table oitelecom
ip rule del fwmark 2 table gvttelecom
ip rule add fwmark 1 table oitelecom
ip rule add fwmark 2 table gvttelecom
ip route flush cache
thanks....any tips is welcome.
Em 25 de fevereiro de 2012 08:53, Andrew Beverley <andy@andybev.com> escreveu:
>> >> If it's not working, there must be another problem. Please show the
>> >> output of "ip rule show", "ip ro" and "ip ro show table <table>" for
>> >> each of your tables where <table> is the name of the tables.
>
>>
>> [root@mtjve sbin]# ip ro
>> 192.168.215.0/24 dev eth1 proto kernel scope link src 192.168.215.1
>> 192.168.217.0/24 dev eth2 proto kernel scope link src 192.168.217.254
>> 192.168.216.0/24 dev eth0 proto kernel scope link src 192.168.216.254
>> default via 192.168.216.1 dev eth0
>>
>>
>> [root@mtjve sbin]# ip rule show
>> 0: from all lookup 255
>> 32763: from all fwmark 0x2 lookup gvttelecom
>> 32764: from all fwmark 0x1 lookup oitelecom
>> 32765: from 192.168.217.254 lookup oitelecom
>> 32766: from all lookup main
>> 32767: from all lookup default
>>
>> [root@mtjve sbin]# ip ro show table oitelecom
>> 192.168.217.0 via 192.168.217.1 dev eth2
>> 192.168.217.254 via 192.168.217.1 dev eth2
>> default via 192.168.217.1 dev eth2
>>
>> [root@mtjve sbin]# ip ro show table gvttelecom
>> 192.168.216.0 via 192.168.216.1 dev eth0
>> default via 192.168.216.1 dev eth0
>>
>>
>> I think here is the issue! there are two PREROUTING rule that they
>> should do the load-balance but I wonder when the socond rule (
>> CONNMARK2 ) coming use it ?? I think never because data flows always
>> going to into at the first rule ( CONNMARK1 ). so...the second rule
>> never is using! today I have done a test download from three diferente
>> sites and the load-balance it wasn't work.
>>
>>
>> eth1: is my lan interface
>>
>> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
>> --state NEW -m statistic --mode nth --every 2 --packet 0 -j CONNMARK1
>> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
>> --state NEW -m statistic --mode nth --every 2 --packet 1 -j CONNMARK2
>
> I can't see any restoration of the marks. You are marking the connection
> when it is NEW, but "ip rule fwmark" will match on the packet's mark, so
> you need to copy the connection mark to the packet mark with "-j
> CONNMARK --restore-mark".
>
> Andy
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-25 10:53 ` Andrew Beverley
2012-02-27 16:40 ` Usuário do Sistema
@ 2012-02-27 17:07 ` Usuário do Sistema
2012-02-27 18:15 ` Usuário do Sistema
2 siblings, 0 replies; 11+ messages in thread
From: Usuário do Sistema @ 2012-02-27 17:07 UTC (permalink / raw)
To: Andrew Beverley; +Cc: Mail List - Netfilter
Thank you Andy, really I have figure out this issue.....go ahead
I forgotten include my CONNMARK RESTORE rule at the last message....
so follow my whole script.
there are two PREROUTING rule for new connections that they should do
the load-balance but I wonder when the socond rule CONNMARK2 coming
use it ?? I think never because data flows always going to into at the
first rule ( CONNMARK1 ). so the second rule never is using! I think
the statistc module could do that rotate at the use rules...but
doesn't work!
eth1: LAN Interface
eth0: WAN1
eth2: WAN2
#!/bin/bash
# flush all iptables entries
iptables -t filter -F
iptables -t filter -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t filter -P INPUT ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT
iptables -t mangle -N CONNMARK1
iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
iptables -t mangle -N CONNMARK2
iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j SNAT --to-source 192.168.217.254
iptables -t nat -N SNAT2
iptables -t nat -A SNAT2 -j SNAT --to-source 192.168.216.254
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state ESTABLISHED,RELATED -j RESTOREMARK
iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
statistic --mode nth --every 2 --packet 0 -j CONNMARK1
iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
statistic --mode nth --every 2 --packet 1 -j CONNMARK2
iptables -t nat -A POSTROUTING -o eth2 -j SNAT1
iptables -t nat -A POSTROUTING -o eth0 -j SNAT2
ip route add 192.168.217.0 via 192.168.217.1 table oitelecom
ip route add 192.168.216.0 via 192.168.216.1 table gvttelecom
ip route add default via 192.168.217.1 table oitelecom
ip route add default via 192.168.216.1 table gvttelecom
ip rule del from 192.168.217.254 table oitelecom
ip rule add from 192.168.217.254 table oitelecom
ip rule del fwmark 1 table oitelecom
ip rule del fwmark 2 table gvttelecom
ip rule add fwmark 1 table oitelecom
ip rule add fwmark 2 table gvttelecom
ip route flush cache
thanks....any tips is welcome.
Em 25 de fevereiro de 2012 08:53, Andrew Beverley <andy@andybev.com> escreveu:
>> >> If it's not working, there must be another problem. Please show the
>> >> output of "ip rule show", "ip ro" and "ip ro show table <table>" for
>> >> each of your tables where <table> is the name of the tables.
>
>>
>> [root@mtjve sbin]# ip ro
>> 192.168.215.0/24 dev eth1 proto kernel scope link src 192.168.215.1
>> 192.168.217.0/24 dev eth2 proto kernel scope link src 192.168.217.254
>> 192.168.216.0/24 dev eth0 proto kernel scope link src 192.168.216.254
>> default via 192.168.216.1 dev eth0
>>
>>
>> [root@mtjve sbin]# ip rule show
>> 0: from all lookup 255
>> 32763: from all fwmark 0x2 lookup gvttelecom
>> 32764: from all fwmark 0x1 lookup oitelecom
>> 32765: from 192.168.217.254 lookup oitelecom
>> 32766: from all lookup main
>> 32767: from all lookup default
>>
>> [root@mtjve sbin]# ip ro show table oitelecom
>> 192.168.217.0 via 192.168.217.1 dev eth2
>> 192.168.217.254 via 192.168.217.1 dev eth2
>> default via 192.168.217.1 dev eth2
>>
>> [root@mtjve sbin]# ip ro show table gvttelecom
>> 192.168.216.0 via 192.168.216.1 dev eth0
>> default via 192.168.216.1 dev eth0
>>
>>
>> I think here is the issue! there are two PREROUTING rule that they
>> should do the load-balance but I wonder when the socond rule (
>> CONNMARK2 ) coming use it ?? I think never because data flows always
>> going to into at the first rule ( CONNMARK1 ). so...the second rule
>> never is using! today I have done a test download from three diferente
>> sites and the load-balance it wasn't work.
>>
>>
>> eth1: is my lan interface
>>
>> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
>> --state NEW -m statistic --mode nth --every 2 --packet 0 -j CONNMARK1
>> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
>> --state NEW -m statistic --mode nth --every 2 --packet 1 -j CONNMARK2
>
> I can't see any restoration of the marks. You are marking the connection
> when it is NEW, but "ip rule fwmark" will match on the packet's mark, so
> you need to copy the connection mark to the packet mark with "-j
> CONNMARK --restore-mark".
>
> Andy
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-25 10:53 ` Andrew Beverley
2012-02-27 16:40 ` Usuário do Sistema
2012-02-27 17:07 ` Usuário do Sistema
@ 2012-02-27 18:15 ` Usuário do Sistema
2012-02-28 11:16 ` Gáspár Lajos
2 siblings, 1 reply; 11+ messages in thread
From: Usuário do Sistema @ 2012-02-27 18:15 UTC (permalink / raw)
To: Andrew Beverley; +Cc: Mail List - Netfilter
I can't see any restoration of the marks. You are marking the connection
when it is NEW, but "ip rule fwmark" will match on the packet's mark, so
you need to copy the connection mark to the packet mark with "-j
CONNMARK --restore-mark".
Thank you Andy, really I have figure out this issue.....go ahead
I forgotten include my CONNMARK RESTORE rule at the last message....
so follow my whole script.
there are two PREROUTING rule for new connections that they should do
the load-balance but I wonder when the socond rule CONNMARK2 coming
use it ?? I think never because data flows always going to into at the
first rule ( CONNMARK1 ). so the second rule never is using! I think
the statistc module could do that rotate at the use rules...but
doesn't work!
eth1: LAN Interface
eth0: WAN1
eth2: WAN2
#!/bin/bash
# flush all iptables entries
iptables -t filter -F
iptables -t filter -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t filter -P INPUT ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT
iptables -t mangle -N CONNMARK1
iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
iptables -t mangle -N CONNMARK2
iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j SNAT --to-source 192.168.217.254
iptables -t nat -N SNAT2
iptables -t nat -A SNAT2 -j SNAT --to-source 192.168.216.254
iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
--state ESTABLISHED,RELATED -j RESTOREMARK
iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
statistic --mode nth --every 2 --packet 0 -j CONNMARK1
iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
statistic --mode nth --every 2 --packet 1 -j CONNMARK2
iptables -t nat -A POSTROUTING -o eth2 -j SNAT1
iptables -t nat -A POSTROUTING -o eth0 -j SNAT2
ip route add 192.168.217.0 via 192.168.217.1 table oitelecom
ip route add 192.168.216.0 via 192.168.216.1 table gvttelecom
ip route add default via 192.168.217.1 table oitelecom
ip route add default via 192.168.216.1 table gvttelecom
ip rule del from 192.168.217.254 table oitelecom
ip rule add from 192.168.217.254 table oitelecom
ip rule del fwmark 1 table oitelecom
ip rule del fwmark 2 table gvttelecom
ip rule add fwmark 1 table oitelecom
ip rule add fwmark 2 table gvttelecom
ip route flush cache
thanks....any tips is welcome.
Em 25 de fevereiro de 2012 08:53, Andrew Beverley <andy@andybev.com> escreveu:
>> >> If it's not working, there must be another problem. Please show the
>> >> output of "ip rule show", "ip ro" and "ip ro show table <table>" for
>> >> each of your tables where <table> is the name of the tables.
>
>>
>> [root@mtjve sbin]# ip ro
>> 192.168.215.0/24 dev eth1 proto kernel scope link src 192.168.215.1
>> 192.168.217.0/24 dev eth2 proto kernel scope link src 192.168.217.254
>> 192.168.216.0/24 dev eth0 proto kernel scope link src 192.168.216.254
>> default via 192.168.216.1 dev eth0
>>
>>
>> [root@mtjve sbin]# ip rule show
>> 0: from all lookup 255
>> 32763: from all fwmark 0x2 lookup gvttelecom
>> 32764: from all fwmark 0x1 lookup oitelecom
>> 32765: from 192.168.217.254 lookup oitelecom
>> 32766: from all lookup main
>> 32767: from all lookup default
>>
>> [root@mtjve sbin]# ip ro show table oitelecom
>> 192.168.217.0 via 192.168.217.1 dev eth2
>> 192.168.217.254 via 192.168.217.1 dev eth2
>> default via 192.168.217.1 dev eth2
>>
>> [root@mtjve sbin]# ip ro show table gvttelecom
>> 192.168.216.0 via 192.168.216.1 dev eth0
>> default via 192.168.216.1 dev eth0
>>
>>
>> I think here is the issue! there are two PREROUTING rule that they
>> should do the load-balance but I wonder when the socond rule (
>> CONNMARK2 ) coming use it ?? I think never because data flows always
>> going to into at the first rule ( CONNMARK1 ). so...the second rule
>> never is using! today I have done a test download from three diferente
>> sites and the load-balance it wasn't work.
>>
>>
>> eth1: is my lan interface
>>
>> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
>> --state NEW -m statistic --mode nth --every 2 --packet 0 -j CONNMARK1
>> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
>> --state NEW -m statistic --mode nth --every 2 --packet 1 -j CONNMARK2
>
> I can't see any restoration of the marks. You are marking the connection
> when it is NEW, but "ip rule fwmark" will match on the packet's mark, so
> you need to copy the connection mark to the packet mark with "-j
> CONNMARK --restore-mark".
>
> Andy
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-27 18:15 ` Usuário do Sistema
@ 2012-02-28 11:16 ` Gáspár Lajos
2012-03-02 21:24 ` Usuário do Sistema
0 siblings, 1 reply; 11+ messages in thread
From: Gáspár Lajos @ 2012-02-28 11:16 UTC (permalink / raw)
To: Usuário do Sistema; +Cc: Andrew Beverley, Mail List - Netfilter
Hi,
A few comments...
2012-02-27 19:15 keltezéssel, Usuário do Sistema írta:
> eth1: LAN Interface
> eth0: WAN1
> eth2: WAN2
>
> #!/bin/bash
>
> # flush all iptables entries
> iptables -t filter -F
> iptables -t filter -X
> iptables -t nat -F
> iptables -t nat -X
> iptables -t mangle -F
> iptables -t mangle -X
> iptables -t filter -P INPUT ACCEPT
> iptables -t filter -P OUTPUT ACCEPT
> iptables -t filter -P FORWARD ACCEPT
By default the policies are set to ACCEPT...
I would set them to DROP and I would write my own "ACCEPT-ing" rules...
iptables -t filter -P INPUT DROP
iptables -t filter -P OUTPUT DROP
iptables -t filter -P FORWARD DROP
iptables -t filter -A INPUT -j ACCEPT -i lo
iptables -t filter -A INPUT -j ACCEPT -s {trusted newtork}
iptables -t filter -A OUTPUT -j ACCEPT -o lo
But don't change them if you don't want to filter the connections... :D
I would ACCEPT every packet (as the first rule) that comes/goes on the
lo interface... (in the PREROUTING/INPUT/OUTPUT/POSTROUTING chains in
the raw/mangle/filter tables)
> iptables -t mangle -N CONNMARK1
> iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
> iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
>
> iptables -t mangle -N CONNMARK2
> iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
> iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
>
> iptables -t mangle -N RESTOREMARK
> iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
> --state ESTABLISHED,RELATED -j RESTOREMARK
> iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
> statistic --mode nth --every 2 --packet 0 -j CONNMARK1
> iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
> statistic --mode nth --every 2 --packet 1 -j CONNMARK2
Hmm... :D
iptables -t mangle -N MARKS
iptables -t mangle -A MARKS - RETURN -m mark ! --mark 0/3
iptables -t mangle -A MARKS -j MARK --set-mark 3/3 -m state --state
ESTABLISED,RELATED
iptables -t mangle -A MARKS -j MARK --set-mark 1/3 -m mark --mark 0/3 -m
mode statistic --mode nth --every 2
iptables -t mangle -A MARKS -j MARK --set-mark 2/3 -m mark --mark 0/3
iptables -t mangle -A PREROUTING -j ACCEPT -i lo
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -j MARKS
iptables -t mangle -A INPUT -j ACCEPT -i lo (put this as the first rule
in the INPUT...)
iptables -t mangle -A INPUT -j CONNMARK --save-mark (put this as the
last rule in the INPUT...)
iptables -t mangle -A OUTPUT -j ACCEPT -o lo
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -j MARKS (use this if you want to
load-balance the traffic that originates from you firewall)
iptables -t mangle -A POSTROUTING -j ACCEPT -o lo (put this as the first
rule in the POSTROUTING...)
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark (put this as
the last rule in the POSTROUTING...)
> iptables -t nat -N SNAT1
> iptables -t nat -A SNAT1 -j SNAT --to-source 192.168.217.254
>
> iptables -t nat -N SNAT2
> iptables -t nat -A SNAT2 -j SNAT --to-source 192.168.216.254
> iptables -t nat -A POSTROUTING -o eth2 -j SNAT1
> iptables -t nat -A POSTROUTING -o eth0 -j SNAT2
How do you like these "one-liners"?
iptables -t nat -A POSTROUTING -j SNAT -o eth0 ! -s 192.168.216.254
--to-source 192.168.216.254
iptables -t nat -A POSTROUTING -j SNAT -o eth2 ! -s 192.168.217.254
--to-source 192.168.217.254
> ip route add 192.168.217.0 via 192.168.217.1 table oitelecom
> ip route add 192.168.216.0 via 192.168.216.1 table gvttelecom
> ip route add default via 192.168.217.1 table oitelecom
> ip route add default via 192.168.216.1 table gvttelecom
Maybe it is better:
ip route add default via 192.168.216.1 src 192.168.216.254 dev eth0
table gvttelecom
ip route add default via 192.168.217.1 src 192.168.217.254 dev eth2
table oitelecom
> ip rule del from 192.168.217.254 table oitelecom
> ip rule add from 192.168.217.254 table oitelecom
>
> ip rule del fwmark 1 table oitelecom
> ip rule del fwmark 2 table gvttelecom
>
> ip rule add fwmark 1 table oitelecom
> ip rule add fwmark 2 table gvttelecom
Use mask in marks:
ip rule add fwmark 1/3 table oitelecom
ip rule add fwmark 2/3 table gvttelecom
Maybe you need to copy other local routes:
ip route show table main | grep -E
'(^10\.|^172\.1[6-9]\.|^172\.2[0-9]\.|^172\.3[01]\.|^192\.168\.)' |
while read ROUTE
do
ip route add table ovtelecom ${ROUTE} 2>/dev/null
ip route add table gvttelecom ${ROUTE} 2>/dev/null
done
Be carefull with this last one, as it copies the 192.168.x.x routes too!!!!!
> ip route flush cache
>
>
> thanks....any tips is welcome.
Swifty
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-02-28 11:16 ` Gáspár Lajos
@ 2012-03-02 21:24 ` Usuário do Sistema
2012-03-05 13:34 ` Gáspár Lajos
0 siblings, 1 reply; 11+ messages in thread
From: Usuário do Sistema @ 2012-03-02 21:24 UTC (permalink / raw)
To: Gáspár Lajos; +Cc: Andrew Beverley, Mail List - Netfilter
Hi,
iptables -t mangle -N MARKS
iptables -t mangle -A MARKS - RETURN -m mark ! --mark 0/3
iptables -t mangle -A MARKS -j MARK --set-mark 3/3 -m state --state
ESTABLISED,RELATED
iptables -t mangle -A MARKS -j MARK --set-mark 1/3 -m mark --mark 0/3
-m mode statistic --mode nth --every 2
iptables -t mangle -A MARKS -j MARK --set-mark 2/3 -m mark --mark 0/3
sorry, but I did understand the mark x/y as you wrote above. why did
you use --set-mark 1 or --set-mark 2 ??
I think when it's use the mark x/y y is an mask but I'dont know what
is the match ?!?! with someone has a doc about that please post here.
I haven't done test yet with your suggested Swifty. I'll tell you when I do.
thanks
Em 28 de fevereiro de 2012 08:16, Gáspár Lajos <swifty@freemail.hu> escreveu:
> Hi,
>
> A few comments...
>
> 2012-02-27 19:15 keltezéssel, Usuário do Sistema írta:
>
>> eth1: LAN Interface
>> eth0: WAN1
>> eth2: WAN2
>>
>> #!/bin/bash
>>
>> # flush all iptables entries
>> iptables -t filter -F
>> iptables -t filter -X
>> iptables -t nat -F
>> iptables -t nat -X
>> iptables -t mangle -F
>> iptables -t mangle -X
>> iptables -t filter -P INPUT ACCEPT
>> iptables -t filter -P OUTPUT ACCEPT
>> iptables -t filter -P FORWARD ACCEPT
>
> By default the policies are set to ACCEPT...
> I would set them to DROP and I would write my own "ACCEPT-ing" rules...
>
> iptables -t filter -P INPUT DROP
> iptables -t filter -P OUTPUT DROP
> iptables -t filter -P FORWARD DROP
>
> iptables -t filter -A INPUT -j ACCEPT -i lo
> iptables -t filter -A INPUT -j ACCEPT -s {trusted newtork}
>
> iptables -t filter -A OUTPUT -j ACCEPT -o lo
>
> But don't change them if you don't want to filter the connections... :D
>
> I would ACCEPT every packet (as the first rule) that comes/goes on the lo
> interface... (in the PREROUTING/INPUT/OUTPUT/POSTROUTING chains in the
> raw/mangle/filter tables)
>
>
>> iptables -t mangle -N CONNMARK1
>> iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
>> iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
>>
>> iptables -t mangle -N CONNMARK2
>> iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
>> iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
>>
>> iptables -t mangle -N RESTOREMARK
>> iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
>
>
>> iptables -t mangle -A PREROUTING -i eth1 -s 0/0 -d 0/0 -m state
>> --state ESTABLISHED,RELATED -j RESTOREMARK
>> iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
>> statistic --mode nth --every 2 --packet 0 -j CONNMARK1
>> iptables -t mangle -A PREROUTING -p tcp -m state --state NEW -m
>> statistic --mode nth --every 2 --packet 1 -j CONNMARK2
>
> Hmm... :D
>
> iptables -t mangle -N MARKS
> iptables -t mangle -A MARKS - RETURN -m mark ! --mark 0/3
> iptables -t mangle -A MARKS -j MARK --set-mark 3/3 -m state --state
> ESTABLISED,RELATED
> iptables -t mangle -A MARKS -j MARK --set-mark 1/3 -m mark --mark 0/3 -m
> mode statistic --mode nth --every 2
> iptables -t mangle -A MARKS -j MARK --set-mark 2/3 -m mark --mark 0/3
>
> iptables -t mangle -A PREROUTING -j ACCEPT -i lo
> iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -j MARKS
>
> iptables -t mangle -A INPUT -j ACCEPT -i lo (put this as the first rule in
> the INPUT...)
> iptables -t mangle -A INPUT -j CONNMARK --save-mark (put this as the last
> rule in the INPUT...)
>
> iptables -t mangle -A OUTPUT -j ACCEPT -o lo
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -j MARKS (use this if you want to load-balance
> the traffic that originates from you firewall)
>
> iptables -t mangle -A POSTROUTING -j ACCEPT -o lo (put this as the first
> rule in the POSTROUTING...)
> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark (put this as the
> last rule in the POSTROUTING...)
>
>
>> iptables -t nat -N SNAT1
>> iptables -t nat -A SNAT1 -j SNAT --to-source 192.168.217.254
>>
>> iptables -t nat -N SNAT2
>> iptables -t nat -A SNAT2 -j SNAT --to-source 192.168.216.254
>
>
>> iptables -t nat -A POSTROUTING -o eth2 -j SNAT1
>> iptables -t nat -A POSTROUTING -o eth0 -j SNAT2
>
> How do you like these "one-liners"?
> iptables -t nat -A POSTROUTING -j SNAT -o eth0 ! -s 192.168.216.254
> --to-source 192.168.216.254
> iptables -t nat -A POSTROUTING -j SNAT -o eth2 ! -s 192.168.217.254
> --to-source 192.168.217.254
>
>
>> ip route add 192.168.217.0 via 192.168.217.1 table oitelecom
>> ip route add 192.168.216.0 via 192.168.216.1 table gvttelecom
>> ip route add default via 192.168.217.1 table oitelecom
>> ip route add default via 192.168.216.1 table gvttelecom
>
> Maybe it is better:
> ip route add default via 192.168.216.1 src 192.168.216.254 dev eth0 table
> gvttelecom
> ip route add default via 192.168.217.1 src 192.168.217.254 dev eth2 table
> oitelecom
>
>
>> ip rule del from 192.168.217.254 table oitelecom
>> ip rule add from 192.168.217.254 table oitelecom
>>
>> ip rule del fwmark 1 table oitelecom
>> ip rule del fwmark 2 table gvttelecom
>>
>> ip rule add fwmark 1 table oitelecom
>> ip rule add fwmark 2 table gvttelecom
>
> Use mask in marks:
> ip rule add fwmark 1/3 table oitelecom
> ip rule add fwmark 2/3 table gvttelecom
>
> Maybe you need to copy other local routes:
> ip route show table main | grep -E
> '(^10\.|^172\.1[6-9]\.|^172\.2[0-9]\.|^172\.3[01]\.|^192\.168\.)' | while
> read ROUTE
> do
> ip route add table ovtelecom ${ROUTE} 2>/dev/null
> ip route add table gvttelecom ${ROUTE} 2>/dev/null
> done
>
> Be carefull with this last one, as it copies the 192.168.x.x routes too!!!!!
>
>> ip route flush cache
>>
>>
>> thanks....any tips is welcome.
>
>
> Swifty
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: fail in the connmark load-balancing
2012-03-02 21:24 ` Usuário do Sistema
@ 2012-03-05 13:34 ` Gáspár Lajos
0 siblings, 0 replies; 11+ messages in thread
From: Gáspár Lajos @ 2012-03-05 13:34 UTC (permalink / raw)
To: Usuário do Sistema; +Cc: Andrew Beverley, Mail List - Netfilter
Hi,
Let me explain my post a bit :D
2012-03-02 22:24 keltezéssel, Usuário do Sistema írta:
> Hi,
>
> iptables -t mangle -N MARKS
> iptables -t mangle -A MARKS - RETURN -m mark ! --mark 0/3
If we have a mark (either 1 or 2 or 3) then we should just RETURN
because we already set the mark...
> iptables -t mangle -A MARKS -j MARK --set-mark 3/3 -m state --state
> ESTABLISED,RELATED
If the connection is already ESTABLISHED or it is RELATED to an another
connetion then just mark it as 3.
> iptables -t mangle -A MARKS -j MARK --set-mark 1/3 -m mark --mark 0/3
> -m mode statistic --mode nth --every 2
Every 2nd. connection should be marked as 2.
> iptables -t mangle -A MARKS -j MARK --set-mark 2/3 -m mark --mark 0/3
Every other connection is marked as 1.
> sorry, but I did understand the mark x/y as you wrote above. why did
> you use --set-mark 1 or --set-mark 2 ??
x is the mark, y is the mask...
0 -> not marked -> new or already establised but not marked (see mark 3)
1 -> new connection that should go out on interface 1.
2 -> new connection that should go out on interface 2.
3 -> any other but marked so we do not check again. (ESTABLISED, RELATED)
> I think when it's use the mark x/y y is an mask but I'dont know what
> is the match ?!?! with someone has a doc about that please post here.
man iptables (MARK target):
--set-mark value[/mask]
Zeroes out the bits given by mask and ORs value into the
packet mark. If mask is omitted, 0xFFFFFFFF is assumed.
> I haven't done test yet with your suggested Swifty. I'll tell you when I do.
>
> thanks
After reading my post I found out that my script can be fine tuned ... :D
In mangle/PREROUTING change these lines:
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -j MARKS
to this:
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -j MARK --set-mark 1/3 -m mark --mark
0/3 -i eth2
iptables -t mangle -A PREROUTING -j MARK --set-mark 2/3 -m mark --mark
0/3 -i eth0
iptables -t mangle -A PREROUTING -j MARKS
This modification ensures that any incoming connection (that either goes
to the firewall or gets FORWARDED) should take the same interface as it
came from.
Swifty
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-03-05 13:34 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-02-11 20:19 fail in the connmark load-balancing Usuário do Sistema
2012-02-12 22:10 ` Andrew Beverley
2012-02-13 11:19 ` Usuário do Sistema
2012-02-13 22:03 ` Usuário do Sistema
2012-02-25 10:53 ` Andrew Beverley
2012-02-27 16:40 ` Usuário do Sistema
2012-02-27 17:07 ` Usuário do Sistema
2012-02-27 18:15 ` Usuário do Sistema
2012-02-28 11:16 ` Gáspár Lajos
2012-03-02 21:24 ` Usuário do Sistema
2012-03-05 13:34 ` Gáspár Lajos
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.