Firewall in Load Balance - Active/Active

Firewall in Load Balance - Active/Active

[Eduardo Sachs]

Hi Friends!

I'm looking for firewall solution for active/active, the clients use
the firewalls randomly.

The conntrackd help me to replicate the state of the connection.

What help to "load balance" the firewalls?

My english is very terrible.. :)

Thanks!

Re: Firewall in Load Balance - Active/Active

[Marek Kierdelewicz]

> 
>Hi Friends!

Hi Bro,

>I'm looking for firewall solution for active/active, the clients use
>the firewalls randomly.

Interesting. How does client choose firewall? Where the randomization
occurs?

>The conntrackd help me to replicate the state of the connection.
>What help to "load balance" the firewalls?

You can use keepalived [1] to have two virtual gateway IP addresses on
the network - GW1 and GW2. Assign half of the clients staticly to GW1,
another half to GW2. If one of the boxes fails, keepalived brings up
missing GWX address on another box. This way you are provided with
redundancy and load balancing.

[1] http://www.keepalived.org/

Re: Firewall in Load Balance - Active/Active

[Eduardo Sachs]

> Hi Bro,

Hi!

> Interesting. How does client choose firewall? Where the randomization
> occurs?

Could have a Virtual IP for sending the requests to the two firewalls.
Could have a ldirectord for gateways. Remember that the ldirectord is
"load balance" only for ports TCP/UDP.

> You can use keepalived [1] to have two virtual gateway IP addresses on
> the network - GW1 and GW2. Assign half of the clients staticly to GW1,
> another half to GW2. If one of the boxes fails, keepalived brings up
> missing GWX address on another box. This way you are provided with
> redundancy and load balancing.

How would I disclose two gateways in the network if I have only one
DHCP? Could create VLANs. Correct?

>
> [1] http://www.keepalived.org/
>

Thanks!

Re: Firewall in Load Balance - Active/Active

[Pablo Neira Ayuso]

Eduardo Sachs wrote:
> Hi Friends!
> 
> I'm looking for firewall solution for active/active, the clients use
> the firewalls randomly.
> 
> The conntrackd help me to replicate the state of the connection.
> 
> What help to "load balance" the firewalls?

Have a look at the "cluster match" in the git tree:

http://git.netfilter.org/cgi-bin/gitweb.cgi?p=iptables.git;a=commit;h=cd958a6c92c84095a439780b53832bb3aae2d512

It will be available in 2.6.30. I'm still finishing some user-level
documentation about it. The integration with conntrackd is also on the way.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Firewall in Load Balance - Active/Active

[Eduardo Sachs]

Well!!

I will create a ambient of firewall active/passive.

But, what better program to do this? Heartbeat? VRRP? UCARP? Keepalived?

Thanks!

2009/5/25 Pablo Neira Ayuso <pablo@netfilter.org>:
> Eduardo Sachs wrote:
>> Hi Friends!
>>
>> I'm looking for firewall solution for active/active, the clients use
>> the firewalls randomly.
>>
>> The conntrackd help me to replicate the state of the connection.
>>
>> What help to "load balance" the firewalls?
>
> Have a look at the "cluster match" in the git tree:
>
> http://git.netfilter.org/cgi-bin/gitweb.cgi?p=iptables.git;a=commit;h=cd958a6c92c84095a439780b53832bb3aae2d512
>
> It will be available in 2.6.30. I'm still finishing some user-level
> documentation about it. The integration with conntrackd is also on the way.
>
> --
> "Los honestos son inadaptados sociales" -- Les Luthiers
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Firewall in Load Balance - Active/Active

[Покотиленко Костик]

В Пнд, 25/05/2009 в 10:35 -0300, Eduardo Sachs пишет:
> Well!!
> 
> I will create a ambient of firewall active/passive.
> 
> But, what better program to do this? Heartbeat? VRRP? UCARP? Keepalived?

Take a look at CARP.

-- 
Покотиленко Костик <casper@meteor.dp.ua>

Choices for virtual IP failover (was Re: Firewall in Load Balance - Active/Active)

[Thomas Jacob]

On Mon, 2009-05-25 at 10:35 -0300, Eduardo Sachs wrote:
> Well!!
> 
> I will create a ambient of firewall active/passive.
> 
> But, what better program to do this? Heartbeat? VRRP? UCARP? Keepalived?

I've checked the available options from a similar perspective and
finally decided on keepalived.

My premises were, that the chosen system should:

1. be lightweight (i.e. using almost no resources)
2. allow for coupled virtual IPs (i.e. move the external
  virtual IP whenever moving the internal IP)
3. allow the execution of simple scripts
4. still be maintained
5. have IPv6 support

Heartbeat (2.0) is an unwieldy monster with umpteen daemons
and is also somewhat tardy, but otherwise provides all
the features (and a million more ;). Might be a choice
if you like heartbeat and/or have a lot of other services
that you want to run on the firewall machines. Heartbeat
also supports more than two nodes.

VrrpD isn't maintained anymore (last release in 2002) and provides 
no coupled IPs, not scripts and no IPv6 support but is very lightweight.

Ucarp (userspace implementation of OpenBSDs Carp) is still being
maintained and seems to be lightweight, provides for simple scripts,
though no coupled IP fail over. Don't know whether or not it provides
IPv6 support, I haven't tried it myself.

Keepalived does not have IPv6 support (yet, VRRP for IPv6 is fairly
recent) but otherwise provides all the features and also can watch
the link states of network devices. The major drawback is that it also
has a IPVS module which is printing harmless error messages when the
underlying kernel doesn't support IPVS but I suppose you could prevent
that if you'd compile keepalived yourself.

I've selected keepalived and am so far quite happy with it.

Finally the problem with all these implementations is that they don't
support virtual MAC addresses in the way VRRP is usually provides
by router vendors and thus have to send gratuitous ARP requests
to inform their networks about the new MAC address after a failover.
Maybe ucarp is an exception, doesn't look like it though.

If people have experiences with other Linux based techniques it would
be great if they could post them here.

   Thomas

RE: Choices for virtual IP failover (was Re: Firewall in Load Balance - Active/Active)

[Thomas Jacob]

On Mon, 2009-05-25 at 15:26 +0100, John Bourke wrote:
> Folks,
> 
> Keepalived does not seem to have a mechanism to failover on the failure of a
> process on the system.  It has a load balancer which can select where
> traffic is sent to based on a HTTP check or a script return code.  But that
> is for server selection in load balancing.

It does in the latest versions, you can now add periodically run check
scripts that can be used to determine a FAULT state.

> One thing you really need to consider is flip flopping.  If you have Node A
> which is master, and when it goes down, Node B becomes master, the when node
> A comes up again it will become master.  A mechanism to "stick to the node
> last used" would be better so that a master with an intermittent failure
> does not cause flip flops.

You can do that with keepalived as well, check out the preemption
control parameters (noprempt etc.)

> Thanks
> 
> John

Re: Firewall in Load Balance - Active/Active

[Marek Kierdelewicz]

>Could have a Virtual IP for sending the requests to the two firewalls.
>Could have a ldirectord for gateways. Remember that the ldirectord is
>"load balance" only for ports TCP/UDP.

So you use third box. Solution I presented allows you to do
load-balancing on routers alone. Without ldirector.

>> You can use keepalived [1] to have two virtual gateway IP addresses
>> on the network - GW1 and GW2. Assign half of the clients staticly to
>> GW1,
>>...
>How would I disclose two gateways in the network if I have only one
>DHCP? Could create VLANs. Correct?

Vlans are not required. You can define more then one router-gateway on
one ethernet segment in dhcp server, example from one of my boxes:

shared-network seg1{

  subnet 10.32.0.0 netmask 255.255.254.0 {
   option subnet-mask 255.255.254.0;
   option broadcast-address 10.32.1.255;
   option routers 10.32.1.254;   <---- first gateway definition

   range 10.32.1.128 10.32.1.191;
   }


  subnet xx.xx.177.0 netmask 255.255.255.224 {
   option subnet-mask 255.255.255.224;
   option broadcast-address xx.xx.177.31;
   option routers xx.xx.177.30; <---- second gateway definition
   }
}

Cheers,
Marek Kierdelewicz

Re: Choices for virtual IP failover (was Re: Firewall in Load Balance - Active/Active)

[Tore Anderson]

Hi,

* Thomas Jacob

> Keepalived does not have IPv6 support (yet, VRRP for IPv6 is fairly
> recent) but otherwise provides all the features and also can watch
> the link states of network devices. The major drawback is that it also
> has a IPVS module which is printing harmless error messages when the
> underlying kernel doesn't support IPVS but I suppose you could prevent
> that if you'd compile keepalived yourself.

I knowthat keepalived has a command line option to only start the VRRP
parts of the code (-P).  Perhaps that will silence the warnings?

The lack of IPv6 support is something I miss, too.  I plan to deal with
it by adding/removing the HA IPv6 addresses from shell scripts ithat
runs when the state changes (the settings notify_{master,backup,fault}).
I didn't try it yet but I see no reason why it wouldn't work.  You'll
need to piggy-back it on an IPv4 VIP though (just use dummy addresses
from 169.254.0.0/16 or RFC1918 space for single-stack IPv6 networks).

> Finally the problem with all these implementations is that they don't
> support virtual MAC addresses in the way VRRP is usually provides
> by router vendors and thus have to send gratuitous ARP requests
> to inform their networks about the new MAC address after a failover.

I think this is due to a limitation in the Linux kernel - it is simply
not possible to have a multiple unicast layer-2 addresses assigned to a
single network interface.  Go bug the people on netdev - I'm sure
keepalived will support VMAC immediately after the necessary kernel
changes have been made.

BR,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/

Re: Choices for virtual IP failover (was Re: Firewall in Load Balance - Active/Active)

[Thomas Jacob]

On Mon, 2009-05-25 at 16:58 +0200, Tore Anderson wrote:
> > the link states of network devices. The major drawback is that it also
> > has a IPVS module which is printing harmless error messages when the
> > underlying kernel doesn't support IPVS but I suppose you could prevent
> > that if you'd compile keepalived yourself.
> 
> I knowthat keepalived has a command line option to only start the VRRP
> parts of the code (-P).  Perhaps that will silence the warnings?

It doesn't but I am simply ignoring them (they are only
printed at startup anyway).

> The lack of IPv6 support is something I miss, too.  I plan to deal with
> it by adding/removing the HA IPv6 addresses from shell scripts ithat
> runs when the state changes (the settings notify_{master,backup,fault}).
> I didn't try it yet but I see no reason why it wouldn't work. 

It does work as that's what I am doing now ;) (Debian 5.0 nodes
with packaged keepalived). Also the Linux IPv6 stack automatically
sends something equivalent to a gratuitous ARP request anyway
(some ICMPv6 type deals with announcing address changes on the link).

>  You'll
> need to piggy-back it on an IPv4 VIP though (just use dummy addresses
> from 169.254.0.0/16 or RFC1918 space for single-stack IPv6 networks).

I'm using a dual stack approach so I don't have that problem.

> > Finally the problem with all these implementations is that they don't
> > support virtual MAC addresses in the way VRRP is usually provides
> > by router vendors and thus have to send gratuitous ARP requests
> > to inform their networks about the new MAC address after a failover.
> 
> I think this is due to a limitation in the Linux kernel - it is simply
> not possible to have a multiple unicast layer-2 addresses assigned to a
> single network interface.  Go bug the people on netdev - I'm sure
> keepalived will support VMAC immediately after the necessary kernel
> changes have been made.

Am aware of this but I simply don't have the time right now, but
surely would be a nice thing to have, also from a security perspective
(relying on what makes ARP-Spoofing possible for your fundamental
infrastructure is not nice ;)

RE: Choices for virtual IP failover (was Re: Firewall in Load Balance - Active/Active)

[Thomas Jacob]

On Mon, 2009-05-25 at 17:39 +0100, John Bourke wrote:
> Thomas,
> 
> This is what I get for reading everything BUT the man page !
> 
> Found the preempt, but the checks seem to refer to the health checks of real
> servers
> 
>                   # one entry for each realserver
>                   real_server <IPADDR> <PORT>
>                      {
> 				 ...
>                          # Script to launch when healthchecker
>                          # considers service as up.
>                          notify_up <STRING>|<QUOTED-STRING>
>                          # Script to launch when healthchecker
>                          # considers service as down.
>                          notify_down <STRING>|<QUOTED-STRING>
> 
>                          # pick one healthchecker
>                          # HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK
> 
> I don't think this can trigger a keepalived VRRp failover if a real server
> fails ??? (not that our scenario uses external real servers)

You really need the bleeding edge for the scripting checks:

http://www.keepalived.org/software/keepalived-1.1.17.tar.gz

cat doc/samples/keepalived.conf.vrrp.localcheck

! Configuration File for keepalived

vrrp_script chk_sshd {
       script "killall -0 sshd"        # cheaper than pidof
       interval 2                      # check every 2 seconds
       weight -4                       # default prio: -4 if KO
}

[...]

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 51
    priority 100
    virtual_ipaddress {
        192.168.200.18/25
    }
    track_interface {
       eth1 weight 2   # prio = +2 if UP
       eth2 weight -2  # prio = -2 if DOWN
       eth3            # no weight, fault if down
    }
    track_script {
       chk_sshd                # use default weight from the script
       chk_haproxy weight 2    # +2 if process is present
       chk_http_port
       chk_https_port
       chk_smtp_port
    }
}


[..]


> Thanks
> 
> John
> 
> 
> -----Original Message-----
> From: Thomas Jacob [mailto:jacob@internet24.de] 
> Sent: 25 May 2009 15:31
> To: John Bourke
> Cc: 'Eduardo Sachs'; netfilter@vger.kernel.org
> Subject: RE: Choices for virtual IP failover (was Re: Firewall in Load
> Balance - Active/Active)
> 
> On Mon, 2009-05-25 at 15:26 +0100, John Bourke wrote:
> > Folks,
> > 
> > Keepalived does not seem to have a mechanism to failover on the failure of
> a
> > process on the system.  It has a load balancer which can select where
> > traffic is sent to based on a HTTP check or a script return code.  But
> that
> > is for server selection in load balancing.
> 
> It does in the latest versions, you can now add periodically run check
> scripts that can be used to determine a FAULT state.
> 
> > One thing you really need to consider is flip flopping.  If you have Node
> A
> > which is master, and when it goes down, Node B becomes master, the when
> node
> > A comes up again it will become master.  A mechanism to "stick to the node
> > last used" would be better so that a master with an intermittent failure
> > does not cause flip flops.
> 
> You can do that with keepalived as well, check out the preemption
> control parameters (noprempt etc.)
> 
> > Thanks
> > 
> > John
> 
> 
> 
> 

Re: Firewall in Load Balance - Active/Active

[Elvir Kuric]

Use CARP, it is best and try to set up it on OpenBSD. On linux it is
ucarp, but I would recommend
OpenBSD + CARP

Regards,

Elvir Kuric

On Mon, May 25, 2009 at 3:35 PM, Eduardo Sachs <edu.sachs@gmail.com> wrote:
> Well!!
>
> I will create a ambient of firewall active/passive.
>
> But, what better program to do this? Heartbeat? VRRP? UCARP? Keepalived?
>
> Thanks!
>
> 2009/5/25 Pablo Neira Ayuso <pablo@netfilter.org>:
>> Eduardo Sachs wrote:
>>> Hi Friends!
>>>
>>> I'm looking for firewall solution for active/active, the clients use
>>> the firewalls randomly.
>>>
>>> The conntrackd help me to replicate the state of the connection.
>>>
>>> What help to "load balance" the firewalls?
>>
>> Have a look at the "cluster match" in the git tree:
>>
>> http://git.netfilter.org/cgi-bin/gitweb.cgi?p=iptables.git;a=commit;h=cd958a6c92c84095a439780b53832bb3aae2d512
>>
>> It will be available in 2.6.30. I'm still finishing some user-level
>> documentation about it. The integration with conntrackd is also on the way.
>>
>> --
>> "Los honestos son inadaptados sociales" -- Les Luthiers
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Firewall in Load Balance - Active/Active

[Thomas Jacob]

Danger. Possible flame bait ahead.

On Tue, May 26, 2009 at 08:39:02PM +0200, Elvir Kuric wrote:
> Use CARP, it is best and try to set up it on OpenBSD. On linux it is
> ucarp, but I would recommend
> OpenBSD + CARP

Given that the OP was looking for an active/active solution and thus is presumably
interested in performance as well, I wonder why you would recommend
OpenBSD's pf+carp. In the last performance comparison I read some years back
pf was almost an order of magnitude slower than iptables at the time.

We for instance run a pair of OpenBSD boxen on some old 933 GHz P3s with
4 ste/Sundance ST201-NICs each and they basically don't even manage data 
transfers at the full 100mbit/s.

Stability?
Features?
Security?

Just curious...