Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Tobias DiPasquale]

Harald et al,

Tonight I installed 2.6.12 and iptables 1.3.1 on my Gentoo box, both
from pristine sources. Two problems:

1. I had to copy in the ipt_CLUSTERIP.h kernel header from 2.6.12 into
iptables-1.3.1/include/linux/netfilter_ipv4 for the
libipt_CLUSTERIP.so library to have to same targinfosize as the kernel
module. Before that, I was getting the following error:

CLUSTERIP: targinfosize 64 != 72

2. I installed this rule just to test things out:

iptables -A INPUT -d 192.168.1.3 -i eth0 -p tcp --dport 3000 -j
CLUSTERIP --new --hashmode sourceip-sourceport --clustermac
01:23:45:67:89:AB --total-nodes 2 --local-node 1

I then tried to remove it (I didn't update iptables rules again since
then) by switching the -A to a -D and that failed. I tried all sorts
of combinations of the arguments to -j CLUSTERIP but they all failed.
Eventually, I was forced to remove the rule by using the iptables rule
# (e.g. iptables -D INPUT 10). I feel that this is in error, but I've
seen at least two patches to CLUSTERIP that claim to fix rule
deletion. Am I doing something wrong?

Anyway, I'm glad that you can now update the node lists dynamically
now. This makes it usable. I'm planning on writing a userspace driver
for this module to make at least the Win2K3 NLB functionality
available to Linux users.

-- 
[ Tobias DiPasquale ]
0x636f6465736c696e67657240676d61696c2e636f6d

Re: Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 2224 bytes --]

On Mon, Jun 20, 2005 at 10:04:15PM -0400, Tobias DiPasquale wrote:
> Harald et al,
> 
> Tonight I installed 2.6.12 and iptables 1.3.1 on my Gentoo box, both
> from pristine sources. Two problems:
> 
> 1. I had to copy in the ipt_CLUSTERIP.h kernel header from 2.6.12 into
> iptables-1.3.1/include/linux/netfilter_ipv4 for the
> libipt_CLUSTERIP.so library to have to same targinfosize as the kernel
> module. Before that, I was getting the following error:

this should never happen.  iptables always prefers
$KERNEL_DIR/include/linux/netfilter_ipv4 above its local copy.  Are you
sure KERNEL_DIR was set correctly while compiling iptables?


> 2. I installed this rule just to test things out:
> 
> iptables -A INPUT -d 192.168.1.3 -i eth0 -p tcp --dport 3000 -j
> CLUSTERIP --new --hashmode sourceip-sourceport --clustermac
> 01:23:45:67:89:AB --total-nodes 2 --local-node 1
> 
> I then tried to remove it (I didn't update iptables rules again since
> then) by switching the -A to a -D and that failed. I tried all sorts
> of combinations of the arguments to -j CLUSTERIP but they all failed.
> Eventually, I was forced to remove the rule by using the iptables rule
> # (e.g. iptables -D INPUT 10). I feel that this is in error, but I've
> seen at least two patches to CLUSTERIP that claim to fix rule
> deletion. Am I doing something wrong?

it is an error.  I will investigate any patches / fixes that have been
sent to the list.

> Anyway, I'm glad that you can now update the node lists dynamically
> now. This makes it usable. 

At which point was this not possible?  Or am I missing something?

> I'm planning on writing a userspace driver for this module to make at
> least the Win2K3 NLB functionality available to Linux users.

I am not familiar with that functionality, can you give me a pointer?

-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Tobias DiPasquale]

On 6/22/05, Harald Welte <laforge@netfilter.org> wrote: 
> this should never happen.  iptables always prefers
> $KERNEL_DIR/include/linux/netfilter_ipv4 above its local copy.  Are you
> sure KERNEL_DIR was set correctly while compiling iptables?

% cd iptables-1.3.1
% make KERNEL_DIR=/usr/src/linux-2.6.12 BINDIR=/sbin LIBDIR=/lib
MANDIR=/usr/share/man INCDIR=/usr/include

KERNEL_DIR above is where I had just two minutes before built a brand
new vanilla 2.6.12 kernel from my 2.6.11 config (with ipt_CLUSTERIP as
a module) and installed it. iptables correctly detected all compiled
extensions. Here's the diff between the two versions of the header
files:

adidas~/iptables-1.3.1/include/linux/netfilter_ipv4> diff -u
ipt_CLUSTERIP.h
/usr/src/linux-2.6.12/include/linux/netfilter_ipv4/ipt_CLUSTERIP.h
--- ipt_CLUSTERIP.h     2005-03-07 09:00:35.000000000 -0500
+++ /usr/src/linux-2.6.12/include/linux/netfilter_ipv4/ipt_CLUSTERIP.h
 2005-06-20 16:37:47.000000000 -0400
@@ -18,6 +18,7 @@
 struct ipt_clusterip_tgt_info {
 
        u_int32_t flags;
+       struct clusterip_config *config;
 
        /* only relevant for new ones */
        u_int8_t clustermac[6];
@@ -26,12 +27,6 @@
        u_int16_t local_nodes[CLUSTERIP_MAX_NODES];
        enum clusterip_hashmode hash_mode;
        u_int32_t hash_initval;
-
-#ifdef KERNEL_64_USERSPACE_32
-       u_int64_t config;
-#else
-       struct clusterip_config *config;
-#endif
 };
 
 #endif /*_IPT_CLUSTERIP_H_target*/
adidas~/iptables-1.3.1/include/linux/netfilter_ipv4>

This is on an x86_64 (Athlon64) machine, btw.

> it is an error.  I will investigate any patches / fixes that have been
> sent to the list.

Nice, thanks. The two previous patches I was referring to were:

1. message from you on May 6, 2005, subject "[PATCH 2.6] Two
ipt_CLUSTERIP fixes"; your second patch deals with iptables -D rule
deletion for ipt_CLUSTERIP rules.

2. message from Pablo Neira on March 6, 2005, subject "[PATCH 2/2] fix
CLUSTERIP rule deletion in iptables" containing only an attached patch
file, "fix-cluster-del.patch".

When deleting a rule that uses --new, should you repeat the --new in
the iptables -D command?

> > Anyway, I'm glad that you can now update the node lists dynamically
> > now. This makes it usable.
> 
> At which point was this not possible?  Or am I missing something?

Sorry, I just wasn't aware of this previously. I misspoke myself. I
was initially confused by the terminology "statically allocated" (in
the iptables man page?) used in referring to how the IP space was
divided between the nodes. Saru had provisions for this, as well, but
I believe they called it something else.
 
> > I'm planning on writing a userspace driver for this module to make at
> > least the Win2K3 NLB functionality available to Linux users.
> 
> I am not familiar with that functionality, can you give me a pointer?

Sure, this is a good introductory article:

http://www.west-wind.com/presentations/loadbalancing/NetworkLoadBalancingWindows2003.asp

Win2K3 NLB seems to have two modes: multicast (akin to CLUSTERIP) and
unicast, the latter of which I'm not really sure how it works. The
best I can figure, since it requires a dedicated interface for the
virtual IP, is that the "master" simply turns into a router that
routes requests to the other dedicated NICs in the cluster when the
traffic is not destined to the master. Those clusters then forward the
traffic from the dedicated NIC to the primary NIC for normal
processing.

Can you think of any other way this might work (my buddy and I can't)?
Would it be of any benefit to mirror this functionality in CLUSTERIP?
(they claim that they unicast mode is the faster of the two, in terms
of aggregated throughput)

-- 
[ Tobias DiPasquale ]
0x636f6465736c696e67657240676d61696c2e636f6d

Re: Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Wed, Jun 22, 2005 at 09:07:02AM -0400, Tobias DiPasquale wrote:
> On 6/22/05, Harald Welte <laforge@netfilter.org> wrote: 
> > this should never happen.  iptables always prefers
> > $KERNEL_DIR/include/linux/netfilter_ipv4 above its local copy.  Are you
> > sure KERNEL_DIR was set correctly while compiling iptables?

Ok, I now found what your problem was.  Instead of copying the
kernel-ipt_CLUSTERIP.h to iptables, you should have done it the other
way around (copy itpables-header to the kernel and recompile both).

The reason is that somehow one of the bugfixes for ipt_CLUSTERIP that I
sent DaveM some 7 weeks ago did not make it into 2.6.12 for unknown
reasons :(
-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

Hi Harald,

Harald Welte wrote:
>>2. I installed this rule just to test things out:
>>
>>iptables -A INPUT -d 192.168.1.3 -i eth0 -p tcp --dport 3000 -j
>>CLUSTERIP --new --hashmode sourceip-sourceport --clustermac
>>01:23:45:67:89:AB --total-nodes 2 --local-node 1
>>
>>I then tried to remove it (I didn't update iptables rules again since
>>then) by switching the -A to a -D and that failed. I tried all sorts
>>of combinations of the arguments to -j CLUSTERIP but they all failed.
>>Eventually, I was forced to remove the rule by using the iptables rule
>># (e.g. iptables -D INPUT 10). I feel that this is in error, but I've
>>seen at least two patches to CLUSTERIP that claim to fix rule
>>deletion. Am I doing something wrong?

fix-del-with-userspacesize-target.patch:

You still need to apply this patch to iptables, it got lost somehow 
since I've posted it some time ago. It fixes ipt_CLUSTERIP deletion. To 
be precise, it fixes rule deletion of targets whose userspace size part 
differs from the kernel part.

BTW, I forgot if you passed the patch here below (link) to davem? I 
can't see it in the current git snapshot. got lost as well ?

http://people.netfilter.org/pablo/patches/fixes/kernel/pending/fix-layout-CLUSTERIP.patch

@Tobias: With both patches applied you'll fix your problems with 
CLUSTERIP rule deletion.

--
Pablo

[-- Attachment #2: fix-del-with-userspacesize-target.patch --]
[-- Type: text/x-patch, Size: 795 bytes --]

Index: libiptc/libip4tc.c
===================================================================
--- libiptc/libip4tc.c	(revision 3742)
+++ libiptc/libip4tc.c	(working copy)
@@ -210,6 +210,7 @@
 	mptr = matchmask + sizeof(STRUCT_ENTRY);
 	if (IPT_MATCH_ITERATE(a, match_different, a->elems, b->elems, &mptr))
 		return NULL;
+	mptr += IPT_ALIGN(sizeof(struct ipt_entry_target));
 
 	return mptr;
 }
Index: libiptc/libip6tc.c
===================================================================
--- libiptc/libip6tc.c	(revision 3742)
+++ libiptc/libip6tc.c	(working copy)
@@ -242,6 +242,7 @@
 	mptr = matchmask + sizeof(STRUCT_ENTRY);
 	if (IP6T_MATCH_ITERATE(a, match_different, a->elems, b->elems, &mptr))
 		return NULL;
+	mptr += IP6T_ALIGN(sizeof(struct ip6t_entry_target));
 
 	return mptr;
 }

Re: Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Tobias DiPasquale]

On 6/22/05, Pablo Neira <pablo@eurodev.net> wrote:
> >>iptables -A INPUT -d 192.168.1.3 -i eth0 -p tcp --dport 3000 -j
> >>CLUSTERIP --new --hashmode sourceip-sourceport --clustermac
> >>01:23:45:67:89:AB --total-nodes 2 --local-node 1
> >>
> >>I then tried to remove it (I didn't update iptables rules again since
> >>then) by switching the -A to a -D and that failed. I tried all sorts
> >>of combinations of the arguments to -j CLUSTERIP but they all failed.
> >>Eventually, I was forced to remove the rule by using the iptables rule
> >># (e.g. iptables -D INPUT 10). I feel that this is in error, but I've
> >>seen at least two patches to CLUSTERIP that claim to fix rule
> >>deletion. Am I doing something wrong?
> 
> fix-del-with-userspacesize-target.patch:
> 
> You still need to apply this patch to iptables, it got lost somehow
> since I've posted it some time ago. It fixes ipt_CLUSTERIP deletion. To
> be precise, it fixes rule deletion of targets whose userspace size part
> differs from the kernel part.
> 
> BTW, I forgot if you passed the patch here below (link) to davem? I
> can't see it in the current git snapshot. got lost as well ?
> 
> http://people.netfilter.org/pablo/patches/fixes/kernel/pending/fix-layout-CLUSTERIP.patch
> 
> @Tobias: With both patches applied you'll fix your problems with
> CLUSTERIP rule deletion.

This did indeed fix both issues that I was having. Thanks, Pablo!

-- 
[ Tobias DiPasquale ]
0x636f6465736c696e67657240676d61696c2e636f6d

Re: Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

On Wed, Jun 22, 2005 at 11:05:29PM +0200, Pablo Neira wrote:
> Hi Harald,
> 
> Harald Welte wrote:
> >>2. I installed this rule just to test things out:
> >>
> >>iptables -A INPUT -d 192.168.1.3 -i eth0 -p tcp --dport 3000 -j
> >>CLUSTERIP --new --hashmode sourceip-sourceport --clustermac
> >>01:23:45:67:89:AB --total-nodes 2 --local-node 1
> >>
> >>I then tried to remove it (I didn't update iptables rules again since
> >>then) by switching the -A to a -D and that failed. I tried all sorts
> >>of combinations of the arguments to -j CLUSTERIP but they all failed.
> >>Eventually, I was forced to remove the rule by using the iptables rule
> >># (e.g. iptables -D INPUT 10). I feel that this is in error, but I've
> >>seen at least two patches to CLUSTERIP that claim to fix rule
> >>deletion. Am I doing something wrong?
> 
> fix-del-with-userspacesize-target.patch:

thanks, Committed revision 3989.


> http://people.netfilter.org/pablo/patches/fixes/kernel/pending/fix-layout-CLUSTERIP.patch

yes, I re-sent it yesterday.


-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux 2.6.12/iptables 1.3.1+CLUSTERIP issues

[Tobias DiPasquale]

On 6/22/05, Harald Welte <laforge@netfilter.org> wrote:
> On Wed, Jun 22, 2005 at 09:07:02AM -0400, Tobias DiPasquale wrote:
> > On 6/22/05, Harald Welte <laforge@netfilter.org> wrote:
> > > this should never happen.  iptables always prefers
> > > $KERNEL_DIR/include/linux/netfilter_ipv4 above its local copy.  Are you
> > > sure KERNEL_DIR was set correctly while compiling iptables?
> 
> Ok, I now found what your problem was.  Instead of copying the
> kernel-ipt_CLUSTERIP.h to iptables, you should have done it the other
> way around (copy itpables-header to the kernel and recompile both).
> 
> The reason is that somehow one of the bugfixes for ipt_CLUSTERIP that I
> sent DaveM some 7 weeks ago did not make it into 2.6.12 for unknown
> reasons :(

I think I see the reason: the new header file from the patch you're
referring to installs into "linux/netfilter_ipv4/ipt_CLUSTERIP.h", not
"include/linux/netfilter_ipv4/ipt_CLUSTERIP.h" where it should go.

-- 
[ Tobias DiPasquale ]
0x636f6465736c696e67657240676d61696c2e636f6d