Second failover failure with conntrackd - INVALID packets

Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

Hi,

I have a testbed cluster with two firewall nodes and conntrack sync. 
Initially, node1 is the primary and node2 the backup.

1) I open a jabber connection. node1 replicates the conntrack session to 
node2 (conntrackd in synchronization mode FT-FW).
2) First failover : node2 becomes the new primary, node1 the backup. 
node2 recovers the TCP session. Everything seems to work fine at this 
moment.
3) Second failover : just 1 to 2 minutes after the first failove, node1 
becomes the primary, node2 the backup. The jabber TCP session is still 
into the conntrack table but the session is broken. I see many packets 
denied due to the INVALID state.

"nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= 
OUT=xxxxxx"

The TCP window tracking rejects packets after the second failober. I 
just have to activate "nf_conntrack_tcp_be_liberal" to make it work 
again, but that's not, for me, a good solution:

echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal

Why conntrackd cannot recover the TCP session in the second failover ? 
Is it a known issue, possibly due to a misconfiguration ? I'm using 
debian/lenny (kernel 2.6.26-1-amd64) with heartbeat2, conntrack 0.9.6 
and a shell script that executes :

** on the new primary :
                conntrackd -c -C /etc/conntrackd.conf
                conntrackd -f -C /etc/conntrackd.conf
                conntrackd -R -C /etc/conntrackd.conf

** on the new backup :
                conntrackd -n -C /etc/conntrackd.conf

Thanks for your help,
Regards,

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Hi Yoann,

Yoann Juet wrote:
> I have a testbed cluster with two firewall nodes and conntrack sync.
> Initially, node1 is the primary and node2 the backup.
> 
> 1) I open a jabber connection. node1 replicates the conntrack session to
> node2 (conntrackd in synchronization mode FT-FW).
> 2) First failover : node2 becomes the new primary, node1 the backup.
> node2 recovers the TCP session. Everything seems to work fine at this
> moment.
> 3) Second failover : just 1 to 2 minutes after the first failove, node1
> becomes the primary, node2 the backup. The jabber TCP session is still
> into the conntrack table but the session is broken. I see many packets
> denied due to the INVALID state.
> 
> "nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN=
> OUT=xxxxxx"
> 
> The TCP window tracking rejects packets after the second failober. I
> just have to activate "nf_conntrack_tcp_be_liberal" to make it work
> again, but that's not, for me, a good solution:
> 
> echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
> 
> Why conntrackd cannot recover the TCP session in the second failover ?
> Is it a known issue, possibly due to a misconfiguration ? I'm using
> debian/lenny (kernel 2.6.26-1-amd64) with heartbeat2, conntrack 0.9.6
> and a shell script that executes :
> 
> ** on the new primary :
>                conntrackd -c -C /etc/conntrackd.conf
>                conntrackd -f -C /etc/conntrackd.conf
>                conntrackd -R -C /etc/conntrackd.conf
> 
> ** on the new backup :
>                conntrackd -n -C /etc/conntrackd.conf

Please, have a look at the conntrackd log file (/var/log/conntrackd.log
or syslog depending on your configuration). I think that it must be
reporting EINVAL while trying to update the entries during the second
fail-over.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 2097 bytes --]

Hi,

I see tons of messages "[warning] delayed packet?", even before the 
first failover, but nothing related to EINVAL. Does it help ?

FYI, the cluster is a KVM guest using hardware virtualization with net 
virtio.

Regards,

Pablo Neira Ayuso wrote:
> Hi Yoann,
> 
> Yoann Juet wrote:
>> I have a testbed cluster with two firewall nodes and conntrack sync.
>> Initially, node1 is the primary and node2 the backup.
>>
>> 1) I open a jabber connection. node1 replicates the conntrack session to
>> node2 (conntrackd in synchronization mode FT-FW).
>> 2) First failover : node2 becomes the new primary, node1 the backup.
>> node2 recovers the TCP session. Everything seems to work fine at this
>> moment.
>> 3) Second failover : just 1 to 2 minutes after the first failove, node1
>> becomes the primary, node2 the backup. The jabber TCP session is still
>> into the conntrack table but the session is broken. I see many packets
>> denied due to the INVALID state.
>>
>> "nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN=
>> OUT=xxxxxx"
>>
>> The TCP window tracking rejects packets after the second failober. I
>> just have to activate "nf_conntrack_tcp_be_liberal" to make it work
>> again, but that's not, for me, a good solution:
>>
>> echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
>>
>> Why conntrackd cannot recover the TCP session in the second failover ?
>> Is it a known issue, possibly due to a misconfiguration ? I'm using
>> debian/lenny (kernel 2.6.26-1-amd64) with heartbeat2, conntrack 0.9.6
>> and a shell script that executes :
>>
>> ** on the new primary :
>>                conntrackd -c -C /etc/conntrackd.conf
>>                conntrackd -f -C /etc/conntrackd.conf
>>                conntrackd -R -C /etc/conntrackd.conf
>>
>> ** on the new backup :
>>                conntrackd -n -C /etc/conntrackd.conf
> 
> Please, have a look at the conntrackd log file (/var/log/conntrackd.log
> or syslog depending on your configuration). I think that it must be
> reporting EINVAL while trying to update the entries during the second
> fail-over.
> 


[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Yoann Juet wrote:
> Hi,
> 
> I see tons of messages "[warning] delayed packet?", even before the
> first failover, but nothing related to EINVAL. Does it help ?

That's another known problem of 0.9.6. Probably, you don't see EINVAL
but a message like "N entries can't be committed". I suggest you to
upgrade to latest. I'm about to release 0.9.10, using current would make
my life easier to provide you support.

> FYI, the cluster is a KVM guest using hardware virtualization with net
> virtio.

Interesting. I have never used it in such environment.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

 > That's another known problem of 0.9.6. Probably, you don't see EINVAL
 > but a message like "N entries can't be committed". I suggest you to
 > upgrade to latest. I'm about to release 0.9.10, using current would 
 >make my life easier to provide you support.

Well, I installed conntrack-tools 0.9.9 and libnetfilter_conntrack 
0.0.99 on the cluster. No more "delayed packet" message or another 
warning or error message. Unfortunately, I get the same result when the 
second failover is triggered. Packets are denied due to INVALID state.

PS: the new configuration subblock "Filter from Kernelspace" in 
conntrackd.conf is not parsed correctly. I get an error message:

"Error parsing config file: line (190), symbol 'from': syntax error"

I have to delete it to make starting conntrackd.

Regards,

Pablo Neira Ayuso wrote:
> Yoann Juet wrote:
>> Hi,
>>
>> I see tons of messages "[warning] delayed packet?", even before the
>> first failover, but nothing related to EINVAL. Does it help ?
> 
> That's another known problem of 0.9.6. Probably, you don't see EINVAL
> but a message like "N entries can't be committed". I suggest you to
> upgrade to latest. I'm about to release 0.9.10, using current would make
> my life easier to provide you support.
> 
>> FYI, the cluster is a KVM guest using hardware virtualization with net
>> virtio.
> 
> Interesting. I have never used it in such environment.
> 


[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Yoann Juet wrote:
>> That's another known problem of 0.9.6. Probably, you don't see EINVAL
>> but a message like "N entries can't be committed". I suggest you to
>> upgrade to latest. I'm about to release 0.9.10, using current would
>>make my life easier to provide you support.
> 
> Well, I installed conntrack-tools 0.9.9 and libnetfilter_conntrack
> 0.0.99 on the cluster. No more "delayed packet" message or another
> warning or error message. Unfortunately, I get the same result when the
> second failover is triggered. Packets are denied due to INVALID state.

When the entries are created or updated, the flag
IP_CT_TCP_FLAG_BE_LIBERAL is set so that the window checking are
skipped, you should not get those "ACK/SEQ is under/over window". I
don't have an answer for the problem that you're reporting at this
moment. I know that there are some bugs in the ctnetlink code of 2.6.26
that were fixed in the subsequent kernel releases, but I did not know
any that affected the internal TCP flags set/unset. As these stuff is
under development, I suggest you to use the latest Linux kernel, please
let me know if the problem persists.

> PS: the new configuration subblock "Filter from Kernelspace" in
> conntrackd.conf is not parsed correctly. I get an error message:
> 
> "Error parsing config file: line (190), symbol 'from': syntax error"
> 
> I have to delete it to make starting conntrackd.

Filter From Kernelspace {
         Protocol Accept {
                TCP
         }
         Address Ignore {
                IPv4_address 127.0.0.1 # loopback
         }
}

It works here fine. Error reporting in the parsing is not very precise
yet, probably the problem is not in Filter but before (something is
missing and the parser gets confused). Could you post your config file
to reproduce it? You can send it to me in private if you want.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

Pablo Neira Ayuso wrote:
> Yoann Juet wrote:
>> "Error parsing config file: line (190), symbol 'from': syntax error"
>>
>> I have to delete it to make starting conntrackd.
> 
> Filter From Kernelspace {
>          Protocol Accept {
>                 TCP
>          }
>          Address Ignore {
>                 IPv4_address 127.0.0.1 # loopback
>          }
> }
> 
> It works here fine. Error reporting in the parsing is not very precise
> yet, probably the problem is not in Filter but before (something is
> missing and the parser gets confused). Could you post your config file
> to reproduce it? You can send it to me in private if you want.

Sorry, you're right. There is a typo in the example files. I have
applied the following fix to the git (will be available in
conntrack-tools-0.9.10).

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

[-- Attachment #2: doc3.patch --]
[-- Type: text/x-diff, Size: 1829 bytes --]

doc: use 'From' instead of 'from' in the example configfiles

This patch fixes a wrong use of 'from' instead of 'From' in the
example configuration files.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---

 doc/sync/alarm/conntrackd.conf   |    2 +-
 doc/sync/ftfw/conntrackd.conf    |    2 +-
 doc/sync/notrack/conntrackd.conf |    2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/doc/sync/alarm/conntrackd.conf b/doc/sync/alarm/conntrackd.conf
index 39741b3..da11887 100644
--- a/doc/sync/alarm/conntrackd.conf
+++ b/doc/sync/alarm/conntrackd.conf
@@ -226,7 +226,7 @@ General {
 	# event filtering, use the keyword 'Kernelspace' instead of 
 	# 'Userspace'.
 	#
-	Filter from Userspace {
+	Filter From Userspace {
 		#
 		# Accept only certain protocols: You may want to replicate
 		# the state of flows depending on their layer 4 protocol.
diff --git a/doc/sync/ftfw/conntrackd.conf b/doc/sync/ftfw/conntrackd.conf
index 93f7a44..3c39291 100644
--- a/doc/sync/ftfw/conntrackd.conf
+++ b/doc/sync/ftfw/conntrackd.conf
@@ -235,7 +235,7 @@ General {
 	# event filtering, use the keyword 'Kernelspace' instead of 
 	# 'Userspace'.
 	#
-	Filter from Userspace {
+	Filter From Userspace {
 		#
 		# Accept only certain protocols: You may want to replicate
 		# the state of flows depending on their layer 4 protocol.
diff --git a/doc/sync/notrack/conntrackd.conf b/doc/sync/notrack/conntrackd.conf
index 39a5faa..f86d17b 100644
--- a/doc/sync/notrack/conntrackd.conf
+++ b/doc/sync/notrack/conntrackd.conf
@@ -216,7 +216,7 @@ General {
 	# event filtering, use the keyword 'Kernelspace' instead of 
 	# 'Userspace'.
 	#
-	Filter from Userspace {
+	Filter From Userspace {
 		#
 		# Accept only certain protocols: You may want to replicate
 		# the state of flows depending on their layer 4 protocol.

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 1601 bytes --]

Hi pablo !

> that were fixed in the subsequent kernel releases, but I did not know
> any that affected the internal TCP flags set/unset. As these stuff is
> under development, I suggest you to use the latest Linux kernel, 
>please let me know if the problem persists.

I still have the same symptoms with a 2.6.28-2 kernel. My testbed is not 
so far away from yours:

You        		Me
----------------------------------
Etch <-> 		Lenny
2.6.28 <-> 		2.6.28-2
conntrack 0.9.9? <-> 	conntrack 0.9.9
ftfw mode <-> 		ftfw mode
keepalived 1.1.15 <-> 	heartbeat 2.1.3
no virtualization <-> 	KVM with net virtio

On your opinion, could it be the side effect of KVM ? Unfortunately, I 
cannot do without KVM, and cannot test easily without...

Regards,

Pablo Neira Ayuso wrote:
> Pablo Neira Ayuso wrote:
>> Yoann Juet wrote:
>>> "Error parsing config file: line (190), symbol 'from': syntax error"
>>>
>>> I have to delete it to make starting conntrackd.
>> Filter From Kernelspace {
>>          Protocol Accept {
>>                 TCP
>>          }
>>          Address Ignore {
>>                 IPv4_address 127.0.0.1 # loopback
>>          }
>> }
>>
>> It works here fine. Error reporting in the parsing is not very precise
>> yet, probably the problem is not in Filter but before (something is
>> missing and the parser gets confused). Could you post your config file
>> to reproduce it? You can send it to me in private if you want.
> 
> Sorry, you're right. There is a typo in the example files. I have
> applied the following fix to the git (will be available in
> conntrack-tools-0.9.10).
> 
> 



[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Hi again Yoann,

Yoann Juet wrote:
> Hi pablo !
> 
>> that were fixed in the subsequent kernel releases, but I did not know
>> any that affected the internal TCP flags set/unset. As these stuff is
>> under development, I suggest you to use the latest Linux kernel,
>> please let me know if the problem persists.
> 
> I still have the same symptoms with a 2.6.28-2 kernel. My testbed is not
> so far away from yours:
> 
> You                Me
> ----------------------------------
> Etch <->         Lenny
> 2.6.28 <->         2.6.28-2
> conntrack 0.9.9? <->     conntrack 0.9.9
> ftfw mode <->         ftfw mode
> keepalived 1.1.15 <->     heartbeat 2.1.3
> no virtualization <->     KVM with net virtio

Indeed, very similar.

> On your opinion, could it be the side effect of KVM ? Unfortunately, I
> cannot do without KVM, and cannot test easily without...

I'm not familiar with KVM, but before pointing to it as the problem
(since I think that it is transparent to conntrackd). Could you try
latest conntrack-tools 0.9.10? I released them yesterday along with
accumulated updates/fixes. Thanks!

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]


 > Could you try latest conntrack-tools 0.9.10? I released them 
yesterday > along with accumulated updates/fixes. Thanks!

I experience right now some difficulties to compile version 0.9.10 on 
lenny. I keep you in touch with test results.

Regards,

Pablo Neira Ayuso wrote:
> Hi again Yoann,
> 
> Yoann Juet wrote:
>> Hi pablo !
>>
>>> that were fixed in the subsequent kernel releases, but I did not know
>>> any that affected the internal TCP flags set/unset. As these stuff is
>>> under development, I suggest you to use the latest Linux kernel,
>>> please let me know if the problem persists.
>> I still have the same symptoms with a 2.6.28-2 kernel. My testbed is not
>> so far away from yours:
>>
>> You                Me
>> ----------------------------------
>> Etch <->         Lenny
>> 2.6.28 <->         2.6.28-2
>> conntrack 0.9.9? <->     conntrack 0.9.9
>> ftfw mode <->         ftfw mode
>> keepalived 1.1.15 <->     heartbeat 2.1.3
>> no virtualization <->     KVM with net virtio
> 
> Indeed, very similar.
> 
>> On your opinion, could it be the side effect of KVM ? Unfortunately, I
>> cannot do without KVM, and cannot test easily without...
> 
> I'm not familiar with KVM, but before pointing to it as the problem
> (since I think that it is transparent to conntrackd). Could you try
> latest conntrack-tools 0.9.10? I released them yesterday along with
> accumulated updates/fixes. Thanks!
> 


[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Hi Yoann,

Yoann Juet wrote:
> 
>> Could you try latest conntrack-tools 0.9.10? I released them yesterday
>> along with accumulated updates/fixes. Thanks!
> 
> I experience right now some difficulties to compile version 0.9.10 on
> lenny. I keep you in touch with test results.

Any update? I'm interested in your setup.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]

Hi pablo,

I still have an error as follows with conntrack 0.9.10:

#make
...
netfilter_conntrack.so -lnfnetlink
mcast.o: In function `mcast_dump_stats_extended':
/root/conntrack-tools-0.9.10/src/mcast.c:529: undefined reference to 
`nlif_get_ifflags'
sync-mode.o: In function `mcast_iface_handler':
/root/conntrack-tools-0.9.10/src/sync-mode.c:203: undefined reference to 
`nlif_get_ifflags'
sync-mode.o: In function `mcast_iface_candidate':
/root/conntrack-tools-0.9.10/src/sync-mode.c:185: undefined reference to 
`nlif_get_ifflags'
collect2: ld returned 1 exit status
make[1]: *** [conntrackd] Erreur 1
make[1]: quittant le répertoire « /root/conntrack-tools-0.9.10/src »
make: *** [all-recursive] Erreur 1

Do you have an idea ? The compilation of conntrack 0.9.9 works on the 
same machine.

Regards,

Pablo Neira Ayuso wrote:
> Hi Yoann,
> 
> Yoann Juet wrote:
>>> Could you try latest conntrack-tools 0.9.10? I released them yesterday
>>> along with accumulated updates/fixes. Thanks!
>> I experience right now some difficulties to compile version 0.9.10 on
>> lenny. I keep you in touch with test results.
> 
> Any update? I'm interested in your setup.
> 


[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Yoann Juet wrote:
> Hi pablo,
> 
> I still have an error as follows with conntrack 0.9.10:
> 
> #make
> ...
> netfilter_conntrack.so -lnfnetlink
> mcast.o: In function `mcast_dump_stats_extended':
> /root/conntrack-tools-0.9.10/src/mcast.c:529: undefined reference to
> `nlif_get_ifflags'
> sync-mode.o: In function `mcast_iface_handler':
> /root/conntrack-tools-0.9.10/src/sync-mode.c:203: undefined reference to
> `nlif_get_ifflags'
> sync-mode.o: In function `mcast_iface_candidate':
> /root/conntrack-tools-0.9.10/src/sync-mode.c:185: undefined reference to
> `nlif_get_ifflags'
> collect2: ld returned 1 exit status
> make[1]: *** [conntrackd] Erreur 1
> make[1]: quittant le répertoire « /root/conntrack-tools-0.9.10/src »
> make: *** [all-recursive] Erreur 1
> 
> Do you have an idea ? The compilation of conntrack 0.9.9 works on the
> same machine.
> 
> Regards,
> 
> Pablo Neira Ayuso wrote:
>> Hi Yoann,
>>
>> Yoann Juet wrote:
>>>> Could you try latest conntrack-tools 0.9.10? I released them yesterday
>>>> along with accumulated updates/fixes. Thanks!
>>> I experience right now some difficulties to compile version 0.9.10 on
>>> lenny. I keep you in touch with test results.
>>
>> Any update? I'm interested in your setup.

Damn. I forgot to update library dependencies. conntrack-tools-0.9.10
requires libnfnetlink-0.0.40. I'm going to fix this now in the git tree.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 2257 bytes --]

Pablo Neira Ayuso wrote:
> Yoann Juet wrote:
>> Hi pablo,
>>
>> I still have an error as follows with conntrack 0.9.10:
>>
>> #make
>> ...
>> netfilter_conntrack.so -lnfnetlink
>> mcast.o: In function `mcast_dump_stats_extended':
>> /root/conntrack-tools-0.9.10/src/mcast.c:529: undefined reference to
>> `nlif_get_ifflags'
>> sync-mode.o: In function `mcast_iface_handler':
>> /root/conntrack-tools-0.9.10/src/sync-mode.c:203: undefined reference to
>> `nlif_get_ifflags'
>> sync-mode.o: In function `mcast_iface_candidate':
>> /root/conntrack-tools-0.9.10/src/sync-mode.c:185: undefined reference to
>> `nlif_get_ifflags'
>> collect2: ld returned 1 exit status
>> make[1]: *** [conntrackd] Erreur 1
>> make[1]: quittant le répertoire « /root/conntrack-tools-0.9.10/src »
>> make: *** [all-recursive] Erreur 1
>>
>> Do you have an idea ? The compilation of conntrack 0.9.9 works on the
>> same machine.
>>
>> Regards,
>>
>> Pablo Neira Ayuso wrote:
>>> Hi Yoann,
>>>
>>> Yoann Juet wrote:
>>>>> Could you try latest conntrack-tools 0.9.10? I released them yesterday
>>>>> along with accumulated updates/fixes. Thanks!
>>>> I experience right now some difficulties to compile version 0.9.10 on
>>>> lenny. I keep you in touch with test results.
>>> Any update? I'm interested in your setup.
> 
> Damn. I forgot to update library dependencies. conntrack-tools-0.9.10
> requires libnfnetlink-0.0.40. I'm going to fix this now in the git tree.
> 

I'm still facing the same difficulties with conntrack-tools 0.9.10 and 
kernel 2.6.28.

Log on FW1 after the second failover:

Feb  6 09:55:46 FW-DSI-1-IRT kernel: [ 1352.601798] RULE -1 -- DENY 
IN=eth0 OUT=eth1 SRC=193.52.101.32 DST=172.18.244.10 LEN=255 TOS=0x00 
PREC=0x00 TTL=62 ID=8698 DF PROTO=TCP SPT=5222 DPT=34189 WINDOW=501 
RES=0x00 ACK PSH URGP=0

As you can see, this TCP connection is present:

root@fw1-irt:~# conntrack -L  |grep 34189
conntrack v0.9.10 (conntrack-tools): 14 flow entries has been shown.
tcp      6 10581 ESTABLISHED src=172.18.244.10 dst=193.52.101.32 
sport=34189 dport=5222 packets=63 bytes=12039 src=193.52.101.32 
dst=172.18.244.10 sport=5222 dport=34189 packets=58 bytes=22146 
[ASSURED] mark=0 secmark=0 use=1


[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

Hi again Yoann,

Yoann Juet wrote:
> I'm still facing the same difficulties with conntrack-tools 0.9.10 and
> kernel 2.6.28.
> 
> Log on FW1 after the second failover:
> 
> Feb  6 09:55:46 FW-DSI-1-IRT kernel: [ 1352.601798] RULE -1 -- DENY
> IN=eth0 OUT=eth1 SRC=193.52.101.32 DST=172.18.244.10 LEN=255 TOS=0x00
> PREC=0x00 TTL=62 ID=8698 DF PROTO=TCP SPT=5222 DPT=34189 WINDOW=501
> RES=0x00 ACK PSH URGP=0
> 
> As you can see, this TCP connection is present:
> 
> root@fw1-irt:~# conntrack -L  |grep 34189
> conntrack v0.9.10 (conntrack-tools): 14 flow entries has been shown.
> tcp      6 10581 ESTABLISHED src=172.18.244.10 dst=193.52.101.32
> sport=34189 dport=5222 packets=63 bytes=12039 src=193.52.101.32
> dst=172.18.244.10 sport=5222 dport=34189 packets=58 bytes=22146
> [ASSURED] mark=0 secmark=0 use=1

This is weird, look like some problem in your scripts or the commit is
not working in node fw1-irt. The packet counters of the entry above show
that this is the old entry which is stuck in the cache after the second
failover. This should be deleted when fw1-irt's script issues the commit
(conntrackd -c). Does the log file tells that the commit was successful?

The following attached patch adds more verbose output to tell you that
some old entries has been deleted. Just in case that you need more
information for troubleshooting.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

[-- Attachment #2: y --]
[-- Type: text/plain, Size: 1425 bytes --]

diff --git a/include/cache.h b/include/cache.h
index 371170d..4d2bbe8 100644
--- a/include/cache.h
+++ b/include/cache.h
@@ -82,6 +82,7 @@ struct cache {
 		uint32_t	upd_fail_enoent;
 
 		uint32_t	commit_ok;
+		uint32_t	commit_delete;
 		uint32_t	commit_fail;
 
 		uint32_t	flush;
diff --git a/src/cache_iterators.c b/src/cache_iterators.c
index e16a621..54613b9 100644
--- a/src/cache_iterators.c
+++ b/src/cache_iterators.c
@@ -134,6 +134,7 @@ retry:
 		if (errno == EEXIST && retry == 1) {
 			ret = nl_destroy_conntrack(tmp->h, ct);
 			if (ret == 0 || (ret == -1 && errno == ENOENT)) {
+				tmp->c->stats.commit_delete++;
 				if (retry) {
 					retry = 0;
 					goto retry;
@@ -179,6 +180,7 @@ void cache_commit(struct cache *c)
 {
 	unsigned int commit_ok = c->stats.commit_ok;
 	unsigned int commit_fail = c->stats.commit_fail;
+	unsigned int commit_delete = c->stats.commit_delete;
 	struct __commit_container tmp;
 	struct timeval commit_start, commit_stop, res;
 
@@ -199,6 +201,11 @@ void cache_commit(struct cache *c)
 	/* calculate new entries committed */
 	commit_ok = c->stats.commit_ok - commit_ok;
 	commit_fail = c->stats.commit_fail - commit_fail;
+	commit_delete = c->stats.commit_delete - commit_delete;
+
+	if (commit_delete)
+		dlog(LOG_NOTICE, "%u old entries deleted before "
+				 "commit", c->stats.commit_delete);
 
 	/* log results */
 	dlog(LOG_NOTICE, "Committed %u new entries", commit_ok);

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 2168 bytes --]

Hi pablo,

Pablo Neira Ayuso wrote:
> Hi again Yoann,
> 
> Yoann Juet wrote:
>> I'm still facing the same difficulties with conntrack-tools 0.9.10 and
>> kernel 2.6.28.
>>
>> Log on FW1 after the second failover:
>>
>> Feb  6 09:55:46 FW-DSI-1-IRT kernel: [ 1352.601798] RULE -1 -- DENY
>> IN=eth0 OUT=eth1 SRC=193.52.101.32 DST=172.18.244.10 LEN=255 TOS=0x00
>> PREC=0x00 TTL=62 ID=8698 DF PROTO=TCP SPT=5222 DPT=34189 WINDOW=501
>> RES=0x00 ACK PSH URGP=0
>>
>> As you can see, this TCP connection is present:
>>
>> root@fw1-irt:~# conntrack -L  |grep 34189
>> conntrack v0.9.10 (conntrack-tools): 14 flow entries has been shown.
>> tcp      6 10581 ESTABLISHED src=172.18.244.10 dst=193.52.101.32
>> sport=34189 dport=5222 packets=63 bytes=12039 src=193.52.101.32
>> dst=172.18.244.10 sport=5222 dport=34189 packets=58 bytes=22146
>> [ASSURED] mark=0 secmark=0 use=1
> 
> This is weird, look like some problem in your scripts or the commit is
> not working in node fw1-irt. The packet counters of the entry above show
> that this is the old entry which is stuck in the cache after the second
> failover. This should be deleted when fw1-irt's script issues the commit
> (conntrackd -c). Does the log file tells that the commit was successful?
> 

I confirm that "conntrackd -c" is executed on FW1-IRT on the second 
failover. FYI, my LSB script executes the following instructions (start= 
new active node ; stop = new passive node):

...
case "$1" in
         start)
                 conntrackd -c -C /etc/conntrackd.conf
                 conntrackd -f -C /etc/conntrackd.conf
                 conntrackd -R -C /etc/conntrackd.conf
                 exit 0
                 ;;

         stop)
                 #conntrackd -t -C /etc/conntrackd.conf
                 conntrackd -n -C /etc/conntrackd.conf
                 exit 0
                 ;;

         *)
                 exit 0
                 ;;
esac


> The following attached patch adds more verbose output to tell you that
> some old entries has been deleted. Just in case that you need more
> information for troubleshooting.
> 
> 

Thanks, maybe with this patch I'll see what's wrong with my setup.




[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Yoann Juet wrote:
> Pablo Neira Ayuso wrote:
>>> As you can see, this TCP connection is present:
>>>
>>> root@fw1-irt:~# conntrack -L  |grep 34189
>>> conntrack v0.9.10 (conntrack-tools): 14 flow entries has been shown.
>>> tcp      6 10581 ESTABLISHED src=172.18.244.10 dst=193.52.101.32
>>> sport=34189 dport=5222 packets=63 bytes=12039 src=193.52.101.32
>>> dst=172.18.244.10 sport=5222 dport=34189 packets=58 bytes=22146
>>> [ASSURED] mark=0 secmark=0 use=1
>>
>> This is weird, look like some problem in your scripts or the commit is
>> not working in node fw1-irt. The packet counters of the entry above show
>> that this is the old entry which is stuck in the cache after the second
>> failover. This should be deleted when fw1-irt's script issues the commit
>> (conntrackd -c). Does the log file tells that the commit was successful?
>>
> 
> I confirm that "conntrackd -c" is executed on FW1-IRT on the second
> failover. FYI, my LSB script executes the following instructions (start=
> new active node ; stop = new passive node):
> 
> ...
> case "$1" in
>         start)
>                 conntrackd -c -C /etc/conntrackd.conf
>                 conntrackd -f -C /etc/conntrackd.conf
>                 conntrackd -R -C /etc/conntrackd.conf
>                 exit 0
>                 ;;
> 
>         stop)
>                 #conntrackd -t -C /etc/conntrackd.conf
>                 conntrackd -n -C /etc/conntrackd.conf
>                 exit 0
>                 ;;

Before the second failover, does conntrackd -e in FW1-IRT (now in backup
mode) show, at least, one entry that talks about the TCP connection that
is in trouble? I think that it will not show any (but it should show
one, so I think that I have found the problem ;). Please, confirm this
and I'll get back to you with a possible solution soon.

BTW, why your script does not invoke the '-t', did you notice any
problem? That schedules the cleanup of the kernel conntrack table after
PurgeTimeout seconds when a node enters backup mode.

Thanks for your patience.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 2330 bytes --]

Pablo,

Pablo Neira Ayuso wrote:
> Yoann Juet wrote:
>> Pablo Neira Ayuso wrote:
>>>> As you can see, this TCP connection is present:
>>>>
>>>> root@fw1-irt:~# conntrack -L  |grep 34189
>>>> conntrack v0.9.10 (conntrack-tools): 14 flow entries has been shown.
>>>> tcp      6 10581 ESTABLISHED src=172.18.244.10 dst=193.52.101.32
>>>> sport=34189 dport=5222 packets=63 bytes=12039 src=193.52.101.32
>>>> dst=172.18.244.10 sport=5222 dport=34189 packets=58 bytes=22146
>>>> [ASSURED] mark=0 secmark=0 use=1
>>> This is weird, look like some problem in your scripts or the commit is
>>> not working in node fw1-irt. The packet counters of the entry above show
>>> that this is the old entry which is stuck in the cache after the second
>>> failover. This should be deleted when fw1-irt's script issues the commit
>>> (conntrackd -c). Does the log file tells that the commit was successful?
>>>
>> I confirm that "conntrackd -c" is executed on FW1-IRT on the second
>> failover. FYI, my LSB script executes the following instructions (start=
>> new active node ; stop = new passive node):
>>
>> ...
>> case "$1" in
>>         start)
>>                 conntrackd -c -C /etc/conntrackd.conf
>>                 conntrackd -f -C /etc/conntrackd.conf
>>                 conntrackd -R -C /etc/conntrackd.conf
>>                 exit 0
>>                 ;;
>>
>>         stop)
>>                 #conntrackd -t -C /etc/conntrackd.conf
>>                 conntrackd -n -C /etc/conntrackd.conf
>>                 exit 0
>>                 ;;
> 
> Before the second failover, does conntrackd -e in FW1-IRT (now in backup
> mode) show, at least, one entry that talks about the TCP connection that
> is in trouble? I think that it will not show any (but it should show
> one, so I think that I have found the problem ;). Please, confirm this
> and I'll get back to you with a possible solution soon.

You're right, the external cache on FW1 is empty before the second failover.

> 
> BTW, why your script does not invoke the '-t', did you notice any
> problem? That schedules the cleanup of the kernel conntrack table after
> PurgeTimeout seconds when a node enters backup mode.
> 

I was just testing with and without this instruction that was not 
available on conntrackd 0.9.6.

> Thanks for your patience.
> 

Thanks for your assistance !

[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Yoann Juet wrote:
> Pablo Neira Ayuso wrote:
>> Yoann Juet wrote:
>>> Pablo Neira Ayuso wrote:
>>>>> As you can see, this TCP connection is present:
>>>>>
>>>>> root@fw1-irt:~# conntrack -L  |grep 34189
>>>>> conntrack v0.9.10 (conntrack-tools): 14 flow entries has been shown.
>>>>> tcp      6 10581 ESTABLISHED src=172.18.244.10 dst=193.52.101.32
>>>>> sport=34189 dport=5222 packets=63 bytes=12039 src=193.52.101.32
>>>>> dst=172.18.244.10 sport=5222 dport=34189 packets=58 bytes=22146
>>>>> [ASSURED] mark=0 secmark=0 use=1
>>>> This is weird, look like some problem in your scripts or the commit is
>>>> not working in node fw1-irt. The packet counters of the entry above
>>>> show
>>>> that this is the old entry which is stuck in the cache after the second
>>>> failover. This should be deleted when fw1-irt's script issues the
>>>> commit
>>>> (conntrackd -c). Does the log file tells that the commit was
>>>> successful?
>>>>
>>> I confirm that "conntrackd -c" is executed on FW1-IRT on the second
>>> failover. FYI, my LSB script executes the following instructions (start=
>>> new active node ; stop = new passive node):
>>>
>>> ...
>>> case "$1" in
>>>         start)
>>>                 conntrackd -c -C /etc/conntrackd.conf
>>>                 conntrackd -f -C /etc/conntrackd.conf
>>>                 conntrackd -R -C /etc/conntrackd.conf

Please, add the following line here to your scripts:

conntrackd -B -C /etc/conntrackd.conf

Let me now if that fixes your problem.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

Pablo Neira Ayuso wrote:
> Pablo Neira Ayuso wrote:
>> Please, add the following line here to your scripts:
>>
>> conntrackd -B -C /etc/conntrackd.conf
>>
>> Let me now if that fixes your problem.
> 
> Updates? I'm intrigued with your problem. Some more info for
> troubleshooting. You have the commands:
> 
> display internal cache (states that belong to this node)
> # conntrackd -i
> 
> display external cache (states that belong to other nodes)
> # conntrackd -e
> 
> While trigering fail-overs, you should see the same states in the
> active's internal cache and the backup's external cache. If that does
> happen, there's a problem somewhere.
> 
> I'm about to release 0.9.11 but before I'd like to close pending issues.
> 

The issue is solved by adding "conntrackd -B" to my script. According to 
the logs, such instruction sends bulk update. What is it for exactly ?

Best regards,

[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Second failover failure with conntrackd - INVALID packets

[Pablo Neira Ayuso]

Yoann Juet wrote:
> Pablo Neira Ayuso wrote:
>> Pablo Neira Ayuso wrote:
>>> Please, add the following line here to your scripts:
>>>
>>> conntrackd -B -C /etc/conntrackd.conf
>>>
>>> Let me now if that fixes your problem.
>>
>> Updates? I'm intrigued with your problem. Some more info for
>> troubleshooting. You have the commands:
>>
>> display internal cache (states that belong to this node)
>> # conntrackd -i
>>
>> display external cache (states that belong to other nodes)
>> # conntrackd -e
>>
>> While trigering fail-overs, you should see the same states in the
>> active's internal cache and the backup's external cache. If that does
>> happen, there's a problem somewhere.
>>
>> I'm about to release 0.9.11 but before I'd like to close pending issues.
> 
> The issue is solved by adding "conntrackd -B" to my script. According to 
> the logs, such instruction sends bulk update. What is it for exactly ?

It forces the new primary to send a bulk update with the current state 
(that has been injected into the kernel) to the backup. You were using 
heartbeat? It seems that heartbeat triggers the backup state transition 
(thus, the request to resync to the new primary) before the new primary 
is itself in sync.

BTW, this change in the primary-backup.sh script has been already 
included in the conntrack-tools 0.9.11 (that I release a couple of days 
ago).

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Second failover failure with conntrackd - INVALID packets

[Yoann Juet]

[-- Attachment #1: Type: text/plain, Size: 1619 bytes --]

Pablo Neira Ayuso wrote:
> Yoann Juet wrote:
>> Pablo Neira Ayuso wrote:
>>> Pablo Neira Ayuso wrote:
>>>> Please, add the following line here to your scripts:
>>>>
>>>> conntrackd -B -C /etc/conntrackd.conf
>>>>
>>>> Let me now if that fixes your problem.
>>>
>>> Updates? I'm intrigued with your problem. Some more info for
>>> troubleshooting. You have the commands:
>>>
>>> display internal cache (states that belong to this node)
>>> # conntrackd -i
>>>
>>> display external cache (states that belong to other nodes)
>>> # conntrackd -e
>>>
>>> While trigering fail-overs, you should see the same states in the
>>> active's internal cache and the backup's external cache. If that does
>>> happen, there's a problem somewhere.
>>>
>>> I'm about to release 0.9.11 but before I'd like to close pending issues.
>>
>> The issue is solved by adding "conntrackd -B" to my script. According 
>> to the logs, such instruction sends bulk update. What is it for exactly ?
> 
> It forces the new primary to send a bulk update with the current state 
> (that has been injected into the kernel) to the backup. You were using 
> heartbeat? It seems that heartbeat triggers the backup state transition 
> (thus, the request to resync to the new primary) before the new primary 
> is itself in sync.
> 
> BTW, this change in the primary-backup.sh script has been already 
> included in the conntrack-tools 0.9.11 (that I release a couple of days 
> ago).
> 
In fact, I'm using Heartbeat2 and that's one major difference between 
our two test beds.

Again, many thanks for your assistance and your great work Pablo !

Best regards,

[-- Attachment #2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 375 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable:;DSI Universit=C3=A9 de Nantes
adr;quoted-printable:BP92208;;2, rue de la Houssini=C3=A8re;Nantes;;44322;France
email;internet:yoann.juet@univ-nantes.fr
title;quoted-printable:Ing=C3=A9nieur s=C3=A9curit=C3=A9 & r=C3=A9seau
tel;work:02.51.12.53.93
tel;fax:02.51.12.58.60
x-mozilla-html:FALSE
version:2.1
end:vcard