conntrackd exits during failover when there are around 30000 connections

conntrackd exits during failover when there are around 30000 connections

[PATEL, SAMEER]

Hi,

I'm having some problems with the following configuration:

- Two firewalls in a master-backup configuration managed by keepalived
- A single dedicated link between the two firewalls managed by conntrackd

Now, if I make around 30000 connections between a computer and a server behind the firewall, and the master firewall fails, then conntrackd exits (or perhaps crashes). I don't think this is an out-of-memory issue because conntrackd didn't have the highest OOM score before it failed. Also, I watched memory usage while this was going on and there seemed to be plenty.

Is there some tweak or configuration parameter that enables support for large numbers of connections? Any insights into this issue would be greatly appreciated.

Thanks,

Sameer Patel
Siemens Canada Limited


This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation

Re: conntrackd exits during failover when there are around 30000 connections

[Arturo Borrero Gonzalez]

On 25 July 2017 at 16:32, PATEL, SAMEER <sameer.patel@siemens.com> wrote:
> Hi,
>
> I'm having some problems with the following configuration:
>
> - Two firewalls in a master-backup configuration managed by keepalived
> - A single dedicated link between the two firewalls managed by conntrackd
>
> Now, if I make around 30000 connections between a computer and a server behind the firewall, and the master firewall fails, then conntrackd exits (or perhaps crashes). I don't think this is an out-of-memory issue because conntrackd didn't have the highest OOM score before it failed. Also, I watched memory usage while this was going on and there seemed to be plenty.
>
> Is there some tweak or configuration parameter that enables support for large numbers of connections? Any insights into this issue would be greatly appreciated.
>

Could you share which version are you running? both of the kernel,
conntrackd (and libnetfilter-conntrackd).
Did you look at the logs? usually /var/log/conntrackd.log.

If conntrackd is hitting some errors, for example, failed to commit
some entries, then some log lines should be there.

RE: conntrackd exits during failover when there are around 30000 connections

[PATEL, SAMEER]

Hi Arturo,

We're using a distribution based on Debian Jessie. The software versions are below:

libnetfilter-conntrack3: 1.0.4
libnetfilter-cthelper0: 1.0.0
libnetfilter-queue1: 1.0.2
conntrackd: 1.4.2
kernel: 3.14.68

There aren't any errors in the logs. The last things I see are "flushing conntrack table in 60" and "request resync"

Also conntrackd fails to restart until I delete a lock file. This might be more evidence that conntrackd isn't exiting cleanly.

Thanks,
Sameer


-----Original Message-----
From: Arturo Borrero Gonzalez [mailto:arturo@netfilter.org]
Sent: July-26-17 3:05 AM
To: PATEL, SAMEER (PD PA CI RC R&D SW)
Cc: netfilter@vger.kernel.org
Subject: Re: conntrackd exits during failover when there are around 30000 connections

On 25 July 2017 at 16:32, PATEL, SAMEER <sameer.patel@siemens.com> wrote:
> Hi,
>
> I'm having some problems with the following configuration:
>
> - Two firewalls in a master-backup configuration managed by keepalived
> - A single dedicated link between the two firewalls managed by
> conntrackd
>
> Now, if I make around 30000 connections between a computer and a server behind the firewall, and the master firewall fails, then conntrackd exits (or perhaps crashes). I don't think this is an out-of-memory issue because conntrackd didn't have the highest OOM score before it failed. Also, I watched memory usage while this was going on and there seemed to be plenty.
>
> Is there some tweak or configuration parameter that enables support for large numbers of connections? Any insights into this issue would be greatly appreciated.
>

Could you share which version are you running? both of the kernel, conntrackd (and libnetfilter-conntrackd).
Did you look at the logs? usually /var/log/conntrackd.log.

If conntrackd is hitting some errors, for example, failed to commit some entries, then some log lines should be there.

This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation

Re: conntrackd exits during failover when there are around 30000 connections

[Arturo Borrero Gonzalez]

On 26 July 2017 at 15:49, PATEL, SAMEER <sameer.patel@siemens.com> wrote:
> Hi Arturo,
>
> We're using a distribution based on Debian Jessie. The software versions are below:
>
> libnetfilter-conntrack3: 1.0.4
> libnetfilter-cthelper0: 1.0.0
> libnetfilter-queue1: 1.0.2
> conntrackd: 1.4.2
> kernel: 3.14.68
>

Well, many fixes happened since conntrackd 1.4.2 which is 4 years old.
Same for the kernel.
It is possible if you try a more recent version of both kernel and conntrackd?

> There aren't any errors in the logs. The last things I see are "flushing conntrack table in 60" and "request resync"
>
> Also conntrackd fails to restart until I delete a lock file. This might be more evidence that conntrackd isn't exiting cleanly.
>

True. This reminds my of the debian bug #796877 [0] that I suffered in the past.

Problem is that back then, the conntrackd package didn't include debug
symbols, so even if you run conntrackd with valgrind or gdb to see
where the crash happens, you wont get the symbol names (i.e.
funcionts) so little clues.

Since then, I took over the debian packages and added debug symbols
(starting with 1:1.4.3-2). But then, again, you need a newer version.

[0] https://bugs.debian.org/796877

RE: conntrackd exits during failover when there are around 30000 connections

[PATEL, SAMEER]

Didn't realize we were using such an old version. I'll try with the latest. Thanks very much for the help.

Sameer

-----Original Message-----
From: Arturo Borrero Gonzalez [mailto:arturo@netfilter.org] 
Sent: July-27-17 6:40 AM
To: PATEL, SAMEER (PD PA CI RC R&D SW)
Cc: netfilter@vger.kernel.org
Subject: Re: conntrackd exits during failover when there are around 30000 connections

On 26 July 2017 at 15:49, PATEL, SAMEER <sameer.patel@siemens.com> wrote:
> Hi Arturo,
>
> We're using a distribution based on Debian Jessie. The software versions are below:
>
> libnetfilter-conntrack3: 1.0.4
> libnetfilter-cthelper0: 1.0.0
> libnetfilter-queue1: 1.0.2
> conntrackd: 1.4.2
> kernel: 3.14.68
>

Well, many fixes happened since conntrackd 1.4.2 which is 4 years old.
Same for the kernel.
It is possible if you try a more recent version of both kernel and conntrackd?

> There aren't any errors in the logs. The last things I see are "flushing conntrack table in 60" and "request resync"
>
> Also conntrackd fails to restart until I delete a lock file. This might be more evidence that conntrackd isn't exiting cleanly.
>

True. This reminds my of the debian bug #796877 [0] that I suffered in the past.

Problem is that back then, the conntrackd package didn't include debug symbols, so even if you run conntrackd with valgrind or gdb to see where the crash happens, you wont get the symbol names (i.e.
funcionts) so little clues.

Since then, I took over the debian packages and added debug symbols (starting with 1:1.4.3-2). But then, again, you need a newer version.

[0] https://bugs.debian.org/796877

Re: conntrackd exits during failover when there are around 30000 connections

[Arturo Borrero Gonzalez]

On 27 July 2017 at 15:15, PATEL, SAMEER <sameer.patel@siemens.com> wrote:
> Didn't realize we were using such an old version. I'll try with the latest. Thanks very much for the help.
>

Great, thanks :-)

BTW, please, in the future avoid top-posting [0].

best regards.

[0] https://en.wikipedia.org/wiki/Posting_style#Top-posting