conntrack accounting

conntrack accounting

[Ben Lentz]

Greetings list!
I am considering using the conntrack-tools userspace package to perform 
byte level accounting for iptables by reading events from the connection 
tracking table for completed connections and logging the statistics for 
the stateful connection to syslog. It appears that conntrack was really 
designed to keep redundant firewalls' state tables in sync, but I'm 
intrigued by it's ability to use the new connection tracking and state 
notification features in netfilter without having to parse or poll 
/proc/net/ip_conntrack.

The goal I'm trying to accomplish is similar to that of:
conntrack -E conntrack -e DESTROY | logger -t conntrack &

which gives me the ability to log completed (e.g. entered the DESTROY 
state) connections to syslog from kernel-triggered events. It's plenty 
hackish though... it'd be nicer to have an actual daemon that fork()s 
and detaches and closes file descriptors and communicates with syslog 
directly. I understand that a patch has been contributed to allow 
conntrackd to use syslog, but it appears that the logging facility in 
conntrackd is limited to recording startup, shutdown, and error 
information. In any event, the current incarnation of conntrackd does 
not support the long-term recording of event messages.

What would you folks recommend to accomplish this goal? Am I simply 
using the wrong tool here, or is it worthwhile to get a-patchin'?

If more appropriate, I'll repost this in netfilter failover, but since 
I'm not actually looking to do failover (at the moment) I'd figure I'd 
start here.

Thanks in advance for any information or opinions you can provide.

Re: conntrack accounting

[Pablo Neira Ayuso]

Hi,

Ben Lentz wrote:
> I am considering using the conntrack-tools userspace package to perform
> byte level accounting for iptables by reading events from the connection
> tracking table for completed connections and logging the statistics for
> the stateful connection to syslog. It appears that conntrack was really
> designed to keep redundant firewalls' state tables in sync, but I'm
> intrigued by it's ability to use the new connection tracking and state
> notification features in netfilter without having to parse or poll
> /proc/net/ip_conntrack.
> 
> The goal I'm trying to accomplish is similar to that of:
> conntrack -E conntrack -e DESTROY | logger -t conntrack &

I just committed a patch to SVN which implements this for the statistics
mode. Have a look at the doc/stats/conntrackd.conf example file and
enable logging to give it a try. This will be available in the upcoming
conntrack-tool 0.9.6 release. Don't forget to run conntrackd with the -S
option.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: conntrack accounting

[Ben Lentz]

> I just committed a patch to SVN which implements this for the statistics
> mode. Have a look at the doc/stats/conntrackd.conf example file and
> enable logging to give it a try. This will be available in the upcoming
> conntrack-tool 0.9.6 release. Don't forget to run conntrackd with the -S
> option.
>   
This sounds great! However, I appear to be having some trouble. I 
checked out, built, and installed conntrack-tools 0.9.6 7164 and 
libnetfilter_conntrack-0.0.87 7164 and am running conntrackd -S. I still 
have libnfnetlink-0.0.30. I don't seem to be getting any statistics 
logging either in Syslog mode or LogFile mode.

- If I set the Stats section to Syslog on, it seems to crash on the 
first attempt to log:
select(5, [3 4], NULL, NULL, {0, 199092}) = 1 (in [4], left {0, 111000})
rt_sigprocmask(SIG_BLOCK, [INT TERM CHLD], NULL, 8) = 0
recvmsg(4, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000004}, 
msg_iov(1)=[...], msg_controllen=0, msg_flags=0}, MSG_PEEK) = 164
recvfrom(4, ""..., 8192, 0, {sa_family=AF_NETLINK, pid=0, 
groups=00000004}, [12]) = 164
time(NULL)                              = 1199383171
open("/etc/localtime", O_RDONLY)        = 6
fstat64(6, {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
fstat64(6, {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7f97000
read(6, ""..., 4096)                    = 3519
close(6)                                = 0
munmap(0xb7f97000, 4096)                = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

- If I set the Stats section to Logfile on (or LogFile filename), it 
doesn't crash, but generates a similar error each time it goes to log a 
connection. I can correlate connections about to close with
$ sudo watch --interval=0.1 'cat /proc/net/ip_conntrack | grep 
"^[a-z]\{3\} *[0-9]* *0"'

to errors in a strace on conntrackd:
rt_sigprocmask(SIG_BLOCK, [INT TERM CHLD], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [INT TERM CHLD], NULL, 8) = 0
gettimeofday({1199383388, 670177}, NULL) = 0
gettimeofday({1199383388, 670286}, NULL) = 0
select(5, [3 4], NULL, NULL, {0, 198979}) = 1 (in [4], left {0, 47000})
rt_sigprocmask(SIG_BLOCK, [INT TERM CHLD], NULL, 8) = 0
recvmsg(4, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000004}, 
msg_iov(1)=[...], msg_controllen=0, msg_flags=0}, MSG_PEEK) = 164
recvfrom(4, ""..., 8192, 0, {sa_family=AF_NETLINK, pid=0, 
groups=00000004}, [12]) = 164
recvfrom(4, 0xbfae01e0, 8192, 0, 0xbfae01ac, 0xbfae01b8) = -1 EAGAIN 
(Resource temporarily unavailable)
rt_sigprocmask(SIG_UNBLOCK, [INT TERM CHLD], NULL, 8) = 0
gettimeofday({1199383388, 822810}, NULL) = 0
gettimeofday({1199383388, 822856}, NULL) = 0

Configuration file is as follows:
$ grep -v '^$\|^#\|^\W#' /etc/conntrackd/conntrackd.conf
General {
        HashSize 8192
        HashLimit 65535
        LogFile on
        Syslog off
        LockFile /var/lock/conntrack.lock
        UNIX {
                Path /tmp/sync.sock
                Backlog 20
        }
        SocketBufferSize 262142
        SocketBufferSizeMaxGrown 655355
}
Stats {
        LogFile on
        Syslog off
}
IgnoreTrafficFor {
}
IgnoreProtocol {
}

Platform is CentOS 5, kernel 2.6.18.

Please let me know if I've done something dumb or if there's anything I 
can do to provide more useful debugging information. It's been a long 
while since I've been in gdb, so I might need some help with that...

Re: conntrack accounting

[Ben Lentz]

> This sounds great! However, I appear to be having some trouble. I 
> checked out, built, and installed conntrack-tools 0.9.6 7164 and 
> libnetfilter_conntrack-0.0.87 7164 and am running conntrackd -S. I 
> still have libnfnetlink-0.0.30. I don't seem to be getting any 
> statistics logging either in Syslog mode or LogFile mode.
I've made some progress... it turns out that the statistics logging via 
LogFile mode works only if I start conntrackd with -C 
/etc/conntrackd/conntrackd.conf... for some reason, leaving the -C 
option off and defaulting to this configuration file doesn't work, even 
though a strace shows a successful open to the file upon initialization.

Syslog statistics mode still crashes... but only if I *disable* LogFile, 
too.

This works perfectly:
Stats {
        LogFile on
        Syslog on
}
This crashes:
Stats {
        LogFile off
        Syslog on
}

CentOS 5.0, kernel 2.6.18, no SELinux, liberal iptables configuration, 
sysklogd 1.4.1

Is there a more appropriate place to report this information?

Thanks for your work on this thus far, this looks like a great addition 
to conntrack-tools!

Re: conntrack accounting

[Ben Lentz]

[-- Attachment #1: Type: text/plain, Size: 174 bytes --]


> Syslog statistics mode still crashes... but only if I *disable* 
> LogFile, too.
>
I've fixed the crash when stats LogFile is off and stats Syslog is on, 
patch attached.

[-- Attachment #2: conntrack-tools_stats-syslog.patch --]
[-- Type: text/x-patch, Size: 553 bytes --]

--- src/log.c.orig	2008-01-04 16:26:50.000000000 -0500
+++ src/log.c	2008-01-04 16:30:26.000000000 -0500
@@ -99,13 +99,14 @@
 	time_t t;
 	char buf[1024];
 	char *tmp;
+		
+	t = time(NULL);
+	ctime_r(&t, buf);
+	tmp = buf + strlen(buf);
+	buf[strlen(buf)-1]='\t';
+	nfct_snprintf(buf+strlen(buf), 1024-strlen(buf), ct, 0, 0, 0);
 
 	if (fd) {
-		t = time(NULL);
-		ctime_r(&t, buf);
-		tmp = buf + strlen(buf);
-		buf[strlen(buf)-1]='\t';
-		nfct_snprintf(buf+strlen(buf), 1024-strlen(buf), ct, 0, 0, 0);
 		fprintf(fd, "%s\n", buf);
 		fflush(fd);
 	}

Re: conntrack accounting

[Pablo Neira Ayuso]

Hi Ben,

Ben Lentz wrote:
>> Syslog statistics mode still crashes... but only if I *disable*
>> LogFile, too.
>>
> I've fixed the crash when stats LogFile is off and stats Syslog is on,
> patch attached.

Applied. Thanks a lot for investigating and fixing this.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: conntrack accounting

[Pablo Neira Ayuso]

Ben Lentz wrote:
>> One more thing... it looks like daemon mode doesn't detach from it's
>> controlling terminal, still shows "pts/1", and hangs upon logout:
> 
> Added setsid(), patch attached.

Applied. Thanks again.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: conntrack accounting

[Ben Lentz]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]


> Ben Lentz wrote:
>   
>>> One more thing... it looks like daemon mode doesn't detach from it's
>>> controlling terminal, still shows "pts/1", and hangs upon logout:
>>>       
>> Added setsid(), patch attached.
>>     
>
> Applied. Thanks again.
>   
Hmmm... file descriptors for STDIN, STDOUT, and STDERR should be closed, 
too... the setsid() detaches the controlling terminal but a terminal 
still hangs at logout after starting the daemon. A quick-and-dirty patch 
is attached, but there's probably a "better" way to do this.

Thanks very much for considering my patches! I really appreciate the 
work you've done implementing my suggestions!

[-- Attachment #2: conntrack-tools-0.9.6-close-fd.patch --]
[-- Type: text/plain, Size: 342 bytes --]

--- src/main.c.orig	2008-01-04 21:19:52.000000000 -0500
+++ src/main.c	2008-01-04 21:42:52.000000000 -0500
@@ -287,6 +287,10 @@
 			exit(EXIT_FAILURE);
 		}
 
+		close(0);
+		close(1);
+		close(2);
+
 		dlog(STATE(log), LOG_NOTICE, "-- starting in daemon mode --");
 	} else
 		dlog(STATE(log), LOG_NOTICE, "-- starting in console mode --");

Re: conntrack accounting

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]

Ben Lentz wrote:
> Hmmm... file descriptors for STDIN, STDOUT, and STDERR should be closed,
> too... the setsid() detaches the controlling terminal but a terminal
> still hangs at logout after starting the daemon. A quick-and-dirty patch
> is attached, but there's probably a "better" way to do this.

I have applied a similar patch. I have also moved the initialization
before the daemonization so that the error messages that may occur are
still printed ;)

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1553 bytes --]

Index: src/main.c
===================================================================
--- src/main.c	(revisión: 7171)
+++ src/main.c	(copia de trabajo)
@@ -270,6 +270,18 @@
 	}
 	close(ret);
 
+	/*
+	 * initialization process
+	 */
+
+	if (init() == -1) {
+		close_log();
+		fprintf(stderr, "ERROR: conntrackd cannot start, please "
+				"check the logfile for more info\n");
+		unlink(CONFIG(lockfile));
+		exit(EXIT_FAILURE);
+	}
+
 	/* Daemonize conntrackd */
 	if (type == DAEMON) {
 		pid_t pid, sid;
@@ -287,23 +299,15 @@
 			exit(EXIT_FAILURE);
 		}
 
+		close(STDIN_FILENO);
+		close(STDOUT_FILENO);
+		close(STDERR_FILENO);
+
 		dlog(STATE(log), LOG_NOTICE, "-- starting in daemon mode --");
 	} else
 		dlog(STATE(log), LOG_NOTICE, "-- starting in console mode --");
 
 	/*
-	 * initialization process
-	 */
-
-	if (init() == -1) {
-		close_log();
-		fprintf(stderr, "ERROR: conntrackd cannot start, please "
-				"check the logfile for more info\n");
-		unlink(CONFIG(lockfile));
-		exit(EXIT_FAILURE);
-	}
-
-	/*
 	 * run main process
 	 */
 	run();
Index: ChangeLog
===================================================================
--- ChangeLog	(revisión: 7171)
+++ ChangeLog	(copia de trabajo)
@@ -29,6 +29,7 @@
 o minor irrelevant fixes for uncommon error paths and fix several typos
 o detach daemon from its terminal (Ben Lenitz <BLentz@channing-bete.com>)
 o obsolete `-S' option: Use information provided by the config file
+o daemonize conntrackd after initialization
 
 version 0.9.5 (2007/07/29)
 ------------------------------

Re: conntrack accounting

[Jan Engelhardt]

On Jan 5 2008 09:22, Ben Lentz wrote:
>> Ben Lentz wrote:
>>   
>> > > One more thing... it looks like daemon mode doesn't detach from it's
>> > > controlling terminal, still shows "pts/1", and hangs upon logout:
>> > >       
>> > Added setsid(), patch attached.
>> >     
>>
>> Applied. Thanks again.
>>   
> Hmmm... file descriptors for STDIN, STDOUT, and STDERR should be closed, too...
> the setsid() detaches the controlling terminal but a terminal still hangs at
> logout after starting the daemon. A quick-and-dirty patch is attached, but
> there's probably a "better" way to do this.
>
> Thanks very much for considering my patches! I really appreciate the work
> you've done implementing my suggestions!
>
>

close(STDIN_FILENO);
close(STDOUT_FILENO);
close(STDERR_FILENO);

Re: conntrack accounting

[Pablo Neira Ayuso]

Ben Lentz wrote:
> Thanks very much for considering my patches! I really appreciate the
> work you've done implementing my suggestions!

I just implemented buffer logging that guarantees that fflush is called
if the buffer is full. Have a look at LogFileBufferSize. This must
improve performance under very busy firewall. Does syslog have any
similar setting?

The logging format must be discussed before the release. It would be
fairly easy to dump the connection logging info in XML instead of plain
text and I'm not sure if current format is fine.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: conntrack accounting

[Ben Lentz]

> I just implemented buffer logging that guarantees that fflush is called
> if the buffer is full. Have a look at LogFileBufferSize. This must
> improve performance under very busy firewall. Does syslog have any
> similar setting?
>   

That's good question, I'm not really sure what buffering is available in 
the various syslog implementations.

> The logging format must be discussed before the release. It would be
> fairly easy to dump the connection logging info in XML instead of plain
> text and I'm not sure if current format is fine.

In my opinion, the existing plain text log format is fine. I have an 
existing tool that's used to parse out the existing plain text iptables 
syslog data, and I plan on adapting it to support the similar format 
produced by conntrackd. If there are changes that could be made to make 
the formats similar, that might be nice (so the "family" netfilter 
softwares log uniformly (?)). XML would be nice, but I won't use it at 
this time.