excluding auditd events

excluding auditd events

[Mr Dash Four]

Further to the discussion I've had with Eric Paris, Steve Grubb and various other members over on the SELinux mailing list, I am now glad I am able to seek help and advice as well as prompt further debate on variety of issues concerning the audit daemon.

The main reason for wanting to join the list was that I was having difficulty in trying to exclude certain type of messages (below) for a particular SELinux type being reported to the auditd daemon.

In particular, I wanted to exclude the following from being reported:

msgtype={USER_ACCT|CRED_ACQ|USER_START|CRED_DISP|USER_END}
obj_type=crond_t
success=0

When I try to add this as a rule with "auditctl -A exclude,never -F msgtype=USER_ACCT -F obj_type=crond_t -F success=0" I get "Only msgtype field can be used with exclude filter".

If left unchecked, I am getting "success" messages on a pretty regular intervals (every time cron daemon runs!), thus filling up my audit logs unnecessarily. This won't normally be a big issue on a small system, but when one has to scan thousands of logs every single hour it becomes a bit of a burden. I won't even go into the issue of filling up disk space and consuming system resources unnecessarily.

After having exchanged a few emails with Eric and Steve on that particular issue, it became apparent that since this is a kernel restriction the only feasible solution would be to use "user,exclude" and then the SELinux role I want filtered, though currently no message-type filtering is implemented - either in the kernel, or the audit daemon.

I haven't studied the auditd code at all, but to me, this is far too restrictive and if I am to filter on just SELinux context/role/user etc, I am running the risk, however small, of not seeing a security-related messages, which are of potential interest to me as a developer and sysadmin.

If I am able to filter on (SELinux) user role and message type (even in userspace) that would be good-enough match!

Another couple of things which immediately struck me as soon as I "acquainted" myself with the audit daemon. To me, it is vitally important if any kind of reporting is done on security-related events on a particular system, that reporting to be sufficiently "verifiable" to prevent tampering. Is there such feature implemented in the audit daemon to spot tampering - both on a record level, as well as the audit file as a whole? I couldn't spot such feature during the (admittedly) short time I have studied the audit daemon.

Finally, another feature which I am badly missing - the ability to see audit files loaded remotely by the audit-viewer (audit logs located on network shares for example) - this is currently missing and the audit viewer bluntly refuses to load audit file if this file is remotely based and not on the local file system. Is something planned in that respect to enable this?

Re: excluding auditd events

[Mr Dash Four]

> That would be "user,never". The audit daemon does no filtering. Its in a race with the 
> kernel to put events to disk before the kernel's backlog overflows.
>   
Yeah, that's it, sorry. Is this backlog configurable (maybe in the same 
way tcp/udp buffers are via the sysctl daemon)?

> The user filter can filter on: pid, uid, gid, auid, and any part of the subject's 
> selinux label. The only thing not being filtered on that is available is the event's 
> record type. There are no other attributes available that can be filtered on.
>   
You mean the message type? If so, filtering by selinux label and message 
type is sufficient, at least for my immediate needs.

> It is protected by file permissions. You must be root to write to the file. If you want 
> to gpg armor your files when you archive them, its possible to script that.
Actually, I was thinking more of having a hash against each record 
(horizontally) and, maybe a separate hash over the current tuple of 
time:audit count (vertically).

It was just an idea and is similar to what I have implemented in my 
database-based log system (using PostgreSQL) - a token (via smartcard) 
is taken when the logging starts (at boot up using dracut - I have 
designed a module for this too) and this token is then used to create a 
hash when each log/record is inserted into the system and inserts that 
has as part of the record itself - that prevents tampering with a single 
record, while a separate hash is kept for a single column across the 
entire table (timestamp and transaction id in my case) - that prevents 
tampering entire logs (i.e. adding/deleting entries).

>  But we've 
> always taken the position that if someone obtains root privileges, tampering with the 
> logs is the last thing you need to worry about.
I am sure someone said the same thing before SELinux was invented and 
implemented in Fedora. In SELinux even if you are root you are still 
restricted by the domain you operate in and by the policies in existence 
for that particular domain. My view has always been that you can never 
be too careful and this adds another level of security - an additional 
barrier for "hackers" to have to break, if you like.

>> Finally, another feature which I am badly missing - the ability to see
>> audit files loaded remotely by the audit-viewer (audit logs located on
>> network shares for example) - this is currently missing and the audit
>> viewer bluntly refuses to load audit file if this file is remotely based
>> and not on the local file system. Is something planned in that respect to
>> enable this?
>>     
>
> No idea.
>   
It is a restriction in audit-viewer - at least in the version I am using 
(stock FC13).

Re: excluding auditd events

[Steve Grubb]

On Thursday, May 26, 2011 09:16:59 AM Mr Dash Four wrote:
> > That would be "user,never". The audit daemon does no filtering. Its in a
> > race with the kernel to put events to disk before the kernel's backlog
> > overflows.
> 
> Yeah, that's it, sorry. Is this backlog configurable (maybe in the same
> way tcp/udp buffers are via the sysctl daemon)?

Yes, that is the -b option to auditctl. No matter what you set it to, it can be 
overflowed by the right conditions. This is why the audit daemon does no filtering.


> > The user filter can filter on: pid, uid, gid, auid, and any part of the
> > subject's selinux label. The only thing not being filtered on that is
> > available is the event's record type. There are no other attributes
> > available that can be filtered on.
> 
> You mean the message type? 

An event is composed of records. Sometimes just one record, sometimes 5 or 6. but they 
are all linked with a timestamp and serial number.


> If so, filtering by selinux label and message type is sufficient, at least for my
> immediate needs.
> 
> > It is protected by file permissions. You must be root to write to the
> > file. If you want to gpg armor your files when you archive them, its
> > possible to script that.
> 
> Actually, I was thinking more of having a hash against each record
> (horizontally) and, maybe a separate hash over the current tuple of
> time:audit count (vertically).

I have been toying with the idea of creating a detached signature where the audit 
daemon leaves a public key for use in verifying the integrity of the log. But that 
still does not prevent someone from mimicing the algorithm themselves after modifying 
the files. For ultimate protection, we suggest remote logging to a box that has 
restricted access.

 
> > always taken the position that if someone obtains root privileges,
> > tampering with the logs is the last thing you need to worry about.
> 
> I am sure someone said the same thing before SELinux was invented and
> implemented in Fedora. In SELinux even if you are root you are still
> restricted by the domain you operate in and by the policies in existence
> for that particular domain. My view has always been that you can never
> be too careful and this adds another level of security - an additional
> barrier for "hackers" to have to break, if you like.

In the case where you are running SE Linux correctly - where users are not logging in 
as unconfined_t, then you have to be audit admin to do anything with the audit system.

-Steve

Re: excluding auditd events

[Mr Dash Four]

> Yes, that is the -b option to auditctl. No matter what you set it to, it can be 
> overflowed by the right conditions. This is why the audit daemon does no filtering.
>   
Thanks, I realise it doesn't bring 100% certainty against overflowing 
(what does?), but it at least helps.

>> You mean the message type? 
>>     
>
> An event is composed of records. Sometimes just one record, sometimes 5 or 6. but they 
> are all linked with a timestamp and serial number.
>   
So filtering could be done on message type and SElinux subject, 
eventually, by using "user,never"? I take it, this isn't implemented yet 
(the message type filter part)?

> I have been toying with the idea of creating a detached signature where the audit 
> daemon leaves a public key for use in verifying the integrity of the log. But that 
> still does not prevent someone from mimicing the algorithm themselves after modifying 
> the files.
That kind of tampering won't be possible if the token is taken from a 
trusted source (smartcard in my case), is then kept in-memory and is 
never visible to anyone/anything except the audit daemon. That token is 
then used to create all the hashes needed to do the verification. When 
sysadmin (or other authorised personnel) need to do record verification 
they simply insert the smartcard, run a separate tool 
(ausearch/aureport-type tool), which retrieves the token again from the 
smartcard and then verifies whatever needs to be verified in the logs 
and produces the report needed.

At least this is how I have it currently implemented in my database.

>  For ultimate protection, we suggest remote logging to a box that has 
> restricted access.
>   
That is certainly a possibility (but then again the box needs to be 
"secure"), though since I am not very familiar with the audit daemon 
I'll just ask - is the connection between the 2 daemons (on the secure 
box as well as the daemon sending the logs) encrypted so to prevent 
tampering in-route (man in the middle etc attacks)?

Re: excluding auditd events

[Steve Grubb]

On Thursday, May 26, 2011 10:07:57 AM Mr Dash Four wrote:
> >  For ultimate protection, we suggest remote logging to a box that has 
> > restricted access.
> 
> That is certainly a possibility (but then again the box needs to be 
> "secure"), though since I am not very familiar with the audit daemon 
> I'll just ask - is the connection between the 2 daemons (on the secure 
> box as well as the daemon sending the logs) encrypted so to prevent 
> tampering in-route (man in the middle etc attacks)?

Sort of. We have kerberos support, but its not enabled at the moment. The reason being 
is that the kerberos libraries were in /usr/lib64 which is a big problem if the audit 
system started before the nfs components (and it does). I think the kerberos libraries 
might have been moved so we could potentially turn that on sometime soon - but I have 
not been updating or testing the code. If you build your own packages, you can turn it 
on now. 

-Steve

Re: excluding auditd events

[Mr Dash Four]

> Sort of. We have kerberos support, but its not enabled at the moment. The reason being 
> is that the kerberos libraries were in /usr/lib64 which is a big problem if the audit 
> system started before the nfs components (and it does). I think the kerberos libraries 
> might have been moved so we could potentially turn that on sometime soon - but I have 
> not been updating or testing the code. If you build your own packages, you can turn it 
> on now. 
>   
Thanks, I'll try as soon as I am able to build a decent version of the 
daemon (as I already pointed out - the show-stopper for me is the older 
version of perl on my FC13 system, which, from what I remember, is a 
dependency for building the remote-logging part of the daemon). When I 
get more confident about understanding the audit daemon better (and 
having looked properly in the code itself) I may come up with few other 
ideas.

Re: excluding auditd events

[Steve Grubb]

On Thursday, May 26, 2011 10:23:09 AM Mr Dash Four wrote:
> > Sort of. We have kerberos support, but its not enabled at the moment. The
> > reason being is that the kerberos libraries were in /usr/lib64 which is
> > a big problem if the audit system started before the nfs components (and
> > it does). I think the kerberos libraries might have been moved so we
> > could potentially turn that on sometime soon - but I have not been
> > updating or testing the code. If you build your own packages, you can
> > turn it on now.
> 
> Thanks, I'll try as soon as I am able to build a decent version of the
> daemon (as I already pointed out - the show-stopper for me is the older
> version of perl on my FC13 system, which, from what I remember, is a
> dependency for building the remote-logging part of the daemon).

The audit package has no dependency on perl.

-Steve

Re: excluding auditd events

[Mr Dash Four]

> The audit package has no dependency on perl.
>   
Oops, sorry, I meant python! My bad!

[rant]
The other dependency I can't overcome (which I could not remember the 
other day when the same issue popped up during our discussion Steve) was 
the libprelude-devel package, simply because in the way the current 
libprelude-devel package is configured,  there is no way on gods green 
Earth I could install both i686 and x86_64 versions of the tool to 
enable cross-compilation.

There are hardcoded lib paths in "/usr/bin/libprelude-config" 
("libdirs=-L/usr/lib" in the i686 version, while "libdirs=-L/usr/lib64" 
in the x86_64 version) which are obviously arch-dependent.

Not to mention this little gem in prelude-intttypes.h - "#define 
__PRELUDE_HAVE_64BIT_LONG", which is commented out in the i686 version, 
instead of using a much-more sensible approach and either 1) define 
these variables depending on the current arch used for compilation - 
i.e. the "--target" value used in rpmbuild for example; or 2) use two 
completely separate versions of these files (say 
libprelude-config-<arch> and prelude-inttypes-<arch>.h) and integrate 
those in the pkgconfig files.

All this is a classic example of careless programming in my view!
[/rant]

Without libprelude-devel I can't compile/build audispd-plugins, without 
audispd-plugins I can't have remote logging in auditd (you get the 
picture!).

Re: excluding auditd events

[LC Bruzenak]

On Thu, 2011-05-26 at 16:22 +0100, Mr Dash Four wrote:
...
> 
> Without libprelude-devel I can't compile/build audispd-plugins, without 
> audispd-plugins I can't have remote logging in auditd (you get the 
> picture!).


I am using mock to compile both the 32-bit and 64-bit versions of all
the prelude code on rhel6, starting with the fc14 src rpm. You should be
able to do this on your distro as well.

LCB

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: excluding auditd events

[Mr Dash Four]

> I am using mock to compile both the 32-bit and 64-bit versions of all
> the prelude code on rhel6, starting with the fc14 src rpm. You should be
> able to do this on your distro as well.
>   
Thanks for the tip - never used mock myself, but will give it a go and 
see where does that take me!

Re: excluding auditd events

[Mr Dash Four]

>> That is certainly a possibility (but then again the box needs to be 
>> "secure"), though since I am not very familiar with the audit daemon 
>> I'll just ask - is the connection between the 2 daemons (on the secure 
>> box as well as the daemon sending the logs) encrypted so to prevent 
>> tampering in-route (man in the middle etc attacks)?
>>     
>
> Sort of. We have kerberos support, but its not enabled at the moment. The reason being 
> is that the kerberos libraries were in /usr/lib64 which is a big problem if the audit 
> system started before the nfs components (and it does). I think the kerberos libraries 
> might have been moved so we could potentially turn that on sometime soon - but I have 
> not been updating or testing the code. If you build your own packages, you can turn it 
> on now. 
>   
A couple of recent developments on that front - I was able to finally 
build the 2.1.1.1-1 version of *all* audit packages even without mock. I 
did what I am already accustomed to do when faced with this sort of 
"multilib" cross-compilation issues and partially installed (understand 
extracted) just the files needed for the host arch in order to pass 
compilation, linking and final build.

That included python libraries and library objects (from python and 
python-devel as I already had the rest installed), making sure there are 
no conflicts (diff was indeed my friend). I also used the kerberos 
option (--enable-gssapi-krb5=yes) though to make this option work it 
required some creativity on my part. There were also some Fedora-related 
tweaks and hacks I did, but I will refer to those privately.

The package was built and installed on my host machine - it works as 
expected, though I am still not getting matches on the AUDIT target - 
even with this new version! There is also a bizarre date matching in 
ausearch as I discovered:

-bash-4.1# ausearch -m AVC -ts "May, 26 2011" | more
Invalid start date (May, 26 2011). Month, Day, and Year are required.
-bash-4.1# ausearch -m AVC -ts "May 26 2011" | more
Invalid start date (May 26 2011). Month, Day, and Year are required.
-bash-4.1# ausearch -m AVC -ts "26 May 2011" | more
Invalid start date (26 May 2011). Month, Day, and Year are required.
-bash-4.1# ausearch -m AVC -ts "26 05 2011" | more
Invalid start date (26 05 2011). Month, Day, and Year are required.
-bash-4.1# ausearch -m AVC -ts "05/26/2011" | more
Error parsing start date (05/26/2011)
-bash-4.1# ausearch -m AVC -ts "26/05/2011" | more
Error parsing start date (26/05/2011)
-bash-4.1# ausearch -m AVC -ts "2011/26/05" | more
Invalid start date (2011/26/05). Month, Day, and Year are required.
-bash-4.1# ausearch -m AVC -ts "2011/05/26" | more
Invalid start date (2011/05/26). Month, Day, and Year are required.

-bash-4.1# ausearch -m AVC -ts "05/26/11" | more <- works!
-bash-4.1# ausearch -m AVC -ts yesterday | more <- works!
-bash-4.1# ausearch -m AVC -ts today | more <- works!

So, it turns out that apart from the human-like date description like 
"yesterday" and "today", ausearch only accepts 2-digit years! I thought 
we have long-passed these Y2K-related issues - that is so 1999. That is 
assuming I didn't mess things up, which is also a possibility, of 
course! The error messages I was getting above did not help my cause either!

Re: excluding auditd events

[LC Bruzenak]

> So, it turns out that apart from the human-like date description like 
> "yesterday" and "today", ausearch only accepts 2-digit years! I thought 
> we have long-passed these Y2K-related issues - that is so 1999. That is 
> assuming I didn't mess things up, which is also a possibility, of 
> course! The error messages I was getting above did not help my cause either!

Too bad on not using mock; it is in my experience easier than grabbing
pieces needed and certainly easier when those pieces get revised.

You must have read the ausearch man page which describes the date usage
and subsequently followed the pointer to the localtime man page. The
dates work as described in those pages:

$ sudo ausearch -ts 05/30/2011 | less  
works fine for me on FC10 & RHEL6.

Look at your system time - is it correct?
Use the "date" command.
Check your LC_TIME ENV variable.


> -bash-4.1# ausearch -m AVC -ts "05/26/11" | more <- works!

$ sudo ausearch -m AVC -ts "05/26/11"
Error - year is 11

This also is the same for me on FC10 & RHEL6 (audit-1.7.16 and
audit-2.1-5 respectively) . So my guess is your LC_TIME or locale value
is set for 2-digit dates or something alone those lines. The "date"
command should yield a clue, especially "date +%x".

LCB
-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: excluding auditd events

[Mr Dash Four]

> Too bad on not using mock; it is in my experience easier than grabbing
> pieces needed and certainly easier when those pieces get revised.
>   
The main reason for not using mock (without drifting too much off topic) 
is because it sets up the chroot environment to mirror the target arch, 
which is not suitable to me at all - the main reason I use 
cross-compilation is to utilise the power of the build machine and its 
architecture - the last think I expected is mock to install GCC and its 
accompanying tools for the (slow) target arch instead of install/build 
them for the build arch with the ability for them to cross-compile.

> $ sudo ausearch -ts 05/30/2011 | less  
> works fine for me on FC10 & RHEL6.
>   
-bash-4.1# ausearch -ts 05/30/2011 | less
Error parsing start date (05/30/2011)

> Look at your system time - is it correct?
> Use the "date" command.
> Check your LC_TIME ENV variable.
>   
-bash-4.1# date
Wed Jun  1 15:41:53 BST 2011
-bash-4.1# echo $LC_TIME

-bash-4.1#

(I am executing this as root as you can imagine).

>> -bash-4.1# ausearch -m AVC -ts "05/26/11" | more <- works!
>>     
>
> $ sudo ausearch -m AVC -ts "05/26/11"
> Error - year is 11
>   
Interesting! I get the desired results and the machine on which this is 
executed has all the latest (and greatest) packages in it, so I am not 
using something which could be considered outdated (even though it is 
all FC13-based a lot of the stuff there is compiled and build from the 
newest available sources).


> This also is the same for me on FC10 & RHEL6 (audit-1.7.16 and
> audit-2.1-5 respectively) . So my guess is your LC_TIME or locale value
> is set for 2-digit dates or something alone those lines. The "date"
> command should yield a clue, especially "date +%x".
>   
-bash-4.1# ausearch --version
ausearch version 2.1.1

-bash-4.1# date +%x
01/06/11