RHEL-AS-4.4 and auditd-1.0.14

RHEL-AS-4.4 and auditd-1.0.14

[Simon Jones]

Hi,

We have been having some OOM problems over the last week.  We think  
it is stemming from auditd-1.0.14.

I've had a quick look over the archives and couldn't find anything,  
so if this has already been fixed, please be kind...

I went from using the standard CAPP.rules example file to the  
following audit.rules file:

-D
-w /etc -p w -k ETC
-w /etc/sysconfig -p w -k SYSCONFIG
-w /caer/e/cnf -p w -k DMS_CNF
-w /caer/g/cnf -p w -k GAS_CNF
-w /bin/su -p x -k SBIN

A glance at cat /proc/slabinfo shows that there may be a memory leak:
After two minutes:
size-32            13447  13447     32  119    1 : tunables  120    
60    8 : slabdata    113    113      0
After several hours:
size-32           18598891 18599105     32  119    1 : tunables   
120   60    8 : slabdata 156295 156295      0


Whereas on a server not running the auditd daemon a cat /proc/ 
slabinfo gives:
After two minutes:
size-32             3556   3808     32  119    1 : tunables  120    
60    8 : slabdata     32     32      0
After several hours:
size-32             3601   3808     32  119    1 : tunables  120    
60    8 : slabdata     32     32      0


OOM starts killing off processes (seemingly at random) even though  
there appears to be plenty of memory free (physical and swap).

The above tests are on DELL 1650's with 2GB of RAM running  
2.6.9-42.ELsmp #1 SMP.

I found this https://bugzilla.redhat.com/bugzilla/show_bug.cgi? 
id=193542#c15 bug that seems to have a similar problem...  Anyone  
else come across this issue?  If so has it been fixed in 1.0.15?

As I mentioned, using the following rules file seems to be  
significantly more stable, (so perhaps my rules file has something to  
do with it too):

This is the original rules file:

##
## This file contains a sample audit configuration.  Combined with the
## system events that are audited by default, this set of rules causes
## audit to generate records for the auditable events specified by the
## Controlled Access Protection Profile (CAPP).
##
## It should be noted that this set of rules identifies directories by
## leaving a / at the end of the path. These need to be updated to be
## a watch for each file in that directory. This is because a watch on
## a directory only triggers when the directory's inode is updated with
## meta data. To have accurate events, a watch should be place on each
## file. Because each installation is different, we leave that as a
## site customization.
##

## Remove any existing rules
-D

## Increase buffer size to handle the increased number of messages.
## Feel free to increase this if the machine panic's
-b 8192

## Set failure mode to panic
-f 2

##
## FAU_SAR.1, FAU_SAR.2, FMT_MTD.1
## successful and unsuccessful attempts to read information from the
## audit records; all modifications to the audit trail
##
# -w /var/log/audit/ -k LOG_audit
#-w /var/log/audit/audit_log -k LOG_audit_log
#-w /var/log/audit/audit_log.1 -k LOG_audit_log
#-w /var/log/audit/audit_log.2 -k LOG_audit_log
#-w /var/log/audit/audit_log.3 -k LOG_audit_log
#-w /var/log/audit/audit_log.4 -k LOG_audit_log

##
## FAU_SEL.1, FMT_MTD.1
## modifications to audit configuration that occur while the audit
## collection functions are operating; all modications to the set of
## audited events
##
-w /etc/auditd.conf -k CFG_auditd.conf
-w /etc/audit.rules -k CFG_audit.rules

##
## FDP_ACF.1, FMT_MSA.1, FMT_MTD.1, FMT_REV.1
## all requests to perform an operation on an object covered by the
## SFP; all modifications of the values of security attributes;
## modifications to TSF data; attempts to revoke security attributes
##

## Objects covered by the Security Functional Policy (SFP) are:
## - File system objects (files, directories, special files, extended  
attributes)
## - IPC objects (SYSV shared memory, message queues, and semaphores)

## Operations on file system objects - by default, only monitor
## files and directories covered by filesystem watches. Replace
## "possible" with "always" to create audit records for all uses of this
## syscall.

## Changes in ownership and permissions
-a entry,possible -S chmod
-a entry,possible -S fchmod
-a entry,possible -S chown
-a entry,possible -S chown32
-a entry,possible -S fchown
-a entry,possible -S fchown32
-a entry,possible -S lchown
-a entry,possible -S lchown32
## For x86_64,ia64 architectures, disable any *32 rules above

## File content modification. Permissions are checked at open time,
## monitoring individual read/write calls is not useful.
-a entry,possible -S creat
-a entry,possible -S open
-a entry,possible -S truncate
-a entry,possible -S truncate64
-a entry,possible -S ftruncate
-a entry,possible -S ftruncate64
## For x86_64,ia64 architectures, disable any *64 rules above

## directory operations
-a entry,possible -S mkdir
-a entry,possible -S rmdir

## moving, removing, and linking
-a entry,possible -S unlink
-a entry,possible -S rename
-a entry,possible -S link
-a entry,possible -S symlink

## Extended attribute operations
## Enable if you are interested in these events
#-a entry,always -S setxattr
#-a entry,always -S lsetxattr
#-a entry,always -S fsetxattr
#-a entry,always -S removexattr
#-a entry,always -S lremovexattr
#-a entry,always -S fremovexattr

## special files
-a entry,always -S mknod

## Other file system operations
-a entry,always -S mount
-a entry,always -S umount
-a entry,always -S umount2
## For x86_64 architecture, disable umount rule
## For ia64 architecture, disable umount2 rule

## SYSV message queues
## Enable if you are interested in these events (x86)
## msgctl
#-a entry,always -S ipc -F a0=14
## msgget
#-a entry,always -S ipc -F a0=13
## Enable if you are interested in these events (x86_64,ia64)
#-a entry,always -S msgctl
#-a entry,always -S msgget

## SYSV semaphores
## Enable if you are interested in these events (x86)
## semctl
#-a entry,always -S ipc -F a0=3
## semget
#-a entry,always -S ipc -F a0=2
## semop
#-a entry,always -S ipc -F a0=1
## semtimedop
#-a entry,always -S ipc -F a0=4
## Enable if you are interested in these events (x86_64, ia64)
#-a entry,always -S semctl
#-a entry,always -S semget
#-a entry,always -S semop
#-a entry,always -S semtimedop

## SYSV shared memory
## Enable if you are interested in these events (x86)
## shmctl
#-a entry,always -S ipc -F a0=24
## shmget
#-a entry,always -S ipc -F a0=23
## Enable if you are interested in these events (x86_64, ia64)
#-a entry,always -S shmctl
#-a entry,always -S shmget

##
## FIA_USB.1
## success and failure of binding user security attributes to a subject
##
## Enable if you are interested in these events
##
#-a entry,always -S clone
#-a entry,always -S fork
#-a entry,always -S vfork
## For ia64 architecture, disable fork and vfork rules above, and
## enable the following:
#-a entry,always -S clone2

##
## FMT_MSA.3
## modifications of the default setting of permissive or restrictive
## rules, all modifications of the initial value of security attributes
##
## Enable if you are interested in these events
##
#-a entry,always -S umask

##
## FPT_STM.1
## changes to the time
##
-a entry,always -S adjtimex
-a entry,always -S settimeofday

##
## FTP_ITC.1
## set-up of trusted channel
##
-w /usr/sbin/stunnel -p x
-a entry,possible -S execve

##
## Security Databases
##

## at configuration & scheduled jobs
-w /var/spool/at -k LOG_at
-w /etc/at.allow -k CFG_at.allow
-w /etc/at.deny -k CFG_at.deny

## cron configuration & scheduled jobs
-w /etc/cron.allow -p wa -k CFG_cron.allow
-w /etc/cron.deny -p wa -k CFG_cron.deny
-w /etc/cron.d/ -p wa -k CFG_cron.d
-w /etc/cron.daily/ -p wa -k CFG_cron.daily
-w /etc/cron.hourly/ -p wa -k CFG_cron.hourly
-w /etc/cron.monthly/ -p wa -k CFG_cron.monthly
-w /etc/cron.weekly/ -p wa -k CFG_cron.weekly
-w /etc/crontab -p wa -k CFG_crontab
-w /var/spool/cron/root -k CFG_crontab_root

## user, group, password databases
-w /etc/group -p wa -k CFG_group
-w /etc/passwd -p wa -k CFG_passwd
-w /etc/gshadow -k CFG_gshadow
-w /etc/shadow -k CFG_shadow
-w /etc/security/opasswd -k CFG_opasswd

## login configuration and information
-w /etc/login.defs -p wa -k CFG_login.defs
-w /etc/securetty -k CFG_securetty
-w /var/log/faillog -k LOG_faillog
-w /var/log/lastlog -k LOG_lastlog

## network configuration
-w /etc/hosts -p wa -k CFG_hosts
-w /etc/sysconfig/

## system startup scripts
-w /etc/inittab -p wa -k CFG_inittab
-w /etc/rc.d/init.d/
-w /etc/rc.d/init.d/auditd -p wa -k CFG_initd_auditd

## library search paths
-w /etc/ld.so.conf -p wa -k CFG_ld.so.conf

## local time zone
-w /etc/localtime -p wa -k CFG_localtime

## kernel parameters
-w /etc/sysctl.conf -p wa -k CFG_sysctl.conf

## modprobe configuration
-w /etc/modprobe.conf -p wa -k CFG_modprobe.conf

## pam configuration
-w /etc/pam.d/

## postfix configuration
-w /etc/aliases -p wa -k CFG_aliases
-w /etc/postfix/ -p wa -k CFG_postfix

## ssh configuration
-w /etc/ssh/sshd_config -k CFG_sshd_config

## stunnel configuration
-w /etc/stunnel/stunnel.conf -k CFG_stunnel.conf
-w /etc/stunnel/stunnel.pem -k CFG_stunnel.pem

## vsftpd configuration
#-w /etc/vsftpd.ftpusers -k CFG_vsftpd.ftpusers
#-w /etc/vsftpd/vsftpd.conf -k CFG_vsftpd.conf

## Not specifically required by CAPP; but common sense items
-a exit,always -S sethostname
-w /etc/issue -p wa -k CFG_issue
-w /etc/issue.net -p wa -k CFG_issue.net

## Put your own watches after this point


Regards,

Simon.

Re: RHEL-AS-4.4 and auditd-1.0.14

[Steve Grubb]

On Thursday 08 February 2007 23:12, Simon Jones wrote:
> I've had a quick look over the archives and couldn't find anything,  
> so if this has already been fixed, please be kind...

No one has reported any problem of this kind.

> I went from using the standard CAPP.rules example file to the  
> following audit.rules file:

This reduces what the kernel is doing. Does this also reduce the number of 
events hitting your audit logs?


> -D
> -w /etc -p w -k ETC

This only records writes to the /etc directory and not the files in the /etc 
directory.

> -w /etc/sysconfig -p w -k SYSCONFIG
> -w /caer/e/cnf -p w -k DMS_CNF
> -w /caer/g/cnf -p w -k GAS_CNF
> -w /bin/su -p x -k SBIN
>
> A glance at cat /proc/slabinfo shows that there may be a memory leak:
> After two minutes:
> size-32            13447  13447     32  119    1 : tunables  120    
> 60    8 : slabdata    113    113      0
> After several hours:
> size-32           18598891 18599105     32  119    1 : tunables  
> 120   60    8 : slabdata 156295 156295      0

I wonder if you still see the leak if you load the rules but do not start the 
audit daemon? We need to see if its a kernel memory leak or user space. I've 
run valgrind against auditd and do not know of any leaks.

>
> Whereas on a server not running the auditd daemon a cat /proc/
> slabinfo gives:
> After two minutes:
> size-32             3556   3808     32  119    1 : tunables  120    
> 60    8 : slabdata     32     32      0
> After several hours:
> size-32             3601   3808     32  119    1 : tunables  120    
> 60    8 : slabdata     32     32      0

But do you still have the CAPP rules loaded?

> I found this https://bugzilla.redhat.com/bugzilla/show_bug.cgi?
> id=193542#c15 bug that seems to have a similar problem...

similar but different.

> If so has it been fixed in 1.0.15?

No one's reported such an issue...so no one's worked on it. The first step is 
determining if the problem is kernel or user space. Please load the CAPP 
rules without starting the audit daemon and see what that shows.

Thanks,
-Steve

Re: RHEL-AS-4.4 and auditd-1.0.14

[Simon Jones]

Hi Steve,

Thanks for the response.
>
>> I went from using the standard CAPP.rules example file to the
>> following audit.rules file:
>
> This reduces what the kernel is doing. Does this also reduce the  
> number of
> events hitting your audit logs?
>

Yes this did reduce the events hitting the logs quite considerably.

>
> I wonder if you still see the leak if you load the rules but do not  
> start the
> audit daemon? We need to see if its a kernel memory leak or user  
> space. I've
> run valgrind against auditd and do not know of any leaks.

I loaded just the rules and left it overnight and it still looks fine.

size-32             3688   3808     32  119    1 : tunables  120    
60    8 : slabdata     32     32      0

[root@blah ~]# auditctl -l
No rules
AUDIT_WATCH_LIST: dev=9:1, path=/etc, filterkey=ETC, perms=w, valid=0
AUDIT_WATCH_LIST: dev=9:1, path=/etc/sysconfig, filterkey=SYSCONFIG,  
perms=w, valid=0
AUDIT_WATCH_LIST: dev=9:3, path=/caer/e/cnf, filterkey=DMS_CNF,  
perms=w, valid=0
AUDIT_WATCH_LIST: dev=9:3, path=/caer/g/cnf, filterkey=GAS_CNF,  
perms=w, valid=0
AUDIT_WATCH_LIST: dev=9:1, path=/bin/su, filterkey=SBIN, perms=x,  
valid=0

>
>>
>> Whereas on a server not running the auditd daemon a cat /proc/
>> slabinfo gives:
>> After two minutes:
>> size-32             3556   3808     32  119    1 : tunables  120
>> 60    8 : slabdata     32     32      0
>> After several hours:
>> size-32             3601   3808     32  119    1 : tunables  120
>> 60    8 : slabdata     32     32      0
>
> But do you still have the CAPP rules loaded?

This was with no CAPP rules nor auditd running.

>
> No one's reported such an issue...so no one's worked on it. The  
> first step is
> determining if the problem is kernel or user space. Please load the  
> CAPP
> rules without starting the audit daemon and see what that shows.
>
> Thanks,
> -Steve

I loaded the CAPP example rules (and auditd) and it appears to leak  
very slowly.  After a couple of days it's sitting at:

size-32            84728  84728     32  119    1 : tunables  120    
60    8 : slabdata    712    712

We never saw the OOM killer in the six months that we ran with the  
CAPP example rules.

When I use the cut down CAPP rules it will kill the box within about  
2 days.

Regards,

Simon.

Re: RHEL-AS-4.4 and auditd-1.0.14

[Steve Grubb]

On Monday 12 February 2007 17:54, Simon Jones wrote:
> I loaded just the rules and left it overnight and it still looks fine.
>
> size-32             3688   3808     32  119    1 : tunables  120    
> 60    8 : slabdata     32     32      0

Hmm...that would seem to point to the audit daemon. I posted the code for the 
1.0.15 audit package here:

http://people.redhat.com/sgrubb/audit/audit-1.0.15-1.fc4.src.rpm

Maybe you want to build that and give it a try? I'd be curious if you see a 
leak in that version. It does have some cleanups, but nothing I recall as 
fixing a memory leak.

-Steve

Re: RHEL-AS-4.4 and auditd-1.0.14

[Simon Jones]

Hi Steve,

I've installed the latest audit package and it seems to be exactly  
the same.  Overnight:

size-32           208310 208369     32  119    1 : tunables  120    
60    8 : slabdata   1751   1751      0

[sysadmin@blah ~]$ rpm -q audit
audit-1.0.15-1.fc4

I've cut down the rules to a single watch on the /etc directory (I  
realise that this only watches the directory and not the files in it).

No rules
AUDIT_WATCH_LIST: dev=9:1, path=/etc, filterkey=ETC, perms=w, valid=0

Every access to /etc seems to add to the size-32 objects and never  
releases them.

Any other suggestions?

Simon.

On 13/02/2007, at 1:33 PM, Steve Grubb wrote:

> On Monday 12 February 2007 17:54, Simon Jones wrote:
>> I loaded just the rules and left it overnight and it still looks  
>> fine.
>>
>> size-32             3688   3808     32  119    1 : tunables  120
>> 60    8 : slabdata     32     32      0
>
> Hmm...that would seem to point to the audit daemon. I posted the  
> code for the
> 1.0.15 audit package here:
>
> http://people.redhat.com/sgrubb/audit/audit-1.0.15-1.fc4.src.rpm
>
> Maybe you want to build that and give it a try? I'd be curious if  
> you see a
> leak in that version. It does have some cleanups, but nothing I  
> recall as
> fixing a memory leak.
>
> -Steve

Re: RHEL-AS-4.4 and auditd-1.0.14

[Simon Jones]

Hi Steve,

I changed the rule from the /etc watch to individual files in the / 
etc directory and that seems to have settled it down.

It seems to be a problem with watching directories only.

Simon.

On 14/02/2007, at 10:07 AM, Simon Jones wrote:

> Hi Steve,
>
> I've installed the latest audit package and it seems to be exactly  
> the same.  Overnight:
>
> size-32           208310 208369     32  119    1 : tunables  120    
> 60    8 : slabdata   1751   1751      0
>
> [sysadmin@blah ~]$ rpm -q audit
> audit-1.0.15-1.fc4
>
> I've cut down the rules to a single watch on the /etc directory (I  
> realise that this only watches the directory and not the files in it).
>
> No rules
> AUDIT_WATCH_LIST: dev=9:1, path=/etc, filterkey=ETC, perms=w, valid=0
>
> Every access to /etc seems to add to the size-32 objects and never  
> releases them.
>
> Any other suggestions?
>
> Simon.
>
> On 13/02/2007, at 1:33 PM, Steve Grubb wrote:
>
>> On Monday 12 February 2007 17:54, Simon Jones wrote:
>>> I loaded just the rules and left it overnight and it still looks  
>>> fine.
>>>
>>> size-32             3688   3808     32  119    1 : tunables  120
>>> 60    8 : slabdata     32     32      0
>>
>> Hmm...that would seem to point to the audit daemon. I posted the  
>> code for the
>> 1.0.15 audit package here:
>>
>> http://people.redhat.com/sgrubb/audit/audit-1.0.15-1.fc4.src.rpm
>>
>> Maybe you want to build that and give it a try? I'd be curious if  
>> you see a
>> leak in that version. It does have some cleanups, but nothing I  
>> recall as
>> fixing a memory leak.
>>
>> -Steve
>

Re: RHEL-AS-4.4 and auditd-1.0.14

[Steve Grubb]

On Tuesday 13 February 2007 18:20:04 Simon Jones wrote:
> I changed the rule from the /etc watch to individual files in the /
> etc directory and that seems to have settled it down.
>
> It seems to be a problem with watching directories only.

Hmm. The daemon doesn't make decisions at all based on what's in the event. 
Offhand, I don't have any other suggestions other than a session with 
valgrind. There's very little memory allocating done by the audit daemon to 
make sure we do not have memory leaks.

-Steve