BIG performance hit with auditd on large systems (>64 CPUs)

BIG performance hit with auditd on large systems (>64 CPUs)

[Klaus Lichtenwalder]

Hi,

we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74
CPUs and >= 400G RAM.
When the system is busy with large SAP jobs, it goes onto its knees with
cpu %system up to 80%, thus making the SAP jobs run twice as long. As
soon as you stop auditd everything returns to normal...

Facts:
RHEL6 instances on RHEL7 hosts.
the rule set (see below) runs fine on any other system with less cpus
(<64, maybe this is the cut off?). We have smaller systems with this
rule set that rotate the audit file nearly every minute without any
noticable performance hit, these SAP systems rotate once every
20-24hours....

Anyone has an idea?

Here's an excerpt from "perf top":
with auditd running:

> Samples: 28M of event 'cpu-clock', Event count (approx.): 236747914918
> Overhead Shared Object Symbol
> 23.13% [kernel] [k] get_task_cred
> 10.05% [kernel] [k] audit_filter_rules
> 4.21% [kernel] [k] _spin_unlock_irqrestore
> 3.30% libdb2e.so.1 [.] sqlbfix
> 2.92% [kernel] [k] finish_task_switch
> 1.69% disp+work [.] rrol_in
> 1.69% disp+work [.] rrol_out
> 0.98% [kernel] [k] run_timer_softirq
> 0.96% [kernel] [k] rcu_process_gp_end
> 

auditd stopped:

> Samples: 3M of event 'cpu-clock', Event count (approx.): 526535382557
> Overhead Shared Object Symbol
> 2.41% disp+work [.] memcmpU16
> 2.32% disp+work [.] MmxMalloc2
> 2.25% disp+work [.] ab_Rudi
> 2.07% disp+work [.] rrol_out
> 1.98% disp+work [.] rrol_in
> 1.95% disp+work [.] ab_CompByCmpCntx
> 1.88% libdb2e.so.1 [.] sqlbfix
> 1.73% disp+work [.] MmxFree2
> 1.62% [kernel] [k] run_timer_softirq
> 1.56% [kernel] [k] __do_softirq
> 1.39% disp+work [.] ab_InitRcDecompress
> 
> These are the audit rules:
> auditctl -l
> -a always,exit -S all -F path=/etc/environment -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F key=USER_EXEC
> -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 -F key=USER_EXEC
> -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F key=CRIT_PAM
> -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA
> -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=5000000 -F auid!=-1 -F key=S3DATA
> 

-- 
------------------------------------------------------------------------
 Klaus Lichtenwalder, Dipl. Inform.,  http://www.lichtenwalder.name/
 PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Paul Moore]

On Fri, May 19, 2017 at 2:52 PM, Klaus Lichtenwalder
<klic@mnet-online.de> wrote:
> Hi,
>
> we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74
> CPUs and >= 400G RAM.
> When the system is busy with large SAP jobs, it goes onto its knees with
> cpu %system up to 80%, thus making the SAP jobs run twice as long. As
> soon as you stop auditd everything returns to normal...
>
> Facts:
> RHEL6 instances on RHEL7 hosts.
> the rule set (see below) runs fine on any other system with less cpus
> (<64, maybe this is the cut off?). We have smaller systems with this
> rule set that rotate the audit file nearly every minute without any
> noticable performance hit, these SAP systems rotate once every
> 20-24hours....

While we might occasionally provide distribution specific advice and
troubleshooting on the upstream lists, you are best off bringing
things like this up via support contract with your distro vendor.

-- 
paul moore
www.paul-moore.com

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Stephen Buchanan]

[-- Attachment #1.1: Type: text/plain, Size: 10562 bytes --]

Agree with Steve's suggestion re: "-S all". Also might help if you sort
your rules to put all the ones with '-F auid>=400' below a single line rule
like this:
-a never,exit -F auid<400

and remove the '-F auid>=400' from all of the rules below it.

Like so:
-a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
key=USER_EXEC
-a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=4294967295
-F key=USER_EXEC
-a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
auid>=5000000 -F auid!=4294967295 -F key=S3DATA

-a never,exit -F auid<400
-a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F
key=CRIT_AUDIT
-a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF
-a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG
-a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG
-a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT
-a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT
-a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT
-a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT
-a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT
-a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM
-a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF
-a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT
-a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000 -F
auid!=4294967295 -F key=S3DATA


On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder <klic@mnet-online.de>
wrote:

> Hi,
>
> we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74
> CPUs and >= 400G RAM.
> When the system is busy with large SAP jobs, it goes onto its knees with
> cpu %system up to 80%, thus making the SAP jobs run twice as long. As
> soon as you stop auditd everything returns to normal...
>
> Facts:
> RHEL6 instances on RHEL7 hosts.
> the rule set (see below) runs fine on any other system with less cpus
> (<64, maybe this is the cut off?). We have smaller systems with this
> rule set that rotate the audit file nearly every minute without any
> noticable performance hit, these SAP systems rotate once every
> 20-24hours....
>
> Anyone has an idea?
>
> Here's an excerpt from "perf top":
> with auditd running:
>
> > Samples: 28M of event 'cpu-clock', Event count (approx.): 236747914918
> > Overhead Shared Object Symbol
> > 23.13% [kernel] [k] get_task_cred
> > 10.05% [kernel] [k] audit_filter_rules
> > 4.21% [kernel] [k] _spin_unlock_irqrestore
> > 3.30% libdb2e.so.1 [.] sqlbfix
> > 2.92% [kernel] [k] finish_task_switch
> > 1.69% disp+work [.] rrol_in
> > 1.69% disp+work [.] rrol_out
> > 0.98% [kernel] [k] run_timer_softirq
> > 0.96% [kernel] [k] rcu_process_gp_end
> >
>
> auditd stopped:
>
> > Samples: 3M of event 'cpu-clock', Event count (approx.): 526535382557
> > Overhead Shared Object Symbol
> > 2.41% disp+work [.] memcmpU16
> > 2.32% disp+work [.] MmxMalloc2
> > 2.25% disp+work [.] ab_Rudi
> > 2.07% disp+work [.] rrol_out
> > 1.98% disp+work [.] rrol_in
> > 1.95% disp+work [.] ab_CompByCmpCntx
> > 1.88% libdb2e.so.1 [.] sqlbfix
> > 1.73% disp+work [.] MmxFree2
> > 1.62% [kernel] [k] run_timer_softirq
> > 1.56% [kernel] [k] __do_softirq
> > 1.39% disp+work [.] ab_InitRcDecompress
> >
> > These are the audit rules:
> > auditctl -l
> > -a always,exit -S all -F path=/etc/environment -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F
> auid>=400 -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F
> perm=wa -F auid>=400 -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa -F
> auid>=400 -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F
> auid>=400 -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F
> key=CRIT_PROG
> > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F
> auid>=400 -F key=USER_MGMT
> > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F auid>=400
> -F key=CRIT_CONF
> > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 -F
> key=USER_MGMT
> > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
> key=USER_EXEC
> > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 -F
> key=USER_EXEC
> > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F
> key=CRIT_PAM
> > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 -F
> key=CRIT_CONF
> > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F auid>=400
> -F key=CRIT_AUDIT
> > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F auid>=400
> -F key=CRIT_AUDIT
> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
> auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA
> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
> auid>=5000000 -F auid!=-1 -F key=S3DATA
> >
>
> --
> ------------------------------------------------------------------------
>  Klaus Lichtenwalder, Dipl. Inform.,  http://www.lichtenwalder.name/
>  PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>

[-- Attachment #1.2: Type: text/html, Size: 12426 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Klaus Lichtenwalder]

[-- Attachment #1.1: Type: text/plain, Size: 11630 bytes --]

Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan <stephenwb@gmail.com>:
>Agree with Steve's suggestion re: "-S all". Also might help if you sort
>your rules to put all the ones with '-F auid>=400' below a single line
>rule
>like this:
>-a never,exit -F auid<400
>
>and remove the '-F auid>=400' from all of the rules below it.
>
>Like so:
>-a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
>key=USER_EXEC
>-a always,exit -F arch=b64 -S execve -F auid>=5000000 -F
>auid!=4294967295
>-F key=USER_EXEC
>-a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
>auid>=5000000 -F auid!=4294967295 -F key=S3DATA
>
>-a never,exit -F auid<400
>-a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F
>key=CRIT_AUDIT
>-a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F
>key=CRIT_AUDIT
>-a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F
>key=CRIT_AUDIT
>-a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF
>-a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG
>-a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG
>-a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT
>-a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT
>-a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT
>-a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT
>-a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT
>-a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM
>-a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF
>-a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT
>-a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT
>-a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000
>-F
>auid!=4294967295 -F key=S3DATA
>
>
>On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder
><klic@mnet-online.de>
>wrote:
>
>> Hi,
>>
>> we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74
>> CPUs and >= 400G RAM.
>> When the system is busy with large SAP jobs, it goes onto its knees
>with
>> cpu %system up to 80%, thus making the SAP jobs run twice as long. As
>> soon as you stop auditd everything returns to normal...
>>
>> Facts:
>> RHEL6 instances on RHEL7 hosts.
>> the rule set (see below) runs fine on any other system with less cpus
>> (<64, maybe this is the cut off?). We have smaller systems with this
>> rule set that rotate the audit file nearly every minute without any
>> noticable performance hit, these SAP systems rotate once every
>> 20-24hours....
>>
>> Anyone has an idea?
>>
>> Here's an excerpt from "perf top":
>> with auditd running:
>>
>> > Samples: 28M of event 'cpu-clock', Event count (approx.):
>236747914918
>> > Overhead Shared Object Symbol
>> > 23.13% [kernel] [k] get_task_cred
>> > 10.05% [kernel] [k] audit_filter_rules
>> > 4.21% [kernel] [k] _spin_unlock_irqrestore
>> > 3.30% libdb2e.so.1 [.] sqlbfix
>> > 2.92% [kernel] [k] finish_task_switch
>> > 1.69% disp+work [.] rrol_in
>> > 1.69% disp+work [.] rrol_out
>> > 0.98% [kernel] [k] run_timer_softirq
>> > 0.96% [kernel] [k] rcu_process_gp_end
>> >
>>
>> auditd stopped:
>>
>> > Samples: 3M of event 'cpu-clock', Event count (approx.):
>526535382557
>> > Overhead Shared Object Symbol
>> > 2.41% disp+work [.] memcmpU16
>> > 2.32% disp+work [.] MmxMalloc2
>> > 2.25% disp+work [.] ab_Rudi
>> > 2.07% disp+work [.] rrol_out
>> > 1.98% disp+work [.] rrol_in
>> > 1.95% disp+work [.] ab_CompByCmpCntx
>> > 1.88% libdb2e.so.1 [.] sqlbfix
>> > 1.73% disp+work [.] MmxFree2
>> > 1.62% [kernel] [k] run_timer_softirq
>> > 1.56% [kernel] [k] __do_softirq
>> > 1.39% disp+work [.] ab_InitRcDecompress
>> >
>> > These are the audit rules:
>> > auditctl -l
>> > -a always,exit -S all -F path=/etc/environment -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F
>> auid>=400 -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F
>> perm=wa -F auid>=400 -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa
>-F
>> auid>=400 -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F
>> auid>=400 -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F
>> key=CRIT_PROG
>> > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F
>auid>=400 -F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F
>> auid>=400 -F key=USER_MGMT
>> > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F
>auid>=400 -F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F
>auid>=400
>> -F key=CRIT_CONF
>> > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400
>-F
>> key=USER_MGMT
>> > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F
>> key=USER_EXEC
>> > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1
>-F
>> key=USER_EXEC
>> > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F
>> key=CRIT_PAM
>> > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400
>-F
>> key=CRIT_CONF
>> > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F
>auid>=400
>> -F key=CRIT_AUDIT
>> > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F
>auid>=400
>> -F key=CRIT_AUDIT
>> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
>> auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA
>> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F
>> auid>=5000000 -F auid!=-1 -F key=S3DATA
>> >
>>
>> --
>>
>------------------------------------------------------------------------
>>  Klaus Lichtenwalder, Dipl. Inform.,  http://www.lichtenwalder.name/
>>  PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA
>0980
>>
>> --
>> Linux-audit mailing list
>> Linux-audit@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-audit
>>

Thanks everybody for these valuable insights!

@Paul: it is in the support hands of the distribution provider, but there were reasons to also go here... I still have a strong indication of a problematic situation with many cpus, maybe you can make something out of it, and these tips only popped up here on this great list

Klaus
-- 
Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten

[-- Attachment #1.2: Type: text/html, Size: 13402 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Klaus Lichtenwalder]

Hi everybody

Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan <stephenwb@gmail.com>:
>Agree with Steve's suggestion re: "-S all". Also might help if you sort

I now know where -S all stems from... Some watches add a -S all by themselves... Probably created an audit.rules file by textually working from there and duplicating rules

>your rules to put all the ones with '-F auid>=400' below a single line
>rule
>like this:
>-a never,exit -F auid<400
>
>and remove the '-F auid>=400' from all of the rules below it.
>
...

I did this, and verified it, but there was absolutely no difference to unsorted rules having​ -S all also specified

Still cpu %system up to 50% and run time of jobs 100% longer. 
This was on a vm with 72 cpus

Klaus


-- 
Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Steve Grubb]

Hello,

On Tue, 23 May 2017 11:05:18 +0200
Klaus Lichtenwalder <klic@mnet-online.de> wrote:
> Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan
> <stephenwb@gmail.com>:
> >Agree with Steve's suggestion re: "-S all". Also might help if you
> >sort  
> 
> I now know where -S all stems from... Some watches add a -S all by
> themselves... Probably created an audit.rules file by textually
> working from there and duplicating rules

What is the source of your rules listed? Is it coming from auditctl -l
or from /etc/audit/audit.rules? There were a couple releases of
auditctl where I think -S all may have been added but if I remember it
was fixed a few releases later. The rules that come from disk would be
more accurate.

-Steve

> >your rules to put all the ones with '-F auid>=400' below a single
> >line rule
> >like this:
> >-a never,exit -F auid<400
> >
> >and remove the '-F auid>=400' from all of the rules below it.
> >  
> ...
> 
> I did this, and verified it, but there was absolutely no difference
> to unsorted rules having​ -S all also specified
> 
> Still cpu %system up to 50% and run time of jobs 100% longer. 
> This was on a vm with 72 cpus
> 
> Klaus
> 
> 


--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Klaus Lichtenwalder]

Am 23. Mai 2017 14:51:29 MESZ schrieb Steve Grubb <sgrubb@redhat.com>:
>Hello,
>
>On Tue, 23 May 2017 11:05:18 +0200
>Klaus Lichtenwalder <klic@mnet-online.de> wrote:
>> Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan
>> <stephenwb@gmail.com>:
>> >Agree with Steve's suggestion re: "-S all". Also might help if you
>> >sort  
>> 
>> I now know where -S all stems from... Some watches add a -S all by
>> themselves... Probably created an audit.rules file by textually
>> working from there and duplicating rules
>
>What is the source of your rules listed? Is it coming from auditctl -l
>or from /etc/audit/audit.rules? There were a couple releases of
>auditctl where I think -S all may have been added but if I remember it
>was fixed a few releases later. The rules that come from disk would be
>more accurate.
>

Well, they came from auditctl -l
System in question is RHEL6.8, can't tell actual package version right now, as I'm on the road...
But thanks, will keep in mind to stick to the files...

Klaus

-- 
Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Klaus Lichtenwalder]

>>> your rules to put all the ones with '-F auid>=400' below a single
>>> line rule
>>> like this:
>>> -a never,exit -F auid<400
>>>
>>> and remove the '-F auid>=400' from all of the rules below it.
>>>  
>> ...
>>
>> I did this, and verified it, but there was absolutely no difference
>> to unsorted rules having​ -S all also specified
>>
>> Still cpu %system up to 50% and run time of jobs 100% longer. 
>> This was on a vm with 72 cpus
>>

Just to give this story some kind of closure: we got a test kernel from
$SUPPORT fixing a specifig bugzilla (which seems to be private) and %cpu
system is in normal (low) ranges again.

So thanks for your advices, they are still heeded!

Klaus
-- 
------------------------------------------------------------------------
 Klaus Lichtenwalder, Dipl. Inform.,  http://www.lichtenwalder.name/
 PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: BIG performance hit with auditd on large systems (>64 CPUs)

[Paul Moore]

On Tue, May 30, 2017 at 2:17 PM, Klaus Lichtenwalder
<klic@mnet-online.de> wrote:
>>>> your rules to put all the ones with '-F auid>=400' below a single
>>>> line rule
>>>> like this:
>>>> -a never,exit -F auid<400
>>>>
>>>> and remove the '-F auid>=400' from all of the rules below it.
>>>>
>>> ...
>>>
>>> I did this, and verified it, but there was absolutely no difference
>>> to unsorted rules having -S all also specified
>>>
>>> Still cpu %system up to 50% and run time of jobs 100% longer.
>>> This was on a vm with 72 cpus
>>>
>
> Just to give this story some kind of closure: we got a test kernel from
> $SUPPORT fixing a specifig bugzilla (which seems to be private) and %cpu
> system is in normal (low) ranges again.
>
> So thanks for your advices, they are still heeded!

For the record the core issue was fixed in f56298835036 ("audit:
acquire creds selectively to reduce atomic op overhead").

-- 
paul moore
www.paul-moore.com