Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan : >Agree with Steve's suggestion re: "-S all". Also might help if you sort >your rules to put all the ones with '-F auid>=400' below a single line >rule >like this: >-a never,exit -F auid<400 > >and remove the '-F auid>=400' from all of the rules below it. > >Like so: >-a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F >key=USER_EXEC >-a always,exit -F arch=b64 -S execve -F auid>=5000000 -F >auid!=4294967295 >-F key=USER_EXEC >-a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F >auid>=5000000 -F auid!=4294967295 -F key=S3DATA > >-a never,exit -F auid<400 >-a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F >key=CRIT_AUDIT >-a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F >key=CRIT_AUDIT >-a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F >key=CRIT_AUDIT >-a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG >-a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT >-a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT >-a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT >-a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT >-a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM >-a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT >-a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT >-a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000 >-F >auid!=4294967295 -F key=S3DATA > > >On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder > >wrote: > >> Hi, >> >> we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74 >> CPUs and >= 400G RAM. >> When the system is busy with large SAP jobs, it goes onto its knees >with >> cpu %system up to 80%, thus making the SAP jobs run twice as long. As >> soon as you stop auditd everything returns to normal... >> >> Facts: >> RHEL6 instances on RHEL7 hosts. >> the rule set (see below) runs fine on any other system with less cpus >> (<64, maybe this is the cut off?). We have smaller systems with this >> rule set that rotate the audit file nearly every minute without any >> noticable performance hit, these SAP systems rotate once every >> 20-24hours.... >> >> Anyone has an idea? >> >> Here's an excerpt from "perf top": >> with auditd running: >> >> > Samples: 28M of event 'cpu-clock', Event count (approx.): >236747914918 >> > Overhead Shared Object Symbol >> > 23.13% [kernel] [k] get_task_cred >> > 10.05% [kernel] [k] audit_filter_rules >> > 4.21% [kernel] [k] _spin_unlock_irqrestore >> > 3.30% libdb2e.so.1 [.] sqlbfix >> > 2.92% [kernel] [k] finish_task_switch >> > 1.69% disp+work [.] rrol_in >> > 1.69% disp+work [.] rrol_out >> > 0.98% [kernel] [k] run_timer_softirq >> > 0.96% [kernel] [k] rcu_process_gp_end >> > >> >> auditd stopped: >> >> > Samples: 3M of event 'cpu-clock', Event count (approx.): >526535382557 >> > Overhead Shared Object Symbol >> > 2.41% disp+work [.] memcmpU16 >> > 2.32% disp+work [.] MmxMalloc2 >> > 2.25% disp+work [.] ab_Rudi >> > 2.07% disp+work [.] rrol_out >> > 1.98% disp+work [.] rrol_in >> > 1.95% disp+work [.] ab_CompByCmpCntx >> > 1.88% libdb2e.so.1 [.] sqlbfix >> > 1.73% disp+work [.] MmxFree2 >> > 1.62% [kernel] [k] run_timer_softirq >> > 1.56% [kernel] [k] __do_softirq >> > 1.39% disp+work [.] ab_InitRcDecompress >> > >> > These are the audit rules: >> > auditctl -l >> > -a always,exit -S all -F path=/etc/environment -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F >> auid>=400 -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F >> perm=wa -F auid>=400 -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa >-F >> auid>=400 -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F >> auid>=400 -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F >auid>=400 -F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F >> auid>=400 -F key=USER_MGMT >> > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F >> key=USER_EXEC >> > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 >-F >> key=USER_EXEC >> > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F >> key=CRIT_PAM >> > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F >auid>=400 >> -F key=CRIT_AUDIT >> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F >> auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA >> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F >> auid>=5000000 -F auid!=-1 -F key=S3DATA >> > >> >> -- >> >------------------------------------------------------------------------ >> Klaus Lichtenwalder, Dipl. Inform., http://www.lichtenwalder.name/ >> PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA >0980 >> >> -- >> Linux-audit mailing list >> Linux-audit@redhat.com >> https://www.redhat.com/mailman/listinfo/linux-audit >> Thanks everybody for these valuable insights! @Paul: it is in the support hands of the distribution provider, but there were reasons to also go here... I still have a strong indication of a problematic situation with many cpus, maybe you can make something out of it, and these tips only popped up here on this great list Klaus -- Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten