* BIG performance hit with auditd on large systems (>64 CPUs) @ 2017-05-19 18:52 Klaus Lichtenwalder 2017-05-19 20:56 ` Paul Moore 2017-05-19 21:41 ` Stephen Buchanan 0 siblings, 2 replies; 9+ messages in thread From: Klaus Lichtenwalder @ 2017-05-19 18:52 UTC (permalink / raw) To: linux-audit Hi, we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74 CPUs and >= 400G RAM. When the system is busy with large SAP jobs, it goes onto its knees with cpu %system up to 80%, thus making the SAP jobs run twice as long. As soon as you stop auditd everything returns to normal... Facts: RHEL6 instances on RHEL7 hosts. the rule set (see below) runs fine on any other system with less cpus (<64, maybe this is the cut off?). We have smaller systems with this rule set that rotate the audit file nearly every minute without any noticable performance hit, these SAP systems rotate once every 20-24hours.... Anyone has an idea? Here's an excerpt from "perf top": with auditd running: > Samples: 28M of event 'cpu-clock', Event count (approx.): 236747914918 > Overhead Shared Object Symbol > 23.13% [kernel] [k] get_task_cred > 10.05% [kernel] [k] audit_filter_rules > 4.21% [kernel] [k] _spin_unlock_irqrestore > 3.30% libdb2e.so.1 [.] sqlbfix > 2.92% [kernel] [k] finish_task_switch > 1.69% disp+work [.] rrol_in > 1.69% disp+work [.] rrol_out > 0.98% [kernel] [k] run_timer_softirq > 0.96% [kernel] [k] rcu_process_gp_end > auditd stopped: > Samples: 3M of event 'cpu-clock', Event count (approx.): 526535382557 > Overhead Shared Object Symbol > 2.41% disp+work [.] memcmpU16 > 2.32% disp+work [.] MmxMalloc2 > 2.25% disp+work [.] ab_Rudi > 2.07% disp+work [.] rrol_out > 1.98% disp+work [.] rrol_in > 1.95% disp+work [.] ab_CompByCmpCntx > 1.88% libdb2e.so.1 [.] sqlbfix > 1.73% disp+work [.] MmxFree2 > 1.62% [kernel] [k] run_timer_softirq > 1.56% [kernel] [k] __do_softirq > 1.39% disp+work [.] ab_InitRcDecompress > > These are the audit rules: > auditctl -l > -a always,exit -S all -F path=/etc/environment -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F key=USER_MGMT > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 -F key=USER_MGMT > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 -F key=USER_MGMT > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F key=CRIT_PROG > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F key=CRIT_PROG > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F key=CRIT_PROG > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F key=CRIT_PROG > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F key=CRIT_PROG > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F auid>=400 -F key=USER_MGMT > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F auid>=400 -F key=USER_MGMT > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 -F key=USER_MGMT > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 -F key=USER_MGMT > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 -F key=USER_MGMT > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F key=USER_EXEC > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 -F key=USER_EXEC > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F key=CRIT_PAM > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 -F key=CRIT_CONF > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F auid>=400 -F key=CRIT_AUDIT > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=5000000 -F auid!=-1 -F key=S3DATA > -- ------------------------------------------------------------------------ Klaus Lichtenwalder, Dipl. Inform., http://www.lichtenwalder.name/ PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-19 18:52 BIG performance hit with auditd on large systems (>64 CPUs) Klaus Lichtenwalder @ 2017-05-19 20:56 ` Paul Moore 2017-05-19 21:41 ` Stephen Buchanan 1 sibling, 0 replies; 9+ messages in thread From: Paul Moore @ 2017-05-19 20:56 UTC (permalink / raw) To: Klaus Lichtenwalder; +Cc: linux-audit On Fri, May 19, 2017 at 2:52 PM, Klaus Lichtenwalder <klic@mnet-online.de> wrote: > Hi, > > we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74 > CPUs and >= 400G RAM. > When the system is busy with large SAP jobs, it goes onto its knees with > cpu %system up to 80%, thus making the SAP jobs run twice as long. As > soon as you stop auditd everything returns to normal... > > Facts: > RHEL6 instances on RHEL7 hosts. > the rule set (see below) runs fine on any other system with less cpus > (<64, maybe this is the cut off?). We have smaller systems with this > rule set that rotate the audit file nearly every minute without any > noticable performance hit, these SAP systems rotate once every > 20-24hours.... While we might occasionally provide distribution specific advice and troubleshooting on the upstream lists, you are best off bringing things like this up via support contract with your distro vendor. -- paul moore www.paul-moore.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-19 18:52 BIG performance hit with auditd on large systems (>64 CPUs) Klaus Lichtenwalder 2017-05-19 20:56 ` Paul Moore @ 2017-05-19 21:41 ` Stephen Buchanan 2017-05-20 7:18 ` Klaus Lichtenwalder 2017-05-23 9:05 ` Klaus Lichtenwalder 1 sibling, 2 replies; 9+ messages in thread From: Stephen Buchanan @ 2017-05-19 21:41 UTC (permalink / raw) To: Klaus Lichtenwalder, linux-audit [-- Attachment #1.1: Type: text/plain, Size: 10562 bytes --] Agree with Steve's suggestion re: "-S all". Also might help if you sort your rules to put all the ones with '-F auid>=400' below a single line rule like this: -a never,exit -F auid<400 and remove the '-F auid>=400' from all of the rules below it. Like so: -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F key=USER_EXEC -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=4294967295 -F key=USER_EXEC -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=5000000 -F auid!=4294967295 -F key=S3DATA -a never,exit -F auid<400 -a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG -a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT -a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT -a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT -a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT -a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM -a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT -a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000 -F auid!=4294967295 -F key=S3DATA On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder <klic@mnet-online.de> wrote: > Hi, > > we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74 > CPUs and >= 400G RAM. > When the system is busy with large SAP jobs, it goes onto its knees with > cpu %system up to 80%, thus making the SAP jobs run twice as long. As > soon as you stop auditd everything returns to normal... > > Facts: > RHEL6 instances on RHEL7 hosts. > the rule set (see below) runs fine on any other system with less cpus > (<64, maybe this is the cut off?). We have smaller systems with this > rule set that rotate the audit file nearly every minute without any > noticable performance hit, these SAP systems rotate once every > 20-24hours.... > > Anyone has an idea? > > Here's an excerpt from "perf top": > with auditd running: > > > Samples: 28M of event 'cpu-clock', Event count (approx.): 236747914918 > > Overhead Shared Object Symbol > > 23.13% [kernel] [k] get_task_cred > > 10.05% [kernel] [k] audit_filter_rules > > 4.21% [kernel] [k] _spin_unlock_irqrestore > > 3.30% libdb2e.so.1 [.] sqlbfix > > 2.92% [kernel] [k] finish_task_switch > > 1.69% disp+work [.] rrol_in > > 1.69% disp+work [.] rrol_out > > 0.98% [kernel] [k] run_timer_softirq > > 0.96% [kernel] [k] rcu_process_gp_end > > > > auditd stopped: > > > Samples: 3M of event 'cpu-clock', Event count (approx.): 526535382557 > > Overhead Shared Object Symbol > > 2.41% disp+work [.] memcmpU16 > > 2.32% disp+work [.] MmxMalloc2 > > 2.25% disp+work [.] ab_Rudi > > 2.07% disp+work [.] rrol_out > > 1.98% disp+work [.] rrol_in > > 1.95% disp+work [.] ab_CompByCmpCntx > > 1.88% libdb2e.so.1 [.] sqlbfix > > 1.73% disp+work [.] MmxFree2 > > 1.62% [kernel] [k] run_timer_softirq > > 1.56% [kernel] [k] __do_softirq > > 1.39% disp+work [.] ab_InitRcDecompress > > > > These are the audit rules: > > auditctl -l > > -a always,exit -S all -F path=/etc/environment -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F > auid>=400 -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F > perm=wa -F auid>=400 -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa -F > auid>=400 -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F > auid>=400 -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F > auid>=400 -F key=USER_MGMT > > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F > key=USER_EXEC > > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 -F > key=USER_EXEC > > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F > key=CRIT_PAM > > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F auid>=400 > -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F auid>=400 > -F key=CRIT_AUDIT > > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F > auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA > > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F > auid>=5000000 -F auid!=-1 -F key=S3DATA > > > > -- > ------------------------------------------------------------------------ > Klaus Lichtenwalder, Dipl. Inform., http://www.lichtenwalder.name/ > PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980 > > -- > Linux-audit mailing list > Linux-audit@redhat.com > https://www.redhat.com/mailman/listinfo/linux-audit > [-- Attachment #1.2: Type: text/html, Size: 12426 bytes --] [-- Attachment #2: Type: text/plain, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-19 21:41 ` Stephen Buchanan @ 2017-05-20 7:18 ` Klaus Lichtenwalder 2017-05-23 9:05 ` Klaus Lichtenwalder 1 sibling, 0 replies; 9+ messages in thread From: Klaus Lichtenwalder @ 2017-05-20 7:18 UTC (permalink / raw) To: linux-audit [-- Attachment #1.1: Type: text/plain, Size: 11630 bytes --] Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan <stephenwb@gmail.com>: >Agree with Steve's suggestion re: "-S all". Also might help if you sort >your rules to put all the ones with '-F auid>=400' below a single line >rule >like this: >-a never,exit -F auid<400 > >and remove the '-F auid>=400' from all of the rules below it. > >Like so: >-a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F >key=USER_EXEC >-a always,exit -F arch=b64 -S execve -F auid>=5000000 -F >auid!=4294967295 >-F key=USER_EXEC >-a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F >auid>=5000000 -F auid!=4294967295 -F key=S3DATA > >-a never,exit -F auid<400 >-a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F >key=CRIT_AUDIT >-a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F >key=CRIT_AUDIT >-a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F >key=CRIT_AUDIT >-a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF >-a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG >-a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG >-a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT >-a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT >-a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT >-a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT >-a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT >-a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM >-a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF >-a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT >-a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT >-a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000 >-F >auid!=4294967295 -F key=S3DATA > > >On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder ><klic@mnet-online.de> >wrote: > >> Hi, >> >> we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74 >> CPUs and >= 400G RAM. >> When the system is busy with large SAP jobs, it goes onto its knees >with >> cpu %system up to 80%, thus making the SAP jobs run twice as long. As >> soon as you stop auditd everything returns to normal... >> >> Facts: >> RHEL6 instances on RHEL7 hosts. >> the rule set (see below) runs fine on any other system with less cpus >> (<64, maybe this is the cut off?). We have smaller systems with this >> rule set that rotate the audit file nearly every minute without any >> noticable performance hit, these SAP systems rotate once every >> 20-24hours.... >> >> Anyone has an idea? >> >> Here's an excerpt from "perf top": >> with auditd running: >> >> > Samples: 28M of event 'cpu-clock', Event count (approx.): >236747914918 >> > Overhead Shared Object Symbol >> > 23.13% [kernel] [k] get_task_cred >> > 10.05% [kernel] [k] audit_filter_rules >> > 4.21% [kernel] [k] _spin_unlock_irqrestore >> > 3.30% libdb2e.so.1 [.] sqlbfix >> > 2.92% [kernel] [k] finish_task_switch >> > 1.69% disp+work [.] rrol_in >> > 1.69% disp+work [.] rrol_out >> > 0.98% [kernel] [k] run_timer_softirq >> > 0.96% [kernel] [k] rcu_process_gp_end >> > >> >> auditd stopped: >> >> > Samples: 3M of event 'cpu-clock', Event count (approx.): >526535382557 >> > Overhead Shared Object Symbol >> > 2.41% disp+work [.] memcmpU16 >> > 2.32% disp+work [.] MmxMalloc2 >> > 2.25% disp+work [.] ab_Rudi >> > 2.07% disp+work [.] rrol_out >> > 1.98% disp+work [.] rrol_in >> > 1.95% disp+work [.] ab_CompByCmpCntx >> > 1.88% libdb2e.so.1 [.] sqlbfix >> > 1.73% disp+work [.] MmxFree2 >> > 1.62% [kernel] [k] run_timer_softirq >> > 1.56% [kernel] [k] __do_softirq >> > 1.39% disp+work [.] ab_InitRcDecompress >> > >> > These are the audit rules: >> > auditctl -l >> > -a always,exit -S all -F path=/etc/environment -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F >> auid>=400 -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F >> perm=wa -F auid>=400 -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa >-F >> auid>=400 -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F >> auid>=400 -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F >> key=CRIT_PROG >> > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F >auid>=400 -F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F >> auid>=400 -F key=USER_MGMT >> > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F >auid>=400 -F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F >auid>=400 >> -F key=CRIT_CONF >> > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 >-F >> key=USER_MGMT >> > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F >> key=USER_EXEC >> > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 >-F >> key=USER_EXEC >> > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F >> key=CRIT_PAM >> > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 >-F >> key=CRIT_CONF >> > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F >auid>=400 >> -F key=CRIT_AUDIT >> > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F >auid>=400 >> -F key=CRIT_AUDIT >> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F >> auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA >> > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F >> auid>=5000000 -F auid!=-1 -F key=S3DATA >> > >> >> -- >> >------------------------------------------------------------------------ >> Klaus Lichtenwalder, Dipl. Inform., http://www.lichtenwalder.name/ >> PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA >0980 >> >> -- >> Linux-audit mailing list >> Linux-audit@redhat.com >> https://www.redhat.com/mailman/listinfo/linux-audit >> Thanks everybody for these valuable insights! @Paul: it is in the support hands of the distribution provider, but there were reasons to also go here... I still have a strong indication of a problematic situation with many cpus, maybe you can make something out of it, and these tips only popped up here on this great list Klaus -- Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten [-- Attachment #1.2: Type: text/html, Size: 13402 bytes --] [-- Attachment #2: Type: text/plain, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-19 21:41 ` Stephen Buchanan 2017-05-20 7:18 ` Klaus Lichtenwalder @ 2017-05-23 9:05 ` Klaus Lichtenwalder 2017-05-23 12:51 ` Steve Grubb 1 sibling, 1 reply; 9+ messages in thread From: Klaus Lichtenwalder @ 2017-05-23 9:05 UTC (permalink / raw) To: Stephen Buchanan, linux-audit Hi everybody Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan <stephenwb@gmail.com>: >Agree with Steve's suggestion re: "-S all". Also might help if you sort I now know where -S all stems from... Some watches add a -S all by themselves... Probably created an audit.rules file by textually working from there and duplicating rules >your rules to put all the ones with '-F auid>=400' below a single line >rule >like this: >-a never,exit -F auid<400 > >and remove the '-F auid>=400' from all of the rules below it. > ... I did this, and verified it, but there was absolutely no difference to unsorted rules having -S all also specified Still cpu %system up to 50% and run time of jobs 100% longer. This was on a vm with 72 cpus Klaus -- Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-23 9:05 ` Klaus Lichtenwalder @ 2017-05-23 12:51 ` Steve Grubb 2017-05-23 14:45 ` Klaus Lichtenwalder 2017-05-30 18:17 ` Klaus Lichtenwalder 0 siblings, 2 replies; 9+ messages in thread From: Steve Grubb @ 2017-05-23 12:51 UTC (permalink / raw) To: Klaus Lichtenwalder; +Cc: linux-audit Hello, On Tue, 23 May 2017 11:05:18 +0200 Klaus Lichtenwalder <klic@mnet-online.de> wrote: > Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan > <stephenwb@gmail.com>: > >Agree with Steve's suggestion re: "-S all". Also might help if you > >sort > > I now know where -S all stems from... Some watches add a -S all by > themselves... Probably created an audit.rules file by textually > working from there and duplicating rules What is the source of your rules listed? Is it coming from auditctl -l or from /etc/audit/audit.rules? There were a couple releases of auditctl where I think -S all may have been added but if I remember it was fixed a few releases later. The rules that come from disk would be more accurate. -Steve > >your rules to put all the ones with '-F auid>=400' below a single > >line rule > >like this: > >-a never,exit -F auid<400 > > > >and remove the '-F auid>=400' from all of the rules below it. > > > ... > > I did this, and verified it, but there was absolutely no difference > to unsorted rules having -S all also specified > > Still cpu %system up to 50% and run time of jobs 100% longer. > This was on a vm with 72 cpus > > Klaus > > -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-23 12:51 ` Steve Grubb @ 2017-05-23 14:45 ` Klaus Lichtenwalder 2017-05-30 18:17 ` Klaus Lichtenwalder 1 sibling, 0 replies; 9+ messages in thread From: Klaus Lichtenwalder @ 2017-05-23 14:45 UTC (permalink / raw) To: Steve Grubb; +Cc: linux-audit Am 23. Mai 2017 14:51:29 MESZ schrieb Steve Grubb <sgrubb@redhat.com>: >Hello, > >On Tue, 23 May 2017 11:05:18 +0200 >Klaus Lichtenwalder <klic@mnet-online.de> wrote: >> Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan >> <stephenwb@gmail.com>: >> >Agree with Steve's suggestion re: "-S all". Also might help if you >> >sort >> >> I now know where -S all stems from... Some watches add a -S all by >> themselves... Probably created an audit.rules file by textually >> working from there and duplicating rules > >What is the source of your rules listed? Is it coming from auditctl -l >or from /etc/audit/audit.rules? There were a couple releases of >auditctl where I think -S all may have been added but if I remember it >was fixed a few releases later. The rules that come from disk would be >more accurate. > Well, they came from auditctl -l System in question is RHEL6.8, can't tell actual package version right now, as I'm on the road... But thanks, will keep in mind to stick to the files... Klaus -- Mit K9 vom Telefon gesendet. Tippfehler und komische Worte darf der Empfänger behalten -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-23 12:51 ` Steve Grubb 2017-05-23 14:45 ` Klaus Lichtenwalder @ 2017-05-30 18:17 ` Klaus Lichtenwalder 2017-05-30 19:49 ` Paul Moore 1 sibling, 1 reply; 9+ messages in thread From: Klaus Lichtenwalder @ 2017-05-30 18:17 UTC (permalink / raw) To: linux-audit >>> your rules to put all the ones with '-F auid>=400' below a single >>> line rule >>> like this: >>> -a never,exit -F auid<400 >>> >>> and remove the '-F auid>=400' from all of the rules below it. >>> >> ... >> >> I did this, and verified it, but there was absolutely no difference >> to unsorted rules having -S all also specified >> >> Still cpu %system up to 50% and run time of jobs 100% longer. >> This was on a vm with 72 cpus >> Just to give this story some kind of closure: we got a test kernel from $SUPPORT fixing a specifig bugzilla (which seems to be private) and %cpu system is in normal (low) ranges again. So thanks for your advices, they are still heeded! Klaus -- ------------------------------------------------------------------------ Klaus Lichtenwalder, Dipl. Inform., http://www.lichtenwalder.name/ PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980 -- Linux-audit mailing list Linux-audit@redhat.com https://www.redhat.com/mailman/listinfo/linux-audit ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: BIG performance hit with auditd on large systems (>64 CPUs) 2017-05-30 18:17 ` Klaus Lichtenwalder @ 2017-05-30 19:49 ` Paul Moore 0 siblings, 0 replies; 9+ messages in thread From: Paul Moore @ 2017-05-30 19:49 UTC (permalink / raw) To: Klaus Lichtenwalder, linux-audit On Tue, May 30, 2017 at 2:17 PM, Klaus Lichtenwalder <klic@mnet-online.de> wrote: >>>> your rules to put all the ones with '-F auid>=400' below a single >>>> line rule >>>> like this: >>>> -a never,exit -F auid<400 >>>> >>>> and remove the '-F auid>=400' from all of the rules below it. >>>> >>> ... >>> >>> I did this, and verified it, but there was absolutely no difference >>> to unsorted rules having -S all also specified >>> >>> Still cpu %system up to 50% and run time of jobs 100% longer. >>> This was on a vm with 72 cpus >>> > > Just to give this story some kind of closure: we got a test kernel from > $SUPPORT fixing a specifig bugzilla (which seems to be private) and %cpu > system is in normal (low) ranges again. > > So thanks for your advices, they are still heeded! For the record the core issue was fixed in f56298835036 ("audit: acquire creds selectively to reduce atomic op overhead"). -- paul moore www.paul-moore.com ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-05-30 19:49 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-05-19 18:52 BIG performance hit with auditd on large systems (>64 CPUs) Klaus Lichtenwalder 2017-05-19 20:56 ` Paul Moore 2017-05-19 21:41 ` Stephen Buchanan 2017-05-20 7:18 ` Klaus Lichtenwalder 2017-05-23 9:05 ` Klaus Lichtenwalder 2017-05-23 12:51 ` Steve Grubb 2017-05-23 14:45 ` Klaus Lichtenwalder 2017-05-30 18:17 ` Klaus Lichtenwalder 2017-05-30 19:49 ` Paul Moore
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.