All of lore.kernel.org
 help / color / mirror / Atom feed
From: Paul Moore <paul@paul-moore.com>
To: burn@swtf.dyndns.org
Cc: Richard Guy Briggs <rgb@redhat.com>,
	Linux Audit <linux-audit@redhat.com>
Subject: Re: Occasional delayed output of events
Date: Tue, 19 Jan 2021 14:11:05 -0500	[thread overview]
Message-ID: <CAHC9VhRNcFappgO1eb40wbtjtzZe1b5RRV6ogN563QTOubwvOA@mail.gmail.com> (raw)
In-Reply-To: <9a7ed1203fa7ec67000aa68281a215354c2ed5f5.camel@iinet.net.au>

On Tue, Jan 19, 2021 at 3:18 AM Burn Alting <burn.alting@iinet.net.au> wrote:
> I tend to have a rigorous auditing posture (see the rules loaded in https://github.com/linux-audit/audit-userspace/issues/148) which is not normal for most. Perhaps, Paul, you have hit the nail on the head by stating that this 'severe delay' is not that unreasonable given my rules posture and we just need to 'deal with it' in user space.
> We still get the event data, I just need to adjust the user space tools to deal with this occurrence.
> As for what the system is doing, in my home case it's a Centos 7 VM running a tomcat service which only gets busy every 20 minutes and the other is a HPE Z800 running Centos 8 with 4-5 VM's mostly dormant. I can put any code in these hosts to assist in 'validating'/testing the delay. Advise and I will run.

I took a (slightly) closer look at the queuing code just now and the
corner case I suspected doesn't look very promising here.  On failure
to send a record to userspace, the record is put back at the front of
the queue to preserve ordering so the kernel *should* still emit
records in order even when auditd is under pressure, dead, or in the
process of a restart.  Also, in this case the kernel kicks the auditd
connection, disconnecting auditd's netlink socket; I imagine that
would be a notable event on your systems.

However, looking at the timestamps in the audit events you posted, I
noticed something a little odd.  If I align the timestamps a bit
better let's see if it jumps out at you guys ...

type=XXX msg=audit(1609519900.159:44606): ...
type=XXX msg=audit(1609519900.161:44607): ...
type=XXX msg=audit(1609519900.163:44608): ...
type=XXX msg=audit(1609519896.829:44609): ...
type=XXX msg=audit(1609519900.170:44610): ...

Let me guess Burn, you're running NTP/PTP :)  I'm not sure how auditd
handles things like this, but it looks like there was a small negative
time correction between events which caused the odd scenario where
event N+1 actually occurred before event N according to the wall
clock.  In other words, your system is time traveling ;)

-- 
paul moore
www.paul-moore.com


--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit


  parent reply	other threads:[~2021-01-19 19:11 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-01 21:22 Occasional delayed output of events Burn Alting
2021-01-03 15:41 ` Steve Grubb
2021-01-04  7:55   ` Burn Alting
2021-01-04 14:46     ` Steve Grubb
2021-01-04 20:12       ` Burn Alting
2021-01-10  4:39         ` Burn Alting
2021-01-15 22:18           ` Burn Alting
2021-01-16  0:35             ` Richard Guy Briggs
2021-01-16  2:42               ` Burn Alting
2021-01-17 14:07                 ` Paul Moore
2021-01-17 21:12                   ` Steve Grubb
2021-01-18 13:54                     ` Paul Moore
2021-01-18 14:31                       ` Steve Grubb
2021-01-18 20:34                         ` Burn Alting
2021-01-18 20:36                         ` Paul Moore
2021-01-19  8:18                           ` Burn Alting
2021-01-19 15:31                             ` Lenny Bruzenak
2021-01-19 19:11                             ` Paul Moore [this message]
2021-01-19 19:38                               ` Burn Alting
2021-01-19 20:26                                 ` Paul Moore
2021-01-19 21:51                                   ` Steve Grubb
2021-01-20  6:38                                     ` Burn Alting
2021-01-20 22:50                                       ` Paul Moore
2021-01-23 22:55                                         ` Burn Alting
2021-01-25 23:53                                           ` Steve Grubb
2021-01-26  0:11                                             ` Burn Alting
2021-01-26  0:20                                               ` Steve Grubb
2021-01-26  0:29                                                 ` Burn Alting
2021-01-26 11:53                                                   ` Burn Alting
2021-01-26 20:42                                                     ` Steve Grubb
2021-01-27 12:12                                                       ` Burn Alting
2021-01-19 20:42                               ` Richard Guy Briggs

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAHC9VhRNcFappgO1eb40wbtjtzZe1b5RRV6ogN563QTOubwvOA@mail.gmail.com \
    --to=paul@paul-moore.com \
    --cc=burn@swtf.dyndns.org \
    --cc=linux-audit@redhat.com \
    --cc=rgb@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.