* Audit format utility
@ 2014-09-23 22:03 Steve Grubb
2014-09-25 16:16 ` LC Bruzenak
0 siblings, 1 reply; 8+ messages in thread
From: Steve Grubb @ 2014-09-23 22:03 UTC (permalink / raw)
To: Linux-audit
Hello,
I have been doing some thinking about allowing user defined formats to be
declared as a parameter to ausearch. Before I commit to that, I thought it
might be interesting to create a "mockup". I have placed a utility here:
http://people.redhat.com/sgrubb/files/auformat.tar.gz
Just untar it and run make.
$ ./auformat
Usage: auformat <format-string> [path-to-log] --fill
What this means is that you can give it a path to a file. If you don't do that,
it will read from stdin. Then it will output the fields that you specify. To
specify fields start with a % and then put the name as seen in logs. A couple examples:
%auid %pid %path %exe
This will dump the raw value. But suppose you wanted the interpreted values?
Tell it to interpret by giving uppercase:
%AUID %SYSCALL %RES %PATH
Not all events have all fields. By default it will emit a message to stderr
about this. But, you can give it --fill which will print "(none)" in that case.
So, a use might be:
# ausearch --start today -m login --raw | ~sgrubb/test/auformat/auformat "%time %AUID %ses %RES\n"
The app also has 4 virtual field names: date, time, milli, serial. These
relate to the event timestamp which you couldn't specify any other way.
In any event, I am putting this out to see what people think.
-Steve
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Audit format utility
2014-09-23 22:03 Audit format utility Steve Grubb
@ 2014-09-25 16:16 ` LC Bruzenak
2014-09-26 3:05 ` Steve Grubb
0 siblings, 1 reply; 8+ messages in thread
From: LC Bruzenak @ 2014-09-25 16:16 UTC (permalink / raw)
To: linux-audit
[-- Attachment #1.1: Type: text/plain, Size: 605 bytes --]
On 09/23/2014 05:03 PM, Steve Grubb wrote:
> Hello,
>
> I have been doing some thinking about allowing user defined formats to be
> declared as a parameter to ausearch. Before I commit to that, I thought it
> might be interesting to create a "mockup". I have placed a utility here:
>
Steve,
I'll be testing this ASAP to see what shakes out for me.
For user-defined formats, are you thinking maybe along the lines of
allowing a regexp ability to ausearch which would be like the
ausearch-expression capability in libauparse?
Thx,
LCB
--
LC (Lenny) Bruzenak
lenny@magitekltd.com
[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Audit format utility
2014-09-25 16:16 ` LC Bruzenak
@ 2014-09-26 3:05 ` Steve Grubb
2014-09-26 15:24 ` LC Bruzenak
0 siblings, 1 reply; 8+ messages in thread
From: Steve Grubb @ 2014-09-26 3:05 UTC (permalink / raw)
To: LC Bruzenak; +Cc: linux-audit
Hello,
On Thu, 25 Sep 2014 11:16:33 -0500
LC Bruzenak <lenny@magitekltd.com> wrote:
> On 09/23/2014 05:03 PM, Steve Grubb wrote:
> > Hello,
> >
> > I have been doing some thinking about allowing user defined formats
> > to be declared as a parameter to ausearch. Before I commit to that,
> > I thought it might be interesting to create a "mockup". I have
> > placed a utility here:
> >
> Steve,
>
> I'll be testing this ASAP to see what shakes out for me.
> For user-defined formats, are you thinking maybe along the lines of
> allowing a regexp ability to ausearch which would be like the
> ausearch-expression capability in libauparse?
Not exactly. A regex would be for searching for events that match some
criteria. That might be useful in and of itself. What this proposal is
about is allowing one to specify how an event is output. For example,
you may want some kind of xml looking thing. Or maybe html, or maybe
json looking. Or maybe you just want it your way...just because.
I know one of the limitations is that each event is unique. I am
planning to address this in the near future by having a classifier that
you just ask, what is the subject, what is the object, what was the
action, what was the result. It would supply the answer. Then I could
go back to this and add some new classifier virtual fields that could
be $SUBJECT, $OBJECT, etc.
But this proposal is purely about output and not searching.
-Steve
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Audit format utility
2014-09-26 3:05 ` Steve Grubb
@ 2014-09-26 15:24 ` LC Bruzenak
2014-09-28 4:52 ` Burn Alting
0 siblings, 1 reply; 8+ messages in thread
From: LC Bruzenak @ 2014-09-26 15:24 UTC (permalink / raw)
Cc: linux-audit
[-- Attachment #1.1: Type: text/plain, Size: 197 bytes --]
On 09/25/2014 10:05 PM, Steve Grubb wrote:
> But this proposal is purely about output and not searching.
I get it now; thanks Steve!
LCB
--
LC (Lenny) Bruzenak
lenny@magitekltd.com
[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Audit format utility
2014-09-26 15:24 ` LC Bruzenak
@ 2014-09-28 4:52 ` Burn Alting
2014-10-01 18:44 ` Steve Grubb
0 siblings, 1 reply; 8+ messages in thread
From: Burn Alting @ 2014-09-28 4:52 UTC (permalink / raw)
To: Steve Grubb; +Cc: linux-audit
Steve,
Had a play around with it. I am not sure about it's value in it's
current form. Rather than specifying the keys to print, it would be
better to print everything in the event and only 'override' the standard
formatting if there is an 'snode' for a key.
Further, it has a couple of immediate issues given it's using
libauparse.
- it is "lossy" in that it wont parse poorly formed audit events (see
the op key value pair below)
[burn@swtf auformat]$ cat add_user.txt
node=swtf.swtf.dyndns.org type=ADD_USER
msg=audit(1411871714.393:47872): user pid=13455 uid=0 auid=500
ses=11
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
msg='op=adding home directory id=502 exe="/usr/sbin/useradd"
hostname=? addr=? terminal=pts/2 res=success'
[burn@swtf auformat]$ ./auformat "%node %date %time %milli %
serial: type=%TYPE msg=%msg op=%op auid=%auid pid=%pid path=%
path exe=%exe subj=%subj hostname=%hostname terminal=%terminal
res=%res\n" add_user.txt
swtf.swtf.dyndns.org 09/28/2014 12:35:14 393 47872:
type=ADD_USER msg= op=adding auid=500 pid=13455 path=
exe="/usr/sbin/useradd"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
hostname=? terminal=pts/2 res=success
[burn@swtf auformat]$
We loose the strings
- 'user' before the pid key
- op='adding home directory' becomes op'adding'
This is particularly important for incorrectly formatted application
level audit sent via auditd.
- 'rewinding' the event's cursor for each possible key, the call to
auparse_first_record() in print_item(), is probably not what one would
want - but then again, auformat is just a mock up at the moment.
- one looses the parsing 'fix-up' that ausearch does in
src/ausearch-report.c:output_interpreted_node()
- to build a complete event, having addressed the 'rewinding' issue,
would make the format look very messy - you would need to include every
possible key to print all key/values.
- one should add event separation so that further tools could process
the data more easily.
At the moment, the only tool I'm aware of that 'correctly' parses a log
file is ausearch. Perhaps we would be better served by adding another
output option to ausearch to print events in a much more parse-able
format (e.g. XML, JSON)
I am happy to work on this (either extending ausearch or working on
auformat).
Rgds
On Fri, 2014-09-26 at 10:24 -0500, LC Bruzenak wrote:
> On 09/25/2014 10:05 PM, Steve Grubb wrote:
> > But this proposal is purely about output and not searching.
> I get it now; thanks Steve!
>
> LCB
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Audit format utility
2014-09-28 4:52 ` Burn Alting
@ 2014-10-01 18:44 ` Steve Grubb
2014-10-01 21:44 ` Burn Alting
0 siblings, 1 reply; 8+ messages in thread
From: Steve Grubb @ 2014-10-01 18:44 UTC (permalink / raw)
To: burn; +Cc: linux-audit
Hello,
On Sunday, September 28, 2014 02:52:53 PM Burn Alting wrote:
> Had a play around with it. I am not sure about it's value in it's
> current form.
This is why I chose to make it separate for now. Its a strawman for people to
poke at and see what's wrong before committing to something that will be
supported long term.
> Rather than specifying the keys to print, it would be better to print
> everything in the event and only 'override' the standard formatting if there
> is an 'snode' for a key.
Sure, perhaps that is a command line option on how to use the format string.
> Further, it has a couple of immediate issues given it's using
> libauparse.
>
> - it is "lossy" in that it wont parse poorly formed audit events (see
> the op key value pair below)
> [burn@swtf auformat]$ cat add_user.txt
> node=swtf.swtf.dyndns.org type=ADD_USER
> msg=audit(1411871714.393:47872): user pid=13455 uid=0 auid=500
> ses=11
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> msg='op=adding home directory id=502 exe="/usr/sbin/useradd"
> hostname=? addr=? terminal=pts/2 res=success'
> [burn@swtf auformat]$ ./auformat "%node %date %time %milli %
> serial: type=%TYPE msg=%msg op=%op auid=%auid pid=%pid path=%
> path exe=%exe subj=%subj hostname=%hostname terminal=%terminal
> res=%res\n" add_user.txt
> swtf.swtf.dyndns.org 09/28/2014 12:35:14 393 47872:
> type=ADD_USER msg= op=adding auid=500 pid=13455 path=
> exe="/usr/sbin/useradd"
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> hostname=? terminal=pts/2 res=success
> [burn@swtf auformat]$
>
> We loose the strings
> - 'user' before the pid key
Which is meaningless in this case.
> - op='adding home directory' becomes op'adding'
> This is particularly important for incorrectly formatted application
> level audit sent via auditd.
This is a problem in the shadow-utils package. It is the one that I'm
currently having to re-do for this reason and many more. Upstream seems to
have taken a stab at re-doming the audit events and pretty much used it like
syslog.
> - 'rewinding' the event's cursor for each possible key, the call to
> auparse_first_record() in print_item(), is probably not what one would
> want - but then again, auformat is just a mock up at the moment.
Well, if you want your fields in a specific order and its not the order in the
event, then we have no choice. Note that the event is alrady parsed at this
point so we are just literally changing the position in a linked list. The
cost is a series of strcmp calls.
> - one looses the parsing 'fix-up' that ausearch does in
> src/ausearch-report.c:output_interpreted_node()
Not sure what "fix-up" we are talking about. The intention is that auparse
completely mimicks ausearch's interpretation ability (which ausearch was
switched over to use auparse a few releases back).
> - to build a complete event, having addressed the 'rewinding' issue,
> would make the format look very messy - you would need to include every
> possible key to print all key/values.
If you wanted that, yeah. But I am thinking of cases where one may not want
every field. For example, you might do something like this to check file access:
# ausearch --start today -m path --raw |
auformat 'auid=%AUID res=%SUCCESS name=%NAME\n'
> - one should add event separation so that further tools could process
> the data more easily.
I am thinking of 1 event per line. This is kind of a requirement of Map
Reduce.
> At the moment, the only tool I'm aware of that 'correctly' parses a log
> file is ausearch.
If there are omissions in auparse, I really want to know. It must be able to
correctly parse events.
> Perhaps we would be better served by adding another
> output option to ausearch to print events in a much more parse-able
> format (e.g. XML, JSON)
I am sort of going that way. I am thinking about logstash/elastic search and
Map reduce and how one might use the audit system when you have say 10,000
systems.
> I am happy to work on this (either extending ausearch or working on
> auformat).
There are a couple needs at the moment are round writing a test suite to 1)
identify new fields that suddenly show up in a record, 2) locate dangling
values so they can be fixed.
Also, we need some performance testing and improvements of auparse. Does
switching to jemalloc make any difference? Is a linked list the best way to do
it? Can the field searching be smarter?
The auformat work is for now a prototype. I have near term plans to assign
specific meaning to each event so that events are more understandable. When I
have this working, then I think we can look at how we want to output the
event.
-Steve
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Audit format utility
2014-10-01 18:44 ` Steve Grubb
@ 2014-10-01 21:44 ` Burn Alting
2014-10-02 21:23 ` Steve Grubb
0 siblings, 1 reply; 8+ messages in thread
From: Burn Alting @ 2014-10-01 21:44 UTC (permalink / raw)
To: Steve Grubb; +Cc: linux-audit
On Wed, 2014-10-01 at 14:44 -0400, Steve Grubb wrote:
> Hello,
>
> On Sunday, September 28, 2014 02:52:53 PM Burn Alting wrote:
> > Had a play around with it. I am not sure about it's value in it's
> > current form.
>
> This is why I chose to make it separate for now. Its a strawman for people to
> poke at and see what's wrong before committing to something that will be
> supported long term.
>
>
> > Rather than specifying the keys to print, it would be better to print
> > everything in the event and only 'override' the standard formatting if there
> > is an 'snode' for a key.
>
> Sure, perhaps that is a command line option on how to use the format string.
>
>
> > Further, it has a couple of immediate issues given it's using
> > libauparse.
> >
> > - it is "lossy" in that it wont parse poorly formed audit events (see
> > the op key value pair below)
> > [burn@swtf auformat]$ cat add_user.txt
> > node=swtf.swtf.dyndns.org type=ADD_USER
> > msg=audit(1411871714.393:47872): user pid=13455 uid=0 auid=500
> > ses=11
> > subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> > msg='op=adding home directory id=502 exe="/usr/sbin/useradd"
> > hostname=? addr=? terminal=pts/2 res=success'
> > [burn@swtf auformat]$ ./auformat "%node %date %time %milli %
> > serial: type=%TYPE msg=%msg op=%op auid=%auid pid=%pid path=%
> > path exe=%exe subj=%subj hostname=%hostname terminal=%terminal
> > res=%res\n" add_user.txt
> > swtf.swtf.dyndns.org 09/28/2014 12:35:14 393 47872:
> > type=ADD_USER msg= op=adding auid=500 pid=13455 path=
> > exe="/usr/sbin/useradd"
> > subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> > hostname=? terminal=pts/2 res=success
> > [burn@swtf auformat]$
> >
> > We loose the strings
> > - 'user' before the pid key
>
> Which is meaningless in this case.
>
>
> > - op='adding home directory' becomes op'adding'
> > This is particularly important for incorrectly formatted application
> > level audit sent via auditd.
>
> This is a problem in the shadow-utils package. It is the one that I'm
> currently having to re-do for this reason and many more. Upstream seems to
> have taken a stab at re-doming the audit events and pretty much used it like
> syslog.
I suppose my concern is that until we have fixed all the incorrectly
formatted key values, auparse is going to loose information.
>
>
> > - 'rewinding' the event's cursor for each possible key, the call to
> > auparse_first_record() in print_item(), is probably not what one would
> > want - but then again, auformat is just a mock up at the moment.
>
> Well, if you want your fields in a specific order and its not the order in the
> event, then we have no choice. Note that the event is alrady parsed at this
> point so we are just literally changing the position in a linked list. The
> cost is a series of strcmp calls.
>
>
> > - one looses the parsing 'fix-up' that ausearch does in
> > src/ausearch-report.c:output_interpreted_node()
>
> Not sure what "fix-up" we are talking about. The intention is that auparse
> completely mimicks ausearch's interpretation ability (which ausearch was
> switched over to use auparse a few releases back).
By 'fix-up' I meant the code like
// Some user messages have msg='uid=500 in this case
// skip the msg= piece since the real stuff is the uid=
...
// Value side has commas and another field exists
// Known: LABEL_LEVEL_CHANGE banners=none,none
// Known: ROLL_ASSIGN new-role=r,r
// Known: any MAC LABEL can potentially have commas
etc
>
>
> > - to build a complete event, having addressed the 'rewinding' issue,
> > would make the format look very messy - you would need to include every
> > possible key to print all key/values.
>
> If you wanted that, yeah. But I am thinking of cases where one may not want
> every field. For example, you might do something like this to check file access:
>
>
> # ausearch --start today -m path --raw |
> auformat 'auid=%AUID res=%SUCCESS name=%NAME\n'
>
>
> > - one should add event separation so that further tools could process
> > the data more easily.
>
> I am thinking of 1 event per line. This is kind of a requirement of Map
> Reduce.
So you expect the complete event of my tailing audit.log
node=swtf.swtf.dyndns.org type=SYSCALL
msg=audit(1412198543.190:141570): arch=c000003e syscall=59
success=yes exit=0 a0=1a2d530 a1=1a2d350 a2=1a06f10 a3=20
items=2 ppid=19529 pid=32647 auid=500 uid=0 gid=0 euid=0 suid=0
fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail"
exe="/usr/bin/tail"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
key="cmds"
node=swtf.swtf.dyndns.org type=EXECVE
msg=audit(1412198543.190:141570): argc=3 a0="tail" a1="-f"
a2="/var/log/audit/audit.log"
node=swtf.swtf.dyndns.org type=CWD
msg=audit(1412198543.190:141570): cwd="/home/burn"
node=swtf.swtf.dyndns.org type=PATH
msg=audit(1412198543.190:141570): item=0 name="/usr/bin/tail"
inode=2135830 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00
obj=system_u:object_r:bin_t:s0 nametype=NORMAL
node=swtf.swtf.dyndns.org type=PATH
msg=audit(1412198543.190:141570): item=1 name=(null)
inode=524293 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00
obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL
to generate one line of output?
>
>
> > At the moment, the only tool I'm aware of that 'correctly' parses a log
> > file is ausearch.
>
> If there are omissions in auparse, I really want to know. It must be able to
> correctly parse events.
By correctly, I meant completely. It currently, in
output_interpreted_node() handles incorrectly formed key values like
op=adding home directory
as per
[burn@swtf auformat]$ /sbin/ausearch -i -if add_user.txt
----
node=swtf.swtf.dyndns.org type=ADD_USER msg=audit(09/28/2014
12:35:14.393:47872) : user pid=13455 uid=root auid=burn ses=11
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
msg='op=adding home directory id=freddo exe=/usr/sbin/useradd hostname=?
addr=? terminal=pts/2 res=success'
[burn@swtf auformat]$
>
>
> > Perhaps we would be better served by adding another
> > output option to ausearch to print events in a much more parse-able
> > format (e.g. XML, JSON)
>
> I am sort of going that way. I am thinking about logstash/elastic search and
> Map reduce and how one might use the audit system when you have say 10,000
> systems.
Which is my use case.
>From my standpoint, I need each host to
- enrich the data e.g. uid=500 to become uid=500(burn) (I want both the
id and interpreted name for checking id mismatches in the enterprise),
syscall=59 to become syscall=execve, etc
- not loose important data (op=adding home directory)
- turn single and multi-line events into well defined and formatted
events (xml/json),
- send the data to an aggregation point within the enterprise.
At the aggregation point I can apply capability such as logstash/elastic
search/map reduce and analyse the data.
Ideally I'd extend ausearch-report.c:output_record() to output events in
a well defined format (xml/json) - probably refactoring
output_interpreted_node() to generate it's current format or xml/json
depending on a flag so we only have one 'parser' to maintain.
>
> > I am happy to work on this (either extending ausearch or working on
> > auformat).
>
> There are a couple needs at the moment are round writing a test suite to 1)
> identify new fields that suddenly show up in a record, 2) locate dangling
> values so they can be fixed.
>
> Also, we need some performance testing and improvements of auparse. Does
> switching to jemalloc make any difference? Is a linked list the best way to do
> it? Can the field searching be smarter?
>
> The auformat work is for now a prototype. I have near term plans to assign
> specific meaning to each event so that events are more understandable. When I
> have this working, then I think we can look at how we want to output the
> event.
>
> -Steve
Burn
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Audit format utility
2014-10-01 21:44 ` Burn Alting
@ 2014-10-02 21:23 ` Steve Grubb
0 siblings, 0 replies; 8+ messages in thread
From: Steve Grubb @ 2014-10-02 21:23 UTC (permalink / raw)
To: burn; +Cc: linux-audit
On Thursday, October 02, 2014 07:44:32 AM Burn Alting wrote:
> On Wed, 2014-10-01 at 14:44 -0400, Steve Grubb wrote:
> > > Further, it has a couple of immediate issues given it's using
> > > libauparse.
> > >
> > > - it is "lossy" in that it wont parse poorly formed audit events (see
> > > the op key value pair below)
> > >
> > > [burn@swtf auformat]$ cat add_user.txt
> > > node=swtf.swtf.dyndns.org type=ADD_USER
> > > msg=audit(1411871714.393:47872): user pid=13455 uid=0 auid=500
> > > ses=11
> > > subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> > > msg='op=adding home directory id=502 exe="/usr/sbin/useradd"
> > > hostname=? addr=? terminal=pts/2 res=success'
> > > [burn@swtf auformat]$ ./auformat "%node %date %time %milli %
> > > serial: type=%TYPE msg=%msg op=%op auid=%auid pid=%pid path=%
> > > path exe=%exe subj=%subj hostname=%hostname terminal=%terminal
> > > res=%res\n" add_user.txt
> > > swtf.swtf.dyndns.org 09/28/2014 12:35:14 393 47872:
> > > type=ADD_USER msg= op=adding auid=500 pid=13455 path=
> > > exe="/usr/sbin/useradd"
> > > subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> > > hostname=? terminal=pts/2 res=success
> > > [burn@swtf auformat]$
> > >
> > > We loose the strings
> > >
> > > - 'user' before the pid key
> >
> > Which is meaningless in this case.
> >
> > > - op='adding home directory' becomes op'adding'
> > >
> > > This is particularly important for incorrectly formatted application
> > > level audit sent via auditd.
> >
> > This is a problem in the shadow-utils package. It is the one that I'm
> > currently having to re-do for this reason and many more. Upstream seems to
> > have taken a stab at re-doming the audit events and pretty much used it
> > like syslog.
>
> I suppose my concern is that until we have fixed all the incorrectly
> formatted key values, auparse is going to loose information.
OK. I see. I really think we should get some tool created that can help
identify these. It could be as simple as pushing into auparse, iterating
across fields to to recreate the record, then diff the original and the
recreated.
>From that, we can get fixes in place. I think shadow-utils is the package most
affected.
> > > - 'rewinding' the event's cursor for each possible key, the call to
> > > auparse_first_record() in print_item(), is probably not what one would
> > > want - but then again, auformat is just a mock up at the moment.
> >
> > Well, if you want your fields in a specific order and its not the order in
> > the event, then we have no choice. Note that the event is alrady parsed
> > at this point so we are just literally changing the position in a linked
> > list. The cost is a series of strcmp calls.
> >
> > > - one looses the parsing 'fix-up' that ausearch does in
> > > src/ausearch-report.c:output_interpreted_node()
> >
> > Not sure what "fix-up" we are talking about. The intention is that auparse
> > completely mimicks ausearch's interpretation ability (which ausearch was
> > switched over to use auparse a few releases back).
>
> By 'fix-up' I meant the code like
> // Some user messages have msg='uid=500 in this case
> // skip the msg= piece since the real stuff is the uid=
> ...
> // Value side has commas and another field exists
> // Known: LABEL_LEVEL_CHANGE banners=none,none
> // Known: ROLL_ASSIGN new-role=r,r
> // Known: any MAC LABEL can potentially have commas
> etc
Auparse should handle these.
> > > - to build a complete event, having addressed the 'rewinding' issue,
> > > would make the format look very messy - you would need to include every
> > > possible key to print all key/values.
> >
> > If you wanted that, yeah. But I am thinking of cases where one may not
> > want
> > every field. For example, you might do something like this to check file
> > access:
> >
> >
> > # ausearch --start today -m path --raw |
> >
> > auformat 'auid=%AUID res=%SUCCESS name=%NAME\n'
> >
> > > - one should add event separation so that further tools could process
> > > the data more easily.
> >
> > I am thinking of 1 event per line. This is kind of a requirement of Map
> > Reduce.
>
> So you expect the complete event of my tailing audit.log
> node=swtf.swtf.dyndns.org type=SYSCALL
> msg=audit(1412198543.190:141570): arch=c000003e syscall=59
> success=yes exit=0 a0=1a2d530 a1=1a2d350 a2=1a06f10 a3=20
> items=2 ppid=19529 pid=32647 auid=500 uid=0 gid=0 euid=0 suid=0
> fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail"
> exe="/usr/bin/tail"
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> key="cmds"
> node=swtf.swtf.dyndns.org type=EXECVE
> msg=audit(1412198543.190:141570): argc=3 a0="tail" a1="-f"
> a2="/var/log/audit/audit.log"
> node=swtf.swtf.dyndns.org type=CWD
> msg=audit(1412198543.190:141570): cwd="/home/burn"
> node=swtf.swtf.dyndns.org type=PATH
> msg=audit(1412198543.190:141570): item=0 name="/usr/bin/tail"
> inode=2135830 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00
> obj=system_u:object_r:bin_t:s0 nametype=NORMAL
> node=swtf.swtf.dyndns.org type=PATH
> msg=audit(1412198543.190:141570): item=1 name=(null)
> inode=524293 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00
> obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL
> to generate one line of output?
I really don't think all of those fields are necessary to understand what is
happening. I have a plan to be able to take that and reduce it down to
[On node] at time, subj [acting-as] results action what using
Your event would become
At 17:22:23 uid-500 acting as root successfully run cmds using tail
I think it will take me a few weeks to get it to this point. But I suspect
that this work will point the way to reducing logs smartly. This of course
doesn't mean getting rid of the full event. But for processing massive amounts
of data, it needs to become normalized.
> > > At the moment, the only tool I'm aware of that 'correctly' parses a log
> > > file is ausearch.
> >
> > If there are omissions in auparse, I really want to know. It must be able
> > to correctly parse events.
>
> By correctly, I meant completely. It currently, in
> output_interpreted_node() handles incorrectly formed key values like
> op=adding home directory
A bugzilla should be opened on the package originating the event. These need
to get fixed.
> as per
> [burn@swtf auformat]$ /sbin/ausearch -i -if add_user.txt
> ----
> node=swtf.swtf.dyndns.org type=ADD_USER msg=audit(09/28/2014
> 12:35:14.393:47872) : user pid=13455 uid=root auid=burn ses=11
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> msg='op=adding home directory id=freddo exe=/usr/sbin/useradd hostname=?
> addr=? terminal=pts/2 res=success'
> [burn@swtf auformat]$
>
> > > Perhaps we would be better served by adding another
> > > output option to ausearch to print events in a much more parse-able
> > > format (e.g. XML, JSON)
> >
> > I am sort of going that way. I am thinking about logstash/elastic search
> > and Map reduce and how one might use the audit system when you have say
> > 10,000 systems.
>
> Which is my use case.
> From my standpoint, I need each host to
> - enrich the data e.g. uid=500 to become uid=500(burn) (I want both the
> id and interpreted name for checking id mismatches in the enterprise),
> syscall=59 to become syscall=execve, etc
> - not loose important data (op=adding home directory)
The best thing here is opening bz. We really need some test tool to search for
malformed events for when upstream tinkers with the events.
> - turn single and multi-line events into well defined and formatted
> events (xml/json),
> - send the data to an aggregation point within the enterprise.
>
> At the aggregation point I can apply capability such as logstash/elastic
> search/map reduce and analyse the data.
Sure. I have not looked at what it takes to make a logstash plugin. But I
could envision feeding the event into auparse and then using it to provide the
interpreted fields as needed.
> Ideally I'd extend ausearch-report.c:output_record() to output events in
> a well defined format (xml/json) - probably refactoring
> output_interpreted_node() to generate it's current format or xml/json
> depending on a flag so we only have one 'parser' to maintain.
Sure. I could see that as a follow on after getting some new capabilities in
place and working through how to enrich audit events in general. I'd like to
make a couple prototypes as standalone utilities to experiment with. When it
seems to work good, then merge into ausearch. I'd like to see others in the
community also help define what this can do.
Thanks,
-Steve
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-10-02 21:23 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-23 22:03 Audit format utility Steve Grubb
2014-09-25 16:16 ` LC Bruzenak
2014-09-26 3:05 ` Steve Grubb
2014-09-26 15:24 ` LC Bruzenak
2014-09-28 4:52 ` Burn Alting
2014-10-01 18:44 ` Steve Grubb
2014-10-01 21:44 ` Burn Alting
2014-10-02 21:23 ` Steve Grubb
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.