Linux C2-Style Audit Capability

Linux C2-Style Audit Capability

[Red Phoenix]

G'day,

We've been working on a kernel module/audit daemon combination that 
implements a C2-style, objective-based audit capability for several months 
now. We're at a stage where it's stable and functional, but a fundamental 
kernel-to-user communication issue is making things difficult at the moment. 
I'm hoping that someone here will have an idea of how to solve things.

At present, communication between audit daemon (user space) and the audit 
module takes the form of a device (/dev/audit). The daemon communicates with 
module via ioctl(), the module communicates with the daemon using read(), 
via a file_operations structure in the module. As such, when a new audit 
event is generated, the kernel module buffers any audit data in a ring 
buffer, SIGIO's the daemon, and waits for the daemon to call a read() on 
/dev/audit.

System calls are overridden by pointing sys_call_table[system call] to a 
replacement function which saves off the data for auditing purposes, then 
calls the original system call.
eg:

// Audit rmdir directory creations.
asmlinkage int audit_rmdir(const char *path)
{
        io_class        io_event;

        // If no daemon, or the daemon has generated the event.
        if((!AUDIT_IS_RUNNING) || (AUDIT_IS_RUNNING && (current->pid == 
auditdaemon_task_struct->pid)))
        {
                return orig_rmdir(path);
        }

        io_event.t_return.returncode=orig_rmdir(path);
        io_event.t_header.event_class=AUDIT_CLASS_IO;
        io_event.t_header.event_id=SYS_rmdir;
        io_event.t_process.pid=current->pid;

        io_event.t_process.name[0]='\0';
        strncpy(io_event.t_process.name,current->comm,MAXCOMMAND);

        // Store the path
        strncpy(io_event.t_path.path,path,MAX_PATH);

        // Everything else can be handled by the generic event handler.
        audit_event(&io_event);

        // Now continue on with the normal exit.
        return io_event.t_return.returncode;
}

This works beautifully in most circumstances. However, when high-volume 
audit events are turned on (eg: open()), the user-space audit daemon cannot 
keep up with the kernel, and therefore my ring buffer fills. As such, we 
lose events.

                if(!write_io((io_class *)event))
                {
                        // Couldnt write it out.
                        // No space available in the ring buffer.
                        signal=0;
                        lost_events++;
                }


Increasing the ring buffer memory allocation helps somewhat, but this only 
serves to postpone the inevitable - the user-space audit daemon cannot keep 
up with the events generated by the kernel if the kernel is generating 
events very quickly. Even with a buffer of 1MB kernel memory, events are 
lost. Unfortunately, (like a normal device driver), I can't afford to slow 
my data source down - I can't just ask the kernel not to execute any more 
system calls until my user-space program catches up ;) - I have to dump 
data.

In normal operations, very few (if any) audit events are lost.. However, 
turn on the open() audit event, load mozilla, and you'll probably miss a 
dozen or so events (out of a few hundred). Do a grep "blah" 
/usr/include/*/*, and the audit daemon will miss about 500 out of perhaps 
3000.

What I've tried:
* Removed the dependance on SIGIO to notify the audit daemon of a new event 
- I used blocking IO in the module, and a while(read) in the daemon. No 
significant difference.

* nice -19 auditd. No significant difference.

* I even briefly attempted having the audit daemon create a pipe (one end of 
which is passed to the audit module), and have the audit module write to it 
using sys_write()... however, since file descriptors are relative to the 
process that opened them, the processes I wish to audit (which call 
SYS_execve for example), do not share the file descriptor, and I cannot 
write to the pipe in the routine called from my routine which overrides 
execve.
This seemed to be almost an ideal implementation before I really thought it 
through - a pipe buffers the data in user space, rather than the module 
buffering the data in kernel space.

What I've considered:
* Opening the output file directly from the module
  - I'd prefer to avoid this if at all possible.
* A file pointer that is 'global' across all processes?
  - Does such a beast exist?

So does anyone have any suggestions on how to effectively push the data out 
from the audit module, without losing audit events, and without putting 
artificial delays in the system calls to facilitate guaranteed event 
delivery?

Ideally, I'd like to be able to send the data straight out the door in the 
kernel, and have it buffered somewhere that is available to user-space until 
the audit daemon can pick the data up and run with it.... However, I'll 
consider ANY suggestions.

Overview available from 
http://www.intersectalliance.com/projects/snare/index.html for those 
interested in more details.

Sorry for the hotmail return address by the way - with all the spambots 
around, I'd rather the hotmail server cope with unrelated spam than mine. :)

I'd appreciate being cc'd at the hotmail address (if you're willing), but 
will try and watch any mailing-list replies via the lkml web archives.

Leigh.
---
Leigh Purdie, Director - InterSect Alliance Pty Ltd
http://www.intersectalliance.com/

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp

Re: Linux C2-Style Audit Capability

[Alan Cox]

> System calls are overridden by pointing sys_call_table[system call] to a 
> replacement function which saves off the data for auditing purposes, then 
> calls the original system call.

Ugly but that bit probably ties in with all the other folks trying to put
together a unified security hook set for 2.5

> audit events are turned on (eg: open()), the user-space audit daemon cannot 
> keep up with the kernel, and therefore my ring buffer fills. As such, we 
> lose events.
> 
>                 if(!write_io((io_class *)event))
>                 {
>                         // Couldnt write it out.
>                         // No space available in the ring buffer.
>                         signal=0;
>                         lost_events++;
>                 }

So why don't you block ? Obviously you must never block logging events
caused by the logging daemon itself but in the other cases since you are 
logging before (and maybe after) a syscall rather than logging during the
syscall where locks may be held I dont see the problem

Re: Linux C2-Style Audit Capability

[Ralf Baechle]

On Sat, Aug 04, 2001 at 10:23:58AM +0100, Alan Cox wrote:

> > System calls are overridden by pointing sys_call_table[system call] to a 
> > replacement function which saves off the data for auditing purposes, then 
> > calls the original system call.
> 
> Ugly but that bit probably ties in with all the other folks trying to put
> together a unified security hook set for 2.5

That approach means you have to write a large number of system specific
wrappers for the various architectures.

  Ralf

Re: Linux C2-Style Audit Capability

[richard offer]

* frm alan@lxorguk.ukuu.org.uk "08/04/01 09:24:11 +0000" | sed '1,$s/^/* /'
*
*> System calls are overridden by pointing sys_call_table[system call] to a 
*> replacement function which saves off the data for auditing purposes,
*> then  calls the original system call.
* 
* Ugly but that bit probably ties in with all the other folks trying to put
* together a unified security hook set for 2.5

Simply wrapping the system calls isn't going to get a CAPP (or C2)
compliant audit implementation. It also isn't how the "unified security
hooks" (aka LSM, Linux Security Modules) are working.

SGI is working towards a CAPP compliant audit implementation under the LSM
framework, I'd suggest that you head over to http://lsm.immunix.org/ for
more details on LSM.


richard.

-----------------------------------------------------------------------
Richard Offer                     Technical Lead, Trust Technology, SGI
"Specialization is for insects"
_______________________________________________________________________

Linux C2-Style Audit Capability

[Hank Leininger]

On 2001-08-04, "Red Phoenix" <redph0enix@hotmail.com> wrote:

[ snip "every syscall can/will be interrupted and it+arguments want to be
  logged somewhere, and a userspace log daemon can't keep up" ]

> What I've considered:
> * Opening the output file directly from the module
>   - I'd prefer to avoid this if at all possible.

While this sounds ugly, it is I suspect the way several commercial UNIXen
handle system call logging[0].  There's some wacky arcane userspace
commands that set the path-to-be-logged-to and tweak what shall be
audited, and then the kernel just goes.  If you think about it, it's the
easiest way to handle the inherent race condition and/or performance hits
of having one userspace process be the "clearing house" for every other
processes's syscalls (each syscall now causes at least 2 extra context
switches).

Also, keep in mind that auditing to a userspace process means it all goes
away if/when someone cracks root.  Doing all of it in kernel space makes
it harder to subvert.  Right now the "root user"<->"kernel" barrier is too
porous, but there are efforts under way/things you can do to make it
better[1] (run around with some chattr +i's, drop privileges early in the
boot process to disable raw memory and raw disk access, etc).  You
shouldn't have to solve that problem, but you do need to keep it in mind
(if nothing else, point these issues out in your first INSTALL doc, caveat
emptor ;).

iif that problem is addressed, an in-kernel audit subsystem with userspace
knobs stands a chance (similar to the old securelevel idea, it should
probably not be possible to "unset" various auditing options once they're
turned on, without a reboot).  A system which relies on a userspace
root-run process (on a conventional UNIXy system) to actually write logs
out isn't going to last two seconds once the system has been compromised;
if the audit trail is stored only locally it will then be targetted
(though the same root<->kernel hardening might make that impossible... but
the logging of new events can certainly be stopped).

The usual log-replication standard of a syslog loghost is worth thinking
about; it'd be pretty easy to have the kernel module spit logs out as
syslog-alike packets (probably stamped with a MAC and possibly
payload-encrypted).  It has some serious shortcomings if you want 100%
guaranteed remote capturing of all logs (in short: it won't happen) but
it's valuable nevertheless; if a box is cracked and its logs wiped,
99.999% complete logs written to another box in real time is better than
0% of the logs being available for analysis[2].

[ Um, after a visit to your project page, I see you're already experienced
  with at least Solaris's audit subsystem, and have things that feed these
  to syslog already.  Heh ;)  Course you know what I'll ask about next:
  a kernel-module version of backlog for Solaris. ]

[0] Not that is necessarily a ringing endorsement.
[1] Note I didn't say "good enough;" that's not possible for all
    definitions of "enough."
[2] Of course, smart attackers will try to make sure they're in the 0.001%

--
Hank Leininger <hlein@progressive-comp.com>

Re: Linux C2-Style Audit Capability

[Elladan]

This is a classic producer-consumer problem.  There are only two solutions:

* Discard log events (unacceptable for security - anyone could just turn off 
  logging whenever they want by spamming the system with spurious log events)
* Block the producer

So, really, you have no choice but to block the producer of the log events, eg.
block all logged system calls until the buffer is depleted again.

Of course, this leads to all sorts of enjoyable deadlock conditions in the
system...  The audit daemon itself cannot block on anything safely, really.
Eg:

What if the audit daemon is just passing the log events on to a log daemon on
the same machine?  The log daemon itself gets logged, so it gets blocked
waiting for the audit buffer.  The audit daemon is trying to deplete its own
buffer by sending to the log daemon, which will never be available...  Poof!

What if the audit daemon is just passing the log events on to an external log
daemon on another machine?  This seems ok, as long as the other machine is
visible.  As soon as it dies, the audited machine will hang until the log host
comes back.  It can't even be shut down, since the shutdown command will be
logged ...

...  But what if the logging machine itself is just a router that sends off
streams from multiple machines to be warehoused?  This seems fine, except...
What if the warehousing machine is itself audited?  It tries to log its audit
stream back to the router, which comes back to itself, which blocks the entire
computer network until someone kicks something.  Ack! 

So it seems that to avoid losing a few system calls in a kernel ring buffer,
you've now crashed an entire NOC.  :-)


The audit daemon will have to be smart and log to some sort of ring buffer on
disk, but of course, disk space is not infinite, so eventually it will have to
start throwing away old log events, just as the kernel does.  So we're right
back where we started.  But perhaps it will take a few days before you have to
do this, with a big disk buffer.  

(Or perhaps not...  If you logged every lstat(), and each log line was 100 
bytes, it seems a malicious user could generate 103 gigabytes of log data in 1
hour, thus erasing their malicious actions at the start of the hour.)

-J