Keeping track of called syscalls in real-time

Keeping track of called syscalls in real-time

[Ben Mezger]

Can the kernel keep track of all the system calls that were called by an
application/module in real-time?
I know I can statically use strace, or even gdb, but I am looking for a
solution in real time when the application/module is already running and
the user has no control over it.

I am not sure if a system call needs to go through a sort of wrapper to
get it from the syscall table, which I'm then assuming I can get such
info from there, but I am not sure.

I am looking for hints/options to archive this.

Many thanks

-- 
- seds
~> https://seds.nl

Keeping track of called syscalls in real-time

[valdis.kletnieks at vt.edu]

On Wed, 28 Jun 2017 17:48:15 -0300, Ben Mezger said:
> Can the kernel keep track of all the system calls that were called by an
> application/module in real-time?
> I know I can statically use strace, or even gdb, but I am looking for a
> solution in real time when the application/module is already running and
> the user has no control over it.

What actual problem are you trying to solve by having the information?

How "real-time" does it have to be?

Have you looked at the syscall audit facility?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170628/15f0c2f0/attachment.bin 

Keeping track of called syscalls in real-time

[Ben Mezger]

I'm actually formulating my thesis project. I am looking for a way to
intercept system calls (those chosen by the users), where I can keep
track of what syscall has been called and by who.

A big picture of the _main_ idea of interception would be: Application
called a syscall -> Intercept and delay call -> do something before the
call -> return back to the syscall.

By real-time I mean as soon as an application called a syscall (i.e.
fopen), I could then receive a reply from the kernel informing me X
called fopen, where X could be a pid or whatever.

>> Have you looked at the syscall audit facility?

I have not. Are you talking about auditctl?

On 06/28/2017 06:19 PM, valdis.kletnieks at vt.edu wrote:
> On Wed, 28 Jun 2017 17:48:15 -0300, Ben Mezger said:
>> Can the kernel keep track of all the system calls that were called by an
>> application/module in real-time?
>> I know I can statically use strace, or even gdb, but I am looking for a
>> solution in real time when the application/module is already running and
>> the user has no control over it.
> 
> What actual problem are you trying to solve by having the information?
> 
> How "real-time" does it have to be?
> 
> Have you looked at the syscall audit facility?
> 
> 
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 

-- 
- seds
~> https://seds.nl

Keeping track of called syscalls in real-time

[valdis.kletnieks at vt.edu]

On Wed, 28 Jun 2017 19:06:56 -0300, Ben Mezger said:
> I'm actually formulating my thesis project. I am looking for a way to
> intercept system calls (those chosen by the users), where I can keep
> track of what syscall has been called and by who.

As I said before - knowing this, what do you *do* with it? Statistics
after the fact?  Apply security rules before the fact?  Something else?
The answer depends *a lot* on what you're planning to *do* with the info.

> A big picture of the _main_ idea of interception would be: Application
> called a syscall -> Intercept and delay call -> do something before the
> call -> return back to the syscall.

"Do something before the syscall".

Congrats - you just re-invented the LSM subsystem.  Or possibly seccomp,
depending on what it is you're trying to accomplish.

Note that LSM's have some restrictions on what they can and can't do,
mostly because it's otherwise almost impossible to do any reasoning about
the security and stability guarantees of a process/system otherwise.

> By real-time I mean as soon as an application called a syscall (i.e.
> fopen), I could then receive a reply from the kernel informing me X
> called fopen, where X could be a pid or whatever.

Yes, but the question is "what value of "I then receive" appropriate?
Do you need it before the syscall is executed? After it is finished?
Or "don't care at all as long as we eventually get a complete trail"?

> >> Have you looked at the syscall audit facility?
>
> I have not. Are you talking about auditctl?

That's part of the userspace utilities that interface to the audit system.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170628/44fa1b2d/attachment.bin 

Keeping track of called syscalls in real-time

[Ben Mezger]

Let me clear things out.

> As I said before - knowing this, what do you *do* with it? Statistics
> after the fact?  Apply security rules before the fact?  Something else?
> The answer depends *a lot* on what you're planning to *do* with the info.

There is no statistics involved. I am trying to intercept *some* system
calls. The list of syscalls I should intercept will be set by the user
in a form of a rule, however, if a specified syscall is meant for file
I/O (i.e. fopen), the user would need to specify which files he would
like the interception to take part on whenever fopen is called.
A simple example of a user rule would be (in a nutshell):

# syscall   #intercept   #file_arg     #action   #onuid  #oncall
fopen       1		 /etc/shadow   verify	 12      0

Where #syscall specifies which syscall to intercept, #intercept is a
bool whenever it should run or not, the #file_arg basically says
"intercept fopen only when fopen is called on /etc/shadow", #action
specifies the name of the procedure the tool would run when intercepting
fopen, #onuid specifies the user uid to intercept (run this just on the
user who has 12 as uid) and finally, #oncall is a bool telling the tool
to intercept after the syscall has returned (1 for after the call, 0 for
before).

Whenever fopen("/etc/shadow", "r") is called, the tool would intercept
it, run the verify() procedure, and return back to the syscall, allowing
it to do it's job.

> Yes, but the question is "what value of "I then receive" appropriate?
> Do you need it before the syscall is executed? After it is finished?
> Or "don't care at all as long as we eventually get a complete trail"?

That all depends on the config for *that* specific call. Using the
previous examples, I would need the kernel to tell me right when fopen
was called;

int foo(...){
    ...
    fopen(arg, "r"); <- need an alert from the kernel here
}

I am using the word "tool" here, but I am willing to get this builtin to
the kernel when compiling it, so as a root user, it would be slightly
more difficult to disable it without having to recompile everything (afaik).
 > Congrats - you just re-invented the LSM subsystem.  Or possibly seccomp,
> depending on what it is you're trying to accomplish.
> 
> Note that LSM's have some restrictions on what they can and can't do,
> mostly because it's otherwise almost impossible to do any reasoning about
> the security and stability guarantees of a process/system otherwise.

I understand seccomp and LSM allows __some__ type of syscall
interposition (where afaik seccomp blocks mostly all of them), but what
I am willing to do here is not *reinvent* the wheel, I am willing to
make things a bit more configurable, where a user has access to an API
where he could write custom procedures to run on the interception side,
without having to dig through the source.

Many thanks


On 06/28/2017 07:26 PM, valdis.kletnieks at vt.edu wrote:
> On Wed, 28 Jun 2017 19:06:56 -0300, Ben Mezger said:
>> I'm actually formulating my thesis project. I am looking for a way to
>> intercept system calls (those chosen by the users), where I can keep
>> track of what syscall has been called and by who.
> 
> As I said before - knowing this, what do you *do* with it? Statistics
> after the fact?  Apply security rules before the fact?  Something else?
> The answer depends *a lot* on what you're planning to *do* with the info.
> 
>> A big picture of the _main_ idea of interception would be: Application
>> called a syscall -> Intercept and delay call -> do something before the
>> call -> return back to the syscall.
> 
> "Do something before the syscall".
> 
> Congrats - you just re-invented the LSM subsystem.  Or possibly seccomp,
> depending on what it is you're trying to accomplish.
> 
> Note that LSM's have some restrictions on what they can and can't do,
> mostly because it's otherwise almost impossible to do any reasoning about
> the security and stability guarantees of a process/system otherwise.
> 
>> By real-time I mean as soon as an application called a syscall (i.e.
>> fopen), I could then receive a reply from the kernel informing me X
>> called fopen, where X could be a pid or whatever.
> 
> Yes, but the question is "what value of "I then receive" appropriate?
> Do you need it before the syscall is executed? After it is finished?
> Or "don't care at all as long as we eventually get a complete trail"?
> 
>>>> Have you looked at the syscall audit facility?
>>
>> I have not. Are you talking about auditctl?
> 
> That's part of the userspace utilities that interface to the audit system.
> 

-- 
- seds
~> https://seds.nl

Keeping track of called syscalls in real-time

[W. Michael Petullo]

> Whenever fopen("/etc/shadow", "r") is called, the tool would intercept
> it, run the verify() procedure, and return back to the syscall, allowing
> it to do it's job.

This sounds like an LSM, possibly with a component which communicates
with userspace, depending on how sophisticated "verify" needs to be.

We've also done some very early work in trying to do this type of thing
from a hypervisor. See:

	https://www.flyn.org/projects/VisorFlow/

-- 
Mike

:wq

Keeping track of called syscalls in real-time

[Ben Mezger]

> This sounds like an LSM, possibly with a component which communicates
> with userspace, depending on how sophisticated "verify" needs to be.

Yes, the component *should* communicate with the userspace. The
sophistication of "verify" varies from user to user. The tool will
provide a few procedures to, say, verify integrity and log call. But
"verify" was a plain example, where my point was that the user could
extend/add these procedures for their own needs.

VisorFlow sounds interesting. I've seen the paper is on submission. When
will it be published?

On 06/28/2017 09:49 PM, W. Michael Petullo wrote:
>> Whenever fopen("/etc/shadow", "r") is called, the tool would intercept
>> it, run the verify() procedure, and return back to the syscall, allowing
>> it to do it's job.
> 
> This sounds like an LSM, possibly with a component which communicates
> with userspace, depending on how sophisticated "verify" needs to be.
> 
> We've also done some very early work in trying to do this type of thing
> from a hypervisor. See:
> 
> 	https://www.flyn.org/projects/VisorFlow/
> 

-- 
- seds
~> https://seds.nl

Keeping track of called syscalls in real-time

[valdis.kletnieks at vt.edu]

On Wed, 28 Jun 2017 20:16:33 -0300, Ben Mezger said:

(By the way - fopen() is a glibc construct, not a syscall)

> I understand seccomp and LSM allows __some__ type of syscall
> interposition (where afaik seccomp blocks mostly all of them), but what
> I am willing to do here is not *reinvent* the wheel, I am willing to
> make things a bit more configurable, where a user has access to an API
> where he could write custom procedures to run on the interception side,
> without having to dig through the source.

AppArmor, SMACK, SELinux, and Tomoyo all allow writing of user-readable
policies in userspace and then applying the policy (through some combination of
labelling and/or loading policy into the kernel at boot), without having to "dig
through the source". In fact, it's probably impossible to write a non-trivial
and yet actually usable LSM that doesn't work this way. So it sounds like you
probably *are* re-inventing some wheels here.

If you have a *very* recent kernel, you can probably get yourself into all
sorts of trouble by creating an LSM that allows attachment of arbitrary BPF
programs to the various hooks listed in include/linux/lsm_hooks.h.  However,
there's no really good way to allow unrestricted "custom procedures to run on
intercept" that doesn't involve having a reasonably good idea of how the rest
of the source works. You *really* want to restrict it to interpreting a policy
language of some sort - at which point BPF is probably overkill.

(For bonus points - spend at least 30-45 minutes thinking deeply about all the
ways an attacker can use the ability to use "custom procedures on intercept" to
attack the system. In particular, examine ways an attacker who gets root and/or
"authority to load policy" can leverage it to load in a kernel-resident
rootkit.  See http://phrack.org/issues/58/7.html#article on SucKIT for a worked
example of loading code into a kernel that doesn't even support modules.)

Note also that you're going to have a *lot* of issues trying to do path-based
security - there's all sorts of ugly things that can be done with deep nested
paths, symlinks, remote NFS servers, and race conditions.  And things like
openat() and namespaces only add to the fun...

And I'm convinced that even after some 4 decades, there's *still* code and
systems configurations out there that allows naughty uses of multiple hard
links to a file.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170628/f52f4e53/attachment.bin