Identifying thread/process termination

Identifying thread/process termination

[L. A. Walsh]

On 2020/11/16 05:43, Paul Moore wrote:

> The most important thing to keep in mind is that all of the threads
> inside a process share the same memory space.  It is the lack of a
> strong, enforceable boundary between threads which makes it difficult,
> if not impossible, to view threads as individual entities from a
> security perspective.
---
    Depends on how much your security policy relies on recognizing
abnormal behavior.  If a program splits function across well defined
areas by a named thread, one may develop a baseline of "normal"
functionality associated with given threads.  Determining that
a thread is operating outside it's normal range can allow for a
earlier detection and better monitoring of "safe" and/or secure
operation.

    How programs operate, especially in regards to what work is
normal for a given thread can only be done with thread level
monitoring.  While given threads _can_ access global-user memory,
that involves how they are coded or programmed to run.  That, in
turn, can be used to help define boundaries and integrity levels
of various processes in a system. 

    For example, even though a logging thread might gather data
from other threads, knowing that it can only write to output
to specific configured destinations would allow swift detection
of aberrations.





--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Identifying thread/process termination

[L. A. Walsh]

Repost from right address.
On 2020/10/08 08:33, Lenny Bruzenak wrote:

> On 10/7/20 7:27 PM, Paul Moore wrote:
>
>   
>> Almost everywhere in the kernel we record the TGID for the "pid="
>> values and not the actual task/thread ID.  That decision was made
>> before my heavy involvement with audit, but my guess is that most
>> audit users are focused more on security relevant events at the
>> process level, not the thread level.  After all, there isn't really
>> much in the way of significant boundaries between threads.
>>     
>
> That's right, Paul. The process (exe/comm) is the discriminator from a 
> security perspective.
>   
----
  So, when different threads perform / execute different functionality
as loaded by a runtime loadable libraries, how is that discriminated
from the initially started program?

  Often, programs with many threads will rename the threads so they
show up differently, though some of those may be processes, on linux
there really aren't any threads as being separate from processes -- i.e.
threads, at the linux kernel level are built on processes AFAIK.  Either
way, there can be a separation of what is executed based on what threads
are assigned what purposes.  I'd be hesitant to label the exe/comm as
the only discriminator in an "arbitrary target environment".  Certainly
it can be in some, but that doesn't mean it has to be sole discriminator
when different threads can be mapped to different functions in
1 starting binary.

  In a similar way, coreutils, can be used as 1 library/binary where
functionality is determined by the invoking name.  While coreutils uses
separate names for each function, there's nothing stopping creating
1 binary with all functions launched in separate threads launched out of
  some shell performing diverse functions based on a thread ID or name.
Certainly it isn't the common case, but it would be a way for a hacker
to make their actions more opaque given current limitations.  At the
same time, it might be the way to create some type of 'all-in-one' shell
that could be configured by runtime presence of loadable objects.

  An audit system supporting appending of arbitrary data types could
support appending new data items/types as needed for extension.  Such
was the Irix audit system that was ported to sgi's linux before the
project was cancelled. It had similar benefits to the various layers and
protocols that have been added on top of IPv4 networking, with wrappers
around the low-level IP layer being added as new protocols demanded.

  Just saying, a case can be made for needed additions not originally
planned -- something that is almost always needed in time.




--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Lenny Bruzenak]

On 11/20/20 1:43 PM, L. A. Walsh wrote:

> Repost from right address.
> On 2020/10/08 08:33, Lenny Bruzenak wrote:
>
>> On 10/7/20 7:27 PM, Paul Moore wrote:
>>
>>  
>>> Almost everywhere in the kernel we record the TGID for the "pid="
>>> values and not the actual task/thread ID.  That decision was made
>>> before my heavy involvement with audit, but my guess is that most
>>> audit users are focused more on security relevant events at the
>>> process level, not the thread level.  After all, there isn't really
>>> much in the way of significant boundaries between threads.
>>>     
>>
>> That's right, Paul. The process (exe/comm) is the discriminator from
>> a security perspective.
>>   
> ----
>  So, when different threads perform / execute different functionality
> as loaded by a runtime loadable libraries, how is that discriminated
> from the initially started program?
>
>  Often, programs with many threads will rename the threads so they
> show up differently, though some of those may be processes, on linux
> there really aren't any threads as being separate from processes -- i.e.
> threads, at the linux kernel level are built on processes AFAIK.  Either
> way, there can be a separation of what is executed based on what threads
> are assigned what purposes.  I'd be hesitant to label the exe/comm as
> the only discriminator in an "arbitrary target environment".  Certainly
> it can be in some, but that doesn't mean it has to be sole discriminator
> when different threads can be mapped to different functions in
> 1 starting binary.
>
>  In a similar way, coreutils, can be used as 1 library/binary where
> functionality is determined by the invoking name.  While coreutils uses
> separate names for each function, there's nothing stopping creating
> 1 binary with all functions launched in separate threads launched out of
>  some shell performing diverse functions based on a thread ID or name.
> Certainly it isn't the common case, but it would be a way for a hacker
> to make their actions more opaque given current limitations.  At the
> same time, it might be the way to create some type of 'all-in-one' shell
> that could be configured by runtime presence of loadable objects.
>
>  An audit system supporting appending of arbitrary data types could
> support appending new data items/types as needed for extension.  Such
> was the Irix audit system that was ported to sgi's linux before the
> project was cancelled. It had similar benefits to the various layers and
> protocols that have been added on top of IPv4 networking, with wrappers
> around the low-level IP layer being added as new protocols demanded.
>
>  Just saying, a case can be made for needed additions not originally
> planned -- something that is almost always needed in time.
>
Sorry for the belated reply. Good points.

The main focus (of an audit system) is on being able to prove
accountability. Because threads cannot be initiated directly by a
person, only the parent executable, this is as much as is reasonably
required, currently.

Sure, technically all you say is pretty reasonable, but using your
example of RTLs, extend that to those entities as well. You often don't
have insight into their internals except via the audit trail (syscalls)
- and you know the launched executable.

Threads/libraries/executables don't face consequences, people do. So
until a person (or systemd/cron/...) can launch a thread on its own, it
doesn't make a lot of difference, as it is identifiable to the parent
process, which so far, gives us enough traceability.

The rules need to be such that for a given system security stance, Who
launched Which When causing What is the main focus. Definitely the more
relevant information the better from a forensics perspective. However,
I'd submit that having the thread ID would not lend much to an
investigation. Probably not even in a subsequent attempt to recreate the
issue you would be investigating. But it is possible I'm overlooking
something. As Paul said in the OP, there isn't that much boundary
between threads instantiated from a common parent process, so examining
the process entity as a whole is what would need to happen.

Your point about differently-named executables being the same is a good
one, sure there are lots of executables hard-linked to the same code,
look at /sbin/lvm - I see :

% ls -al /sbin | grep "\-> lvm" | wc -l
43

Doesn't really matter; you still should have the syscall records you
need, surrounded by all the relevant facts (auid, date/time, executable
name, etc., etc.). The thread ID (maybe really even the parent process
ID, except possibly for inclusion with other events, but that's a
stretch) is irrelevant AFAIK.

If the current granularity proves inadequate to achieve the auditing
goals, then IMHO this would need to be revisited, because I agree, as
I'm sure most everyone would, with your final statement - things change.


V/R,

LCB

-- 
Lenny Bruzenak
MagitekLTD

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Identifying thread/process termination

[Natan Yellin]

[-- Attachment #1.1: Type: text/plain, Size: 644 bytes --]

Hello,
I've been tracking all process terminations using a rule for the exit and
exit_group syscalls. However, by looking at the audit events for exit it is
impossible to differentiate between the death of different threads in the
same thread group. Is there an alternative way to track this?

For my use case, I would like to know when either processes or individual
threads execute and terminate. (I'm fine tracking at either granularity.)
Right now I can track the creation properly using fork/clone/etc but for
termination I receive multiple exit events with identical information that
doesn't let me know which thread died.

Thanks,
Natan

[-- Attachment #1.2: Type: text/html, Size: 746 bytes --]

[-- Attachment #2: Type: text/plain, Size: 102 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Steve Grubb]

Hello,

On Monday, October 5, 2020 3:07:12 PM EDT Natan Yellin wrote:
> I've been tracking all process terminations using a rule for the exit and
> exit_group syscalls. However, by looking at the audit events for exit it is
> impossible to differentiate between the death of different threads in the
> same thread group. Is there an alternative way to track this?

I don't think the audit system was ever designed to distinguish between 
threads. But there is a general need to determine the exit of a process 
rather than a thread. 

Paul, Richard, Do you have any thoughts?

-Steve

> For my use case, I would like to know when either processes or individual
> threads execute and terminate. (I'm fine tracking at either granularity.)
> Right now I can track the creation properly using fork/clone/etc but for
> termination I receive multiple exit events with identical information that
> doesn't let me know which thread died.




--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Paul Moore]

On Tue, Oct 6, 2020 at 4:20 PM Steve Grubb <sgrubb@redhat.com> wrote:
> On Monday, October 5, 2020 3:07:12 PM EDT Natan Yellin wrote:
> > I've been tracking all process terminations using a rule for the exit and
> > exit_group syscalls. However, by looking at the audit events for exit it is
> > impossible to differentiate between the death of different threads in the
> > same thread group. Is there an alternative way to track this?
>
> I don't think the audit system was ever designed to distinguish between
> threads. But there is a general need to determine the exit of a process
> rather than a thread.
>
> Paul, Richard, Do you have any thoughts?

Almost everywhere in the kernel we record the TGID for the "pid="
values and not the actual task/thread ID.  That decision was made
before my heavy involvement with audit, but my guess is that most
audit users are focused more on security relevant events at the
process level, not the thread level.  After all, there isn't really
much in the way of significant boundaries between threads.

To get the information you are looking for, I think we would need to
add an additional task/thread ID to the relevant records and that
would be *very* messy.

-- 
paul moore
www.paul-moore.com

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Natan Yellin]

[-- Attachment #1.1: Type: text/plain, Size: 1684 bytes --]

What would be so messy about adding the extra field?

I'm happy to put together a patch myself which adds it to all syscalls and
to process lifecycle events. My goal isn't to identify the exact thread
that performs every audit event but rather to allow tracking thread
lifecycle which isn't currently possible.

Natan

On Thu, Oct 8, 2020, 04:27 Paul Moore <paul@paul-moore.com> wrote:

> On Tue, Oct 6, 2020 at 4:20 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > On Monday, October 5, 2020 3:07:12 PM EDT Natan Yellin wrote:
> > > I've been tracking all process terminations using a rule for the exit
> and
> > > exit_group syscalls. However, by looking at the audit events for exit
> it is
> > > impossible to differentiate between the death of different threads in
> the
> > > same thread group. Is there an alternative way to track this?
> >
> > I don't think the audit system was ever designed to distinguish between
> > threads. But there is a general need to determine the exit of a process
> > rather than a thread.
> >
> > Paul, Richard, Do you have any thoughts?
>
> Almost everywhere in the kernel we record the TGID for the "pid="
> values and not the actual task/thread ID.  That decision was made
> before my heavy involvement with audit, but my guess is that most
> audit users are focused more on security relevant events at the
> process level, not the thread level.  After all, there isn't really
> much in the way of significant boundaries between threads.
>
> To get the information you are looking for, I think we would need to
> add an additional task/thread ID to the relevant records and that
> would be *very* messy.
>
> --
> paul moore
> www.paul-moore.com
>

[-- Attachment #1.2: Type: text/html, Size: 2357 bytes --]

[-- Attachment #2: Type: text/plain, Size: 102 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Paul Moore]

On Thu, Oct 8, 2020 at 4:00 AM Natan Yellin <aantny@gmail.com> wrote:
>
> What would be so messy about adding the extra field?
>
> I'm happy to put together a patch myself which adds it to all syscalls and to process lifecycle events. My goal isn't to identify the exact thread that performs every audit event but rather to allow tracking thread lifecycle which isn't currently possible.

*Please* don't top post, it's a pain to read and it messes up the thread.

As far as recording the thread information, what I meant by messy is
that any new fields added to a record need to be added to the end[1],
which may result in some ugly code.  Of course, if it's important to
you I would encourage you to develop a RFC patch and send it off to
the list for review.  Maybe it won't be so messy after all! :)

[1] It's a really long story, involving a lot of screaming, so just
trust me on this one.  If you really want to challenge this assertion
go read the past seven to eight years of linux-audit archives first ;)

> On Thu, Oct 8, 2020, 04:27 Paul Moore <paul@paul-moore.com> wrote:
>>
>> On Tue, Oct 6, 2020 at 4:20 PM Steve Grubb <sgrubb@redhat.com> wrote:
>> > On Monday, October 5, 2020 3:07:12 PM EDT Natan Yellin wrote:
>> > > I've been tracking all process terminations using a rule for the exit and
>> > > exit_group syscalls. However, by looking at the audit events for exit it is
>> > > impossible to differentiate between the death of different threads in the
>> > > same thread group. Is there an alternative way to track this?
>> >
>> > I don't think the audit system was ever designed to distinguish between
>> > threads. But there is a general need to determine the exit of a process
>> > rather than a thread.
>> >
>> > Paul, Richard, Do you have any thoughts?
>>
>> Almost everywhere in the kernel we record the TGID for the "pid="
>> values and not the actual task/thread ID.  That decision was made
>> before my heavy involvement with audit, but my guess is that most
>> audit users are focused more on security relevant events at the
>> process level, not the thread level.  After all, there isn't really
>> much in the way of significant boundaries between threads.
>>
>> To get the information you are looking for, I think we would need to
>> add an additional task/thread ID to the relevant records and that
>> would be *very* messy.
>>
>> --
>> paul moore
>> www.paul-moore.com



-- 
paul moore
www.paul-moore.com


--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Richard Guy Briggs]

On 2020-10-07 21:27, Paul Moore wrote:
> On Tue, Oct 6, 2020 at 4:20 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > On Monday, October 5, 2020 3:07:12 PM EDT Natan Yellin wrote:
> > > I've been tracking all process terminations using a rule for the exit and
> > > exit_group syscalls. However, by looking at the audit events for exit it is
> > > impossible to differentiate between the death of different threads in the
> > > same thread group. Is there an alternative way to track this?
> >
> > I don't think the audit system was ever designed to distinguish between
> > threads. But there is a general need to determine the exit of a process
> > rather than a thread.
> >
> > Paul, Richard, Do you have any thoughts?
> 
> Almost everywhere in the kernel we record the TGID for the "pid="
> values and not the actual task/thread ID.  That decision was made
> before my heavy involvement with audit, but my guess is that most
> audit users are focused more on security relevant events at the
> process level, not the thread level.  After all, there isn't really
> much in the way of significant boundaries between threads.
> 
> To get the information you are looking for, I think we would need to
> add an additional task/thread ID to the relevant records and that
> would be *very* messy.

I would say that adding a thread ID rather than changing any existing
fields would be the safe way to go, but adds overhead and information to
wade through.

> paul moore

- RGB

--
Richard Guy Briggs <rgb@redhat.com>
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Lenny Bruzenak]

On 10/7/20 7:27 PM, Paul Moore wrote:

> Almost everywhere in the kernel we record the TGID for the "pid="
> values and not the actual task/thread ID.  That decision was made
> before my heavy involvement with audit, but my guess is that most
> audit users are focused more on security relevant events at the
> process level, not the thread level.  After all, there isn't really
> much in the way of significant boundaries between threads.

That's right, Paul. The process (exe/comm) is the discriminator from a 
security perspective.

LCB

-- 
Lenny Bruzenak
MagitekLTD

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[L. A. Walsh]

On 2020/10/08 08:33, Lenny Bruzenak wrote:
> On 10/7/20 7:27 PM, Paul Moore wrote:
>
>   
>> Almost everywhere in the kernel we record the TGID for the "pid="
>> values and not the actual task/thread ID.  That decision was made
>> before my heavy involvement with audit, but my guess is that most
>> audit users are focused more on security relevant events at the
>> process level, not the thread level.  After all, there isn't really
>> much in the way of significant boundaries between threads.
>>     
>
> That's right, Paul. The process (exe/comm) is the discriminator from a 
> security perspective.
>   
----
  So, when different threads perform / execute different functionality
as loaded by a runtime loadable libraries, how is that discriminated
from the initially started program?

  Often, programs with many threads will rename the threads so they
show up differently, though some of those may be processes, on linux
there really aren't any threads as being separate from processes -- i.e.
threads, at the linux kernel level are built on processes AFAIK.  Either
way, there can be a separation of what is executed based on what threads
are assigned what purposes.  I'd be hesitant to label the exe/comm as
the only discriminator in an "arbitrary target environment".  Certainly
it can be in some, but that doesn't mean it has to be sole discriminator
when different threads can be mapped to different functions in
1 starting binary.

  In a similar way, coreutils, can be used as 1 library/binary where
functionality is determined by the invoking name.  While coreutils uses
separate names for each function, there's nothing stopping creating
1 binary with all functions launched in separate threads launched out of
  some shell performing diverse functions based on a thread ID or name.
Certainly it isn't the common case, but it would be a way for a hacker
to make their actions more opaque given current limitations.  At the
same time, it might be the way to create some type of 'all-in-one' shell
that could be configured by runtime presence of loadable objects.

  An audit system supporting appending of arbitrary data types could
support appending new data items/types as needed for extension.  Such
was the Irix audit system that was ported to sgi's linux before the
project was cancelled. It had similar benefits to the various layers and
protocols that have been added on top of IPv4 networking, with wrappers
around the low-level IP layer being added as new protocols demanded.

  Just saying, a case can be made for needed additions not originally
planned -- something that is almost always needed in time.


--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[Paul Moore]

On Mon, Nov 16, 2020 at 8:16 AM L. A. Walsh <lkml@tlinx.org> wrote:
> On 2020/10/08 08:33, Lenny Bruzenak wrote:
> > On 10/7/20 7:27 PM, Paul Moore wrote:
> >> Almost everywhere in the kernel we record the TGID for the "pid="
> >> values and not the actual task/thread ID.  That decision was made
> >> before my heavy involvement with audit, but my guess is that most
> >> audit users are focused more on security relevant events at the
> >> process level, not the thread level.  After all, there isn't really
> >> much in the way of significant boundaries between threads.
> >>
> >
> > That's right, Paul. The process (exe/comm) is the discriminator from a
> > security perspective.
> >
> ----
>   So, when different threads perform / execute different functionality
> as loaded by a runtime loadable libraries, how is that discriminated
> from the initially started program?
>
>   Often, programs with many threads will rename the threads so they
> show up differently, though some of those may be processes, on linux
> there really aren't any threads as being separate from processes -- i.e.
> threads, at the linux kernel level are built on processes AFAIK.  Either
> way, there can be a separation of what is executed based on what threads
> are assigned what purposes.  I'd be hesitant to label the exe/comm as
> the only discriminator in an "arbitrary target environment".  Certainly
> it can be in some, but that doesn't mean it has to be sole discriminator
> when different threads can be mapped to different functions in
> 1 starting binary.

The most important thing to keep in mind is that all of the threads
inside a process share the same memory space.  It is the lack of a
strong, enforceable boundary between threads which makes it difficult,
if not impossible, to view threads as individual entities from a
security perspective.

>   In a similar way, coreutils, can be used as 1 library/binary where
> functionality is determined by the invoking name.  While coreutils uses
> separate names for each function, there's nothing stopping creating
> 1 binary with all functions launched in separate threads launched out of
>   some shell performing diverse functions based on a thread ID or name.
> Certainly it isn't the common case, but it would be a way for a hacker
> to make their actions more opaque given current limitations.  At the
> same time, it might be the way to create some type of 'all-in-one' shell
> that could be configured by runtime presence of loadable objects.

First, and perhaps most importantly, see the earlier comments about
threads and the lack of strong boundaries inside a process.  Second,
the busybox problem (different behavior based on the executable name)
is one of the many reasons why relying on executable names, pathnames,
etc. for identification of entities in a security policy is generally
ill advised.

-- 
paul moore
www.paul-moore.com

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Identifying thread/process termination

[L A Walsh]

On 2020/11/16 05:43, Paul Moore wrote:
> The most important thing to keep in mind is that all of the threads
> inside a process share the same memory space.  It is the lack of a
> strong, enforceable boundary between threads which makes it difficult,
> if not impossible, to view threads as individual entities from a
> security perspective.
---
    Depends on how much your security policy relies on recognizing
abnormal behavior.  If a program splits function across well defined
areas by a named thread, one may develop a baseline of "normal"
functionality associated with given threads.  Determining that
a thread is operating outside it's normal range can allow for a
earlier detection and better monitoring of "safe" and/or secure
operation.

    How programs operate, especially in regards to what work is
normal for a given thread can only be done with thread level
monitoring.  While given threads _can_ access global-user memory,
that involves how they are coded or programmed to run.  That, in
turn, can be used to help define boundaries and integrity levels
of various processes in a system. 

    For example, even though a logging thread might gather data
from other threads, knowing that it can only write to output
to specific configured destinations would allow swift detection
of aberrations.



--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit