Re: Notify special task kill using wait* functions

Re: Notify special task kill using wait* functions

[John Wood]

Hi Valdis,

On Tue, Apr 06, 2021 at 07:55:36PM -0400, Valdis Klētnieks wrote:
> On Mon, 05 Apr 2021 09:31:47 +0200, John Wood said:
>
> > > And how does the kernel know that it's notifying a "real" supervisor process,
> > > and not a process started by the bad guy, who can receive the notification
> > > and decide to respawn?
> > >
> > Well, I think this is not possible to know. Anyway, I believe that the "bad
> > guy" not rely on the wait* notification to decide to respawn or not. He
> > will do the attack without waiting any notification.
>
> You believe wrong. After my 4 decades of interacting with the computer security
> community, the only thing that remains a constant is that if you say "I believe
> that...", there will be *somebody* who will say "Challenge accepted" and try to
> do the opposite just for the lulz. Then there will be a second guy saying "Hmm..
> I wonder how much I could sell a 0-day for..."

Ok, lesson learned. I agree.

> [Great explanation and information]

Wow, I'm impressed. Thank you very much for this great explanation and info.

Thanks a lot for do that (insist about this subject). During the discussion [1]
you made me realize that I'm totally wrong (and you are totally right :) ).
The detection of brute force attacks that happen through the execve system
call can be easily bypassed -> Well, I bypass it during the tests using a
double exec. So, this part needs more work.

[1] https://lore.kernel.org/kernelnewbies/20210330173459.GA3163@ubuntu/

A first thought:

Scenario:
A process [p1] execs. The child [p2] execs again. The child [p3] crashes.

Problem:
The brute LSM kills p3 if it forks and crashes with a fast crash rate (fork
brute force attack). But the p2 process can start again the p3. Then brute
kills p2 (exec brute force attack). Now, if p1 starts p2 the attack can
follow without mitigation.

New proposal:
When brute detects a brute force attack through the fork system call
(killing p3) it will mark the binary file executed by p3 as "not allowed".
From now on, any execve that try to run this binary will fail. This way it
is not necessary to notify nothing to userspace and also we avoid an exec
brute force attack due to the respawn of processes [2] by a supervisor
(abused or not by a bad guy).

[2] https://lore.kernel.org/kernel-hardening/878s78dnrm.fsf@linux.intel.com/

This would imply remove the update (walking up in the processes tree) of
the exec stats and add a list of not allowed binaries.

What do you think? Any ideas are welcome. I'm open minded :)

Regards,
John Wood

Re: Notify special task kill using wait* functions

[Valdis Klētnieks]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

On Wed, 07 Apr 2021 19:51:51 +0200, John Wood said:

> When brute detects a brute force attack through the fork system call
> (killing p3) it will mark the binary file executed by p3 as "not allowed".
> From now on, any execve that try to run this binary will fail. This way it
> is not necessary to notify nothing to userspace and also we avoid an exec
> brute force attack due to the respawn of processes [2] by a supervisor
> (abused or not by a bad guy).

You're not thinking evil enough. :)

I didn't even finish the line that starts "From now on.." before I started
wondering "How can I abuse this to hang or crash a system?"  And it only took
me a few seconds to come up with an attack. All you need to do is find a way to
sigsegv /bin/bash... and that's easy to do by forking, excecve /bin/bash, and
then use ptrace() to screw the child process's stack and cause a sigsegv.

Say goodnight Gracie...

[-- Attachment #2: Type: application/pgp-signature, Size: 832 bytes --]

Re: Notify special task kill using wait* functions

[Andi Kleen]

> I didn't even finish the line that starts "From now on.." before I started
> wondering "How can I abuse this to hang or crash a system?"  And it only took
> me a few seconds to come up with an attack. All you need to do is find a way to
> sigsegv /bin/bash... and that's easy to do by forking, excecve /bin/bash, and
> then use ptrace() to screw the child process's stack and cause a sigsegv.
> 
> Say goodnight Gracie...

Yes there is certainly DoS potential, but that's kind of inevitable
for the proposal. It's a trade between allowing attacks and allowing DoS,
with the idea that a DoS is more benign.

I'm more worried that it doesn't actually prevent the attacks 
unless we make sure systemd and other supervisor daemons understand it,
so that they don't restart.

Any caching of state is inherently insecure because any caches of limited
size can be always thrashed by a purposeful attacker. I suppose the
only thing that would work is to actually write something to the 
executable itself on disk, but of course that doesn't always work either.

-Andi

Re: Notify special task kill using wait* functions

[John Wood]

Hi Valdis and Andi. Thanks for your comments.

On Wed, Apr 07, 2021 at 06:51:48PM -0700, Andi Kleen wrote:
> > I didn't even finish the line that starts "From now on.." before I started
> > wondering "How can I abuse this to hang or crash a system?"  And it only took
> > me a few seconds to come up with an attack. All you need to do is find a way to
> > sigsegv /bin/bash... and that's easy to do by forking, excecve /bin/bash, and
> > then use ptrace() to screw the child process's stack and cause a sigsegv.
> >
> > Say goodnight Gracie...
>
> Yes there is certainly DoS potential, but that's kind of inevitable
> for the proposal. It's a trade between allowing attacks and allowing DoS,
> with the idea that a DoS is more benign.
>
> I'm more worried that it doesn't actually prevent the attacks
> unless we make sure systemd and other supervisor daemons understand it,
> so that they don't restart.

I'm working on it. I will send a formal proposal in the next version.

> Any caching of state is inherently insecure because any caches of limited
> size can be always thrashed by a purposeful attacker. I suppose the
> only thing that would work is to actually write something to the
> executable itself on disk, but of course that doesn't always work either.

I'm also working on this. In the next version I will try to find a way to
prevent brute force attacks through the execve system call with more than
one level of forking.

> -Andi

Again, thank you very much.
John Wood

Re: Notify special task kill using wait* functions

[Andi Kleen]

> > Any caching of state is inherently insecure because any caches of limited
> > size can be always thrashed by a purposeful attacker. I suppose the
> > only thing that would work is to actually write something to the
> > executable itself on disk, but of course that doesn't always work either.
> 
> I'm also working on this. In the next version I will try to find a way to
> prevent brute force attacks through the execve system call with more than
> one level of forking.

Thanks.

Thinking more about it what I wrote above wasn't quite right. The cache
would only need to be as big as the number of attackable services/suid
binaries. Presumably on many production systems that's rather small,
so a cache (which wouldn't actually be a cache, but a complete database)
might actually work.

-Andi

Re: Notify special task kill using wait* functions

[John Wood]

Hi,

On Fri, Apr 09, 2021 at 08:06:21AM -0700, Andi Kleen wrote:
> > > Any caching of state is inherently insecure because any caches of limited
> > > size can be always thrashed by a purposeful attacker. I suppose the
> > > only thing that would work is to actually write something to the
> > > executable itself on disk, but of course that doesn't always work either.
> >
> > I'm also working on this. In the next version I will try to find a way to
> > prevent brute force attacks through the execve system call with more than
> > one level of forking.
>
> Thanks.
>
> Thinking more about it what I wrote above wasn't quite right. The cache
> would only need to be as big as the number of attackable services/suid
> binaries. Presumably on many production systems that's rather small,
> so a cache (which wouldn't actually be a cache, but a complete database)
> might actually work.

Thanks. I will keep it in mind.

>
> -Andi

Regards,
John Wood

Re: Notify special task kill using wait* functions

[Valdis Klētnieks]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Fri, 09 Apr 2021 08:06:21 -0700, Andi Kleen said:

> Thinking more about it what I wrote above wasn't quite right. The cache
> would only need to be as big as the number of attackable services/suid
> binaries. Presumably on many production systems that's rather small,
> so a cache (which wouldn't actually be a cache, but a complete database)
> might actually work.

You also need to consider non-suid things called by suid things that don't
sanitize input sufficiently before invocation...

Thinking about at - is it really a good thing to try to do this in kernelspace?
Or is 'echo 1 > /proc/sys/kernel/print-fatal-signals' and a program to watch
the dmesg and take action more appropriate?  A userspace monitor would
have more options (though a slightly higher risk of race conditions).


[-- Attachment #2: Type: application/pgp-signature, Size: 832 bytes --]

Re: Notify special task kill using wait* functions

[John Wood]

Hi,

On Fri, Apr 09, 2021 at 07:28:20PM -0400, Valdis Klētnieks wrote:
> On Fri, 09 Apr 2021 08:06:21 -0700, Andi Kleen said:
>
> > Thinking more about it what I wrote above wasn't quite right. The cache
> > would only need to be as big as the number of attackable services/suid
> > binaries. Presumably on many production systems that's rather small,
> > so a cache (which wouldn't actually be a cache, but a complete database)
> > might actually work.
>
> You also need to consider non-suid things called by suid things that don't
> sanitize input sufficiently before invocation...
>
> Thinking about at - is it really a good thing to try to do this in kernelspace?
> Or is 'echo 1 > /proc/sys/kernel/print-fatal-signals' and a program to watch
> the dmesg and take action more appropriate?  A userspace monitor would
> have more options (though a slightly higher risk of race conditions).
>

Thanks for the ideas. I need some time to send a formal proposal that
works properly. I would like to get feedback at that moment. I think it
would be better to discuss about the real patch.

Again, thanks.
John Wood