[kernel-hardening] Use-after-free and management of reference counts

[kernel-hardening] Use-after-free and management of reference counts

[Vaishali Thakkar]

Hi,

Recently I studied the reported CVEs of last 2 years and there were
fair number of use-after-free bugs. Usually we also see many reports
of use-after-free bugs in the LKML [reported by one or other tools].

Also, at Kernel recipes Jonathan Corbet  mentioned reference counts as
a security issue. I believe if we have more kernel hardening patches
then we can avoid such bugs. I was wondering if there is some ongoing
work in the both [use-after-free and management of reference counts]
of these areas?

If not, then I would like to work on the same. Any pointers on this
would be appreciated.

Thank you.

-- 
Vaishali

Re: [kernel-hardening] Use-after-free and management of reference counts

[David Windsor]

Hi,

On Tue, Oct 18, 2016 at 8:22 AM, Vaishali Thakkar
<vaishali.thakkar@oracle.com> wrote:
> Hi,
>
> Recently I studied the reported CVEs of last 2 years and there were
> fair number of use-after-free bugs. Usually we also see many reports
> of use-after-free bugs in the LKML [reported by one or other tools].
>
> Also, at Kernel recipes Jonathan Corbet  mentioned reference counts as
> a security issue. I believe if we have more kernel hardening patches
> then we can avoid such bugs. I was wondering if there is some ongoing
> work in the both [use-after-free and management of reference counts]
> of these areas?
>

There is a patchset currently on this mailing list (kernel-hardening)
that directly addresses use-after free bugs.  The feature is called
HARDENED_ATOMIC and is based off of earlier work done by PaX
(https://forums.grsecurity.net/viewtopic.php?f=7&t=4173).

Here is a link to the feature's latest RFC patchset:
http://www.openwall.com/lists/kernel-hardening/2016/10/03/1

Here is the work we've done thus far in documenting the feature:
http://progbits.org

If you're interested in helping with this feature, we're always
looking for people to help move the feature to other architectures.
Someone else is already working on the arm64 port, so you may want to
look at either arm (this is most preferable), mips, sparc or powerpc.
You may want to talk to Takahiro Akashi, who is doing the arm64 work,
to get his thoughts on the arm port.  See this thread for more
information: http://www.openwall.com/lists/kernel-hardening/2016/10/12/2.

> If not, then I would like to work on the same. Any pointers on this
> would be appreciated.
>

Great, thanks for offering to help!  Let me know if you'd like any
other pointers.

> Thank you.
>
> --
> Vaishali

Re: [kernel-hardening] Use-after-free and management of reference counts

[Jann Horn]

[-- Attachment #1: Type: text/plain, Size: 1117 bytes --]

On Tue, Oct 18, 2016 at 05:52:46PM +0530, Vaishali Thakkar wrote:
> Hi,
> 
> Recently I studied the reported CVEs of last 2 years and there were
> fair number of use-after-free bugs. Usually we also see many reports
> of use-after-free bugs in the LKML [reported by one or other tools].
> 
> Also, at Kernel recipes Jonathan Corbet  mentioned reference counts as
> a security issue. I believe if we have more kernel hardening patches
> then we can avoid such bugs. I was wondering if there is some ongoing
> work in the both [use-after-free and management of reference counts]
> of these areas?

Use-after-frees are really hard to deal with. I think I saw some patch
semi-recently for randomizing kernel allocations, to make it harder to
exploit memory safety bugs, but actually fixing use-after-free is a
really hard problem.

For reference counting, refcount overdecrements are pretty much as
hard to deal with as UAFs, but for refcount overincrements, you can
take a look at Elena Reshetova's refcount hardening patch series (see
http://www.openwall.com/lists/kernel-hardening/2016/10/10/1).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [kernel-hardening] Use-after-free and management of reference counts

[Colin Vidal]

Hi Vaishali, David,

> If you're interested in helping with this feature, we're always
> looking for people to help move the feature to other architectures.
> Someone else is already working on the arm64 port, so you may want to
> look at either arm (this is most preferable), mips, sparc or powerpc.

Actually, I've started to work on arm (see a little patch-set I've just
send for review). But as beginning kernel developer, I am glad if
others people can have a look/give help on that stuff :-)

Thanks,

Colin

Re: [kernel-hardening] Use-after-free and management of reference counts

[Kees Cook]

On Tue, Oct 18, 2016 at 5:22 AM, Vaishali Thakkar
<vaishali.thakkar@oracle.com> wrote:
> Hi,
>
> Recently I studied the reported CVEs of last 2 years and there were
> fair number of use-after-free bugs. Usually we also see many reports
> of use-after-free bugs in the LKML [reported by one or other tools].
>
> Also, at Kernel recipes Jonathan Corbet  mentioned reference counts as
> a security issue. I believe if we have more kernel hardening patches
> then we can avoid such bugs. I was wondering if there is some ongoing
> work in the both [use-after-free and management of reference counts]
> of these areas?
>
> If not, then I would like to work on the same. Any pointers on this
> would be appreciated.

Others have already mentioned the HARDENED_ATOMIC work that is
underway, but one place I'd really like to take a little bit more time
to examine is how to improve the zero-poisoning work that Laura Abbott
did recently. Having this feature reduces the availability of a subset
of kernel memory exposures and can frustrate a subset of
use-after-free exploits. It's been in the kernel since v4.6, but it
needs some performance tuning and likely some other adjustments. I
talk about it a bit here:

https://outflux.net/blog/archives/2016/09/30/security-things-in-linux-v4-6/

I think to really dial in this protection so it is more like
PAX_MEMORY_SANITIZE, we'd need to do the following:

- adjust the slab poisoning to do zeroing (upstream's slab poisoning
uses a different value)
- consolidate the CONFIG and run-time cmdlines to give us a single "do
the right thing" CONFIG
- identify frequently-freed slabs that are less security-sensitive and
whitelist them from the poisoning (as done by PAX_MEMORY_SANITIZE)
- add kernel command-line argument to control poisoning (as done by
PAX_MEMORY_SANITIZE)

-Kees

-- 
Kees Cook
Nexus Security

Re: [kernel-hardening] Use-after-free and management of reference counts

[Sandy Harris]

On Tue, Oct 18, 2016 at 9:13 AM, Jann Horn <jann@thejh.net> wrote:

> Use-after-frees are really hard to deal with.

I want to ask a possibly naive questions -- why can't we avoid the
whole class of bugs in the first place? I can think of only two main
sources for use-after-free bugs, both avoidable.

One is allocating something within a function, then returning a
pointer to it. I know I've done that on occasion, got a warning from
lint(1) or the compiler, & fixed it; I have forgotten details. Surely
current static analysis tools can catch nearly all of these. If not,
writing one looks fairly straightforward. Sure, complex code with a
lot of indirection might fool such tools, but in general such code
should not be used anyway.

The other is misusing malloc()/free() or their kernel code analogs, &
the obvious solution is to avoid using those wherever possible. There
are some kernel data structures that need to grow dynamically, but I
do not think there need to be a lot. Code the few very carefully and
refuse to accept patches elsewhere that use dynamic allocation; the
problem may not vanish, but it looks like this reduces it to
manageable size.

I have the feeling I might be missing something here. Pointers welcome.

Re: [kernel-hardening] Use-after-free and management of reference counts

[Jann Horn]

On Tue, Oct 18, 2016 at 06:56:24PM -0400, Sandy Harris wrote:
> On Tue, Oct 18, 2016 at 9:13 AM, Jann Horn <jann@thejh.net> wrote:
> 
> > Use-after-frees are really hard to deal with.
> 
> I want to ask a possibly naive questions -- why can't we avoid the
> whole class of bugs in the first place? I can think of only two main
> sources for use-after-free bugs, both avoidable.
> 
> One is allocating something within a function, then returning a
> pointer to it. I know I've done that on occasion, got a warning from
> lint(1) or the compiler, & fixed it; I have forgotten details. Surely
> current static analysis tools can catch nearly all of these. If not,
> writing one looks fairly straightforward. Sure, complex code with a
> lot of indirection might fool such tools, but in general such code
> should not be used anyway.

I'm pretty sure the kernel does that in a lot of places, actually.


> The other is misusing malloc()/free() or their kernel code analogs, &
> the obvious solution is to avoid using those wherever possible. There
> are some kernel data structures that need to grow dynamically, but I
> do not think there need to be a lot.

Actually, nearly everything in the kernel is allocated dynamically.
Even data that you might normally put on the stack is allocated
dynamically because kernel stacks are so small.
And thanks to namespaces, even things like the hostname can be stored
in dynamically allocated memory.

Re: [kernel-hardening] Use-after-free and management of reference counts

[David Windsor]

On Tue, Oct 18, 2016 at 6:56 PM, Sandy Harris <sandyinchina@gmail.com> wrote:
> On Tue, Oct 18, 2016 at 9:13 AM, Jann Horn <jann@thejh.net> wrote:
>
>> Use-after-frees are really hard to deal with.
>
> I want to ask a possibly naive questions -- why can't we avoid the
> whole class of bugs in the first place? I can think of only two main
> sources for use-after-free bugs, both avoidable.
>
> One is allocating something within a function, then returning a
> pointer to it. I know I've done that on occasion, got a warning from
> lint(1) or the compiler, & fixed it; I have forgotten details. Surely
> current static analysis tools can catch nearly all of these. If not,
> writing one looks fairly straightforward. Sure, complex code with a
> lot of indirection might fool such tools, but in general such code
> should not be used anyway.
>

The main use case in the kernel of reference counters is garbage
collection: the kernel manages a particular object that has multiple
users.  It cannot safely delete this object until all users of the
object have finished using it.   Users acquire and release references
to this object via a get()/put() API.  When all users have called
put(), the reference counter becomes 0 (or -1, or some other
sentinel), and the kernel frees the object.

Things go wrong when the reference counter gets overflowed.  If the
reference counter is represented with a signed integer type,
overflowing the reference counter causes it to go from INT_MAX to
INT_MIN, then approach 0.  Depending on the logic, the transition to
INT_MIN may be enough to trigger the bug, but when the reference
counter becomes 0, the kernel will free the underlying object guarded
by the reference counter while it still has valid users.

> The other is misusing malloc()/free() or their kernel code analogs, &
> the obvious solution is to avoid using those wherever possible. There
> are some kernel data structures that need to grow dynamically, but I
> do not think there need to be a lot. Code the few very carefully and
> refuse to accept patches elsewhere that use dynamic allocation; the
> problem may not vanish, but it looks like this reduces it to
> manageable size.
>
> I have the feeling I might be missing something here. Pointers welcome.

Re: [kernel-hardening] Use-after-free and management of reference counts

[Vaishali Thakkar]

On Wednesday 19 October 2016 05:49 AM, David Windsor wrote:
> On Tue, Oct 18, 2016 at 6:56 PM, Sandy Harris <sandyinchina@gmail.com> wrote:
>> On Tue, Oct 18, 2016 at 9:13 AM, Jann Horn <jann@thejh.net> wrote:
>>
>>> Use-after-frees are really hard to deal with.
>>
>> I want to ask a possibly naive questions -- why can't we avoid the
>> whole class of bugs in the first place? I can think of only two main
>> sources for use-after-free bugs, both avoidable.
>>
>> One is allocating something within a function, then returning a
>> pointer to it. I know I've done that on occasion, got a warning from
>> lint(1) or the compiler, & fixed it; I have forgotten details. Surely
>> current static analysis tools can catch nearly all of these. If not,
>> writing one looks fairly straightforward. Sure, complex code with a
>> lot of indirection might fool such tools, but in general such code
>> should not be used anyway.
>>
> 
> The main use case in the kernel of reference counters is garbage
> collection: the kernel manages a particular object that has multiple
> users.  It cannot safely delete this object until all users of the
> object have finished using it.   Users acquire and release references
> to this object via a get()/put() API.  When all users have called
> put(), the reference counter becomes 0 (or -1, or some other
> sentinel), and the kernel frees the object.
> 
> Things go wrong when the reference counter gets overflowed.  If the
> reference counter is represented with a signed integer type,
> overflowing the reference counter causes it to go from INT_MAX to
> INT_MIN, then approach 0.  Depending on the logic, the transition to
> INT_MIN may be enough to trigger the bug, but when the reference
> counter becomes 0, the kernel will free the underlying object guarded
> by the reference counter while it still has valid users.

One of the other case is when users forget to call put() functions.
I think we have seen many such missing put() cases in the kernel.
Sometimes in some cases it is also possible that users may call put()
little early. 

>> The other is misusing malloc()/free() or their kernel code analogs, &
>> the obvious solution is to avoid using those wherever possible. There
>> are some kernel data structures that need to grow dynamically, but I
>> do not think there need to be a lot. Code the few very carefully and
>> refuse to accept patches elsewhere that use dynamic allocation; the
>> problem may not vanish, but it looks like this reduces it to
>> manageable size.
>>
>> I have the feeling I might be missing something here. Pointers welcome.
> 

-- 
Vaishali

Re: [kernel-hardening] Use-after-free and management of reference counts

[Vaishali Thakkar]

On Wednesday 19 October 2016 03:06 AM, Kees Cook wrote:
> On Tue, Oct 18, 2016 at 5:22 AM, Vaishali Thakkar
> <vaishali.thakkar@oracle.com> wrote:
>> Hi,
>>
>> Recently I studied the reported CVEs of last 2 years and there were
>> fair number of use-after-free bugs. Usually we also see many reports
>> of use-after-free bugs in the LKML [reported by one or other tools].
>>
>> Also, at Kernel recipes Jonathan Corbet  mentioned reference counts as
>> a security issue. I believe if we have more kernel hardening patches
>> then we can avoid such bugs. I was wondering if there is some ongoing
>> work in the both [use-after-free and management of reference counts]
>> of these areas?
>>
>> If not, then I would like to work on the same. Any pointers on this
>> would be appreciated.
> 
> Others have already mentioned the HARDENED_ATOMIC work that is
> underway, but one place I'd really like to take a little bit more time
> to examine is how to improve the zero-poisoning work that Laura Abbott
> did recently. Having this feature reduces the availability of a subset
> of kernel memory exposures and can frustrate a subset of
> use-after-free exploits. It's been in the kernel since v4.6, but it
> needs some performance tuning and likely some other adjustments. I
> talk about it a bit here:

Hi,

Sorry, I missed this as I was not CC'ed. Thanks for the pointers.

> https://outflux.net/blog/archives/2016/09/30/security-things-in-linux-v4-6/
> 
> I think to really dial in this protection so it is more like
> PAX_MEMORY_SANITIZE, we'd need to do the following:
> 
> - adjust the slab poisoning to do zeroing (upstream's slab poisoning
> uses a different value)
> - consolidate the CONFIG and run-time cmdlines to give us a single "do
> the right thing" CONFIG
> - identify frequently-freed slabs that are less security-sensitive and
> whitelist them from the poisoning (as done by PAX_MEMORY_SANITIZE)
> - add kernel command-line argument to control poisoning (as done by
> PAX_MEMORY_SANITIZE)

I am looking in to the work done by Laura and PAX team. I'll get back to you
on this soon.

> -Kees
> 

-- 
Vaishali

Re: [kernel-hardening] Use-after-free and management of reference counts

[Sandy Harris]

On Tue, Oct 18, 2016 at 7:58 PM, Jann Horn <jann@thejh.net> wrote:
> On Tue, Oct 18, 2016 at 06:56:24PM -0400, Sandy Harris wrote:
>> On Tue, Oct 18, 2016 at 9:13 AM, Jann Horn <jann@thejh.net> wrote:
>>
>> > Use-after-frees are really hard to deal with.
>>
>> I want to ask a possibly naive questions -- why can't we avoid the
>> whole class of bugs in the first place? I can think of only two main
>> sources for use-after-free bugs, both avoidable.
>>
>> One is allocating something within a function, then returning a
>> pointer to it. I know I've done that on occasion, got a warning from
>> lint(1) or the compiler, & fixed it; I have forgotten details. Surely
>> current static analysis tools can catch nearly all of these. If not,
>> writing one looks fairly straightforward. Sure, complex code with a
>> lot of indirection might fool such tools, but in general such code
>> should not be used anyway.
>
> I'm pretty sure the kernel does that in a lot of places, actually.

I meant code like this:

char * something(void)
{
      char array[SIZE], *p ;

>
>
>> The other is misusing malloc()/free() or their kernel code analogs, &
>> the obvious solution is to avoid using those wherever possible. There
>> are some kernel data structures that need to grow dynamically, but I
>> do not think there need to be a lot.
>
> Actually, nearly everything in the kernel is allocated dynamically.
> Even data that you might normally put on the stack is allocated
> dynamically because kernel stacks are so small.
> And thanks to namespaces, even things like the hostname can be stored
> in dynamically allocated memory.

Re: [kernel-hardening] Use-after-free and management of reference counts

[Sandy Harris]

On Tue, Oct 18, 2016 at 7:58 PM, Jann Horn <jann@thejh.net> wrote:
> On Tue, Oct 18, 2016 at 06:56:24PM -0400, Sandy Harris wrote:
>> On Tue, Oct 18, 2016 at 9:13 AM, Jann Horn <jann@thejh.net> wrote:
>>
>> > Use-after-frees are really hard to deal with.
>>
>> ...
>>
>> One is allocating something within a function, then returning a
>> pointer to it. I know I've done that on occasion, got a warning from
>> lint(1) or the compiler, & fixed it; I have forgotten details. Surely
>> current static analysis tools can catch nearly all of these. If not,
>> writing one looks fairly straightforward. Sure, complex code with a
>> lot of indirection might fool such tools, but in general such code
>> should not be used anyway.
>
> I'm pretty sure the kernel does that in a lot of places, actually.

I meant code that allocates things on the stack & then returns a
pointer to them. That memory is guaranteed invalid since the
function's stack area is invalid after return. Surely such errors
are rare & can be caught by static analysis. Their frequency
should be near zero in any well-written code and zero once it
has gone through the sort of testing kernel code gets.

I think you meant calling something from the malloc() family
of functions, doing some error checking or initialisation, &
then returning the pointer. That is valid code; it just requires
great care to balance *alloc()s and free()s,