[Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Andy Lutomirski]

Are there useful things to discuss in person about hardening?  (I
don't want to bikeshed about the name at the kernel summit if we can
possibly avoid it.)

Plausible sub-topics include:

 - "USERCOPY" hardening

 - Virtually mapped stacks (I'm hoping to have that in for x86 before
kernel summit...)

 - Refcount

I don't how much of this really needs an in-person meeting, but maybe
some if it would benefit.

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Rafael J. Wysocki]

On Sunday, July 10, 2016 09:28:53 PM Andy Lutomirski wrote:
> Are there useful things to discuss in person about hardening?  (I
> don't want to bikeshed about the name at the kernel summit if we can
> possibly avoid it.)
> 
> Plausible sub-topics include:
> 
>  - "USERCOPY" hardening
> 
>  - Virtually mapped stacks (I'm hoping to have that in for x86 before
> kernel summit...)
> 
>  - Refcount
> 
> I don't how much of this really needs an in-person meeting, but maybe
> some if it would benefit.

I, for one, would like to know in advance if things are in planning that may
interfere with stuff like hibernation or it gets really ugly when breakage
has to be fixed after the fact.

Thanks,
Rafael

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Eric W. Biederman]

Andy Lutomirski <luto@amacapital.net> writes:

> Are there useful things to discuss in person about hardening?  (I
> don't want to bikeshed about the name at the kernel summit if we can
> possibly avoid it.)
>
> Plausible sub-topics include:
>
>  - "USERCOPY" hardening
>
>  - Virtually mapped stacks (I'm hoping to have that in for x86 before
> kernel summit...)
>
>  - Refcount
>
> I don't how much of this really needs an in-person meeting, but maybe
> some if it would benefit.

Given the history of the previous work on which much of this kernel
hardening work is inspired, I think it makes some sense to discuss what
is needed to get various features ready for mainline.

Many kernel hardening features are perceived as having downsides affect
performance or code maintainability.  Which quite frankly is a security
issue of another flavor because if something does not work well, and or
is not maintainable after a while it won't get used.  Rarely are we in
the situation where defense against attack is the most important thing
that people who are deploying linux are worried about.

Eric

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Jann Horn]

[-- Attachment #1: Type: text/plain, Size: 2130 bytes --]

On Sun, Jul 10, 2016 at 09:28:53PM -0700, Andy Lutomirski wrote:
> Are there useful things to discuss in person about hardening? [...]

I think that an interesting question to discuss might be whether, and
if so, how, it makes sense to add restrictions to namespaces.

Namespaces, as a concept, aren't very scary when you keep in mind that
they only grant privileges to otherwise unprivileged users when they
interact with things inside their namespaces. However, in their
implementation, they are somewhat scary because they expose code to
unprivileged users that was written as code only root could reach. As
an example, have a look at NCC Group's netfilter bugs (and netfilter
in general; iirc, the filter parsing code has exponential complexity
without process death checks, which afaik shouldn't happen in any
code normal users can reach).

User namespaces alone are pretty simple. I don't know everything
about mount namespaces, but I think they also don't expose big masses
of kernel code, and IPC, PID and UTS namespaces are pretty simple.

I think that network namespaces, compared to other namespace types,
expose a lot of code. Grepping for CAP_SYS_ADMIN with
`egrep -R '(ns_capable|netlink_net_capable).*CAP_NET_ADMIN'`
returns a bunch of things, including netlink stuff, netfilter,
sysctls, AF_KEY stuff, bridges, routing, socket repair, ARP and
tunnel devices. At the same time, they are one of the lesser-used
namespace types: Containers need them, but sandboxes don't really
need them for much apart from making abstract unix sockets and
networking in general inaccessible.

For this reason, I'm wondering whether it might make sense to create
a new global capability set that specifies which capabilities should
be inherited on namespace entry / namespace access from outside
instead of being set unconditionally, and then let distros and/or
system administrators use that to e.g. restrict CAP_NET_ADMIN.  


But I'm not sure whether I'd want to fly over to the US to attend
the summit, and I'm not sure whether a discussion on this would
benefit from happening in person.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Mon, Jul 11, 2016 at 12:28 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> Are there useful things to discuss in person about hardening?  (I
> don't want to bikeshed about the name at the kernel summit if we can
> possibly avoid it.)
>
> Plausible sub-topics include:
>
>  - "USERCOPY" hardening

The whitelisting stuff might be interesting, but I think it's mostly
about standardizing how architectures define their *copy_*_user()
implementations so that things like KASan and now the hardening
infrastructure can hook it reliably.

>  - Virtually mapped stacks (I'm hoping to have that in for x86 before
> kernel summit...)

Yeah, this should just go in. :) Perhaps a discussion for other
architectures, and the specific requirements (which are mostly well
documented, excepting some notes on stuff like the guard page at
either end, etc).

>  - Refcount

This one needs a solid technical step first; I'd like to avoid
bikeshedding without real code to work from.

> I don't how much of this really needs an in-person meeting, but maybe
> some if it would benefit.

Perhaps some discussion on new/interesting/better gcc plugins, as the
infrastructure and several good examples should have landed by then?

-Kees

-- 
Kees Cook
Chrome OS & Brillo Security

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Mon, Jul 11, 2016 at 12:30 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Andy Lutomirski <luto@amacapital.net> writes:
>
>> Are there useful things to discuss in person about hardening?  (I
>> don't want to bikeshed about the name at the kernel summit if we can
>> possibly avoid it.)
>>
>> Plausible sub-topics include:
>>
>>  - "USERCOPY" hardening
>>
>>  - Virtually mapped stacks (I'm hoping to have that in for x86 before
>> kernel summit...)
>>
>>  - Refcount
>>
>> I don't how much of this really needs an in-person meeting, but maybe
>> some if it would benefit.
>
> Given the history of the previous work on which much of this kernel
> hardening work is inspired, I think it makes some sense to discuss what
> is needed to get various features ready for mainline.

This is a much larger topic, I think. The work being done already by
the Kernel Self Protection Project is highlighting what's needed to
bring various features into mainline. Generally they require a lot of
chopping up into distinct pieces, expanding their portability to other
architectures, more testing, etc.

> Many kernel hardening features are perceived as having downsides affect
> performance or code maintainability.  Which quite frankly is a security
> issue of another flavor because if something does not work well, and or

Sure, but I think this will be done on a case-by-case basis, as each
thing has wildly different aspects. :)

> is not maintainable after a while it won't get used.  Rarely are we in
> the situation where defense against attack is the most important thing
> that people who are deploying linux are worried about.

Yup, of course. This is why I'm trying to make sure that things have
either near-zero impact or are easily configurable.

-Kees


-- 
Kees Cook
Chrome OS & Brillo Security

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Josh Triplett]

On Mon, Jul 11, 2016 at 01:53:42PM -0400, Kees Cook wrote:
> On Mon, Jul 11, 2016 at 12:28 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> > I don't how much of this really needs an in-person meeting, but maybe
> > some if it would benefit.
> 
> Perhaps some discussion on new/interesting/better gcc plugins, as the
> infrastructure and several good examples should have landed by then?

I'd be interested in that as well.  One item for discussion: for some of
the ideas proposed for implementation via GCC plugins, should the code
rely on the plugin to provide functionality at compile time, or should
the plugin identify places in the source that need editing and/or
explicit annotation?  The former provides the possibility of removing
annotations in favor of autodetection, which seems more maintainable;
the latter provides the functionality even without the plugin.

- Josh Triplett

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Mon, Jul 11, 2016 at 2:07 PM, Josh Triplett <josh@joshtriplett.org> wrote:
> On Mon, Jul 11, 2016 at 01:53:42PM -0400, Kees Cook wrote:
>> On Mon, Jul 11, 2016 at 12:28 AM, Andy Lutomirski <luto@amacapital.net> wrote:
>> > I don't how much of this really needs an in-person meeting, but maybe
>> > some if it would benefit.
>>
>> Perhaps some discussion on new/interesting/better gcc plugins, as the
>> infrastructure and several good examples should have landed by then?
>
> I'd be interested in that as well.  One item for discussion: for some of
> the ideas proposed for implementation via GCC plugins, should the code
> rely on the plugin to provide functionality at compile time, or should
> the plugin identify places in the source that need editing and/or
> explicit annotation?  The former provides the possibility of removing
> annotations in favor of autodetection, which seems more maintainable;
> the latter provides the functionality even without the plugin.

If we can get the same results without gcc plugins, we should do that,
since I think it would good to play nice with other compilers.

That said, not all things that the plugins do can be done natively by
gcc even with annotation. So, I think a hybrid approach is probably
best: warn about things that could be changed with annotation, but
make the changes too.

-Kees

-- 
Kees Cook
Chrome OS & Brillo Security

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Eric W. Biederman]

Kees Cook <keescook@chromium.org> writes:

> On Mon, Jul 11, 2016 at 12:30 PM, Eric W. Biederman
> <ebiederm@xmission.com> wrote:
>> Andy Lutomirski <luto@amacapital.net> writes:
>>
>>> Are there useful things to discuss in person about hardening?  (I
>>> don't want to bikeshed about the name at the kernel summit if we can
>>> possibly avoid it.)
>>>
>>> Plausible sub-topics include:
>>>
>>>  - "USERCOPY" hardening
>>>
>>>  - Virtually mapped stacks (I'm hoping to have that in for x86 before
>>> kernel summit...)
>>>
>>>  - Refcount
>>>
>>> I don't how much of this really needs an in-person meeting, but maybe
>>> some if it would benefit.
>>
>> Given the history of the previous work on which much of this kernel
>> hardening work is inspired, I think it makes some sense to discuss what
>> is needed to get various features ready for mainline.
>
> This is a much larger topic, I think. The work being done already by
> the Kernel Self Protection Project is highlighting what's needed to
> bring various features into mainline. Generally they require a lot of
> chopping up into distinct pieces, expanding their portability to other
> architectures, more testing, etc.
>
>> Many kernel hardening features are perceived as having downsides affect
>> performance or code maintainability.  Which quite frankly is a security
>> issue of another flavor because if something does not work well, and or
>
> Sure, but I think this will be done on a case-by-case basis, as each
> thing has wildly different aspects. :)
>
>> is not maintainable after a while it won't get used.  Rarely are we in
>> the situation where defense against attack is the most important thing
>> that people who are deploying linux are worried about.
>
> Yup, of course. This is why I'm trying to make sure that things have
> either near-zero impact or are easily configurable.

Please make certain to leave a big fat comment explaining the security
that is added and why.

I have two brand new patchs that I received today or yesterday sitting
in my inbox to remove security features in the kernel.

A patch to remove the limit on /proc/self/exec only being able to be
changed once.

A patch to muck with kexec_file_load, and get it to accept a new flavor
of unsigned data.  At least I think that is what the patch is.

Eric

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Eric W. Biederman]

Jann Horn <jann@thejh.net> writes:

> On Sun, Jul 10, 2016 at 09:28:53PM -0700, Andy Lutomirski wrote:
>> Are there useful things to discuss in person about hardening? [...]
>
> I think that an interesting question to discuss might be whether, and
> if so, how, it makes sense to add restrictions to namespaces.
>
> Namespaces, as a concept, aren't very scary when you keep in mind that
> they only grant privileges to otherwise unprivileged users when they
> interact with things inside their namespaces. However, in their
> implementation, they are somewhat scary because they expose code to
> unprivileged users that was written as code only root could reach. As
> an example, have a look at NCC Group's netfilter bugs (and netfilter
> in general; iirc, the filter parsing code has exponential complexity
> without process death checks, which afaik shouldn't happen in any
> code normal users can reach).
>
> User namespaces alone are pretty simple. I don't know everything
> about mount namespaces, but I think they also don't expose big masses
> of kernel code, and IPC, PID and UTS namespaces are pretty simple.

Mount namespaces share a lot by default and as such there have been a
lot of hard to resolve semantic difficulties that had to be sorted out.

I am very grateful right now that the issues we are primary issues we
are seeing now are primarily human error.

> I think that network namespaces, compared to other namespace types,
> expose a lot of code. Grepping for CAP_SYS_ADMIN with
> `egrep -R '(ns_capable|netlink_net_capable).*CAP_NET_ADMIN'`
> returns a bunch of things, including netlink stuff, netfilter,
> sysctls, AF_KEY stuff, bridges, routing, socket repair, ARP and
> tunnel devices. At the same time, they are one of the lesser-used
> namespace types: Containers need them, but sandboxes don't really
> need them for much apart from making abstract unix sockets and
> networking in general inaccessible.

Sort of.  A lot of the code is already exposed as the networking stack,
and is exposed from the underside to packets from random strangers from
the internet if not from the control side.

Eric

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Eric W. Biederman]

Andy Lutomirski <luto@amacapital.net> writes:

> On Tue, Jul 19, 2016 at 8:40 AM, Eric W. Biederman
> <ebiederm@xmission.com> wrote:
>> Jann Horn <jann@thejh.net> writes:
>>
>>> On Sun, Jul 10, 2016 at 09:28:53PM -0700, Andy Lutomirski wrote:
>>>> Are there useful things to discuss in person about hardening? [...]
>>>
>>> I think that an interesting question to discuss might be whether, and
>>> if so, how, it makes sense to add restrictions to namespaces.
>>>
>>> Namespaces, as a concept, aren't very scary when you keep in mind that
>>> they only grant privileges to otherwise unprivileged users when they
>>> interact with things inside their namespaces. However, in their
>>> implementation, they are somewhat scary because they expose code to
>>> unprivileged users that was written as code only root could reach. As
>>> an example, have a look at NCC Group's netfilter bugs (and netfilter
>>> in general; iirc, the filter parsing code has exponential complexity
>>> without process death checks, which afaik shouldn't happen in any
>>> code normal users can reach).
>>>
>>> User namespaces alone are pretty simple. I don't know everything
>>> about mount namespaces, but I think they also don't expose big masses
>>> of kernel code, and IPC, PID and UTS namespaces are pretty simple.
>>
>> Mount namespaces share a lot by default and as such there have been a
>> lot of hard to resolve semantic difficulties that had to be sorted out.
>>
>> I am very grateful right now that the issues we are primary issues we
>> are seeing now are primarily human error.
>>
>>> I think that network namespaces, compared to other namespace types,
>>> expose a lot of code. Grepping for CAP_SYS_ADMIN with
>>> `egrep -R '(ns_capable|netlink_net_capable).*CAP_NET_ADMIN'`
>>> returns a bunch of things, including netlink stuff, netfilter,
>>> sysctls, AF_KEY stuff, bridges, routing, socket repair, ARP and
>>> tunnel devices. At the same time, they are one of the lesser-used
>>> namespace types: Containers need them, but sandboxes don't really
>>> need them for much apart from making abstract unix sockets and
>>> networking in general inaccessible.
>>
>> Sort of.  A lot of the code is already exposed as the networking stack,
>> and is exposed from the underside to packets from random strangers from
>> the internet if not from the control side.
>>
>
> At least when that code was written the authors *knew* it was
> security-sensitive.  The control stuff wasn't security sensitive in
> the past.

True.  I tried to review things to make certain they were safe in user
namespaces when I enabled things but clearly a few things slipped
through the cracks.

Eric

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Andy Lutomirski]

On Tue, Jul 19, 2016 at 8:40 AM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Jann Horn <jann@thejh.net> writes:
>
>> On Sun, Jul 10, 2016 at 09:28:53PM -0700, Andy Lutomirski wrote:
>>> Are there useful things to discuss in person about hardening? [...]
>>
>> I think that an interesting question to discuss might be whether, and
>> if so, how, it makes sense to add restrictions to namespaces.
>>
>> Namespaces, as a concept, aren't very scary when you keep in mind that
>> they only grant privileges to otherwise unprivileged users when they
>> interact with things inside their namespaces. However, in their
>> implementation, they are somewhat scary because they expose code to
>> unprivileged users that was written as code only root could reach. As
>> an example, have a look at NCC Group's netfilter bugs (and netfilter
>> in general; iirc, the filter parsing code has exponential complexity
>> without process death checks, which afaik shouldn't happen in any
>> code normal users can reach).
>>
>> User namespaces alone are pretty simple. I don't know everything
>> about mount namespaces, but I think they also don't expose big masses
>> of kernel code, and IPC, PID and UTS namespaces are pretty simple.
>
> Mount namespaces share a lot by default and as such there have been a
> lot of hard to resolve semantic difficulties that had to be sorted out.
>
> I am very grateful right now that the issues we are primary issues we
> are seeing now are primarily human error.
>
>> I think that network namespaces, compared to other namespace types,
>> expose a lot of code. Grepping for CAP_SYS_ADMIN with
>> `egrep -R '(ns_capable|netlink_net_capable).*CAP_NET_ADMIN'`
>> returns a bunch of things, including netlink stuff, netfilter,
>> sysctls, AF_KEY stuff, bridges, routing, socket repair, ARP and
>> tunnel devices. At the same time, they are one of the lesser-used
>> namespace types: Containers need them, but sandboxes don't really
>> need them for much apart from making abstract unix sockets and
>> networking in general inaccessible.
>
> Sort of.  A lot of the code is already exposed as the networking stack,
> and is exposed from the underside to packets from random strangers from
> the internet if not from the control side.
>

At least when that code was written the authors *knew* it was
security-sensitive.  The control stuff wasn't security sensitive in
the past.

--Andy

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Herbert Xu]

On Tue, Jul 19, 2016 at 09:14:03PM -0500, Eric W. Biederman wrote:
> True.  I tried to review things to make certain they were safe in user
> namespaces when I enabled things but clearly a few things slipped
> through the cracks.

What's worse is that after you enable them someone else can come
along and add a new piece of functionality but still assuming
that only root has access to it.  I encountered this with netfilter
and rhashtable where the argument put forward was that as long as
only root had access to a hash table then we don't have to worry
about hash collissions.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Mark Rutland]

On Mon, Jul 11, 2016 at 11:30:13AM -0500, Eric W. Biederman wrote:
> Many kernel hardening features are perceived as having downsides affect
> performance or code maintainability.  Which quite frankly is a security
> issue of another flavor because if something does not work well, and or
> is not maintainable after a while it won't get used. 

In some cases (e.g. copy_{to,from}_user checks), I think that's
indicative of the structure of the existing code, and there are existing
opportunities for cleanup/simplification and/or unification regardless
of particular hardening features.

To some extent, it would be better if we could do more ground work
making those areas more maintainable, rather than focussing solely on
getting particular features in ASAP.

Thanks,
Mark.

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Eric W. Biederman]

Herbert Xu <herbert@gondor.apana.org.au> writes:

> On Tue, Jul 19, 2016 at 09:14:03PM -0500, Eric W. Biederman wrote:
>> True.  I tried to review things to make certain they were safe in user
>> namespaces when I enabled things but clearly a few things slipped
>> through the cracks.
>
> What's worse is that after you enable them someone else can come
> along and add a new piece of functionality but still assuming
> that only root has access to it.  I encountered this with netfilter
> and rhashtable where the argument put forward was that as long as
> only root had access to a hash table then we don't have to worry
> about hash collissions.

Wow.  I missed that hash table discussion.

Yes.  People working on old assumptions is problematic.  Although I have
seen people in code reviews ask the question and what happens if someone
enables your code with user namespaces?  Which seems like a good sign.

Hopefully we can just retire the argument only root can do this, we
don't need to care about code quality.

Eric

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Paul Burton]

On 11/07/16 18:53, Kees Cook wrote:
> On Mon, Jul 11, 2016 at 12:28 AM, Andy Lutomirski <luto@amacapital.net> wrote:
>> Are there useful things to discuss in person about hardening?  (I
>> don't want to bikeshed about the name at the kernel summit if we can
>> possibly avoid it.)
>>
>> Plausible sub-topics include:
>>
>>  - "USERCOPY" hardening
>
> The whitelisting stuff might be interesting, but I think it's mostly
> about standardizing how architectures define their *copy_*_user()
> implementations so that things like KASan and now the hardening
> infrastructure can hook it reliably.
>
>>  - Virtually mapped stacks (I'm hoping to have that in for x86 before
>> kernel summit...)
>
> Yeah, this should just go in. :) Perhaps a discussion for other
> architectures, and the specific requirements (which are mostly well
> documented, excepting some notes on stuff like the guard page at
> either end, etc).

It would be very interesting to discuss what's needed from arch code for 
various hardening features, both those currently in mainline & those in 
development.

For example MIPS systems are currently showing the "This architecture 
does not have kernel memory protection." message since d2aa1acad22f 
(which to a user sounds pretty dire as though user code can freely 
access kernel data) and which I'd like for MIPS to implement the 
security to avoid. However because TLB refills are performed by software 
it's non-trivial, since we generally rely upon the kernel being placed 
in an unmapped region of the virtual address space & being unmapped 
there is no TLB entry to mark read-only.

Usercopy sounds like an interesting one for us too, with the MIPS 
copy_*_user implementations ripe for some cleanup.

It would be interesting to hear from people focused on security what 
their requirements, current & coming soon, from arch code are.

Thanks,
     Paul

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com> wrote:
> On 11/07/16 18:53, Kees Cook wrote:
>>
>> On Mon, Jul 11, 2016 at 12:28 AM, Andy Lutomirski <luto@amacapital.net>
>> wrote:
>>>
>>> Are there useful things to discuss in person about hardening?  (I
>>> don't want to bikeshed about the name at the kernel summit if we can
>>> possibly avoid it.)
>>>
>>> Plausible sub-topics include:
>>>
>>>  - "USERCOPY" hardening
>>
>>
>> The whitelisting stuff might be interesting, but I think it's mostly
>> about standardizing how architectures define their *copy_*_user()
>> implementations so that things like KASan and now the hardening
>> infrastructure can hook it reliably.
>>
>>>  - Virtually mapped stacks (I'm hoping to have that in for x86 before
>>> kernel summit...)
>>
>>
>> Yeah, this should just go in. :) Perhaps a discussion for other
>> architectures, and the specific requirements (which are mostly well
>> documented, excepting some notes on stuff like the guard page at
>> either end, etc).
>
>
> It would be very interesting to discuss what's needed from arch code for
> various hardening features, both those currently in mainline & those in
> development.

Yeah, there are a number of arch-specific things on my radar:

- Standardizing copy_*_user() infrastructure. Each architecture does
their usercopy work in slightly different ways, and hooking it for
things like KASan and hardened usercopy can be weird and error-prone.
What's landing in arm64 is, I think, likely the start of what things
should look like for other architectures: there is a low-level
function (__arch_copy_*_user) that does the actual work of the copy.
Above that needs to be the place to hook KASan and hardened usercopy,
but still within the __copy* and copy* functions. (And x86 has a
single-underscore _copy* set of functions too!) Deciding the ordering
of KASan/hardened-usercopy vs access_ok checks may be worth discussing
too. For example, it's silly to check hardened-usercopy first if
access_ok is going to reject it.

- Cleaning up CONFIG_DEBUG_RODATA. This config should not be called
"debug", and, frankly, it should be mandatory for all architectures.
But (as you mention below) we need to sort out each architecture's
various capabilities in this regard. Some use delayed RO-marking (e.g.
x86, arm), some do it at load time (e.g. s390). As part of this clean
up, making sure the __ro_after_init section is being handled correctly
is needed too (and more people need to start using it). But we've got
to have correct memory protections in kernel memory; this is a
fundamental protection.

- Supporting gcc plugins. Validating each architecture correctly
supports the gcc plugin infrastructure is important since a number of
features may start landing there (already we'll have a latent entropy
plugin that can help especially embedded devices).

- Handling userspace/kernelspace memory segregation. (This is the SMAP
of x86, PAN of ARM, and just native on s390.) For architectures (or
chipsets within an architecture) that don't support unprivileged
memory access restrictions in hardware, we must find a way to emulate
it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
etc.) Keeping these regions separate is extremely important in
stopping exploitation.

- Significant reduction in kernel memory attack surface by marking
rarely-changed function pointers as read-only. We need architectures
to have a way to make (uncommonly changed) function pointers
temporarily writable so that they are read-only (see
CONFIG_DEBUG_RODATA above) during most of their lifetime, thus
removing them as viable attack targets. There is nothing implemented
for this in the kernel yet.

- Stacks without thread_info and with guard pages. Each architecture
needs to keep sensitive values off the kernel stack so that they can't
be targeted via stack-based attacks (e.g. via offsets or exhaustion),
and that faulting pages should live at either end to catch exhaustion
or large writes/reads trying to reach into other stacks. x86 is
starting to work on this now.

- Atomic overflow handling. Each architecture needs to provide a way
to catch atomic_t when it wraps around. This isn't implemented
anywhere in the kernel yet, but it'll need per-architecture support.
Some background on the rationale and the work involved (from the
perspective of this protection in PaX/Grsecurity, which has support
for x86, arm, mips, powerpc, and sparc) can be read here:
https://forums.grsecurity.net/viewtopic.php?f=7&t=4173

- Kernel ASLR: each architecture should (especially 64-bit ones)
separately randomize the base address of the kernel text, kernel
modules, and any large memory areas. Already x86, arm64, and MIPS are
doing base text address and module ASLR, with x86 now landing memory
ASLR too. Getting more architectures doing this would be a nice
addition. (And this will raise the need to provide better kernel
memory address exposure handling, since there are still so many ways
to leak the KASLR offsets. But breaking this catch-22 is important:
we'll never fix exposures if they're not important to fix, and we'll
never implement KASLR if there are so many exposures.)

> For example MIPS systems are currently showing the "This architecture does
> not have kernel memory protection." message since d2aa1acad22f (which to a
> user sounds pretty dire as though user code can freely access kernel data)
> and which I'd like for MIPS to implement the security to avoid. However
> because TLB refills are performed by software it's non-trivial, since we
> generally rely upon the kernel being placed in an unmapped region of the
> virtual address space & being unmapped there is no TLB entry to mark
> read-only.

This is actually from the perspective of the kernel, so when the
kernel code is running, it's mapped, and those entries should all be
marked read-only. As I mentioned above, we need to consider W^X memory
permissions a fundamental security protection, and architectures
should make sure this is completely fixed. (And that is stays that
way: x86 actually scans kernel memory permissions at boot now to make
sure no writable and executable regions have appeared.)

> Usercopy sounds like an interesting one for us too, with the MIPS
> copy_*_user implementations ripe for some cleanup.

Yeah, I'd love to get MIPS added to the hardened usercopy series: I
just hadn't had time to dig in. It _should_ be pretty straight
forward. If following the arm64 example is desired, the changes are
pretty clear in the arm64 patch:
http://git.kernel.org/cgit/linux/kernel/git/kees/linux.git/commit/?h=for-linus/usercopy&id=faf5b63e294151d6ac24ca6906d6f221bd3496cd
If you have time to look it over and send a patch, I'd love to land it. :)

> It would be interesting to hear from people focused on security what their
> requirements, current & coming soon, from arch code are.

I did a bit of a brain-dump into here already:
https://www.kernel.org/doc/Documentation/security/self-protection.txt

For the things that are implemented in the kernel, making sure each
architecture fully supports them would be a good first step. I'd like
to make a little chart of feature vs architecture, but it's a little
hard to compile, since it tends to have a third dimension: chipset.
For example, the PAN/SMAP protection (emulated or in hardware) looks
like this:
http://kernsec.org/wiki/index.php/Exploit_Methods/Userspace_data_usage#Mitigations
so it can be a bit of an eye-chart. :P

There is, of course, more than everything above, but I think these
things will keep us busy for a while. :)

-Kees

-- 
Kees Cook
Chrome OS & Brillo Security

Re: [Ksummit-discuss] [nominations] kernel hardening / self-protection / whatever

[Mark Rutland]

On Sun, Jul 31, 2016 at 10:55:29AM +0100, Paul Burton wrote:
> It would be interesting to hear from people focused on security what
> their requirements, current & coming soon, from arch code are.

I'd be interested in this, and I think it would also be worth having:

- Ard Biesheuvel
- Laura Abbott

Who have both done work on various protection improvements for arm/arm64
(e.g. DEBUG_RODATA & KASLR), and generic infrastructure pieces (e.g.
LKDTM).

Thanks,
Mark.

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Mark Rutland]

On Sun, Jul 31, 2016 at 03:04:58PM -0700, Kees Cook wrote:
> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com> wrote:
> > It would be very interesting to discuss what's needed from arch code for
> > various hardening features, both those currently in mainline & those in
> > development.
> 
> Yeah, there are a number of arch-specific things on my radar:
> 
> - Standardizing copy_*_user() infrastructure. Each architecture does
> their usercopy work in slightly different ways, and hooking it for
> things like KASan and hardened usercopy can be weird and error-prone.
> What's landing in arm64 is, I think, likely the start of what things
> should look like for other architectures: there is a low-level
> function (__arch_copy_*_user) that does the actual work of the copy.
> Above that needs to be the place to hook KASan and hardened usercopy,
> but still within the __copy* and copy* functions. (And x86 has a
> single-underscore _copy* set of functions too!) Deciding the ordering
> of KASan/hardened-usercopy vs access_ok checks may be worth discussing
> too. For example, it's silly to check hardened-usercopy first if
> access_ok is going to reject it.

I think that's basically implied by the copy_* and __copy_* variants,
per the asm-generic version the former simply add an access_ok check
prior to calling the latter.

Modulo figuring out the specifics for x86, perhaps this is just a matter
of proposing patches?

Given arm64 looks to be roughly the right shape already, I'd be happy to
see how much of the arm64 code we can shift out to
<asm-generic/uaccess.h> or <linux/uaccess.h>.

> - Cleaning up CONFIG_DEBUG_RODATA. This config should not be called
> "debug", and, frankly, it should be mandatory for all architectures.

Likewise for DEBUG_SET_MODULE_RONX (which is currently independent of
DEBUG_RODATA). I think we should do the same thing there, and perhaps
fold the options together, or remove the config symbols entirely.

We don't currently have a boot-time option to disable
DEBUG_SET_MODULE_RONX for testing, as we do for DEBUG_RODATA. Perhaps
adding one is the first step to making that default y for all
architectures?

[...]

> - Significant reduction in kernel memory attack surface by marking
> rarely-changed function pointers as read-only. We need architectures
> to have a way to make (uncommonly changed) function pointers
> temporarily writable so that they are read-only (see
> CONFIG_DEBUG_RODATA above) during most of their lifetime, thus
> removing them as viable attack targets. There is nothing implemented
> for this in the kernel yet.

For reference, do you have any specific examples of such pointers? Most
things I can think of are perhaps more suitable for ro_after_init (e.g.
handle_arch_irq), or are embedded in structures with RW fields like
refcounts.

I'll also read "temporarily writeable" as "modifiable through some alias
somehow". For arm/arm64 it's easier/faster/safer to set up a temporary
R/W alias for modification than to modify the active kernel image
mapping.

> - Stacks without thread_info and with guard pages. Each architecture
> needs to keep sensitive values off the kernel stack so that they can't
> be targeted via stack-based attacks (e.g. via offsets or exhaustion),
> and that faulting pages should live at either end to catch exhaustion
> or large writes/reads trying to reach into other stacks. x86 is
> starting to work on this now.

I've also begun looking at this for arm64. It should be possible, though
so far I've found a couple of things that mean we can't do a trivial
port of the x86 approach:

- Unlike x86, we don't have a double-fault vector which can move to a
  new stack. We do have separate "thread" and "handler" stacks, but the
  hardware always moves to the handler stack when an exception is taken.
  So far, ideas of what we can do include:

  * do some early entry work on the handler stack, then migrate to the
    thread stack. This probably involves a memcpy of stashed context, so
    the trivial version is likely to be measurably slower than what we
    do now.

  * always use the handler stack, but detect overflow before stacking
    any context.

    For this, it looks like we need at least a register's worth of
    scratch space, so it's not clear how to do this. We could perhaps
    have per-cpu vectors so as to give us pc-relative addressable
    scratch space.

    I'd initially hoped we could over-align the stack and use TBNZ to
    detect the overflow, but it looks like that accepts the zero
    register rather than the SP.

- Our per-cpu primitives depend on preempt_{disable,enable}(), which
  depend on modifying preempt_count in our thread_info. This means that
  we can't use a per-cpu thread_info pointer like x86 now does.
  
  We might be able to safely access a per-cpu thread_info pointer in our
  entry code to initialise a cached value in sp_el0, though, given we
  don't expect to take any exceptions there (and thus aren't
  preemptible).
  
  I've also been looking at per-cpu primitives that don't need the
  preempt_{disable,enable}() dance, but the approach I've come up with
  so far (reserving a general purpose register) is rather invasive and
  scary.
  
[...]

> For the things that are implemented in the kernel, making sure each
> architecture fully supports them would be a good first step. I'd like
> to make a little chart of feature vs architecture, but it's a little
> hard to compile, since it tends to have a third dimension: chipset.
> For example, the PAN/SMAP protection (emulated or in hardware) looks
> like this:
> http://kernsec.org/wiki/index.php/Exploit_Methods/Userspace_data_usage#Mitigations
> so it can be a bit of an eye-chart. :P

Somewhat an aside, a while back I wanted to clean-up:
http://kernsec.org/wiki/index.php/Feature_List

To be a feature x arch chart, with version+fixups notes in each cell, as
that would help to highlight what was implemented/missing per-arch.

I couldn't see how to register for the wiki to do so. If the above
sounds useful, is there any way I can get an account?

One final thing that I didn't spot on the list was testing. For example,
recent patches to LKDTM were somewhat hindered by the OBJCOPYFLAGS mess.
Having tests work across architectures (and having tests at all!) is
really important to for both development and ongoing regression testing
of features.

As a cross-track thing, it would be great if we could have test projects
like kernelci run security regression tests. We should see if there's
anything we need to do cross-arch to make that happen.

Thanks,
Mark.

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Mon, Aug 1, 2016 at 3:47 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> On Sun, Jul 31, 2016 at 03:04:58PM -0700, Kees Cook wrote:
>> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com> wrote:
>> > It would be very interesting to discuss what's needed from arch code for
>> > various hardening features, both those currently in mainline & those in
>> > development.
>>
>> Yeah, there are a number of arch-specific things on my radar:
>>
>> - Standardizing copy_*_user() infrastructure. Each architecture does
>> their usercopy work in slightly different ways, and hooking it for
>> things like KASan and hardened usercopy can be weird and error-prone.
>> What's landing in arm64 is, I think, likely the start of what things
>> should look like for other architectures: there is a low-level
>> function (__arch_copy_*_user) that does the actual work of the copy.
>> Above that needs to be the place to hook KASan and hardened usercopy,
>> but still within the __copy* and copy* functions. (And x86 has a
>> single-underscore _copy* set of functions too!) Deciding the ordering
>> of KASan/hardened-usercopy vs access_ok checks may be worth discussing
>> too. For example, it's silly to check hardened-usercopy first if
>> access_ok is going to reject it.
>
> I think that's basically implied by the copy_* and __copy_* variants,
> per the asm-generic version the former simply add an access_ok check
> prior to calling the latter.

Well, it's certainly the intended design, but some architectures use
inlines, some use a common function for the copy (with in/out as a
parameter), and x86 uses different paths for __copy* than copy*

> Modulo figuring out the specifics for x86, perhaps this is just a matter
> of proposing patches?

Totally agreed: though there is enough arch-specific nuance in here
that I'd love to see arch maintainers helping (and I only have so much
time...)

> Given arm64 looks to be roughly the right shape already, I'd be happy to
> see how much of the arm64 code we can shift out to
> <asm-generic/uaccess.h> or <linux/uaccess.h>.

Right, I think that'd be the first step.

>> - Cleaning up CONFIG_DEBUG_RODATA. This config should not be called
>> "debug", and, frankly, it should be mandatory for all architectures.
>
> Likewise for DEBUG_SET_MODULE_RONX (which is currently independent of
> DEBUG_RODATA). I think we should do the same thing there, and perhaps
> fold the options together, or remove the config symbols entirely.

Yes, totally agreed.

> We don't currently have a boot-time option to disable
> DEBUG_SET_MODULE_RONX for testing, as we do for DEBUG_RODATA. Perhaps
> adding one is the first step to making that default y for all
> architectures?

Or tie it to the existing rodata flag, then drop the MODULE_RONX
config? In the back of my mind there's some reason that these are
separate configs, but I can't remember now. Maybe Laura remembers?

>> - Significant reduction in kernel memory attack surface by marking
>> rarely-changed function pointers as read-only. We need architectures
>> to have a way to make (uncommonly changed) function pointers
>> temporarily writable so that they are read-only (see
>> CONFIG_DEBUG_RODATA above) during most of their lifetime, thus
>> removing them as viable attack targets. There is nothing implemented
>> for this in the kernel yet.
>
> For reference, do you have any specific examples of such pointers? Most
> things I can think of are perhaps more suitable for ro_after_init (e.g.
> handle_arch_irq), or are embedded in structures with RW fields like
> refcounts.

So, a lot of things that look like they're good for ro_after_init
aren't exactly, like x86 smp_ops, which gets written to during suspend
(or resume?). Or vector, interrupt, and exception tables which need to
be able to change under rare situations. There are also structures
like page tables, etc. And there are function tables that get set up
outside of __init that could be read-only. Doing a quick grep through
the Grsecurity patchset shows pax_open_kernel() calls on things like
struct cdrom_device_ops. (Which could be const, except for the
rewriting of the generic_packet callback.)

Basically, doing an analysis of all things that could be const (which
the (currently out-of-tree) constify plugin does quite a bit of), and
dealing with the weird cases.

> I'll also read "temporarily writeable" as "modifiable through some alias
> somehow". For arm/arm64 it's easier/faster/safer to set up a temporary
> R/W alias for modification than to modify the active kernel image
> mapping.

Sure, though whatever the mechanism, it should be ROP-safe, i.e.
keeping it inline around the write, otherwise it may come to haunt us
as a new exploit path. :)

>> - Stacks without thread_info and with guard pages. Each architecture
>> needs to keep sensitive values off the kernel stack so that they can't
>> be targeted via stack-based attacks (e.g. via offsets or exhaustion),
>> and that faulting pages should live at either end to catch exhaustion
>> or large writes/reads trying to reach into other stacks. x86 is
>> starting to work on this now.
>
> I've also begun looking at this for arm64. It should be possible, though
> so far I've found a couple of things that mean we can't do a trivial
> port of the x86 approach:
>
> - Unlike x86, we don't have a double-fault vector which can move to a
>   new stack. We do have separate "thread" and "handler" stacks, but the
>   hardware always moves to the handler stack when an exception is taken.
>   So far, ideas of what we can do include:
>
>   * do some early entry work on the handler stack, then migrate to the
>     thread stack. This probably involves a memcpy of stashed context, so
>     the trivial version is likely to be measurably slower than what we
>     do now.
>
>   * always use the handler stack, but detect overflow before stacking
>     any context.
>
>     For this, it looks like we need at least a register's worth of
>     scratch space, so it's not clear how to do this. We could perhaps
>     have per-cpu vectors so as to give us pc-relative addressable
>     scratch space.
>
>     I'd initially hoped we could over-align the stack and use TBNZ to
>     detect the overflow, but it looks like that accepts the zero
>     register rather than the SP.
>
> - Our per-cpu primitives depend on preempt_{disable,enable}(), which
>   depend on modifying preempt_count in our thread_info. This means that
>   we can't use a per-cpu thread_info pointer like x86 now does.
>
>   We might be able to safely access a per-cpu thread_info pointer in our
>   entry code to initialise a cached value in sp_el0, though, given we
>   don't expect to take any exceptions there (and thus aren't
>   preemptible).
>
>   I've also been looking at per-cpu primitives that don't need the
>   preempt_{disable,enable}() dance, but the approach I've come up with
>   so far (reserving a general purpose register) is rather invasive and
>   scary.

Heh, yeah, a lot of the hardening changes end up with some really
tight design restrictions, but just getting people thinking about them
at all is a great first step. At the very least, it can help guide
architectures away from digging deeper holes when other things get
designed.

>> For the things that are implemented in the kernel, making sure each
>> architecture fully supports them would be a good first step. I'd like
>> to make a little chart of feature vs architecture, but it's a little
>> hard to compile, since it tends to have a third dimension: chipset.
>> For example, the PAN/SMAP protection (emulated or in hardware) looks
>> like this:
>> http://kernsec.org/wiki/index.php/Exploit_Methods/Userspace_data_usage#Mitigations
>> so it can be a bit of an eye-chart. :P
>
> Somewhat an aside, a while back I wanted to clean-up:
> http://kernsec.org/wiki/index.php/Feature_List
>
> To be a feature x arch chart, with version+fixups notes in each cell, as
> that would help to highlight what was implemented/missing per-arch.
>
> I couldn't see how to register for the wiki to do so. If the above
> sounds useful, is there any way I can get an account?

Yeah, that'd be great. James Morris should be able to set you up an account.

> One final thing that I didn't spot on the list was testing. For example,
> recent patches to LKDTM were somewhat hindered by the OBJCOPYFLAGS mess.
> Having tests work across architectures (and having tests at all!) is
> really important to for both development and ongoing regression testing
> of features.

I thought briefly about including that on my "radar" list above, but
decided my email had gotten long enough already. ;) But yes, testing
is absolutely central to these protections. If we can't test it, we
have no idea if even works in the first place, and that it doesn't
regress later. I've been trying to keep LKDTM up to date with as many
tests for things as possible. (In fact, LKDTM actually has some tests
for things that aren't even in the kernel yet (e.g. the atomic wrap
test and the usercopy whitelisting).) But things slip by once in a
while, but I try to keep them on the KSPP TODO list (e.g. the eBPF JIT
constant blinding test).

Trying to write the tests so they are strictly architecture agnostic
is also important, since I want to test a protection, not an
architecture. :) This is why I went through so many weird hoops to get
the rodata-isn't-executable test done with objcopy: no arch-specific
code.

> As a cross-track thing, it would be great if we could have test projects
> like kernelci run security regression tests. We should see if there's
> anything we need to do cross-arch to make that happen.

I talked with the kernelci folks last year, and they were interested
in using the lkdtm stuff, but it was a bit sideway to how they expect
tests to operate. For lkdtm, testing is inverted: if we Oops, it's
working. And in some tests, the entire system is taken down (e.g.
stack buffer overflow detection), so catching that the device under
test has _correctly_ crashed needs a non-trivial amount of test
framework infrastructure to handle. I'd love it if someone could step
up to take the time to get what's needed into kernelci.

-Kees

-- 
Kees Cook
Chrome OS & Brillo Security

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Catalin Marinas]

On 1 Aug 2016, at 00:05, Kees Cook <keescook@chromium.org> wrote:
> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com> wrote:
>> It would be very interesting to discuss what's needed from arch code for
>> various hardening features, both those currently in mainline & those in
>> development.

I'm interested in such topic as well, primarily from an arm/arm64 perspective. 

> - Handling userspace/kernelspace memory segregation. (This is the SMAP
> of x86, PAN of ARM, and just native on s390.) For architectures (or
> chipsets within an architecture) that don't support unprivileged
> memory access restrictions in hardware, we must find a way to emulate
> it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
> etc.) Keeping these regions separate is extremely important in
> stopping exploitation.

For arm64 ARMv8.0 (without hardware PAN), I'm going to post a patch 
in a week or so which emulates PAN by switching the user page table (TTBR0) 
to the zero page. I guess a similar approach could work for other architectures, 
maybe using swapper_pg_dir as the PAN page table. 

>> For example MIPS systems are currently showing the "This architecture does
>> not have kernel memory protection." message since d2aa1acad22f (which to a
>> user sounds pretty dire as though user code can freely access kernel data)
>> and which I'd like for MIPS to implement the security to avoid. However
>> because TLB refills are performed by software it's non-trivial, since we
>> generally rely upon the kernel being placed in an unmapped region of the
>> virtual address space & being unmapped there is no TLB entry to mark
>> read-only.
> 
> This is actually from the perspective of the kernel, so when the
> kernel code is running, it's mapped, and those entries should all be
> marked read-only. As I mentioned above, we need to consider W^X memory
> permissions a fundamental security protection, and architectures
> should make sure this is completely fixed.

On arm64 we have such feature in hardware: the WXN (write execute never) 
bit in SCTLR_EL1. However, this would affect user space as well, so it needs to 
be switched only when entering the kernel to avoid ABI breakage. I don't have 
the ARM specs at hand but one possible complication is the WXN bit 
being cached in the TLB (and invalidating the TLBs on kernel entry/exit 
is not practical). 

> (And that is stays that
> way: x86 actually scans kernel memory permissions at boot now to make
> sure no writable and executable regions have appeared.)

That's probably a good enough workaround without additional hardware support. 

BTW, while not a kernel security feature, I've been asked in the past to enable 
execute-only (no read) permissions on arm64 (e.g. mmap(PROT_EXEC)). 
I have a simple patch for this, though I'm not 100% sure about user ABI implications. 
So far I'm not aware of any user application using PROT_EXEC only and also 
expecting PROT_READ. 


(Sorry for the email formatting, I'm on holiday and with only a phone as communication device)

-- 
Catalin 

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Wed, Aug 3, 2016 at 3:53 PM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On 1 Aug 2016, at 00:05, Kees Cook <keescook@chromium.org> wrote:
>> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com> wrote:
>>> It would be very interesting to discuss what's needed from arch code for
>>> various hardening features, both those currently in mainline & those in
>>> development.
>
> I'm interested in such topic as well, primarily from an arm/arm64 perspective.
>
>> - Handling userspace/kernelspace memory segregation. (This is the SMAP
>> of x86, PAN of ARM, and just native on s390.) For architectures (or
>> chipsets within an architecture) that don't support unprivileged
>> memory access restrictions in hardware, we must find a way to emulate
>> it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
>> etc.) Keeping these regions separate is extremely important in
>> stopping exploitation.
>
> For arm64 ARMv8.0 (without hardware PAN), I'm going to post a patch
> in a week or so which emulates PAN by switching the user page table (TTBR0)
> to the zero page. I guess a similar approach could work for other architectures,
> maybe using swapper_pg_dir as the PAN page table.

At least on x86 I've heard grumblings that it can be prohibitively
expensive due to TLB-flushing, but I'd still like to see an
implementation doing it first. :)

>>> For example MIPS systems are currently showing the "This architecture does
>>> not have kernel memory protection." message since d2aa1acad22f (which to a
>>> user sounds pretty dire as though user code can freely access kernel data)
>>> and which I'd like for MIPS to implement the security to avoid. However
>>> because TLB refills are performed by software it's non-trivial, since we
>>> generally rely upon the kernel being placed in an unmapped region of the
>>> virtual address space & being unmapped there is no TLB entry to mark
>>> read-only.
>>
>> This is actually from the perspective of the kernel, so when the
>> kernel code is running, it's mapped, and those entries should all be
>> marked read-only. As I mentioned above, we need to consider W^X memory
>> permissions a fundamental security protection, and architectures
>> should make sure this is completely fixed.
>
> On arm64 we have such feature in hardware: the WXN (write execute never)
> bit in SCTLR_EL1. However, this would affect user space as well, so it needs to
> be switched only when entering the kernel to avoid ABI breakage. I don't have
> the ARM specs at hand but one possible complication is the WXN bit
> being cached in the TLB (and invalidating the TLBs on kernel entry/exit
> is not practical).

Oh, that's even more literal W^X than I was expecting. I just meant
having clean page permissions on the kernel text. If we could actually
put WXN to use, that would be very interesting. (Though I guess I
don't see its advantage over just being non-executable?)

>> (And that is stays that
>> way: x86 actually scans kernel memory permissions at boot now to make
>> sure no writable and executable regions have appeared.)
>
> That's probably a good enough workaround without additional hardware support.
>
> BTW, while not a kernel security feature, I've been asked in the past to enable
> execute-only (no read) permissions on arm64 (e.g. mmap(PROT_EXEC)).
> I have a simple patch for this, though I'm not 100% sure about user ABI implications.
> So far I'm not aware of any user application using PROT_EXEC only and also
> expecting PROT_READ.

x86 is working on this too, and IIRC, they uncovered some "fun" ELF
corner cases. I've added Dave for some more background...

-Kees

-- 
Kees Cook
Brillo & Chrome OS Security

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Andy Lutomirski]

On Wed, Aug 3, 2016 at 10:32 PM, Kees Cook <keescook@chromium.org> wrote:
> On Wed, Aug 3, 2016 at 3:53 PM, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> On 1 Aug 2016, at 00:05, Kees Cook <keescook@chromium.org> wrote:
>>> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com> wrote:
>>>> It would be very interesting to discuss what's needed from arch code for
>>>> various hardening features, both those currently in mainline & those in
>>>> development.
>>
>> I'm interested in such topic as well, primarily from an arm/arm64 perspective.
>>
>>> - Handling userspace/kernelspace memory segregation. (This is the SMAP
>>> of x86, PAN of ARM, and just native on s390.) For architectures (or
>>> chipsets within an architecture) that don't support unprivileged
>>> memory access restrictions in hardware, we must find a way to emulate
>>> it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
>>> etc.) Keeping these regions separate is extremely important in
>>> stopping exploitation.
>>
>> For arm64 ARMv8.0 (without hardware PAN), I'm going to post a patch
>> in a week or so which emulates PAN by switching the user page table (TTBR0)
>> to the zero page. I guess a similar approach could work for other architectures,
>> maybe using swapper_pg_dir as the PAN page table.
>
> At least on x86 I've heard grumblings that it can be prohibitively
> expensive due to TLB-flushing, but I'd still like to see an
> implementation doing it first. :)

No TLB flush needed if we use PCID.  Linus will attack you with a
pitchfork either way, though :)

It shouldn't be *that* hard to build this thing on top of my PCID
patchset.  I need to dust that off, which I'll do right after vmap
stacks land and I finish fsgsbase.  Sigh, so many things.

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Wed, Aug 3, 2016 at 10:45 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Wed, Aug 3, 2016 at 10:32 PM, Kees Cook <keescook@chromium.org> wrote:
>> On Wed, Aug 3, 2016 at 3:53 PM, Catalin Marinas <catalin.marinas@arm.com> wrote:
>>> On 1 Aug 2016, at 00:05, Kees Cook <keescook@chromium.org> wrote:
>>>> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com> wrote:
>>>>> It would be very interesting to discuss what's needed from arch code for
>>>>> various hardening features, both those currently in mainline & those in
>>>>> development.
>>>
>>> I'm interested in such topic as well, primarily from an arm/arm64 perspective.
>>>
>>>> - Handling userspace/kernelspace memory segregation. (This is the SMAP
>>>> of x86, PAN of ARM, and just native on s390.) For architectures (or
>>>> chipsets within an architecture) that don't support unprivileged
>>>> memory access restrictions in hardware, we must find a way to emulate
>>>> it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
>>>> etc.) Keeping these regions separate is extremely important in
>>>> stopping exploitation.
>>>
>>> For arm64 ARMv8.0 (without hardware PAN), I'm going to post a patch
>>> in a week or so which emulates PAN by switching the user page table (TTBR0)
>>> to the zero page. I guess a similar approach could work for other architectures,
>>> maybe using swapper_pg_dir as the PAN page table.
>>
>> At least on x86 I've heard grumblings that it can be prohibitively
>> expensive due to TLB-flushing, but I'd still like to see an
>> implementation doing it first. :)
>
> No TLB flush needed if we use PCID.  Linus will attack you with a
> pitchfork either way, though :)

Hmm, my asbestos suit won't help me there. I will need to invest in titanium. :)

The TLB flushing way I can understand being pitchfork-worthy, though
I'm curious why a PCID implementation would be upsetting?

> It shouldn't be *that* hard to build this thing on top of my PCID
> patchset.  I need to dust that off, which I'll do right after vmap
> stacks land and I finish fsgsbase.  Sigh, so many things.

What does your PCID series do?

-Kees

-- 
Kees Cook
Brillo & Chrome OS Security

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Dave Hansen]

On 08/03/2016 10:32 PM, Kees Cook wrote:
>> > BTW, while not a kernel security feature, I've been asked in the past to enable
>> > execute-only (no read) permissions on arm64 (e.g. mmap(PROT_EXEC)).
>> > I have a simple patch for this, though I'm not 100% sure about user ABI implications.
>> > So far I'm not aware of any user application using PROT_EXEC only and also
>> > expecting PROT_READ.
> x86 is working on this too, and IIRC, they uncovered some "fun" ELF
> corner cases. I've added Dave for some more background...

I haven't been able to find anything in the wild that actually uses
PROT_EXEC by itself.  The corner cases I hit were because I took a
PROT_READ|PROT_EXEC mapping and munged it to really be PROT_EXEC only as
an experiment.  It blew up pretty spectacularly because of
non-page-aligned ELF sections creating pages that really do contain
instructions _and_ read-only data.

The exec-only support got in 4.6 and does work under qemu today if
anyone wants to give it a try.

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Catalin Marinas]

On 4 Aug 2016, at 16:18, Dave Hansen <dave.hansen@linux.intel.com> wrote:
> On 08/03/2016 10:32 PM, Kees Cook wrote:
>>>> BTW, while not a kernel security feature, I've been asked in the past to enable
>>>> execute-only (no read) permissions on arm64 (e.g. mmap(PROT_EXEC)).
>>>> I have a simple patch for this, though I'm not 100% sure about user ABI implications.
>>>> So far I'm not aware of any user application using PROT_EXEC only and also
>>>> expecting PROT_READ.
>> x86 is working on this too, and IIRC, they uncovered some "fun" ELF
>> corner cases. I've added Dave for some more background...
> 
> I haven't been able to find anything in the wild that actually uses
> PROT_EXEC by itself.  

I'm not aware of anything in the wild either but there are people 
looking into compiler improvements to allow executable-only ELF sections. 

> The corner cases I hit were because I took a
> PROT_READ|PROT_EXEC mapping and munged it to really be PROT_EXEC only as
> an experiment.  It blew up pretty spectacularly because of
> non-page-aligned ELF sections creating pages that really do contain
> instructions _and_ read-only data.

There is a similar issue on ARM with the literal pool in the executable
section (at least with a standard, unmodified compiler). 

> The exec-only support got in 4.6 and does work under qemu today if
> anyone wants to give it a try.

I plan to push the arm64 equivalent into 4.9 as well on the assumption 
that no existing user code would be broken. Thanks for sharing the x86
findings. 

-- 
Catalin 

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Andy Lutomirski]

[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]

On Aug 3, 2016 10:54 PM, "Kees Cook" <keescook@chromium.org> wrote:
>
> On Wed, Aug 3, 2016 at 10:45 PM, Andy Lutomirski <luto@amacapital.net>
wrote:
> > On Wed, Aug 3, 2016 at 10:32 PM, Kees Cook <keescook@chromium.org>
wrote:
> >> On Wed, Aug 3, 2016 at 3:53 PM, Catalin Marinas <
catalin.marinas@arm.com> wrote:
> >>> On 1 Aug 2016, at 00:05, Kees Cook <keescook@chromium.org> wrote:
> >>>> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com>
wrote:
> >>>>> It would be very interesting to discuss what's needed from arch
code for
> >>>>> various hardening features, both those currently in mainline &
those in
> >>>>> development.
> >>>
> >>> I'm interested in such topic as well, primarily from an arm/arm64
perspective.
> >>>
> >>>> - Handling userspace/kernelspace memory segregation. (This is the
SMAP
> >>>> of x86, PAN of ARM, and just native on s390.) For architectures (or
> >>>> chipsets within an architecture) that don't support unprivileged
> >>>> memory access restrictions in hardware, we must find a way to emulate
> >>>> it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
> >>>> etc.) Keeping these regions separate is extremely important in
> >>>> stopping exploitation.
> >>>
> >>> For arm64 ARMv8.0 (without hardware PAN), I'm going to post a patch
> >>> in a week or so which emulates PAN by switching the user page table
(TTBR0)
> >>> to the zero page. I guess a similar approach could work for other
architectures,
> >>> maybe using swapper_pg_dir as the PAN page table.
> >>
> >> At least on x86 I've heard grumblings that it can be prohibitively
> >> expensive due to TLB-flushing, but I'd still like to see an
> >> implementation doing it first. :)
> >
> > No TLB flush needed if we use PCID.  Linus will attack you with a
> > pitchfork either way, though :)
>
> Hmm, my asbestos suit won't help me there. I will need to invest in
titanium. :)
>
> The TLB flushing way I can understand being pitchfork-worthy, though
> I'm curious why a PCID implementation would be upsetting?

I think the problem is the page table switch, not the PCID.

>
> > It shouldn't be *that* hard to build this thing on top of my PCID
> > patchset.  I need to dust that off, which I'll do right after vmap
> > stacks land and I finish fsgsbase.  Sigh, so many things.
>
> What does your PCID series do?

It uses PCID for what it looks like it was designed for: faster context
switches.

AFAICT it was actually designed for user/kernel switching on Mac OS X,
which, if true, would explain some design oddities.

[-- Attachment #2: Type: text/html, Size: 3590 bytes --]

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Kees Cook]

On Thu, Aug 4, 2016 at 5:12 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Aug 3, 2016 10:54 PM, "Kees Cook" <keescook@chromium.org> wrote:
>>
>> On Wed, Aug 3, 2016 at 10:45 PM, Andy Lutomirski <luto@amacapital.net>
>> wrote:
>> > On Wed, Aug 3, 2016 at 10:32 PM, Kees Cook <keescook@chromium.org>
>> > wrote:
>> >> On Wed, Aug 3, 2016 at 3:53 PM, Catalin Marinas
>> >> <catalin.marinas@arm.com> wrote:
>> >>> On 1 Aug 2016, at 00:05, Kees Cook <keescook@chromium.org> wrote:
>> >>>> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com>
>> >>>> wrote:
>> >>>>> It would be very interesting to discuss what's needed from arch code
>> >>>>> for
>> >>>>> various hardening features, both those currently in mainline & those
>> >>>>> in
>> >>>>> development.
>> >>>
>> >>> I'm interested in such topic as well, primarily from an arm/arm64
>> >>> perspective.
>> >>>
>> >>>> - Handling userspace/kernelspace memory segregation. (This is the
>> >>>> SMAP
>> >>>> of x86, PAN of ARM, and just native on s390.) For architectures (or
>> >>>> chipsets within an architecture) that don't support unprivileged
>> >>>> memory access restrictions in hardware, we must find a way to emulate
>> >>>> it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
>> >>>> etc.) Keeping these regions separate is extremely important in
>> >>>> stopping exploitation.
>> >>>
>> >>> For arm64 ARMv8.0 (without hardware PAN), I'm going to post a patch
>> >>> in a week or so which emulates PAN by switching the user page table
>> >>> (TTBR0)
>> >>> to the zero page. I guess a similar approach could work for other
>> >>> architectures,
>> >>> maybe using swapper_pg_dir as the PAN page table.
>> >>
>> >> At least on x86 I've heard grumblings that it can be prohibitively
>> >> expensive due to TLB-flushing, but I'd still like to see an
>> >> implementation doing it first. :)
>> >
>> > No TLB flush needed if we use PCID.  Linus will attack you with a
>> > pitchfork either way, though :)
>>
>> Hmm, my asbestos suit won't help me there. I will need to invest in
>> titanium. :)
>>
>> The TLB flushing way I can understand being pitchfork-worthy, though
>> I'm curious why a PCID implementation would be upsetting?
>
> I think the problem is the page table switch, not the PCID.
>
>>
>> > It shouldn't be *that* hard to build this thing on top of my PCID
>> > patchset.  I need to dust that off, which I'll do right after vmap
>> > stacks land and I finish fsgsbase.  Sigh, so many things.

What's the state of the vmap stacks? Are they just waiting for 4.9 to
open? I haven't seen any further discussion...

>> What does your PCID series do?
>
> It uses PCID for what it looks like it was designed for: faster context
> switches.
>
> AFAICT it was actually designed for user/kernel switching on Mac OS X,
> which, if true, would explain some design oddities.

Yeah, if it looks anything like the arm64 emulation for swapping page
tables in and out, it should work very nicely.

-Kees

-- 
Kees Cook
Nexus Security

Re: [Ksummit-discuss] [TOPIC] kernel hardening / self-protection / whatever

[Andy Lutomirski]

On Thu, Sep 8, 2016 at 4:54 PM, Kees Cook <keescook@chromium.org> wrote:
> On Thu, Aug 4, 2016 at 5:12 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>> On Aug 3, 2016 10:54 PM, "Kees Cook" <keescook@chromium.org> wrote:
>>>
>>> On Wed, Aug 3, 2016 at 10:45 PM, Andy Lutomirski <luto@amacapital.net>
>>> wrote:
>>> > On Wed, Aug 3, 2016 at 10:32 PM, Kees Cook <keescook@chromium.org>
>>> > wrote:
>>> >> On Wed, Aug 3, 2016 at 3:53 PM, Catalin Marinas
>>> >> <catalin.marinas@arm.com> wrote:
>>> >>> On 1 Aug 2016, at 00:05, Kees Cook <keescook@chromium.org> wrote:
>>> >>>> On Sun, Jul 31, 2016 at 2:55 AM, Paul Burton <paul.burton@imgtec.com>
>>> >>>> wrote:
>>> >>>>> It would be very interesting to discuss what's needed from arch code
>>> >>>>> for
>>> >>>>> various hardening features, both those currently in mainline & those
>>> >>>>> in
>>> >>>>> development.
>>> >>>
>>> >>> I'm interested in such topic as well, primarily from an arm/arm64
>>> >>> perspective.
>>> >>>
>>> >>>> - Handling userspace/kernelspace memory segregation. (This is the
>>> >>>> SMAP
>>> >>>> of x86, PAN of ARM, and just native on s390.) For architectures (or
>>> >>>> chipsets within an architecture) that don't support unprivileged
>>> >>>> memory access restrictions in hardware, we must find a way to emulate
>>> >>>> it. (e.g. 32-bit ARM uses Domains, and 64-bit x86 could use PCIDs,
>>> >>>> etc.) Keeping these regions separate is extremely important in
>>> >>>> stopping exploitation.
>>> >>>
>>> >>> For arm64 ARMv8.0 (without hardware PAN), I'm going to post a patch
>>> >>> in a week or so which emulates PAN by switching the user page table
>>> >>> (TTBR0)
>>> >>> to the zero page. I guess a similar approach could work for other
>>> >>> architectures,
>>> >>> maybe using swapper_pg_dir as the PAN page table.
>>> >>
>>> >> At least on x86 I've heard grumblings that it can be prohibitively
>>> >> expensive due to TLB-flushing, but I'd still like to see an
>>> >> implementation doing it first. :)
>>> >
>>> > No TLB flush needed if we use PCID.  Linus will attack you with a
>>> > pitchfork either way, though :)
>>>
>>> Hmm, my asbestos suit won't help me there. I will need to invest in
>>> titanium. :)
>>>
>>> The TLB flushing way I can understand being pitchfork-worthy, though
>>> I'm curious why a PCID implementation would be upsetting?
>>
>> I think the problem is the page table switch, not the PCID.
>>
>>>
>>> > It shouldn't be *that* hard to build this thing on top of my PCID
>>> > patchset.  I need to dust that off, which I'll do right after vmap
>>> > stacks land and I finish fsgsbase.  Sigh, so many things.
>
> What's the state of the vmap stacks? Are they just waiting for 4.9 to
> open? I haven't seen any further discussion...

Vmapped stacks themselves are in tip:x86/asm.  The thread_info cleanup
isn't yet, though.  I'm guessing that Ingo is still wrangling the
patches into his queue.