[kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Daniel Cegiełka]

https://grsecurity.net/passing_the_baton_faq.php

Best regards,
Daniel

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

On Wed, Apr 26, 2017 at 2:05 PM, Daniel Cegiełka
<daniel.cegielka@gmail.com> wrote:
> https://grsecurity.net/passing_the_baton_faq.php

Yeah, I'm sad to see them go. PaX Team was just last night helping
with some details of PAX_REFCOUNT as it could appear in the upstream
refcount_t API. I hope they'll still help out from time to time.

It does underscore the critical need to upstream stuff, though. Forks
of projects might disappear at any time. :(

-Kees

-- 
Kees Cook
Pixel Security

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Mathias Krause]

On 27 April 2017 at 00:04, Kees Cook <keescook@chromium.org> wrote:
> On Wed, Apr 26, 2017 at 2:05 PM, Daniel Cegiełka
> <daniel.cegielka@gmail.com> wrote:
>> https://grsecurity.net/passing_the_baton_faq.php
>
> Yeah, I'm sad to see them go. PaX Team was just last night helping
> with some details of PAX_REFCOUNT as it could appear in the upstream
> refcount_t API. I hope they'll still help out from time to time.
>
> It does underscore the critical need to upstream stuff, though. Forks
> of projects might disappear at any time. :(

Now, since grsecurity and PaX went private, quite a few users
(including me) are left in the dark and are, to say the least,
slightly pissed. But not everybody seems to be barking at the right
tree. So I've a few comments and questions for the KSPP.

I think the main reason for Brad and PaX Team to make their work
private is the increased amount of work KSSP has put on them without
providing any valuable work in return. They just don't want to be
forced to maintain and fix-up the variants of grsecurity/PaX features
KSPP lands upstream. And no, the work of KSPP does not make their life
easier, in fact, it makes it harder. Harder for two reasons: First,
the code does not end up verbatim, it's always changed, taken out of
context and "enhanced". Second is the loose of control over the code.
But let me elaborate those two a little further.

The first point directly forces them to think through the upstream
incarnation of a feature, how it differs from their original one and,
for those parts, how to fix them up to work properly to fit their
needs, to fit their security requirements. Those changes generate a
lot of conflicts with their version of the code which takes time to
fiddle out. As happened for __ro_after_init, MEMORY_SANITIZE,
USERCOPY,... which only went in in reduced and modified form,
generating needless work on their side -- from their point of view.

The second point, the loose of control over the code, is even worse.
Not getting any more conflicts when porting the grsecurity/PaX patch
to a new kernel release makes changes to code that used to live in
grsecurity/PaX probably go unnoticed. And, as it seem to be the case,
upstream developers are not always familiar with all the gory details
and might introduce weaknesses and bugs. This not only makes upstream
Linux less secure, it makes grsecurity and PaX less secure, too.

So I can understand why they've done this. Still, with the loose of
the public availability of the patch, Linux security has suffered a
lot. Not only is upstream far far away to reach a level that is
available today in grsecurity and PaX, no, it also won't benefit from
new developments any more. What a great achievement! :(

*sigh*

I think the intention of the KSPP is good -- making vanilla Linux more
secure. But the way it does its work harms overall Linux security. It
does hurt mine, that's for sure! I know the value of grsecurity and
PaX in particular and know how easy it still is to exploit a vanilla
Linux. Features like KERNEXEC and RAP make it almost impossible for an
attacker to abuse a memory corruption bug in the kernel. Many unnamed
features -- unnamed because not under an #ifdef -- make exploiting
use-after-free bugs much less interesting or reduce the race window
for TOCTTOU bugs involving user copy operations. None of this can be
found in vanilla Linux. Probably never will...

So, here's my list of questions for the KSPP:
1/ When will I be able to switch to a vanilla Linux kernel that is
equivalently hardened as a grsecurity/PaX kernel used to be?
2/ Who will maintain this code and how?
3/ Who ensures the coverage and quality won't suffer for each new
kernel release?

Judging from the planed EOL for the v4.9 LTS kernel -- the last one a
grsecurity patch was publicly available for --, there are less than
two years to finish the work for item 1 to ensure a secure and smooth
transition from grsecurity to upstream Linux. Will the KSPP be able to
achieve this?... I have my doubts.

Even if so, item 3 will be a hard problem to solve as the Linux kernel
development model does not support such a tree-wide review point in
the release cycle. Assuming there are people able and willing to judge
and review the code base for required changes (e.g. atomic_t ->
refcount_t changes or function pointer cleanups for RAP), how would
those be able to simply get in those changes at, say, time of rc5? --
without having to argue with a horde of maintainers of the individual
subsystems involved?

Quite a lot of questions. The external patch solved them all by not
having to deal with upstream Linux development and not having the code
available until it's ready. But now it's gone and no adequate
replacement is on the horizon. What will KSPP do about it? I had a
reasonably secure kernel, now it's gone. :(


Regards,
Mathias

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Rik van Riel]

On Tue, 2017-05-02 at 00:01 +0200, Mathias Krause wrote:

> I think the intention of the KSPP is good -- making vanilla Linux
> more
> secure. But the way it does its work harms overall Linux security. It
> does hurt mine, that's for sure!

Yeah, no. The grsecurity people produced patches
that were used on maybe a few tens of thousands
of systems, while the KSPP code will end up
enhancing the security of over a billion Android
devices.

Those Android devices are more likely to require
hardening, too, since they do not receive security
updates as quickly as the systems maintained by
grsecurity users.

Integrating hardening into the upstream kernel is
a good thing for security, not a bad thing.

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Olof Johansson]

On Mon, May 1, 2017 at 3:01 PM, Mathias Krause <minipli@googlemail.com> wrote:

> The first point directly forces them to think through the upstream
> incarnation of a feature, how it differs from their original one and,
> for those parts, how to fix them up to work properly to fit their
> needs, to fit their security requirements. Those changes generate a
> lot of conflicts with their version of the code which takes time to
> fiddle out. As happened for __ro_after_init, MEMORY_SANITIZE,
> USERCOPY,... which only went in in reduced and modified form,
> generating needless work on their side -- from their point of view.

This is the classic pain point of initial upstreaming when you have
been out-of-tree for a long time. We see this all the time for
hardware support as well.

It's a hump to climb, and once you're over it, things settle down
again. The quicker you can get past it, the better off you are.

> The second point, the loose of control over the code, is even worse.
> Not getting any more conflicts when porting the grsecurity/PaX patch
> to a new kernel release makes changes to code that used to live in
> grsecurity/PaX probably go unnoticed. And, as it seem to be the case,
> upstream developers are not always familiar with all the gory details
> and might introduce weaknesses and bugs. This not only makes upstream
> Linux less secure, it makes grsecurity and PaX less secure, too.

The solution to this is to get involved and review code and educate
your peers when they're doing the wrong thing. Linux is developed and
managed based on influence, not control. Influence requires
participation and interaction.


-Olof

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Casey Schaufler]

On 5/1/2017 3:01 PM, Mathias Krause wrote:
> On 27 April 2017 at 00:04, Kees Cook <keescook@chromium.org> wrote:
>> On Wed, Apr 26, 2017 at 2:05 PM, Daniel Cegiełka
>> <daniel.cegielka@gmail.com> wrote:
>>> https://grsecurity.net/passing_the_baton_faq.php
>> Yeah, I'm sad to see them go. PaX Team was just last night helping
>> with some details of PAX_REFCOUNT as it could appear in the upstream
>> refcount_t API. I hope they'll still help out from time to time.
>>
>> It does underscore the critical need to upstream stuff, though. Forks
>> of projects might disappear at any time. :(
> Now, since grsecurity and PaX went private, quite a few users
> (including me) are left in the dark and are, to say the least,
> slightly pissed. But not everybody seems to be barking at the right
> tree. So I've a few comments and questions for the KSPP.
>
> I think the main reason for Brad and PaX Team to make their work
> private is the increased amount of work KSSP has put on them without
> providing any valuable work in return. They just don't want to be
> forced to maintain and fix-up the variants of grsecurity/PaX features
> KSPP lands upstream. And no, the work of KSPP does not make their life
> easier, in fact, it makes it harder. Harder for two reasons: First,
> the code does not end up verbatim, it's always changed, taken out of
> context and "enhanced". Second is the loose of control over the code.
> But let me elaborate those two a little further.

I can't speak for Brad and the PaX team, and I certainly
can't claim to understand their motivation for abandoning
the community they had built. I can say that maintaining a
fork of the kernel for any purpose is a lot of work, and
keeping a patch set as large and complex as grsecurity up
to date is no mean feat. This would be true whether KSPP
were active or not. As anyone who does kernel development
outside of an isolated area can tell you, there's always
someone messing up everything.

There's good reason that code hasn't often been taken from
grsecurity verbatim. Not the least of these is that teasing
a particular "feature" out of the public patches is not easy.
Another is that maintainers of other aspects of the system
often dislike the affect the grsecurity code has on the
aspect near and dear to their hearts, and won't accept the
"value of security" argument. KSPP is committed to upstream.
KSPP has to make the trade-offs grsecurity was never motivated
to.

> The first point directly forces them to think through the upstream
> incarnation of a feature, how it differs from their original one and,
> for those parts, how to fix them up to work properly to fit their
> needs, to fit their security requirements. Those changes generate a
> lot of conflicts with their version of the code which takes time to
> fiddle out. As happened for __ro_after_init, MEMORY_SANITIZE,
> USERCOPY,... which only went in in reduced and modified form,
> generating needless work on their side -- from their point of view.
>
> The second point, the loose of control over the code, is even worse.
> Not getting any more conflicts when porting the grsecurity/PaX patch
> to a new kernel release makes changes to code that used to live in
> grsecurity/PaX probably go unnoticed. And, as it seem to be the case,
> upstream developers are not always familiar with all the gory details
> and might introduce weaknesses and bugs. This not only makes upstream
> Linux less secure, it makes grsecurity and PaX less secure, too.
>
> So I can understand why they've done this. Still, with the loose of
> the public availability of the patch, Linux security has suffered a
> lot. Not only is upstream far far away to reach a level that is
> available today in grsecurity and PaX, no, it also won't benefit from
> new developments any more. What a great achievement! :(

You are confusing "Linux security" with "grsecurity (tm) security".
grsecurity is a commercial product. Brad Spender has made it very
clear that this is how he wants it viewed (Brad, feel free to correct
me) and that is his call. Linux is a community project. As such, Linux
has never before had the facilities KSPP is putting in now. Similar
features have been available in a commercial fork, and thanks to the
terms of the GPLv2, we have access to that code.

> *sigh*
>
> I think the intention of the KSPP is good -- making vanilla Linux more
> secure. But the way it does its work harms overall Linux security.

No. Again, you are confusing a commercial product based on Linux
with Linux.

>  It
> does hurt mine, that's for sure! I know the value of grsecurity and
> PaX in particular and know how easy it still is to exploit a vanilla
> Linux. Features like KERNEXEC and RAP make it almost impossible for an
> attacker to abuse a memory corruption bug in the kernel. Many unnamed
> features -- unnamed because not under an #ifdef -- make exploiting
> use-after-free bugs much less interesting or reduce the race window
> for TOCTTOU bugs involving user copy operations. None of this can be
> found in vanilla Linux. Probably never will...

Patches welcome!

> So, here's my list of questions for the KSPP:
> 1/ When will I be able to switch to a vanilla Linux kernel that is
> equivalently hardened as a grsecurity/PaX kernel used to be?

Can't answer that. I spent months trying to unravel USERCOPY
from the rest of the patch and can safely say that there's so
much going on in grsecurity that I would never believe someone
who told me they could identify all the features.

> 2/ Who will maintain this code and how?

Most often the authors of code will maintain it, but we've
already seen cases where architecture specialists have begun
treating KSPP code as their own. How? The usual community
spirit of keeping things working.

> 3/ Who ensures the coverage and quality won't suffer for each new
> kernel release?

Do you understand how community works?

>
> Judging from the planed EOL for the v4.9 LTS kernel -- the last one a
> grsecurity patch was publicly available for --, there are less than
> two years to finish the work for item 1 to ensure a secure and smooth
> transition from grsecurity to upstream Linux. Will the KSPP be able to
> achieve this?... I have my doubts.

Grsecurity was a fork, and part of the reason for that is that
parts of what is in it where not acceptable to other members of
the community. Support of what is in a fork when the fork goes
away is one reason not to rely on forks. 

> Even if so, item 3 will be a hard problem to solve as the Linux kernel
> development model does not support such a tree-wide review point in
> the release cycle. Assuming there are people able and willing to judge
> and review the code base for required changes (e.g. atomic_t ->
> refcount_t changes or function pointer cleanups for RAP), how would
> those be able to simply get in those changes at, say, time of rc5? --
> without having to argue with a horde of maintainers of the individual
> subsystems involved?

You *always* have to argue with a horde of maintainers!
That's what we do! Drop in the refcount_t change without
getting buy-in from Al and David? I think not.

> Quite a lot of questions. The external patch solved them all by not
> having to deal with upstream Linux development and not having the code
> available until it's ready. But now it's gone and no adequate
> replacement is on the horizon. What will KSPP do about it? I had a
> reasonably secure kernel, now it's gone. :(

Pay Mr. Spender.
Contribute patches to KSPP.

>
>
> Regards,
> Mathias
>

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

On Mon, May 1, 2017 at 3:01 PM, Mathias Krause <minipli@googlemail.com> wrote:
> On 27 April 2017 at 00:04, Kees Cook <keescook@chromium.org> wrote:
>> On Wed, Apr 26, 2017 at 2:05 PM, Daniel Cegiełka
>> <daniel.cegielka@gmail.com> wrote:
>>> https://grsecurity.net/passing_the_baton_faq.php
>>
>> Yeah, I'm sad to see them go. PaX Team was just last night helping
>> with some details of PAX_REFCOUNT as it could appear in the upstream
>> refcount_t API. I hope they'll still help out from time to time.
>>
>> It does underscore the critical need to upstream stuff, though. Forks
>> of projects might disappear at any time. :(
>
> Now, since grsecurity and PaX went private, quite a few users
> (including me) are left in the dark and are, to say the least,
> slightly pissed. But not everybody seems to be barking at the right
> tree. So I've a few comments and questions for the KSPP.

First, let me adjust language here a bit to make things more clear.
Talking about KSPP is like talking about net-dev: it is upstream, not
a separate fork. It has a focus, like net-dev, but it is upstream. So
there is no difference semantically between talking about KSPP and
upstream. Additionally, while PaX Team, grsecurity, and ephox's work
were technically separate development efforts (for example, just look
at how PAX_USERCOPY differed between PaX and grsecurity), most people
used the combination, which I'll just call grsecurity here. Both
definitions will be helpful below...

> I think the main reason for Brad and PaX Team to make their work
> private is the increased amount of work KSSP has put on them without
> providing any valuable work in return. They just don't want to be

First, to think they didn't get valuable work in return from upstream
is missing the forest for the trees. With every release of upstream,
grsecurity would get tens of thousands of commits. If there wasn't
benefit in these changes, grsecurity would never forward port to the
latest upstream. The fact that it is a notable event that grsecurity
has ceased updating their public patches is because people using
grsecurity suddenly aren't getting the upstream changes, in addition
to them not getting new grsecurity features. It is a totally false
equivalency to say "upstream has [created work for|used code from]
grsecurity without giving anything valuable in return". I'll come back
to this later.

Second, when HARDENED_USERCOPY was landing, I specifically asked
grsecurity about the best way to make the upstreaming efforts easiest
to deal with for them, since I knew there might be some work with
their forward porting, and I didn't want to create undue work for
them. They ignored me. I asked again a while later and was continued
to be met with silence. Looking at the results in grsecurity, though,
it's clear that they chose to integrate with upstream instead of
maintaining a forked implementation. For example, the core
PAX_USERCOPY logic lived in fs/exec.c. When I upstreamed it, I moved
it to a place that made more sense in mm/usercopy.c. When grsecurity
forward ported, they could have trivially left HARDENED_USERCOPY
disabled, and kept PAX_USERCOPY using fs/exec.c, but they didn't.
Since grsecurity has talked in the past about how their
implementations are frequently arranged to make their forward porting
easier, I think this is strong evidence that using upstream's
implementation (with whatever further changes they wanted) was easier
than keeping their own fork. This is further supported by grsecurity
being paid by CII to upstream the gcc plugin infrastructure, where
they used almost entirely the same paths, Kconfigs, and Makefile
changes. So, given that grsecurity did not communicate at all about
wanting forward porting made easier and that they've reused gcc
plugins, hardened usercopy, etc, I think it's an entirely false claim
that upstream is creating more work that normal forward porting. In
some places, it even appears to be easier.

Finally, even if you can somehow disregard the thousands of upstream
changes benefiting grsecurity, and just want to look at the areas
touched while upstreaming things from grsecurity, they have benefited
from the upstreaming review by upstream finding bugs in various
grsecurity features (some of which haven't even landed in upstream
yet: Arnd Bergmann alone found tons of issues with the initify plugin
and grsecurity fixed them). Hardened usercopy found bugs in
grsecurity's slab implementation, gained strncpy_from_user() coverage,
and triggered a massive consolidation of per-arch uaccess fixes which
have found numerous upstream bugs that will now be fixed in grsecurity
too. The current refcount_t work based on PAX_REFCOUNT uncovered a
crash bug in grsecurity's implementation as well as a corner case in
bounds checking. Upstreaming grsecurity features brings a huge amount
of testing to bear on the code, which, like all the other things in
upstream, grsecurity directly benefits from. And to top all of this
off, while upstreaming the latent_entropy plugin, I noticed a English
typo that was present in all the grsecurity gcc plugins, and when I
sent them a trivial spelling fix patch for all their plugins (rather
than just letting it stand and making work for them to catch it during
forward porting and fix it everywhere else), they publicly mocked the
patch (which they applied). So, no, unless you count fixing newly
discovered bugs in their code, I again summarily reject the notion
that upstreaming grsecurity features creates "more work for grsecurity
without value in return".

> forced to maintain and fix-up the variants of grsecurity/PaX features
> KSPP lands upstream. And no, the work of KSPP does not make their life
> easier, in fact, it makes it harder. Harder for two reasons: First,
> the code does not end up verbatim, it's always changed, taken out of
> context and "enhanced". Second is the loose of control over the code.
> But let me elaborate those two a little further.

See above. In many cases, they chose integration over maintaining a
fork. In others, the two features coexist (for example, to various
levels of needless redundancy, see arch/arm/mm/init.c's KERNEXEC
overlap with RODATA).

> The first point directly forces them to think through the upstream
> incarnation of a feature, how it differs from their original one and,
> for those parts, how to fix them up to work properly to fit their
> needs, to fit their security requirements. Those changes generate a
> lot of conflicts with their version of the code which takes time to
> fiddle out. As happened for __ro_after_init, MEMORY_SANITIZE,
> USERCOPY,... which only went in in reduced and modified form,
> generating needless work on their side -- from their point of view.

One of the costs of forking a project is dealing with forward porting.
This problem is faced by all kinds of organizations (usually with
small engineering teams), where some set of features is desired on top
of Linux, and there are three choices for handling it, in ascending
order of difficulty:

- pick a kernel version, fork, develop features, use, abandon, repeat
(see most IoT vendors)

- pick a kernel version, fork, develop features, use, forward port to
next kernel version, repeat (see most phone vendors)

- develop features, upstream, use whatever future kernel desired (see
most CPU vendors and distros)

Upstreaming can be extremely time consuming. Grsecurity made it clear
from long ago that they had no intention of upstreaming things because
they wanted to use their time differently. This is entirely their
choice, and one that, frankly, the vast majority of Linux forks make.
But it means accepting the literally unending work of forward porting.
That said, when I asked grsecurity if there I was any way they would
accept payment to upstream things, they did briefly agree and
upstreamed the gcc plugin infrastructure. So they were willing to
upstream if paid, yet ultimately decided to stop, and to continue to
forward port their patches. But now they made their work private,
which, I should point out, doesn't change the amount of work for them,
except maybe no longer getting bug reports from their users.

To your specific examples, __ro_after_init is literally a one-line
change: they just make in __read_only. This was some of my first
attempts to make their forward porting work easy while upstream slowly
incorporated features. The work around PAX_MEMORY_SANITIZE made the
slab debug paths faster for everyone. I already talked about usercopy.
So, neither of us can speak for grsecurity, but I reject your belief
that upstream has somehow created needless work for them.

Which brings me to a question I haven't seen anyone ask yet: why does
grsecurity exist? If it was created to showcase strong security
mechanisms to benefit Linux users, then why fight upstreaming? Why
appear to abandon their users as soon as some of their work starts
entering upstream? The number of users protected by these features has
gone up by orders of magnitude, benefiting all Linux users, not just
grsecurity's users. One could take the position that grsecurity
desires to be differentiated from upstream so that they can point at
how more protected grsecurity users are over upstream users, but as
the gap shrinks, it will become harder to see the benefits. But that
would seem to be still a long way down the road, given how time
consuming upstreaming security features can be (regardless of whether
they come from grsecurity or not). So if they just want to be better
than upstream without caring about how many users are protected, then
it's not hard to imagine grsecurity taking their patches private. I
don't know that this is the actual rationale, but it really doesn't
look to me like they want to protect as many Linux users as possible.
And if protecting users isn't their goal, what is?

> The second point, the loose of control over the code, is even worse.
> Not getting any more conflicts when porting the grsecurity/PaX patch
> to a new kernel release makes changes to code that used to live in
> grsecurity/PaX probably go unnoticed. And, as it seem to be the case,
> upstream developers are not always familiar with all the gory details
> and might introduce weaknesses and bugs. This not only makes upstream
> Linux less secure, it makes grsecurity and PaX less secure, too.

You're speaking entirely in theoreticals, but I understand what you're
trying to say. All forward porting runs this risk. Kernel internals
change in ways that threaten even unchanged grsecurity features. To
bring this down to earth, I would ask "how does grsecurity perform
testing?" I can point to the many ways how upstream performs testing,
including LKDTM for several of the security-sensitive features. As
seen in the PAX_REFCOUNT porting work, the 4.9 grsecurity patch was
clearly never actually tested since _any_ exercise of the PAX_REFCOUNT
protection would Oops the kernel. I'm not saying they don't have
tests, nor am I saying they didn't just make a one-time mistake, but I
can point you directly to how upstream tests, how upstream developers
find bugs in grsecurity features, and how things can improve once
upstreamed (again, for example, the massive uaccess consolidation). As
to familiarity with code, this is part of upstreaming. Upstreaming
isn't just throwing a patch at the wall and running away: it needs to
have an understandable and compelling changelog, it needs to clearly
implemented and easy to maintain, etc. This differs greatly from
grsecurity changes which were rarely commented, had nearly unusable
changelogs, but only needed to be understood by a couple people. What
would happen to grsecurity if one of their developers decided to give
up on technology and go live in the woods? I would argue that features
in upstream are significantly more resilient to bad things happening,
but, we're now back where we started: a hard to prove opinion.

> So I can understand why they've done this. Still, with the loose of
> the public availability of the patch, Linux security has suffered a
> lot. Not only is upstream far far away to reach a level that is
> available today in grsecurity and PaX, no, it also won't benefit from
> new developments any more. What a great achievement! :(

Yeah, I'm quite disappointed too. When grsecurity talked at the first
Linux Security Summit in 2010, they detailed a "ten year todo list"
for upstream, itemizing all the features grsecurity had that upstream
needed. Over the years I took them at their word that they wanted to
see upstream improved but that they didn't have time to be bothered
with upstreaming. And even finding money for upstreaming didn't really
help things. And now that the needle has finally moved, and things are
starting to actually land upstream, they go private. I thought they
wanted to protect Linux users but not at the cost of upstreaming
(which, again, I can certainly understand: it can be very hard). But
now they've gone private. To me, that says I misunderstood their
intentions from the very start.

I'm glad to see that folks from the Hardened Gentoo project have
started organizing to upstream grsecurity features too. I'm excited to
find more people willing to risk their sanity and bring stuff
upstream: https://wiki.gentoo.org/wiki/Hardened/Hardened_Kernel_Project

> *sigh*
>
> I think the intention of the KSPP is good -- making vanilla Linux more
> secure. But the way it does its work harms overall Linux security. It
> does hurt mine, that's for sure! I know the value of grsecurity and

I think I have thoroughly disproved this position. Something that I
think is hard to see for people not involved in day-to-day upstream
work is how very different the development workflows are between
upstream and grsecurity. Upstream is normally evolutionary, doing
things in easy to digest pieces, where as grsecurity could just land
massive changes between releases. This makes it look like upstreaming
is just landing tiny bits of features, instead of looking at the long
view over time. Look at how __ro_after_init (while, yes, not being
anywhere close to __read_only) has been slowly gaining ground in
upstream. Look at how maybe one gcc plugin appears every couple of
releases, or how usercopy continues to expand coverage.

> PaX in particular and know how easy it still is to exploit a vanilla
> Linux. Features like KERNEXEC and RAP make it almost impossible for an
> attacker to abuse a memory corruption bug in the kernel. Many unnamed
> features -- unnamed because not under an #ifdef -- make exploiting
> use-after-free bugs much less interesting or reduce the race window
> for TOCTTOU bugs involving user copy operations. None of this can be
> found in vanilla Linux. Probably never will...

These are all complaints to be made to grsecurity. They are the ones
who provides those features originally, did not upstream them, and
then took them away. As far as never finding them in upstream, I would
remind you that grsecurity, while very impressive, is not the single
source for all Linux kernel security improvements. For example, ARM
created the PAN emulation code, HP is working on XPFO, etc. How much
further along would upstream be with security features if you yourself
had started sending patches like I've asked you repeatedly over the
years? I wasn't being glib; I was being genuine. If you have time to
criticize, you have time to write a patch. That said, I shouldn't give
you too hard a time: you have 5 times the number of commits in the
kernel than grsecurity has.

> So, here's my list of questions for the KSPP:
> 1/ When will I be able to switch to a vanilla Linux kernel that is
> equivalently hardened as a grsecurity/PaX kernel used to be?

How could anyone answer that question? I can't see the future. And
besides, it's not like grsecurity was providing comprehensive
protections. If you're using arm64, you might feel like you're already
in a better position with upstream (got PAN emu, got hardened
usercopy, still no RAP). If you're on x86_64, you might feel like
you've very far (no SMAP emu, no RAP). Upstream's goal is protecting
as many people as possible.

> 2/ Who will maintain this code and how?
> 3/ Who ensures the coverage and quality won't suffer for each new
> kernel release?

The upstream development community, just like everything else. I
answered both of these already above.

> Judging from the planed EOL for the v4.9 LTS kernel -- the last one a
> grsecurity patch was publicly available for --, there are less than
> two years to finish the work for item 1 to ensure a secure and smooth
> transition from grsecurity to upstream Linux. Will the KSPP be able to
> achieve this?... I have my doubts.

Again, it depends on your perspective. And if you have doubts, then be
part of the solution. What would you need to do to achieve your goal?

> Even if so, item 3 will be a hard problem to solve as the Linux kernel
> development model does not support such a tree-wide review point in
> the release cycle. Assuming there are people able and willing to judge
> and review the code base for required changes (e.g. atomic_t ->
> refcount_t changes or function pointer cleanups for RAP), how would
> those be able to simply get in those changes at, say, time of rc5? --
> without having to argue with a horde of maintainers of the individual
> subsystems involved?

This is an upstream development workflow problem that is especially
frustrating for security work since it can regularly touch lots of
areas, but it's one that I've been working to resolve. It's actually
on my list of things to discuss at the next Kernel Summit, since
making these wide changes is really irritating if it's going in via
each maintainer. This doesn't have to be the case, but that's how I've
been doing it lately since I can work on those pieces while waiting
for reviews on other things. And in the case of refcount_t, the
arduous work has provided some important discussions and reviews of
existing refcounting in the kernel, so I can't say this workflow is
entirely without merit.

> Quite a lot of questions. The external patch solved them all by not
> having to deal with upstream Linux development and not having the code
> available until it's ready. But now it's gone and no adequate
> replacement is on the horizon. What will KSPP do about it? I had a
> reasonably secure kernel, now it's gone. :(

Upstream was never in the business of providing the grsecurity patch,
so you'll need to ask them why they abandoned their users. Upstream's
work is unchanged. We'll still continue to work to bring security
features into the kernel, and we'll continue to welcome new
contributors.

-Kees

-- 
Kees Cook
Pixel Security

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[David Gens]

On 2017-05-02 00:01, Mathias Krause wrote:
> On 27 April 2017 at 00:04, Kees Cook <keescook@chromium.org> wrote:
>> On Wed, Apr 26, 2017 at 2:05 PM, Daniel Cegiełka
>> <daniel.cegielka@gmail.com> wrote:
>>> https://grsecurity.net/passing_the_baton_faq.php
>> 
>> Yeah, I'm sad to see them go. PaX Team was just last night helping
>> with some details of PAX_REFCOUNT as it could appear in the upstream
>> refcount_t API. I hope they'll still help out from time to time.
>> 
>> It does underscore the critical need to upstream stuff, though. Forks
>> of projects might disappear at any time. :(
> 
> Now, since grsecurity and PaX went private, quite a few users
> (including me) are left in the dark and are, to say the least,
> slightly pissed. But not everybody seems to be barking at the right
> tree. So I've a few comments and questions for the KSPP.
> 
> I think the main reason for Brad and PaX Team to make their work
> private is the increased amount of work KSSP has put on them without
> providing any valuable work in return. They just don't want to be
> forced to maintain and fix-up the variants of grsecurity/PaX features
> KSPP lands upstream. And no, the work of KSPP does not make their life
> easier, in fact, it makes it harder. Harder for two reasons: First,
> the code does not end up verbatim, it's always changed, taken out of
> context and "enhanced". Second is the loose of control over the code.
> But let me elaborate those two a little further.
> 
> The first point directly forces them to think through the upstream
> incarnation of a feature, how it differs from their original one and,
> for those parts, how to fix them up to work properly to fit their
> needs, to fit their security requirements. Those changes generate a
> lot of conflicts with their version of the code which takes time to
> fiddle out. As happened for __ro_after_init, MEMORY_SANITIZE,
> USERCOPY,... which only went in in reduced and modified form,
> generating needless work on their side -- from their point of view.
> 
> The second point, the loose of control over the code, is even worse.
> Not getting any more conflicts when porting the grsecurity/PaX patch
> to a new kernel release makes changes to code that used to live in
> grsecurity/PaX probably go unnoticed. And, as it seem to be the case,
> upstream developers are not always familiar with all the gory details
> and might introduce weaknesses and bugs. This not only makes upstream
> Linux less secure, it makes grsecurity and PaX less secure, too.
> 
> So I can understand why they've done this. Still, with the loose of
> the public availability of the patch, Linux security has suffered a
> lot. Not only is upstream far far away to reach a level that is
> available today in grsecurity and PaX, no, it also won't benefit from
> new developments any more. What a great achievement! :(
> 
> *sigh*
> 
> I think the intention of the KSPP is good -- making vanilla Linux more
> secure. But the way it does its work harms overall Linux security. It
> does hurt mine, that's for sure! I know the value of grsecurity and
> PaX in particular and know how easy it still is to exploit a vanilla
> Linux. Features like KERNEXEC and RAP make it almost impossible for an
> attacker to abuse a memory corruption bug in the kernel. Many unnamed
> features -- unnamed because not under an #ifdef -- make exploiting
> use-after-free bugs much less interesting or reduce the race window
> for TOCTTOU bugs involving user copy operations. None of this can be
> found in vanilla Linux. Probably never will...

So tell me how exactly a PaX-hardened kernel with a memory-corruption 
bug
prevents an adversary from modifying critical data, such as the page 
tables
in a data-only attack? IMHO the "PaX mitigates Memory-corruption" 
argument
is wishful thinking, and now that PaX is gone users cannot fool 
themselves
any longer into thinking "My System is Secure". That awareness is 
actually
good no?


> So, here's my list of questions for the KSPP:
> 1/ When will I be able to switch to a vanilla Linux kernel that is
> equivalently hardened as a grsecurity/PaX kernel used to be?
> 2/ Who will maintain this code and how?
> 3/ Who ensures the coverage and quality won't suffer for each new
> kernel release?
> 
> Judging from the planed EOL for the v4.9 LTS kernel -- the last one a
> grsecurity patch was publicly available for --, there are less than
> two years to finish the work for item 1 to ensure a secure and smooth
> transition from grsecurity to upstream Linux. Will the KSPP be able to
> achieve this?... I have my doubts.
> 
> Even if so, item 3 will be a hard problem to solve as the Linux kernel
> development model does not support such a tree-wide review point in
> the release cycle. Assuming there are people able and willing to judge
> and review the code base for required changes (e.g. atomic_t ->
> refcount_t changes or function pointer cleanups for RAP), how would
> those be able to simply get in those changes at, say, time of rc5? --
> without having to argue with a horde of maintainers of the individual
> subsystems involved?
> 
> Quite a lot of questions. The external patch solved them all by not
> having to deal with upstream Linux development and not having the code
> available until it's ready. But now it's gone and no adequate
> replacement is on the horizon. What will KSPP do about it? I had a
> reasonably secure kernel, now it's gone. :(
> 
> 
> Regards,
> Mathias

Best,
David

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Shawn]

On Tue, May 2, 2017 at 8:09 AM, Rik van Riel <riel@redhat.com> wrote:
> On Tue, 2017-05-02 at 00:01 +0200, Mathias Krause wrote:
>
>> I think the intention of the KSPP is good -- making vanilla Linux
>> more
>> secure. But the way it does its work harms overall Linux security. It
>> does hurt mine, that's for sure!
>
> Yeah, no. The grsecurity people produced patches
> that were used on maybe a few tens of thousands
> of systems, while the KSPP code will end up
> enhancing the security of over a billion Android
> devices.
>
Are you serious think KSPP can solve Android security issue without
changing the current Android eco-system? It's about one billion
Android device( most are old armv7 based) can be exploited and being
used by criminal and BIGBROs( NSA/CIA/PLA/whatever-A*/etc) with low
cost. I made hardened PoC( based on PaX + PXN + some code hardening)
for Nexus 7 2013 back in 2015. It proved that it can defeat "massive"
exp without much perf impact. But I don't see any sign those cellphone
vendors would port any KSPP features into the old devices. How about
new phone? I ported a couple of KSPP features( all originally done or
inspired by PaX/Grsecurity) to Pixel XL:

https://github.com/hardenedlinux/armv7-nexus7-grsec

Google's Pixel/Pixel2 may be the one of few cellphones can getting
benefit from KSPP. Otherwise, my own phone is running with much more
hardening features than Android O( released in Oct?) but it's still
not secure enough to defeat customized exploit.

> Those Android devices are more likely to require
> hardening, too, since they do not receive security
> updates as quickly as the systems maintained by
> grsecurity users.
>
Don't worry about it. PaX/Grsecurity can defeat multiple public
exploits without any fix. Maybe some ppl will go to maintain 4.9 LTS.

> Integrating hardening into the upstream kernel is
> a good thing for security, not a bad thing.
>
It depends how much security you need.

-- 
GNU powered it...
GPL protect it...
God blessing it...

regards
Shawn

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

On Tue, May 2, 2017 at 7:46 AM, Shawn <citypw@gmail.com> wrote:
> On Tue, May 2, 2017 at 8:09 AM, Rik van Riel <riel@redhat.com> wrote:
>> On Tue, 2017-05-02 at 00:01 +0200, Mathias Krause wrote:
>>
>>> I think the intention of the KSPP is good -- making vanilla Linux
>>> more
>>> secure. But the way it does its work harms overall Linux security. It
>>> does hurt mine, that's for sure!
>>
>> Yeah, no. The grsecurity people produced patches
>> that were used on maybe a few tens of thousands
>> of systems, while the KSPP code will end up
>> enhancing the security of over a billion Android
>> devices.
>>
> Are you serious think KSPP can solve Android security issue without
> changing the current Android eco-system? It's about one billion
> Android device( most are old armv7 based) can be exploited and being
> used by criminal and BIGBROs( NSA/CIA/PLA/whatever-A*/etc) with low
> cost. I made hardened PoC( based on PaX + PXN + some code hardening)
> for Nexus 7 2013 back in 2015. It proved that it can defeat "massive"
> exp without much perf impact. But I don't see any sign those cellphone
> vendors would port any KSPP features into the old devices. How about
> new phone? I ported a couple of KSPP features( all originally done or
> inspired by PaX/Grsecurity) to Pixel XL:
>
> https://github.com/hardenedlinux/armv7-nexus7-grsec

I would love it if things would get rolled out to older phones. This
isn't something upstream has much influence over, though. But we have
to look to the future. Eventually all the old phones will get
replaced, and the new ones will be running newer kernels, etc.

The problem with the phone (and IoT) ecosystem is an entirely separate
problem. There are some folks chipping away at it, but it's tangential
to upstream gaining more defenses. I'd also note that this problem
isn't solved by grsecurity either --- how many phones are running the
grsecurity patch?

> Google's Pixel/Pixel2 may be the one of few cellphones can getting
> benefit from KSPP. Otherwise, my own phone is running with much more
> hardening features than Android O( released in Oct?) but it's still
> not secure enough to defeat customized exploit.
>
>> Those Android devices are more likely to require
>> hardening, too, since they do not receive security
>> updates as quickly as the systems maintained by
>> grsecurity users.
>>
> Don't worry about it. PaX/Grsecurity can defeat multiple public
> exploits without any fix. Maybe some ppl will go to maintain 4.9 LTS.

This is kind of true. RAP was x86_64 only, so it didn't help armv7 nor
arm64. Hardened usercopy wasn't ported by grsecurity to arm64
(upstream did that, though it was trivial). UDEREF wasn't ported by
grsecurity to arm64, ARM filled that gap by inventing the mechanisms
to provide PAN emulation. There are all kinds of other protections
aren't arch-specific (e.g. heap hardening). Those would make good
things to focus on upstreaming if you were interested. CopperheadOS is
using a bunch of those things already:

https://github.com/CopperheadOS/kernel_google_marlin/commits/nougat-mr2-release

So really, upstreaming defenses has really helped the future phone
ecosystem, even if the "don't update old devices" attitude continues
to not change with vendors.

-Kees

-- 
Kees Cook
Pixel Security

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Mathias Krause]

On 2 May 2017 at 02:09, Rik van Riel <riel@redhat.com> wrote:
> On Tue, 2017-05-02 at 00:01 +0200, Mathias Krause wrote:
>
>> I think the intention of the KSPP is good -- making vanilla Linux
>> more
>> secure. But the way it does its work harms overall Linux security. It
>> does hurt mine, that's for sure!
>
> Yeah, no.

Well, yes, it does! Losing access to the grsecurity patch makes the
systems I do care about much less secure.

> The grsecurity people produced patches
> that were used on maybe a few tens of thousands
> of systems,

Where did you pull that number from? Out of thin air, I guess. I know,
for sure, there are many more installations.

> while the KSPP code will end up
> enhancing the security of over a billion Android
> devices.

Or making them more easily to DoS because features like VMAP_STACK and
HARDENED_USERCOPY will likely fail hard when hitting a vendor's diver
code base. Probably making them disable the problematic config
options. Even upstream still has to fix related fallout.

> Those Android devices are more likely to require
> hardening, too, since they do not receive security
> updates as quickly as the systems maintained by
> grsecurity users.

Why couldn't those devices benefit from grsecurity as well? Couldn't
google or Samsung just integrate grsecurity into their Android
kernels? They're far away from vanilla Linux anyway, so why not add
just another patch to provide some matured security code base to
protect those billion of Android devices? I'd guess, if a big player
like google would sponsor / pay grsecurity to provide a patch for the
relevant Android kernels, all sides would be happy: grsecurity for
getting wider adoption, Android users for having secured systems.

> Integrating hardening into the upstream kernel is
> a good thing for security, not a bad thing.

I never said it's a bad thing. Indeed I'm all for making vanilla Linux
more secure. Just how KSPP tries to do it is IMHO wrong. Ripping hunks
out of grsecurity and trying to integrate them into vanilla Linux
without understanding all the interdependencies or even the features
themselves, how would that provide security? By chance, maybe. But not
intentional, as that requires having thought of every corner case and
boundary condition.


Regards,
Mathias

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Mathias Krause]

On 2 May 2017 at 13:11, David Gens <david.gens@cs.tu-darmstadt.de> wrote:
> So tell me how exactly a PaX-hardened kernel with a memory-corruption bug
> prevents an adversary from modifying critical data, such as the page tables
> in a data-only attack?

It doesn't as it has not mitigation for data-only attacks yet. But
neither does a vanilla kernel. It's even worse on vanilla Linux where
the page tables simply can be written to with a write-what-where
primitive. PaX can, however, protect the page tables from getting
modified by having them write-protected. So yes, that makes it more
secure.

> IMHO the "PaX mitigates Memory-corruption" argument
> is wishful thinking, and now that PaX is gone users cannot fool themselves
> any longer into thinking "My System is Secure". That awareness is actually
> good no?

What about KERNEXEC and CONSTIFY taking care of protecting a lot of
data structures? What about SIZE_OVERFLOW successfully preventing know
overflow bugs, like the various keys related ones? What about RAP,
ensuring control flow integrity? That's nothing, you say? Ridiculous.


Thanks,
Mathias

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Casey Schaufler]

On 5/2/2017 2:16 PM, Mathias Krause wrote:
> On 2 May 2017 at 02:09, Rik van Riel <riel@redhat.com> wrote:
>> On Tue, 2017-05-02 at 00:01 +0200, Mathias Krause wrote:
>>
>>> I think the intention of the KSPP is good -- making vanilla Linux
>>> more
>>> secure. But the way it does its work harms overall Linux security. It
>>> does hurt mine, that's for sure!
>> Yeah, no.
> Well, yes, it does! Losing access to the grsecurity patch makes the
> systems I do care about much less secure.

Then pay for support.

>> The grsecurity people produced patches
>> that were used on maybe a few tens of thousands
>> of systems,
> Where did you pull that number from? Out of thin air, I guess. I know,
> for sure, there are many more installations.

What number are you suggesting, and where did you get it?

>> while the KSPP code will end up
>> enhancing the security of over a billion Android
>> devices.
> Or making them more easily to DoS because features like VMAP_STACK and
> HARDENED_USERCOPY will likely fail hard when hitting a vendor's diver
> code base. Probably making them disable the problematic config
> options. Even upstream still has to fix related fallout.

So you're saying that didn't happen when vendors tried to
apply grsecurity patches?

>> Those Android devices are more likely to require
>> hardening, too, since they do not receive security
>> updates as quickly as the systems maintained by
>> grsecurity users.
> Why couldn't those devices benefit from grsecurity as well? Couldn't
> google or Samsung just integrate grsecurity into their Android
> kernels? They're far away from vanilla Linux anyway, so why not add
> just another patch to provide some matured security code base to
> protect those billion of Android devices? I'd guess, if a big player
> like google would sponsor / pay grsecurity to provide a patch for the
> relevant Android kernels, all sides would be happy: grsecurity for
> getting wider adoption, Android users for having secured systems.

What makes you think that the "big players" didn't carefully
consider using grsecurity? And calling grsecurity "just another
patch" is like the Black Knight referring to his missing limbs
as "flesh wounds". Grsecurity isn't a thing. It's a jumble of
collections of tweaks and clever checks. Without some amount of
breaking out the implementations into discrete features there's
no way anyone doing long term upstream community maintenance
was ever going to be happy.


>> Integrating hardening into the upstream kernel is
>> a good thing for security, not a bad thing.
> I never said it's a bad thing. Indeed I'm all for making vanilla Linux
> more secure. Just how KSPP tries to do it is IMHO wrong. Ripping hunks
> out of grsecurity and trying to integrate them into vanilla Linux
> without understanding all the interdependencies or even the features
> themselves, how would that provide security? By chance, maybe. But not
> intentional, as that requires having thought of every corner case and
> boundary condition.

I seriously doubt that grsecurity is as wonderful as you seem
convinced it is. The fact that one can't identify independent
features or their interdependencies unless you're one of the
authors does not speak well for it. No, much better to examine
the flying saucer at Roswell and use what you've learned to build
understandable components than to "just clone" the starship.

> Regards,
> Mathias
>

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

On Tue, May 2, 2017 at 2:16 PM, Mathias Krause <minipli@googlemail.com> wrote:
> On 2 May 2017 at 02:09, Rik van Riel <riel@redhat.com> wrote:
>> while the KSPP code will end up
>> enhancing the security of over a billion Android
>> devices.
>
> Or making them more easily to DoS because features like VMAP_STACK and
> HARDENED_USERCOPY will likely fail hard when hitting a vendor's diver
> code base. Probably making them disable the problematic config
> options. Even upstream still has to fix related fallout.

A vendor who would disable HARDENED_USERCOPY would never have
integrated grsecurity, so that's a totally false equivalency.

I don't disagree that VMAP_STACK might have been implemented in a way
to better catch driver mistakes, but the condition was deemed rare
enough that it got implemented the way it did. You may object to the
difference in engineering practices, but pragmatism comes in many
flavors. And no one else developed vmap stacks for upstream, so I'd
rather have it than not.

>> Those Android devices are more likely to require
>> hardening, too, since they do not receive security
>> updates as quickly as the systems maintained by
>> grsecurity users.
>
> Why couldn't those devices benefit from grsecurity as well? Couldn't
> google or Samsung just integrate grsecurity into their Android
> kernels? They're far away from vanilla Linux anyway, so why not add
> just another patch to provide some matured security code base to
> protect those billion of Android devices? I'd guess, if a big player

The very fact that you're asking this question means you don't
understand how vendors deal with the kernel. With a small engineering
team, you can't afford to have changes from upstream you can't
support, so you leave as much of the kernel at stock upstream as
possible so you can get help from upstream when you need it. Now,
ironically, these same vendors don't realize that the moment their
kernel ages a few years, they're on the hook for supporting the ENTIRE
kernel again, since upstream has left that kernel in the dust.

Those organizations can't justify the resources needed to support an
out-of-tree kernel as their starting point. Chrome OS was probably one
of the most paranoid OS designs in a commercially available product
and we still couldn't get agreement to use grsecurity. (Though I would
point out support isn't the only issue: another is the risk of the
fork disappearing. *cough*)

> like google would sponsor / pay grsecurity to provide a patch for the
> relevant Android kernels, all sides would be happy: grsecurity for
> getting wider adoption, Android users for having secured systems.

And yet this never happened, and so the only way to get the defenses
to the general public was the upstream them. Which is something
grsecurity refused to do, even when offered money to do it.

>> Integrating hardening into the upstream kernel is
>> a good thing for security, not a bad thing.
>
> I never said it's a bad thing. Indeed I'm all for making vanilla Linux
> more secure. Just how KSPP tries to do it is IMHO wrong. Ripping hunks
> out of grsecurity and trying to integrate them into vanilla Linux
> without understanding all the interdependencies or even the features
> themselves, how would that provide security? By chance, maybe. But not
> intentional, as that requires having thought of every corner case and
> boundary condition.

So what's the solution? Give up? No. Want upstream to be some how
better at porting or developing defenses? Okay, help us. "Do it
better" isn't a particularly useful suggestion.

Besides, the defenses aren't opaque, even if they're virtually
undocumented. They can be extracted, studied, ported, and landed. For
the defenses that we're interested in that came from grsecurity, this
doesn't seem like a bad approach. If you have expertise in some of the
areas that are being upstreamed, speak up and contribute. Want
something to be upstreamed to your liking? Please do it.

-Kees

-- 
Kees Cook
Pixel Security

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Shawn]

On Wed, May 3, 2017 at 2:55 AM, Kees Cook <keescook@chromium.org> wrote:
> On Tue, May 2, 2017 at 7:46 AM, Shawn <citypw@gmail.com> wrote:
>> On Tue, May 2, 2017 at 8:09 AM, Rik van Riel <riel@redhat.com> wrote:
>>> On Tue, 2017-05-02 at 00:01 +0200, Mathias Krause wrote:
>>>
>>>> I think the intention of the KSPP is good -- making vanilla Linux
>>>> more
>>>> secure. But the way it does its work harms overall Linux security. It
>>>> does hurt mine, that's for sure!
>>>
>>> Yeah, no. The grsecurity people produced patches
>>> that were used on maybe a few tens of thousands
>>> of systems, while the KSPP code will end up
>>> enhancing the security of over a billion Android
>>> devices.
>>>
>> Are you serious think KSPP can solve Android security issue without
>> changing the current Android eco-system? It's about one billion
>> Android device( most are old armv7 based) can be exploited and being
>> used by criminal and BIGBROs( NSA/CIA/PLA/whatever-A*/etc) with low
>> cost. I made hardened PoC( based on PaX + PXN + some code hardening)
>> for Nexus 7 2013 back in 2015. It proved that it can defeat "massive"
>> exp without much perf impact. But I don't see any sign those cellphone
>> vendors would port any KSPP features into the old devices. How about
>> new phone? I ported a couple of KSPP features( all originally done or
>> inspired by PaX/Grsecurity) to Pixel XL:
>>
>> https://github.com/hardenedlinux/armv7-nexus7-grsec
>
> I would love it if things would get rolled out to older phones. This
> isn't something upstream has much influence over, though. But we have
> to look to the future. Eventually all the old phones will get
> replaced, and the new ones will be running newer kernels, etc.
>
> The problem with the phone (and IoT) ecosystem is an entirely separate
> problem. There are some folks chipping away at it, but it's tangential
> to upstream gaining more defenses. I'd also note that this problem
> isn't solved by grsecurity either --- how many phones are running the
> grsecurity patch?
>
The fragmentation of Android eco-system may be inevitable. The whole
chains is too long from ASOP/BSP/Vendors and it affect the security
fix being delivered to the end user. According to my own statistic
from my customers, there will be more than 7 millions of Android phone
will be using some features of PaX/Grsec this year. I've been pushing
those "simplified" mitigation from PaX/Grsecurity to some vendors.
That's why I have to find out what exploitable bugs is possible to
turned to be "massive" exp and which one is being massive used
already. To me, it's obviously my problem solved by PaX/Grsecurity. I
can't wait for Google's hepl to solve the problem. I shared what I
found about "massive" exp with you and other Google folksl at last
linux security summit. I really appreciate you guys helped some
patches goes into AOSP. But it's just a starting point. Maybe in
Google's point of view, everybody should buy the new phone( btw, I
really love Pixel XL). But I have responsible to taking care of old
devices.


>> Google's Pixel/Pixel2 may be the one of few cellphones can getting
>> benefit from KSPP. Otherwise, my own phone is running with much more
>> hardening features than Android O( released in Oct?) but it's still
>> not secure enough to defeat customized exploit.
>>
>>> Those Android devices are more likely to require
>>> hardening, too, since they do not receive security
>>> updates as quickly as the systems maintained by
>>> grsecurity users.
>>>
>> Don't worry about it. PaX/Grsecurity can defeat multiple public
>> exploits without any fix. Maybe some ppl will go to maintain 4.9 LTS.
>
> This is kind of true. RAP was x86_64 only, so it didn't help armv7 nor
> arm64. Hardened usercopy wasn't ported by grsecurity to arm64
> (upstream did that, though it was trivial). UDEREF wasn't ported by
> grsecurity to arm64, ARM filled that gap by inventing the mechanisms
> to provide PAN emulation. There are all kinds of other protections
> aren't arch-specific (e.g. heap hardening). Those would make good
> things to focus on upstreaming if you were interested. CopperheadOS is
> using a bunch of those things already:
>
> https://github.com/CopperheadOS/kernel_google_marlin/commits/nougat-mr2-release
>
Yeah, I know the history. Intel/ARM never credited PaX/Grsecurity
about the contribution of KERNEXEC/UDEREF, which is the origins of
hardware implementation, e.g: SMEP/SMAP in x86, PXN/PAN in
armv7/arm64. I don't blame Spender/PaX team;-) I thank to ARM
maintainers who did those mitigation for arm64. But I thank more to
PaX/Grsecurity who created it from nothing.

btw: I share the same view with Mathias Krause and other ppl who
really concern the real sense of security. I like KSPP in the 1st
place. But now I lost PaX/Grsecurity test patch. Who should I blame? I
think we made our point very clear here:

https://hardenedlinux.github.io/announcement/2017/04/29/hardenedlinux-statement2.html

> So really, upstreaming defenses has really helped the future phone
> ecosystem, even if the "don't update old devices" attitude continues
> to not change with vendors.
>
Agree! I'm afraid I can't help much on upstream because that's not my
customer want in short-term. S0rry, my idealist side never fit in
KSPP.



-- 
GNU powered it...
GPL protect it...
God blessing it...

regards
Shawn

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[David Gens]

On 2017-05-02 23:27, Mathias Krause wrote:
> On 2 May 2017 at 13:11, David Gens <david.gens@cs.tu-darmstadt.de> 
> wrote:
>> So tell me how exactly a PaX-hardened kernel with a memory-corruption 
>> bug
>> prevents an adversary from modifying critical data, such as the page 
>> tables
>> in a data-only attack?
> 
> It doesn't as it has not mitigation for data-only attacks yet. But
> neither does a vanilla kernel. It's even worse on vanilla Linux where
> the page tables simply can be written to with a write-what-where
> primitive. PaX can, however, protect the page tables from getting
> modified by having them write-protected. So yes, that makes it more
> secure.
> 
>> IMHO the "PaX mitigates Memory-corruption" argument
>> is wishful thinking, and now that PaX is gone users cannot fool 
>> themselves
>> any longer into thinking "My System is Secure". That awareness is 
>> actually
>> good no?
> 
> What about KERNEXEC and CONSTIFY taking care of protecting a lot of
> data structures? What about SIZE_OVERFLOW successfully preventing know
> overflow bugs, like the various keys related ones? What about RAP,
> ensuring control flow integrity? That's nothing, you say? Ridiculous.

Nope, I never said that. I said it is insufficient to mitigate memory 
corruption.
Actually, KSPP even has the potential to improve things here. Granted, 
for the time
being these PaX features are unavailable, but that was a one-sided 
political decision,
and blaming others (based on non-factual claims) is not helping.

I guess it would be nice to have a list of features that will be ported 
into KSPP,
e.g., sorted by priority, and then users could easily check when to 
switch from the
last PaX patches. But thats not how kernel development works 
(traditionally), because
developers push features they (or their employers) are interested in.

> Thanks,
> Mathias

Best,
David

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]

On Wed, 2017-05-03 at 12:50 +0800, Shawn wrote:

> The fragmentation of Android eco-system may be inevitable. The whole
> chains is too long from ASOP/BSP/Vendors and it affect the security
> fix being delivered to the end user. According to my own statistic
> from my customers, there will be more than 7 millions of Android
> phone
> will be using some features of PaX/Grsec this year.

That is great news. I am glad to hear the hardening features
are being used on that many phones.

Of course, given the fragmentation of the eco-system, the
only thing that can get the hardening on all of the (new)
phones in the future will be getting the hardening features
into the upstream kernel.

> btw: I share the same view with Mathias Krause and other ppl who
> really concern the real sense of security. I like KSPP in the 1st
> place. But now I lost PaX/Grsecurity test patch. Who should I blame?

I am not sure anyone deserves blame for this situation.

Spender has been doing what is best for his business,
and his work is an important asset for security-minded
people.

Kees and the other KSPP contributors have been doing what
is best for the community, and wide-spread adoption of
hardening functionality.

The important question to ask is "what do we do now?"

I suspect the answer is upstreaming more and more of the
grsecurity functionality, so nobody needs to carry around
that patch any more.

-- 
All rights reversed

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

On Tue, 2017-05-02 at 23:16 +0200, Mathias Krause wrote:
> On 2 May 2017 at 02:09, Rik van Riel <riel@redhat.com> wrote:
> > On Tue, 2017-05-02 at 00:01 +0200, Mathias Krause wrote:
> > 
> > > I think the intention of the KSPP is good -- making vanilla Linux
> > > more
> > > secure. But the way it does its work harms overall Linux
> > > security. It
> > > does hurt mine, that's for sure!
> > 
> > Yeah, no.
> 
> Well, yes, it does! Losing access to the grsecurity patch makes the
> systems I do care about much less secure.

Your systems are not necessarily representative of the
overall security of all Linux systems in the world,
though.

> > The grsecurity people produced patches
> > that were used on maybe a few tens of thousands
> > of systems,
> 
> Where did you pull that number from? Out of thin air, I guess. I
> know,
> for sure, there are many more installations.

I have seen a few emails suggesting there are millions
of systems with grsecurity patches around, and I got
the number wrong by several orders of magnitude.

However, that does not invalidate my original point:
the number of grsecurity protected systems is only a tiny
fraction of the entire Linux installed base.

It would be good to get hardening functionality upstream,
in order to get a majority of Linux systems (that get
deployed in the future) hardened out of the box.

> > Those Android devices are more likely to require
> > hardening, too, since they do not receive security
> > updates as quickly as the systems maintained by
> > grsecurity users.
> 
> Why couldn't those devices benefit from grsecurity as well? Couldn't
> google or Samsung just integrate grsecurity into their Android
> kernels? They're far away from vanilla Linux anyway, so why not add
> just another patch to provide some matured security code base to
> protect those billion of Android devices? I'd guess, if a big player
> like google would sponsor / pay grsecurity to provide a patch for the
> relevant Android kernels, all sides would be happy: grsecurity for
> getting wider adoption, Android users for having secured systems.

They could, but the reality is many vendors will not
bother enabling hardening features unless the code is
already there and enabled by default in the upstream
kernel.

> > Integrating hardening into the upstream kernel is
> > a good thing for security, not a bad thing.
> 
> I never said it's a bad thing. Indeed I'm all for making vanilla
> Linux
> more secure. Just how KSPP tries to do it is IMHO wrong. Ripping
> hunks
> out of grsecurity and trying to integrate them into vanilla Linux
> without understanding all the interdependencies or even the features
> themselves, how would that provide security? By chance, maybe. But
> not
> intentional, as that requires having thought of every corner case and
> boundary condition.

How would you do it?

Maintainers integrate code one patch series at a
time. That is not a constraint you can work around,
because code does need to be reviewed.

-- 
All rights reversed

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Wed, 2017-05-03 at 10:59 +0200, David Gens wrote:

> I guess it would be nice to have a list of features that will be
> ported 
> into KSPP,
> e.g., sorted by priority, and then users could easily check when to 
> switch from the
> last PaX patches. But thats not how kernel development works 
> (traditionally), because
> developers push features they (or their employers) are interested in.

That is the difference between development and support.

Developers will continue to do things, on their own schedule
and/or the schedule of their employers.

If you want grsecurity support, your best bet right now
is to pay grsecurity.

-- 
All rights reversed

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Daniel Micay]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

> Maintainers integrate code one patch series at a
> time. That is not a constraint you can work around,
> because code does need to be reviewed.

By not requiring fixes for issues like undefined function pointer usage
or other undefined behavior to be split up in hundreds of patches to go
through many different maintainers / many different trees. The same goes
for things like basic constification  (i.e. const and __ro_after_init)
that are clear cut and don't involve a performance compromise (so an
equivalent to pax_{open_close}_kernel would be different).

It's extremely unrealistic to get type-based Control Flow Integrity like
RAP for the mainline kernel if the fixes cannot be queued up in a single
tree. It also deters people from working on any of these small,
incremental improvements since they need to split it all up and try to
deal with dozens of maintainers, with most of the patches being lost.
See what happened to past attempts at this stuff.

Of course it needs to be reviewed and hopefully tested, but for fixing
clear cases of undefined behavior (i.e. those caught by UBSan or other
sanitizers and not covered by flags like -fno-strict-overflow -fno-
strict-aliasing) or uncontroversial things like missing const (without
other code changes beyond making pointers const), it should be possible
to land fixes via one tree. It isn't an issue that's specific to
security patches.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 866 bytes --]

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Daniel Micay]

[-- Attachment #1: Type: text/plain, Size: 2029 bytes --]

On Wed, 2017-05-03 at 14:56 -0400, Rik van Riel wrote:
> On Wed, 2017-05-03 at 12:50 +0800, Shawn wrote:
> 
> > The fragmentation of Android eco-system may be inevitable. The whole
> > chains is too long from ASOP/BSP/Vendors and it affect the security
> > fix being delivered to the end user. According to my own statistic
> > from my customers, there will be more than 7 millions of Android
> > phone
> > will be using some features of PaX/Grsec this year.
> 
> That is great news. I am glad to hear the hardening features
> are being used on that many phones.
> 
> Of course, given the fragmentation of the eco-system, the
> only thing that can get the hardening on all of the (new)
> phones in the future will be getting the hardening features
> into the upstream kernel.

Just worth noting that the upstream in this case almost always includes
the Android common kernel. There's still some baseline out-of-tree
Android code, although there's much less than there used to be so the
vast majority of the code these days is SoC vendor code needed by a non-
Android Linux distribution on those devices too.

That's forked into the SoC vendor kernels and then the device kernels if
applicable (some are device-specific, but some vendors like Sony have
moved to having a shared kernel and the Pixel / Pixel XL share a kernel
since they're basically the same thing). Google can also verify that
hardening is present via the Compatibility Test Suite if it can be
detected from an unprivileged userspace app. Hopefully they'll turn
passing their new privileged vts test suite into a requirement too so
they can test for kernel self protection features.

For example, Android devices are required to have perf_event_paranoid=3
even though it was rejected upstream for the time being. If a clearly
useful change is rejected, that doesn't mean Google won't add it to
their common kernel which will then propagate at least to new devices
from other vendors and their own first-party released devices.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 866 bytes --]

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

On Tue, May 2, 2017 at 9:50 PM, Shawn <citypw@gmail.com> wrote:
> The fragmentation of Android eco-system may be inevitable. The whole
> chains is too long from ASOP/BSP/Vendors and it affect the security
> fix being delivered to the end user. According to my own statistic
> from my customers, there will be more than 7 millions of Android phone
> will be using some features of PaX/Grsec this year. I've been pushing
> those "simplified" mitigation from PaX/Grsecurity to some vendors.
> That's why I have to find out what exploitable bugs is possible to
> turned to be "massive" exp and which one is being massive used
> already. To me, it's obviously my problem solved by PaX/Grsecurity. I
> can't wait for Google's hepl to solve the problem. I shared what I
> found about "massive" exp with you and other Google folksl at last
> linux security summit. I really appreciate you guys helped some
> patches goes into AOSP. But it's just a starting point. Maybe in
> Google's point of view, everybody should buy the new phone( btw, I
> really love Pixel XL). But I have responsible to taking care of old
> devices.

My concerns aren't limited to just phones, since the pattern of "stick
with one kernel version" is endemic in the industry. Even distros are
mostly guilty of this, though their production cycle time is much
shorter. I'm glad to have people like you working on this, especially
in places where it's traditionally not gotten a lot of attention!

> Yeah, I know the history. Intel/ARM never credited PaX/Grsecurity
> about the contribution of KERNEXEC/UDEREF, which is the origins of
> hardware implementation, e.g: SMEP/SMAP in x86, PXN/PAN in
> armv7/arm64. I don't blame Spender/PaX team;-) I thank to ARM
> maintainers who did those mitigation for arm64. But I thank more to
> PaX/Grsecurity who created it from nothing.

Yeah, no doubt. The stuff PaX and grsecurity invented was amazing! And
I credit them as helping me see that killing individual bugs isn't
sufficient for security defense work.

> btw: I share the same view with Mathias Krause and other ppl who
> really concern the real sense of security. I like KSPP in the 1st
> place. But now I lost PaX/Grsecurity test patch. Who should I blame? I
> think we made our point very clear here:
>
> https://hardenedlinux.github.io/announcement/2017/04/29/hardenedlinux-statement2.html

Well, there's exactly one group that decided to pull the patches. If
you want to assign blame, I think it's clear it falls squarely on
their shoulders.

To the first point at your URL, there's a lot of inaccuracies. KSPP is
an upstream group; CII funds individuals (not the whole of KSPP) that
make independent proposals. In fact, the very first CII funded person
was ephox from grsecurity. So ... if CII funding contributors to
upstream was the problem, then grsecurity created their own problem?
That makes no sense.

As for "hardly accomplished anything compared to grsecurity" that's a
pretty unfair comparison to make. Upstream has been really focusing on
this for maybe 2 years at best now? And grsecurity has been doing it
for over a decade? And besides, "upstream doesn't have all of
grsecurity upstreamed" isn't a reason to blame upstream for grsecurity
going private.

How can you say "introduced more bugs"? I'd be happy to fix them, if
you see them. And as I talked about in my reply to Mathias, of course
things look incomplete in upstream: they're being developed like
everything else in upstream: incrementally. It is not possible to just
fork-lift everything from grsecurity into upstream. It's just not how
upstream development works.

To this notion that there is some kind of conspiracy by LF to snub
grsecurity is just flat out silly. LF isn't upstream (though, sure,
they support a lot of the work). Upstream developers can't control
what or how LF says things (and few of us have time to deal with
that). If you look at things in upstream that came from grsecurity,
you'll see plenty of credit given. That sure seems like upstream
"revealing the truth to the public". Look at the copyright notices in
the gcc plugins, in mm/usercopy.c, and the notes in the Kconfigs, and
the commit messages. Look at my own blog posts covering the features,
look at the KSPP wiki page. The vast majority of things attributable
to grsecurity, do, in fact, give credit. So upstream certainly isn't
"blatantly stealing credit from grsecurity", so why would LF? And if
this single PR statement from LF is still seen as a sudden marketing
war against grsecurity, then I would ask "How is making the patches
private the solution?" Grsecurity has spent over a decade mocking and
berating upstream (just look at LWN), so how does a single article
from LF suddenly undo all their work to differentiate themselves? This
whole line of reasoning is convoluted, and frankly, has absolutely
nothing to do with upstream's efforts to protect Linux users.

It'd be nice if I could convince you otherwise (see my reply to
Mathias), but upstream hasn't "created more work" for grsecurity. And
even if you still believe that, making their patches private doesn't
reduce the amount of work they have to do. It just forces people to
pay them.

You later speak of philosophy, so let me ask you: if they've forced
their users to pay for access because their features were being
upstreamed, then they do not want these protections available to
everyone, they want them available only to those who would pay for
them. To me, though, it still can't be this simplistic. They've mostly
rejected spending free time to upstream things, they've mostly
rejected being paid to upstream things, and their efforts to be paid
to just work on grsecurity in the open (see the CII thread from a
while back) haven't worked, or are insufficient in some way. What is
it that grsecurity wants?

It looks very much like they want to stay a fork and be paid for it. I
certainly can't blame them for this; upstreaming is hard and as
Mathias says, they lose a certain level of control over the results.
Fundamentally, it's their time; they can do whatever they want with
it. And looking to be paid for your time is of course perfectly
sensible. But if this desire to stay a fork is true, then their
primary goal is NOT protecting as many people as possible, it's
protecting a subset of people as perfectly as they can. Again, this is
fine. Upstream and grsecurity can coexist in this state (since again,
upstreaming doesn't increase their work), and this was the state for
over a decade. But suddenly grsecurity went private.

So it would appear that grsecurity couldn't make enough money with
their patches staying public, so they've made them private. If that's
true, then it means that everyone who was using grsecurity but wasn't
a sponsor of grsecurity's work holds some level of responsibility for
grsecurity's decision.

It could be argued that upstream is "using grsecurity but wasn't a
sponsor", but this would be missing an important point: upstream's
contribution to grsecurity is Linux itself, which has giant value.
(And for my part personally, I did everything I could to pay
grsecurity, and succeeded a couple times over the years.) With this we
have come back around to looking at the GPL, which you mentioned. This
is specifically what the GPL was created to protect against. Why
hasn't grsecurity paid money to the tens of thousands of contributors
to the Linux kernel? Because they don't have to, and that is part of
the understanding about using the GPL for a project: everyone
contributes what they can, and everyone benefits. And if there are
forks, people can use each other's work so no one is risking being in
a situation where someone is unfairly benefiting from someone else's
work.

You had invoked RMS, so, here we are. It looks like grsecurity has
gone private because upstream started exercising the princples of the
GPL to expand how many upstream users would benefit from grsecurity's
work in the same way grsecurity's users benefited from upstream's
work. In answer to "who should I blame?" I'd say this is squarely on
grsecurity, not anyone else. They've chosen to benefit from upstream's
future work without making their future work available to upstream.

But maybe this is all wrong. Neither of us can speak for grsecurity,
nor know their motivations. I've always tried to avoid these kinds of
discussions for that very reason, and to stay focused on the technical
work. I can only truly speak for myself. I was faced with a seemingly
simple problem: Linux users need to be protected, the vast majority of
Linux users run upstream-derived kernels, but many of the best
defenses lived in grsecurity. What I'd learned from working at an
industry group (OSDL, the pre-LF), working at a distro (Canonical on
Ubuntu), and now working on devices and services (Google), is that
engineering teams in most places won't risk basing their work on a
fork of upstream, whether it be support concerns, cost concerns,
complexity concerns, or whatever. At the end of the day, it was clear
that the only path forward was to focus upstream on the same defense
principles grsecurity espoused, and to then start porting or creating
these kinds of defenses in upstream. (Which is the very thing
grsecurity told upstream to do at the first Linux Security Summit in
2010!) Doing this work was very time consuming, so I tried to collect
interested people together to work under a single umbrella to help
change the upstream culture, examine flaws, port or invent defenses,
test, document, etc.

So, in the end, I can't speak to grsecurity's true motivations, but I
can speak to mine, and what I'd like upstream to be doing: protecting
as many Linux users as completely as possible. What's possible in
upstream differs from what was possible in grsecurity, so no one
should expect the same things between the two. Comparing the two
projects is fraught with error, and blaming anyone other than
grsecurity for going private is ridiculous.

> Agree! I'm afraid I can't help much on upstream because that's not my
> customer want in short-term. S0rry, my idealist side never fit in
> KSPP.

Sure, understood. I'm delighted you've spent time getting kernel
hardening stuff landed in AOSP, though; for vendors that can be
convinced to care about backporting these things, it's great to have a
place to point to for them to get your work! And it's fine not to
contribute to upstream, since luckily there will be some people who
can. Upstream will continue to work on security defenses, but at the
very least, it'd be nice if people would be constructive in their
criticisms. We've gone over a decade with grsecurity's frequently
toxic rhetoric only alienating people instead of productively working
together to reach what should be the common goal of security work:
defending users.

-Kees

-- 
Kees Cook
Pixel Security

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Lionel Debroux]

Hi,

(I hope that silly webmail is not going to butcher my e-mail's
layout, this time...)

> It'd be nice if I could convince you otherwise (see my reply to
> Mathias), but upstream hasn't "created more work" for grsecurity.
> [...]
> (since again, upstreaming doesn't increase their work)
Uh, do you really mean that ?
Because on the contrary, it's a (real) _fact_ that upstreaming
something directly taken from, or inspired by, the PaX/grsecurity
work _does_ increase their work, through at least three
well-identified mechanisms:
* adjusting their work to match upstream's changes. That's relatively
simple if the changes are pure renames, but as we know, upstream
changes are not always (not often, even, they'd probably say) that
simple - and do not always come without drawbacks on complexity or
protective abilities, either. spender has been posting (complaining)
multiple times about that, giving specific examples - which you may
have seen by yourself.

* helping upstream perform the integration in the first place. They
had to spend much time helping the mainlining process of some bits.
See it indirectly in spender's posts questioning mainline's ability
to perform _sustained_ maintenance of the integrated features (even
if they received some minor fixes on some GCC plugins in return for
the integration, as you mentioned).

Don't let spender's usual writing style (not necessarily more abrasive
than Linus', BTW) shadow the technical reasons behind his complaints,
some are arguably valid.

* unapplying, one by one, the hunks mentioned earlier for improvements
related to (mostly) CONSTIFY, RANDSTRUCT, RAP. They've been carrying
these patches for years, with very little maintenance: the vast
majority of these hunks didn't change from one upstream version to the
next one. We've already discussed at length the non-technical reasons
why nobody picked them up.

To tell the truth, the fact that upstreaming said changes does, as a
matter of fact, increase their work load was the second, auxiliary
reason why, after several patches in 2010 and 2012, I didn't spend
more of my free time mainlining some of these small, scattered changes,
rather than on other endeavours. I find myself caring quite a bit less
about that second reason now that the patches have become private, but
the main reason of the dishearteningly expensive mainlining process
largely remains valid... I do understand the need for reviews in
general, but CONSTIFY or RANDSTRUCT-related hunks are arguably special.


> And even if you still believe that, making their patches private
> doesn't reduce the amount of work they have to do. It just forces
> people to pay them.
Yes, that is true. Arguably, in some ways, since making their patches
private increases the incentive to mainline hunks thereof, it
_increases_ their work over some areas in the short and mid-term.


Bye,
Lionel.

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Shawn]

On Thu, May 4, 2017 at 1:45 PM, Kees Cook <keescook@chromium.org> wrote:
>> btw: I share the same view with Mathias Krause and other ppl who
>> really concern the real sense of security. I like KSPP in the 1st
>> place. But now I lost PaX/Grsecurity test patch. Who should I blame? I
>> think we made our point very clear here:
>>
>> https://hardenedlinux.github.io/announcement/2017/04/29/hardenedlinux-statement2.html
>
> Well, there's exactly one group that decided to pull the patches. If
> you want to assign blame, I think it's clear it falls squarely on
> their shoulders.
>
> To the first point at your URL, there's a lot of inaccuracies. KSPP is
> an upstream group; CII funds individuals (not the whole of KSPP) that
> make independent proposals. In fact, the very first CII funded person
> was ephox from grsecurity. So ... if CII funding contributors to
> upstream was the problem, then grsecurity created their own problem?
> That makes no sense.
>
> As for "hardly accomplished anything compared to grsecurity" that's a
> pretty unfair comparison to make. Upstream has been really focusing on
> this for maybe 2 years at best now? And grsecurity has been doing it
> for over a decade? And besides, "upstream doesn't have all of
> grsecurity upstreamed" isn't a reason to blame upstream for grsecurity
> going private.
>
> How can you say "introduced more bugs"? I'd be happy to fix them, if
> you see them. And as I talked about in my reply to Mathias, of course
> things look incomplete in upstream: they're being developed like
> everything else in upstream: incrementally. It is not possible to just
> fork-lift everything from grsecurity into upstream. It's just not how
> upstream development works.
>
> To this notion that there is some kind of conspiracy by LF to snub
> grsecurity is just flat out silly. LF isn't upstream (though, sure,
> they support a lot of the work). Upstream developers can't control
> what or how LF says things (and few of us have time to deal with
> that). If you look at things in upstream that came from grsecurity,
> you'll see plenty of credit given. That sure seems like upstream
> "revealing the truth to the public". Look at the copyright notices in
> the gcc plugins, in mm/usercopy.c, and the notes in the Kconfigs, and
> the commit messages. Look at my own blog posts covering the features,
> look at the KSPP wiki page. The vast majority of things attributable
> to grsecurity, do, in fact, give credit. So upstream certainly isn't
> "blatantly stealing credit from grsecurity", so why would LF? And if
> this single PR statement from LF is still seen as a sudden marketing
> war against grsecurity, then I would ask "How is making the patches
> private the solution?" Grsecurity has spent over a decade mocking and
> berating upstream (just look at LWN), so how does a single article
> from LF suddenly undo all their work to differentiate themselves? This
> whole line of reasoning is convoluted, and frankly, has absolutely
> nothing to do with upstream's efforts to protect Linux users.
>
I don't have the problem with ephox funded by CII for upstreaming GCC
plugins. I blame LF/CII's( I can't speak for other ppl from
HardenedLinux) PR because they've been doing PR since the day 1 of
KSPP. A lot of ppl being misled by their improper PR including BLUG(
Beijing GNU/Linux User Group) and my customers. I don't know how LF
sells their "service" or "membership" and what stories they tell to
enterprise management. Eventually some of my customers indeed being
affected by some false sense of security advertisement about linux
security. As a security consultant, how am I suppose to answer the
question like "there are a bunch of exploits in the wild and the
linux( KSPP?) can't mitigate, which isn't the stories we've had heard"
from customers? I understand it's unfair to compare KSPP with
PaX/Grsecurity. But the fact is there are multiple exploitable bugs in
the wild that's the reason I can not rely on KSPP now:

https://github.com/hardenedlinux/grsecurity-101-tutorials/blob/master/kernel_vuln_exp.md

You may still remember those days that "one null-ptr deref exploit can
rule them all". It was my nightmare that endless breaches from both
outside and inside. PaX's KERNEXEC/UDEREF saved my time back then. As
the evidences show that PaX/Grsec is still the most effective defense
in RING0 so far.

That announcement only represented the POV from a group of ppl. From
my( and other ppl from HardenedLinux) perspective, Linux foundation is
a commercial company and very good at PR but zero integrity to us.
They don't respect individuals and the community. But I never said
KSPP don't credited PaX/Grsecurity in the commits or Kconfig. I've
been reviewing you guys work, which porting some pieces of
PaX/Grsecurity code/features to the mainline. I'm glad to see linux
kernel is raising the bar. I translated KSPP documentation into
Chinese and let more ppl( both community and commercial customers need
education) know there are some ppl care security and trying to achieve
the security level what PaX/Grsecurity can provided. But what LF PR
does is really harmful to my efforts. I'll have to spend my extra time
on explanation and writing reports for those victims( customers, other
vendors, even one of my investors). As I see, LF was trying to
sabotage the small open source consulting business like mine.

Otherwise, I don't care about Spender's personality stuff. All I
concern is the practical defense solution to protect our asset from
those threats not only from Ring 0. Both fortunate and unfortunate
that PaX/Grsec is still an option. We build our defense solution based
on PaX/Grsecurity and some of our work is open source at
HardenedLinux's github.

> It'd be nice if I could convince you otherwise (see my reply to
> Mathias), but upstream hasn't "created more work" for grsecurity. And
> even if you still believe that, making their patches private doesn't
> reduce the amount of work they have to do. It just forces people to
> pay them.
>
> You later speak of philosophy, so let me ask you: if they've forced
> their users to pay for access because their features were being
> upstreamed, then they do not want these protections available to
> everyone, they want them available only to those who would pay for
> them. To me, though, it still can't be this simplistic. They've mostly
> rejected spending free time to upstream things, they've mostly
> rejected being paid to upstream things, and their efforts to be paid
> to just work on grsecurity in the open (see the CII thread from a
> while back) haven't worked, or are insufficient in some way. What is
> it that grsecurity wants?
>
> It looks very much like they want to stay a fork and be paid for it. I
> certainly can't blame them for this; upstreaming is hard and as
> Mathias says, they lose a certain level of control over the results.
> Fundamentally, it's their time; they can do whatever they want with
> it. And looking to be paid for your time is of course perfectly
> sensible. But if this desire to stay a fork is true, then their
> primary goal is NOT protecting as many people as possible, it's
> protecting a subset of people as perfectly as they can. Again, this is
> fine. Upstream and grsecurity can coexist in this state (since again,
> upstreaming doesn't increase their work), and this was the state for
> over a decade. But suddenly grsecurity went private.
>
> So it would appear that grsecurity couldn't make enough money with
> their patches staying public, so they've made them private. If that's
> true, then it means that everyone who was using grsecurity but wasn't
> a sponsor of grsecurity's work holds some level of responsibility for
> grsecurity's decision.
>
> It could be argued that upstream is "using grsecurity but wasn't a
> sponsor", but this would be missing an important point: upstream's
> contribution to grsecurity is Linux itself, which has giant value.
> (And for my part personally, I did everything I could to pay
> grsecurity, and succeeded a couple times over the years.) With this we
> have come back around to looking at the GPL, which you mentioned. This
> is specifically what the GPL was created to protect against. Why
> hasn't grsecurity paid money to the tens of thousands of contributors
> to the Linux kernel? Because they don't have to, and that is part of
> the understanding about using the GPL for a project: everyone
> contributes what they can, and everyone benefits. And if there are
> forks, people can use each other's work so no one is risking being in
> a situation where someone is unfairly benefiting from someone else's
> work.
>
> You had invoked RMS, so, here we are. It looks like grsecurity has
> gone private because upstream started exercising the princples of the
> GPL to expand how many upstream users would benefit from grsecurity's
> work in the same way grsecurity's users benefited from upstream's
> work. In answer to "who should I blame?" I'd say this is squarely on
> grsecurity, not anyone else. They've chosen to benefit from upstream's
> future work without making their future work available to upstream.
>
> But maybe this is all wrong. Neither of us can speak for grsecurity,
> nor know their motivations. I've always tried to avoid these kinds of
> discussions for that very reason, and to stay focused on the technical
> work. I can only truly speak for myself. I was faced with a seemingly
> simple problem: Linux users need to be protected, the vast majority of
> Linux users run upstream-derived kernels, but many of the best
> defenses lived in grsecurity. What I'd learned from working at an
> industry group (OSDL, the pre-LF), working at a distro (Canonical on
> Ubuntu), and now working on devices and services (Google), is that
> engineering teams in most places won't risk basing their work on a
> fork of upstream, whether it be support concerns, cost concerns,
> complexity concerns, or whatever. At the end of the day, it was clear
> that the only path forward was to focus upstream on the same defense
> principles grsecurity espoused, and to then start porting or creating
> these kinds of defenses in upstream. (Which is the very thing
> grsecurity told upstream to do at the first Linux Security Summit in
> 2010!) Doing this work was very time consuming, so I tried to collect
> interested people together to work under a single umbrella to help
> change the upstream culture, examine flaws, port or invent defenses,
> test, document, etc.
>
Maybe you are right from your perspective. I appreciate what you've
contributed to the FLOSS community since you were working for
Canonical. Upstream is good but it's always raising the bar but never
being good enough for the defensive side. It never bothers me when I
worked for two towers of GNU/Linux vendor( SUSE/RH). It bothers me now
and the desert of the real is happening every single day, while spend
another years of waiting is not practical to those who concerns their
security.

Speaking of GPL, closing the public access isn't violate GPL
technically. On the other hand, the authors of PaX/Grsecurity can do
anything they want. It's their rights. To be honest, ppl who concerns
security are minority. It's sad to see this group of ppl lost test
patch. They shared their work with FLOSS world for 16 years. I don't
think I have the rights to blame them. Anyway, I'd still blame LF.

> So, in the end, I can't speak to grsecurity's true motivations, but I
> can speak to mine, and what I'd like upstream to be doing: protecting
> as many Linux users as completely as possible. What's possible in
> upstream differs from what was possible in grsecurity, so no one
> should expect the same things between the two. Comparing the two
> projects is fraught with error, and blaming anyone other than
> grsecurity for going private is ridiculous.
>
>> Agree! I'm afraid I can't help much on upstream because that's not my
>> customer want in short-term. S0rry, my idealist side never fit in
>> KSPP.
>
> Sure, understood. I'm delighted you've spent time getting kernel
> hardening stuff landed in AOSP, though; for vendors that can be
> convinced to care about backporting these things, it's great to have a
> place to point to for them to get your work! And it's fine not to
> contribute to upstream, since luckily there will be some people who
> can. Upstream will continue to work on security defenses, but at the
> very least, it'd be nice if people would be constructive in their
> criticisms. We've gone over a decade with grsecurity's frequently
> toxic rhetoric only alienating people instead of productively working
> together to reach what should be the common goal of security work:
> defending users.
>
Yeah, we have different battle to fight with. I truely hope KSPP can
be an alternative mitigation solution in the future because the users
could get benefit from open competitions. I think PaX team and Spender
never fear to compete with any other defense solution.



-- 
GNU powered it...
GPL protect it...
God blessing it...

regards
Shawn

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Greg KH]

On Thu, May 04, 2017 at 10:11:04PM +0800, Shawn wrote:
> That announcement only represented the POV from a group of ppl. From
> my( and other ppl from HardenedLinux) perspective, Linux foundation is
> a commercial company and very good at PR but zero integrity to us.

A slight correction here please.  The LF is a non-profit organization[1]
set up to promote Linux and allow companies who want to see Linux
succeed, get together and do this.  The LF happens to sponsor a few
kernel developers (me and Linus), but they can not tell us what to do at
all.

They also are a place that companies have come together to help with the
state of security in the Linux and Open Source ecosystem, starting CII
which offers grants to anyone who wants to get paid to do security work
(new features, support, audits, etc.)  CII doesn't make any money, it
gives money away!  Of course it does press releases saying what projects
it funds in order to get other projects and people to submit project
proposals to continue this work.  I know of at least 2 new kernel
security projects that recently got funding because of this.

So there is no "integrity" that the LF can, or can not, have when it
comes to anyone here as the LF doesn't actually _do_ anything when it
comes to kernel development (again, other than funding 2 developers
directly).

> They don't respect individuals and the community.

That's a load of crap, really.  The LF has always had a kernel community
developer as a full board member, and sponsors conferences, travel
funding, hardware acquisition, intern programs, and lots of other stuff.
I don't know of any kernel community request that the LF has _not_
funded, do you?

The LF is all about making the whole community work well together, and
that includes both individual developers and companies as this is a
symbiotic relationship (companies use Linux, fund its development,
create new hardware for Linux to run on, etc.)  Without one part of the
group, Linux would not succeed at all, and they know that quite well.

If the LF didn't "individuals and the community", I know I wouldn't be
working for them.

So I don't know why anyone would be "upset" at the LF here, all they
have done is actually fund people to do kernel security work, including
members of the grsecurity team!  How is doing that somehow "bad"?  Do
you want to go back to 2+ years ago when they were not doing this
funding at all?

And does no one remember how things were before there was a LF?  Do you
really want to go back to those days?  Were they somehow better than
things are now?  As someone who remembers those times quite well, I can
assure you that they were not.

Sorry for the digression,

greg k-h

[1] Yes, it's structured as a trade organization, it has to be that way
    from a legal point of view in order for companies to be able to
    help Linux and work together.  Without it, companies would be
    violating anti-trust laws and would not be able to help the
    community out at all.  Think of the LF as the "Milk Advisory Board"
    for Linux.

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Shawn]

Hi Greg

On Fri, May 5, 2017 at 12:03 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, May 04, 2017 at 10:11:04PM +0800, Shawn wrote:
>> That announcement only represented the POV from a group of ppl. From
>> my( and other ppl from HardenedLinux) perspective, Linux foundation is
>> a commercial company and very good at PR but zero integrity to us.
>
> A slight correction here please.  The LF is a non-profit organization[1]
> set up to promote Linux and allow companies who want to see Linux
> succeed, get together and do this.  The LF happens to sponsor a few
> kernel developers (me and Linus), but they can not tell us what to do at
> all.
>
Oh, that's new to me. LF is a non-profit organization. Maybe some
"rumors" isn't true: I thought LF hired two "sales" people in HK last
year? Can you confirm that?

Otherwise, which NGO's runner has $344,220 salary per year?
http://news.idg.no/cw/art.cfm?id=5A9F8343-BAA1-6432-72A26555784BF05E

LF is growing so fast, look at that. Compensation is seems very
promising even in 2013:
https://projects.propublica.org/nonprofits/organizations/460503801

$499,705 is the total compensation in 2014? Wow..why this magic
number? Because $500,000 would be excessive?
http://pdfs.citizenaudit.org/2016_02_EO/46-0503801_990O_201412.pdf

> They also are a place that companies have come together to help with the
> state of security in the Linux and Open Source ecosystem, starting CII
> which offers grants to anyone who wants to get paid to do security work
> (new features, support, audits, etc.)  CII doesn't make any money, it
> gives money away!  Of course it does press releases saying what projects
> it funds in order to get other projects and people to submit project
> proposals to continue this work.  I know of at least 2 new kernel
> security projects that recently got funding because of this.
>
> So there is no "integrity" that the LF can, or can not, have when it
> comes to anyone here as the LF doesn't actually _do_ anything when it
> comes to kernel development (again, other than funding 2 developers
> directly).
>
I'm curious what's your offical relationship with Google? Cu'z you
have this account( gregkh@google.com) from Google:
https://android.googlesource.com/kernel/common.git/+/e88bb963b88d5579805b90e8d505739692095042

But you seems still working for LF, aren't you?:
https://www.linuxfoundation.org/about/linux-foundation-fellows

>> They don't respect individuals and the community.
>
> That's a load of crap, really.  The LF has always had a kernel community
> developer as a full board member, and sponsors conferences, travel
> funding, hardware acquisition, intern programs, and lots of other stuff.
> I don't know of any kernel community request that the LF has _not_
> funded, do you?
>
Really? That's also new to me. Let me give you some hints:

The Linux Foundation: Not a Friend of Desktop Linux, the GPL, or Openness:
http://fossforce.com/2017/04/lin-desktop-linux-gpl-openness/

OpenSSL after Heartbleed:
https://lwn.net/Articles/703000/

Linux Foundation quietly drops community representation:
https://mjg59.dreamwidth.org/39546.html

> The LF is all about making the whole community work well together, and
> that includes both individual developers and companies as this is a
> symbiotic relationship (companies use Linux, fund its development,
> create new hardware for Linux to run on, etc.)  Without one part of the
> group, Linux would not succeed at all, and they know that quite well.
>
> If the LF didn't "individuals and the community", I know I wouldn't be
> working for them.
>
> So I don't know why anyone would be "upset" at the LF here, all they
> have done is actually fund people to do kernel security work, including
> members of the grsecurity team!  How is doing that somehow "bad"?  Do
> you want to go back to 2+ years ago when they were not doing this
> funding at all?
>
> And does no one remember how things were before there was a LF?  Do you
> really want to go back to those days?  Were they somehow better than
> things are now?  As someone who remembers those times quite well, I can
> assure you that they were not.
>
In my very "narrow" POV( only as FLOSS supporter and security
consutlant), KSPP would be so much better without LF's involvement and
PR


-- 
GNU powered it...
GPL protect it...
God blessing it...

regards
Shawn

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Greg KH]

On Fri, May 05, 2017 at 01:12:37AM +0800, Shawn wrote:
> Hi Greg
> 
> On Fri, May 5, 2017 at 12:03 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Thu, May 04, 2017 at 10:11:04PM +0800, Shawn wrote:
> >> That announcement only represented the POV from a group of ppl. From
> >> my( and other ppl from HardenedLinux) perspective, Linux foundation is
> >> a commercial company and very good at PR but zero integrity to us.
> >
> > A slight correction here please.  The LF is a non-profit organization[1]
> > set up to promote Linux and allow companies who want to see Linux
> > succeed, get together and do this.  The LF happens to sponsor a few
> > kernel developers (me and Linus), but they can not tell us what to do at
> > all.
> >
> Oh, that's new to me. LF is a non-profit organization. Maybe some
> "rumors" isn't true: I thought LF hired two "sales" people in HK last
> year? Can you confirm that?

And it's always been a non-profit, that should not be a surprise.

And I don't have any idea what a "sales" person at the LF would do,
perhaps the fact that the LF is hiring people to work with local
companies to become members?  Why not ask them what they do?

> > So there is no "integrity" that the LF can, or can not, have when it
> > comes to anyone here as the LF doesn't actually _do_ anything when it
> > comes to kernel development (again, other than funding 2 developers
> > directly).
> >
> I'm curious what's your offical relationship with Google? Cu'z you
> have this account( gregkh@google.com) from Google:
> https://android.googlesource.com/kernel/common.git/+/e88bb963b88d5579805b90e8d505739692095042
> 
> But you seems still working for LF, aren't you?:
> https://www.linuxfoundation.org/about/linux-foundation-fellows

What does my google account have to do with the LF?  I've had that
account for many many years now, to help Google out with kernel security
work, including stable kernel releases.

I have lots of other email addresses too, doesn't everyone?  :)

And yes, I still work for the LF.

> In my very "narrow" POV( only as FLOSS supporter and security
> consutlant), KSPP would be so much better without LF's involvement and
> PR

So the funding of grsecurity developers to get features merged into
mainline was bad?  That's a very strange viewpoint of "so much better".

good luck!

greg k-h

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

On Wed, May 3, 2017 at 11:47 PM, Lionel Debroux <lionel_debroux@yahoo.fr> wrote:
>> It'd be nice if I could convince you otherwise (see my reply to
>> Mathias), but upstream hasn't "created more work" for grsecurity.
>> [...]
>> (since again, upstreaming doesn't increase their work)
> Uh, do you really mean that ?
> Because on the contrary, it's a (real) _fact_ that upstreaming
> something directly taken from, or inspired by, the PaX/grsecurity

I think it would be noise compared to all the regular work needed when
doing forward porting, yes. As I said in my reply to Mathias, I think
some things make it harder, some things make it easier, and others
don't change anything. Other stuff happening in upstream can be WAY
more disruptive, like rewriting x86_64 entry code. I bet that was
terrible to do a forward port after.

> work _does_ increase their work, through at least three
> well-identified mechanisms:
> * adjusting their work to match upstream's changes. That's relatively
> simple if the changes are pure renames, but as we know, upstream
> changes are not always (not often, even, they'd probably say) that
> simple - and do not always come without drawbacks on complexity or
> protective abilities, either. spender has been posting (complaining)
> multiple times about that, giving specific examples - which you may
> have seen by yourself.

This is what I think is covered by regular upstreaming, mentioned
above. On the level, it looks like it's not a meaningfully different
amount of work. Maybe I'm wrong, but you can go look at the evidence I
linked to about how forward porting was performed in the past.

> * helping upstream perform the integration in the first place. They
> had to spend much time helping the mainlining process of some bits.
> See it indirectly in spender's posts questioning mainline's ability
> to perform _sustained_ maintenance of the integrated features (even
> if they received some minor fixes on some GCC plugins in return for
> the integration, as you mentioned).

Well, again, they get all of upstream. The hardened usercopy
upstreaming triggered a massive uaccess rewrite that means all of the
arch-specific code has gone away so there's one place for usercopy
stuff. That implies both reduction in forward porting work and
evidence of "sustained maintenance". My little typo fix patch was
something I sent because I noticed it affected things that weren't yet
upstream, so it felt silly not to send it since I was right there
looking at the code.

> Don't let spender's usual writing style (not necessarily more abrasive
> than Linus', BTW) shadow the technical reasons behind his complaints,
> some are arguably valid.

Neither of their styles excuse the other, but yes, I absolutely
understand that if I can dig through the mockery and insults, I can
almost always find very compelling technical analysis from grsecurity.

> * unapplying, one by one, the hunks mentioned earlier for improvements
> related to (mostly) CONSTIFY, RANDSTRUCT, RAP. They've been carrying
> these patches for years, with very little maintenance: the vast
> majority of these hunks didn't change from one upstream version to the
> next one. We've already discussed at length the non-technical reasons
> why nobody picked them up.

Well, again, the evidence in the recent kernel patches would indicate
that's not how things have been dealt with.

> To tell the truth, the fact that upstreaming said changes does, as a
> matter of fact, increase their work load was the second, auxiliary
> reason why, after several patches in 2010 and 2012, I didn't spend
> more of my free time mainlining some of these small, scattered changes,
> rather than on other endeavours. I find myself caring quite a bit less
> about that second reason now that the patches have become private, but
> the main reason of the dishearteningly expensive mainlining process
> largely remains valid... I do understand the need for reviews in
> general, but CONSTIFY or RANDSTRUCT-related hunks are arguably special.

Just out of curiosity, what changes did you upstream and what was
communicated to you about them being disruptive?

Constify is quite special-purpose, yes. Randstruct less so (and
linux-next now only needs a few more fix-ups, I've upstreamed the rest
I've identified).

>> And even if you still believe that, making their patches private
>> doesn't reduce the amount of work they have to do. It just forces
>> people to pay them.
> Yes, that is true. Arguably, in some ways, since making their patches
> private increases the incentive to mainline hunks thereof, it
> _increases_ their work over some areas in the short and mid-term.

I wish grsecurity had responded to my requests for guidance about how
to make their forward-porting easier. Having that kind of work myself
in the past, I'll just continue to try to organize things in ways that
seems the least disruptive to me, and make suggestions to other people
about doing the same.

-Kees

-- 
Kees Cook
Pixel Security

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[PaX Team]

i think i'm not the only one to notice that the past two weeks
have been an interesting experience to say the least. ever since
our announcement about the fate of our test patch series we've
been silent but not because we had nothing to say (as you've no
doubt guessed by now, this mail is it). rather, we've used the
opportunity the announcement created to see just what comes out
of the woodwork. you see, we've always wondered about this kernel
security game, who wants what, who's doing what, who's p(l)aying
whom, etc. in other words, we wondered if the participants of the
game would show their true colors in the situation. and show they
did, it was well worth the wait.

as far as i could piece together the narrative propagated by Kees
Cook of Google, Greg Kroah-Hartman of the Linux Foundation, LWN's
Jonathan Corbet, the Linux Foundation's CII, and other less known
players, the story is that said entities have found (some of them
after years of ignoring the problem) that their security needs
could be best solved by upstreaming our code but all their attempts
to work out a deal with us for said purpose have failed and thus we
are to blame for everything under the sun and then some. and if
that weren't enough, they then went on the offensive with lots of
self-congratulating bravado and outright attacks on us and our
projects.

one can't help but notice that behind all the garbage spewed at
us, someone was getting very worried about the fate of the KSPP
fork of our work. what i can tell them now is that they will have
a whole lot more to worry about once they finished reading our
response. before we'll get to particular claims, we'll first look
at some history to set the stage for some of our answers.

my side started in 2000 as a hobby project that i worked on
entirely in my free time. i used to jokingly refer to it as a
'weekend project' as that's literally when i had time for it
due to my day jobs. later years didn't change much except for
the periods when i was without a job (amounting to over a decade)
and thus had more free time on my hand. Brad started his project
a bit later than me and has always worked on it in his free time.

as a true community project (KSPP's is very far from it, see
below) we have received lots of generous help over the years in
the form of hardware and financial donations that however didn't
quite cover our costs so at some point we were forced to run the
project in a more formal manner to be able to accept corporate
sponsorship (this is what OSS was originally created for). at
some point we also decided to begin supporting selected kernel
versions for a longer period of time in addition to the latest
upstream kernel. this proved to be a nice bonus and even
incentive for corporate sponsors.

all was well until a few years ago when we learned of several
blatant copyright violations (not of just our code but by extension
that of the upstream kernel too) and also trademark violations.
pursuing these wasn't too successful and in order to prevent
further abuse we decided to stop the public distribution of
our stable patches and provide it as an entirely commercial
product from then on (all still under the GPL, despite much
speculation to the contrary).

this proved to be a better decision and all seemed well for a few
months until the Washington Post article on kernel security appeared.
this turned out to be a game changer event in that linux security
got a serious look at from high level corporate executives and
resulted in Google tasking Kees Cook to start to work on the KSPP
with the express purpose of upstreaming grsecurity - all without us.

at this point we'd like to make it clear that any statements from
Kees/Greg/etc to the contrary are simply *lies*, neither i nor Brad
have ever been contacted by any of the corporations paying for the
KSPP. since proving a negative is hard, instead hereby we give any
corporate attorney, executive (or any other officer who thinks
that they gave us an offer that we refused) permission to post any
private emails including financial details, contract terms, etc that
prove us wrong. as a sidenote, there's some irony in that Kees and
Greg managed to tell contradicting lies as according to one we refused
to get paid to upstream our code whereas the other said that the
Linux Foundation had funded 'members of the grsecurity team'. in
reality, neither happened as such offers that we could refuse or
accept have never been made. it's also telling that when upstreaming
our code came up in private conversations where Kees was present,
his best suggestion at the time (around 2015 summer) was to try
the freshly expanded CII instead of, say, his own employer Google
(last month after he got an early notification about our plans
regarding the test patches, he in fact admitted that Google was
never going to pay us for the upstreaming work).

the CII is also an interesting animal in that they can't seem to
figure out what exactly it is they want. during the summer of
2015 i was CC'd into an email thread on cii-discuss where some
of our users suggested that the CII take up funding our projects.
at the time the CII stated that they would not provide funds
that'd amount to effective employment and also that they'd be
only interested in upstreaming our work. when i asked if they
were then willing to fund thousands of hours of such work, i got
no response and left it at that. now imagine my surprise when i
saw their recently published annual report for 2016 which has
this following beauty in there:

   One challenge we’ve encountered this year is finding skilled
   people to take on the work. While the desire to work on
   open source exists, without compensation it’s simply not
   feasible for many developers to do so. Emese Renfy, for
   example, had to step back from Kernel Self Protection
   Work because funding from CII took so long to approve.
   We’ve worked to resolve this with our new online grant
   system. Another solution is to find those who are already
   working on open source as a hobby and allow them to
   continue their work at a fully-funded, professional level. For
   example, Chris Lamb and Ximin Luo, have been able to give
   up their day job to focus exclusively on the Reproducible
   Builds project.

so it now looks like that funds amounting to effective employment
are possible, too bad they never told me about it (this is of course
in direct contradiction to what they claimed in their blog and i
again give them permission to post any emails/etc that prove the
contrary). on a somewhat amusing sidenote, i can't help but notice
that someone at the CII must have taken lessons from the KSPP in
their copy-pasting skills as they managed to get Emese's last name
wrong not once but twice then repeated the same in their blog for
good measure.

so what do we have so far about this KSPP business? Google and
other companies made a decision to get our work upstream using
their own resources only - so far so good, it's their business
in every sense of the word. not involving us however turned out to
be an unwise decision when they realized that their employees are
completely unprepared for this work (we'll discuss some examples
later). this then led to all kinds of social engineering attempts
at trying to get us to help them out - all without getting paid
for our time, as if we were just supposed to drop all other work
and spend our free time on fixing their botched up attempts. as
if this still wasn't disgusting and outragous enough, they went
on the combative and had the guts to accuse us of not wanting
to cooperate without also mentioning that they expected all this
work from us for free taken away from our free time. at this point
it should be clear that the KSPP is anything but a community
project. rather, it's a joint effort of commercial companies
such as Google/Intel/etc to spend their corporate resources on
ripping our code and abusing our hard-earned reputation *and*
have the audacity to expect us to assist in all this using our
free time.

this state of affairs made us realize that if this is what this
'community' wants, then they shall get it. now they're on their
own and by looking at all the reactive mud slinging ever since,
they're terrified about the fate of their fork. based on the
level of incompetence they showed so far, it's not hard to see
why. this brings us to second part of our response where we'll
take a look at some of the claims Kees made and show them for
what they really are.

> It does underscore the critical need to upstream stuff, though.
> Forks of projects might disappear at any time. :(

this is somewhat rich coming from a Google employee considering
the fate of various linux forks used in their devices. one may
also wonder what will happen to the KSPP fork, especially now
that Fuchsia is also set to obsolete all that work and remove
the incentive to fund it much longer.

upstreaming stuff also isn't at all enough to prevent code
from disappearing, one needs to look no further than the recent
removal of avr32 support due to bitrot. considering the force
that sustains the KSPP (corporate money and interest) one can
guess how long said code will survive bitrot vs. our track
record that is the result of a fundamentally different kind of
motivation.

as another counterexample to why that 'critical need' can be
overrated, one can also look to the unprivileged ping code
(somewhat ironically, it's a security related feature, not
unlike the topic here), introduced on this very same list
years ago by Vasiliy Kulikov via a Google Summer of Code
project. it was later maintained by upstream developers who
then introduced several exploitable vulnerabilities in it.

> Additionally, while PaX Team, grsecurity, and ephox's work
> were technically separate development efforts (for example, just look
> at how PAX_USERCOPY differed between PaX and grsecurity),

FYI, USERCOPY wasn't separate efforts at all, we worked together
on USERCOPY and decided to carry certain features in grsec only.
this is a typical workflow for us, there've always been things that
started out in PaX and were made into a complete feature in grsec
or where i took back some code into PaX that i considered needed
infrastructure/features in there.

> Looking at the results in grsecurity, though, it's clear that they
> chose to integrate with upstream instead of maintaining a forked
> implementation.

i think you got that backwards, our code is the original, the KSPP
(and consequently upstream's) copy is just that, a fork (look at
where you get the gcc plugin updates from among others). that said,
why would we keep two similar copies of our own code around? as for
'integration', what i actually did was to clean up the mess you
created, just look at the bits i 'awarded' with CONFIG_BROKEN_SECURITY
and some more examples below.

> This is further supported by grsecurity being paid by CII to upstream
> the gcc plugin infrastructure,

correction, grsecurity, which according to your own definition is
a kernel patch, can't have been and thus never was paid by the CII,
Emese was (and she isn't a grsecurity developer any more than i am
an upstream linux developer).

> I think it's an entirely false claim that upstream is creating
> more work that normal forward porting.

and i think you're wrong on that (see below). it's in fact yet
another example to show your lack of understanding of what some
of our features do.

> Finally, even if you can somehow disregard the thousands of upstream
> changes benefiting grsecurity[...]

what exactly benefits *grsecurity* (you know, what you defined
as a kernel patch) and not kernel users in general (which then
means that said changes have nothing to do with us)? i can't think
of much if anything in recent memory.

> they have benefited from the upstreaming review by upstream
> finding bugs in various grsecurity features

oh yes, this myth that just can't die. we'll see in about a second
what exactly those 'bugs' are.

> Arnd Bergmann alone found tons of issues with the initify plugin
> and grsecurity fixed them)

actually that was something like 4 bugs in the plugin and we
didn't fix them, Emese did.

> Hardened usercopy found bugs in grsecurity's slab implementation,

now we're getting somewhere, let's dive in.

the first 'bug' was that *4.6* introduced red_left_pad to SLUB
objects which needs to be taken into account when computing
object boundaries and my code didn't. what you forgot to tell
the world at the time (and ever since) is that it was you who
took my code from *4.5* which didn't have red_left_pad and thus
was perfectly fine as is. the actual bug was that you blindly
copy-pasted my code (a recurring theme) without doing the one
job you had: maintain this code and follow upstream evolution.

the second 'bug' was the use of ksize vs. ->object_size which, if
memory serves, a slab maintainer insisted on using. what this
'bugfix' achieved in one fell swoop is that all SLUB objects with
padding can now be used to leak that padding which pretty much
destroys the purpose of (your copy of) USERCOPY. just think about
it, by definition that padding is uninitialized memory and thus a
prime target for memory leak attempts that thanks to your 'bugfix'
can now occur unimpeded. as an aspiring maintainer your proper
response should have been to explain this to the slab maintainers
and also why ksize is an abomination itself (its users by definition
exercise UB) and that the problem it sets out to solve should be
an API redesign instead (remove ksize and create a new allocation
function that takes the size as reference and updates it on return
so that the caller can learn how much memory it ended up getting).

so that's two bugs so far, except none of them is ours but yours
instead but nice try selling them otherwise. unfortunately the
buck doesn's stop just there yet, there're more problems with
this expert reviewed and maintained code (i wonder how on earth
we managed to survive this far without such expertise).

the third bug is that the comment describing __check_heap_object
is meaningless garbage as already allocated objects by definition
cannot be 'incorrectly sized' (at least not at this level,
SIZE_OVERFLOW would be something closer to that purpose) and of
course the purpose of USERCOPY has nothing to do with verifying
object sizes anyway. what USERCOPY does want to verify is the
size of the memory *copy* attempt (that is, the copy must fully
fall inside the kernel object).

this brings us to the fourth bug which is the verification of the
kernel object pointer for being 'impossible'. that of course was
never the purpose of USERCOPY as its design assumes a threat
model where an attacker controls only the size of the copy but
not the object pointer. if the latter is also assumed, all bets
are off, there's no way such checks can catch an actual attack
(checking pointer provenance is a very hard problem).

there're further problems caused by your 'improvements' on our
code. one is the needless change of the log message that breaks
existing log analyzers looking for a pattern. the other problem
is that the replacement of do_group_exit with a BUG is entirely
unnecessary as a USERCOPY event is recoverable, there's no need
to panic the whole system because of it (a bit like how a REFCOUNT
event is recoverable to an extent). i also wonder what locks would
be broken by BUG when copy*user functions can sleep and thus are
not supposed to be called under locks anyway. (and if you didn't
mean spinlocks then how would BUG handling break them?)

last but not least, a quick look at your claim to fame due to:

> [USERCOPY] gained strncpy_from_user() coverage [...]

this is an entirely pointless exercise as an audit of the few
callers of said function can show their correctness instead. so
your 'coverage' adds nothing but pointless performance impact
to it. FWIW, similar considerations made us not check copying
with compile-time constant sizes at the time.

i believe this brief overview at your achievements regarding
USERCOPY shows that instead of fixing any bugs in our code, you
in fact added a bunch of them.

> The current refcount_t work based on PAX_REFCOUNT uncovered a
> crash bug in grsecurity's implementation

for the record, it was a bug introduced in the PaX 4.9 port only
and you failed to spot it when copying it to a newer kernel (you
also failed to notice what the underlying problem i wanted to fix
was and thus didn't fix it yourself either at the time). note also
that the reporting problem i tried to fix there (and fixed it
differently in later patches) is still not properly addressed in
your refcount port but at least you documented it now.

> Upstreaming grsecurity features brings a huge amount of testing
> to bear on the code, which, like all the other things in upstream,
> grsecurity directly benefits from.

not all things upstreamed benefit us (or other users for that matter).
just look at VMAP_STACK (upstream NIH'd version of Brad's KSTACKOVERFLOW
feature) and the entirely avoidable damage (and security bugs) it
caused to end users ever since its release.

> And to top all of this off, while upstreaming the latent_entropy
> plugin, I noticed a English typo that was present in all the
> grsecurity gcc plugins, and when I sent them a trivial spelling
> fix patch for all their plugins[...]

if only it had been all plugins... clearly you never tried a recursive
grep and thus missed the same typo in RAP and SIZE_OVERFLOW.

> [...] (rather than just letting it stand and making work for
> them to catch it during forward porting and fix it everywhere
> else), they publicly mocked the patch (which they applied).

given the above, it'd have been better to just send an email about
the typo and let us fix it ourselves properly which we ended up
doing anyway instead of using your incomplete patch.

> Upstreaming can be extremely time consuming. Grsecurity made it clear
> from long ago that they had no intention of upstreaming things because
> they wanted to use their time differently.

not true at all. what we've always said was that upstreaming our
code requires so much time that noone can reasonably expect us to
do it in our free time. that the KSPP exists entirely by corporate
decree is proof of that. what would actually limit any potential
upstreaming effort on our side is the need to balance our time
spent on upstreaming vs. time spent on new R&D. come to think of
it, if the companies running the KSPP had played it smart, they'd
have gone out of their way to engage us as that's the only way
they'd have a chance to close the ever widening feature gap.

> That said, when I asked grsecurity if there I was any way they would
> accept payment to upstream things, they did briefly agree and
> upstreamed the gcc plugin infrastructure. So they were willing to
> upstream if paid, yet ultimately decided to stop, and to continue to
> forward port their patches.

i don't know who you asked as 'grsecurity' but the only thing i
recall was some private conversations back in 2015 where you
suggested to try to get the CII to fund an upstreaming project
(where you yourself had no saying) and that turned out to be a
downer. you also knew full well that Brad wouldn't have anything
to do with the CII due to one of their member companies (that
entirely coincidentally is also behind the KSPP) having violated
his trademark. one can't also help but wonder why you didn't
offer Google's money? because you have never been authorized by
Google to act on their behalf (and thus can't have possibly made
any such offers)? so i'm not sure what you think you did to get
us 'accept payment to upstream things' but you've clearly never
been in a position to make such an offer (and no, poorly disguised
social engineering attempts don't count).

by the way, it wasn't Brad who upstreamed the gcc plugins stuff
but Emese (she's neither a PaX nor a grsecurity developer).

> [...] I again summarily reject the notion that upstreaming
> grsecurity features creates "more work for grsecurity without
> value in return".
> [...]
> To your specific examples, __ro_after_init is literally a one-line
> change: they just make in __read_only. This was some of my first
> attempts to make their forward porting work easy while upstream slowly
> incorporated features.

this is false and only shows that you don't at all understand what
__read_only really does. in particular, __read_only is enforced
much earlier in modules (so much for it being 'after init') and
thus all writes to such objects must be instrumented. this means
that all new __ro_after_init uses must be carefully audited and
any writes to them instrumented. this is anything but 'literally
a one-line change' (though i can at least get the compiler to do
the work for me to an extent).

> The work around PAX_MEMORY_SANITIZE made the slab debug paths
> faster for everyone.

i wonder, how many systems actually enable those slab debug features
at runtime due to their performance impact.

> So, neither of us can speak for grsecurity, but I reject your belief
> that upstream has somehow created needless work for them.

fortunately it's got nothing to do with belief but simple facts
that happen to contradict your belief.

> Which brings me to a question I haven't seen anyone ask yet: why does
> grsecurity exist?

thanks for your concern for our existence but i think you should
ask why the KSPP exists instead and how much longer it will be
around. based on your statements here and in later mails it seems
that you're very worried that your ability to copy-paste yet more
of our code will diminish over time. unfortunately for you, putting
up a fake happy face and pretending nothing happened won't actually
make it any better for the KSPP.

back to your question, you don't have to guess and can instead just
look at what world class domain experts think of our work in their
testimonials on the grsecurity website. it seems to me that you and
the KSPP are stuck in the world of linux whereas our work has
affected much more than that already. so in a sense you're right to
say that we don't care about all linux users since we actually care
about everyone else too and use our projects (the code, conference
presentations, blogs, etc) to showcase defense technologies that
everyone can benefit from, not just linux users. when will the KSPP
reach this level, if it's on the corporate agenda all, that is? and
if all this counts for nothing for you then consider our existence
as the apparent enabler of your career.

> You're speaking entirely in theoreticals, but I understand what you're
> trying to say. All forward porting runs this risk. Kernel internals
> change in ways that threaten even unchanged grsecurity features. To
> bring this down to earth, I would ask "how does grsecurity perform
> testing?" I can point to the many ways how upstream performs testing,
> including LKDTM for several of the security-sensitive features. As
> seen in the PAX_REFCOUNT porting work, the 4.9 grsecurity patch was
> clearly never actually tested since _any_ exercise of the PAX_REFCOUNT
> protection would Oops the kernel.

you're wrong, i actually tested the code just not a last-minute
one-liner change that was 'obviously correct' (well, it wasn't,
it happens). on the other hand it's pretty funny that you pointed
out this one bug as it was you who blindly copy-pasted it *and*
never tested it yourself otherwise you would have found it, not
Jann Horn. another funny thing is that you clearly never understood
the REFCOUNT reporting code as LKDTM didn't even exercise those
paths at the time. last but not least, the actual effect of the
bug wasn't necessarily an oops, it all depended on how the offsets
were encoded in the conditional jumps (as i explained all this in
my response to Jann and you).

> [...] but I can point you directly to how upstream tests, how upstream
> developers find bugs in grsecurity features, and how things can improve once
> upstreamed (again, for example, the massive uaccess consolidation).

yep, things improve indeed, like the new vulnerabilities caused by
ping sockets, VMAP_STACK, HARDENED_USERCOPY, SLAB_FREELIST_RANDOM,
HARDENED_REFCOUNT, etc...

> I think I have thoroughly disproved this position. Something that I
> think is hard to see for people not involved in day-to-day upstream
> work is how very different the development workflows are between
> upstream and grsecurity. Upstream is normally evolutionary, doing
> things in easy to digest pieces, where as grsecurity could just land
> massive changes between releases.

believe it or not but we also develop our features step-by-step, not
unlike how any sane developer, upstream or not, does it.

> These are all complaints to be made to grsecurity. They are the ones
> who provides those features originally, did not upstream them, and
> then took them away.

i'm wondering, when are you going to stop the hypocrisy and this
rhetoric of blaming us for not upstreaming our own code in our free
time while the KSPP companies have been happily paying you and
others for doing the same?

> > So, here's my list of questions for the KSPP:
> > 1/ When will I be able to switch to a vanilla Linux kernel that is
> > equivalently hardened as a grsecurity/PaX kernel used to be?
>
> How could anyone answer that question? I can't see the future.

well, it seems that not that long ago you did see the future as:

  grsec is going to stay a decade ahead of anyone else (not just Linux)

> And besides, it's not like grsecurity was providing comprehensive
> protections. If you're using arm64, you might feel like you're already
> in a better position with upstream (got PAN emu, got hardened
> usercopy, still no RAP).

actually, by definition you can't be in a better position with
upstream unless you're claiming that we actually reduce security
with our code (please, do bring up my disabling of KASLR, happy
to discuss it again). in fact due to the bugfixes we have for
your USERCOPY fork (that is, we kept our own working code), our
code *is* actually in a better position already.

as for RAP, it works on any arch by design, i just don't enable it
on archs where i haven't fixed all the function pointer abuse yet
and marked up asm code (both arm and arm64 work already here).

> Upstream's goal is protecting as many people as possible.

the KSPP's goal is to further the agenda of the companies behind
it (which is extracting profits for shareholders). that has nothing
to do with "protecting as many people as possible" but everything
to do with business goals du jour. if what you claim was true,
they would have done it since the beginning and in a way that is
not restricted to only linux users.

> > 2/ Who will maintain this code and how?
> > 3/ Who ensures the coverage and quality won't suffer for each new
> > kernel release?
>
> The upstream development community, just like everything else. I
> answered both of these already above.

the correct answer is 'whoever the KSPP companies put on the job'
though their performance so far doesn't leave one with great
confidence as to the sustainability of the project.

cheers till the next round,
 PaX Team

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Daniel Micay]

KSPP is going to change without public patches. There was a substantial
community built around PaX and grsecurity. Hardened Gentoo, Alpine and
Arch Linux all had/have official support integration and a significant
number of users. Other people used it on their own. Few of those users
are going to have access now so the best we have is upstream work (KSPP)
and maintaining stuff downstream if it hasn't landed yet or is blocked
by politics, etc. People aren't going to accept only having a 4.9 LTS
with the last public patch. New hardware support is already chipping
away at the usefulness of that.

I agree with a lot of what you're saying, but that doesn't really matter
anymore. I care about code, not politics and historical grievances. I
think PaX / grsecurity has almost always been on the right side of these
feuds but that changed when people trying to improve security upstream
by landing changes from PaX / grsecurity or even just new code became
the enemy. People get banned from the community (IRC) for even stating
that they want to help improve upstream security.

I was quite happy with the grsecurity patches so I lacked the motivation
to make upstream contributions. I only tried to push things in the right
direction on the mailing lists, which ended up getting me banned from a
community that I put a lot of time/effort into. Other than the few with
access to the commercial patches, the community either has the choice of
staying loyal to you and having nothing or using and contributing to the
remaining public efforts.

In terms of funding contracting out work isn't how companies like Google
do things. They hire people and do everything in-house, even if that
means losing on lots of talented people not willing to do things their
way. CII / OTF are the exception to the rule and then your funding would
be subject to politics and could vanish at any time even if you succeed
in getting it from them initially. They didn't single out grsecurity for
this treatment, it's how they do things in general. We were encouraged
to apply for OTF funding for CopperheadOS by people who supposedly had
influence there and were rejected based on astoundingly stupid reasons
after wasting time on it. AOSP is primarily permissively licensed, so we
moved away from FOSS licensing -> non-commercial usage licensing for
everything but the kernel due to getting nothing back from it while
companies were taking and not giving back. Unfortunately, you don't have
that option. I've always tried (and mostly failed) to get changes
upstream into AOSP though. I'm not worried about running out of features
to implement and it makes it easier to maintain since that stuff doesn't
need to keep being ported forward, although there's a lot more time
pressure to port than there is for Linux due to how it's released.

The VMAP_STACK feature is upstream, so while reinventing KSTACKOVERFLOW
was silly and rediscovering the same bugs even more so, that time ended
up being wasted and the temporary stability hit was accepted. It's too
late now and the upstream feature is going to be perfectly good. KASLR
is a weak mitigation, but people decided to spend their time on that and
it exists so I don't see the point in not getting the marginal utility
it offers rather than disabling it or marginally useful checks in the
USERCOPY code via CONFIG_BROKEN_SECURITY, where you're taking something
away from your users essentially out of spite. I also don't understand
the point of maintaining stuff like KSTACKOVERFLOW and a bunch of other
features once the upstream solution works fine. Why keep a separate ASLR
implementation with a legacy random number generation scheme vs. a few
small changes to the upstream one, etc?

https://github.com/thestinger/linux-hardened now exists because I am
quite aware of the limitations of upstream work and I don't plan on
waiting years to have features like RAP and page table protection.
Still, it's only maintainable for us if we actually split everything up
and document it which is half of the way to upstreaming everything. It
makes sense to try to land as much as possible upstream for free code
review / testing and a reduction of the maintenance burden. The goal
isn't to upstream as much as possible, but rather than the means to an
end which is getting back what we already had available via the work you
published. If you can't tolerate people doing stuff like this, you're
going to end up considering a lot of your old community as enemies... I
don't see what else we are supposed to do. There's no way for the PaX /
grsecurity commercial model to work for regular users.

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

PaX Team,

I've always valued your thoughts and contributions. Your work has made
the world a little more secure. I'm doing my best to make the world a
lot more secure -- and it's nice when I get to follow your lead. You
can think or say what you want about me, even if it's untrue, but I
(and many others here) are still very willing and hopeful to
collaborate with you. If a financial arrangement is desired, I believe
my employer continues to be willing to explore it now just as they
have in the past.

It's unfortunate that the discourse has dropped to this level and
continuing at that level isn't going to help. I don't really have much
new to say beyond what I've already posted, but I'd ask that you take
your harassment and antagonism somewhere else; we're trying to get
work done.

-Kees

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Hunger]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

On Thu, May 11, 2017 at 8:02 PM, Kees Cook <keescook@chromium.org> wrote:

> Your work has made
> the world a little more secure. I'm doing my best to make the world a
> lot more secure


PaX and grsecurity had an influence on the whole industry. Every modern OS
have NX support and ASLR now from Apple to Microsoft. PIE binaries and
mprotect restrictions in Apple iOS. PaX UDEREF and KERNEXEC features
implemented in hardware level, like ARM PXN, PAN and Intel SMEP, SMAP
(still weaker protection than PaX btw). PaX RAP is leading in Control Flow
Integrity.

So how you make the world a lot more secure while just copy+pasting the 10+
years old grsecurity and PaX features and innovating nothing new?


> If a financial arrangement is desired, I believe
> my employer continues to be willing to explore it now just as they
> have in the past.
>

If i read it correctly, pipacs and spender got no financial arrangement or
any contact attempt about it in the past, so what will your employer
continue?

[-- Attachment #2: Type: text/html, Size: 1515 bytes --]

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Solar Designer]

Hi all,

I've just re-read PaX Team's lengthy message from May (back then, I only
skimmed it).  I felt I owed PaX/grsecurity at least as much to hopefully
understand their position better, which I admit I still don't understand
very well.  But that's not why I post this belated reply.  Rather, I
found in the message a detail I feel I have to correct as the issue is
partially my responsibility and I want to share the lessons learned:

On Thu, May 11, 2017 at 03:24:35AM +0200, PaX Team wrote:
> > It does underscore the critical need to upstream stuff, though.
> > Forks of projects might disappear at any time. :(
[...]

> as another counterexample to why that 'critical need' can be
> overrated, one can also look to the unprivileged ping code
> (somewhat ironically, it's a security related feature, not
> unlike the topic here), introduced on this very same list
> years ago by Vasiliy Kulikov via a Google Summer of Code
> project. it was later maintained by upstream developers who
> then introduced several exploitable vulnerabilities in it.

The exploitable vulnerabilities that I think PaX Team is referring to
were introduced during forward-porting of the patch prior to it getting
accepted upstream.  Thus, in one way yes the bugs were part of the
upstreaming effort, but in another way no they were not by upstream
developers.  Rather, in hindsight neither Vasiliy nor I (his GSoC
mentor) were familiar enough with the target kernel version's APIs (the
forward-porting was from 2.4.32 to 3.0), and we didn't perform enough
testing (not of obscure cases not occurring in ping(1), such as
disconnecting and reconnecting the same ping socket).

Those bugs were found by different third-parties through fuzzing, and
fixed by upstream kernel developers.

Lessons (re-)learned include:

1. Tough to make non-trivial contributions to the kernel (especially by
"outsiders" or/and after having been "on hiatus") without introducing
bugs.  Even if tests appear to pass at first (ping(1) worked fine via
the new sockets, including if invoked repeatedly and concurrently).

2. Kernel APIs are very brittle, locking conventions are non-obvious.

3. Kernel hardening efforts should include making the APIs less brittle
or/and to have things fail in less dangerous or at least less likely
exploitable ways.  In fact, in the aftermath of one of these bugs the
list pointer poison values have been adjusted not to be valid addresses
(which I think had been done in grsecurity before).  The refcount
overflow work is also relevant.

4. Forward-ports (and perhaps distro vendors' back-ports too?) tend to
receive less scrutiny than brand new code, but bugs are just as easily
introduced (or even easier if the porting isn't by the original author).
Patch made to apply and the feature works, so we're OK?  No, we are not.

5. One thing we did right was making the feature unavailable to non-root
by default.  We knew we'd be increasing the attack surface otherwise,
and instead we wanted to introduce this layer of privsep - having ping
SGID to a group (instead of SUID root), and having it use the new ping
sockets, without exposing those sockets to potential exploits directly.
This caution paid off.

6. As expected, some systems ended up exposing the sockets to users
other than host root anyway.  I expected this in container setups, where
container root would have access (can't do much about this while also
keeping root as powerful in their container as root would normally be on
a Unix system).  I didn't expect it, but Android also exposed the
sockets to all users.

7. The advances in kernel syscall fuzzing are great.  These days, if
similar bugs get in I hope most would be found by someone running a
fuzzer sooner (the ping socket bugs were found over quite a long period
of time).  Preferably, contributors themselves should run focused
fuzzing efforts on the syscalls, socket types, etc. they add or modify.
And non-focused too, so that interactions with other calls are exposed.

Alexander

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Kees Cook]

On Tue, May 2, 2017 at 12:06 PM, Lionel Debroux <lionel_debroux@yahoo.fr> wrote:
> Hi,
>
>
> Thanks Mathias for bringing up the matter, despite the way the attempt
> was bound to be received by upstream, let alone trolls...
>
>> > Assuming there are people able and willing to judge and review the
>> > code base for required changes (e.g. atomic_t -> refcount_t changes
>> > or function pointer cleanups for RAP), how would those be able to
>> > simply get in those changes at, say, time of rc5? -- without having
>> > to argue with a horde of maintainers of the individual subsystems
>> > involved?
>> This is an upstream development workflow problem that is especially
>
>> frustrating for security work since it can regularly touch lots of
>> areas, but it's one that I've been working to resolve. It's actually
>> on my list of things to discuss at the next Kernel Summit, since
>> making these wide changes is really irritating if it's going in via
>> each maintainer.
>
> Really irritating because really time-consuming, indeed. There's a
> genuine issue here. I too agree that mainline needs to grow a more
> efficient, more streamlined way to integrate a number of smaller
> cleanups and improvements, which _currently_ bring minor or even no
> functional changes on a mainline kernel. Creating such a process would
> be a major improvement.

Yup, and being able to show the cases for this is what I'm hoping to
do at Kernel Summit. I currently think that for minor things, having a
process to notify maintainers (seeking their Ack), but committing the
changes if there is no NAK would be the best approach. This is how
similar large changes happen now (see the uaccess conversion, for
example), but it requires such a tree be maintained by someone that
Linus will trust to pull from. I'm hoping to be that person, but I'll
need to demonstrate the need with a few examples, and build that trust
by doing it safely for a while. I've already started this a little bit
with my KSPP tree that is pulled for -next. From this tree I've been
carrying hardened usercopy changes (which touched lots of
architectures initially), gcc plugin changes (which touch lots of
files for annotations), and lately the CONFIG_DEBUG_RODATA renaming.
The trick is finding the right balance and making sure maintainers
have a chance to see things. The massive refcount_t conversion, for
example, while really time-consuming has resulted in a lot of
excellent discussion, identified many places where things were
refcounting incorrectly, etc. In those cases, the "via maintainers"
approach continues to make sense, even if painful.

>> If you have time to criticize, you have time to write a patch.
>
> True, and as you pointed, Mathias did.
>
> I can't speak for others, but right now, the prospect of spending some
> of my free time making the jump from
> browsing through the grsecurity patch and trying to determine which
> hunks relate to what and how they're useful to
> actually making + testing + writing a changelog explaining to
> maintainers not necessarily caring about security why the change

If it helps, the other people on this mailing list care very much
about security, so you're not alone in the effort of convincing
maintainers of things they might not understand. With that said, that
is fundamentally the cost of protecting the users of Linux. Linux is
successful because of the specialization and diversity of its
maintainers. In the same way not everyone can be a security expert,
not everyone can be a USB expert, or a filesystem expert.

> makes sense and how it changes the binary's size + submitting
> patches to mainline (with wrong credit, it's not my work in the
> first place) and most of all
> watch a sizable subset of these patches be either dropped on the
> floor without a comment (this is not a theoretical concern, that's

And this flaw in the upstream culture is why I started KSPP: there IS
a group of upstream developers that care about security, and we needed
a single umbrella to stand under to help each other with developing
and landing important defenses. Sometimes there is silence or
incredulity for a patch, but that is part of what I hope KSPP will
help shield contributors from.

> precisely what happened to most patches in Emese Revfy's large
> constification patchset years ago) or outright rejected because

Yup, that was a shame. And it's that past cultural problem I want KSPP
to help fix. And I think it's had some success there. Take the tty
hardening patch being recently discussed. It hasn't landed yet, sure,
but it wasn't ignored. Ways to improve it were found, etc. This is a
big shift, IMO. Maybe I'm seeing it wrong, but I think the culture has
moved.

> they _currently_ bring no functional change on a mainline kernel
> is... unappealing.

Agreed. At the end of the day, no matter how the upstream culture
improves, it is time consuming and frequently frustrating work to
upstream things. That's why I don't think less of anyone who doesn't
contribute. But when someone can spend the time criticizing things
without trying to actually fix it... that's pretty disappointing. A
lot of people see upstreaming just as developing code, but there's a
lot more to helping fix the culture. Chime in on threads to help
explain stuff, test code, etc. This is a community, and building it
constructively takes time and effort.

> Let's list several families of smaller cleanups and improvements made
> in PaX/grsec. You (Kees, Mathias) are fully aware about them but maybe
> some other readers aren't familiar with the contents of the monster
> patches:
> * using explicit struct members throughout the code, which is a
> prerequisite for making RANDSTRUCT (probably bound for mainline at some
> point, AFAIK) more broadly useful than if it's limited to a handful
> of whitelisted structs. grsecurity contains hundreds of such hunks. I
> think that 4 randomly chosen, simple and related samples are in
>
> drivers/net/wan/z85230.c;

Yup. And getting those into upstream isn't hard at all. Most
maintainers trivially take those patches, as you can see:

$ git log --oneline | grep -i 'use designated initializers' | wc -l
30

As for z85230.c itself, that appears not to get covered by an x86,
arm, or arm64 allyesconfig nor allmodconfig, since I didn't trip over
it when porting randstruct:
https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/log/?h=kspp/gcc-plugin/randstruct-next-20170418

And that's what I mean about incremental improvements. I hope to land
the core of rand struct soon, and as others find things that need the
designated initializers, they'll get fixed. And then we can move on to
the discussions about how to deal with unnamed structs in task_struct
(a gcc 4.6+ feature) for controlling what/where randomization is
possible, etc, etc. Without forward progress, there is NO progress.

And by the way, while porting randstruct, I landed
-Werror=designated-init and adjusted the plugin to emit that
annotation so mistakes could be found at build time:
https://lkml.org/lkml/2017/3/21/604

And then offered the improvements to grsecurity, which was met with silence:
http://www.openwall.com/lists/kernel-hardening/2017/02/01/9

> * building constant structures at compile-time (and making them const)
> rather than emitting both the room for that structure in a writable
> section (...) or on the stack, and the code to initialize that
> structure. That's related to CONSTIFY, it yields safer and usually
> smaller code+data. Over the years, overwriting needlessly writable ops
> structs proved to be used multiple times in exploit PoCs for mainline
> kernels, which often failed miserably on PaX/grsecurity (not just
> thanks to CONSTIFY, two or three among MEMORY_UDEREF, KERNEXEC and
> RANDSTRUCT foiled the PoC as well).
> See the hunks for (I think)
>
> drivers/cpufreq/sparc-us3-cpufreq.c
> drivers/gpu/drm/armada/armada_drv.c
> others in drivers/gpu/drm/ (constifying "*_max_ioctl(s?)" variables
> initialized with ARRAY_SIZE(a constant array of structs))
> drivers/media/platform/omap/omap_vout.c
> drivers/memory/omap-gpmc.c
> drivers/mfd/rn5t618.c
> drivers/net/ethernet/hisilicon/hns/hns_dsaf_misc.*
> drivers/net/wireless/ath/wil6210/pcie_bus.c
> drivers/net/ethernet/amd/xgbe/* (most of the hunks in these files)
> drivers/video/fbdev/aty/aty128fb.c
> net/can/gw.c
> include/net/sctp/checksum.h
> ... and hundreds of others;

Yup, this is a major area of kernel attack surface reduction. I
kick-started upstream work on this with the introduction of
__ro_after_init which could cover a subset of these changes, and it's
being used more and more. Some coccinelle work went into further
constification (and checkpatch work) of ops structures, and with the
port of __read_only and the constify plugin:
https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/log/?h=kspp/write-rarely
upstream is getting closer to having even more of this coverage. In
the meantime, these are all excellent individual patches to send
upstream. Here's a less trivial one I ported from grsecurity, again,
taking by the maintainer immediately:
https://patchwork.kernel.org/patch/9570905/

> * getting rid of some of the wrong casts between incompatible function
> pointer types. This often does not require adding wrappers, and
> sometimes even makes the code _more_ readable. These would be non-
> functional changes on a mainline kernel, and therefore especially
> likely to be ignored or rejected by maintainers. Again, hundreds of
> such hunks, which made up the bulk of the PaX & grsecurity patches'
> size increase when RAP was added. For instance, AFAICS:
>
> drivers/gpu/drm/*/psb_intel_sdvo.c and others: returning "enum
> drm_mode_status" instead of the less precise "int".
>
> drivers/media/dvb-frontends/cx*.c and others: likewise, returning
> "enum dvbfe_algo" instead of "int".
>
> drivers/net/wireless/atmel/atmel.c
> drivers/net/wireless/cisco/airo.c
> drivers/net/wireless/intersil/hostap/hostap_ioctl.c
> drivers/net/wireless/realtek/rtlwifi/pci.c
> (+ other WiFi drivers)
> arch/{arm,x86}/crypto/sha*glue.*
> fs/coda/dir.c

The function pointer type fixes are a huge clean-up and would have
appeared "unjustified" to maintainers. (For a sense of scale, just
look at the delta between the v4.4 grsecurity patch and the v4.5
patch. The bulk of the changes there were for RAP function pointer
type fixes.) However that's a matter of describing why it's a useful
change, and this is kind of thing the KSPP tree would be perfect for
carrying. No one has spent the time on these patches yet, since RAP is
pretty far down on the TODO list, especially given that it's
x86_64-only. But that doesn't mean the fixes wouldn't be welcome.

> * other samples of the untapped tidbits from PaX/grsec:
>
>   * removing spurious "static" keywords on function-local variable
>     declarations when the first thing the function does with the
>     variable is to reinitialize it. See e.g. the
>
> drivers/mfd/max8925-i2c.c
> drivers/mfd/tps65910.c
>
>     hunks, both of which were already in a grsecurity patch for 3.14
>     from May 2014 laying around on my HDD.

This sounds like a great coccinelle rule to generate and fix these
everywhere in the kernel.

>   * annotations in e.g.
>
> include/linux/seqlock.h
> include/linux/spinlock.h

It'd be great to find which testing tool noticed these being missing
and see if there are others. Again, please upstream them.

> Integrating into mainline hundreds of these smaller improvements and
> cleanups from PaX/grsec, i.e. even a single-digit percentage of the
> grsecurity patch's hunks, would demonstrably improve the mainline code
> immediately or in the mid-term... but that integration work is harder
> than it could be :(

And that is what KSPP is here to help with. We'll help new
contributors get this work done.

Thanks!

-Kees

-- 
Kees Cook
Pixel Security

Re: [kernel-hardening] It looks like there will be no more public versions of PaX and Grsec.

[Lionel Debroux]

Hi,


Thanks Mathias for bringing up the matter, despite the way the attempt
was bound to be received by upstream, let alone trolls...

> > Assuming there are people able and willing to judge and review the

> > code base for required changes (e.g. atomic_t -> refcount_t changes

> > or function pointer cleanups for RAP), how would those be able to

> > simply get in those changes at, say, time of rc5? -- without having

> > to argue with a horde of maintainers of the individual subsystems

> > involved?

> This is an upstream development workflow problem that is especially

> frustrating for security work since it can regularly touch lots of

> areas, but it's one that I've been working to resolve. It's actually

> on my list of things to discuss at the next Kernel Summit, since

> making these wide changes is really irritating if it's going in via

> each maintainer.

Really irritating because really time-consuming, indeed. There's a

genuine issue here. I too agree that mainline needs to grow a more
efficient, more streamlined way to integrate a number of smaller
cleanups and improvements, which _currently_ bring minor or even no

functional changes on a mainline kernel. Creating such a process would

be a major improvement.


> If you have time to criticize, you have time to write a patch.

True, and as you pointed, Mathias did.

I can't speak for others, but right now, the prospect of spending some

of my free time making the jump from

    browsing through the grsecurity patch and trying to determine which

    hunks relate to what and how they're useful

to

    actually making + testing + writing a changelog explaining to

    maintainers not necessarily caring about security why the change

    makes sense and how it changes the binary's size + submitting

    patches to mainline (with wrong credit, it's not my work in the

    first place)

and most of all

    watch a sizable subset of these patches be either dropped on the

    floor without a comment (this is not a theoretical concern, that's

    precisely what happened to most patches in Emese Revfy's large

    constification patchset years ago) or outright rejected because

    they _currently_ bring no functional change on a mainline kernel

is... unappealing.



Let's list several families of smaller cleanups and improvements made

in PaX/grsec. You (Kees, Mathias) are fully aware about them but maybe
some other readers aren't familiar with the contents of the monster
patches:
* using explicit struct members throughout the code, which is a

prerequisite for making RANDSTRUCT (probably bound for mainline at some

point, AFAIK) more broadly useful than if it's limited to a handful

of whitelisted structs. grsecurity contains hundreds of such hunks. I

think that 4 randomly chosen, simple and related samples are in

drivers/net/wan/z85230.c;


* building constant structures at compile-time (and making them const)

rather than emitting both the room for that structure in a writable

section (...) or on the stack, and the code to initialize that

structure. That's related to CONSTIFY, it yields safer and usually

smaller code+data. Over the years, overwriting needlessly writable ops

structs proved to be used multiple times in exploit PoCs for mainline

kernels, which often failed miserably on PaX/grsecurity (not just
thanks to CONSTIFY, two or three among MEMORY_UDEREF, KERNEXEC and
RANDSTRUCT foiled the PoC as well).
See the hunks for (I think)

drivers/cpufreq/sparc-us3-cpufreq.c

drivers/gpu/drm/armada/armada_drv.c

others in drivers/gpu/drm/ (constifying "*_max_ioctl(s?)" variables

initialized with ARRAY_SIZE(a constant array of structs))

drivers/media/platform/omap/omap_vout.c

drivers/memory/omap-gpmc.c

drivers/mfd/rn5t618.c

drivers/net/ethernet/hisilicon/hns/hns_dsaf_misc.*

drivers/net/wireless/ath/wil6210/pcie_bus.c

drivers/net/ethernet/amd/xgbe/* (most of the hunks in these files)

drivers/video/fbdev/aty/aty128fb.c

net/can/gw.c

include/net/sctp/checksum.h

... and hundreds of others;


* getting rid of some of the wrong casts between incompatible function

pointer types. This often does not require adding wrappers, and

sometimes even makes the code _more_ readable. These would be non-

functional changes on a mainline kernel, and therefore especially

likely to be ignored or rejected by maintainers. Again, hundreds of

such hunks, which made up the bulk of the PaX & grsecurity patches'

size increase when RAP was added. For instance, AFAICS:

drivers/gpu/drm/*/psb_intel_sdvo.c and others: returning "enum

drm_mode_status" instead of the less precise "int".

drivers/media/dvb-frontends/cx*.c and others: likewise, returning

"enum dvbfe_algo" instead of "int".


drivers/net/wireless/atmel/atmel.c

drivers/net/wireless/cisco/airo.c

drivers/net/wireless/intersil/hostap/hostap_ioctl.c

drivers/net/wireless/realtek/rtlwifi/pci.c

(+ other WiFi drivers)


arch/{arm,x86}/crypto/sha*glue.*


fs/coda/dir.c

* other samples of the untapped tidbits from PaX/grsec:

  * removing spurious "static" keywords on function-local variable

    declarations when the first thing the function does with the

    variable is to reinitialize it. See e.g. the

drivers/mfd/max8925-i2c.c

drivers/mfd/tps65910.c

    hunks, both of which were already in a grsecurity patch for 3.14

    from May 2014 laying around on my HDD.

  * annotations in e.g.

include/linux/seqlock.h 

include/linux/spinlock.h



Integrating into mainline hundreds of these smaller improvements and

cleanups from PaX/grsec, i.e. even a single-digit percentage of the

grsecurity patch's hunks, would demonstrably improve the mainline code
immediately or in the mid-term... but that integration work is harder
than it could be :(


Regards,

Lionel.