[Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[James Morris]

I'd like to propose a security topic, "Kernel Hardening" (or "Kernel Self 
Protection"), to discuss how we can better mitigate vulnerabilities 
arising from kernel bugs.

We have some measures in place, although we are really not doing 
everything we can, as demonstrated from time to time when vulnerabilities 
arise which are mitigated by protections in grsecurity (for example), but 
not by mainline.  Much of the necessary work has already been done in that 
project, and as many will know, there have been significant challenges 
involved in past efforts to bring these techniques into mainline.  In some 
cases, the performance hit has been too high for maintainers to accept, 
and I wonder if we can re-visit some of these cases, with new approaches 
or perspectives on cost/benefit.

There are also potentially promising approaches to mitigation with other 
technologies such as KASan and gcc plugins, as well as evolving hardware 
features.

The aim of this session would be to bring relevant core kernel maintainers 
together with representatives of the research community and figure out a 
way to work together to improve hardening and mitigation in the Linux 
kernel.  We'd discuss what gaps we currently have, and what code or 
techniques already exist that can be incorporated into mainline to close 
them.  We'd identify issues that maintainers may have and try and find 
ways to address those issues.  From this, I'd hope that we'd develop an 
overall picture of what needs to be done and a practical idea of how to 
move forward.  We may not necessarily resolve all issues in this session, 
but we can at least characterize them and go away and think more about 
them.

We could also talk to the Core Infrastructure Initiative folk if we 
discover potentially useful tasks with no owners -- they may be able to 
fund developers for them.  It would likely be useful to provide CII with a 
status report after the session in any case.

I'd recommend Kees Cook be involved, due to his existing efforts in kernel 
hardening.  I think it would be good to invite one or two expert security 
researchers in this area -- Kees would know who.  In terms of core kernel 
folk, I'd suggest Ingo and akpm, as a starting point.

Comments?


-- 
James Morris
<jmorris@namei.org>

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Jiri Kosina]

On Mon, 24 Aug 2015, James Morris wrote:

> I'd recommend Kees Cook be involved, due to his existing efforts in 
> kernel hardening.  I think it would be good to invite one or two expert 
> security researchers in this area -- Kees would know who.  In terms of 
> core kernel folk, I'd suggest Ingo and akpm, as a starting point.

I am pretty sure spender will also have a lot to tell us :p

This is partially sarcasm (due to his often vocally expressed opinions 
about our processess (and even code) with respect to security) and 
partially meant seriously (there is always a possibility that at least 
some misunderstandings will be clarified and we'll get closer to 
convergency of the trees).

As a very long time grsecurity user, I'd definitely be interested.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Jiri Kosina]

[ resending with proper list-member From: sorry for duplicates to those 
  directly CCed ]

On Mon, 24 Aug 2015, James Morris wrote:

> I'd recommend Kees Cook be involved, due to his existing efforts in 
> kernel hardening.  I think it would be good to invite one or two expert 
> security researchers in this area -- Kees would know who.  In terms of 
> core kernel folk, I'd suggest Ingo and akpm, as a starting point.

I am pretty sure spender will also have a lot to tell us :p

This is partially sarcasm (due to his often vocally expressed opinions 
about our processess (and even code) with respect to security) and 
partially meant seriously (there is always a possibility that at least 
some misunderstandings will be clarified and we'll get closer to 
convergency of the trees).

As a very long time grsecurity user, I'd definitely be interested.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[James Morris]

On Mon, 24 Aug 2015, Jiri Kosina wrote:

> On Mon, 24 Aug 2015, James Morris wrote:
> 
> > I'd recommend Kees Cook be involved, due to his existing efforts in 
> > kernel hardening.  I think it would be good to invite one or two expert 
> > security researchers in this area -- Kees would know who.  In terms of 
> > core kernel folk, I'd suggest Ingo and akpm, as a starting point.
> 
> I am pretty sure spender will also have a lot to tell us :p

He actually presented at the 2010 security summit:
https://grsecurity.net/spender_summit.pdf

I'm not sure he'd be interested in working with us on this, though.


- James
-- 
James Morris
<jmorris@namei.org>

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Linus Walleij]

On Mon, Aug 24, 2015 at 6:20 AM, James Morris <jmorris@namei.org> wrote:

> There are also potentially promising approaches to mitigation with other
> technologies such as KASan and gcc plugins, as well as evolving hardware
> features.

What I've discovered when running KASan the last few weeks is that
this points back to the question of tests ... I've been using Trinity
to find bugs, but it is more likely to kill itself or cause OOM than
trigger any boundary overrun bugs.

Kselftest may be helpful, but basically any loads that heavily
exercise the kernel internals are helpful to harden the kernel.
Some of these are custom test suites I suspect. Any good hints
for a simple embedded developer like me?

Linus Walleij

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Jason Cooper]

On Mon, Aug 24, 2015 at 02:29:41PM +0200, Linus Walleij wrote:
> Kselftest may be helpful, but basically any loads that heavily
> exercise the kernel internals are helpful to harden the kernel.
> Some of these are custom test suites I suspect. Any good hints
> for a simple embedded developer like me?

A great place to start is this paper:

  http://messenger.jhuapl.edu/the_mission/publications/Hill.2007.pdf

It's a walk-through of how NASA's JHU/APL facility stress-tests software
intended for deep space missions, including Messenger, New Horizons, and
STEREO.  Lot's of good concepts, no specific test harnesses, though.

thx,

Jason.

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Aneesh Kumar K.V]

James Morris <jmorris@namei.org> writes:

> I'd like to propose a security topic, "Kernel Hardening" (or "Kernel Self 
> Protection"), to discuss how we can better mitigate vulnerabilities 
> arising from kernel bugs.
>
> We have some measures in place, although we are really not doing 
> everything we can, as demonstrated from time to time when vulnerabilities 
> arise which are mitigated by protections in grsecurity (for example), but 
> not by mainline.  Much of the necessary work has already been done in that 
> project, and as many will know, there have been significant challenges 
> involved in past efforts to bring these techniques into mainline.  In some 
> cases, the performance hit has been too high for maintainers to accept, 
> and I wonder if we can re-visit some of these cases, with new approaches 
> or perspectives on cost/benefit.
>
> There are also potentially promising approaches to mitigation with other 
> technologies such as KASan and gcc plugins, as well as evolving hardware 
> features.
>

We also have to make sure that the compiler based approach work with
architectures other than x86. Archs like ppc64 have different memory
layout and features like KASan may not really map easily with the
layout. For example we may not be able to implement inline kasan
instrumentation on ppc64. Also we have issues with stack and
global out of bounds access check.

I would be interested in this discussion, if we are scheduling this for
ksummit. I work mostly on ppc64 memory-management subsystem and can
bring in details of challenges faced with KASan implementation on ppc64.

-aneesh

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 5:29 AM, Linus Walleij <linus.walleij@linaro.org> wrote:
> On Mon, Aug 24, 2015 at 6:20 AM, James Morris <jmorris@namei.org> wrote:
>
>> There are also potentially promising approaches to mitigation with other
>> technologies such as KASan and gcc plugins, as well as evolving hardware
>> features.
>
> What I've discovered when running KASan the last few weeks is that
> this points back to the question of tests ... I've been using Trinity
> to find bugs, but it is more likely to kill itself or cause OOM than
> trigger any boundary overrun bugs.
>
> Kselftest may be helpful, but basically any loads that heavily
> exercise the kernel internals are helpful to harden the kernel.
> Some of these are custom test suites I suspect. Any good hints
> for a simple embedded developer like me?

I agree with the sentiment here, but not with the language. Finding
flaws (which is what selftests, KASan, Trinity, etc do) isn't
hardening. Hardening is stopping the exploitation of flaws. The
hardening the kernel needs is about taking away exploitation tools,
not killing bugs. (Though killing bugs is still great.)

As an example, making the kernel code memory read-only means an
attacker cannot just directly change the kernel's execution path when
they use an arbitrary memory-writing flaw. (This feature is mostly
enabled via CONFIG_DEBUG_RODATA, and was very recently added to ARM,
though isn't at 100% coverage for all the physical memory aliases.)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
> On Mon, 24 Aug 2015, Jiri Kosina wrote:
>
>> On Mon, 24 Aug 2015, James Morris wrote:
>>
>> > I'd recommend Kees Cook be involved, due to his existing efforts in
>> > kernel hardening.  I think it would be good to invite one or two expert
>> > security researchers in this area -- Kees would know who.  In terms of

Many of the folks that are good at kernel exploitation don't want to
help us fix the situation. :)

I'd recommend Lee Campbell, he's got a solid bit of experience from
the offense side. I think we should extend an invite to spender and
pageexec as well. They've been on the cutting edge of this for
decades, and it would be silly not to invite them.

>> > core kernel folk, I'd suggest Ingo and akpm, as a starting point.

Perhaps also Linus and rmk? Some of the protections are very central
to the kernel (e.g. constification, "read-mostly", segmentation
through page table swaps or domains, etc). I'd also want Andy
Lutomirski around, as he's got a lot of deep chipset knowledge. :)

>>
>> I am pretty sure spender will also have a lot to tell us :p
>
> He actually presented at the 2010 security summit:
> https://grsecurity.net/spender_summit.pdf

This was his bullet list of things that grsecurity/PaX already does
and that should be in mainline (with my notes in parens). He suggested
it would take us 10 years to catch up. We're 5 years into that, with
only a few things partially off this list:

 Remove infoleaks
 o Symbol information (partial improvement via kptr_restrict)
 o Slabinfo (partial improvement with 0400 root perms)
 o PAX_USERCOPY (even if gcc fixed their sizeof bug, we'd be no where
near this plugin's level of protection)
 Remove RWX from kernel (in good shape on x86, started on arm)
 Protect sensitive data
 o Constify function pointers! (no where close to this plugin)
 o IDT/GDT/syscall table/etc (this is partially done, aliases are
writable still)
 o Vsyscall shadow table, see sgrakkyu's remote
SELinux-disabling exploit
(http://sgrakkyu.antifork.org/sctp_houdini.c, luto fixed this via
vsyscall emulation)
Protect against invalid userland memory
accesses in general (SMEP and SMAP on x86 get us this, but it can be
emulated or done differently e.g. ARM domains)
Make refcount overflows unexploitable (nothing done on this front)
 o Currently equivalent to use-after-free
kmalloc(sizeof(somestruct) * attacker_len) (some partial improvements
here to replace this anti-pattern manually)
 o See recent ethtool get_rxnfc() vulnerability
Basically, secure the kernel! Your super finegrained
security systems will thank you

This is far from a comprehensive list, though. The biggest value, I
think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
for constification and integer overflow.

> I'm not sure he'd be interested in working with us on this, though.

As I understand him, he's quite binary about his methods: either it's
done totally right, or it's not worth doing at all. With that in mind,
I think his position is clear, even if he's rather prickly. :)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 9:20 AM, Aneesh Kumar K.V
<aneesh.kumar@linux.vnet.ibm.com> wrote:
> James Morris <jmorris@namei.org> writes:
>
>> I'd like to propose a security topic, "Kernel Hardening" (or "Kernel Self
>> Protection"), to discuss how we can better mitigate vulnerabilities
>> arising from kernel bugs.
>>
>> We have some measures in place, although we are really not doing
>> everything we can, as demonstrated from time to time when vulnerabilities
>> arise which are mitigated by protections in grsecurity (for example), but
>> not by mainline.  Much of the necessary work has already been done in that
>> project, and as many will know, there have been significant challenges
>> involved in past efforts to bring these techniques into mainline.  In some
>> cases, the performance hit has been too high for maintainers to accept,
>> and I wonder if we can re-visit some of these cases, with new approaches
>> or perspectives on cost/benefit.
>>
>> There are also potentially promising approaches to mitigation with other
>> technologies such as KASan and gcc plugins, as well as evolving hardware
>> features.
>>
>
> We also have to make sure that the compiler based approach work with
> architectures other than x86. Archs like ppc64 have different memory
> layout and features like KASan may not really map easily with the
> layout. For example we may not be able to implement inline kasan
> instrumentation on ppc64. Also we have issues with stack and
> global out of bounds access check.
>
> I would be interested in this discussion, if we are scheduling this for
> ksummit. I work mostly on ppc64 memory-management subsystem and can
> bring in details of challenges faced with KASan implementation on ppc64.

Should we have a separate discussion for bug-hunting? KASan needs to
expand its architecture coverage, Trinity needs a maintainer, and
there's always new things happening with smatch and coccinelle.

I'd just like to keep bug-hunting entirely separate from
self-protection, as I feel they're distinct topics (though related).

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Andy Lutomirski]

On Mon, Aug 24, 2015 at 10:17 AM, Kees Cook <keescook@chromium.org> wrote:
> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
>> On Mon, 24 Aug 2015, Jiri Kosina wrote:
>>
>>> On Mon, 24 Aug 2015, James Morris wrote:
>>>
>>> > I'd recommend Kees Cook be involved, due to his existing efforts in
>>> > kernel hardening.  I think it would be good to invite one or two expert
>>> > security researchers in this area -- Kees would know who.  In terms of
>
> Many of the folks that are good at kernel exploitation don't want to
> help us fix the situation. :)
>
> I'd recommend Lee Campbell, he's got a solid bit of experience from
> the offense side. I think we should extend an invite to spender and
> pageexec as well. They've been on the cutting edge of this for
> decades, and it would be silly not to invite them.
>
>>> > core kernel folk, I'd suggest Ingo and akpm, as a starting point.
>
> Perhaps also Linus and rmk? Some of the protections are very central
> to the kernel (e.g. constification, "read-mostly", segmentation
> through page table swaps or domains, etc). I'd also want Andy
> Lutomirski around, as he's got a lot of deep chipset knowledge. :)
>

What is this chipset knowledge you speak of? :)

One thing that grsecurity addresses (partially or fully?  I haven't
looked that closely): we have tons of static, non-const data structure
that contain function pointers, and we can't make them const because
they get filled in when things are initialized.  Grsecurity mitigates
this with some combination of compiler plugins and pax_open_kernel,
but we could probably come up with a more straightforward solution.
We could add an ro_after_init section, or we could even have a section
for things that are const but are writable through a special function.

>>>
>>> I am pretty sure spender will also have a lot to tell us :p
>>
>> He actually presented at the 2010 security summit:
>> https://grsecurity.net/spender_summit.pdf
>
> This was his bullet list of things that grsecurity/PaX already does
> and that should be in mainline (with my notes in parens). He suggested
> it would take us 10 years to catch up. We're 5 years into that, with
> only a few things partially off this list:
>
>  Remove infoleaks
>  o Symbol information (partial improvement via kptr_restrict)
>  o Slabinfo (partial improvement with 0400 root perms)
>  o PAX_USERCOPY (even if gcc fixed their sizeof bug, we'd be no where
> near this plugin's level of protection)
>  Remove RWX from kernel (in good shape on x86, started on arm)

We still need to do something about the direct map, IIRC.  Or did we
already fix that?

>  Protect sensitive data
>  o Constify function pointers! (no where close to this plugin)

As above, this would be nice.  If we did this upstream, we could do
even better if we outright disallowed non-const function pointers (as
opposed to fixing them up semi-transparently as the plugin does).

>  o IDT/GDT/syscall table/etc (this is partially done, aliases are
> writable still)

I don't think the RO GDT patches ever happened.

FWIW, we can't really eliminate a writable GDT alias without killing
compat performance: set_thread_area rather critically depends on
writing to the GDT at context switch time.

>  o Vsyscall shadow table, see sgrakkyu's remote
> SELinux-disabling exploit
> (http://sgrakkyu.antifork.org/sctp_houdini.c, luto fixed this via
> vsyscall emulation)

Not done yet, because even modern binaries still have the thing
mapped.  We could devote two or three minutes of a KS session to
figuring out how to kill it off for real for modern binaries.

>
> This is far from a comprehensive list, though. The biggest value, I
> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
> for constification and integer overflow.

At the risk of poking a big elephant, I think we should do something
about perf, too.  Perf is really useful, but it's also a *huge* attack
surface.

--Andy

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Julia Lawall]

> One thing that grsecurity addresses (partially or fully?  I haven't
> looked that closely): we have tons of static, non-const data structure
> that contain function pointers, and we can't make them const because
> they get filled in when things are initialized.  Grsecurity mitigates
> this with some combination of compiler plugins and pax_open_kernel,
> but we could probably come up with a more straightforward solution.
> We could add an ro_after_init section, or we could even have a section
> for things that are const but are writable through a special function.

Would it be helpful/possible to make a substructure for the constant parts?

julia

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 10:28 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Mon, Aug 24, 2015 at 10:17 AM, Kees Cook <keescook@chromium.org> wrote:
>> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
>>> On Mon, 24 Aug 2015, Jiri Kosina wrote:
>>>
>>>> On Mon, 24 Aug 2015, James Morris wrote:
>>>>
>>>> > I'd recommend Kees Cook be involved, due to his existing efforts in
>>>> > kernel hardening.  I think it would be good to invite one or two expert
>>>> > security researchers in this area -- Kees would know who.  In terms of
>>
>> Many of the folks that are good at kernel exploitation don't want to
>> help us fix the situation. :)
>>
>> I'd recommend Lee Campbell, he's got a solid bit of experience from
>> the offense side. I think we should extend an invite to spender and
>> pageexec as well. They've been on the cutting edge of this for
>> decades, and it would be silly not to invite them.
>>
>>>> > core kernel folk, I'd suggest Ingo and akpm, as a starting point.
>>
>> Perhaps also Linus and rmk? Some of the protections are very central
>> to the kernel (e.g. constification, "read-mostly", segmentation
>> through page table swaps or domains, etc). I'd also want Andy
>> Lutomirski around, as he's got a lot of deep chipset knowledge. :)
>>
>
> What is this chipset knowledge you speak of? :)

You appear to enjoy fixing deep x86 madness. :)

> One thing that grsecurity addresses (partially or fully?  I haven't
> looked that closely): we have tons of static, non-const data structure
> that contain function pointers, and we can't make them const because
> they get filled in when things are initialized.  Grsecurity mitigates
> this with some combination of compiler plugins and pax_open_kernel,

Right. That's the constification and KERNEXEC pieces.

> but we could probably come up with a more straightforward solution.
> We could add an ro_after_init section, or we could even have a section
> for things that are const but are writable through a special function.

I don't think it would be that clean: there are many targets that
legitimately need updating at runtime, but should be otherwise
read-only. The idea with solving that is to use inline
open/close_kernel calls to make them writable briefly and in a way
that is ROP-defensive.

>>>>
>>>> I am pretty sure spender will also have a lot to tell us :p
>>>
>>> He actually presented at the 2010 security summit:
>>> https://grsecurity.net/spender_summit.pdf
>>
>> This was his bullet list of things that grsecurity/PaX already does
>> and that should be in mainline (with my notes in parens). He suggested
>> it would take us 10 years to catch up. We're 5 years into that, with
>> only a few things partially off this list:
>>
>>  Remove infoleaks
>>  o Symbol information (partial improvement via kptr_restrict)
>>  o Slabinfo (partial improvement with 0400 root perms)
>>  o PAX_USERCOPY (even if gcc fixed their sizeof bug, we'd be no where
>> near this plugin's level of protection)
>>  Remove RWX from kernel (in good shape on x86, started on arm)
>
> We still need to do something about the direct map, IIRC.  Or did we
> already fix that?

We have a lot of targets still, but we're headed (slowly) in the right
direction.

>>  Protect sensitive data
>>  o Constify function pointers! (no where close to this plugin)
>
> As above, this would be nice.  If we did this upstream, we could do
> even better if we outright disallowed non-const function pointers (as
> opposed to fixing them up semi-transparently as the plugin does).

I would love this. :)

>>  o IDT/GDT/syscall table/etc (this is partially done, aliases are
>> writable still)
>
> I don't think the RO GDT patches ever happened.
>
> FWIW, we can't really eliminate a writable GDT alias without killing
> compat performance: set_thread_area rather critically depends on
> writing to the GDT at context switch time.

If that's what's needed, I am fine killing compat performance. (We
could attempt to make this a CONFIG for those that expect to run
compat loads for their kernels.)

>>  o Vsyscall shadow table, see sgrakkyu's remote
>> SELinux-disabling exploit
>> (http://sgrakkyu.antifork.org/sctp_houdini.c, luto fixed this via
>> vsyscall emulation)
>
> Not done yet, because even modern binaries still have the thing
> mapped.  We could devote two or three minutes of a KS session to
> figuring out how to kill it off for real for modern binaries.

What uses it? I can run with vsyscall=none without problems...

>> This is far from a comprehensive list, though. The biggest value, I
>> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
>> for constification and integer overflow.
>
> At the risk of poking a big elephant, I think we should do something
> about perf, too.  Perf is really useful, but it's also a *huge* attack
> surface.

No disagreement from me.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Andy Lutomirski]

On Mon, Aug 24, 2015 at 11:01 AM, Kees Cook <keescook@chromium.org> wrote:
> On Mon, Aug 24, 2015 at 10:28 AM, Andy Lutomirski <luto@amacapital.net> wrote:
>> On Mon, Aug 24, 2015 at 10:17 AM, Kees Cook <keescook@chromium.org> wrote:
>>> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
>>>> On Mon, 24 Aug 2015, Jiri Kosina wrote:
>>>>
>>>>> On Mon, 24 Aug 2015, James Morris wrote:
>>>>>
>>>>> > I'd recommend Kees Cook be involved, due to his existing efforts in
>>>>> > kernel hardening.  I think it would be good to invite one or two expert
>>>>> > security researchers in this area -- Kees would know who.  In terms of
>>>
>>> Many of the folks that are good at kernel exploitation don't want to
>>> help us fix the situation. :)
>>>
>>> I'd recommend Lee Campbell, he's got a solid bit of experience from
>>> the offense side. I think we should extend an invite to spender and
>>> pageexec as well. They've been on the cutting edge of this for
>>> decades, and it would be silly not to invite them.
>>>
>>>>> > core kernel folk, I'd suggest Ingo and akpm, as a starting point.
>>>
>>> Perhaps also Linus and rmk? Some of the protections are very central
>>> to the kernel (e.g. constification, "read-mostly", segmentation
>>> through page table swaps or domains, etc). I'd also want Andy
>>> Lutomirski around, as he's got a lot of deep chipset knowledge. :)
>>>
>>
>> What is this chipset knowledge you speak of? :)
>
> You appear to enjoy fixing deep x86 madness. :)
>
>> One thing that grsecurity addresses (partially or fully?  I haven't
>> looked that closely): we have tons of static, non-const data structure
>> that contain function pointers, and we can't make them const because
>> they get filled in when things are initialized.  Grsecurity mitigates
>> this with some combination of compiler plugins and pax_open_kernel,
>
> Right. That's the constification and KERNEXEC pieces.
>
>> but we could probably come up with a more straightforward solution.
>> We could add an ro_after_init section, or we could even have a section
>> for things that are const but are writable through a special function.
>
> I don't think it would be that clean: there are many targets that
> legitimately need updating at runtime, but should be otherwise
> read-only. The idea with solving that is to use inline
> open/close_kernel calls to make them writable briefly and in a way
> that is ROP-defensive.

I really dislike open_kernel.  It's basically "turn off RO completely
-- please ROP me".  If we had a nice, conservative "poke this value
into this place that is hopefully actually a compile-time constant
address", I'd be much happier with it.

>
>>>  Protect sensitive data
>>>  o Constify function pointers! (no where close to this plugin)
>>
>> As above, this would be nice.  If we did this upstream, we could do
>> even better if we outright disallowed non-const function pointers (as
>> opposed to fixing them up semi-transparently as the plugin does).
>
> I would love this. :)
>
>>>  o IDT/GDT/syscall table/etc (this is partially done, aliases are
>>> writable still)
>>
>> I don't think the RO GDT patches ever happened.
>>
>> FWIW, we can't really eliminate a writable GDT alias without killing
>> compat performance: set_thread_area rather critically depends on
>> writing to the GDT at context switch time.
>
> If that's what's needed, I am fine killing compat performance. (We
> could attempt to make this a CONFIG for those that expect to run
> compat loads for their kernels.)

We would need to change the default behavior or arch_set_fs, too,
since some 64-bit processes use that mechanism as well.  But that's
easy.

>
>>>  o Vsyscall shadow table, see sgrakkyu's remote
>>> SELinux-disabling exploit
>>> (http://sgrakkyu.antifork.org/sctp_houdini.c, luto fixed this via
>>> vsyscall emulation)
>>
>> Not done yet, because even modern binaries still have the thing
>> mapped.  We could devote two or three minutes of a KS session to
>> figuring out how to kill it off for real for modern binaries.
>
> What uses it? I can run with vsyscall=none without problems...

Nothing, but it's still mapped by default.  I want something like the
stack access flags or an arch_prctl so glibc can turn it off by
default.

--Andy

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[James Morris]

On Mon, 24 Aug 2015, Kees Cook wrote:

> Should we have a separate discussion for bug-hunting? KASan needs to
> expand its architecture coverage, Trinity needs a maintainer, and
> there's always new things happening with smatch and coccinelle.
> 
> I'd just like to keep bug-hunting entirely separate from
> self-protection, as I feel they're distinct topics (though related).

I agree -- this needs to be a separate, related discussion.


-- 
James Morris
<jmorris@namei.org>

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Thomas Gleixner]

On Mon, 24 Aug 2015, Kees Cook wrote:
> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
> This is far from a comprehensive list, though. The biggest value, I
> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
> for constification and integer overflow.

There is another aspect. We need to make developers more aware of the
potential attack issues. I learned my lesson with the futex disaster
and since then I certainly look with a different set of eyes at user
space facing code. I doubt that we want that everyone experiences the
disaster himself (though that's a very enlightening experience), but
we should try to document incidents and the lessons learned from
them. Right now we just rely on those who are deep into the security
realm or the few people who learned it the hard way.

Thanks,

	tglx

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 11:19 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Mon, Aug 24, 2015 at 11:01 AM, Kees Cook <keescook@chromium.org> wrote:
>> On Mon, Aug 24, 2015 at 10:28 AM, Andy Lutomirski <luto@amacapital.net> wrote:
>>> On Mon, Aug 24, 2015 at 10:17 AM, Kees Cook <keescook@chromium.org> wrote:
>>>> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
>>>>> On Mon, 24 Aug 2015, Jiri Kosina wrote:
>>>>>
>>>>>> On Mon, 24 Aug 2015, James Morris wrote:
>>>>>>
>>>>>> > I'd recommend Kees Cook be involved, due to his existing efforts in
>>>>>> > kernel hardening.  I think it would be good to invite one or two expert
>>>>>> > security researchers in this area -- Kees would know who.  In terms of
>>>>
>>>> Many of the folks that are good at kernel exploitation don't want to
>>>> help us fix the situation. :)
>>>>
>>>> I'd recommend Lee Campbell, he's got a solid bit of experience from
>>>> the offense side. I think we should extend an invite to spender and
>>>> pageexec as well. They've been on the cutting edge of this for
>>>> decades, and it would be silly not to invite them.
>>>>
>>>>>> > core kernel folk, I'd suggest Ingo and akpm, as a starting point.
>>>>
>>>> Perhaps also Linus and rmk? Some of the protections are very central
>>>> to the kernel (e.g. constification, "read-mostly", segmentation
>>>> through page table swaps or domains, etc). I'd also want Andy
>>>> Lutomirski around, as he's got a lot of deep chipset knowledge. :)
>>>>
>>>
>>> What is this chipset knowledge you speak of? :)
>>
>> You appear to enjoy fixing deep x86 madness. :)
>>
>>> One thing that grsecurity addresses (partially or fully?  I haven't
>>> looked that closely): we have tons of static, non-const data structure
>>> that contain function pointers, and we can't make them const because
>>> they get filled in when things are initialized.  Grsecurity mitigates
>>> this with some combination of compiler plugins and pax_open_kernel,
>>
>> Right. That's the constification and KERNEXEC pieces.
>>
>>> but we could probably come up with a more straightforward solution.
>>> We could add an ro_after_init section, or we could even have a section
>>> for things that are const but are writable through a special function.
>>
>> I don't think it would be that clean: there are many targets that
>> legitimately need updating at runtime, but should be otherwise
>> read-only. The idea with solving that is to use inline
>> open/close_kernel calls to make them writable briefly and in a way
>> that is ROP-defensive.
>
> I really dislike open_kernel.  It's basically "turn off RO completely
> -- please ROP me".  If we had a nice, conservative "poke this value
> into this place that is hopefully actually a compile-time constant
> address", I'd be much happier with it.

Well... open_kernel turns off preempt. With the inlining, no-preempt,
and barriers, it should be correct, yes? Used as:

kernel_open();
readonly_thing = whatever;
kernel_close();

This should be way better than what we have already.

>>>>  Protect sensitive data
>>>>  o Constify function pointers! (no where close to this plugin)
>>>
>>> As above, this would be nice.  If we did this upstream, we could do
>>> even better if we outright disallowed non-const function pointers (as
>>> opposed to fixing them up semi-transparently as the plugin does).
>>
>> I would love this. :)
>>
>>>>  o IDT/GDT/syscall table/etc (this is partially done, aliases are
>>>> writable still)
>>>
>>> I don't think the RO GDT patches ever happened.
>>>
>>> FWIW, we can't really eliminate a writable GDT alias without killing
>>> compat performance: set_thread_area rather critically depends on
>>> writing to the GDT at context switch time.
>>
>> If that's what's needed, I am fine killing compat performance. (We
>> could attempt to make this a CONFIG for those that expect to run
>> compat loads for their kernels.)
>
> We would need to change the default behavior or arch_set_fs, too,
> since some 64-bit processes use that mechanism as well.  But that's
> easy.
>
>>
>>>>  o Vsyscall shadow table, see sgrakkyu's remote
>>>> SELinux-disabling exploit
>>>> (http://sgrakkyu.antifork.org/sctp_houdini.c, luto fixed this via
>>>> vsyscall emulation)
>>>
>>> Not done yet, because even modern binaries still have the thing
>>> mapped.  We could devote two or three minutes of a KS session to
>>> figuring out how to kill it off for real for modern binaries.
>>
>> What uses it? I can run with vsyscall=none without problems...
>
> Nothing, but it's still mapped by default.  I want something like the
> stack access flags or an arch_prctl so glibc can turn it off by
> default.

Oh, instead of just 87983c66bc02 plus =none, you want a per-process
runtime config? I think =none should just become the global default.
:)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Thomas Gleixner]

On Mon, 24 Aug 2015, Thomas Gleixner wrote:
> On Mon, 24 Aug 2015, Kees Cook wrote:
> > On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
> > This is far from a comprehensive list, though. The biggest value, I
> > think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
> > for constification and integer overflow.
> 
> There is another aspect. We need to make developers more aware of the
> potential attack issues. I learned my lesson with the futex disaster
> and since then I certainly look with a different set of eyes at user
> space facing code. I doubt that we want that everyone experiences the
> disaster himself (though that's a very enlightening experience), but
> we should try to document incidents and the lessons learned from
> them. Right now we just rely on those who are deep into the security
> realm or the few people who learned it the hard way.

A good way to start would be to actually force developers to document
meticulously the possible states of user space variable(s) which
influence the behaviour of the interface. That can be a mindboggling
exercise depending on the complexity of the interface, but it helps
both the implementer and the reviewer.

Thanks,

	tglx

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 11:52 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Mon, 24 Aug 2015, Kees Cook wrote:
>> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
>> This is far from a comprehensive list, though. The biggest value, I
>> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
>> for constification and integer overflow.
>
> There is another aspect. We need to make developers more aware of the
> potential attack issues. I learned my lesson with the futex disaster
> and since then I certainly look with a different set of eyes at user
> space facing code. I doubt that we want that everyone experiences the
> disaster himself (though that's a very enlightening experience), but
> we should try to document incidents and the lessons learned from
> them. Right now we just rely on those who are deep into the security
> realm or the few people who learned it the hard way.

Yeah, it can be a hard perspective shift to make. And shifting the
thinking about the kernel itself to always operating in a compromised
state makes thinking about how to protect it much easier. User space
is trying to hurt us! :)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[James Bottomley]

On Mon, 2015-08-24 at 09:35 -0700, Kees Cook wrote:
> On Mon, Aug 24, 2015 at 5:29 AM, Linus Walleij <linus.walleij@linaro.org> wrote:
> > On Mon, Aug 24, 2015 at 6:20 AM, James Morris <jmorris@namei.org> wrote:
> >
> >> There are also potentially promising approaches to mitigation with other
> >> technologies such as KASan and gcc plugins, as well as evolving hardware
> >> features.
> >
> > What I've discovered when running KASan the last few weeks is that
> > this points back to the question of tests ... I've been using Trinity
> > to find bugs, but it is more likely to kill itself or cause OOM than
> > trigger any boundary overrun bugs.
> >
> > Kselftest may be helpful, but basically any loads that heavily
> > exercise the kernel internals are helpful to harden the kernel.
> > Some of these are custom test suites I suspect. Any good hints
> > for a simple embedded developer like me?
> 
> I agree with the sentiment here, but not with the language. Finding
> flaws (which is what selftests, KASan, Trinity, etc do) isn't
> hardening. Hardening is stopping the exploitation of flaws.

Um, forgive me for being dense, but doesn't fixing the flaws stop their
exploitation?  In any event, Hardening means "reducing the attack
surface" and that encompasses both active and passive means (including
actual bug fixing).

>  The
> hardening the kernel needs is about taking away exploitation tools,
> not killing bugs. (Though killing bugs is still great.)

It's both.  One of the old standards for attacking C code was buffer
overruns.  Remove those via detection tools and you reduce the attack
surface.

> As an example, making the kernel code memory read-only means an
> attacker cannot just directly change the kernel's execution path when
> they use an arbitrary memory-writing flaw. (This feature is mostly
> enabled via CONFIG_DEBUG_RODATA, and was very recently added to ARM,
> though isn't at 100% coverage for all the physical memory aliases.)

Yes, I like this one a lot.  Adding mechanisms that don't increase
complexity like this are good active means.  However, I become less
enamoured of things like selinux and grsecurity which add complexity in
the name of active attack surface reduction.  That's not to say never do
it, it's just to say that attack surface is directly related to
complexity.

James

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[James Morris]

On Mon, 24 Aug 2015, James Bottomley wrote:

> Um, forgive me for being dense, but doesn't fixing the flaws stop their
> exploitation?  In any event, Hardening means "reducing the attack
> surface" and that encompasses both active and passive means (including
> actual bug fixing).

Hardening is mitigating those flaws.  You'll never find every flaw, but 
you can mitigate against entire classes of flaws being exploited.

> >  The
> > hardening the kernel needs is about taking away exploitation tools,
> > not killing bugs. (Though killing bugs is still great.)
> 
> It's both.  One of the old standards for attacking C code was buffer
> overruns.  Remove those via detection tools and you reduce the attack
> surface.

In this case, we're specifically talking about hardening the kernel to 
mitigate exploitation of flaws.  Kernel self-protection may be a better 
term (and recently surfaced in an NSA presentation at LSS: p.23 of

http://kernsec.org/files/lss2015/lss2015_selinuxinandroidlollipopandm_smalley.pdf


-- 
James Morris
<jmorris@namei.org>

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[josh]

On Mon, Aug 24, 2015 at 01:09:01PM -0700, James Bottomley wrote:
> On Mon, 2015-08-24 at 09:35 -0700, Kees Cook wrote:
> > On Mon, Aug 24, 2015 at 5:29 AM, Linus Walleij <linus.walleij@linaro.org> wrote:
> > > On Mon, Aug 24, 2015 at 6:20 AM, James Morris <jmorris@namei.org> wrote:
> > >
> > >> There are also potentially promising approaches to mitigation with other
> > >> technologies such as KASan and gcc plugins, as well as evolving hardware
> > >> features.
> > >
> > > What I've discovered when running KASan the last few weeks is that
> > > this points back to the question of tests ... I've been using Trinity
> > > to find bugs, but it is more likely to kill itself or cause OOM than
> > > trigger any boundary overrun bugs.
> > >
> > > Kselftest may be helpful, but basically any loads that heavily
> > > exercise the kernel internals are helpful to harden the kernel.
> > > Some of these are custom test suites I suspect. Any good hints
> > > for a simple embedded developer like me?
> > 
> > I agree with the sentiment here, but not with the language. Finding
> > flaws (which is what selftests, KASan, Trinity, etc do) isn't
> > hardening. Hardening is stopping the exploitation of flaws.
> 
> Um, forgive me for being dense, but doesn't fixing the flaws stop their
> exploitation?  In any event, Hardening means "reducing the attack
> surface" and that encompasses both active and passive means (including
> actual bug fixing).

There's a difference between fixing an individual vulnerability
discovered by fuzzing and other tools, and fixing (or at least making it
easier to do the right thing for) an entire *class* of
vulnerabilities.  We should fix the former when we find them, but any
time we see more than a couple bugs of the same time, we should be
asking "how can we never have another?".

- Josh Triplett

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Thomas Gleixner]

On Tue, 25 Aug 2015, James Morris wrote:
> On Mon, 24 Aug 2015, James Bottomley wrote:
> 
> > Um, forgive me for being dense, but doesn't fixing the flaws stop their
> > exploitation?  In any event, Hardening means "reducing the attack
> > surface" and that encompasses both active and passive means (including
> > actual bug fixing).
> 
> Hardening is mitigating those flaws.  You'll never find every flaw, but 
> you can mitigate against entire classes of flaws being exploited.

Fair enough, but I agree with James, that we should not make this an
isolated topic.

Let's look at it from a different POV. 10+ years ago locking was
something which was only understood by a minority of the developers
and they had to figure out (or not) the horrible once per year
deadlocks.

Then RT happened to expose a gazillion of those issues and once we got
tired of debugging this hard to understand problems we came up with a
tool which tells Joe Developer where he screwed up.

As a consequence developers got more aware of locking semantics and
the number of hard to figure out lock inversion problems got reduced
significantly.

We added a lot of other mechanisms to detect different classes of
failure (use after free, ....). Both runtime debugging facilities and
static code analysis have helped to improve the code quality and the
awareness of developers.

Security and the pitfalls of user space facing code are still a riddle
wrapped up in an enigma for most developers, so the question is how
can we improve that situation.

While we certainly want to add mechanisms which prevent flaws to be
exploited we surely want to do something about educating people how to
avoid the flaws in the first place.

One can read about interesting ways to subvert a lot of the hardening
techniques every other day, so while we certainly want to add
hardening techniques, it's even more important that these techniques
become the last parachute and not the catch it all mechanism because
we gave up on the underlying issues.

I totally agree that we cannot prevent all flaws, but we certainly can
do better in reducing the quantity. And that means that we need to
educate people. And that education involves traditional training,
documentation and clever usage of tools. If we can use hardening
techniques to slap developers on their fingers, that's certainly a
good thing. But we don't want to decouple the hardening from 'reduce
the flaws' as you might create the impression that it's not that
important to think security aware because the hardering techniques
will prevent the exploits anyway.

Thanks,

	tglx

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Greg KH]

On Mon, Aug 24, 2015 at 12:00:15PM -0700, Kees Cook wrote:
> On Mon, Aug 24, 2015 at 11:52 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Mon, 24 Aug 2015, Kees Cook wrote:
> >> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
> >> This is far from a comprehensive list, though. The biggest value, I
> >> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
> >> for constification and integer overflow.
> >
> > There is another aspect. We need to make developers more aware of the
> > potential attack issues. I learned my lesson with the futex disaster
> > and since then I certainly look with a different set of eyes at user
> > space facing code. I doubt that we want that everyone experiences the
> > disaster himself (though that's a very enlightening experience), but
> > we should try to document incidents and the lessons learned from
> > them. Right now we just rely on those who are deep into the security
> > realm or the few people who learned it the hard way.
> 
> Yeah, it can be a hard perspective shift to make. And shifting the
> thinking about the kernel itself to always operating in a compromised
> state makes thinking about how to protect it much easier. User space
> is trying to hurt us! :)

Microsoft's security team, which was responsible for forcing all of
their developers to undergo some security training every year, has
boiled it all down to these simple 4 words:

	All input is evil.

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[James Morris]

On Mon, 24 Aug 2015, Thomas Gleixner wrote:

> While we certainly want to add mechanisms which prevent flaws to be
> exploited we surely want to do something about educating people how to
> avoid the flaws in the first place.

What if we had a session where a security researcher presented an overview 
of classes kernel security flaws and how they're typically exploited?  
This would be aimed at core kernel developers, specifically.

> 
> One can read about interesting ways to subvert a lot of the hardening
> techniques every other day, so while we certainly want to add
> hardening techniques, it's even more important that these techniques
> become the last parachute and not the catch it all mechanism because
> we gave up on the underlying issues.

Nobody is giving up on underlying issues, in fact, we are doing pretty 
well there, but not so well with kernel mitigations, which keeps biting us 
in uncomfortable places.

> I totally agree that we cannot prevent all flaws, but we certainly can 
> do better in reducing the quantity. And that means that we need to 
> educate people. And that education involves traditional training, 
> documentation and clever usage of tools. If we can use hardening 
> techniques to slap developers on their fingers, that's certainly a good 
> thing. But we don't want to decouple the hardening from 'reduce the 
> flaws' as you might create the impression that it's not that important 
> to think security aware because the hardering techniques will prevent 
> the exploits anyway.

You're right, but as mentioned, I do believe that we are doing pretty well 
at code quality vs., say, proprietary development, as demonstrated by 
numerous quantitative studies.  The Linux kernel is used heavily and 
across many different types of environments, with a tight user/developer 
feedback loop, and all the code is subject to public review and 
accountability.  We actually pioneered this stuff :)

There is a lot of good testing happening, both in the community and via 
corporate QA, and of course there can be more, but the point of this 
proposal is that we are not leading in kernel self-protection and in fact 
somewhat behind the state of the art.  This is a problem which we 
specifically need to address in consultation with the research community 
and with core kernel folk on board.

I'm not saying the other stuff is not important, it absolutely is, but I 
wanted to specifically highlight kernel self-protection as a significant 
issue requiring collaboration between different groups.

There's no reason we can't have an expanded track, including an education 
session for kernel developers as mentioned above, as well as a testing 
session, and collect these together with kernel self-protection as a 
"kernel security hardening" track.  (This is of course distinct from 
functional security mechanisms, which we covered at LSS last week). I 
didn't want to have such a wide scope given limited availability of core 
developers on any one topic, but if folk are happy to consider expanding 
it, then fine.


- James
-- 
James Morris
<jmorris@namei.org>

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 1:09 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Mon, 2015-08-24 at 09:35 -0700, Kees Cook wrote:
>> On Mon, Aug 24, 2015 at 5:29 AM, Linus Walleij <linus.walleij@linaro.org> wrote:
>> > On Mon, Aug 24, 2015 at 6:20 AM, James Morris <jmorris@namei.org> wrote:
>> >
>> >> There are also potentially promising approaches to mitigation with other
>> >> technologies such as KASan and gcc plugins, as well as evolving hardware
>> >> features.
>> >
>> > What I've discovered when running KASan the last few weeks is that
>> > this points back to the question of tests ... I've been using Trinity
>> > to find bugs, but it is more likely to kill itself or cause OOM than
>> > trigger any boundary overrun bugs.
>> >
>> > Kselftest may be helpful, but basically any loads that heavily
>> > exercise the kernel internals are helpful to harden the kernel.
>> > Some of these are custom test suites I suspect. Any good hints
>> > for a simple embedded developer like me?
>>
>> I agree with the sentiment here, but not with the language. Finding
>> flaws (which is what selftests, KASan, Trinity, etc do) isn't
>> hardening. Hardening is stopping the exploitation of flaws.
>
> Um, forgive me for being dense, but doesn't fixing the flaws stop their
> exploitation?  In any event, Hardening means "reducing the attack
> surface" and that encompasses both active and passive means (including
> actual bug fixing).

I think "hardening" isn't a very precise word for security work,
actually. The language I try to use around these things (each in the
context of "security"):

flaw - a bug that provides some type of attack primitive (generally
memory reads and writes).
exploit - delivery of executable payload via flaws (this is a
technique to gain control via the flaw primitives)

Many times all we see are the flaws and a theoretical example. If
we're lucky, we see a brief PoC that triggers an Oops or something.
Then we fix the flaw and move on. We are not as exposed to fully
functional exploits. Overwriting syscall tables, IDT entries, function
pointers, or the kernel itself, chaining code snippets with ROP, heap
manipulation, and on and on.

So, elimination of flaws is certainly good, but there will always be
more. And it is worth noting that they exist _now_. There are multiple
full privileges escalation flaws in the kernel you're using right now.
We -- the upstream kernel community -- just don't know where they are.
Even Jon's conservative estimate is that these flaws tend to live at
least 5 years: https://lwn.net/Articles/410606/

We are finding the bugs, and we can do better, but that's not what I
think needs the most attention right now. We need to kill classes of
bugs and classes of exploits. To kill a bug class, we must remove the
possibility that it can ever go wrong in the first place. Many people
stop here when thinking about hardening, but we must move on to
killing classes of exploit. I'll continue to use my W^X kernel code
example, since it was not, from an operational stance, a flaw that
kernel code was writable. But it's a exploitation weakness. If an
attacker just needs to directly target a kernel memory location with
their memory-write primitive, all their work is done, that user loses
control of their kernel, game over.

We need to add the safety nets under the acrobats, since they can fall
at any time.

>>  The
>> hardening the kernel needs is about taking away exploitation tools,
>> not killing bugs. (Though killing bugs is still great.)
>
> It's both.  One of the old standards for attacking C code was buffer
> overruns.  Remove those via detection tools and you reduce the attack
> surface.

Well, attack surface is just what interfaces are exposed to an
attacker. But yes, killing bug classes is important. Making bugs
unexploitable is even better because then security bugs really do just
become bugs.

>> As an example, making the kernel code memory read-only means an
>> attacker cannot just directly change the kernel's execution path when
>> they use an arbitrary memory-writing flaw. (This feature is mostly
>> enabled via CONFIG_DEBUG_RODATA, and was very recently added to ARM,
>> though isn't at 100% coverage for all the physical memory aliases.)
>
> Yes, I like this one a lot.  Adding mechanisms that don't increase
> complexity like this are good active means.  However, I become less
> enamoured of things like selinux and grsecurity which add complexity in
> the name of active attack surface reduction.  That's not to say never do
> it, it's just to say that attack surface is directly related to
> complexity.

FWIW, really only reduces userspace attack surface (all syscalls are
still available to a confined process). seccomp reduces attack
surface. grsecurity has an MAC component, but grsecurity (with PaX) is
much larger. Regardless, some of these nets will increase complexity.
It's the same for anti-lock brakes and airbags[1]. We have to take on
this burden to protect our users from our mistakes.

-Kees

[1] Borrowing mricon's great analogy as presented at the Security
Summit this year:
http://kernsec.org/files/lss2015/giant-bags-of-mostly-water.pdf

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 1:17 PM, James Morris <jmorris@namei.org> wrote:
> On Mon, 24 Aug 2015, James Bottomley wrote:
>
>> Um, forgive me for being dense, but doesn't fixing the flaws stop their
>> exploitation?  In any event, Hardening means "reducing the attack
>> surface" and that encompasses both active and passive means (including
>> actual bug fixing).
>
> Hardening is mitigating those flaws.  You'll never find every flaw, but
> you can mitigate against entire classes of flaws being exploited.
>
>> >  The
>> > hardening the kernel needs is about taking away exploitation tools,
>> > not killing bugs. (Though killing bugs is still great.)
>>
>> It's both.  One of the old standards for attacking C code was buffer
>> overruns.  Remove those via detection tools and you reduce the attack
>> surface.
>
> In this case, we're specifically talking about hardening the kernel to
> mitigate exploitation of flaws.  Kernel self-protection may be a better
> term (and recently surfaced in an NSA presentation at LSS: p.23 of
>
> http://kernsec.org/files/lss2015/lss2015_selinuxinandroidlollipopandm_smalley.pdf

Yup. Using "kernel self-protection" cuts right to the point. I'm in
debt to Stephen for finding the right marketing term for "kernel
memory corruption vulnerability exploit mitigation". :)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 1:46 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Tue, 25 Aug 2015, James Morris wrote:
>> On Mon, 24 Aug 2015, James Bottomley wrote:
>>
>> > Um, forgive me for being dense, but doesn't fixing the flaws stop their
>> > exploitation?  In any event, Hardening means "reducing the attack
>> > surface" and that encompasses both active and passive means (including
>> > actual bug fixing).
>>
>> Hardening is mitigating those flaws.  You'll never find every flaw, but
>> you can mitigate against entire classes of flaws being exploited.
>
> Fair enough, but I agree with James, that we should not make this an
> isolated topic.
>
> Let's look at it from a different POV. 10+ years ago locking was
> something which was only understood by a minority of the developers
> and they had to figure out (or not) the horrible once per year
> deadlocks.
>
> Then RT happened to expose a gazillion of those issues and once we got
> tired of debugging this hard to understand problems we came up with a
> tool which tells Joe Developer where he screwed up.
>
> As a consequence developers got more aware of locking semantics and
> the number of hard to figure out lock inversion problems got reduced
> significantly.
>
> We added a lot of other mechanisms to detect different classes of
> failure (use after free, ....). Both runtime debugging facilities and
> static code analysis have helped to improve the code quality and the
> awareness of developers.
>
> Security and the pitfalls of user space facing code are still a riddle
> wrapped up in an enigma for most developers, so the question is how
> can we improve that situation.
>
> While we certainly want to add mechanisms which prevent flaws to be
> exploited we surely want to do something about educating people how to
> avoid the flaws in the first place.

Absolutely. I just think they are separate discussions. Teaching
someone how to avoid integer overflows is totally different from
teaching them why a fix-position writeable and executable memory
region should never exist.

>
> One can read about interesting ways to subvert a lot of the hardening
> techniques every other day, so while we certainly want to add
> hardening techniques, it's even more important that these techniques
> become the last parachute and not the catch it all mechanism because
> we gave up on the underlying issues.

Yes, it's absolutely an arms race. But there's no real way around
this. What we have to do is actually start running the race. :)

> I totally agree that we cannot prevent all flaws, but we certainly can
> do better in reducing the quantity. And that means that we need to
> educate people. And that education involves traditional training,
> documentation and clever usage of tools. If we can use hardening
> techniques to slap developers on their fingers, that's certainly a
> good thing. But we don't want to decouple the hardening from 'reduce
> the flaws' as you might create the impression that it's not that
> important to think security aware because the hardering techniques
> will prevent the exploits anyway.

Kernel self-protection systems aren't for slapping developers on their
fingers: it's about maintaining kernel integrity for a user's system
that is under active attack. The harder it is to attack, the better
the chance they will have to detect it.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Andy Lutomirski]

On Mon, Aug 24, 2015 at 3:55 PM, Kees Cook <keescook@chromium.org> wrote:
> On Mon, Aug 24, 2015 at 1:09 PM, James Bottomley
> <James.Bottomley@hansenpartnership.com> wrote:
>> On Mon, 2015-08-24 at 09:35 -0700, Kees Cook wrote:
>>>  The
>>> hardening the kernel needs is about taking away exploitation tools,
>>> not killing bugs. (Though killing bugs is still great.)
>>
>> It's both.  One of the old standards for attacking C code was buffer
>> overruns.  Remove those via detection tools and you reduce the attack
>> surface.
>
> Well, attack surface is just what interfaces are exposed to an
> attacker. But yes, killing bug classes is important. Making bugs
> unexploitable is even better because then security bugs really do just
> become bugs.

There are plenty of cases where mitigation of an exploit technique
class changes a family of bugs from privilege escalation to DoS, and
that's a big deal.  For example, min mmap addresses mostly turn
NULL-pointer dereferences into DoS.  SMAP (and, to a lesser extent,
UDEREF) turns "upside-down" gsbase (a lovely x86ism that's involved in
the BadIRET privesc and quite a few others) into just DoS.

These aren't cases where it's a PITA for an attacker to get more than
DoS -- these are cases where the mitigations really do prevent
privilege escalation completely because they kick in before the
attacker gets to control a value that's used for anything.

--Andy

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 3:22 PM, James Morris <jmorris@namei.org> wrote:
> On Mon, 24 Aug 2015, Thomas Gleixner wrote:
>
>> While we certainly want to add mechanisms which prevent flaws to be
>> exploited we surely want to do something about educating people how to
>> avoid the flaws in the first place.
>
> What if we had a session where a security researcher presented an overview
> of classes kernel security flaws and how they're typically exploited?
> This would be aimed at core kernel developers, specifically.

I think this would be great, but I wouldn't want to sacrifice
discussion time on which self-protections to start working on, and the
best ways to approach them.

Could we assign this as homework instead? There are countless examples
of well described kernel exploits already visible on the web. Here's
one that used ARM's historical lack of text protection (from Lee
Campbell):
http://googleprojectzero.blogspot.com/2015/01/exploiting-nvmap-to-escape-chrome.html
or overwriting function pointers (from the attack mentioned in
spender's 2010 LSS slides):
http://kernelbof.blogspot.com/2009/07/even-when-one-byte-matters.html
or Andy's exploit of the NMI issue:
http://www.openwall.com/lists/oss-security/2015/08/04/8

>> One can read about interesting ways to subvert a lot of the hardening
>> techniques every other day, so while we certainly want to add
>> hardening techniques, it's even more important that these techniques
>> become the last parachute and not the catch it all mechanism because
>> we gave up on the underlying issues.
>
> Nobody is giving up on underlying issues, in fact, we are doing pretty
> well there, but not so well with kernel mitigations, which keeps biting us
> in uncomfortable places.
>
>> I totally agree that we cannot prevent all flaws, but we certainly can
>> do better in reducing the quantity. And that means that we need to
>> educate people. And that education involves traditional training,
>> documentation and clever usage of tools. If we can use hardening
>> techniques to slap developers on their fingers, that's certainly a good
>> thing. But we don't want to decouple the hardening from 'reduce the
>> flaws' as you might create the impression that it's not that important
>> to think security aware because the hardering techniques will prevent
>> the exploits anyway.
>
> You're right, but as mentioned, I do believe that we are doing pretty well
> at code quality vs., say, proprietary development, as demonstrated by
> numerous quantitative studies.  The Linux kernel is used heavily and
> across many different types of environments, with a tight user/developer
> feedback loop, and all the code is subject to public review and
> accountability.  We actually pioneered this stuff :)
>
> There is a lot of good testing happening, both in the community and via
> corporate QA, and of course there can be more, but the point of this
> proposal is that we are not leading in kernel self-protection and in fact
> somewhat behind the state of the art.  This is a problem which we
> specifically need to address in consultation with the research community
> and with core kernel folk on board.
>
> I'm not saying the other stuff is not important, it absolutely is, but I
> wanted to specifically highlight kernel self-protection as a significant
> issue requiring collaboration between different groups.

Right. We already know how to do bug-hunting, testing, etc. They can
surely be improved, but right now I think we need to spend time
learning how to build these kinds of defenses.

> There's no reason we can't have an expanded track, including an education
> session for kernel developers as mentioned above, as well as a testing
> session, and collect these together with kernel self-protection as a
> "kernel security hardening" track.  (This is of course distinct from
> functional security mechanisms, which we covered at LSS last week). I
> didn't want to have such a wide scope given limited availability of core
> developers on any one topic, but if folk are happy to consider expanding
> it, then fine.

We could have everyone write their first kernel privilege escalation exploit. :)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 3:57 PM, Kees Cook <keescook@chromium.org> wrote:
> On Mon, Aug 24, 2015 at 1:17 PM, James Morris <jmorris@namei.org> wrote:
>> On Mon, 24 Aug 2015, James Bottomley wrote:
>>
>>> Um, forgive me for being dense, but doesn't fixing the flaws stop their
>>> exploitation?  In any event, Hardening means "reducing the attack
>>> surface" and that encompasses both active and passive means (including
>>> actual bug fixing).
>>
>> Hardening is mitigating those flaws.  You'll never find every flaw, but
>> you can mitigate against entire classes of flaws being exploited.
>>
>>> >  The
>>> > hardening the kernel needs is about taking away exploitation tools,
>>> > not killing bugs. (Though killing bugs is still great.)
>>>
>>> It's both.  One of the old standards for attacking C code was buffer
>>> overruns.  Remove those via detection tools and you reduce the attack
>>> surface.
>>
>> In this case, we're specifically talking about hardening the kernel to
>> mitigate exploitation of flaws.  Kernel self-protection may be a better
>> term (and recently surfaced in an NSA presentation at LSS: p.23 of
>>
>> http://kernsec.org/files/lss2015/lss2015_selinuxinandroidlollipopandm_smalley.pdf
>
> Yup. Using "kernel self-protection" cuts right to the point. I'm in
> debt to Stephen for finding the right marketing term for "kernel
> memory corruption vulnerability exploit mitigation". :)

pageexec informs me he used the term earlier...
https://pax.grsecurity.net/docs/PaXTeam-H2HC12-PaX-kernel-self-protection.pdf

Regardless, you're all awesome.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Theodore Ts'o]

On Mon, Aug 24, 2015 at 04:20:47PM -0700, Kees Cook wrote:
> 
> Could we assign this as homework instead?  There are countless examples
> of well described kernel exploits already visible on the web....

Might I suggest a somewhat higher-level homework?  What are the kernel
self-protection features that would be most useful for us to
implement, and --- this is critically important --- why aren't we
doing them already, and how can we fix that higher-order issue?

Is it because adding a particular feature would incur a huge
performance penalty?

Is it because no company has been willing to fund developers to work
on that particular feature to date?  (BTW, I consider the fact that
various companies collectively wasn't able to find a place for the
trinity maintainer to find a place to land to be somewhat of a failure
of the ecosystem, but maybe the tool wasn't as useful as we think, or
it maybe we failed to make the case to the correct set of
bean-counters.)

If the answer is that it's obvious what needs to be done, but (a) we
can't find anyone to bell the cat, or (b) the patches are going to be
rejected out of hand for one reason or another, the kernel summit is a
great opportunity to see if some face-to-face discussion address the
problem.  OTOH, if the fundamental problem is that we can't get the
headcount funded, then discussion at the kernel summit is probably not
going to be a good use of our time.  :-/

      	      	       	      	     - Ted

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[James Morris]

On Mon, 24 Aug 2015, Theodore Ts'o wrote:

> If the answer is that it's obvious what needs to be done, but (a) we
> can't find anyone to bell the cat, or (b) the patches are going to be
> rejected out of hand for one reason or another, the kernel summit is a
> great opportunity to see if some face-to-face discussion address the
> problem.  OTOH, if the fundamental problem is that we can't get the
> headcount funded, then discussion at the kernel summit is probably not
> going to be a good use of our time.  :-/

With my corp hat on, I find that it's much more difficult to find an 
appropriate developer to hire for this kind of work than it is to find 
funding for it.

Emily seems to be facing the same issue, from her presentation at LSS.



-- 
James Morris
<jmorris@namei.org>

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 4:54 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Aug 24, 2015 at 04:20:47PM -0700, Kees Cook wrote:
>>
>> Could we assign this as homework instead?  There are countless examples
>> of well described kernel exploits already visible on the web....
>
> Might I suggest a somewhat higher-level homework?  What are the kernel
> self-protection features that would be most useful for us to
> implement, and --- this is critically important --- why aren't we

I listed a bunch in my earlier reply. You can see a nice breakdown of
them in pageexec's slides on the topic:
https://pax.grsecurity.net/docs/PaXTeam-H2HC12-PaX-kernel-self-protection.pdf

> doing them already, and how can we fix that higher-order issue?

They are each complex, and no one has the interest, skills, tenacity,
and work-priority available to them yet. I continue to chip away at
pieces here and there, but I'm just one person, and I'm probably
lacking in skill too.

> Is it because adding a particular feature would incur a huge
> performance penalty?

Some do, yes. Most don't. Mostly it's just the will to fight for the
changes. I think the concepts behind grsec/PaX's KERNEXEC would be a
great first target. Many other features depend on being able to trust
various aspects of kernel's resulting memory footprint.

> Is it because no company has been willing to fund developers to work
> on that particular feature to date?  (BTW, I consider the fact that

Yes, that's a big part of it. Also, the political navigation of the
feature is time-consuming. But I've been trying to socialize these
ideas for a while now, so they shouldn't be too much of a shock to
anyone. The concept and benefits of defensive features (as opposed to
bug killing) has been traditionally difficult to get people to see.

> various companies collectively wasn't able to find a place for the
> trinity maintainer to find a place to land to be somewhat of a failure
> of the ecosystem, but maybe the tool wasn't as useful as we think, or
> it maybe we failed to make the case to the correct set of
> bean-counters.)

Agreed. Though AIUI, loss of trinity maintenance had more to do with
people not contributing back to it. He should speak for himself,
though. :)

> If the answer is that it's obvious what needs to be done, but (a) we
> can't find anyone to bell the cat, or (b) the patches are going to be
> rejected out of hand for one reason or another, the kernel summit is a
> great opportunity to see if some face-to-face discussion address the
> problem.  OTOH, if the fundamental problem is that we can't get the
> headcount funded, then discussion at the kernel summit is probably not
> going to be a good use of our time.  :-/

Understanding the value of self-protection technologies will go a long
way to accepting them when someone tries to present them for upstream.
Accepting the limitations they present (e.g. gcc plugins may only work
with certain compiler versions, etc) can be a hard pill to swallow,
but I think it's best for the end users.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Rafael J. Wysocki]

On Monday, August 24, 2015 05:05:25 PM Greg KH wrote:
> On Mon, Aug 24, 2015 at 12:00:15PM -0700, Kees Cook wrote:
> > On Mon, Aug 24, 2015 at 11:52 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > > On Mon, 24 Aug 2015, Kees Cook wrote:
> > >> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
> > >> This is far from a comprehensive list, though. The biggest value, I
> > >> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
> > >> for constification and integer overflow.
> > >
> > > There is another aspect. We need to make developers more aware of the
> > > potential attack issues. I learned my lesson with the futex disaster
> > > and since then I certainly look with a different set of eyes at user
> > > space facing code. I doubt that we want that everyone experiences the
> > > disaster himself (though that's a very enlightening experience), but
> > > we should try to document incidents and the lessons learned from
> > > them. Right now we just rely on those who are deep into the security
> > > realm or the few people who learned it the hard way.
> > 
> > Yeah, it can be a hard perspective shift to make. And shifting the
> > thinking about the kernel itself to always operating in a compromised
> > state makes thinking about how to protect it much easier. User space
> > is trying to hurt us! :)
> 
> Microsoft's security team, which was responsible for forcing all of
> their developers to undergo some security training every year, has
> boiled it all down to these simple 4 words:
> 
> 	All input is evil.

Right.

And not just input, but also everything you created and then allowed
someone else to modify.

Thanks,
Rafael

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Shuah Khan]

On Mon, Aug 24, 2015 at 10:35 AM, Kees Cook <keescook@chromium.org> wrote:
> On Mon, Aug 24, 2015 at 5:29 AM, Linus Walleij <linus.walleij@linaro.org> wrote:
>> On Mon, Aug 24, 2015 at 6:20 AM, James Morris <jmorris@namei.org> wrote:
>>
>>> There are also potentially promising approaches to mitigation with other
>>> technologies such as KASan and gcc plugins, as well as evolving hardware
>>> features.
>>
>> What I've discovered when running KASan the last few weeks is that
>> this points back to the question of tests ... I've been using Trinity
>> to find bugs, but it is more likely to kill itself or cause OOM than
>> trigger any boundary overrun bugs.
>>
>> Kselftest may be helpful, but basically any loads that heavily
>> exercise the kernel internals are helpful to harden the kernel.
>> Some of these are custom test suites I suspect. Any good hints
>> for a simple embedded developer like me?
>
> I agree with the sentiment here, but not with the language. Finding
> flaws (which is what selftests, KASan, Trinity, etc do) isn't
> hardening. Hardening is stopping the exploitation of flaws. The
> hardening the kernel needs is about taking away exploitation tools,
> not killing bugs. (Though killing bugs is still great.)

I agree with Kees on this. Kselftest or any other test suites can help
with regression testing and make sure Kernel works the way it should.
Also these tests can tell us if kernel is hardened or not.

Hardening means something different to me. i.e making sure kernel
can protect against attacks and fail gracefully. This is something to
address during design and development process.

>
> As an example, making the kernel code memory read-only means an
> attacker cannot just directly change the kernel's execution path when
> they use an arbitrary memory-writing flaw. (This feature is mostly
> enabled via CONFIG_DEBUG_RODATA, and was very recently added to ARM,
> though isn't at 100% coverage for all the physical memory aliases.)
>

This sounds similar to ExecShield (NX bit) on Intel. Yes this is a good example.

thanks,
-- Shuah

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Tue, Aug 25, 2015 at 8:15 AM, Shuah Khan <shuahkhan@gmail.com> wrote:
> On Mon, Aug 24, 2015 at 10:35 AM, Kees Cook <keescook@chromium.org> wrote:
>> As an example, making the kernel code memory read-only means an
>> attacker cannot just directly change the kernel's execution path when
>> they use an arbitrary memory-writing flaw. (This feature is mostly
>> enabled via CONFIG_DEBUG_RODATA, and was very recently added to ARM,
>> though isn't at 100% coverage for all the physical memory aliases.)
>>
>
> This sounds similar to ExecShield (NX bit) on Intel. Yes this is a good example.

Yup! That's exactly the NX bit (or other architecture equivalent). The
trouble tends to be around correctly setting up the kernel memory maps
to actually split up regions and mark the permissions correctly, and
make sure nothing was depending on the side-effects of the old
permissions.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

On Tue, Aug 25, 2015 at 09:15:32AM -0600, Shuah Khan wrote:
> On Mon, Aug 24, 2015 at 10:35 AM, Kees Cook <keescook@chromium.org> wrote:

> > I agree with the sentiment here, but not with the language. Finding
> > flaws (which is what selftests, KASan, Trinity, etc do) isn't
> > hardening. Hardening is stopping the exploitation of flaws. The
> > hardening the kernel needs is about taking away exploitation tools,
> > not killing bugs. (Though killing bugs is still great.)

> I agree with Kees on this. Kselftest or any other test suites can help
> with regression testing and make sure Kernel works the way it should.
> Also these tests can tell us if kernel is hardened or not.

> Hardening means something different to me. i.e making sure kernel
> can protect against attacks and fail gracefully. This is something to
> address during design and development process.

Testsuites can help here if we get into the habit of making sure they
exercise error conditions; they're off to the side a bit but they can
be a useful way of promoting good practice (at least in my experience).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Tue, Aug 25, 2015 at 9:30 AM, Mark Brown <broonie@kernel.org> wrote:
> On Tue, Aug 25, 2015 at 09:15:32AM -0600, Shuah Khan wrote:
>> On Mon, Aug 24, 2015 at 10:35 AM, Kees Cook <keescook@chromium.org> wrote:
>
>> > I agree with the sentiment here, but not with the language. Finding
>> > flaws (which is what selftests, KASan, Trinity, etc do) isn't
>> > hardening. Hardening is stopping the exploitation of flaws. The
>> > hardening the kernel needs is about taking away exploitation tools,
>> > not killing bugs. (Though killing bugs is still great.)
>
>> I agree with Kees on this. Kselftest or any other test suites can help
>> with regression testing and make sure Kernel works the way it should.
>> Also these tests can tell us if kernel is hardened or not.
>
>> Hardening means something different to me. i.e making sure kernel
>> can protect against attacks and fail gracefully. This is something to
>> address during design and development process.
>
> Testsuites can help here if we get into the habit of making sure they
> exercise error conditions; they're off to the side a bit but they can
> be a useful way of promoting good practice (at least in my experience).

Yeah, this is what I've done with a bunch of the newer tests in the
lkdtm module. They're designed to Oops the machine by performing
actions that should be caught by various mitigations (e.g. writing to
kernel text, executing userspace memory from the kernel, etc).

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Luis R. Rodriguez]

On Mon, Aug 24, 2015 at 10:46:33PM +0200, Thomas Gleixner wrote:
> While we certainly want to add mechanisms which prevent flaws to be
> exploited we surely want to do something about educating people how to
> avoid the flaws in the first place.

<-- snip -->

> I totally agree that we cannot prevent all flaws, but we certainly can
> do better in reducing the quantity. And that means that we need to
> educate people. And that education involve documentation and clever usage of
> tools.

What folks have been discussing mostly after this post are reactive security
solutions, that's fine but I think we need to also be a bit more proactive
about what we let folks design in terms of *new* userspace APIs. For instance
although it would not solve all userspace facing issues, I certainly think
things like struct nla_policy helps move away from loose userspace APIs.
The next evolutionary step was generic netlink but that may be too tied down
to networking.

  Luis

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Shuah Khan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/25/2015 10:30 AM, Mark Brown wrote:
> On Tue, Aug 25, 2015 at 09:15:32AM -0600, Shuah Khan wrote:
>> On Mon, Aug 24, 2015 at 10:35 AM, Kees Cook
>> <keescook@chromium.org> wrote:
> 
>>> I agree with the sentiment here, but not with the language.
>>> Finding flaws (which is what selftests, KASan, Trinity, etc do)
>>> isn't hardening. Hardening is stopping the exploitation of
>>> flaws. The hardening the kernel needs is about taking away
>>> exploitation tools, not killing bugs. (Though killing bugs is
>>> still great.)
> 
>> I agree with Kees on this. Kselftest or any other test suites can
>> help with regression testing and make sure Kernel works the way
>> it should. Also these tests can tell us if kernel is hardened or
>> not.
> 
>> Hardening means something different to me. i.e making sure
>> kernel can protect against attacks and fail gracefully. This is
>> something to address during design and development process.
> 
> Testsuites can help here if we get into the habit of making sure
> they exercise error conditions; they're off to the side a bit but
> they can be a useful way of promoting good practice (at least in my
> experience).
> 

Yes. I agree. We can expand the Kselftest suite to do error checking,
maybe error injection. There are several tests now that are focused
on error checking and make sure syscalls fail correctly when used
incorrectly etc.

We could look into enhancing existing tests and add new tests as needed
to focus on finding kernel hardening related bugs and continue use them
for regression testing.

thanks,
- -- Shuah

- -- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJV3J7LAAoJEAsCRMQNDUMcT1sQANiUpiZuYMjEtUFAkaxjEWGq
wI8xv69csf25ZClnOnjya3ALFksiIFbkdi7n4NWe6okKOgACpe5T0muVnSGbd6bz
ynw8zrBtCn0Kv9x+ee805wZ+VE5LtZe1bLqmhEIV0pDR+9wfn+BEmi2wsdeorW/R
tdE1REiyD9EThkfR6A15pVutD1lSPAxoPN1CfizO4VL/MSqa+/whU9V0Nrsj4CLl
MD9a4TP1TKgDdSFWWO6Gmdlh4cqXcJtXvGlnMxh3CZOuXolKMvye3Zgp9eyCp8HD
lHdJvbeON2z9lugFFfhF/iNvToHAuce3ArdlrlEcOJMDWBQ5ndUgGDBGYHNgWdGo
JoUKMJkI3g46ZsA7jfBJZ/G5wkH6k3b0Rwj+RPI2uPmu43TU80At3GwdiLq+aYGR
yp1p9ZuwrvtzL2sZOlPrj/uV1ZnIBj//fOuJWG0bJbsPYsnXe2vkZ4lQ/MsAe+sA
WKBYx8EO1YWCHbbsf37Vs1q8GRSUYuoOR2x7fUnvQlGO+QRfpb8ZcboMC7L0eC0C
plujBslA8E155TVUel9nreBK0WBWjynWOVlMf/GevCzWpiVirowlbKsOzttVHVCE
FczNYNd+CAzqxDfTrtzrjX3g5pEfjdmUgT4+6gwieG2AnilupV3yRvSW0e1Gqkii
R2LPXaZ0cMKJQ+2GYdD9
=kpAC
-----END PGP SIGNATURE-----

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 24, 2015 at 10:17 AM, Kees Cook <keescook@chromium.org> wrote:
> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
>> On Mon, 24 Aug 2015, Jiri Kosina wrote:
>>
>>> On Mon, 24 Aug 2015, James Morris wrote:
>>>
>>> > I'd recommend Kees Cook be involved, due to his existing efforts in
>>> > kernel hardening.  I think it would be good to invite one or two expert
>>> > security researchers in this area -- Kees would know who.  In terms of
>
> Many of the folks that are good at kernel exploitation don't want to
> help us fix the situation. :)
>
> I'd recommend Lee Campbell, he's got a solid bit of experience from
> the offense side. I think we should extend an invite to spender and
> pageexec as well. They've been on the cutting edge of this for
> decades, and it would be silly not to invite them.
>
>>> > core kernel folk, I'd suggest Ingo and akpm, as a starting point.
>
> Perhaps also Linus and rmk? Some of the protections are very central
> to the kernel (e.g. constification, "read-mostly", segmentation
> through page table swaps or domains, etc). I'd also want Andy
> Lutomirski around, as he's got a lot of deep chipset knowledge. :)

I think another valuable developer to invite would be Matthew Garrett.
He's been looking at hardening the line between root and kernel for a
while now.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Matthew Garrett]

(Resending now this address is subscribed)

On Wed, Aug 26, 2015 at 1:51 PM, Kees Cook <keescook@chromium.org> wrote:
> I think another valuable developer to invite would be Matthew Garrett.
> He's been looking at hardening the line between root and kernel for a
> while now.

Yeah, we can't rely on userspace to avoid or mitigate all privilege
escalation bugs - avoiding userspace escalation inherently turning
into owning the kernel remains valuable. There's various approaches to
validating userspace state that are only achievable if the kernel is
still trustworthy, so the assumption that root = game over is one that
we shouldn't allow the kernel to embody.

[Ksummit-discuss] grsecurity and kernel hardening

[Stephen Hemminger]

Looks like grsecurity has had a turn for the worse
 http://www.theregister.co.uk/2015/08/27/grsecurity/

But can't say that I blame him for being mad about vendors
using without giving back...

Re: [Ksummit-discuss] grsecurity and kernel hardening

[James Bottomley]

On Thu, 2015-08-27 at 15:08 -0700, Stephen Hemminger wrote:
> Looks like grsecurity has had a turn for the worse
>  http://www.theregister.co.uk/2015/08/27/grsecurity/

We could add this to the legal discussion.  The above article reads like
a classic how not to do things right.  Trademarks aren't like copyrights
or patents, they're really tricky things to use successfully.  You can't
register them and only pull them out of a drawer when someone does
something you don't like.  For trademarks, you have to establish your
mark and the norms first.  This usually requires some sort of policy
statement, the most critical element of which is (especially if the mark
is actually part of the distributed open source base or a tag attached
to the patch set) what acceptable and non-acceptable use is.  In
particular, you can't allow people to use it all over the place and then
later turn around and decide X use is a trademark infringement because
you now decide you don't like it or whoever's using it hasn't paid you
any money.

The Iceweasel controversy notwithstanding, Mozilla seems to have about
the most effective Open Source trademark practises.  They certainly
demonstrate that a trademark gives you more control than you'd otherwise
expect over a permissively licensed project.

>  But can't say that I blame him for being mad about vendors
>  using without giving back...

We also have to be a little careful about this.  The strict requirements
are whatever the licence says and nothing more.  We like it when vendors
give back, either by becoming contributing members of the community or
sponsoring conferences, or even just giving developers cash, but we have
to acknowledge that they don't do it because they're required to, or
even because they feel morally obliged to ... they mostly do it because
they have sound business reasons for doing so.

Getting a vendor to see that they have sound business reasons for giving
back is a specialised skill usually practised over long time period and
only successfully by a few people in the industry.  It doesn't happen
automatically and certainly not because you have a dummy spit over
someone taking something you advertised as "for free" and having the
temerity to expect not to have to pay you.

Perhaps it would be useful to run a "how to get your company to regard
open source as an investment" type training course, but it's probable
that the kernel summit isn't the best venue.

James

Re: [Ksummit-discuss] grsecurity and kernel hardening

[Stephen Hemminger]

On Thu, 27 Aug 2015 15:49:52 -0700
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> Getting a vendor to see that they have sound business reasons for giving
> back is a specialised skill usually practised over long time period and
> only successfully by a few people in the industry.  It doesn't happen
> automatically and certainly not because you have a dummy spit over
> someone taking something you advertised as "for free" and having the
> temerity to expect not to have to pay you.
> 
> Perhaps it would be useful to run a "how to get your company to regard
> open source as an investment" type training course, but it's probable
> that the kernel summit isn't the best venue.

Agreed, plus the KS is not necessarily going to be the right venue
for this. Kind of hard to preach to the converted.

[Ksummit-discuss] Self nomination

[Matthew Garrett]

Unsure whether Kees suggesting me is sufficient, so:

I'd like to be involved if we're going to have a meaningful discussion 
about more proactively mitigating attacks on the kernel. I've been 
working full-time in the security field for about the past three years, 
and it's certainly resulted in a fairly strong shift in attitude towards 
how well we're doing here. At this point even Microsoft are showing more 
aggressive security development than we are, and when the only person 
who appears to be doing meaningful work in adding mitigation features to 
the kernel is Kees, that's kind of a bad sign.

I've been doing development in this field for some time now - the secure 
boot patchset is an attempt to avoid allowing userspace privilege 
escalations to turn into persistent kernel compromises, for instance. As 
a member of the SFC's kernel copyright enforcement group, I'm also 
interested in being involved in any legal discussions that come up. Even 
if I don't make the invite list, I'd hope that these both end up as high 
priority topics.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Eric W. Biederman]

Greg KH <greg@kroah.com> writes:

> On Mon, Aug 24, 2015 at 12:00:15PM -0700, Kees Cook wrote:
>> On Mon, Aug 24, 2015 at 11:52 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> > On Mon, 24 Aug 2015, Kees Cook wrote:
>> >> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
>> >> This is far from a comprehensive list, though. The biggest value, I
>> >> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
>> >> for constification and integer overflow.
>> >
>> > There is another aspect. We need to make developers more aware of the
>> > potential attack issues. I learned my lesson with the futex disaster
>> > and since then I certainly look with a different set of eyes at user
>> > space facing code. I doubt that we want that everyone experiences the
>> > disaster himself (though that's a very enlightening experience), but
>> > we should try to document incidents and the lessons learned from
>> > them. Right now we just rely on those who are deep into the security
>> > realm or the few people who learned it the hard way.
>> 
>> Yeah, it can be a hard perspective shift to make. And shifting the
>> thinking about the kernel itself to always operating in a compromised
>> state makes thinking about how to protect it much easier. User space
>> is trying to hurt us! :)
>
> Microsoft's security team, which was responsible for forcing all of
> their developers to undergo some security training every year, has
> boiled it all down to these simple 4 words:
>
> 	All input is evil.

I have a linux version of that one.

An unprivileged user can call that.

It is absolutely amazing how much kernel code does not worry
about evil input after it checks that the caller is root.

Eric

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[josh]

On Mon, Aug 31, 2015 at 03:10:01PM -0500, Eric W. Biederman wrote:
> Greg KH <greg@kroah.com> writes:
> 
> > On Mon, Aug 24, 2015 at 12:00:15PM -0700, Kees Cook wrote:
> >> On Mon, Aug 24, 2015 at 11:52 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> >> > On Mon, 24 Aug 2015, Kees Cook wrote:
> >> >> On Mon, Aug 24, 2015 at 4:56 AM, James Morris <jmorris@namei.org> wrote:
> >> >> This is far from a comprehensive list, though. The biggest value, I
> >> >> think, would be in using KERNEXEC, UDEREF, USERCOPY, and the plugins
> >> >> for constification and integer overflow.
> >> >
> >> > There is another aspect. We need to make developers more aware of the
> >> > potential attack issues. I learned my lesson with the futex disaster
> >> > and since then I certainly look with a different set of eyes at user
> >> > space facing code. I doubt that we want that everyone experiences the
> >> > disaster himself (though that's a very enlightening experience), but
> >> > we should try to document incidents and the lessons learned from
> >> > them. Right now we just rely on those who are deep into the security
> >> > realm or the few people who learned it the hard way.
> >> 
> >> Yeah, it can be a hard perspective shift to make. And shifting the
> >> thinking about the kernel itself to always operating in a compromised
> >> state makes thinking about how to protect it much easier. User space
> >> is trying to hurt us! :)
> >
> > Microsoft's security team, which was responsible for forcing all of
> > their developers to undergo some security training every year, has
> > boiled it all down to these simple 4 words:
> >
> > 	All input is evil.
> 
> I have a linux version of that one.
> 
> An unprivileged user can call that.
> 
> It is absolutely amazing how much kernel code does not worry
> about evil input after it checks that the caller is root.

Another that has only started getting consideration recently: relative to the
kernel, root has less privilege.

- Josh Triplett

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Eric W. Biederman]

Kees Cook <keescook@chromium.org> writes:

> We are finding the bugs, and we can do better, but that's not what I
> think needs the most attention right now. We need to kill classes of
> bugs and classes of exploits. To kill a bug class, we must remove the
> possibility that it can ever go wrong in the first place. Many people
> stop here when thinking about hardening, but we must move on to
> killing classes of exploit. I'll continue to use my W^X kernel code
> example, since it was not, from an operational stance, a flaw that
> kernel code was writable. But it's a exploitation weakness. If an
> attacker just needs to directly target a kernel memory location with
> their memory-write primitive, all their work is done, that user loses
> control of their kernel, game over.
>
> We need to add the safety nets under the acrobats, since they can fall
> at any time.

I think it makes sense to close the classes of vulnerabilities that we
can.

At the same time I think we to serious consider tossing attempts that
fails to close a class of exploits.

The kernel address space randomization on x86_64 I find disturbing.  We
have a 2GB address space for kernel code.  We have pages that are 2MB in
that address space.  So we only have 10 bits that can change.  Only 9
bits that can change if the kernel needs more than one 2MB page.  Which
means that at most we need to brute force 1024 things to exploit any
weakness.

I don't see that attempt at kernel self protection actually
accomplishing anything in the way of protection, and I do see it costing
us debuggability which impacts kernel maintenance.  That is enabling
this protection seems to increase the effort to fix kernel bugs and as
such increases the number of bugs overall.

Is it reasonable to suggest when we have kernel security features that
only make people feel good, but don't actually protect them that we toss
the feature?

>> Yes, I like this one a lot.  Adding mechanisms that don't increase
>> complexity like this are good active means.  However, I become less
>> enamoured of things like selinux and grsecurity which add complexity in
>> the name of active attack surface reduction.  That's not to say never do
>> it, it's just to say that attack surface is directly related to
>> complexity.>

> FWIW, really only reduces userspace attack surface (all syscalls are
> still available to a confined process). seccomp reduces attack
> surface. grsecurity has an MAC component, but grsecurity (with PaX) is
> much larger. Regardless, some of these nets will increase complexity.
> It's the same for anti-lock brakes and airbags[1]. We have to take on
> this burden to protect our users from our mistakes.

Except given your reference[1] what we need to do is protect our users
from their own mistakes.  Which means we need to do more than just tell
our users no it is not ok to do that thing you want to do (they will do
it anyway).  We need to figure out safe ways to allow our users to do
the things they want or need to do.

This means things like not hiding new features behind CAP_SYS_ADMIN so
that we don't have to bother with securing kernel code.

This means things like figuring out how to make it possible for users to
mount that usb key they found in the parking lot and not have their
computer get owned.

All of which says that we need to increase the amount of the kernel
code that we are willing to defend from attacks, and figure out how to
defend that code.

Allowing our users to reduce the kernel attack surface is valid, but I
don't think for us as kernel developers it is valid to rely on users
reducing the kernel attack surface.

> -Kees
>
> [1] Borrowing mricon's great analogy as presented at the Security
> Summit this year:
> http://kernsec.org/files/lss2015/giant-bags-of-mostly-water.pdf

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Jiri Kosina]

On Mon, 31 Aug 2015, Eric W. Biederman wrote:

> At the same time I think we to serious consider tossing attempts that
> fails to close a class of exploits.
> 
> The kernel address space randomization on x86_64 I find disturbing.  We
> have a 2GB address space for kernel code.  We have pages that are 2MB in
> that address space.  So we only have 10 bits that can change.  Only 9
> bits that can change if the kernel needs more than one 2MB page.  Which
> means that at most we need to brute force 1024 things to exploit any
> weakness.

I understand what you are saying. OTOH, assuming that kernel doesn't leak 
addressess when restrict_kptr is set, it's rather likely that bruteforcing 
attempt to exploit the kernel is going to crash it instead of privilege 
escalation happenin (because the address wouldn't match, and either the 
kernel will #GP directly, or do some totally random garbage and crash 
shortly afterwards).

It's still *pretty* bad, but arguably better than being silently 
compromised without anyone noticing.

> I don't see that attempt at kernel self protection actually 
> accomplishing anything in the way of protection, and I do see it costing 
> us debuggability which impacts kernel maintenance.  

When kaslr appeared, my first reaction also was "this will be disaster for 
debugability and post-mortem crashdump analysis", but it turns out to work 
out pretty well.

What particular concerncs do you have in this respect?

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Mon, Aug 31, 2015 at 1:58 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Kees Cook <keescook@chromium.org> writes:
>
>> We are finding the bugs, and we can do better, but that's not what I
>> think needs the most attention right now. We need to kill classes of
>> bugs and classes of exploits. To kill a bug class, we must remove the
>> possibility that it can ever go wrong in the first place. Many people
>> stop here when thinking about hardening, but we must move on to
>> killing classes of exploit. I'll continue to use my W^X kernel code
>> example, since it was not, from an operational stance, a flaw that
>> kernel code was writable. But it's a exploitation weakness. If an
>> attacker just needs to directly target a kernel memory location with
>> their memory-write primitive, all their work is done, that user loses
>> control of their kernel, game over.
>>
>> We need to add the safety nets under the acrobats, since they can fall
>> at any time.
>
> I think it makes sense to close the classes of vulnerabilities that we
> can.

Absolutely, though I'm also suggesting we add proactive defenses. A
net isn't fixing a class of vulnerabilities in my bad analogy. A net
is proactive defense. Making the trapeze cables out of steel is a
vuln-fix.

> At the same time I think we to serious consider tossing attempts that
> fails to close a class of exploits.

I would agree if no one wanted them.

> The kernel address space randomization on x86_64 I find disturbing.  We
> have a 2GB address space for kernel code.  We have pages that are 2MB in
> that address space.  So we only have 10 bits that can change.  Only 9
> bits that can change if the kernel needs more than one 2MB page.  Which
> means that at most we need to brute force 1024 things to exploit any
> weakness.
>
> I don't see that attempt at kernel self protection actually
> accomplishing anything in the way of protection, and I do see it costing
> us debuggability which impacts kernel maintenance.  That is enabling
> this protection seems to increase the effort to fix kernel bugs and as
> such increases the number of bugs overall.

We'll have to agree to disagree. The kernel knows where its symbols
are, relocations are exported in crash dumps, etc. Everything needed
to debug a randomized base offset kernel is already there. For system
owners that run tight containers or remote services, kASLR provides a
real (if statistical) defense.

> Is it reasonable to suggest when we have kernel security features that
> only make people feel good, but don't actually protect them that we toss
> the feature?

I'm sure such a situation would be met with debate, like all kernel topics. :)

>>> Yes, I like this one a lot.  Adding mechanisms that don't increase
>>> complexity like this are good active means.  However, I become less
>>> enamoured of things like selinux and grsecurity which add complexity in
>>> the name of active attack surface reduction.  That's not to say never do
>>> it, it's just to say that attack surface is directly related to
>>> complexity.>
>
>> FWIW, really only reduces userspace attack surface (all syscalls are
>> still available to a confined process). seccomp reduces attack
>> surface. grsecurity has an MAC component, but grsecurity (with PaX) is
>> much larger. Regardless, some of these nets will increase complexity.
>> It's the same for anti-lock brakes and airbags[1]. We have to take on
>> this burden to protect our users from our mistakes.
>
> Except given your reference[1] what we need to do is protect our users
> from their own mistakes.  Which means we need to do more than just tell
> our users no it is not ok to do that thing you want to do (they will do
> it anyway).  We need to figure out safe ways to allow our users to do
> the things they want or need to do.
>
> This means things like not hiding new features behind CAP_SYS_ADMIN so
> that we don't have to bother with securing kernel code.

The kernel cannot be one-size-fits all. But we all have to play in the
same codebase, which means sometimes a feature is inherently opposed
to another. See all the interfaces that equate root with kernel access
(e.g. x86 msr module). These things are useful to those that need it
and a disaster to those that don't. We're always going to have the
play a balancing game and deal with these kinds of things on a
case-by-case basis. All that said, I agree: I don't like hiding stuff
behind CAP_SYS_ADMIN either.

> This means things like figuring out how to make it possible for users to
> mount that usb key they found in the parking lot and not have their
> computer get owned.

Well, that use-case is a whole different story. But it's a great
example of need to close bug classes, program defensively, add
self-protection features, etc.

> All of which says that we need to increase the amount of the kernel
> code that we are willing to defend from attacks, and figure out how to
> defend that code.

Sure, I totally agree. But in parallel to that is creating systems in
the kernel that stop exploitation methods (that exist regardless of
the flaw used as the primitive). There is a difference between killing
bug classes and killing exploitation methods. Our kernel has way too
few of the latter.

> Allowing our users to reduce the kernel attack surface is valid, but I
> don't think for us as kernel developers it is valid to rely on users
> reducing the kernel attack surface.

Totally agreed. Distro kernels are a good example, since they DO need
a one-size-fits-all config. :(

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Kees Cook]

On Tue, Sep 1, 2015 at 2:03 AM, Jiri Kosina <jikos@kernel.org> wrote:
> On Mon, 31 Aug 2015, Eric W. Biederman wrote:
>
>> At the same time I think we to serious consider tossing attempts that
>> fails to close a class of exploits.
>>
>> The kernel address space randomization on x86_64 I find disturbing.  We
>> have a 2GB address space for kernel code.  We have pages that are 2MB in
>> that address space.  So we only have 10 bits that can change.  Only 9
>> bits that can change if the kernel needs more than one 2MB page.  Which
>> means that at most we need to brute force 1024 things to exploit any
>> weakness.
>
> I understand what you are saying. OTOH, assuming that kernel doesn't leak
> addressess when restrict_kptr is set, it's rather likely that bruteforcing
> attempt to exploit the kernel is going to crash it instead of privilege
> escalation happenin (because the address wouldn't match, and either the
> kernel will #GP directly, or do some totally random garbage and crash
> shortly afterwards).
>
> It's still *pretty* bad, but arguably better than being silently
> compromised without anyone noticing.

This reminds me: we must assume all users on a system (and all devices
being attached) have access to a write-anywhere primitive (since they
DO -- we just haven't found them all yet, and we never will), what can
we do to close the windows of exploitability? (e.g. making memory
read-only, adding CFI to defend against ROP via the stack, etc?)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Dan Carpenter]

On Mon, Aug 24, 2015 at 09:35:50AM -0700, Kees Cook wrote:
> As an example, making the kernel code memory read-only means an
> attacker cannot just directly change the kernel's execution path when
> they use an arbitrary memory-writing flaw.

Part of the problem is that C defaults to writable and you have to
explicitly add a const if you want it read only.  Then when you are
reading the code and you see:

static struct uvc_format_desc uvc_fmts[] = {

You can't immediately tell if it is writeable because it needs to be or
because that's just the default.  We should make a new keyword that
means non constant then we could grep for structs which have a function
pointer and complain if they weren't specified as const or __writable.

Maybe there is a way to do this with GCC attributes but I have never
used them before and I couldn't immediately see how to do it.

regards,
dan carpenter

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

On mar., 2015-09-22 at 15:24 +0300, Dan Carpenter wrote:
> On Mon, Aug 24, 2015 at 09:35:50AM -0700, Kees Cook wrote:
> > As an example, making the kernel code memory read-only means an
> > attacker cannot just directly change the kernel's execution path when
> > they use an arbitrary memory-writing flaw.
> 
> Part of the problem is that C defaults to writable and you have to
> explicitly add a const if you want it read only.  Then when you are
> reading the code and you see:
> 
> static struct uvc_format_desc uvc_fmts[] = {
> 
> You can't immediately tell if it is writeable because it needs to be or
> because that's just the default.  We should make a new keyword that
> means non constant then we could grep for structs which have a function
> pointer and complain if they weren't specified as const or __writable.
> 
> Maybe there is a way to do this with GCC attributes but I have never
> used them before and I couldn't immediately see how to do it.

If you look at the constify plugin (part of PaX), you'll see it uses a
gcc plugin to do something like this for structure containing only
function pointers. Historically it was done manually by adding “const”
where needed.

Now with the gcc plugin it's the opposite. All structure like these are
const by default, and a no_const attribute has been added for some of
them (around a hundred afair).

Regards,
-- 
Yves-Alexis


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Julia Lawall]

On Tue, 22 Sep 2015, Dan Carpenter wrote:

> On Mon, Aug 24, 2015 at 09:35:50AM -0700, Kees Cook wrote:
> > As an example, making the kernel code memory read-only means an
> > attacker cannot just directly change the kernel's execution path when
> > they use an arbitrary memory-writing flaw.
>
> Part of the problem is that C defaults to writable and you have to
> explicitly add a const if you want it read only.  Then when you are
> reading the code and you see:
>
> static struct uvc_format_desc uvc_fmts[] = {
>
> You can't immediately tell if it is writeable because it needs to be or
> because that's just the default.  We should make a new keyword that
> means non constant then we could grep for structs which have a function
> pointer and complain if they weren't specified as const or __writable.
>
> Maybe there is a way to do this with GCC attributes but I have never
> used them before and I couldn't immediately see how to do it.

But the bigger problem is that some of these kinds of structs have a
single writeable field lurking inside fo them.  For example,
platform_driver contains struct device_driver driver; that is initialized
in the platform driver library, and not in the device driver itself.
Would it be beneficial to put const on the individual function-typed
fields, if that is indeed possible?  For example, I don't find any
assignments to a platform_driver remove field.

julia

Re: [Ksummit-discuss] [TECH TOPIC] Kernel Hardening

[Andy Lutomirski]

On Tue, Sep 22, 2015 at 5:59 AM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>
>
> On Tue, 22 Sep 2015, Dan Carpenter wrote:
>
>> On Mon, Aug 24, 2015 at 09:35:50AM -0700, Kees Cook wrote:
>> > As an example, making the kernel code memory read-only means an
>> > attacker cannot just directly change the kernel's execution path when
>> > they use an arbitrary memory-writing flaw.
>>
>> Part of the problem is that C defaults to writable and you have to
>> explicitly add a const if you want it read only.  Then when you are
>> reading the code and you see:
>>
>> static struct uvc_format_desc uvc_fmts[] = {
>>
>> You can't immediately tell if it is writeable because it needs to be or
>> because that's just the default.  We should make a new keyword that
>> means non constant then we could grep for structs which have a function
>> pointer and complain if they weren't specified as const or __writable.
>>
>> Maybe there is a way to do this with GCC attributes but I have never
>> used them before and I couldn't immediately see how to do it.
>
> But the bigger problem is that some of these kinds of structs have a
> single writeable field lurking inside fo them.  For example,
> platform_driver contains struct device_driver driver; that is initialized
> in the platform driver library, and not in the device driver itself.
> Would it be beneficial to put const on the individual function-typed
> fields, if that is indeed possible?  For example, I don't find any
> assignments to a platform_driver remove field.
>

PaX gets around this with special tricks to write to fields in
otherwise const structures in appropriately annotated code.  We could
clean that up and do it upstream.

--Andy

Re: [Ksummit-discuss] Self nomination

[Grant Likely]

On Mon, Aug 8, 2016 at 12:07 PM, Lorenzo Pieralisi
<lorenzo.pieralisi@arm.com> wrote:
> On Thu, Jul 28, 2016 at 11:14:51AM +0100, Marc Zyngier wrote:
>> On 26/07/16 23:30, Dmitry Torokhov wrote:
>> > I'd like to nominate myself for the kernel summit this year. I am part
>> > of Chrome OS kernel team and I also maintain drivers/input in mainline.
>>
>> [...]
>>
>> > - I would like to sync up with people and discuss [lack of] progress
>> >   on topic of device probe ordering (including handling of deferred
>> >   probes, asynchronous probes, etc).
>>
>> I'm extremely interested in discussing this.
>>
>> It has wide reaching consequences as (with my irqchip maintainer hat on)
>> we've had to pretend that some bits of HW (timers, interrupt
>> controllers) are not "devices". Not a massive issue for most, except
>> when your interrupt controller has requirements that are very similar to
>> the DMA mapping API (which you cannot use because "not a device"). Other
>> problems are introduced by things like wire-MSI bridges, and most people
>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
>> probes in specific drivers.
>>
>> I've seen a number of proposal so far, but the subject seems to have
>> gone quiet (well, not really, but hardly any progress has been made).
>>
>> Happy to make this a tech discussion or a hallway track.
>
> I am very interested in this discussion too in whatever form it takes
> place and I really think it is the right time to find a way forward
> for DT and ACPI probing alike given that we have started facing
> these probe ordering issues in ARM/ACPI world too, it would be nice to
> find a solution that works seamlessly.

[Responding very late to this thread]

I also want to be involved with this discussion. We've got to come up
with something better than what we've got now.

g.

Re: [Ksummit-discuss] Self nomination

[Rafael J. Wysocki]

On Wednesday, August 24, 2016 02:12:18 PM Marek Szyprowski wrote:
> Hi Rafael,
> 
> 
> On 2016-08-06 02:20, Rafael J. Wysocki wrote:
> > On Wednesday, August 03, 2016 10:12:00 AM Marek Szyprowski wrote:
> >> Dear All,
> >>
> >>
> >> On 2016-08-03 01:00, Rafael J. Wysocki wrote:
> >>> On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
> >>>> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> >>>>> On 26/07/16 23:30, Dmitry Torokhov wrote:
> >>>>>> - I would like to sync up with people and discuss [lack of] progress
> >>>>>>     on topic of device probe ordering (including handling of deferred
> >>>>>>     probes, asynchronous probes, etc).
> >>>>> I'm extremely interested in discussing this.
> >>>> I've also tried to pitch in on it in the past but I just feel stupid
> >>>> whenever we try to come up with something better than what
> >>>> we have :(
> >>>>
> >>>>> It has wide reaching consequences as (with my irqchip maintainer hat on)
> >>>>> we've had to pretend that some bits of HW (timers, interrupt
> >>>>> controllers) are not "devices". Not a massive issue for most, except
> >>>>> when your interrupt controller has requirements that are very similar to
> >>>>> the DMA mapping API (which you cannot use because "not a device"). Other
> >>>>> problems are introduced by things like wire-MSI bridges, and most people
> >>>>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
> >>>>> probes in specific drivers.
> >>>> Same feeling here. I'm accepting patches for random initcall
> >>>> reordering because there is nothing else I can do, people need to
> >>>> have their systems running. But it feels really fragile.
> >>>>
> >>>> Deferred probe alleviated the problem, but I remember saying at
> >>>> the time that what we really need to do is build a dependency
> >>>> graph and resolve it the same way e.g. systemd does. (Someone
> >>>> may have called me BS on that, either for being wrong about everything
> >>>> as usual or because of mentioning systemd, I don't know which one.)
> >>>>
> >>>> The latest proposal I saw came from Rafael and he had a scratch
> >>>> idea for a dependency graph that I really liked, but I guess he's been
> >>>> sidetracked since. Rafael, what happened with that?
> >>> I got distracted, but Marek Szyprowski has revived it recently.
> >>>
> >>> It needs to be cleaned up somewhat, but other than that I think it's in
> >>> a good enough shape to make some progress in that direction, at least in
> >>> principle.
> >> I really like the idea of pm dependencies between device and the patches
> >> prepared by Rafael. They are exactly what we need for our case (PM for
> >> Exynos IOMMU), but they will also help solving PM issues with complex
> >> devices (like DRM for SoCs and ASoC audio).
> >>
> >> Rafael: do you plan to do any update on them?
> > Yes, I do, but to make some cosmetic changes rather.
> >
> >> Some time ago you wrote, that you had such plan, but real life proved
> >> something else.
> > Well, I was working on other things in the meantime, but I still had that
> > plan. :-)
> >
> >> If needed I can continue works on them, but I need some directions what has
> >> to be improved and fixed.
> > Thanks so much!
> >
> > First off, the networking people claimed the "devlink" term in the meantime
> > and it's better to avoid confusion here, so I'd change it to "devdep" or
> > similar in the patches.
> >
> > In addition to that Tomeu Vizoso complained that the supplier_links and
> > consumer_links list heads in struct device were confusing and I see why that
> > could be the case, so I'd change them to something more direct, like maybe
> > links_to_suppliers and links_to_consumers.
> >
> > Please let me know what you think.
> 
> I think that both name changes (devlink -> devdep and adding "_to_") 
> make sense
> and such change will make the code easier to understand.
> 
> I've also managed to find what is the source of the problem with reboot 
> hang that
> Tobias reported. It was my fault of incorrect use of device links. 
> Adding a link
> to not-yet-fully-registered device results in trashing devices_kset and 
> dpm lists.
> I will check if it is possible to a warning or proper support for such case
> (iommu support for given device is initialized before given device's 
> struct device
> is added to the system by device_add() function).
> 
> Do you want me to resend the patches with the above mentioned name 
> changes or do
> you want to do it on your own and then I will send my updated patches?

I'd like to post new versions myself and then let you rebase on top of
them if that's not a problem.

I'll get to that when I get back home from the current travels.

Thanks,
Rafael

Re: [Ksummit-discuss] Self nomination

[Rafael J. Wysocki]

On Wednesday, August 24, 2016 02:12:18 PM Marek Szyprowski wrote:
> Hi Rafael,
> 
> 
> On 2016-08-06 02:20, Rafael J. Wysocki wrote:
> > On Wednesday, August 03, 2016 10:12:00 AM Marek Szyprowski wrote:
> >> Dear All,
> >>
> >>
> >> On 2016-08-03 01:00, Rafael J. Wysocki wrote:
> >>> On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
> >>>> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> >>>>> On 26/07/16 23:30, Dmitry Torokhov wrote:
> >>>>>> - I would like to sync up with people and discuss [lack of] progress
> >>>>>>     on topic of device probe ordering (including handling of deferred
> >>>>>>     probes, asynchronous probes, etc).
> >>>>> I'm extremely interested in discussing this.
> >>>> I've also tried to pitch in on it in the past but I just feel stupid
> >>>> whenever we try to come up with something better than what
> >>>> we have :(
> >>>>
> >>>>> It has wide reaching consequences as (with my irqchip maintainer hat on)
> >>>>> we've had to pretend that some bits of HW (timers, interrupt
> >>>>> controllers) are not "devices". Not a massive issue for most, except
> >>>>> when your interrupt controller has requirements that are very similar to
> >>>>> the DMA mapping API (which you cannot use because "not a device"). Other
> >>>>> problems are introduced by things like wire-MSI bridges, and most people
> >>>>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
> >>>>> probes in specific drivers.
> >>>> Same feeling here. I'm accepting patches for random initcall
> >>>> reordering because there is nothing else I can do, people need to
> >>>> have their systems running. But it feels really fragile.
> >>>>
> >>>> Deferred probe alleviated the problem, but I remember saying at
> >>>> the time that what we really need to do is build a dependency
> >>>> graph and resolve it the same way e.g. systemd does. (Someone
> >>>> may have called me BS on that, either for being wrong about everything
> >>>> as usual or because of mentioning systemd, I don't know which one.)
> >>>>
> >>>> The latest proposal I saw came from Rafael and he had a scratch
> >>>> idea for a dependency graph that I really liked, but I guess he's been
> >>>> sidetracked since. Rafael, what happened with that?
> >>> I got distracted, but Marek Szyprowski has revived it recently.
> >>>
> >>> It needs to be cleaned up somewhat, but other than that I think it's in
> >>> a good enough shape to make some progress in that direction, at least in
> >>> principle.
> >> I really like the idea of pm dependencies between device and the patches
> >> prepared by Rafael. They are exactly what we need for our case (PM for
> >> Exynos IOMMU), but they will also help solving PM issues with complex
> >> devices (like DRM for SoCs and ASoC audio).
> >>
> >> Rafael: do you plan to do any update on them?
> > Yes, I do, but to make some cosmetic changes rather.
> >
> >> Some time ago you wrote, that you had such plan, but real life proved
> >> something else.
> > Well, I was working on other things in the meantime, but I still had that
> > plan. :-)
> >
> >> If needed I can continue works on them, but I need some directions what has
> >> to be improved and fixed.
> > Thanks so much!
> >
> > First off, the networking people claimed the "devlink" term in the meantime
> > and it's better to avoid confusion here, so I'd change it to "devdep" or
> > similar in the patches.
> >
> > In addition to that Tomeu Vizoso complained that the supplier_links and
> > consumer_links list heads in struct device were confusing and I see why that
> > could be the case, so I'd change them to something more direct, like maybe
> > links_to_suppliers and links_to_consumers.
> >
> > Please let me know what you think.
> 
> I think that both name changes (devlink -> devdep and adding "_to_") 
> make sense
> and such change will make the code easier to understand.
> 
> I've also managed to find what is the source of the problem with reboot 
> hang that
> Tobias reported. It was my fault of incorrect use of device links. 
> Adding a link
> to not-yet-fully-registered device results in trashing devices_kset and 
> dpm lists.
> I will check if it is possible to a warning or proper support for such case
> (iommu support for given device is initialized before given device's 
> struct device
> is added to the system by device_add() function).
> 
> Do you want me to resend the patches with the above mentioned name 
> changes or do
> you want to do it on your own and then I will send my updated patches?

I'd like to post new versions myself and then let you rebase on top of
them if that's not a problem.

I'll get to that when I get back home from the current travels.

Thanks,
Rafael

Re: [Ksummit-discuss] Self nomination

[Marek Szyprowski]

Hi Rafael,


On 2016-08-06 02:20, Rafael J. Wysocki wrote:
> On Wednesday, August 03, 2016 10:12:00 AM Marek Szyprowski wrote:
>> Dear All,
>>
>>
>> On 2016-08-03 01:00, Rafael J. Wysocki wrote:
>>> On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
>>>> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>>> On 26/07/16 23:30, Dmitry Torokhov wrote:
>>>>>> - I would like to sync up with people and discuss [lack of] progress
>>>>>>     on topic of device probe ordering (including handling of deferred
>>>>>>     probes, asynchronous probes, etc).
>>>>> I'm extremely interested in discussing this.
>>>> I've also tried to pitch in on it in the past but I just feel stupid
>>>> whenever we try to come up with something better than what
>>>> we have :(
>>>>
>>>>> It has wide reaching consequences as (with my irqchip maintainer hat on)
>>>>> we've had to pretend that some bits of HW (timers, interrupt
>>>>> controllers) are not "devices". Not a massive issue for most, except
>>>>> when your interrupt controller has requirements that are very similar to
>>>>> the DMA mapping API (which you cannot use because "not a device"). Other
>>>>> problems are introduced by things like wire-MSI bridges, and most people
>>>>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
>>>>> probes in specific drivers.
>>>> Same feeling here. I'm accepting patches for random initcall
>>>> reordering because there is nothing else I can do, people need to
>>>> have their systems running. But it feels really fragile.
>>>>
>>>> Deferred probe alleviated the problem, but I remember saying at
>>>> the time that what we really need to do is build a dependency
>>>> graph and resolve it the same way e.g. systemd does. (Someone
>>>> may have called me BS on that, either for being wrong about everything
>>>> as usual or because of mentioning systemd, I don't know which one.)
>>>>
>>>> The latest proposal I saw came from Rafael and he had a scratch
>>>> idea for a dependency graph that I really liked, but I guess he's been
>>>> sidetracked since. Rafael, what happened with that?
>>> I got distracted, but Marek Szyprowski has revived it recently.
>>>
>>> It needs to be cleaned up somewhat, but other than that I think it's in
>>> a good enough shape to make some progress in that direction, at least in
>>> principle.
>> I really like the idea of pm dependencies between device and the patches
>> prepared by Rafael. They are exactly what we need for our case (PM for
>> Exynos IOMMU), but they will also help solving PM issues with complex
>> devices (like DRM for SoCs and ASoC audio).
>>
>> Rafael: do you plan to do any update on them?
> Yes, I do, but to make some cosmetic changes rather.
>
>> Some time ago you wrote, that you had such plan, but real life proved
>> something else.
> Well, I was working on other things in the meantime, but I still had that
> plan. :-)
>
>> If needed I can continue works on them, but I need some directions what has
>> to be improved and fixed.
> Thanks so much!
>
> First off, the networking people claimed the "devlink" term in the meantime
> and it's better to avoid confusion here, so I'd change it to "devdep" or
> similar in the patches.
>
> In addition to that Tomeu Vizoso complained that the supplier_links and
> consumer_links list heads in struct device were confusing and I see why that
> could be the case, so I'd change them to something more direct, like maybe
> links_to_suppliers and links_to_consumers.
>
> Please let me know what you think.

I think that both name changes (devlink -> devdep and adding "_to_") 
make sense
and such change will make the code easier to understand.

I've also managed to find what is the source of the problem with reboot 
hang that
Tobias reported. It was my fault of incorrect use of device links. 
Adding a link
to not-yet-fully-registered device results in trashing devices_kset and 
dpm lists.
I will check if it is possible to a warning or proper support for such case
(iommu support for given device is initialized before given device's 
struct device
is added to the system by device_add() function).

Do you want me to resend the patches with the above mentioned name 
changes or do
you want to do it on your own and then I will send my updated patches?

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [Ksummit-discuss] Self nomination

[Marek Szyprowski]

Hi Rafael,


On 2016-08-06 02:20, Rafael J. Wysocki wrote:
> On Wednesday, August 03, 2016 10:12:00 AM Marek Szyprowski wrote:
>> Dear All,
>>
>>
>> On 2016-08-03 01:00, Rafael J. Wysocki wrote:
>>> On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
>>>> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>>> On 26/07/16 23:30, Dmitry Torokhov wrote:
>>>>>> - I would like to sync up with people and discuss [lack of] progress
>>>>>>     on topic of device probe ordering (including handling of deferred
>>>>>>     probes, asynchronous probes, etc).
>>>>> I'm extremely interested in discussing this.
>>>> I've also tried to pitch in on it in the past but I just feel stupid
>>>> whenever we try to come up with something better than what
>>>> we have :(
>>>>
>>>>> It has wide reaching consequences as (with my irqchip maintainer hat on)
>>>>> we've had to pretend that some bits of HW (timers, interrupt
>>>>> controllers) are not "devices". Not a massive issue for most, except
>>>>> when your interrupt controller has requirements that are very similar to
>>>>> the DMA mapping API (which you cannot use because "not a device"). Other
>>>>> problems are introduced by things like wire-MSI bridges, and most people
>>>>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
>>>>> probes in specific drivers.
>>>> Same feeling here. I'm accepting patches for random initcall
>>>> reordering because there is nothing else I can do, people need to
>>>> have their systems running. But it feels really fragile.
>>>>
>>>> Deferred probe alleviated the problem, but I remember saying at
>>>> the time that what we really need to do is build a dependency
>>>> graph and resolve it the same way e.g. systemd does. (Someone
>>>> may have called me BS on that, either for being wrong about everything
>>>> as usual or because of mentioning systemd, I don't know which one.)
>>>>
>>>> The latest proposal I saw came from Rafael and he had a scratch
>>>> idea for a dependency graph that I really liked, but I guess he's been
>>>> sidetracked since. Rafael, what happened with that?
>>> I got distracted, but Marek Szyprowski has revived it recently.
>>>
>>> It needs to be cleaned up somewhat, but other than that I think it's in
>>> a good enough shape to make some progress in that direction, at least in
>>> principle.
>> I really like the idea of pm dependencies between device and the patches
>> prepared by Rafael. They are exactly what we need for our case (PM for
>> Exynos IOMMU), but they will also help solving PM issues with complex
>> devices (like DRM for SoCs and ASoC audio).
>>
>> Rafael: do you plan to do any update on them?
> Yes, I do, but to make some cosmetic changes rather.
>
>> Some time ago you wrote, that you had such plan, but real life proved
>> something else.
> Well, I was working on other things in the meantime, but I still had that
> plan. :-)
>
>> If needed I can continue works on them, but I need some directions what has
>> to be improved and fixed.
> Thanks so much!
>
> First off, the networking people claimed the "devlink" term in the meantime
> and it's better to avoid confusion here, so I'd change it to "devdep" or
> similar in the patches.
>
> In addition to that Tomeu Vizoso complained that the supplier_links and
> consumer_links list heads in struct device were confusing and I see why that
> could be the case, so I'd change them to something more direct, like maybe
> links_to_suppliers and links_to_consumers.
>
> Please let me know what you think.

I think that both name changes (devlink -> devdep and adding "_to_") 
make sense
and such change will make the code easier to understand.

I've also managed to find what is the source of the problem with reboot 
hang that
Tobias reported. It was my fault of incorrect use of device links. 
Adding a link
to not-yet-fully-registered device results in trashing devices_kset and 
dpm lists.
I will check if it is possible to a warning or proper support for such case
(iommu support for given device is initialized before given device's 
struct device
is added to the system by device_add() function).

Do you want me to resend the patches with the above mentioned name 
changes or do
you want to do it on your own and then I will send my updated patches?

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [Ksummit-discuss] Self nomination

[Lorenzo Pieralisi]

On Thu, Jul 28, 2016 at 11:14:51AM +0100, Marc Zyngier wrote:
> On 26/07/16 23:30, Dmitry Torokhov wrote:
> > I'd like to nominate myself for the kernel summit this year. I am part
> > of Chrome OS kernel team and I also maintain drivers/input in mainline.
> 
> [...]
> 
> > - I would like to sync up with people and discuss [lack of] progress
> >   on topic of device probe ordering (including handling of deferred
> >   probes, asynchronous probes, etc).
> 
> I'm extremely interested in discussing this.
> 
> It has wide reaching consequences as (with my irqchip maintainer hat on)
> we've had to pretend that some bits of HW (timers, interrupt
> controllers) are not "devices". Not a massive issue for most, except
> when your interrupt controller has requirements that are very similar to
> the DMA mapping API (which you cannot use because "not a device"). Other
> problems are introduced by things like wire-MSI bridges, and most people
> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
> probes in specific drivers.
> 
> I've seen a number of proposal so far, but the subject seems to have
> gone quiet (well, not really, but hardly any progress has been made).
> 
> Happy to make this a tech discussion or a hallway track.

I am very interested in this discussion too in whatever form it takes
place and I really think it is the right time to find a way forward
for DT and ACPI probing alike given that we have started facing
these probe ordering issues in ARM/ACPI world too, it would be nice to
find a solution that works seamlessly.

Lorenzo

Re: [Ksummit-discuss] Self nomination

[Rafael J. Wysocki]

On Wednesday, August 03, 2016 10:12:00 AM Marek Szyprowski wrote:
> Dear All,
> 
> 
> On 2016-08-03 01:00, Rafael J. Wysocki wrote:
> > On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
> >> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> >>> On 26/07/16 23:30, Dmitry Torokhov wrote:
> >>>> - I would like to sync up with people and discuss [lack of] progress
> >>>>    on topic of device probe ordering (including handling of deferred
> >>>>    probes, asynchronous probes, etc).
> >>> I'm extremely interested in discussing this.
> >> I've also tried to pitch in on it in the past but I just feel stupid
> >> whenever we try to come up with something better than what
> >> we have :(
> >>
> >>> It has wide reaching consequences as (with my irqchip maintainer hat on)
> >>> we've had to pretend that some bits of HW (timers, interrupt
> >>> controllers) are not "devices". Not a massive issue for most, except
> >>> when your interrupt controller has requirements that are very similar to
> >>> the DMA mapping API (which you cannot use because "not a device"). Other
> >>> problems are introduced by things like wire-MSI bridges, and most people
> >>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
> >>> probes in specific drivers.
> >> Same feeling here. I'm accepting patches for random initcall
> >> reordering because there is nothing else I can do, people need to
> >> have their systems running. But it feels really fragile.
> >>
> >> Deferred probe alleviated the problem, but I remember saying at
> >> the time that what we really need to do is build a dependency
> >> graph and resolve it the same way e.g. systemd does. (Someone
> >> may have called me BS on that, either for being wrong about everything
> >> as usual or because of mentioning systemd, I don't know which one.)
> >>
> >> The latest proposal I saw came from Rafael and he had a scratch
> >> idea for a dependency graph that I really liked, but I guess he's been
> >> sidetracked since. Rafael, what happened with that?
> > I got distracted, but Marek Szyprowski has revived it recently.
> >
> > It needs to be cleaned up somewhat, but other than that I think it's in
> > a good enough shape to make some progress in that direction, at least in
> > principle.
> 
> I really like the idea of pm dependencies between device and the patches
> prepared by Rafael. They are exactly what we need for our case (PM for
> Exynos IOMMU), but they will also help solving PM issues with complex
> devices (like DRM for SoCs and ASoC audio).
> 
> Rafael: do you plan to do any update on them?

Yes, I do, but to make some cosmetic changes rather.

> Some time ago you wrote, that you had such plan, but real life proved
> something else.

Well, I was working on other things in the meantime, but I still had that
plan. :-)

> If needed I can continue works on them, but I need some directions what has
> to be improved and fixed.

Thanks so much!

First off, the networking people claimed the "devlink" term in the meantime
and it's better to avoid confusion here, so I'd change it to "devdep" or
similar in the patches.

In addition to that Tomeu Vizoso complained that the supplier_links and
consumer_links list heads in struct device were confusing and I see why that
could be the case, so I'd change them to something more direct, like maybe
links_to_suppliers and links_to_consumers.

Please let me know what you think.

Thanks,
Rafael

Re: [Ksummit-discuss] Self nomination

[Rafael J. Wysocki]

On Wednesday, August 03, 2016 10:12:00 AM Marek Szyprowski wrote:
> Dear All,
> 
> 
> On 2016-08-03 01:00, Rafael J. Wysocki wrote:
> > On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
> >> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> >>> On 26/07/16 23:30, Dmitry Torokhov wrote:
> >>>> - I would like to sync up with people and discuss [lack of] progress
> >>>>    on topic of device probe ordering (including handling of deferred
> >>>>    probes, asynchronous probes, etc).
> >>> I'm extremely interested in discussing this.
> >> I've also tried to pitch in on it in the past but I just feel stupid
> >> whenever we try to come up with something better than what
> >> we have :(
> >>
> >>> It has wide reaching consequences as (with my irqchip maintainer hat on)
> >>> we've had to pretend that some bits of HW (timers, interrupt
> >>> controllers) are not "devices". Not a massive issue for most, except
> >>> when your interrupt controller has requirements that are very similar to
> >>> the DMA mapping API (which you cannot use because "not a device"). Other
> >>> problems are introduced by things like wire-MSI bridges, and most people
> >>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
> >>> probes in specific drivers.
> >> Same feeling here. I'm accepting patches for random initcall
> >> reordering because there is nothing else I can do, people need to
> >> have their systems running. But it feels really fragile.
> >>
> >> Deferred probe alleviated the problem, but I remember saying at
> >> the time that what we really need to do is build a dependency
> >> graph and resolve it the same way e.g. systemd does. (Someone
> >> may have called me BS on that, either for being wrong about everything
> >> as usual or because of mentioning systemd, I don't know which one.)
> >>
> >> The latest proposal I saw came from Rafael and he had a scratch
> >> idea for a dependency graph that I really liked, but I guess he's been
> >> sidetracked since. Rafael, what happened with that?
> > I got distracted, but Marek Szyprowski has revived it recently.
> >
> > It needs to be cleaned up somewhat, but other than that I think it's in
> > a good enough shape to make some progress in that direction, at least in
> > principle.
> 
> I really like the idea of pm dependencies between device and the patches
> prepared by Rafael. They are exactly what we need for our case (PM for
> Exynos IOMMU), but they will also help solving PM issues with complex
> devices (like DRM for SoCs and ASoC audio).
> 
> Rafael: do you plan to do any update on them?

Yes, I do, but to make some cosmetic changes rather.

> Some time ago you wrote, that you had such plan, but real life proved
> something else.

Well, I was working on other things in the meantime, but I still had that
plan. :-)

> If needed I can continue works on them, but I need some directions what has
> to be improved and fixed.

Thanks so much!

First off, the networking people claimed the "devlink" term in the meantime
and it's better to avoid confusion here, so I'd change it to "devdep" or
similar in the patches.

In addition to that Tomeu Vizoso complained that the supplier_links and
consumer_links list heads in struct device were confusing and I see why that
could be the case, so I'd change them to something more direct, like maybe
links_to_suppliers and links_to_consumers.

Please let me know what you think.

Thanks,
Rafael

Re: [Ksummit-discuss] Self nomination

[Geert Uytterhoeven]

On Wed, Jul 27, 2016 at 11:25 AM, Linus Walleij
<linus.walleij@linaro.org> wrote:
> Qualifying Linux for Functional Safety
> http://events.linuxfoundation.org/sites/events/files/slides/20160713_SIL2LinuxMP_Min_ALS_0.9_pub.pdf

404

http://events.linuxfoundation.org/sites/events/files/slides/20160713_SIL2LinuxMP_Min_ALS_1.1.pdf

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] Self nomination

[Marek Szyprowski]

Dear All,


On 2016-08-03 01:00, Rafael J. Wysocki wrote:
> On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
>> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>> On 26/07/16 23:30, Dmitry Torokhov wrote:
>>>> - I would like to sync up with people and discuss [lack of] progress
>>>>    on topic of device probe ordering (including handling of deferred
>>>>    probes, asynchronous probes, etc).
>>> I'm extremely interested in discussing this.
>> I've also tried to pitch in on it in the past but I just feel stupid
>> whenever we try to come up with something better than what
>> we have :(
>>
>>> It has wide reaching consequences as (with my irqchip maintainer hat on)
>>> we've had to pretend that some bits of HW (timers, interrupt
>>> controllers) are not "devices". Not a massive issue for most, except
>>> when your interrupt controller has requirements that are very similar to
>>> the DMA mapping API (which you cannot use because "not a device"). Other
>>> problems are introduced by things like wire-MSI bridges, and most people
>>> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
>>> probes in specific drivers.
>> Same feeling here. I'm accepting patches for random initcall
>> reordering because there is nothing else I can do, people need to
>> have their systems running. But it feels really fragile.
>>
>> Deferred probe alleviated the problem, but I remember saying at
>> the time that what we really need to do is build a dependency
>> graph and resolve it the same way e.g. systemd does. (Someone
>> may have called me BS on that, either for being wrong about everything
>> as usual or because of mentioning systemd, I don't know which one.)
>>
>> The latest proposal I saw came from Rafael and he had a scratch
>> idea for a dependency graph that I really liked, but I guess he's been
>> sidetracked since. Rafael, what happened with that?
> I got distracted, but Marek Szyprowski has revived it recently.
>
> It needs to be cleaned up somewhat, but other than that I think it's in
> a good enough shape to make some progress in that direction, at least in
> principle.

I really like the idea of pm dependencies between device and the patches
prepared by Rafael. They are exactly what we need for our case (PM for
Exynos IOMMU), but they will also help solving PM issues with complex
devices (like DRM for SoCs and ASoC audio).

Rafael: do you plan to do any update on them? Some time ago you wrote,
that you had such plan, but real life proved something else. If needed
I can continue works on them, but I need some directions what has to be
improved and fixed.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [Ksummit-discuss] Self nomination

[Rafael J. Wysocki]

On Tuesday, August 02, 2016 10:09:17 AM Linus Walleij wrote:
> On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> > On 26/07/16 23:30, Dmitry Torokhov wrote:
> 
> >> - I would like to sync up with people and discuss [lack of] progress
> >>   on topic of device probe ordering (including handling of deferred
> >>   probes, asynchronous probes, etc).
> >
> > I'm extremely interested in discussing this.
> 
> I've also tried to pitch in on it in the past but I just feel stupid
> whenever we try to come up with something better than what
> we have :(
> 
> > It has wide reaching consequences as (with my irqchip maintainer hat on)
> > we've had to pretend that some bits of HW (timers, interrupt
> > controllers) are not "devices". Not a massive issue for most, except
> > when your interrupt controller has requirements that are very similar to
> > the DMA mapping API (which you cannot use because "not a device"). Other
> > problems are introduced by things like wire-MSI bridges, and most people
> > end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
> > probes in specific drivers.
> 
> Same feeling here. I'm accepting patches for random initcall
> reordering because there is nothing else I can do, people need to
> have their systems running. But it feels really fragile.
> 
> Deferred probe alleviated the problem, but I remember saying at
> the time that what we really need to do is build a dependency
> graph and resolve it the same way e.g. systemd does. (Someone
> may have called me BS on that, either for being wrong about everything
> as usual or because of mentioning systemd, I don't know which one.)
> 
> The latest proposal I saw came from Rafael and he had a scratch
> idea for a dependency graph that I really liked, but I guess he's been
> sidetracked since. Rafael, what happened with that?

I got distracted, but Marek Szyprowski has revived it recently.

It needs to be cleaned up somewhat, but other than that I think it's in
a good enough shape to make some progress in that direction, at least in
principle.

Thanks,
Rafael

Re: [Ksummit-discuss] Self nomination

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

On Sun, Jul 31, 2016 at 08:57:18AM +0200, Olof Johansson wrote:

> * Stable tree discussions. Having the discussion at last KS about
> timing of releases was hugely beneficial, picking 4.4 as a solid base
> for the year's devices.

Definitely.  I'm seeing both companies who have been able to adopt v4.4
and companies who are seeing costs from not jumping forward as everyone
else did and so are likely to move forward if the same pattern is
followed again.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] Self nomination

[Linus Walleij]

On Thu, Jul 28, 2016 at 12:14 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On 26/07/16 23:30, Dmitry Torokhov wrote:

>> - I would like to sync up with people and discuss [lack of] progress
>>   on topic of device probe ordering (including handling of deferred
>>   probes, asynchronous probes, etc).
>
> I'm extremely interested in discussing this.

I've also tried to pitch in on it in the past but I just feel stupid
whenever we try to come up with something better than what
we have :(

> It has wide reaching consequences as (with my irqchip maintainer hat on)
> we've had to pretend that some bits of HW (timers, interrupt
> controllers) are not "devices". Not a massive issue for most, except
> when your interrupt controller has requirements that are very similar to
> the DMA mapping API (which you cannot use because "not a device"). Other
> problems are introduced by things like wire-MSI bridges, and most people
> end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
> probes in specific drivers.

Same feeling here. I'm accepting patches for random initcall
reordering because there is nothing else I can do, people need to
have their systems running. But it feels really fragile.

Deferred probe alleviated the problem, but I remember saying at
the time that what we really need to do is build a dependency
graph and resolve it the same way e.g. systemd does. (Someone
may have called me BS on that, either for being wrong about everything
as usual or because of mentioning systemd, I don't know which one.)

The latest proposal I saw came from Rafael and he had a scratch
idea for a dependency graph that I really liked, but I guess he's been
sidetracked since. Rafael, what happened with that?

Yours,
Linus Walleij

[Ksummit-discuss] Self nomination

[Olof Johansson]

Hi,

I'd like to nominate myself to the kernel summit this year.

I co-maintain arm-soc together with Arnd Bergmann (and Kevin Hilman),
often merging code that spans between our tree and various other
subsystem. Meeting other maintainers at Kernel Summit is usually
highly beneficial since it makes the whole process run smoother.

Topics I care about:

* Embedded/mobile platforms: We merge embedded as well as server
platform code, and interact with a lot of the various camps, including
the people stuck between the nasty downstream vendor trees and
upstream.
* Stable tree discussions. Having the discussion at last KS about
timing of releases was hugely beneficial, picking 4.4 as a solid base
for the year's devices.
* Maintainership, process discussions. arm-soc is one of the fairly
smooth-running maintainer groups but always looking to improve (and
share our experiences)
* Testing and making it easier for people to get tests in with their
code, even if we don't mandate it.
* Devicetree and platform description discussions always seem to be
going on, even when we think we've sorted out most existing issues. I
care a good amount about these as well.


-Olof

[Ksummit-discuss] Self nomination

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

I am nominating myself to attend the kernel summit.

I continue to maintain Debian's kernel package (unstable and stable
branches), and the 3.2 and 3.16 longterm branches at kernel.org.

I'm interested in:

- Stable/longterm maintenance
- Tracking (or avoiding) regressions
- Code signing for modules, firmware, etc.  (Debian is now starting to
  implement this.)
  - Relatedly, the securelevel patches or similar restrictions on
    userland to prevent subversion of code signing
- Documentation - making sure the post-DocBook documentation remains
  packageable and reproducible

Ben.

-- 

Ben Hutchings
Experience is directly proportional to the value of equipment
destroyed.
                                                         - Carolyn
Scheppner

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] self nomination

[Davidlohr Bueso]

On Fri, 29 Jul 2016, Wangnan (F) wrote:

>Yes, we need a performance related session. I posted a proposal but
>not very responsive.
>
>I can talk about tracing and tools/perf, especially the progress of
>BPF. We can discuss something about performance profiling based on
>these tracing result.

Fair enough, although I was referring to something a little more tangible,
such as specific pressure points and bottlenecks in (hopefully) real
workloads/systems. In the past this has worked well, and some issues that
have been brought to our attention have been addressed.

Like others already said, this obviously requires enough interest and
content; so other performance related topics could fit as well.

Thanks,
Davidlohr

[Ksummit-discuss] self nomination

[Mimi Zohar]

Hi,

I would like to be included on any discussions relating to:
- key management/trust
- signature verification

I would be willing to give a status update on closing file
measurement/appraisal (signature verification) gaps, and hopefully
impress upon kernel developers not to introduce new gaps.

Mimi

[Ksummit-discuss] Self nomination

[Bartlomiej Zolnierkiewicz]

Hello,

I would like to self-nominate for the Kernel Summit 2016.

I'm the leader of the Tizen Kernel team in Samsung R&D Institute
Poland (SRPOL) which (together with Korean HQ Kernel team) develops
Linux Kernel for Tizen.org reference devices and works on support
for ARM Samsung Exynos SoCs in the mainline kernel.  I'm also
currently co-maintainer of mainline libata PATA drivers.

I'm particularly interested in the following topics proposed this
year:

- Kernel unit testing and generally all topics related to kernel
  testing.

  We have developed internal Tizen kernel testing system in SRPOL
  and are still working on enhancing it.  If possible in the future
  we would like to make it (or at least some parts of it) available
  to external developers (i.e. using kernelci.org infrastructure).

  I would also like to discuss the ways to make it easier for testing
  kernels by developers on old and/or rare devices (especially in
  the context of IDE subsystem to libata PATA migration).

- stable workflow - we are basing Tizen.org kernels on LTS kernels
  so any ways to improve the stable/LTS kernels quality are in
  our area of interest.

- Bus IPC.  I can act as a bridge to developers from Tizen System
  team in SRPOL who are working on IPC in Tizen (dbus/kdbus).  I can
  provide input to the discussion from Tizen's requirements POV (i.e.
  we need our IPC to be container-aware).

Thank you for considering.

Best regards,
--
Bartlomiej Zolnierkiewicz
Samsung R&D Institute Poland
Samsung Electronics

Re: [Ksummit-discuss] self nomination

[Wangnan (F)]

On 2016/7/28 7:20, Davidlohr Bueso wrote:
> Hi,
>
> I would like to nominate myself for kernel summit this year.
>
> I hack on various core subsystems mainly in the name of performance and
> am particularly interested in the following topics/discussions:
>
> - C++11 atomics/kernel memory model.
> - Upstreaming PREEMPT_RT.
> - Regression tracking.
> - (and to a lesser degree unit testing)
>
> In addition, I'm wondering if there's any interest in a performance
> session (getting to know some of the bottlenecks that are currently
> making folks cry), like Chris Mason has been doing in past kernel
> summits for fb specific workloads.
>

Yes, we need a performance related session. I posted a proposal but
not very responsive.

I can talk about tracing and tools/perf, especially the progress of
BPF. We can discuss something about performance profiling based on
these tracing result.

Thank you.

> Thanks,
> Davidlohr
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] self nomination

[James Bottomley]

On Thu, 2016-07-28 at 13:29 -0400, James Bottomley wrote:
> I have been known to be OK with the occasional storage or obscure non
> -x86 architecture or even the occasional mm issues.
> 
> I'm principally interested in
> 
>  * Storage issues (not that these ever come up at the kernel summit)
>  * In kernel container technologies (gcroups, namespaces and how we
>    apply the granular virtualizations) since this has been one of my
>    foci for five years now.
>  * Security, keys and the TPM.  I'm mostly working on this above the
>    kernel for a more secure cloud environment, but this means I have a
>    lot of TSS and TPM knowledge and I'd love to share the pain ...

I suppose I should add that running a top level tree, I have an
interest in process issues.  I'd be somewhat interested in a stable
tree discussion but only really to make sure it doesn't increase my
maintainer workload ...

James

[Ksummit-discuss] self nomination

[James Bottomley]

I have been known to be OK with the occasional storage or obscure non
-x86 architecture or even the occasional mm issues.

I'm principally interested in

 * Storage issues (not that these ever come up at the kernel summit)
 * In kernel container technologies (gcroups, namespaces and how we
   apply the granular virtualizations) since this has been one of my
   foci for five years now.
 * Security, keys and the TPM.  I'm mostly working on this above the
   kernel for a more secure cloud environment, but this means I have a
   lot of TSS and TPM knowledge and I'd love to share the pain ...

James

Re: [Ksummit-discuss] self nomination

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

On Wed, 2016-07-27 at 16:20 -0700, Davidlohr Bueso wrote:
> 
> - Upstreaming PREEMPT_RT.

> In addition, I'm wondering if there's any interest in a performance
> session (getting to know some of the bottlenecks that are currently
> making folks cry), like Chris Mason has been doing in past kernel
> summits for fb specific workloads.

I would be interested in these topics, too.

I guess I should nominate myself, in case either
Johannes's MM topic comes up, or the above topics.

-- 
All rights reversed

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] Self nomination

[Luis de Bethencourt]

On 27/07/16 00:59, Stephen Rothwell wrote:
> I'd like to self nominate for the kernel summit this year.
> 
> As the linux-next maintainer, I have interest in most process topics in
> particular the maintainership processes.
> 
> Also, being a bit isolated down under, meeting in person allows me to
> remember that you are all people really :-)
> 

I am also interested in the maintainership processes. I think these need
to be better defined and documented.

Linux is a project that would benefit of having a bigger revolving door of
new joiners, at all levels.

Thanks Stephen,
Luis

Re: [Ksummit-discuss] Self nomination

[Marc Zyngier]

On 26/07/16 23:30, Dmitry Torokhov wrote:
> I'd like to nominate myself for the kernel summit this year. I am part
> of Chrome OS kernel team and I also maintain drivers/input in mainline.

[...]

> - I would like to sync up with people and discuss [lack of] progress
>   on topic of device probe ordering (including handling of deferred
>   probes, asynchronous probes, etc).

I'm extremely interested in discussing this.

It has wide reaching consequences as (with my irqchip maintainer hat on)
we've had to pretend that some bits of HW (timers, interrupt
controllers) are not "devices". Not a massive issue for most, except
when your interrupt controller has requirements that are very similar to
the DMA mapping API (which you cannot use because "not a device"). Other
problems are introduced by things like wire-MSI bridges, and most people
end-up resorting to hacks like ad-hoc initcalls and sprinkling deferred
probes in specific drivers.

I've seen a number of proposal so far, but the subject seems to have
gone quiet (well, not really, but hardly any progress has been made).

Happy to make this a tech discussion or a hallway track.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [Ksummit-discuss] self nomination

[Jan Kara]

Hi!

On Wed 27-07-16 16:20:54, Davidlohr Bueso wrote:
> In addition, I'm wondering if there's any interest in a performance
> session (getting to know some of the bottlenecks that are currently
> making folks cry), like Chris Mason has been doing in past kernel
> summits for fb specific workloads.

I'd be interested in that session. But we need to find enough content. We
can certainly talk about our experiences with updating kernel for SLE12 SP2
(to 4.4) but I don't think it would fill the whole session. So it would
be good if other "vendors" shared their experiences as well.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[Ksummit-discuss] self nomination

[Davidlohr Bueso]

Hi,

I would like to nominate myself for kernel summit this year.

I hack on various core subsystems mainly in the name of performance and
am particularly interested in the following topics/discussions:

- C++11 atomics/kernel memory model.
- Upstreaming PREEMPT_RT.
- Regression tracking.
- (and to a lesser degree unit testing)

In addition, I'm wondering if there's any interest in a performance
session (getting to know some of the bottlenecks that are currently
making folks cry), like Chris Mason has been doing in past kernel
summits for fb specific workloads.

Thanks,
Davidlohr

Re: [Ksummit-discuss] Self nomination

[Darren Hart]

On Wed, Jul 27, 2016 at 11:25:52AM +0200, Linus Walleij wrote:
> On Wed, Jul 27, 2016 at 6:46 AM, Darren Hart <dvhart@infradead.org> wrote:
> 
> >   - Developing a "safety culture" and any overlap that may have with security
> 
> Do you mean safety as in "Linux in airbags and smoke detectors"?

It's more like Linux in consolidated automotive ECUs, industrial robot and
control, medical devices, avionics, etc. When the computational requirements of
safety critical systems exceed the capabilities of the traditional MCUs.
Automotive and Industrial Control are two of the driving forces currently.

> 
> This area is interesting for GPIO and IIO as well for natural reasons:
> these systems all tend to use GPIO and sensors. (Albeit more often
> than not with some horrific userspace hodge-podge but this is not the
> time to be grumpy about that.)
> 
> This presentation appeared at LinuxCon Japan (got the link from
> my colleague Takahiro Akashi):
> 
> Qualifying Linux for Functional Safety
> http://events.linuxfoundation.org/sites/events/files/slides/20160713_SIL2LinuxMP_Min_ALS_0.9_pub.pdf
> 

There are others, but this one is the core of the approach to a successful
compliance route, and is based on the most rigorous and experienced approach.
Nicholas is a well respected leader in this area. It may be worth inviting him
to participate in this tech topic. I'll see what his interest level is.

> I have been in contact with a Swedish company working in the fire alarm
> business as well.
> 
> My overall feeling is that world regulations (standards) on safety-critical
> software seem to be centered around code inspection.
> 
> These regulations approach is not to trust any third party but have all
> code inspected by independent reviewers for functional safety. All the
> time. For every new deployment. Kernel, libc, busybox userspace.
> (Or whatever they use.)

This is true, and all changes (fixes, features, security patches, etc.) are
treated as modifications. The compliance route, however, for complex software
and complex hardware is an active area of development (see SIL2LinuxMP above),
as these standards (IEC 61508) were not developed with such software or hardware
in mind.

> 
> Thus Hitachi developed this minimizing stripping out all not-compiled code
> and #ifdefs etc too to get down the code size they have to manually inspect
> to a minimum. (It easily translates into work hours.)
>
> My loose thoughts on the issue is twofold:
> 
> - We will have an influx of professional safety reviewers that do not
>   share their review comments with us, instead apply fixes locally and
>   not upstream. This is potentially dangerous if the next reviewer for
>   a safety-critical system misses the same bug. (Not to say unethical
>   vs the community but I have come to understand that some people
>   out there do not care about that.) So we need to send a message to
>   the safety-critical industry that any issues found in such safety
>   inspections need to go upstream pronto. No vendor tree:ing of this.

SIL2LinuxMP will help significantly with this, and the TUeV is happy to see open
source entering the Functional Safety space as the developer culture rewards
finding and fixing issues. Bugs can be found and fixed early, rather than
waiting for accidents and deaths to drive changes.

> 
> - Can we record external inspection-only code reviews done by these
>   independent code reviewers (post-minimization) into the kernel (etc) git?
>   That I guess is pretty useful for building formal trust for the code,
>   but I never heard of git annotations to some random code lines like
>   that.
> 
> - Should minimization be a part of the kernel standard tooling for use
>   cases like this?
> 
> Incidentally that may overlap with the footprint minimizing goal: if you can
> configure code out (such as the modular syscalls things that Nico has
> been working on), that makes this kind of code minimization easier and
> may employ similar tooling.

The SIL2LinuxMP project works with a minimized Linux kernel of ~400k loc.

> 
> Yours,
> Linus Walleij
> 

-- 
Darren Hart
Intel Open Source Technology Center

[Ksummit-discuss] Self nomination

[Mark Rutland]

Hi,

I would like to self nominate for the kernel summit this year.

I co-maintain device tree bindings, and the ARM PSCI driver. I also
contribute to the low-level architecture code for arm and arm64.

I'm interested in hardening efforts that are applicable either
generically or to arm/arm64, and have been involved with some of the
work so far.

Having been bitten by a few issues in the past, I'm also interested in
what we can do to ensure that changes to common infrastructure don't
leave architectures broken (or in a sub-optimal state). I'm interested
in unit testing, regression tracking, and stable workflow.

I'm interested in discussions regarding:

- device tree bindings / subsystem divisions
- kernel hardening / self protection
- kernel unit testing
- regression tracking
- stable workflow

I will also be attending Linux Plumbers Conference 2016.

Thanks,
Mark.

[Ksummit-discuss] Self nomination

[Lorenzo Pieralisi]

Hello,

I would like to self-nominate for the 2016 kernel summit.

I am currently co-maintaining the ARM PSCI firmware interface,
some ARM CPU idle drivers and ARM platforms code and I am a core
contributor of ARM power management, ACPI and PCI code.

I will attend the Linux Plumbers Conference 2016 as PCI
microconference co-leader and take part in the Power
Management microconference, among other topics.

For my maintainership role, I am quite interested in attending the
plenary KS sessions core tracks (that are currently under discussion
in this mailing list), in particular:

  - (group) maintainership models
  - Stable kernels process and workflow
  - kthread freezer clean-up

Since I am involved in power management and firmware interfaces I
am particularly interested in topics revolving around ACPI and
device tree integration in the corresponding workshops and breakout
sessions (other than the LPC microconferences already mentioned).

Thank you very much for considering.

Regards,
Lorenzo

Re: [Ksummit-discuss] Self nomination

[Linus Walleij]

On Wed, Jul 27, 2016 at 6:46 AM, Darren Hart <dvhart@infradead.org> wrote:

>   - Developing a "safety culture" and any overlap that may have with security

Do you mean safety as in "Linux in airbags and smoke detectors"?

This area is interesting for GPIO and IIO as well for natural reasons:
these systems all tend to use GPIO and sensors. (Albeit more often
than not with some horrific userspace hodge-podge but this is not the
time to be grumpy about that.)

This presentation appeared at LinuxCon Japan (got the link from
my colleague Takahiro Akashi):

Qualifying Linux for Functional Safety
http://events.linuxfoundation.org/sites/events/files/slides/20160713_SIL2LinuxMP_Min_ALS_0.9_pub.pdf

I have been in contact with a Swedish company working in the fire alarm
business as well.

My overall feeling is that world regulations (standards) on safety-critical
software seem to be centered around code inspection.

These regulations approach is not to trust any third party but have all
code inspected by independent reviewers for functional safety. All the
time. For every new deployment. Kernel, libc, busybox userspace.
(Or whatever they use.)

Thus Hitachi developed this minimizing stripping out all not-compiled code
and #ifdefs etc too to get down the code size they have to manually inspect
to a minimum. (It easily translates into work hours.)

My loose thoughts on the issue is twofold:

- We will have an influx of professional safety reviewers that do not
  share their review comments with us, instead apply fixes locally and
  not upstream. This is potentially dangerous if the next reviewer for
  a safety-critical system misses the same bug. (Not to say unethical
  vs the community but I have come to understand that some people
  out there do not care about that.) So we need to send a message to
  the safety-critical industry that any issues found in such safety
  inspections need to go upstream pronto. No vendor tree:ing of this.

- Can we record external inspection-only code reviews done by these
  independent code reviewers (post-minimization) into the kernel (etc) git?
  That I guess is pretty useful for building formal trust for the code,
  but I never heard of git annotations to some random code lines like
  that.

- Should minimization be a part of the kernel standard tooling for use
  cases like this?

Incidentally that may overlap with the footprint minimizing goal: if you can
configure code out (such as the modular syscalls things that Nico has
been working on), that makes this kind of code minimization easier and
may employ similar tooling.

Yours,
Linus Walleij

[Ksummit-discuss] Self nomination

[Darren Hart]

I'd like to self nominate for this kernel summit this year.

I'm the platform-drivers-x86 maintainer, and am interested in discussions
regarding:

  - maintainer group process and tooling
  - easing the non-kernel-code barriers to contribution
    (eliminating "mechanical failures")

I'm involved with various efforts around Real-Time and Safety Critical Linux and
would be interested in discussions around:

  - Upstreaming PREEMPT_RT
  - Developing a "safety culture" and any overlap that may have with security

Finally, as part of my platform enabling responsibilities, I'm interested in any
developments in the stable process.

Regards,

-- 
Darren Hart
Intel Open Source Technology Center

[Ksummit-discuss] Self nomination

[Sergey Senozhatsky]

Hello,

I'd like to self nominate for the kernel summit this year.

Interested in printk-related discussions (async printk, KERN_CONT printing,
async console_unlock(), panic printk(), etc.), stable workflow and some
other topics.

	-ss

[Ksummit-discuss] Self nomination

[Stephen Rothwell]

I'd like to self nominate for the kernel summit this year.

As the linux-next maintainer, I have interest in most process topics in
particular the maintainership processes.

Also, being a bit isolated down under, meeting in person allows me to
remember that you are all people really :-)

-- 
Cheers,
Stephen Rothwell

[Ksummit-discuss] Self nomination

[Dmitry Torokhov]

I'd like to nominate myself for the kernel summit this year. I am part
of Chrome OS kernel team and I also maintain drivers/input in mainline.

I am interested in the following topics/discussions:

- stable process and regression tracking;

- maintainership process. I am still struggling to figure out how I
  can spread the load and would love to hear some ideas;

- discussions regarding subsystem boundaries (iio, input, hwmon, etc)
  and how we can provide better guidance to driver authors to find a
  home for their drivers;

- I would like to sync up with people and discuss [lack of] progress
  on topic of device probe ordering (including handling of deferred
  probes, asynchronous probes, etc).

Thanks!

-- 
Dmitry

[Ksummit-discuss] Self nomination

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

I'd like to nominate myself for the kernel summit this year.

I am very interested in toolchain-related discussion, looking at what
the compiler(s) can do better and how we can make better use of them.
Including types and static analysis, and potentially making use of
LLVM.

And also the memory model — like the byteswapping that I've worked on
before, it makes a lot of sense to let the compiler *see* what's going
on rather than hiding it in opaque inline assembly which precludes
optimisations. If we have to make an effort to ensure the ordering
models are compatible, that may well be a price worth paying. It's not
like we didn't already have work to do on ensuring that the coherency
models we offer to (e.g.) device drivers through things like
readl_relaxed() are well-specified in a way that can be implemented
efficiently across architectures.

I'm interested in testing topics, and how we can make it easier for
contributors (and 'janitors') to build simple test cases which run in a
consistent environment, rather than doing their own ad-hoc testing.

I would very much like to be involved with any discussion on
asynchronous and early console output.

I'm also interested in the signature/key management topic if that
happens.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5760 bytes --]

Re: [Ksummit-discuss] Self nomination

[Vlastimil Babka]

On 07/26/2016 12:56 PM, Jan Kara wrote:
> On Mon 25-07-16 14:15:18, Rik van Riel wrote:
>> On Mon, 2016-07-25 at 13:11 -0400, Johannes Weiner wrote:
>>> Hi,
>>>
>>> I would like to self-nominate myself for this year's kernel summit.
>>>
>>> I co-maintain cgroups and the memory controller and have been a
>>> long-time contributor to the memory management subsystem. At
>>> Facebook,
>>> I'm in charge of MM scalability and reliability in our fleet. Most
>>> recently I have been working on reviving swap for SSDs and persistent
>>> memory devices (https://lwn.net/Articles/690079/) as part of a bigger
>>> anti-thrashing effort to make the VM recover swiftly and predictably
>>> from load spikes. This has been a bit of a "lock yourself in a
>>> basement" type project, which is why I missed the mechanimal
>>> nomination based on sign-offs this year.
>>
>> I am interested in discussing that, either at kernel
>> summit, or at next year's LSF/MM.
>
> Me as well.

+1

>
> 								Honza
>

Re: [Ksummit-discuss] Self nomination

[Jan Kara]

On Mon 25-07-16 14:15:18, Rik van Riel wrote:
> On Mon, 2016-07-25 at 13:11 -0400, Johannes Weiner wrote:
> > Hi,
> > 
> > I would like to self-nominate myself for this year's kernel summit.
> > 
> > I co-maintain cgroups and the memory controller and have been a
> > long-time contributor to the memory management subsystem. At
> > Facebook,
> > I'm in charge of MM scalability and reliability in our fleet. Most
> > recently I have been working on reviving swap for SSDs and persistent
> > memory devices (https://lwn.net/Articles/690079/) as part of a bigger
> > anti-thrashing effort to make the VM recover swiftly and predictably
> > from load spikes. This has been a bit of a "lock yourself in a
> > basement" type project, which is why I missed the mechanimal
> > nomination based on sign-offs this year.
> 
> I am interested in discussing that, either at kernel
> summit, or at next year's LSF/MM.

Me as well.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[Ksummit-discuss] self nomination

[Kevin Hilman]

Hello,

I would like to self-nominate for the 2016 kernel summit.

I'm the lead kernel developer working on kernelci.org,
backup/assistant/3rd-string/sick-leave co-maintainer of the arm-soc
tree, as well as co-maintainer of a couple of the many ARM
sub-architectures.

I'm most interested in discussions around kernel testing infrastructure,
and in particular, how to improve testing for the various stable and
LTS-based trees.

Thank you for considering,

Kevin

Re: [Ksummit-discuss] Self nomination

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On Mon, 2016-07-25 at 13:11 -0400, Johannes Weiner wrote:
> Hi,
> 
> I would like to self-nominate myself for this year's kernel summit.
> 
> I co-maintain cgroups and the memory controller and have been a
> long-time contributor to the memory management subsystem. At
> Facebook,
> I'm in charge of MM scalability and reliability in our fleet. Most
> recently I have been working on reviving swap for SSDs and persistent
> memory devices (https://lwn.net/Articles/690079/) as part of a bigger
> anti-thrashing effort to make the VM recover swiftly and predictably
> from load spikes. This has been a bit of a "lock yourself in a
> basement" type project, which is why I missed the mechanimal
> nomination based on sign-offs this year.

I am interested in discussing that, either at kernel
summit, or at next year's LSF/MM.

-- 

All Rights Reversed.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Ksummit-discuss] Self nomination

[Johannes Weiner]

Hi,

I would like to self-nominate myself for this year's kernel summit.

I co-maintain cgroups and the memory controller and have been a
long-time contributor to the memory management subsystem. At Facebook,
I'm in charge of MM scalability and reliability in our fleet. Most
recently I have been working on reviving swap for SSDs and persistent
memory devices (https://lwn.net/Articles/690079/) as part of a bigger
anti-thrashing effort to make the VM recover swiftly and predictably
from load spikes. This has been a bit of a "lock yourself in a
basement" type project, which is why I missed the mechanimal
nomination based on sign-offs this year.

Thanks
Johannes

[Ksummit-discuss] Self nomination

[Haggai Eran]

Hi,

Excuse me for the late email, but I would like to nominate myself for
the core discussion. I'm interested in participating in the
compute-offload devices discussions, as well as the FPGA programming topic.

I've been part of the team that developed the RDMA subsystem support for
IO paging, and the mlx5 driver implementation.

Regards,
Haggai

Re: [Ksummit-discuss] Self nomination

[Sasha Levin]

On 08/01/2015 04:30 PM, Dave Jones wrote:
> On Sat, Aug 01, 2015 at 11:26:17AM -0400, Sasha Levin wrote:
>  > On 08/01/2015 09:45 AM, Dan Carpenter wrote:
>  > > On Fri, Jul 31, 2015 at 01:49:08PM -0400, Sasha Levin wrote:
>  > >>  - Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
>  > >> testing code to the various testing projects around (trinity and such).
>  > > 
>  > > Which other fuzzers are people using besides trinity?
>  > 
>  > I'm not sure, I think trinity sort of took over :)
>  > 
>  > My point was that people should be submitting a more significant amount
>  > of test code along with their new ABI - none of the recent syscalls added
>  > has any mention in LTP for example.
>  > 
>  > > Also why don't we merge trinity under tools/testing/?  I bet people
>  > > would keep it in sync better if we did that.
>  > 
>  > No objections on my end. I believe that Dave previously objected because
>  > it's hard to make it work on sync with the kernel's release cycle.
> 
> A big factor is the same reason I'm not a huge fan of tools/testing/ in general.
> People want to do things like "run latest testing tools on old LTS/enterprise kernels".
> 
> Having to suck a subtree out of the latest kernel tarball, and then spend
> time trying to make it even compile/run against headers from an old kernel
> is something I've already lost way too many weeks of my life to, so I'm
> not really a fan of making that problem even worse by adding other tools to it.

An interesting model we can try here is to let development happen both in the kernel
tree and outside of it, and patches will be exchanged between the two development trees
(possibly automatically). Think of it as a "fork".

On one hand it'll provide the kernel with a proven testing framework, and on
the other it'll give trinity a wider user/developer base.


Thanks,
Sasha

Re: [Ksummit-discuss] Self nomination

[Sasha Levin]

On 08/01/2015 12:22 PM, Greg KH wrote:
>> > My point was that people should be submitting a more significant amount
>> > of test code along with their new ABI - none of the recent syscalls added
>> > has any mention in LTP for example.
> Because most of us hate LTP.
> 
> Seriously, I've lost days of my life many years ago tracing down "kernel
> bugs" that in the end were just bugs in LTP due to the horrid code in
> it.
> 
> Now yes, this was a long time ago, and hopefully it has gotten a lot
> better, but lots of people have this attitude which is why we are
> encouraging people to use the tests in the kernel tree itself, and is
> why new syscalls are all adding their tests there.
> 
> I've encouraged people to take the "good bits" of LTP and add them to
> the kernel test framework, but no one seems to want to do that...

I didn't really mean to focus on LTP here: pick any project that has
a critical mass of users and send them ways to test your code now and
for future changes.


Thanks,
Sasha

Re: [Ksummit-discuss] Self nomination

[Dave Jones]

On Sat, Aug 01, 2015 at 11:26:17AM -0400, Sasha Levin wrote:
 > On 08/01/2015 09:45 AM, Dan Carpenter wrote:
 > > On Fri, Jul 31, 2015 at 01:49:08PM -0400, Sasha Levin wrote:
 > >>  - Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
 > >> testing code to the various testing projects around (trinity and such).
 > > 
 > > Which other fuzzers are people using besides trinity?
 > 
 > I'm not sure, I think trinity sort of took over :)
 > 
 > My point was that people should be submitting a more significant amount
 > of test code along with their new ABI - none of the recent syscalls added
 > has any mention in LTP for example.
 > 
 > > Also why don't we merge trinity under tools/testing/?  I bet people
 > > would keep it in sync better if we did that.
 > 
 > No objections on my end. I believe that Dave previously objected because
 > it's hard to make it work on sync with the kernel's release cycle.

A big factor is the same reason I'm not a huge fan of tools/testing/ in general.
People want to do things like "run latest testing tools on old LTS/enterprise kernels".

Having to suck a subtree out of the latest kernel tarball, and then spend
time trying to make it even compile/run against headers from an old kernel
is something I've already lost way too many weeks of my life to, so I'm
not really a fan of making that problem even worse by adding other tools to it.

	Dave

Re: [Ksummit-discuss] Self nomination

[Greg KH]

On Sat, Aug 01, 2015 at 11:26:17AM -0400, Sasha Levin wrote:
> On 08/01/2015 09:45 AM, Dan Carpenter wrote:
> > On Fri, Jul 31, 2015 at 01:49:08PM -0400, Sasha Levin wrote:
> >>  - Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
> >> testing code to the various testing projects around (trinity and such).
> > 
> > Which other fuzzers are people using besides trinity?
> 
> I'm not sure, I think trinity sort of took over :)
> 
> My point was that people should be submitting a more significant amount
> of test code along with their new ABI - none of the recent syscalls added
> has any mention in LTP for example.

Because most of us hate LTP.

Seriously, I've lost days of my life many years ago tracing down "kernel
bugs" that in the end were just bugs in LTP due to the horrid code in
it.

Now yes, this was a long time ago, and hopefully it has gotten a lot
better, but lots of people have this attitude which is why we are
encouraging people to use the tests in the kernel tree itself, and is
why new syscalls are all adding their tests there.

I've encouraged people to take the "good bits" of LTP and add them to
the kernel test framework, but no one seems to want to do that...

thanks,

greg k-h

Re: [Ksummit-discuss] Self nomination

[Sasha Levin]

On 08/01/2015 09:45 AM, Dan Carpenter wrote:
> On Fri, Jul 31, 2015 at 01:49:08PM -0400, Sasha Levin wrote:
>>  - Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
>> testing code to the various testing projects around (trinity and such).
> 
> Which other fuzzers are people using besides trinity?

I'm not sure, I think trinity sort of took over :)

My point was that people should be submitting a more significant amount
of test code along with their new ABI - none of the recent syscalls added
has any mention in LTP for example.

> Also why don't we merge trinity under tools/testing/?  I bet people
> would keep it in sync better if we did that.

No objections on my end. I believe that Dave previously objected because
it's hard to make it work on sync with the kernel's release cycle.

I've actually presented a proposal in linuxcon back in 2013 for unifying
trinity into the kernel, and using common headers for syscalls. This would
force people who add/modify syscalls to update trinity, and it'll also make
the ABI self documenting as one would need to be very specific about what
parameters are expected for each syscall.


Thanks,
Sasha

Re: [Ksummit-discuss] Self nomination

[Dan Carpenter]

On Fri, Jul 31, 2015 at 01:49:08PM -0400, Sasha Levin wrote:
>  - Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
> testing code to the various testing projects around (trinity and such).

Which other fuzzers are people using besides trinity?

Also why don't we merge trinity under tools/testing/?  I bet people
would keep it in sync better if we did that.

regards,
dan carpenter

Re: [Ksummit-discuss] Self nomination

[Jiri Kosina]

On Fri, 31 Jul 2015, Dan Williams wrote:

> One more thought on this tangent... I've had a task stuck at the bottom 
> of my backlog to look at extending dynamic_debug call sites to 
> optionally be tracepoints so that you could feasibly have debug on all 
> the time with less run time impact. 

Making dynamic debug use jump labels should be enough to achieve this.

That used to actually be the case, but got reverted later, as it didn't 
properly handle gccs that didn't support asm goto (see 2d75af2f2a7).

Once that is fixed to be more general and handle older gccs as well, your 
task is done :)

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] Self nomination

[Dan Williams]

On Fri, Jul 31, 2015 at 11:22 AM, Greg KH <greg@kroah.com> wrote:
> On Fri, Jul 31, 2015 at 02:17:58PM -0400, Dave Jones wrote:
>> On Fri, Jul 31, 2015 at 11:12:45AM -0700, Greg Kroah-Hartman wrote:
>>  > On Fri, Jul 31, 2015 at 12:51:53PM -0500, Bjorn Helgaas wrote:
>>  > > I prefer that a dmesg collected in the simplest possible way, with no
>>  > > special config or boot flags, be as useful as possible.  So converting
>>  > > to dynamic debug requires much more thought about which messages
>>  > > should be always printed and which should become dynamic.
>>  >
>>  > Why would a debugging message ever not be dynamic?  They are there for
>>  > you to use, and turn off when you are done.  If you want a user to
>>  > report the output of them, then of course they should be dynamic so they
>>  > can just write a line to a debugfs file and then start seeing them
>>
>> This implies a user knows ahead of time what bugs they are going to hit,
>> and which messages they need to enable.  For hard-to-reproduce bugs,
>> or bugs that exhibit non-obvious symptoms, this isn't workable.
>
> Fair enough, but wouldn't those messages be "errors"?
>
> Anyway, this is way off-topic from the original thread, it all comes
> down to specifics of the message that is being written and the
> surrounding issues of why it would be written.

One more thought on this tangent... I've had a task stuck at the
bottom of my backlog to look at extending dynamic_debug call sites to
optionally be tracepoints so that you could feasibly have debug on all
the time with less run time impact.

Re: [Ksummit-discuss] Self nomination

[Greg KH]

On Fri, Jul 31, 2015 at 02:17:58PM -0400, Dave Jones wrote:
> On Fri, Jul 31, 2015 at 11:12:45AM -0700, Greg Kroah-Hartman wrote:
>  > On Fri, Jul 31, 2015 at 12:51:53PM -0500, Bjorn Helgaas wrote:
>  > > I prefer that a dmesg collected in the simplest possible way, with no
>  > > special config or boot flags, be as useful as possible.  So converting
>  > > to dynamic debug requires much more thought about which messages
>  > > should be always printed and which should become dynamic.
>  > 
>  > Why would a debugging message ever not be dynamic?  They are there for
>  > you to use, and turn off when you are done.  If you want a user to
>  > report the output of them, then of course they should be dynamic so they
>  > can just write a line to a debugfs file and then start seeing them
> 
> This implies a user knows ahead of time what bugs they are going to hit,
> and which messages they need to enable.  For hard-to-reproduce bugs,
> or bugs that exhibit non-obvious symptoms, this isn't workable.

Fair enough, but wouldn't those messages be "errors"?

Anyway, this is way off-topic from the original thread, it all comes
down to specifics of the message that is being written and the
surrounding issues of why it would be written.

thanks,

greg k-h

Re: [Ksummit-discuss] Self nomination

[Dave Jones]

On Fri, Jul 31, 2015 at 11:12:45AM -0700, Greg Kroah-Hartman wrote:
 > On Fri, Jul 31, 2015 at 12:51:53PM -0500, Bjorn Helgaas wrote:
 > > I prefer that a dmesg collected in the simplest possible way, with no
 > > special config or boot flags, be as useful as possible.  So converting
 > > to dynamic debug requires much more thought about which messages
 > > should be always printed and which should become dynamic.
 > 
 > Why would a debugging message ever not be dynamic?  They are there for
 > you to use, and turn off when you are done.  If you want a user to
 > report the output of them, then of course they should be dynamic so they
 > can just write a line to a debugfs file and then start seeing them

This implies a user knows ahead of time what bugs they are going to hit,
and which messages they need to enable.  For hard-to-reproduce bugs,
or bugs that exhibit non-obvious symptoms, this isn't workable.

	Dave

Re: [Ksummit-discuss] Self nomination

[Greg KH]

On Fri, Jul 31, 2015 at 12:51:53PM -0500, Bjorn Helgaas wrote:
> I prefer that a dmesg collected in the simplest possible way, with no
> special config or boot flags, be as useful as possible.  So converting
> to dynamic debug requires much more thought about which messages
> should be always printed and which should become dynamic.

Why would a debugging message ever not be dynamic?  They are there for
you to use, and turn off when you are done.  If you want a user to
report the output of them, then of course they should be dynamic so they
can just write a line to a debugfs file and then start seeing them, they
shouldn't be seeing them all the time, and they should never require a
rebuild of the kernel to see them (which is why we deleted
CONFIG_USB_DEBUG).

thanks,

greg k-h

Re: [Ksummit-discuss] Self nomination

[Bjorn Helgaas]

On Fri, Jul 31, 2015 at 12:15 PM, Guenter Roeck <linux@roeck-us.net> wrote:
> On Fri, Jul 31, 2015 at 10:08:25AM -0700, Greg KH wrote:
>> On Fri, Jul 31, 2015 at 09:59:15AM -0700, Guenter Roeck wrote:
>> > Hi Bjorn,
>> >
>> > On Fri, Jul 31, 2015 at 11:27:38AM -0500, Bjorn Helgaas wrote:
>> > > On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
>> > > > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
>> > > > and by improving the quality of debug output that comes out of the kernel.
>> > >
>> > > What sort of debug output improvements are you interested in?  I spend
>> > > a fair amount of time converting to dev_printk and %pR.  They make the
>> >
>> > I have been wondering about that, especially since dev_dbg() and
>> > 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason
>> > for preferring dev_printk() over dev_dbg() ?
>>
>> The opposite is true, please always use dev_dbg() as it properly ties
>
> Hi Greg,
>
> maybe my semantics wasn't perfect - my question was why Bjorn prefers
> dev_printk(KERN_DEBUG, ..) over dev_dbg(), which he answered.
>
> I did not (want to) make the claim or even suggest that dev_printk()
> would be preferred over dev_dbg() in general.

Lest I give the wrong impression, I'm not opposed to dev_dbg().  I
just think there are two separate changes that don't need to be made
at the same time: (1) convert to dev_printk style, and (2) convert to
the dynamic debug stuff.  As a consumer of dmesg logs, I'm most
interested in the former.

I prefer that a dmesg collected in the simplest possible way, with no
special config or boot flags, be as useful as possible.  So converting
to dynamic debug requires much more thought about which messages
should be always printed and which should become dynamic.

Re: [Ksummit-discuss] Self nomination

[Sasha Levin]

On 07/31/2015 01:26 PM, James Bottomley wrote:
> Other than to confuse us all over minute details, is there a reason for
> this difference to exist?  I realise right at the moment dev_dbg can be
> configured off by a variety of symbols and dev_printk(KERN_DEBUG) can't,
> conversely dev_printk doesn't pick up the location information and
> dev_dbg does, but there's no reason all that couldn't be harmonised so
> we didn't have to have arguments about which to use (and endless patches
> switching from one to the other)?

I suppose it's a good sub-topic: deciding which one of the 3.5 methods
to print debug information we should be using across the kernel and
remove the other 2.5.


Thanks,
Sasha

Re: [Ksummit-discuss] Self nomination

[Sasha Levin]

On 07/31/2015 12:51 AM, Julia Lawall wrote:
>> I'd like to nominate myself to this year's kernel summit.
>> > 
>> > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
>> > and by improving the quality of debug output that comes out of the kernel.
> Improving it in what way?  We are doing some research related to logging 
> at the moment, and so I would be interested to hear what you think are the 
> problems.

>From my perspective the biggest issue is making a given error state more reproducible. Quite
often I am able to trigger a bug on my testing box, but the information the kernel spits out
(backtrace + registers) aren't too useful if the problem is complicated.

The result is that finding out what happens depends on few people who are able to "magically"
trigger bugs and result in a long and inefficient back and forth while the bug manages to
sneak in upstream and to users hands.

To sum it up, I think we need to figure out a way to produce enough information to make bugs
more reproducible, but not too much to avoid a needle in a haystack situation.

In this regard, the new Intel PT technology is interesting and it's also worth figuring out
a good way to integrate it with our current infrastructure/tools.


Among other, less urgent ideas I'd like to discuss are:

 - Using KASan for poison fields. Quite a few structures have embedded poison fields. Rather
than detecting a corruption after it happens we are able to detect it as it happens - making
bugs more obvious.

 - Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
testing code to the various testing projects around (trinity and such).

 - Improving userspace tooling to make transfer of information simpler (along the lines of
scripts/decode_stacktrace.sh that no one seems to be using).

I suppose it'll be interesting to discuss the idea of making the kernel dump a "black box"
on panic. Not just the core memory, but also various parameters regarding configuration and
such, to enable folks to provide us with more than just backtraces.


Thanks,
Sasha

Re: [Ksummit-discuss] Self nomination

[Greg KH]

On Fri, Jul 31, 2015 at 10:26:41AM -0700, James Bottomley wrote:
> On Fri, 2015-07-31 at 10:08 -0700, Greg KH wrote:
> > On Fri, Jul 31, 2015 at 09:59:15AM -0700, Guenter Roeck wrote:
> > > Hi Bjorn,
> > > 
> > > On Fri, Jul 31, 2015 at 11:27:38AM -0500, Bjorn Helgaas wrote:
> > > > On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> > > > > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
> > > > > and by improving the quality of debug output that comes out of the kernel.
> > > > 
> > > > What sort of debug output improvements are you interested in?  I spend
> > > > a fair amount of time converting to dev_printk and %pR.  They make the
> > > 
> > > I have been wondering about that, especially since dev_dbg() and
> > > 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason
> > > for preferring dev_printk() over dev_dbg() ?
> > 
> > The opposite is true, please always use dev_dbg() as it properly ties
> > into the overall kernel-wide dynamic debug infrastructure, providing a
> > unified way to enable/disable debug messages, or even, compiling them
> > out if none are wanted due to size constraints.
> 
> Other than to confuse us all over minute details, is there a reason for
> this difference to exist?

dev_dbg() uses dev_printk(KERN_DEBUG) to build on, we can't delete
dev_printk() from the tree, so we are stuck with both.  If you can
figure out a way to prevent this, that would be great.

thanks,

greg k-h

Re: [Ksummit-discuss] Self nomination

[James Bottomley]

On Fri, 2015-07-31 at 10:08 -0700, Greg KH wrote:
> On Fri, Jul 31, 2015 at 09:59:15AM -0700, Guenter Roeck wrote:
> > Hi Bjorn,
> > 
> > On Fri, Jul 31, 2015 at 11:27:38AM -0500, Bjorn Helgaas wrote:
> > > On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> > > > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
> > > > and by improving the quality of debug output that comes out of the kernel.
> > > 
> > > What sort of debug output improvements are you interested in?  I spend
> > > a fair amount of time converting to dev_printk and %pR.  They make the
> > 
> > I have been wondering about that, especially since dev_dbg() and
> > 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason
> > for preferring dev_printk() over dev_dbg() ?
> 
> The opposite is true, please always use dev_dbg() as it properly ties
> into the overall kernel-wide dynamic debug infrastructure, providing a
> unified way to enable/disable debug messages, or even, compiling them
> out if none are wanted due to size constraints.

Other than to confuse us all over minute details, is there a reason for
this difference to exist?  I realise right at the moment dev_dbg can be
configured off by a variety of symbols and dev_printk(KERN_DEBUG) can't,
conversely dev_printk doesn't pick up the location information and
dev_dbg does, but there's no reason all that couldn't be harmonised so
we didn't have to have arguments about which to use (and endless patches
switching from one to the other)?

James

Re: [Ksummit-discuss] Self nomination

[Guenter Roeck]

On Fri, Jul 31, 2015 at 10:08:25AM -0700, Greg KH wrote:
> On Fri, Jul 31, 2015 at 09:59:15AM -0700, Guenter Roeck wrote:
> > Hi Bjorn,
> > 
> > On Fri, Jul 31, 2015 at 11:27:38AM -0500, Bjorn Helgaas wrote:
> > > On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> > > > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
> > > > and by improving the quality of debug output that comes out of the kernel.
> > > 
> > > What sort of debug output improvements are you interested in?  I spend
> > > a fair amount of time converting to dev_printk and %pR.  They make the
> > 
> > I have been wondering about that, especially since dev_dbg() and
> > 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason
> > for preferring dev_printk() over dev_dbg() ?
> 
> The opposite is true, please always use dev_dbg() as it properly ties

Hi Greg,

maybe my semantics wasn't perfect - my question was why Bjorn prefers
dev_printk(KERN_DEBUG, ..) over dev_dbg(), which he answered.

I did not (want to) make the claim or even suggest that dev_printk()
would be preferred over dev_dbg() in general.

Thanks,
Guenter

Re: [Ksummit-discuss] Self nomination

[Greg KH]

On Fri, Jul 31, 2015 at 09:59:15AM -0700, Guenter Roeck wrote:
> Hi Bjorn,
> 
> On Fri, Jul 31, 2015 at 11:27:38AM -0500, Bjorn Helgaas wrote:
> > On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> > > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
> > > and by improving the quality of debug output that comes out of the kernel.
> > 
> > What sort of debug output improvements are you interested in?  I spend
> > a fair amount of time converting to dev_printk and %pR.  They make the
> 
> I have been wondering about that, especially since dev_dbg() and
> 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason
> for preferring dev_printk() over dev_dbg() ?

The opposite is true, please always use dev_dbg() as it properly ties
into the overall kernel-wide dynamic debug infrastructure, providing a
unified way to enable/disable debug messages, or even, compiling them
out if none are wanted due to size constraints.

thanks,

greg k-h

Re: [Ksummit-discuss] Self nomination

[Bjorn Helgaas]

On Fri, Jul 31, 2015 at 11:59 AM, Guenter Roeck <linux@roeck-us.net> wrote:
> Hi Bjorn,
>
> On Fri, Jul 31, 2015 at 11:27:38AM -0500, Bjorn Helgaas wrote:
>> On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
>> > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
>> > and by improving the quality of debug output that comes out of the kernel.
>>
>> What sort of debug output improvements are you interested in?  I spend
>> a fair amount of time converting to dev_printk and %pR.  They make the
>
> I have been wondering about that, especially since dev_dbg() and
> 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason
> for preferring dev_printk() over dev_dbg() ?

Personally I think that difference is a hassle.  When I convert
printk(KERN_DEBUG) to use dev_printk, I use dev_printk(KERN_DEBUG) to
preserve the fact that the printk happens unconditionally.  If I were
smarter, I would figure out how to use dev_dbg() effectively, but
usually I want my patch to be "convert to dev_printk", not "convert to
dev_dbg and only print when some magic incantation is given".

Re: [Ksummit-discuss] Self nomination

[Guenter Roeck]

Hi Bjorn,

On Fri, Jul 31, 2015 at 11:27:38AM -0500, Bjorn Helgaas wrote:
> On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> > Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
> > and by improving the quality of debug output that comes out of the kernel.
> 
> What sort of debug output improvements are you interested in?  I spend
> a fair amount of time converting to dev_printk and %pR.  They make the

I have been wondering about that, especially since dev_dbg() and
'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason
for preferring dev_printk() over dev_dbg() ?

Thank,
Guenter

Re: [Ksummit-discuss] Self nomination

[Bjorn Helgaas]

On Thu, Jul 30, 2015 at 9:55 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
> and by improving the quality of debug output that comes out of the kernel.

What sort of debug output improvements are you interested in?  I spend
a fair amount of time converting to dev_printk and %pR.  They make the
kernel more consistent and approachable, but they're minor and I'm
sure you have more substantive things in mind.

[Ksummit-discuss] Self nomination

[David Howells]

I would like to nominate myself for this years KS.  I'm interested in:

 (*) Compiler features and cross-compilers having been maintaining a set of
     cross compilers for kernel.  I've been watching gcc-5 come out and find
     new things to warning about in the kernel.

 (*) Filesystem unioning - overlayfs, unionmount and the way LSMs interact and
     the way unioning metadata is stored in the fs.

 (*) Signature management - keys, modules, firmware.

 (*) Media drivers - I've been working to improve a couple of these.

David

[Ksummit-discuss] Self nomination

[Sasha Levin]

Hi folks,

I'd like to nominate myself to this year's kernel summit.

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs
and by improving the quality of debug output that comes out of the kernel.

As the maintainer of the 3.18 LTS line I'd also like to discuss various -stable kernel
topics that came up previously on the mailing list.


Thanks,
Sasha