[Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Jason Cooper]

All,

This is a topic of interest to me that I think would best benefit from a
conference room discussion.

Items to discuss:

 - Survey the room on workflows and security posture for kernel work
 - Discussion of threat models, attack vectors
 - Discuss mitigation methods, tools and techniques
 - Identify missing tools or features of tools

The intent is to discuss end point security with regards to protecting
the kernel source tree.

This would *not* be about changing anyones workflow or DE or $editor or
other religious items. ;-)  It would be more about increasing awareness.
Both of attack vectors and tools to mitigate risk which would fit into
current workflows.

In order to encourage open and honest discussion ("I can only afford one
box.  My kid does unrestricted web browsing on it every day when I'm at
work"-type stuff) we could consider doing Chatham House Rule [0] for
this discussion.

thx,

Jason.

[0] https://en.wikipedia.org/wiki/Chatham_House_Rule

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Josh Boyer]

On Fri, Jul 10, 2015 at 10:38 AM, Jason Cooper <jason@lakedaemon.net> wrote:
> All,
>
> This is a topic of interest to me that I think would best benefit from a
> conference room discussion.
>
> Items to discuss:
>
>  - Survey the room on workflows and security posture for kernel work
>  - Discussion of threat models, attack vectors
>  - Discuss mitigation methods, tools and techniques
>  - Identify missing tools or features of tools
>
> The intent is to discuss end point security with regards to protecting
> the kernel source tree.

Interesting.  Though I think it needs a broader audience to be honest.
It would be far easier to use distros as an attack vector than to try
subverting the upstream source code.  This might be a good topic for
something like Linux Plumbers.

josh

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Theodore Ts'o]

On Fri, Jul 10, 2015 at 11:50:30AM -0400, Josh Boyer wrote:
> > This is a topic of interest to me that I think would best benefit from a
> > conference room discussion.
> >
> > Items to discuss:
> >
> >  - Survey the room on workflows and security posture for kernel work
> >  - Discussion of threat models, attack vectors
> >  - Discuss mitigation methods, tools and techniques
> >  - Identify missing tools or features of tools
> >
> > The intent is to discuss end point security with regards to protecting
> > the kernel source tree.
> 
> Interesting.  Though I think it needs a broader audience to be honest.
> It would be far easier to use distros as an attack vector than to try
> subverting the upstream source code.  This might be a good topic for
> something like Linux Plumbers.

I agree that the issue of trusting distros and the possibility of
hiding malware in distro software is an important one, and one which
is best addressed at something like Plumbers.

However, there are a number of Kernel-specific security issues which
are just as important --- how we maintain our own personal security,
and how we make sure a backdoor doesn't get slipped into the kernel
source tree.

This ends up relating to the code reviewer problem, and I've always
been convinced that if I had a a few dozen students to coach in a
computer security class, tasked as part of a semester-long class
project to attempt to sneak an obfuscated security bug as part of some
code cleanup or refactoring patch, at least one of them would end up
getting into Linus's tree.  The only reason why I haven't done this is
no one would have forgiven me afterwards, and the resulting bad
publicity wouldn't be good for Linux.

I wonder if this might be better done as a panel session during the
wider technical session day?

						- Ted

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Steven Rostedt]

On Fri, 10 Jul 2015 12:23:28 -0400
Theodore Ts'o <tytso@mit.edu> wrote:

> I wonder if this might be better done as a panel session during the
> wider technical session day?

Or both. Have this brought up as a panel session as well as a topic for
the core day. The panel session (which would come first), could be
about what types of attacks there could be, and concerns that people
have, and other general ideas about the topic.

The core day can be about what to do with all the info we got from the
panel session.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Olof Johansson]

On Fri, Jul 10, 2015 at 9:45 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 10 Jul 2015 12:23:28 -0400
> Theodore Ts'o <tytso@mit.edu> wrote:
>
>> I wonder if this might be better done as a panel session during the
>> wider technical session day?
>
> Or both. Have this brought up as a panel session as well as a topic for
> the core day. The panel session (which would come first), could be
> about what types of attacks there could be, and concerns that people
> have, and other general ideas about the topic.
>
> The core day can be about what to do with all the info we got from the
> panel session.

Agreed. I suspect nobody will have anything else than stringent best
practices advice to give in an open forum, while hopefully in a closed
one we might learn a bit about what convenience-vs-security trade-offs
people have done in reality, if any.

Ideal outcome to me from a closed session would be learning how to get
more convenience without sacrificing security, which can probably be
presented widely (open session and/or LWN article, etc). To get there
we might need to hear a bit about what level of convenience people
want.


-Olof

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Kees Cook]

On Fri, Jul 10, 2015 at 9:23 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Fri, Jul 10, 2015 at 11:50:30AM -0400, Josh Boyer wrote:
>> > This is a topic of interest to me that I think would best benefit from a
>> > conference room discussion.
>> >
>> > Items to discuss:
>> >
>> >  - Survey the room on workflows and security posture for kernel work
>> >  - Discussion of threat models, attack vectors
>> >  - Discuss mitigation methods, tools and techniques
>> >  - Identify missing tools or features of tools
>> >
>> > The intent is to discuss end point security with regards to protecting
>> > the kernel source tree.
>>
>> Interesting.  Though I think it needs a broader audience to be honest.
>> It would be far easier to use distros as an attack vector than to try
>> subverting the upstream source code.  This might be a good topic for
>> something like Linux Plumbers.
>
> I agree that the issue of trusting distros and the possibility of
> hiding malware in distro software is an important one, and one which
> is best addressed at something like Plumbers.
>
> However, there are a number of Kernel-specific security issues which
> are just as important --- how we maintain our own personal security,
> and how we make sure a backdoor doesn't get slipped into the kernel
> source tree.

I'm curious about the larger set of security topics. Jason's topic
seems to surround "supply chain security", but I think we've got a lot
of other areas. Is focusing on just that topic the right thing to do?

- supply chain security: I see two parts:
  - intentionally malicious commits (as Ted describes below)
  - personal security (keep commit credentials secure from theft)
- reactive security: bug fix workflow
  - getting fixes _to end users_ (not the same as publishing to stable)
  - documenting impact when known (avoiding intentional obfuscation)
- proactive security: stop security flaws from happening in the first place
  - scope analysis (defending both userspace and kernel from attack)
  - threat analysis (how are attacks being made now and in future?)
  - exposure analysis (syscall interface, device firmware, etc)
  - static checkers (find and eliminate bug classes in the code)
  - run-time mitigation features (endless list: memory protection, CFI,
    ASLR, anti-bruteforcing, etc)

My question would be: where do we need discussion? "Supply chain
security" was discussed a fair bit after the kernel.org incident.
"Reactive security" has seen endless discussion, and there are a
number of stale-mate topics. "Proactive security" is what I'm most
interested in, but tends to lack enough engineering resources to
really make strong headway. There are a number of dedicated folks, but
we need more people on that topic, not really more discussion.

I'm not saying we shouldn't have a discussion: I'm more curious to
hear where people think we should focus for a single session's time.

> This ends up relating to the code reviewer problem, and I've always
> been convinced that if I had a a few dozen students to coach in a
> computer security class, tasked as part of a semester-long class
> project to attempt to sneak an obfuscated security bug as part of some
> code cleanup or refactoring patch, at least one of them would end up
> getting into Linus's tree.  The only reason why I haven't done this is
> no one would have forgiven me afterwards, and the resulting bad
> publicity wouldn't be good for Linux.

My paranoid brain thinks this has likely already happened. :)

> I wonder if this might be better done as a panel session during the
> wider technical session day?

As mentioned in other replies, I think this would be best served as a
panel session to keep the audience tighter. It's a large field of
discussion and we might have problems staying on track even in the
smaller group.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Jason Cooper]

On Fri, Jul 10, 2015 at 10:34:16PM +0200, Olof Johansson wrote:
> On Fri, Jul 10, 2015 at 9:45 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> > On Fri, 10 Jul 2015 12:23:28 -0400
> > Theodore Ts'o <tytso@mit.edu> wrote:
> >
> >> I wonder if this might be better done as a panel session during the
> >> wider technical session day?
> >
> > Or both. Have this brought up as a panel session as well as a topic for
> > the core day. The panel session (which would come first), could be
> > about what types of attacks there could be, and concerns that people
> > have, and other general ideas about the topic.
> >
> > The core day can be about what to do with all the info we got from the
> > panel session.
> 
> Agreed. I suspect nobody will have anything else than stringent best
> practices advice to give in an open forum, while hopefully in a closed
> one we might learn a bit about what convenience-vs-security trade-offs
> people have done in reality, if any.

This is what I was driving at.  Thanks for the clarification.  I was
hoping the idea of Chatham House Rule would encourage the desired
honesty.  Basically, everyone agrees not to tie anything said in the
closed session to the person/org that said it.  Other than that,
everything is publishable, etc.

> Ideal outcome to me from a closed session would be learning how to get
> more convenience without sacrificing security, which can probably be
> presented widely (open session and/or LWN article, etc). To get there
> we might need to hear a bit about what level of convenience people
> want.

I think it makes more sense to have the larger, open session after the
closed session.  We can first collect and distill the honest trade-offs
from the closed session.  Then boil it down into into a set of
recommendations or a report.  This would help keep the open session on a
more formal, presentation-style track.

thx,

Jason.

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Jason Cooper]

On Fri, Jul 10, 2015 at 03:08:57PM -0700, Kees Cook wrote:
> On Fri, Jul 10, 2015 at 9:23 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> > On Fri, Jul 10, 2015 at 11:50:30AM -0400, Josh Boyer wrote:
> >> > This is a topic of interest to me that I think would best benefit from a
> >> > conference room discussion.
> >> >
> >> > Items to discuss:
> >> >
> >> >  - Survey the room on workflows and security posture for kernel work
> >> >  - Discussion of threat models, attack vectors
> >> >  - Discuss mitigation methods, tools and techniques
> >> >  - Identify missing tools or features of tools
> >> >
> >> > The intent is to discuss end point security with regards to protecting
> >> > the kernel source tree.
> >>
> >> Interesting.  Though I think it needs a broader audience to be honest.
> >> It would be far easier to use distros as an attack vector than to try
> >> subverting the upstream source code.  This might be a good topic for
> >> something like Linux Plumbers.
> >
> > I agree that the issue of trusting distros and the possibility of
> > hiding malware in distro software is an important one, and one which
> > is best addressed at something like Plumbers.
> >
> > However, there are a number of Kernel-specific security issues which
> > are just as important --- how we maintain our own personal security,
> > and how we make sure a backdoor doesn't get slipped into the kernel
> > source tree.
> 
> I'm curious about the larger set of security topics.

I am as well, but most of those can be discussed openly.  I thought the
KS would be a good opportunity to sit down and compare notes about
things we may not be comfortable discussing in public.

> Jason's topic seems to surround "supply chain security", but I think
> we've got a lot of other areas. Is focusing on just that topic the
> right thing to do?

I'm open to these for sure, but it's time dependent.  This could easily
turn into the KSS, Kernel Security Summit.  *I* think that would fun,
but a lot of others might not.

> - supply chain security: I see two parts:
>   - intentionally malicious commits (as Ted describes below)
>   - personal security (keep commit credentials secure from theft)

> - reactive security: bug fix workflow
>   - getting fixes _to end users_ (not the same as publishing to stable)
>   - documenting impact when known (avoiding intentional obfuscation)

This one is a can of worms that no one has solved yet.  It's the
conflict between the 'open' in "open source" and the reality of patch
propagation.  Or complete lack there of when users don't update.  I
doubt we could add anything new to the discussion, let alone solve the
problem at KS.

> - proactive security: stop security flaws from happening in the first place
>   - scope analysis (defending both userspace and kernel from attack)
>   - threat analysis (how are attacks being made now and in future?)
>   - exposure analysis (syscall interface, device firmware, etc)
>   - static checkers (find and eliminate bug classes in the code)
>   - run-time mitigation features (endless list: memory protection, CFI,
>     ASLR, anti-bruteforcing, etc)

These are fun, but in the traditional security vein.  Not to say they
aren't worthy of discussion, but that they can and are discussed in
public.  If we choose to add this to the agenda, perhaps a summary of
the state of the art from the parties involved would suffice?

> My question would be: where do we need discussion? "Supply chain
> security" was discussed a fair bit after the kernel.org incident.

That was pre-Snowden.  Also prior to a bunch of game-changing hacks and
vulnerability disclosures.  I think it's safe to say everyone's security
awareness and imagined art-of-the-possible has opened up in the past
couple of years.  Particularly for non-security minded folks.

I think that merits revisiting the topic.  As Olof said, what are the
convenience-vs-security tradeoffs everyone is doing today?  Which of
them sound sane, which need tweaking?

thx,

Jason.

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[James Bottomley]

On Fri, 2015-07-10 at 15:08 -0700, Kees Cook wrote:
> On Fri, Jul 10, 2015 at 9:23 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> > On Fri, Jul 10, 2015 at 11:50:30AM -0400, Josh Boyer wrote:
> >> > This is a topic of interest to me that I think would best benefit from a
> >> > conference room discussion.
> >> >
> >> > Items to discuss:
> >> >
> >> >  - Survey the room on workflows and security posture for kernel work
> >> >  - Discussion of threat models, attack vectors
> >> >  - Discuss mitigation methods, tools and techniques
> >> >  - Identify missing tools or features of tools
> >> >
> >> > The intent is to discuss end point security with regards to protecting
> >> > the kernel source tree.
> >>
> >> Interesting.  Though I think it needs a broader audience to be honest.
> >> It would be far easier to use distros as an attack vector than to try
> >> subverting the upstream source code.  This might be a good topic for
> >> something like Linux Plumbers.
> >
> > I agree that the issue of trusting distros and the possibility of
> > hiding malware in distro software is an important one, and one which
> > is best addressed at something like Plumbers.
> >
> > However, there are a number of Kernel-specific security issues which
> > are just as important --- how we maintain our own personal security,
> > and how we make sure a backdoor doesn't get slipped into the kernel
> > source tree.
> 
> I'm curious about the larger set of security topics. Jason's topic
> seems to surround "supply chain security", but I think we've got a lot
> of other areas. Is focusing on just that topic the right thing to do?
> 
> - supply chain security: I see two parts:
>   - intentionally malicious commits (as Ted describes below)

You're missing a huge one and the usual way we get security breaches:
accidental commits that weren't reviewed properly (or which the
reviewers didn't spot the potential security problem).  All of the
recent spate of serious vulnerabilities (heartbleed, shellshock, etc)
have been accidental.

If we can concentrate on this one, and find a way of fixing it, the
malicious but obfuscated commit will be covered under that.

>   - personal security (keep commit credentials secure from theft)

This second one is a bit of a red herring:  Assuming you did steal my
credentials, how would you use them without being detected?

Security is not an absolute, it's a tradeoff.  The point of the tradeoff
is to make sure you address the significant threats while not impeding
the workflow too much.  If we start worrying about and addressing
insignificant threats, eventually you won't get on to kernel.org without
going through airport theatre style security.

> - reactive security: bug fix workflow
>   - getting fixes _to end users_ (not the same as publishing to stable)

Stable is our last point.  After that, it's up to the distros

>   - documenting impact when known (avoiding intentional obfuscation)
> - proactive security: stop security flaws from happening in the first place
>   - scope analysis (defending both userspace and kernel from attack)
>   - threat analysis (how are attacks being made now and in future?)
>   - exposure analysis (syscall interface, device firmware, etc)
>   - static checkers (find and eliminate bug classes in the code)
>   - run-time mitigation features (endless list: memory protection, CFI,
>     ASLR, anti-bruteforcing, etc)

Perhaps the question here is would we be interested in making use of the
core infrastructure initiative to give us a security analysis of parts
of the kernel (and if so, which parts).

James

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Jason Cooper]

On Sat, Jul 11, 2015 at 08:31:13AM +0100, James Bottomley wrote:
> On Fri, 2015-07-10 at 15:08 -0700, Kees Cook wrote:
> > On Fri, Jul 10, 2015 at 9:23 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> > > On Fri, Jul 10, 2015 at 11:50:30AM -0400, Josh Boyer wrote:
> > >> > This is a topic of interest to me that I think would best benefit from a
> > >> > conference room discussion.
> > >> >
> > >> > Items to discuss:
> > >> >
> > >> >  - Survey the room on workflows and security posture for kernel work
> > >> >  - Discussion of threat models, attack vectors
> > >> >  - Discuss mitigation methods, tools and techniques
> > >> >  - Identify missing tools or features of tools
> > >> >
> > >> > The intent is to discuss end point security with regards to protecting
> > >> > the kernel source tree.
> > >>
> > >> Interesting.  Though I think it needs a broader audience to be honest.
> > >> It would be far easier to use distros as an attack vector than to try
> > >> subverting the upstream source code.  This might be a good topic for
> > >> something like Linux Plumbers.
> > >
> > > I agree that the issue of trusting distros and the possibility of
> > > hiding malware in distro software is an important one, and one which
> > > is best addressed at something like Plumbers.
> > >
> > > However, there are a number of Kernel-specific security issues which
> > > are just as important --- how we maintain our own personal security,
> > > and how we make sure a backdoor doesn't get slipped into the kernel
> > > source tree.
> > 
> > I'm curious about the larger set of security topics. Jason's topic
> > seems to surround "supply chain security", but I think we've got a lot
> > of other areas. Is focusing on just that topic the right thing to do?
> > 
> > - supply chain security: I see two parts:
> >   - intentionally malicious commits (as Ted describes below)
> 
> You're missing a huge one and the usual way we get security breaches:
> accidental commits that weren't reviewed properly (or which the
> reviewers didn't spot the potential security problem).  All of the
> recent spate of serious vulnerabilities (heartbleed, shellshock, etc)
> have been accidental.
> 
> If we can concentrate on this one, and find a way of fixing it, the
> malicious but obfuscated commit will be covered under that.

Good point.  Addressing maintainer / reviewer burnout with co- roles and
lead rotation will help significantly.  With that addressed, people
still need to know *what* to look for.

Could we have some sort of post-vuln/CVE conversation dissecting the
vulnerability and how it got there?  Or, perhaps select a few for
process-dissection to be presented at the summit?

No matter how we do it, I think the best mitigation is education for
reviewers.  And there's no better examples than actual events.  How we
get there and how public it can be needs to be discussed though.

> Security is not an absolute, it's a tradeoff.  The point of the tradeoff
> is to make sure you address the significant threats while not impeding
> the workflow too much.  If we start worrying about and addressing
> insignificant threats, eventually you won't get on to kernel.org without
> going through airport theatre style security.

Ack.  My main goal was to give everyone an opportunity to say exactly
what tradeoffs they're making and tools being used.  Then see if there
are some common weaknesses that can be addressed.

The example that runs through my mind was after the Core Initiative was
stood up, the bash exploit was announced (shellshock).  Bash wasn't on
the CI list at the time.  It's certainly relied on by everybody, and was
simply a failure of imagination.

thx,

Jason.

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Theodore Ts'o]

On Sat, Jul 11, 2015 at 04:02:02PM +0000, Jason Cooper wrote:
> 
> Could we have some sort of post-vuln/CVE conversation dissecting the
> vulnerability and how it got there?  Or, perhaps select a few for
> process-dissection to be presented at the summit?

Kees did a really good presentation entitled "security anti-patterns"
a year or two ago at a kernel summit (the one at Edinburgh if I
remember correctly?).  Kees, do you think it would be worth updating
and re-doing that presentation?  And perhaps at a wider set of venues
beyond just the kernel summit....

						- Ted

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Jiri Kosina]

On Sat, 11 Jul 2015, James Bottomley wrote:

> >   - personal security (keep commit credentials secure from theft)
> 
> This second one is a bit of a red herring:  Assuming you did steal my
> credentials, how would you use them without being detected?

If the credentials can be used both to push to ra.kernel.org and to access 
your "local" copy of the GIT repo (on your notebook / desktop / storage), 
I can just push the malicious commit (*) to both repos and you might not 
notice immediately (because you wouldn't get non-fast-forward hint from 
git).

(*) or just ammend some already existing one so that you wouldn't
    notice extra commit when preparing pull request

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

On Mon, Jul 13, 2015 at 10:32:06AM +0200, Jiri Kosina wrote:
> If the credentials can be used both to push to ra.kernel.org and to access 
> your "local" copy of the GIT repo (on your notebook / desktop / storage), 
> I can just push the malicious commit (*) to both repos and you might not 
> notice immediately (because you wouldn't get non-fast-forward hint from 
> git).

This is mitigated somewhat by existing 2-factor mechanisms placed on
select git repositories.

https://korg.wiki.kernel.org/userdoc/gitolite_2fa

To successfully attack in this manner, you would need to push to
gitolite.kernel.org from an IP address that's been previously
2fa-validated by the developer.

Which brings me around to grumbling a bit -- since we've made 2-factor
auth available, only 30 people have set up a token[*] (not even 10% of all
account holders) and only 25 repositories/subdirs have a 2fa requirement
on them, out of 450 defined.

I'm far from suggesting that we make this mandatory, but I'm open to
any suggestions on how we can make more developers enroll with 2fa.

Best,
-- 
Konstantin Ryabitsev
Linux Foundation Collab Projects
Montréal, Québec

[*] Not counting a couple of people using GPG smartcards/yubikeys
    for their ssh authentication.


[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[James Bottomley]

[-- Attachment #1: Type: text/plain, Size: 2318 bytes --]

On Mon, 2015-07-13 at 10:07 -0400, Konstantin Ryabitsev wrote:
> On Mon, Jul 13, 2015 at 10:32:06AM +0200, Jiri Kosina wrote:
> > If the credentials can be used both to push to ra.kernel.org and to access 
> > your "local" copy of the GIT repo (on your notebook / desktop / storage), 
> > I can just push the malicious commit (*) to both repos and you might not 
> > notice immediately (because you wouldn't get non-fast-forward hint from 
> > git).
> 
> This is mitigated somewhat by existing 2-factor mechanisms placed on
> select git repositories.
> 
> https://korg.wiki.kernel.org/userdoc/gitolite_2fa
> 
> To successfully attack in this manner, you would need to push to
> gitolite.kernel.org from an IP address that's been previously
> 2fa-validated by the developer.
>
> Which brings me around to grumbling a bit -- since we've made 2-factor
> auth available, only 30 people have set up a token[*] (not even 10% of all
> account holders) and only 25 repositories/subdirs have a 2fa requirement
> on them, out of 450 defined.
>
> I'm far from suggesting that we make this mandatory, but I'm open to
> any suggestions on how we can make more developers enroll with 2fa.

It's a bit painful for those of us who move around a lot and no-one has
ever articulated a clear threat vector it's supposed to counter.

In fact, I'd argue it gives a false sense of security: the ssh keys and
authentication factors aren't what I'd go after if I were attacking
kernel.org because anything I pushed using a stolen key would instantly
be noticed the next time the maintainer pushed and the tree wouldn't
fast forward.  If I were trying to get a bogus commit into the tree, I'd
be attacking the maintainer's laptop to put it into their personal git
tree (I'd actually tack the code on to an existing commit via rebase ...
cleverly choosing a commit they hadn't yet pushed), so no-one would
notice when it was pushed to kernel.org and it would be properly
accounted for in the subsequent pull request to Linus.  2 factor
authentication does nothing to counter this.

James


> Best,
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On Mon, Jul 13, 2015 at 04:39:20PM +0100, James Bottomley wrote:
> On Mon, 2015-07-13 at 10:07 -0400, Konstantin Ryabitsev wrote:

> > Which brings me around to grumbling a bit -- since we've made 2-factor
> > auth available, only 30 people have set up a token[*] (not even 10% of all
> > account holders) and only 25 repositories/subdirs have a 2fa requirement
> > on them, out of 450 defined.

> It's a bit painful for those of us who move around a lot and no-one has
> ever articulated a clear threat vector it's supposed to counter.

As the owner of about 16% of those repositories and someone who travels
a reasonable amount I have to say I actually find it a lot easier than
the previous system (especially given that previously I had to share a
single kernel.org SSH key over all my machines).  That's got more to do
with being able to use a self supplied SSH key but still.

> fast forward.  If I were trying to get a bogus commit into the tree, I'd
> be attacking the maintainer's laptop to put it into their personal git
> tree (I'd actually tack the code on to an existing commit via rebase ...
> cleverly choosing a commit they hadn't yet pushed), so no-one would
> notice when it was pushed to kernel.org and it would be properly
> accounted for in the subsequent pull request to Linus.  2 factor
> authentication does nothing to counter this.

For me the second factor is as much a defence in depth thing as anything
else.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 2945 bytes --]

On Mon, Jul 13, 2015 at 04:39:20PM +0100, James Bottomley wrote:
> > I'm far from suggesting that we make this mandatory, but I'm open to
> > any suggestions on how we can make more developers enroll with 2fa.
> 
> It's a bit painful for those of us who move around a lot and no-one has
> ever articulated a clear threat vector it's supposed to counter.
> 
> In fact, I'd argue it gives a false sense of security: the ssh keys and
> authentication factors aren't what I'd go after if I were attacking
> kernel.org because anything I pushed using a stolen key would instantly
> be noticed the next time the maintainer pushed and the tree wouldn't
> fast forward.  If I were trying to get a bogus commit into the tree, I'd
> be attacking the maintainer's laptop to put it into their personal git
> tree (I'd actually tack the code on to an existing commit via rebase ...
> cleverly choosing a commit they hadn't yet pushed), so no-one would
> notice when it was pushed to kernel.org and it would be properly
> accounted for in the subsequent pull request to Linus.  2 factor
> authentication does nothing to counter this.

It counters several vectors of attack:

1. Someone makes a commit to your repo while you are on an extended trip
   and it percolates to other repos that pull from you. When you notice
   this upon your return, it's already been widely disseminated.
2. When multiple developers are working on the same repository, an attacker
   can time a commit when one of the developers is about to step away for a
   few days (travel, conference, etc). When the developer is back, they
   are going to do a "git pull" as first thing, to get the latest
   commits from other developers and will therefore likely miss "their
   own" missing commit, pulling it in their tree.
3. Developers who use multiple systems (home workstation and a travel
   laptop, for example) will routinely perform a "git pull" to keep
   their trees in sync and it would be fairly easy to time an injection
   into the tree to sneak in a commit.

Getting private ssh keys is a lot easier than getting full access to a
developer's workstation:

- Many developers ssh to systems without restricting agent forwarding to
  a set of trusted systems only, which would allow an attacker on a
  compromised system to ssh to gitolite.kernel.org with developer's
  credentials.
- Many 0-days in client tools allow full access to local content, but
  not necessarily an ability to execute arbitrary commands.
- I'm willing to bet there are lots of removable storage floating around
  people's workdesks and travel bags that contain full copies of their
  homedirs for backup purposes (tell me it ain't true!).

The 2fa solution we have certainly doesn't solve all possible problems,
but it does have very good reasons to exist.

Regards,
-- 
Konstantin Ryabitsev
Linux Foundation Collab Projects
Montréal, Québec

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[James Bottomley]

[-- Attachment #1: Type: text/plain, Size: 3882 bytes --]

On Mon, 2015-07-13 at 12:05 -0400, Konstantin Ryabitsev wrote:
> On Mon, Jul 13, 2015 at 04:39:20PM +0100, James Bottomley wrote:
> > > I'm far from suggesting that we make this mandatory, but I'm open to
> > > any suggestions on how we can make more developers enroll with 2fa.
> > 
> > It's a bit painful for those of us who move around a lot and no-one has
> > ever articulated a clear threat vector it's supposed to counter.
> > 
> > In fact, I'd argue it gives a false sense of security: the ssh keys and
> > authentication factors aren't what I'd go after if I were attacking
> > kernel.org because anything I pushed using a stolen key would instantly
> > be noticed the next time the maintainer pushed and the tree wouldn't
> > fast forward.  If I were trying to get a bogus commit into the tree, I'd
> > be attacking the maintainer's laptop to put it into their personal git
> > tree (I'd actually tack the code on to an existing commit via rebase ...
> > cleverly choosing a commit they hadn't yet pushed), so no-one would
> > notice when it was pushed to kernel.org and it would be properly
> > accounted for in the subsequent pull request to Linus.  2 factor
> > authentication does nothing to counter this.
> 
> It counters several vectors of attack:
> 
> 1. Someone makes a commit to your repo while you are on an extended trip
>    and it percolates to other repos that pull from you. When you notice
>    this upon your return, it's already been widely disseminated.

If I'm away and not doing any activity, it's only going to be in trees
that are basing on mine, so not really wide dissemnation

> 2. When multiple developers are working on the same repository, an attacker
>    can time a commit when one of the developers is about to step away for a
>    few days (travel, conference, etc). When the developer is back, they
>    are going to do a "git pull" as first thing, to get the latest
>    commits from other developers and will therefore likely miss "their
>    own" missing commit, pulling it in their tree.

How multiple developers wish to work on the same repo is largely up to
them.  I could design a work flow that would detect injections without
2fa, but if they choose another way, it's fine by me.

> 3. Developers who use multiple systems (home workstation and a travel
>    laptop, for example) will routinely perform a "git pull" to keep
>    their trees in sync and it would be fairly easy to time an injection
>    into the tree to sneak in a commit.

Well, that's not what I do: I have one master tree and I work from that.

> Getting private ssh keys is a lot easier than getting full access to a
> developer's workstation:
> 
> - Many developers ssh to systems without restricting agent forwarding to
>   a set of trusted systems only, which would allow an attacker on a
>   compromised system to ssh to gitolite.kernel.org with developer's
>   credentials.

It's possible, but it's a really short window to trick an agent.  The
default of ssh is no agent forwarding, anyway.

> - Many 0-days in client tools allow full access to local content, but
>   not necessarily an ability to execute arbitrary commands.

I'm not sure I get this: you mean client tools may compromise my ssh
keys?

> - I'm willing to bet there are lots of removable storage floating around
>   people's workdesks and travel bags that contain full copies of their
>   homedirs for backup purposes (tell me it ain't true!).

Well, not me ... I use a mirror between my server and my laptop for
that.

> The 2fa solution we have certainly doesn't solve all possible problems,
> but it does have very good reasons to exist.

So: I admit that if I'm careless, 2fa helps protect everyone else.
However, I think you can see that if I'm careful (as I claim I am) 2fa
doesn't buy me much.

James


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Geert Uytterhoeven]

On Mon, Jul 13, 2015 at 6:05 PM, Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
> - I'm willing to bet there are lots of removable storage floating around
>   people's workdesks and travel bags that contain full copies of their
>   homedirs for backup purposes (tell me it ain't true!).

Given how easy it is these days to use luksformatted removable media,
there's not really an excuse for that.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[josh]

On Mon, Jul 13, 2015 at 06:46:42PM +0200, Geert Uytterhoeven wrote:
> On Mon, Jul 13, 2015 at 6:05 PM, Konstantin Ryabitsev
> <konstantin@linuxfoundation.org> wrote:
> > - I'm willing to bet there are lots of removable storage floating around
> >   people's workdesks and travel bags that contain full copies of their
> >   homedirs for backup purposes (tell me it ain't true!).
> 
> Given how easy it is these days to use luksformatted removable media,
> there's not really an excuse for that.

Or "tar ... | gpg", or duplicity, or any number of encrypted backup
solutions.  Keeping around any unencrypted copy of any of your keys is
irresponsible.

- Josh Triplett

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Theodore Ts'o]

On Mon, Jul 13, 2015 at 05:14:16PM +0100, James Bottomley wrote:
> 
> So: I admit that if I'm careless, 2fa helps protect everyone else.
> However, I think you can see that if I'm careful (as I claim I am) 2fa
> doesn't buy me much.

The whole point of defense in depth is that even if you normally are
very careful, if you screw up, there are backup protections that
hopefully will prevent the lapse from being a disaster.

With security, it's always about "belt and suspenders".  Sure, we need
to trade off security gains versus the impacts to convenience.  For
me, using 2FA to protect my ssh and GPG keys makes more sense, so I'm
using a Yubikey Neo to provide that 2FA protection.

						- Ted

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Jiri Kosina]

On Mon, 13 Jul 2015, Konstantin Ryabitsev wrote:

> Getting private ssh keys is a lot easier than getting full access to a
> developer's workstation:

Well ... even the recent example on this very list (a bug in script for 
applying patches being used by prominent maintainers) could be used by an 
attacker to open remote shell with repository access credentials on the 
local system of the maintainer. So I would be rather careful with stating 
that all this is just theoretical excercise.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Kees Cook]

On Sat, Jul 11, 2015 at 9:38 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Sat, Jul 11, 2015 at 04:02:02PM +0000, Jason Cooper wrote:
>>
>> Could we have some sort of post-vuln/CVE conversation dissecting the
>> vulnerability and how it got there?  Or, perhaps select a few for
>> process-dissection to be presented at the summit?
>
> Kees did a really good presentation entitled "security anti-patterns"
> a year or two ago at a kernel summit (the one at Edinburgh if I
> remember correctly?).  Kees, do you think it would be worth updating
> and re-doing that presentation?  And perhaps at a wider set of venues
> beyond just the kernel summit....

I can, yeah, but I sort of think stuff like that is really just
"reference material", and sometimes too specific. Reviewers (and
authors) need to think about high-level risks, not just mechanical
flaws. I would agree that beefing up reviewer roles could really help
this part of the problem, though.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Kees Cook]

On Sat, Jul 11, 2015 at 12:31 AM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Fri, 2015-07-10 at 15:08 -0700, Kees Cook wrote:
>>   - personal security (keep commit credentials secure from theft)
>
> This second one is a bit of a red herring:  Assuming you did steal my
> credentials, how would you use them without being detected?

Well, I meant it in a general sense. Whether that's your ssh key, or
direct access to your entire network through backdoored network card
firmware and SMM code, there are TONS of way to be owned without being
detected. :)

> Security is not an absolute, it's a tradeoff.  The point of the tradeoff
> is to make sure you address the significant threats while not impeding
> the workflow too much.  If we start worrying about and addressing
> insignificant threats, eventually you won't get on to kernel.org without
> going through airport theatre style security.

We have one thing that a lot of other workflows don't: transparency,
so we can examine commit histories, etc. This makes credential theft
much less useful (which I think was your point).

>> - reactive security: bug fix workflow
>>   - getting fixes _to end users_ (not the same as publishing to stable)
>
> Stable is our last point.  After that, it's up to the distros

I don't agree with this. Distros are just one consumer. I think it's
worth examining how real-world devices end up running Linux. Telcos
pushing kernel updates, for example, jumps to mind. I think it's a
weak stance to say "well, they should update to the latest kernel". Is
it a failing of our community that it's so much work for these vendors
to update kernels? Is offering an LTS "good enough", or can we do
more? It's Linux's name that gets smeared by vendors who are terrible
at updating kernels. :(

>>   - documenting impact when known (avoiding intentional obfuscation)
>> - proactive security: stop security flaws from happening in the first place
>>   - scope analysis (defending both userspace and kernel from attack)
>>   - threat analysis (how are attacks being made now and in future?)
>>   - exposure analysis (syscall interface, device firmware, etc)
>>   - static checkers (find and eliminate bug classes in the code)
>>   - run-time mitigation features (endless list: memory protection, CFI,
>>     ASLR, anti-bruteforcing, etc)
>
> Perhaps the question here is would we be interested in making use of the
> core infrastructure initiative to give us a security analysis of parts
> of the kernel (and if so, which parts).

I actually think the issue is body count. We have a lot of tools
already. We have coverity, for example, but it needs full-time work
(by a few people, I think) to trim false-positives, improve rules, and
extract the real bugs. Which companies are paying people to do this
full-time? Our numbers aren't improving much in this area. We've
actually been getting smaller... Dave Jones, come back, we all still
love Trinity! :)

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[James Bottomley]

On Mon, 2015-07-13 at 16:25 -0700, Kees Cook wrote:
> On Sat, Jul 11, 2015 at 12:31 AM, James Bottomley
> <James.Bottomley@hansenpartnership.com> wrote:
> > On Fri, 2015-07-10 at 15:08 -0700, Kees Cook wrote:
> >>   - personal security (keep commit credentials secure from theft)
> >
> > This second one is a bit of a red herring:  Assuming you did steal my
> > credentials, how would you use them without being detected?
> 
> Well, I meant it in a general sense. Whether that's your ssh key, or
> direct access to your entire network through backdoored network card
> firmware and SMM code, there are TONS of way to be owned without being
> detected. :)

Right, so there's no point having a huge lock on your front door if you
know there's a weak catch on the back one.

> > Security is not an absolute, it's a tradeoff.  The point of the tradeoff
> > is to make sure you address the significant threats while not impeding
> > the workflow too much.  If we start worrying about and addressing
> > insignificant threats, eventually you won't get on to kernel.org without
> > going through airport theatre style security.
> 
> We have one thing that a lot of other workflows don't: transparency,
> so we can examine commit histories, etc. This makes credential theft
> much less useful (which I think was your point).

No, that was part of my point: detection of forged commits via stolen
ssh credentials without compromising the laptop are easily detectable.

The other part is that security is a chain: it's only as strong as its
weakest link.  One consideration in making a chain is that you try to
have all the links be of roughly equal strength.  In security terms you
do this because security is a tradeoff: there's no point having onerous
security on one link if another is weak because people just bitch about
the pointless problems this causes.  That's precisely why security is a
tradeoff: you assess the threats and counter what you can in a way that
makes the least impact to usability.  What you should never do (unless
you're a government) is make an elaborate show of security to give a
false impression because clever people notice and real security suffers.

> >> - reactive security: bug fix workflow
> >>   - getting fixes _to end users_ (not the same as publishing to stable)
> >
> > Stable is our last point.  After that, it's up to the distros
> 
> I don't agree with this. Distros are just one consumer. I think it's
> worth examining how real-world devices end up running Linux. Telcos
> pushing kernel updates, for example, jumps to mind. I think it's a
> weak stance to say "well, they should update to the latest kernel". Is
> it a failing of our community that it's so much work for these vendors
> to update kernels? Is offering an LTS "good enough", or can we do
> more? It's Linux's name that gets smeared by vendors who are terrible
> at updating kernels. :(

While security fixes (and the kernel security list) aren't transparent,
I don't really see what else we can do.  Stable is our last best effort
before it gets handed off, unless you have another proposal?

> >>   - documenting impact when known (avoiding intentional obfuscation)
> >> - proactive security: stop security flaws from happening in the first place
> >>   - scope analysis (defending both userspace and kernel from attack)
> >>   - threat analysis (how are attacks being made now and in future?)
> >>   - exposure analysis (syscall interface, device firmware, etc)
> >>   - static checkers (find and eliminate bug classes in the code)
> >>   - run-time mitigation features (endless list: memory protection, CFI,
> >>     ASLR, anti-bruteforcing, etc)
> >
> > Perhaps the question here is would we be interested in making use of the
> > core infrastructure initiative to give us a security analysis of parts
> > of the kernel (and if so, which parts).
> 
> I actually think the issue is body count. We have a lot of tools
> already. We have coverity, for example, but it needs full-time work
> (by a few people, I think) to trim false-positives, improve rules, and
> extract the real bugs. Which companies are paying people to do this
> full-time? Our numbers aren't improving much in this area. We've
> actually been getting smaller... Dave Jones, come back, we all still
> love Trinity! :)

So you seem to be implying this is a funding problem?  We could easily
apply to the CII for a full time position if that's the case (and we
have a good job description).

James

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Kees Cook]

On Tue, Jul 14, 2015 at 12:47 AM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Mon, 2015-07-13 at 16:25 -0700, Kees Cook wrote:
>> I don't agree with this. Distros are just one consumer. I think it's
>> worth examining how real-world devices end up running Linux. Telcos
>> pushing kernel updates, for example, jumps to mind. I think it's a
>> weak stance to say "well, they should update to the latest kernel". Is
>> it a failing of our community that it's so much work for these vendors
>> to update kernels? Is offering an LTS "good enough", or can we do
>> more? It's Linux's name that gets smeared by vendors who are terrible
>> at updating kernels. :(
>
> While security fixes (and the kernel security list) aren't transparent,
> I don't really see what else we can do.  Stable is our last best effort
> before it gets handed off, unless you have another proposal?

I don't presently, no. Stable is certainly good, but I think we're
missing a "real world workload testing" element somewhere. Again,
though, this should probably be on the vendors, but there is very
little interest in it, since the ROI appears bad.

>
>> >>   - documenting impact when known (avoiding intentional obfuscation)
>> >> - proactive security: stop security flaws from happening in the first place
>> >>   - scope analysis (defending both userspace and kernel from attack)
>> >>   - threat analysis (how are attacks being made now and in future?)
>> >>   - exposure analysis (syscall interface, device firmware, etc)
>> >>   - static checkers (find and eliminate bug classes in the code)
>> >>   - run-time mitigation features (endless list: memory protection, CFI,
>> >>     ASLR, anti-bruteforcing, etc)
>> >
>> > Perhaps the question here is would we be interested in making use of the
>> > core infrastructure initiative to give us a security analysis of parts
>> > of the kernel (and if so, which parts).
>>
>> I actually think the issue is body count. We have a lot of tools
>> already. We have coverity, for example, but it needs full-time work
>> (by a few people, I think) to trim false-positives, improve rules, and
>> extract the real bugs. Which companies are paying people to do this
>> full-time? Our numbers aren't improving much in this area. We've
>> actually been getting smaller... Dave Jones, come back, we all still
>> love Trinity! :)
>
> So you seem to be implying this is a funding problem?  We could easily
> apply to the CII for a full time position if that's the case (and we
> have a good job description).

It'll take more than 1 person. :) It's a cultural problem with the
tech industry: defensive security is very hard to measure, so it tends
to be ignored until something bad happens publicly. Unfortunately,
there is a LOT happening secretly that we need to defend against, and
everyone depending on Linux should be putting forward people to work
on that. Like testing, it appears to be poor ROI, though.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [CORE TOPIC] dev/maintainer workflow security

[Steven Rostedt]

On Mon, 13 Jul 2015 10:32:06 +0200 (CEST)
Jiri Kosina <jkosina@suse.com> wrote:

> On Sat, 11 Jul 2015, James Bottomley wrote:
> 
> > >   - personal security (keep commit credentials secure from theft)
> > 
> > This second one is a bit of a red herring:  Assuming you did steal my
> > credentials, how would you use them without being detected?
> 
> If the credentials can be used both to push to ra.kernel.org and to access 
> your "local" copy of the GIT repo (on your notebook / desktop / storage), 
> I can just push the malicious commit (*) to both repos and you might not 
> notice immediately (because you wouldn't get non-fast-forward hint from 
> git).

Actually, I do development on a different box than I push with. Thus,
if someone did modify both that box and my korg repo, I would notice a
problem as soon as I push my development box to the box I push with.

Now the attacker would need to compromise that development box too.

-- Steve