[Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

I believe we have reasonably well-established process for handling 
security issues that are a matter of a single, reasonably self-contained 
fixup.

Past experiences with Meltdown, Spectre and L1TF have shown that we're not 
really ready to handle that in a reasonably sane way yet.

Yeah, at the end of the day we managed to have the fixes propagated in 
time to Linus' tree, to stable, and to distros as well, but it was 
completely out of anything regular, and definitely had permanent damaging 
effects (I'd say both in personal and business aspects for almost 
everybody who had to participate).

I'd believe that everybody involved would agree that this didn't work 
really well, and if it potentially would have to happen again (and we 
already went through it at least twice this year), it would not be 
sustainable.

I am not completely sure what we could do to improve this, especially with 
our kernel community hats on -- I am pretty sure a lot is happening on the 
corporate level between individual "corporate stakeholders". Ideas I think 
would be worth discussing:

- how to adapt our processess to be able to deal with such situations 
  better should they happen in the future again. So far all our 
  longer-term development has been concentrated around LKML (and other 
  MLs) and the existing maintainership communities / structures, but the 
  embargos for new big features don't really fit into this

- how to make sure that proper pressure is applied on the companies that 
  are handling embargoes irresponsibly wrt. linux/opensource development 
  (well, even some proprietary vendors were rather unhappy with those 
  events) from us as the linux kernel community
  + and perhaps even more importantly, what exactly we should be pressing 
    for

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Linus Torvalds]

On Thu, Sep 6, 2018 at 12:18 PM Jiri Kosina <jikos@kernel.org> wrote:
>
> I am not completely sure what we could do to improve this, especially with
> our kernel community hats on -- I am pretty sure a lot is happening on the
> corporate level between individual "corporate stakeholders".

One particular pain point this last time around were the stable
backports, I feel.

A lot of that was that the actual *fixes* were marked for stable, but
quite often they were preceded by cleanups and other updates that
didn't actually fix things directly, and that weren't in themselves
explicitly marked for stable and didn't have a Fixes: tag, because
they were prep-work.

So we had _several_ nasty regressions in stable that never showed up
in mainline, because there was some non-obvious dependency that didn't
cause a merge conflict, but did cause a "this commit needed that other
commit to work right".

We should probably at least think about having a way to mark those.
Something like a "for-stable-because-of-subsequent-patches" tag?

Or just more eager use of the table cc? I often feel bad about adding
"cc: stable" to preparatory patches that don't actually fix the bug,
but I think it was bad this time around.

Of course, I also hope that we're over the worst.

               Linus

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

On Thu, 6 Sep 2018, Linus Torvalds wrote:

> > I am not completely sure what we could do to improve this, especially with
> > our kernel community hats on -- I am pretty sure a lot is happening on the
> > corporate level between individual "corporate stakeholders".
> 
> One particular pain point this last time around were the stable
> backports, I feel.
> 
> A lot of that was that the actual *fixes* were marked for stable, but
> quite often they were preceded by cleanups and other updates that
> didn't actually fix things directly, and that weren't in themselves
> explicitly marked for stable and didn't have a Fixes: tag, because
> they were prep-work.
> 
> So we had _several_ nasty regressions in stable that never showed up
> in mainline, because there was some non-obvious dependency that didn't
> cause a merge conflict, but did cause a "this commit needed that other
> commit to work right".

I fully agree that this is an issue for stable. On the other hand, I would 
be reasonably sure this has been equally painful issue for stable even 
before this particular disaster (and all the preceeding stable discussions 
on this very ML sort of do support that).

> We should probably at least think about having a way to mark those. 
> Something like a "for-stable-because-of-subsequent-patches" tag?
> 
> Or just more eager use of the table cc? I often feel bad about adding
> "cc: stable" to preparatory patches that don't actually fix the bug,
> but I think it was bad this time around.

Maybe at least partial solution (or first step) to this would be to 
somehow make sure that "these patches form an actual patchset that belongs 
together and is in fact one single thing" information somehow gets 
preserved in maintainer's / your tree.

It's sort-of achievable if everybody (not only the patchset producers, but 
also the consumers) would be very familiar with the idea of strictly topic 
git branches, but that's probably not realistic.

I currently have no good idea how exactly this should be done technically, 
but certainly it's doable and would be of a tremendous help to downstream, 
older-codebase consumers of your tree.

> Of course, I also hope that we're over the worst.

Fully agreed. Also, I hope that world is flat :)

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Eduardo Valentin]

Hello,

On Thu, Sep 06, 2018 at 11:14:08PM +0200, Jiri Kosina wrote:
> On Thu, 6 Sep 2018, Linus Torvalds wrote:
> 
> > > I am not completely sure what we could do to improve this, especially with
> > > our kernel community hats on -- I am pretty sure a lot is happening on the
> > > corporate level between individual "corporate stakeholders".
> > 
> > One particular pain point this last time around were the stable
> > backports, I feel.
> > 

Totally agree.

> > A lot of that was that the actual *fixes* were marked for stable, but
> > quite often they were preceded by cleanups and other updates that
> > didn't actually fix things directly, and that weren't in themselves
> > explicitly marked for stable and didn't have a Fixes: tag, because
> > they were prep-work.
> > 
> > So we had _several_ nasty regressions in stable that never showed up
> > in mainline, because there was some non-obvious dependency that didn't
> > cause a merge conflict, but did cause a "this commit needed that other
> > commit to work right".
> 

Yeah, I think the later case of hidden dependencies is nastier to 
deal with than the first case of long patchset with lots of rework. 
And I remember the meltdown spectre case required quite a lot of elbow
grease from different involved developers to find those hidden patches,
specially on backports to kernels older than 4.14. Obviously with help 
from more experienced x86 maintainers, eventually things were fixed.

Now, one thing that really helped was the fact that 4.14.y backport was
done by the x86 maintainers. However, it is not feasible to expect that
happening for all stable branches, like it did not happen during the
spectre/meltdown story.

> I fully agree that this is an issue for stable. On the other hand, I would 
> be reasonably sure this has been equally painful issue for stable even 
> before this particular disaster (and all the preceeding stable discussions 
> on this very ML sort of do support that).
> 
> > We should probably at least think about having a way to mark those. 
> > Something like a "for-stable-because-of-subsequent-patches" tag?
> > 
> > Or just more eager use of the table cc? I often feel bad about adding
> > "cc: stable" to preparatory patches that don't actually fix the bug,
> > but I think it was bad this time around.
> 
> Maybe at least partial solution (or first step) to this would be to 
> somehow make sure that "these patches form an actual patchset that belongs 
> together and is in fact one single thing" information somehow gets 
> preserved in maintainer's / your tree.
> 
> It's sort-of achievable if everybody (not only the patchset producers, but 
> also the consumers) would be very familiar with the idea of strictly topic 
> git branches, but that's probably not realistic.
> 
> I currently have no good idea how exactly this should be done technically, 
> but certainly it's doable and would be of a tremendous help to downstream, 
> older-codebase consumers of your tree.

Well, keeping the topic branch for future reference can help yes. Now,
the more problematic part is the dependencies needed due to changes that
happened after the target backport kernel (say 4.9.y)  and the reference
backport / topic branch (say 4.15), but that were not necessarily part
of the original patch set that either prepare the code or really fixes
the issue finally.

And that even brings the question if all of those should be brought to
stable, or if some other version of the fix should be considered.

> 
> > Of course, I also hope that we're over the worst.
> 
> Fully agreed. Also, I hope that world is flat :)
> 

hehe..

> Thanks,
> 
> -- 
> Jiri Kosina
> SUSE Labs
> 
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Eduardo Valentin]

Hey,

On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote:
> I believe we have reasonably well-established process for handling 
> security issues that are a matter of a single, reasonably self-contained 
> fixup.
> 
> Past experiences with Meltdown, Spectre and L1TF have shown that we're not 
> really ready to handle that in a reasonably sane way yet.
> 
> Yeah, at the end of the day we managed to have the fixes propagated in 
> time to Linus' tree, to stable, and to distros as well, but it was 
> completely out of anything regular, and definitely had permanent damaging 
> effects (I'd say both in personal and business aspects for almost 
> everybody who had to participate).
> 
> I'd believe that everybody involved would agree that this didn't work 
> really well, and if it potentially would have to happen again (and we 
> already went through it at least twice this year), it would not be 
> sustainable.
> 
> I am not completely sure what we could do to improve this, especially with 
> our kernel community hats on -- I am pretty sure a lot is happening on the 
> corporate level between individual "corporate stakeholders". Ideas I think 
> would be worth discussing:
> 
> - how to adapt our processess to be able to deal with such situations 
>   better should they happen in the future again. So far all our 
>   longer-term development has been concentrated around LKML (and other 
>   MLs) and the existing maintainership communities / structures, but the 
>   embargos for new big features don't really fit into this
> 
> - how to make sure that proper pressure is applied on the companies that 
>   are handling embargoes irresponsibly wrt. linux/opensource development 
>   (well, even some proprietary vendors were rather unhappy with those 
>   events) from us as the linux kernel community
>   + and perhaps even more importantly, what exactly we should be pressing 
>     for

Should we add maybe a point here to discuss which kernels are to be
considered for patching in these cases? All the stable branches? Only
mainline? Obviously, either extreme cases can hurt people. Patching
older kernels requires insane amount of work and patching only mainline
leaves distros on limbo.

Honestly, the fact that somehow the community managed to make this to
stable (and eventually to distros) is really good. Imagine for a second
a world in which these made only mainline and no stable branch..

In any case, maybe the community should consider what really is going to
be effectively patched. That may be an extra push for distros to upgrade
older kernels as well.


> 
> Thanks,
> 
> -- 
> Jiri Kosina
> SUSE Labs
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Geert Uytterhoeven]

Hi Eduardo,

On Fri, Sep 7, 2018 at 12:55 AM Eduardo Valentin <edubezval@gmail.com> wrote:
> On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote:
> Honestly, the fact that somehow the community managed to make this to
> stable (and eventually to distros) is really good. Imagine for a second
> a world in which these made only mainline and no stable branch..

It could have been the disaster needed to trigger a paradigm shift in the
software industry?

For a fully isolated/controlled system old stable may make sense, but in
the post-Internet connected world, where the whole world population can
peek at your front door, you'd better be prepared to upgrade any time.
Despite backporting, you'll always be vulnerable to some security issues
that were fixed in mainline.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jani Nikula]

On Thu, 06 Sep 2018, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> A lot of that was that the actual *fixes* were marked for stable, but
> quite often they were preceded by cleanups and other updates that
> didn't actually fix things directly, and that weren't in themselves
> explicitly marked for stable and didn't have a Fixes: tag, because
> they were prep-work.

There's always the struggle of whether to put fixes or cleanups first in
the series. The former makes backports easier, the latter makes the
actual fixes easier. And the prep work could be committed quickly while
the fixes are still being debated.

> So we had _several_ nasty regressions in stable that never showed up
> in mainline, because there was some non-obvious dependency that didn't
> cause a merge conflict, but did cause a "this commit needed that other
> commit to work right".
>
> We should probably at least think about having a way to mark those.
> Something like a "for-stable-because-of-subsequent-patches" tag?

If you know the commit ids of the dependencies, i.e. they've been
applied, per Documentation/process/stable-kernel-rules.rst you could
add:

     Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched: Check for idle
     Cc: <stable@vger.kernel.org> # 3.3.x: 1b9508f: sched: Rate-limit newidle
     Cc: <stable@vger.kernel.org> # 3.3.x: fd21073: sched: Fix affinity logic
     Cc: <stable@vger.kernel.org> # 3.3.x

listing the dependencies. I don't see that used much, though, so not
sure defining something new could gain more traction.

> Or just more eager use of the table cc? I often feel bad about adding
> "cc: stable" to preparatory patches that don't actually fix the bug,
> but I think it was bad this time around.

If the fix with prep work is in a topic branch, one could add cc: stable
to the merge commit.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Graphics Center

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

On Thu, 6 Sep 2018, Eduardo Valentin wrote:

> Should we add maybe a point here to discuss which kernels are to be 
> considered for patching in these cases? All the stable branches? Only 
> mainline? Obviously, either extreme cases can hurt people. Patching 
> older kernels requires insane amount of work and patching only mainline 
> leaves distros on limbo.

That'd be mostly question for the stable guys I guess. I am not sure how 
often did they in the past have to say "sorry, the backport is horribly 
complex, so we are not backporting the fix and we're keeping the bug 
unfixed".

Greg, is this something that actually has been happening for real in the 
past? Or would that absolutely break the expectations that stable tree 
consumers have?

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]



On Thu, 2018-09-06 at 13:56 -0700, Linus Torvalds wrote:
> 
> A lot of that was that the actual *fixes* were marked for stable, but
> quite often they were preceded by cleanups and other updates that
> didn't actually fix things directly, and that weren't in themselves
> explicitly marked for stable and didn't have a Fixes: tag, because
> they were prep-work.
> 
> So we had _several_ nasty regressions in stable that never showed up
> in mainline, because there was some non-obvious dependency that didn't
> cause a merge conflict, but did cause a "this commit needed that other
> commit to work right".

This time round, for L1TF I think we did relatively OK on the stable
backports. The dependencies we were missing were stuff that was just
randomly cleaned up in the intervening kernels.

For example, the 'Simplify p[g4um]d_page() macros' commit (fd7e315988)
start out as *purely* a cosmetic improvement to reduce duplication by
introducing a pgd_pfn() macro.

Some time later, completely unpredicted at the time of that commit, we
came along and made pgd_pfn() do something different. It's *those*
"dependencies" which tend to get missed, and they can't be predicted at
the time of the original commit.

I don't think we did so badly with the commits which were done as
cleanups in preparation for the event itself.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Andy Lutomirski]

On Thu, Sep 6, 2018 at 12:18 PM, Jiri Kosina <jikos@kernel.org> wrote:
> I believe we have reasonably well-established process for handling
> security issues that are a matter of a single, reasonably self-contained
> fixup.
>
> Past experiences with Meltdown, Spectre and L1TF have shown that we're not
> really ready to handle that in a reasonably sane way yet.
>

To a large extend, I think that the problems we had with Meltdown and
Spectre weren't the fault of our processes so much as the fact that we
weren't allowed to have a sane process.  As far as I know, the number
of upstream-focused non-Intel-employed Linux developers who were told
about Meltdown and Spectre in a timely manner can be counted on a very
small number of fingers on one hand.  Those people had no channel to
communicate with any of the distro people involved in a timely manner.
This situation was not good.

Perhaps the Linux community could have a small team that handles
coordination of severe incidents along with a documented process for
what happens when the process is invoked.  It would need to be
designed in such a way that vendors would be willing to use it when
needed and that it gave the Linux world enough flexibility to actually
handle the issue.  At the very least, the relevant maintainers and
perhaps some relevant upstream reviewers would get notified, and the
people working on the issue for distros could actually communicate
with the upstream folks.  The total set of people involved would be
small.  It could be useful to try to get the process to work similarly
to whatever Microsoft and Apple do.

(There was another CVE that got much less press that I was involved,
and it didn't get much attention in Linux land because Linux was only
minimally affected.  Despite being an Intel/AMD issue, all of the
coordination was done by Microsoft, and it was done remarkably well
once the process actually got started.)

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Thomas Gleixner]

On Fri, 7 Sep 2018, Andy Lutomirski wrote:
> (There was another CVE that got much less press that I was involved,
> and it didn't get much attention in Linux land because Linux was only
> minimally affected.  Despite being an Intel/AMD issue, all of the
> coordination was done by Microsoft, and it was done remarkably well
> once the process actually got started.)

The point is that the coordination done by the entity who 'owns' the thing
is the key.

Contrary to Meltdown/Spectre Intel informed us about L1Tf halfways early
and allowed _all_ involved parties to talk to each other. There were still
some rough edges to bring key people like Greg in, but that was a minor
nuisance compared to the whole Meltdown/Spectre mess.

Of course there was no communication channel which allowed us to talk in a
workable way, but that got resolved by ourself setting up a encrypted
mailing list which made halfways normal kernel style cooperation
possible. That still has its rough edges vs. limited review capacity and
testing, but compared to Meltdown/Spectre L1TF was halfways workable.

So we surely have the ability to communicate properly in an embargo
situation, but that requires that the entity who controls the issue is

 1) Telling us in time and putting all cards on the table
 2) Setting no silly restrictions vs. who can talk to whom

That said, I agree that a more formal process with clear rules might make
it easier for companies to handle that proper. It's definitely worth to
try.

Though it won't make the issues which come inherently with embargo
development go away. Mechanisms we rely on like 0-bot, kernel-ci and others
need to grow an embargo mode as well.

Thanks,

	tglx

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Mauro Carvalho Chehab]

Em Sat, 8 Sep 2018 10:56:35 +0200 (CEST)
Thomas Gleixner <tglx@linutronix.de> escreveu:

> On Fri, 7 Sep 2018, Andy Lutomirski wrote:
> > (There was another CVE that got much less press that I was involved,
> > and it didn't get much attention in Linux land because Linux was only
> > minimally affected.  Despite being an Intel/AMD issue, all of the
> > coordination was done by Microsoft, and it was done remarkably well
> > once the process actually got started.)  
> 
> The point is that the coordination done by the entity who 'owns' the thing
> is the key.

I didn't participate on handing those bugs. So, I'll give just my
2 cents here.

IMO, the problem with embargoed security issues is that the affected
vendor will very likely want to contain the ones that have access to
the bug-related information. They also want to have some legal ways
to protect their asses^Wassets if data gets disclosed.

In other words, they'll very likely require some sort of NDA. As 
we're working on different places, each one is under a different NDA
agreement with the affected vendor (and some may even not have any
signed NDA at all). I suspect that identifying who has a signed NDA
that would cover the assets for everyone that would have access to
the embargoed data can take quite some time.

> Contrary to Meltdown/Spectre Intel informed us about L1Tf halfways early
> and allowed _all_ involved parties to talk to each other.

That's probably because they had already gave some sort of 
"clearance" to the ones that handled Meltdown/Spectre.

I suspect that, if L1Tf were affecting some other vendor, it would 
also be as painful as it was for Meltdown/Spectre.

IMHO, the best would be to have a formal/legal way to handle it.

I suspect that having a single entity like LF handling a formal
process for embargoed issues could help making it smoother.
That would likely envolve having a single NDA model that could
be signed with vendors (either before of after an event like that),
and LF having pre-signed NDA with the group of maintainers/developers
that would more likely handle such issues.

> There were still
> some rough edges to bring key people like Greg in, but that was a minor
> nuisance compared to the whole Meltdown/Spectre mess.
> 
> Of course there was no communication channel which allowed us to talk in a
> workable way, but that got resolved by ourself setting up a encrypted
> mailing list which made halfways normal kernel style cooperation
> possible. That still has its rough edges vs. limited review capacity and
> testing, but compared to Meltdown/Spectre L1TF was halfways workable.

A formal agreement with a single entity could help there too.

I mean, provided that everyone's involved is under a legal umbrella,
it should be possible for the vendor to give logical access to some
of their own labs where it would be possible to run the tests with
the real thing.

> 
> So we surely have the ability to communicate properly in an embargo
> situation, but that requires that the entity who controls the issue is
> 
>  1) Telling us in time and putting all cards on the table
>  2) Setting no silly restrictions vs. who can talk to whom
> 
> That said, I agree that a more formal process with clear rules might make
> it easier for companies to handle that proper. It's definitely worth to
> try.
> 
> Though it won't make the issues which come inherently with embargo
> development go away. Mechanisms we rely on like 0-bot, kernel-ci and others
> need to grow an embargo mode as well.
> 
> Thanks,
> 
> 	tglx
> 
> 
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss



Thanks,
Mauro

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab wrote:
> IMHO, the best would be to have a formal/legal way to handle it.

No, sorry, some of us are not allowed legally to sign NDAs for stuff
like this.  So keeping legal out of is it the best solution and we have
done that pretty well so far.

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Andy Lutomirski]

> On Sep 8, 2018, at 4:34 AM, Greg KH <greg@kroah.com> wrote:
> 
>> On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab wrote:
>> IMHO, the best would be to have a formal/legal way to handle it.
> 
> No, sorry, some of us are not allowed legally to sign NDAs for stuff
> like this.  So keeping legal out of is it the best solution and we have
> done that pretty well so far.
> 

A lot us us (such as yours truly) have NDAs in place. I would love a clear mechanism by which a vendor gives explicit permission for me to communicate with other relevant parties. It doesn’t need to be fancy and legalistic, but having it written down would be very, very nice.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[James Bottomley]

On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote:
> On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab
> wrote:
> > IMHO, the best would be to have a formal/legal way to handle it.
> 
> No, sorry, some of us are not allowed legally to sign NDAs for stuff
> like this.

As a blanket statement this simply isn't true.  It is certainly
possible for a specific NDA to conflict with other agreed obligations
an individual has in which case that specific NDA can't be signed. 
However, knowing the obligations, it's also possible to craft a
different version of the NDA that can be signed.

There's also significant problems with badly worded NDAs and Open
Source: given you're going to produce a thing which everyone can see,
disclosure ipso facto eventually occurs so the NDA has to take this
correctly into account but again, this can be done.

I can respect the moral position that NDAs are incompatible with the
values of open source but it's an individual choice and conscience
issue not a legal one.

>   So keeping legal out of is it the best solution and we have
> done that pretty well so far.

I think we might benefit from a discussion of whether we could have
handled Meltdown/Spectre better in an NDA framework ... I'm not saying
it would have been any better, just that we might consider if some
driving need for secrecy caused us to be left out of the loop and
whether a small cabal in the know with an NDA might have steered us
better.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Sat, Sep 08, 2018 at 07:20:00AM -0700, Andy Lutomirski wrote:
> 
> 
> > On Sep 8, 2018, at 4:34 AM, Greg KH <greg@kroah.com> wrote:
> > 
> >> On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab wrote:
> >> IMHO, the best would be to have a formal/legal way to handle it.
> > 
> > No, sorry, some of us are not allowed legally to sign NDAs for stuff
> > like this.  So keeping legal out of is it the best solution and we have
> > done that pretty well so far.
> > 
> 
> A lot us us (such as yours truly) have NDAs in place. I would love a
> clear mechanism by which a vendor gives explicit permission for me to
> communicate with other relevant parties. It doesn’t need to be fancy
> and legalistic, but having it written down would be very, very nice.

I agree, what we have now is a mis-match of different people working for
different companies with different types of NDAs through different
companies or not, trying to deal with different companies trying to work
with us.

Having something "simple" for us to work off of would be good, and this
isn't the first thread that this topic has come up on.  I know people
want to solve this, and maybe the LF is the way to go here.  There was a
meeting last week at Harvard to start to consider how this could happen.
Hopefully in a few months we will have something "solid" to discuss.

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Sat, Sep 08, 2018 at 08:00:29AM -0700, James Bottomley wrote:
> On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote:
> > On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab
> > wrote:
> > > IMHO, the best would be to have a formal/legal way to handle it.
> > 
> > No, sorry, some of us are not allowed legally to sign NDAs for stuff
> > like this.
> 
> As a blanket statement this simply isn't true.

Um, I said "some of us".  Some of us can, some of us can not.  That's
a blanket statement that has to be true :)

> It is certainly
> possible for a specific NDA to conflict with other agreed obligations
> an individual has in which case that specific NDA can't be signed. 
> However, knowing the obligations, it's also possible to craft a
> different version of the NDA that can be signed.
> 
> There's also significant problems with badly worded NDAs and Open
> Source: given you're going to produce a thing which everyone can see,
> disclosure ipso facto eventually occurs so the NDA has to take this
> correctly into account but again, this can be done.
> 
> I can respect the moral position that NDAs are incompatible with the
> values of open source but it's an individual choice and conscience
> issue not a legal one.

I never brought up moral positions, you did.  I do not have a moral
position on this, sorry.

> >   So keeping legal out of is it the best solution and we have
> > done that pretty well so far.
> 
> I think we might benefit from a discussion of whether we could have
> handled Meltdown/Spectre better in an NDA framework ... I'm not saying
> it would have been any better, just that we might consider if some
> driving need for secrecy caused us to be left out of the loop and
> whether a small cabal in the know with an NDA might have steered us
> better.

This is what happened for L1TF, and yes, it was steered better and
worked amazingly well for the conditions under which we were forced to
work.

So things are getting better, Meltdown/Spectre showed some people that
that was _not_ the way to handle it, with hard proof.  So they changed,
and we will continue to revise and work through these issues.

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[James Bottomley]

On Sat, 2018-09-08 at 17:32 +0200, Greg KH wrote:
> On Sat, Sep 08, 2018 at 08:00:29AM -0700, James Bottomley wrote:
> > On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote:
> > > On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab
> > > wrote:
> > > > IMHO, the best would be to have a formal/legal way to handle
> > > > it.
> > > 
> > > No, sorry, some of us are not allowed legally to sign NDAs for
> > > stuff like this.
> > 
> > As a blanket statement this simply isn't true.
> 
> Um, I said "some of us".  Some of us can, some of us can not.  That's
> a blanket statement that has to be true :)

OK, let me make it more specific: there exists no individual
contributing to open source in a leadership capacity for whom a
signable NDA cannot be crafted.

The reason is fairly simple: leadership positions come with duties of
care which include duties of confidentiality (think adjudication of
anti-harassment policy or code of conflict) and an NDA could be crafted
directly along the lines of the duty of confidentiality.  Now it's not
that an employment or other agreement couldn't forbid this, but if it
did the leader in question would be in a false position already because
their agreement effectively forbids them from taking a leadership
position in open source in the first place.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

On Sat, 8 Sep 2018, James Bottomley wrote:

> I think we might benefit from a discussion of whether we could have
> handled Meltdown/Spectre better in an NDA framework ... 

Well, at the end of the day, we actually did handle it in the NDA 
framework (otherwise the end result wrt. stable and distros really 
wouldn't look like it did by the release date), but everything else 
(timing, information sharing, feedback channels to the "owners", ...) 
didn't really work all that well.

--
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[James Bottomley]

On Sat, 2018-09-08 at 21:26 +0200, Jiri Kosina wrote:
> On Sat, 8 Sep 2018, James Bottomley wrote:
> 
> > I think we might benefit from a discussion of whether we could have
> > handled Meltdown/Spectre better in an NDA framework ... 
> 
> Well, at the end of the day, we actually did handle it in the NDA 
> framework (otherwise the end result wrt. stable and distros really 
> wouldn't look like it did by the release date), but everything else 
> (timing, information sharing, feedback channels to the "owners",
> ...) didn't really work all that well.

Right, but my impression was the NDAs came late in the game.  I think
what happened early is we got an -ENONDA from the various lawyers
involved which lead them to cut most of us out of the handling loop. 
Eventually the distros had to sign NDAs to get back in.  Now this
situation is undoubtedly due to lack of open source awareness among the
various disclosing parties, but it does beg the question if we'd had an
NDA framework in place (constructed by someone who knows what they're
doing in open source) and we'd been able to say: "sure, this is the way
we handle NDA issues in open source, sign here" we might have been in
the loop much earlier.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Linus Torvalds]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

On Sat, Sep 8, 2018, 08:54 James Bottomley <
James.Bottomley@hansenpartnership.com> wrote:

>
> OK, let me make it more specific: there exists no individual
> contributing to open source in a leadership capacity for whom a
> signable NDA cannot be crafted.
>

No.

I don't sign NDA's. I just don't do it.

It's that simple.

It's actually worked pretty well. It started because I worked for a direct
competitor to Intel, and couldn't sign an NDA for the really old f0 0f
lockup issue.

Not having an NDA back then turned out to be a good thing, because it made
it a non-issue when leaks happened. So I started the policy that I never
want to be in the position that I had to worry legally about being in the
position of being under an NDA and knowing things outside of the leaks.

Instead, I've had a gentleman's agreement with companies - nothing legally
binding, but over the years people have come to realize that the leaks
don't come from me.

So I don't do NDA's. Maybe some Linux Foundation NDA agreement technically
covers me, but at least with the Intel cases, Intel is actually aware of my
non-NDA situation and is fine with it.

      Linus

>

[-- Attachment #2: Type: text/html, Size: 2027 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[James Bottomley]

On Sat, 2018-09-08 at 12:49 -0700, Linus Torvalds wrote:
> On Sat, Sep 8, 2018, 08:54 James Bottomley <
> James.Bottomley@hansenpartnership.com> wrote:
> 
> > 
> > OK, let me make it more specific: there exists no individual
> > contributing to open source in a leadership capacity for whom a
> > signable NDA cannot be crafted.
> > 
> 
> No.
> 
> I don't sign NDA's. I just don't do it.
> 
> It's that simple.

But that's you're choice; it's not because legally you can't.

> It's actually worked pretty well. It started because I worked for a
> direct competitor to Intel, and couldn't sign an NDA for the really
> old f0 0f lockup issue.
> 
> Not having an NDA back then turned out to be a good thing, because it
> made it a non-issue when leaks happened. So I started the policy that
> I never want to be in the position that I had to worry legally about
> being in the position of being under an NDA and knowing things
> outside of the leaks.
> 
> Instead, I've had a gentleman's agreement with companies - nothing
> legally binding, but over the years people have come to realize that
> the leaks don't come from me.
> 
> So I don't do NDA's. Maybe some Linux Foundation NDA agreement
> technically covers me, but at least with the Intel cases, Intel is
> actually aware of my non-NDA situation and is fine with it.

I'm fine with all of this as an argument.  If we believe that signing
NDAs would eventually lead to worse disasters because agreeing to them
now means corporations never change and never take our views into
account, then we should have the debate and make the decision for sound
policy reasons not because there's some spurious legal bar.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Andy Lutomirski]

On Sat, Sep 8, 2018 at 2:24 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Sat, 2018-09-08 at 12:49 -0700, Linus Torvalds wrote:
>> On Sat, Sep 8, 2018, 08:54 James Bottomley <
>> James.Bottomley@hansenpartnership.com> wrote:
>>
>> >
>> > OK, let me make it more specific: there exists no individual
>> > contributing to open source in a leadership capacity for whom a
>> > signable NDA cannot be crafted.
>> >
>>
>> No.
>>
>> I don't sign NDA's. I just don't do it.
>>
>> It's that simple.
>
> But that's you're choice; it's not because legally you can't.
>
>> It's actually worked pretty well. It started because I worked for a
>> direct competitor to Intel, and couldn't sign an NDA for the really
>> old f0 0f lockup issue.
>>
>> Not having an NDA back then turned out to be a good thing, because it
>> made it a non-issue when leaks happened. So I started the policy that
>> I never want to be in the position that I had to worry legally about
>> being in the position of being under an NDA and knowing things
>> outside of the leaks.
>>
>> Instead, I've had a gentleman's agreement with companies - nothing
>> legally binding, but over the years people have come to realize that
>> the leaks don't come from me.
>>
>> So I don't do NDA's. Maybe some Linux Foundation NDA agreement
>> technically covers me, but at least with the Intel cases, Intel is
>> actually aware of my non-NDA situation and is fine with it.
>
> I'm fine with all of this as an argument.  If we believe that signing
> NDAs would eventually lead to worse disasters because agreeing to them
> now means corporations never change and never take our views into
> account, then we should have the debate and make the decision for sound
> policy reasons not because there's some spurious legal bar.
>

My NDA is through my company.  I would *love* to cancel it and set up
a replacement arrangement through LF or a similar entity, or to just
not replace it at all.  My company is not equipped for the kind of
wrangling that would have helped during Meltdown and a couple of other
situations, whereas anything reasonable set up for the purpose would
work much better.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Mauro Carvalho Chehab]

Em Sat, 8 Sep 2018 15:33:22 -0700
Andy Lutomirski <luto@kernel.org> escreveu:

> On Sat, Sep 8, 2018 at 2:24 PM, James Bottomley
> <James.Bottomley@hansenpartnership.com> wrote:
> > On Sat, 2018-09-08 at 12:49 -0700, Linus Torvalds wrote:  
> >> On Sat, Sep 8, 2018, 08:54 James Bottomley <  
> >> James.Bottomley@hansenpartnership.com> wrote:  
> >>  
> >> >
> >> > OK, let me make it more specific: there exists no individual
> >> > contributing to open source in a leadership capacity for whom a
> >> > signable NDA cannot be crafted.
> >> >  
> >>
> >> No.
> >>
> >> I don't sign NDA's. I just don't do it.
> >>
> >> It's that simple.  
> >
> > But that's you're choice; it's not because legally you can't.
> >  
> >> It's actually worked pretty well. It started because I worked for a
> >> direct competitor to Intel, and couldn't sign an NDA for the really
> >> old f0 0f lockup issue.
> >>
> >> Not having an NDA back then turned out to be a good thing, because it
> >> made it a non-issue when leaks happened. So I started the policy that
> >> I never want to be in the position that I had to worry legally about
> >> being in the position of being under an NDA and knowing things
> >> outside of the leaks.
> >>
> >> Instead, I've had a gentleman's agreement with companies - nothing
> >> legally binding, but over the years people have come to realize that
> >> the leaks don't come from me.
> >>
> >> So I don't do NDA's. Maybe some Linux Foundation NDA agreement
> >> technically covers me, but at least with the Intel cases, Intel is
> >> actually aware of my non-NDA situation and is fine with it.  
> >
> > I'm fine with all of this as an argument.  If we believe that signing
> > NDAs would eventually lead to worse disasters because agreeing to them
> > now means corporations never change and never take our views into
> > account, then we should have the debate and make the decision for sound
> > policy reasons not because there's some spurious legal bar.
> >  
> 
> My NDA is through my company.  I would *love* to cancel it and set up
> a replacement arrangement through LF or a similar entity, or to just
> not replace it at all.  My company is not equipped for the kind of
> wrangling that would have helped during Meltdown and a couple of other
> situations, whereas anything reasonable set up for the purpose would
> work much better.

I guess this is the situation of most of all. My contract has a
NDA covering my work, so I'm legally bound to whatever NDA my
employer has to some other companies. I'm almost certain that 
none of those were designed to cover Open Source. 

So, I'm pretty sure that, if I had to deal with an embargoed
security issue that would require an NDA to get access to details,
I would need to spend a lot of time and effort talking with the
legal department to explain the needs and discuss about a set of
clauses that would work for both sides(with won't be trivial, 
as they usually don't usually deal with open source).

There's also a language barrier: some lawyers expect NDAs to be
under the Country's official language, in order for it to have
legal value on that Country.

All that process can take weeks to happen, as that would likely
envolve discussions between several parties. During that period of
time, I won't likely be able to access the data relevant to solve
the embargoed issue.

So, as much as I would prefer to live in a world that would 
work without any NDAs (and on a first world Country), reality 
takes place. Having something prepared in forehand would
significantly improve the process.

Granted, it is unlikely that I would have to deal with those issues
with the stuff I currently handle, as usually most serious security
threats are not at drivers, but I suspect that others would have 
to deal with similar issues.

Thanks,
Mauro

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Sat, Sep 08, 2018 at 08:54:26AM -0700, James Bottomley wrote:
> On Sat, 2018-09-08 at 17:32 +0200, Greg KH wrote:
> > On Sat, Sep 08, 2018 at 08:00:29AM -0700, James Bottomley wrote:
> > > On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote:
> > > > On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab
> > > > wrote:
> > > > > IMHO, the best would be to have a formal/legal way to handle
> > > > > it.
> > > > 
> > > > No, sorry, some of us are not allowed legally to sign NDAs for
> > > > stuff like this.
> > > 
> > > As a blanket statement this simply isn't true.
> > 
> > Um, I said "some of us".  Some of us can, some of us can not.  That's
> > a blanket statement that has to be true :)
> 
> OK, let me make it more specific: there exists no individual
> contributing to open source in a leadership capacity for whom a
> signable NDA cannot be crafted.

"can be crafted eventually" :)

There are language issues, corporate issues, and lots and lots of other
issues involved here, you know this.  Look at Mauro's situation for one
example.

Anyway, if the main goal here is to somehow have the LF provide some
sort of situation where we can invoke the old "3-way" NDA process to
handle security issues, then fine, let's propose that and see if the LF
wishes to do this.

But remember, this is only needed for the "crazy" issues, like Meltdown.
What we put together add-hoc for L1TF worked well, and what we do every
week in handling security issues sent to security@k.org works very well
also.  So well that no one really realizes what we do there :)

So again, if this is something that people strongly feel the LF should
handle, let the TAB know and they will be glad to work on it.

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Fri, Sep 07, 2018 at 03:30:32PM +0200, Jiri Kosina wrote:
> On Thu, 6 Sep 2018, Eduardo Valentin wrote:
> 
> > Should we add maybe a point here to discuss which kernels are to be 
> > considered for patching in these cases? All the stable branches? Only 
> > mainline? Obviously, either extreme cases can hurt people. Patching 
> > older kernels requires insane amount of work and patching only mainline 
> > leaves distros on limbo.
> 
> That'd be mostly question for the stable guys I guess. I am not sure how 
> often did they in the past have to say "sorry, the backport is horribly 
> complex, so we are not backporting the fix and we're keeping the bug 
> unfixed".
> 
> Greg, is this something that actually has been happening for real in the 
> past? Or would that absolutely break the expectations that stable tree 
> consumers have?

Yes, this is something that is happening today.

If you look, L1TF is not fully backported to 4.4.y, for anyone running
4.4.y as a host operating system.  The backport was just too horrible
and no one wanted to do it and test it as all of the major hosting
services have moved on to 4.9.y or better.

There are other examples of this, spectre fixes for arm32 are not in any
stable tree older than 4.18.y.  Same for other arches and kernel
versions.

I tried to write up "what kernel version to use" on my blog a few weeks
back to answer this type of question.  Basically, only "trust" the
latest LTS stable kernel for security issues to be able to use it to run
untrusted users.  When you start getting older kernels involved, nasty
problems like what Meltdown and the like are having to implement, it
just does not work.

So only "stay" with on old LTS kernel if your hardware requires you to
(i.e. the horrid SoC nightmare).  And even then, be careful about things
(sandboxes, selinux, etc.) and go yell at your SoC vendor for forcing
you into this nightmare of a problem.  If they do not hear from
companies, they will not change.

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Linus Torvalds]

On Sun, Sep 9, 2018 at 5:51 AM Greg KH <greg@kroah.com> wrote:
>
> But remember, this is only needed for the "crazy" issues, like Meltdown.
> What we put together add-hoc for L1TF worked well, and what we do every
> week in handling security issues sent to security@k.org works very well
> also.  So well that no one really realizes what we do there :)

Note that at some point, we should just say "f*ck it".

For hardware bugs, we should remember that *we* aren't the ones that
are in trouble. If a hardware company makes it too hard for us to work
with them, we should literally say "go the f*ck away" and stop talking
to them.

It's *their* problem, not ours.  If they only work with vendors unable
to talk to core maintainers, I guarantee that it will *remain* their
problem. I will happily tell the world that the hardware company
screwed up and didn't even help us try to fix things right.

Their lawyers and PR people can go screw themselves.

Seriously. People need to be aware that it's not us that should be
bending over backwards over hardware issues. If some hardware company
wants an NDA from me for their own screw-ups, I'll laugh in their
face, and then I'll tell journalists about how they actively made it
harder to fix their mess.

          Linus

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[James Bottomley]

On Sun, 2018-09-09 at 07:20 -0700, Linus Torvalds wrote:
> On Sun, Sep 9, 2018 at 5:51 AM Greg KH <greg@kroah.com> wrote:
> > 
> > But remember, this is only needed for the "crazy" issues, like
> > Meltdown. What we put together add-hoc for L1TF worked well, and
> > what we do every week in handling security issues sent to
> > security@k.org works very well also.  So well that no one really
> > realizes what we do there :)
> 
> Note that at some point, we should just say "f*ck it".
> 
> For hardware bugs, we should remember that *we* aren't the ones that
> are in trouble. If a hardware company makes it too hard for us to
> work with them, we should literally say "go the f*ck away" and stop
> talking to them.
> 
> It's *their* problem, not ours.  If they only work with vendors
> unable to talk to core maintainers, I guarantee that it will *remain*
> their problem. I will happily tell the world that the hardware
> company screwed up and didn't even help us try to fix things right.
> 
> Their lawyers and PR people can go screw themselves.
> 
> Seriously. People need to be aware that it's not us that should be
> bending over backwards over hardware issues. If some hardware company
> wants an NDA from me for their own screw-ups, I'll laugh in their
> face, and then I'll tell journalists about how they actively made it
> harder to fix their mess.

So it seems we have the two choices:

   1. Conform to industry norms for disclosures and find a way of bringing
      an NDA framework to Linux Security fix handling 
   2. Force industry to adopt new norms that actually work well with open
      source.

I think I already hear a majority for number 2.

However, to make 2 work we need to use every tool at our disposal to
push for change, including our PR relationships and, to be true to
that, we really should publish a critique of what went wrong with
spectre/meltdown and how it should have gone better.  That way we have
something to point to when someone asks what to do about the next
hardware side channel problem.  I'm sure lwn.net would be up for doing
something to help with this provided we give them access to the raw
material and maintainer interviews so they can present a coherent story
rather than a gripe fest (which is what we've mostly got in this
thread).

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Andy Lutomirski]

> On Sep 9, 2018, at 7:38 AM, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
>> On Sun, 2018-09-09 at 07:20 -0700, Linus Torvalds wrote:
>>> On Sun, Sep 9, 2018 at 5:51 AM Greg KH <greg@kroah.com> wrote:
>>> 
>>> But remember, this is only needed for the "crazy" issues, like
>>> Meltdown. What we put together add-hoc for L1TF worked well, and
>>> what we do every week in handling security issues sent to
>>> security@k.org works very well also.  So well that no one really
>>> realizes what we do there :)
>> 
>> Note that at some point, we should just say "f*ck it".
>> 
>> For hardware bugs, we should remember that *we* aren't the ones that
>> are in trouble. If a hardware company makes it too hard for us to
>> work with them, we should literally say "go the f*ck away" and stop
>> talking to them.
>> 
>> It's *their* problem, not ours.  If they only work with vendors
>> unable to talk to core maintainers, I guarantee that it will *remain*
>> their problem. I will happily tell the world that the hardware
>> company screwed up and didn't even help us try to fix things right.
>> 
>> Their lawyers and PR people can go screw themselves.
>> 
>> Seriously. People need to be aware that it's not us that should be
>> bending over backwards over hardware issues. If some hardware company
>> wants an NDA from me for their own screw-ups, I'll laugh in their
>> face, and then I'll tell journalists about how they actively made it
>> harder to fix their mess.
> 
> So it seems we have the two choices:
> 
>   1. Conform to industry norms for disclosures and find a way of bringing
>      an NDA framework to Linux Security fix handling 
>   2. Force industry to adopt new norms that actually work well with open
>      source.
> 

Or my proposal of 3: have a policy, get lawyers to agree to it, and make it barely be an NDA.  I don’t know how practical it is, but it could be a nice middle ground.

> I think I already hear a majority for number 2.
> 
> However, to make 2 work we need to use every tool at our disposal to
> push for change, including our PR relationships and, to be true to
> that, we really should publish a critique of what went wrong with
> spectre/meltdown and how it should have gone better.  That way we have
> something to point to when someone asks what to do about the next
> hardware side channel problem.  I'm sure lwn.net would be up for doing
> something to help with this provided we give them access to the raw
> material and maintainer interviews so they can present a coherent story
> rather than a gripe fest (which is what we've mostly got in this
> thread).
> 
> James
> 
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Theodore Y. Ts'o]

On Sun, Sep 09, 2018 at 07:51:09AM -0700, Andy Lutomirski wrote:
> > So it seems we have the two choices:
> > 
> >   1. Conform to industry norms for disclosures and find a way of bringing
> >      an NDA framework to Linux Security fix handling 
> >   2. Force industry to adopt new norms that actually work well with open
> >      source.
> > 
> 
> Or my proposal of 3: have a policy, get lawyers to agree to it, and
> make it barely be an NDA.  I don’t know how practical it is, but it
> could be a nice middle ground.

The only middle ground is "gentleman's agreement".  The main problem
any legal agreement is what are the teeth if someone violates the NDA
and breaks the embargo.  The reason why it will be very hard for some
third-party, like the LF, to sign any kind of NDA on behalf of
independent developers is that it puts the liability risk on the LF.
And the LF's lawyers aren't going to be comfortable with this.

We've been through this before with the TAB and getting all of the TAB
members under an NDA so we could talk about pre-standardized UEFI
proposals.  We looked at trying to get the LF to sign an NDA for the
TAB members who didn't work for companies which had an NDA with UEFI,
and it just didn't work.  Ultimately, what we did is we negotiated a
specific NDA just for me (where it would be my house on the line in
terms of an NDA violation), and I then had to get the Google's lawyers
to OK my signing it as a personal NDA.  The whole process took
**months**.

If we need to do this when we're in a hurry, by the time the lawyers
get over their "we must reduce our client's risk to zero" obsession,
the security problem will have leaked, and the embargo and the NDA
will be moot.

					- Ted

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On Sun, 2018-09-09 at 13:20 -0400, Theodore Y. Ts'o wrote:
> 
> The only middle ground is "gentleman's agreement".  The main problem
> any legal agreement is what are the teeth if someone violates the NDA
> and breaks the embargo.  The reason why it will be very hard for some
> third-party, like the LF, to sign any kind of NDA on behalf of
> independent developers is that it puts the liability risk on the LF.
> And the LF's lawyers aren't going to be comfortable with this.
> 
> We've been through this before with the TAB and getting all of the TAB
> members under an NDA so we could talk about pre-standardized UEFI
> proposals.  We looked at trying to get the LF to sign an NDA for the
> TAB members who didn't work for companies which had an NDA with UEFI,
> and it just didn't work.  Ultimately, what we did is we negotiated a
> specific NDA just for me (where it would be my house on the line in
> terms of an NDA violation), and I then had to get the Google's lawyers
> to OK my signing it as a personal NDA.  The whole process took
> **months**.

In practice there's not really a lot of difference between a "real" NDA
and a gentlepersons' agreement. Nobody's *really* going to lose their
house; all that's likely to happen if you screw up is that they won't
include you in the party next time round.

We *had* a breach before the Spectre/Meltdown embargo was supposed to
be over, and to my knowledge (and I *hope*) nothing actually happened
except a bit of tutting.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Andy Lutomirski]

> On Sep 9, 2018, at 10:20 AM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> 
> On Sun, Sep 09, 2018 at 07:51:09AM -0700, Andy Lutomirski wrote:
>>> So it seems we have the two choices:
>>> 
>>>  1. Conform to industry norms for disclosures and find a way of bringing
>>>     an NDA framework to Linux Security fix handling 
>>>  2. Force industry to adopt new norms that actually work well with open
>>>     source.
>>> 
>> 
>> Or my proposal of 3: have a policy, get lawyers to agree to it, and
>> make it barely be an NDA.  I don’t know how practical it is, but it
>> could be a nice middle ground.
> 
> The only middle ground is "gentleman's agreement".  The main problem
> any legal agreement is what are the teeth if someone violates the NDA
> and breaks the embargo.  The reason why it will be very hard for some
> third-party, like the LF, to sign any kind of NDA on behalf of
> independent developers is that it puts the liability risk on the LF.
> And the LF's lawyers aren't going to be comfortable with this.


What I want is the opposite of an NDA. I want a gentlemen’s agreement plus an explicit statement that the relevant people *may* talk about the issue among themselves despite any NDAs that might already exist. And that they may release patches when the embargo is up. And that the embargo has an end date, and that the developers may decline an extension.

The business people are well aware that even the strongest NDA isn’t worth a damn thing in these scenarios. If Meltdown had leaked in September, then Intel would have had egg all over its face and no amount of lawsuits or injunctions would have helped. In fact, it would have been worse than nothing because now the news would say “Intel failed to permit vendors to fix it *and* the embargo got blown, and rather than doing something about it, they’re suing people.”

I personally would have liked such an arrangement for Meltdown.

So maybe a business case could be made to vendors that they should sign the thing.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Theodore Y. Ts'o]

On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote:
> 
> What I want is the opposite of an NDA. I want a gentlemen’s
> agreement plus an explicit statement that the relevant people *may*
> talk about the issue among themselves despite any NDAs that might
> already exist. And that they may release patches when the embargo is
> up. And that the embargo has an end date, and that the developers
> may decline an extension.

So what you're talking about is some kind of "Memo of Understanding"
that has no talk about "if this leaks it will Intel will suffer
millons and billons and zillons of dollars and Intel well sue you
until your assets are a smoking crater in the ground"?

If there are no consequences to violating the Gentleman's agreement
(other than not being included the next time *when* another CPU
vulnerability comes up), then nothing really needs to be signed, since
it has no legal impact.

I'd certainly support such a thing, but in my view it's really no
different from Linus's #2:

   2. Force industry to adopt new norms that actually work well with open
      source.

If the MOU with no teeth is enough to save the lawyer's face, that
would be great.

					- Ted

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Andy Lutomirski]

> On Sep 9, 2018, at 11:56 AM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> 
>> On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote:
>> 
>> What I want is the opposite of an NDA. I want a gentlemen’s
>> agreement plus an explicit statement that the relevant people *may*
>> talk about the issue among themselves despite any NDAs that might
>> already exist. And that they may release patches when the embargo is
>> up. And that the embargo has an end date, and that the developers
>> may decline an extension.
> 
> So what you're talking about is some kind of "Memo of Understanding"
> that has no talk about "if this leaks it will Intel will suffer
> millons and billons and zillons of dollars and Intel well sue you
> until your assets are a smoking crater in the ground"?

Yes

> 
> If there are no consequences to violating the Gentleman's agreement
> (other than not being included the next time *when* another CPU
> vulnerability comes up), then nothing really needs to be signed, since
> it has no legal impact.

Here I disagree. The consequence to *Intel* for signing needs to be clear. If I’m included, and Intel thinks I leaked it or their attorneys get overzealous and complain that I talked to someone at SUSE or whatever or that I *gasp* published a patch on the day the embargo ended and they sue *me* for zillions under my preexisting, then I want to point to this agreement and say “no, and by suing me you are in breach of this contract”.

> 
> I'd certainly support such a thing, but in my view it's really no
> different from Linus's #2:
> 
>   2. Force industry to adopt new norms that actually work well with open
>      source.
> If the MOU with no teeth is enough to save the lawyer's face, that
> would be great.
> 

That too. 

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

On Sun, 9 Sep 2018, Linus Torvalds wrote:

> > But remember, this is only needed for the "crazy" issues, like Meltdown.
> > What we put together add-hoc for L1TF worked well, and what we do every
> > week in handling security issues sent to security@k.org works very well
> > also.  So well that no one really realizes what we do there :)
> 
> Note that at some point, we should just say "f*ck it".

I absolutely agree (even more for stable tree backports).

One thing that is absolutely necessary then though (especially if this is 
just about not doing crazy backport to -stable), to make it absolutely 
clear to the stable downstreams/consumers, that this is the case.

Rationale: turns out it might be even illegal in some countries to release 
software with known security issue for which the fix exists (yeah, well, 
whatever that means). So the downstreams really should be made aware, so 
that we don't put them into an uncomfortable situation and they could 
adapt somehow.

(on a first sight, this might be like buying into what grsecurity folk(s) 
have been asking for for ages -- that is annotating ideally each and every 
commit with its security implications -- but that's not the case here; 
it's quite the opposite: explicitly stating that certain security 
fix/backport is *not* happening).

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

On Sun, 9 Sep 2018, Greg KH wrote:

> Yes, this is something that is happening today.
> 
> If you look, L1TF is not fully backported to 4.4.y, for anyone running
> 4.4.y as a host operating system.  The backport was just too horrible
> and no one wanted to do it and test it as all of the major hosting
> services have moved on to 4.9.y or better.

Unrelated sidenote: we have the whole thing backported to SUSE 4.4 kernel, 
so it can be cherry-picked from there if needed.

> There are other examples of this, spectre fixes for arm32 are not in any
> stable tree older than 4.18.y.  Same for other arches and kernel
> versions.
> 
> I tried to write up "what kernel version to use" on my blog a few weeks
> back to answer this type of question.  Basically, only "trust" the
> latest LTS stable kernel for security issues to be able to use it to run
> untrusted users.  When you start getting older kernels involved, nasty
> problems like what Meltdown and the like are having to implement, it
> just does not work.

OK, so as long as this message is completely clear to the stable tree 
consumers (see my other mail about potential legal implications for the 
downstream consumers in case they are not aware of this), then all is 
fine.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote:

>    2. Force industry to adopt new norms that actually work well with open
>       source.

The big quesiton mark here obviously is -- how?

I would hope that there are other ways than having the whole thing once 
fail completely, in order to provide enough of a shock effect.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[James Bottomley]

On Sun, 2018-09-09 at 22:20 +0200, Jiri Kosina wrote:
> On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote:
> 
> >    2. Force industry to adopt new norms that actually work well
> > with open source.
> 
> The big quesiton mark here obviously is -- how?
> 
> I would hope that there are other ways than having the whole thing
> once fail completely, in order to provide enough of a shock effect.

Well, as I said in the original email apart from peer pressure, and
perhaps some institutional help from the various open source
foundations, all we really have at our disposal is the PR factor: if we
produce a detailed history and critique of the failures plus a set of
suggestions for improvements we have at least begun the conversation.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Eduardo Valentin]

On Sun, Sep 09, 2018 at 09:48:58PM +0200, Jiri Kosina wrote:
> On Sun, 9 Sep 2018, Greg KH wrote:
> 
> > Yes, this is something that is happening today.
> > 
> > If you look, L1TF is not fully backported to 4.4.y, for anyone running
> > 4.4.y as a host operating system.  The backport was just too horrible
> > and no one wanted to do it and test it as all of the major hosting
> > services have moved on to 4.9.y or better.
> 
> Unrelated sidenote: we have the whole thing backported to SUSE 4.4 kernel, 
> so it can be cherry-picked from there if needed.
> 
> > There are other examples of this, spectre fixes for arm32 are not in any
> > stable tree older than 4.18.y.  Same for other arches and kernel
> > versions.
> > 
> > I tried to write up "what kernel version to use" on my blog a few weeks
> > back to answer this type of question.  Basically, only "trust" the
> > latest LTS stable kernel for security issues to be able to use it to run
> > untrusted users.  When you start getting older kernels involved, nasty
> > problems like what Meltdown and the like are having to implement, it
> > just does not work.
> 
> OK, so as long as this message is completely clear to the stable tree 
> consumers (see my other mail about potential legal implications for the 
> downstream consumers in case they are not aware of this), then all is 
> fine.

If I got this right, the lastest LTS gets what is closest to upstream,
everything else gets "best effort" backport, I assume.

> 
> Thanks,
> 
> -- 
> Jiri Kosina
> SUSE Labs
> 

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Eduardo Valentin]

On Sun, Sep 09, 2018 at 02:55:54PM +0200, Greg KH wrote:
> On Fri, Sep 07, 2018 at 03:30:32PM +0200, Jiri Kosina wrote:
> > On Thu, 6 Sep 2018, Eduardo Valentin wrote:
> > 
> > > Should we add maybe a point here to discuss which kernels are to be 
> > > considered for patching in these cases? All the stable branches? Only 
> > > mainline? Obviously, either extreme cases can hurt people. Patching 
> > > older kernels requires insane amount of work and patching only mainline 
> > > leaves distros on limbo.
> > 
> > That'd be mostly question for the stable guys I guess. I am not sure how 
> > often did they in the past have to say "sorry, the backport is horribly 
> > complex, so we are not backporting the fix and we're keeping the bug 
> > unfixed".
> > 
> > Greg, is this something that actually has been happening for real in the 
> > past? Or would that absolutely break the expectations that stable tree 
> > consumers have?
> 
> Yes, this is something that is happening today.
> 
> If you look, L1TF is not fully backported to 4.4.y, for anyone running
> 4.4.y as a host operating system.  The backport was just too horrible
> and no one wanted to do it and test it as all of the major hosting
> services have moved on to 4.9.y or better.
> 
> There are other examples of this, spectre fixes for arm32 are not in any
> stable tree older than 4.18.y.  Same for other arches and kernel
> versions.


Another example of divergence is meltdown itself, which made in 4.14
(Latest LTS) pretty close to what went into mainline, but anything older
than 4.14 got the core of meltdown implementation, but not all the
optimization in the x86 entry code, which was horrible to backport
entirely.

> 
> I tried to write up "what kernel version to use" on my blog a few weeks
> back to answer this type of question.  Basically, only "trust" the
> latest LTS stable kernel for security issues to be able to use it to run
> untrusted users.  When you start getting older kernels involved, nasty
> problems like what Meltdown and the like are having to implement, it
> just does not work.
> 
> So only "stay" with on old LTS kernel if your hardware requires you to
> (i.e. the horrid SoC nightmare).  And even then, be careful about things
> (sandboxes, selinux, etc.) and go yell at your SoC vendor for forcing
> you into this nightmare of a problem.  If they do not hear from
> companies, they will not change.

For these horrendous cases, should the community simply not patch the 
oldest kernels? Patching only the latest LTS may be a very strong
reason for everyone on older to simply go ahead and upgrade their kernels.

> 
> thanks,
> 
> greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 2926 bytes --]

On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote:
> On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote:
> > 
> > What I want is the opposite of an NDA. I want a gentlemen’s
> > agreement plus an explicit statement that the relevant people *may*
> > talk about the issue among themselves despite any NDAs that might
> > already exist. And that they may release patches when the embargo is
> > up. And that the embargo has an end date, and that the developers
> > may decline an extension.
> 
> So what you're talking about is some kind of "Memo of Understanding"
> that has no talk about "if this leaks it will Intel will suffer
> millons and billons and zillons of dollars and Intel well sue you
> until your assets are a smoking crater in the ground"?
> 
> If there are no consequences to violating the Gentleman's agreement
> (other than not being included the next time *when* another CPU
> vulnerability comes up), then nothing really needs to be signed, since
> it has no legal impact.

Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster (which
was completely FUBARed by Intel), having something like this in place could
have certainly solved the main gap which we had. We were able to
communicate freely between the informed parties and their allowed to know
kernel developers, even accross vendors. But there was no simple way to
bring in anybody else. It tooks us almost 2 months to get GregKH on board,
but there was no way to talk to e.g. the BPF folks in time.

I think this needs to have some formal setup. The way disclosure to
companies work is through coordinators, who then disclose it internaly to
the relevant people.

We should provide something similar, i.e. an embargo coordination group,
which coordinates the issue with the disclosing party. And yes, this only
can be based on a general Memo of Understanding, as there is no way to make
that whole NDA mess work when the group needs to bring in individual
developers.

Having something formal and halfways familiar in place is definitely
something we need before we are starting to communicate and negotiate that
through all channels.

What I came up with so far is:

 - work out a Memo of Understanding
   
 - appoint an initial group of embargo coordinators, ideally people who
   have already an established trust relationship in the industry.

 - come up with a clear and well defined set of rules what this embargo
   group is doing and what not.

   It's sole purpose is to handle and coordinate the kind of embargo
   issues, which have a long preparation time, need coordination with other
   OSes etc, i.e. the Meltdown/Spectre/L1TF scenarios.

   It won't deal with NDAs and has to be free to disclose to individual
   developers based on trust under the MOU.

Creating this kind of formal entity is probably the closest thing to the
established inter corporate embargo handling which we can provide.

Thanks,

	tglx

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

On Sun, Sep 09, 2018 at 02:55:54PM +0200, Greg KH wrote:

> There are other examples of this, spectre fixes for arm32 are not in any
> stable tree older than 4.18.y.  Same for other arches and kernel
> versions.

Not really relevant to your point but just for clarity those should make
it back to the LTS versions at some point for arm32, they've been held
up behind some of the other entertaining backports.  Which is itself an
issue but there we are.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[James Bottomley]

On Mon, 2018-09-10 at 11:25 +0200, Thomas Gleixner wrote:
> On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote:
> > On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote:
> > > 
> > > What I want is the opposite of an NDA. I want a gentlemen’s
> > > agreement plus an explicit statement that the relevant people
> > > *may* talk about the issue among themselves despite any NDAs that
> > > might already exist. And that they may release patches when the
> > > embargo is up. And that the embargo has an end date, and that the
> > > developers may decline an extension.
> > 
> > So what you're talking about is some kind of "Memo of
> > Understanding" that has no talk about "if this leaks it will Intel
> > will suffer millons and billons and zillons of dollars and Intel
> > well sue you until your assets are a smoking crater in the ground"?
> > 
> > If there are no consequences to violating the Gentleman's agreement
> > (other than not being included the next time *when* another CPU
> > vulnerability comes up), then nothing really needs to be signed,
> > since it has no legal impact.
> 
> Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster
> (which was completely FUBARed by Intel), having something like this
> in place could have certainly solved the main gap which we had. We
> were able to communicate freely between the informed parties and
> their allowed to know kernel developers, even accross vendors. But
> there was no simple way to bring in anybody else. It tooks us almost
> 2 months to get GregKH on board, but there was no way to talk to e.g.
> the BPF folks in time.
> 
> I think this needs to have some formal setup. The way disclosure to
> companies work is through coordinators, who then disclose it
> internaly to the relevant people.
> 
> We should provide something similar, i.e. an embargo coordination
> group, which coordinates the issue with the disclosing party. And
> yes, this only can be based on a general Memo of Understanding, as
> there is no way to make that whole NDA mess work when the group needs
> to bring in individual developers.

The good thing about doing this is we can set the rules for onward
disclosure from the embargo co-ordination group.  We could probably get
away with something that said (co-ordinate with required linux kernel
subsystem maintainers on a need to know basis) i.e. under our rules we
could disclose to a maintainer if they needed to know without an NDA.

> Having something formal and halfways familiar in place is definitely
> something we need before we are starting to communicate and negotiate
> that through all channels.
> 
> What I came up with so far is:
> 
>  - work out a Memo of Understanding
>    
>  - appoint an initial group of embargo coordinators, ideally people
> who    have already an established trust relationship in the
> industry.
> 
>  - come up with a clear and well defined set of rules what this
> embargo group is doing and what not.

This is the key for better co-ordination.  One of the rules should be
"take responsibility for determining who needs to know in the Linux
Kernel maintainer community and communicating relevant information to
them on a strict need to know basis".

It can probably be better phrased and we'd need a lawyer to look it
over because this is the point at which the NDA gives way to a
"gentleman's agreement".

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Dave Hansen]

On 09/08/2018 12:49 PM, Linus Torvalds wrote:
> So I don't do NDA's. Maybe some Linux Foundation NDA agreement
> technically covers me, but at least with the Intel cases, Intel is
> actually aware of my non-NDA situation and is fine with it.

My *personal* observation on the NDAs:

Companies don't actually care about the NDA being an NDA per se.  They
really only want to feel like they are in control of the information.
They get that warm and fuzzy feeling from NDAs for normal
company-to-company interactions, which makes NDAs the go-to tool when
these security things pop up.

We (the community) are slowly showing the NDA-loving folks that <gasp>
they are not the _only_ tool available.  But, it's going to take time to
change the mindset.

I *do* wish that companies like Intel who are actively doing these
non-NDA things would find some way to share their methods.  Maybe the LF
can help here by providing a semi-anonymous way for folks to share what
has worked.  Or, maybe folks like Intel need to just to it ourselves.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Eduardo Valentin]

Hello Geert,

On Fri, Sep 07, 2018 at 10:21:15AM +0200, Geert Uytterhoeven wrote:
> Hi Eduardo,
> 
> On Fri, Sep 7, 2018 at 12:55 AM Eduardo Valentin <edubezval@gmail.com> wrote:
> > On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote:
> > Honestly, the fact that somehow the community managed to make this to
> > stable (and eventually to distros) is really good. Imagine for a second
> > a world in which these made only mainline and no stable branch..
> 
> It could have been the disaster needed to trigger a paradigm shift in the
> software industry?

Well yeah. Just to clarify, what I meant is that the community did a
really great job handling the event, despite the fact that we could have
dont better. But I still think pushing everyone to make sure they can
move to newer kernels is best thing to do. How achieve it is that is a
different discussion.

> 
> For a fully isolated/controlled system old stable may make sense, but in
> the post-Internet connected world, where the whole world population can
> peek at your front door, you'd better be prepared to upgrade any time.

Well yeah, I would do the same. But then there is thing called vendor
trees. That makes developers life even more horrible. As you hear in
this thread, even top noch developer would avoid backporting stuff to
too old kernels, and latest LTS is probably the oldest one can trust.
But vendor trees, which is the stuff that really ends up running on
those cameras running Linux at your front door, are usually old and
have bunch of code that community is not aware (and really should not be). 
And backporting stuff to those kernels is even a worse nighmare than
backporting to older stables.

> Despite backporting, you'll always be vulnerable to some security issues
> that were fixed in mainline.
> 
> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> -- 
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Jiri Kosina]

On Mon, 10 Sep 2018, Thomas Gleixner wrote:

> Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster (which
> was completely FUBARed by Intel), having something like this in place could
> have certainly solved the main gap which we had. We were able to
> communicate freely between the informed parties and their allowed to know
> kernel developers, even accross vendors. 

Agreed, this worked pretty well this time.

> But there was no simple way to bring in anybody else. It tooks us almost 
> 2 months to get GregKH on board, but there was no way to talk to e.g. 
> the BPF folks in time.

But this was what has caused real pain indeed. Do we know / can it be 
publicly said what exactly was the issue in those cases? Was it perhaps 
that those people were not employed by a company the disclosing party had 
a NDA in place already (like it probably had with all the involved 
vendors, etc)?

Thanks,

-- 
Jiri Kosina

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Mon, Sep 10, 2018 at 04:26:53PM -0700, Eduardo Valentin wrote:
> Hello Geert,
> 
> On Fri, Sep 07, 2018 at 10:21:15AM +0200, Geert Uytterhoeven wrote:
> > Hi Eduardo,
> > 
> > On Fri, Sep 7, 2018 at 12:55 AM Eduardo Valentin <edubezval@gmail.com> wrote:
> > > On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote:
> > > Honestly, the fact that somehow the community managed to make this to
> > > stable (and eventually to distros) is really good. Imagine for a second
> > > a world in which these made only mainline and no stable branch..
> > 
> > It could have been the disaster needed to trigger a paradigm shift in the
> > software industry?
> 
> Well yeah. Just to clarify, what I meant is that the community did a
> really great job handling the event, despite the fact that we could have
> dont better.

What do you feel we could have done "better" given the constraints
placed on us?

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Mon, Sep 10, 2018 at 03:59:26PM -0700, Dave Hansen wrote:
> I *do* wish that companies like Intel who are actively doing these
> non-NDA things would find some way to share their methods.  Maybe the LF
> can help here by providing a semi-anonymous way for folks to share what
> has worked.  Or, maybe folks like Intel need to just to it ourselves.

There is a framework where Intel can share these types of methods
already.  The LF has at least one meeting a year for all of the
Linux-related company's legal people to come together and share
information and talk in a way that all can benefit, using the Chatham
House rules.

I'm sure a presentation from Intel to that group would always be most
welcome.

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Thomas Gleixner]

On Tue, 11 Sep 2018, Jiri Kosina wrote:
> On Mon, 10 Sep 2018, Thomas Gleixner wrote:
> 
> > Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster (which
> > was completely FUBARed by Intel), having something like this in place could
> > have certainly solved the main gap which we had. We were able to
> > communicate freely between the informed parties and their allowed to know
> > kernel developers, even accross vendors. 
> 
> Agreed, this worked pretty well this time.
> 
> > But there was no simple way to bring in anybody else. It tooks us almost 
> > 2 months to get GregKH on board, but there was no way to talk to e.g. 
> > the BPF folks in time.
> 
> But this was what has caused real pain indeed. Do we know / can it be 
> publicly said what exactly was the issue in those cases? Was it perhaps 
> that those people were not employed by a company the disclosing party had 
> a NDA in place already (like it probably had with all the involved 
> vendors, etc)?

Greg is not employed by one of the companies to which the issue was
disclosed and had no NDA in place. AFAIK he's not doing the NDA mess
either, so it took time to sort it out.

For others, who were not employed by one of the involved parties and were
not covered by some magic NDA construct, we just gave up because at the
point we realized that we need their expertise it was way too late to wait
a couple of month/weeks to get that sorted.

That's where I see the embargo coordination to have it's place. If the
people working on the embargoed issue need the expertise of somebody who is
not covered in one way or the other, then the embargo team can disclose it
due to technical necessarity and based on trust.

And I can see that working, because if someone gets pulled in and spills
the beans then he has not only violated the trust put into him for this
particular embargo issue, he also killed his reputation and trust
relationships in the community as a whole.

I doubt that any coordinator in a company - despite the whole NDA/work
contract/whatever stuff in place - will disclose a sensitive embargo issue
to an engineer whom he doesn't trust fully.

Thanks,

	tglx

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Dave Hansen]

On 09/11/2018 01:45 AM, Greg KH wrote:
> What do you feel we could have done "better" given the constraints
> placed on us?

Hindsight is 20/20.  But, here are a few things I wish we would have had
in place a year or two ago.  These are utterly _minor_ nits in the grand
scheme of things.  Intel had the most room for improvement here, not the
community.  But, here's what the community could have done better:

1. Have a documented procedure for submitting issues that the submitter
   perceives can not go to security@.  Folks are always going to think
   they are a special snowflake.  This will get them talking to
   *someone* at least.
2. Document that stable updates require stable maintainers to be
   involved.  If you want fixes in mainline, tell Linus.  If you want
   stable fixes, tell the stable maintainers.  Also document the time
   required to integrate a fix, even if it is a worst-case estimate.
   (The distros who consume stable can help here, too)

Why?  From what I've seen, the folks controlling the embargo respect
*processes*.  A written-down document is a process that's hard to argue
with and represents the weight of the community.  But if I tell them,
it's just "one person's opinion".

Giving timelines is also very important.  Folks spend a lot of time
counting months and weeks back on the calendar from a disclosure date.
The timeline gives them a discrete date to *do* something.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Tue, Sep 11, 2018 at 10:10:37AM -0700, Dave Hansen wrote:
> On 09/11/2018 01:45 AM, Greg KH wrote:
> > What do you feel we could have done "better" given the constraints
> > placed on us?
> 
> Hindsight is 20/20.  But, here are a few things I wish we would have had
> in place a year or two ago.  These are utterly _minor_ nits in the grand
> scheme of things.  Intel had the most room for improvement here, not the
> community.  But, here's what the community could have done better:
> 
> 1. Have a documented procedure for submitting issues that the submitter
>    perceives can not go to security@.  Folks are always going to think
>    they are a special snowflake.  This will get them talking to
>    *someone* at least.

We have Documentation/admin-guide/security-bugs.rst.  If you feel it is
lacking in some way that would help out here, please submit patches.

I know some people did submit patches for this after the Meltdown mess
happened to hopefully clear some things up.  Oh wait, it was you who did
that, why am I even saying this then? :)

Anyway, Willy posted a patch somewhere to also update it, but that patch
seems to have gotten dropped, we should poke him to resend it.

> 2. Document that stable updates require stable maintainers to be
>    involved.  If you want fixes in mainline, tell Linus.  If you want
>    stable fixes, tell the stable maintainers.  Also document the time
>    required to integrate a fix, even if it is a worst-case estimate.
>    (The distros who consume stable can help here, too)

Note, I don't always _have_ to be involved.  I usually don't want to be
involved.  But if you expect me to start taking random non-obvious
patches into a stable kernel, you had better get me involved, otherwise
you will start to get the "WTF" emails from me.  That's just common
sense, right?  If not, how/where do I have to document this?

Also, for some people like me, perhaps you might want to get us involved
because we happen to know a bit about security and bugfixing given we
have been doing it for many decades now (not just me, most of the
security@ people are included in that category, hence them being on that
team).  Why _wouldn't_ you want them helping to fix your problem?

> Why?  From what I've seen, the folks controlling the embargo respect
> *processes*.  A written-down document is a process that's hard to argue
> with and represents the weight of the community.  But if I tell them,
> it's just "one person's opinion".

Great, the security process page can always need help, like you
provided.  Hopefully that looks sufficient to you now?

> Giving timelines is also very important.  Folks spend a lot of time
> counting months and weeks back on the calendar from a disclosure date.
> The timeline gives them a discrete date to *do* something.

Giving timelines to whom?  Us getting a timeline or us giving a timeline
to others?  It's hard to "predict" how long some things will take to be
fixed (obviously, we once took 6 months to fix a nasty bug that some
people thought it was already resolved as they provided a proposed fix.)

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Thomas Gleixner]

On Tue, 11 Sep 2018, Dave Hansen wrote:
>
> Giving timelines is also very important.  Folks spend a lot of time
> counting months and weeks back on the calendar from a disclosure date.
> The timeline gives them a discrete date to *do* something.

Giving a timeline whatfor? How long it takes to fix something? We need to
know about the issue first in order to do so.

So the simple answer here is ASAP and not when some disclosure manager
thinks it's about time. I rather have the fix simmering in my hidden
repository for a month or two than having to rush things toward the
disclosure date or when the embargo breaks early.

But then if we know what it is, it might be trivial to give an ETA and it
might be complete guess work for a while until we wrapped our brains around
it.

Thanks,

	tglx

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Balbir Singh]

On Thu, Sep 06, 2018 at 03:55:33PM -0700, Eduardo Valentin wrote:
> Hey,
> 
> On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote:
> > I believe we have reasonably well-established process for handling 
> > security issues that are a matter of a single, reasonably self-contained 
> > fixup.
> > 
> > Past experiences with Meltdown, Spectre and L1TF have shown that we're not 
> > really ready to handle that in a reasonably sane way yet.
> > 
> > Yeah, at the end of the day we managed to have the fixes propagated in 
> > time to Linus' tree, to stable, and to distros as well, but it was 
> > completely out of anything regular, and definitely had permanent damaging 
> > effects (I'd say both in personal and business aspects for almost 
> > everybody who had to participate).
> > 
> > I'd believe that everybody involved would agree that this didn't work 
> > really well, and if it potentially would have to happen again (and we 
> > already went through it at least twice this year), it would not be 
> > sustainable.
> > 
> > I am not completely sure what we could do to improve this, especially with 
> > our kernel community hats on -- I am pretty sure a lot is happening on the 
> > corporate level between individual "corporate stakeholders". Ideas I think 
> > would be worth discussing:
> > 
> > - how to adapt our processess to be able to deal with such situations 
> >   better should they happen in the future again. So far all our 
> >   longer-term development has been concentrated around LKML (and other 
> >   MLs) and the existing maintainership communities / structures, but the 
> >   embargos for new big features don't really fit into this
> > 
> > - how to make sure that proper pressure is applied on the companies that 
> >   are handling embargoes irresponsibly wrt. linux/opensource development 
> >   (well, even some proprietary vendors were rather unhappy with those 
> >   events) from us as the linux kernel community
> >   + and perhaps even more importantly, what exactly we should be pressing 
> >     for
> 
> Should we add maybe a point here to discuss which kernels are to be
> considered for patching in these cases? All the stable branches? Only
> mainline? Obviously, either extreme cases can hurt people. Patching
> older kernels requires insane amount of work and patching only mainline
> leaves distros on limbo.
>

I don't think we can decide, may be the right thing to do is to consider where
the pressure is coming from

1. Security researcher's wanting to set aggressive embargo schedules?
2. Are distros supporting older kernels for too long?

The spectre/meltdown affected all architectures/variants of CPUs and
so the overhead was massive and I don't think that will change depending
the layer in which a security vulnerability is discovered.
 
> Honestly, the fact that somehow the community managed to make this to
> stable (and eventually to distros) is really good. Imagine for a second
> a world in which these made only mainline and no stable branch..
> 

Half the world would have bugs in their patches :)

> In any case, maybe the community should consider what really is going to
> be effectively patched. That may be an extra push for distros to upgrade
> older kernels as well.
> 
>

Balbir Singh. 

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues

[Greg KH]

On Sun, Sep 09, 2018 at 09:04:57PM -0700, Eduardo Valentin wrote:
> On Sun, Sep 09, 2018 at 09:48:58PM +0200, Jiri Kosina wrote:
> > On Sun, 9 Sep 2018, Greg KH wrote:
> > 
> > > Yes, this is something that is happening today.
> > > 
> > > If you look, L1TF is not fully backported to 4.4.y, for anyone running
> > > 4.4.y as a host operating system.  The backport was just too horrible
> > > and no one wanted to do it and test it as all of the major hosting
> > > services have moved on to 4.9.y or better.
> > 
> > Unrelated sidenote: we have the whole thing backported to SUSE 4.4 kernel, 
> > so it can be cherry-picked from there if needed.
> > 
> > > There are other examples of this, spectre fixes for arm32 are not in any
> > > stable tree older than 4.18.y.  Same for other arches and kernel
> > > versions.
> > > 
> > > I tried to write up "what kernel version to use" on my blog a few weeks
> > > back to answer this type of question.  Basically, only "trust" the
> > > latest LTS stable kernel for security issues to be able to use it to run
> > > untrusted users.  When you start getting older kernels involved, nasty
> > > problems like what Meltdown and the like are having to implement, it
> > > just does not work.
> > 
> > OK, so as long as this message is completely clear to the stable tree 
> > consumers (see my other mail about potential legal implications for the 
> > downstream consumers in case they are not aware of this), then all is 
> > fine.
> 
> If I got this right, the lastest LTS gets what is closest to upstream,
> everything else gets "best effort" backport, I assume.

That is correct.

thanks,

greg k-h