* [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues @ 2018-09-06 19:18 Jiri Kosina 2018-09-06 20:56 ` Linus Torvalds ` (2 more replies) 0 siblings, 3 replies; 54+ messages in thread From: Jiri Kosina @ 2018-09-06 19:18 UTC (permalink / raw) To: ksummit-discuss I believe we have reasonably well-established process for handling security issues that are a matter of a single, reasonably self-contained fixup. Past experiences with Meltdown, Spectre and L1TF have shown that we're not really ready to handle that in a reasonably sane way yet. Yeah, at the end of the day we managed to have the fixes propagated in time to Linus' tree, to stable, and to distros as well, but it was completely out of anything regular, and definitely had permanent damaging effects (I'd say both in personal and business aspects for almost everybody who had to participate). I'd believe that everybody involved would agree that this didn't work really well, and if it potentially would have to happen again (and we already went through it at least twice this year), it would not be sustainable. I am not completely sure what we could do to improve this, especially with our kernel community hats on -- I am pretty sure a lot is happening on the corporate level between individual "corporate stakeholders". Ideas I think would be worth discussing: - how to adapt our processess to be able to deal with such situations better should they happen in the future again. So far all our longer-term development has been concentrated around LKML (and other MLs) and the existing maintainership communities / structures, but the embargos for new big features don't really fit into this - how to make sure that proper pressure is applied on the companies that are handling embargoes irresponsibly wrt. linux/opensource development (well, even some proprietary vendors were rather unhappy with those events) from us as the linux kernel community + and perhaps even more importantly, what exactly we should be pressing for Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 19:18 [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues Jiri Kosina @ 2018-09-06 20:56 ` Linus Torvalds 2018-09-06 21:14 ` Jiri Kosina ` (2 more replies) 2018-09-06 22:55 ` Eduardo Valentin 2018-09-08 4:21 ` Andy Lutomirski 2 siblings, 3 replies; 54+ messages in thread From: Linus Torvalds @ 2018-09-06 20:56 UTC (permalink / raw) To: Jiri Kosina; +Cc: ksummit On Thu, Sep 6, 2018 at 12:18 PM Jiri Kosina <jikos@kernel.org> wrote: > > I am not completely sure what we could do to improve this, especially with > our kernel community hats on -- I am pretty sure a lot is happening on the > corporate level between individual "corporate stakeholders". One particular pain point this last time around were the stable backports, I feel. A lot of that was that the actual *fixes* were marked for stable, but quite often they were preceded by cleanups and other updates that didn't actually fix things directly, and that weren't in themselves explicitly marked for stable and didn't have a Fixes: tag, because they were prep-work. So we had _several_ nasty regressions in stable that never showed up in mainline, because there was some non-obvious dependency that didn't cause a merge conflict, but did cause a "this commit needed that other commit to work right". We should probably at least think about having a way to mark those. Something like a "for-stable-because-of-subsequent-patches" tag? Or just more eager use of the table cc? I often feel bad about adding "cc: stable" to preparatory patches that don't actually fix the bug, but I think it was bad this time around. Of course, I also hope that we're over the worst. Linus ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 20:56 ` Linus Torvalds @ 2018-09-06 21:14 ` Jiri Kosina 2018-09-06 22:51 ` Eduardo Valentin 2018-09-07 9:17 ` Jani Nikula 2018-09-07 14:43 ` David Woodhouse 2 siblings, 1 reply; 54+ messages in thread From: Jiri Kosina @ 2018-09-06 21:14 UTC (permalink / raw) To: Linus Torvalds; +Cc: ksummit On Thu, 6 Sep 2018, Linus Torvalds wrote: > > I am not completely sure what we could do to improve this, especially with > > our kernel community hats on -- I am pretty sure a lot is happening on the > > corporate level between individual "corporate stakeholders". > > One particular pain point this last time around were the stable > backports, I feel. > > A lot of that was that the actual *fixes* were marked for stable, but > quite often they were preceded by cleanups and other updates that > didn't actually fix things directly, and that weren't in themselves > explicitly marked for stable and didn't have a Fixes: tag, because > they were prep-work. > > So we had _several_ nasty regressions in stable that never showed up > in mainline, because there was some non-obvious dependency that didn't > cause a merge conflict, but did cause a "this commit needed that other > commit to work right". I fully agree that this is an issue for stable. On the other hand, I would be reasonably sure this has been equally painful issue for stable even before this particular disaster (and all the preceeding stable discussions on this very ML sort of do support that). > We should probably at least think about having a way to mark those. > Something like a "for-stable-because-of-subsequent-patches" tag? > > Or just more eager use of the table cc? I often feel bad about adding > "cc: stable" to preparatory patches that don't actually fix the bug, > but I think it was bad this time around. Maybe at least partial solution (or first step) to this would be to somehow make sure that "these patches form an actual patchset that belongs together and is in fact one single thing" information somehow gets preserved in maintainer's / your tree. It's sort-of achievable if everybody (not only the patchset producers, but also the consumers) would be very familiar with the idea of strictly topic git branches, but that's probably not realistic. I currently have no good idea how exactly this should be done technically, but certainly it's doable and would be of a tremendous help to downstream, older-codebase consumers of your tree. > Of course, I also hope that we're over the worst. Fully agreed. Also, I hope that world is flat :) Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 21:14 ` Jiri Kosina @ 2018-09-06 22:51 ` Eduardo Valentin 0 siblings, 0 replies; 54+ messages in thread From: Eduardo Valentin @ 2018-09-06 22:51 UTC (permalink / raw) To: Jiri Kosina; +Cc: ksummit Hello, On Thu, Sep 06, 2018 at 11:14:08PM +0200, Jiri Kosina wrote: > On Thu, 6 Sep 2018, Linus Torvalds wrote: > > > > I am not completely sure what we could do to improve this, especially with > > > our kernel community hats on -- I am pretty sure a lot is happening on the > > > corporate level between individual "corporate stakeholders". > > > > One particular pain point this last time around were the stable > > backports, I feel. > > Totally agree. > > A lot of that was that the actual *fixes* were marked for stable, but > > quite often they were preceded by cleanups and other updates that > > didn't actually fix things directly, and that weren't in themselves > > explicitly marked for stable and didn't have a Fixes: tag, because > > they were prep-work. > > > > So we had _several_ nasty regressions in stable that never showed up > > in mainline, because there was some non-obvious dependency that didn't > > cause a merge conflict, but did cause a "this commit needed that other > > commit to work right". > Yeah, I think the later case of hidden dependencies is nastier to deal with than the first case of long patchset with lots of rework. And I remember the meltdown spectre case required quite a lot of elbow grease from different involved developers to find those hidden patches, specially on backports to kernels older than 4.14. Obviously with help from more experienced x86 maintainers, eventually things were fixed. Now, one thing that really helped was the fact that 4.14.y backport was done by the x86 maintainers. However, it is not feasible to expect that happening for all stable branches, like it did not happen during the spectre/meltdown story. > I fully agree that this is an issue for stable. On the other hand, I would > be reasonably sure this has been equally painful issue for stable even > before this particular disaster (and all the preceeding stable discussions > on this very ML sort of do support that). > > > We should probably at least think about having a way to mark those. > > Something like a "for-stable-because-of-subsequent-patches" tag? > > > > Or just more eager use of the table cc? I often feel bad about adding > > "cc: stable" to preparatory patches that don't actually fix the bug, > > but I think it was bad this time around. > > Maybe at least partial solution (or first step) to this would be to > somehow make sure that "these patches form an actual patchset that belongs > together and is in fact one single thing" information somehow gets > preserved in maintainer's / your tree. > > It's sort-of achievable if everybody (not only the patchset producers, but > also the consumers) would be very familiar with the idea of strictly topic > git branches, but that's probably not realistic. > > I currently have no good idea how exactly this should be done technically, > but certainly it's doable and would be of a tremendous help to downstream, > older-codebase consumers of your tree. Well, keeping the topic branch for future reference can help yes. Now, the more problematic part is the dependencies needed due to changes that happened after the target backport kernel (say 4.9.y) and the reference backport / topic branch (say 4.15), but that were not necessarily part of the original patch set that either prepare the code or really fixes the issue finally. And that even brings the question if all of those should be brought to stable, or if some other version of the fix should be considered. > > > Of course, I also hope that we're over the worst. > > Fully agreed. Also, I hope that world is flat :) > hehe.. > Thanks, > > -- > Jiri Kosina > SUSE Labs > > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 20:56 ` Linus Torvalds 2018-09-06 21:14 ` Jiri Kosina @ 2018-09-07 9:17 ` Jani Nikula 2018-09-07 14:43 ` David Woodhouse 2 siblings, 0 replies; 54+ messages in thread From: Jani Nikula @ 2018-09-07 9:17 UTC (permalink / raw) To: Linus Torvalds, Jiri Kosina; +Cc: ksummit On Thu, 06 Sep 2018, Linus Torvalds <torvalds@linux-foundation.org> wrote: > A lot of that was that the actual *fixes* were marked for stable, but > quite often they were preceded by cleanups and other updates that > didn't actually fix things directly, and that weren't in themselves > explicitly marked for stable and didn't have a Fixes: tag, because > they were prep-work. There's always the struggle of whether to put fixes or cleanups first in the series. The former makes backports easier, the latter makes the actual fixes easier. And the prep work could be committed quickly while the fixes are still being debated. > So we had _several_ nasty regressions in stable that never showed up > in mainline, because there was some non-obvious dependency that didn't > cause a merge conflict, but did cause a "this commit needed that other > commit to work right". > > We should probably at least think about having a way to mark those. > Something like a "for-stable-because-of-subsequent-patches" tag? If you know the commit ids of the dependencies, i.e. they've been applied, per Documentation/process/stable-kernel-rules.rst you could add: Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched: Check for idle Cc: <stable@vger.kernel.org> # 3.3.x: 1b9508f: sched: Rate-limit newidle Cc: <stable@vger.kernel.org> # 3.3.x: fd21073: sched: Fix affinity logic Cc: <stable@vger.kernel.org> # 3.3.x listing the dependencies. I don't see that used much, though, so not sure defining something new could gain more traction. > Or just more eager use of the table cc? I often feel bad about adding > "cc: stable" to preparatory patches that don't actually fix the bug, > but I think it was bad this time around. If the fix with prep work is in a topic branch, one could add cc: stable to the merge commit. BR, Jani. -- Jani Nikula, Intel Open Source Graphics Center ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 20:56 ` Linus Torvalds 2018-09-06 21:14 ` Jiri Kosina 2018-09-07 9:17 ` Jani Nikula @ 2018-09-07 14:43 ` David Woodhouse 2 siblings, 0 replies; 54+ messages in thread From: David Woodhouse @ 2018-09-07 14:43 UTC (permalink / raw) To: Linus Torvalds, Jiri Kosina; +Cc: ksummit [-- Attachment #1: Type: text/plain, Size: 1354 bytes --] On Thu, 2018-09-06 at 13:56 -0700, Linus Torvalds wrote: > > A lot of that was that the actual *fixes* were marked for stable, but > quite often they were preceded by cleanups and other updates that > didn't actually fix things directly, and that weren't in themselves > explicitly marked for stable and didn't have a Fixes: tag, because > they were prep-work. > > So we had _several_ nasty regressions in stable that never showed up > in mainline, because there was some non-obvious dependency that didn't > cause a merge conflict, but did cause a "this commit needed that other > commit to work right". This time round, for L1TF I think we did relatively OK on the stable backports. The dependencies we were missing were stuff that was just randomly cleaned up in the intervening kernels. For example, the 'Simplify p[g4um]d_page() macros' commit (fd7e315988) start out as *purely* a cosmetic improvement to reduce duplication by introducing a pgd_pfn() macro. Some time later, completely unpredicted at the time of that commit, we came along and made pgd_pfn() do something different. It's *those* "dependencies" which tend to get missed, and they can't be predicted at the time of the original commit. I don't think we did so badly with the commits which were done as cleanups in preparation for the event itself. [-- Attachment #2: smime.p7s --] [-- Type: application/x-pkcs7-signature, Size: 5213 bytes --] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 19:18 [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues Jiri Kosina 2018-09-06 20:56 ` Linus Torvalds @ 2018-09-06 22:55 ` Eduardo Valentin 2018-09-07 8:21 ` Geert Uytterhoeven ` (2 more replies) 2018-09-08 4:21 ` Andy Lutomirski 2 siblings, 3 replies; 54+ messages in thread From: Eduardo Valentin @ 2018-09-06 22:55 UTC (permalink / raw) To: Jiri Kosina; +Cc: ksummit-discuss Hey, On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote: > I believe we have reasonably well-established process for handling > security issues that are a matter of a single, reasonably self-contained > fixup. > > Past experiences with Meltdown, Spectre and L1TF have shown that we're not > really ready to handle that in a reasonably sane way yet. > > Yeah, at the end of the day we managed to have the fixes propagated in > time to Linus' tree, to stable, and to distros as well, but it was > completely out of anything regular, and definitely had permanent damaging > effects (I'd say both in personal and business aspects for almost > everybody who had to participate). > > I'd believe that everybody involved would agree that this didn't work > really well, and if it potentially would have to happen again (and we > already went through it at least twice this year), it would not be > sustainable. > > I am not completely sure what we could do to improve this, especially with > our kernel community hats on -- I am pretty sure a lot is happening on the > corporate level between individual "corporate stakeholders". Ideas I think > would be worth discussing: > > - how to adapt our processess to be able to deal with such situations > better should they happen in the future again. So far all our > longer-term development has been concentrated around LKML (and other > MLs) and the existing maintainership communities / structures, but the > embargos for new big features don't really fit into this > > - how to make sure that proper pressure is applied on the companies that > are handling embargoes irresponsibly wrt. linux/opensource development > (well, even some proprietary vendors were rather unhappy with those > events) from us as the linux kernel community > + and perhaps even more importantly, what exactly we should be pressing > for Should we add maybe a point here to discuss which kernels are to be considered for patching in these cases? All the stable branches? Only mainline? Obviously, either extreme cases can hurt people. Patching older kernels requires insane amount of work and patching only mainline leaves distros on limbo. Honestly, the fact that somehow the community managed to make this to stable (and eventually to distros) is really good. Imagine for a second a world in which these made only mainline and no stable branch.. In any case, maybe the community should consider what really is going to be effectively patched. That may be an extra push for distros to upgrade older kernels as well. > > Thanks, > > -- > Jiri Kosina > SUSE Labs > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 22:55 ` Eduardo Valentin @ 2018-09-07 8:21 ` Geert Uytterhoeven 2018-09-10 23:26 ` Eduardo Valentin 2018-09-07 13:30 ` Jiri Kosina 2018-09-12 4:22 ` Balbir Singh 2 siblings, 1 reply; 54+ messages in thread From: Geert Uytterhoeven @ 2018-09-07 8:21 UTC (permalink / raw) To: Eduardo Valentin; +Cc: ksummit-discuss Hi Eduardo, On Fri, Sep 7, 2018 at 12:55 AM Eduardo Valentin <edubezval@gmail.com> wrote: > On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote: > Honestly, the fact that somehow the community managed to make this to > stable (and eventually to distros) is really good. Imagine for a second > a world in which these made only mainline and no stable branch.. It could have been the disaster needed to trigger a paradigm shift in the software industry? For a fully isolated/controlled system old stable may make sense, but in the post-Internet connected world, where the whole world population can peek at your front door, you'd better be prepared to upgrade any time. Despite backporting, you'll always be vulnerable to some security issues that were fixed in mainline. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-07 8:21 ` Geert Uytterhoeven @ 2018-09-10 23:26 ` Eduardo Valentin 2018-09-11 8:45 ` Greg KH 0 siblings, 1 reply; 54+ messages in thread From: Eduardo Valentin @ 2018-09-10 23:26 UTC (permalink / raw) To: Geert Uytterhoeven; +Cc: ksummit-discuss Hello Geert, On Fri, Sep 07, 2018 at 10:21:15AM +0200, Geert Uytterhoeven wrote: > Hi Eduardo, > > On Fri, Sep 7, 2018 at 12:55 AM Eduardo Valentin <edubezval@gmail.com> wrote: > > On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote: > > Honestly, the fact that somehow the community managed to make this to > > stable (and eventually to distros) is really good. Imagine for a second > > a world in which these made only mainline and no stable branch.. > > It could have been the disaster needed to trigger a paradigm shift in the > software industry? Well yeah. Just to clarify, what I meant is that the community did a really great job handling the event, despite the fact that we could have dont better. But I still think pushing everyone to make sure they can move to newer kernels is best thing to do. How achieve it is that is a different discussion. > > For a fully isolated/controlled system old stable may make sense, but in > the post-Internet connected world, where the whole world population can > peek at your front door, you'd better be prepared to upgrade any time. Well yeah, I would do the same. But then there is thing called vendor trees. That makes developers life even more horrible. As you hear in this thread, even top noch developer would avoid backporting stuff to too old kernels, and latest LTS is probably the oldest one can trust. But vendor trees, which is the stuff that really ends up running on those cameras running Linux at your front door, are usually old and have bunch of code that community is not aware (and really should not be). And backporting stuff to those kernels is even a worse nighmare than backporting to older stables. > Despite backporting, you'll always be vulnerable to some security issues > that were fixed in mainline. > > Gr{oetje,eeting}s, > > Geert > > -- > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org > > In personal conversations with technical people, I call myself a hacker. But > when I'm talking to journalists I just say "programmer" or something like that. > -- Linus Torvalds ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-10 23:26 ` Eduardo Valentin @ 2018-09-11 8:45 ` Greg KH 2018-09-11 17:10 ` Dave Hansen 0 siblings, 1 reply; 54+ messages in thread From: Greg KH @ 2018-09-11 8:45 UTC (permalink / raw) To: Eduardo Valentin; +Cc: ksummit-discuss On Mon, Sep 10, 2018 at 04:26:53PM -0700, Eduardo Valentin wrote: > Hello Geert, > > On Fri, Sep 07, 2018 at 10:21:15AM +0200, Geert Uytterhoeven wrote: > > Hi Eduardo, > > > > On Fri, Sep 7, 2018 at 12:55 AM Eduardo Valentin <edubezval@gmail.com> wrote: > > > On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote: > > > Honestly, the fact that somehow the community managed to make this to > > > stable (and eventually to distros) is really good. Imagine for a second > > > a world in which these made only mainline and no stable branch.. > > > > It could have been the disaster needed to trigger a paradigm shift in the > > software industry? > > Well yeah. Just to clarify, what I meant is that the community did a > really great job handling the event, despite the fact that we could have > dont better. What do you feel we could have done "better" given the constraints placed on us? thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-11 8:45 ` Greg KH @ 2018-09-11 17:10 ` Dave Hansen 2018-09-11 18:28 ` Greg KH 2018-09-11 18:44 ` Thomas Gleixner 0 siblings, 2 replies; 54+ messages in thread From: Dave Hansen @ 2018-09-11 17:10 UTC (permalink / raw) To: Greg KH, Eduardo Valentin; +Cc: ksummit-discuss On 09/11/2018 01:45 AM, Greg KH wrote: > What do you feel we could have done "better" given the constraints > placed on us? Hindsight is 20/20. But, here are a few things I wish we would have had in place a year or two ago. These are utterly _minor_ nits in the grand scheme of things. Intel had the most room for improvement here, not the community. But, here's what the community could have done better: 1. Have a documented procedure for submitting issues that the submitter perceives can not go to security@. Folks are always going to think they are a special snowflake. This will get them talking to *someone* at least. 2. Document that stable updates require stable maintainers to be involved. If you want fixes in mainline, tell Linus. If you want stable fixes, tell the stable maintainers. Also document the time required to integrate a fix, even if it is a worst-case estimate. (The distros who consume stable can help here, too) Why? From what I've seen, the folks controlling the embargo respect *processes*. A written-down document is a process that's hard to argue with and represents the weight of the community. But if I tell them, it's just "one person's opinion". Giving timelines is also very important. Folks spend a lot of time counting months and weeks back on the calendar from a disclosure date. The timeline gives them a discrete date to *do* something. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-11 17:10 ` Dave Hansen @ 2018-09-11 18:28 ` Greg KH 2018-09-11 18:44 ` Thomas Gleixner 1 sibling, 0 replies; 54+ messages in thread From: Greg KH @ 2018-09-11 18:28 UTC (permalink / raw) To: Dave Hansen; +Cc: ksummit-discuss On Tue, Sep 11, 2018 at 10:10:37AM -0700, Dave Hansen wrote: > On 09/11/2018 01:45 AM, Greg KH wrote: > > What do you feel we could have done "better" given the constraints > > placed on us? > > Hindsight is 20/20. But, here are a few things I wish we would have had > in place a year or two ago. These are utterly _minor_ nits in the grand > scheme of things. Intel had the most room for improvement here, not the > community. But, here's what the community could have done better: > > 1. Have a documented procedure for submitting issues that the submitter > perceives can not go to security@. Folks are always going to think > they are a special snowflake. This will get them talking to > *someone* at least. We have Documentation/admin-guide/security-bugs.rst. If you feel it is lacking in some way that would help out here, please submit patches. I know some people did submit patches for this after the Meltdown mess happened to hopefully clear some things up. Oh wait, it was you who did that, why am I even saying this then? :) Anyway, Willy posted a patch somewhere to also update it, but that patch seems to have gotten dropped, we should poke him to resend it. > 2. Document that stable updates require stable maintainers to be > involved. If you want fixes in mainline, tell Linus. If you want > stable fixes, tell the stable maintainers. Also document the time > required to integrate a fix, even if it is a worst-case estimate. > (The distros who consume stable can help here, too) Note, I don't always _have_ to be involved. I usually don't want to be involved. But if you expect me to start taking random non-obvious patches into a stable kernel, you had better get me involved, otherwise you will start to get the "WTF" emails from me. That's just common sense, right? If not, how/where do I have to document this? Also, for some people like me, perhaps you might want to get us involved because we happen to know a bit about security and bugfixing given we have been doing it for many decades now (not just me, most of the security@ people are included in that category, hence them being on that team). Why _wouldn't_ you want them helping to fix your problem? > Why? From what I've seen, the folks controlling the embargo respect > *processes*. A written-down document is a process that's hard to argue > with and represents the weight of the community. But if I tell them, > it's just "one person's opinion". Great, the security process page can always need help, like you provided. Hopefully that looks sufficient to you now? > Giving timelines is also very important. Folks spend a lot of time > counting months and weeks back on the calendar from a disclosure date. > The timeline gives them a discrete date to *do* something. Giving timelines to whom? Us getting a timeline or us giving a timeline to others? It's hard to "predict" how long some things will take to be fixed (obviously, we once took 6 months to fix a nasty bug that some people thought it was already resolved as they provided a proposed fix.) thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-11 17:10 ` Dave Hansen 2018-09-11 18:28 ` Greg KH @ 2018-09-11 18:44 ` Thomas Gleixner 1 sibling, 0 replies; 54+ messages in thread From: Thomas Gleixner @ 2018-09-11 18:44 UTC (permalink / raw) To: Dave Hansen; +Cc: ksummit-discuss On Tue, 11 Sep 2018, Dave Hansen wrote: > > Giving timelines is also very important. Folks spend a lot of time > counting months and weeks back on the calendar from a disclosure date. > The timeline gives them a discrete date to *do* something. Giving a timeline whatfor? How long it takes to fix something? We need to know about the issue first in order to do so. So the simple answer here is ASAP and not when some disclosure manager thinks it's about time. I rather have the fix simmering in my hidden repository for a month or two than having to rush things toward the disclosure date or when the embargo breaks early. But then if we know what it is, it might be trivial to give an ETA and it might be complete guess work for a while until we wrapped our brains around it. Thanks, tglx ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 22:55 ` Eduardo Valentin 2018-09-07 8:21 ` Geert Uytterhoeven @ 2018-09-07 13:30 ` Jiri Kosina 2018-09-09 12:55 ` Greg KH 2018-09-12 4:22 ` Balbir Singh 2 siblings, 1 reply; 54+ messages in thread From: Jiri Kosina @ 2018-09-07 13:30 UTC (permalink / raw) To: Eduardo Valentin; +Cc: ksummit-discuss On Thu, 6 Sep 2018, Eduardo Valentin wrote: > Should we add maybe a point here to discuss which kernels are to be > considered for patching in these cases? All the stable branches? Only > mainline? Obviously, either extreme cases can hurt people. Patching > older kernels requires insane amount of work and patching only mainline > leaves distros on limbo. That'd be mostly question for the stable guys I guess. I am not sure how often did they in the past have to say "sorry, the backport is horribly complex, so we are not backporting the fix and we're keeping the bug unfixed". Greg, is this something that actually has been happening for real in the past? Or would that absolutely break the expectations that stable tree consumers have? Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-07 13:30 ` Jiri Kosina @ 2018-09-09 12:55 ` Greg KH 2018-09-09 19:48 ` Jiri Kosina ` (2 more replies) 0 siblings, 3 replies; 54+ messages in thread From: Greg KH @ 2018-09-09 12:55 UTC (permalink / raw) To: Jiri Kosina; +Cc: ksummit-discuss On Fri, Sep 07, 2018 at 03:30:32PM +0200, Jiri Kosina wrote: > On Thu, 6 Sep 2018, Eduardo Valentin wrote: > > > Should we add maybe a point here to discuss which kernels are to be > > considered for patching in these cases? All the stable branches? Only > > mainline? Obviously, either extreme cases can hurt people. Patching > > older kernels requires insane amount of work and patching only mainline > > leaves distros on limbo. > > That'd be mostly question for the stable guys I guess. I am not sure how > often did they in the past have to say "sorry, the backport is horribly > complex, so we are not backporting the fix and we're keeping the bug > unfixed". > > Greg, is this something that actually has been happening for real in the > past? Or would that absolutely break the expectations that stable tree > consumers have? Yes, this is something that is happening today. If you look, L1TF is not fully backported to 4.4.y, for anyone running 4.4.y as a host operating system. The backport was just too horrible and no one wanted to do it and test it as all of the major hosting services have moved on to 4.9.y or better. There are other examples of this, spectre fixes for arm32 are not in any stable tree older than 4.18.y. Same for other arches and kernel versions. I tried to write up "what kernel version to use" on my blog a few weeks back to answer this type of question. Basically, only "trust" the latest LTS stable kernel for security issues to be able to use it to run untrusted users. When you start getting older kernels involved, nasty problems like what Meltdown and the like are having to implement, it just does not work. So only "stay" with on old LTS kernel if your hardware requires you to (i.e. the horrid SoC nightmare). And even then, be careful about things (sandboxes, selinux, etc.) and go yell at your SoC vendor for forcing you into this nightmare of a problem. If they do not hear from companies, they will not change. thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 12:55 ` Greg KH @ 2018-09-09 19:48 ` Jiri Kosina 2018-09-10 4:04 ` Eduardo Valentin 2018-09-10 4:12 ` Eduardo Valentin 2018-09-10 11:10 ` Mark Brown 2 siblings, 1 reply; 54+ messages in thread From: Jiri Kosina @ 2018-09-09 19:48 UTC (permalink / raw) To: Greg KH; +Cc: ksummit-discuss On Sun, 9 Sep 2018, Greg KH wrote: > Yes, this is something that is happening today. > > If you look, L1TF is not fully backported to 4.4.y, for anyone running > 4.4.y as a host operating system. The backport was just too horrible > and no one wanted to do it and test it as all of the major hosting > services have moved on to 4.9.y or better. Unrelated sidenote: we have the whole thing backported to SUSE 4.4 kernel, so it can be cherry-picked from there if needed. > There are other examples of this, spectre fixes for arm32 are not in any > stable tree older than 4.18.y. Same for other arches and kernel > versions. > > I tried to write up "what kernel version to use" on my blog a few weeks > back to answer this type of question. Basically, only "trust" the > latest LTS stable kernel for security issues to be able to use it to run > untrusted users. When you start getting older kernels involved, nasty > problems like what Meltdown and the like are having to implement, it > just does not work. OK, so as long as this message is completely clear to the stable tree consumers (see my other mail about potential legal implications for the downstream consumers in case they are not aware of this), then all is fine. Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 19:48 ` Jiri Kosina @ 2018-09-10 4:04 ` Eduardo Valentin 2018-09-12 7:03 ` Greg KH 0 siblings, 1 reply; 54+ messages in thread From: Eduardo Valentin @ 2018-09-10 4:04 UTC (permalink / raw) To: Jiri Kosina; +Cc: ksummit-discuss On Sun, Sep 09, 2018 at 09:48:58PM +0200, Jiri Kosina wrote: > On Sun, 9 Sep 2018, Greg KH wrote: > > > Yes, this is something that is happening today. > > > > If you look, L1TF is not fully backported to 4.4.y, for anyone running > > 4.4.y as a host operating system. The backport was just too horrible > > and no one wanted to do it and test it as all of the major hosting > > services have moved on to 4.9.y or better. > > Unrelated sidenote: we have the whole thing backported to SUSE 4.4 kernel, > so it can be cherry-picked from there if needed. > > > There are other examples of this, spectre fixes for arm32 are not in any > > stable tree older than 4.18.y. Same for other arches and kernel > > versions. > > > > I tried to write up "what kernel version to use" on my blog a few weeks > > back to answer this type of question. Basically, only "trust" the > > latest LTS stable kernel for security issues to be able to use it to run > > untrusted users. When you start getting older kernels involved, nasty > > problems like what Meltdown and the like are having to implement, it > > just does not work. > > OK, so as long as this message is completely clear to the stable tree > consumers (see my other mail about potential legal implications for the > downstream consumers in case they are not aware of this), then all is > fine. If I got this right, the lastest LTS gets what is closest to upstream, everything else gets "best effort" backport, I assume. > > Thanks, > > -- > Jiri Kosina > SUSE Labs > ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-10 4:04 ` Eduardo Valentin @ 2018-09-12 7:03 ` Greg KH 0 siblings, 0 replies; 54+ messages in thread From: Greg KH @ 2018-09-12 7:03 UTC (permalink / raw) To: Eduardo Valentin; +Cc: ksummit-discuss On Sun, Sep 09, 2018 at 09:04:57PM -0700, Eduardo Valentin wrote: > On Sun, Sep 09, 2018 at 09:48:58PM +0200, Jiri Kosina wrote: > > On Sun, 9 Sep 2018, Greg KH wrote: > > > > > Yes, this is something that is happening today. > > > > > > If you look, L1TF is not fully backported to 4.4.y, for anyone running > > > 4.4.y as a host operating system. The backport was just too horrible > > > and no one wanted to do it and test it as all of the major hosting > > > services have moved on to 4.9.y or better. > > > > Unrelated sidenote: we have the whole thing backported to SUSE 4.4 kernel, > > so it can be cherry-picked from there if needed. > > > > > There are other examples of this, spectre fixes for arm32 are not in any > > > stable tree older than 4.18.y. Same for other arches and kernel > > > versions. > > > > > > I tried to write up "what kernel version to use" on my blog a few weeks > > > back to answer this type of question. Basically, only "trust" the > > > latest LTS stable kernel for security issues to be able to use it to run > > > untrusted users. When you start getting older kernels involved, nasty > > > problems like what Meltdown and the like are having to implement, it > > > just does not work. > > > > OK, so as long as this message is completely clear to the stable tree > > consumers (see my other mail about potential legal implications for the > > downstream consumers in case they are not aware of this), then all is > > fine. > > If I got this right, the lastest LTS gets what is closest to upstream, > everything else gets "best effort" backport, I assume. That is correct. thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 12:55 ` Greg KH 2018-09-09 19:48 ` Jiri Kosina @ 2018-09-10 4:12 ` Eduardo Valentin 2018-09-10 11:10 ` Mark Brown 2 siblings, 0 replies; 54+ messages in thread From: Eduardo Valentin @ 2018-09-10 4:12 UTC (permalink / raw) To: Greg KH; +Cc: ksummit-discuss On Sun, Sep 09, 2018 at 02:55:54PM +0200, Greg KH wrote: > On Fri, Sep 07, 2018 at 03:30:32PM +0200, Jiri Kosina wrote: > > On Thu, 6 Sep 2018, Eduardo Valentin wrote: > > > > > Should we add maybe a point here to discuss which kernels are to be > > > considered for patching in these cases? All the stable branches? Only > > > mainline? Obviously, either extreme cases can hurt people. Patching > > > older kernels requires insane amount of work and patching only mainline > > > leaves distros on limbo. > > > > That'd be mostly question for the stable guys I guess. I am not sure how > > often did they in the past have to say "sorry, the backport is horribly > > complex, so we are not backporting the fix and we're keeping the bug > > unfixed". > > > > Greg, is this something that actually has been happening for real in the > > past? Or would that absolutely break the expectations that stable tree > > consumers have? > > Yes, this is something that is happening today. > > If you look, L1TF is not fully backported to 4.4.y, for anyone running > 4.4.y as a host operating system. The backport was just too horrible > and no one wanted to do it and test it as all of the major hosting > services have moved on to 4.9.y or better. > > There are other examples of this, spectre fixes for arm32 are not in any > stable tree older than 4.18.y. Same for other arches and kernel > versions. Another example of divergence is meltdown itself, which made in 4.14 (Latest LTS) pretty close to what went into mainline, but anything older than 4.14 got the core of meltdown implementation, but not all the optimization in the x86 entry code, which was horrible to backport entirely. > > I tried to write up "what kernel version to use" on my blog a few weeks > back to answer this type of question. Basically, only "trust" the > latest LTS stable kernel for security issues to be able to use it to run > untrusted users. When you start getting older kernels involved, nasty > problems like what Meltdown and the like are having to implement, it > just does not work. > > So only "stay" with on old LTS kernel if your hardware requires you to > (i.e. the horrid SoC nightmare). And even then, be careful about things > (sandboxes, selinux, etc.) and go yell at your SoC vendor for forcing > you into this nightmare of a problem. If they do not hear from > companies, they will not change. For these horrendous cases, should the community simply not patch the oldest kernels? Patching only the latest LTS may be a very strong reason for everyone on older to simply go ahead and upgrade their kernels. > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 12:55 ` Greg KH 2018-09-09 19:48 ` Jiri Kosina 2018-09-10 4:12 ` Eduardo Valentin @ 2018-09-10 11:10 ` Mark Brown 2 siblings, 0 replies; 54+ messages in thread From: Mark Brown @ 2018-09-10 11:10 UTC (permalink / raw) To: Greg KH; +Cc: ksummit-discuss [-- Attachment #1: Type: text/plain, Size: 453 bytes --] On Sun, Sep 09, 2018 at 02:55:54PM +0200, Greg KH wrote: > There are other examples of this, spectre fixes for arm32 are not in any > stable tree older than 4.18.y. Same for other arches and kernel > versions. Not really relevant to your point but just for clarity those should make it back to the LTS versions at some point for arm32, they've been held up behind some of the other entertaining backports. Which is itself an issue but there we are. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 22:55 ` Eduardo Valentin 2018-09-07 8:21 ` Geert Uytterhoeven 2018-09-07 13:30 ` Jiri Kosina @ 2018-09-12 4:22 ` Balbir Singh 2 siblings, 0 replies; 54+ messages in thread From: Balbir Singh @ 2018-09-12 4:22 UTC (permalink / raw) To: Eduardo Valentin; +Cc: ksummit-discuss On Thu, Sep 06, 2018 at 03:55:33PM -0700, Eduardo Valentin wrote: > Hey, > > On Thu, Sep 06, 2018 at 09:18:07PM +0200, Jiri Kosina wrote: > > I believe we have reasonably well-established process for handling > > security issues that are a matter of a single, reasonably self-contained > > fixup. > > > > Past experiences with Meltdown, Spectre and L1TF have shown that we're not > > really ready to handle that in a reasonably sane way yet. > > > > Yeah, at the end of the day we managed to have the fixes propagated in > > time to Linus' tree, to stable, and to distros as well, but it was > > completely out of anything regular, and definitely had permanent damaging > > effects (I'd say both in personal and business aspects for almost > > everybody who had to participate). > > > > I'd believe that everybody involved would agree that this didn't work > > really well, and if it potentially would have to happen again (and we > > already went through it at least twice this year), it would not be > > sustainable. > > > > I am not completely sure what we could do to improve this, especially with > > our kernel community hats on -- I am pretty sure a lot is happening on the > > corporate level between individual "corporate stakeholders". Ideas I think > > would be worth discussing: > > > > - how to adapt our processess to be able to deal with such situations > > better should they happen in the future again. So far all our > > longer-term development has been concentrated around LKML (and other > > MLs) and the existing maintainership communities / structures, but the > > embargos for new big features don't really fit into this > > > > - how to make sure that proper pressure is applied on the companies that > > are handling embargoes irresponsibly wrt. linux/opensource development > > (well, even some proprietary vendors were rather unhappy with those > > events) from us as the linux kernel community > > + and perhaps even more importantly, what exactly we should be pressing > > for > > Should we add maybe a point here to discuss which kernels are to be > considered for patching in these cases? All the stable branches? Only > mainline? Obviously, either extreme cases can hurt people. Patching > older kernels requires insane amount of work and patching only mainline > leaves distros on limbo. > I don't think we can decide, may be the right thing to do is to consider where the pressure is coming from 1. Security researcher's wanting to set aggressive embargo schedules? 2. Are distros supporting older kernels for too long? The spectre/meltdown affected all architectures/variants of CPUs and so the overhead was massive and I don't think that will change depending the layer in which a security vulnerability is discovered. > Honestly, the fact that somehow the community managed to make this to > stable (and eventually to distros) is really good. Imagine for a second > a world in which these made only mainline and no stable branch.. > Half the world would have bugs in their patches :) > In any case, maybe the community should consider what really is going to > be effectively patched. That may be an extra push for distros to upgrade > older kernels as well. > > Balbir Singh. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-06 19:18 [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues Jiri Kosina 2018-09-06 20:56 ` Linus Torvalds 2018-09-06 22:55 ` Eduardo Valentin @ 2018-09-08 4:21 ` Andy Lutomirski 2018-09-08 8:56 ` Thomas Gleixner 2 siblings, 1 reply; 54+ messages in thread From: Andy Lutomirski @ 2018-09-08 4:21 UTC (permalink / raw) To: Jiri Kosina; +Cc: ksummit-discuss On Thu, Sep 6, 2018 at 12:18 PM, Jiri Kosina <jikos@kernel.org> wrote: > I believe we have reasonably well-established process for handling > security issues that are a matter of a single, reasonably self-contained > fixup. > > Past experiences with Meltdown, Spectre and L1TF have shown that we're not > really ready to handle that in a reasonably sane way yet. > To a large extend, I think that the problems we had with Meltdown and Spectre weren't the fault of our processes so much as the fact that we weren't allowed to have a sane process. As far as I know, the number of upstream-focused non-Intel-employed Linux developers who were told about Meltdown and Spectre in a timely manner can be counted on a very small number of fingers on one hand. Those people had no channel to communicate with any of the distro people involved in a timely manner. This situation was not good. Perhaps the Linux community could have a small team that handles coordination of severe incidents along with a documented process for what happens when the process is invoked. It would need to be designed in such a way that vendors would be willing to use it when needed and that it gave the Linux world enough flexibility to actually handle the issue. At the very least, the relevant maintainers and perhaps some relevant upstream reviewers would get notified, and the people working on the issue for distros could actually communicate with the upstream folks. The total set of people involved would be small. It could be useful to try to get the process to work similarly to whatever Microsoft and Apple do. (There was another CVE that got much less press that I was involved, and it didn't get much attention in Linux land because Linux was only minimally affected. Despite being an Intel/AMD issue, all of the coordination was done by Microsoft, and it was done remarkably well once the process actually got started.) ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 4:21 ` Andy Lutomirski @ 2018-09-08 8:56 ` Thomas Gleixner 2018-09-08 11:21 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 54+ messages in thread From: Thomas Gleixner @ 2018-09-08 8:56 UTC (permalink / raw) To: Andy Lutomirski; +Cc: ksummit-discuss On Fri, 7 Sep 2018, Andy Lutomirski wrote: > (There was another CVE that got much less press that I was involved, > and it didn't get much attention in Linux land because Linux was only > minimally affected. Despite being an Intel/AMD issue, all of the > coordination was done by Microsoft, and it was done remarkably well > once the process actually got started.) The point is that the coordination done by the entity who 'owns' the thing is the key. Contrary to Meltdown/Spectre Intel informed us about L1Tf halfways early and allowed _all_ involved parties to talk to each other. There were still some rough edges to bring key people like Greg in, but that was a minor nuisance compared to the whole Meltdown/Spectre mess. Of course there was no communication channel which allowed us to talk in a workable way, but that got resolved by ourself setting up a encrypted mailing list which made halfways normal kernel style cooperation possible. That still has its rough edges vs. limited review capacity and testing, but compared to Meltdown/Spectre L1TF was halfways workable. So we surely have the ability to communicate properly in an embargo situation, but that requires that the entity who controls the issue is 1) Telling us in time and putting all cards on the table 2) Setting no silly restrictions vs. who can talk to whom That said, I agree that a more formal process with clear rules might make it easier for companies to handle that proper. It's definitely worth to try. Though it won't make the issues which come inherently with embargo development go away. Mechanisms we rely on like 0-bot, kernel-ci and others need to grow an embargo mode as well. Thanks, tglx ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 8:56 ` Thomas Gleixner @ 2018-09-08 11:21 ` Mauro Carvalho Chehab 2018-09-08 11:34 ` Greg KH 0 siblings, 1 reply; 54+ messages in thread From: Mauro Carvalho Chehab @ 2018-09-08 11:21 UTC (permalink / raw) To: Thomas Gleixner; +Cc: ksummit-discuss Em Sat, 8 Sep 2018 10:56:35 +0200 (CEST) Thomas Gleixner <tglx@linutronix.de> escreveu: > On Fri, 7 Sep 2018, Andy Lutomirski wrote: > > (There was another CVE that got much less press that I was involved, > > and it didn't get much attention in Linux land because Linux was only > > minimally affected. Despite being an Intel/AMD issue, all of the > > coordination was done by Microsoft, and it was done remarkably well > > once the process actually got started.) > > The point is that the coordination done by the entity who 'owns' the thing > is the key. I didn't participate on handing those bugs. So, I'll give just my 2 cents here. IMO, the problem with embargoed security issues is that the affected vendor will very likely want to contain the ones that have access to the bug-related information. They also want to have some legal ways to protect their asses^Wassets if data gets disclosed. In other words, they'll very likely require some sort of NDA. As we're working on different places, each one is under a different NDA agreement with the affected vendor (and some may even not have any signed NDA at all). I suspect that identifying who has a signed NDA that would cover the assets for everyone that would have access to the embargoed data can take quite some time. > Contrary to Meltdown/Spectre Intel informed us about L1Tf halfways early > and allowed _all_ involved parties to talk to each other. That's probably because they had already gave some sort of "clearance" to the ones that handled Meltdown/Spectre. I suspect that, if L1Tf were affecting some other vendor, it would also be as painful as it was for Meltdown/Spectre. IMHO, the best would be to have a formal/legal way to handle it. I suspect that having a single entity like LF handling a formal process for embargoed issues could help making it smoother. That would likely envolve having a single NDA model that could be signed with vendors (either before of after an event like that), and LF having pre-signed NDA with the group of maintainers/developers that would more likely handle such issues. > There were still > some rough edges to bring key people like Greg in, but that was a minor > nuisance compared to the whole Meltdown/Spectre mess. > > Of course there was no communication channel which allowed us to talk in a > workable way, but that got resolved by ourself setting up a encrypted > mailing list which made halfways normal kernel style cooperation > possible. That still has its rough edges vs. limited review capacity and > testing, but compared to Meltdown/Spectre L1TF was halfways workable. A formal agreement with a single entity could help there too. I mean, provided that everyone's involved is under a legal umbrella, it should be possible for the vendor to give logical access to some of their own labs where it would be possible to run the tests with the real thing. > > So we surely have the ability to communicate properly in an embargo > situation, but that requires that the entity who controls the issue is > > 1) Telling us in time and putting all cards on the table > 2) Setting no silly restrictions vs. who can talk to whom > > That said, I agree that a more formal process with clear rules might make > it easier for companies to handle that proper. It's definitely worth to > try. > > Though it won't make the issues which come inherently with embargo > development go away. Mechanisms we rely on like 0-bot, kernel-ci and others > need to grow an embargo mode as well. > > Thanks, > > tglx > > > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss Thanks, Mauro ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 11:21 ` Mauro Carvalho Chehab @ 2018-09-08 11:34 ` Greg KH 2018-09-08 14:20 ` Andy Lutomirski 2018-09-08 15:00 ` James Bottomley 0 siblings, 2 replies; 54+ messages in thread From: Greg KH @ 2018-09-08 11:34 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: ksummit-discuss On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab wrote: > IMHO, the best would be to have a formal/legal way to handle it. No, sorry, some of us are not allowed legally to sign NDAs for stuff like this. So keeping legal out of is it the best solution and we have done that pretty well so far. thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 11:34 ` Greg KH @ 2018-09-08 14:20 ` Andy Lutomirski 2018-09-08 15:29 ` Greg KH 2018-09-08 15:00 ` James Bottomley 1 sibling, 1 reply; 54+ messages in thread From: Andy Lutomirski @ 2018-09-08 14:20 UTC (permalink / raw) To: Greg KH; +Cc: Mauro Carvalho Chehab, ksummit-discuss > On Sep 8, 2018, at 4:34 AM, Greg KH <greg@kroah.com> wrote: > >> On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab wrote: >> IMHO, the best would be to have a formal/legal way to handle it. > > No, sorry, some of us are not allowed legally to sign NDAs for stuff > like this. So keeping legal out of is it the best solution and we have > done that pretty well so far. > A lot us us (such as yours truly) have NDAs in place. I would love a clear mechanism by which a vendor gives explicit permission for me to communicate with other relevant parties. It doesn’t need to be fancy and legalistic, but having it written down would be very, very nice. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 14:20 ` Andy Lutomirski @ 2018-09-08 15:29 ` Greg KH 0 siblings, 0 replies; 54+ messages in thread From: Greg KH @ 2018-09-08 15:29 UTC (permalink / raw) To: Andy Lutomirski; +Cc: Mauro Carvalho Chehab, ksummit-discuss On Sat, Sep 08, 2018 at 07:20:00AM -0700, Andy Lutomirski wrote: > > > > On Sep 8, 2018, at 4:34 AM, Greg KH <greg@kroah.com> wrote: > > > >> On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab wrote: > >> IMHO, the best would be to have a formal/legal way to handle it. > > > > No, sorry, some of us are not allowed legally to sign NDAs for stuff > > like this. So keeping legal out of is it the best solution and we have > > done that pretty well so far. > > > > A lot us us (such as yours truly) have NDAs in place. I would love a > clear mechanism by which a vendor gives explicit permission for me to > communicate with other relevant parties. It doesn’t need to be fancy > and legalistic, but having it written down would be very, very nice. I agree, what we have now is a mis-match of different people working for different companies with different types of NDAs through different companies or not, trying to deal with different companies trying to work with us. Having something "simple" for us to work off of would be good, and this isn't the first thread that this topic has come up on. I know people want to solve this, and maybe the LF is the way to go here. There was a meeting last week at Harvard to start to consider how this could happen. Hopefully in a few months we will have something "solid" to discuss. thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 11:34 ` Greg KH 2018-09-08 14:20 ` Andy Lutomirski @ 2018-09-08 15:00 ` James Bottomley 2018-09-08 15:32 ` Greg KH 2018-09-08 19:26 ` Jiri Kosina 1 sibling, 2 replies; 54+ messages in thread From: James Bottomley @ 2018-09-08 15:00 UTC (permalink / raw) To: Greg KH, Mauro Carvalho Chehab; +Cc: ksummit-discuss On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote: > On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab > wrote: > > IMHO, the best would be to have a formal/legal way to handle it. > > No, sorry, some of us are not allowed legally to sign NDAs for stuff > like this. As a blanket statement this simply isn't true. It is certainly possible for a specific NDA to conflict with other agreed obligations an individual has in which case that specific NDA can't be signed. However, knowing the obligations, it's also possible to craft a different version of the NDA that can be signed. There's also significant problems with badly worded NDAs and Open Source: given you're going to produce a thing which everyone can see, disclosure ipso facto eventually occurs so the NDA has to take this correctly into account but again, this can be done. I can respect the moral position that NDAs are incompatible with the values of open source but it's an individual choice and conscience issue not a legal one. > So keeping legal out of is it the best solution and we have > done that pretty well so far. I think we might benefit from a discussion of whether we could have handled Meltdown/Spectre better in an NDA framework ... I'm not saying it would have been any better, just that we might consider if some driving need for secrecy caused us to be left out of the loop and whether a small cabal in the know with an NDA might have steered us better. James ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 15:00 ` James Bottomley @ 2018-09-08 15:32 ` Greg KH 2018-09-08 15:54 ` James Bottomley 2018-09-08 19:26 ` Jiri Kosina 1 sibling, 1 reply; 54+ messages in thread From: Greg KH @ 2018-09-08 15:32 UTC (permalink / raw) To: James Bottomley; +Cc: Mauro Carvalho Chehab, ksummit-discuss On Sat, Sep 08, 2018 at 08:00:29AM -0700, James Bottomley wrote: > On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote: > > On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab > > wrote: > > > IMHO, the best would be to have a formal/legal way to handle it. > > > > No, sorry, some of us are not allowed legally to sign NDAs for stuff > > like this. > > As a blanket statement this simply isn't true. Um, I said "some of us". Some of us can, some of us can not. That's a blanket statement that has to be true :) > It is certainly > possible for a specific NDA to conflict with other agreed obligations > an individual has in which case that specific NDA can't be signed. > However, knowing the obligations, it's also possible to craft a > different version of the NDA that can be signed. > > There's also significant problems with badly worded NDAs and Open > Source: given you're going to produce a thing which everyone can see, > disclosure ipso facto eventually occurs so the NDA has to take this > correctly into account but again, this can be done. > > I can respect the moral position that NDAs are incompatible with the > values of open source but it's an individual choice and conscience > issue not a legal one. I never brought up moral positions, you did. I do not have a moral position on this, sorry. > > So keeping legal out of is it the best solution and we have > > done that pretty well so far. > > I think we might benefit from a discussion of whether we could have > handled Meltdown/Spectre better in an NDA framework ... I'm not saying > it would have been any better, just that we might consider if some > driving need for secrecy caused us to be left out of the loop and > whether a small cabal in the know with an NDA might have steered us > better. This is what happened for L1TF, and yes, it was steered better and worked amazingly well for the conditions under which we were forced to work. So things are getting better, Meltdown/Spectre showed some people that that was _not_ the way to handle it, with hard proof. So they changed, and we will continue to revise and work through these issues. thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 15:32 ` Greg KH @ 2018-09-08 15:54 ` James Bottomley 2018-09-08 19:49 ` Linus Torvalds 2018-09-09 12:51 ` Greg KH 0 siblings, 2 replies; 54+ messages in thread From: James Bottomley @ 2018-09-08 15:54 UTC (permalink / raw) To: Greg KH; +Cc: Mauro Carvalho Chehab, ksummit-discuss On Sat, 2018-09-08 at 17:32 +0200, Greg KH wrote: > On Sat, Sep 08, 2018 at 08:00:29AM -0700, James Bottomley wrote: > > On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote: > > > On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab > > > wrote: > > > > IMHO, the best would be to have a formal/legal way to handle > > > > it. > > > > > > No, sorry, some of us are not allowed legally to sign NDAs for > > > stuff like this. > > > > As a blanket statement this simply isn't true. > > Um, I said "some of us". Some of us can, some of us can not. That's > a blanket statement that has to be true :) OK, let me make it more specific: there exists no individual contributing to open source in a leadership capacity for whom a signable NDA cannot be crafted. The reason is fairly simple: leadership positions come with duties of care which include duties of confidentiality (think adjudication of anti-harassment policy or code of conflict) and an NDA could be crafted directly along the lines of the duty of confidentiality. Now it's not that an employment or other agreement couldn't forbid this, but if it did the leader in question would be in a false position already because their agreement effectively forbids them from taking a leadership position in open source in the first place. James ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 15:54 ` James Bottomley @ 2018-09-08 19:49 ` Linus Torvalds 2018-09-08 21:24 ` James Bottomley 2018-09-10 22:59 ` Dave Hansen 2018-09-09 12:51 ` Greg KH 1 sibling, 2 replies; 54+ messages in thread From: Linus Torvalds @ 2018-09-08 19:49 UTC (permalink / raw) To: James Bottomley; +Cc: mchehab+samsung, ksummit-discuss [-- Attachment #1: Type: text/plain, Size: 1154 bytes --] On Sat, Sep 8, 2018, 08:54 James Bottomley < James.Bottomley@hansenpartnership.com> wrote: > > OK, let me make it more specific: there exists no individual > contributing to open source in a leadership capacity for whom a > signable NDA cannot be crafted. > No. I don't sign NDA's. I just don't do it. It's that simple. It's actually worked pretty well. It started because I worked for a direct competitor to Intel, and couldn't sign an NDA for the really old f0 0f lockup issue. Not having an NDA back then turned out to be a good thing, because it made it a non-issue when leaks happened. So I started the policy that I never want to be in the position that I had to worry legally about being in the position of being under an NDA and knowing things outside of the leaks. Instead, I've had a gentleman's agreement with companies - nothing legally binding, but over the years people have come to realize that the leaks don't come from me. So I don't do NDA's. Maybe some Linux Foundation NDA agreement technically covers me, but at least with the Intel cases, Intel is actually aware of my non-NDA situation and is fine with it. Linus > [-- Attachment #2: Type: text/html, Size: 2027 bytes --] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 19:49 ` Linus Torvalds @ 2018-09-08 21:24 ` James Bottomley 2018-09-08 22:33 ` Andy Lutomirski 2018-09-10 22:59 ` Dave Hansen 1 sibling, 1 reply; 54+ messages in thread From: James Bottomley @ 2018-09-08 21:24 UTC (permalink / raw) To: Linus Torvalds; +Cc: mchehab+samsung, ksummit-discuss On Sat, 2018-09-08 at 12:49 -0700, Linus Torvalds wrote: > On Sat, Sep 8, 2018, 08:54 James Bottomley < > James.Bottomley@hansenpartnership.com> wrote: > > > > > OK, let me make it more specific: there exists no individual > > contributing to open source in a leadership capacity for whom a > > signable NDA cannot be crafted. > > > > No. > > I don't sign NDA's. I just don't do it. > > It's that simple. But that's you're choice; it's not because legally you can't. > It's actually worked pretty well. It started because I worked for a > direct competitor to Intel, and couldn't sign an NDA for the really > old f0 0f lockup issue. > > Not having an NDA back then turned out to be a good thing, because it > made it a non-issue when leaks happened. So I started the policy that > I never want to be in the position that I had to worry legally about > being in the position of being under an NDA and knowing things > outside of the leaks. > > Instead, I've had a gentleman's agreement with companies - nothing > legally binding, but over the years people have come to realize that > the leaks don't come from me. > > So I don't do NDA's. Maybe some Linux Foundation NDA agreement > technically covers me, but at least with the Intel cases, Intel is > actually aware of my non-NDA situation and is fine with it. I'm fine with all of this as an argument. If we believe that signing NDAs would eventually lead to worse disasters because agreeing to them now means corporations never change and never take our views into account, then we should have the debate and make the decision for sound policy reasons not because there's some spurious legal bar. James ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 21:24 ` James Bottomley @ 2018-09-08 22:33 ` Andy Lutomirski 2018-09-09 12:18 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 54+ messages in thread From: Andy Lutomirski @ 2018-09-08 22:33 UTC (permalink / raw) To: James Bottomley; +Cc: Mauro Carvalho Chehab, ksummit-discuss On Sat, Sep 8, 2018 at 2:24 PM, James Bottomley <James.Bottomley@hansenpartnership.com> wrote: > On Sat, 2018-09-08 at 12:49 -0700, Linus Torvalds wrote: >> On Sat, Sep 8, 2018, 08:54 James Bottomley < >> James.Bottomley@hansenpartnership.com> wrote: >> >> > >> > OK, let me make it more specific: there exists no individual >> > contributing to open source in a leadership capacity for whom a >> > signable NDA cannot be crafted. >> > >> >> No. >> >> I don't sign NDA's. I just don't do it. >> >> It's that simple. > > But that's you're choice; it's not because legally you can't. > >> It's actually worked pretty well. It started because I worked for a >> direct competitor to Intel, and couldn't sign an NDA for the really >> old f0 0f lockup issue. >> >> Not having an NDA back then turned out to be a good thing, because it >> made it a non-issue when leaks happened. So I started the policy that >> I never want to be in the position that I had to worry legally about >> being in the position of being under an NDA and knowing things >> outside of the leaks. >> >> Instead, I've had a gentleman's agreement with companies - nothing >> legally binding, but over the years people have come to realize that >> the leaks don't come from me. >> >> So I don't do NDA's. Maybe some Linux Foundation NDA agreement >> technically covers me, but at least with the Intel cases, Intel is >> actually aware of my non-NDA situation and is fine with it. > > I'm fine with all of this as an argument. If we believe that signing > NDAs would eventually lead to worse disasters because agreeing to them > now means corporations never change and never take our views into > account, then we should have the debate and make the decision for sound > policy reasons not because there's some spurious legal bar. > My NDA is through my company. I would *love* to cancel it and set up a replacement arrangement through LF or a similar entity, or to just not replace it at all. My company is not equipped for the kind of wrangling that would have helped during Meltdown and a couple of other situations, whereas anything reasonable set up for the purpose would work much better. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 22:33 ` Andy Lutomirski @ 2018-09-09 12:18 ` Mauro Carvalho Chehab 0 siblings, 0 replies; 54+ messages in thread From: Mauro Carvalho Chehab @ 2018-09-09 12:18 UTC (permalink / raw) To: Andy Lutomirski; +Cc: James Bottomley, ksummit-discuss Em Sat, 8 Sep 2018 15:33:22 -0700 Andy Lutomirski <luto@kernel.org> escreveu: > On Sat, Sep 8, 2018 at 2:24 PM, James Bottomley > <James.Bottomley@hansenpartnership.com> wrote: > > On Sat, 2018-09-08 at 12:49 -0700, Linus Torvalds wrote: > >> On Sat, Sep 8, 2018, 08:54 James Bottomley < > >> James.Bottomley@hansenpartnership.com> wrote: > >> > >> > > >> > OK, let me make it more specific: there exists no individual > >> > contributing to open source in a leadership capacity for whom a > >> > signable NDA cannot be crafted. > >> > > >> > >> No. > >> > >> I don't sign NDA's. I just don't do it. > >> > >> It's that simple. > > > > But that's you're choice; it's not because legally you can't. > > > >> It's actually worked pretty well. It started because I worked for a > >> direct competitor to Intel, and couldn't sign an NDA for the really > >> old f0 0f lockup issue. > >> > >> Not having an NDA back then turned out to be a good thing, because it > >> made it a non-issue when leaks happened. So I started the policy that > >> I never want to be in the position that I had to worry legally about > >> being in the position of being under an NDA and knowing things > >> outside of the leaks. > >> > >> Instead, I've had a gentleman's agreement with companies - nothing > >> legally binding, but over the years people have come to realize that > >> the leaks don't come from me. > >> > >> So I don't do NDA's. Maybe some Linux Foundation NDA agreement > >> technically covers me, but at least with the Intel cases, Intel is > >> actually aware of my non-NDA situation and is fine with it. > > > > I'm fine with all of this as an argument. If we believe that signing > > NDAs would eventually lead to worse disasters because agreeing to them > > now means corporations never change and never take our views into > > account, then we should have the debate and make the decision for sound > > policy reasons not because there's some spurious legal bar. > > > > My NDA is through my company. I would *love* to cancel it and set up > a replacement arrangement through LF or a similar entity, or to just > not replace it at all. My company is not equipped for the kind of > wrangling that would have helped during Meltdown and a couple of other > situations, whereas anything reasonable set up for the purpose would > work much better. I guess this is the situation of most of all. My contract has a NDA covering my work, so I'm legally bound to whatever NDA my employer has to some other companies. I'm almost certain that none of those were designed to cover Open Source. So, I'm pretty sure that, if I had to deal with an embargoed security issue that would require an NDA to get access to details, I would need to spend a lot of time and effort talking with the legal department to explain the needs and discuss about a set of clauses that would work for both sides(with won't be trivial, as they usually don't usually deal with open source). There's also a language barrier: some lawyers expect NDAs to be under the Country's official language, in order for it to have legal value on that Country. All that process can take weeks to happen, as that would likely envolve discussions between several parties. During that period of time, I won't likely be able to access the data relevant to solve the embargoed issue. So, as much as I would prefer to live in a world that would work without any NDAs (and on a first world Country), reality takes place. Having something prepared in forehand would significantly improve the process. Granted, it is unlikely that I would have to deal with those issues with the stuff I currently handle, as usually most serious security threats are not at drivers, but I suspect that others would have to deal with similar issues. Thanks, Mauro ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 19:49 ` Linus Torvalds 2018-09-08 21:24 ` James Bottomley @ 2018-09-10 22:59 ` Dave Hansen 2018-09-11 8:48 ` Greg KH 1 sibling, 1 reply; 54+ messages in thread From: Dave Hansen @ 2018-09-10 22:59 UTC (permalink / raw) To: Linus Torvalds, James Bottomley; +Cc: mchehab+samsung, ksummit-discuss On 09/08/2018 12:49 PM, Linus Torvalds wrote: > So I don't do NDA's. Maybe some Linux Foundation NDA agreement > technically covers me, but at least with the Intel cases, Intel is > actually aware of my non-NDA situation and is fine with it. My *personal* observation on the NDAs: Companies don't actually care about the NDA being an NDA per se. They really only want to feel like they are in control of the information. They get that warm and fuzzy feeling from NDAs for normal company-to-company interactions, which makes NDAs the go-to tool when these security things pop up. We (the community) are slowly showing the NDA-loving folks that <gasp> they are not the _only_ tool available. But, it's going to take time to change the mindset. I *do* wish that companies like Intel who are actively doing these non-NDA things would find some way to share their methods. Maybe the LF can help here by providing a semi-anonymous way for folks to share what has worked. Or, maybe folks like Intel need to just to it ourselves. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-10 22:59 ` Dave Hansen @ 2018-09-11 8:48 ` Greg KH 0 siblings, 0 replies; 54+ messages in thread From: Greg KH @ 2018-09-11 8:48 UTC (permalink / raw) To: Dave Hansen; +Cc: James Bottomley, mchehab+samsung, ksummit-discuss On Mon, Sep 10, 2018 at 03:59:26PM -0700, Dave Hansen wrote: > I *do* wish that companies like Intel who are actively doing these > non-NDA things would find some way to share their methods. Maybe the LF > can help here by providing a semi-anonymous way for folks to share what > has worked. Or, maybe folks like Intel need to just to it ourselves. There is a framework where Intel can share these types of methods already. The LF has at least one meeting a year for all of the Linux-related company's legal people to come together and share information and talk in a way that all can benefit, using the Chatham House rules. I'm sure a presentation from Intel to that group would always be most welcome. thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 15:54 ` James Bottomley 2018-09-08 19:49 ` Linus Torvalds @ 2018-09-09 12:51 ` Greg KH 2018-09-09 14:20 ` Linus Torvalds 1 sibling, 1 reply; 54+ messages in thread From: Greg KH @ 2018-09-09 12:51 UTC (permalink / raw) To: James Bottomley; +Cc: Mauro Carvalho Chehab, ksummit-discuss On Sat, Sep 08, 2018 at 08:54:26AM -0700, James Bottomley wrote: > On Sat, 2018-09-08 at 17:32 +0200, Greg KH wrote: > > On Sat, Sep 08, 2018 at 08:00:29AM -0700, James Bottomley wrote: > > > On Sat, 2018-09-08 at 13:34 +0200, Greg KH wrote: > > > > On Sat, Sep 08, 2018 at 08:21:41AM -0300, Mauro Carvalho Chehab > > > > wrote: > > > > > IMHO, the best would be to have a formal/legal way to handle > > > > > it. > > > > > > > > No, sorry, some of us are not allowed legally to sign NDAs for > > > > stuff like this. > > > > > > As a blanket statement this simply isn't true. > > > > Um, I said "some of us". Some of us can, some of us can not. That's > > a blanket statement that has to be true :) > > OK, let me make it more specific: there exists no individual > contributing to open source in a leadership capacity for whom a > signable NDA cannot be crafted. "can be crafted eventually" :) There are language issues, corporate issues, and lots and lots of other issues involved here, you know this. Look at Mauro's situation for one example. Anyway, if the main goal here is to somehow have the LF provide some sort of situation where we can invoke the old "3-way" NDA process to handle security issues, then fine, let's propose that and see if the LF wishes to do this. But remember, this is only needed for the "crazy" issues, like Meltdown. What we put together add-hoc for L1TF worked well, and what we do every week in handling security issues sent to security@k.org works very well also. So well that no one really realizes what we do there :) So again, if this is something that people strongly feel the LF should handle, let the TAB know and they will be glad to work on it. thanks, greg k-h ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 12:51 ` Greg KH @ 2018-09-09 14:20 ` Linus Torvalds 2018-09-09 14:38 ` James Bottomley 2018-09-09 19:41 ` Jiri Kosina 0 siblings, 2 replies; 54+ messages in thread From: Linus Torvalds @ 2018-09-09 14:20 UTC (permalink / raw) To: Greg KH; +Cc: James Bottomley, mchehab+samsung, ksummit On Sun, Sep 9, 2018 at 5:51 AM Greg KH <greg@kroah.com> wrote: > > But remember, this is only needed for the "crazy" issues, like Meltdown. > What we put together add-hoc for L1TF worked well, and what we do every > week in handling security issues sent to security@k.org works very well > also. So well that no one really realizes what we do there :) Note that at some point, we should just say "f*ck it". For hardware bugs, we should remember that *we* aren't the ones that are in trouble. If a hardware company makes it too hard for us to work with them, we should literally say "go the f*ck away" and stop talking to them. It's *their* problem, not ours. If they only work with vendors unable to talk to core maintainers, I guarantee that it will *remain* their problem. I will happily tell the world that the hardware company screwed up and didn't even help us try to fix things right. Their lawyers and PR people can go screw themselves. Seriously. People need to be aware that it's not us that should be bending over backwards over hardware issues. If some hardware company wants an NDA from me for their own screw-ups, I'll laugh in their face, and then I'll tell journalists about how they actively made it harder to fix their mess. Linus ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 14:20 ` Linus Torvalds @ 2018-09-09 14:38 ` James Bottomley 2018-09-09 14:51 ` Andy Lutomirski 2018-09-09 19:41 ` Jiri Kosina 1 sibling, 1 reply; 54+ messages in thread From: James Bottomley @ 2018-09-09 14:38 UTC (permalink / raw) To: Linus Torvalds, Greg KH; +Cc: mchehab+samsung, ksummit On Sun, 2018-09-09 at 07:20 -0700, Linus Torvalds wrote: > On Sun, Sep 9, 2018 at 5:51 AM Greg KH <greg@kroah.com> wrote: > > > > But remember, this is only needed for the "crazy" issues, like > > Meltdown. What we put together add-hoc for L1TF worked well, and > > what we do every week in handling security issues sent to > > security@k.org works very well also. So well that no one really > > realizes what we do there :) > > Note that at some point, we should just say "f*ck it". > > For hardware bugs, we should remember that *we* aren't the ones that > are in trouble. If a hardware company makes it too hard for us to > work with them, we should literally say "go the f*ck away" and stop > talking to them. > > It's *their* problem, not ours. If they only work with vendors > unable to talk to core maintainers, I guarantee that it will *remain* > their problem. I will happily tell the world that the hardware > company screwed up and didn't even help us try to fix things right. > > Their lawyers and PR people can go screw themselves. > > Seriously. People need to be aware that it's not us that should be > bending over backwards over hardware issues. If some hardware company > wants an NDA from me for their own screw-ups, I'll laugh in their > face, and then I'll tell journalists about how they actively made it > harder to fix their mess. So it seems we have the two choices: 1. Conform to industry norms for disclosures and find a way of bringing an NDA framework to Linux Security fix handling 2. Force industry to adopt new norms that actually work well with open source. I think I already hear a majority for number 2. However, to make 2 work we need to use every tool at our disposal to push for change, including our PR relationships and, to be true to that, we really should publish a critique of what went wrong with spectre/meltdown and how it should have gone better. That way we have something to point to when someone asks what to do about the next hardware side channel problem. I'm sure lwn.net would be up for doing something to help with this provided we give them access to the raw material and maintainer interviews so they can present a coherent story rather than a gripe fest (which is what we've mostly got in this thread). James ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 14:38 ` James Bottomley @ 2018-09-09 14:51 ` Andy Lutomirski 2018-09-09 17:20 ` Theodore Y. Ts'o 0 siblings, 1 reply; 54+ messages in thread From: Andy Lutomirski @ 2018-09-09 14:51 UTC (permalink / raw) To: James Bottomley; +Cc: mchehab+samsung, ksummit > On Sep 9, 2018, at 7:38 AM, James Bottomley <James.Bottomley@HansenPartnership.com> wrote: > >> On Sun, 2018-09-09 at 07:20 -0700, Linus Torvalds wrote: >>> On Sun, Sep 9, 2018 at 5:51 AM Greg KH <greg@kroah.com> wrote: >>> >>> But remember, this is only needed for the "crazy" issues, like >>> Meltdown. What we put together add-hoc for L1TF worked well, and >>> what we do every week in handling security issues sent to >>> security@k.org works very well also. So well that no one really >>> realizes what we do there :) >> >> Note that at some point, we should just say "f*ck it". >> >> For hardware bugs, we should remember that *we* aren't the ones that >> are in trouble. If a hardware company makes it too hard for us to >> work with them, we should literally say "go the f*ck away" and stop >> talking to them. >> >> It's *their* problem, not ours. If they only work with vendors >> unable to talk to core maintainers, I guarantee that it will *remain* >> their problem. I will happily tell the world that the hardware >> company screwed up and didn't even help us try to fix things right. >> >> Their lawyers and PR people can go screw themselves. >> >> Seriously. People need to be aware that it's not us that should be >> bending over backwards over hardware issues. If some hardware company >> wants an NDA from me for their own screw-ups, I'll laugh in their >> face, and then I'll tell journalists about how they actively made it >> harder to fix their mess. > > So it seems we have the two choices: > > 1. Conform to industry norms for disclosures and find a way of bringing > an NDA framework to Linux Security fix handling > 2. Force industry to adopt new norms that actually work well with open > source. > Or my proposal of 3: have a policy, get lawyers to agree to it, and make it barely be an NDA. I don’t know how practical it is, but it could be a nice middle ground. > I think I already hear a majority for number 2. > > However, to make 2 work we need to use every tool at our disposal to > push for change, including our PR relationships and, to be true to > that, we really should publish a critique of what went wrong with > spectre/meltdown and how it should have gone better. That way we have > something to point to when someone asks what to do about the next > hardware side channel problem. I'm sure lwn.net would be up for doing > something to help with this provided we give them access to the raw > material and maintainer interviews so they can present a coherent story > rather than a gripe fest (which is what we've mostly got in this > thread). > > James > > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 14:51 ` Andy Lutomirski @ 2018-09-09 17:20 ` Theodore Y. Ts'o 2018-09-09 17:48 ` David Woodhouse 2018-09-09 18:17 ` Andy Lutomirski 0 siblings, 2 replies; 54+ messages in thread From: Theodore Y. Ts'o @ 2018-09-09 17:20 UTC (permalink / raw) To: Andy Lutomirski; +Cc: James Bottomley, mchehab+samsung, ksummit On Sun, Sep 09, 2018 at 07:51:09AM -0700, Andy Lutomirski wrote: > > So it seems we have the two choices: > > > > 1. Conform to industry norms for disclosures and find a way of bringing > > an NDA framework to Linux Security fix handling > > 2. Force industry to adopt new norms that actually work well with open > > source. > > > > Or my proposal of 3: have a policy, get lawyers to agree to it, and > make it barely be an NDA. I don’t know how practical it is, but it > could be a nice middle ground. The only middle ground is "gentleman's agreement". The main problem any legal agreement is what are the teeth if someone violates the NDA and breaks the embargo. The reason why it will be very hard for some third-party, like the LF, to sign any kind of NDA on behalf of independent developers is that it puts the liability risk on the LF. And the LF's lawyers aren't going to be comfortable with this. We've been through this before with the TAB and getting all of the TAB members under an NDA so we could talk about pre-standardized UEFI proposals. We looked at trying to get the LF to sign an NDA for the TAB members who didn't work for companies which had an NDA with UEFI, and it just didn't work. Ultimately, what we did is we negotiated a specific NDA just for me (where it would be my house on the line in terms of an NDA violation), and I then had to get the Google's lawyers to OK my signing it as a personal NDA. The whole process took **months**. If we need to do this when we're in a hurry, by the time the lawyers get over their "we must reduce our client's risk to zero" obsession, the security problem will have leaked, and the embargo and the NDA will be moot. - Ted ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 17:20 ` Theodore Y. Ts'o @ 2018-09-09 17:48 ` David Woodhouse 2018-09-09 18:17 ` Andy Lutomirski 1 sibling, 0 replies; 54+ messages in thread From: David Woodhouse @ 2018-09-09 17:48 UTC (permalink / raw) To: Theodore Y. Ts'o, Andy Lutomirski Cc: James Bottomley, mchehab+samsung, ksummit [-- Attachment #1: Type: text/plain, Size: 1511 bytes --] On Sun, 2018-09-09 at 13:20 -0400, Theodore Y. Ts'o wrote: > > The only middle ground is "gentleman's agreement". The main problem > any legal agreement is what are the teeth if someone violates the NDA > and breaks the embargo. The reason why it will be very hard for some > third-party, like the LF, to sign any kind of NDA on behalf of > independent developers is that it puts the liability risk on the LF. > And the LF's lawyers aren't going to be comfortable with this. > > We've been through this before with the TAB and getting all of the TAB > members under an NDA so we could talk about pre-standardized UEFI > proposals. We looked at trying to get the LF to sign an NDA for the > TAB members who didn't work for companies which had an NDA with UEFI, > and it just didn't work. Ultimately, what we did is we negotiated a > specific NDA just for me (where it would be my house on the line in > terms of an NDA violation), and I then had to get the Google's lawyers > to OK my signing it as a personal NDA. The whole process took > **months**. In practice there's not really a lot of difference between a "real" NDA and a gentlepersons' agreement. Nobody's *really* going to lose their house; all that's likely to happen if you screw up is that they won't include you in the party next time round. We *had* a breach before the Spectre/Meltdown embargo was supposed to be over, and to my knowledge (and I *hope*) nothing actually happened except a bit of tutting. [-- Attachment #2: smime.p7s --] [-- Type: application/x-pkcs7-signature, Size: 5213 bytes --] ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 17:20 ` Theodore Y. Ts'o 2018-09-09 17:48 ` David Woodhouse @ 2018-09-09 18:17 ` Andy Lutomirski 2018-09-09 18:56 ` Theodore Y. Ts'o 1 sibling, 1 reply; 54+ messages in thread From: Andy Lutomirski @ 2018-09-09 18:17 UTC (permalink / raw) To: Theodore Y. Ts'o; +Cc: James Bottomley, mchehab+samsung, ksummit > On Sep 9, 2018, at 10:20 AM, Theodore Y. Ts'o <tytso@mit.edu> wrote: > > On Sun, Sep 09, 2018 at 07:51:09AM -0700, Andy Lutomirski wrote: >>> So it seems we have the two choices: >>> >>> 1. Conform to industry norms for disclosures and find a way of bringing >>> an NDA framework to Linux Security fix handling >>> 2. Force industry to adopt new norms that actually work well with open >>> source. >>> >> >> Or my proposal of 3: have a policy, get lawyers to agree to it, and >> make it barely be an NDA. I don’t know how practical it is, but it >> could be a nice middle ground. > > The only middle ground is "gentleman's agreement". The main problem > any legal agreement is what are the teeth if someone violates the NDA > and breaks the embargo. The reason why it will be very hard for some > third-party, like the LF, to sign any kind of NDA on behalf of > independent developers is that it puts the liability risk on the LF. > And the LF's lawyers aren't going to be comfortable with this. What I want is the opposite of an NDA. I want a gentlemen’s agreement plus an explicit statement that the relevant people *may* talk about the issue among themselves despite any NDAs that might already exist. And that they may release patches when the embargo is up. And that the embargo has an end date, and that the developers may decline an extension. The business people are well aware that even the strongest NDA isn’t worth a damn thing in these scenarios. If Meltdown had leaked in September, then Intel would have had egg all over its face and no amount of lawsuits or injunctions would have helped. In fact, it would have been worse than nothing because now the news would say “Intel failed to permit vendors to fix it *and* the embargo got blown, and rather than doing something about it, they’re suing people.” I personally would have liked such an arrangement for Meltdown. So maybe a business case could be made to vendors that they should sign the thing. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 18:17 ` Andy Lutomirski @ 2018-09-09 18:56 ` Theodore Y. Ts'o 2018-09-09 19:19 ` Andy Lutomirski ` (2 more replies) 0 siblings, 3 replies; 54+ messages in thread From: Theodore Y. Ts'o @ 2018-09-09 18:56 UTC (permalink / raw) To: Andy Lutomirski; +Cc: James Bottomley, mchehab+samsung, ksummit On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote: > > What I want is the opposite of an NDA. I want a gentlemen’s > agreement plus an explicit statement that the relevant people *may* > talk about the issue among themselves despite any NDAs that might > already exist. And that they may release patches when the embargo is > up. And that the embargo has an end date, and that the developers > may decline an extension. So what you're talking about is some kind of "Memo of Understanding" that has no talk about "if this leaks it will Intel will suffer millons and billons and zillons of dollars and Intel well sue you until your assets are a smoking crater in the ground"? If there are no consequences to violating the Gentleman's agreement (other than not being included the next time *when* another CPU vulnerability comes up), then nothing really needs to be signed, since it has no legal impact. I'd certainly support such a thing, but in my view it's really no different from Linus's #2: 2. Force industry to adopt new norms that actually work well with open source. If the MOU with no teeth is enough to save the lawyer's face, that would be great. - Ted ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 18:56 ` Theodore Y. Ts'o @ 2018-09-09 19:19 ` Andy Lutomirski 2018-09-09 20:20 ` Jiri Kosina 2018-09-10 9:25 ` Thomas Gleixner 2 siblings, 0 replies; 54+ messages in thread From: Andy Lutomirski @ 2018-09-09 19:19 UTC (permalink / raw) To: Theodore Y. Ts'o; +Cc: James Bottomley, mchehab+samsung, ksummit > On Sep 9, 2018, at 11:56 AM, Theodore Y. Ts'o <tytso@mit.edu> wrote: > >> On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote: >> >> What I want is the opposite of an NDA. I want a gentlemen’s >> agreement plus an explicit statement that the relevant people *may* >> talk about the issue among themselves despite any NDAs that might >> already exist. And that they may release patches when the embargo is >> up. And that the embargo has an end date, and that the developers >> may decline an extension. > > So what you're talking about is some kind of "Memo of Understanding" > that has no talk about "if this leaks it will Intel will suffer > millons and billons and zillons of dollars and Intel well sue you > until your assets are a smoking crater in the ground"? Yes > > If there are no consequences to violating the Gentleman's agreement > (other than not being included the next time *when* another CPU > vulnerability comes up), then nothing really needs to be signed, since > it has no legal impact. Here I disagree. The consequence to *Intel* for signing needs to be clear. If I’m included, and Intel thinks I leaked it or their attorneys get overzealous and complain that I talked to someone at SUSE or whatever or that I *gasp* published a patch on the day the embargo ended and they sue *me* for zillions under my preexisting, then I want to point to this agreement and say “no, and by suing me you are in breach of this contract”. > > I'd certainly support such a thing, but in my view it's really no > different from Linus's #2: > > 2. Force industry to adopt new norms that actually work well with open > source. > If the MOU with no teeth is enough to save the lawyer's face, that > would be great. > That too. ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 18:56 ` Theodore Y. Ts'o 2018-09-09 19:19 ` Andy Lutomirski @ 2018-09-09 20:20 ` Jiri Kosina 2018-09-09 21:36 ` James Bottomley 2018-09-10 9:25 ` Thomas Gleixner 2 siblings, 1 reply; 54+ messages in thread From: Jiri Kosina @ 2018-09-09 20:20 UTC (permalink / raw) To: Theodore Y. Ts'o; +Cc: James Bottomley, mchehab+samsung, ksummit On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote: > 2. Force industry to adopt new norms that actually work well with open > source. The big quesiton mark here obviously is -- how? I would hope that there are other ways than having the whole thing once fail completely, in order to provide enough of a shock effect. Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 20:20 ` Jiri Kosina @ 2018-09-09 21:36 ` James Bottomley 0 siblings, 0 replies; 54+ messages in thread From: James Bottomley @ 2018-09-09 21:36 UTC (permalink / raw) To: Jiri Kosina, Theodore Y. Ts'o; +Cc: mchehab+samsung, ksummit On Sun, 2018-09-09 at 22:20 +0200, Jiri Kosina wrote: > On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote: > > > 2. Force industry to adopt new norms that actually work well > > with open source. > > The big quesiton mark here obviously is -- how? > > I would hope that there are other ways than having the whole thing > once fail completely, in order to provide enough of a shock effect. Well, as I said in the original email apart from peer pressure, and perhaps some institutional help from the various open source foundations, all we really have at our disposal is the PR factor: if we produce a detailed history and critique of the failures plus a set of suggestions for improvements we have at least begun the conversation. James ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 18:56 ` Theodore Y. Ts'o 2018-09-09 19:19 ` Andy Lutomirski 2018-09-09 20:20 ` Jiri Kosina @ 2018-09-10 9:25 ` Thomas Gleixner 2018-09-10 14:40 ` James Bottomley 2018-09-11 8:20 ` Jiri Kosina 2 siblings, 2 replies; 54+ messages in thread From: Thomas Gleixner @ 2018-09-10 9:25 UTC (permalink / raw) To: Theodore Y. Ts'o; +Cc: James Bottomley, mchehab+samsung, ksummit [-- Attachment #1: Type: text/plain, Size: 2926 bytes --] On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote: > On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote: > > > > What I want is the opposite of an NDA. I want a gentlemen’s > > agreement plus an explicit statement that the relevant people *may* > > talk about the issue among themselves despite any NDAs that might > > already exist. And that they may release patches when the embargo is > > up. And that the embargo has an end date, and that the developers > > may decline an extension. > > So what you're talking about is some kind of "Memo of Understanding" > that has no talk about "if this leaks it will Intel will suffer > millons and billons and zillons of dollars and Intel well sue you > until your assets are a smoking crater in the ground"? > > If there are no consequences to violating the Gentleman's agreement > (other than not being included the next time *when* another CPU > vulnerability comes up), then nothing really needs to be signed, since > it has no legal impact. Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster (which was completely FUBARed by Intel), having something like this in place could have certainly solved the main gap which we had. We were able to communicate freely between the informed parties and their allowed to know kernel developers, even accross vendors. But there was no simple way to bring in anybody else. It tooks us almost 2 months to get GregKH on board, but there was no way to talk to e.g. the BPF folks in time. I think this needs to have some formal setup. The way disclosure to companies work is through coordinators, who then disclose it internaly to the relevant people. We should provide something similar, i.e. an embargo coordination group, which coordinates the issue with the disclosing party. And yes, this only can be based on a general Memo of Understanding, as there is no way to make that whole NDA mess work when the group needs to bring in individual developers. Having something formal and halfways familiar in place is definitely something we need before we are starting to communicate and negotiate that through all channels. What I came up with so far is: - work out a Memo of Understanding - appoint an initial group of embargo coordinators, ideally people who have already an established trust relationship in the industry. - come up with a clear and well defined set of rules what this embargo group is doing and what not. It's sole purpose is to handle and coordinate the kind of embargo issues, which have a long preparation time, need coordination with other OSes etc, i.e. the Meltdown/Spectre/L1TF scenarios. It won't deal with NDAs and has to be free to disclose to individual developers based on trust under the MOU. Creating this kind of formal entity is probably the closest thing to the established inter corporate embargo handling which we can provide. Thanks, tglx ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-10 9:25 ` Thomas Gleixner @ 2018-09-10 14:40 ` James Bottomley 2018-09-11 8:20 ` Jiri Kosina 1 sibling, 0 replies; 54+ messages in thread From: James Bottomley @ 2018-09-10 14:40 UTC (permalink / raw) To: Thomas Gleixner, Theodore Y. Ts'o; +Cc: mchehab+samsung, ksummit On Mon, 2018-09-10 at 11:25 +0200, Thomas Gleixner wrote: > On Sun, 9 Sep 2018, Theodore Y. Ts'o wrote: > > On Sun, Sep 09, 2018 at 11:17:20AM -0700, Andy Lutomirski wrote: > > > > > > What I want is the opposite of an NDA. I want a gentlemen’s > > > agreement plus an explicit statement that the relevant people > > > *may* talk about the issue among themselves despite any NDAs that > > > might already exist. And that they may release patches when the > > > embargo is up. And that the embargo has an end date, and that the > > > developers may decline an extension. > > > > So what you're talking about is some kind of "Memo of > > Understanding" that has no talk about "if this leaks it will Intel > > will suffer millons and billons and zillons of dollars and Intel > > well sue you until your assets are a smoking crater in the ground"? > > > > If there are no consequences to violating the Gentleman's agreement > > (other than not being included the next time *when* another CPU > > vulnerability comes up), then nothing really needs to be signed, > > since it has no legal impact. > > Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster > (which was completely FUBARed by Intel), having something like this > in place could have certainly solved the main gap which we had. We > were able to communicate freely between the informed parties and > their allowed to know kernel developers, even accross vendors. But > there was no simple way to bring in anybody else. It tooks us almost > 2 months to get GregKH on board, but there was no way to talk to e.g. > the BPF folks in time. > > I think this needs to have some formal setup. The way disclosure to > companies work is through coordinators, who then disclose it > internaly to the relevant people. > > We should provide something similar, i.e. an embargo coordination > group, which coordinates the issue with the disclosing party. And > yes, this only can be based on a general Memo of Understanding, as > there is no way to make that whole NDA mess work when the group needs > to bring in individual developers. The good thing about doing this is we can set the rules for onward disclosure from the embargo co-ordination group. We could probably get away with something that said (co-ordinate with required linux kernel subsystem maintainers on a need to know basis) i.e. under our rules we could disclose to a maintainer if they needed to know without an NDA. > Having something formal and halfways familiar in place is definitely > something we need before we are starting to communicate and negotiate > that through all channels. > > What I came up with so far is: > > - work out a Memo of Understanding > > - appoint an initial group of embargo coordinators, ideally people > who have already an established trust relationship in the > industry. > > - come up with a clear and well defined set of rules what this > embargo group is doing and what not. This is the key for better co-ordination. One of the rules should be "take responsibility for determining who needs to know in the Linux Kernel maintainer community and communicating relevant information to them on a strict need to know basis". It can probably be better phrased and we'd need a lawyer to look it over because this is the point at which the NDA gives way to a "gentleman's agreement". James ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-10 9:25 ` Thomas Gleixner 2018-09-10 14:40 ` James Bottomley @ 2018-09-11 8:20 ` Jiri Kosina 2018-09-11 9:03 ` Thomas Gleixner 1 sibling, 1 reply; 54+ messages in thread From: Jiri Kosina @ 2018-09-11 8:20 UTC (permalink / raw) To: Thomas Gleixner; +Cc: James Bottomley, mchehab+samsung, ksummit On Mon, 10 Sep 2018, Thomas Gleixner wrote: > Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster (which > was completely FUBARed by Intel), having something like this in place could > have certainly solved the main gap which we had. We were able to > communicate freely between the informed parties and their allowed to know > kernel developers, even accross vendors. Agreed, this worked pretty well this time. > But there was no simple way to bring in anybody else. It tooks us almost > 2 months to get GregKH on board, but there was no way to talk to e.g. > the BPF folks in time. But this was what has caused real pain indeed. Do we know / can it be publicly said what exactly was the issue in those cases? Was it perhaps that those people were not employed by a company the disclosing party had a NDA in place already (like it probably had with all the involved vendors, etc)? Thanks, -- Jiri Kosina ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-11 8:20 ` Jiri Kosina @ 2018-09-11 9:03 ` Thomas Gleixner 0 siblings, 0 replies; 54+ messages in thread From: Thomas Gleixner @ 2018-09-11 9:03 UTC (permalink / raw) To: Jiri Kosina; +Cc: James Bottomley, mchehab+samsung, ksummit On Tue, 11 Sep 2018, Jiri Kosina wrote: > On Mon, 10 Sep 2018, Thomas Gleixner wrote: > > > Looking at SSBD/L1TF only and ignoring the Meltdown/Spectre disaster (which > > was completely FUBARed by Intel), having something like this in place could > > have certainly solved the main gap which we had. We were able to > > communicate freely between the informed parties and their allowed to know > > kernel developers, even accross vendors. > > Agreed, this worked pretty well this time. > > > But there was no simple way to bring in anybody else. It tooks us almost > > 2 months to get GregKH on board, but there was no way to talk to e.g. > > the BPF folks in time. > > But this was what has caused real pain indeed. Do we know / can it be > publicly said what exactly was the issue in those cases? Was it perhaps > that those people were not employed by a company the disclosing party had > a NDA in place already (like it probably had with all the involved > vendors, etc)? Greg is not employed by one of the companies to which the issue was disclosed and had no NDA in place. AFAIK he's not doing the NDA mess either, so it took time to sort it out. For others, who were not employed by one of the involved parties and were not covered by some magic NDA construct, we just gave up because at the point we realized that we need their expertise it was way too late to wait a couple of month/weeks to get that sorted. That's where I see the embargo coordination to have it's place. If the people working on the embargoed issue need the expertise of somebody who is not covered in one way or the other, then the embargo team can disclose it due to technical necessarity and based on trust. And I can see that working, because if someone gets pulled in and spills the beans then he has not only violated the trust put into him for this particular embargo issue, he also killed his reputation and trust relationships in the community as a whole. I doubt that any coordinator in a company - despite the whole NDA/work contract/whatever stuff in place - will disclose a sensitive embargo issue to an engineer whom he doesn't trust fully. Thanks, tglx ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-09 14:20 ` Linus Torvalds 2018-09-09 14:38 ` James Bottomley @ 2018-09-09 19:41 ` Jiri Kosina 1 sibling, 0 replies; 54+ messages in thread From: Jiri Kosina @ 2018-09-09 19:41 UTC (permalink / raw) To: Linus Torvalds; +Cc: James Bottomley, ksummit, mchehab+samsung On Sun, 9 Sep 2018, Linus Torvalds wrote: > > But remember, this is only needed for the "crazy" issues, like Meltdown. > > What we put together add-hoc for L1TF worked well, and what we do every > > week in handling security issues sent to security@k.org works very well > > also. So well that no one really realizes what we do there :) > > Note that at some point, we should just say "f*ck it". I absolutely agree (even more for stable tree backports). One thing that is absolutely necessary then though (especially if this is just about not doing crazy backport to -stable), to make it absolutely clear to the stable downstreams/consumers, that this is the case. Rationale: turns out it might be even illegal in some countries to release software with known security issue for which the fix exists (yeah, well, whatever that means). So the downstreams really should be made aware, so that we don't put them into an uncomfortable situation and they could adapt somehow. (on a first sight, this might be like buying into what grsecurity folk(s) have been asking for for ages -- that is annotating ideally each and every commit with its security implications -- but that's not the case here; it's quite the opposite: explicitly stating that certain security fix/backport is *not* happening). Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 15:00 ` James Bottomley 2018-09-08 15:32 ` Greg KH @ 2018-09-08 19:26 ` Jiri Kosina 2018-09-08 19:47 ` James Bottomley 1 sibling, 1 reply; 54+ messages in thread From: Jiri Kosina @ 2018-09-08 19:26 UTC (permalink / raw) To: James Bottomley; +Cc: Mauro Carvalho Chehab, ksummit-discuss On Sat, 8 Sep 2018, James Bottomley wrote: > I think we might benefit from a discussion of whether we could have > handled Meltdown/Spectre better in an NDA framework ... Well, at the end of the day, we actually did handle it in the NDA framework (otherwise the end result wrt. stable and distros really wouldn't look like it did by the release date), but everything else (timing, information sharing, feedback channels to the "owners", ...) didn't really work all that well. -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues 2018-09-08 19:26 ` Jiri Kosina @ 2018-09-08 19:47 ` James Bottomley 0 siblings, 0 replies; 54+ messages in thread From: James Bottomley @ 2018-09-08 19:47 UTC (permalink / raw) To: Jiri Kosina; +Cc: Mauro Carvalho Chehab, ksummit-discuss On Sat, 2018-09-08 at 21:26 +0200, Jiri Kosina wrote: > On Sat, 8 Sep 2018, James Bottomley wrote: > > > I think we might benefit from a discussion of whether we could have > > handled Meltdown/Spectre better in an NDA framework ... > > Well, at the end of the day, we actually did handle it in the NDA > framework (otherwise the end result wrt. stable and distros really > wouldn't look like it did by the release date), but everything else > (timing, information sharing, feedback channels to the "owners", > ...) didn't really work all that well. Right, but my impression was the NDAs came late in the game. I think what happened early is we got an -ENONDA from the various lawyers involved which lead them to cut most of us out of the handling loop. Eventually the distros had to sign NDAs to get back in. Now this situation is undoubtedly due to lack of open source awareness among the various disclosing parties, but it does beg the question if we'd had an NDA framework in place (constructed by someone who knows what they're doing in open source) and we'd been able to say: "sure, this is the way we handle NDA issues in open source, sign here" we might have been in the loop much earlier. James ^ permalink raw reply [flat|nested] 54+ messages in thread
end of thread, other threads:[~2018-09-12 7:03 UTC | newest] Thread overview: 54+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-06 19:18 [Ksummit-discuss] [MAINTAINERS SUMMIT] Handling of embargoed security issues Jiri Kosina 2018-09-06 20:56 ` Linus Torvalds 2018-09-06 21:14 ` Jiri Kosina 2018-09-06 22:51 ` Eduardo Valentin 2018-09-07 9:17 ` Jani Nikula 2018-09-07 14:43 ` David Woodhouse 2018-09-06 22:55 ` Eduardo Valentin 2018-09-07 8:21 ` Geert Uytterhoeven 2018-09-10 23:26 ` Eduardo Valentin 2018-09-11 8:45 ` Greg KH 2018-09-11 17:10 ` Dave Hansen 2018-09-11 18:28 ` Greg KH 2018-09-11 18:44 ` Thomas Gleixner 2018-09-07 13:30 ` Jiri Kosina 2018-09-09 12:55 ` Greg KH 2018-09-09 19:48 ` Jiri Kosina 2018-09-10 4:04 ` Eduardo Valentin 2018-09-12 7:03 ` Greg KH 2018-09-10 4:12 ` Eduardo Valentin 2018-09-10 11:10 ` Mark Brown 2018-09-12 4:22 ` Balbir Singh 2018-09-08 4:21 ` Andy Lutomirski 2018-09-08 8:56 ` Thomas Gleixner 2018-09-08 11:21 ` Mauro Carvalho Chehab 2018-09-08 11:34 ` Greg KH 2018-09-08 14:20 ` Andy Lutomirski 2018-09-08 15:29 ` Greg KH 2018-09-08 15:00 ` James Bottomley 2018-09-08 15:32 ` Greg KH 2018-09-08 15:54 ` James Bottomley 2018-09-08 19:49 ` Linus Torvalds 2018-09-08 21:24 ` James Bottomley 2018-09-08 22:33 ` Andy Lutomirski 2018-09-09 12:18 ` Mauro Carvalho Chehab 2018-09-10 22:59 ` Dave Hansen 2018-09-11 8:48 ` Greg KH 2018-09-09 12:51 ` Greg KH 2018-09-09 14:20 ` Linus Torvalds 2018-09-09 14:38 ` James Bottomley 2018-09-09 14:51 ` Andy Lutomirski 2018-09-09 17:20 ` Theodore Y. Ts'o 2018-09-09 17:48 ` David Woodhouse 2018-09-09 18:17 ` Andy Lutomirski 2018-09-09 18:56 ` Theodore Y. Ts'o 2018-09-09 19:19 ` Andy Lutomirski 2018-09-09 20:20 ` Jiri Kosina 2018-09-09 21:36 ` James Bottomley 2018-09-10 9:25 ` Thomas Gleixner 2018-09-10 14:40 ` James Bottomley 2018-09-11 8:20 ` Jiri Kosina 2018-09-11 9:03 ` Thomas Gleixner 2018-09-09 19:41 ` Jiri Kosina 2018-09-08 19:26 ` Jiri Kosina 2018-09-08 19:47 ` James Bottomley
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.