* [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ @ 2023-08-15 9:28 Jiri Kosina 2023-08-15 10:17 ` Vegard Nossum 0 siblings, 1 reply; 29+ messages in thread From: Jiri Kosina @ 2023-08-15 9:28 UTC (permalink / raw) To: ksummit Hi, I believe that reporters of embargoed security issues have always been confused about reporting to security@kernel.org vs. reporting to linux-distros@, as those two lists have completely different ways of dealing with the report (different purpose, different deadlines, different obligations imposed on the reporters, etc). Our documentation originally suggested reporting to both in some "hybrid" mode, and the results were not great, quite often leading to confusion left and right. This led to a slight update to our documentation [1], where reporters are discouraged from reporting kernel issues to linux-distros@ ever. While I generally agree with the change *now*, given the current conditions, I would like to bring this up for discussion on how to deal with this longer term. With my distro hat on, I really want the kernel security bugs to be *eventually* processed through linux-distros@ somehow, for one sole reason: it means that our distro security people will be aware of it, track it internally, and keep an eye on the fix being ported to all of our codestreams. This is exactly how this is done for all other packages. I would be curious to hear about this from other distros, but I sort of expect that they would agree. If this process doesn't happen, many kernel security (with CVE potentially eventually assigned retroactively) fixes will go by unnoticed, as distro kernel people will never be explicitly made aware of them, and distros will be missing many of the patches. Which I don't think is a desired end effect. I have been discussing this with Greg already some time ago, and I know that his response to this is "then use -stable, and you'll get everything automatically" (which is obviously true, because stable is represented at security@kernel.org), but: - Neither us (nor RedHat, nor Ubuntu, as far as I am aware) are picking stable as a primary base for distro kernels. There are many reasons for this (lifecycles not matching, stable picking up way too many things for taste of some of us, etc), but that's probably slightly off-topic for this discussion - For several varying reasons, our security people really struggle with ensuring that whenever CVE is published, we as a distro can guarantee, that fix for that particular CVE is included. linux-distros@ provides that connection between bugfix and CVE report, and that is lost if the fix goes only through security@kernel.org And yes, I hate the whole CVE thing with passion, but it unfortunately exists and is seen as an industry standard by many. And with us not being able to systematically / automatically guarantee that fix for particulart CVE is included, Linux will be not allowed into many places. I am currently not sure what exactly would be the solution to this. One thing to try would of course be to discuss with linux-distros@ people whether they'd be willing to adjust their rules to fit our needs better; but before that happens, we should be ourselves clear on what our needs towards them actually are. Another option might be to ensure representation of distros at security@kernel.org, but that would completely change the purpose of it, and I don't think that's desired. ... ? [1] https://git.kernel.org/linus/4fee0915e649b Thanks, -- Jiri Kosina Director, SUSE Labs Core ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 9:28 [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ Jiri Kosina @ 2023-08-15 10:17 ` Vegard Nossum 2023-08-15 10:34 ` Jiri Kosina ` (2 more replies) 0 siblings, 3 replies; 29+ messages in thread From: Vegard Nossum @ 2023-08-15 10:17 UTC (permalink / raw) To: Jiri Kosina, ksummit (Sending this with a few extra people in Bcc so they'll see it without getting spammed with replies if they don't want them.) On 8/15/23 11:28, Jiri Kosina wrote: > Hi, > > I believe that reporters of embargoed security issues have always been > confused about reporting to security@kernel.org vs. reporting to > linux-distros@, as those two lists have completely different ways of > dealing with the report (different purpose, different deadlines, different > obligations imposed on the reporters, etc). > > Our documentation originally suggested reporting to both in some "hybrid" > mode, and the results were not great, quite often leading to confusion > left and right. > > This led to a slight update to our documentation [1], where reporters are > discouraged from reporting kernel issues to linux-distros@ ever. > > While I generally agree with the change *now*, given the current > conditions, I would like to bring this up for discussion on how to deal > with this longer term. Sorry to toot this particular horn all the time, but before this update to the security documentation I had also submitted patches to update it to be much clearer and more explicit about the s@k.o and linux-distros distinction: https://lore.kernel.org/all/20230305220010.20895-1-vegard.nossum@oracle.com/ The main objection was probably this response from Greg KH, but I had the impression he was (at the time) not totally against wording things in this direction at some point: https://lore.kernel.org/all/ZAWB5kwcG9IpWvE%2F@kroah.com/ I think it's worth pointing out that the latest update to the documentation (where reporters are discouraged from reporting kernel issues to linux-distros@ ever) came just after a private discussion (on both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where Linus in particular came out hard against the linux-distros policy, in particular the requirement to disclose PoCs if those were shared with the list in the first place. I therefore think that Linus himself needs to be involved in this discussion and that his arguments need to be made in public instead of just paraphrased by me. > With my distro hat on, I really want the kernel security bugs to be > *eventually* processed through linux-distros@ somehow, for one sole > reason: it means that our distro security people will be aware of it, > track it internally, and keep an eye on the fix being ported to all of our > codestreams. This is exactly how this is done for all other packages. > > I would be curious to hear about this from other distros, but I sort of > expect that they would agree. +1 from me; the distros list has been an extremely important resource for distros in order to get fixes shipped out with minimal delay. I keep saying this as well: almost all users get their kernels through distros. Very few end users build their own kernels from source; in other words, it's not enough that a fix has been committed to mainline/stable git to consider it fixed. The connection between upstream git and end users is clearly the distros, so distros should be in the loop _at some point_. > If this process doesn't happen, many kernel security (with CVE potentially > eventually assigned retroactively) fixes will go by unnoticed, as distro > kernel people will never be explicitly made aware of them, and distros > will be missing many of the patches. Which I don't think is a desired end > effect. > > I have been discussing this with Greg already some time ago, and I know > that his response to this is "then use -stable, and you'll get everything > automatically" (which is obviously true, because stable is represented at > security@kernel.org), but: > > - Neither us (nor RedHat, nor Ubuntu, as far as I am aware) are picking > stable as a primary base for distro kernels. There are many reasons for > this (lifecycles not matching, stable picking up way too many things for > taste of some of us, etc), but that's probably slightly off-topic for > this discussion > > - For several varying reasons, our security people really struggle with > ensuring that whenever CVE is published, we as a distro can guarantee, > that fix for that particular CVE is included. linux-distros@ provides > that connection between bugfix and CVE report, and that is lost if the > fix goes only through security@kernel.org > > And yes, I hate the whole CVE thing with passion, but it unfortunately > exists and is seen as an industry standard by many. And with us not > being able to systematically / automatically guarantee that fix for > particulart CVE is included, Linux will be not allowed into many places. > > I am currently not sure what exactly would be the solution to this. > > One thing to try would of course be to discuss with linux-distros@ people > whether they'd be willing to adjust their rules to fit our needs better; > but before that happens, we should be ourselves clear on what our needs > towards them actually are. > > Another option might be to ensure representation of distros at > security@kernel.org, but that would completely change the purpose of it, > and I don't think that's desired. > > ... ? I'll throw in another idea: distros@kernel.org. A closed list which will be notified by security@kernel.org once they feel patches for a particular issue are ready for testing/consumption by distros (and hopefully before the issue is disclosed publicly, if the reporter still wishes to do that). The members and list rules would be totally up to the security team to decide. > > [1] https://git.kernel.org/linus/4fee0915e649b > > Thanks, > Vegard ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 10:17 ` Vegard Nossum @ 2023-08-15 10:34 ` Jiri Kosina 2023-08-15 11:23 ` Greg KH 2023-08-16 15:26 ` Solar Designer 2 siblings, 0 replies; 29+ messages in thread From: Jiri Kosina @ 2023-08-15 10:34 UTC (permalink / raw) To: Vegard Nossum; +Cc: ksummit On Tue, 15 Aug 2023, Vegard Nossum wrote: > I think it's worth pointing out that the latest update to the > documentation (where reporters are discouraged from reporting kernel > issues to linux-distros@ ever) came just after a private discussion (on > both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where > Linus in particular came out hard against the linux-distros policy, in > particular the requirement to disclose PoCs if those were shared with > the list in the first place. I therefore think that Linus himself needs > to be involved in this discussion and that his arguments need to be made > in public instead of just paraphrased by me. And I acutally agree with Linus there -- I see no huge value with POCs being shared there. I know the linux-distros@ people would probably argue that POCs are required so that they can integrate them into their QA / CI / regression testing pipeline, but, quite honestly, I don't believe this is happening in reality anyway. > > With my distro hat on, I really want the kernel security bugs to be > > *eventually* processed through linux-distros@ somehow, for one sole > > reason: it means that our distro security people will be aware of it, > > track it internally, and keep an eye on the fix being ported to all of our > > codestreams. This is exactly how this is done for all other packages. > > > > I would be curious to hear about this from other distros, but I sort of > > expect that they would agree. > > +1 from me; the distros list has been an extremely important resource > for distros in order to get fixes shipped out with minimal delay. > > I keep saying this as well: almost all users get their kernels through > distros. Very few end users build their own kernels from source; in > other words, it's not enough that a fix has been committed to > mainline/stable git to consider it fixed. The connection between > upstream git and end users is clearly the distros, so distros should be > in the loop _at some point_. Thanks for underlying exactly my point here. > I'll throw in another idea: distros@kernel.org. > > A closed list which will be notified by security@kernel.org once they > feel patches for a particular issue are ready for testing/consumption by > distros (and hopefully before the issue is disclosed publicly, if the > reporter still wishes to do that). > > The members and list rules would be totally up to the security team to > decide. That might be a viable option as well. It'll be a little bit inconvenient for the distro security people to follow different sets of rules, but still better than current situation. They'll probably also want the "assign a CVE" process that currently works on linux-distros@, but that's doable. I can ask SUSE security people what they'd think about such an idea, input from others would also be welcome. (and maybe, if the distros@ process really gets established, linux-distros@ will finally realize that they'd better adjust their process to accomodate kernel :) ). Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 10:17 ` Vegard Nossum 2023-08-15 10:34 ` Jiri Kosina @ 2023-08-15 11:23 ` Greg KH 2023-08-15 12:42 ` Steven Rostedt 2023-08-16 15:26 ` Solar Designer 2 siblings, 1 reply; 29+ messages in thread From: Greg KH @ 2023-08-15 11:23 UTC (permalink / raw) To: Vegard Nossum; +Cc: Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote: > I'll throw in another idea: distros@kernel.org. > > A closed list which will be notified by security@kernel.org once they > feel patches for a particular issue are ready for testing/consumption by > distros (and hopefully before the issue is disclosed publicly, if the > reporter still wishes to do that). > > The members and list rules would be totally up to the security team to > decide. As per the lawyers, and government officials we have worked with in the past, having a closed list for preannouncements like this will be either: - deemed illegal in some countries - made to have all "major"[1] Linux users on it. Neither of which actually will work out at all, the whole "preannouncement" stuff just is not possible, sorry. I'm amazed that other projects have been able to "get away with it" for as long as they have without either being infiltrated by "the powers that be" or shutdown yet. Politics is a rough game, the only way to survive is to not play it for stuff like this. So no, "distros@k.o" isn't going to be possible for the LF to host, and any other group that wants to run such a thing will quickly have these issues as well, it's amazing that linux-distros has been able to survive for as long as it has. greg k-h [1] "Major" includes most government agencies in most countries. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 11:23 ` Greg KH @ 2023-08-15 12:42 ` Steven Rostedt 2023-08-15 13:17 ` Daniel Borkmann ` (2 more replies) 0 siblings, 3 replies; 29+ messages in thread From: Steven Rostedt @ 2023-08-15 12:42 UTC (permalink / raw) To: Greg KH; +Cc: Vegard Nossum, Jiri Kosina, ksummit On Tue, 15 Aug 2023 13:23:36 +0200 Greg KH <gregkh@linuxfoundation.org> wrote: > On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote: > > I'll throw in another idea: distros@kernel.org. > > > > A closed list which will be notified by security@kernel.org once they > > feel patches for a particular issue are ready for testing/consumption by > > distros (and hopefully before the issue is disclosed publicly, if the > > reporter still wishes to do that). > > > > The members and list rules would be totally up to the security team to > > decide. > > As per the lawyers, and government officials we have worked with in the > past, having a closed list for preannouncements like this will be > either: I guess the question is, what "preannouncements" are the lawyers worried about? It looks like Jiri's concern is just to make sure that distros have security patches in. Would just a list of "here's the commits that fix security issues" be considered a preannouncement, without having any reference to *what* security issue they fix? It would basically be a subset of the stable tree. That is, anything that came from security@k.o, and does not include all the AUTOSEL and other minor fixes that stable pulls in. > > - deemed illegal in some countries > - made to have all "major"[1] Linux users on it. And if this list only has a list of patches that are already fixed, how dangerous is it to not be 100% closed? I mean, it was pretty obvious that the huge churn with spectre/meltdown patches that were being added to the kernel at the late stage of an -rc release was fixing "something big". > > Neither of which actually will work out at all, the whole > "preannouncement" stuff just is not possible, sorry. I'm amazed that > other projects have been able to "get away with it" for as long as they > have without either being infiltrated by "the powers that be" or > shutdown yet. Have there been lists shutdown by the powers that be already? Or is this just a threat made by the power that be? > > Politics is a rough game, the only way to survive is to not play it for > stuff like this. > > So no, "distros@k.o" isn't going to be possible for the LF to host, and > any other group that wants to run such a thing will quickly have these > issues as well, it's amazing that linux-distros has been able to survive > for as long as it has. I'll have to talk with some laywers, as I'm curious to what would be considered illegal about linux-distros. Are you aware of any government specific laws I could go read? I'm not a lawyer, but I've read quite a bit of laws that I can usually get an idea of the problem for my own references (and my experience is that lawyers don't even know until something is settled in court). Note, linux-distros is not about "Linux users" but "Linux distributors". They are not end users but are distributing a product (and having to follow all the rules of the GPL and such to do so). They are not users, they are part of a supply chain. How is security@kernel.org different? If the bug is in the kernel, then the bug is in the distro. Perhaps if we find a bug in the kernel, we should send it to the distro we are using, and not to the Linux community? As Jiri said, most people use a distro kernel, and not the mainline or even the stable one. Really, if you think about it. The closest product to the user is the distro. If someone finds a bug in the kernel, they can see if they can exploit a distro with it. If they can, perhaps they should send it to the security folks of the distro first. Then the distro can send it to security@kernel.org. Maybe this already happens? -- Steve > > greg k-h > > [1] "Major" includes most government agencies in most countries. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 12:42 ` Steven Rostedt @ 2023-08-15 13:17 ` Daniel Borkmann 2023-08-15 14:19 ` Laurent Pinchart 2023-08-15 22:04 ` Jiri Kosina 2023-08-15 14:20 ` Catalin Marinas 2023-08-15 15:08 ` Greg KH 2 siblings, 2 replies; 29+ messages in thread From: Daniel Borkmann @ 2023-08-15 13:17 UTC (permalink / raw) To: Steven Rostedt, Greg KH; +Cc: Vegard Nossum, Jiri Kosina, ksummit On 8/15/23 2:42 PM, Steven Rostedt wrote: > On Tue, 15 Aug 2023 13:23:36 +0200 > Greg KH <gregkh@linuxfoundation.org> wrote: >> On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote: >>> I'll throw in another idea: distros@kernel.org. >>> >>> A closed list which will be notified by security@kernel.org once they >>> feel patches for a particular issue are ready for testing/consumption by >>> distros (and hopefully before the issue is disclosed publicly, if the >>> reporter still wishes to do that). >>> >>> The members and list rules would be totally up to the security team to >>> decide. >> >> As per the lawyers, and government officials we have worked with in the >> past, having a closed list for preannouncements like this will be >> either: > > I guess the question is, what "preannouncements" are the lawyers worried about? > > It looks like Jiri's concern is just to make sure that distros have > security patches in. Would just a list of "here's the commits that fix > security issues" be considered a preannouncement, without having any > reference to *what* security issue they fix? It would basically be a subset > of the stable tree. That is, anything that came from security@k.o, and does > not include all the AUTOSEL and other minor fixes that stable pulls in. Not really answering your question, but the above is also a somewhat misleading "assurance" for distros. Some security fixes might potentially have come in via AUTOSEL, and some others might not have been discussed at security@k.o in the first place. How would you classify which ones of the whole set are relevant for distros and which ones are not? Imho, if distros decide to maintain their own trees outside of stable it would still require manual triaging of the set of stable patches that went into a minor release (and if in doubt, just cherry-pick the patch) - that's just the cost to pay for maintaining non-stable base. It's the same issue as discussed in [0]. [0] https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/ >> - deemed illegal in some countries >> - made to have all "major"[1] Linux users on it. > > And if this list only has a list of patches that are already fixed, how > dangerous is it to not be 100% closed? > > I mean, it was pretty obvious that the huge churn with spectre/meltdown > patches that were being added to the kernel at the late stage of an -rc > release was fixing "something big". > >> Neither of which actually will work out at all, the whole >> "preannouncement" stuff just is not possible, sorry. I'm amazed that >> other projects have been able to "get away with it" for as long as they >> have without either being infiltrated by "the powers that be" or >> shutdown yet. > > Have there been lists shutdown by the powers that be already? Or is this > just a threat made by the power that be? > >> Politics is a rough game, the only way to survive is to not play it for >> stuff like this. >> >> So no, "distros@k.o" isn't going to be possible for the LF to host, and >> any other group that wants to run such a thing will quickly have these >> issues as well, it's amazing that linux-distros has been able to survive >> for as long as it has. > > I'll have to talk with some laywers, as I'm curious to what would be > considered illegal about linux-distros. Are you aware of any government > specific laws I could go read? I'm not a lawyer, but I've read quite a bit > of laws that I can usually get an idea of the problem for my own > references (and my experience is that lawyers don't even know until > something is settled in court). > > Note, linux-distros is not about "Linux users" but "Linux distributors". > They are not end users but are distributing a product (and having to follow > all the rules of the GPL and such to do so). They are not users, they are > part of a supply chain. > > How is security@kernel.org different? If the bug is in the kernel, then the > bug is in the distro. Perhaps if we find a bug in the kernel, we should > send it to the distro we are using, and not to the Linux community? As Jiri > said, most people use a distro kernel, and not the mainline or even the > stable one. Really, if you think about it. The closest product to the user > is the distro. If someone finds a bug in the kernel, they can see if they > can exploit a distro with it. If they can, perhaps they should send it to > the security folks of the distro first. Then the distro can send it to > security@kernel.org. Maybe this already happens? I mainly just wanted to chime in on this one and mention that from my past experience (at least I've seen it couple of times from RH/Canonical) this will not work overly well. We had seen users reporting kernel security bugs there and they were stuck in the security team's triage/bug queue for 3+ months before someone got in contact with upstream. :-( Presumably the teams were overloaded when it happened, or the bugs were misclassified due to lack of domain specific knowledge or they wanted to fix it themselves but didn't get to it yet. Had they been reported to s@k.org, then the relevant maintainers/developers could have been pulled in and it would have been fixed upstream possibly the same day it got reported. So my biggest concern with reporting to distro first is that "things get stuck in the process", unfortunately. The less additional/artificial hops, the better, imho. Cheers, Daniel > -- Steve > >> >> greg k-h >> >> [1] "Major" includes most government agencies in most countries. > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 13:17 ` Daniel Borkmann @ 2023-08-15 14:19 ` Laurent Pinchart 2023-08-15 22:04 ` Jiri Kosina 1 sibling, 0 replies; 29+ messages in thread From: Laurent Pinchart @ 2023-08-15 14:19 UTC (permalink / raw) To: Daniel Borkmann Cc: Steven Rostedt, Greg KH, Vegard Nossum, Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 03:17:00PM +0200, Daniel Borkmann wrote: > On 8/15/23 2:42 PM, Steven Rostedt wrote: > > On Tue, 15 Aug 2023 13:23:36 +0200 Greg KH wrote: > >> On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote: > >>> I'll throw in another idea: distros@kernel.org. > >>> > >>> A closed list which will be notified by security@kernel.org once they > >>> feel patches for a particular issue are ready for testing/consumption by > >>> distros (and hopefully before the issue is disclosed publicly, if the > >>> reporter still wishes to do that). > >>> > >>> The members and list rules would be totally up to the security team to > >>> decide. > >> > >> As per the lawyers, and government officials we have worked with in the > >> past, having a closed list for preannouncements like this will be > >> either: > > > > I guess the question is, what "preannouncements" are the lawyers worried about? > > > > It looks like Jiri's concern is just to make sure that distros have > > security patches in. Would just a list of "here's the commits that fix > > security issues" be considered a preannouncement, without having any > > reference to *what* security issue they fix? It would basically be a subset > > of the stable tree. That is, anything that came from security@k.o, and does > > not include all the AUTOSEL and other minor fixes that stable pulls in. > > Not really answering your question, but the above is also a somewhat misleading > "assurance" for distros. Some security fixes might potentially have come in via > AUTOSEL, and some others might not have been discussed at security@k.o in the > first place. How would you classify which ones of the whole set are relevant > for distros and which ones are not? *LOTS* of fixes for driver bugs that can cause security-related issues are never reported to security@k.o. The author of the bug fix may not even realize that a security issue is fixed. Those issues may have a narrower area of impact individually as one would need to run Linux on specific devices, but collectively they should be considered as important as other security problems. > Imho, if distros decide to maintain their own trees outside of stable it would > still require manual triaging of the set of stable patches that went into a minor > release (and if in doubt, just cherry-pick the patch) - that's just the cost to > pay for maintaining non-stable base. It's the same issue as discussed in [0]. > > [0] https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/ > > >> - deemed illegal in some countries > >> - made to have all "major"[1] Linux users on it. > > > > And if this list only has a list of patches that are already fixed, how > > dangerous is it to not be 100% closed? > > > > I mean, it was pretty obvious that the huge churn with spectre/meltdown > > patches that were being added to the kernel at the late stage of an -rc > > release was fixing "something big". > > > >> Neither of which actually will work out at all, the whole > >> "preannouncement" stuff just is not possible, sorry. I'm amazed that > >> other projects have been able to "get away with it" for as long as they > >> have without either being infiltrated by "the powers that be" or > >> shutdown yet. > > > > Have there been lists shutdown by the powers that be already? Or is this > > just a threat made by the power that be? > > > >> Politics is a rough game, the only way to survive is to not play it for > >> stuff like this. > >> > >> So no, "distros@k.o" isn't going to be possible for the LF to host, and > >> any other group that wants to run such a thing will quickly have these > >> issues as well, it's amazing that linux-distros has been able to survive > >> for as long as it has. > > > > I'll have to talk with some laywers, as I'm curious to what would be > > considered illegal about linux-distros. Are you aware of any government > > specific laws I could go read? I'm not a lawyer, but I've read quite a bit > > of laws that I can usually get an idea of the problem for my own > > references (and my experience is that lawyers don't even know until > > something is settled in court). > > > > Note, linux-distros is not about "Linux users" but "Linux distributors". > > They are not end users but are distributing a product (and having to follow > > all the rules of the GPL and such to do so). They are not users, they are > > part of a supply chain. > > > > How is security@kernel.org different? If the bug is in the kernel, then the > > bug is in the distro. Perhaps if we find a bug in the kernel, we should > > send it to the distro we are using, and not to the Linux community? As Jiri > > said, most people use a distro kernel, and not the mainline or even the > > stable one. Really, if you think about it. The closest product to the user > > is the distro. If someone finds a bug in the kernel, they can see if they > > can exploit a distro with it. If they can, perhaps they should send it to > > the security folks of the distro first. Then the distro can send it to > > security@kernel.org. Maybe this already happens? > > I mainly just wanted to chime in on this one and mention that from my past > experience (at least I've seen it couple of times from RH/Canonical) this > will not work overly well. > > We had seen users reporting kernel security bugs there and they were stuck > in the security team's triage/bug queue for 3+ months before someone got in > contact with upstream. :-( > > Presumably the teams were overloaded when it happened, or the bugs were > misclassified due to lack of domain specific knowledge or they wanted to fix > it themselves but didn't get to it yet. > > Had they been reported to s@k.org, then the relevant maintainers/developers > could have been pulled in and it would have been fixed upstream possibly the > same day it got reported. > > So my biggest concern with reporting to distro first is that "things get stuck > in the process", unfortunately. The less additional/artificial hops, the better, > imho. > > >> [1] "Major" includes most government agencies in most countries. -- Regards, Laurent Pinchart ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 13:17 ` Daniel Borkmann 2023-08-15 14:19 ` Laurent Pinchart @ 2023-08-15 22:04 ` Jiri Kosina 1 sibling, 0 replies; 29+ messages in thread From: Jiri Kosina @ 2023-08-15 22:04 UTC (permalink / raw) To: Daniel Borkmann; +Cc: Steven Rostedt, Greg KH, Vegard Nossum, ksummit On Tue, 15 Aug 2023, Daniel Borkmann wrote: > Not really answering your question, but the above is also a somewhat > misleading "assurance" for distros. Some security fixes might > potentially have come in via AUTOSEL, and some others might not have > been discussed at security@k.o in the first place. How would you > classify which ones of the whole set are relevant for distros and which > ones are not? > > Imho, if distros decide to maintain their own trees outside of stable it > would still require manual triaging of the set of stable patches that > went into a minor release (and if in doubt, just cherry-pick the patch) > - that's just the cost to pay for maintaining non-stable base. It's the > same issue as discussed in [0]. > > [0] > https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/ Whith my kernel developer hat on, I fully agree. The whole CVE handling has been a disaster for many reasons, and it's sad that people still believe in it. Fixes are fixes, and you ideally want them all, right? With my distro kernel person hat on, the linux-distros@ process helps us a lot with a first iteration of identifying the big set of things that we (well, our security team) should take a look into immediately, because they have been explicitly reported (and verified) to be security relevant. It has been the case with pretty much every opensource/Linux-relevant project out there. Kernel has now removed itself from it. Again, from very good and understandable reasons for now, but I don't think that should mark the ultimate end of the story. That doesn't mean that we don't have a process for evaluating what to merge/cherry-pick from other trees (including stable, of course). We absolutely do. We just can't relistically base on -stable, for multitude of reasons. Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 12:42 ` Steven Rostedt 2023-08-15 13:17 ` Daniel Borkmann @ 2023-08-15 14:20 ` Catalin Marinas 2023-08-15 14:41 ` Greg KH 2023-08-15 15:08 ` Greg KH 2 siblings, 1 reply; 29+ messages in thread From: Catalin Marinas @ 2023-08-15 14:20 UTC (permalink / raw) To: Steven Rostedt; +Cc: Greg KH, Vegard Nossum, Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 08:42:53AM -0400, Steven Rostedt wrote: > On Tue, 15 Aug 2023 13:23:36 +0200 > Greg KH <gregkh@linuxfoundation.org> wrote: > > Politics is a rough game, the only way to survive is to not play it for > > stuff like this. > > > > So no, "distros@k.o" isn't going to be possible for the LF to host, and > > any other group that wants to run such a thing will quickly have these > > issues as well, it's amazing that linux-distros has been able to survive > > for as long as it has. > > I'll have to talk with some laywers, as I'm curious to what would be > considered illegal about linux-distros. Are you aware of any government > specific laws I could go read? I'm not a lawyer, but I've read quite a bit > of laws that I can usually get an idea of the problem for my own > references (and my experience is that lawyers don't even know until > something is settled in court). One thing that comes to mind is the hosting location of openwall.org. For some reason my employer decided to block access to it, though apparently one-way emails to linux-distros still work. I can see some politicians wondering why one sends security pre-announcements to a sanctioned country. For this reason, I'd much prefer an equivalent hosted by the LF but the foundation may not want to be in a position to police who's on that list, what qualifies as a Linux distro, which country they come from, potentially removing them based on the geopolitical situation of the day. I guess security@kernel.org can be easier to justify as a closed forum for fixing the actual bug with the aim to release the fix to the world as soon as possible. But yeah, from the disclosure perspective, I don't see much difference other than fewer people (well, the LF could ask for US gov security clearance to be on this list ;)). FWIW, Arm does want some official pre-announcement mechanism for kernel bug-fixes with severe security implications. Some of our partners (especially those with large cloud deployments or distro vendors providing them with software) need a bit of time to roll out a fix and consider that the public disclosure via the kernel stable/linus trees is too late, pretty much a zero-day for them. So far we pointed them at linux-distros but there is a growing pressure for private disclosure mechanisms (only for reports originating from Arm). Maybe your idea of first reporting to distro security teams is not that bad. -- Catalin ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 14:20 ` Catalin Marinas @ 2023-08-15 14:41 ` Greg KH 2023-08-15 15:04 ` Steven Rostedt 0 siblings, 1 reply; 29+ messages in thread From: Greg KH @ 2023-08-15 14:41 UTC (permalink / raw) To: Catalin Marinas; +Cc: Steven Rostedt, Vegard Nossum, Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 03:20:00PM +0100, Catalin Marinas wrote: > On Tue, Aug 15, 2023 at 08:42:53AM -0400, Steven Rostedt wrote: > > On Tue, 15 Aug 2023 13:23:36 +0200 > > Greg KH <gregkh@linuxfoundation.org> wrote: > > > Politics is a rough game, the only way to survive is to not play it for > > > stuff like this. > > > > > > So no, "distros@k.o" isn't going to be possible for the LF to host, and > > > any other group that wants to run such a thing will quickly have these > > > issues as well, it's amazing that linux-distros has been able to survive > > > for as long as it has. > > > > I'll have to talk with some laywers, as I'm curious to what would be > > considered illegal about linux-distros. Are you aware of any government > > specific laws I could go read? I'm not a lawyer, but I've read quite a bit > > of laws that I can usually get an idea of the problem for my own > > references (and my experience is that lawyers don't even know until > > something is settled in court). > > One thing that comes to mind is the hosting location of openwall.org. > For some reason my employer decided to block access to it, though > apparently one-way emails to linux-distros still work. I can see some > politicians wondering why one sends security pre-announcements to a > sanctioned country. For this reason, I'd much prefer an equivalent > hosted by the LF but the foundation may not want to be in a position to > police who's on that list, what qualifies as a Linux distro, which > country they come from, potentially removing them based on the > geopolitical situation of the day. Lots of people want the LF to do such a thing, and last I heard, there were some people potentially working on this but I do not know the status of it, sorry. Try asking the owners of openwall.org if you are curious, they should know the current state. Note, this is independant of the kernel, hence my not wanting to get involved in it at all. > I guess security@kernel.org can be easier to justify as a closed forum > for fixing the actual bug with the aim to release the fix to the world > as soon as possible. But yeah, from the disclosure perspective, I don't > see much difference other than fewer people (well, the LF could ask for > US gov security clearance to be on this list ;)). > > FWIW, Arm does want some official pre-announcement mechanism for > kernel bug-fixes with severe security implications. Some of our partners > (especially those with large cloud deployments or distro vendors > providing them with software) need a bit of time to roll out a fix and > consider that the public disclosure via the kernel stable/linus trees is > too late, pretty much a zero-day for them. So far we pointed them at > linux-distros but there is a growing pressure for private disclosure > mechanisms (only for reports originating from Arm). Maybe your idea of > first reporting to distro security teams is not that bad. Loads of companies/governments have been pestering us for access to security@k.o for decades now, this isn't going to change for the obvious reason that having such groups on the list is not going to help us fix any problem, but instead, just give everyone early access to known security problems. Same thing would happen for any potential distro@k.o list, remember who some of the largest users of Linux is (i.e. governments) and many of them have their own custom "distros" for their systems for valid reasons. So no, we can't do that if you care about security overall, this would make things insecure. sorry, greg k-h ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 14:41 ` Greg KH @ 2023-08-15 15:04 ` Steven Rostedt 2023-08-15 15:51 ` Greg KH 0 siblings, 1 reply; 29+ messages in thread From: Steven Rostedt @ 2023-08-15 15:04 UTC (permalink / raw) To: Greg KH; +Cc: Catalin Marinas, Vegard Nossum, Jiri Kosina, ksummit On Tue, 15 Aug 2023 16:41:37 +0200 Greg KH <gregkh@linuxfoundation.org> wrote: > Loads of companies/governments have been pestering us for access to > security@k.o for decades now, this isn't going to change for the obvious > reason that having such groups on the list is not going to help us fix > any problem, but instead, just give everyone early access to known > security problems. > > Same thing would happen for any potential distro@k.o list, remember who > some of the largest users of Linux is (i.e. governments) and many of > them have their own custom "distros" for their systems for valid > reasons. > > So no, we can't do that if you care about security overall, this would > make things insecure. Even if the only thing that is shown is a commit sha that should be taken? -- Steve ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 15:04 ` Steven Rostedt @ 2023-08-15 15:51 ` Greg KH 0 siblings, 0 replies; 29+ messages in thread From: Greg KH @ 2023-08-15 15:51 UTC (permalink / raw) To: Steven Rostedt; +Cc: Catalin Marinas, Vegard Nossum, Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 11:04:22AM -0400, Steven Rostedt wrote: > On Tue, 15 Aug 2023 16:41:37 +0200 > Greg KH <gregkh@linuxfoundation.org> wrote: > > > Loads of companies/governments have been pestering us for access to > > security@k.o for decades now, this isn't going to change for the obvious > > reason that having such groups on the list is not going to help us fix > > any problem, but instead, just give everyone early access to known > > security problems. > > > > Same thing would happen for any potential distro@k.o list, remember who > > some of the largest users of Linux is (i.e. governments) and many of > > them have their own custom "distros" for their systems for valid > > reasons. > > > > So no, we can't do that if you care about security overall, this would > > make things insecure. > > Even if the only thing that is shown is a commit sha that should be taken? I provide a list of all of those commit sha in every release today. You sound like someone in a meeting I had many years ago at a "big chip company", the conversation went something like: Security Manager - "Just tell us which releases in every stable release you make that fix a security bug." Me - "I can't do that as that would imply we have audited every commit to determine if they are, or are not, a security fix, and so you would have a false sense of security if you don't take all of them." SM - "But I don't care about any unknown security fixes, I only want to know the known security fixes!" Me - "You will then have insecure systems when it is determined later that those other fixes were security fixes." SM - "I don't care about future security issues, I only want to know about known ones now!" Me - "..." Luckily a VP stepped in then and said, "The community is giving you a set of known bugfixes for all issues that they know of at the moment, for free, why are you refusing to take all of them and insisting that they do more work for you for free so that you can have a more insecure system?" And yes, this company is not using a distro at all, they have their own "releases" based on the kernel.org tree. And they are HUGE, one might argue much bigger users of Linux than many of the "enterprise" distros are on quantity of Linux systems in use. So again, what's preventing you from just taking all published fixes? Android does this, are your systems somehow more special than Android, the largest user of Linux by far in the world? :) Also, to further drive home this issue, a "big car manufacturer" a few years ago told me, "we now have a team of developers that are going to audit every stable kernel commit to determine if it is, or is not, a security fix for our systems before we will take it." They did this for about a year and came back saying "we could not keep up with the flow at all, and everytime we guessed wrong, we put the security of our systems at risk. It ended up being cheaper, and safer, to just take all of the fixes." Which they now do. So, why are you refusing to do the cheaper (from a business point of view), and safer (from a security point of view) and not just take all of the fixes we provide? Again, the Google security team proved that taking the stable kernel releases fixes 90%+ of all known security problems in their systems BEFORE they are known! They analyzed all of the data on this for 2 years straight and published their results before they just gave in and said "just take them all, to not do so is crazy". (the 10% not fixed were attributed to out-of-tree code that was not upstream yet, nothing we can do about that...) And yes, I'm arguing strongly that the old model of the "enterprise kernel releases" is wrong. So much so that I fail to understand that if Android can provide a secure, up-to-date, internal ABI STABLE, kernel based on the LTS releases, why can't these enterprise distros do the same for their customers? And I think Oracle does do this for their enterprise kernel releases, which is nice to see. Yes, I'm saying more people need to be like Oracle and Android, what is the world coming to... :) thanks, greg k-h ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 12:42 ` Steven Rostedt 2023-08-15 13:17 ` Daniel Borkmann 2023-08-15 14:20 ` Catalin Marinas @ 2023-08-15 15:08 ` Greg KH 2023-08-15 18:46 ` Konrad Rzeszutek Wilk ` (2 more replies) 2 siblings, 3 replies; 29+ messages in thread From: Greg KH @ 2023-08-15 15:08 UTC (permalink / raw) To: Steven Rostedt; +Cc: Vegard Nossum, Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 08:42:53AM -0400, Steven Rostedt wrote: > On Tue, 15 Aug 2023 13:23:36 +0200 > Greg KH <gregkh@linuxfoundation.org> wrote: > > > On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote: > > > I'll throw in another idea: distros@kernel.org. > > > > > > A closed list which will be notified by security@kernel.org once they > > > feel patches for a particular issue are ready for testing/consumption by > > > distros (and hopefully before the issue is disclosed publicly, if the > > > reporter still wishes to do that). > > > > > > The members and list rules would be totally up to the security team to > > > decide. > > > > As per the lawyers, and government officials we have worked with in the > > past, having a closed list for preannouncements like this will be > > either: > > I guess the question is, what "preannouncements" are the lawyers worried about? Access to known security problems before they are fixed for everyone. > It looks like Jiri's concern is just to make sure that distros have > security patches in. Would just a list of "here's the commits that fix > security issues" be considered a preannouncement, without having any > reference to *what* security issue they fix? It would basically be a subset > of the stable tree. That is, anything that came from security@k.o, and does > not include all the AUTOSEL and other minor fixes that stable pulls in. As Laurent said, and as many who have analyzed the data have proven, the HUGE majority of "security fixes" flow through the normal stable release process. And yet, many distros/companies don't even keep up with that, why would stuff sent to security@k.o be any different? We used to have someone on security@k.o that would notify linux-distros about problems when we had fixed them, but I think they got tired of constantly keeping on top of this and stopped doing it. It's a thankless job, just like being on the security@k.o alias is, and I don't blame them for stopping. > > - deemed illegal in some countries > > - made to have all "major"[1] Linux users on it. > > And if this list only has a list of patches that are already fixed, how > dangerous is it to not be 100% closed? > > I mean, it was pretty obvious that the huge churn with spectre/meltdown > patches that were being added to the kernel at the late stage of an -rc > release was fixing "something big". Hardware embargoes have different "rules" that we have agreed to follow, based on the lawyers for all companies involved agreeing to those rules, that's independent of security@k.o which doesn't get involved in this process at all (despite some of us overlapping both lists, but that's just an individual person thing, not a "whole group" thing.) And I would honestly say that everyone involved in the hw-embargo process currently hates it and I don't think it works well but it's the best that we have at the moment until it can be re-negotiated OR hardware companies stop doing stupid things with their CPUs. > > Neither of which actually will work out at all, the whole > > "preannouncement" stuff just is not possible, sorry. I'm amazed that > > other projects have been able to "get away with it" for as long as they > > have without either being infiltrated by "the powers that be" or > > shutdown yet. > > Have there been lists shutdown by the powers that be already? Or is this > just a threat made by the power that be? No lists have been stopped that I know of, because we have been told by governments that our current process is ok and in fact is "the only reasonable way it can be done that we know of." See my US Senate testimony/interview about Spectre/Meltdown for details about this. I think it's online somewhere, but don't know where... > > Politics is a rough game, the only way to survive is to not play it for > > stuff like this. > > > > So no, "distros@k.o" isn't going to be possible for the LF to host, and > > any other group that wants to run such a thing will quickly have these > > issues as well, it's amazing that linux-distros has been able to survive > > for as long as it has. > > I'll have to talk with some laywers, as I'm curious to what would be > considered illegal about linux-distros. Are you aware of any government > specific laws I could go read? I'm not a lawyer, but I've read quite a bit > of laws that I can usually get an idea of the problem for my own > references (and my experience is that lawyers don't even know until > something is settled in court). > > Note, linux-distros is not about "Linux users" but "Linux distributors". > They are not end users but are distributing a product (and having to follow > all the rules of the GPL and such to do so). They are not users, they are > part of a supply chain. > > How is security@kernel.org different? If the bug is in the kernel, then the > bug is in the distro. Perhaps if we find a bug in the kernel, we should > send it to the distro we are using, and not to the Linux community? As Jiri > said, most people use a distro kernel, and not the mainline or even the > stable one. security@k.o is very different in that our job is to triage reported bugs and fix them as soon as possible and get the fix out to the world. We almost never have embargoes (there is a limited, very very short, potential embargo we will agree to in some limited situations), but for the huge majority of what we do, all we are here for is to fix problems. That's it, no notifications, no delays, nothing else except fix the issue as soon as we can. Yes, sometimes this takes a long time, but once we have a working fix, we get it merged quickly and move on to the next issue. > Really, if you think about it. The closest product to the user > is the distro. If someone finds a bug in the kernel, they can see if they > can exploit a distro with it. If they can, perhaps they should send it to > the security folks of the distro first. Then the distro can send it to > security@kernel.org. Maybe this already happens? The huge majority of Linux use in the world is Android, everything else is a rounding error. Android does now have projects where security bugs reported to them are required for the reporter to submit the problem to security@k.o as well. We fix the issue, get it pushed out, and the reporter gets the credit. Sometime in the future Android picks the fixes up and pushed them out to their users (the delay there is much argued about, I've worked with many companies over the years about this and I understand the issues involved, but really, it should be sooner than the 4-6 months it currently takes, but this is on those companies, not us. Governments now know this and are pushing laws to resolve this delay (i.e. the CRA)). After Android, Debian is by far the largest Linux user, and the Debian kernel developers do an awesome job of tracking the stable kernel releases which have been documented to fix 99% of known security issues _BEFORE_ they are known (data produced by Google security team for 2 years straight). After that, the use of Linux tapers off to "roll your own kernel.org releases" (still a huge number of absolute boxes thanks to many government instances and corporate clouds) to the various "enterprise" distros, down to the other community distros (Fedora/openSUSE/Arch/etc.) So the top end (Android and Debian and kernel.org) are covered today with stable/LTS releases. Same for the bottom end (Fedora/openSUSE/Arch/etc.) It's that corporate sponsored "middle tier" (which is still quite small overall) that likes complaining about this stuff because they don't want to take the stable kernel updates for various (in my opinion stupid) reasons. So who would such a "distros@" list help? Only that middle-tier, small group of very well financed companies that want to go their own way with their kernel releases because they want to. Why are they not just doing what the huge majority of Linux users doing and taking the "feed of known issues that resolve problems before they are public knowledge" that we provide today for free to them? Because they somehow think that knowing a specific single bugfix is more special than all of those other bugfixes, which honestly, is just loony. Anyway, sorry for the long rant, that's my take on the kernel ecosystem as someone who has been seeing this for a very long time and working with loads of different companies using the kernel. As for the legal issues involved in such a preannouncement list, think about all of those huge government agencies in many many different countries who use Linux and would insist that they get access to that feed for their own security reasons. I would say that they honestly have a better reason than the smaller "enterprise" distros out there, and they can back up their request for access with legal measures.[1] If we do not even have such a list, there's no legal measures that can be used, which is why kernel.org should not host such a list. thanks, greg k-h [1] Someone who is involved in EU governments told me once that if the EU were to be setting up a MITRE-like organization to get pre-announcements of security problems as the CRA recommends, we would instantly solve the "bugs are kept secret" problem because one thing the EU governments are really good at is making secret things public :) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 15:08 ` Greg KH @ 2023-08-15 18:46 ` Konrad Rzeszutek Wilk 2023-08-15 19:41 ` Greg KH 2023-08-15 22:13 ` Jiri Kosina 2023-08-15 22:17 ` Jiri Kosina 2 siblings, 1 reply; 29+ messages in thread From: Konrad Rzeszutek Wilk @ 2023-08-15 18:46 UTC (permalink / raw) To: Greg KH; +Cc: Steven Rostedt, Vegard Nossum, Jiri Kosina, ksummit ..snip.. > > We used to have someone on security@k.o that would notify linux-distros > about problems when we had fixed them, but I think they got tired of > constantly keeping on top of this and stopped doing it. It's a > thankless job, just like being on the security@k.o alias is, and I don't > blame them for stopping. Hi Greg, Oracle will happily pay someone this "thankless job" (actually I think it is a pretty exciting job as you get to learn and try your hand on everything) to do this and also help with the security fixes. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 18:46 ` Konrad Rzeszutek Wilk @ 2023-08-15 19:41 ` Greg KH 0 siblings, 0 replies; 29+ messages in thread From: Greg KH @ 2023-08-15 19:41 UTC (permalink / raw) To: Konrad Rzeszutek Wilk; +Cc: Steven Rostedt, Vegard Nossum, Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 02:46:27PM -0400, Konrad Rzeszutek Wilk wrote: > ..snip.. > > > > We used to have someone on security@k.o that would notify linux-distros > > about problems when we had fixed them, but I think they got tired of > > constantly keeping on top of this and stopped doing it. It's a > > thankless job, just like being on the security@k.o alias is, and I don't > > blame them for stopping. > > Hi Greg, > > Oracle will happily pay someone this "thankless job" (actually I think it > is a pretty exciting job as you get to learn and try your hand on > everything) to do this and also help with the security fixes. The thing is, people on security@k.o are there on their own recognition, and can not represent, nor notify, their employer of things discussed there (otherwise the group can't really be called independent.) We have had to remove members in the past who were only using access there for their employer so I'm a bit hesitant to only add someone for the single reason to funnel stuff from the list elsewhere for obvious reasons. Others in the group are free to disagree with me about this, it's run as a "collective" by those doing the work there, not by fiat. Note, the people on security@k.o almost always do NOT fix the issue reported, they are there to triage and drag in the correct maintainers and then help review proposed changes. If you as a maintainer are drug into the list enough times, you're asked if you want to join to save the round-trip emails. So when people are added, it's because of problems in their kernel area, or because they have done lots of reviews of subsystems in ways relating to security issues, not because they are there to fix issues in other parts of the kernel. And again, if only the issues that are reported to security@k.o are sent to linux-distros, the distros will only get a tiny tiny subset of the actual bugs being fixed in the kernel on a weekly basis. Trying to get access to this tiny feed does not solve the real issue of distros not properly updating to get all of the needed fixes. Also remember that some subsystems refuse to participate in security@k.o, their fixes come in through the "normal" stable releases, with work done on mailing lists. So again, if you only see the security@k.o issues, you will miss major problems being resolved. Work on solving the root problem here for your distro please, don't fixate on the CVE nonsense dance, that provides a false sense of security and not actual security at all. thanks, greg k-h ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 15:08 ` Greg KH 2023-08-15 18:46 ` Konrad Rzeszutek Wilk @ 2023-08-15 22:13 ` Jiri Kosina 2023-08-15 22:31 ` Steven Rostedt 2023-08-15 22:17 ` Jiri Kosina 2 siblings, 1 reply; 29+ messages in thread From: Jiri Kosina @ 2023-08-15 22:13 UTC (permalink / raw) To: Greg KH; +Cc: Steven Rostedt, Vegard Nossum, ksummit On Tue, 15 Aug 2023, Greg KH wrote: > > Really, if you think about it. The closest product to the user > > is the distro. If someone finds a bug in the kernel, they can see if they > > can exploit a distro with it. If they can, perhaps they should send it to > > the security folks of the distro first. Then the distro can send it to > > security@kernel.org. Maybe this already happens? > > The huge majority of Linux use in the world is Android, everything else > is a rounding error. Sorry, but in my view this is a slight oversimplification. Sure, Android is ultra-huge, with userbase being the metric. But you very well aware of where Linux as an server/enterprise distro is running, and that's big as well. And you can't just count that by "number of deployments/users" metric. Stock exchanges, air traffic control, NASA, govermental insitutions of all sorts, you name it. And, at the end of the day, all those huge deployments then directly contribute back to kernel development, by allowing companies like Red Hat, SUSE, Canonical, ... to put kernel engineers on their payroll. So I'd like to (with both my community and distro hats on now :) ) make sure they/we can proceed with that without unnecessary hurdles. [ .. paragraps about how enterprise/server distributions are irrelevant stripped here :) .. ] > So the top end (Android and Debian and kernel.org) are covered today > with stable/LTS releases. Same for the bottom end > (Fedora/openSUSE/Arch/etc.) Are they? I don't think Fedora is picking LTS as a base for the kernel. openSUSE definitely is not. So how are they covered? Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 22:13 ` Jiri Kosina @ 2023-08-15 22:31 ` Steven Rostedt 2023-08-16 14:55 ` Greg KH 0 siblings, 1 reply; 29+ messages in thread From: Steven Rostedt @ 2023-08-15 22:31 UTC (permalink / raw) To: Jiri Kosina; +Cc: Greg KH, Vegard Nossum, ksummit On Wed, 16 Aug 2023 00:13:56 +0200 (CEST) Jiri Kosina <jikos@kernel.org> wrote: > > The huge majority of Linux use in the world is Android, everything else > > is a rounding error. > > Sorry, but in my view this is a slight oversimplification. I agree. And that's just taking in "numbers". What about impact. If there's a security flaw in an android phone, it opens up each individual for an attack, but it usually requires an attacker to hit them individually. But if an enterprise is hit. All it takes is to go after one server, and you have access to thousands of users and their private data. It's not just the number of installations that count. -- Steve ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 22:31 ` Steven Rostedt @ 2023-08-16 14:55 ` Greg KH 2024-02-16 17:14 ` Michal Suchánek 0 siblings, 1 reply; 29+ messages in thread From: Greg KH @ 2023-08-16 14:55 UTC (permalink / raw) To: Steven Rostedt; +Cc: Jiri Kosina, Vegard Nossum, ksummit On Tue, Aug 15, 2023 at 06:31:20PM -0400, Steven Rostedt wrote: > On Wed, 16 Aug 2023 00:13:56 +0200 (CEST) > Jiri Kosina <jikos@kernel.org> wrote: > > > > The huge majority of Linux use in the world is Android, everything else > > > is a rounding error. > > > > Sorry, but in my view this is a slight oversimplification. > > I agree. And that's just taking in "numbers". What about impact. If there's > a security flaw in an android phone, it opens up each individual for an > attack, but it usually requires an attacker to hit them individually. > > But if an enterprise is hit. All it takes is to go after one server, and > you have access to thousands of users and their private data. > > It's not just the number of installations that count. Very true, but as an individual, we regard the private data in our phones usually more important than the data stored in corporate systems :) Anyway, my whole point here is that there are huge swaths of users of Linux, the majority in quantity (not judging quality), that are now doing the right thing here by taking all known fixes into their kernel trees. It's only a small (in absolute numbers, not importance, I'm not judging importance as everyone is more important than everyone else, like always), number that are not doing this and asking for us to give them access to a tiny feed of fixes instead so that they can have a false sense of security. Please, let's work on fixing that false sense of security. It's wrong, and our users deserve better as we (the community) ARE doing all of the fixing the best we can, but not all of our users seem to be able to take advantage of this due to the "policy decisions" of others outside of our community. And note, those "policy decisions of companies" are now known by governments to be incorrect, and soon will be made illegal in many countries, so we are on the right side here. Together we were able to solve the "impossible" problem of Android kernel security. Now that that windmill is semi-successfully conquered despite many who thought it never could be, why can't we help other users of our product to do the same? If not, we run the risk of having Linux be blamed for bad security, when in reality, it's the policy of companies not taking advantage of what we are providing them, a nuance that Linux users will never really understand, nor should they have to. Let's fix this properly please, access to the security@k.o reports will do nothing to solve the root issue, and only paper it over for a few more years. thanks, greg k-h ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-16 14:55 ` Greg KH @ 2024-02-16 17:14 ` Michal Suchánek 2024-02-16 17:34 ` Greg KH 0 siblings, 1 reply; 29+ messages in thread From: Michal Suchánek @ 2024-02-16 17:14 UTC (permalink / raw) To: Greg KH; +Cc: Steven Rostedt, Jiri Kosina, Vegard Nossum, ksummit Hello, On Wed, Aug 16, 2023 at 04:55:39PM +0200, Greg KH wrote: > On Tue, Aug 15, 2023 at 06:31:20PM -0400, Steven Rostedt wrote: > > On Wed, 16 Aug 2023 00:13:56 +0200 (CEST) > > Jiri Kosina <jikos@kernel.org> wrote: > > > > > > The huge majority of Linux use in the world is Android, everything else > > > > is a rounding error. > > > > > > Sorry, but in my view this is a slight oversimplification. > > > > I agree. And that's just taking in "numbers". What about impact. If there's > > a security flaw in an android phone, it opens up each individual for an > > attack, but it usually requires an attacker to hit them individually. > > > > But if an enterprise is hit. All it takes is to go after one server, and > > you have access to thousands of users and their private data. > > > > It's not just the number of installations that count. > > Very true, but as an individual, we regard the private data in our > phones usually more important than the data stored in corporate systems :) If my data happens to be on a corporate system I do care about their security. With everything moving to the cloud today most of the data users can see on their Android phones is in fact on such a corporate system. > Together we were able to solve the "impossible" problem of Android > kernel security. Now that that windmill is semi-successfully conquered > despite many who thought it never could be, why can't we help other > users of our product to do the same? If not, we run the risk of having > Linux be blamed for bad security, when in reality, it's the policy of > companies not taking advantage of what we are providing them, a nuance > that Linux users will never really understand, nor should they have to. The real impossible problem of Android security is that those Android systems aren't really opensource, and it's (next to) impossible to get updataes when the device vendor does not provide one. So many Android device users are running long-obsolete systems because there is nothing more recent that runs on their device. That won't change no matter what stable does or does not. Thanks Michal ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2024-02-16 17:14 ` Michal Suchánek @ 2024-02-16 17:34 ` Greg KH 2024-02-16 18:13 ` Michal Suchánek 0 siblings, 1 reply; 29+ messages in thread From: Greg KH @ 2024-02-16 17:34 UTC (permalink / raw) To: Michal Suchánek; +Cc: Steven Rostedt, Jiri Kosina, Vegard Nossum, ksummit On Fri, Feb 16, 2024 at 06:14:27PM +0100, Michal Suchánek wrote: > On Wed, Aug 16, 2023 at 04:55:39PM +0200, Greg KH wrote: That was a very old thread, why dig it up now? > > Together we were able to solve the "impossible" problem of Android > > kernel security. Now that that windmill is semi-successfully conquered > > despite many who thought it never could be, why can't we help other > > users of our product to do the same? If not, we run the risk of having > > Linux be blamed for bad security, when in reality, it's the policy of > > companies not taking advantage of what we are providing them, a nuance > > that Linux users will never really understand, nor should they have to. > > The real impossible problem of Android security is that those Android > systems aren't really opensource, and it's (next to) impossible to get > updataes when the device vendor does not provide one. > > So many Android device users are running long-obsolete systems because > there is nothing more recent that runs on their device. > > That won't change no matter what stable does or does not. That is why governments are making laws to require this to happen. The EU just did this for all mobile devices, and I expect other governments to do the same over time. thanks, greg k-h ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2024-02-16 17:34 ` Greg KH @ 2024-02-16 18:13 ` Michal Suchánek 2024-02-16 18:16 ` Jiri Kosina 0 siblings, 1 reply; 29+ messages in thread From: Michal Suchánek @ 2024-02-16 18:13 UTC (permalink / raw) To: Greg KH; +Cc: Steven Rostedt, Jiri Kosina, Vegard Nossum, ksummit On Fri, Feb 16, 2024 at 06:34:37PM +0100, Greg KH wrote: > On Fri, Feb 16, 2024 at 06:14:27PM +0100, Michal Suchánek wrote: > > On Wed, Aug 16, 2023 at 04:55:39PM +0200, Greg KH wrote: > > That was a very old thread, why dig it up now? It was linked to recently and I did not realize how old it is. > > > Together we were able to solve the "impossible" problem of Android > > > kernel security. Now that that windmill is semi-successfully conquered > > > despite many who thought it never could be, why can't we help other > > > users of our product to do the same? If not, we run the risk of having > > > Linux be blamed for bad security, when in reality, it's the policy of > > > companies not taking advantage of what we are providing them, a nuance > > > that Linux users will never really understand, nor should they have to. > > > > The real impossible problem of Android security is that those Android > > systems aren't really opensource, and it's (next to) impossible to get > > updataes when the device vendor does not provide one. > > > > So many Android device users are running long-obsolete systems because > > there is nothing more recent that runs on their device. > > > > That won't change no matter what stable does or does not. > > That is why governments are making laws to require this to happen. The > EU just did this for all mobile devices, and I expect other governments > to do the same over time. Indeed, with the requirement to ship updates for several years, and the CRA requirement to not ship broken updates things may improve, at least it looks so on paper. Thanks Michal ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2024-02-16 18:13 ` Michal Suchánek @ 2024-02-16 18:16 ` Jiri Kosina 0 siblings, 0 replies; 29+ messages in thread From: Jiri Kosina @ 2024-02-16 18:16 UTC (permalink / raw) To: Michal Suchánek; +Cc: Greg KH, Steven Rostedt, Vegard Nossum, ksummit On Fri, 16 Feb 2024, Michal Suchánek wrote: > > That was a very old thread, why dig it up now? > > It was linked to recently and I did not realize how old it is. Yeah, I take the "blame" for that. I referenced that thread on our internal IRC in order to point out another case where the role of (commercial) distros was downplayed. -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 15:08 ` Greg KH 2023-08-15 18:46 ` Konrad Rzeszutek Wilk 2023-08-15 22:13 ` Jiri Kosina @ 2023-08-15 22:17 ` Jiri Kosina 2023-08-16 14:57 ` Greg KH 2023-08-16 18:38 ` Vegard Nossum 2 siblings, 2 replies; 29+ messages in thread From: Jiri Kosina @ 2023-08-15 22:17 UTC (permalink / raw) To: Greg KH; +Cc: Steven Rostedt, Vegard Nossum, ksummit On Tue, 15 Aug 2023, Greg KH wrote: > Why are they not just doing what the huge majority of Linux users doing > and taking the "feed of known issues that resolve problems before they > are public knowledge" that we provide today for free to them? Because > they somehow think that knowing a specific single bugfix is more special > than all of those other bugfixes, which honestly, is just loony. If you'd like me to turn this proposal into "What can we do to make sure that most major distros are eventually basing their kernels on -stable" discussion, I'd be happy to do that, but I believe this has been discussed quite extensively already. Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 22:17 ` Jiri Kosina @ 2023-08-16 14:57 ` Greg KH 2023-08-16 17:22 ` Jiri Kosina 2023-08-16 18:38 ` Vegard Nossum 1 sibling, 1 reply; 29+ messages in thread From: Greg KH @ 2023-08-16 14:57 UTC (permalink / raw) To: Jiri Kosina; +Cc: Steven Rostedt, Vegard Nossum, ksummit On Wed, Aug 16, 2023 at 12:17:36AM +0200, Jiri Kosina wrote: > On Tue, 15 Aug 2023, Greg KH wrote: > > > Why are they not just doing what the huge majority of Linux users doing > > and taking the "feed of known issues that resolve problems before they > > are public knowledge" that we provide today for free to them? Because > > they somehow think that knowing a specific single bugfix is more special > > than all of those other bugfixes, which honestly, is just loony. > > If you'd like me to turn this proposal into "What can we do to make sure > that most major distros are eventually basing their kernels on -stable" > discussion, I'd be happy to do that, but I believe this has been discussed > quite extensively already. It has, and note that no one in the room would be in a real position to solve the problem (Project Managers and VPs are not usually invited to the kernel summit.) Now that many laws are being passed to mandate this soon, I think the governments might solve this issue for us automatically :) thanks, greg k-h ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-16 14:57 ` Greg KH @ 2023-08-16 17:22 ` Jiri Kosina 0 siblings, 0 replies; 29+ messages in thread From: Jiri Kosina @ 2023-08-16 17:22 UTC (permalink / raw) To: Greg KH; +Cc: Steven Rostedt, Vegard Nossum, ksummit On Wed, 16 Aug 2023, Greg KH wrote: > > > Why are they not just doing what the huge majority of Linux users doing > > > and taking the "feed of known issues that resolve problems before they > > > are public knowledge" that we provide today for free to them? Because > > > they somehow think that knowing a specific single bugfix is more special > > > than all of those other bugfixes, which honestly, is just loony. > > > > If you'd like me to turn this proposal into "What can we do to make sure > > that most major distros are eventually basing their kernels on -stable" > > discussion, I'd be happy to do that, but I believe this has been discussed > > quite extensively already. > > It has, and note that no one in the room would be in a real position to > solve the problem (Project Managers and VPs are not usually invited to > the kernel summit.) Not sure about other distros, but at least at SUSE, this decision powers are held by the kernel developers / maintainers of the branches in question. Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 22:17 ` Jiri Kosina 2023-08-16 14:57 ` Greg KH @ 2023-08-16 18:38 ` Vegard Nossum 1 sibling, 0 replies; 29+ messages in thread From: Vegard Nossum @ 2023-08-16 18:38 UTC (permalink / raw) To: Jiri Kosina, Greg KH; +Cc: Steven Rostedt, ksummit On 8/16/23 00:17, Jiri Kosina wrote: > On Tue, 15 Aug 2023, Greg KH wrote: > >> Why are they not just doing what the huge majority of Linux users doing >> and taking the "feed of known issues that resolve problems before they >> are public knowledge" that we provide today for free to them? Because >> they somehow think that knowing a specific single bugfix is more special >> than all of those other bugfixes, which honestly, is just loony. > > If you'd like me to turn this proposal into "What can we do to make sure > that most major distros are eventually basing their kernels on -stable" > discussion, I'd be happy to do that, but I believe this has been discussed > quite extensively already. I really think these are two different and orthogonal topics and we ought to treat them separately: tracking stable vs. having advance notice for some vulnerabilities. (Regarding tracking stable, I fully agree that all distros should be keeping up with stable and not just cherry-picking specific fixes or whatever commits happen to have a CVE assigned. No contest there.) This proposal is about embargoed security issues, issues that were reported because somebody knows (or has good reason to suspect) that an issue is exploitable and is planning to disclose that fact publicly. I posit that there is objectively a difference between: a) vulnerabilities that have been discovered and reported by a security researcher and where a detailed analysis and/or PoCs/exploits are certain to be made published publicly; and b) patches that appear in mainline/stable with no vulnerability analysis and no known exploit vector. I agree that both categories of issues may be equally serious or equally exploitable, so the difference is in the fact that somebody has done a thorough analysis of specific bugs and/or is planning to disclose that publicly. If distros are only made aware of a vulnerability when it is disclosed publicly by a security researcher, that leaves a window of opportunity for attackers -- patches that appear in stable do not immediately make it onto end-user systems for a variety of reasons: the time it takes to backport the patch (as most distros carry at least some out-of-tree patches), testing/validation (especially if there were conflicts or other issues with the patch), and then of course the time it takes end-user to upgrade. I think several distros at this point are making use of live patching technology -- for really critical vulnerabilities, this has the potential to reduce that window to very close to zero: live patching updates that are available and can be applied from the moment the vulnerability is disclosed publicly. So it really isn't a choice between one or the other -- the best security for end users is when we do both. (I should add that I don't think anybody is really asking for a private feed of s@k.o -- IMHO the best solution would be for security researchers to notify the distros list when s@k.o has a tentative or final patch, before disclosing the vulnerability publicly. The best path I see for that is: 1) relax the linux-distros policy to not require posting more than what the reporter is comfortable disclosing, 2) fix the kernel.org documentation to be clearer for reporters, 3) have a person on s@k.o who can guide the reporter towards linux-distros if they are planning to do a public disclosure anyway.) Thanks, Vegard ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-15 10:17 ` Vegard Nossum 2023-08-15 10:34 ` Jiri Kosina 2023-08-15 11:23 ` Greg KH @ 2023-08-16 15:26 ` Solar Designer 2023-08-25 11:17 ` Donald Buczek 2 siblings, 1 reply; 29+ messages in thread From: Solar Designer @ 2023-08-16 15:26 UTC (permalink / raw) To: Vegard Nossum; +Cc: Jiri Kosina, ksummit On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote: > (Sending this with a few extra people in Bcc so they'll see it without > getting spammed with replies if they don't want them.) Thank you, Vegard! I've also now skimmed this thread at https://lore.kernel.org/ksummit/nycvar.YFH.7.76.2308160014330.14207@cbobk.fhfr.pm/T/#t I also found interesting the adjacent thread on "Quality standards for embargoed code". I notice these are tagged "MAINTAINERS SUMMIT", I assume in reference to the one in Richmond, VA on November 16th, 2023? I cannot easily get to the US, but I'd consider getting to the summit in Bilbao, Spain on September 19-21 if the relevant people would be there as well and my attendance would be expected to help? I don't think there's anything we can't discuss over e-mail (and in fact e-mail is more specific), but meeting in person is a gesture that might help establish an atmosphere of trust and assurance of good intent. > I think it's worth pointing out that the latest update to the > documentation (where reporters are discouraged from reporting kernel > issues to linux-distros@ ever) It isn't "ever" - rather, it is "NEVER contact the "linux-distros" mailing list until AFTER discussing it with the kernel security team." So the kernel security team can, perhaps after having arrived at a fix, choose to direct the reporter to also contact linux-distros. Now, I don't know whether this would actually be happening or not. Maybe some friendly dialogue and agreeing on things could affect that. > came just after a private discussion (on > both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where > Linus in particular came out hard against the linux-distros policy, in > particular the requirement to disclose PoCs if those were shared with > the list in the first place. I therefore think that Linus himself needs > to be involved in this discussion and that his arguments need to be made > in public instead of just paraphrased by me. In that private thread, two linux-distros policy aspects came up as being inconsistent with s@k.o preferences: 1. For s@k.o, public disclosure is typically in up to 7 days since fix is ready, but for linux-distros the deadline is 14 days max since linux-distros is notified even if the fix is not ready. Apparently, this one makes Greg particularly unhappy about linux-distros. 2. _If_ the reporter shares PoCs/exploits with linux-distros, then per current policy those should also be made public (within up to 7 days more after the vulnerability is publicly disclosed as such). Linus said that this must be up to the reporter only, and we should not be imposing such policy. In a sense, this is already up to the reporter - if they disagree, they just shouldn't post PoCs/exploits to linux-distros (can instead post an offer to share with individual distros) - however, this is problematic in practice because not everyone reads the rules before posting and sometimes people change their mind during the embargo time (or are required to "change their mind" by their employer, etc.) I agree both of these are problems, and I suggest discussing them on oss-security. Maybe we can arrive at satisfactory solutions/exceptions. While we're at it, I'd like to point out that while the wording (in the commit above) that "the linux-distros rules do not contribute to actually fixing any potential security problems" is technically correct (yes, the _rules_ do not contribute to _upstream_ fixes), it may be misleading. It implies that linux-distros members would not help fix bugs, whereas in that very StackRot/CVE-2023-3269 thread Vegard, receiving the messages as part of Oracle Linux's linux-distros membership, has actually contributed to fixing the bug upstream. Vegard is probably too humble to bring this up, so I do. Also, even when not contributing to upstream fixes, linux-distros members will generally "contribute to actually fixing any potential security problems" in their respective distros (even if e.g. by back-porting upstream fixes when those are ready). So I would omit this wording if it were up to me. > On 8/15/23 11:28, Jiri Kosina wrote: > >With my distro hat on, I really want the kernel security bugs to be > >*eventually* processed through linux-distros@ somehow, for one sole > >reason: it means that our distro security people will be aware of it, > >track it internally, and keep an eye on the fix being ported to all of our > >codestreams. This is exactly how this is done for all other packages. > > > >I would be curious to hear about this from other distros, but I sort of > >expect that they would agree. > > +1 from me; the distros list has been an extremely important resource > for distros in order to get fixes shipped out with minimal delay. Yes, I'd like this to be happening as well, and the current kernel documentation does not preclude that from happening. So now it's up to the people on s@k.o. > >[1] https://git.kernel.org/linus/4fee0915e649b Alexander ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-16 15:26 ` Solar Designer @ 2023-08-25 11:17 ` Donald Buczek 2023-08-29 8:46 ` Miroslav Benes 0 siblings, 1 reply; 29+ messages in thread From: Donald Buczek @ 2023-08-25 11:17 UTC (permalink / raw) To: Solar Designer, oss-security; +Cc: Vegard Nossum, Jiri Kosina, ksummit Hi, Alexander and all, I was more or less waiting for a public discussion to start on oss-security or somewhere else, because I'd like to express how important the announcements of Linux kernel issues via linux-distros is to us. We operate the IT infrastructure of a scientific research institute (456 users, 144 Linux servers, 98 Linux workstations, about 6 PB used storage). We are not using any distro, everything is build locally and we maintain the systems by continuous rolling upgrades. All systems from workstations to cluster servers run the same (set of) kernels and identical userspace. The setting is rather unusual these days, all shared storage including homes is on software raid and available to the other systems via automounted nfs. We have a few public servers and a rather big computer cluster with a job scheduler. Everything is multi-user. We need to maintain separation between users and to defend the systems against attacks from the (possible abused) unprivileged user accounts. Our scientist run software on cluster-servers which often take many, many days to complete because of the size of the input data. Other might, for example, experiment with Jupyter Notebooks and have a lot of in-memory state. We have long running remote login sessions round the clock. For efficiency, most things run natively on hardware and not in virtual machines. System reboots have a lot of negative impact for our users. Even nowadays not everything is single-user or microservices. "All users of the X:X kernel series must upgrade" or "just run mainline" is not a helpful strategy for us, because there is the operational issue of booting into a new kernel. Also, because we have a rather unusual settings, it is not unusual that we run into bugs others haven't discovered before. That's why we rather test new releases and roll them out slowly. That may be a corner case, but these corner cases exist, too. We heavily rely on the information about kernel security issues published to linux-distros, which we, of course, can only receive via oss-security after the embargo. We analyze each and every new topic on oss-security to decide, whether it is relevant to us and what we can do about it. Nearly all of the userspace issues are of no relevance to us, but many of the kernel issues are, if we happen to run affected kernel versions. We go a long way to avoid rebooting. This might be as easy as disabling unused dynamic modules by just removing the .ko files from userspace, but sometimes we even convert an upstream fix into a loadable module which uses ftrace to replace or wrap the buggy functions in the running systems. A "reboot party" would only be a measure of last resort. oss-security gives us the notifications and enough information or references so that we can deal with the problems. We can not just follow mainline (or a stable series) and analyze every patch ourself to identify, whether a security bug was silently fixed. We can not boot into a new kernel every few days, just in case. Exploits are sometimes helpful, too, for example to quickly identify whether we run affected versions, which sometimes isn't that trivial to find out otherwise, or to verify that our mitigations are working as intended. I would very much appreciate if a policy could be found with the kernel community that encourages to send information to linux-distros. If that requires that the rather strict procedures and deadlines of linux-distros are relaxed, I hope there is room for compromise. Best Donald On 8/16/23 5:26 PM, Solar Designer wrote: > On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote: >> (Sending this with a few extra people in Bcc so they'll see it without >> getting spammed with replies if they don't want them.) > > Thank you, Vegard! > > I've also now skimmed this thread at > https://lore.kernel.org/ksummit/nycvar.YFH.7.76.2308160014330.14207@cbobk.fhfr.pm/T/#t > > I also found interesting the adjacent thread on "Quality standards for > embargoed code". > > I notice these are tagged "MAINTAINERS SUMMIT", I assume in reference to > the one in Richmond, VA on November 16th, 2023? I cannot easily get to > the US, but I'd consider getting to the summit in Bilbao, Spain on > September 19-21 if the relevant people would be there as well and my > attendance would be expected to help? I don't think there's anything we > can't discuss over e-mail (and in fact e-mail is more specific), but > meeting in person is a gesture that might help establish an atmosphere > of trust and assurance of good intent. > >> I think it's worth pointing out that the latest update to the >> documentation (where reporters are discouraged from reporting kernel >> issues to linux-distros@ ever) > > It isn't "ever" - rather, it is "NEVER contact the "linux-distros" > mailing list until AFTER discussing it with the kernel security team." > So the kernel security team can, perhaps after having arrived at a fix, > choose to direct the reporter to also contact linux-distros. Now, I > don't know whether this would actually be happening or not. Maybe some > friendly dialogue and agreeing on things could affect that. > >> came just after a private discussion (on >> both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where >> Linus in particular came out hard against the linux-distros policy, in >> particular the requirement to disclose PoCs if those were shared with >> the list in the first place. I therefore think that Linus himself needs >> to be involved in this discussion and that his arguments need to be made >> in public instead of just paraphrased by me. > > In that private thread, two linux-distros policy aspects came up as > being inconsistent with s@k.o preferences: > > 1. For s@k.o, public disclosure is typically in up to 7 days since fix > is ready, but for linux-distros the deadline is 14 days max since > linux-distros is notified even if the fix is not ready. Apparently, > this one makes Greg particularly unhappy about linux-distros. > > 2. _If_ the reporter shares PoCs/exploits with linux-distros, then per > current policy those should also be made public (within up to 7 days > more after the vulnerability is publicly disclosed as such). Linus said > that this must be up to the reporter only, and we should not be imposing > such policy. In a sense, this is already up to the reporter - if they > disagree, they just shouldn't post PoCs/exploits to linux-distros (can > instead post an offer to share with individual distros) - however, this > is problematic in practice because not everyone reads the rules before > posting and sometimes people change their mind during the embargo time > (or are required to "change their mind" by their employer, etc.) > > I agree both of these are problems, and I suggest discussing them on > oss-security. Maybe we can arrive at satisfactory solutions/exceptions. > > While we're at it, I'd like to point out that while the wording (in the > commit above) that "the linux-distros rules do not contribute to > actually fixing any potential security problems" is technically correct > (yes, the _rules_ do not contribute to _upstream_ fixes), it may be > misleading. It implies that linux-distros members would not help fix > bugs, whereas in that very StackRot/CVE-2023-3269 thread Vegard, > receiving the messages as part of Oracle Linux's linux-distros > membership, has actually contributed to fixing the bug upstream. Vegard > is probably too humble to bring this up, so I do. Also, even when not > contributing to upstream fixes, linux-distros members will generally > "contribute to actually fixing any potential security problems" in their > respective distros (even if e.g. by back-porting upstream fixes when > those are ready). So I would omit this wording if it were up to me. > >> On 8/15/23 11:28, Jiri Kosina wrote: >>> With my distro hat on, I really want the kernel security bugs to be >>> *eventually* processed through linux-distros@ somehow, for one sole >>> reason: it means that our distro security people will be aware of it, >>> track it internally, and keep an eye on the fix being ported to all of our >>> codestreams. This is exactly how this is done for all other packages. >>> >>> I would be curious to hear about this from other distros, but I sort of >>> expect that they would agree. >> >> +1 from me; the distros list has been an extremely important resource >> for distros in order to get fixes shipped out with minimal delay. > > Yes, I'd like this to be happening as well, and the current kernel > documentation does not preclude that from happening. So now it's up to > the people on s@k.o. > >>> [1] https://git.kernel.org/linus/4fee0915e649b > > Alexander > -- Donald Buczek buczek@molgen.mpg.de Tel: +49 30 8413 1433 ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ 2023-08-25 11:17 ` Donald Buczek @ 2023-08-29 8:46 ` Miroslav Benes 0 siblings, 0 replies; 29+ messages in thread From: Miroslav Benes @ 2023-08-29 8:46 UTC (permalink / raw) To: Donald Buczek Cc: Solar Designer, oss-security, Vegard Nossum, Jiri Kosina, ksummit [ apologies for a slight off topic ] Hi, On Fri, 25 Aug 2023, Donald Buczek wrote: > We go a long way to avoid rebooting. This might be as easy as disabling > unused dynamic modules by just removing the .ko files from userspace, > but sometimes we even convert an upstream fix into a loadable module > which uses ftrace to replace or wrap the buggy functions in the running > systems. A "reboot party" would only be a measure of last resort. the kernel live patching infrastructure might help you with this. See Documentation/livepatch/ and samples/livepatch/ in the kernel tree. Regards, Miroslav ^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2024-02-16 18:16 UTC | newest] Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-08-15 9:28 [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@ Jiri Kosina 2023-08-15 10:17 ` Vegard Nossum 2023-08-15 10:34 ` Jiri Kosina 2023-08-15 11:23 ` Greg KH 2023-08-15 12:42 ` Steven Rostedt 2023-08-15 13:17 ` Daniel Borkmann 2023-08-15 14:19 ` Laurent Pinchart 2023-08-15 22:04 ` Jiri Kosina 2023-08-15 14:20 ` Catalin Marinas 2023-08-15 14:41 ` Greg KH 2023-08-15 15:04 ` Steven Rostedt 2023-08-15 15:51 ` Greg KH 2023-08-15 15:08 ` Greg KH 2023-08-15 18:46 ` Konrad Rzeszutek Wilk 2023-08-15 19:41 ` Greg KH 2023-08-15 22:13 ` Jiri Kosina 2023-08-15 22:31 ` Steven Rostedt 2023-08-16 14:55 ` Greg KH 2024-02-16 17:14 ` Michal Suchánek 2024-02-16 17:34 ` Greg KH 2024-02-16 18:13 ` Michal Suchánek 2024-02-16 18:16 ` Jiri Kosina 2023-08-15 22:17 ` Jiri Kosina 2023-08-16 14:57 ` Greg KH 2023-08-16 17:22 ` Jiri Kosina 2023-08-16 18:38 ` Vegard Nossum 2023-08-16 15:26 ` Solar Designer 2023-08-25 11:17 ` Donald Buczek 2023-08-29 8:46 ` Miroslav Benes
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).