Is the Linux kernel underfunded? Lack of quality and security?

Is the Linux kernel underfunded? Lack of quality and security?

[Evan Rudford]

The problem of underfunding plagues many open source projects.
I wonder whether the Linux kernel suffers from underfunding in
comparison to its global reach.
Although code reviews and technical discussions are working well, I
argue that the testing infrastructure of the kernel is lacking.
Severe bugs are discovered late, and they are discovered by developers
that should not be exposed to that amount of breakage.
Moreover, I feel that security issues do not receive enough resources.

I argue that the cost of those bugs is vastly higher than the cost
that it would take to setup a better quality assurance.
With sufficient funding, the kernel might do all of the following:

- Make serious efforts to rewrite code with a bad security track
record, instead of only fixing security vulnerabilities on an ad hoc
basis.
- Although the kernel will always remain in C, make serious efforts to
introduce a safe language for kernel modules and perhaps for some
subsystems.
- Build an efficient continuous integration (CI) infrastructure.
- Run a fast subset of the CI tests as a gatekeeper for all patch sets.
- Run strict CI tests to ensure that userspace compatibility does not break.
- Run CI tests not only in virtual environments, but also on real hardware.
- Run CI tests that aim to detect performance regressions.

I realize that some companies are already running kernel testing
infrastructure like this.
However, the development process seems to either lack the resources or
the willingness to build a better quality assurance?

Re: Is the Linux kernel underfunded? Lack of quality and security?

[Greg KH]

On Sun, Jan 05, 2020 at 04:49:32AM +0100, Evan Rudford wrote:
> The problem of underfunding plagues many open source projects.

Does it?  Citation please :)

And compared to what exactly?

> I wonder whether the Linux kernel suffers from underfunding in
> comparison to its global reach.

Does it?  Again, specifics would be great to have.

> Although code reviews and technical discussions are working well, I
> argue that the testing infrastructure of the kernel is lacking.

Does it?  No one can argue we are "doing to much testing", and more
testing is always wanted, and happening, can you help with that effort?

> Severe bugs are discovered late, and they are discovered by developers
> that should not be exposed to that amount of breakage.

Specifics please.

Remember that Linux runs on _EVERYTHING_ so testing on _EVERYTHING_ is
sometimes a bit hard and bugs only show up later on when people get
around to running newer kernels on their specific hardware/workload.

> Moreover, I feel that security issues do not receive enough resources.

Again, citation please?  I would argue that right now we have too many
people/resources working on security issues that are really really minor
in the overall scheme of things.

What specific "security issues" are not currently being addressed?

> I argue that the cost of those bugs is vastly higher than the cost
> that it would take to setup a better quality assurance.

Why do you think that?

> With sufficient funding, the kernel might do all of the following:

Define "sufficient" :)

> - Make serious efforts to rewrite code with a bad security track
> record, instead of only fixing security vulnerabilities on an ad hoc
> basis.

What code do you think meets this criteria?

> - Although the kernel will always remain in C, make serious efforts to
> introduce a safe language for kernel modules and perhaps for some
> subsystems.

That is already happening for those people that really like those types
of languages.  Why not help them out with that effort as it seems to be
going slowly.

> - Build an efficient continuous integration (CI) infrastructure.

What is wrong with the one(s) that we currently have and rely on today?

> - Run a fast subset of the CI tests as a gatekeeper for all patch sets.

Um, this already happens, what needs to be added?  What tests are not
being run that would catch issues?  Why not add them to the existing
tools we all use today?

> - Run strict CI tests to ensure that userspace compatibility does not break.

What tests are those that are not being run today?

> - Run CI tests not only in virtual environments, but also on real hardware.

That's happening today, what specific platforms/hardware is not being
tested in this manner?

> - Run CI tests that aim to detect performance regressions.

Again, we are doing that, what tests need to be added to the tools?

> I realize that some companies are already running kernel testing
> infrastructure like this.

Exactly :)

> However, the development process seems to either lack the resources or
> the willingness to build a better quality assurance?

Why do you think this?  Again, specifics please.

greg k-h

Re: Is the Linux kernel underfunded? Lack of quality and security?

[Evan Rudford]

Thanks for your detailed response, I will try to address those points
one by one.

Am So., 5. Jan. 2020 um 09:15 Uhr schrieb Greg KH <greg@kroah.com>:
>
> On Sun, Jan 05, 2020 at 04:49:32AM +0100, Evan Rudford wrote:
> > The problem of underfunding plagues many open source projects.
>
> Does it?  Citation please :)
> And compared to what exactly?

Linux might be hard to compare with other open source projects because
of its enormous scale.
But anyways, I saw many different open source projects that were
underfunded based on their "GitHub-situation".
Even large projects like "webpack"  seem to suffer from underfunding
right now. Here is a citation for you:
https://webpack.js.org/blog/2020-10-10-webpack-5-release/
Also some "medium-sized" projects like
https://github.com/typeorm/typeorm tend to be underfunded unless a
company is willing to sponsor them.

> > Although code reviews and technical discussions are working well, I
> > argue that the testing infrastructure of the kernel is lacking.
>
> Does it?  No one can argue we are "doing to much testing", and more
> testing is always wanted, and happening, can you help with that effort?

Well, yes I would help, but it seems to be hard unless you are working
for one of those companies who are actually doing kernel-testing.

> > Severe bugs are discovered late, and they are discovered by developers
> > that should not be exposed to that amount of breakage.
>
> Specifics please.

This is perhaps only relevant for some specific users.
When I see a critical bug report, then I always ask the question:
Could this bug have been catched by a test-suite with reasonable
efforts compared to the size of the project?
Or is it such a weird corner case that no test-suite could have
realistically catched this bug, other than by pure luck?
For most projects, I tend to lean towards the first answer.

> Remember that Linux runs on _EVERYTHING_ so testing on _EVERYTHING_ is
> sometimes a bit hard and bugs only show up later on when people get
> around to running newer kernels on their specific hardware/workload.
>
> > Moreover, I feel that security issues do not receive enough resources.

This is perhaps hard to argue because the competition isn't good.
To be honest, I feel that neither Linux nor any other "major" OS is
reaching "high" security-standards.
It is a fallacy to think that the security-situation is good just
because nobody else is better.
And of course, rewriting Linux is nearly impossible, but I doubt that
Linux will ever become "truly secure" as long as everything is written
in C.
Let's face the reality: C is an excellent systems programming
language, but it is like an "unprotected chainsaw" with respect to
security.

> Again, citation please?  I would argue that right now we have too many
> people/resources working on security issues that are really really minor
> in the overall scheme of things.
> greg k-h

I agree that the current security-efforts might not be well-directed
for the overall scheme of things.
However, I don't think that security has "too many" people in total.
It might be true that "minor" security-issues are eating too many
resources, but there are still "non-minor" security issues that are
not yet addressed.

Re: Is the Linux kernel underfunded? Lack of quality and security?

[Steven Rostedt]

On Wed, 18 Nov 2020 18:59:09 +0100
Evan Rudford <zocker76@gmail.com> wrote:

> This is perhaps hard to argue because the competition isn't good.
> To be honest, I feel that neither Linux nor any other "major" OS is
> reaching "high" security-standards.
> It is a fallacy to think that the security-situation is good just
> because nobody else is better.
> And of course, rewriting Linux is nearly impossible, but I doubt that
> Linux will ever become "truly secure" as long as everything is written
> in C.
> Let's face the reality: C is an excellent systems programming
> language, but it is like an "unprotected chainsaw" with respect to
> security.
> 

I call "bull" on the above statement. This C isn't secure, is just a
blanket statement. Yes, C has issues, and so does assembly (which there's
plenty of that in the kernel). But with the amount of static analyzers and
fuzz testing going on, the typical C bugs that are in most projects are
well discovered in the Linux kernel.

> > Again, citation please?  I would argue that right now we have too many
> > people/resources working on security issues that are really really minor
> > in the overall scheme of things.
> > greg k-h  
> 
> I agree that the current security-efforts might not be well-directed
> for the overall scheme of things.
> However, I don't think that security has "too many" people in total.
> It might be true that "minor" security-issues are eating too many
> resources, but there are still "non-minor" security issues that are
> not yet addressed.

Funny, I find that the biggest threat to security today is coming from the
hardware. Issues like spectre and meltdown, and everything to do with
parallel programming is going to be the new age of cracking the system. And
ironically, C and assembly are probably the best languages to counter it ;-)

-- Steve

Re: Is the Linux kernel underfunded? Lack of quality and security?

[Evan Rudford]

Am Mi., 18. Nov. 2020 um 19:13 Uhr schrieb Steven Rostedt <rostedt@goodmis.org>:
>
> On Wed, 18 Nov 2020 18:59:09 +0100
> Evan Rudford <zocker76@gmail.com> wrote:
>
> > This is perhaps hard to argue because the competition isn't good.
> > To be honest, I feel that neither Linux nor any other "major" OS is
> > reaching "high" security-standards.
> > It is a fallacy to think that the security-situation is good just
> > because nobody else is better.
> > And of course, rewriting Linux is nearly impossible, but I doubt that
> > Linux will ever become "truly secure" as long as everything is written
> > in C.
> > Let's face the reality: C is an excellent systems programming
> > language, but it is like an "unprotected chainsaw" with respect to
> > security.
> >
>
> I call "bull" on the above statement. This C isn't secure, is just a
> blanket statement. Yes, C has issues, and so does assembly (which there's
> plenty of that in the kernel). But with the amount of static analyzers and
> fuzz testing going on, the typical C bugs that are in most projects are
> well discovered in the Linux kernel.

I fully agree that Linux uses many fuzzers/analyzers that are not
typically used by a lot of C-projects.
However, to claim that C is still "good practice" would be an insult
against all the research on memory safety vulnerabilities over the
last decades.
We should not trash all this research just because many programmers
are more convenient with C.
In other words, I argue that we should avoid a hostile environment
where new research-results are destroyed just because some people
think that this is "not practical".
Rust and other languages were not only invented as fun side projects,
but because the knowledge of today is way better than the knowledge
back in the 1990s when Linux wrote the initial kernel.

> > > Again, citation please?  I would argue that right now we have too many
> > > people/resources working on security issues that are really really minor
> > > in the overall scheme of things.
> > > greg k-h
> >
> > I agree that the current security-efforts might not be well-directed
> > for the overall scheme of things.
> > However, I don't think that security has "too many" people in total.
> > It might be true that "minor" security-issues are eating too many
> > resources, but there are still "non-minor" security issues that are
> > not yet addressed.
>
> Funny, I find that the biggest threat to security today is coming from the
> hardware. Issues like spectre and meltdown, and everything to do with
> parallel programming is going to be the new age of cracking the system. And
> ironically, C and assembly are probably the best languages to counter it ;-)
>
> -- Steve

I believe that Spectre and Meltdown are kind of orthogonal to many
other security threats.
Yes, I fully agree that Spectre and Meltdown need to be addressed, but
I still consider arbitrary buffer overflows in parsing libraries as
more dangerous than "typical" Spectre/Meltdown threats.

Re: Is the Linux kernel underfunded? Lack of quality and security?

[Steven Rostedt]

On Wed, 18 Nov 2020 20:30:48 +0100
Evan Rudford <zocker76@gmail.com> wrote:

> Am Mi., 18. Nov. 2020 um 19:13 Uhr schrieb Steven Rostedt <rostedt@goodmis.org>:
> >
> > On Wed, 18 Nov 2020 18:59:09 +0100
> > Evan Rudford <zocker76@gmail.com> wrote:
> >  
> > > This is perhaps hard to argue because the competition isn't good.
> > > To be honest, I feel that neither Linux nor any other "major" OS is
> > > reaching "high" security-standards.
> > > It is a fallacy to think that the security-situation is good just
> > > because nobody else is better.
> > > And of course, rewriting Linux is nearly impossible, but I doubt that
> > > Linux will ever become "truly secure" as long as everything is written
> > > in C.
> > > Let's face the reality: C is an excellent systems programming
> > > language, but it is like an "unprotected chainsaw" with respect to
> > > security.
> > >  
> >
> > I call "bull" on the above statement. This C isn't secure, is just a
> > blanket statement. Yes, C has issues, and so does assembly (which there's
> > plenty of that in the kernel). But with the amount of static analyzers and
> > fuzz testing going on, the typical C bugs that are in most projects are
> > well discovered in the Linux kernel.  
> 
> I fully agree that Linux uses many fuzzers/analyzers that are not
> typically used by a lot of C-projects.
> However, to claim that C is still "good practice" would be an insult
> against all the research on memory safety vulnerabilities over the
> last decades.
> We should not trash all this research just because many programmers
> are more convenient with C.
> In other words, I argue that we should avoid a hostile environment
> where new research-results are destroyed just because some people
> think that this is "not practical".
> Rust and other languages were not only invented as fun side projects,
> but because the knowledge of today is way better than the knowledge
> back in the 1990s when Linux wrote the initial kernel.

I'm a fan of Rust. And I'm all for moving to Rust for user space
applications, and perhaps even as Linux kernel modules (there's work on
supporting this).

But the Linux kernel itself is driven to be close to optimal in
performance, and for tooling. We push C to its limit. The Linux kernel even
does a bit of dynamic code modifications at run time. It's not about
convenience, it's about systems programing and pushing the limits of how
powerful you can make your hardware run.

There's a lot that Linux does that is out of scope of normal user space,
and the kernel is also limited in what it does as well (it has a limited
stack, and needs to take extra care not to overflow it). It limits options
of the compiler because it expects the compiler to be deterministic where
optimizations can break some of the tricks that the kernel performs.

I like to think of Linux as the German autobahn and everything else like US
highways. Linux has no speed limit, but is very strict on the rules (must
stay in the right lane, and only pass on the left, etc). When things go
wrong, there can be huge consequences. Everything else is like the US
highway. More lenient on the rules, but have to have speed limits. Things
can still go badly, but not as bad as when you are going over 100 mph (160
kph).


> 
> > > > Again, citation please?  I would argue that right now we have too many
> > > > people/resources working on security issues that are really really minor
> > > > in the overall scheme of things.
> > > > greg k-h  
> > >
> > > I agree that the current security-efforts might not be well-directed
> > > for the overall scheme of things.
> > > However, I don't think that security has "too many" people in total.
> > > It might be true that "minor" security-issues are eating too many
> > > resources, but there are still "non-minor" security issues that are
> > > not yet addressed.  
> >
> > Funny, I find that the biggest threat to security today is coming from the
> > hardware. Issues like spectre and meltdown, and everything to do with
> > parallel programming is going to be the new age of cracking the system. And
> > ironically, C and assembly are probably the best languages to counter it ;-)
> >
> > -- Steve  
> 
> I believe that Spectre and Meltdown are kind of orthogonal to many
> other security threats.
> Yes, I fully agree that Spectre and Meltdown need to be addressed, but
> I still consider arbitrary buffer overflows in parsing libraries as
> more dangerous than "typical" Spectre/Meltdown threats.

I expect that there will be a lot more vulnerable attacks in user space
than what you'll find in the kernel. Rust can help with that. If user space
is written more in Rust and we even have more Rust Linux modules, perhaps
things can get better. But there's no interest in rewriting the Linux core
code in any other language if that's what you are getting at.

-- Steve

Re: Is the Linux kernel underfunded? Lack of quality and security?

[Theodore Y. Ts'o]

On Wed, Nov 18, 2020 at 08:30:48PM +0100, Evan Rudford wrote:
> I believe that Spectre and Meltdown are kind of orthogonal to many
> other security threats.
> Yes, I fully agree that Spectre and Meltdown need to be addressed, but
> I still consider arbitrary buffer overflows in parsing libraries as
> more dangerous than "typical" Spectre/Meltdown threats.

You do realize that the kernel does very little parsing, and doesn't
use any parsing libraries, right?

In any cases, do you have a specific proposal you'd like to make?  Or
do you have something specific you'd be interested in doing, yourself,
to help make things better?

						- Ted