Re: Some questions about the patching process

Re: Some questions about the patching process

[Greg Kroah-Hartman]

On Thu, Aug 27, 2020 at 12:34:57PM -0500, Qiushi Wu wrote:
> Dear Linux maintainers:
> 
> I'm Qiushi Wu, a Ph.D. student from the secure and reliable systems
> research group at the University of Minnesota. Our group strives to improve
> the security and reliability of the Linux kernel and has contributed quite
> a number of patches. We appreciate the openness of the Linux community, but
> would also like to discuss some questions about the patching process. It
> would be great if you could let us know your thoughts.
> 
> We recently found that minor patches such as one fixing a memory leak may
> introduce more critical security bugs like double free. Sometimes the
> maintainers can capture the introduced security bugs, sometimes not. This
> is understandable because the introduced bugs can be subtle and hard to
> catch due to the complexity. We are more concerned when “bad” submitters
> intentionally and stealthily introduce such security bugs via seemingly
> good patches, as they indeed have a chance to get the actually bad patches
> accepted. This is not impossible because people have incentives---to plant
> a vulnerability in a targeted driver, to get bounty rewards, etc.
> 
> Based on our patching experience, we observe several things related to the
> risks.
> 
> 1. Linux allows anyone to submit a patch because it is an open community.
> 
> 2. Maintainers tend to not accept preventive patches, i.e., a patch for a
> bug that is not really there yet, but it can be likely formed in the
> future.

How can you create a patch to prevent a bug that is not present?

But no, this is not true, look at all of the kernel hardening features
added to the kernel over the past 5+ years.  Those are specifically to
help handle the problem when there are bugs in the kernel, so that "bad
things" do not happen when they occur.

> 3. The patch review is mainly manual, so sometimes the introduced security
> bugs could be missed.

We are human, no development process can prevent this.

> We would like to know how the Linux maintainers think about these risks.

I think 2. is wrong, so it's not a risk.

And how is 1. a "risk"?

And of course, 3., humans, well, what can you do about them?  :)

> We
> would like to know if maintainers have some methods and tools (such as
> Smatch, Syzbot?) to mitigate these potential issues. We are happy to
> discuss these issues and hope our observations could raise some awareness
> of them.

How do you "raise awareness" among a developer community that is 4000
people each year (1000 are new each year), consisting of 450+ different
companies?

And yes, we have lots of tools, and run them all the time on all of our
public trees constantly.  And they fix things before they get merged and
sent out to the rest of the world.

So what specific things are you wanting to discuss here?

thanks,

greg k-h

Re: Some questions about the patching process

[Greg Kroah-Hartman]

On Thu, Aug 27, 2020 at 02:17:20PM -0500, Qiushi Wu wrote:
> Hi Greg,
> Thanks for your response!

<snip>

You responded in html format which got rejected by the public list,
please resend in text-only and I will be glad to reply.

thanks,

greg k-h

Re: Some questions about the patching process

[Qiushi Wu]

Hi Greg,
Thanks for your response!

> You responded in html format which got rejected by the public list,
> please resend in text-only and I will be glad to reply.
>
Sorry about this!


> > 1. Linux allows anyone to submit a patch because it is an open community.
> >
> And how is 1. a "risk"?

We are assuming the possibility of potential malicious commit contributors
and want to reduce the risk of accepting vulnerable patches from them.

> > We would like to know if maintainers have some methods and tools (such as
> > Smatch, Syzbot?) to mitigate these potential issues. We are happy to
> > discuss these issues and hope our observations could raise some awareness
> > of them.
>
> How do you "raise awareness" among a developer community that is 4000
> people each year (1000 are new each year), consisting of 450+ different
> companies?

Yes, this is a problem. Maybe people can summarize and pubic some security
coding guidelines for different modules of the kernel, or recommend maintainers
to use some bug detection tools to test the patches.

> And yes, we have lots of tools, and run them all the time on all of our
> public trees constantly.  And they fix things before they get merged and
> sent out to the rest of the world.
>
> So what specific things are you wanting to discuss here?

Specifically, we are curious about what kind of tools maintainers are often
used to test potential bugs or vulnerabilities?  Also, can these tools have a
high coverage rate to test corner cases like error-paths, indirect calls,
concurrency issues, etc?

Thanks again for your patience and reply : ),
Best regards,
Qiushi

Re: Some questions about the patching process

[Greg Kroah-Hartman]

On Fri, Aug 28, 2020 at 02:59:25AM -0500, Qiushi Wu wrote:
> Hi Greg,
> Thanks for your response!
> 
> > You responded in html format which got rejected by the public list,
> > please resend in text-only and I will be glad to reply.
> >
> Sorry about this!
> 
> 
> > > 1. Linux allows anyone to submit a patch because it is an open community.
> > >
> > And how is 1. a "risk"?
> 
> We are assuming the possibility of potential malicious commit contributors
> and want to reduce the risk of accepting vulnerable patches from them.

No, you are thinking about this all wrong.

ALL contributors make mistakes, you should not be treating anyone
different from anyone else.  I think I probably have contributed more
bugs than many contributors, does that make me a "malicious"
contributor?  Or just someone who contributes a lot?

So checking on patches needs to be done for everyone, right?

We have an idea of "trust" in kernel development, it's how we work so
well.  I don't trust people not that they will always get things
"correct", but rather that they will be around to fix it when they get
it "wrong", as everyone makes mistakes, we are all human.

So we trust people who we accept pull requests from, we don't review
their contributions because we trust that they did, and again, they will
fix it when it goes wrong.

> > > We would like to know if maintainers have some methods and tools (such as
> > > Smatch, Syzbot?) to mitigate these potential issues. We are happy to
> > > discuss these issues and hope our observations could raise some awareness
> > > of them.
> >
> > How do you "raise awareness" among a developer community that is 4000
> > people each year (1000 are new each year), consisting of 450+ different
> > companies?
> 
> Yes, this is a problem. Maybe people can summarize and pubic some security
> coding guidelines for different modules of the kernel, or recommend maintainers
> to use some bug detection tools to test the patches.

We do both today quite well, why do you think this is not the case?

> > And yes, we have lots of tools, and run them all the time on all of our
> > public trees constantly.  And they fix things before they get merged and
> > sent out to the rest of the world.
> >
> > So what specific things are you wanting to discuss here?
> 
> Specifically, we are curious about what kind of tools maintainers are often
> used to test potential bugs or vulnerabilities?

We use lots, everything we do is in the open, I suggest doing some
research first please.

> Also, can these tools have a
> high coverage rate to test corner cases like error-paths, indirect calls,
> concurrency issues, etc?

Since when does code coverage actually matter as a viable metric?

Look at the tools we use, again, it's all in the open, and tell us what
we could be doing differently by offering to help us implement those
tools into our workflows.  That would be the best way to contribute
here, don't you agree?

thanks,

greg k-h

Re: Some questions about the patching process

[Qiushi Wu]

On Fri, Aug 28, 2020 at 3:25 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Fri, Aug 28, 2020 at 02:59:25AM -0500, Qiushi Wu wrote:
> > Hi Greg,
> > Thanks for your response!
> >
> > > You responded in html format which got rejected by the public list,
> > > please resend in text-only and I will be glad to reply.
> > >
> > Sorry about this!
> >
> >
> > > > 1. Linux allows anyone to submit a patch because it is an open community.
> > > >
> > > And how is 1. a "risk"?
> >
> > We are assuming the possibility of potential malicious commit contributors
> > and want to reduce the risk of accepting vulnerable patches from them.
>
> No, you are thinking about this all wrong.
>
> ALL contributors make mistakes, you should not be treating anyone
> different from anyone else.  I think I probably have contributed more
> bugs than many contributors, does that make me a "malicious"
> contributor?  Or just someone who contributes a lot?

Sorry for my confusion!  I don't mean to say that our kernel
contributors are 'malicious' or some similar, and previously I have also
made mistakes and send buggy code into the kernel accidentally.
Also, we are trying to summarize the methods to efficiently auditing
patches to prevent potential issues in the patch.


> So checking on patches needs to be done for everyone, right?
>
> We have an idea of "trust" in kernel development, it's how we work so
> well.  I don't trust people not that they will always get things
> "correct", but rather that they will be around to fix it when they get
> it "wrong", as everyone makes mistakes, we are all human.
>
> So we trust people who we accept pull requests from, we don't review
> their contributions because we trust that they did, and again, they will
> fix it when it goes wrong.

agree : )


> We use lots, everything we do is in the open, I suggest doing some
> research first please.
> > Also, can these tools have a
> > high coverage rate to test corner cases like error-paths, indirect calls,
> > concurrency issues, etc?
>
> Since when does code coverage actually matter as a viable metric?
>
> Look at the tools we use, again, it's all in the open, and tell us what
> we could be doing differently by offering to help us implement those
> tools into our workflows.  That would be the best way to contribute
> here, don't you agree?
Okay, I see.

Thanks again for your patient reply, and apologies for my confusing description.

Best regards,
Qiushi