Re: Coverity, was Re: What's cooking in git.git (Oct 2021, #02; Wed, 6)

[Johannes Schindelin <Johannes.Schindelin@gmx.de>]

[-- Attachment #1: Type: text/plain, Size: 3679 bytes --]

Hi Peff,

On Tue, 16 Aug 2022, Jeff King wrote:

> On Tue, Aug 16, 2022 at 11:05:48AM +0200, Johannes Schindelin wrote:
>
> > > It sounds like Taylor is volunteering to set up the Coverity side for
> > > git.git, and I can help him with getting those COVERITY_* variables into
> > > the GitHub environment.
> >
> > Given the challenges with Coverity (false positives, lack of support on
> > Synopsys' side, severely limited access to the reports), and given the
> > renewed efforts by OSTIF that focus not on Coverity but on CodeQL, I am
> > in favor of abandoning the idea to integrate Coverity in our GitHub
> > workflow.
> >
> > Regarding CodeQL, I am still uncertain what level of integration we will
> > end up with, and the contacts I am working with are currently all on
> > vacation, but I am confident that we will have an easier time going
> > forward with static analysis using CodeQL instead of Coverity.
>
> OK. I haven't been that impressed with CodeQL for C so far, but it may
> be getting better.

If your lack of being impressed stems from CodeQL's default suite catching
very, very few issues, I agree with you.

The reason why I am more hopeful about CodeQL than about Coverity is that
I now have a contact to work with (although they are currently on
vacation, so I get to work on other things for now, including `merge-ort`
and trying to integrate `mimalloc` in Git for Windows). And that is one
more contact who is willing to work with me than I ever had on the
Coverity side of things.

And apparently CodeQL's default settings optimize for reducing false
positives in an attempt to avoid scaring potential users away.

But I have credible assurances that CodeQL has many more checks in store
that simply need to be enabled in order to catch way more issues, at the
expense of risking more false positives.

> I certainly would be happier with a system that made it easier to
> display and share reports.
>
> Coverity does have a lot of false positives, but I've at least been able
> to pick useful fixes out of them (especially because it is good about
> saying "here are 5 new things to look at"). I've been continuing to
> build my private branch with it, so we'll see if it turns up anything
> useful. I do agree that inflicting it on ordinary users may be
> counter-productive (I often have to stare really hard to understand why
> its false positives are false, and that is not something I would wish
> on, say, a GGG user).

Don't get me wrong, I do not plan on dropping the Coverity builds of the
Git for Windows project's `main` branch at
https://dev.azure.com/git-for-windows/git/_build?definitionId=35.

As you probably recall, I specifically looked through the Coverity report
in advance of v2.37.0 and tried to address some of the issues in
https://lore.kernel.org/git/pull.1264.git.1655336146.gitgitgadget@gmail.com/,
with mixed results (mainly because of some time constraints on my side,
combined with your willingness to help fix some issues with my patches, as
well as René's, Taylor's and Junio's excellent assistance).

However, the sheer amount of false positives, with intentional issues
being a close second (in test helpers, we are not all that strict about
releasing memory, for example), makes it a tough sell to ask more than
just a few very dedicated contributors to have a look at the reports.

With CodeQL, I am optimistic that we can get it to a point where the
burden can be carried by a larger group of people, with the help of
enthusiastic CodeQL experts, which also means that the reports have a
bigger chance at making Git safer.

Ciao,
Dscho