Re: Git Test Coverage Report (Tuesday, Sept 25)

[Ben Peart <peartben@gmail.com>]

On 9/25/2018 2:42 PM, Derrick Stolee wrote:
> In an effort to ensure new code is reasonably covered by the test suite, 
> we now have contrib/coverage-diff.sh to combine the gcov output from 
> 'make coverage-test ; make coverage-report' with the output from 'git 
> diff A B' to discover _new_ lines of code that are not covered.
> 
> This report takes the output of these results after running on four 
> branches:
> 
>          pu: 80e728fa913dc3a1165b6dec9a7afa6052a86325
> 
>         jch: 0c10634844314ab89666ed0a1c7d36dde7ac9689
> 
>        next: 76f2f5c1e34c4dbef1029e2984c2892894c444ce
> 
>      master: fe8321ec057f9231c26c29b364721568e58040f7
> 
> master@{1}: 2d3b1c576c85b7f5db1f418907af00ab88e0c303
> 
> I ran the test suite on each of these branches on an Ubuntu Linux VM, 
> and I'm missing some dependencies (like apache, svn, and perforce) so 
> not all tests are run.
> 
> I submit this output without comment. I'm taking a look especially at my 
> own lines to see where coverage can be improved.
> 

Thanks for driving this.  I think it provides an interesting view into 
new code and how well it is being tested.  In an effort to make this as 
useful as possible, we should be looking to eliminate as much noise as 
possible otherwise people will stop looking at it.

I looked at the lines that came from my patches and most if not all of 
them are only going to be executed by the test suite if the correct 
"special setup" option is enabled.  In my particular case, that is the 
option "GIT_TEST_INDEX_THREADS=<n>" as documented in t/README.

I suspect this will be the case for other code as well so I wonder if 
the tests should be run with each the GIT_TEST_* options that exist to 
exercise uncommon code paths with the test suite.  This should prevent 
false positives on code paths that are actually covered by the test 
suite as long as it is run with the appropriate option set.

I realize it would take a long time to run the entire test suite with 
all GIT_TEST_* variables so perhaps they can only be tested "as needed" 
(ie when patches add new variables in the "special setups" section of 
t/README).  This should reduce the number of combinations that need to 
be run while still eliminating many of the false positive hits.