[Qemu-devel] [BUG] gcov support appears to be broken

[Qemu-devel] [BUG] gcov support appears to be broken

[Aleksandar Markovic]

Hello, according to out docs, here is the procedure that should produce coverage report for execution of the complete "make check":

#./configure --enable-gcov
#make
#make check
#make coverage-report

It seems that first three commands execute as expected. (For example, there are plenty of files generated by "make check" that would've not been generated if "enable-gcov" hadn't been chosen.) However, the last command complains about some missing files related to FP support. If those files are added (for example, artificially, using "touch <missing-file"), that it starts complaining about missing some decodetree-generated files. Other kinds of files are involved too.

It would be nice to have coverage support working. Please somebody take a look, or explain if I make a mistake or misunderstood our gcov support.

Yours,
Aleksandar

Re: [Qemu-devel] [BUG] gcov support appears to be broken

[Peter Maydell]

On Mon, 5 Aug 2019 at 11:39, Aleksandar Markovic <amarkovic@wavecomp.com> wrote:
>
> Hello, according to out docs, here is the procedure that should produce coverage report for execution of the complete "make check":
>
> #./configure --enable-gcov
> #make
> #make check
> #make coverage-report
>
> It seems that first three commands execute as expected. (For example, there are plenty of files generated by "make check" that would've not been generated if "enable-gcov" hadn't been chosen.) However, the last command complains about some missing files related to FP support. If those files are added (for example, artificially, using "touch <missing-file"), that it starts complaining about missing some decodetree-generated files. Other kinds of files are involved too.
>
> It would be nice to have coverage support working. Please somebody take a look, or explain if I make a mistake or misunderstood our gcov support.

Cc'ing Alex who's probably the closest we have to a gcov expert.

(make/make check of a --enable-gcov build is in the set of things our
Travis CI setup runs, so we do defend that part against regressions.)

thanks
-- PMM

Re: [Qemu-devel] [BUG] gcov support appears to be broken

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Mon, 5 Aug 2019 at 11:39, Aleksandar Markovic <amarkovic@wavecomp.com> wrote:
>>
>> Hello, according to out docs, here is the procedure that should produce coverage report for execution of the complete "make check":
>>
>> #./configure --enable-gcov
>> #make
>> #make check
>> #make coverage-report
>>
>> It seems that first three commands execute as expected. (For example,
>> there are plenty of files generated by "make check" that would've not
>> been generated if "enable-gcov" hadn't been chosen.) However, the
>> last command complains about some missing files related to FP
>> support. If those files are added (for example, artificially, using
>> "touch <missing-file"), that it starts complaining about missing some
>> decodetree-generated files. Other kinds of files are involved too.

The gcov tool is fairly noisy about missing files but that just
indicates the tests haven't exercised those code paths. "make check"
especially doesn't touch much of the TCG code and a chunk of floating
point.

>>
>> It would be nice to have coverage support working. Please somebody
>> take a look, or explain if I make a mistake or misunderstood our gcov
>> support.

So your failure mode is no report is generated at all? It's working for
me here.

>
> Cc'ing Alex who's probably the closest we have to a gcov expert.
>
> (make/make check of a --enable-gcov build is in the set of things our
> Travis CI setup runs, so we do defend that part against regressions.)

We defend the build but I have just checked and it seems our
check_coverage script is currently failing:

  https://travis-ci.org/stsquad/qemu/jobs/567809808#L10328

But as it's an after_success script it doesn't fail the build.

>
> thanks
> -- PMM


--
Alex Bennée

Re: [Qemu-devel] [BUG] gcov support appears to be broken

[Aleksandar Markovic]

>> #./configure --enable-gcov
>> #make
>> #make check
>> #make coverage-report
>>
>> It seems that first three commands execute as expected. (For example,
>> there are plenty of files generated by "make check" that would've not
>> been generated if "enable-gcov" hadn't been chosen.) However, the
>> last command complains about some missing files related to FP

> So your failure mode is no report is generated at all? It's working for
> me here.

Alex, no report is generated for my test setups - in fact, "make coverage-report" even says that it explicitly deletes what appears to be the main coverage report html file).

This is the terminal output of an unsuccessful executions of "make coverage-report" for recent ToT:

~/Build/qemu-TOT-TEST$ make coverage-report
make[1]: Entering directory '/home/user/Build/qemu-TOT-TEST/slirp'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/user/Build/qemu-TOT-TEST/slirp'
	CHK version_gen.h
  GEN     coverage-report.html
Traceback (most recent call last):
  File "/usr/bin/gcovr", line 1970, in <module>
    print_html_report(covdata, options.html_details)
  File "/usr/bin/gcovr", line 1473, in print_html_report
    INPUT = open(data['FILENAME'], 'r')
IOError: [Errno 2] No such file or directory: 'wrap.inc.c'
Makefile:1048: recipe for target '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html' failed
make: *** [/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html] Error 1
make: *** Deleting file '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html'

This instance is executed in QEMU 3.0 source tree: (so, it looks the problem existed for quite some time)

~/Build/qemu-3.0$ make coverage-report
	CHK version_gen.h
  GEN     coverage-report.html
Traceback (most recent call last):
  File "/usr/bin/gcovr", line 1970, in <module>
    print_html_report(covdata, options.html_details)
  File "/usr/bin/gcovr", line 1473, in print_html_report
    INPUT = open(data['FILENAME'], 'r')
IOError: [Errno 2] No such file or directory: '/home/user/Build/qemu-3.0/target/openrisc/decode.inc.c'
Makefile:992: recipe for target '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html' failed
make: *** [/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html] Error 1
make: *** Deleting file '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html'

Fond regards,
Aleksandar


> Alex Bennée

Re: [Qemu-devel] [BUG] gcov support appears to be broken

[Aleksandar Markovic]

>> #./configure --enable-gcov
>> #make
>> #make check
>> #make coverage-report
>>
>> It seems that first three commands execute as expected. (For example,
>> there are plenty of files generated by "make check" that would've not
>> been generated if "enable-gcov" hadn't been chosen.) However, the
>> last command complains about some missing files related to FP

> So your failure mode is no report is generated at all? It's working for
> me here.

Another piece of info:

~/Build/qemu-TOT-TEST$ gcov --version
gcov (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or 
FITNESS FOR A PARTICULAR PURPOSE.

:~/Build/qemu-TOT-TEST$ gcc --version
gcc (Ubuntu 7.2.0-1ubuntu1~16.04) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.




Alex, no report is generated for my test setups - in fact, "make coverage-report" even says that it explicitly deletes what appears to be the main coverage report html file).

This is the terminal output of an unsuccessful executions of "make coverage-report" for recent ToT:

~/Build/qemu-TOT-TEST$ make coverage-report
make[1]: Entering directory '/home/user/Build/qemu-TOT-TEST/slirp'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/user/Build/qemu-TOT-TEST/slirp'
        CHK version_gen.h
  GEN     coverage-report.html
Traceback (most recent call last):
  File "/usr/bin/gcovr", line 1970, in <module>
    print_html_report(covdata, options.html_details)
  File "/usr/bin/gcovr", line 1473, in print_html_report
    INPUT = open(data['FILENAME'], 'r')
IOError: [Errno 2] No such file or directory: 'wrap.inc.c'
Makefile:1048: recipe for target '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html' failed
make: *** [/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html] Error 1
make: *** Deleting file '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html'

This instance is executed in QEMU 3.0 source tree: (so, it looks the problem existed for quite some time)

~/Build/qemu-3.0$ make coverage-report
        CHK version_gen.h
  GEN     coverage-report.html
Traceback (most recent call last):
  File "/usr/bin/gcovr", line 1970, in <module>
    print_html_report(covdata, options.html_details)
  File "/usr/bin/gcovr", line 1473, in print_html_report
    INPUT = open(data['FILENAME'], 'r')
IOError: [Errno 2] No such file or directory: '/home/user/Build/qemu-3.0/target/openrisc/decode.inc.c'
Makefile:992: recipe for target '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html' failed
make: *** [/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html] Error 1
make: *** Deleting file '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html'

Fond regards,
Aleksandar


> Alex Bennée

Re: [Qemu-devel] [BUG] gcov support appears to be broken - solved?

[Aleksandar Markovic]

>> #./configure --enable-gcov
>> #make
>> #make check
>> #make coverage-report
>>
>> It seems that first three commands execute as expected. (For example,
>> there are plenty of files generated by "make check" that would've not
>> been generated if "enable-gcov" hadn't been chosen.) However, the
>> last command complains about some missing files related to FP

> So your failure mode is no report is generated at all? It's working for
> me here.

Alex, here is the thing:

Seeing that my gcovr is relatively old (2014) 3.2 version, I upgraded it from git repo to the most recent 4.1 (actually, to a dev version, from the very tip of the tree), and "make coverage-report" started generating coverage reports. It did emit some error messages (totally different than previous), but still it did not stop like it used to do with gcovr 3.2.

Perhaps you would want to add some gcov/gcovr minimal version info in our docs. (or at least a statement "this was tested with such and such gcc, gcov and gcovr", etc.?)

Coverage report looked fine at first glance, but it a kind of disappointed me when I digged deeper into its content - for example, it shows very low coverage for our FP code (softfloat), while, in fact, we know that "make check" contains detailed tests on FP functionalities. But this is most likely a separate problem of a very different nature, perhaps the issue of separate git repo for FP tests (testfloat) that our FP tests use as a mid-layer.

I'll try how everything works with my test examples, and will let you know.

Your help is greatly appreciated,
Aleksandar

Fond regards,
Aleksandar


> Alex Bennée

Re: [Qemu-devel] [BUG] gcov support appears to be broken - solved?

[Alex Bennée]

Aleksandar Markovic <amarkovic@wavecomp.com> writes:

>>> #./configure --enable-gcov
>>> #make
>>> #make check
>>> #make coverage-report
>>>
>>> It seems that first three commands execute as expected. (For example,
>>> there are plenty of files generated by "make check" that would've not
>>> been generated if "enable-gcov" hadn't been chosen.) However, the
>>> last command complains about some missing files related to FP
>
>> So your failure mode is no report is generated at all? It's working for
>> me here.
>
> Alex, here is the thing:
>
> Seeing that my gcovr is relatively old (2014) 3.2 version, I upgraded it from git repo to the most recent 4.1 (actually, to a dev version, from the very tip of the tree), and "make coverage-report" started generating coverage reports. It did emit some error messages (totally different than previous), but still it did not stop like it used to do with gcovr 3.2.
>
> Perhaps you would want to add some gcov/gcovr minimal version info in our docs. (or at least a statement "this was tested with such and such gcc, gcov and gcovr", etc.?)
>
> Coverage report looked fine at first glance, but it a kind of
> disappointed me when I digged deeper into its content - for example,
> it shows very low coverage for our FP code (softfloat), while, in
> fact, we know that "make check" contains detailed tests on FP
> functionalities. But this is most likely a separate problem of a very
> different nature, perhaps the issue of separate git repo for FP tests
> (testfloat) that our FP tests use as a mid-layer.

I get:

68.6 %	2593 / 3782	62.2 %	1690 / 2718

Which is not bad considering we don't exercise the 80 and 128 bit
softfloat code at all (which is not shared by the re-factored 16/32/64
bit code).

>
> I'll try how everything works with my test examples, and will let you know.
>
> Your help is greatly appreciated,
> Aleksandar
>
> Fond regards,
> Aleksandar
>
>
>> Alex Bennée


--
Alex Bennée

Re: [Qemu-devel] [BUG] gcov support appears to be broken - solved?

[Aleksandar Markovic]

> > it shows very low coverage for our FP code (softfloat), while, in
> > fact, we know that "make check" contains detailed tests on FP
> > functionalities. But this is most likely a separate problem of a very
> > different nature, perhaps the issue of separate git repo for FP tests
> > (testfloat) that our FP tests use as a mid-layer.
> 
> I get:
> 
> 68.6 %  2593 / 3782     62.2 %  1690 / 2718
> 

I would expect that kind of result too.

However, I get:

File: 	fpu/softfloat.c 		Lines: 	8 	3334 	0.2 %
Date: 	2019-08-05 19:56:58 		Branches: 	3 	2376 	0.1 %

:(

OK, I'll try to figure that out, and most likely I could live with it if it is an isolated problem.

Thank you for your assistance in this matter,
Aleksandar

> Which is not bad considering we don't exercise the 80 and 128 bit
> softfloat code at all (which is not shared by the re-factored 16/32/64
> bit code).
> 
> Alex Bennée

[Qemu-devel] ]Re: [BUG] gcov support appears to be broken - solved?

[Aleksandar Markovic]

> > it shows very low coverage for our FP code (softfloat), while, in
> > fact, we know that "make check" contains detailed tests on FP
> > functionalities. But this is most likely a separate problem of a very
> > different nature, perhaps the issue of separate git repo for FP tests
> > (testfloat) that our FP tests use as a mid-layer.
> 
> I get:
> 
> 68.6 %  2593 / 3782     62.2 %  1690 / 2718
> 

This problem is solved too. (and it is my fault)

I worked with multiple versions of QEMU, and my previous low-coverage results were for QEMU 3.0, and for that version the directory tests/fp did not even exist. :D (<blush>)

For QEMU ToT, I get now:

fpu/softfloat.c 	
	68.8 % 	2592 / 3770 	62.3 % 	1693 / 2718

which is identical for all intents and purposes to your result.

Yours cordially,
Aleksandar