* [OpenBMC]: Unit test coverage analysis on openbmc repositories
@ 2020-02-28 15:35 Lakshminarayana R Kammath
2020-02-28 16:18 ` Brad Bishop
` (4 more replies)
0 siblings, 5 replies; 16+ messages in thread
From: Lakshminarayana R Kammath @ 2020-02-28 15:35 UTC (permalink / raw)
To: openbmc
[-- Attachment #1: Type: text/html, Size: 22121 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 15:35 [OpenBMC]: Unit test coverage analysis on openbmc repositories Lakshminarayana R Kammath
@ 2020-02-28 16:18 ` Brad Bishop
2020-02-28 16:30 ` Lakshminarayana R Kammath
` (3 subsequent siblings)
4 siblings, 0 replies; 16+ messages in thread
From: Brad Bishop @ 2020-02-28 16:18 UTC (permalink / raw)
To: Lakshminarayana R Kammath; +Cc: openbmc
at 10:35 AM, Lakshminarayana R Kammath <lkammath@in.ibm.com> wrote:
> Hi All,
>
> Recently we worked on pulling the UT coverage analysis from all the
> available OpenBMC repositories
>
> https://github.com/openbmc/openbmc-build-scripts/commit/229b76a95f87af60c976a0c0dfe84716c9ce5318
>
> This script does following
> • Clone the repo
> • Use the CI build environment to build code
> • Publish the result in the following format
> Latest report:
> *************************UNIT TEST COVERAGE REPORT*************************
> https://github.com/openbmc/bmcweb.git NO
> https://github.com/openbmc/boost-dbus.git ERROR
> https://github.com/openbmc/btbridge.git NO
> https://github.com/openbmc/dbus-sensors.git NO
> https://github.com/openbmc/entity-manager.git NO
> https://github.com/openbmc/fb-ipmi-oem.git NO
> https://github.com/openbmc/google-ipmi-i2c.git ERROR
> https://github.com/openbmc/google-ipmi-sys.git YES, UNIT TEST
> https://github.com/openbmc/gpioplus.git YES, COVERAGE
> https://github.com/openbmc/hiomapd.git YES, UNIT TEST
> https://github.com/openbmc/ibm-dbus-interfaces.git NO
> https://github.com/openbmc/ibm-logging.git YES, UNIT TEST
> https://github.com/openbmc/ibm-pldm-oem.git ERROR
> https://github.com/openbmc/inarp.git ERROR
> https://github.com/openbmc/intel-dbus-interfaces.git NO
> https://github.com/openbmc/intel-ipmi-oem.git NO
> https://github.com/openbmc/ipmbbridge.git NO
> https://github.com/openbmc/ipmi-blob-tool.git YES, UNIT TEST
> https://github.com/openbmc/ipmi-fru-parser.git NO
> https://github.com/openbmc/ipmitool.git ERROR
> https://github.com/openbmc/jsnbd.git ERROR
> https://github.com/openbmc/kcsbridge.git NO
> https://github.com/openbmc/lenovo-ipmi-oem.git NO
> https://github.com/openbmc/libmctp.git YES, UNIT TEST
> https://github.com/openbmc/librrdplus.git NO
> https://github.com/openbmc/obmc-console.git YES, UNIT TEST
> https://github.com/openbmc/obmc-ikvm.git NO
> https://github.com/openbmc/openbmc.git NO
> https://github.com/openbmc/openbmc-tools.git NO
> https://github.com/openbmc/openbmc.github.io.git NO
> https://github.com/openbmc/openpower-dbus-interfaces.git NO
> https://github.com/openbmc/openpower-debug-collector.git NO
> https://github.com/openbmc/openpower-host-ipmi-flash.git NO
> https://github.com/openbmc/openpower-host-ipmi-oem.git NO
> https://github.com/openbmc/openpower-hw-diags.git YES, COVERAGE
> https://github.com/openbmc/openpower-inventory-upload.git NO
> https://github.com/openbmc/openpower-libhei.git YES, COVERAGE
> https://github.com/openbmc/openpower-logging.git NO
> https://github.com/openbmc/openpower-occ-control.git YES, UNIT TEST
> https://github.com/openbmc/openpower-pnor-code-mgmt.git YES, UNIT TEST
> https://github.com/openbmc/openpower-power-control.git NO
> https://github.com/openbmc/openpower-proc-control.git YES, UNIT TEST
> https://github.com/openbmc/openpower-sbe-interface.git NO
> https://github.com/openbmc/openpower-vpd-parser.git YES, UNIT TEST
> https://github.com/openbmc/pam-ipmi.git NO
> https://github.com/openbmc/phosphor-bmc-code-mgmt.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-buttons.git NO
> https://github.com/openbmc/phosphor-certificate-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-dbus-interfaces.git NO
> https://github.com/openbmc/phosphor-dbus-monitor.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-debug-collector.git NO
> https://github.com/openbmc/phosphor-ecc.git NO
> https://github.com/openbmc/phosphor-event.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-fan-presence.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-gpio-monitor.git YES, COVERAGE
> https://github.com/openbmc/phosphor-host-ipmid.git ERROR
> https://github.com/openbmc/phosphor-host-postd.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-hostlogger.git NO
> https://github.com/openbmc/phosphor-hwmon.git NO
> https://github.com/openbmc/phosphor-inventory-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-ipmi-blobs.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-ipmi-blobs-binarystore.git YES, UNIT
> TEST
> https://github.com/openbmc/phosphor-ipmi-ethstats.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-ipmi-flash.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-led-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-led-sysfs.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-logging.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-mboxd.git ERROR
> https://github.com/openbmc/phosphor-misc.git NO
> https://github.com/openbmc/phosphor-mrw-tools.git NO
> https://github.com/openbmc/phosphor-net-ipmid.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-networkd.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-nvme.git NO
> https://github.com/openbmc/phosphor-objmgr.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-pcie-presence.git NO
> https://github.com/openbmc/phosphor-pid-control.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-post-code-manager.git NO
> https://github.com/openbmc/phosphor-power.git YES, COVERAGE
> https://github.com/openbmc/phosphor-power-control.git NO
> https://github.com/openbmc/phosphor-power-monitor.git NO
> https://github.com/openbmc/phosphor-psu-code-mgmt.git YES, COVERAGE
> https://github.com/openbmc/phosphor-rest-server.git NO
> https://github.com/openbmc/phosphor-sel-logger.git NO
> https://github.com/openbmc/phosphor-settingsd.git NO
> https://github.com/openbmc/phosphor-snmp.git ERROR
> https://github.com/openbmc/phosphor-state-manager.git YES, COVERAGE
> https://github.com/openbmc/phosphor-time-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-u-boot-env-mgr.git NO
> https://github.com/openbmc/phosphor-user-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-watchdog.git YES, COVERAGE
> https://github.com/openbmc/phosphor-webui.git NO
> https://github.com/openbmc/pldm.git YES, COVERAGE
> https://github.com/openbmc/pyphosphor.git NO
> https://github.com/openbmc/qualcomm-ipmi-oem.git NO
> https://github.com/openbmc/qualcomm-misc.git NO
> https://github.com/openbmc/quanta-ipmi-oem.git NO
> https://github.com/openbmc/rest-dbus.git NO
> https://github.com/openbmc/rrd-ipmi-blobs.git NO
> https://github.com/openbmc/s2600wf-misc.git NO
> https://github.com/openbmc/sdbusplus.git YES, UNIT TEST
> https://github.com/openbmc/sdeventplus.git YES, COVERAGE
> https://github.com/openbmc/skeleton.git ERROR
> https://github.com/openbmc/slpd-lite.git NO
> https://github.com/openbmc/smbios-mdr.git ERROR
> https://github.com/openbmc/stdplus.git YES, COVERAGE
> https://github.com/openbmc/telemetry.git NO
> https://github.com/openbmc/u-boot NO
> https://github.com/openbmc/webui-vue.git NO
> https://github.com/openbmc/witherspoon-pfault-analysis.git YES, COVERAGE
> https://github.com/openbmc/x86-power-control.git NO
> *************************UNIT TEST COVERAGE REPORT*************************
> SUMMARY:
> TOTAL REPOSITORIES : 110
> TESTED REPOSITORIES : 43
> ERROR : 11
> COVERAGE REPORT : 12
> UNIT TEST REPORT : 31
> NO REPORT : 56
> ***************************************************************************
Very nice Lakshmi!
Some of the repos have been archived. Can the script be updated to check
for that and skip it if so?
boost-dbus, inarp are a couple - there are likely more.
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 15:35 [OpenBMC]: Unit test coverage analysis on openbmc repositories Lakshminarayana R Kammath
2020-02-28 16:18 ` Brad Bishop
@ 2020-02-28 16:30 ` Lakshminarayana R Kammath
2020-02-28 16:31 ` Brad Bishop
2020-02-28 18:20 ` Patrick Venture
` (2 subsequent siblings)
4 siblings, 1 reply; 16+ messages in thread
From: Lakshminarayana R Kammath @ 2020-02-28 16:30 UTC (permalink / raw)
To: bradleyb; +Cc: openbmc
[-- Attachment #1: Type: text/html, Size: 19589 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 16:30 ` Lakshminarayana R Kammath
@ 2020-02-28 16:31 ` Brad Bishop
2020-02-28 16:36 ` Brad Bishop
2020-02-28 16:39 ` Lakshminarayana R Kammath
0 siblings, 2 replies; 16+ messages in thread
From: Brad Bishop @ 2020-02-28 16:31 UTC (permalink / raw)
To: Lakshminarayana R Kammath; +Cc: openbmc
at 11:30 AM, Lakshminarayana R Kammath <lkammath@in.ibm.com> wrote:
> sure Brad!
>
> If Some one can confirm the list of repo's that can be ignored we can put
> them to blacklist
Is it possible to use the GitHub api to query this? Then we don’t need to
maintain a blacklist. How are you building the list of repos?
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 16:31 ` Brad Bishop
@ 2020-02-28 16:36 ` Brad Bishop
2020-03-04 6:42 ` Lakshminarayana R Kammath
2020-02-28 16:39 ` Lakshminarayana R Kammath
1 sibling, 1 reply; 16+ messages in thread
From: Brad Bishop @ 2020-02-28 16:36 UTC (permalink / raw)
To: Lakshminarayana R Kammath; +Cc: openbmc
at 11:31 AM, Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
> at 11:30 AM, Lakshminarayana R Kammath <lkammath@in.ibm.com> wrote:
>
>> sure Brad!
>>
>> If Some one can confirm the list of repo's that can be ignored we can
>> put them to blacklist
>
> Is it possible to use the GitHub api to query this? Then we don’t need
> to maintain a blacklist. How are you building the list of repos?
https://developer.github.com/v3/repos/#list-organization-repositories
there is an “archived” property in the response to this list-org-repos api
that seems like it would work?
This looks like an interesting python module that might make the API
interaction easier...
https://pygithub.readthedocs.io/en/latest/
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 16:31 ` Brad Bishop
2020-02-28 16:36 ` Brad Bishop
@ 2020-02-28 16:39 ` Lakshminarayana R Kammath
2020-02-28 18:12 ` Nancy Yuen
1 sibling, 1 reply; 16+ messages in thread
From: Lakshminarayana R Kammath @ 2020-02-28 16:39 UTC (permalink / raw)
To: bradleyb; +Cc: openbmc
[-- Attachment #1: Type: text/html, Size: 1735 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 16:39 ` Lakshminarayana R Kammath
@ 2020-02-28 18:12 ` Nancy Yuen
0 siblings, 0 replies; 16+ messages in thread
From: Nancy Yuen @ 2020-02-28 18:12 UTC (permalink / raw)
To: Lakshminarayana R Kammath; +Cc: Brad Bishop, OpenBMC Maillist
[-- Attachment #1: Type: text/plain, Size: 935 bytes --]
This is great information! Thank
----------
Nancy
On Fri, Feb 28, 2020 at 8:40 AM Lakshminarayana R Kammath <
lkammath@in.ibm.com> wrote:
> Yes, We can do that. Just need to figure out the API's
>
> Thanks & Regards,
> Lakshminarayana Kamath
>
>
>
> ----- Original message -----
> From: Brad Bishop <bradleyb@fuzziesquirrel.com>
> To: Lakshminarayana R Kammath <lkammath@in.ibm.com>
> Cc: openbmc@lists.ozlabs.org
> Subject: [EXTERNAL] Re: [OpenBMC]: Unit test coverage analysis on openbmc
> repositories
> Date: Fri, Feb 28, 2020 10:01 PM
>
> at 11:30 AM, Lakshminarayana R Kammath <lkammath@in.ibm.com> wrote:
>
> > sure Brad!
> >
> > If Some one can confirm the list of repo's that can be ignored we can
> put
> > them to blacklist
>
> Is it possible to use the GitHub api to query this? Then we don’t need to
>
> maintain a blacklist. How are you building the list of repos?
>
>
>
>
>
[-- Attachment #2: Type: text/html, Size: 2395 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 15:35 [OpenBMC]: Unit test coverage analysis on openbmc repositories Lakshminarayana R Kammath
2020-02-28 16:18 ` Brad Bishop
2020-02-28 16:30 ` Lakshminarayana R Kammath
@ 2020-02-28 18:20 ` Patrick Venture
2020-03-05 1:25 ` James Feist
2020-03-03 4:48 ` Andrew Jeffery
2020-03-04 18:43 ` Andrew Geissler
4 siblings, 1 reply; 16+ messages in thread
From: Patrick Venture @ 2020-02-28 18:20 UTC (permalink / raw)
To: Lakshminarayana R Kammath; +Cc: OpenBMC Maillist
On Fri, Feb 28, 2020 at 7:36 AM Lakshminarayana R Kammath
<lkammath@in.ibm.com> wrote:
>
> Hi All,
>
> Recently we worked on pulling the UT coverage analysis from all the available OpenBMC repositories
>
> https://github.com/openbmc/openbmc-build-scripts/commit/229b76a95f87af60c976a0c0dfe84716c9ce5318
>
> This script does following
>
> Clone the repo
> Use the CI build environment to build code
> Publish the result in the following format
>
> Latest report:
> *************************UNIT TEST COVERAGE REPORT*************************
> https://github.com/openbmc/bmcweb.git NO
> https://github.com/openbmc/boost-dbus.git ERROR
> https://github.com/openbmc/btbridge.git NO
> https://github.com/openbmc/dbus-sensors.git NO
> https://github.com/openbmc/entity-manager.git NO
> https://github.com/openbmc/fb-ipmi-oem.git NO
> https://github.com/openbmc/google-ipmi-i2c.git ERROR
> https://github.com/openbmc/google-ipmi-sys.git YES, UNIT TEST
> https://github.com/openbmc/gpioplus.git YES, COVERAGE
> https://github.com/openbmc/hiomapd.git YES, UNIT TEST
> https://github.com/openbmc/ibm-dbus-interfaces.git NO
> https://github.com/openbmc/ibm-logging.git YES, UNIT TEST
> https://github.com/openbmc/ibm-pldm-oem.git ERROR
> https://github.com/openbmc/inarp.git ERROR
> https://github.com/openbmc/intel-dbus-interfaces.git NO
> https://github.com/openbmc/intel-ipmi-oem.git NO
> https://github.com/openbmc/ipmbbridge.git NO
> https://github.com/openbmc/ipmi-blob-tool.git YES, UNIT TEST
> https://github.com/openbmc/ipmi-fru-parser.git NO
> https://github.com/openbmc/ipmitool.git ERROR
> https://github.com/openbmc/jsnbd.git ERROR
> https://github.com/openbmc/kcsbridge.git NO
> https://github.com/openbmc/lenovo-ipmi-oem.git NO
> https://github.com/openbmc/libmctp.git YES, UNIT TEST
> https://github.com/openbmc/librrdplus.git NO
> https://github.com/openbmc/obmc-console.git YES, UNIT TEST
> https://github.com/openbmc/obmc-ikvm.git NO
> https://github.com/openbmc/openbmc.git NO
> https://github.com/openbmc/openbmc-tools.git NO
> https://github.com/openbmc/openbmc.github.io.git NO
> https://github.com/openbmc/openpower-dbus-interfaces.git NO
> https://github.com/openbmc/openpower-debug-collector.git NO
> https://github.com/openbmc/openpower-host-ipmi-flash.git NO
> https://github.com/openbmc/openpower-host-ipmi-oem.git NO
> https://github.com/openbmc/openpower-hw-diags.git YES, COVERAGE
> https://github.com/openbmc/openpower-inventory-upload.git NO
> https://github.com/openbmc/openpower-libhei.git YES, COVERAGE
> https://github.com/openbmc/openpower-logging.git NO
> https://github.com/openbmc/openpower-occ-control.git YES, UNIT TEST
> https://github.com/openbmc/openpower-pnor-code-mgmt.git YES, UNIT TEST
> https://github.com/openbmc/openpower-power-control.git NO
> https://github.com/openbmc/openpower-proc-control.git YES, UNIT TEST
> https://github.com/openbmc/openpower-sbe-interface.git NO
> https://github.com/openbmc/openpower-vpd-parser.git YES, UNIT TEST
> https://github.com/openbmc/pam-ipmi.git NO
> https://github.com/openbmc/phosphor-bmc-code-mgmt.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-buttons.git NO
> https://github.com/openbmc/phosphor-certificate-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-dbus-interfaces.git NO
> https://github.com/openbmc/phosphor-dbus-monitor.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-debug-collector.git NO
> https://github.com/openbmc/phosphor-ecc.git NO
> https://github.com/openbmc/phosphor-event.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-fan-presence.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-gpio-monitor.git YES, COVERAGE
> https://github.com/openbmc/phosphor-host-ipmid.git ERROR
> https://github.com/openbmc/phosphor-host-postd.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-hostlogger.git NO
> https://github.com/openbmc/phosphor-hwmon.git NO
^---- This one should be YES, UNIT TEST.
Thanks for working on identifying missing coverage. I wanted to point
this out so you can take another swing at this, -- Thanks!
> https://github.com/openbmc/phosphor-inventory-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-ipmi-blobs.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-ipmi-blobs-binarystore.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-ipmi-ethstats.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-ipmi-flash.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-led-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-led-sysfs.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-logging.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-mboxd.git ERROR
> https://github.com/openbmc/phosphor-misc.git NO
> https://github.com/openbmc/phosphor-mrw-tools.git NO
> https://github.com/openbmc/phosphor-net-ipmid.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-networkd.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-nvme.git NO
> https://github.com/openbmc/phosphor-objmgr.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-pcie-presence.git NO
> https://github.com/openbmc/phosphor-pid-control.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-post-code-manager.git NO
> https://github.com/openbmc/phosphor-power.git YES, COVERAGE
> https://github.com/openbmc/phosphor-power-control.git NO
> https://github.com/openbmc/phosphor-power-monitor.git NO
> https://github.com/openbmc/phosphor-psu-code-mgmt.git YES, COVERAGE
> https://github.com/openbmc/phosphor-rest-server.git NO
> https://github.com/openbmc/phosphor-sel-logger.git NO
> https://github.com/openbmc/phosphor-settingsd.git NO
> https://github.com/openbmc/phosphor-snmp.git ERROR
> https://github.com/openbmc/phosphor-state-manager.git YES, COVERAGE
> https://github.com/openbmc/phosphor-time-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-u-boot-env-mgr.git NO
> https://github.com/openbmc/phosphor-user-manager.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-watchdog.git YES, COVERAGE
> https://github.com/openbmc/phosphor-webui.git NO
> https://github.com/openbmc/pldm.git YES, COVERAGE
> https://github.com/openbmc/pyphosphor.git NO
> https://github.com/openbmc/qualcomm-ipmi-oem.git NO
> https://github.com/openbmc/qualcomm-misc.git NO
> https://github.com/openbmc/quanta-ipmi-oem.git NO
> https://github.com/openbmc/rest-dbus.git NO
> https://github.com/openbmc/rrd-ipmi-blobs.git NO
> https://github.com/openbmc/s2600wf-misc.git NO
> https://github.com/openbmc/sdbusplus.git YES, UNIT TEST
> https://github.com/openbmc/sdeventplus.git YES, COVERAGE
> https://github.com/openbmc/skeleton.git ERROR
> https://github.com/openbmc/slpd-lite.git NO
> https://github.com/openbmc/smbios-mdr.git ERROR
> https://github.com/openbmc/stdplus.git YES, COVERAGE
> https://github.com/openbmc/telemetry.git NO
> https://github.com/openbmc/u-boot NO
> https://github.com/openbmc/webui-vue.git NO
> https://github.com/openbmc/witherspoon-pfault-analysis.git YES, COVERAGE
> https://github.com/openbmc/x86-power-control.git NO
> *************************UNIT TEST COVERAGE REPORT*************************
> SUMMARY:
> TOTAL REPOSITORIES : 110
> TESTED REPOSITORIES : 43
> ERROR : 11
> COVERAGE REPORT : 12
> UNIT TEST REPORT : 31
> NO REPORT : 56
> ***************************************************************************
>
>
> Following actions, we can draw from above
>
> There are 12 repo's that have GCOV/LCOV plugged in - We need to maintain which has good coverage and try to improve those which has less coverage
> There are 31 repo's that have unit test cases added but no coverage analysis because they aren't plugged in yet to GCOV/LCOV - Is there a plan for plugging in GCOV
> There are 11 repo's which fail to compile and errors out - Fix compilation issue. Some of these maybe not used currently?
> There are 56 repo's for which there is no UT tests available - This is a major issue we need to tackle and improve upon
>
> Feedback's , views highly appreciated
>
> Thanks & Regards,
> Lakshminarayana Kamath
> --------------------------------------------------------------------------------------------------
> Senior Engineer,
> IBM India Software Labs
> Bangalore
> github:lkammath
> ---------------------------------------------------------------------------------------------------
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 15:35 [OpenBMC]: Unit test coverage analysis on openbmc repositories Lakshminarayana R Kammath
` (2 preceding siblings ...)
2020-02-28 18:20 ` Patrick Venture
@ 2020-03-03 4:48 ` Andrew Jeffery
2020-03-03 6:16 ` Lakshminarayana R Kammath
2020-03-05 7:57 ` Lakshminarayana R Kammath
2020-03-04 18:43 ` Andrew Geissler
4 siblings, 2 replies; 16+ messages in thread
From: Andrew Jeffery @ 2020-03-03 4:48 UTC (permalink / raw)
To: Lakshminarayana R Kammath, openbmc
On Sat, 29 Feb 2020, at 02:05, Lakshminarayana R Kammath wrote:
> https://github.com/openbmc/obmc-console.git YES, UNIT TEST
I need to add coverage support to obmc-console.
> https://github.com/openbmc/openbmc-tools.git NO
Not sure we can expect anything for openbmc-tools - it's a collection
of random scripts people have written to make their life developing
openbmc easier. It's not something that should have strict QA
processes.
> https://github.com/openbmc/openpower-host-ipmi-flash.git NO
I should add some tests here where I can.
> https://github.com/openbmc/hiomapd.git YES, UNIT TEST
> https://github.com/openbmc/libmctp.git YES, UNIT TEST
> https://github.com/openbmc/phosphor-led-sysfs.git YES, UNIT TEST
These three do in-fact have GCOV/LCOV support, you need to do the usual autotools invocations to get it:
```
./configure --enable-code-coverage && make check-code-coverage
```
> https://github.com/openbmc/phosphor-mboxd.git ERROR
This is archived.
...
> https://github.com/openbmc/u-boot NO
This uses Kbuild as a build system. I notice openbmc/linux and openbmc/qemu aren't included in the list.
> https://github.com/openbmc/webui-vue.git NO
> https://github.com/openbmc/witherspoon-pfault-analysis.git YES, COVERAGE
> https://github.com/openbmc/x86-power-control.git NO
> *************************UNIT TEST COVERAGE
> REPORT*************************
> SUMMARY:
> TOTAL REPOSITORIES : 110
> TESTED REPOSITORIES : 43
> ERROR : 11
> COVERAGE REPORT : 12
> UNIT TEST REPORT : 31
> NO REPORT : 56
> ***************************************************************************
> Following actions, we can draw from above
>
> 1. *There are 12 repo's that have GCOV/LCOV plugged in* - We need to
> maintain which has good coverage and try to improve those which has
> less coverage
> 2. *There are 31 repo's that have unit test cases added but no
> coverage analysis because they aren't plugged in yet to GCOV/LCOV* - Is
> there a plan for plugging in GCOV
As above at least three of these are false-negatives for test coverage.
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-03-03 4:48 ` Andrew Jeffery
@ 2020-03-03 6:16 ` Lakshminarayana R Kammath
2020-03-05 7:57 ` Lakshminarayana R Kammath
1 sibling, 0 replies; 16+ messages in thread
From: Lakshminarayana R Kammath @ 2020-03-03 6:16 UTC (permalink / raw)
To: andrew; +Cc: openbmc
[-- Attachment #1: Type: text/html, Size: 5325 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 16:36 ` Brad Bishop
@ 2020-03-04 6:42 ` Lakshminarayana R Kammath
0 siblings, 0 replies; 16+ messages in thread
From: Lakshminarayana R Kammath @ 2020-03-04 6:42 UTC (permalink / raw)
To: bradleyb; +Cc: openbmc
[-- Attachment #1: Type: text/html, Size: 4411 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 15:35 [OpenBMC]: Unit test coverage analysis on openbmc repositories Lakshminarayana R Kammath
` (3 preceding siblings ...)
2020-03-03 4:48 ` Andrew Jeffery
@ 2020-03-04 18:43 ` Andrew Geissler
4 siblings, 0 replies; 16+ messages in thread
From: Andrew Geissler @ 2020-03-04 18:43 UTC (permalink / raw)
To: Lakshminarayana R Kammath; +Cc: openbmc
> On Feb 28, 2020, at 9:35 AM, Lakshminarayana R Kammath <lkammath@in.ibm.com> wrote:
>
> Hi All,
>
> Recently we worked on pulling the UT coverage analysis from all the available OpenBMC repositories
>
> https://github.com/openbmc/openbmc-build-scripts/commit/229b76a95f87af60c976a0c0dfe84716c9ce5318
>
> This script does following
> • Clone the repo
> • Use the CI build environment to build code
> • Publish the result in the following format
> Feedback's , views highly appreciated
Good stuff. I added this to a jenkins job which will run each day.
https://openpower.xyz/job/openbmc-unit-test-coverage/
You can then look at coverage info for your repo of interest by
clicking through the artifacts and getting to your coverage info.
For example:
https://openpower.xyz/job/openbmc-unit-test-coverage/lastSuccessfulBuild/artifact/openbmc-build-scripts/scripts/unit-test-meta-data/pldm/build/meson-logs/coveragereport/index.html
The job uses the following to know which repositories to check so feel
free to throw up some commits if there are missing or invalid repos:
https://github.com/openbmc/openbmc-build-scripts/blob/master/scripts/repositories.txt
>
> Thanks & Regards,
> Lakshminarayana Kamath
> --------------------------------------------------------------------------------------------------
> Senior Engineer,
> IBM India Software Labs
> Bangalore
> github:lkammath
> ---------------------------------------------------------------------------------------------------
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-02-28 18:20 ` Patrick Venture
@ 2020-03-05 1:25 ` James Feist
0 siblings, 0 replies; 16+ messages in thread
From: James Feist @ 2020-03-05 1:25 UTC (permalink / raw)
To: Patrick Venture, Lakshminarayana R Kammath; +Cc: OpenBMC Maillist
On 2/28/20 10:20 AM, Patrick Venture wrote:
> On Fri, Feb 28, 2020 at 7:36 AM Lakshminarayana R Kammath
> <lkammath@in.ibm.com> wrote:
>>
>> Hi All,
>>
>> Recently we worked on pulling the UT coverage analysis from all the available OpenBMC repositories
>>
>> https://github.com/openbmc/openbmc-build-scripts/commit/229b76a95f87af60c976a0c0dfe84716c9ce5318
>>
>> This script does following
>>
>> Clone the repo
>> Use the CI build environment to build code
>> Publish the result in the following format
>>
>> Latest report:
>> *************************UNIT TEST COVERAGE REPORT*************************
>> https://github.com/openbmc/bmcweb.git NO
>> https://github.com/openbmc/boost-dbus.git ERROR
>> https://github.com/openbmc/btbridge.git NO
>> https://github.com/openbmc/dbus-sensors.git NO
>> https://github.com/openbmc/entity-manager.git NO
This has some tests:
https://github.com/openbmc/entity-manager/blob/master/test/test_entity-manager.cpp
>> https://github.com/openbmc/fb-ipmi-oem.git NO
>> https://github.com/openbmc/google-ipmi-i2c.git ERROR
>> https://github.com/openbmc/google-ipmi-sys.git YES, UNIT TEST
>> https://github.com/openbmc/gpioplus.git YES, COVERAGE
>> https://github.com/openbmc/hiomapd.git YES, UNIT TEST
>> https://github.com/openbmc/ibm-dbus-interfaces.git NO
>> https://github.com/openbmc/ibm-logging.git YES, UNIT TEST
>> https://github.com/openbmc/ibm-pldm-oem.git ERROR
>> https://github.com/openbmc/inarp.git ERROR
>> https://github.com/openbmc/intel-dbus-interfaces.git NO
>> https://github.com/openbmc/intel-ipmi-oem.git NO
So does this:
https://github.com/openbmc/intel-ipmi-oem/blob/master/tests/test_sensorcommands.cpp
>> https://github.com/openbmc/ipmbbridge.git NO
>> https://github.com/openbmc/ipmi-blob-tool.git YES, UNIT TEST
>> https://github.com/openbmc/ipmi-fru-parser.git NO
>> https://github.com/openbmc/ipmitool.git ERROR
>> https://github.com/openbmc/jsnbd.git ERROR
>> https://github.com/openbmc/kcsbridge.git NO
>> https://github.com/openbmc/lenovo-ipmi-oem.git NO
>> https://github.com/openbmc/libmctp.git YES, UNIT TEST
>> https://github.com/openbmc/librrdplus.git NO
>> https://github.com/openbmc/obmc-console.git YES, UNIT TEST
>> https://github.com/openbmc/obmc-ikvm.git NO
>> https://github.com/openbmc/openbmc.git NO
>> https://github.com/openbmc/openbmc-tools.git NO
>> https://github.com/openbmc/openbmc.github.io.git NO
>> https://github.com/openbmc/openpower-dbus-interfaces.git NO
>> https://github.com/openbmc/openpower-debug-collector.git NO
>> https://github.com/openbmc/openpower-host-ipmi-flash.git NO
>> https://github.com/openbmc/openpower-host-ipmi-oem.git NO
>> https://github.com/openbmc/openpower-hw-diags.git YES, COVERAGE
>> https://github.com/openbmc/openpower-inventory-upload.git NO
>> https://github.com/openbmc/openpower-libhei.git YES, COVERAGE
>> https://github.com/openbmc/openpower-logging.git NO
>> https://github.com/openbmc/openpower-occ-control.git YES, UNIT TEST
>> https://github.com/openbmc/openpower-pnor-code-mgmt.git YES, UNIT TEST
>> https://github.com/openbmc/openpower-power-control.git NO
>> https://github.com/openbmc/openpower-proc-control.git YES, UNIT TEST
>> https://github.com/openbmc/openpower-sbe-interface.git NO
>> https://github.com/openbmc/openpower-vpd-parser.git YES, UNIT TEST
>> https://github.com/openbmc/pam-ipmi.git NO
>> https://github.com/openbmc/phosphor-bmc-code-mgmt.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-buttons.git NO
>> https://github.com/openbmc/phosphor-certificate-manager.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-dbus-interfaces.git NO
>> https://github.com/openbmc/phosphor-dbus-monitor.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-debug-collector.git NO
>> https://github.com/openbmc/phosphor-ecc.git NO
>> https://github.com/openbmc/phosphor-event.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-fan-presence.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-gpio-monitor.git YES, COVERAGE
>> https://github.com/openbmc/phosphor-host-ipmid.git ERROR
>> https://github.com/openbmc/phosphor-host-postd.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-hostlogger.git NO
>> https://github.com/openbmc/phosphor-hwmon.git NO
>
> ^---- This one should be YES, UNIT TEST.
>
> Thanks for working on identifying missing coverage. I wanted to point
> this out so you can take another swing at this, -- Thanks!
>
>> https://github.com/openbmc/phosphor-inventory-manager.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-ipmi-blobs.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-ipmi-blobs-binarystore.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-ipmi-ethstats.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-ipmi-flash.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-led-manager.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-led-sysfs.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-logging.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-mboxd.git ERROR
>> https://github.com/openbmc/phosphor-misc.git NO
>> https://github.com/openbmc/phosphor-mrw-tools.git NO
>> https://github.com/openbmc/phosphor-net-ipmid.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-networkd.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-nvme.git NO
>> https://github.com/openbmc/phosphor-objmgr.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-pcie-presence.git NO
>> https://github.com/openbmc/phosphor-pid-control.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-post-code-manager.git NO
>> https://github.com/openbmc/phosphor-power.git YES, COVERAGE
>> https://github.com/openbmc/phosphor-power-control.git NO
>> https://github.com/openbmc/phosphor-power-monitor.git NO
>> https://github.com/openbmc/phosphor-psu-code-mgmt.git YES, COVERAGE
>> https://github.com/openbmc/phosphor-rest-server.git NO
>> https://github.com/openbmc/phosphor-sel-logger.git NO
>> https://github.com/openbmc/phosphor-settingsd.git NO
>> https://github.com/openbmc/phosphor-snmp.git ERROR
>> https://github.com/openbmc/phosphor-state-manager.git YES, COVERAGE
>> https://github.com/openbmc/phosphor-time-manager.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-u-boot-env-mgr.git NO
>> https://github.com/openbmc/phosphor-user-manager.git YES, UNIT TEST
>> https://github.com/openbmc/phosphor-watchdog.git YES, COVERAGE
>> https://github.com/openbmc/phosphor-webui.git NO
>> https://github.com/openbmc/pldm.git YES, COVERAGE
>> https://github.com/openbmc/pyphosphor.git NO
>> https://github.com/openbmc/qualcomm-ipmi-oem.git NO
>> https://github.com/openbmc/qualcomm-misc.git NO
>> https://github.com/openbmc/quanta-ipmi-oem.git NO
>> https://github.com/openbmc/rest-dbus.git NO
>> https://github.com/openbmc/rrd-ipmi-blobs.git NO
>> https://github.com/openbmc/s2600wf-misc.git NO
>> https://github.com/openbmc/sdbusplus.git YES, UNIT TEST
>> https://github.com/openbmc/sdeventplus.git YES, COVERAGE
>> https://github.com/openbmc/skeleton.git ERROR
>> https://github.com/openbmc/slpd-lite.git NO
>> https://github.com/openbmc/smbios-mdr.git ERROR
>> https://github.com/openbmc/stdplus.git YES, COVERAGE
>> https://github.com/openbmc/telemetry.git NO
>> https://github.com/openbmc/u-boot NO
>> https://github.com/openbmc/webui-vue.git NO
>> https://github.com/openbmc/witherspoon-pfault-analysis.git YES, COVERAGE
>> https://github.com/openbmc/x86-power-control.git NO
>> *************************UNIT TEST COVERAGE REPORT*************************
>> SUMMARY:
>> TOTAL REPOSITORIES : 110
>> TESTED REPOSITORIES : 43
>> ERROR : 11
>> COVERAGE REPORT : 12
>> UNIT TEST REPORT : 31
>> NO REPORT : 56
>> ***************************************************************************
>>
>>
>> Following actions, we can draw from above
>>
>> There are 12 repo's that have GCOV/LCOV plugged in - We need to maintain which has good coverage and try to improve those which has less coverage
>> There are 31 repo's that have unit test cases added but no coverage analysis because they aren't plugged in yet to GCOV/LCOV - Is there a plan for plugging in GCOV
>> There are 11 repo's which fail to compile and errors out - Fix compilation issue. Some of these maybe not used currently?
>> There are 56 repo's for which there is no UT tests available - This is a major issue we need to tackle and improve upon
>>
>> Feedback's , views highly appreciated
>>
>> Thanks & Regards,
>> Lakshminarayana Kamath
>> --------------------------------------------------------------------------------------------------
>> Senior Engineer,
>> IBM India Software Labs
>> Bangalore
>> github:lkammath
>> ---------------------------------------------------------------------------------------------------
>>
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-03-03 4:48 ` Andrew Jeffery
2020-03-03 6:16 ` Lakshminarayana R Kammath
@ 2020-03-05 7:57 ` Lakshminarayana R Kammath
2020-03-06 9:27 ` Lei YU
1 sibling, 1 reply; 16+ messages in thread
From: Lakshminarayana R Kammath @ 2020-03-05 7:57 UTC (permalink / raw)
To: andrew; +Cc: openbmc
[-- Attachment #1: Type: text/html, Size: 8614 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
2020-03-05 7:57 ` Lakshminarayana R Kammath
@ 2020-03-06 9:27 ` Lei YU
0 siblings, 0 replies; 16+ messages in thread
From: Lei YU @ 2020-03-06 9:27 UTC (permalink / raw)
To: Lakshminarayana R Kammath; +Cc: Andrew Jeffery, OpenBMC Maillist
It's found that the test coverage result is not correct for some repos.
E.g. for phosphor-psu-code-mgmt, there are quite a few tests for code
in `src`, but it only shows 2.5%:
https://openpower.xyz/job/openbmc-unit-test-coverage/2/artifact/openbmc-build-scripts/scripts/unit-test-meta-data/phosphor-psu-code-mgmt/build/meson-logs/coveragereport/index.html
It looks like `lcov` is not handling such case well, and there is a
similar question on StackOverflow:
https://stackoverflow.com/questions/27556872/lcov-doesnt-match-gcov-coverage-results-for-a-googletest-unit-test
I tried to remove `lcov` and use `gcovr` in local CI, and the result
looks correct.
`lcov` is required for autotools' coverage, so we can not just remove it.
If I add `gcovr` into the docker image without removing `lcov`,
unfortunately, meson still uses `lcov` for `coverage-html` when both
tools are available.
An issue is created at
https://github.com/mesonbuild/meson/issues/6747, hopefully we could
get some feedback from meson.
In OpenBMC, do you have any ideas on how to "fix" the coverage result?
On Thu, Mar 5, 2020 at 3:59 PM Lakshminarayana R Kammath
<lkammath@in.ibm.com> wrote:
>
> Hi All,
>
> please confirm on following
>
> Boost-bus - compilation fails . Is it used? Should be archived @andrewj?
>
> Google-ipmi-i2c - code compiles well, complain in formatting script @pstrinkle
> subsys_initcall(i2c_via_ipmi_init);^[[m^M
> Traceback (most recent call last):^M
> File "/home/lkammath/final_data/unit-test.py", line 826, in <module>^M
> check_call_cmd("./format-code.sh", CODE_SCAN_DIR)^M
> File "/home/lkammath/final_data/unit-test.py", line 229, in check_call_cmd^M
> check_call(cmd)^M
> File "/usr/lib/python2.7/subprocess.py", line 190, in check_call^M
> raise CalledProcessError(retcode, cmd)^M
> subprocess.CalledProcessError: Command '('./format-code.sh', '/home/lkammath/final_data/google-ipmi-i2c')' returned non-zero exit status 1^M
> Traceback (most recent call last):^M
> File "/home/lkammath/final_data/dbus-unit-test.py", line 91, in <module>^M
>
> Inarp.git - Should be archived @andrewj?
> Compilation issue
> inarp.c: In function ‘arp_recv’:
> inarp.c:433:4: error: taking address of packed member of ‘struct arp_packet’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
> 433 | &inarp_req.src_ip);
> | ^~~~~~~~~~~~~~~~~
> In function ‘do_ifreq’,
> inlined from ‘get_local_ipaddr’ at inarp.c:182:7,
> inlined from ‘arp_recv’ at inarp.c:419:7,
> inlined from ‘main’ at inarp.c:480:4:
> inarp.c:170:2: error: ‘strncpy’ output may be truncated copying 16 bytes from a string of length 16 [-Werror=stringop-truncation]
> 170 | strncpy(ifreq->ifr_name, ifname, sizeof(ifreq->ifr_name));
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> cc1: all warnings being treated as errors
> make[1]: *** [inarp.o] Error 1
> make[1]: Leaving directory `/esw/san5/lkammath/boost/inarp'
> make: *** [all] Error 2
> In CI environment
> ool.la -lncurses -ltinfo -lcrypto^M
> copying selected object files to avoid basename conflicts...^M
> ar: `u' modifier ignored since `D' is the default (see `U')^M
> make[4]: Leaving directory '/home/lkammath/final_data/ipmitool/src/plugins/lan'^M
> Making all in lanplus^M
> make[4]: Entering directory '/home/lkammath/final_data/ipmitool/src/plugins/lanplus'^M
> /bin/bash ../../../libtool --silent --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../include -g -O2 -Wall -Wextra -std=c99 -pedantic -Wformat -Wformat-nonliteral -MT lanplus_crypt_impl.lo -MD -MP -MF .deps/lanplus_crypt_impl.Tpo -c -o lanplus_crypt_impl.lo lanplus_crypt_impl.c^M
> lanplus_crypt_impl.c: In function 'lanplus_encrypt_aes_cbc_128':^M
> lanplus_crypt_impl.c:158:17: error: storage size of 'ctx' isn't known^M
> 158 | EVP_CIPHER_CTX ctx;^M
> | ^~~^M
> lanplus_crypt_impl.c:158:17: warning: unused variable 'ctx' [-Wunused-variable]^M
> lanplus_crypt_impl.c: In function 'lanplus_decrypt_aes_cbc_128':^M
> lanplus_crypt_impl.c:233:17: error: storage size of 'ctx' isn't known^M
> 233 | EVP_CIPHER_CTX ctx;^M
> | ^~~^M
>
> Smbios-mdr - @brad. Please confirm if used?
> No automake env (make)
>
> Thanks & Regards,
> Lakshminarayana Kamath
> --------------------------------------------------------------------------------------------------
>
>
>
> ----- Original message -----
> From: "Andrew Jeffery" <andrew@aj.id.au>
> To: "Lakshminarayana R Kammath" <lkammath@in.ibm.com>, openbmc@lists.ozlabs.org
> Cc:
> Subject: [EXTERNAL] Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
> Date: Tue, Mar 3, 2020 10:18 AM
>
>
> On Sat, 29 Feb 2020, at 02:05, Lakshminarayana R Kammath wrote:
>
> > https://github.com/openbmc/obmc-console.git YES, UNIT TEST
>
> I need to add coverage support to obmc-console.
>
> > https://github.com/openbmc/openbmc-tools.git NO
>
> Not sure we can expect anything for openbmc-tools - it's a collection
> of random scripts people have written to make their life developing
> openbmc easier. It's not something that should have strict QA
> processes.
>
> > https://github.com/openbmc/openpower-host-ipmi-flash.git NO
>
> I should add some tests here where I can.
>
> > https://github.com/openbmc/hiomapd.git YES, UNIT TEST
> > https://github.com/openbmc/libmctp.git YES, UNIT TEST
> > https://github.com/openbmc/phosphor-led-sysfs.git YES, UNIT TEST
>
> These three do in-fact have GCOV/LCOV support, you need to do the usual autotools invocations to get it:
>
> ```
> ./configure --enable-code-coverage && make check-code-coverage
> ```
>
> > https://github.com/openbmc/phosphor-mboxd.git ERROR
>
> This is archived.
>
> ...
>
> > https://github.com/openbmc/u-boot NO
>
> This uses Kbuild as a build system. I notice openbmc/linux and openbmc/qemu aren't included in the list.
>
> > https://github.com/openbmc/webui-vue.git NO
> > https://github.com/openbmc/witherspoon-pfault-analysis.git YES, COVERAGE
> > https://github.com/openbmc/x86-power-control.git NO
> > *************************UNIT TEST COVERAGE
> > REPORT*************************
> > SUMMARY:
> > TOTAL REPOSITORIES : 110
> > TESTED REPOSITORIES : 43
> > ERROR : 11
> > COVERAGE REPORT : 12
> > UNIT TEST REPORT : 31
> > NO REPORT : 56
> > ***************************************************************************
> > Following actions, we can draw from above
> >
> > 1. *There are 12 repo's that have GCOV/LCOV plugged in* - We need to
> > maintain which has good coverage and try to improve those which has
> > less coverage
> > 2. *There are 31 repo's that have unit test cases added but no
> > coverage analysis because they aren't plugged in yet to GCOV/LCOV* - Is
> > there a plan for plugging in GCOV
>
> As above at least three of these are false-negatives for test coverage.
>
> Andrew
>
>
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* [OpenBMC]: Unit test coverage analysis on openbmc repositories
@ 2020-02-28 16:08 lakshminarayan kamath
0 siblings, 0 replies; 16+ messages in thread
From: lakshminarayan kamath @ 2020-02-28 16:08 UTC (permalink / raw)
To: openbmc
[-- Attachment #1: Type: text/plain, Size: 7920 bytes --]
Hi All,
Somehow my earlier mail did not get displayed properly
Recently we worked on pulling the UT coverage analysis from all the
available OpenBMC repositories
https://github.com/openbmc/openbmc-build-scripts/commit/229b76a95f87af60c976a0c0dfe84716c9ce5318
This script does following
1. Clone the repo
2. Use the CI build environment to build code
3. Publish the result in the following format
Latest report:
*************************UNIT TEST COVERAGE REPORT*************************
https://github.com/openbmc/bmcweb.git NO
https://github.com/openbmc/boost-dbus.git ERROR
https://github.com/openbmc/btbridge.git NO
https://github.com/openbmc/dbus-sensors.git NO
https://github.com/openbmc/entity-manager.git NO
https://github.com/openbmc/fb-ipmi-oem.git NO
https://github.com/openbmc/google-ipmi-i2c.git ERROR
https://github.com/openbmc/google-ipmi-sys.git YES, UNIT TEST
https://github.com/openbmc/gpioplus.git YES, COVERAGE
https://github.com/openbmc/hiomapd.git YES, UNIT TEST
https://github.com/openbmc/ibm-dbus-interfaces.git NO
https://github.com/openbmc/ibm-logging.git YES, UNIT TEST
https://github.com/openbmc/ibm-pldm-oem.git ERROR
https://github.com/openbmc/inarp.git ERROR
https://github.com/openbmc/intel-dbus-interfaces.git NO
https://github.com/openbmc/intel-ipmi-oem.git NO
https://github.com/openbmc/ipmbbridge.git NO
https://github.com/openbmc/ipmi-blob-tool.git YES, UNIT TEST
https://github.com/openbmc/ipmi-fru-parser.git NO
https://github.com/openbmc/ipmitool.git ERROR
https://github.com/openbmc/jsnbd.git ERROR
https://github.com/openbmc/kcsbridge.git NO
https://github.com/openbmc/lenovo-ipmi-oem.git NO
https://github.com/openbmc/libmctp.git YES, UNIT TEST
https://github.com/openbmc/librrdplus.git NO
https://github.com/openbmc/obmc-console.git YES, UNIT TEST
https://github.com/openbmc/obmc-ikvm.git NO
https://github.com/openbmc/openbmc.git NO
https://github.com/openbmc/openbmc-tools.git NO
https://github.com/openbmc/openbmc.github.io.git NO
https://github.com/openbmc/openpower-dbus-interfaces.git NO
https://github.com/openbmc/openpower-debug-collector.git NO
https://github.com/openbmc/openpower-host-ipmi-flash.git NO
https://github.com/openbmc/openpower-host-ipmi-oem.git NO
https://github.com/openbmc/openpower-hw-diags.git YES, COVERAGE
https://github.com/openbmc/openpower-inventory-upload.git NO
https://github.com/openbmc/openpower-libhei.git YES, COVERAGE
https://github.com/openbmc/openpower-logging.git NO
https://github.com/openbmc/openpower-occ-control.git YES, UNIT TEST
https://github.com/openbmc/openpower-pnor-code-mgmt.git YES, UNIT TEST
https://github.com/openbmc/openpower-power-control.git NO
https://github.com/openbmc/openpower-proc-control.git YES, UNIT TEST
https://github.com/openbmc/openpower-sbe-interface.git NO
https://github.com/openbmc/openpower-vpd-parser.git YES, UNIT TEST
https://github.com/openbmc/pam-ipmi.git NO
https://github.com/openbmc/phosphor-bmc-code-mgmt.git YES, UNIT TEST
https://github.com/openbmc/phosphor-buttons.git NO
https://github.com/openbmc/phosphor-certificate-manager.git YES, UNIT TEST
https://github.com/openbmc/phosphor-dbus-interfaces.git NO
https://github.com/openbmc/phosphor-dbus-monitor.git YES, UNIT TEST
https://github.com/openbmc/phosphor-debug-collector.git NO
https://github.com/openbmc/phosphor-ecc.git NO
https://github.com/openbmc/phosphor-event.git YES, UNIT TEST
https://github.com/openbmc/phosphor-fan-presence.git YES, UNIT TEST
https://github.com/openbmc/phosphor-gpio-monitor.git YES, COVERAGE
https://github.com/openbmc/phosphor-host-ipmid.git ERROR
https://github.com/openbmc/phosphor-host-postd.git YES, UNIT TEST
https://github.com/openbmc/phosphor-hostlogger.git NO
https://github.com/openbmc/phosphor-hwmon.git NO
https://github.com/openbmc/phosphor-inventory-manager.git YES, UNIT TEST
https://github.com/openbmc/phosphor-ipmi-blobs.git YES, UNIT TEST
https://github.com/openbmc/phosphor-ipmi-blobs-binarystore.git YES, UNIT
TEST
https://github.com/openbmc/phosphor-ipmi-ethstats.git YES, UNIT TEST
https://github.com/openbmc/phosphor-ipmi-flash.git YES, UNIT TEST
https://github.com/openbmc/phosphor-led-manager.git YES, UNIT TEST
https://github.com/openbmc/phosphor-led-sysfs.git YES, UNIT TEST
https://github.com/openbmc/phosphor-logging.git YES, UNIT TEST
https://github.com/openbmc/phosphor-mboxd.git ERROR
https://github.com/openbmc/phosphor-misc.git NO
https://github.com/openbmc/phosphor-mrw-tools.git NO
https://github.com/openbmc/phosphor-net-ipmid.git YES, UNIT TEST
https://github.com/openbmc/phosphor-networkd.git YES, UNIT TEST
https://github.com/openbmc/phosphor-nvme.git NO
https://github.com/openbmc/phosphor-objmgr.git YES, UNIT TEST
https://github.com/openbmc/phosphor-pcie-presence.git NO
https://github.com/openbmc/phosphor-pid-control.git YES, UNIT TEST
https://github.com/openbmc/phosphor-post-code-manager.git NO
https://github.com/openbmc/phosphor-power.git YES, COVERAGE
https://github.com/openbmc/phosphor-power-control.git NO
https://github.com/openbmc/phosphor-power-monitor.git NO
https://github.com/openbmc/phosphor-psu-code-mgmt.git YES, COVERAGE
https://github.com/openbmc/phosphor-rest-server.git NO
https://github.com/openbmc/phosphor-sel-logger.git NO
https://github.com/openbmc/phosphor-settingsd.git NO
https://github.com/openbmc/phosphor-snmp.git ERROR
https://github.com/openbmc/phosphor-state-manager.git YES, COVERAGE
https://github.com/openbmc/phosphor-time-manager.git YES, UNIT TEST
https://github.com/openbmc/phosphor-u-boot-env-mgr.git NO
https://github.com/openbmc/phosphor-user-manager.git YES, UNIT TEST
https://github.com/openbmc/phosphor-watchdog.git YES, COVERAGE
https://github.com/openbmc/phosphor-webui.git NO
https://github.com/openbmc/pldm.git YES, COVERAGE
https://github.com/openbmc/pyphosphor.git NO
https://github.com/openbmc/qualcomm-ipmi-oem.git NO
https://github.com/openbmc/qualcomm-misc.git NO
https://github.com/openbmc/quanta-ipmi-oem.git NO
https://github.com/openbmc/rest-dbus.git NO
https://github.com/openbmc/rrd-ipmi-blobs.git NO
https://github.com/openbmc/s2600wf-misc.git NO
https://github.com/openbmc/sdbusplus.git YES, UNIT TEST
https://github.com/openbmc/sdeventplus.git YES, COVERAGE
https://github.com/openbmc/skeleton.git ERROR
https://github.com/openbmc/slpd-lite.git NO
https://github.com/openbmc/smbios-mdr.git ERROR
https://github.com/openbmc/stdplus.git YES, COVERAGE
https://github.com/openbmc/telemetry.git NO
https://github.com/openbmc/u-boot NO
https://github.com/openbmc/webui-vue.git NO
https://github.com/openbmc/witherspoon-pfault-analysis.git YES, COVERAGE
https://github.com/openbmc/x86-power-control.git NO
*************************UNIT TEST COVERAGE REPORT*************************
SUMMARY:
TOTAL REPOSITORIES : 110
TESTED REPOSITORIES : 43
ERROR : 11
COVERAGE REPORT : 12
UNIT TEST REPORT : 31
NO REPORT : 56
***************************************************************************
Following actions, we can draw from above
1. *There are 12 repo's that have GCOV/LCOV plugged in* - We need to
maintain which has good coverage and try to improve those which has less
coverage
2. *There are 31 repo's that have unit test cases added but no coverage
analysis because they aren't plugged in yet to GCOV/LCOV* - Is there a
plan for plugging in GCOV
3. *There are 11 repo's which fail to compile and errors out* - Fix
compilation issue. Some of these maybe not used currently?
4. *There are 56 repo's for which there is no UT tests availabl*e -
This is a major issue we need to tackle and improve upon
Feedback's, views highly appreciated
Thanks & Regards,
Lakshminarayana Kamath
--------------------------------------------------------------------------------------------------
Senior Engineer,
IBM India Software Labs
Bangalore
github:lkammath
---------------------------------------------------------------------------------------------------
[-- Attachment #2: Type: text/html, Size: 22009 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2020-03-06 9:27 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-28 15:35 [OpenBMC]: Unit test coverage analysis on openbmc repositories Lakshminarayana R Kammath
2020-02-28 16:18 ` Brad Bishop
2020-02-28 16:30 ` Lakshminarayana R Kammath
2020-02-28 16:31 ` Brad Bishop
2020-02-28 16:36 ` Brad Bishop
2020-03-04 6:42 ` Lakshminarayana R Kammath
2020-02-28 16:39 ` Lakshminarayana R Kammath
2020-02-28 18:12 ` Nancy Yuen
2020-02-28 18:20 ` Patrick Venture
2020-03-05 1:25 ` James Feist
2020-03-03 4:48 ` Andrew Jeffery
2020-03-03 6:16 ` Lakshminarayana R Kammath
2020-03-05 7:57 ` Lakshminarayana R Kammath
2020-03-06 9:27 ` Lei YU
2020-03-04 18:43 ` Andrew Geissler
2020-02-28 16:08 lakshminarayan kamath
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.